Galaxy: Error Uploading Directory Of Files

Question

[nk]

Hi,

Our lab recently installed a local version of Galaxy on a mid-2012 Mac Pro computer. We can access the Galaxy server and sign in as an administrator. Today we tried creating a Data Library, adding a dataset to it, and uploading a directory of files. We followed this Galaxy documentation to setup this feature:

• Admin > Data Library > Add datasets > Upload directory of files
• file format was set to auto-detect
• and we chose the option to link to files instead of copying them

Galaxy confirmed that the files were successfully uploaded. However, in the data library, under the Message column, is a message in red saying "Job error (click name for more info)". Clicking on one of the uploaded files displays a page with this information:

Date uploaded: 2013-06-28

File size: 7.5 GB

Data type: auto

Build: sacCer2

Miscellaneous information:
Traceback (most recent call last): File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 386, in __main__() File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 357, in __main__ output_paths =

Job Standard Error
Traceback (most recent call last):
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 386, in
__main__()
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 357, in __main__
output_paths = parse_outputs( sys.argv[4:] )
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 64, in parse_outputs
id, files_path, path = arg.split( ':', 2 )
ValueError: need more than 1 value to unpack
error

Database/Build: sacCer2
Number of data lines: None
Disk file: /Volumes/G-SPEED Q/data/Person 2012 project/DCP2 mef.fastq

Questions:

1. Should we be concerned about this error?
2. If so, what is the right way to fix it?
3. If not, how do we remove the red error message next to each file: "Job error (click name for more info)" ?

[Björn]

Which Galaxy version did you use? Can you give us the corresponding log snippet from paster.log? The line in which tools/data_source/upload.py is called, is especially interesting. Did you have spaces in your file path (/Volumes/G-SPEED Q/data/Person 2012 project/DCP2 mef.fastq)? Can you reproduce it with files without spaces in there path?

[Peter]

Hi Bjoern - thanks for the link via http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-July/015390.html noticing I suggested the same thing http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-July/015385.html when this was asked on the mailing list a few days later http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-July/015384.html

[Peter]

I'm closing this Question as it turning into a debugging discussion better suited to the mailing list thread Nicholas Kline already started on the Galaxy-dev mailing list, http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-July/015384.html

Answer 1

[nk]

Thank you for the quick replies. Here is what I tried today...

I removed all spaces from paths and filenames and tried uploading the contents of the directory again. The upload failed but with a different error message:

Date uploaded: 2013-07-08
File size: 9.1 GB
Data type: auto
Build: sacCer2

Miscellaneous information:
Traceback (most recent call last): File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 386, in __main__() 
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 375, in __main__ add_file( data Job Standard Error

Traceback (most recent call last):
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 386, in 
    __main__()
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 375, in __main__
    add_file( dataset, registry, json_file, output_path )
File "/Users/administrator/galaxy-dist/tools/data_source/upload.py", line 284, in add_file
    if datatype.dataset_content_needs_grooming( dataset.path ):
File "/Users/administrator/galaxy-dist/lib/galaxy/datatypes/binary.py", line 130, in dataset_content_needs_grooming
    version = self._get_samtools_version()
File "/Users/administrator/galaxy-dist/lib/galaxy/datatypes/binary.py", line 114, in _get_samtools_version
    output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1]   
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child

OSError: [Errno 2] No such file or directory error

Database/Build: sacCer2 
Number of data lines: None 
Disk file: /Volumes/G-SPEED-Q/data/Person2012project/WT.fastq

So, I tried something different. I created a directory named test and tried several upload scenarios:

works (1 fastq file):

• /Volumes/G-SPEED-Q/data/test/WT.fastq

fails (1 fastq, 1 bam with misc chars in filename):

• /Volumes/G-SPEED-Q/data/test/WT.fastq
• /Volumes/G-SPEED-Q/data/test/Galaxy61-[WT_minus_strand_-ribo_SAM-to-BAM_on_data_55].bam

fails (1 fastq, 1 bam with underscores):

• /Volumes/G-SPEED-Q/data/test/WT.fastq
• /Volumes/G-SPEED-Q/data/test/Galaxy61_WT_minus_strand_ribo_SAM_to_BAM_on_data_55.bam

fails (1 fastq, 1 bam no misc chars):

• /Volumes/G-SPEED-Q/data/test/WT.fastq
• /Volumes/G-SPEED-Q/data/test/test.bam

works (2 fastq files):

• /Volumes/G-SPEED-Q/data/test/WT.fastq
• /Volumes/G-SPEED-Q/data/test/WTmef.fastq

Questions:

1. It appears Galaxy does not like to upload bam files ?
2. How do I check our Galaxy version ?
3. Where is paster.log located ?

[Björn]

Hi,

I can be wrong but I think you should edit your original question. That is supposed to be the answer :-). Nevertheless, here a few hints and ideas:

1. Yes, you can upload bam files
2. 'hg tip' will show you the latest revision
3. paster.log is in your galaxy root directory if you start galaxy with sh ./run.sh --daemon

You can watch the output with tail -f paster.log.

I'm not able to reproduce the crash. I tried with 1.bam and 1.fastq from the test-data directory. Is your bam file a link? The error occurs during samtools index calculation step. For some reason your file is not found.

[Peter]

The error is "No such file or directory error" when trying to run samtools. So, is samtools installed and on your path? As I recall, Galaxy will automatically call it to sort and index uploaded BAM files - using samtools.

The paster.log file is by default in the main Galaxy folder.

Assuming you installed Galaxy via mercurial (hg), then use hg to get the last commit hash - that is what the Galaxy team use instead of traditional version numbers.