Entering edit mode
8.3 years ago
JoeDoasi
▴
10
Hi all,
After I dowloaded the MISO software and run the test command as requested in the manual, I ended up having this message:
===========================
Error: BAM /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam cannot be found.
02/06/2016 05:39:49 PM - miso_main - ERROR - BAM file /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam not found.
----------------------------------------------------------------------
FAIL: test_a_sam_to_bam (misopy.test_miso.TestMISO)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/misopy/test_miso.py", line 51, in test_a_sam_to_bam
"c2c12.Atp2b1.sorted.bam")))
AssertionError"
===========================
Please guide me on how to solve this issue. FYI, I have downloaded all prerequisite softwares that needed by MISO and Samtools can be run from terminal without any problems.
I appreciate your help
How did you run the test? What is the result if you run
which python?I ran this command: $ python -m unittest discover misopy
according to the manual : https://miso.readthedocs.io/en/fastmiso/#installing-the-fastmiso-version-of-miso
Python version is 2.7.6
The result is this error (part 1 - due to post restrictions to 5000 charcters):
Testing gene-level Psi... Executing: python /usr/local/lib/python2.7/dist-packages/misopy/index_gff.py --index /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/indexed Indexing GFF... /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff appears to already be indexed. Aborting. Executing: python /usr/local/lib/python2.7/dist-packages/misopy/run_events_analysis.py --compute-genes-psi /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/indexed /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output --read-len 36 --paired-end 250 30 --use-cluster MISO (Mixture of Isoforms model) To run MISO, please use "miso" instead. .Testing single-end SE event interface... Executing: python /usr/local/lib/python2.7/dist-packages/misopy/run_events_analysis.py --compute-events-psi se-sample /usr/local/lib/python2.7/dist-packages/misopy/test-data/se-counts/se_test.counts --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/SE-output --read-len 35 --overhang-len 4 --event-type SE --use-cluster MISO (Mixture of Isoforms model) To run MISO, please use "miso" instead. .Testing fr-unstranded... Checking read f_read against + Checking read f_read against - Checking read r_read against + Checking read r_read against - Testing fr-firststrand... .Testing conversion of SAM to BAM... Executing: sam_to_bam --convert /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output Converting SAM to BAM... - Executing: samtools view -Sbh /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam > /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam Sorting BAM file... - Executing: samtools sort /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted [bam_sort] Use -T PREFIX / -o FILE to specify temporary and final output files Usage: samtools sort [options...] [in.bam] Options: -l INT Set compression level, from 0 (uncompressed) to 9 (best) -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] -n Sort by read name -o FILE Write final output to FILE rather than standard output -T PREFIX Write temporary files to PREFIX.nnnn.bam -@, --threads INT Set number of sorting and compression threads [1] --input-fmt-option OPT[=VAL] Specify a single input file format option in the form of OPTION or OPTION=VALUE -O, --output-fmt FORMAT[,OPT[=VAL]]... Specify output format (SAM, BAM, CRAM) --output-fmt-option OPT[=VAL] Specify a single output file format option in the form of OPTION or OPTION=VALUE --reference FILE Reference sequence FASTA FILE [null] Indexing BAM... - Executing: samtools index /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam [E::hts_open_format] fail to open file '/usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam' samtools index: failed to open "/usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam": No such file or directory Conversion took 0.00 minutes. FTesting gene-level Psi... Testing GFF indexing of: /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff Executing: index_gff --index /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed Indexing GFF... /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff appears to already be indexed. Aborting. Executing: miso --run /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output --read-len 36 MISO (Mixture of Isoforms model) Probabilistic analysis of RNA-Seq data for detecting differential isoforms Use --help argument to view options.
part 2:
Using MISO settings file: /usr/local/lib/python2.7/dist-packages/misopy/settings/miso_settings.txt Computing Psi values... - GFF index: /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed - BAM: /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam - Read length: 36 - Output directory: /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output Checking your GFF annotation and BAM for mismatches... Error: BAM /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam cannot be found. 08/20/2016 05:33:07 AM - miso_main - ERROR - BAM file /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam not found.
.
FAIL: test_a_sam_to_bam (misopy.test_miso.TestMISO)
Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/misopy/test_miso.py", line 51, in test_a_sam_to_bam "c2c12.Atp2b1.sorted.bam"))) AssertionError
Ran 5 tests in 1.192s
FAILED (failures=1)
I found this: https://github.com/yarden/MISO/blob/fastmiso/misopy/sam_to_bam.py The samtools sort function in is incorrect and that call fails. Perhaps in an older version it did work, I wouldn't know. Try to adjust the script on line 28 to have correct samtools synthax Samtools even prints it in your traceback:
So instead of having
cmd = "samtools sort %s %s" %(bam_filename, sorted_filename)you should havecmd = "samtools sort %s -o %s" %(bam_filename, sorted_filename)Thanks a lot for your responses. I changed the command by adding -o and still the get the same result. I also tried older version of samtools 0.1.19 as @genomax2 suggested and still same problem..
I don't know if this page have something meaningful: https://github.com/nservant/HiC-Pro/issues/23
could you please look at it and see if it makes sense !
Is the error message exactly the same? And you did add the
-oin front of the second%s? Remarkable.Yes I added -o in front of the second %s . The same error message.
My bad, you also need to change the line above:
sorted_filename = "%s.sorted" %(bam_filename.split(".bam")[0])should becomesorted_filename = "%s.sorted.bam" %(bam_filename.split(".bam")[0])Nothing you have to worry about, but the script should use pysam instead of os.system() calls to samtools o.0 or at least subprocess.call() with proper error handling. It now executes shell commands without checking whether those were successful...
Thanks for your help..
It failed again even after editing the two lines!, so I took a screenshot for the screen for more convenience..please check the attached photo
I'm new to programming and I worked on perl not python so imagine how I'm lost !!!
Debugging from a distance isn't easy but I'm confident we'll get there.
Actually, the error is not completely the same, and that's a good thing. You'll notice the samtools error about usage is no longer there. It's just a wrong output file that is created, if I'm not mistaken :p Could you edit the code once more to print out
print(sorted_filename)before thecmd = "samtools sort %s %s" %(bam_filename, sorted_filename)? Then we'll find out which file exactly is created.Ehm edit: I notice multiple lines in which
samtools not foundis stated. You have samtools installed and in $PATH right?Ah, got another error a bit lower. You can change line 34
final_filename = "%s.bam" %(sorted_filename)tofinal_filename = sorted_filenameI took a screenshot for the terminal when I run samtools and also when I ran the modules check command as requested in the installation manual..
what really kills is that modules check states that samtools is installed !!!!!!!
also please check the test script.
please see the outputs of both commands, $ samtools & $ module_availability
test_miso.py
This is getting rather odd. Where did you install samtools? Where do you test? Is samtools in your $PATH?
I really appreciate your help and time ...
samtools is downloaded on the desktop
Kindly look at the attached photo ($PATH and /.bashrc) screenshot for the $PATH
Good, could you tell me the output of
which samtools? What I find most odd is that finding samtools wasn't the problem earlier, but now it is. Makes me wonder whether starting over from the beginning would be worth it.ya73@Bioinftop2016-06 ~ $ which samtools /home/ya73/Desktop/tuxedo/samtools-1.3.1//samtools
I gave up !!!! more than a week trying to troubleshoot, yet the people made it do not respond !!!
I deleted it and I'm gonna reinstall it.. awaiting for your advice.
Alright, run the test again and let's try solve the error message again. I had the feeling we almost did fix it :p If this doesn't work out quickly I might rewrite the damn script myself.
Unfortunately didn't work..
I was doubting that sorting the bam file is the problem so I copied the original sam file (in the test directory: c2c12.Atp2b1.sam) to the desktop and converted it to bam (it worked using samtools conversion command). then I run the sorting command :
$ samtools sort c2c12.Atp2b1.bam c2c12.Atp2b1.sorted
guess what happened ?!
samtools 1.3.1 had an error samtools 0.1.19 worked and sorted the bam file !
I went again to run the test using samtools-0.1.19 and it failed !!!!!!!!!!!!!
please find here a snapshot for the terminal using version 0.1.19!
Thank you
Samtools sort command has changed between those two versions. For v. 1.3.1 you need to use
samtools sort -o c2c12.Atp2b1.sorted.bam c2c12.Atp2b1.bamthanks. it worked.
Could it be the pysam failed to call samtools ?
pysam is downloaded on : /usr/local/lib/python2.7/dist-packages
miso is downloaded on : /usr/local/bin
please look at this part of the process which is realated to samtools and pysam:
Testing conversion of SAM to BAM... Executing: sam_to_bam --convert /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output Converting SAM to BAM... - Executing: samtools view -Sbh /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam > /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam sh: 1: samtools: not found Sorting BAM file... /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam - Executing: samtools sort /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam -o /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam sh: 1: samtools: not found Indexing BAM... - Executing: samtools index /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam sh: 1: samtools: not found
From what I saw MISO doesn't use pysam, it just does lazy calls to
os.system(samtools)Okay, could you try editing the script to replace samtools everywhere with the absolute path to the samtools executable?
Finally - Finalmente
I seems like it worked..
BIG THANKS to you @WouterDeCoster and also to @genomax2
take a look at snapshots 1 and 2
Believe me, I'm as happy as you that we finally got through. Now go do some awesome research!
one week to start my graduation project ... that was so stressful .. especially that I'm not a python dude !
Thanks again
You can either try @Wouter's change or install an old version of samtools.
Can you verify that this file is present on your system
ls -lh /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam?I got this message :
ls: cannot access /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam: No such file or directory
I don't know why it doesn't create this file, c2c12.Atp2b1.sorted.bam when I run the test command !!!!