Hi, I am using ubuntu 16.04 and tried using a command suggested in one of the posts () to add tophat and bowtie to my path but I think I wasn't successful.
echo $PATH shows neither of them in the path. I also tried running tophat from a directory , it also shows that tophat can't find the b2 index files. What am I doing wrong?
Well, thanks for the language. You might be an expert in linux but I am really new to it and am trying to learn so that I can run these tools for my research work.
Anyways, I did realize my mistake that I was checking in the same terminal.
Considering, you are an expert and I am stupid, may be you can also shed some light on this, when I am trying to run tophat from a dir containing the files to be mapped, it returns the error,
Error: Could not find Bowtie 2 index files (I already built the b2 index, contained in the b2 index dir)
Chill :D the stupid thing was my question, but every question is worth asking!
If you export a variable from terminal 1, and then try to call it from terminal 2, you might not retrieve it (they load environmental variables when you open them).
If you want to have the command in your path, best thing is to edit your .bash_profile (or .profile, or .bashrc, depending on where your $PATH variable is declared) adding:
For the bowtie2 indexes, you have to specify the --basename. This means that if your indexes are all called like whatever.bt2 you have to specify -x whatever in the bowtie2 command. ;)
I think $home should really be /home. export PATH=$PATH:/home/gjjha/Downloads/softwares/bowtie2-2.2.9/index:/home/gjjha/Downloads/softwares/tophat-2.1.1 Having said that, it would be better not to put random folders in the PATH, as Michael mentioned.
Hi everyone.
@Macspider sorry, I misunderstood, my bad. Couldn't clarify yesterday since I am a new user and can post only 5 times.
I am writing in details so that I don’t miss any point this time. I am using ubuntu 16.04 on my workstn (256gb RAM). I have downloaded and installed bowtie2.2.9 and tophat2 according to the instruction given on tophat page of ccb.jhu.edu. I am successful in building indices and running tophat but only from the bowtie2 index directory, where the b2 index files are. This limits my usage coz I need to copy my raw read files in the index directory every time and can run one set of reads at a time only.
So, I tried adding the location of these tools and the index dir to the path thinking it will enable me to run multiple files parallelly, from any other directory.
Now I understand that these tools are installed in /usr/bin as “which” returns /usr/bin
$ which bowtie2 returns /usr/bin/bowtie2
$ which tophat2 returns /usr/bin/tophat2
But yesterday, using answers in some post, I used this command for adding the location of index directory to my PATH
This limits my usage coz I need to copy my raw read files in the index directory every time and can run one set of reads at a time only
So this means that if you use the full path of the read files you don't get them?
If you have these files:
R1 = /path/to/reads_1.fq
R2 = /path/to/reads_2.fq
INDEX = /path/to/index/basename (don't consider the *.ht2 extension)
You should be able to run tophat2 from everywhere specifying full paths. Like:
Moreover, if you need to run many sets, you can either run many tophat runs changing the input files or using their special list comprehension in the input, separating by a comma the files you want to input (comma, without space afterwards), like they write in the program's help:
@Macspider
One question, when we run tophat for a file, the results are saved in the tophat_out directory. So, when we are using the command
tophat [options] <bowtie_index> <reads1[,reads2,...]> [reads1[,reads2,...]]<="" p="">
to run multiple PE sets, do we need to specify an output dir for each pair too or it will create separate directories itself?
Yes you do need to specify a unique new output directory for each sample. Every sample needs to be run independently. This is important since output from tophat has files with exactly the same name(s) for every sample.
There is a difference between running 2 tophat runs on two sets, and 1 run on two sets. In detail:
If you run separately the two runs, you will specify an output folder in each, and it's obviously the right thing to do to name them differently cause files inside will have the same name and will be therefore overwriting theirselves (a mess).
If you run them together, you will get a single BAM file that has all the results merged. This might be what you want, might be not. It's up to you! For example, if you have 3 replicates it is very unlikely that you want this, while if you are doing a gene prediction you do want this because you pile up evidence, and don't aim for expression levels.
@Macspider
I hv paired reads for replicates and different experimental conditions. I definitely want separate results not merged. Now how can I run them in parallel, to save time & efforts and how can I define separate output dir for each pair in one command?
Tophat has a -p option that sets how many threads you want to use. This has an upper limit in how many cores your machine / cluster has available, or in how many cores you are allowed to use if you are using some sort of queue manager.
That said, if you want to do parallel runs there is no built-in pipeline to do that, you have to do it yourself, but it's just as easy as launch N times the same command changing the files.
Since you don't want to overheat your system or to force more threads than the tolerated number in your cores, you'd probably be safe if you use for each run a number of threads (-p option) that is the result of:
Number of cores you can use (f.e. 30) / Number of runs (let's say 6) = 5 threads (in this example)
The power tool for running things in parallel is gnu-parallel, which can also restrict memory usage, specify the number of jobs, avoid swapping, show progress...
Or alternatively more complicated pipeline tools such as snakemake
You were lucky that this didn't mess up your data! The positional arguments at the end of the tophat command are considered as the input data, and the ouptut directory is one optional argument (--output-dir). If the output was as well a positional argument, you could have attempted to overwrite your input data with a null output file just by placing it in the wrong slot of the command line.
Bottom line: always be sure of what the manuals and helps say about arguments, if they're to be placed here or there, because this can really mess up things sometimes :)
Edit: note to put the local directory first, so the newer binary overrides the default path
shell variables are case sensitive.
Btw: don't put random download directories in your path, 'install' binaries in a 'proper' location (using unix tools such as install, stow, or even cp if you must), e.g. /usr/local/bin or $HOME/bin, then add export PATH=$HOME/bin:$PATH to your .bashrc or .bashprofile
thanks, will try and let you know. If you have answer for my above que regarding the output dir for tophat too (my comment on Macspider answer), I'll be grateful.
@Michael Dondrop
Hi,
I followed your suggestion and created the $HOME/bin dir as you have suggested above. Other than tophat, bowtie, I also copied blast+2.5 to the dir and added the path to my .bashrc using
but both return the following error,
BLAST Database error: No alias or index file found for nucleotide database [home/mirza/Downloads/softwares/blastdb/my_db] in search path [/home/mirza:$/home/mirza/Downloads/softwares/blastdb:]
I tried this path too,
blastn -task blastn -query test.fasta -db home/mirza/Downloads/softwares/blastdb/my_db -outfmt 6
I am not sure we can or should solve your path problems here. I can just recommend the follwoing:
check carefully your .profile, .bashrc, .bash_profile for additional path entries and remove :
:/home/mirza/mirza/Downloads/softwares/bowtie2-2.2.9/index:/home/mirza/Downloads/softwares/tophat-2.1.1: wherever you find that.
I recommended PATH=$HOME/bin:$PATH because you want your own software
@Macspider
I did let the run complete, just checked if its creating the output dir and terminated the process, when I use -o flag after the PEreads file names, so my output didn't get messed up.
I really want to thank everyone here for their help. I worked out the running parallel mapping sessions by using your help,
I have solved this problem with a very easy solution.
Problem occurs when you set $PATH to a directory it clears the previous paths from the system which are essential for to run the software sometimes. So,
echo PATH
copy all the paths comes in the results
the use
export PATH
command and add the path of the software you are using : at the end of previous paths
If you cant understand this please see on google how to add multiple paths in linux/ubuntu. Also be-careful about terminal. You should use the same terminal in which you have added the path.
@Shahzad Thank you for your reply. Can you please send me the link to the article you are referring to coz sometimes there are so many similar hits on Google and it all gets confusing. I am new but really interested in learning.
You should also be able to install bowtie2 (2.2.6-2) and tophat (2.1.0) from the Ubuntu Software Center.
Very stupid check question:
are you doing
echo $PATHfrom the same terminal, or from another one?Well, thanks for the language. You might be an expert in linux but I am really new to it and am trying to learn so that I can run these tools for my research work. Anyways, I did realize my mistake that I was checking in the same terminal.
Considering, you are an expert and I am stupid, may be you can also shed some light on this, when I am trying to run tophat from a dir containing the files to be mapped, it returns the error, Error: Could not find Bowtie 2 index files (I already built the b2 index, contained in the b2 index dir)
Macspider wasn't calling you stupid, he/she was calling his/her own question "stupid".
Anyway, what are you using for the
<bowtie_index>argument and where are the indices?Chill :D the stupid thing was my question, but every question is worth asking!
If you export a variable from terminal 1, and then try to call it from terminal 2, you might not retrieve it (they load environmental variables when you open them).
If you want to have the command in your path, best thing is to edit your .bash_profile (or .profile, or .bashrc, depending on where your $PATH variable is declared) adding:
at the end of it.
For the bowtie2 indexes, you have to specify the --basename. This means that if your indexes are all called like whatever.bt2 you have to specify
-x whateverin the bowtie2 command. ;)I think $home should really be /home. export PATH=$PATH:/home/gjjha/Downloads/softwares/bowtie2-2.2.9/index:/home/gjjha/Downloads/softwares/tophat-2.1.1 Having said that, it would be better not to put random folders in the PATH, as Michael mentioned.
Hi everyone. @Macspider sorry, I misunderstood, my bad. Couldn't clarify yesterday since I am a new user and can post only 5 times. I am writing in details so that I don’t miss any point this time. I am using ubuntu 16.04 on my workstn (256gb RAM). I have downloaded and installed bowtie2.2.9 and tophat2 according to the instruction given on tophat page of ccb.jhu.edu. I am successful in building indices and running tophat but only from the bowtie2 index directory, where the b2 index files are. This limits my usage coz I need to copy my raw read files in the index directory every time and can run one set of reads at a time only. So, I tried adding the location of these tools and the index dir to the path thinking it will enable me to run multiple files parallelly, from any other directory. Now I understand that these tools are installed in /usr/bin as “which” returns /usr/bin
$ which bowtie2 returns /usr/bin/bowtie2
$ which tophat2 returns /usr/bin/tophat2
But yesterday, using answers in some post, I used this command for adding the location of index directory to my PATH
$ export PATH=$PATH:$home/mirza/Downloads/softwares/bowtie2-2.2.9/index:/home/mirza/Downloads/softwares/tophat-2.1.1
echo $PATH returns
:/gjjha/Downloads/softwares/bowtie2-2.2.9/index:/home/gjjha/Downloads/softwares/tophat-2.1.1:/gjjha/Downloads/softwares/bowtie2-2.2.9/index:/home/gjjha/gjjha/Downloads/softwares/bowtie2-2.2.9/index:/home/gjjha/Downloads/softwares/tophat-2.1.1
I am still unable to run tophat2 from any directory other than the index (containing the indices) dir within bowtie2 dir. I need help with,
How to add these tools to my path?
How to undo the above command I used to add the index dir & tools to my path?
So this means that if you use the full path of the read files you don't get them? If you have these files: R1 = /path/to/reads_1.fq R2 = /path/to/reads_2.fq INDEX = /path/to/index/basename (don't consider the *.ht2 extension)
You should be able to run tophat2 from everywhere specifying full paths. Like:
No need to copy files!
Moreover, if you need to run many sets, you can either run many tophat runs changing the input files or using their special list comprehension in the input, separating by a comma the files you want to input (comma, without space afterwards), like they write in the program's help:
@Macspider One question, when we run tophat for a file, the results are saved in the tophat_out directory. So, when we are using the command tophat [options] <bowtie_index> <reads1[,reads2,...]> [reads1[,reads2,...]]<="" p="">
to run multiple PE sets, do we need to specify an output dir for each pair too or it will create separate directories itself?
Yes you do need to specify a unique new output directory for each sample. Every sample needs to be run independently. This is important since output from tophat has files with exactly the same name(s) for every sample.
There is a difference between running 2 tophat runs on two sets, and 1 run on two sets. In detail:
@Macspider I hv paired reads for replicates and different experimental conditions. I definitely want separate results not merged. Now how can I run them in parallel, to save time & efforts and how can I define separate output dir for each pair in one command?
Tophat has a -p option that sets how many threads you want to use. This has an upper limit in how many cores your machine / cluster has available, or in how many cores you are allowed to use if you are using some sort of queue manager.
That said, if you want to do parallel runs there is no built-in pipeline to do that, you have to do it yourself, but it's just as easy as launch N times the same command changing the files.
Since you don't want to overheat your system or to force more threads than the tolerated number in your cores, you'd probably be safe if you use for each run a number of threads (-p option) that is the result of:
Number of cores you can use (f.e. 30) / Number of runs (let's say 6) = 5 threads (in this example)
The power tool for running things in parallel is gnu-parallel, which can also restrict memory usage, specify the number of jobs, avoid swapping, show progress...
Or alternatively more complicated pipeline tools such as snakemake
Did you read my answer?
@Macspider thanks I'll try these options and let you know tomorrow. @Michael Dondrup I have answered your reply below.
@Macspider
@Devon Ryan
Hi, can you please tell me how to use the -output-dir option in tophat I have tried all these
-o my_out;
--output-dir my_out;
-o ./my_out
to creat my tophat_out directory but still no success. It creates its default dir tophat_out everytime.
I want to thank you all, learning a lot through biostar.
You should just specify a word and include an eventual path, it will create the folder. Can you paste here the command?
Hi, I am simply using
tophat b2.index input_file1 input_file2
where 1 & 2 are the left and right reads and its working fine. The output is generated in the default tophat_out directory.
But how did you use the -o flag?
as I have written in my earlier reply above, I tried using it in 3 different ways but still got the default dir not mine
tophat b2index input1 input2 -o my_out also,
tophat b2index input1 input2 --output-dir my_out and
tophat b2index input1 input2 -o ./my_out
Sometimes order matters, try
tophat -o my_out b2index input1 input2.ok, may be that's the problem. I can try this tomorrow only. Will post the outcome here. Thanks.
Indeed that's likely the problem. From the manual:
-o is an option and should come before the index. That works for me.
Just a side note to this:
You were lucky that this didn't mess up your data! The positional arguments at the end of the tophat command are considered as the input data, and the ouptut directory is one optional argument (--output-dir). If the output was as well a positional argument, you could have attempted to overwrite your input data with a null output file just by placing it in the wrong slot of the command line.
Bottom line: always be sure of what the manuals and helps say about arguments, if they're to be placed here or there, because this can really mess up things sometimes :)