how to download all the complete genomes for mycobacteria from NCBI?

Question

[Paul]

How to download all the complete genomes for mycobacteria from NCBI?

I tried downloading the complete genomes from the NCBI site

ftp://ftp.ncbi.nlm.nih.gov/genomes/GENOME_REPORTS/

But couldn't get the exact fasta files with respective mycobacteria. And https://www.ncbi.nlm.nih.gov/genome/?term=mycobacteria gave me 421 hits

Answer 1

[5heikki]

#Get GenBank assembly summary file
wget ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/assembly_summary_genbank.txt

#Get all lines that have "Mycobacter", if 12th field is "Complete Genome", print the 20th field (url to file).
#But the actual filename ends _genomic.fna.gz so include that too..
grep Mycobacter assembly_summary_genbank.txt \
    | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' \
    | awk 'BEGIN{OFS=FS="/"}{print $0,$NF"_genomic.fna.gz"}' \
    > urls.txt

#Now you can go through your urls file
IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done

[Paul]

Thanks.. This worked

[Debut]

I tried your method but I have an empty urls.txt file. has the format changed please?

[5heikki]

It hasn't changed. I just tried the above and see 2,481 Mycobacter genomes with the status "Complete Genome"..

[Debut]

OKAY, THANK YOU FOR YOUR ANSWER.

[5heikki]

$ grep klebsiella assembly_summary_genbank.txt | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' | wc -l
0
$ grep Klebsiella assembly_summary_genbank.txt | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' | wc -l
1523

[Debut]

please, is it possible to put all the output sequences in one file (file with several FASTA files) ?

[5heikki]

$ ls
file1.fna  file2.fna
$ cat file1.fna
>seq1
aaaaaaaaaa
$ cat file2.fna
>seq2
gggggg
$ cat file1.fna file2.fna > file3.fna
$ cat file3.fna
>seq1
aaaaaaaaaa
>seq2
gggggg

[Debut]

thank you very much for your answer. but i have 10668 outputs it doesn't have a command to add for example after "IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done" i tried IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done >doc.txt" it didn't work

[5heikki]

The output files all end in ".gz", right?

So zcat *.gz > all.fna

zcat instead of cat because they're gz archieves

[Debut]

Hi, I'm trying to do this with python, I've already loaded my table with pandas, and I'd like to do the same thing I've got the FTP Path back but I have to go from :""ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/316/945/GCA_001316945.3_ASM131694v3"""" to this : """ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/316/945/GCA_001316945.3_ASM131694v3/GCA_001316945.3_ASM131694v3_genomic.fna.gz""""" Thanks