How to keep my data and analysis up to date across new genome releases

Question

[Sheila]

What are the best methods to maintain up-to-date biological database?

I used a gnome from NCBI which was a large gz file. Now there is a newer release available in gz format. Every-time there is a change in database I have to download it completely and reanalysis the data. Same is the problem with some other databases. Is there a way to just only get the information which have been updated only?

Answer 1

[t.candelli]

It depends on which kind of analysis you do. if your outputs are simple data representation such as .bed or .bedgraph files the liftOver tool from UCSC can help keep everything up to date. it might require more work in generating the chain files if you work with unusual organisms. this method works for almost any form of feature-based annotation (or even .wig files), the only thing you need is to convert it into a .bed or .bedgraph.

A brief word on how liftOver works. the program is designed to convert a set of genomic coordinates (with respective values if present) between different assemblies of the same organism. in order to do this, liftOver requires a special "chain file", a file containing the differencese between the two target assemblies.

a number of pre-computed chainfiles for a number of different organisms, along with liftOver itself, is downloadable here.

[Sheila]

Thanks. I am interested in gz files or any flat file data. Is there any tool/method of just only downloading changes!

[Alex Reynolds]

No, it doesn't look like UCSC publishes diffs. It is probably easier for them to publish compressed files. You may need to download the entire file.

[t.candelli]

UCSC does publish a number of "diffs", or in this context chain files, between a number of different assemblies in different organisms here. i will update the answer to be more detailed.

Answer 2

[Elisabeth Gasteiger]

For UniProtKB, you could explore the advanced query mechanisms by date and use them programmatically (cf http://www.uniprot.org/faq/28 and http://www.uniprot.org/help/query-fields ), e.g.

lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+created:[current+TO+current]&format=txt" new_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+sequence_modified:[current+TO+current]&format=txt" upd_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+modified:[current+TO+current]&format=txt" upd_ann.dat