How Do I Import Data From A Torrent Into A Bioperl, R, Bioclipse, Or Taverna Application?

Question

[Egon Willighagen]

While BioTorrent is still rather unpopulated, I quite like the idea of having such torrents seeded by university IT or library departments. Now, I want to analyze the data, so the question is, how can the data be used directly in my workflow system (BioPerl or R scripts, Taverna or Bioclipse workflows)?

Are there libraries in a suitable programming language to download torrents automatically, without human interaction?

[Giovanni M Dall'Olio]

Very nice idea, you convinced me completely. I am trying to convince the 1000genomes maintainers to release their data as torrents.

[Egon Willighagen]

Cool, that is much appreciated!

Answer 1

[Giovanni M Dall'Olio]

I don't think there is an R library for torrents. I have tried searching here and on CRAN, but there is nothing. Perl, on the other side, has many libraries for downloading torrents. Python supports torrents as well.

Once you have chosen a library, just write a script to download the torrent and to give a return signal when it is finished.

However, I would object on the reproducibility of using data downloaded with a torrent: what would you do if the torrent disappears or is modified after you have used it? How do you take into account the version of the data?

Moreover, consider that scientific databases have to release their data often, and every release would require a different torrent to be seed. That would make it more difficult to control and keep.

[Istvan Albert]

The data underlying the torrents cannot be modified because unlike files that are downloaded by name it has check-sums that must match. You would need to publish a new torrent for modified data. So torrents are actually more reliable than a download. The second issue of disappearing data is also a vote for torrents. Unlike a download that can disappear unless someone republishes it on an FTP site a torrent is automatically distributed and shared thus it will live on even if the original source is gone.

Answer 2

[Paulo Nuin]

There are different libraries and APIs for bittorrent downloads. I don't have enough knowledge about them to say if you actually need or not a bittorrent client installed in your system (something like uTorrent) but the API is quite rich to allow automation and inclusion in a pipeline. You can see more here