Forum:Data sharing via Bittorrent is coming to Biostar

Question

[Istvan Albert]

Hello Everyone,

We are adding bittorrent data sharing to Biostars. Help us identify bugs and issues by creating a few torrents and adding them to posts on the test site. Also feel free to comment and provide suggestions and feedback. The description of how it works is at:

http://test.biostars.org/info/data/

An example post with data can be seen at:

http://test.biostars.org/p/101/

A few details on how it works:

1. Torrents can get attached to posts, answers or comments
2. A post may have multiple torrents attached.
3. Biostars will attempt to connect the IP number of the Bittorrent peer connection to the IP number of the Biostar user account. This allows you to see who the person that shares the data is.
4. Anonymous users cannot create torrents but they may share existing datasets.
5. Data may be shared without making it visible on Biostar (although this should not be considered a secure way to share data)

(note: the test site will not log you into your old account since the emails are protected so don't report that as an issue)

[Roman Valls Guimerà]

Istvan, I guess you got notified now but visiting the following link:

http://test.biostars.org/info/data/

I get a 500, Internal Server Error, right now. Same in production.

[Istvan Albert]

This is a test server and as such it may run different versions. We were troubleshooting an email handler for the Bioconductor folks so the data server was taken offline and the current version is being run.

Answer 1

[DG]

Looks great. Two thoughts:

1. I think it would have been great to simply provide integration and support for BioTorrents. Disclosure: I know and work with the author of BioTorrents, but I think it is a great tool that is currently seriously underutilised.
2. One of the reasons Torrents are currently underutlisied in biology is that for many of is, it simply isn't an option. Institutional IT policies often straight out block torrent traffic and the IT admins have no desire to whitelist torrents with trackers hosted at specific places for legitimate purposes. They would rather just not deal with torrent traffic period.

[Istvan Albert]

Yes, I do expect that some institutions will block that traffic because it has a bad reputation. I don't know how prevalent that is. It will probably vary even within organizations, time will allow us to get a sense of that. Bittorrent is the simplest and most effective way to share large datasets and we have to start using it for legitimate reasons to erase the stigma.

If we can provide the level of utility that makes scientists make their needs clear their IT will adapt to that.

As for integrating with other sites sadly I haven't heard of it BioTorrents so never reached out to them (though I have know of AcademicTorrents) Looks like BioTorrents has been published in PLoS CompBio: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010071

I am open to any kind of collaboration though.

[DG]

Yes BioTorrents was published. Morgan Langille is now a new faculty member here at the same University I am at and we have worked together in the past. I would be happy to point him to this and see if he is interested. I am not sure how much he still actively works with BioTorrents. Our University at least monitors for bittorrent traffic on the network and getting caught results in a loss of network privileges.

[martenson]

IT infrastructure and network problems can be solved easily when there is a will. The need to share data is on the rise and academic IT has to react soon. I doubt there is now a better solution to it than this one.

Banning whole bittorent protocol because of piracy is like banning http because of porn.

Answer 2

[Roman Valls Guimerà]

I just heard at a conference that a big institution such as Max Planck blocks bittorrent traffic. I'm not sure if this has been brainstormed before, but I just had two features on that line that might be interesting to explore:

1. Provide an optional VPN service to BioStar users (crowdfunded?) and/or refer to third party services to circumvent institutional network restrictions.
2. Have a google maps overlay on the institutions that block legit bittorrent traffic.

Thoughts?

[Istvan Albert]

It is definitely a problem with no easy solution at hand - you know maybe the answer is a new simple http based protocol that works like BitTorrent and we come up and define it. Thing is, it is not all that complicated.

[Roman Valls Guimerà]

I'm not sure that would be a good strategy. You'll have to implement support on clients, getting it accepted upstream, known by the userbase.... we have all this already in place today with bittorrent clients.

I'm not saying that VPNs would be that much easier for users to adopt, but definitely less effort when it comes to implementation and adoption.

[Roman Valls Guimerà]

Hmmm...: http://peerjs.com/

Apparently it does require proxies anyway for NAT'd environments:

Are there any caveats?

A small percentage of users are behind symmetric NATs. When two symmetric NAT users try to connect to each other, NAT traversal is impossible and no connection can be made. A workaround is to proxy through the connection through a TURN server. The PeerServer cloud service does not provide a TURN server. You'll have to find your own. You can pass a TURN server into the Peer object options. This will allow your PeerJS app to work seamlessly for this situation

[Roman Valls Guimerà]

http://webp2p.org/

[Roman Valls Guimerà]

https://torrentfreak.com/popcorn-time-gives-users-anonymity-with-a-free-built-in-vpn-140607/

[Roman Valls Guimerà]

This topic is getting more and more interesting:

https://github.com/feross/webtorrent

Answer 3

[Daniel]

I think this is a great idea. I do think that some institutes might block torrent data outright though.

Does anyone know the fidelity of torrents? I know that it should be self checking but is that 100% accurate?

[Istvan Albert]

each piece is protected by a cryptographic hash, altering that is practically unfeasible

Answer 4

[martenson]

Seems working, however only 1 of 3 test torrents will download (the largest one).

see: https://www.dropbox.com/s/knozqs9gitmf6ai/Screenshot%202014-07-15%2016.17.25.png

[Istvan Albert]

Strange may be some sort of traffic filtering or DHCP problem yesterday they worked but today the don't - both of those were shared via wireless computers at PSU.

The large file was shared from my home system.