Help needed to troubleshoot a Bioconda recipe for a Rust tool

Question

[Matthias Zepper]

Hello everyone,

Does somebody here have experience with adding Rust tools to Bioconda?

I would like to add BarcodEX-rs to Bioconda, but unfortunately the build fails. Since there is no dedicated documentation for adding Rust packages and my attempts to copy the recipes from other Rust packages have not been successful (I tried those of perbase, rust-bio-tools and fastqc-rs ), I am somewhat out of ideas how to proceed with troubleshooting.

Regardless of what I try, the build fails with:

error[E0432]: unresolved import
error[E0433]: failed to resolve. Use of undeclared type or module
error[E0599]: no function or associated item found for struct in the current scope

However, if I build the tool locally with cargo build --release everything works smoothly and all imports are automatically resolved. What peculiarities of the Bioconda build system am I missing?

The (somewhat desperate) PR can be found here.

[GenoMax]

Tagging - Devon Ryan (bioconda) and Rob (rust advocate)

Answer 1

[Rob]

It looks like the compiler itself throws many errors (looking e.g. here). Does it compile for you locally and cleanly on the tagged release you are trying to submit?

EDIT: Ok, I just tried locally and the latest tag version seems to build fine. I assume the issue may have something to do with a many of the changes being made in the build.sh file? Could you try a minimal build.sh like the one we use for alevin-fry?

[Matthias Zepper]

Foremost: Thanks a lot for the instant replies!

Yes, I already committed a new clean build.sh similar to that of alevin-fry, so fingers crossed. Ironically, I did take a look at your tool and the recipe, but that patch file was very enigmatic to me. Therefore, I opted for other recipes to use as template.

PS: That cluttered build.sh was the result of increasingly desperate attempts and all those commits will of course be squashed before merging to Bioconda ;-)

[Rob]

I can see that, unfortunately, that didn't work. You can ignore the patch in the alevin-fry recipe. That is no longer needed, but was for a while because of an upstream issue for a dependency when building on OSX.

However, in your case I think I can see what the issue may be. When I built locally, I got the requested version of all of the dependencies in the Cargo.toml file. However, on the Circle CI, it seems that the versions being fetched are too new, and several are breaking semantic compatibility with what the barcodeEX-rs program assumes. I think the Cargo.toml needs to be updated to pin the specified versions (or the program code should be updated to work with the newer versions of these crates, like Clap). For example, Cargo.toml asks for the clap 3.0-beta1, but in Circle CI 3.2.something is pulled down, and this is not compatible with the beta in terms of usage. However, when I built locally and it pulled down the requested version of everything, it built and ran fine.

[Matthias Zepper]

Unfortunately, the build.sh from alevin-fry did not help, those errors persist.

[Rob]

In fact, I see that if I remove / delete the Cargo.lock file from either the current HEAD commit or from the release tarball, then the program fails to build (in almost exactly the same way as it does on bioconda). At least this may help debug more quickly locally. I'd recommend submitting a bug report upstream in the repo about this.

[Matthias Zepper]

Thank you very much for your thorough investigation of the issue! This helped me a lot, since I myself do not code Rust and therefore did not notice the legacy versions and outdated calls.

A build failure with more modern versions of the packages is nonetheless strange, since I made a Docker image to benchmark the tool on our HPC and this worked nicely on rust:latest as builder.

I will try to submit a bug report, but to my best knowledge the tool is unmaintained, since its author has moved to an industry job. Admittedly, this somewhat argues against adding it to Bioconda in the first place, but it does exactly what we need and nf-core modules rely on Bioconda for their container images.

[Rob]

I believe this works because your repo (building the docker) also includes the Cargo.lock file. The problem is not to do with the version of rust (the rust compiler), but with the versions of the dependencies that are being relied upon. Generally, Cargo.toml says what versions of dependencies your program should use. There is different notation e.g. =1.2.3 means exactly 1.2.3 while ^1.2 means e.g. >= 1.2.0 and <1.3.0, etc.

This repo also contains a Cargo.lock file. Generally, it is not recommended to distribute .lock files or include them in your repo. The lockfile contains the exact resolution of package versions that your computer created at the time it solved for the dependencies. If a lock file is present, then when cargo builds, it will download these exact versions from the urls specified in the lock file. However, if you don't have a lockfile, and just have a Cargo.toml, then cargo will use the rules to solve the dependencies given the later versions of the packages.

So, what I'm suggesting is if you just go to your repo, delete Cargo.lock and run cargo clean. Then the next time you try to build, you'll get exactly the same errors you're seeing on bioconda! This is because there seem to be some issues in the way that traits are being resolved with the latest packages in how the code is written. This is why I think it's worth report the issue upstream.

[Matthias Zepper]

Thank you very much for this comprehensive and immensely helpful explanation. I had believed that the Cargo.lock file is there on purpose, because that is what I had found in the documentation, but that is why it is so useful to have a seasoned Rust veteran to kindly answer all my stupid noob questions ;-).

Thanks again, I will proceed as you suggested, try to fix and/or to report the issue upstream!

[Rob]

Thanks for the link, Matthias! I'd actually not seen the recommendation in the link that you included. So, it seems that including the Cargo.lock file in an end-user target application isn't actually as discouraged as I'd thought, and the reasoning there makes total sense!

However, even if the developer (justifiably) would like to keep a Cargo.lock file committed, there are still at least 2 problems I see. The first is in the repo itself. While the application was updated to release 1.2, the lockfile remained frozen at that for the 1.1 release. At the very least, the lockfile should be updated to represent the state of the environment solve when 1.2 was tagged.

The other issue is on the bioconda side, and @Devon is probably the best person to address that. The question there is why the bioconda CI is not respecting the Cargo.lock file!

[Matthias Zepper]

Thanks a lot for your help. It now finally worked after indeed changing the source code and the recipe as well.