BBTools - Repair.sh bug (?)

Question

[cacampbell]

Hello Brian Bushnell,

I get an ambiguous stack trace in 1/8 - 1/10 of repair.sh jobs (but not other tools), and I cannot seem to reliably reproduce the problem. Running the same jobs again on the same files only produces errors in the same files by rare coincidence, if at all.

Here is the full output:

java -ea -Xmx38G -cp /home/cacampbe/.prog/bbmap/current/ jgi.SplitPairsAndSingles rp in1=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz in2=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz out1=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz out2=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz -Xmx38G t=9
Executing jgi.SplitPairsAndSingles [rp, in1=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz, in2=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz, out1=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz, out2=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz, -Xmx38G, t=9]

Set threads to 9
Set INTERLEAVED to false
Exception in thread "main" java.lang.NoClassDefFoundError: java/lang/ProcessBuilder$Redirect
    at fileIO.ReadWrite.getOutputStreamFromProcess(ReadWrite.java:619)
    at fileIO.ReadWrite.getPigzStream(ReadWrite.java:550)
    at fileIO.ReadWrite.getGZipOutputStream(ReadWrite.java:521)
    at fileIO.ReadWrite.getOutputStream(ReadWrite.java:382)
    at fileIO.ReadWrite.getOutputStream(ReadWrite.java:346)
    at stream.ReadStreamWriter.<init>(ReadStreamWriter.java:69)
    at stream.ReadStreamByteWriter.<init>(ReadStreamByteWriter.java:17)
    at stream.ConcurrentGenericReadOutputStream.<init>(ConcurrentGenericReadOutputStream.java:39)
    at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:53)
    at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:27)
    at jgi.SplitPairsAndSingles.process2(SplitPairsAndSingles.java:287)
    at jgi.SplitPairsAndSingles.process(SplitPairsAndSingles.java:230)
    at jgi.SplitPairsAndSingles.main(SplitPairsAndSingles.java:45)
Caused by: java.lang.ClassNotFoundException: java.lang.ProcessBuilder$Redirect
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    ... 13 more

Some (relevant?) background information:

• Java 1.8

• pigz

• BBMap 35.92

The process then hangs seemingly indefinitely.

[Brian Bushnell]

That's really strange, and I can't seem to replicate it. Are you using OpenJDK or the Oracle version of Java?

Either way, if you add these flags:

pigz=f unpigz=f

...the problem should go away. Please let me know if it doesn't.

It kind of sounds to me like sometimes an old version of Java is being used. For example, if you are on a cluster and the job is running on some random node, then there is one node with perhaps Java 1.6 and all the others have Java 1.8; or one node has pigz installed and the others don't. Basically, they only explanation I can imagine is that the environment is different for the jobs that fail compared to the ones that succeed.

[cacampbell]

Hi Brian, I believe that the java problem was the issue.

We do use an environment module system, and I was loading java 1.8 for each of these jobs. Nonetheless, it appears that three nodes are unable to find the modulepath and thus default to java 1.6. I have notified our system administrators.

I didn't see this issue with other processes because the three nodes in question all have very little RAM. Reformat doesn't take much RAM, so it is the only process that ever ran on these nodes (of the BBTools that I have been running so far)

Thanks for the help!

[GenoMax]

Ah. Modules are wonderful but when individual nodes misbehave it can be hard to track down .. specially when you are a regular user.

Answer 1

[cacampbell]

Java version error due to nodes not finding modulepath -- see comments