SRAdb DBI errors on NFS and LustreFS

Question

[Casey Bergman]

I've been trying to use the bioconductor SRAdb on our Scientific Linux 6 cluster and have run into a problem with the the SQLlite DBI giving errors when the db_connection is first used. I observe this problem in both my NFS home directory and my LustreFS scratch directory, but the error is slightly different in each location.

Here is the code snippet I am using:

#install the SRAdb bioconductor package
source("http://www.bioconductor.org/biocLite.R")
biocLite("SRAdb")
library(SRAdb)

#download and connect to the SRA SQLlite database
sqlfile <- getSRAdbFile()
sra_con <- dbConnect(SQLite(), "SRAmetadb.sqlite")

# make the SRA id conversion
conversion <- sraConvert(c("ERP001893"), sra_con=sra_con)

Here is the error I see in my home directory on NFS:

Error in sqliteExecStatement(con, statement, bind.data) :
RS-DBI driver: (error in statement: database is locked)
Calls: sraConvert ... dbGetQuery -> sqliteQuickSQL -> sqliteExecStatement -> .Call
Execution halted

Here is the error I see in my scratch directory on LustreFS:

Error in sqliteExecStatement(con, statement, bind.data) :
RS-DBI driver: (error in statement: disk I/O error)
Calls: sraConvert ... dbGetQuery -> sqliteQuickSQL -> sqliteExecStatement -> .Call
Execution halted

Any ideas what might be going on here?

Answer 1

[Casey Bergman]

This is a general problem with SQLlite on network filesystems and is not specifically associated with the SRAdb package. You may see this error for other bioconductor packages that use SQLlite as well. The problem is described in the SQLlite page on "File Locking And Concurrency In SQLite Version 3" (http://www.sqlite.org/lockingv3.html) and has to do with the fact that many many network filesystems are not POSIX compliant:

"SQLite uses POSIX advisory locks to implement locking on Unix. On Windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem."

To work around this problem, download your SRAdb database into a non-networked location e.g. /local and access it from there:

#install the SRAdb bioconductor package
source("http://www.bioconductor.org/biocLite.R")
biocLite("SRAdb")
library(SRAdb)

#download and connect to the SRA SQLlite database
sqlfile <- getSRAdbFile(destdir = "/local", destfile = "SRAmetadb.sqlite.gz")
sra_con <- dbConnect(SQLite(), "/local/SRAmetadb.sqlite")

# make the SRA id conversion
conversion <- sraConvert(c("ERP001893"), sra_con=sra_con)