Following up on Mikael's answer, you may have reasons for doing what you're doing. If you have access to a computational cluster, you might look into compiling meme_p
, a variant of meme
that incorporates OpenMPI components to spread out the work on multiple nodes.
You might build it like so:
$ cd /home/foo/meme_4.9.0
$ ./configure \
--prefix=/home/foo/meme_4.9.0 \
--with-url="http://meme.nbcr.net/meme" \
--enable-openmp \
--enable-debug \
--with-mpicc=/opt/openmpi-1.6.3/bin/mpicc \
--enable-opt
You need to add the OpenMPI lib
path to your LD_LIBRARY_PATH
environment variable, _e.g._ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi-1.6.3/lib
etc. in your environment setup. Your OpenMPI installation must also be present on or available to each cluster node.
A Sun Grid Engine-based script called runall.cluster
would fire off the search as follows, supposing your sys admin has set up a parallel environment called mpi_pe
(for example) with at least 64 slots:
#!/bin/bash
#
# runall.cluster
#
#$ -N memeCluster64
#$ -S /bin/bash
#$ -pe mpi_pe 64
#$ -v -np=64
#$ -cwd
#$ -o "memeCluster64.out"
#$ -e "memeCluster64.err"
#$ -notify
#$ -V
time /opt/openmpi-1.6.3/bin/mpirun \
-np 64 \
/home/foo/meme_4.9.0/bin/meme_p \
/home/foo/meme_4.9.0/data/myReads.fa \
-oc /home/foo/meme_4.9.0/output/myReads.fa.meme \
-dna \
-text \
-nmotifs 30 \
-maxsize 100000000 \
-maxw 15
To run it:
$ qsub ./runall.cluster
In our environment, testing showed immediate benefit with as few as 8 or 16 nodes, with diminishing returns after about 32-64 nodes. You could use GNU time
to do the same runtime testing on your end, i.e., measuring execution time vs nodes on a small test sequence set, in order to find a "sweet spot" where your job will run faster without taking up too much of the cluster.
Concretely, the running time of MEME grows as the square of the total number of characters of the sequence and the cube of the number of sequences. This makes running MEME on more than about 10,000 sequences impractical on commodity hardware. MEME-ChIP works around this by sampling sequences from the input set and running MEME on only the sampled sequences. DREME's running time grows roughly linearly with the number of characters in the sequence data, but it's limited to motifs of width 8 or less.
Many thanks for the detailed explanation!