The Hardware Request For Repeatmasker

Question

[chjiao3456]

I installed RepeatMasker on my PC(ubuntu 10.04, 8G, cpu:i5), and this software can program very well on small sequence data(<10M), but for larger data(i.e. human chromosome 100M+) RepeatMasker always stopped before it can report a result, it that the hardware of my computer not enough for such size data? Or something else? And what is the usual request to run this software, anyone knows about these?

[JC]

I think is something else, I already ran RM on linux boxes with 2 cores and 4 GB, masking human genome sizes.

[chjiao3456]

And how long did the process cost?

[JC]

it take a few hours, ~10 hrs

[chjiao3456]

Thanks very much~ There are three kinds of Sequence Search Engine (Cross_Match, RMBlast and ABBlast) available and I am using RMBlast, which one is on your machine?

[JC]

I have the crossmatch and rmblast installed, did you find what is the problem?

[chjiao3456]

I think it's the limitation of RAM size, RepeatMasker occupy nearly all my RAM(8G) while working, and the request grows as time lasts I think that's why the program always stopped before results occur. The option -maxsize I used the default number and it is said in the help document that "Memory requirements go up with higher maxsize." Did you change this or other options when running RepeatMasker?

[JC]

no, I use defaults

[Biomonika (Noolean)]

I used this settings: RepeatMasker -nolow -norna -div 18 -gccalc -maxsize 999000000 -species human

[JC]

you're using a big MAXSIZE, the default is 4000000 (4Mb), whe you increase that value your RAM will blow. Try to reduce it.

Answer 1

[Biomonika (Noolean)]

If RepeatMasker produced .cat files and stopped just before producing .out files, you can get those by running ProcessRepeats (program which you run similiarly as RepeatMasker and is also part of the package) on the .cat files. These two out files will in fact differ just a little, so you can actually neglect the differencies and use just newly produced file.

(I have no idea why this freezing happens - in our case restarting the server helped.)

[chjiao3456]

Thanks for your answer! RepeatMasker is stopped during the producing .out files process, and I am using ProcessRepeats on .cat file now, however, ProcessRepeats occupy nearly all my RAM(8G) while working, I think that's why RepeatMasker always stopped during the process, maybe I should try a server with larger RAM.

[chjiao3456]

I can get results now! Thanks, so can I use RepeatMasker to produce only .cat files before I use ProcessRepeats?

[Biomonika (Noolean)]

Hi, unfortunately i dont know how to produce just .cat files with RepeatMasker:/ However, I am glad that I could help you (you can upvote my answer in return ;-)

Answer 2

[Brian Bushnell]

I have tested RepeatMasker and Dust, and found both of them to be incredibly slow, in addition to being over-aggressive at masking things that should not be masked, resulting in incorrect output. I find the concept of masking to be very mysterious. I'm not sure where it started, but I can't imagine it has advanced science as often as it has caused created misinformation; it seems to me like masking is generally used to allow a bad tool to run in finite time and get published.

These days, there are good tools that don't need masking. I recommend you use them.

That said, I have written masking software, for those times when it's a good idea. It's calibrated it to ensure that unlike RepeatMasker, it only excludes the most repetitive portions of a genome (under 3% of the human reference genome at default settings). It's intended for specialized cases and is not generally relevant, and I absolutely do not recommend using it blase as people seem to use ReapeatMasker. But it's 1000x faster than RepeatMasker and Dust, so if you need to mask low-complexity portions of a genome, or the portions of a genome that are homologous to some other genome, it's available in the BBMap package as bbmask.sh. I designed it for contamination removal (to prevent false positives) and to prevent spurious results when BLASTing unknown sequence against a huge database like nt (again, to prevent false positives). I do not recommend masking to be used in other contexts.

Answer 3

[miya]

Hi,I meet the same problem. When run the repeatmasker with human genome sequence(3 G), before the last step to get the result,the computer will restar without any information about errors and result.However, 1 G sequence can work well.Linux 128g RAM.Have you solved the problem?