Repeat masked gtf files from ensembl

Question

[kevin.stachelek]

I am looking for a repeat masked version of a reference genome from ensembl. I find information posted here that suggests accessing via ftp. I can't find figure out what subdirectory it might be hiding in. Searching via command-line didn't turn anything up.

I could run repeat-masker myself but as I remember setup is pretty involved (you need to download a repeat library, etc.)

I can post in some ensembl forum if that's more appropriate.

thanks,

[GenoMax]

Repeat masked gtf files from ensembl

Do you actually want a GFF file with regions of the genome that are repeat masked or a repeat masked genome sequence (answer provided by @h.mon below).

[kevin.stachelek]

Hi, yes the gff (or gtf) file is what I'm looking for. Something analogous to this file from UCSC.

[Emily]

Do you mean a GTF or GFF containing the repeats, or just the genes as annotated on repeat-masked sequence?

[kevin.stachelek]

Thanks for helping me clarify. I'm looking for a gtf file containing the locations of all masked regions in the ensembl reference genome. It might look like this:

chr1    hg38_rmsk   exon    67108754    67109046    1892.000000 +.  gene_id "L1P5"; transcript_id "L1P5";

[Emily]

All the gene annotation in Ensembl is done after repeat masking. There are folders of GFF3 and GTF files on the Ensembl FTP site.

[kevin.stachelek]

Sorry, it appears to me that these gtf files are for the gene annotation in ensembl rather than an annotation of the masked regions (types of transposable elements, repeats, etc.)

Am I misunderstanding?

[Emily]

Yes, they are. I was still quite confused by your comment as it said you were after the repeats, but then the line you gave was an exon line, which would suggest you want the genes.

[kevin.stachelek]

Ah, I see. think the 'exon' annotation be a confusing shortcut; the actual annotated feature is a LINE-1 element

[liorglic]

Did you ever find a solution? I'm looking to do the same now.

[kevin.stachelek]

I didn't find a solution. I think I eventually ran repeatmasker myself. The best reference in ensembl that I can find states: "Repeats can be viewed in the browser and extracted using our APIs. You can also download repeat-masked sequence from our FTP site, either hard-masked (rm) where repeats are replaced with Ns, or soft-masked (sm) where repeats are in lower-case text."

[liorglic]

Turns out this information can be fetched via the Ensembl APIs. See my answer here.

[e.r.zakiev]

So you took one of the fasta files for masked sequences (like Homo_sapiens.GRCh38.dna_rm.toplevel.fa.gz) and ran it through the repeatmasker? Did it spew out a gtf file as a result? I was looking to run velocyto RNA velocity analysis and thus was also looking for a gtf file for ensemble-based annotation

[cat.lou]

Did you figure out how to generate the file e.r.zakiev ? I'm trying to do the same.

[e.r.zakiev]

hello cat.lou & thank you for reaching out! I'm still in the process of troubleshooting the whole scVelo pipeline (i get very low splicing rates, ~6%, here is the issue on biostars as well), so the end result might be wrong with my approach, but I did, in fact ran repeatmasker on the whole genome (it ran for 3 weeks straight on our cluster) and it spewed out an .out file which I then converted to .gff3 format using the repeatmasker's rmOutToGFF3.pl and then I converted the .gff3 file to gtf file using the AGAT suite (while not forgetting to rename all the 'masked repeats' entries to 'exon' in the chromosome column of the gff3 file) and it's currently where I stand. I didn't try to run scVelo using this masked repeats file yet, but will try today. If you want I can keep you updated!

[e.r.zakiev]

also, i just tested the solution provided by liorglic above and it actually works! It just involved ~4 hours of setting up the necessary perl libraries (if you haven't set them up yet), but that's still much faster than 3 weeks!

I just put his code in a file called fetch_rmsk.pl and ran the following command:

perl fetch_rmsk.pl 'homo sapiens' 'vertebrates' rmsk_hs_ensembl.gtf

perl fetch_rmsk.pl 'mus musculus' 'vertebrates' rmsk_mm_ensembl.gtf

This fetching for human took ~50 minutes of downloading at a quite a low speed (the final gtf (bed?) file size is 783Mb) and ~30 min for mouse (461Mb)

[cat.lou]

Thanks e.r.zakiev - the perl scripts sound promising! Have you managed to successfully convert the bed file to GTF?

[e.r.zakiev]

well, not yet. There is AGAT, which I didn't test yet for this purpose (but it worked for other stuff for me before, but for which - I couldn't remember now). I also asked liorglic in his topic if he was willing to reformat the script so that the output is a gtf file 5 days ago, but so far there was no response. Admittedly, it is a relatively simple modification, I just need to look up what are the fields of a standard GTF file and modify the perl script accordingly. I plan to do it this friday, so stay tuned!

[cat.lou]

Maybe this post helps with the conversion.. Fastest way to convert BED to GTF/GFF with gene_ids?

Naive question - how do you define which version ensembl to use when running the perl script?

[e.r.zakiev]

that's a good question. I'd assume it downloads the latest build's repeat masked data, which is 112. Coming to think about it, you raised a good concern, as i would also be interested in a particular build version which is definitely not the current one, but one corresponding to the genome my data was aligned to... I am reasonably certain it should be possible to download the info from a certain build but i don't know how yet

Answer 1

[h.mon]

The easiest way is to go to the organism Ensembl page, e.g. https://www.ensembl.org/Acanthochromis_polyacanthus/Info/Index for the spiny chromis. There, you will find a "Download DNA sequence (FASTA)" link which will take you directly to the ftp folder containing the complete genome, soft-masked genome, and hard-masked genome.