ENCODE ChIP-Seq Control Experiments

Question

[ghazalhaddad]

Hello, I have a question regarding the ChIP-seq data from the ENCODE Project available on goo.gl/uTlPgE I have downloaded all of the data available from Homo sapien transcription factor ChIP-seq experiments (which ends up being more than 2500 fastq files), and I want to do peak calling on these files after alignment, therefore, I would need a control file for each of the ChIP-seq experiment files to do peak-calling on. However, I don't know how to find the ID for the control experiment corresponding to each experiment, since there is no file on the ENCODE website that has all the control experiments for each experiment laid out. I was wondering if anyone has any idea about how to go about finding the control experiments for this very large number of downloaded files without having to look each experiment up separately on the ENCODE website.

I would appreciate any responses!

[Sinji]

You'd have to go through them manually, each experiment has a control dataset specified. Well ... almost all of them do. ENCODE is funny sometimes and this right here is one of the most frustrating reasons.

[ghazalhaddad]

That's the only way I have found to work as well, the problem is that it will take more time that I'm willing to spend to go through all the files I have downloaded and find their control experiments manually. I really hope there is another way!

Answer 1

[Ryan Dale]

I recently had to deal with this as well.

I typically interactively identify a data subset of interest on encodeproject.org and then get the general metadata from the first line of the file downloaded by clicking the "Download" button e.g.,

https://www.encodeproject.org/metadata/type=Experiment&biosample_term_name=HepG2&assay_title=ChIP-seq&limit=all/metadata.tsv

But this file doesn't contain info on controls. The trick is to use their REST API to query for individual accessions. Here's a working example that assigns controls to that metadata.tsv which you should be able to adapt for your use-case:

	#!/usr/bin/env python
	# Install into new environment and activate it before running this script:
	#
	# conda create -n biostars-212519 python=3 requests numpy pandas
	# source activate biostars-212519
	# python identify_encode_controls.py
	import pandas
	import numpy as np
	import requests
	def accession_metadata(acc):
	"""
	Returns the metadata for ENCODE accession `acc` (e.g., ENCSR000BJN)
	"""
	HEADERS = {'accept': 'application/json'}
	URL = (
	'https://www.encodeproject.org/experiments/{0}/?frame=object'
	.format(acc)
	)
	response = requests.get(URL, headers=HEADERS)
	return response.content
	# You can get a URL interactively on encodeproject.org by subsetting your
	# query, clicking on the "download" button, and extracting the first line of
	# that file. This example is all HepG2 ChIP-seq data.
	metadata_url = (
	"https://www.encodeproject.org/metadata/type=Experiment&"
	"biosample_term_name=HepG2&assay_title=ChIP-seq&limit=all/metadata.tsv"
	)
	df = pandas.read_table(metadata_url)
	# subset just the first 100 rows for this example
	df = df.iloc[:100]
	def find_controls(acc):
	"""
	The metadata for an accession contains a "possible controls" field. I'm
	taking that to mean there can be multiple controls, so to be safe I'm
	returning a list of them.
	"""
	m = pandas.read_json(accession_metadata(acc), typ='series')
	c = m['possible_controls']
	return [i.split('/')[2] for i in c]
	# the metadata has multiple rows for each accession. To speed things up
	# dramatically, only look for controls for the unique set of accessions, and
	# then join them to the dataframe afterwards.
	#
	ds = []
	for acc in df['Experiment accession'].unique():
	print('getting metadata for accession:', acc)
	ds.append(
	{
	'Experiment accession': acc,
	'controls': find_controls(acc)
	}
	)
	controls = pandas.DataFrame(ds).set_index('Experiment accession')
	# join controls to full metadata
	df = df.join(controls, on='Experiment accession')

view raw identify_encode_controls.py hosted with ❤ by GitHub

[ghazalhaddad]

This works perfectly, thank you so much!

[Vanilla]

Now the'possible_controls' has been changed to 'Controlled by'.