GEM is a scientific software for studying protein-DNA interaction at high resolution using ChIP-seq/ChIP-exo data. It can also be applied to CLIP-seq and Branch-seq data.
GEM links binding event discovery and motif discovery with positional priors in the context of a generative probabilistic model of ChIP data and genome sequence, resolves ChIP data into explanatory motifs and binding events at unsurpassed spatial resolution. GEM reciprocally improves motif discovery using binding event locations, and binding event predictions using discovered motifs.

GEM has following features:

1. Exceptionally high spatial resolution on binding event prediction (aka peak calling)
2. Highly accurate de novo motif discovery
3. Resolve closely spaced (less than 500bp) homotypic events that appear as a single cluster of reads
4. Enable analysis of spatial binding constraints of multiple transcription factors, for predicting TF dimer binding sites, enhanceosomes, etc.
5. Analyze ChIP-seq, ChIP-exo, CLIP-seq and Branch-seq data, single-end or paired-end
6. Run in single-condition mode or multi-condition mode

See an example of GEM output for ES cell Sox2 binding, including binding events, K-mer Set motifs (KSMs), PWM motifs, and motif spatial distribution plots.

Download, unzip, and run ... see command line examples.

Download GEM software (version 2.6) and test data

GEM is a superset of GPS. GEM uses both ChIP-seq read data and genome sequence to perform de novo motif discovery and binding event calling, while GPS uses only ChIP-seq read data for binding event calling.

GEM can be activated by giving a genome sequence (--genome) and using any one of the following command line options:

• --k: the length of the k-mers
• --k_min and --k_max: the range for the length of k-mers
• --seed: the seed k-mer to jump start k-mer set motif discovery. The length of the seed k-mer will be used to set k.

GEM is a java software. Not installation is required. It works across most of computer systems.

Java 7 is required to execute the JAR. For analysis with mamalian genome, GEM requires about 5-15G memory. It can be specified at the command line with the option -Xmx (i.e. java -jar -Xmx10G gem.jar allocates 10GB of memory).

As GPS, read distribution file is required for GEM. The user can use the default read distribution file provided with the software as starting point (ChIP-seq default read distribution file ChIP-exo default read distribution file). After one round of prediction, GEM will re-estimate the read distribution using the predicted events.

The read distribution file specifies the empirical spatial distribution of reads for a given binding event. The file contains tab-delimited position/density pairs. The first field is the position relative to the binding position (i.e. binding event is at position 0). The second field contains the corresponding read density at that position. For example,

-344    1.42285E-4
-343    1.42275E-4
-342    1.42265E-4
... ...

Alternatively, it can be estimated directly from the ChIP-seq data. Given a set of events, we count all the reads at each position (the 5' end of the reads) relative to the corresponding event positions. The initial set of events for estimating the empirical spatial distribution can be defined by using known motifs or by finding the center of the forward and reverse read profiles. GEM has a tool to calculate the read distribution from a user provided file (coords.txt) containing the coordinates (format chr:coord, e.g. 1:3498832, or chr:coord:strand, e.g. 1:3498832:+, each coordinate on a separate line).

java -Xmx1G -cp gem.jar edu.mit.csail.cgs.deepseq.analysis.GPS_ReadDistribution --g "mm8.chrom.sizes" --coords "coords.txt" --chipseq "SRX000540_mES_CTCF.bed" --f "BED" --name "CTCF" --range 250 --smooth 5 --mrc 4

After GPS/GEM makes the prediction, it will re-estimate the read distribution using the predicted events.

If the data are too noisy or too few events are used for re-estimation, the new read distribution may not be accurate. The users are encouraged to examine the read distribution using the plot of read distributions (X_All_Read_Distributions.png) output by GEM.

GEM takes an alignment file of ChIP-seq reads and a genome sequence as input and reports a list of predicted binding events and the explanatory binding motifs.

ChIP-seq alignment file

ChIP-seq alignment file formats that are supported:

• BED
• SAM

1. Send me a few example lines (and the version of the aligner software), I will update it in the next release.
2. A quick fix is to convert the read alignment file into BED format, and then try again.

Genome sequence

To run de novo motif discovery, a genome sequence (UCSC download) is needed. The path to directory containing the genome sequence files (by chromosome, *.fa or *.fasta files, with the prefix "chr") can be specified using option --genome (for example, --genome your_path/mm8/). Note that the chromosome name should match those in the "--g" genome.chrom.sizes file, as well as those in your read alignment file.

For example, the fasta file for chromosome 2 is chr2.fa:


Output files
GEM outputs both the binding event files and the motif files. Because of the read distribution re-estimation, GEM outputs event prediction and read distribution files for multiple rounds. (See more details)

1. GEM event text file (GEM_events.txt, see more details)
2. K-mer set motifs (KSM.txt, see more details)
3. PFM file of PWM motifs (PFM.txt)
4. HTML file summarizing the GEM event and motif results (see an example and explanations)
5. GEM output folder containing more detailed result files (Round 1 and 2 are GPS and GEM results, respectively (See more details))
   • GEM event text files (significant, insignificant and filtered)
   • K-mer set motifs
   • PFM file of PWM motifs
   • Read distribution file
   • Spatial distribution between primary motif and all the secondary motifs in the 61bp around the GEM events (this is based on PWM motif match, not based on the GEM binding sites). You will have the files such as Name_Spatial_dist_0_1.txt/png showing the spatial distribution of secondary motifs (#1) with respect to the primary motif (#0) in text format or as a PNG image. If you click on the PNG image on the HTML output page, you also get the txt file with the values.

GEM also outputs the list of insignificant events (those do not pass the statistical test), and the filtered events (those would pass the statistical test using the read count, but have a low IP/Ctrl ratio, or the distribution of reads are quite different from the empirical distribution).

Optionally, GEM can be set to output BED files (using option --outBED) for loading the GEM results to Genome Browser as custom tracks for visualization. Note that in BED file, the coordinates are offset to left/right 100bp to give a region for visualization.

GEM event file is a tab-delimited file (xxx_n_GEM_events.txt ) with following fields:

The KSM file is a tab-delimited file (xxx_n_KSM.txt ) with following fields:

This data can be used to test GEM. It comes from a Ng lab publication (PMID: 18555785) and consists of Bowtie alignments of mouse ES cell CTCF ChIP-seq and GFP control reads.

Once everything is unpacked, use the following command:
java -Xmx10G -jar gem.jar --d Read_Distribution_default.txt --g mm8.chrom.sizes --genome your_path/mm8 --s 2000000000 --expt SRX000540_mES_CTCF.bed --ctrl SRX000543_mES_GFP.bed --f BED --out mouseCTCF --k_min 6 --k_max 13

Note the double dashes (--) for GEM parameters.

For ChIP-exo data, use ChIP-exo read distribution --d Read_Distribution_ChIP-exo.txt, add one more option --smooth 3 to estimate the read distribution without too much smoothing. Depending on the quality of the data, you may want to turn off the read filtering by option --nrf.

The command line parameters are in the format of --flag/name pairs. Note the double dashes (--); GEM does not accept single dash parameters.

Some parameters are required:

Some parameters (those relevant to motif discovery are in bold) are optional:

Optional flags:

• xxx_0_Read_distribution.txt: The input read distribution (specified by --d).
• xxx_0_GEM_events.txt: GPS Events used to re-estimate the read distribution of current dataset.
• xxx_1_Read_distribution.txt: The read distribution estimated from xxx_0_GEM_events.
• xxx_1_GEM_events.txt: GPS Events predicted using xxx_1_Read_distribution.
• xxx_1_KSM/PFMs.txt: Motifs discovered using GPS events.
• xxx_2_Read_distribution.txt: The read distribution estimated from GPS events.
• xxx_2_GEM_events.txt: GEM Events predicted using xxx_2_Read_distribution and xxx_1_KSM/PFMs motifs.
• xxx_2_KSM/PFMs.txt: Motifs discovered using GEM events.

Some best practices?

• For each GEM result set, create a folder with the same name as --out. Then run GEM within this folder. This will be useful if later you need to write scripts to process multiple GEM results.
• After each GEM run, check the PNG image file that plot all the read distribution curves (xxx_All_Read_Distributions.png). If the estimated curves are not smooth, you may have too few binding events to estimate from.
• If you get something unexpected (or an error), check the command line options, make sure that each option starts with "--", and there is a space between each option/value pair.

Can GEM/GPS process paired-end data?
Yes. As version 2.4, GEM/GPS can process paired-end SAM/BAM format data (using the same --f SAM option) by treating each mate-pair as two single-end reads. This works well, but it is not optimal. We are developing a new version that explicitly models the paired-end data, which should give more accurate results.

What if GEM finds wrong motif?
Clearly, GEM's binding call accuracy is depending on finding the correct primary motif. Some times a co-factor motif may be more statistically significant in the data, and it is subsequently used to direct the binding calls. There are several alternative options to try:

• If you know the consensus motif of the TF, use --seed option to set a starting k-mer for the motif discovery process.
• You may want to try some different k values, or different sequence window size --k_win.
• Try to use a different negative sequence set, for example --k_neg_dinu_shuffle option will use di-nucleotide shuffled sequences as negative set, instead of the default set taking from 300bp away from binding sites. This may be useful for TF that binds in CG-rich promoter regions (e.g. SP1).

Multi-condition v.s. Multi-replicates
GEM can analyze binding data from multiple conditions (time points) simultaneously. The user need to give them different names, for example, -–exptCond1 CTCF_cond1.bed -–exptCond2 CTCF_cond2.bed.

For multiple replicates of same condition, you can specify multiple replicates as separate files, for example, -–exptCond1 CTCF_cond1_rep1.bed -–exptCond1 CTCF_cond1_rep2.bed (note that they need to have the same name). GEM will combine the replicates as one large dataset for analysis.

Read filtering and event filtering
PCR amplification artifacts typically manifest as the observation of many reads mapping to the exact same base positions. These artifacts are quite variable and dataset-specific. Therefore, a generic approach to exclude those regions might result in the loss of true events.

GEM implements an event filtering method by comparing the read distribution of the predicted event to the expected event read distribution. A shape deviation score (IPvsEMP field) is computed using Kullback–Leibler divergence (see method section 2.6 of GPS paper). A higher score means the event is more divergent from the expected read distribution, hence more likely to be artifact or noise. A cutoff score can be specified by user to filter out spurious events using option (--sd). GEM also excludes events with less than 3 fold enrichment (IP/Control). GEM reports the filtered events, hence allows the user to verify and adjust cutoff threshold for a particular dataset. The shape deviation filter is on by default, but can be turned off using option (--nf).

In addition, GEM also applies a Poisson filter for abnormal high read count at a base position. For each base, we obtain an average read count by estimating a Gaussian Kernel density (with std=20bp) on the read counts of nearby base positions (excluding the base of interest). The estimated value is used to set Lambda parameter of Poisson distribution. The actual read count value is then set to the value corresponding to p-value=0.001 if it is larger.

Contact Yuchun Guo (yguo at mit dot edu) with any problems, comments, or suggestions.

Sign up for GPS mailing list to receive emails related to GEM/GPS updates, release, etc.

This software is for research use only.