4 [Bo Li](http://pages.cs.wisc.edu/~bli) \(bli at cs dot wisc dot edu\)
11 * [Introduction](#introduction)
12 * [Compilation & Installation](#compilation)
15 * [Simulation](#simulation)
16 * [Generate Transcript-to-Gene-Map from Trinity Output](#gen_trinity)
17 * [Differential Expression Analysis](#de)
19 * [Acknowledgements](#acknowledgements)
24 ## <a name="introduction"></a> Introduction
26 RSEM is a software package for estimating gene and isoform expression
27 levels from RNA-Seq data. The RSEM package provides an user-friendly
28 interface, supports threads for parallel computation of the EM
29 algorithm, single-end and paired-end read data, quality scores,
30 variable-length reads and RSPD estimation. In addition, it provides
31 posterior mean and 95% credibility interval estimates for expression
32 levels. For visualization, It can generate BAM and Wiggle files in
33 both transcript-coordinate and genomic-coordinate. Genomic-coordinate
34 files can be visualized by both UCSC Genome browser and Broad
35 Institute's Integrative Genomics Viewer (IGV). Transcript-coordinate
36 files can be visualized by IGV. RSEM also has its own scripts to
37 generate transcript read depth plots in pdf format. The unique feature
38 of RSEM is, the read depth plots can be stacked, with read depth
39 contributed to unique reads shown in black and contributed to
40 multi-reads shown in red. In addition, models learned from data can
41 also be visualized. Last but not least, RSEM contains a simulator.
43 ## <a name="compilation"></a> Compilation & Installation
45 To compile RSEM, simply run
49 For cygwin users, please uncomment the 3rd and 7th line in
50 'sam/Makefile' before you run 'make'.
52 To compile EBSeq, which is included in the RSEM package, run
56 To install, simply put the rsem directory in your environment's PATH
59 If you prefer to put all RSEM executables to a bin directory, please
60 also remember to put 'rsem_perl_utils.pm' and 'WHAT_IS_NEW' to the
61 same bin directory. 'rsem_perl_utils.pm' is required for most RSEM's
62 perl scripts and 'WHAT_IS_NEW' contains the RSEM version information.
66 C++, Perl and R are required to be installed.
68 To take advantage of RSEM's built-in support for the Bowtie/Bowtie 2
69 alignment program, you must have
70 [Bowtie](http://bowtie-bio.sourceforge.net) and/or [Bowtie
71 2](http://bowtie-bio.sourceforge.net/bowtie2) installed.
73 ## <a name="usage"></a> Usage
75 ### I. Preparing Reference Sequences
77 RSEM can extract reference transcripts from a genome if you provide it
78 with gene annotations in a GTF file. Alternatively, you can provide
79 RSEM with transcript sequences directly.
81 Please note that GTF files generated from the UCSC Table Browser do not
82 contain isoform-gene relationship information. However, if you use the
83 UCSC Genes annotation track, this information can be recovered by
84 downloading the knownIsoforms.txt file for the appropriate genome.
86 To prepare the reference sequences, you should run the
87 'rsem-prepare-reference' program. Run
89 rsem-prepare-reference --help
91 to get usage information or visit the [rsem-prepare-reference
92 documentation page](http://deweylab.biostat.wisc.edu/rsem/rsem-prepare-reference.html).
94 ### II. Calculating Expression Values
96 To calculate expression values, you should run the
97 'rsem-calculate-expression' program. Run
99 rsem-calculate-expression --help
101 to get usage information or visit the [rsem-calculate-expression
102 documentation page](http://deweylab.biostat.wisc.edu/rsem/rsem-calculate-expression.html).
104 #### Calculating expression values from single-end data
106 For single-end models, users have the option of providing a fragment
107 length distribution via the '--fragment-length-mean' and
108 '--fragment-length-sd' options. The specification of an accurate fragment
109 length distribution is important for the accuracy of expression level
110 estimates from single-end data. If the fragment length mean and sd are
111 not provided, RSEM will not take a fragment length distribution into
114 #### Using an alternative aligner
116 By default, RSEM automates the alignment of reads to reference
117 transcripts using the Bowtie alignment program. Turn on '--bowtie2'
118 for 'rsem-prepare-reference' and 'rsem-calculate-expression' will
119 allow RSEM to use the Bowtie 2 alignment program instead. Please note
120 that indel alignments, local alignments and discordant alignments are
121 disallowed when RSEM uses Bowtie 2 since RSEM currently cannot handle
122 them. See the description of '--bowtie2' option in
123 'rsem-calculate-expression' for more details. To use an alternative
124 alignment program, align the input reads against the file
125 'reference_name.idx.fa' generated by 'rsem-prepare-reference', and
126 format the alignment output in SAM or BAM format. Then, instead of
127 providing reads to 'rsem-calculate-expression', specify the '--sam' or
128 '--bam' option and provide the SAM or BAM file as an argument. When
129 using an alternative aligner, you may also want to provide the
130 '--no-bowtie' option to 'rsem-prepare-reference' so that the Bowtie
131 indices are not built.
133 RSEM requires all alignments of the same read group together. For
134 paired-end reads, RSEM also requires the two mates of any alignment be
135 adjacent. To check if your SAM/BAM file satisfy the requirements,
138 rsem-sam-validator <input.sam/input.bam>
140 If your file does not satisfy the requirements, you can use
141 'convert-sam-for-rsem' to convert it into a BAM file which RSEM can
144 convert-sam-for-rsem --help
146 to get usage information or visit the [convert-sam-for-rsem
148 page](http://deweylab.biostat.wisc.edu/rsem/convert-sam-for-rsem.html).
150 However, please note that RSEM does ** not ** support gapped
151 alignments. So make sure that your aligner does not produce alignments
152 with intersions/deletions. Also, please make sure that you use
153 'reference_name.idx.fa' , which is generated by RSEM, to build your
156 ### III. Visualization
158 RSEM contains a version of samtools in the 'sam' subdirectory. RSEM
159 will always produce three files:'sample_name.transcript.bam', the
160 unsorted BAM file, 'sample_name.transcript.sorted.bam' and
161 'sample_name.transcript.sorted.bam.bai' the sorted BAM file and
162 indices generated by the samtools included. All three files are in
163 transcript coordinates. When users specify the --output-genome-bam
164 option RSEM will produce three files: 'sample_name.genome.bam', the
165 unsorted BAM file, 'sample_name.genome.sorted.bam' and
166 'sample_name.genome.sorted.bam.bai' the sorted BAM file and indices
167 generated by the samtools included. All these files are in genomic
170 #### a) Converting transcript BAM file into genome BAM file
172 Normally, RSEM will do this for you via '--output-genome-bam' option
173 of 'rsem-calculate-expression'. However, if you have run
174 'rsem-prepare-reference' and use 'reference_name.idx.fa' to build
175 indices for your aligner, you can use 'rsem-tbam2gbam' to convert your
176 transcript coordinate BAM alignments file into a genomic coordinate
177 BAM alignments file without the need to run the whole RSEM
178 pipeline. Please note that 'rsem-prepare-reference' will convert all
179 'N' into 'G' by default for 'reference_name.idx.fa'. If you do not
180 want this to happen, please use '--no-ntog' option.
184 rsem-tbam2gbam reference_name unsorted_transcript_bam_input genome_bam_output
186 reference_name : The name of reference built by 'rsem-prepare-reference'
187 unsorted_transcript_bam_input : This file should satisfy: 1) the alignments of a same read are grouped together, 2) for any paired-end alignment, the two mates should be adjacent to each other, 3) this file should not be sorted by samtools
188 genome_bam_output : The output genomic coordinate BAM file's name
190 #### b) Generating a Wiggle file
192 A wiggle plot representing the expected number of reads overlapping
193 each position in the genome/transcript set can be generated from the
194 sorted genome/transcript BAM file output. To generate the wiggle
195 plot, run the 'rsem-bam2wig' program on the
196 'sample_name.genome.sorted.bam'/'sample_name.transcript.sorted.bam' file.
200 rsem-bam2wig sorted_bam_input wig_output wiggle_name [--no-fractional-weight]
202 sorted_bam_input : Input BAM format file, must be sorted
203 wig_output : Output wiggle file's name, e.g. output.wig
204 wiggle_name : The name of this wiggle plot
205 --no-fractional-weight : If this is set, RSEM will not look for "ZW" tag and each alignment appeared in the BAM file has weight 1. Set this if your BAM file is not generated by RSEM. Please note that this option must be at the end of the command line
207 #### c) Loading a BAM and/or Wiggle file into the UCSC Genome Browser or Integrative Genomics Viewer(IGV)
209 For UCSC genome browser, please refer to the [UCSC custom track help page](http://genome.ucsc.edu/goldenPath/help/customTrack.html).
211 For integrative genomics viewer, please refer to the [IGV home page](http://www.broadinstitute.org/software/igv/home). Note: Although IGV can generate read depth plot from the BAM file given, it cannot recognize "ZW" tag RSEM puts. Therefore IGV counts each alignment as weight 1 instead of the expected weight for the plot it generates. So we recommend to use the wiggle file generated by RSEM for read depth visualization.
213 Here are some guidance for visualizing transcript coordinate files using IGV:
215 1) Import the transcript sequences as a genome
217 Select File -> Import Genome, then fill in ID, Name and Fasta file. Fasta file should be 'reference_name.idx.fa'. After that, click Save button. Suppose ID is filled as 'reference_name', a file called 'reference_name.genome' will be generated. Next time, we can use: File -> Load Genome, then select 'reference_name.genome'.
219 2) Load visualization files
221 Select File -> Load from File, then choose one transcript coordinate visualization file generated by RSEM. IGV might require you to convert wiggle file to tdf file. You should use igvtools to perform this task. One way to perform the conversion is to use the following command:
223 igvtools tile reference_name.transcript.wig reference_name.transcript.tdf reference_name.genome
225 #### d) Generating Transcript Wiggle Plots
227 To generate transcript wiggle plots, you should run the
228 'rsem-plot-transcript-wiggles' program. Run
230 rsem-plot-transcript-wiggles --help
232 to get usage information or visit the [rsem-plot-transcript-wiggles
233 documentation page](http://deweylab.biostat.wisc.edu/rsem/rsem-plot-transcript-wiggles.html).
235 #### e) Visualize the model learned by RSEM
237 RSEM provides an R script, 'rsem-plot-model', for visulazing the model learned.
241 rsem-plot-model sample_name output_plot_file
243 sample_name: the name of the sample analyzed
244 output_plot_file: the file name for plots generated from the model. It is a pdf file
246 The plots generated depends on read type and user configuration. It
247 may include fragment length distribution, mate length distribution,
248 read start position distribution (RSPD), quality score vs observed
249 quality given a reference base, position vs percentage of sequencing
250 error given a reference base and histogram of reads with different
251 number of alignments.
253 fragment length distribution and mate length distribution: x-axis is fragment/mate length, y axis is the probability of generating a fragment/mate with the associated length
255 RSPD: Read Start Position Distribution. x-axis is bin number, y-axis is the probability of each bin. RSPD can be used as an indicator of 3' bias
257 Quality score vs. observed quality given a reference base: x-axis is Phred quality scores associated with data, y-axis is the "observed quality", Phred quality scores learned by RSEM from the data. Q = -10log_10(P), where Q is Phred quality score and P is the probability of sequencing error for a particular base
259 Position vs. percentage sequencing error given a reference base: x-axis is position and y-axis is percentage sequencing error
261 Histogram of reads with different number of alignments: x-axis is the number of alignments a read has and y-axis is the number of such reads. The inf in x-axis means number of reads filtered due to too many alignments
263 ## <a name="example"></a> Example
265 Suppose we download the mouse genome from UCSC Genome Browser. We
266 will use a reference_name of 'mouse_125'. We have a FASTQ-formatted file,
267 'mmliver.fq', containing single-end reads from one sample, which we
268 call 'mmliver_single_quals'. We want to estimate expression values by
269 using the single-end model with a fragment length distribution. We
270 know that the fragment length distribution is approximated by a normal
271 distribution with a mean of 150 and a standard deviation of 35. We
272 wish to generate 95% credibility intervals in addition to maximum
273 likelihood estimates. RSEM will be allowed 1G of memory for the
274 credibility interval calculation. We will visualize the probabilistic
275 read mappings generated by RSEM on UCSC genome browser. We will
276 generate a list of genes' transcript wiggle plots in 'output.pdf'. The
277 list is 'gene_ids.txt'. We will visualize the models learned in
278 'mmliver_single_quals.models.pdf'
280 The commands for this scenario are as follows:
282 rsem-prepare-reference --gtf mm9.gtf --mapping knownIsoforms.txt --bowtie-path /sw/bowtie /data/mm9 /ref/mouse_125
283 rsem-calculate-expression --bowtie-path /sw/bowtie --phred64-quals --fragment-length-mean 150.0 --fragment-length-sd 35.0 -p 8 --output-genome-bam --calc-ci --memory-allocate 1024 /data/mmliver.fq /ref/mouse_125 mmliver_single_quals
284 rsem-bam2wig mmliver_single_quals.sorted.bam mmliver_single_quals.sorted.wig mmliver_single_quals
285 rsem-plot-transcript-wiggles --gene-list --show-unique mmliver_single_quals gene_ids.txt output.pdf
286 rsem-plot-model mmliver_single_quals mmliver_single_quals.models.pdf
288 ## <a name="simulation"></a> Simulation
290 RSEM provides users the 'rsem-simulate-reads' program to simulate RNA-Seq data based on parameters learned from real data sets. Run
294 to get usage information or read the following subsections.
298 rsem-simulate-reads reference_name estimated_model_file estimated_isoform_results theta0 N output_name [-q]
300 __reference_name:__ The name of RSEM references, which should be already generated by 'rsem-prepare-reference'
302 __estimated_model_file:__ This file describes how the RNA-Seq reads will be sequenced given the expression levels. It determines what kind of reads will be simulated (single-end/paired-end, w/o quality score) and includes parameters for fragment length distribution, read start position distribution, sequencing error models, etc. Normally, this file should be learned from real data using 'rsem-calculate-expression'. The file can be found under the 'sample_name.stat' folder with the name of 'sample_name.model'. 'model_file_description.txt' provides the format and meanings of this file.
304 __estimated_isoform_results:__ This file contains expression levels for all isoforms recorded in the reference. It can be learned using 'rsem-calculate-expression' from real data. The corresponding file users want to use is 'sample_name.isoforms.results'. If simulating from user-designed expression profile is desired, start from a learned 'sample_name.isoforms.results' file and only modify the 'TPM' column. The simulator only reads the TPM column. But keeping the file format the same is required. If the RSEM references built are aware of allele-specific transcripts, 'sample_name.alleles.results' should be used instead.
306 __theta0:__ This parameter determines the fraction of reads that are coming from background "noise" (instead of from a transcript). It can also be estimated using 'rsem-calculate-expression' from real data. Users can find it as the first value of the third line of the file 'sample_name.stat/sample_name.theta'.
308 __N:__ The total number of reads to be simulated. If 'rsem-calculate-expression' is executed on a real data set, the total number of reads can be found as the 4th number of the first line of the file 'sample_name.stat/sample_name.cnt'.
310 __output_name:__ Prefix for all output files.
312 __--seed seed:__ Set seed for the random number generator used in simulation. The seed should be a 32-bit unsigned integer.
314 __-q:__ Set it will stop outputting intermediate information.
318 output_name.sim.isoforms.results, output_name.sim.genes.results: Expression levels estimated by counting where each simulated read comes from.
319 output_name.sim.alleles.results: Allele-specific expression levels estimated by counting where each simulated read comes from.
321 output_name.fa if single-end without quality score;
322 output_name.fq if single-end with quality score;
323 output_name_1.fa & output_name_2.fa if paired-end without quality
325 output_name_1.fq & output_name_2.fq if paired-end with quality score.
327 **Format of the header line**: Each simulated read's header line encodes where it comes from. The header line has the format:
329 {>/@}_rid_dir_sid_pos[_insertL]
331 __{>/@}:__ Either '>' or '@' must appear. '>' appears if FASTA files are generated and '@' appears if FASTQ files are generated
333 __rid:__ Simulated read's index, numbered from 0
335 __dir:__ The direction of the simulated read. 0 refers to forward strand ('+') and 1 refers to reverse strand ('-')
337 __sid:__ Represent which transcript this read is simulated from. It ranges between 0 and M, where M is the total number of transcripts. If sid=0, the read is simulated from the background noise. Otherwise, the read is simulated from a transcript with index sid. Transcript sid's transcript name can be found in the 'transcript_id' column of the 'sample_name.isoforms.results' file (at line sid + 1, line 1 is for column names)
339 __pos:__ The start position of the simulated read in strand dir of transcript sid. It is numbered from 0
341 __insertL:__ Only appear for paired-end reads. It gives the insert length of the simulated read.
345 Suppose we want to simulate 50 millon single-end reads with quality scores and use the parameters learned from [Example](#example). In addition, we set theta0 as 0.2 and output_name as 'simulated_reads'. The command is:
347 rsem-simulate-reads /ref/mouse_125 mmliver_single_quals.stat/mmliver_single_quals.model mmliver_single_quals.isoforms.results 0.2 50000000 simulated_reads
349 ## <a name="gen_trinity"></a> Generate Transcript-to-Gene-Map from Trinity Output
351 For Trinity users, RSEM provides a perl script to generate transcript-to-gene-map file from the fasta file produced by Trinity.
355 extract-transcript-to-gene-map-from-trinity trinity_fasta_file map_file
357 trinity_fasta_file: the fasta file produced by trinity, which contains all transcripts assembled.
358 map_file: transcript-to-gene-map file's name.
360 ## <a name="de"></a> Differential Expression Analysis
362 Popular differential expression (DE) analysis tools such as edgeR and
363 DESeq do not take variance due to read mapping uncertainty into
364 consideration. Because read mapping ambiguity is prevalent among
365 isoforms and de novo assembled transcripts, these tools are not ideal
366 for DE detection in such conditions.
368 EBSeq, an empirical Bayesian DE analysis tool developed in UW-Madison,
369 can take variance due to read mapping ambiguity into consideration by
370 grouping isoforms with parent gene's number of isoforms. In addition,
371 it is more robust to outliers. For more information about EBSeq
372 (including the paper describing their method), please visit [EBSeq's
373 website](http://www.biostat.wisc.edu/~ningleng/EBSeq_Package).
376 RSEM includes EBSeq in its folder named 'EBSeq'. To use it, first type
380 to compile the EBSeq related codes.
382 EBSeq requires gene-isoform relationship for its isoform DE
383 detection. However, for de novo assembled transcriptome, it is hard to
384 obtain an accurate gene-isoform relationship. Instead, RSEM provides a
385 script 'rsem-generate-ngvector', which clusters transcripts based on
386 measures directly relating to read mappaing ambiguity. First, it
387 calcualtes the 'unmappability' of each transcript. The 'unmappability'
388 of a transcript is the ratio between the number of k mers with at
389 least one perfect match to other transcripts and the total number of k
390 mers of this transcript, where k is a parameter. Then, Ng vector is
391 generated by applying Kmeans algorithm to the 'unmappability' values
392 with number of clusters set as 3. This program will make sure the mean
393 'unmappability' scores for clusters are in ascending order. All
394 transcripts whose lengths are less than k are assigned to cluster
397 rsem-generate-ngvector --help
399 to get usage information or visit the [rsem-generate-ngvector
401 page](http://deweylab.biostat.wisc.edu/rsem/rsem-generate-ngvector.html).
403 If your reference is a de novo assembled transcript set, you should
404 run 'rsem-generate-ngvector' first. Then load the resulting
405 'output_name.ngvec' into R. For example, you can use
407 NgVec <- scan(file="output_name.ngvec", what=0, sep="\n")
409 . After that, set "NgVector = NgVec" for your differential expression
410 test (either 'EBTest' or 'EBMultiTest').
413 For users' convenience, RSEM also provides a script
414 'rsem-generate-data-matrix' to extract input matrix from expression
417 rsem-generate-data-matrix sampleA.[genes/isoforms].results sampleB.[genes/isoforms].results ... > output_name.counts.matrix
419 The results files are required to be either all gene level results or
420 all isoform level results. You can load the matrix into R by
422 IsoMat <- data.matrix(read.table(file="output_name.counts.matrix"))
424 before running either 'EBTest' or 'EBMultiTest'.
426 Lastly, RSEM provides two scripts, 'rsem-run-ebseq' and
427 'rsem-control-fdr', to help users find differential expressed
428 genes/transcripts. First, 'rsem-run-ebseq' calls EBSeq to calculate related statistics
429 for all genes/transcripts. Run
431 rsem-run-ebseq --help
433 to get usage information or visit the [rsem-run-ebseq documentation
434 page](http://deweylab.biostat.wisc.edu/rsem/rsem-run-ebseq.html). Second,
435 'rsem-control-fdr' takes 'rsem-run-ebseq' 's result and reports called
436 differentially expressed genes/transcripts by controlling the false
439 rsem-control-fdr --help
441 to get usage information or visit the [rsem-control-fdr documentation
442 page](http://deweylab.biostat.wisc.edu/rsem/rsem-control-fdr.html). These
443 two scripts can perform DE analysis on either 2 conditions or multiple
446 Please note that 'rsem-run-ebseq' and 'rsem-control-fdr' use EBSeq's
447 default parameters. For advanced use of EBSeq or information about how
448 EBSeq works, please refer to [EBSeq's
449 manual](http://www.bioconductor.org/packages/devel/bioc/vignettes/EBSeq/inst/doc/EBSeq_Vignette.pdf).
451 Questions related to EBSeq should
452 be sent to <a href="mailto:nleng@wisc.edu">Ning Leng</a>.
454 ## <a name="authors"></a> Authors
456 The RSEM algorithm is developed by Bo Li and Colin Dewey. The RSEM software is mainly implemented by Bo Li.
458 ## <a name="acknowledgements"></a> Acknowledgements
460 RSEM uses the [Boost C++](http://www.boost.org) and
461 [samtools](http://samtools.sourceforge.net) libraries. RSEM includes
462 [EBSeq](http://www.biostat.wisc.edu/~ningleng/EBSeq_Package/) for
463 differential expression analysis.
465 We thank earonesty and Dr. Samuel Arvidsson for contributing patches.
467 We thank Han Lin, j.miller, Joël Fillon, Dr. Samuel G. Younkin and Malcolm Cook for suggesting possible fixes.
469 ## <a name="license"></a> License
471 RSEM is licensed under the [GNU General Public License
472 v3](http://www.gnu.org/licenses/gpl-3.0.html).