1 @c -*- coding: utf-8; mode: texinfo; -*-
3 @chapter Regression tests
6 * Introduction to regression tests::
7 * Precompiled regression tests::
8 * Compiling regression tests::
9 * Identifying code regressions::
10 * Memory and coverage tests::
15 @node Introduction to regression tests
16 @section Introduction to regression tests
18 LilyPond has a complete suite of regression tests that are used
19 to ensure that changes to the code do not break existing behavior.
20 These regression tests comprise small LilyPond snippets that test
21 the functionality of each part of LilyPond.
23 Regression tests are added when new functionality is added to
24 LilyPond. They are also added when bugs are identified. The
25 snippet that causes the bug becomes a regression test to verify
26 that the bug has been fixed.
28 The regression tests are compiled using special @code{make}
29 targets. There are three primary uses for the regression
30 tests. First, successful completion of the regression tests means
31 that LilyPond has been properly built. Second, the output of the
32 regression tests can be manually checked to ensure that
33 the graphical output matches the description of the intended
34 output. Third, the regression test output from two different
35 versions of LilyPond can be automatically compared to identify
36 any differences. These differences should then be manually
37 checked to ensure that the differences are intended.
39 Regression tests (@qq{regtests}) are available in precompiled form
40 as part of the documentation. Regtests can also be compiled
41 on any machine that has a properly configured LilyPond build
45 @node Precompiled regression tests
46 @section Precompiled regression tests
48 @subheading Regression test output
50 As part of the release process, the regression tests are run
51 for every LilyPond release. Full regression test output is
52 available for every stable version and the most recent development
55 Regression test output is available in HTML and PDF format. Links
56 to the regression test output are available at the developer's
57 resources page for the version of interest.
59 The latest stable version of the regtests is found at:
62 @uref{http://lilypond.org/doc/stable/input/regression/collated-files.html}
65 The latest development version of the regtests is found at:
68 @uref{http://lilypond.org/doc/latest/input/regression/collated-files.html}
72 @subheading Regression test comparison
74 Each time a new version is released, the regtests are
75 compiled and the output is automatically compared with the
76 output of the previous release. The result of these
77 comparisons is archived online:
80 @uref{http://lilypond.org/test/}
83 Checking these pages is a very important task for the LilyPond project.
84 You are invited to report anything that looks broken, or any case
85 where the output quality is not on par with the previous release,
86 as described in @rweb{Bug reports}.
88 @warning{ The special regression test
89 @file{test-output-distance.ly} will always show up as a
90 regression. This test changes each time it is run, and serves to
91 verify that the regression tests have, in fact, run.}
94 @subheading What to look for
96 The test comparison shows all of the changes that occurred between
97 the current release and the prior release. Each test that has a
98 significant difference in output is displayed, with the old
99 version on the left and the new version on the right.
101 Regression tests whose output is the same for both versions are
102 not shown in the test comparison.
106 Images: green blurs in the new version show the approximate
107 location of elements in the old version.
109 There are often minor adjustments in spacing which do not indicate
113 Log files: show the difference in command-line output.
115 The main thing to examine are any changes in page counts -- if a
116 file used to fit on 1 page but now requires 4 or 5 pages,
117 something is suspicious!
120 Profile files: give information about
121 TODO? I don't know what they're for.
126 The automatic comparison of the regtests checks the LilyPond
127 bounding boxes. This means that Ghostscript changes and changes
128 in lyrics or text are not found.
131 @node Compiling regression tests
132 @section Compiling regression tests
134 Developers may wish to see the output of the complete regression
135 test suite for the current version of the source repository
136 between releases. Current source code is available; see
137 @ref{Working with source code}. Then you will need
138 to build the LilyPond binary; see @ref{Compiling LilyPond}.
140 Uninstalling the previous LilyPond version is not necessary, nor is
141 running @code{make install}, since the tests will automatically be
142 compiled with the LilyPond binary you have just built in your source
145 From this point, the regtests are compiled with:
151 If you have a multi-core machine you may want to use the @option{-j}
152 option and @var{CPU_COUT} variable, as
153 described in @ref{Saving time with CPU_COUNT}.
154 For a quad-core processor the complete command would be:
157 make -j5 CPU_COUNT=5 test
160 The regtest output will then be available in
161 @file{input/regression/out-test}.
162 @file{input/regression/out-test/collated-examples.html}
163 contains a listing of all the regression tests that were run,
164 but none of the images are included. Individual images are
165 also available in this directory.
167 The primary use of @samp{make@tie{}test} is to verify that the
168 regression tests all run without error. The regression test
169 page that is part of the documentation is created only when the
170 documentation is built, as described in @ref{Generating documentation}.
171 Note that building the documentation requires more installed components
172 than building the source code, as described in
173 @ref{Requirements for building documentation}.
176 @node Identifying code regressions
177 @section Identifying code regressions
179 Before modified code is committed to master, a regression test
180 comparison must be completed to ensure that the changes have
181 not caused problems with previously working code. The comparison
182 is made automatically upon compiling the regression test suite
188 Before making changes, a baseline should be established by
196 Make your changes, or apply the patch(es) to consider.
199 Compile the source with @samp{make} as usual.
202 Check for unintentional changes to the regtests:
208 After this has finished, a regression test comparison will be
212 out/test-results/index.html
215 For each regression test that differs between the baseline and the
216 changed code, a regression test entry will displayed. Ideally,
217 the only changes would be the changes that you were working on.
218 If regressions are introduced, they must be fixed before
222 The special regression test @file{test-output-distance.ly} will always
223 show up as a regression. This test changes each time it is run, and
224 serves to verify that the regression tests have, in fact, run.}
227 If you are happy with the results, then stop now.
229 If you want to continue programming, then make any additional code
230 changes, and continue.
233 Compile the source with @samp{make} as usual.
236 To re-check files that differed between the initial
237 @samp{make@tie{}test-baseline} and your post-changes
238 @samp{make@tie{}check}, run:
244 This updates the regression list at @file{out/test-results/index.html}.
245 It does @emph{not} redo @file{test-output-distance.ly}.
248 When all regressions have been resolved, the output list will be empty.
251 Once all regressions have been resolved, a final check should be completed
259 This cleans the results of the previous @samp{make@tie{}check}, then does the
260 automatic regression comparison again.
265 @node Memory and coverage tests
266 @section Memory and coverage tests
268 In addition to the graphical output of the regression tests, it is
269 possible to test memory usage and to determine how much of the source
270 code has been exercised by the tests.
272 @subheading Memory usage
274 For tracking memory usage as part of this test, you will need
275 GUILE CVS; especially the following patch:
276 @uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.
278 @subheading Code coverage
280 For checking the coverage of the test suite, do the following
283 ./scripts/auxiliar/build-coverage.sh
284 @emph{# uncovered files, least covered first}
285 ./scripts/auxiliar/coverage.py --summary out-cov/*.cc
286 @emph{# consecutive uncovered lines, longest first}
287 ./scripts/auxiliar/coverage.py --uncovered out-cov/*.cc
292 @section MusicXML tests
295 LilyPond comes with a complete set of regtests for the
296 @uref{http://www.musicxml.org/,MusicXML} language. Originally
297 developed to test @samp{musicxml2ly}, these regression tests
298 can be used to test any MusicXML implementation.
300 The MusicXML regression tests are found at
301 @file{input/regression/musicxml/}.
303 The output resulting from running these tests
304 through @samp{muscxml2ly} followed by @samp{lilypond} is
305 available in the LilyPond documentation:
308 @uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}