1 @c -*- coding: utf-8; mode: texinfo; -*-
3 @chapter Regression tests
6 * Introduction to regression tests::
7 * Precompiled regression tests::
8 * Compiling regression tests::
10 * Pixel-based regtest comparison::
11 * Finding the cause of a regression::
12 * Memory and coverage tests::
17 @node Introduction to regression tests
18 @section Introduction to regression tests
20 LilyPond has a complete suite of regression tests that are used
21 to ensure that changes to the code do not break existing behavior.
22 These regression tests comprise small LilyPond snippets that test
23 the functionality of each part of LilyPond.
25 Regression tests are added when new functionality is added to
27 We do not yet have a policy on when it is appropriate to add or
28 modify a regtest when bugs are fixed. Individual developers
29 should use their best judgement until this is clarified during the
30 @ref{Grand Organization Project (GOP)}.
32 The regression tests are compiled using special @code{make}
33 targets. There are three primary uses for the regression
34 tests. First, successful completion of the regression tests means
35 that LilyPond has been properly built. Second, the output of the
36 regression tests can be manually checked to ensure that
37 the graphical output matches the description of the intended
38 output. Third, the regression test output from two different
39 versions of LilyPond can be automatically compared to identify
40 any differences. These differences should then be manually
41 checked to ensure that the differences are intended.
43 Regression tests (@qq{regtests}) are available in precompiled form
44 as part of the documentation. Regtests can also be compiled
45 on any machine that has a properly configured LilyPond build
49 @node Precompiled regression tests
50 @section Precompiled regression tests
52 @subheading Regression test output
54 As part of the release process, the regression tests are run
55 for every LilyPond release. Full regression test output is
56 available for every stable version and the most recent development
59 Regression test output is available in HTML and PDF format. Links
60 to the regression test output are available at the developer's
61 resources page for the version of interest.
63 The latest stable version of the regtests is found at:
66 @uref{http://lilypond.org/doc/stable/input/regression/collated-files.html}
69 The latest development version of the regtests is found at:
72 @uref{http://lilypond.org/doc/latest/input/regression/collated-files.html}
76 @subheading Regression test comparison
78 Each time a new version is released, the regtests are
79 compiled and the output is automatically compared with the
80 output of the previous release. The result of these
81 comparisons is archived online:
84 @uref{http://lilypond.org/test/}
87 Checking these pages is a very important task for the LilyPond project.
88 You are invited to report anything that looks broken, or any case
89 where the output quality is not on par with the previous release,
90 as described in @rweb{Bug reports}.
92 @warning{ The special regression test
93 @file{test-output-distance.ly} will always show up as a
94 regression. This test changes each time it is run, and serves to
95 verify that the regression tests have, in fact, run.}
98 @subheading What to look for
100 The test comparison shows all of the changes that occurred between
101 the current release and the prior release. Each test that has a
102 significant (noticeable) difference in output is displayed, with
103 the old version on the left and the new version on the right.
105 Some of the small changes can be ignored (slightly different slur
106 shapes, small variations in note spacing), but this is not always
107 the case: sometimes even the smallest change means that something
108 is wrong. To help in distinguishing these cases, we use bigger
109 staff size when small differences matter.
111 Staff size 30 generally means "pay extra attention to details".
112 Staff size 40 (two times bigger than default size) or more means
113 that the regtest @strong{is} about the details.
115 Staff size smaller than default doesn't mean anything.
117 Regression tests whose output is the same for both versions are
118 not shown in the test comparison.
122 Images: green blurs in the new version show the approximate
123 location of elements in the old version.
125 There are often minor adjustments in spacing which do not indicate
129 Log files: show the difference in command-line output.
131 The main thing to examine are any changes in page counts -- if a
132 file used to fit on 1 page but now requires 4 or 5 pages,
133 something is suspicious!
136 Profile files: give information about
137 TODO? I don't know what they're for.
138 Apparently they give some information about CPU usage. If you got
139 tons of changes in cell counts, this probably means that you compiled
140 @code{make test-baseline} with a different amount of CPU threads than
141 @code{make check}. Try redoing tests from scratch with the same
142 number of threads each time -- see @ref{Saving time with the -j option}.
147 The automatic comparison of the regtests checks the LilyPond
148 bounding boxes. This means that Ghostscript changes and changes
149 in lyrics or text are not found.
152 @node Compiling regression tests
153 @section Compiling regression tests
155 Developers may wish to see the output of the complete regression
156 test suite for the current version of the source repository
157 between releases. Current source code is available; see
158 @ref{Working with source code}.
160 For regression testing @code{../configure} should be run with the
161 @code{--disable-optimising} option. Then you will need
162 to build the LilyPond binary; see @ref{Compiling LilyPond}.
164 Uninstalling the previous LilyPond version is not necessary, nor is
165 running @code{make install}, since the tests will automatically be
166 compiled with the LilyPond binary you have just built in your source
169 From this point, the regtests are compiled with:
175 If you have a multi-core machine you may want to use the @option{-j}
176 option and @var{CPU_COUNT} variable, as
177 described in @ref{Saving time with CPU_COUNT}.
178 For a quad-core processor the complete command would be:
181 make -j5 CPU_COUNT=5 test
184 The regtest output will then be available in
185 @file{input/regression/out-test}.
186 @file{input/regression/out-test/collated-examples.html}
187 contains a listing of all the regression tests that were run,
188 but none of the images are included. Individual images are
189 also available in this directory.
191 The primary use of @samp{make@tie{}test} is to verify that the
192 regression tests all run without error. The regression test
193 page that is part of the documentation is created only when the
194 documentation is built, as described in @ref{Generating documentation}.
195 Note that building the documentation requires more installed components
196 than building the source code, as described in
197 @ref{Requirements for building documentation}.
200 @node Regtest comparison
201 @section Regtest comparison
203 Before modified code is committed to @code{master} (via @code{staging}),
205 comparison must be completed to ensure that the changes have
206 not caused problems with previously working code. The comparison
207 is made automatically upon compiling the regression test suite
213 Run @code{make} with current git master without any of your changes.
216 Before making changes to the code, establish a baseline for the comparison by
217 going to the @file{$LILYPOND_GIT/build/} directory and running:
224 Make your changes, or apply the patch(es) to consider.
227 Compile the source with @samp{make} as usual.
230 Check for unintentional changes to the regtests:
236 After this has finished, a regression test comparison will be
237 available (relative to the current @file{build/} directory) at:
240 out/test-results/index.html
243 For each regression test that differs between the baseline and the
244 changed code, a regression test entry will be displayed. Ideally,
245 the only changes would be the changes that you were working on.
246 If regressions are introduced, they must be fixed before
250 The special regression test @file{test-output-distance.ly} will always
251 show up as a regression. This test changes each time it is run, and
252 serves to verify that the regression tests have, in fact, run.}
255 If you are happy with the results, then stop now.
257 If you want to continue programming, then make any additional code
258 changes, and continue.
261 Compile the source with @samp{make} as usual.
264 To re-check files that differed between the initial
265 @samp{make@tie{}test-baseline} and your post-changes
266 @samp{make@tie{}check}, run:
272 This updates the regression list at @file{out/test-results/index.html}.
273 It does @emph{not} redo @file{test-output-distance.ly}.
276 When all regressions have been resolved, the output list will be empty.
279 Once all regressions have been resolved, a final check should be completed
287 This cleans the results of the previous @samp{make@tie{}check}, then does the
288 automatic regression comparison again.
293 Once a test baseline has been established, there is no need to run it again
294 unless git master changed. In other words, if you work with several branches
295 and want to do regtests comparison for all of them, you can
296 @code{make test-baseline} with git master, checkout some branch,
297 @code{make} and @code{make check} it, then switch to another branch,
298 @code{make test-clean}, @code{make} and @code{make check} it without doing
299 @code{make test-baseline} again.}
301 @node Pixel-based regtest comparison
302 @section Pixel-based regtest comparison
304 As an alternative to the @code{make test} method for regtest checking (which
305 relies upon @code{.signature} files created by a LilyPond run and which describe
306 the placing of grobs) there is a script which compares the output of two
307 LilyPond versions pixel-by-pixel. To use this, start by checking out the
308 version of LilyPond you want to use as a baseline, and run @code{make}. Then,
312 cd $LILYPOND_GIT/scripts/auxiliar/
313 ./make-regtest-pngs.sh -j9 -o
316 The @code{-j9} option tells the script to use 9 CPUs to create the
317 images - change this to your own CPU count+1. @code{-o} means this is the "old"
318 version. This will create images of all the regtests in
321 $LILYPOND_BUILD_DIR/out-png-check/old-regtest-results/
324 Now checkout the version you want to compare with the baseline. Run
325 @code{make} again to recreate the LilyPond binary. Then, do the following:
328 cd $LILYPOND_GIT/scripts/auxiliar/
329 ./make-regtest-pngs.sh -j9 -n
332 The @code{-n} option tells the script to make a "new" version of the
333 images. They are created in
336 $LILYPOND_BUILD_DIR/out-png-check/new-regtest-results/
339 Once the new images have been created, the script compares the old images with
340 the new ones pixel-by-pixel and prints a list of the different images to the
341 terminal, together with a count of how many differences were found. The
342 results of the checks are in
345 $LILYPOND_BUILD_DIR/out-png-check/regtest-diffs/
348 To check for differences, browse that directory with an image
349 viewer. Differences are shown in red. Be aware that some images with complex
350 fonts or spacing annotations always display a few minor differences. These can
354 @node Finding the cause of a regression
355 @section Finding the cause of a regression
357 Git has special functionality to help tracking down the exact
358 commit which causes a problem. See the git manual page for
359 @code{git bisect}. This is a job that non-programmers can do,
360 although it requires familiarity with git, ability to compile
361 LilyPond, and generally a fair amount of technical knowledge. A
362 brief summary is given below, but you may need to consult other
363 documentation for in-depth explanations.
365 Even if you are not familiar with git or are not able to compile
366 LilyPond you can still help to narrow down the cause of a
367 regression simply by downloading the binary releases of different
368 LilyPond versions and testing them for the regression. Knowing
369 which version of LilyPond first exhibited the regression is
370 helpful to a developer as it shortens the @code{git bisect}
373 Once a problematic commit is identified, the programmers' job is
374 much easier. In fact, for most regression bugs, the majority of
375 the time is spent simply finding the problematic commit.
377 More information is in @ref{Regression tests}.
379 @subheading git bisect setup
381 We need to set up the bisect for each problem we want to
384 Suppose we have an input file which compiled in version 2.13.32,
385 but fails in version 2.13.38 and above.
396 Give it the earliest known bad tag:
399 git bisect bad release/2.13.38-1
402 (you can see tags with: @code{git tag} )
405 Give it the latest known good tag:
408 git bisect good release/2.13.32-1
411 You should now see something like:
413 Bisecting: 195 revisions left to test after this (roughly 8 steps)
414 [b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
415 update for the new @qq{LilyPond Report}.
420 @subheading git bisect actual
432 Test your input file:
435 out/bin/lilypond test.ly
443 Does it crash, or is the output bad? If so:
450 Does your input file produce good output? If so:
459 Once the exact problem commit has been identified, git will inform
460 you with a message like:
463 6d28aebbaaab1be9961a00bf15a1ef93acb91e30 is the first bad commit
464 %%% ... blah blah blah ...
467 If there is still a range of commits, then git will automatically
468 select a new version for you to test. Go to step #1.
472 @subheading Recommendation: use two terminal windows
476 One window is open to the @code{build/} directory, and alternates
477 between these commands:
481 out/bin/lilypond test.ly
485 One window is open to the top source directory, and alternates
486 between these commands:
496 @node Memory and coverage tests
497 @section Memory and coverage tests
499 In addition to the graphical output of the regression tests, it is
500 possible to test memory usage and to determine how much of the source
501 code has been exercised by the tests.
503 @subheading Memory usage
505 For tracking memory usage as part of this test, you will need
506 GUILE CVS; especially the following patch:
508 @uref{http://lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.
511 @subheading Code coverage
513 For checking the coverage of the test suite, do the following
516 ./scripts/auxiliar/build-coverage.sh
517 @emph{# uncovered files, least covered first}
518 ./scripts/auxiliar/coverage.py --summary out-cov/*.cc
519 @emph{# consecutive uncovered lines, longest first}
520 ./scripts/auxiliar/coverage.py --uncovered out-cov/*.cc
525 @section MusicXML tests
528 LilyPond comes with a complete set of regtests for the
529 @uref{http://www.musicxml.org/,MusicXML} language. Originally
530 developed to test @samp{musicxml2ly}, these regression tests
531 can be used to test any MusicXML implementation.
533 The MusicXML regression tests are found at
534 @file{input/regression/musicxml/}.
536 The output resulting from running these tests
537 through @samp{musicxml2ly} followed by @samp{lilypond} is
538 available in the LilyPond documentation:
541 @uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}