+After this has finished, a regression test comparison will be
+available (relative to the current @file{build/} directory) at:
+
+@example
+out/test-results/index.html
+@end example
+
+For each regression test that differs between the baseline and the
+changed code, a regression test entry will be displayed. Ideally,
+the only changes would be the changes that you were working on.
+If regressions are introduced, they must be fixed before
+committing the code.
+
+@warning{
+The special regression test @file{test-output-distance.ly} will always
+show up as a regression. This test changes each time it is run, and
+serves to verify that the regression tests have, in fact, run.}
+
+@item
+If you are happy with the results, then stop now.
+
+If you want to continue programming, then make any additional code
+changes, and continue.
+
+@item
+Compile the source with @samp{make} as usual.
+
+@item
+To re-check files that differed between the initial
+@samp{make@tie{}test-baseline} and your post-changes
+@samp{make@tie{}check}, run:
+
+@example
+make test-redo
+@end example
+
+This updates the regression list at @file{out/test-results/index.html}.
+It does @emph{not} redo @file{test-output-distance.ly}.
+
+@item
+When all regressions have been resolved, the output list will be empty.
+
+@item
+Once all regressions have been resolved, a final check should be completed
+by running:
+
+@example
+make test-clean
+make check
+@end example
+
+This cleans the results of the previous @samp{make@tie{}check}, then does the
+automatic regression comparison again.
+
+@end enumerate
+
+@advanced{
+Once a test baseline has been established, there is no need to run it again
+unless git master changed. In other words, if you work with several branches
+and want to do regtests comparison for all of them, you can
+@code{make test-baseline} with git master, checkout some branch,
+@code{make} and @code{make check} it, then switch to another branch,
+@code{make test-clean}, @code{make} and @code{make check} it without doing
+@code{make test-baseline} again.}
+
+@node Pixel-based regtest comparison
+@section Pixel-based regtest comparison
+
+As an alternative to the @code{make test} method for regtest checking (which
+relies upon @code{.signature} files created by a LilyPond run and which describe
+the placing of grobs) there is a script which compares the output of two
+LilyPond versions pixel-by-pixel. To use this, start by checking out the
+version of LilyPond you want to use as a baseline, and run @code{make}. Then,
+do the following:
+
+@example
+cd $LILYPOND_GIT/scripts/auxiliar/
+./make-regtest-pngs.sh -j9 -o
+@end example
+
+The @code{-j9} option tells the script to use 9 CPUs to create the
+images - change this to your own CPU count+1. @code{-o} means this is the "old"
+version. This will create images of all the regtests in
+
+@example
+$LILYPOND_BUILD_DIR/out-png-check/old-regtest-results/
+@end example
+
+Now checkout the version you want to compare with the baseline. Run
+@code{make} again to recreate the LilyPond binary. Then, do the following:
+
+@example
+cd $LILYPOND_GIT/scripts/auxiliar/
+./make-regtest-pngs.sh -j9 -n
+@end example
+
+The @code{-n} option tells the script to make a "new" version of the
+images. They are created in
+
+@example
+$LILYPOND_BUILD_DIR/out-png-check/new-regtest-results/
+@end example
+
+Once the new images have been created, the script compares the old images with
+the new ones pixel-by-pixel and prints a list of the different images to the
+terminal, together with a count of how many differences were found. The
+results of the checks are in
+
+@example
+$LILYPOND_BUILD_DIR/out-png-check/regtest-diffs/
+@end example
+
+To check for differences, browse that directory with an image
+viewer. Differences are shown in red. Be aware that some images with complex
+fonts or spacing annotations always display a few minor differences. These can
+safely be ignored.
+
+
+@node Finding the cause of a regression
+@section Finding the cause of a regression
+
+Git has special functionality to help tracking down the exact
+commit which causes a problem. See the git manual page for
+@code{git bisect}. This is a job that non-programmers can do,
+although it requires familiarity with git, ability to compile
+LilyPond, and generally a fair amount of technical knowledge. A
+brief summary is given below, but you may need to consult other
+documentation for in-depth explanations.
+
+Even if you are not familiar with git or are not able to compile
+LilyPond you can still help to narrow down the cause of a
+regression simply by downloading the binary releases of different
+LilyPond versions and testing them for the regression. Knowing
+which version of LilyPond first exhibited the regression is
+helpful to a developer as it shortens the @code{git bisect}
+procedure.
+
+Once a problematic commit is identified, the programmers' job is
+much easier. In fact, for most regression bugs, the majority of
+the time is spent simply finding the problematic commit.
+
+More information is in @ref{Regression tests}.
+
+@subheading git bisect setup
+
+We need to set up the bisect for each problem we want to
+investigate.
+
+Suppose we have an input file which compiled in version 2.13.32,
+but fails in version 2.13.38 and above.
+
+@enumerate
+@item
+Begin the process:
+
+@example
+git bisect start
+@end example
+
+@item
+Give it the earliest known bad tag:
+
+@example
+git bisect bad release/2.13.38-1
+@end example
+
+(you can see tags with: @code{git tag} )
+
+@item
+Give it the latest known good tag:
+
+@example
+git bisect good release/2.13.32-1
+@end example
+
+You should now see something like:
+@example
+Bisecting: 195 revisions left to test after this (roughly 8 steps)
+[b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
+update for the new @qq{LilyPond Report}.
+@end example
+
+@end enumerate
+
+@subheading git bisect actual
+
+@enumerate
+
+@item
+Compile the source:
+
+@example
+make
+@end example
+
+@item
+Test your input file:
+
+@example
+out/bin/lilypond test.ly
+@end example
+
+@item
+Test results?
+
+@itemize
+@item
+Does it crash, or is the output bad? If so:
+
+@example
+git bisect bad
+@end example
+
+@item
+Does your input file produce good output? If so:
+
+@example
+git bisect good
+@end example
projects / lilypond.git / blobdiff