* Introduction to regression tests::
* Precompiled regression tests::
* Compiling regression tests::
-* Identifying code regressions::
+* Regtest comparison::
+* Finding the cause of a regression::
* Memory and coverage tests::
* MusicXML tests::
@end menu
@uref{http://lilypond.org/test/}
@end example
-The test comparison shows all of the changes that occured between
-the current release and the prior release. Each test that has
-a significant difference in output is displayed, with the old
-version on the left and the new version on the right. Blurs
-in the regression tests for the new version show the approximate
-location of elements in the old version.
-
-Regression tests whose output is the same for both versions are not
-shown in the test comparison.
-
Checking these pages is a very important task for the LilyPond project.
You are invited to report anything that looks broken, or any case
where the output quality is not on par with the previous release,
as described in @rweb{Bug reports}.
+@warning{ The special regression test
+@file{test-output-distance.ly} will always show up as a
+regression. This test changes each time it is run, and serves to
+verify that the regression tests have, in fact, run.}
+
+
+@subheading What to look for
+
+The test comparison shows all of the changes that occurred between
+the current release and the prior release. Each test that has a
+significant difference in output is displayed, with the old
+version on the left and the new version on the right.
+
+Regression tests whose output is the same for both versions are
+not shown in the test comparison.
+
+@itemize
+@item
+Images: green blurs in the new version show the approximate
+location of elements in the old version.
+
+There are often minor adjustments in spacing which do not indicate
+any problem.
+
+@item
+Log files: show the difference in command-line output.
+
+The main thing to examine are any changes in page counts -- if a
+file used to fit on 1 page but now requires 4 or 5 pages,
+something is suspicious!
+
+@item
+Profile files: give information about
+TODO? I don't know what they're for.
+
+@end itemize
+
@warning{
-The automatic comparison of the regtests checks the
-LilyPond bounding boxes and the log files. This means that
-Ghostscript changes and changes in
-lyrics or text are not found. However, errors and warnings that
-are printed to the log file but
-do not change the graphical layout are also identified.
+The automatic comparison of the regtests checks the LilyPond
+bounding boxes. This means that Ghostscript changes and changes
+in lyrics or text are not found.
}
-Periodic manual checking of the
-complete regtest output should be performed to catch anything
-missed by the automatic comparison.}
-
@node Compiling regression tests
@section Compiling regression tests
@ref{Requirements for building documentation}.
-@node Identifying code regressions
-@section Identifying code regressions
+@node Regtest comparison
+@section Regtest comparison
Before modified code is committed to master, a regression test
-comparision must be completed to ensure that the changes have
+comparison must be completed to ensure that the changes have
not caused problems with previously working code. The comparison
is made automatically upon compiling the regression test suite
twice.
-Before making changes, a baseline should be established by running:
+@enumerate
+
+@item
+Before making changes, a baseline should be established by
+running:
@example
make test-baseline
@end example
-After making the changes, the code should be checked by running:
+@item
+Make your changes, or apply the patch(es) to consider.
+
+@item
+Compile the source with @samp{make} as usual.
+
+@item
+Check for unintentional changes to the regtests:
@example
make check
@end example
-After @samp{make@tie{}check} is complete, a regression test comparison
-will be available at @file{out/test-results/index.html}.
+After this has finished, a regression test comparison will be
+available at:
+
+@example
+out/test-results/index.html
+@end example
+
For each regression test that differs between the baseline and the
-changed code, a regression test entry will displayed. Ideally, the
-only changes would be the changes that you were working on. If
-regressions are introduced, they must be fixed before committing the
-code.
+changed code, a regression test entry will displayed. Ideally,
+the only changes would be the changes that you were working on.
+If regressions are introduced, they must be fixed before
+committing the code.
@warning{
The special regression test @file{test-output-distance.ly} will always
show up as a regression. This test changes each time it is run, and
serves to verify that the regression tests have, in fact, run.}
+@item
+If you are happy with the results, then stop now.
+
+If you want to continue programming, then make any additional code
+changes, and continue.
+
+@item
+Compile the source with @samp{make} as usual.
+
+@item
+To re-check files that differed between the initial
+@samp{make@tie{}test-baseline} and your post-changes
+@samp{make@tie{}check}, run:
+
+@example
+make test-redo
+@end example
+
+This updates the regression list at @file{out/test-results/index.html}.
+It does @emph{not} redo @file{test-output-distance.ly}.
+
+@item
+When all regressions have been resolved, the output list will be empty.
+
+@item
+Once all regressions have been resolved, a final check should be completed
+by running:
+
+@example
+make test-clean
+make check
+@end example
+
+This cleans the results of the previous @samp{make@tie{}check}, then does the
+automatic regression comparison again.
+
+@end enumerate
+
+
+@node Finding the cause of a regression
+@section Finding the cause of a regression
+
+Git has special functionality to help tracking down the exact
+commit which causes a problem. See the git manual page for
+@code{git bisect}. This is a job that non-programmers can do,
+although it requires familiarity with git, ability to compile
+LilyPond, and generally a fair amount of technical knowledge. A
+brief summary is given below, but you may need to consult other
+documentation for in-depth explanations.
+
+Even if you are not familiar with git or are not able to compile
+LilyPond you can still help to narrow down the cause of a
+regression simply by downloading the binary releases of different
+LilyPond versions and testing them for the regression. Knowing
+which version of LilyPond first exhibited the regression is
+helpful to a developer as it shortens the @code{git bisect}
+procedure.
+
+Once a problematic commit is identified, the programmers' job is
+much easier. In fact, for most regression bugs, the majority of
+the time is spent simply finding the problematic commit.
+
+More information is in @ref{Regression tests}.
+
+@subheading git bisect setup
+
+We need to set up the bisect for each problem we want to
+investigate.
+
+Suppose we have an input file which compiled in version 2.13.32,
+but fails in version 2.13.38 and above.
+
+@enumerate
+@item
+Begin the process:
+
+@example
+git bisect start
+@end example
+
+@item
+Give it the earliest known bad tag:
+
+@example
+git bisect bad release/2.13.38-1
+@end example
+
+(you can see tags with: @code{git tag} )
+
+@item
+Give it the latest known good tag:
+
+@example
+git bisect good release/2.13.32-1
+@end example
+
+You should now see something like:
+@example
+Bisecting: 195 revisions left to test after this (roughly 8 steps)
+[b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
+update for the new @qq{LilyPond Report}.
+@end example
+
+@end enumerate
+
+@subheading git bisect actual
+
+@enumerate
+
+@item
+Compile the source:
+
+@example
+make
+@end example
+
+@item
+Test your input file:
+
+@example
+out/bin/lilypond test.ly
+@end example
+
+@item
+Test results?
+
+@itemize
+@item
+Does it crash, or is the output bad? If so:
+
+@example
+git bisect bad
+@end example
+
+@item
+Does your input file produce good output? If so:
+
+@example
+git bisect good
+@end example
+
+@end itemize
+
+@item
+Once the exact problem commit has been identified, git will inform
+you with a message like:
+
+@example
+6d28aebbaaab1be9961a00bf15a1ef93acb91e30 is the first bad commit
+%%% ... blah blah blah ...
+@end example
+
+If there is still a range of commits, then git will automatically
+select a new version for you to test. Go to step #1.
+
+@end enumerate
+
+@subheading Recommendation: use two terminal windows
+
+@itemize
+@item
+One window is open to the @code{build/} directory, and alternates
+between these commands:
+
+@example
+make
+out/bin/lilypond test.ly
+@end example
+
+@item
+One window is open to the top source directory, and alternates
+between these commands:
+
+@example
+git bisect good
+git bisect bad
+@end example
+
+@end itemize
+
+
@node Memory and coverage tests
@section Memory and coverage tests
projects / lilypond.git / blobdiff