* Introduction to regression tests::
* Precompiled regression tests::
* Compiling regression tests::
-* Identifying code regressions::
+* Regtest comparison::
+* Finding the cause of a regression::
* Memory and coverage tests::
* MusicXML tests::
@end menu
where the output quality is not on par with the previous release,
as described in @rweb{Bug reports}.
+@warning{ The special regression test
+@file{test-output-distance.ly} will always show up as a
+regression. This test changes each time it is run, and serves to
+verify that the regression tests have, in fact, run.}
+
+
@subheading What to look for
The test comparison shows all of the changes that occurred between
@end example
The regtest output will then be available in
-@file{input/@/regression/@/out@/-test}.
-@file{input/@/regression/@/out@/-test/@/collated@/-examples@/.html}
+@file{input/regression/out-test}.
+@file{input/regression/out-test/collated-examples.html}
contains a listing of all the regression tests that were run,
but none of the images are included. Individual images are
also available in this directory.
@ref{Requirements for building documentation}.
-@node Identifying code regressions
-@section Identifying code regressions
+@node Regtest comparison
+@section Regtest comparison
Before modified code is committed to master, a regression test
comparison must be completed to ensure that the changes have
is made automatically upon compiling the regression test suite
twice.
-Before making changes, a baseline should be established by running:
+@enumerate
+
+@item
+Before making changes, a baseline should be established by
+running:
@example
make test-baseline
@end example
-After making the changes, the code should be checked by running:
+@item
+Make your changes, or apply the patch(es) to consider.
+
+@item
+Compile the source with @samp{make} as usual.
+
+@item
+Check for unintentional changes to the regtests:
@example
make check
@end example
-After @samp{make@tie{}check} is complete, a regression test comparison
-will be available at @file{out/@/test@/-results/@/index@/.html}.
+After this has finished, a regression test comparison will be
+available at:
+
+@example
+out/test-results/index.html
+@end example
+
For each regression test that differs between the baseline and the
-changed code, a regression test entry will displayed. Ideally, the
-only changes would be the changes that you were working on. If
-regressions are introduced, they must be fixed before committing the
-code.
+changed code, a regression test entry will displayed. Ideally,
+the only changes would be the changes that you were working on.
+If regressions are introduced, they must be fixed before
+committing the code.
@warning{
-The special regression test @file{test@/-output@/-distance@/.ly} will always
+The special regression test @file{test-output-distance.ly} will always
show up as a regression. This test changes each time it is run, and
serves to verify that the regression tests have, in fact, run.}
-Once @samp{make@tie{}test-baseline} and @samp{make@tie{}check} have been
-run, the files that differ between @samp{test-baseline} and @samp{check}
-can be repeatedly examined by doing:
+@item
+If you are happy with the results, then stop now.
+
+If you want to continue programming, then make any additional code
+changes, and continue.
+
+@item
+Compile the source with @samp{make} as usual.
+
+@item
+To re-check files that differed between the initial
+@samp{make@tie{}test-baseline} and your post-changes
+@samp{make@tie{}check}, run:
@example
make test-redo
@end example
-This updates the regression list at @file{out/@/test@/-results/@/index@/.html}.
-It does @emph{not} redo @file{test@/-output@/-distance@/.ly}.
+This updates the regression list at @file{out/test-results/index.html}.
+It does @emph{not} redo @file{test-output-distance.ly}.
+@item
When all regressions have been resolved, the output list will be empty.
+@item
Once all regressions have been resolved, a final check should be completed
by running:
This cleans the results of the previous @samp{make@tie{}check}, then does the
automatic regression comparison again.
+@end enumerate
+
+
+@node Finding the cause of a regression
+@section Finding the cause of a regression
+
+Git has special functionality to help tracking down the exact
+commit which causes a problem. See the git manual page for
+@code{git bisect}. This is a job that non-programmers can do,
+although it requires familiarity with git, ability to compile
+LilyPond, and generally a fair amount of technical knowledge. A
+brief summary is given below, but you may need to consult other
+documentation for in-depth explanations.
+
+Even if you are not familiar with git or are not able to compile
+LilyPond you can still help to narrow down the cause of a
+regression simply by downloading the binary releases of different
+LilyPond versions and testing them for the regression. Knowing
+which version of LilyPond first exhibited the regression is
+helpful to a developer as it shortens the @code{git bisect}
+procedure.
+
+Once a problematic commit is identified, the programmers' job is
+much easier. In fact, for most regression bugs, the majority of
+the time is spent simply finding the problematic commit.
+
+More information is in @ref{Regression tests}.
+
+@subheading git bisect setup
+
+We need to set up the bisect for each problem we want to
+investigate.
+
+Suppose we have an input file which compiled in version 2.13.32,
+but fails in version 2.13.38 and above.
+
+@enumerate
+@item
+Begin the process:
+
+@example
+git bisect start
+@end example
+
+@item
+Give it the earliest known bad tag:
+
+@example
+git bisect bad release/2.13.38-1
+@end example
+
+(you can see tags with: @code{git tag} )
+
+@item
+Give it the latest known good tag:
+
+@example
+git bisect good release/2.13.32-1
+@end example
+
+You should now see something like:
+@example
+Bisecting: 195 revisions left to test after this (roughly 8 steps)
+[b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
+update for the new @qq{LilyPond Report}.
+@end example
+
+@end enumerate
+
+@subheading git bisect actual
+
+@enumerate
+
+@item
+Compile the source:
+
+@example
+make
+@end example
+
+@item
+Test your input file:
+
+@example
+out/bin/lilypond test.ly
+@end example
+
+@item
+Test results?
+
+@itemize
+@item
+Does it crash, or is the output bad? If so:
+
+@example
+git bisect bad
+@end example
+
+@item
+Does your input file produce good output? If so:
+
+@example
+git bisect good
+@end example
+
+@end itemize
+
+@item
+Once the exact problem commit has been identified, git will inform
+you with a message like:
+
+@example
+6d28aebbaaab1be9961a00bf15a1ef93acb91e30 is the first bad commit
+%%% ... blah blah blah ...
+@end example
+
+If there is still a range of commits, then git will automatically
+select a new version for you to test. Go to step #1.
+
+@end enumerate
+
+@subheading Recommendation: use two terminal windows
+
+@itemize
+@item
+One window is open to the @code{build/} directory, and alternates
+between these commands:
+
+@example
+make
+out/bin/lilypond test.ly
+@end example
+
+@item
+One window is open to the top source directory, and alternates
+between these commands:
+
+@example
+git bisect good
+git bisect bad
+@end example
+
+@end itemize
+
@node Memory and coverage tests
@section Memory and coverage tests
can be used to test any MusicXML implementation.
The MusicXML regression tests are found at
-@file{input/@/regression/@/musicxml/}.
+@file{input/regression/musicxml/}.
The output resulting from running these tests
through @samp{muscxml2ly} followed by @samp{lilypond} is
projects / lilypond.git / blobdiff