* Introduction to regression tests::
* Precompiled regression tests::
* Compiling regression tests::
-* Identifying code regressions::
+* Regtest comparison::
+* Finding the cause of a regression::
* Memory and coverage tests::
* MusicXML tests::
@end menu
the functionality of each part of LilyPond.
Regression tests are added when new functionality is added to
-LilyPond. They are also added when bugs are identified. The
-snippet that causes the bug becomes a regression test to verify
-that the bug has been fixed.
+LilyPond.
+We do not yet have a policy on when it is appropriate to add or
+modify a regtest when bugs are fixed. Individual developers
+should use their best judgement until this is clarified during the
+@ref{Grand Organization Project (GOP)}.
The regression tests are compiled using special @code{make}
targets. There are three primary uses for the regression
@ref{Requirements for building documentation}.
-@node Identifying code regressions
-@section Identifying code regressions
+@node Regtest comparison
+@section Regtest comparison
Before modified code is committed to master, a regression test
comparison must be completed to ensure that the changes have
@enumerate
@item
-Before making changes, a baseline should be established by
-running:
+Run @code{make} with current git master without any of your changes.
+
+@item
+Before making changes to the code, establish a baseline for the comparison by
+going to the @file{lilypond-git/build/} directory and running:
@example
make test-baseline
@item
Make your changes, or apply the patch(es) to consider.
+@item
+Compile the source with @samp{make} as usual.
+
@item
Check for unintentional changes to the regtests:
@end example
After this has finished, a regression test comparison will be
-available at:
+available (relative to the current @file{build/} directory) at:
@example
out/test-results/index.html
If you want to continue programming, then make any additional code
changes, and continue.
+@item
+Compile the source with @samp{make} as usual.
+
@item
To re-check files that differed between the initial
@samp{make@tie{}test-baseline} and your post-changes
@end enumerate
+@advanced{
+Once a test baseline has been established, there is no need to run it again
+unless git master changed. In other words, if you work with several branches
+and want to do regtests comparison for all of them, you can
+@code{make test-baseline} with git master, checkout some branch,
+@code{make} and @code{make check} it, then switch to another branch,
+@code{make test-clean}, @code{make} and @code{make check} it without doing
+@code{make test-baseline} again.}
+
+
+@node Finding the cause of a regression
+@section Finding the cause of a regression
+
+Git has special functionality to help tracking down the exact
+commit which causes a problem. See the git manual page for
+@code{git bisect}. This is a job that non-programmers can do,
+although it requires familiarity with git, ability to compile
+LilyPond, and generally a fair amount of technical knowledge. A
+brief summary is given below, but you may need to consult other
+documentation for in-depth explanations.
+
+Even if you are not familiar with git or are not able to compile
+LilyPond you can still help to narrow down the cause of a
+regression simply by downloading the binary releases of different
+LilyPond versions and testing them for the regression. Knowing
+which version of LilyPond first exhibited the regression is
+helpful to a developer as it shortens the @code{git bisect}
+procedure.
+
+Once a problematic commit is identified, the programmers' job is
+much easier. In fact, for most regression bugs, the majority of
+the time is spent simply finding the problematic commit.
+
+More information is in @ref{Regression tests}.
+
+@subheading git bisect setup
+
+We need to set up the bisect for each problem we want to
+investigate.
+
+Suppose we have an input file which compiled in version 2.13.32,
+but fails in version 2.13.38 and above.
+
+@enumerate
+@item
+Begin the process:
+
+@example
+git bisect start
+@end example
+
+@item
+Give it the earliest known bad tag:
+
+@example
+git bisect bad release/2.13.38-1
+@end example
+
+(you can see tags with: @code{git tag} )
+
+@item
+Give it the latest known good tag:
+
+@example
+git bisect good release/2.13.32-1
+@end example
+
+You should now see something like:
+@example
+Bisecting: 195 revisions left to test after this (roughly 8 steps)
+[b17e2f3d7a5853a30f7d5a3cdc6b5079e77a3d2a] Web: Announcement
+update for the new @qq{LilyPond Report}.
+@end example
+
+@end enumerate
+
+@subheading git bisect actual
+
+@enumerate
+
+@item
+Compile the source:
+
+@example
+make
+@end example
+
+@item
+Test your input file:
+
+@example
+out/bin/lilypond test.ly
+@end example
+
+@item
+Test results?
+
+@itemize
+@item
+Does it crash, or is the output bad? If so:
+
+@example
+git bisect bad
+@end example
+
+@item
+Does your input file produce good output? If so:
+
+@example
+git bisect good
+@end example
+
+@end itemize
+
+@item
+Once the exact problem commit has been identified, git will inform
+you with a message like:
+
+@example
+6d28aebbaaab1be9961a00bf15a1ef93acb91e30 is the first bad commit
+%%% ... blah blah blah ...
+@end example
+
+If there is still a range of commits, then git will automatically
+select a new version for you to test. Go to step #1.
+
+@end enumerate
+
+@subheading Recommendation: use two terminal windows
+
+@itemize
+@item
+One window is open to the @code{build/} directory, and alternates
+between these commands:
+
+@example
+make
+out/bin/lilypond test.ly
+@end example
+
+@item
+One window is open to the top source directory, and alternates
+between these commands:
+
+@example
+git bisect good
+git bisect bad
+@end example
+
+@end itemize
+
@node Memory and coverage tests
@section Memory and coverage tests
For tracking memory usage as part of this test, you will need
GUILE CVS; especially the following patch:
+@smallexample
@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.
+@end smallexample
@subheading Code coverage
projects / lilypond.git / blobdiff