@menu
* Introduction to regression tests::
-* Current regtest output::
-* Comparison regtest output::
+* Precompiled regression tests::
+* Compiling regression tests::
+* Identifying code regressions::
+* Memory and coverage tests::
* MusicXML tests::
@end menu
snippet that causes the bug becomes a regression test to verify
that the bug has been fixed.
-The regression tests are automatically compiled using special @code{make}
-targets. The output of the regression tests is also automatically
-checked to identify changes in LilyPond output.
+The regression tests are compiled using special @code{make}
+targets. There are three primary uses for the regression
+tests. First, successful completion of the regression tests means
+that LilyPond has been properly built. Second, the output of the
+regression tests can be manually checked to ensure that
+the graphical output matches the description of the intended
+output. Third, the regression test output from two different
+versions of LilyPond can be automatically compared to identify
+any differences. These differences should then be manually
+checked to ensure that the differences are intended.
-The output of the regression tests is available on the website
-for every stable version of LilyPond. This allows the comparison
-of different versions to see when bugs appeared.
+Regression tests (@qq{regtests}) are available in precompiled form
+as part of the documentation. Regtests can also be compiled
+on any machine that has a properly configured LilyPond build
+system.
-@node Current regtest output
-@section Current regtest output
+@node Precompiled regression tests
+@section Precompiled regression tests
+@subheading Regression test output
-@node Comparison regtest output
-@section Comparison regtest output
+As part of the release process, the regression tests are run
+for every LilyPond release. Full regression test output is
+available for every stable version and the most recent development
+version.
+
+Regression test output is available in HTML and PDF format. Links
+to the regression test output are available at the developer's
+resources page for the version of interest.
+
+The latest stable version of the regtests is found at:
+
+@example
+@uref{http://lilypond.org/doc/stable/input/regression/collated-files.html}
+@end example
+
+The latest development version of the regtests is found at:
+
+@example
+@uref{http://lilypond.org/doc/latest/input/regression/collated-files.html}
+@end example
+
+
+@subheading Regression test comparison
+
+Each time a new version is released, the regtests are
+compiled and the output is automatically compared with the
+output of the previous release. The result of these
+comparisons is archived online:
+
+@example
+@uref{http://lilypond.org/test/}
+@end example
+
+Checking these pages is a very important task for the LilyPond project.
+You are invited to report anything that looks broken, or any case
+where the output quality is not on par with the previous release,
+as described in @rweb{Bug reports}.
+
+@subheading What to look for
+
+The test comparison shows all of the changes that occurred between
+the current release and the prior release. Each test that has a
+significant difference in output is displayed, with the old
+version on the left and the new version on the right.
+
+Regression tests whose output is the same for both versions are
+not shown in the test comparison.
+
+@itemize
+@item
+Images: green blurs in the new version show the approximate
+location of elements in the old version.
+
+There are often minor adjustments in spacing which do not indicate
+any problem.
+
+@item
+Log files: show the difference in command-line output.
+
+The main thing to examine are any changes in page counts -- if a
+file used to fit on 1 page but now requires 4 or 5 pages,
+something is suspicious!
+
+@item
+Profile files: give information about
+TODO? I don't know what they're for.
+
+@end itemize
+
+@warning{
+The automatic comparison of the regtests checks the LilyPond
+bounding boxes. This means that Ghostscript changes and changes
+in lyrics or text are not found.
+}
+
+@node Compiling regression tests
+@section Compiling regression tests
+
+Developers may wish to see the output of the complete regression
+test suite for the current version of the source repository
+between releases. Current source code is available; see
+@ref{Working with source code}. Then you will need
+to build the LilyPond binary; see @ref{Compiling LilyPond}.
+
+Uninstalling the previous LilyPond version is not necessary, nor is
+running @code{make install}, since the tests will automatically be
+compiled with the LilyPond binary you have just built in your source
+directory.
+
+From this point, the regtests are compiled with:
+
+@example
+make test
+@end example
+
+If you have a multi-core machine you may want to use the @option{-j}
+option and @var{CPU_COUT} variable, as
+described in @ref{Saving time with CPU_COUNT}.
+For a quad-core processor the complete command would be:
+
+@example
+make -j5 CPU_COUNT=5 test
+@end example
+
+The regtest output will then be available in
+@file{input/@/regression/@/out@/-test}.
+@file{input/@/regression/@/out@/-test/@/collated@/-examples@/.html}
+contains a listing of all the regression tests that were run,
+but none of the images are included. Individual images are
+also available in this directory.
+
+The primary use of @samp{make@tie{}test} is to verify that the
+regression tests all run without error. The regression test
+page that is part of the documentation is created only when the
+documentation is built, as described in @ref{Generating documentation}.
+Note that building the documentation requires more installed components
+than building the source code, as described in
+@ref{Requirements for building documentation}.
+
+
+@node Identifying code regressions
+@section Identifying code regressions
+
+Before modified code is committed to master, a regression test
+comparison must be completed to ensure that the changes have
+not caused problems with previously working code. The comparison
+is made automatically upon compiling the regression test suite
+twice.
+
+Before making changes, a baseline should be established by running:
+
+@example
+make test-baseline
+@end example
+
+After making the changes, the code should be checked by running:
+
+@example
+make check
+@end example
+
+After @samp{make@tie{}check} is complete, a regression test comparison
+will be available at @file{out/@/test@/-results/@/index@/.html}.
+For each regression test that differs between the baseline and the
+changed code, a regression test entry will displayed. Ideally, the
+only changes would be the changes that you were working on. If
+regressions are introduced, they must be fixed before committing the
+code.
+
+@warning{
+The special regression test @file{test@/-output@/-distance@/.ly} will always
+show up as a regression. This test changes each time it is run, and
+serves to verify that the regression tests have, in fact, run.}
+
+Once @samp{make@tie{}test-baseline} and @samp{make@tie{}check} have been
+run, the files that differ between @samp{test-baseline} and @samp{check}
+can be repeatedly examined by doing:
+
+@example
+make test-redo
+@end example
+
+This updates the regression list at @file{out/@/test@/-results/@/index@/.html}.
+It does @emph{not} redo @file{test@/-output@/-distance@/.ly}.
+
+When all regressions have been resolved, the output list will be empty.
+
+Once all regressions have been resolved, a final check should be completed
+by running:
+
+@example
+make test-clean
+make check
+@end example
+
+This cleans the results of the previous @samp{make@tie{}check}, then does the
+automatic regression comparison again.
+
+
+@node Memory and coverage tests
+@section Memory and coverage tests
+
+In addition to the graphical output of the regression tests, it is
+possible to test memory usage and to determine how much of the source
+code has been exercised by the tests.
+
+@subheading Memory usage
+
+For tracking memory usage as part of this test, you will need
+GUILE CVS; especially the following patch:
+@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.
+
+@subheading Code coverage
+
+For checking the coverage of the test suite, do the following
+
+@example
+./scripts/auxiliar/build-coverage.sh
+@emph{# uncovered files, least covered first}
+./scripts/auxiliar/coverage.py --summary out-cov/*.cc
+@emph{# consecutive uncovered lines, longest first}
+./scripts/auxiliar/coverage.py --uncovered out-cov/*.cc
+@end example
@node MusicXML tests
@section MusicXML tests
+LilyPond comes with a complete set of regtests for the
+@uref{http://www.musicxml.org/,MusicXML} language. Originally
+developed to test @samp{musicxml2ly}, these regression tests
+can be used to test any MusicXML implementation.
+
+The MusicXML regression tests are found at
+@file{input/@/regression/@/musicxml/}.
+
+The output resulting from running these tests
+through @samp{muscxml2ly} followed by @samp{lilypond} is
+available in the LilyPond documentation:
+
+@example
+@uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}
+@end example
+
projects / lilypond.git / blobdiff