@c -*- coding: utf-8; mode: texinfo; -*-
@node Regression tests
@chapter Regression tests

@menu
* Introduction to regression tests::
* Precompiled regression tests::
* Compiling regression tests::
* Identifying code regressions::
* Memory and coverage tests::
* MusicXML tests::
@end menu


@node Introduction to regression tests
@section Introduction to regression tests

LilyPond has a complete suite of regression tests that are used
to ensure that changes to the code do not break existing behavior.
These regression tests comprise small LilyPond snippets that test
the functionality of each part of LilyPond.

Regression tests are added when new functionality is added to
LilyPond.  They are also added when bugs are identified.  The
snippet that causes the bug becomes a regression test to verify
that the bug has been fixed.

The regression tests are compiled using special @code{make}
targets.  There are three primary uses for the regression
tests.  First, successful completion of the regression tests means
that LilyPond has been properly built.  Second, the output of the
regression tests can be manually checked to ensure that
the graphical output matches the description of the intended
output.  Third, the regression test output from two different
versions of LilyPond can be automatically compared to identify
any differences.  These differences should then be manually
checked to ensure that the differences are intended.

Regression tests (@qq{regtests}) are available in precompiled form
as part of the documentation.  Regtests can also be compiled
on any machine that has a properly configured LilyPond build
system.


@node Precompiled regression tests
@section Precompiled regression tests

@subheading Regression test output

As part of the release process, the regression tests are run
for every LilyPond release.  Full regression test output is
available for every stable version and the most recent development
version.

Regression test output is available in HTML and PDF format.  Links
to the regression test output are available at the developer's
resources page for the version of interest.

The latest stable version of the regtests is found at:

@example
@uref{http://lilypond.org/doc/stable/input/regression/collated-files.html}
@end example

The latest development version of the regtests is found at:

@example
@uref{http://lilypond.org/doc/latest/input/regression/collated-files.html}
@end example


@subheading Regression test comparison

Each time a new version is released, the regtests are
compiled and the output is automatically compared with the
output of the previous release.  The result of these
comparisons is archived online:

@example
@uref{http://lilypond.org/test/}
@end example

The test comparison shows all of the changes that occured between
the current release and the prior release.  Each test that has
a significant difference in output is displayed, with the old
version on the left and the new version on the right.  Blurs
in the regression tests for the new version show the approximate
location of elements in the old version.

Regression tests whose output is the same for both versions are not
shown in the test comparison.

Checking these pages is a very important task for the LilyPond project.
You are invited to report anything that looks broken, or any case
where the output quality is not on par with the previous release,
as described in @rweb{Bug reports}.

@warning{
The automatic comparison of the regtests checks the
LilyPond bounding boxes and the log files.  This means that
Ghostscript changes and changes in
lyrics or text are not found.  However, errors and warnings that
are printed to the log file but
do not change the graphical layout are also identified.
}

@node Compiling regression tests
@section Compiling regression tests

Developers may wish to see the output of the complete regression
test suite for the current version of the source repository
between releases.  Current source code is available; see
@ref{Working with source code}.  Then you will need
to build the LilyPond binary; see @ref{Compiling LilyPond}.

Uninstalling the previous LilyPond version is not necessary, nor is
running @code{make install}, since the tests will automatically be
compiled with the LilyPond binary you have just built in your source
directory.

From this point, the regtests are compiled with:

@example
make test
@end example

If you have a multi-core machine you may want to use the @option{-j}
option and @var{CPU_COUT} variable, as
described in @ref{Saving time with CPU_COUNT}.
For a quad-core processor the complete command would be:

@example
make -j5 CPU_COUNT=5 test
@end example

The regtest output will then be available in
@file{input/regression/out-test}.
@file{input/regression/out-test/collated-examples.html}
contains a listing of all the regression tests that were run,
but none of the images are included.  Individual images are
also available in this directory.

The primary use of @samp{make@tie{}test} is to verify that the
regression tests all run without error.  The regression test
page that is part of the documentation is created only when the
documentation is built, as described in @ref{Generating documentation}.
Note that building the documentation requires more installed components
than building the source code, as described in
@ref{Requirements for building documentation}.


@node Identifying code regressions
@section Identifying code regressions

Before modified code is committed to master, a regression test
comparision must be completed to ensure that the changes have
not caused problems with previously working code.  The comparison
is made automatically upon compiling the regression test suite
twice.

Before making changes, a baseline should be established by running:

@example
make test-baseline
@end example

After making the changes, the code should be checked by running:

@example
make check
@end example

After @samp{make@tie{}check} is complete, a regression test comparison
will be available at @file{out/test-results/index.html}.
For each regression test that differs between the baseline and the
changed code, a regression test entry will displayed.  Ideally, the
only changes would be the changes that you were working on.  If
regressions are introduced, they must be fixed before committing the
code.

@warning{
The special regression test @file{test-output-distance.ly} will always
show up as a regression.  This test changes each time it is run, and
serves to verify that the regression tests have, in fact, run.}

Once @samp{make@tie{}test-baseline} and @samp{make@tie{}check} have been
run, the files that differ between @samp{test-baseline} and @samp{check}
can be repeatedly examined by doing:

@example
make test-redo
@end example

This updates the regression list at @file{out/test-results/index.html}.
It does @emph{not} redo @file{test-output-distance.ly}.

When all regressions have been resolved, the output list will be empty.

Once all regressions have been resolved, a final check should be completed
by running:

@example
make test-clean
make check
@end example

This cleans the results of the previous @samp{make@tie{}check}, then does the
automatic regression comparison again.  


@node Memory and coverage tests
@section Memory and coverage tests

In addition to the graphical output of the regression tests, it is
possible to test memory usage and to determine how much of the source
code has been exercised by the tests.

@subheading Memory usage

For tracking memory usage as part of this test, you will need
GUILE CVS; especially the following patch:
@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.

@subheading Code coverage

For checking the coverage of the test suite, do the following

@example
./scripts/auxiliar/build-coverage.sh
@emph{# uncovered files, least covered first}
./scripts/auxiliar/coverage.py  --summary out-cov/*.cc
@emph{# consecutive uncovered lines, longest first}
./scripts/auxiliar/coverage.py  --uncovered out-cov/*.cc
@end example


@node MusicXML tests
@section MusicXML tests


LilyPond comes with a complete set of regtests for the
@uref{http://www.musicxml.org/,MusicXML} language.  Originally
developed to test @samp{musicxml2ly}, these regression tests
can be used to test any MusicXML implementation.

The MusicXML regression tests are found at
@file{input/regression/musicxml/}.

The output resulting from running these tests
through @samp{muscxml2ly} followed by @samp{lilypond} is
available in the LilyPond documentation:

@example
@uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}
@end example