* Finding the cause of a regression::
* Memory and coverage tests::
* MusicXML tests::
+* Grand Regression Test Checking::
@end menu
The test comparison shows all of the changes that occurred between
the current release and the prior release. Each test that has a
-significant difference in output is displayed, with the old
-version on the left and the new version on the right.
+significant (noticeable) difference in output is displayed, with
+the old version on the left and the new version on the right.
+
+Some of the small changes can be ignored (slightly different slur
+shapes, small variations in note spacing), but this is not always
+the case: sometimes even the smallest change means that something
+is wrong. To help in distinguishing these cases, we use bigger
+staff size when small differences matter.
+
+Staff size 30 generally means "pay extra attention to details".
+Staff size 40 (two times bigger than default size) or more means
+that the regtest @strong{is} about the details.
+
+Staff size smaller than default doesn't mean anything.
Regression tests whose output is the same for both versions are
not shown in the test comparison.
Developers may wish to see the output of the complete regression
test suite for the current version of the source repository
between releases. Current source code is available; see
-@ref{Working with source code}. Then you will need
+@ref{Working with source code}.
+
+For regression testing @code{../configure} should be run with the
+@code{--disable-optimising} option. Then you will need
to build the LilyPond binary; see @ref{Compiling LilyPond}.
Uninstalling the previous LilyPond version is not necessary, nor is
@end example
If you have a multi-core machine you may want to use the @option{-j}
-option and @var{CPU_COUT} variable, as
+option and @var{CPU_COUNT} variable, as
described in @ref{Saving time with CPU_COUNT}.
For a quad-core processor the complete command would be:
@node Regtest comparison
@section Regtest comparison
-Before modified code is committed to master, a regression test
+Before modified code is committed to @code{master} (via @code{staging}),
+a regression test
comparison must be completed to ensure that the changes have
not caused problems with previously working code. The comparison
is made automatically upon compiling the regression test suite
@enumerate
@item
-Before making changes, a baseline should be established by
-running:
+Run @code{make} with current git master without any of your changes.
+
+@item
+Before making changes to the code, establish a baseline for the comparison by
+going to the @file{lilypond-git/build/} directory and running:
@example
make test-baseline
@end example
After this has finished, a regression test comparison will be
-available at:
+available (relative to the current @file{build/} directory) at:
@example
out/test-results/index.html
@end example
For each regression test that differs between the baseline and the
-changed code, a regression test entry will displayed. Ideally,
+changed code, a regression test entry will be displayed. Ideally,
the only changes would be the changes that you were working on.
If regressions are introduced, they must be fixed before
committing the code.
@end enumerate
+@advanced{
+Once a test baseline has been established, there is no need to run it again
+unless git master changed. In other words, if you work with several branches
+and want to do regtests comparison for all of them, you can
+@code{make test-baseline} with git master, checkout some branch,
+@code{make} and @code{make check} it, then switch to another branch,
+@code{make test-clean}, @code{make} and @code{make check} it without doing
+@code{make test-baseline} again.}
+
@node Finding the cause of a regression
@section Finding the cause of a regression
For tracking memory usage as part of this test, you will need
GUILE CVS; especially the following patch:
+@smallexample
@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}.
+@end smallexample
@subheading Code coverage
@file{input/regression/musicxml/}.
The output resulting from running these tests
-through @samp{muscxml2ly} followed by @samp{lilypond} is
+through @samp{musicxml2ly} followed by @samp{lilypond} is
available in the LilyPond documentation:
@example
@uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}
@end example
+
+@node Grand Regression Test Checking
+@section Grand Regression Test Checking
+
+@subheading What is this all about?
+
+Regression tests (usually abbreviated "regtests") is a collection
+of @file{.ly} files used to check whether LilyPond is working correctly.
+Example: before version 2.15.12 breve noteheads had incorrect width,
+which resulted in collisions with other objects. After the issue was fixed,
+a small @file{.ly} file demonstrating the problem was added to the regression
+tests as a proof that the fix works. If someone will accidentally break
+breve width again, we will notice this in the output of that regression test.
+
+@subheading How can I help?
+
+We ask you to help us by checking one or two regtests from time to time.
+You don't need programming skills to do this, not even LilyPond skills -
+just basic music notation knowledge; checking one regtest takes less than
+a minute. Simply go here:
+
+@example
+@uref{http://www.philholmes.net/lilypond/regtests/}
+@end example
+
+@subheading Some tips on checking regtests
+
+@subsubheading Description text
+
+The description should be clear even for a music beginner.
+If there are any special terms used in the description,
+they all should be explained in our @rglosnamed{Top, Music Glossary}
+or @rinternalsnamed{Top, Internals Reference}.
+Vague descriptions (like "behaves well", "looks reasonable") shouldn't be used.
+
+@ignore
+this may be useful for advanced regtest checking
+@subsubheading Is regtest straightforward and systematic?
+
+Unfortunately some regtests are written poorly. A good regtest should be
+straightforward: it should be obvious what it checks and how. Also, it
+usually shouldn't check everything at once. For example it's a bad idea to test
+accidental placement by constucting one huge chord with many suspended notes
+and loads of accidentals. It's better to divide such problem into a series
+of clearly separated cases.
+@end ignore
projects / lilypond.git / blobdiff