* Finding the cause of a regression::
* Memory and coverage tests::
* MusicXML tests::
+* Grand Regression Test Checking::
@end menu
The test comparison shows all of the changes that occurred between
the current release and the prior release. Each test that has a
-significant difference in output is displayed, with the old
-version on the left and the new version on the right.
+significant (noticeable) difference in output is displayed, with
+the old version on the left and the new version on the right.
+
+Some of the small changes can be ignored (slightly different slur
+shapes, small variations in note spacing), but this is not always
+the case: sometimes even the smallest change means that something
+is wrong. To help in distinguishing these cases, we use bigger
+staff size when small differences matter.
+
+Staff size 30 generally means "pay extra attention to details".
+Staff size 40 (two times bigger than default size) or more means
+that the regtest @strong{is} about the details.
+
+Staff size smaller than default doesn't mean anything.
Regression tests whose output is the same for both versions are
not shown in the test comparison.
@node Regtest comparison
@section Regtest comparison
-Before modified code is committed to master, a regression test
+Before modified code is committed to @code{master} (via @code{staging}),
+a regression test
comparison must be completed to ensure that the changes have
not caused problems with previously working code. The comparison
is made automatically upon compiling the regression test suite
@uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files}
@end example
+
+@node Grand Regression Test Checking
+@section Grand Regression Test Checking
+
+@subheading What is this all about?
+
+Regression tests (usually abbreviated "regtests") is a collection
+of @file{.ly} files used to check whether LilyPond is working correctly.
+Example: before version 2.15.12 breve noteheads had incorrect width,
+which resulted in collisions with other objects. After the issue was fixed,
+a small @file{.ly} file demonstrating the problem was added to the regression
+tests as a proof that the fix works. If someone will accidentally break
+breve width again, we will notice this in the output of that regression test.
+
+@subheading How can I help?
+
+We ask you to help us by checking one or two regtests from time to time.
+You don't need programming skills to do this, not even LilyPond skills -
+just basic music notation knowledge; checking one regtest takes less than
+a minute. Simply go here:
+
+@example
+@uref{http://www.philholmes.net/lilypond/regtests/}
+@end example
+
+@subheading Some tips on checking regtests
+
+@subsubheading Description text
+
+The description should be clear even for a music beginner.
+If there are any special terms used in the description,
+they all should be explained in our @rglosnamed{Top, Music Glossary}
+or @rinternalsnamed{Top, Internals Reference}.
+Vague descriptions (like "behaves well", "looks reasonable") shouldn't be used.
+
+@ignore
+this may be useful for advanced regtest checking
+@subsubheading Is regtest straightforward and systematic?
+
+Unfortunately some regtests are written poorly. A good regtest should be
+straightforward: it should be obvious what it checks and how. Also, it
+usually shouldn't check everything at once. For example it's a bad idea to test
+accidental placement by constucting one huge chord with many suspended notes
+and loads of accidentals. It's better to divide such problem into a series
+of clearly separated cases.
+@end ignore
projects / lilypond.git / blobdiff