@uref{http://lilypond.org/test/}
@end example
-The test comparison shows all of the changes that occured between
-the current release and the prior release. Each test that has
-a significant difference in output is displayed, with the old
-version on the left and the new version on the right. Blurs
-in the regression tests for the new version show the approximate
-location of elements in the old version.
-
-Regression tests whose output is the same for both versions are not
-shown in the test comparison.
-
Checking these pages is a very important task for the LilyPond project.
You are invited to report anything that looks broken, or any case
where the output quality is not on par with the previous release,
as described in @rweb{Bug reports}.
+@warning{ The special regression test
+@file{test-output-distance.ly} will always show up as a
+regression. This test changes each time it is run, and serves to
+verify that the regression tests have, in fact, run.}
+
+
+@subheading What to look for
+
+The test comparison shows all of the changes that occurred between
+the current release and the prior release. Each test that has a
+significant difference in output is displayed, with the old
+version on the left and the new version on the right.
+
+Regression tests whose output is the same for both versions are
+not shown in the test comparison.
+
+@itemize
+@item
+Images: green blurs in the new version show the approximate
+location of elements in the old version.
+
+There are often minor adjustments in spacing which do not indicate
+any problem.
+
+@item
+Log files: show the difference in command-line output.
+
+The main thing to examine are any changes in page counts -- if a
+file used to fit on 1 page but now requires 4 or 5 pages,
+something is suspicious!
+
+@item
+Profile files: give information about
+TODO? I don't know what they're for.
+
+@end itemize
+
@warning{
-The automatic comparison of the regtests checks the
-LilyPond bounding boxes and the log files. This means that
-Ghostscript changes and changes in
-lyrics or text are not found. However, errors and warnings that
-are printed to the log file but
-do not change the graphical layout are also identified.
+The automatic comparison of the regtests checks the LilyPond
+bounding boxes. This means that Ghostscript changes and changes
+in lyrics or text are not found.
}
@node Compiling regression tests
@section Identifying code regressions
Before modified code is committed to master, a regression test
-comparision must be completed to ensure that the changes have
+comparison must be completed to ensure that the changes have
not caused problems with previously working code. The comparison
is made automatically upon compiling the regression test suite
twice.
show up as a regression. This test changes each time it is run, and
serves to verify that the regression tests have, in fact, run.}
+Once @samp{make@tie{}test-baseline} and @samp{make@tie{}check} have been
+run, the files that differ between @samp{test-baseline} and @samp{check}
+can be repeatedly examined by doing:
+
+@example
+make test-redo
+@end example
+
+This updates the regression list at @file{out/test-results/index.html}.
+It does @emph{not} redo @file{test-output-distance.ly}.
+
+When all regressions have been resolved, the output list will be empty.
+
+Once all regressions have been resolved, a final check should be completed
+by running:
+
+@example
+make test-clean
+make check
+@end example
+
+This cleans the results of the previous @samp{make@tie{}check}, then does the
+automatic regression comparison again.
+
+
@node Memory and coverage tests
@section Memory and coverage tests
projects / lilypond.git / blobdiff