X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;f=Documentation%2Fcontributor%2Fregressions.itexi;h=1963a6bebcfac7ae5a1bd4b90cd849423184fadc;hb=68dbc841e3401f14b900f6750ebb49b406caf357;hp=c155a8b923ad946a6beec6aaa2772eb26353935c;hpb=e98a84d3adfccbce3792b1176c937cc23589e669;p=lilypond.git diff --git a/Documentation/contributor/regressions.itexi b/Documentation/contributor/regressions.itexi index c155a8b923..1963a6bebc 100644 --- a/Documentation/contributor/regressions.itexi +++ b/Documentation/contributor/regressions.itexi @@ -7,10 +7,10 @@ * Precompiled regression tests:: * Compiling regression tests:: * Regtest comparison:: +* Pixel-based regtest comparison:: * Finding the cause of a regression:: * Memory and coverage tests:: * MusicXML tests:: -* Grand Regression Test Checking:: @end menu @@ -135,6 +135,11 @@ something is suspicious! @item Profile files: give information about TODO? I don't know what they're for. +Apparently they give some information about CPU usage. If you got +tons of changes in cell counts, this probably means that you compiled +@code{make test-baseline} with a different amount of CPU threads than +@code{make check}. Try redoing tests from scratch with the same +number of threads each time -- see @ref{Saving time with the -j option}. @end itemize @@ -293,6 +298,58 @@ and want to do regtests comparison for all of them, you can @code{make test-clean}, @code{make} and @code{make check} it without doing @code{make test-baseline} again.} +@node Pixel-based regtest comparison +@section Pixel-based regtest comparison + +As an alternative to the @code{make test} method for regtest checking (which +relies upon @code{.signature} files created by a LilyPond run and which describe +the placing of grobs) there is a script which compares the output of two +LilyPond versions pixel-by-pixel. To use this, start by checking out the +version of LilyPond you want to use as a baseline, and run @code{make}. Then, +do the following: + +@example +cd $LILYPOND_GIT/scripts/auxiliar/ +./make-regtest-pngs.sh -j9 -o +@end example + +The @code{-j9} option tells the script to use 9 CPUs to create the +images - change this to your own CPU count+1. @code{-o} means this is the "old" +version. This will create images of all the regtests in + +@example +$LILYPOND_BUILD_DIR/out-png-check/old-regtest-results/ +@end example + +Now checkout the version you want to compare with the baseline. Run +@code{make} again to recreate the LilyPond binary. Then, do the following: + +@example +cd $LILYPOND_GIT/scripts/auxiliar/ +./make-regtest-pngs.sh -j9 -n +@end example + +The @code{-n} option tells the script to make a "new" version of the +images. They are created in + +@example +$LILYPOND_BUILD_DIR/out-png-check/new-regtest-results/ +@end example + +Once the new images have been created, the script compares the old images with +the new ones pixel-by-pixel and prints a list of the different images to the +terminal, together with a count of how many differences were found. The +results of the checks are in + +@example +$LILYPOND_BUILD_DIR/out-png-check/regtest-diffs/ +@end example + +To check for differences, browse that directory with an image +viewer. Differences are shown in red. Be aware that some images with complex +fonts or spacing annotations always display a few minor differences. These can +safely be ignored. + @node Finding the cause of a regression @section Finding the cause of a regression @@ -448,7 +505,7 @@ code has been exercised by the tests. For tracking memory usage as part of this test, you will need GUILE CVS; especially the following patch: @smallexample -@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}. +@uref{http://lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}. @end smallexample @subheading Code coverage @@ -484,49 +541,3 @@ available in the LilyPond documentation: @uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files} @end example - -@node Grand Regression Test Checking -@section Grand Regression Test Checking - -@subheading What is this all about? - -Regression tests (usually abbreviated "regtests") is a collection -of @file{.ly} files used to check whether LilyPond is working correctly. -Example: before version 2.15.12 breve noteheads had incorrect width, -which resulted in collisions with other objects. After the issue was fixed, -a small @file{.ly} file demonstrating the problem was added to the regression -tests as a proof that the fix works. If someone will accidentally break -breve width again, we will notice this in the output of that regression test. - -@subheading How can I help? - -We ask you to help us by checking one or two regtests from time to time. -You don't need programming skills to do this, not even LilyPond skills - -just basic music notation knowledge; checking one regtest takes less than -a minute. Simply go here: - -@example -@uref{http://www.philholmes.net/lilypond/regtests/} -@end example - -@subheading Some tips on checking regtests - -@subsubheading Description text - -The description should be clear even for a music beginner. -If there are any special terms used in the description, -they all should be explained in our @rglosnamed{Top, Music Glossary} -or @rinternalsnamed{Top, Internals Reference}. -Vague descriptions (like "behaves well", "looks reasonable") shouldn't be used. - -@ignore -this may be useful for advanced regtest checking -@subsubheading Is regtest straightforward and systematic? - -Unfortunately some regtests are written poorly. A good regtest should be -straightforward: it should be obvious what it checks and how. Also, it -usually shouldn't check everything at once. For example it's a bad idea to test -accidental placement by constucting one huge chord with many suspended notes -and loads of accidentals. It's better to divide such problem into a series -of clearly separated cases. -@end ignore