From df37b81e8f279c4f9f1e3b6f9b0c1ea07ffd9219 Mon Sep 17 00:00:00 2001 From: Carl Sorensen Date: Mon, 7 Jun 2010 07:14:08 -0600 Subject: [PATCH] Doc: Contributor's: Update Regression tests Make CG8 the home source for all Regression test information, with links from Programming work, Compiling, and INSTALL.txt. --- .../contributor/programming-work.itexi | 68 +++++- Documentation/contributor/regressions.itexi | 230 ++++++++++++------ Documentation/included/compile.itexi | 94 +------ 3 files changed, 234 insertions(+), 158 deletions(-) diff --git a/Documentation/contributor/programming-work.itexi b/Documentation/contributor/programming-work.itexi index 4c915e7f51..b131c423c2 100644 --- a/Documentation/contributor/programming-work.itexi +++ b/Documentation/contributor/programming-work.itexi @@ -1293,8 +1293,72 @@ get help from a Linux or OSX developer to do the make tests. @subsection Verify regression tests In order to avoid breaking LilyPond, it is important to verify that -the regression tests all succeed. This process is described in -@ref{Regression tests}. +the regression tests succeed, and that no unwanted changes are +introduced into the output. This process is described in +@ref{Identifying code regressions}. + +@subheading Typical developer's edit/compile/test cycle + +TODO: is @code{[-j@var{X} CPU_COUNT=@var{X}]} useful for +@code{test-baseline}, @code{check}, @code{clean}, +@code{test-redo}? Neil Puttock says it is useful for +everything but @code{clean}, which is disk-limited. +Need to check formally. + +@itemize +@item +Initial test: + +@example +make [-j@var{X}] +make test-baseline +make [-j@var{X} CPU_COUNT=@var{X}] check +@end example + +@item +Edit/compile/test cycle: + +@example +@emph{## edit source files, then...} + +make clean @emph{## only if needed (see below)} +make [-j@var{X}] @emph{## only if needed (see below)} +make test-redo @emph{## redo files differing from baseline} +make [-j@var{X} CPU_COUNT=@var{X}] check @emph{## CPU_COUNT here?} +@end example + +@item +Reset: + +@example +make test-clean +@end example +@end itemize + +If you modify any source files that have to be compiled (such as +@file{.cc} or @file{.hh} files in @file{flower/} or @file{lily/}), +then you must run @command{make} before @command{make test-redo}, +so @command{make} can compile the modified files and relink all +the object files. If you only modify files which are interpreted, +like those in the @file{scm/} and @file{ly/} directories, then +@command{make} is not needed before @command{make test-redo}. + +TODO: Fix the following paragraph. You can do @command{rm mf/out/*} +instead of make clean, and you can probably do +@command{make -C mf/ clean} as well, but I haven't checked it -- cds + +Also, if you modify any font definitions in the @file{mf/} +directory then you must run @command{make clean} and +@command{make} before running @command{make test-redo}. This will +recompile everything, whether modified or not, and takes a lot +longer. + +Running @command{make@tie{}check} will leave an HTML page +@file{out/test-results/index.html}. This page shows all the +important differences that your change introduced, whether in the +layout, MIDI, performance or error reporting. + + @node Post patch for comments diff --git a/Documentation/contributor/regressions.itexi b/Documentation/contributor/regressions.itexi index 178d3e4a4b..405d0d6bec 100644 --- a/Documentation/contributor/regressions.itexi +++ b/Documentation/contributor/regressions.itexi @@ -4,8 +4,10 @@ @menu * Introduction to regression tests:: -* Current regtest output:: -* Comparison regtest output:: +* Precompiled regression tests:: +* Compiling regression tests:: +* Identifying code regressions:: +* Memory and coverage tests:: * MusicXML tests:: @end menu @@ -23,132 +25,210 @@ LilyPond. They are also added when bugs are identified. The snippet that causes the bug becomes a regression test to verify that the bug has been fixed. -The regression tests are automatically compiled using special @code{make} -targets. The output of the regression tests is also automatically -checked to identify changes in LilyPond output. +The regression tests are compiled using special @code{make} +targets. There are three primary uses for the regression +tests. First, successful completion of the regression tests means +that LilyPond has been properly built. Second, the output of the +regression tests can be manually checked to ensure that +the graphical output matches the description of the intended +output. Third, the regression test output from two different +versions of LilyPond can be automatically compared to identify +any differences. These differences should then be manually +checked to ensure that the differences are intended. -The output of the regression tests is available on the website -for every stable version of LilyPond. This allows the comparison -of different versions to see when bugs appeared. +Regression tests (@qq{regtests}) are available in precompiled form +as part of the documentation. Regtests can also be compiled +on any machine that has a properly configured LilyPond build +system. -@node Current regtest output -@section Current regtest output +@node Precompiled regression tests +@section Precompiled regression tests +@subheading Regression test output -TODO: To be checked and completed -vv +As part of the release process, the regression tests are run +for every LilyPond release. Full regression test output is +available for every stable version and the most recent development +version. -Regression tests (@qq{regtests}) are available in two ways: either -in a compiled form, for instance on the website, or as source code -that needs to be compiled locally, using the most recent LilyPond -binary as possible. The latter is recommended, although more -technically involved. +Regression test output is available in HTML and PDF format. Links +to the regression test output are available at the developer's +resources page for the version of interest. +The latest stable version of the regtests is found at: -@subheading Precompiled regtests +@example +@uref{http://lilypond.org/doc/stable/input/regression/collated-files.html} +@end example -The easiest way to see the @q{current} regtest output (meaning, -the ouput of the latest stable or development version) is -to look at the online compiled regtest page: +The latest development version of the regtests is found at: @example @uref{http://lilypond.org/doc/latest/input/regression/collated-files.html} @end example -However, depending on how many changes have been made to the code -since the latest release, this page may not reflect the latest -features, bugfixes... or new bugs that may have been introduced! -Therefore, if you have an appropriate environment to build LilyPond -yourself, it is recommended that you compile the software yourself. +@subheading Regression test comparison + +Each time a new version is released, the regtests are +compiled and the output is automatically compared with the +output of the previous release. The result of these +comparisons is archived online: +@example +@uref{http://lilypond.org/test/} +@end example -@subheading Compiling regtests +The test comparison shows all of the changes that occured between +the current release and the prior release. Each test that has +a significant difference in output is displayed, with the old +version on the left and the new version on the right. Blurs +in the regression tests for the new version show the approximate +location of elements in the old version. -The first step is to download the latest available source code, -as explained in @ref{Working with source code}. Then you will need -to build the LilyPond binary: see -@ref{Compiling LilyPond}. +Regression tests whose output is the same for both versions are not +shown in the test comparison. -@noindent -(Uninstalling the previous LilyPond version is not necessary, nor is +Checking these pages is a very important task for the LilyPond project. +You are invited to report anything that looks broken, or any case +where the output quality is not on par with the previous release, +as described in @rweb{Bug reports}. + +@warning{ +The automatic comparison of the regtests checks the +LilyPond bounding boxes and the log files. This means that +Ghostscript changes and changes in +lyrics or text are not found. However, errors and warnings that +are printed to the log file but +do not change the graphical layout are also identified. + +Periodic manual checking of the +complete regtest output should be performed to catch anything +missed by the automatic comparison.} + +@node Compiling regression tests +@section Compiling regression tests + +Developers may wish to see the output of the complete regression +test suite for the current version of the source repository +between releases. Current source code is available; see +@ref{Working with source code}. Then you will need +to build the LilyPond binary; see @ref{Compiling LilyPond}. + +Uninstalling the previous LilyPond version is not necessary, nor is running @code{make install}, since the tests will automatically be compiled with the LilyPond binary you have just built in your source -directory.) +directory. -From this point, compiling the regtests is as simple as running +From this point, the regtests are compiled with: @example make test @end example -However, as there are many snippets to compile, if you have a multi-core -machine it is highly recommended to use the @option{-j} option, as -described in @ref{Saving time with the -j option}. Another -useful optimization is to set the @var{CPU_COUNT} variable; for a -quad-core processor the complete command would look like +If you have a multi-core machine you may want to use the @option{-j} +option and @var{CPU_COUT} variable, as +described in @ref{Saving time with CPU_COUNT}. +For a quad-core processor the complete command would be: @example -make -j5 CPU_COUNT=4 test +make -j5 CPU_COUNT=5 test @end example -The regtest output will then be available in one of the -@file{input/regression/out-*} directories, depending on the -exact command you used. See @ref{Testing LilyPond} for -more information. - +The regtest output will then be available in +@file{input/regression/out-test}. +@file{input/regression/out-test/collated-examples.html} +contains a listing of all the regression tests that were run, +but none of the images are included. Individual images are +also available in this directory. -@node Comparison regtest output -@section Comparison regtest output +The primary use of @samp{make@tie{}test} is to verify that the +regression tests all run without error. The regression test +page that is part of the documentation is created only when the +documentation is built, as described in @ref{Generating documentation}. +Note that building the documentation requires more installed components +than building the source code, as described in +@ref{Requirements for building documentation}. -Regtests are an useful way to compare what has changed between two -versions of LilyPond, or to verify on a fine-grained level if a -particular change may have unwanted side-effects, such as introducing -a bug or breaking existing features. +@node Identifying code regressions +@section Identifying code regressions -For such cases, LilyPond's build system provides an automated way of -comparing regtests output. +Before modified code is committed to master, a regression test +comparision must be completed to ensure that the changes have +not caused problems with previously working code. The comparison +is made automatically upon compiling the regression test suite +twice. +Before making changes, a baseline should be established by running: -@subheading Comparing regtests for two development releases +@example +make test-baseline +@end example -Each time a new development version is released, a set of regtests is -compiled and compared with the previous release. The result of these -comparisons is archived online, and may be seen at the following address: +After making the changes, the code should be checked by running: @example -@uref{http://lilypond.org/test/} +make check @end example -@noindent -Checking these pages is a very important task for the LilyPond project. -You are invited to report anything that looks broken, or any case -where the output quality is not on par with the previous release, -either to the Bug Squad, following our guidelines for -@rweb{Bug reports}, or directly in the bug tracker, as explained in -@ref{Issues}. +After @samp{make@tie{}check} is complete, a regression test comparison +will be available at @file{out/test-results/index.html}. +For each regression test that differs between the baseline and the +changed code, a regression test entry will displayed. Ideally, the +only changes would be the changes that you were working on. If +regressions are introduced, they must be fixed before committing the +code. + +@warning{ +The special regression test @file{test-output-distance.ly} will always +show up as a regression. This test changes each time it is run, and +serves to verify that the regression tests have, in fact, run.} + +@node Memory and coverage tests +@section Memory and coverage tests + +In addition to the graphical output of the regression tests, it is +possible to test memory usage and to determine how much of the source +code has been exercised by the tests. + +@subheading Memory usage +For tracking memory usage as part of this test, you will need +GUILE CVS; especially the following patch: +@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}. -@subheading Comparing regtests when modifying the source code +@subheading Code coverage -When changing any piece of code, developers are asked to verify that the -regtests still compile successfuly (i.e., not only without error, but -with an output quality equivalent or superior). This may be done as -described in @ref{Testing LilyPond}. +For checking the coverage of the test suite, do the following + +@example +./scripts/auxiliar/build-coverage.sh +@emph{# uncovered files, least covered first} +./scripts/auxiliar/coverage.py --summary out-cov/*.cc +@emph{# consecutive uncovered lines, longest first} +./scripts/auxiliar/coverage.py --uncovered out-cov/*.cc +@end example @node MusicXML tests @section MusicXML tests -LilyPond comes with a fairly complete set of regtests for the -@uref{http://www.musicxml.org/,MusicXML} language. These tests may -be seen online at the following address: +LilyPond comes with a complete set of regtests for the +@uref{http://www.musicxml.org/,MusicXML} language. Originally +developed to test @samp{musicxml2ly}, these regression tests +can be used to test any MusicXML implementation. + +The MusicXML regression tests are found at +@file{input/regression/musicxml/}. + +The output resulting from running these tests +through @samp{muscxml2ly} followed by @samp{lilypond} is +available in the LilyPond documentation: @example @uref{http://lilypond.org/doc/latest/input/regression/musicxml/collated-files} @end example -TBC - diff --git a/Documentation/included/compile.itexi b/Documentation/included/compile.itexi index 3966ccfce4..34f66a231c 100644 --- a/Documentation/included/compile.itexi +++ b/Documentation/included/compile.itexi @@ -524,7 +524,7 @@ command line, or in @file{local.make} at top of the build tree. @menu * Installing LilyPond from a local build:: * Generating documentation:: -* Testing LilyPond:: +* Testing LilyPond binary:: @end menu @@ -853,93 +853,25 @@ exec /opt/local/bin/pngtopnm "$@" @end verbatim -@node Testing LilyPond -@subsection Testing LilyPond +@node Testing LilyPond binary +@subsection Testing LilyPond binary LilyPond comes with an extensive suite that exercises the entire -program. This suite can be used to automatically check the impact -of a change. +program. This suite can be used to test that the binary has +been built correctly. -@menu -* Developer's edit/compile/test cycle:: -* Other tests:: -@end menu - -@node Developer's edit/compile/test cycle -@unnumberedsubsubsec Developer's edit/compile/test cycle - -TODO: is @code{[-j@var{X} CPU_COUNT=@var{X}]} useful for -@code{test-baseline}, @code{check}, @code{clean}, -@code{test-redo}? - -@itemize -@item -Initial test: - -@example -make [-j@var{X}] -make test-baseline -make [-j@var{X} CPU_COUNT=@var{X}] check -@end example - -@item -Edit/compile/test cycle: - -@example -@emph{## edit source files, then...} - -make clean @emph{## only if needed (see below)} -make [-j@var{X}] @emph{## only if needed (see below)} -make test-redo @emph{## redo files differing from baseline} -make [-j@var{X} CPU_COUNT=@var{X}] check @emph{## CPU_COUNT here?} -@end example +The test suite can be executed with: -@item -Reset: - -@example -make test-clean -@end example -@end itemize - -If you modify any source files that have to be compiled (such as -@file{.cc} or @file{.hh} files in @file{flower/} or @file{lily/}), -then you must run @command{make} before @command{make test-redo}, -so @command{make} can compile the modified files and relink all -the object files. If you only modify files which are interpreted, -like those in the @file{scm/} and @file{ly/} directories, then -@command{make} is not needed before @command{make test-redo}. - -Also, if you modify any font definitions in the @file{mf/} -directory then you must run @command{make clean} and -@command{make} before running @command{make test-redo}. This will -recompile everything, whether modified or not, and takes a lot -longer. - -Running @command{make@tie{}check} will leave an HTML page -@file{out/test-results/index.html}. This page shows all the -important differences that your change introduced, whether in the -layout, MIDI, performance or error reporting. - - -@node Other tests -@unnumberedsubsubsec Other tests - -For tracking memory usage as part of this test, you will need -GUILE CVS; especially the following patch: -@uref{http://www.lilypond.org/vc/old/gub.darcs/patches/guile-1.9-gcstats.patch}. - -For checking the coverage of the test suite, do the following +@verbatim +make test +@end verbatim -@example -./scripts/auxiliar/build-coverage.sh -@emph{# uncovered files, least covered first} -./scripts/auxiliar/coverage.py --summary out-cov/*.cc -@emph{# consecutive uncovered lines, longest first} -./scripts/auxiliar/coverage.py --uncovered out-cov/*.cc -@end example +If the test suite completes successfully, the LilyPond binary +has been verified. +More information on the regression test suite is found at +@rcontrib{Regression tests}. @node Problems @section Problems -- 2.39.5