-@c -*- coding: utf-8; mode: texinfo; -*-
@node Administrative policies
@chapter Administrative policies
@menu
* Meta-policy for this document::
+* Environment variables::
* Meisters::
+* Automated testing with Patchy::
* Administrative mailing list::
* Grand Organization Project (GOP)::
* Grand LilyPond Input Syntax Standardization (GLISS)::
The Contributor's Guide as a whole is still a work in progress,
but some chapters are much more complete than others. Chapters
which are @qq{almost finished} should not have major changes
-without a discussion on @code{-devel}; in other chapters, a
+without a discussion on @w{@code{-devel}}; in other chapters, a
disorganized @qq{wiki-style dump} of information is encouraged.
Do not change (other than spelling mistakes) without discussion:
@end itemize
+@node Environment variables
+@section Environment variables
+
+Some maintenance scripts and instructions in this guide rely on
+the following environment variables. They should be predefined in
+LilyDev distribution (see @ref{LilyDev}); if you set up your own
+development environment, you can set them by appending these settings to
+your @file{~/.bashrc} (or whatever defines your default environment
+variables for the user account for LilyPond development), then logging
+out and in (adapt directories to your setup):
+
+@example
+LILYPOND_GIT=~/lilypond-git
+export LILYPOND_GIT
+LILYPOND_BUILD_DIR=~/lilypond-git/build
+export LILYPOND_BUILD_DIR
+@end example
+
+The standard build and install procedure (with @code{autogen.sh},
+@code{configure}, @code{make}, @code{make install}, @code{make doc}
+@dots{}) does not rely on them.
+
+In addition, for working on the website, @code{LILYPOND_WEB_MEDIA_GIT}
+should be set to the repository lilypond-extra, see
+@ref{lilypond-extra}.
+
@node Meisters
@section Meisters
on which part of their job, checks to make sure that everything is
running smoothly, and has final say on our policy for Issues.
-Currently: Phil
+Currently: Colin H
@item
Doc Meister: trains new doc editors/writers, organizes who works
running smoothly, and has final say on our policy for
Documentation. Also includes LSR work.
-Currently: Graham
+Currently: None
@item
Translation Meister: trains new translators, updates the
Currently: Francisco
+@end itemize
+
+
+@node Automated testing with Patchy
+@section Automated testing with Patchy
+
+@menu
+* Overview of Patchy::
+* Patchy requirements::
+* Installing Patchy::
+* Configuring Patchy::
+* Running the scripts::
+* Automating Patchy::
+* Troubleshooting Patchy::
+@end menu
+
+@node Overview of Patchy
+@subsection Overview of Patchy
+
+Patchy is a set of Python scripts used for testing patches or testing &
+pushing new commits added to @code{remote/origin/staging} to
+@code{remote/origin/master}.
+
+No programmatic skill is required to run either of the scripts; although
+knowledge of compiling LilyPond and its documentation along with
+understanding how to configure the @var{PATH} environment of your
+computer is required. See @ref{Working with source code}.
+
+The two scripts that are used for each function are:
+
+@itemize
+@item
+@code{test-patches.py}. This script tests issues labelled as
+@qq{Patch-New} on the LilyPond issue tracker. Part of the testing
+process involves running the regression tests, so this script always
+requires some human intervention in order to visually check for any
+differences that may be present after a successful test has occurred.
+
+@item
+@code{lilypond-patchy-staging.py}. This script checks for any new
+commits in @code{remote/origin/staging}, makes sure that the new HEAD
+compiles along with all the LilyPond documentation. Then finally
+pushing to @code{remote/origin/master}. This script can be run and left
+unattended, requiring no human intervention.
+
+@end itemize
+
+Both of the scripts can be run independently of each other and it is not
+necessary to be able to run both. So if you wanted to contribute to
+LilyPond development, for example by @emph{just} testing patches then
+this would still be a helpful contribution to LilyPond's development.
+
+Patchy can also be configured to send emails after each successful (or
+unsuccessful) operation. This is not a requirement and is turned off
+by default.
+
+@c Need to explain in more detail how to set up Patchy for email but
+@c as I don't use myself it I have no experience - JL
+
+
+@node Patchy requirements
+@subsection Patchy requirements
+
+@subsubsubheading Testing new patches
+
+@itemize
+
+@item
+A full local copy of the source code. See
+@ref{Working with source code}.
+
+@item
+All the software needed for compiling LilyPond @emph{and} the
+documentation. Although being able to build the full set of LilyPond's
+manuals is not mandatory for testing (most) patches, part of the patch
+testing process requires that the regression tests are run and it is
+this that requires the software normally used for compiling
+documentation. See @ref{Compiling}.
+
+@item
+Commit access is @emph{not} required to test patches, but a valid login
+to @uref{http://code.google.com/} @emph{is}.
+
+@end itemize
+
+
+@subsubsubheading Testing & pushing new commits
+
+@itemize
+
+@item
+A full local copy of the source code. See
+@ref{Working with source code}.
+
+@item
+All the software needed for compiling LilyPond @emph{and} the
+documentation. Unlike testing patches, being able to build the full set
+of LilyPond's documentation is required to be able to test & push new
+commits. See @ref{Compiling}.
+
+@item
+Commit access @emph{is} required to test and push new commits, but a
+valid login to @uref{http://code.google.com/} is @emph{not}. See
+@ref{Commit access}.
+
+@end itemize
+
+
+@node Installing Patchy
+@subsection Installing Patchy
+
+The Patchy scripts are not part of the LilyPond code base, but can be
+downloaded from @uref{https://github.com/gperciva/lilypond-extra/}. The
+scripts and related Python libraries are all located in the
+@file{patches/} directory.
+
+Alternatively, use @code{git clone};
+
+@example
+git clone https://github.com/gperciva/lilypond-extra/
+@end example
+
+This makes it simpler to update the scripts if any changes are ever made
+to them. Finally, add the location of the @file{patches/} directory to
+your @var{PATH}.
+
+
+@node Configuring Patchy
+@subsection Configuring Patchy
+
+@warning{It is recommended to create a new user on your computer
+specifically to run the Patchy scripts as a security precaution and that
+this user should not have any administrative privileges. Also do not
+set password protection for your ssh key else you will not be able to
+run the scripts unattended.}
+
+@enumerate
+
+@item
+Make sure the environment variables @var{LILYPOND_GIT} and
+@var{LILYPOND_BUILD_DIR} are configured appropriately. See
+@ref{Environment variables}.
+
+@item
+To save being prompted for your login and password to
+@uref{http://code.google.com/} when testing patches. create a
+@emph{plain-text} file in your Patchy user's @code{$HOME} directory
+called @code{.lilypond-project-hosting-login} containing your login and
+password, each on a separate line.
+
+@example
+joe_smith123@@gmail.com
+mYp455w0rd!
+@end example
+
+@item
+Manually run either the @code{test-patches.py} or
+@code{lilypond-patchy-staging.py} scripts and when prompted:
+
+@smallexample
+Warning: using default config; please edit /home/joe/.lilypond-patchy-config
+Are you sure that you want to continue with the default config? (y/[n])
+@end smallexample
+
+Answer @qq{@code{n}} and press enter.
+
+The next time either of the scripts are run they will use the
+@code{.lilypond-patchy-config} settings copied to your @code{$HOME}
+directory.
+
+@item
+Manually edit the @file{.lilypond-patchy-config} file, located in your
+@code{$HOME} directory to change any of the default settings.
+
+@end enumerate
+
+These include:
+
+@itemize
+
+@item
+All @code{make} operations are run with;
+@example
+extra_make_options = -j3 CPU_COUNT=3
+@end example
+
+See @ref{Saving time with the -j option}
+
+@item
+A complete build of all the LilyPond documentation is @emph{not}
+performed;
+@example
+patch_test_build_docs = no
+@end example
+
+@item
+Each instance of either a patch test or commit test & push is logged in;
+@example
+auto_compile_results_dir = ~/lilypond-auto-compile-results/
+@end example
+
+@item
+Both scripts will perform their build operations in;
+@example
+build_dir = /tmp/lilypond-autobuild/
+@end example
+
+@end itemize
+
+Each completed patch test will also generate its own directory in
+@file{/tmp/...} labelled with the tracker issue number prefixed by
+@code{show-}.
+@example
+\tmp\show-3446
+@end example
+
+Both the scripts create clones of @code{staging} and @code{master}
+branches (prefixed with @code{test-}) with a third branch, called
+@code{test-master-lock} used as a check to protect against two or more
+instances of Patchy being run locally at the same time.
+
+
+@node Running the scripts
+@subsection Running the scripts
+
+@subsubsubheading Testing & pushing new commits
+
+@code{lilypond-patchy-staging.py} is run @emph{without} any arguments.
+It then checks to see if @code{remote/origin/staging} is
+@qq{further ahead} than @code{remote/origin/master}.
+
+@noindent
+If there are no new differences between the two branches since the last
+run check, the script will report something like this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Success: No new commits in staging
+@end smallexample
+
+@noindent
+If there are any differences between the two branches since the last
+run check, (or if the script cannot for any reason, locate the last
+instance of a commit that it checked) it will report something like
+this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Merged staging, now at: 79e98a773b6570cfa28a15775a9dea3d3e54d6b5
+ Success: ./autogen.sh --noconfigure
+ Success: /tmp/lilypond-autobuild/configure --disable-optimising
+...
+@end smallexample
+
+and proceed with running @code{make}, @code{make test} and a
+@code{make doc}. Unlike @code{test-patches.py} if all the tests pass,
+the script then pushes the changes to @code{remote/origin/master}.
+
+@smallexample
+...
+Success: nice make clean
+Success: nice make -j7 CPU_COUNT=7
+Success: nice make test -j7 CPU_COUNT=7
+Success: nice make doc -j7 CPU_COUNT=7
+To ssh://joe@@git.sv.gnu.org/srv/git/lilypond.git
+ 79e98a7..4726764 test-staging -> master
+ Success: pushed to master
+@end smallexample
+
+@warning{In the case where any of the @code{lilypond-patchy-staging.py}
+tests fail, do not try to push your own fixes but report the failures to
+the Developers List <lilypond-devel@@gnu.org> for advice.}
+
+
+@subsubsubheading Testing new patches
+
+When run without any argument, @code{test-patches.py} will check
+@uref{http://code.google.com/p/lilypond/issues/list} for all tracker
+issues that are marked with the label @code{Patch-new}. It then scrapes
+the issue, looking for the last Rietveld URL entered. It then downloads
+the patch file and applies it to @code{test-master}.
+
+Here is an example where two tracker issues labeled as @code{Patch-new}
+were detected:
+
+@smallexample
+...
+issues [4007, 4008]
+Trying issue 4007
+Found url: http://codereview.appspot.com/112210043
+Found patch: 4007,/home/joe/lilypond-git/issue112210043_1.diff,
+Trying issue 4008
+Found url: http://codereview.appspot.com/115770043
+Found patch: 4008,/home/joe/lilypond-git/issue115770043_1.diff,
+Fetching, cloning, compiling master.
+...
+@end smallexample
+
+If run no tracker items with the @var{Patch-New} label are found it will
+report:
+
+@example
+issues []
+@end example
+
+The script can also be run using the tracker issue number(s) as an
+argument regardless if the @var{Patch-New} label has been assigned;
+
+@example
+test-patches.py 4006
+@end example
+
+or
+
+@example
+test-patches.py 4006 3992 4020
+@end example
+
+The script then checks to see if any previously
+@code{make test-baseline}s have been generated and if the commit ID of
+@code{remote/origin/master} is different from that previously completed
+test.
+
+@noindent
+If no previous @code{make test-baseline} test is discovered or if the
+commit ID of @code{remote/origin/master} has changed, then a new
+@code{make test-baseline} will run first automatically before the patch
+is tested:
+
+This shows when the commit ID has changed:
+
+@smallexample
+...
+(UTC) Begin LilyPond compile, previous commit at 3f92dcb2c81dcd2755542b57a0a5f2039f29a211
+Merged master, now at: 4726764cb591f622e7893407db0e7d42bcde90d9
+ Success: ./autogen.sh --noconfigure
+ Success: /tmp/lilypond-autobuild/configure --disable-optimising
+ Success: nice make clean
+ Success: nice make -j3 CPU_COUNT=3
+ Success: nice make test-baseline -j3 CPU_COUNT=3
+ Success: nice make doc -j3 CPU_COUNT=3
+ Success: nice make doc-clean
+...
+@end smallexample
+
+@noindent
+If a previous regression test @emph{is} discovered @emph{and} if the
+commit ID of @code{remote/origin/master} has not changed, then the patch
+will be tested against the previous @code{make test-baseline} without
+the need to re-generate a new one:
+
+@smallexample
+...
+issues [4009]
+Trying issue 4009
+Found url: http://codereview.appspot.com/110540043
+Found patch: 4009,/home/joe/lilypond-extra/patches/issue110540043_1.diff,
+Fetching, cloning, compiling master.
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+ Success: No new commits in master
+Using test baseline from previous build.
+...
+@end smallexample
+
+The patch is then applied and a @code{make} and @code{make check} are
+run. A full @code{make doc} is also run if the
+@file{.lilypond-patchy-config} file has been edited accordingly;
+
+@smallexample
+...
+Issue 4009:
+Issue 4009: Testing patch issue110540043_1.diff
+ Success: git apply --index /home/joe/lilypond-git/issue112210043_1.diff
+ Success: ./autogen.sh --noconfigure
+ Success: /tmp/lilypond-autobuild/configure --disable-optimising
+ Success: nice make clean
+ Success: nice make -j3 CPU_COUNT=3
+ Success: nice make check -j3 CPU_COUNT=3
+ Success: nice make doc -j3 CPU_COUNT=3
+...
+@end smallexample
+
+Once all the tests have run (successfully or not), the script will clean
+up from the previous patch and, if required, start testing the next
+issue;
+
+@smallexample
+...
+Issue 4007: Cleaning up
+ Success: nice make test-clean
+ Success: nice make doc-clean
+ Success: nice make clean
+ Success: git reset --hard
+Issue 4007: Done.
+Issue 4008:
+Issue 4008: Testing patch issue115770043_1_diff
+ Success: git apply --index /home/joe/lilypond-git/issue115770043_1.diff
+ Success: ./autogen.sh --noconfigure
+...
+@end smallexample
+
+and so on.
+
+@subsubsubheading Checking the regression test results
+
+Assuming the patch passed all the @code{make} tests, the regression
+differences will be located in the @file{/test-results/} directory
+within the build location for the patch issue number;
+
+@example
+/tmp/show-4007/test-results/
+@end example
+
+Open @file{index.html} in a browser of your choice to view any
+differences.
+
+@noindent
+Alternatively if the Firefox browser is installed, then the regression
+test results can be opened by calling the appropriate
+@file{show-regtests-} file located in the auto-compile log location;
+
+@example
+sh ~/lilypond-auto-compile-results/show-regtests-4007
+@end example
+
+See @ref{Regression tests}.
+
+@subsubsubheading Reporting test results
+
+Once a patch has been tested and the regression tests have been
+manually checked, the tracker can be updated manually by editing the
+tracker issue directly in the web browser or by using two additional
+python scripts that are included as part of the Patchy suite.
+
+@unnumberedsubsubsec For patches that have passed
+
+Use the @code{accept-patch.py} script and run it with the Google issue
+tracker number (not the Rietveld issue number) as an argument;
+
+@example
+accept-patch.py 4007
+@end example
+
+This will automatically update the tracker issue with the phrase
+@qq{Patchy the autobot says: passes tests.}.
+
+@noindent
+It is also possible to add additional information to the default
+message by adding a second argument within double-quote marks.
+
+@example
+accept-patch.py 4007 "This also includes a full documentation build."
+@end example
+
+The tracker issue's label is then changed automatically to
+@qq{Patch-review}.
+
+@unnumberedsubsubsec Patches that have failed
+
+Use the @code{reject-patch.py} script and run it with the Google issue
+tracker number (not the Rietveld issue number) as an argument but you
+@emph{must} also include a second argument, in double-quotes, stating
+the reason the patch has been rejected;
+
+@example
+reject-patch.py 4007 "Fails the 'make check' test."
+@end example
+
+Once the @code{reject-patch.py} script has been run, the tracker issue's
+label is changed automatically to @qq{Patch-Needs_work}.
+
+
+@node Automating Patchy
+@subsection Automating Patchy
+
+To run as a cron job make sure you have;
+
+@example
+[notification]
+notify_non_action = no
+@end example
+
+in @file{$HOME/.lilypond-patchy-config} to avoid any unintentional email
+flooding:
+
+Assuming that Patchy run a user @qq{patchy}, create a file called
+@file{$HOME/lilypond-patchy.cron}, adapting it as necessary (the
+@code{/2} means @qq{run this every 2 hours}):
+
+@smallexample
+02 0-23/2 * * * /home/patchy/lilypond-extra/patches/lilypond-patchy-staging.py
+@end smallexample
+
+@warning{@code{cron} will not inherit environment variables so you must
+re-define any variables inside @file{$HOME/lilypond-patchy.cron}. For
+instance, @var{LILYPOND_GIT} may need to be defined if
+@var{git_repository_dir} is not correctly set in
+@file{$HOME/.lilypond-patchy-config}.}
+
+Finally, apply the cron job (you may need superuser privileges for
+this):
+
+@example
+crontab -u patchy /home/patchy/lilypond-patchy.cron
+@end example
+
+
+@node Troubleshooting Patchy
+@subsection Troubleshooting Patchy
+
+The following is a list of the most common messages that the scripts
+may report with explanations.
+
+@itemize
+
+@item
+@example
+issues []
+@end example
+
+@itemize
+
+@item
+There are currently no tracker issues with the @code{Patch-New} status.
+
+@item
+If specific tracker issue number has been used as an argument when
+running @code{test-patches.py}, then the issue contains no URL to
+Rietveld.
+
+@end itemize
+
+@item
+@smallexample
+this Git revision has already been pushed by an operator other than this Patchy.
+@end smallexample
+
+@itemize
+
+@item
+Another, remote, machine has already tested and pushed the new commits
+in staging.
+
+@item
+You may also see this on your local machine if the auto-build files
+have been deleted and this computer has previously already pushed the
+listed commit ID to @code{master}.
+
+@end itemize
+
+
+
+@item
+@smallexample
+Git revision has not changed but checksum of test baseline has, must rebuild.
+@end smallexample
+
+This occurs when Patchy detects that the commit ID has not changed
+since the last test but it cannot locate the last
+@code{make test-baseline} (usually because it has been deleted or moved)
+and so a new @code{test-baseline} is rebuilt.
+
@item
-Frog Meister: is responsible for code patches from (relatively)
-inexperienced contributors. Keeps track of patches, does initial
-reviewing of those patches, sends them to @code{-devel} when
-they've had some initial review on the Frog list, pesters the
-@code{-devel} community into actually reviewing said patches, and
-finally pushes the patches once they're accepted. This person is
-@emph{not} responsible for training new programmers, because that
-would be far too much work -- he job is @qq{only} to guide
-completed patches through our process.
+@example
+Last patch for issue xxxx already tested or under testing
+by another Patchy instance, skipping.
+@end example
+
+@itemize
+
+@item
+There is another instance of Patchy running on your computer that is
+testing the same tracker issue.
+
+@item
+A previous test attempt was unsuccessful for some reason and the scripts
+were not able to tidy up after themselves (for example if you manually
+halt the testing process by killing it or closing the terminal you may
+have been running the script in).
+
+@noindent
+There is a hidden file located in the @code{$HOME} directory of the user
+running Patchy called @code{.lilypond-patchy-cache} that records the
+current patches that are being tested, have been tested and the commit
+ID of @code{remote/origin/master} since the last test. It will contain
+a line like this:
+
+@example
+[3995]
+issue105560044_120001_diff = testing
+@end example
+
+for any issue that it thinks is still in the process of being tested.
+
+@noindent
+Manually remove this entry and re-run the script.
+
+@end itemize
+
+@item
+@example
+test-master-lock and PID entry exist but previous Patchy
+run (PID xxxxx) died, resetting test-master-lock anyway.
+@end example
+
+@item
+A previous test attempt was unsuccessful for some reason and the scripts
+were not able to tidy up after themselves (for example if you manually
+halt the testing process by killing it or closing the terminal you may
+have been running the script in). The @code{test-master-lock} branch
+was therefore not able to be deleted cleanly however, nothing needs to
+be done the scripts will rebuild any tests it needs to.
+
+@item
+@example
+fatal: A branch named 'test-master-lock' already exists.
+@end example
+
+@itemize
+
+@item
+There is another instance of Patchy running on your computer that is
+testing the same tracker issue.
+
+@item
+A previous test attempt was unsuccessful for some reason and the scripts
+were not able to tidy up after themselves (for example if you manually
+halt the testing process by killing it or closing the terminal you may
+have been running the script in). The @code{test-master-lock} branch
+was therefore not able to be deleted cleanly, in this case you must
+manually delete the @code{test-master-lock} branch in your
+@code{$LILYPOND_GIT} directory.
+
+@example
+git branch -d test-master-lock
+@end example
+
+@noindent
+It may be wise to also manually delete @code{test-master} and
+@code{test-staging} too, just to be safe.
+
+@end itemize
+
+@item
+@example
+*** FAILED STEP ***
+ merge from staging
+ Another instance (PID xxxxx) is already running.
+@end example
+
+@noindent
+This occurs when trying to run @code{lilypond-patchy-staging.py} when
+another instance of either script is already running locally.
+
+@item
+@example
+Warning: something went wrong; omitting patch for issue 3976
+@end example
+
+@itemize
-Currently: Carl
+@item
+The Rietveld URL as listed in the tracker is incorrect (e.g. missing or
+incorrect issue number
+
+@item
+The patch on Rietveld is too large to download
+
+@end itemize
@end itemize
+
@node Administrative mailing list
@section Administrative mailing list
-An mailing list for administrative issues is maintained at
+A mailing list for administrative issues is maintained at
@code{lilypond-hackers@@gnu.org}.
This list is intended to be used for discussions that should be kept
@itemize
@item
-Clarify the various development tasks by writing down the polices
+Clarify the various development tasks by writing down the policies
and techniques and/or simplifying the tasks directly.
@item
* Motivation::
* Ongoing jobs::
* Policy decisions::
+* Policy decisions (finished)::
@end menu
@node Motivation
code.
Having 1 more normal user answering emails on lilypond-user won't
-have a dramatic trick-up affect all by himself, of course. But if
+have a dramatic @q{trickle-up} effect all by itself, of course. But if
we had 8 users volunteering to answer emails, 6 users starting to
write documentation, and 2 users editing LSR... well, that would
free up a lot of current bug-fixing-capable contributors to focus
Although GOP is a short-term project, the main goal is to train
more people to handle ongoing jobs. The more people doing these
-jobs, the ligher the work will be, and the more we can get done
+jobs, the lighter the work will be, and the more we can get done
with lilypond!
Also, it would be nice if we had at least one "replacement" /
@item Consultant:
LilyPond is sometimes critized for not listening to users, but
whenever we ask for opinions about specific issues, we never get
-enough feedback. This is somewhat aggrevating.
+enough feedback. This is somewhat aggravating.
We need a group of users to make a dedicated effort to test and
give feedback. If there's new documentation, read it. If there's
an experimental binary, download it and try compiling a score with
documentation. It would be great if somebody could create
properly-formatted patches for these corrections.
-Technical requirements: ability to run @ref{Lilybuntu}.
+Technical requirements: ability to run @ref{LilyDev}.
@item LSR editor:
LSR contains many useful examples of lilypond, but some snippets
often find them in Ponds of Lilies) and new feature implementors.
Technical requirements: development environment (such as
-@ref{Lilybuntu}), ability to read+write scheme and/or C++ code.
+@ref{LilyDev}), ability to read+write scheme and/or C++ code.
@end itemize
@subsection Policy decisions
There are a number of policy decisions -- some of them fairly
-important -- which we have been postponing for a few years. When
-GOP begins, we will start discussing them.
+important -- which we have been postponing for a few years. We
+are now discussing them slowly and thoroughly; agenda and exact
+proposals are online:
+
+@example
+@uref{http://lilypond.org/~graham/gop/index.html}
+@end example
-@warning{The fact that we are not arguing about them right now is
-not, I repeat @strong{not}, an indication that we do not feel that
-these issues are not important. It is simply that if we began
-talking about them now, it would postpone the 2.14 release for a
-few months.}
+Below is a list of policies which are not @qq{on the agenda} yet.
Note that the presence of an item on this list does @emph{not}
mean that everybody thinks that something needs to be done.
developer thinks we should discuss something, just add it to the
bottom of the list. (the list is unsorted)
-Once GOP starts, the list will be sorted into a rough agenda. We
-will probably introduce one topic each week -- yes, it will
-therefore take months to get through everything, but we must
-balance productive work vs. policy administration. If we find
-that we settle questions faster (or slower) than predicted, we
-will of course change the speed of new topic introductions.
+As GOP progresses, items from this list will be put on the agenda
+and removed from this list. I generally try to have one month's
+discussion planned in advance, but I may shuffle things around to
+respond to any immediate problems in the developer community.
There are some item(s) not displayed here; these are questions
that were posed to me privately, and I do not feel justified in
@warning{The estimated time required for "prep work", and the
following discussion, has been added to each item. At the moment,
-there is an estimated 30 hours of prep work and 125 hours of
+there is an estimated 30 hours of prep work and 140 hours of
discussion.}
@itemize
(prep: 2 hours. discuss: 10 hours)
-@item @strong{Lessons from the 2.14 release; future release policy}:
-What went well; what went badly? (how) should we change any
-policies pertaining to releases? Should an undocumented new
-feature count as release-blocking?
+@item @strong{Official links to other organizations?}:
+There's something called the "software freedom conservancy", and
+in general, there's a bunch of "umbrella organizations". Joining
+some of these might give us more visibility, possibly leading to
+more users, more developers, maybe even financial grants or use in
+schools, etc.
-(prep: 1 hour. discuss: 15 hours)
+(prep: 2 hours. discuss: 5 hours)
-@item @strong{lilypond-hackers mailing list}:
-Should we have a private mailing list for senior developers? If
-so, who should be on it?
+@item @strong{Issue tracking with google code}:
+We use the google issue tracker, but this means that we are
+relying on a commercial entity for a large part of our
+development. Would it be better (safer in the long run) to use the
+savannah bug tracker?
+
+(prep: 1 hour. discuss: 5 hours)
+
+@item @strong{Patch review tool}:
+Reitveld is inconvenient in some respects: it requires a google
+account, and there's no way to see all patches relating to
+lilypond. Should we switch to something like gerritt?
+@uref{http://code.google.com/p/lilypond/issues/detail?id=1184}
-(prep: 2 hours+3 weeks. discuss: 10 hours)
+(prep: 5 hours. discuss: 15 hours)
-@item @strong{Hackers B}:
+@item @strong{Clarity for sponsorships}:
+We currently do not advertize bounties and sponsorships on the
+webpage. How much advertising do we want, and what type?
+Should we change the "structure" / "framework" for bounties?
+(prep: 2 hours. discuss: 10 hours)
-@item @strong{Code style}:
-New contributors sometimes struggle to follow our indentation and
-code style -- this is especially difficult when parts of our
-existing source code doesn't have a consistent style. This is
-problematic... we want new contributors to be struggling with the
-lilypond architecture, not playing games in their text editors!
-(ok, we don't actually want them to be struggling with lilypond
-internals... but given the current state of the CG, it's
-understandable, and at any rate it's still better than struggling
-with code style)
-Speaking academically, C++ code style is a "solved problem". Let's
-pick one of the existing solutions (probably either astyle,
-uncrustify, or emacs), and let a computer deal with this.
+@item @strong{code readability}:
+"Our aim when producing source code for Lilypond in whatever
+language is that it should be totally comprehensible to a
+relatively inexperienced developer at the second reading."
+
+Rationale:
+- aids maintainability of code base
+- "second reading" so newer developers can look up unfamiliar
+ stuff
+- will help to keep things simple, even if the code is doing
+ complex stuff discourages "secret squirrel" coding, e.g. "how
+ much functionality can I squeeze into as few characters as
+ possible" "comments are for wimps"
+- will aid not *discouraging* new developers to join the project
+
+(prep: 2 hours. discuss: 10 hours)
+
+@item @strong{C++ vs. scheme}:
+what should be in scheme, what should be in C++, what can/should
+be ported from one to the other, etc. Questions of
+maintainability, speed (especially considering guile 2.0), and the
+amount of current material in either form, are important.
(prep: 5 hours. discuss: 15 hours)
-@item @strong{Git repository(s)}:
-We currently have a web/ branch in our main repo; this seems
-misleading to new developers. More generally, should we have
-branches that aren't related to the master? i.e. should we
-restrict a git branch to code which is an actual "branch" of
-development? Also, some of our code (notably the windows and osx
-lilypad) isn't in a git repository at all.
-We can add new repositories very easily; should make repositories
-like
+@item @strong{always make an issue number for patches}:
+there is a proposal that we should always have a google code issue
+number for every patch. This proposal is closely tied to our
+choice of patch review tool; if we switch to a different tool (as
+suggested in a different proposal), this proposal may become moot.
+
+(prep: 1 hour. discuss: 5 hours)
+
+@item @strong{initalizer lists}:
+shoudl we use initalizer lists for C++? AFAIK they make no
+difference for built-in types, but there's some weird case where
+it's more efficient for objects, or something.
+
+Probably not worth making this a weekly thing on its own, but we
+can probably wrap it up with some other code-related questions.
+
+(prep: 15 minutes. discuss: 3 hours)
+
+@end itemize
+
+@node Policy decisions (finished)
+@subsection Policy decisions (finished)
+
+Here is a record the final decisions, along with links to the
+discussions.
+
+@menu
+* GOP-PROP 1 - python formatting::
+* GOP-PROP 2 - mentors and frogs::
+* GOP-PROP 3 - C++ formatting::
+* GOP-PROP 4 - lessons from 2.14::
+* GOP-PROP 5 - build system output (not accepted)::
+* GOP-PROP 6 - private mailing lists::
+* GOP-PROP 7 - developers as resources::
+* GOP-PROP 8 - issue priorities::
+* GOP-PROP 9 - behavior of make doc::
+@end menu
+
+@node GOP-PROP 1 - python formatting
+@subsubsection GOP-PROP 1 - python formatting
+
+We will follow the indentation described in PEP-8.
+@uref{http://www.python.org/dev/peps/pep-0008/}
+
+@itemize
+@item
+use 4 spaces per indentation level
+
+@item
+never mix tabs and spaces (for indentation)
+
+@item
+Code indented with a mixture of tabs and spaces should be
+converted to using spaces exclusively
+
+Once this is done, we should add @code{python -tt} to the build
+system to avoid such errors in the future.
+
+@end itemize
+
+There should be absolutely no tab characters for indentation in
+any @code{.py} file in lilypond git. All such files should be
+converted to use spaces only.
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00060.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00084.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00310.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00574.html}
+@end smallexample
+
+
+@node GOP-PROP 2 - mentors and frogs
+@subsubsection GOP-PROP 2 - mentors and frogs
+
+Nothing much was decided. The list of responsibilities was
+slightly altered; see the new one in @ref{Mentors}. We should
+encourage more use of the Frogs mailing list. There's a list of
+contributor-mentor pairs in:
+
+@smallexample
+@uref{https://github.com/gperciva/lilypond-extra/blob/master/people/mentors.txt}
+@end smallexample
+
+That's pretty much it.
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00311.html}
+@uref{}
+@uref{}
+@end smallexample
+
+
+@node GOP-PROP 3 - C++ formatting
+@subsubsection GOP-PROP 3 - C++ formatting
+
+Speaking academically, C++ code style is a "solved problem". Let's
+pick one of the existing solutions, and let a computer deal with
+this. Humans should not waste their time, energy, and creativity
+manually adding tabs or spaces to source code.
+
+We have modified @code{fixcc.py} to use astyle, along with extra
+regex tweaks.
+
+@itemize
+@item
+the final script will be run @strong{blindly} on the lilypond
+source code. We will accept whatever formatting the final version
+of this script produces, with no manual tweaking.
+
+@item
+patches which have been run through this tool will not be rejected
+for style reasons. Any code formatting @qq{desires} which are not
+enforced by @code{fixcc.py} will not be considered grounds for
+rejecting a patch.
+
+@item
+for now, this style will not be enforced. It is not cause for
+concern if patches which do not follow the formatting done by
+@code{fixcc.py} are pushed. From time to time, Graham will run
+the formatter on the entire code base, and commit the resulting
+changes.
+
+In a few months, we will tighten up this policy item (with some
+sort of automatic processing), but that is outside the scope of
+this policy item and is a matter for later discussion.
+
+@item
+after the proposal is accepted, we will leave some time for
+existing patches to be accepted and pushed. The script was
+run on the source code on @strong{2011 August 01}.
+
+@end itemize
+
+@subheading GNU code
+
+LilyPond is a GNU project, so it makes sense to follow the GNU
+coding standards. These standards state:
+
+@quotation
+We don’t think of these recommendations as requirements, because
+it causes no problems for users if two different programs have
+different formatting styles.
+
+But whatever style you use, please use it consistently, since a
+mixture of styles within one program tends to look ugly. If you
+are contributing changes to an existing program, please follow the
+style of that program.
+@end quotation
+
+(@uref{http://www.gnu.org/prep/standards/html_node/Formatting.html})
+
+With that in mind, we do not think that we must blindly follow the
+formatting given by the currrent version of Emacs.
+
+@subheading Implementation notes
+
+We can avoid some of the style change pollution in git history by
+ignoring whitespaces changes:
+
@example
-git://git.sv.gnu.org/lilypond/gub.git
-git://git.sv.gnu.org/lilypond/lilypad.git
-git://git.sv.gnu.org/lilypond/misc.git
+git diff -w
@end example
-? More information here:
-@uref{http://code.google.com/p/lilypond/issues/detail?id=980}
-(prep: 2 hours. discuss: 10 hours)
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00526.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00796.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00200.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00525.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00751.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00751.html}
+@end smallexample
+
+
+@node GOP-PROP 4 - lessons from 2.14
+@subsubsection GOP-PROP 4 - lessons from 2.14
+
+@subheading History
+
+A brief history of releases:
+
+@multitable @columnfractions .2 .2 .3
+@headitem date (YYYY-MM-DD) @tab version @tab comment
+@item 2008-10-28 @tab 2.11.63 @tab nobody checking regtests
+@item 2008-11-17 @tab 2.11.64
+@item 2008-11-29 @tab 2.11.65
+@item 2008-12-23 @tab 2.12.0
+@item 2009-01-01 @tab @tab somewhere around here, Graham becomes
+officially release manager, but Han-Wen still builds the actual
+releases
+@item 2009-01-01 @tab 2.12.1
+@item 2009-01-25 @tab 2.12.2
+@item 2009-02-28 @tab 2.13.0
+@item 2009-06-01 @tab 2.13.1 @tab note jump in time!
+@item 2009-06-27 @tab 2.13.2 @tab first Graham release?
+@item 2009-07-03 @tab 2.13.3
+@item 2009-09-09 @tab @tab Graham arrives in Glasgow, gets a
+powerful desktop computer, and begins serious work on GUB (sending
+bug reports to Jan). It takes approximately 100 hours until GUB
+is stable enough to make regular releases.
+@item 2009-09-24 @tab 2.13.4
+@item 2009-10-02 @tab 2.13.5
+@item 2009-10-22 @tab 2.13.6
+@item 2009-11-05 @tab 2.13.7
+@item ...
+@item 2010-01-13 @tab 2.12.3
+@item ...
+@item 2010-03-19 @tab 2.13.16 @tab Bug squad starts doing a few
+regtest comparisons, but IIRC the effort dies out after a few
+weeks (BLUE)
+@item ...
+@item 2010-08-04 @tab 2.13.29 @tab Phil starts checking regtests (BLUE)
+@item ...
+@item 2011-01-12 @tab 2.13.46 @tab release candidate 1 (GREEN)
+@item ...
+@item 2011-05-30 @tab 2.13.63 @tab release candidate 7 (GREEN)
+@item 2011-06-06 @tab 2.14.0
+@end multitable
+
+@c A graphical display of bugs:
+@c
+@c @image{bugs-2.13-visualization,png}
+@c @image{zoom-2.13-visualization,png}
+
+@subheading Carl's analysis of the bugs
+
+A @file{csv} spreadsheet is available.
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00852.html}
+@end smallexample
+
+@example
+@uref{lilypond-issues-analysis.csv}
+@uref{lilypond-issues-analysis-trim-duplicates.csv}
+@end example
-@item @strong{Roadmap of future development}:
-Many projects have a roadmap of planned (or desired) future work.
-Should we use one? If so, what should go on it, bearing in mind
-our volunteer status? Is there any way of having a roadmap that
-isn't vaporware?
+There 148 issues marked with Priority=Critical in the tracker.
-(prep: 1 hour. discuss: 5 hours)
+I've done an analysis, and it looks to me like there was initially
+a backlog of critical issues that weren't fixed, and little work
+was being done to eliminate critical issues.
-@item @strong{Official links to other organizations?}:
-There's something called the "software freedom conservancy", and
-in general, there's a bunch of "umbrella organizations". Joining
-some of these might give us more visibility, possibly leading to
-more users, more developers, maybe even financial grants or use in
-schools, etc.
+Somewhere about 2010-08-01, critical issues started to disappear,
+but occasional new ones appeared.
-(prep: 2 hours. discuss: 5 hours)
+There were a couple of major changes that introduced unanticipated
+regressions (new spacing code, beam collision avoidance). These
+produced more than the expected number of regressions.
-@item @strong{Mailing lists}:
-We currently have a mix of official GNU mailing lists and lilynet
-lists. Is there a strong rationale for having separate mailing
-list servers? Why not pick one place, and put all our lists there?
-(or at least, all "permenant" lists?)
+It appears to me that we didn't really get serious about
+eliminating critical bugs until about 2010-06-15 or so. After
+that point, the number of critical bugs more-or-less steadily
+decreased until we got to a release candidate.
-(prep: 1 hour. discuss: 5 hours)
+Of particular interest, the first release candidate of 2.14 was
+released on 2011-01-12. Over the next 10 days, about a dozen bugs
+were reported and fixed. Release candidate 2 came out on
+2011-02-09. No surge of bugs occurred with this release.
+Candidate 3 came out on 2011-03-13; we got 2 bugs per week.
+Candidate 4 came out on 2011-03-29; 2 new bugs. Candidate 6 came
+out on 2011-04-07. We got a couple of bugs per week.
-@item @strong{Issue tracking with google code}:
-We use the google issue tracker, but this means that we are
-relying on a commercial entity for a large part of our
-development. Would it be better (safer in the long run) to use the
-savannah bug tracker?
+@subheading Notes, commentary, and opinions
-(prep: 1 hour. discuss: 5 hours)
+@example
+Han-Wen: Overall, I think this cycle took too long
+Mike: I agree
+Graham: +1
+@end example
-@item @strong{Patch review tool}:
-Reitveld is inconvenient in some respects: it requires a google
-account, and there's no way to see all patches relating to
-lilypond. Should we switch to something like gerritt?
-@uref{http://code.google.com/p/lilypond/issues/detail?id=1184}
+@subsubheading Discussions
-(prep: 5 hours. discuss: 15 hours)
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00797.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00364.html}
+@uref{}
+@end smallexample
-@item @strong{Subdomains of *.lilypond.org}:
-Unless Jan has a really weird DNS hosting setup, there are no
-technical barriers to having names like lsr.lilypond.org,
-frog.lilypond.org, or news.lilypond.org. Is this something that we
-want to do?
-(prep: 1 hours+2 weeks. discuss: 5 hours)
+@node GOP-PROP 5 - build system output (not accepted)
+@subsubsection GOP-PROP 5 - build system output (not accepted)
-@item @strong{Authorship in source files}:
-Our documentation currently does not attempt to track individual
-authors of each file, while our source code makes a confused and
-jumbled attempt to track this. A number of guidelines for F/OSS
-projects explicitly recommends _not_ tracking this in individual
-files, since the code repository will track that for you.
+This proposal was too broad; after a month of discussion, Graham
+withdrew the proposal. Portions of it will be introduced in later
+proposals.
-(prep: 2 hours. discuss: 15 hours)
+@subsubheading Discussions
-@item @strong{Clarity for sponsorships}:
-We currently do not advertize bounties and sponsorships on the
-webpage. How much advertising do we want, and what type?
-Should we change the "structure" / "framework" for bounties?
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00320.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00527.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00753.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg01042.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00116.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00310.html}
+@end smallexample
-(prep: 2 hours. discuss: 10 hours)
+
+@node GOP-PROP 6 - private mailing lists
+@subsubsection GOP-PROP 6 - private mailing list
+
+Potentially sensitive or private matters will be referred to
+Graham. He will then decide who should discuss the matter on an
+ad-hoc basis, and forward or CC them on future emails.
+
+For emphasis, the project administrators are Han-Wen, Jan, and
+Graham; those three will always be CC'd on any important
+discussions.
+
+The lilypond-hackers mailing list will be removed.
+
+@subheading History
+
+There is some unhappy history about this idea in our development
+community:
+
+@example
+@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-09/msg00178.html}
+@uref{http://news.lilynet.net/spip.php?article121}
+@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00076.html}
+@end example
+
+@subheading Other projects
+
+The idea of private mailing lists is hardly uncommon in
+open-source software. For example,
+
+@example
+@uref{http://lwn.net/Articles/394660/} about debian-private
+@uref{http://subversion.apache.org/mailing-lists.html} private@@
+@uref{http://www.freebsd.org/administration.html#t-core}
+@uref{http://foundation.gnome.org/legal/} board members pledge
+to keep certain matters confidential
+
+every security team of every GNU/Linux distribution and OS
+@end example
+
+In fact, Karl Fogel's @qq{Producing Open Source Software}
+explicitly suggests a private mailing list for some circumstances:
+
+@example
+[on granting commit/push access to a contributor]
+
+But here is one of the rare instances where secrecy is
+appropriate. You can't have votes about potential committers
+posted to a public mailing list, because the candidate's feelings
+(and reputation) could be hurt.
+
+@uref{http://producingoss.com/en/consensus-democracy.html#electorate}
+@end example
+
+@subheading Board of governers, voting, etc?
+
+Many projects have an official board of directors, or a list of
+@qq{core developers}, with set term limits and elections and
+stuff.
+
+I don't think that we're that big. I think we're still small
+enough, and there's enough trust and consensus decisions, that we
+can avoid that. I would rather that we kept on going with
+trust+consensus for at least the next 2-3 years, and spent more
+time+energy on bug fixes and new features instead of
+administrative stuff.
+
+Project administrators are Han-Wen, Jan, and Graham.
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00783.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg01004.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00117.html}
+@end smallexample
+
+
+@node GOP-PROP 7 - developers as resources
+@subsubsection GOP-PROP 7 - developers as resources
+
+We shall treat developers (and contributors) as
+@strong{Independent volunteers}: each person does whatever they
+want, whenever they want. We have busy careers and lives; we make
+no expectations of action from anybody (with the exception of the
+6 people in @qq{Meister} positions).
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg01092.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00087.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00497.html}
+@end smallexample
+
+
+@node GOP-PROP 8 - issue priorities
+@subsubsection GOP-PROP 8 - issue priorities
+
+We will delete the @qq{priority} field of the issue tracker
+altogether. The @qq{type} system will be tweaked.
+
+Type-critical:
+
+@itemize
+
+@item
+a reproducible failure to build either @code{make} or @code{make
+doc}, from an empty build tree, in a first run, if
+@code{configure} does not report any errors.
+
+@item
+any program behaviour which is @strong{unintentionally} worse than
+the previous stable version or the current development version.
+Developers may always use the @qq{this is intentional}, or even
+the @qq{this is an unavoidable effect of an improvement in another
+area}, reason to move this to a different type.
+
+@item
+anything which stops contributors from helping out (e.g.
+lily-git.tcl not working, source tree(s) not being available,
+LilyDev being unable to compile git master, inaccurate
+instructions in the Contributor's Guide 2 Quick start).
+
+To limit this scope of this point, we will assume that the
+contributor is using the latest LilyDev and has read the relevant
+part(s) of the Contributor's Guide. Problems in other chapters of
+the CG are not sufficient to qualify as Type-Critical.
@end itemize
+@subsubheading More new/changed types and labels
+
+Unless otherwise specified, the current types and labels will
+continue to be used. The new types introduced by this proposal
+are:
+
+@itemize
+
+@item
+Type-crash: any segfault, regardless of what the input file looks
+like or which options are given. Disclaimer: this might not be
+possible in some cases, for example certain guile programs (we
+certainly can't predict if a piece of scheme will ever stop
+running, i.e. the halting problem), or if we rely on other
+programs (i.e. ghostscript). If there are any such cases that
+make segfault-prevention impossible, we will document those
+exceptions (and the issue will remain as a "crash" instead of
+"documentation" until the warning has been pushed).
+
+@item
+Type-maintainability: anything which makes it difficult for
+serious contributors to help out (e.g. difficult to find the
+relevant source tree(s), confusing policies, problems with
+automatic indentation tools, etc).
+
+@item
+Type-ugly: replaces Type-collision, and it will include things
+like bad slurs in addition to actual collision.
+
+@end itemize
+
+A new label will be added:
+
+@itemize
+@item
+(label) Needs_evidence: it is not clear what the correct output
+should look like. We need scans, references, examples, etc.
+
+@end itemize
+
+@subheading Reminding users about stars
+
+We can remind users that they can @qq{star} an issue to indicate
+that they care about it. Since we resolved to treat developers as
+independent volunteers, there is no expectation that anybody will
+look at those stars, but if any developer want to organize their
+work schedule according to the stars, they are welcome to do so.
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00019.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00277.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00413.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00624.html}
+@uref{}
+@end smallexample
+
+
+@node GOP-PROP 9 - behavior of make doc
+@subsubsection GOP-PROP 9 - behavior of make doc
+
+If there are build problems, then it should be easier to find out
+why it's failing. This will be achieved with log files, as well
+as possibly including scripts which automatically display portions
+of those log files for a failing build.
+
+We will also add targets for building a specific manual (for
+quick+easy checking of doc work), as well as for building all
+documentation in a specific language (either English or a
+translated language).
+
+When you run @code{make doc},
+
+@itemize
+
+@item
+All output will be saved to various log files, with the exception
+of output directly from @code{make(1)}.
+
+Note that @code{make(1)} refers to a specific executable file on
+unix computers, and is not a general term for the build system.
+
+@item
+By default, no other output will be displayed on the console, with
+one exception: if a build fails, we might display some portion(s)
+of log file(s) which give useful clues about the reason for the
+failure.
+
+The user may optionally request additional output to be printed;
+this is controlled with the @code{VERBOSE=x} flag. In such cases,
+all output will still be written to log files; the console output
+is strictly additional to the log files.
+
+@item
+Logfiles from calling lilypond (as part of lilypond-book) will go in
+the relevant
+@file{$LILYPOND_BUILD_DIR/out/lybook-db/12/lily-123456.log} file. All
+other logfiles will go in the @file{$LILYPOND_BUILD_DIR/logfiles/}
+directory.
+
+A single @code{make doc} will therefore result in hundreds of log
+files. Log files produced from individual lilypond runs are not
+under our control; apart from that, I anticipate having one or two
+dozen log files. As long as it is clear which log file is
+associated with which operation(s), I think this is entirely
+appropriate. The precise implementation will be discussed for
+specific patches as they appear.
+
+@item
+Both stderr and stdout will be saved in @code{*.log}. The order
+of lines from these streams should be preserved.
+
+@item
+There will be no additional @qq{progress messages} during the
+build process. If you run @code{make --silent}, a non-failing
+build should print absolutely nothing to the screen.
+
+@item
+Assuming that the loglevels patch is accepted, lilypond (inside
+lilypond-book) will be run with --loglevel=WARN.
+@uref{http://codereview.appspot.com/4822055/}
+
+@item
+Ideally, a failing build should provide hints about the reason why
+it failed, or at least hints about which log file(s) to examine.
+
+@end itemize
+
+If this proposal is accepted, none of these policies will be
+assumed to apply to any other aspect of the build system.
+Policies for any other aspect of the build system will be
+discussed in separate proposals.
+
+@subheading Don't cause more build problems
+
+However, there is a danger in this approach, that vital error
+messages can also be lost, thus preventing the cause of the
+failure of a make being found. We therefore need to be
+exceptionally careful to move cautiously, include plenty of tests,
+and give time for people to experiment/find problems in each stage
+before proceeding to the next stage.
+
+This will be done by starting from individual lilypond calls
+within lilypond-book, and slowly moving to @qq{larger} targets of
+the build system -- after the individual lilypond calls are are
+producing the appropriate amount of output and this is saved in
+the right place and we can automatically isolate parts of a
+failing build, we will work on lilypond-book in general, and only
+then will we look at the build system itself.
+
+@subheading Implementation notes
+
+There is an existing make variable QUIET_BUILD, which
+alter the amount of output being displayed
+(@uref{
+http://lilypond.org/doc/v2.15/Documentation/contributor/useful-make-variables}
+). We are not planning on keeping this make variable.
+
+The standard way for GNU packages to give more output is with a
+@code{V=x} option. Presumably this is done by increasing
+@code{x}? If we support this option, we should still write log
+files; we would simply print more of the info in those log files
+to screen.
+
+The command @code{tee} may be useful to write to a file and
+display to stdout (in the case of VERBOSE).
+
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00378.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00703.html}
+@end smallexample
+
+
+@ignore
+@n ode GOP-PROP 10 - scheme indentation
+@s ubsubsection GOP-PROP 10 - scheme indentation
+
+still under discussion
+
+@subsubheading Discussions
+
+@smallexample
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg00625.html}
+@uref{https://lists.gnu.org/archive/html/lilypond-devel/2011-08/msg01026.html}
+@c @uref{}
+@end smallexample
+@end ignore
+
@node Grand LilyPond Input Syntax Standardization (GLISS)
@section Grand LilyPond Input Syntax Standardization (GLISS)
+@subheading Summary
+
+@itemize
+@item
+Start: sortly after 2.14 comes out, which is currently estimated
+to happen in January 2011.
+
+@item
+Length: 6-12 months. We're not going to rush this.
+
+@item
+Goal: define an input which we commit to being
+machine-updateable for the forseeable future. Any future patches
+which change the syntax in a non-convert-ly-able format will be
+rejected. (subject to the limitations, below)
+Once this is finished, we will release lilypond 3.0.
+
+@end itemize
+
+
+@subheading The Problem
+
+One of the biggest complaints people have with lilypond -- other
+than silly thing like "there's no gui" -- is the changing syntax.
+Now, inventing a language or standards is difficult. If you set
+it in stone too soon, you risk being stuck with decisions which
+may limit matters. If you keep on updating the syntax,
+interaction with older data (and other programs!) becomes complex.
+
+@subheading Scope and Limitations
+
+@itemize
+@item
+tweaks will not be included. Anything with \override, \set,
+\overrideProperty, \tweak, \revert, \unset... including even those
+command names themselves... is still fair game for NOT_SMART
+convert-ly updates.
+
+@item
+other than that, everything is on the table. Is it a problem to
+have the tagline inside \header? What should the default behavior
+of \include be?
+
+@item
+we need to get standards for command names. This will help users
+remember them, and reduce the options for future names (and
+potential renamings later on). \commandOn and \commandOff seem to
+work well (should we *always* have an Off command?), but what
+about the "command" part? Should it be \nounVerbOn, or
+\verbNounOn ? Or \verbNotesWithExtraInformationOn ?
+
+@item
+we need standards for the location of commands. Ligature
+brackets, I'm looking at you. (non-postfix notation must die!)
+
+@item
+this Grand Project doesn't affect whether we have a 2.16 or not.
+The main problem will be deciding what to do (with a bit of
+messiness anticipated for \tuplet); we should definitely release a
+2.16 before merging _any_ of these changes.
+
+@item
+we obviously can't /guarantee/ that we'll /never/ make any
+non-convert-ly changes in the basic format. But we *can*
+guarantee that such changes would force lilypond 4.0, and that we
+would only do so for overwhelmingly good reasons.
+
+@end itemize
+
+@subheading Workflow
+
+@itemize
+@item
+We're going to have lots and lots of emails flying around. The
+vast majority won't really fit into either -devel or -user, so
+we'll use a list devoted to syntax issues.
+
+@item
+Once we have a serious proposal that gained general acceptance
+from the separate syntax mailing list, I'll bring it to -devel.
+We're not going to make any changes without discussing it on
+-devel, but if we're going to have huge threads about English
+grammar and silly ideas, and I don't want to clutter up -devel.
+Once whatever chaotic silliness on the syntax list is settled
+down, I'll bring the ideas to -devel.
+
+@item
+as with GDP, I'll moderate the discussion. Not as with mailist
+moderation, but rather by introducing issues at specific times.
+We don't want a free-for-all discussion of all parts of the syntax
+at once; nothing will get resolved.
+
+@item
+Whenever possible, we'll decide on policies at the highest level
+of abstraction. For example, consider \numericTimeSignature,
+\slurUp, \xNotesOn, \startTextSpan, and \verylongfermata. One of
+them starts with the name of the notation first (slur). One has
+an abbreviation (x instead of cross). One has the verb at the end
+(On), another has it at the beginning (start). The adjective can
+come at the beginning (numeric, x) or end (Up). Most are in
+camelCase, but one isn't (verylongfermata).
+
+@item
+Instead of arguing about each individual command, we'll decide on
+abstract questions. Should each command begin the notation-noun,
+or the verb? Should all commands be in camelCase, or should we
+make everything other than articulations in camelCase but make
+articulations all lower-case? Are abbreviations allowed?
+
+@item
+Once we've answered such fundamental questions, most of the syntax
+should fall into place pretty easily. There might be a few odd
+questions left ("is it a span, or a spanner?"), but those can be
+settled fairly quickly.
+
+@end itemize
+
+@subheading Implementation
+
+Nothing until the project is finished, then we declare the next
+stable release (2.16.0 or 2.18.0 ?) to be the final 2.x version,
+release it, then apply all the GLISS syntax changes and start
+testing a beta for 3.0 a week or two later.
+
+@subheading Discussion
+
+Don't respond to any of the specifics yet. Yes, we all have our
+pet irritations (like "what's up with \paper and \layout?!").
+There will be plenty of time to discuss them once GLISS starts.
+
+That said, we have a list of specific items that people really
+wanted to have written down. See @ref{Specific GLISS issues}.
+
+@menu
+* Specific GLISS issues::
+@end menu
+
+
+@node Specific GLISS issues
+@subsection Specific GLISS issues
+
+@itemize
+@item
+add regtests for every piece of syntax (not one-command-per-file,
+but making a few files which, between them, use every single piece
+of syntax.) This is a great test for convert-ly.
+
+@item
+should GLISS cover suggested conventions? (indentation,
+one-bar-per-line, etc -- the kind of stuff we list for the
+lilypond formatting in the docs ?)
+
+@item
+how much (if any) syntactic sugar should we add? i.e.
+@example
+ \instrumentName #'foo
+% instead of
+ \set Staff.instrumentName
+@end example
+? Carl: maybe yes, Neil: no. (for example, it fails for
+pianostaff)
+
+@item
+the values that are used as arguments to common used overrides.
+Sometimes they are a symbol (e.g. #'around), sometimes a
+predefined variable referring to a Scheme value or object (e.g.
+#LEFT, #all-visible ). The main trouble is that for novice users
+it is not clear when there should be an apostrophe and when not.
+
+@item
+When do we need -\command and when is it just \command ?
+
+
+@item
+Command-line options to the lilypond binary. -dfoo counts as a
+tweak; we won't be trying to pin those down.
+
+@item
+@verbatim
+\layout {
+ \context { \Score
+% vs.
+\layout {
+ \context {
+ \Score
+@end verbatim
+
+@item
+If would be pedagogically simpler to realize this difference if
+the syntax was separate if you define a context from scratch (as
+is the case with \RemoveEmptyStaffContext) or if it's defined by
+adding onto an existing context. For example, a syntax like
+
+@verbatim
+\context{
+ % Copy the current settings of the Staff context:
+ \use Staff
+ % do whatever additional settings
+}
+%%% could be used to distinguish from
+\context{
+ % Take settings from a variable:
+ \Variable
+ % do whatever additional settings
+}
+
+%%% and
+
+\context{
+ % Start from scratch:
+ \type ...
+ \name ...
+ \consists ...
+ ...
+}
+@end verbatim
+
+@item
+Capitalization of identifiers: \VoiceOne ?
+
+@item
+@verbatim
+%%% Allow
+{ music expression } * 4
+%%% instead of
+\repeat unfold 4 { music expression }
+@end verbatim
+
+? patch here:
+@smallexample
+@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-04/msg00467.html}
+@end smallexample
+
+@item
+Personally, I find it easier to understand when there's a repeated
+8 in the half-bar position; it's much easier to see that you have
+two groups of 4:
+
+@example
+c8 c c c c8 c c c
+%%% instead of one group of eight:
+c8 c c c c c c c
+@end example
+
+@item
+trivially simple bar-lines:
+
+c1 | c1 |
+
+encourage, allow, or discourage, or disallow?
+
+@item
+indentation of \\ inside a @{@} construct.
+
+
+@item
+barline checks at the end of line should be preceded by at least 2
+spaces? barline checks should line up if possible (i.e. if you
+can use less than 4, 8, X empty spaces before a barline check to
+make them line up?)
+
+@item
+Why doesn't \transpose respect \relative mode?
+
+
+@item
+on \score vs. \new Score
+
+But in the light of a consistent syntax and semantic, I see no
+reason (from the users POV) to disallow it. After all, the real
+top-level context is a \book @{@}, isn't it, and I don't see a point
+in disallowing a \new Score construct just like \new Staff.
+
+From a syntactical POV, I see the following pros for \new Score:
+- You can write \with @{ ... @} for every other context but \Score,
+which (for consistency) should also work with \new Score.
+- When there's a \new Foo Bar, there's also a \context Foo Bar,
+ which makes the same as a parallel instantiation of all Bar's.
+- [Quoting Rune from
+@uref{http://www.mail-archive.com/lilypond-devel@@gnu.org/msg14713.html}
+ "I know that the \score-statement is a syntactical construct,
+but I think it would be nice to hide this fact from the users. I
+think we could make the use of score-block much more intuitive if
+changing the syntax to \new \Score and adding an implicit
+sequential-statement to the score."
+
+
+@item
+Discussion on
+@uref{http://code.google.com/p/lilypond/issues/detail?id=1322}
+about \new vs. \context.
+
+
+@item
+Let users add their own items to the parser? comment 11 on:
+@uref{http://code.google.com/p/lilypond/issues/detail?id=1322}
+
+@item
+should engravers be pluralized (note_heads_engraver) or not
+(note_head_engraver) ?
+
+@item
+should we allow numbers in identifier names? Issue:
+@uref{http://code.google.com/p/lilypond/issues/detail?id=1670}
+
+@item
+should we officially allow accented characters? in general, how
+do we feel about utf-8 stuff?
+
+@item
+for the sake of completeness/simplicity, what about *disallowing*
+the "one-note" form of a music expression? i.e. only allowing
+stuff like
+@verbatim
+ \transpose c d { e1 }
+ \transpose c d << e1 >>
+@end verbatim
+
+and never allowing
+@verbatim
+ \transpose c d e1
+@end verbatim
+
+@item
+What should be the officially encouraged way of writing music for
+transposing instruments? Maybe it should be simplified?
+See http://lists.gnu.org/archive/html/lilypond-user/2011-07/msg00130.html
+
+@end itemize
@node Unsorted policies
make grand-replace
@end example
-Internally, this invokes the script @file{scripts/@/build/@/grand@/-replace@/.py},
+Internally, this invokes the script @file{scripts/build/grand-replace.py},
which performs a regular expression substitution for old-year -> new-year
wherever it finds a valid copyright notice.
-Note that snapshots of third party files such as @file{texinfo@/.tex} should
-not be included in the automatic update; @file{grand@/-replace@/.py} ignores these
+Note that snapshots of third party files such as @file{texinfo.tex} should
+not be included in the automatic update; @file{grand-replace.py} ignores these
files if they are listed in the variable @code{copied_files}.
discuss giving them push access. Unsolicited requests from
contributors for access will almost always be turned down.
-
projects / lilypond.git / blobdiff