-@c -*- coding: utf-8; mode: texinfo; -*-
@node Administrative policies
@chapter Administrative policies
@menu
* Meta-policy for this document::
+* Environment variables::
* Meisters::
+* Managing Staging and Master branches with Patchy::
* Administrative mailing list::
* Grand Organization Project (GOP)::
* Grand LilyPond Input Syntax Standardization (GLISS)::
@end itemize
+@node Environment variables
+@section Environment variables
+
+Some maintenance scripts and instructions in this guide rely on
+the following environment variables. They should be predefined in
+LilyDev distribution (see @ref{LilyDev}); if you set up your own
+development environment, you can set them by appending these settings to
+your @file{~/.bashrc} (or whatever defines your default environment
+variables for the user account for LilyPond development), then logging
+out and in (adapt directories to your setup):
+
+@example
+LILYPOND_GIT=~/lilypond-git
+export LILYPOND_GIT
+LILYPOND_BUILD_DIR=~/lilypond-git/build
+export LILYPOND_BUILD_DIR
+@end example
+
+The standard build and install procedure (with @code{autogen.sh},
+@code{configure}, @code{make}, @code{make install}, @code{make doc}
+@dots{}) does not rely on them.
+
+In addition, for working on the website, @code{LILYPOND_WEB_MEDIA_GIT}
+should be set to the repository lilypond-extra, see
+@ref{lilypond-extra}.
+
@node Meisters
@section Meisters
-We have four jobs for organizing a team of contributors:
+We have four primary jobs to help organize all our contributors:
+
+@unnumberedsubsec The Bug Meister
+
+The Bug Meister's responsibilities are:
+
+@itemize
+
+@item
+To organize the individual Bug Squad volunteers, making sure that
+each member is aware of their responsibilities. See
+@ref{The Bug Squad}.
+
+@item
+To train new Bug Squad volunteers in the Issue Tracker process. See
+@ref{Issues}.
+
+@item
+To have the final say on our policies for Issues and their
+classification. See @ref{Issue classification}.
+
+@end itemize
+
+Current Bug Meister: Colin Hall @email{bug-lilypond@@gnu.org}
+
+
+@unnumberedsubsec The Doc Meister
+
+The Doc Meister's responsibilities are:
@itemize
@item
-Bug Meister: trains new Bug Squad volunteers, organizes who works
-on which part of their job, checks to make sure that everything is
-running smoothly, and has final say on our policy for Issues.
+To train new volunteers in our Documentation style and policy,
+including organizing LilyPond Snippet Repository (LSR) work.
+
+@item
+To organize the individual volunteers -- who does what on which job --
+and to check that everything is running smoothly.
+
+@item
+To have final say on any Documentation policy. See
+@ref{Documentation policy}.
+
+@end itemize
+
+Current Doc Meister: None
+
+
+@unnumberedsubsec The Patch Meister
-Currently: Phil
+The Patch Meister's responsibilities are:
+
+@itemize
@item
-Doc Meister: trains new doc editors/writers, organizes who works
-on which part of the job, checks to make sure that everything is
-running smoothly, and has final say on our policy for
-Documentation. Also includes LSR work.
+To keep track of all patches submitted for testing and review. This
+includes scanning the bug and dev email lists looking for any patches
+submitted by @q{random} contributors and advising them on how to submit
+a patch for testing and review. See @ref{Uploading a patch for review}
+and @ref{The patch review cycle}.
-Currently: Graham
+@item
+To makes sure that any patch submitted has a corresponding Issue Tracker
+and Rietveld Issue created for it before it enters the testing and
+review process. See @ref{Issues}.
@item
-Translation Meister: trains new translators, updates the
-translation priority list, and handles merging branches (in both
-directions).
+Updates all Issue statuses for all patches that are currently in the
+testing and review process periodically -- currently every 3 - 4 days.
+See @ref{Patch handling}.
+
+@end itemize
+
+@warning{The Patch Meister's role is a purely administrative one and no
+programming skill or judgement is assumed or required.}
+
+Currently: James Lowe @email{pkx@@gnu.org}
+
+
+@unnumberedsubsec The Translation Meister
+
+The Translation Meister's responsibilities are:
-Currently: Francisco
+@itemize
+
+@item
+To train new documentation translators in the translation process. See
+@ref{Translating the documentation}.
@item
-Frog Meister: is responsible for code patches from (relatively)
-inexperienced contributors. Keeps track of patches, does initial
-reviewing of those patches, sends them to @w{@code{-devel}} when
-they've had some initial review on the Frog list, pesters the
-@w{@code{-devel}} community into actually reviewing said patches, and
-finally pushes the patches once they're accepted. This person is
-@emph{not} responsible for training new programmers, because that
-would be far too much work -- he job is @qq{only} to guide
-completed patches through our process.
+To update the translation priority list and handle the merging of the
+translation branches (in both directions).
-Currently: Carl
+@item
+To have final say on any Translation management policies. See
+@ref{Translating the documentation}.
@end itemize
+Currently: Francisco Vila @email{translations@@lilynet.net}
+
+
+
+@node Managing Staging and Master branches with Patchy
+@section Managing Staging and Master branches with Patchy
+
+@ignore
+The script 'test-patches.py' does not currently work with Allura.
+@end ignore
+
+@menu
+* Overview of Patchy::
+* Patchy requirements::
+* Installing Patchy::
+* Configuring Patchy::
+* Running the script::
+* Automating Patchy::
+* Troubleshooting Patchy::
+@end menu
+
+@node Overview of Patchy
+@subsection Overview of Patchy
+
+No programmatic skill is required to run Patchy; although knowledge of
+compiling LilyPond and its documentation along with understanding how to
+configure the @var{PATH} environment of your computer is required. See
+@ref{Working with source code}.
+
+The script @code{lilypond-patchy-staging.py} checks for any new commits
+in @code{remote/origin/staging}, makes sure that the new HEAD compiles
+along with all the LilyPond documentation. Then finally pushing to
+@code{remote/origin/master}. This script can be run and left
+unattended, requiring no human intervention.
+
+Patchy can also be configured to send emails after each successful (or
+unsuccessful) operation. This is not a requirement and is turned off
+by default.
+
+@c Need to explain in more detail how to set up Patchy for email but
+@c as I don't use myself it I have no experience - JL
+
+
+@node Patchy requirements
+@subsection Patchy requirements
+
+@itemize
+
+@item
+A full local copy of the source code. See
+@ref{Working with source code}.
+
+@item
+All the software needed for compiling LilyPond @emph{and} the
+documentation. Unlike testing patches, being able to build the full set
+of LilyPond's documentation is required to be able to test & push new
+commits. See @ref{Compiling}.
+
+@item
+Commit access @emph{is} required to test and push new commits, but a
+valid login to @uref{https://sourceforge.net} is @emph{not}. See
+@ref{Commit access}.
+
+@end itemize
+
+
+@node Installing Patchy
+@subsection Installing Patchy
+
+The Patchy scripts are not part of the LilyPond code base, but can be
+downloaded from @uref{https://github.com/gperciva/lilypond-extra/}. The
+scripts and related Python libraries are all located in the
+@file{patches/} directory.
+
+Alternatively, use @code{git clone};
+
+@example
+git clone https://github.com/gperciva/lilypond-extra/
+@end example
+
+This makes it simpler to update the scripts if any changes are ever made
+to them. Finally, add the location of the @file{patches/} directory to
+your @var{PATH}.
+
+
+@node Configuring Patchy
+@subsection Configuring Patchy
+
+@warning{It is recommended to create a new user on your computer
+specifically to run the Patchy scripts as a security precaution and that
+this user should not have any administrative privileges. Also do not
+set password protection for your ssh key else you will not be able to
+run the scripts unattended.}
+
+@enumerate
+
+@item
+Make sure the environment variables @var{LILYPOND_GIT} and
+@var{LILYPOND_BUILD_DIR} are configured appropriately. See
+@ref{Environment variables}.
+
+@item
+Manually run either the @code{lilypond-patchy-staging.py} script and
+when prompted:
+
+@smallexample
+Warning: using default config; please edit /home/joe/.lilypond-patchy-config
+Are you sure that you want to continue with the default config? (y/[n])
+@end smallexample
+
+Answer @qq{@code{n}} and press enter.
+
+The next time either of the scripts are run they will use the
+@code{.lilypond-patchy-config} settings copied to your @code{$HOME}
+directory.
+
+@item
+Manually edit the @file{.lilypond-patchy-config} file, located in your
+@code{$HOME} directory to change any of the default settings.
+
+@end enumerate
+
+These include:
+
+@itemize
+
+@item
+All @code{make} operations are run with;
+@example
+extra_make_options = -j3 CPU_COUNT=3
+@end example
+
+See @ref{Saving time with the -j option}
+
+@item
+A complete build of all the LilyPond documentation is @emph{not}
+performed;
+@example
+patch_test_build_docs = no
+@end example
+
+@item
+Each instance of either a patch test or commit test & push is logged in;
+@example
+auto_compile_results_dir = ~/lilypond-auto-compile-results/
+@end example
+
+@item
+Both scripts will perform their build operations in;
+@example
+build_dir = /tmp/lilypond-autobuild/
+@end example
+
+@end itemize
+
+The script creates a clones of @code{staging} and @code{master}
+branches (prefixed with @code{test-}) with a third branch, called
+@code{test-master-lock} used as a check to protect against two or more
+instances of Patchy being run locally at the same time.
+
+
+@node Running the script
+@subsection Running the script
+
+@code{lilypond-patchy-staging.py} is run @emph{without} any arguments.
+It then checks to see if @code{remote/origin/staging} is
+@qq{further ahead} than @code{remote/origin/master}.
+
+@noindent
+If there are no new differences between the two branches since the last
+run check, the script will report something like this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Success: No new commits in staging
+@end smallexample
+
+@noindent
+If there are any differences between the two branches since the last
+run check, (or if the script cannot for any reason, locate the last
+instance of a commit that it checked) it will report something like
+this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Merged staging, now at: 79e98a773b6570cfa28a15775a9dea3d3e54d6b5
+ Success: ./autogen.sh --noconfigure
+ Success: /tmp/lilypond-autobuild/configure --disable-optimising
+...
+@end smallexample
+
+and proceed with running @code{make}, @code{make test} and a
+@code{make doc}. Unlike @code{test-patches.py} if all the tests pass,
+the script then pushes the changes to @code{remote/origin/master}.
+
+@smallexample
+...
+Success: nice make clean
+Success: nice make -j7 CPU_COUNT=7
+Success: nice make test -j7 CPU_COUNT=7
+Success: nice make doc -j7 CPU_COUNT=7
+To ssh://joe@@git.sv.gnu.org/srv/git/lilypond.git
+ 79e98a7..4726764 test-staging -> master
+ Success: pushed to master
+@end smallexample
+
+@warning{In the case where any of the @code{lilypond-patchy-staging.py}
+tests fail, do not try to push your own fixes but report the failures to
+the Developers List <lilypond-devel@@gnu.org> for advice.}
+
+
+@node Automating Patchy
+@subsection Automating Patchy
+
+To run as a cron job make sure you have;
+
+@example
+[notification]
+notify_non_action = no
+@end example
+
+in @file{$HOME/.lilypond-patchy-config} to avoid any unintentional email
+flooding:
+
+Assuming that Patchy run a user @qq{patchy}, create a file called
+@file{$HOME/lilypond-patchy.cron}, adapting it as necessary (the
+@code{/2} means @qq{run this every 2 hours}):
+
+@smallexample
+02 0-23/2 * * * /home/patchy/lilypond-extra/patches/lilypond-patchy-staging.py
+@end smallexample
+
+@warning{@code{cron} will not inherit environment variables so you must
+re-define any variables inside @file{$HOME/lilypond-patchy.cron}. For
+instance, @var{LILYPOND_GIT} may need to be defined if
+@var{git_repository_dir} is not correctly set in
+@file{$HOME/.lilypond-patchy-config}.}
+
+Finally, apply the cron job (you may need superuser privileges for
+this):
+
+@example
+crontab -u patchy /home/patchy/lilypond-patchy.cron
+@end example
+
+
+@node Troubleshooting Patchy
+@subsection Troubleshooting Patchy
+
+The following is a list of the most common messages that the scripts
+may report with explanations.
+
+@smallexample
+this Git revision has already been pushed by an operator other than this Patchy.
+@end smallexample
+
+@itemize
+
+@item
+Another, remote, machine has already tested and pushed the new commits
+in staging.
+
+@item
+You may also see this if the auto-build files have been deleted and this
+computer has previously already pushed the listed commit ID to
+@code{master}.
+
+@end itemize
+
+@example
+test-master-lock and PID entry exist but previous Patchy
+run (PID xxxxx) died, resetting test-master-lock anyway.
+@end example
+
+@noindent
+A previous attempt was unsuccessful for some reason and the scripts were
+not able to tidy up after themselves (for example if you manually halt
+the process by killing it or closing the terminal you may have been
+running the script in). The @code{test-master-lock} branch was
+therefore not able to be deleted cleanly however, nothing needs to be
+done the scripts will rebuild any tests it needs to.
+
+@example
+fatal: A branch named 'test-master-lock' already exists.
+@end example
+
+@itemize
+
+@item
+There is another instance of Patchy running on your computer that is
+testing the same tracker issue.
+
+@item
+A previous test attempt was unsuccessful for some reason and the scripts
+were not able to tidy up after themselves (for example if you manually
+halt the testing process by killing it or closing the terminal you may
+have been running the script in). The @code{test-master-lock} branch
+was therefore not able to be deleted cleanly, in this case you must
+manually delete the @code{test-master-lock} branch in your
+@code{$LILYPOND_GIT} directory.
+
+@example
+git branch -d test-master-lock
+@end example
+
+@noindent
+It may be wise to also manually delete @code{test-master} and
+@code{test-staging} too, just to be safe.
+
+@end itemize
+
+@example
+*** FAILED STEP ***
+ merge from staging
+ Another instance (PID xxxxx) is already running.
+@end example
+
+@noindent
+This occurs when trying to run @code{lilypond-patchy-staging.py} when
+another instance of either script is already running locally.
+
+
+
+
@node Administrative mailing list
@section Administrative mailing list
-An mailing list for administrative issues is maintained at
+A mailing list for administrative issues is maintained at
@code{lilypond-hackers@@gnu.org}.
This list is intended to be used for discussions that should be kept
@itemize
@item
-Clarify the various development tasks by writing down the polices
+Clarify the various development tasks by writing down the policies
and techniques and/or simplifying the tasks directly.
@item
code.
Having 1 more normal user answering emails on lilypond-user won't
-have a dramatic trick-up affect all by himself, of course. But if
+have a dramatic @q{trickle-up} effect all by itself, of course. But if
we had 8 users volunteering to answer emails, 6 users starting to
write documentation, and 2 users editing LSR... well, that would
free up a lot of current bug-fixing-capable contributors to focus
Although GOP is a short-term project, the main goal is to train
more people to handle ongoing jobs. The more people doing these
-jobs, the ligher the work will be, and the more we can get done
+jobs, the lighter the work will be, and the more we can get done
with lilypond!
Also, it would be nice if we had at least one "replacement" /
documentation. It would be great if somebody could create
properly-formatted patches for these corrections.
-Technical requirements: ability to run @ref{Lilydev}.
+Technical requirements: ability to run @ref{LilyDev}.
@item LSR editor:
LSR contains many useful examples of lilypond, but some snippets
often find them in Ponds of Lilies) and new feature implementors.
Technical requirements: development environment (such as
-@ref{Lilydev}), ability to read+write scheme and/or C++ code.
+@ref{LilyDev}), ability to read+write scheme and/or C++ code.
@end itemize
(prep: 2 hours. discuss: 10 hours)
-@item @strong{Future release policy}:
-(how) should we change any policies pertaining to releases? Should
-an undocumented new feature count as release-blocking?
-
-(prep: 1 hour. discuss: 15 hours)
-
-@item @strong{lilypond-hackers mailing list}:
-Should we have a private mailing list for senior developers? If
-so, who should be on it?
-
-(prep: 2 hours+3 weeks. discuss: 10 hours)
-
-@item @strong{Hackers B}:
-
-
-@item @strong{Git repository(s)}:
-We currently have a web/ branch in our main repo; this seems
-misleading to new developers. More generally, should we have
-branches that aren't related to the master? i.e. should we
-restrict a git branch to code which is an actual "branch" of
-development? Also, some of our code (notably the windows and osx
-lilypad) isn't in a git repository at all.
-We can add new repositories very easily; should make repositories
-like
-@example
-git://git.sv.gnu.org/lilypond/gub.git
-git://git.sv.gnu.org/lilypond/lilypad.git
-git://git.sv.gnu.org/lilypond/misc.git
-@end example
-? More information here:
-@uref{http://code.google.com/p/lilypond/issues/detail?id=980}
-
-(prep: 2 hours. discuss: 10 hours)
-
-@item @strong{Roadmap of future development}:
-Many projects have a roadmap of planned (or desired) future work.
-Should we use one? If so, what should go on it, bearing in mind
-our volunteer status? Is there any way of having a roadmap that
-isn't vaporware?
-
-(prep: 1 hour. discuss: 5 hours)
-
@item @strong{Official links to other organizations?}:
There's something called the "software freedom conservancy", and
in general, there's a bunch of "umbrella organizations". Joining
(prep: 2 hours. discuss: 5 hours)
-@item @strong{Mailing lists}:
-We currently have a mix of official GNU mailing lists and lilynet
-lists. Is there a strong rationale for having separate mailing
-list servers? Why not pick one place, and put all our lists there?
-(or at least, all "permanent" lists?)
-
-(prep: 1 hour. discuss: 5 hours)
-
@item @strong{Issue tracking with google code}:
We use the google issue tracker, but this means that we are
relying on a commercial entity for a large part of our
Reitveld is inconvenient in some respects: it requires a google
account, and there's no way to see all patches relating to
lilypond. Should we switch to something like gerritt?
-@uref{http://code.google.com/p/lilypond/issues/detail?id=1184}
+@uref{https://sourceforge.net/p/testlilyissues/issues/1184/}
(prep: 5 hours. discuss: 15 hours)
-@item @strong{Subdomains of *.lilypond.org}:
-Unless Jan has a really weird DNS hosting setup, there are no
-technical barriers to having names like lsr.lilypond.org,
-frog.lilypond.org, or news.lilypond.org. Is this something that we
-want to do?
-
-(prep: 1 hours+2 weeks. discuss: 5 hours)
-
-@item @strong{Authorship in source files}:
-Our documentation currently does not attempt to track individual
-authors of each file, while our source code makes a confused and
-jumbled attempt to track this. A number of guidelines for F/OSS
-projects explicitly recommends _not_ tracking this in individual
-files, since the code repository will track that for you.
-
-(prep: 2 hours. discuss: 15 hours)
-
@item @strong{Clarity for sponsorships}:
We currently do not advertize bounties and sponsorships on the
webpage. How much advertising do we want, and what type?
(prep: 2 hours. discuss: 10 hours)
-@item @strong{Separate branches for active development}:
-it might be good to have @emph{everybody} working on separate
-branches. This complicates the git setup, but with sufficient
-logic in lily-git.tcl, we can probably make it transparent to
-newbies. However, we'd need a reliable person to handle all the
-required merging and stuff.
-
-(prep: 2 hours. discuss: 10 hours)
-
-@item @strong{When do we add regtests?}:
-There is a discrepancy between our stated policy on adding
-regtests, and our actual practice in handling bugs and patches.
-Clarify.
-
-There is also a wider question how to organize the regtests, such
-as where to put interesting-console-output regtests, including
-stuff like lilypond-book and midi2ly in a sensible manner, and
-possibly including regtests for currently-broken functionality.
-
-(prep: 2 hours. discuss: 5 hours)
-
@item @strong{code readability}:
-"Our aim when producing source code for Lilypond in whatever
+"Our aim when producing source code for LilyPond in whatever
language is that it should be totally comprehensible to a
relatively inexperienced developer at the second reading."
@example
@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-09/msg00178.html}
-@uref{http://news.lilynet.net/spip.php?article121}
+@uref{http://web.archive.org/web/20110325004849/http://news.lilynet.net/spip.php?article121}
@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00076.html}
@end example
@uref{http://foundation.gnome.org/legal/} board members pledge
to keep certain matters confidential
-every security team of every linux distribution and OS
+every security team of every GNU/Linux distribution and OS
@end example
In fact, Karl Fogel's @qq{Producing Open Source Software}
@item
anything which stops contributors from helping out (e.g.
lily-git.tcl not working, source tree(s) not being available,
-lilydev being unable to compile git master, inaccurate
+LilyDev being unable to compile git master, inaccurate
instructions in the Contributor's Guide 2 Quick start).
To limit this scope of this point, we will assume that the
-contributor is using the latest lilydev and has read the relevant
+contributor is using the latest LilyDev and has read the relevant
part(s) of the Contributor's Guide. Problems in other chapters of
the CG are not sufficient to qualify as Type-Critical.
is strictly additional to the log files.
@item
-Logfiles from calling lilypond (as part of lilypond-book) will go
-in the relevant @file{build/out/lybook-db/12/lily-123456.log}
-file. All other logfiles will go in the @file{build/logfiles/}
+Logfiles from calling lilypond (as part of lilypond-book) will go in
+the relevant
+@file{$LILYPOND_BUILD_DIR/out/lybook-db/12/lily-123456.log} file. All
+other logfiles will go in the @file{$LILYPOND_BUILD_DIR/logfiles/}
directory.
A single @code{make doc} will therefore result in hundreds of log
@item
other than that, everything is on the table. Is it a problem to
have the tagline inside \header? What should the default behavior
-of \include be? When we abolish \times, do we move to \tuplet 3:2
-or \tuplet 2/3 or what (for typical triplets in 4/4 time)?
+of \include be?
@item
we need to get standards for command names. This will help users
@item
Discussion on
-@uref{http://code.google.com/p/lilypond/issues/detail?id=1322}
+@uref{https://sourceforge.net/p/testlilyissues/issues/1322/}
about \new vs. \context.
@item
Let users add their own items to the parser? comment 11 on:
-@uref{http://code.google.com/p/lilypond/issues/detail?id=1322}
+@uref{https://sourceforge.net/p/testlilyissues/issues/1322/}
@item
should engravers be pluralized (note_heads_engraver) or not
@item
should we allow numbers in identifier names? Issue:
-@uref{http://code.google.com/p/lilypond/issues/detail?id=1670}
+@uref{https://sourceforge.net/p/testlilyissues/issues/1670/}
@item
should we officially allow accented characters? in general, how
projects / lilypond.git / blobdiff