LSR updates

[lilypond.git] / Documentation / contributor / administration.itexi

diff --git a/Documentation/contributor/administration.itexi b/Documentation/contributor/administration.itexi
index 88c66bea775ba1c741463d11c79d42fb1bf98499..5420b04b05728ea772d379ea1c12f26f1bc418d6 100644 (file)
--- a/Documentation/contributor/administration.itexi
+++ b/Documentation/contributor/administration.itexi
@@ -1,4 +1,3 @@
-@c -*- coding: utf-8; mode: texinfo; -*-

 @node Administrative policies
 @chapter Administrative policies
 
 @node Administrative policies
 @chapter Administrative policies
 


@@ -9,7 +8,7 @@ don't fit anywhere else.
 * Meta-policy for this document::
 * Environment variables::
 * Meisters::
 * Meta-policy for this document::
 * Environment variables::
 * Meisters::

-* Patchy::
+* Managing Staging and Master branches with Patchy::

 * Administrative mailing list::
 * Grand Organization Project (GOP)::
 * Grand LilyPond Input Syntax Standardization (GLISS)::
 * Administrative mailing list::
 * Grand Organization Project (GOP)::
 * Grand LilyPond Input Syntax Standardization (GLISS)::


@@ -126,7 +125,7 @@ on which part of the job, checks to make sure that everything is
 running smoothly, and has final say on our policy for
 Documentation.  Also includes LSR work.
 
 running smoothly, and has final say on our policy for
 Documentation.  Also includes LSR work.
 

-Currently: Graham
+Currently: None

 
 @item
 Translation Meister: trains new translators, updates the
 
 @item
 Translation Meister: trains new translators, updates the


@@ -135,165 +134,328 @@ directions).
 
 Currently: Francisco
 
 
 Currently: Francisco
 

-@item
-Frog Meister: is responsible for code patches from (relatively)
-inexperienced contributors.  Keeps track of patches, does initial
-reviewing of those patches, sends them to @w{@code{-devel}} when
-they've had some initial review on the Frog list, pesters the
-@w{@code{-devel}} community into actually reviewing said patches, and
-finally pushes the patches once they're accepted.  This person is
-@emph{not} responsible for training new programmers, because that
-would be far too much work -- his/her job is @qq{only} to guide
-completed patches through our process.
+@end itemize

 
 

-Currently: Mike Solomon

 
 

-@end itemize
+@node Managing Staging and Master branches with Patchy
+@section Managing Staging and Master branches with Patchy
+
+@ignore
+The script 'test-patches.py' no longer works with code.google.com since
+Google changed their authentication method.
+@end ignore
+
+@menu
+* Overview of Patchy::
+* Patchy requirements::
+* Installing Patchy::
+* Configuring Patchy::
+* Running the script::
+* Automating Patchy::
+* Troubleshooting Patchy::
+@end menu
+
+@node Overview of Patchy
+@subsection Overview of Patchy
+
+No programmatic skill is required to run Patchy; although knowledge of
+compiling LilyPond and its documentation along with understanding how to
+configure the @var{PATH} environment of your computer is required.  See
+@ref{Working with source code}.

 
 

-@node Patchy
-@section Patchy
+The script @code{lilypond-patchy-staging.py} checks for any new commits
+in @code{remote/origin/staging}, makes sure that the new HEAD compiles
+along with all the LilyPond documentation.  Then finally pushing to
+@code{remote/origin/master}.  This script can be run and left
+unattended, requiring no human intervention.

 
 

-@subheading Introduction
+Patchy can also be configured to send emails after each successful (or
+unsuccessful) operation.  This is not a requirement and is turned off
+by default.

 
 

-Patchy is a set of Python scripts to automate two administrative
-tasks:
+@c Need to explain in more detail how to set up Patchy for email but
+@c as I don't use myself it I have no experience - JL
+
+
+@node Patchy requirements
+@subsection Patchy requirements

 
 @itemize
 
 @itemize

-@item
-@code{lilypond-patchy-staging.py}: checks that new commits in
-@code{staging} can compile the regtests and documentation before
-merging @code{staging} into @code{master}.

 
 

-(completely automatic)
+@item
+A full local copy of the source code.  See
+@ref{Working with source code}.

 
 @item
 
 @item

-@code{test-patches.py}: checks that patches apply to Git @code{master},
-compile, and lets a human check that there are no big unintended
-changes to the regtests.
+All the software needed for compiling LilyPond @emph{and} the
+documentation.  Unlike testing patches, being able to build the full set
+of LilyPond's documentation is required to be able to test & push new
+commits.  See @ref{Compiling}.

 
 

-(requires some human input)
+@item
+Commit access @emph{is} required to test and push new commits, but a
+valid login to @uref{http://code.google.com/} is @emph{not}.  See
+@ref{Commit access}.

 
 @end itemize
 
 
 @end itemize
 

-@subheading Installing Patchy

 
 

-To install Patchy, you should do the following:
+@node Installing Patchy
+@subsection Installing Patchy
+
+The Patchy scripts are not part of the LilyPond code base, but can be
+downloaded from @uref{https://github.com/gperciva/lilypond-extra/}.  The
+scripts and related Python libraries are all located in the
+@file{patches/} directory.
+
+Alternatively, use @code{git clone};
+
+@example
+git clone https://github.com/gperciva/lilypond-extra/
+@end example
+
+This makes it simpler to update the scripts if any changes are ever made
+to them.  Finally, add the location of the @file{patches/} directory to
+your @var{PATH}.
+
+
+@node Configuring Patchy
+@subsection Configuring Patchy
+
+@warning{It is recommended to create a new user on your computer
+specifically to run the Patchy scripts as a security precaution and that
+this user should not have any administrative privileges.  Also do not
+set password protection for your ssh key else you will not be able to
+run the scripts unattended.}

 
 @enumerate
 
 @enumerate

+

 @item
 @item

-Create a new user on your box to run Patchy; this is a security
-step for your own protection.  It is recommended that this should
-not be an administrator.  New users are created from System;
-Administration; Users and Groups.
+Make sure the environment variables @var{LILYPOND_GIT} and
+@var{LILYPOND_BUILD_DIR} are configured appropriately.  See
+@ref{Environment variables}.

 
 @item
 
 @item

-Get the Patchy scripts from
-@example
-@uref{https://github.com/gperciva/lilypond-extra/}
-@end example
-Patchy is in the @file{patches/} directory.
+Manually run either the @code{lilypond-patchy-staging.py} script and
+when prompted:
+
+@smallexample
+Warning: using default config; please edit /home/joe/.lilypond-patchy-config
+Are you sure that you want to continue with the default config? (y/[n])
+@end smallexample
+
+Answer @qq{@code{n}} and press enter.
+
+The next time either of the scripts are run they will use the
+@code{.lilypond-patchy-config} settings copied to your @code{$HOME}
+directory.

 
 @item
 
 @item

-Put the scripts and Python libraries contained in @file{patches} in a
-sensible place on your system; this can be done by appending
-@file{patches/} full path to the @var{PATH} of the user that runs
-Patchy.
+Manually edit the @file{.lilypond-patchy-config} file, located in your
+@code{$HOME} directory to change any of the default settings.
+
+@end enumerate
+
+These include:
+
+@itemize

 
 @item
 
 @item

-Create a new git repository with
+All @code{make} operations are run with;

 @example
 @example

-git clone git://git.sv.gnu.org/lilypond.git
+extra_make_options = -j3 CPU_COUNT=3

 @end example
 @end example

-This will create a directory called lilypond with the repo in it.
-Make sure it's where you want it and name it lilypond-git
-(assuming you want to follow the standard naming conventions).
+
+See @ref{Saving time with the -j option}

 
 @item
 
 @item

-Create environment variables @var{LILYPOND_GIT} and
-@var{LILYPOND_BUILD_DIR}, see @ref{Environment variables}.
+A complete build of all the LilyPond documentation is @emph{not}
+performed;
+@example
+patch_test_build_docs = no
+@end example

 
 @item
 
 @item

-Run Patchy once to set up config files, answer @q{@code{n}} when it
-asks for going on, unless the default config file happens to suit your
-setup:
+Each instance of either a patch test or commit test & push is logged in;

 @example
 @example

-lilypond-patchy-staging.py
+auto_compile_results_dir = ~/lilypond-auto-compile-results/

 @end example
 
 @item
 @end example
 
 @item

-Edit @file{$HOME/.lilypond-patchy-config} to provide the location of
-your local lilypond Git repository, working directories for your build
-directory, your results directory, compiler options and notification
-method.  If you don't want to use email notification, then delete
-everything after @code{smtp_command:}.
+Both scripts will perform their build operations in;
+@example
+build_dir = /tmp/lilypond-autobuild/
+@end example
+
+@end itemize
+
+The script creates a clones of @code{staging} and @code{master}
+branches (prefixed with @code{test-}) with a third branch, called
+@code{test-master-lock} used as a check to protect against two or more
+instances of Patchy being run locally at the same time.
+
+
+@node Running the script
+@subsection Running the script
+
+@code{lilypond-patchy-staging.py} is run @emph{without} any arguments.
+It then checks to see if @code{remote/origin/staging} is
+@qq{further ahead} than @code{remote/origin/master}.
+
+@noindent
+If there are no new differences between the two branches since the last
+run check, the script will report something like this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Success:               No new commits in staging
+@end smallexample
+
+@noindent
+If there are any differences between the two branches since the last
+run check, (or if the script cannot for any reason, locate the last
+instance of a commit that it checked) it will report something like
+this:
+
+@smallexample
+(UTC) Begin LilyPond compile, previous commit at 4726764cb591f622e7893407db0e7d42bcde90d9
+Merged staging, now at:        79e98a773b6570cfa28a15775a9dea3d3e54d6b5
+       Success:                ./autogen.sh --noconfigure
+       Success:                /tmp/lilypond-autobuild/configure --disable-optimising
+...
+@end smallexample
+
+and proceed with running @code{make}, @code{make test} and a
+@code{make doc}.  Unlike @code{test-patches.py} if all the tests pass,
+the script then pushes the changes to @code{remote/origin/master}.
+
+@smallexample
+...
+Success:               nice make clean
+Success:               nice make -j7 CPU_COUNT=7
+Success:               nice make test -j7 CPU_COUNT=7
+Success:               nice make doc -j7 CPU_COUNT=7
+To ssh://joe@@git.sv.gnu.org/srv/git/lilypond.git
+   79e98a7..4726764  test-staging -> master
+       Success:                pushed to master
+@end smallexample
+
+@warning{In the case where any of the @code{lilypond-patchy-staging.py}
+tests fail, do not try to push your own fixes but report the failures to
+the Developers List <lilypond-devel@@gnu.org> for advice.}
+
+
+@node Automating Patchy
+@subsection Automating Patchy
+
+To run as a cron job make sure you have;
+
+@example
+[notification]
+notify_non_action = no
+@end example
+
+in @file{$HOME/.lilypond-patchy-config} to avoid any unintentional email
+flooding:
+
+Assuming that Patchy run a user @qq{patchy}, create a file called
+@file{$HOME/lilypond-patchy.cron}, adapting it as necessary (the
+@code{/2} means @qq{run this every 2 hours}):
+
+@smallexample
+02 0-23/2 * * * /home/patchy/lilypond-extra/patches/lilypond-patchy-staging.py
+@end smallexample
+
+@warning{@code{cron} will not inherit environment variables so you must
+re-define any variables inside @file{$HOME/lilypond-patchy.cron}.  For
+instance, @var{LILYPOND_GIT} may need to be defined if
+@var{git_repository_dir} is not correctly set in
+@file{$HOME/.lilypond-patchy-config}.}
+
+Finally, apply the cron job (you may need superuser privileges for
+this):
+
+@example
+crontab -u patchy /home/patchy/lilypond-patchy.cron
+@end example
+
+
+@node Troubleshooting Patchy
+@subsection Troubleshooting Patchy
+
+The following is a list of the most common messages that the scripts
+may report with explanations.
+
+@smallexample
+this Git revision has already been pushed by an operator other than this Patchy.
+@end smallexample
+
+@itemize

 
 @item
 
 @item

-Ensure that your new user has git push access.  Follow the
-instructions in the CG at @ref{Commit access}.  Do not set
-password protection for the key --- if you do you will not be able
-to run patchy unattended.
+Another, remote, machine has already tested and pushed the new commits
+in staging.

 
 

-@end enumerate
+@item
+You may also see this if the auto-build files have been deleted and this
+computer has previously already pushed the listed commit ID to
+@code{master}.

 
 

-@subheading lilypond-patchy-staging.py
+@end itemize

 
 

-@code{lilypond-patchy-staging.py} is run with

 @example
 @example

-python lilypond-patchy-staging.py
+test-master-lock and PID entry exist but previous Patchy
+run (PID xxxxx) died, resetting test-master-lock anyway.

 @end example
 @end example

-Not much appears to happen except you can see a lot of CPU gets used
-if you open System Monitor. There's not much point running
-@code{lilypond-patchy-staging.py} unless there is something in
-@code{staging} to be merged to @code{master}, however, if there's
-nothing new in @code{staging} then the script won't waste resources by
-compiling anything.
-
-The script fetches the current patches in staging and runs
-@code{make}, @code{make test} and @code{make doc} to ensure that all of
-these complete error-free. If you have set Patchy up to use email,
-it emails its results to you.  If you haven't, then you can view
-them in a logfile.  It also merges @code{staging} into @code{master}.
-
-When you have run Patchy a few successful times with email sending,
-you are ready for running it as a cron job. First, make sure you have
-the following in @file{$HOME/.lilypond-patchy-config} to avoid
-email flood:
+
+@noindent
+A previous attempt was unsuccessful for some reason and the scripts were
+not able to tidy up after themselves (for example if you manually halt
+the process by killing it or closing the terminal you may have been
+running the script in).  The @code{test-master-lock} branch was
+therefore not able to be deleted cleanly however, nothing needs to be
+done the scripts will rebuild any tests it needs to.

 
 @example
 
 @example

-[notification]
-notify_non_action = no
+fatal: A branch named 'test-master-lock' already exists.

 @end example
 
 @end example
 

-Then, assuming Patchy run with user account @code{patchy}, write the
-following to @file{$HOME/lilypond-patchy.cron}, adapting it as
-necessary (the @code{/2} means @qq{run this every 2 hours}):
+@itemize
+
+@item
+There is another instance of Patchy running on your computer that is
+testing the same tracker issue.
+
+@item
+A previous test attempt was unsuccessful for some reason and the scripts
+were not able to tidy up after themselves (for example if you manually
+halt the testing process by killing it or closing the terminal you may
+have been running the script in).  The @code{test-master-lock} branch
+was therefore not able to be deleted cleanly, in this case you must
+manually delete the @code{test-master-lock} branch in your
+@code{$LILYPOND_GIT} directory.
+

 @example
 @example

-02 0-23/2 * * * /home/patchy/git/lilypond-extra/patches/lilypond-patchy-staging.py
+git branch -d test-master-lock

 @end example
 
 @end example
 

-@warning{@code{cron} will not inherit environment variables from
-your main setup, so you must re-define any variables inside
-@file{$HOME/lilypond-patchy.cron}. For instance, @var{LILYPOND_GIT}
-may need to be defined if @var{git_repository_dir} is not correctly
-set in @file{$HOME/.lilypond-patchy-config}.}
+@noindent
+It may be wise to also manually delete @code{test-master} and
+@code{test-staging} too, just to be safe.
+
+@end itemize

 
 

-Finally, install the cron job (you may need superuser privileges for
-this):

 @example
 @example

-crontab -u patchy /home/patchy/lilypond-patchy.cron
+*** FAILED STEP ***
+        merge from staging
+        Another instance (PID xxxxx) is already running.

 @end example
 
 @end example
 

-@subheading test-patches.py
-@code{test-patches.py} prepares a regtest comparison for a human to
-quickly glance at, to determine if the patch is ready for a review.
-After looking at the comparison (or the lack of a comparison in the
-case of problems), run @code{accept-patch.py} or
-@code{reject-patch.py}.
+@noindent
+This occurs when trying to run @code{lilypond-patchy-staging.py} when
+another instance of either script is already running locally.
+

 
 

-Once a patch has gotten a "LGTM" from Patchy, it should be
-reviewed by relevant developers, and if it passes this, it can be
-considered for countdown (see @ref{Commits and patches}) and
-pushing to staging (see @ref{Pushing to staging}).

 
 
 @node Administrative mailing list
 
 
 @node Administrative mailing list


@@ -916,7 +1078,7 @@ community:
 
 @example
 @uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-09/msg00178.html}
 
 @example
 @uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-09/msg00178.html}

-@uref{http://news.lilynet.net/spip.php?article121}
+@uref{http://web.archive.org/web/20110325004849/http://news.lilynet.net/spip.php?article121}

 @uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00076.html}
 @end example
 
 @uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00076.html}
 @end example
 


@@ -932,7 +1094,7 @@ open-source software.  For example,
 @uref{http://foundation.gnome.org/legal/}   board members pledge
 to keep certain matters confidential
 
 @uref{http://foundation.gnome.org/legal/}   board members pledge
 to keep certain matters confidential
 

-every security team of every linux distribution and OS
+every security team of every GNU/Linux distribution and OS

 @end example
 
 In fact, Karl Fogel's @qq{Producing Open Source Software}
 @end example
 
 In fact, Karl Fogel's @qq{Producing Open Source Software}


@@ -1264,8 +1426,7 @@ convert-ly updates.
 @item
 other than that, everything is on the table.  Is it a problem to
 have the tagline inside \header?  What should the default behavior
 @item
 other than that, everything is on the table.  Is it a problem to
 have the tagline inside \header?  What should the default behavior

-of \include be?  When we abolish \times, do we move to \tuplet 3:2
-or \tuplet 2/3 or what (for typical triplets in 4/4 time)?
+of \include be?

 
 @item
 we need to get standards for command names.  This will help users
 
 @item
 we need to get standards for command names.  This will help users