@chapter Working with source code
@warning{New contributors should read @ref{Quick start}, and in
-particular @ref{Using lily-git}, instead of this chapter.}
+particular @ref{lily-git}, instead of this chapter.}
Advanced contributors will find this material quite useful,
particularly if they are working on major new features.
skip ahead to @ref{Starting with Git}.
@warning{These instructions are only for people who are @emph{not}
-using @ref{Lilydev}.}
+using @ref{LilyDev}.}
@c there's some duplication in this section with stuff covered in
@c Quick Start, but moving it into a macro inside included/ would
@qq{Full installer for official Git} from:
@example
-@uref{http://code.google.com/p/msysgit/downloads/list}
+@uref{https://git-for-windows.github.io/}
@end example
@item Other operating systems: either install @command{git} with
This will create a directory called @file{lilypond-git/} within
your home directory, and will download the source code into that
-directory (around 55Mb). When the process is finished, the
+directory (around 150@tie{}Mb). When the process is finished, the
@qq{Command output} window will display @qq{Done}, and the button
label will change to say @qq{Update source}.
@end enumerate
@warning{Throughout the rest of this manual, most command-line
-input should be entered from @file{~/lilypond-git/}. This is
+input should be entered from @file{$LILYPOND_GIT}. This is
referred to as the @emph{top source directory}.}
-Further instructions are in @ref{Daily use of lily-git.tcl}.
-
-@advanced{the @qq{Get source} button does not fetch the entire
-history of the git repository, so utilities like @command{gitk}
-will only be able to display the most recent additions. As you
-continue to work with @command{lily-git.tcl}, the @qq{Update
-source} button will take any new additions and add it to whatever
-is currently in your repository's history.}
+Further instructions are in @ref{How to use lily-git}.
@node Starting with Git
@menu
* Setting up::
+* Git for the impatient::
+* Other repositories::
* Downloading remote branches::
@end menu
@node Setting up
@subsection Setting up
-
-TODO: Remove this note if incorporating Windows instructions
-throughout this section:
-
@warning{These instructions assume that you are using the
command-line version of Git 1.5 or higher. Windows users should
skip to @ref{Git on Windows}.}
@node Installing Git
@unnumberedsubsubsec Installing Git
-
If you are using a Unix-based machine, the easiest way to download
and install Git is through a package manager such as @command{rpm}
-or @command{apt-get}---the installation is generally automatic.
+or @command{apt-get} -- the installation is generally automatic.
The only required package is (usually) called @command{git-core},
although some of the auxiliary @command{git@var{*}} packages are
also useful (such as @command{gitk}).
(@uref{http://git-scm.com/}) for downloadable binaries and
tarballs.
-TODO: add Windows installation instructions (or @@ref@{Git on
-Windows@}).
-
@node Initializing a repository
@unnumberedsubsubsec Initializing a repository
-
-Once Git is installed, you'll need to create a new directory where
-your initial repository will be stored (the example below uses
-@file{~/lilypond-git/}, where @code{~} represents your home
-directory). Run @command{git@tie{}init} from within the new
-directory to initialize an empty repository:
+Once Git is installed, get a copy of the source code:
@example
-mkdir ~/lilypond-git/; cd ~/lilypond-git/
-git init
+git clone git://git.sv.gnu.org/lilypond.git ~/lilypond-git
@end example
+The above command will put the it in @file{~/lilypond-git}, where
+@code{~} represents your home directory.
+
@subsubheading Technical details
-This creates (within the @file{~/lilypond-git/} directory) a
+This creates (within the @file{$LILYPOND_GIT} directory) a
subdirectory called @file{.git/}, which Git uses to keep track of
changes to the repository, among other things. Normally you don't
need to access it, but it's good to know it's there.
@warning{Throughout the rest of this manual, all command-line
input should be entered from the top directory of the Git
-repository being discussed (eg. @file{~/lilypond-git/}). This is
+repository being discussed (eg. @file{$LILYPOND_GIT}). This is
referred to as the @emph{top source directory}.}
-Before downloading a copy of the main LilyPond repository, you
+Before working with the copy of the main LilyPond repository, you
should configure some basic settings with the
@command{git@tie{}config} command. Git allows you to set both
global and repository-specific options.
git config --global core.editor @var{nano}
@end example
-TODO: Add instructions for changing the editor on Windows, which
-is a little different, I think. -mp
+Finally, and in some ways most importantly, let's make sure that
+we can easily see the state of our working copy, without the need
+of typing @code{git status} repeatedly. If you're not using
+LilyDev, add the following lines to your @file{~/.bashrc}:
+
+@verbatim
+export PS1="\u@\h \w\$(__git_ps1)$ "
+export GIT_PS1_SHOWDIRTYSTATE=true
+export GIT_PS1_SHOWUNTRACKEDFILES=true
+export GIT_PS1_SHOWUPSTREAM=auto
+@end verbatim
+
+The first line will show the branch we're on. The other lines
+will use some symbols next to the branch name to indicate some
+kind of state. @qq{*} means that there are unstaged changes,
+@qq{+} indicates staged changes; if there are untracked files,
+a @qq{%} will appear. Finally, we can also see if our HEAD is
+behind (@qq{<}) or ahead (@qq{>}) of its upstream, and if they
+have diverged (@qq{<>}) or they are synced (@qq{=}).
+
+You may need to install the additional @code{bash-completion}
+package, but it is definitely worth it. After installation
+you must log out, and then log back in again to enable it.
+
@subsubheading Technical details
in this manual.
+@node Git for the impatient
+@subsection Git for the impatient
+
+@advanced{The intent of this subsection is to get you working on lilypond as
+soon as possible. If you want to learn about git, go read
+@ref{Other Git documentation}.
+@*
+Also, these instructions are designed to eliminate the most common
+problems we have found in using git. If you already know git and
+have a different way of working, great! Feel free to ignore the
+advice in this subsection.}
+
+
+Ok, so you've been using @command{lily-git.tcl} for a while, but
+it's time to take the next step. Since our review process delays
+patches by 60-120 hours, and you want to be able to work on other
+stuff while your previous work is getting reviewed, you're going
+to use @strong{branches}.
+
+You can think of a branch as being a separate copy of the source
+code. But don't worry about it.
+
+@subsubheading Start work: make a new branch
+
+Let's pretend you want to add a section to the Contributor's Guide
+about using branches.
+
+Start by updating the repository, then making a new branch. Call
+the branch anything you want as long as the name starts with
+@code{dev/}. Branch names that don't begin with @code{dev/} are
+reserved for special things in lilypond.
+
+@example
+git checkout master
+git pull -r origin master
+git branch dev/cg
+@end example
+
+@subsubheading Switch to that branch
+
+Nothing has happened to the files yet. Let's change into the new
+branch. You can think of this as @qq{loading a file}, although in
+this case it's really @qq{loading a directory and subdirectories
+full of files}.
+
+@example
+git checkout dev/cg
+@end example
+
+Your prompt now shows you that you're on the other branch:
+
+@example
+gperciva@@LilyDev:~/lilypond-git (dev/cg)$
+@end example
+
+To be able to manage multiple lilypond issues at once, you'll need to switch
+branches. You should have each lilypond issue on a separate branch.
+Switching branches is easy:
+
+@example
+git checkout master
+git checkout origin/staging
+git checkout origin/release/unstable
+git checkout dev/cg
+@end example
+
+Branches that begin with @code{origin/} are part of the remote repository,
+rather than your local repository, so when you check them out you get a
+temporary local branch. You should never make changes directly on a
+branch beginning with @code{origin/}. You get changes into the remote
+repository by making them in local branches, and then pushing them to
+@code{origin/staging} as described below.
+
+@subsubheading Make your changes
+
+Edit files, then commit them.
+
+@example
+git commit -a
+@end example
+
+
+Remember how I said that switching to a branch was like
+@qq{loading a directory}? Well, you've just @qq{saved a
+directory}, so that you can @qq{load} it later.
+
+@advanced{If you have used @command{cvs} or @command{svn}, you may
+be very confused: those programs use @qq{commit} to mean
+@qq{upload my changes to the shared source repository}.
+Unfortunately, just to be different, @w{@command{git commit}}
+means @qq{save my changes to the files}.}
+
+When you create a new file, you need to add it to git, then commit it:
+
+@example
+git add input/regression/avoid-crash-on-condition.ly
+git commit -a
+@end example
+
+
+Edit more files. Commit them again. Edit yet more files, commit
+them again. Go eat dinner. Switch to @code{master} so you can
+play with the latest changes from other developers. Switch back
+to your branch and edit some more. Commit those changes.
+
+At this stage, don't worry about how many commits you have.
+
+
+@subsubheading Save commits to external files
+
+Branches are nerve-wracking until you get used to them. You can
+save your hard work as individual @file{.patch} files. Be sure to
+commit your changes first.
+
+@example
+git commit -a
+git format-patch master
+@end example
+
+I personally have between 4 and 20 of those files saved in a
+special folder at any point in time. Git experts might laugh as
+that behavior, but I feel a @emph{lot} better knowing that I've
+got those backups.
+
+
+@subsubheading Prepare your branch for review
+
+After committing, you can update your branch with the latest master:
+
+@example
+git commit -a
+git checkout master
+git pull -r origin master
+git checkout dev/cg
+git rebase master
+@end example
+
+
+Due to the speed of lilypond development, sometimes
+@code{master} has changed so much that your branch can no
+longer be applied to it. In that happens, you will have a merge
+conflict. Stop for a moment to either cry or have a stiff drink,
+then proceed to @ref{Merge conflicts}.
+
+
+@subsubheading Upload your branch
+
+Finally, you're finished your changes. Time to upload for review.
+Make sure that you're on your branch, then upload:
+
+@example
+git checkout dev/cg
+git-cl upload master
+@end example
+
+
+@subsubheading Wait for reviews
+
+While you're waiting for a countdown and reviews, go back to
+master, make a @code{dev/doc-beams} branch, and start adding doc
+suggestions from issue 12345 from the tracker. Or make a
+@code{dev/page-breaks} and fix bug in page breaking. Or whatever.
+Don't worry, your @code{dev/cg} is safe.
+
+
+@subsubheading Combining commits (optional unless you have broken commits)
+
+Does the history of your branch look good?
+
+@example
+gitk
+@end example
+
+If you have a lot of commits on your branch, you might want to
+combine some of them. Alternately, you may like your commits, but
+want to edit the commit messages.
+
+@example
+git rebase -i master
+@end example
+
+Follow instructions on the screen.
+
+@warning{This step gives you the power to completely lose your
+work. Make a backup of your commits by saving them to
+@file{.patch} files before playing with this. If you do lose
+your work, don't despair. You can get it back by using @code{git reflog}.
+The use of @code{git reflog} is not covered here.}
+
+@warning{If any of the commits on your branch represent partial work that will
+not pass @var{make && make doc}, you @strong{must} squash these
+commits into a working commit. Otherwise, your push will break staging
+and will not be able to be merged to master. In general, you will
+be safer to have one commit per push.}
+
+
+@subsubheading Push to staging
+
+When you've got the coveted @code{Patch-push} status, time to
+prepare your upload:
+
+@example
+git fetch
+git rebase origin/staging dev/cg~0
+gitk HEAD
+@end example
+
+@warning{Do not skip the @command{gitk} step; a quick 5-second
+check of the visual history can save a great deal of frustration
+later on. You should see a set of your commits that are ahead of
+@code{origin/staging}, with no label for the top commit -- only a
+SHA1 id.}
+
+@warning{If @code{origin/staging} and @code{origin/master} are the
+same commit, your branch (@code{dev/cg} in the example) will also
+be at the top of the @code{gitk} tree. This is normal.}
+
+If everything looks good, push it:
+
+@example
+git push origin HEAD:staging
+@end example
+
+Then change back to your working branch:
+
+@example
+git checkout dev/cg
+@end example
+
+@warning{It is a best practice to avoid rebasing any of your branches
+to @code{origin/staging}. If @code{origin/staging} is broken, it
+will be deleted and rebuilt. If you have rebased one of your branches
+to @code{origin/staging}, the broken commits can end up in your branch.
+The commands given above do the rebase on a temporary branch, and avoid
+changing your working branch.}
+
+
+@subsubheading Delete your branch (safe)
+
+After a few hours, if there's nothing wrong with your branch, it
+should be automatically moved to @code{origin/master}. Update,
+then try removing your branch:
+
+@example
+git checkout master
+git pull -r origin master
+git branch -d dev/cg
+@end example
+
+The last command will fail if the contents of @code{dev/cg} are
+not present in @code{origin/master}.
+
+
+@subsubheading Delete your branch (UNSAFE)
+
+@c don't give explicit commands here -- this is too dangerous to copy and paste
+Sometimes everything goes wrong. If you want to remove a branch even though
+it will cause your work to be lost (that is, if the contents of @code{dev/cg}
+are @strong{not} present in master), follow the instructions in @qq{Delete
+your branch (safe)}, but replace the @code{-d} on the final line with
+a @code{-D}.
+
+
+@node Other repositories
+@subsection Other repositories
+
+We have a few other code repositories.
+
+@menu
+* lilypond-extra::
+* Grand Unified Builder (GUB)::
+* LilyPad::
+* yet more repositories::
+@end menu
+
+@node lilypond-extra
+@unnumberedsubsubsec lilypond-extra
+
+There is a separate repository for general administrative scripts,
+as well as pictures and media files for the website. People
+interested in working on the website should download this
+repository, and set their @code{$LILYPOND_WEB_MEDIA_GIT}
+environment variable to point to that repository.
+
+@example
+@uref{https://github.com/gperciva/lilypond-extra}
+@end example
+
+To configure an environment variable in bash (the default for most
+GNU/Linux distributions),
+
+@example
+export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/
+@end example
+
+Be aware that @code{lilypond-extra} is the definitive source for some binary
+files - in particular PDF versions of papers concerning LilyPond. To add
+further PDFs of this sort, all that is necessary is to add the PDF to
+@code{lilypond-extra} and then add a reference to it in the documentation. The
+file will then be copied to the website when @code{make website} is run.
+
+However, pictures that are also used in the documentation build are mastered in
+the main git repository. If any of these is changed, it should be updated in
+git, and then the updates copied to @code{lilypond-extra}.
+
+@node Grand Unified Builder (GUB)
+@unnumberedsubsubsec Grand Unified Builder (GUB)
+
+Another item of interest might be the Grand Unified Builder, our
+cross-platform building tool. Since it is used by other projects as
+well, it is not stored in our gub repository. For more info, see
+@uref{http://lilypond.org/gub}.
+
+There are two locations for this repository: the version being used to
+build lilypond, which is at
+
+@example
+@uref{http://github.com/gperciva/gub}
+@end example
+
+and the original version by Jan Nieuwenhuizen, kept at
+
+@example
+@uref{http://github.com/janneke/gub}
+@end example
+
+
+@node LilyPad
+@unnumberedsubsubsec LilyPad
+
+Our binary releases on MacOS X and Windows contain a lightweight
+text editor.
+
+To make any modifications the Windows editor, you will need to do the
+following:
+
+@enumerate
+@item
+Clone the git repository from @code{https://github.com/gperciva/lilypad}
+
+@item
+Make changes to the source, and check it compiles. In a Windows environment
+@code{MinGW} provides both a @code{Git} installation and a @code{gcc}
+compiler. This can be obtained from @code{http://www.mingw.org/}
+
+@item
+Update the version which is contained in the @file{rsrc.rc}. Check
+this compiles, too.
+
+@item
+Commit the changes with an informative commit message.
+
+@item
+Push the changes to github. You will need to use syntax similiar to this:
+
+@example
+git push https://UserName@@github.com/gperciva/lilypad.git
+@end example
+
+You will need to have push access to the git repository for this to be
+successful.
+
+@item
+Make a tarball of the source code to be used by GUB by pulling the updated
+repository from GitHub. Ensure that the tarball has the correct Version
+number.
+
+@item
+Copy the tarball to @code{http://lilypond.org/downloads/gub-sources/lilypad/}.
+You will need to have SSH access to @code{lilypond.org}. If you do not, contact
+the Release Manager via the lilypond-devel mailing list.
+
+@item
+Update GUB to make it use the new tarball by editing
+@file{gub/specs/lilypad.py} and changing the @code{source =} line to point to
+the new source.
+
+@item
+Push this updated @file{lilypad.py} version to the GUB repository on GitHub.
+
+@item
+Test the changes with a new GUB compile.
+
+@end enumerate
+
+@node yet more repositories
+@unnumberedsubsubsec yet more repositories
+
+There are a few other repositories floating around, which will
+hopefully be documented in the near future.
+
+
@node Downloading remote branches
@subsection Downloading remote branches
+@warning{contains obsolete + misleading info}
@menu
* Organization of remote branches::
and the website. Generally, the @code{master} branch is expected
to compile successfully.
-The @code{lilypond/translation} branch is a side branch that
+The @code{translation} branch is a side branch that
allows translators to work without needing to worry about
compilation problems. Periodically, the Translation Meister
(after verifying that it doesn't break compilation), will
-@emph{merge} this branch back into @code{master} to incorporate
+@emph{merge} this branch into @code{staging} to incorporate
recent translations. Similarly, the @code{master} branch is
-usually merged into the @code{lilypond/translation} branch after
+usually merged into the @code{translation} branch after
significant changes to the English documentation. See
@ref{Translating the documentation} for details.
@node Downloading individual branches
@unnumberedsubsubsec Downloading individual branches
+@warning{obsolete, should be deleted!}
+
Once you have initialized an empty Git repository on your system
(see @ref{Initializing a repository}), you can download a remote
origin git://git.sv.gnu.org/lilypond.git/
@end example
-To download the @code{lilypond/translation} branch, enter:
+To download the @code{translation} branch, enter:
@example
-git remote add -ft lilypond/translation -m \
- lilypond/translation origin git://git.sv.gnu.org/lilypond.git/
+git remote add -ft translation -m \
+ translation origin git://git.sv.gnu.org/lilypond.git/
@end example
The @command{git@tie{}remote@tie{}add} process could take up to
@noindent
where @code{@var{branch}} is the name of your tracking branch,
-either @code{master} or @code{lilypond/translation}.
+either @code{master} or @code{translation}.
Git will issue some warnings; this is normal:
@end example
By now the source files should be accessible---you should be able
-to edit any files in the @file{lilypond-git/} directory using a
+to edit any files in the @file{$LILYPOND_GIT} directory using a
text editor of your choice. But don't start just yet! Before
editing any source files, learn how to keep your changes organized
and prevent problems later---read @ref{Basic Git procedures}.
@item @code{stable/XYZ}:
The branches are kept for archival reasons.
-@end itemize
+@item @code{archive/XYZ}:
+The branches are kept for archival reasons.
-Another item of interest might be the Grand Unified Builder, our
-cross-platform building tool. Since it is used by projects as
-well, it is not stored in our gub repository. For more info, see
-@uref{http://lilypond.org/gub}. The git location is
-@uref{http://github.com/janneke/gub}.
+@end itemize
@node Basic Git procedures
* The Git contributor's cycle::
* Pulling and rebasing::
* Using local branches::
-* Commits and patches::
+* Commits::
+* Patches::
+* Uploading a patch for review::
+* The patch review cycle::
@end menu
@noindent
it means that you have modified some files in you working tree
-without committing changes (see @ref{Commits and patches}); you
+without committing changes (see @ref{Commits}); you
can use the @command{git@tie{}stash} command to work around this:
@example
you have not yet pushed to @code{git.sv.gnu.org}, please do not
rebase. If you want to avoid wondering whether you should rebase
each time you pull, please always use committishes from master
-and/or lilypond/translation branch on @code{git.sv.gnu.org}, which
+and/or translation branch on @code{git.sv.gnu.org}, which
in particular implies that you must push your changes to
documentation except committishes updates (possibly after having
rebased), then update the committishes and push them.}
If any conflict happens, see @ref{Resolving conflicts}.
-There are common usage cases for merging: as a translator, you
-will often want to merge @code{master} into
-@code{lilypond/translation}; on the other hand, the Translations
-meister wants to merge @code{lilypond/translation} into
-@code{master} whenever he has checked that
-@code{lilypond/translation} builds successfully.
-
+There are common usage cases for merging: as a translator, you will
+often want the Translations meister to merge @code{master} into
+@code{translation}; on the other hand, the Translations meister wants
+to merge @code{translation} into @code{staging} whenever he has
+checked that @code{translation} builds successfully.
-@node Commits and patches
-@subsection Commits and patches
+@node Commits
+@subsection Commits
@menu
* Understanding commits::
-* Making commits::
+* How to make a commit::
* Commit messages::
-* Making patches::
-* Uploading a patch for review::
@end menu
@uref{http://git.sv.gnu.org/cgit/lilypond.git/log/}.
-@node Making commits
-@unnumberedsubsubsec Making commits
+@node How to make a commit
+@unnumberedsubsubsec How to make a commit
Once you have modified some source files in your working
@end example
Commit messages often start with a short prefix describing the
-general location of the changes. If a commit affects the
+general location of the changes.
+
+@itemize
+@item
+Doc: and Doc-@var{**}: If a commit affects the
documentation in English (or in several languages simultaneously)
-the commit message should be prefixed with @qq{Doc:@tie{}}. If
-the commit affects only one of the translations, the commit
+the commit message should be prefixed with @qq{Doc:@tie{}}. If the
+commit affects only one of the translations, the commit
message should be prefixed with @qq{Doc-@var{**}:@tie{}}, where
-@var{**} is the two-letter language code. Commits that affect the
+@var{**} is the two-letter language code.
+
+@item
+Web: and Web-@var{**}: Commits that affect the
website should use @qq{Web:@tie{}} for English, and
-@qq{Web-@var{**}:@tie{}} for the other languages. Also, changes
-to a single file are often prefixed with the name of the file
-involved. Visit the links listed in @ref{Understanding commits}
-for examples.
+@qq{Web-@var{**}:@tie{}} for other languages.
+
+@item
+CSS: Commits that change CSS files should use @qq{Web:@tie{}CSS:@tie{}}
+or @qq{Doc:@tie{}CSS:@tie{}} depending on whether they affect the
+website or the documentation/manuals.
+
+@item
+Changes to a single file are often prefixed with the name of the file
+involved.
+@end itemize
+
+Visit the links listed in @ref{Understanding commits} for examples.
-@node Making patches
-@unnumberedsubsubsec Making patches
+
+@node Patches
+@subsection Patches
+
+@menu
+* How to make a patch::
+* Emailing patches::
+@end menu
+
+@node How to make a patch
+@unnumberedsubsubsec How to make a patch
If you want to share your changes with other contributors and
developers, you need to generate @emph{patches} from your commits.
repository or discuss them with you.
-@node Uploading a patch for review
-@unnumberedsubsubsec Uploading a patch for review
-
-Any non-trivial change should be uploaded to our @qq{Rietveld}
-code review website:
-
-@example
-@uref{http://codereview.appspot.com/}
-@end example
-
-@subsubheading @command{git-cl} install
-
-LilyDev users should skip over these @q{install} instructions.
-
-@enumerate
-
-@item
-Install @command{git-cl} by entering:
+@node Emailing patches
+@unnumberedsubsubsec Emailing patches
-@example
-git clone https://github.com/gperciva/git-cl.git
-@end example
+The default @code{x-diff} MIME type associated with patch files
+(i.e., files whose name ends in @code{.patch}) means that the
+encoding of line endings may be changed from UNIX to DOS format
+when they are sent as attachments. Attempting to apply such an
+inadvertently altered patch will cause git to fail with a message
+about @q{whitespace errors}.
-If that command fails for some reason, try this instead:
+The solution to such problems is surprisingly simple---just change
+the default file extension of patches generated by git to end in
+@code{.txt}, for example:
@example
-git clone git://github.com/gperciva/git-cl.git
+git config format.suffix '.patch.txt'
@end example
-@item
-Add the @file{git-cl/} directory to your PATH, or create a
-symbolic link to the @command{git-cl} and @command{upload.py}
-scripts in one of your PATH directories (such as
-@file{$HOME/bin}).
-
-@end enumerate
+This should cause email programs to apply the correct base64
+encoding to attached patches.
-@subsubheading @command{git-cl} configuration
+If you receive a patch with DOS instead of UNIX line-endings, it
+can be converted back using the @code{dos2unix} utility.
-LilyDev users should perform these @q{configuration} instructions.
+Lots of useful information on email complications with patches is
+provided on the Wine wiki at
+@uref{http://wiki.winehq.org/GitWine}.
-@enumerate
-@item
-You must have a gmail account; please create one if you do not
-have one already.
-@item
-Move into the top source directory and then configure @command{git
-cl} with the following commands. If you do not understand any
-question, just answer with a newline (CR).
-
-@example
-cd $HOME/lilypond-git/
-git cl config
-@end example
+@node Uploading a patch for review
+@subsection Uploading a patch for review
-The @qq{CC list} question should be answered with:
+Any non-trivial change should be uploaded to our @qq{Rietveld}
+code review website:
@example
-lilypond-devel@@gnu.org
+@uref{http://codereview.appspot.com/}
@end example
-@end enumerate
-
-@subsubheading Uploading patch set
+You can upload a patch for review by using our custom @code{git-cl}
+@q{helper-script}. This section assumes you have already installed,
+updated, and configured @code{git-cl}. See @ref{git-cl}.
@warning{Unless you are familiar with branches, only work on one
set of changes at once.}
@itemize
@item
-@strong{Master branch}: (easy option, and used in @command{lily-git.tcl})
+@strong{Master branch}: (easy option)
If you added your patch to @code{master}, then:
@example
git pull -r
-git cl upload origin/master
+git-cl upload origin/master
@end example
+@c Mention staging here?
If you have git push ability, make sure that you @emph{remove}
your patch (with @command{git rebase} or @command{git reset})
before pushing other stuff.
+@c don't make this one an @example; we don't want to make it easy
+@c for people to use this accidently
+Notifications of patches are automatically added to our issue
+tracker to reduce the chance of patches getting lost. To suppress
+this (not recommended), add the @code{-n / --no-code-issue}
+option.
+
@item
@strong{Separate branch}: (complicated option)
-Ensure your changes are committed in a separate branch, which
-should differ from the reference branch to be used by just the
-changes to be uploaded. If the reference branch is to be
-origin/master, ensure this is up-to-date. If necessary, use git
-rebase to rebase the branch containing the changes to the head of
-origin/master. Finally, check out branch with the changes and
-enter the command:
+Ensure your changes are committed in a separate branch, which should
+differ from the reference branch to be used (usually
+@code{origin/master}) by just the changes to be uploaded. Checkout the
+branch with the changes:
+
+@example
+git checkout some-branch-with-changes
+@end example
+
+If the reference branch is to be @code{origin/master}, ensure that the
+branch containing the changes is up-to-date with it. Use
+@command{git rebase} or @command{git pull -r} to rebase the branch to
+the head of @code{origin/master}. For example:
@example
-git cl upload <reference SHA1 ID>
+git pull -r origin master
+@end example
+
+Finally, start the upload by entering:
+
+@example
+git-cl upload <reference SHA1 ID>
@end example
@noindent
where <reference SHA1 ID> is the SHA1 ID of the commit to be used
as a reference source for the patch. Generally, this will be the
-SHA1 ID of origin/master, and in that case the command:
+SHA1 ID of origin/master, and in that case you can just use the command:
@example
-git cl upload origin/master
+git-cl upload origin/master
@end example
-@noindent
-can be used.
-
@end itemize
+First you will see a terminal editor where you can edit the
+message that will accompany your patch. @command{git-cl} will
+respect the @env{EDITOR} environment variable if defined,
+otherwise it will use @command{vi} as the default editor.
+
After prompting for your Google email address and password, the
patch set will be posted to Rietveld, and you will be given a URL
for your patch.
@warning{Some installations of git-cl fail when uploading a patch
-set that includes a .scm file. When this happens, it can
-generally be fixed by editing the file @file{/etc/mime.types}.
-Add a line to this file containing @code{text/x-script.scheme scm}.}
+with certain filename extensions. If this happens, it can
+generally be fixed by editing the list of exceptions at top of
+@file{git-cl.py}.}
@subsubheading Announcing your patch set
the following command can be used:
@example
-git cl issue issue-number
+git-cl issue issue-number
@end example
@noindent
where @code{issue-number} is the number of the existing Rietveld
issue.
-@subsubheading Resetting git cl
+@subsubheading Resetting git-cl
-If @command{git cl} becomes confused, you can @qq{reset} it by
+If @command{git-cl} becomes confused, you can @qq{reset} it by
running:
@example
-git cl issue 0
+git-cl issue 0
+@end example
+
+
+@node The patch review cycle
+@subsection The patch review cycle
+
+Your patch will be available for reviews for the next few hours or
+days. Three times a week, patches with no known problems are
+gathered into a @qq{patch countdown} and their status changed to
+@code{patch-countdown}. The countdown is a 48-hour waiting period
+in which any final reviews or complaints should be made.
+
+During the countdown, your patch may be set to
+@code{patch-needs_work}, indicating that you should fix something
+(or at least discuss why the patch needs no modification). If no
+problems are found, the patch will be set to @code{patch-push}.
+
+Once a patch has @code{patch-push}, it should be sent to your
+mentor for uploading. If you have git push ability, look at
+@ref{Pushing to staging}.
+
+@itemize
+
+@item
+Patches get added to the tracker and to Rietveld by the @qq{git-cl} tool, with
+a status of @qq{patch-new}.
+
+@item
+The automated tester, Patchy, verifies that the patch can be applied
+to current master. By default, it checks that the patch allows @code{make}
+and @code{make test} to complete successfully. It can also be configured to
+check that @code{make doc} is successful. If it passes, Patchy changes the
+status to @qq{patch-review} and emails the developer list. If the patch
+fails, Patchy sets it to @qq{patch-needs_work} and notifies the developer list.
+
+@item
+The Patch Meister reviews the tracker periodically, to list patches
+which have been on review for at least 24 hours. The list is found at
+
+@smallexample
+@uref{http://code.google.com/p/lilypond/issues/list?can=2&q=label:patch%20patch=review&sort=modified+patch&colspec=ID%20Type%20Status%20Priority%20Owner%20Patch%20Summary%20Modified}
+@end smallexample
+
+@item
+For each patch, the Handler reviews any discussion on the tracker
+and on Rietveld, to determine whether the patch can go forward. If
+there is any indication that a developer thinks the patch is not
+ready, the Handler marks it @qq{patch-needs_work} and makes a comment
+regarding the reason, referring to the Rietveld item if needed.
+
+@item
+Patches with explicit approval, or at least no negative comment, can
+be updated to @qq{patch-countdown}. When saving the tracker item,
+clear the @qq{send email} box to prevent sending notification for
+each patch.
+
+@item
+The Patch Meister sends an email to the developer list, with a fixed
+subject line, to enable filtering by email clients:
+
+@example
+PATCH: Countdown to 20130113
+@end example
+
+The text of the email sets the deadline for this countdown batch. At
+present, batches are done on Tuesday, Thursday and Sunday evenings.
+
+To create the countdown announcement, use the
+@code{make-countdown-announcement.sh} script, which takes the
+deadline date, and optionally your name. Follow the instructions
+provided:
+
+@example
+cd $LILYPOND_GIT
+scripts/auxiliar/make-countdown-announcement.sh "Jan 1, 2001" James
+@end example
+
+The script produces an announcement that is easily readable in all
+email clients. Also, whenever a new contributor submits a patch,
+you will be prompted to add the new username and author name to
+the script itself, and then commit those changes to the main git
+repository.
+
+
+@item
+On the scheduled countdown day, the Patch Meister reviews the
+previous list of patches on countdown, with the same procedure and
+criteria as before. Patches with no controversy can be set to
+@qq{patch-push} with a courtesy message added to the comment block.
+
+@item
+Roughly at six month intervals, the Patch Meister can list the
+patches which have been set to @qq{patch-needs-work} and send the
+results to the developer list for review. In most cases, these
+patches should be marked @qq{patch-abandoned} but this should come
+from the developer if possible.
+
+@item
+As in most organisations of unpaid volunteers, fixed procedures are
+useful in as much as they get the job done. In our community, there
+is room for senior developers to bypass normal patch handling flows,
+particularly now that the testing of patches is largely automated.
+Similarly, the minimum age of 24 hours can reasonably be waived if
+the patch is minor and from an experienced developer.
+
+
+@end itemize
+
+@ignore
+There is a single Patch Meister, and a number of Patch Helpers
+(rename this?). The list of known patches awaiting review is:
+
+@example
+@uref{http://code.google.com/p/lilypond/issues/list?can=2&q=label:patch&sort=patch}
+@end example
+
+
+@subheading Helpers: adding patches
+
+The primary duty is to add patches to the google tracker; we have
+a bad track record of losing patches in email. Patches generally
+come to the @code{lilypond-devel} mailing list, but are sometimes
+sent to @code{bug-lilypond}, @code{lilypond-users}, or
+@code{frogs} mailing list instead.
+
+@itemize
+@item
+Unless a patch is clearly in response to an existing issue, add a
+new issue with the @code{Patch-new} label and a link to the patch
+(either on the mailing list archives or the codereview url).
+
+Issue numbers are cheap; losing developers because they got fed up
+with us losing their hard work is expensive.
+
+
+@c if we enter patches immediately, I don't think this is relevant.
+
+@item
+Before adding a patch-reminder issue, do a quick check to see if
+it was pushed without sending any email. This can be checked for
+searching for relevant terms (from the patch subject or commit
+message) on the webgit page:
+
+@example
+@uref{http://git.savannah.gnu.org/gitweb/?p=lilypond.git}
@end example
+@item
+If the patch is clearly in response to an existing issue, then
+update that issue with the @code{Patch-new} label and a link to
+the patch (either on the mailing list archives or the codereview
+url).
+
+@item
+After adding the issue, please send a response email to the same
+group(s) that the initial patch was sent to.
+
+If the initial email was sent to multiple mailing lists (such as
+both @code{bugs} and @code{devel}), then reply to all those
+mailing lists as well. The email should contain a link to the
+issue you just added.
+
+@end itemize
+
+@subheading Helpers: @code{Patch-review} label
+
+The secondary duty is to do make sure that every issue in the
+tracker with a @code{Patch-review} label has passed these
+@qq{obvious} tests:
+
+@itemize
+@item
+Applies automatically to git master.
+
+It's ok to have offsets, but not conflicts.
+
+@item
+Regtest comparison looks ok; no unexpected changes.
+
+@item
+Descriptive subject line.
+
+Avoid subjects like @qq{fixes 123}; instead write @qq{Doc: discuss
+stacking-dir for BassFigureAlignment (fix 123)}.
+
+@item
+Compiles docs from scratch. Only check this if you have reason to
+suspect it might not work.
+
+@item
+(maybe)
+
+Check code indentation and style. This should be easier post-GOP
+when we have a better-defined code style.
+
+@end itemize
+
+
+@subheading Patch Meister
+
+The Patch Meister will:
+
+@itemize
+
+@item
+send @qq{countdown} emails to
+@code{lilypond-devel} when patches appear to be ready.
+
+@item
+send general requests to review patches, or even nasty requests to
+review patches.
+
+@item
+downgrade patches from @code{Patch-review} to
+@code{Patch-needs_work} as appropriate.
+
+@item
+downgrade patches from @code{Patch-needs_work} to
+@code{Patch-abandoned} if no actions have been taken in four
+weeks.
+
+@end itemize
+
+@end ignore
+
+
@node Advanced Git procedures
@section Advanced Git procedures
It is possible to work with several branches on the same local Git
repository; this is especially useful for translators who may have
-to deal with both @code{lilypond/translation} and a stable branch,
+to deal with both @code{translation} and a stable branch,
e.g. @code{stable/2.12}.
Some Git commands are introduced first, then a workflow with
@menu
+* Merge conflicts::
* Advanced Git concepts::
* Resolving conflicts::
* Reverting all local changes::
* Working with remote branches::
* Git log::
* Applying remote patches::
-* Sending and receiving patches via email::
* Cleaning up multiple patches::
* Commit access::
+* Pushing to staging::
@end menu
+@node Merge conflicts
+@subsection Merge conflicts
+
+To be filled in later, and/or moved to a different section. I
+just wanted to make sure that I had a stub ready somewhere.
+
+
@node Advanced Git concepts
@subsection Advanced Git concepts
@noindent
where @code{@var{branch}} is typically @code{master} or
-@code{lilypond/translation}; if you do not know or remember, see
+@code{translation}; if you do not know or remember, see
@ref{Downloading remote branches} to remember which commands you
issued or which source code you wanted to get.
@subsubheading Local clones, or having several working trees
If you play with several Git branches, e.g. @code{master},
-@code{lilypond/translation}, @code{stable/2.12}), you may want to
+@code{translation}, @code{stable/2.12}), you may want to
have one source and build tree for each branch; this is possible
with subdirectories of your local Git repository, used as local
cloned subrepositories. To create a local clone for the branch
Patches created without @code{git@tie{}format-patch} can be
applied in two steps. The first step is to apply the patch to the
-working tree:
+working tree and the index:
@example
-git apply @var{patch}
+git apply --index @var{patch}
@end example
@noindent
author of the patch. This can be done with the following command:
@example
-git commit -a --author="@var{John Smith} <@var{john@@example.com}>"
+git commit --author="@var{John Smith} <@var{john@@example.com}>"
@end example
+Please note that using the @code{--index} option for patching is quite
+important here and @emph{cannot} reliably be replaced by using the
+@code{-a} option when committing: that would only commit files from the
+working tree that are already registered with git, so every file that
+the patch actually @emph{adds}, like a regtest for a fixed bug, would
+get lost. For the same reason, you should not use the git-independent
+@samp{patch} program for applying patches.
-@node Sending and receiving patches via email
-@subsection Sending and receiving patches via email
-
-
-The default @code{x-diff} MIME type associated with patch files
-(i.e., files whose name ends in @code{.patch}) means that the
-encoding of line endings may be changed from UNIX to DOS format
-when they are sent as attachments. Attempting to apply such an
-inadvertently altered patch will cause git to fail with a message
-about @q{whitespace errors}.
-
-The solution to such problems is surprisingly simple---just change
-the default file extension of patches generated by git to end in
-@code{.txt}, for example:
-
-@example
-git config format.suffix '.patch.txt'
-@end example
-
-This should cause email programs to apply the correct base64
-encoding to attached patches.
-
-If you receive a patch with DOS instead of UNIX line-endings, it
-can be converted back using the @code{dos2unix} utility.
-
-Lots of useful information on email complications with patches is
-provided on the Wine wiki at
-@uref{http://wiki.winehq.org/GitWine}.
@node Cleaning up multiple patches
@node Commit access
@subsection Commit access
-
Most contributors are not able to commit patches directly to the
main repository---only members of the LilyPond development team
have @emph{commit access}. If you are a contributor and are
will include a @q{From} line that shows @q{ssh} as the protocol:
@example
-From ssh://@var{user}@@git.sv.gnu.org/srv/git/lilypond
+From ssh://git.sv.gnu.org/srv/git/lilypond
@end example
If the protocol shown is not @q{ssh}, check that you configured
@item
Test your commit access with a dry run:
+@warning{Do not push directly to master; instead, push to staging.
+See @ref{Pushing to staging}.}
+
@example
git push --dry-run --verbose
@end example
over ATM. If this problem is encountered a possible work-around is
to set the MTU in the local router to 1500.
+
+@node Pushing to staging
+@subsection Pushing to staging
+
+Do not push directly to the git @code{master} branch. Instead,
+push to @code{staging}.
+
+You will not see your patch on @code{origin/master} until some
+automatic tests have been run. These tests are run every couple
+of hours; please wait at least 12 hours before wondering if your
+patch has been lost. Note that you can check the commits on
+@code{origin/staging} by looking at the git web interface on
+savannah.
+
+It may happen occasionally that the staging branch breaks automated
+testing. In this case the automatic move of staging material to
+master gets halted in order to avoid broken material entering master.
+This is a safety net. Please do not try breaking out from it by
+adding fixes on top of staging: in that case the whole sequence will
+end up in master after all, defeating the purpose of the system. The
+proper fix usually involves rewriting the staging branch and is best
+left to core developers after discussion on the developer list.
+
+Before pushing to staging it is a good practice to check whether
+staging is ahead of master, and if so, wait until master has caught up
+with staging before pushing. This simplifies things if changes to
+staging have to be backed out for some reason. To check whether
+master has caught up with staging you can look at the git web interface
+on savannah, or do:
+
+@example
+git fetch
+gitk
+@end example
+
+and check that @code{origin/master} is at the same commit as
+@code{origin/staging}. Another option is to see if any commits are
+listed when you do:
+
+@example
+git fetch
+git log origin/master..origin/staging
+@end example
+
+@subsubheading If your work is in a patch file
+
+Assuming that your patch is in a file called
+@file{0001-my-patch.patch} (see @ref{Patches}), and you are currently
+on git master, do:
+
+@example
+git checkout staging
+git pull -r
+git am 0001-my-patch.patch
+gitk
+git push origin staging
+git checkout master
+@end example
+
+@warning{Do not skip the @command{gitk} step; a quick 5-second
+check of the visual history can save a great deal of frustration
+later on. You should only see that @command{staging} is only 1
+commit ahead of @code{origin/staging}.}
+
+@subsubheading If your work is in a branch
+
+If you are working on branches and your work is in
+@code{my_branch_name}, then do:
+
+@example
+git checkout my_branch_name
+git pull -r origin staging
+@end example
+
+This will rebase your branch on @code{origin/staging}. At this point
+git will let you know if there are any conflicts. If so, resolve them
+before continuing:
+
+@example
+gitk
+git push origin HEAD:staging
+@end example
+
+@warning{Do not skip the @command{gitk} step; a quick 5-second
+check of the visual history can save a great deal of frustration
+later on. You should see that @code{my_branch_name} is only ahead of
+@code{origin/staging} by the commits from your branch.}
+
+
+
@node Git on Windows
@section Git on Windows
+@warning{We heavily recommend that development be done with our
+virtual machine @ref{LilyDev}.}
+
@c Some of this may duplicate stuff in other sections
@c But it is probably best for windows users to have it all together
@c If necessary, clear this up later -td
@subsection Installing git
Obtain Git from
-@uref{http://code.google.com/p/msysgit/downloads/list} (note, not
-msysGit, which is for Git developers and not PortableGit, which is
-not a full git installation) and install it.
+@uref{https://git-for-windows.github.io/}.
Note that most users will not need to install SSH. That is not
required until you have been granted direct push permissions to
@item
Book about git: @uref{http://progit.org/,Pro Git}
+
+@item
+Github help: @uref{http://help.github.com/}
+(very highly recommended by Graham)
+
@end itemize
projects / lilypond.git / blobdiff