[lilypond.git] / Documentation / contributor / source-code.itexi

@c -*- coding: utf-8; mode: texinfo; -*-


@node Working with source code
@chapter Working with source code

@warning{New contributors should read @ref{Quick start}, and in
particular @ref{lily-git}, instead of this chapter.}

Advanced contributors will find this material quite useful,
particularly if they are working on major new features.

@menu
* Manually installing lily-git.tcl::
* Starting with Git::
* Basic Git procedures::
* Advanced Git procedures::
* Git on Windows::
* Repository directory structure::
* Other Git documentation::
@end menu


@node Manually installing lily-git.tcl
@section Manually installing lily-git.tcl

We have created an easy-to-use GUI to simplify git for new
contributors.  If you are comfortable with the command-line, then
skip ahead to @ref{Starting with Git}.

@warning{These instructions are only for people who are @emph{not}
using @ref{LilyDev}.}

@c there's some duplication in this section with stuff covered in
@c Quick Start, but moving it into a macro inside included/ would
@c be getting a bit icky.  -gp

@enumerate
@item
If you haven't already, download and install Git.

@itemize

@item Windows users: download the @code{.exe} file labeled
@qq{Full installer for official Git} from:

@example
@uref{https://git-for-windows.github.io/}
@end example

@item Other operating systems: either install @command{git} with
your package manager, or download it from the @qq{Binaries}
section of:

@example
@uref{http://git-scm.com/download}
@end example

@end itemize


@item
Download the @command{lily-git.tcl} script from:

@c don't change the cgit link below to gitweb; gitweb uses
@c long filenames like "scripts_auxiliar_lily-git.tcl"

@smallexample
@uref{http://git.sv.gnu.org/cgit/lilypond.git/plain/scripts/auxiliar/lily-git.tcl}
@end smallexample

@item
To run the program from the command line, navigate to the
directory containing @command{lily-git.tcl} and enter:

@example
wish lily-git.tcl
@end example

@item
Click on the @qq{Get source} button.

This will create a directory called @file{lilypond-git/} within
your home directory, and will download the source code into that
directory (around 150@tie{}Mb).  When the process is finished, the
@qq{Command output} window will display @qq{Done}, and the button
label will change to say @qq{Update source}.

@item
Navigate to the @file{lilypond-git/} directory to view the source
files.

@end enumerate

@warning{Throughout the rest of this manual, most command-line
input should be entered from @file{$LILYPOND_GIT}.  This is
referred to as the @emph{top source directory}.}

Further instructions are in @ref{How to use lily-git}.


@node Starting with Git
@section Starting with Git

Using the Git program directly (as opposed to using the
@command{lily-git.tcl} GUI) allows you to have much greater control
over the contributing process.  You should consider using Git if
you want to work on complex projects, or if you want to work on
multiple projects concurrently.


@menu
* Setting up::
* Git for the impatient::
* Other repositories::
* Downloading remote branches::
@end menu


@node Setting up
@subsection Setting up

@warning{These instructions assume that you are using the
command-line version of Git 1.5 or higher.  Windows users should
skip to @ref{Git on Windows}.}

@menu
* Installing Git::
* Initializing a repository::
* Configuring Git::
@end menu


@node Installing Git
@unnumberedsubsubsec Installing Git

If you are using a Unix-based machine, the easiest way to download
and install Git is through a package manager such as @command{rpm}
or @command{apt-get} -- the installation is generally automatic.
The only required package is (usually) called @command{git-core},
although some of the auxiliary @command{git@var{*}} packages are
also useful (such as @command{gitk}).

Alternatively, you can visit the Git website
(@uref{http://git-scm.com/}) for downloadable binaries and
tarballs.


@node Initializing a repository
@unnumberedsubsubsec Initializing a repository

Once Git is installed, get a copy of the source code:

@example
git clone git://git.sv.gnu.org/lilypond.git ~/lilypond-git
@end example

The above command will put the it in @file{~/lilypond-git}, where
@code{~} represents your home directory.

@subsubheading Technical details

This creates (within the @file{$LILYPOND_GIT} directory) a
subdirectory called @file{.git/}, which Git uses to keep track of
changes to the repository, among other things.  Normally you don't
need to access it, but it's good to know it's there.


@node Configuring Git
@unnumberedsubsubsec Configuring Git

@warning{Throughout the rest of this manual, all command-line
input should be entered from the top directory of the Git
repository being discussed (eg. @file{$LILYPOND_GIT}).  This is
referred to as the @emph{top source directory}.}

Before working with the copy of the main LilyPond repository, you
should configure some basic settings with the
@command{git@tie{}config} command.  Git allows you to set both
global and repository-specific options.

To configure settings that affect all repositories, use the
@option{--global} command line option.  For example, the first
two options that you should always set are your @var{name} and
@var{email}, since Git needs these to keep track of commit
authors:

@example
git config --global user.name "@var{John Smith}"
git config --global user.email @var{john@@example.com}
@end example

To configure Git to use colored output where possible, use:

@example
git config --global color.ui auto
@end example

The text editor that opens when using @command{git@tie{}commit}
can also be changed.  If none of your editor-related environment
variables are set ($GIT_EDITOR, $VISUAL, or $EDITOR), the default
editor is usually @command{vi} or @command{vim}.  If you're not
familiar with either of these, you should probably change the
default to an editor that you know how to use.  For example, to
change the default editor to @command{nano}, enter:

@example
git config --global core.editor @var{nano}
@end example

Finally, and in some ways most importantly, let's make sure that
we know what branch we're on.  If you're not using LilyDev, add
this to your @file{~/.bashrc}:

@verbatim
export PS1="\u@\h \w\$(__git_ps1)$ "
@end verbatim

You may need to install the additional @code{bash-completion}
package, but it is definitely worth it.  After installation
you must log out, and then log back in again to enable it.


@subsubheading Technical details

Git stores the information entered with
@command{git@tie{}config@tie{}--global} in the file
@file{.gitconfig}, located in your home directory.  This file can
also be modified directly, without using
@command{git@tie{}config}.  The @file{.gitconfig} file generated
by the above commands would look like this:

@example
[user]
        name = John Smith
        email = john@@example.com
[color]
        ui = auto
[core]
        editor = nano
@end example

Using the @command{git@tie{}config} command @emph{without} the
@option{--global} option configures repository-specific settings,
which are stored in the file @file{.git/config}.  This file is
created when a repository is initialized (using
@command{git@tie{}init}), and by default contains these lines:

@example
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
@end example

However, since different repository-specific options are
recommended for different development tasks, it is best to avoid
setting any now.  Specific recommendations will be mentioned later
in this manual.


@node Git for the impatient
@subsection Git for the impatient

@advanced{The intent of this subsection is to get you working on lilypond as
soon as possible.  If you want to learn about git, go read
@ref{Other Git documentation}.
@*
Also, these instructions are designed to eliminate the most common
problems we have found in using git.  If you already know git and
have a different way of working, great!  Feel free to ignore the
advice in this subsection.}


Ok, so you've been using @command{lily-git.tcl} for a while, but
it's time to take the next step.  Since our review process delays
patches by 60-120 hours, and you want to be able to work on other
stuff while your previous work is getting reviewed, you're going
to use @strong{branches}.

You can think of a branch as being a separate copy of the source
code.  But don't worry about it.

@subsubheading Start work: make a new branch

Let's pretend you want to add a section to the Contributor's Guide
about using branches.

Start by updating the repository, then making a new branch.  Call
the branch anything you want as long as the name starts with
@code{dev/}.  Branch names that don't begin with @code{dev/} are
reserved for special things in lilypond.

@example
git checkout master
git pull -r origin master
git branch dev/cg
@end example

@subsubheading Switch to that branch

Nothing has happened to the files yet.  Let's change into the new
branch.  You can think of this as @qq{loading a file}, although in
this case it's really @qq{loading a directory and subdirectories
full of files}.

@example
git checkout dev/cg
@end example

Your prompt now shows you that you're on the other branch:

@example
gperciva@@LilyDev:~/lilypond-git (dev/cg)$
@end example

To be able to manage multiple lilypond issues at once, you'll need to switch
branches.  You should have each lilypond issue on a separate branch.
Switching branches is easy:

@example
git checkout master
git checkout origin/staging
git checkout origin/release/unstable
git checkout dev/cg
@end example

Branches that begin with @code{origin/} are part of the remote repository,
rather than your local repository, so when you check them out you get a 
temporary local branch.  You should never make changes directly on a
branch beginning with @code{origin/}.  You get changes into the remote
repository by making them in local branches, and then pushing them to
@code{origin/staging} as described below. 

@subsubheading Make your changes

Edit files, then commit them.

@example
git commit -a
@end example


Remember how I said that switching to a branch was like
@qq{loading a directory}?  Well, you've just @qq{saved a
directory}, so that you can @qq{load} it later.

@advanced{If you have used @command{cvs} or @command{svn}, you may
be very confused: those programs use @qq{commit} to mean
@qq{upload my changes to the shared source repository}.
Unfortunately, just to be different, @w{@command{git commit}}
means @qq{save my changes to the files}.}

When you create a new file, you need to add it to git, then commit it:

@example
git add input/regression/avoid-crash-on-condition.ly
git commit -a
@end example


Edit more files.  Commit them again.  Edit yet more files, commit
them again.  Go eat dinner.  Switch to @code{master} so you can
play with the latest changes from other developers.  Switch back
to your branch and edit some more.  Commit those changes.

At this stage, don't worry about how many commits you have.


@subsubheading Save commits to external files

Branches are nerve-wracking until you get used to them.  You can
save your hard work as individual @file{.patch} files.  Be sure to
commit your changes first.

@example
git commit -a
git format-patch master
@end example

I personally have between 4 and 20 of those files saved in a
special folder at any point in time.  Git experts might laugh as
that behavior, but I feel a @emph{lot} better knowing that I've
got those backups.


@subsubheading Prepare your branch for review

After committing, you can update your branch with the latest master:

@example
git commit -a
git checkout master
git pull -r origin master
git checkout dev/cg
git rebase master
@end example


Due to the speed of lilypond development, sometimes
@code{master} has changed so much that your branch can no
longer be applied to it.  In that happens, you will have a merge
conflict.  Stop for a moment to either cry or have a stiff drink,
then proceed to @ref{Merge conflicts}.


@subsubheading Upload your branch

Finally, you're finished your changes.  Time to upload for review.
Make sure that you're on your branch, then upload:

@example
git checkout dev/cg
git-cl upload master
@end example


@subsubheading Wait for reviews

While you're waiting for a countdown and reviews, go back to
master, make a @code{dev/doc-beams} branch, and start adding doc
suggestions from issue 12345 from the tracker.  Or make a
@code{dev/page-breaks} and fix bug in page breaking.  Or whatever.
Don't worry, your @code{dev/cg} is safe.


@subsubheading Combining commits (optional unless you have broken commits)

Does the history of your branch look good?

@example
gitk
@end example

If you have a lot of commits on your branch, you might want to
combine some of them.  Alternately, you may like your commits, but
want to edit the commit messages.

@example
git rebase -i master
@end example

Follow instructions on the screen.

@warning{This step gives you the power to completely lose your
work.  Make a backup of your commits by saving them to
@file{.patch} files before playing with this.  If you do lose
your work, don't despair.  You can get it back by using @code{git reflog}.
The use of @code{git reflog} is not covered here.}

@warning{If any of the commits on your branch represent partial work that will
not pass @var{make && make doc}, you @strong{must} squash these 
commits into a working commit.  Otherwise, your push will break staging
and will not be able to be merged to master.  In general, you will
be safer to have one commit per push.}


@subsubheading Push to staging

When you've got the coveted @code{Patch-push} status, time to
prepare your upload:

@example
git fetch
git rebase origin/staging dev/cg~0
gitk HEAD
@end example

@warning{Do not skip the @command{gitk} step; a quick 5-second
check of the visual history can save a great deal of frustration
later on.  You should see a set of your commits that are ahead of
@code{origin/staging}, with no label for the top commit -- only a
SHA1 id.}

@warning{If @code{origin/staging} and @code{origin/master} are the
same commit, your branch (@code{dev/cg} in the example) will also
be at the top of the @code{gitk} tree.  This is normal.}

If everything looks good, push it:

@example
git push origin HEAD:staging
@end example

Then change back to your working branch:

@example
git checkout dev/cg
@end example

@warning{It is a best practice to avoid rebasing any of your branches
to @code{origin/staging}.  If @code{origin/staging} is broken, it
will be deleted and rebuilt.  If you have rebased one of your branches
to @code{origin/staging}, the broken commits can end up in your branch.
The commands given above do the rebase on a temporary branch, and avoid
changing your working branch.}


@subsubheading Delete your branch (safe)

After a few hours, if there's nothing wrong with your branch, it
should be automatically moved to @code{origin/master}.  Update,
then try removing your branch:

@example
git checkout master
git pull -r origin master
git branch -d dev/cg
@end example

The last command will fail if the contents of @code{dev/cg} are
not present in @code{origin/master}.


@subsubheading Delete your branch (UNSAFE)

@c don't give explicit commands here -- this is too dangerous to copy and paste
Sometimes everything goes wrong.  If you want to remove a branch even though 
it will cause your work to be lost (that is, if the contents of @code{dev/cg}
are @strong{not} present in master), follow the instructions in @qq{Delete
your branch (safe)}, but replace the @code{-d} on the final line with 
a @code{-D}.


@node Other repositories
@subsection Other repositories

We have a few other code repositories.

@menu
* lilypond-extra::
* Grand Unified Builder (GUB)::
* LilyPad::
* yet more repositories::
@end menu

@node lilypond-extra
@unnumberedsubsubsec lilypond-extra

There is a separate repository for general administrative scripts,
as well as pictures and media files for the website.  People
interested in working on the website should download this
repository, and set their @code{$LILYPOND_WEB_MEDIA_GIT}
environment variable to point to that repository.

@example
@uref{https://github.com/gperciva/lilypond-extra}
@end example

To configure an environment variable in bash (the default for most
GNU/Linux distributions),

@example
export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/
@end example

Be aware that @code{lilypond-extra} is the definitive source for some binary
files - in particular PDF versions of papers concerning LilyPond.  To add
further PDFs of this sort, all that is necessary is to add the PDF to
@code{lilypond-extra} and then add a reference to it in the documentation.  The
file will then be copied to the website when @code{make website} is run.

However, pictures that are also used in the documentation build are mastered in
the main git repository.  If any of these is changed, it should be updated in
git, and then the updates copied to @code{lilypond-extra}.

@node Grand Unified Builder (GUB)
@unnumberedsubsubsec Grand Unified Builder (GUB)

Another item of interest might be the Grand Unified Builder, our
cross-platform building tool.  Since it is used by other projects as
well, it is not stored in our gub repository.  For more info, see
@uref{http://lilypond.org/gub}.

There are two locations for this repository: the version being used to
build lilypond, which is at

@example
@uref{http://github.com/gperciva/gub}
@end example

and the original version by Jan Nieuwenhuizen, kept at

@example
@uref{http://github.com/janneke/gub}
@end example


@node LilyPad
@unnumberedsubsubsec LilyPad

Our binary releases on MacOS X and Windows contain a lightweight
text editor.

To make any modifications the Windows editor, you will need to do the
following:

@enumerate
@item
Clone the git repository from @code{https://github.com/gperciva/lilypad}

@item
Make changes to the source, and check it compiles.  In a Windows environment
@code{MinGW} provides both a @code{Git} installation and a @code{gcc}
compiler.  This can be obtained from @code{http://www.mingw.org/}

@item
Update the version which is contained in the @file{rsrc.rc}.  Check
this compiles, too.

@item
Commit the changes with an informative commit message.

@item
Push the changes to github.  You will need to use syntax similiar to this:

@example
git push https://UserName@@github.com/gperciva/lilypad.git
@end example

You will need to have push access to the git repository for this to be
successful.

@item
Make a tarball of the source code to be used by GUB by pulling the updated
repository from GitHub.  Ensure that the tarball has the correct Version
number.

@item
Copy the tarball to @code{http://lilypond.org/downloads/gub-sources/lilypad/}.
You will need to have SSH access to @code{lilypond.org}.  If you do not, contact
the Release Manager via the lilypond-devel mailing list.

@item
Update GUB to make it use the new tarball by editing
@file{gub/specs/lilypad.py} and changing the @code{source =} line to point to
the new source.

@item
Push this updated @file{lilypad.py} version to the GUB repository on GitHub.

@item
Test the changes with a new GUB compile.

@end enumerate

@node yet more repositories
@unnumberedsubsubsec yet more repositories

There are a few other repositories floating around, which will
hopefully be documented in the near future.


@node Downloading remote branches
@subsection Downloading remote branches

@warning{contains obsolete + misleading info}

@menu
* Organization of remote branches::
* LilyPond repository sources::
* Downloading individual branches::
* Downloading all remote branches::
* Other branches::
@end menu


@node Organization of remote branches
@unnumberedsubsubsec Organization of remote branches


The main LilyPond repository is organized into @emph{branches} to
facilitate development.  These are often called @emph{remote}
branches to distinguish them from @emph{local} branches you might
create yourself (see @ref{Using local branches}).

The @code{master} branch contains all the source files used to
build LilyPond, which includes the program itself (both stable and
development releases), the documentation (and its translations),
and the website.  Generally, the @code{master} branch is expected
to compile successfully.

The @code{translation} branch is a side branch that
allows translators to work without needing to worry about
compilation problems.  Periodically, the Translation Meister
(after verifying that it doesn't break compilation), will
@emph{merge} this branch into @code{staging} to incorporate
recent translations.  Similarly, the @code{master} branch is
usually merged into the @code{translation} branch after
significant changes to the English documentation.  See
@ref{Translating the documentation} for details.


@node LilyPond repository sources
@unnumberedsubsubsec LilyPond repository sources


The recommended source for downloading a copy of the main
repository is:

@example
git://git.sv.gnu.org/lilypond.git
@end example

However, if your internet router filters out connections using the
GIT protocol, or if you experience difficulty connecting via GIT,
you can try these other sources:

@example
ssh://git.sv.gnu.org/srv/git/lilypond.git
http://git.sv.gnu.org/r/lilypond.git
@end example

The SSH protocol can only be used if your system is properly set
up to use it.  Also, the HTTP protocol is slowest, so it should
only be used as a last resort.


@node Downloading individual branches
@unnumberedsubsubsec Downloading individual branches

@warning{obsolete, should be deleted!}


Once you have initialized an empty Git repository on your system
(see @ref{Initializing a repository}), you can download a remote
branch into it.  Make sure you know which branch you want to start
with.

To download the @code{master} branch, enter the following:

@example
git remote add -ft master -m master \
  origin git://git.sv.gnu.org/lilypond.git/
@end example

To download the @code{translation} branch, enter:

@example
git remote add -ft translation -m \
  translation origin git://git.sv.gnu.org/lilypond.git/
@end example

The @command{git@tie{}remote@tie{}add} process could take up to
ten minutes, depending on the speed of your connection.  The
output will be something like this:

@example
Updating origin
remote: Counting objects: 235967, done.
remote: Compressing objects: 100% (42721/42721), done.
remote: Total 235967 (delta 195098), reused 233311 (delta 192772)
Receiving objects: 100% (235967/235967), 68.37 MiB | 479 KiB/s, done.
Resolving deltas: 100% (195098/195098), done.
From git://git.sv.gnu.org/lilypond
 * [new branch]      master     -> origin/master
From git://git.sv.gnu.org/lilypond
 * [new tag]         flower/1.0.1 -> flower/1.0.1
 * [new tag]         flower/1.0.10 -> flower/1.0.10
⋮
 * [new tag]         release/2.9.6 -> release/2.9.6
 * [new tag]         release/2.9.7 -> release/2.9.7
@end example

When @command{git@tie{}remote@tie{}add} is finished, the remote
branch should be downloaded into your repository---though not yet
in a form that you can use.  In order to browse the source code
files, you need to @emph{create} and @emph{checkout} your own
local branch.  In this case, however, it is easier to have Git
create the branch automatically by using the @command{checkout}
command on a non-existent branch.  Enter the following:

@example
git checkout -b @var{branch} origin/@var{branch}
@end example

@noindent
where @code{@var{branch}} is the name of your tracking branch,
either @code{master} or @code{translation}.

Git will issue some warnings; this is normal:

@example
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of origin/master.
Branch master set up to track remote branch master from origin.
Already on 'master'
@end example

By now the source files should be accessible---you should be able
to edit any files in the @file{$LILYPOND_GIT} directory using a
text editor of your choice.  But don't start just yet!  Before
editing any source files, learn how to keep your changes organized
and prevent problems later---read @ref{Basic Git procedures}.

@subsubheading Technical Details

The @command{git@tie{}remote@tie{}add} command should add some
lines to your local repository's @file{.git/config} file:

@example
[remote "origin"]
        url = git://git.sv.gnu.org/lilypond.git/
        fetch = +refs/heads/master:refs/remotes/origin/master
@end example


@node Downloading all remote branches
@unnumberedsubsubsec Downloading all remote branches


To download all remote branches at once, you can @command{clone}
the entire repository:

@example
git clone git://git.sv.gnu.org/lilypond.git
@end example


@node Other branches
@unnumberedsubsubsec Other branches

Most contributors will never need to touch the other branches.  If
you wish to do so, you will need more familiarity with Git; please
see @ref{Other Git documentation}.

@itemize
@item @code{dev/XYZ}:
These branches are for individual developers.  They store code
which is not yet stable enough to be added to the @code{master}
branch.

@item @code{stable/XYZ}:
The branches are kept for archival reasons.

@item @code{archive/XYZ}:
The branches are kept for archival reasons.

@end itemize


@node Basic Git procedures
@section Basic Git procedures


@menu
* The Git contributor's cycle::
* Pulling and rebasing::
* Using local branches::
* Commits::
* Patches::
* Uploading a patch for review::
* The patch review cycle::
@end menu


@node The Git contributor's cycle
@subsection The Git contributor's cycle


Here is a simplified view of the contribution process on Git:

@enumerate
@item
Update your local repository by @emph{pulling} the most recent
updates from the remote repository.

@item
Edit source files within your local repository's @emph{working
directory}.

@item
@emph{Commit} the changes you've made to a local @emph{branch}.

@item
Generate a @emph{patch} to share your changes with the developers.
@end enumerate


@node Pulling and rebasing
@subsection Pulling and rebasing


When developers push new patches to the @code{git.sv.gnu.org}
repository, your local repository is @strong{not} automatically
updated.  It is important to keep your repository up-to-date by
periodically @emph{pulling} the most recent @emph{commits} from
the remote branch.  Developers expect patches to be as current as
possible, since outdated patches require extra work before they
can be used.

Occasionally you may need to rework some of your own modifications
to match changes made to the remote branch (see @ref{Resolving
conflicts}), and it's considerably easier to rework things
incrementally.  If you don't update your repository along the way,
you may have to spend a lot of time resolving branch conflicts and
reconfiguring much of the work you've already done.

Fortunately, Git is able to resolve certain types of branch
conflicts automatically with a process called @emph{rebasing}.
When rebasing, Git tries to modify your old commits so they appear
as new commits (based on the latest updates).  For a more involved
explanation, see the @command{git-rebase} man page.

To pull without rebasing (recommended for translators), use the
following command:

@example
git pull    # recommended for translators
@end example

If you're tracking the remote @code{master} branch, you should add
the @option{-r} option (short for @option{--rebase}) to keep commits
on your local branch current:

@example
git pull -r # use with caution when translating
@end example

If you don't edit translated documentation and don't want to type
@option{-r} every time, configure the master branch to rebase by
default with this command:

@example
git config branch.master.rebase true
@end example

If pull fails because of a message like

@example
error: Your local changes to 'Documentation/learning/tutorial.itely'
would be overwritten by merge.  Aborting.
@end example

@noindent
or

@example
Documentation/learning/tutorial.itely: needs update
refusing to pull with rebase: your working tree is not up-to-date
@end example

@noindent
it means that you have modified some files in you working tree
without committing changes (see @ref{Commits}); you
can use the @command{git@tie{}stash} command to work around this:

@example
git stash      # save uncommitted changes
git pull -r    # pull using rebase (translators omit "-r")
git stash pop  # reapply previously saved changes
@end example

Note that @command{git@tie{}stash@tie{}pop} will try to apply a
patch, and this may create a conflict.  If this happens, see
@ref{Resolving conflicts}.

TODO: I think the next paragraph is confusing.  Perhaps prepare
the reader for new terms `committish' and `head'?  -mp

@warning{translators and documentation editors, if you have
changed committishes in the head of translated files using commits
you have not yet pushed to @code{git.sv.gnu.org}, please do not
rebase.  If you want to avoid wondering whether you should rebase
each time you pull, please always use committishes from master
and/or translation branch on @code{git.sv.gnu.org}, which
in particular implies that you must push your changes to
documentation except committishes updates (possibly after having
rebased), then update the committishes and push them.}

TODO: when committishes automatic conditional update have been
tested and documented, append the following to the warning above:
Note that using update-committishes make target generally touches
committishes.

@subsubheading Technical details

The @command{git@tie{}config} command mentioned above adds the
line @code{rebase = true} to the master branch in your local
repository's @file{.git/config} file:

@example
[branch "master"]
        remote = origin
        merge = refs/heads/master
        rebase = true
@end example


@node Using local branches
@subsection Using local branches


@menu
* Creating and removing branches::
* Listing branches and remotes::
* Checking out branches::
* Merging branches::
@end menu


@node Creating and removing branches
@unnumberedsubsubsec Creating and removing branches


Local branches are useful when you're working on several different
projects concurrently.  To create a new branch, enter:

@example
git branch @var{name}
@end example

To delete a branch, enter:

@example
git branch -d @var{name}
@end example

Git will ask you for confirmation if it sees that data would be
lost by deleting the branch.  Use @option{-D} instead of @option{-d}
to bypass this.  Note that you cannot delete a branch if it is
currently checked out.


@node Listing branches and remotes
@unnumberedsubsubsec Listing branches and remotes

You can get the exact path or URL of all remote branches by
running:

@example
git remote -v
@end example

To list Git branches on your local repositories, run

@example
git branch     # list local branches only
git branch -r  # list remote branches
git branch -a  # list all branches
@end example


@node Checking out branches
@unnumberedsubsubsec Checking out branches

To know the currently checked out branch, i.e. the branch whose
source files are present in your working tree, read the first line
of the output of

@example
git status
@end example

@noindent
The currently checked out branch is also marked with an asterisk
in the output of @command{git branch}.

You can check out another branch @code{@var{other_branch}}, i.e.
check out @code{@var{other_branch}} to the working tree, by
running

@example
git checkout @var{other_branch}
@end example

Note that it is possible to check out another branch while having
uncommitted changes, but it is not recommended unless you know
what you are doing; it is recommended to run @command{git status}
to check this kind of issue before checking out another branch.

@node Merging branches
@unnumberedsubsubsec Merging branches

To merge branch @code{@var{foo}} into branch @code{@var{bar}},
i.e. to @qq{add} all changes made in branch @code{@var{foo}} to
branch @code{@var{bar}}, run

@example
git checkout @var{bar}
git merge @var{foo}
@end example

If any conflict happens, see @ref{Resolving conflicts}.

There are common usage cases for merging: as a translator, you will
often want the Translations meister to merge @code{master} into
@code{translation}; on the other hand, the Translations meister wants
to merge @code{translation} into @code{staging} whenever he has
checked that @code{translation} builds successfully.


@node Commits
@subsection Commits

@menu
* Understanding commits::
* How to make a commit::
* Commit messages::
@end menu


@node Understanding commits
@unnumberedsubsubsec Understanding commits

Technically, a @emph{commit} is a single point in the history of a
branch, but most developers use the term to mean a @emph{commit
object}, which stores information about a particular revision.  A
single commit can record changes to multiple source files, and
typically represents one logical set of related changes (such as a
bug-fix).  You can list the ten most recent commits in your
current branch with this command:

@example
git log -10 --oneline
@end example

If you're using an older version of Git and get an @q{unrecognized
argument} error, use this instead:

@example
git log -10 --pretty=oneline --abbrev-commit
@end example

More interactive lists of the commits on the remote @code{master}
branch are available at
@uref{http://git.sv.gnu.org/gitweb/?p=lilypond.git;a=shortlog} and
@uref{http://git.sv.gnu.org/cgit/lilypond.git/log/}.


@node How to make a commit
@unnumberedsubsubsec How to make a commit


Once you have modified some source files in your working
directory, you can make a commit with the following procedure:

@enumerate
@item
Make sure you've configured Git properly (see @ref{Configuring
Git}).  Check that your changes meet the requirements described in
@ref{Code style} and/or @ref{Documentation policy}.  For advanced
edits, you may also want to verify that the changes don't break
the compilation process.

@item
Run the following command:

@example
git status
@end example

@noindent
to make sure you're on the right branch, and to see which files
have been modified, added or removed, etc.  You may need to tell
Git about any files you've added by running one of these:

@example
git add @var{file}  # add untracked @var{file} individually
git add .     # add all untracked files in current directory
@end example

@noindent
After @command{git@tie{}add}, run @command{git@tie{}status} again
to make sure you got everything.  You may also need to modify
@file{GNUmakefile}.

@item
Preview the changes about to be committed (to make sure everything
looks right) with:

@example
git diff HEAD
@end example

@noindent
The @code{HEAD} argument refers to the most recent commit on the
currently checked-out branch.

@item
Generate the commit with:

@example
git commit -a
@end example

@noindent
The @option{-a} is short for @option{--all} which includes modified
and deleted files, but only those newly created files that have
previously been added.

@end enumerate


@node Commit messages
@unnumberedsubsubsec Commit messages


When you run the @command{git@tie{}commit@tie{}-a} command, Git
automatically opens the default text editor so you can enter a
@emph{commit message}.  If you find yourself in a foreign editing
environment, you're probably in @command{vi} or @command{vim}.  If
you want to switch to an editor you're more familiar with, quit by
typing @code{:q!} and pressing @code{<Enter>}.  See
@ref{Configuring Git} for instructions on changing the default
editor.

In any case, Git will open a text file for your commit message
that looks like this:

@example

# Please enter the commit message for your changes.  Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   working.itexi
#
@end example

Your commit message should begin with a one-line summary
describing the change (no more than 50 characters long), and if
necessary a blank line followed by several lines giving the
details:

@c $ git log -1 --pretty=medium 4d6f1e5
@example
Doc: add Baerenreiter and Henle solo cello suites

Added comparison of solo cello suite engravings to new essay with
high-res images, fixed cropping on Finale example.
@end example

Commit messages often start with a short prefix describing the
general location of the changes.

@itemize
@item
Doc: and Doc-@var{**}:  If a commit affects the
documentation in English (or in several languages simultaneously)
the commit message should be prefixed with @qq{Doc:@tie{}}.  If the
commit affects only one of the translations, the commit
message should be prefixed with @qq{Doc-@var{**}:@tie{}}, where
@var{**} is the two-letter language code.

@item
Web: and Web-@var{**}:  Commits that affect the
website should use @qq{Web:@tie{}} for English, and
@qq{Web-@var{**}:@tie{}} for other languages.

@item
CSS: Commits that change CSS files should use @qq{Web:@tie{}CSS:@tie{}}
or @qq{Doc:@tie{}CSS:@tie{}} depending on whether they affect the
website or the documentation/manuals.

@item
Changes to a single file are often prefixed with the name of the file
involved.
@end itemize

Visit the links listed in @ref{Understanding commits} for examples.



@node Patches
@subsection Patches

@menu
* How to make a patch::
* Emailing patches::
@end menu

@node How to make a patch
@unnumberedsubsubsec How to make a patch

If you want to share your changes with other contributors and
developers, you need to generate @emph{patches} from your commits.
We prefer it if you follow the instructions in
@ref{Uploading a patch for review}.  However, we present an
alternate method here.

You should always run @command{git@tie{}pull@tie{}-r} (translators
should leave off the @option{-r}) before doing this to ensure that
your patches are as current as possible.

Once you have made one or more commits in your local repository,
and pulled the most recent commits from the remote branch, you can
generate patches from your local commits with the command:

@example
git format-patch origin
@end example

The @code{origin} argument refers to the remote tracking branch at
@code{git.sv.gnu.org}.  This command generates a separate patch
for each commit that's in the current branch but not in the remote
branch.  Patches are placed in the current working directory and
will have names that look something like this:

@example
0001-Doc-Fix-typos.patch
0002-Web-Remove-dead-links.patch
⋮
@end example

Send an email (must be less than 64 KB) to
@email{lilypond-devel@@gnu.org} briefly explaining your work, with
the patch files attached.  Translators should send patches to
@email{translations@@lilynet.net}.  After your patches are
reviewed, the developers may push one or more of them to the main
repository or discuss them with you.


@node Emailing patches
@unnumberedsubsubsec Emailing patches

The default @code{x-diff} MIME type associated with patch files
(i.e., files whose name ends in @code{.patch}) means that the
encoding of line endings may be changed from UNIX to DOS format
when they are sent as attachments.  Attempting to apply such an
inadvertently altered patch will cause git to fail with a message
about @q{whitespace errors}.

The solution to such problems is surprisingly simple---just change
the default file extension of patches generated by git to end in
@code{.txt}, for example:

@example
git config format.suffix '.patch.txt'
@end example

This should cause email programs to apply the correct base64
encoding to attached patches.

If you receive a patch with DOS instead of UNIX line-endings, it
can be converted back using the @code{dos2unix} utility.

Lots of useful information on email complications with patches is
provided on the Wine wiki at
@uref{http://wiki.winehq.org/GitWine}.


@node Uploading a patch for review
@subsection Uploading a patch for review

Any non-trivial change should be uploaded to our @qq{Rietveld}
code review website:

@example
@uref{http://codereview.appspot.com/}
@end example

You can upload a patch for review by using our custom @code{git-cl}
@q{helper-script}.  This section assumes you have already installed,
updated, and configured @code{git-cl}.  See @ref{git-cl}.

@warning{Unless you are familiar with branches, only work on one
set of changes at once.}

There are two methods, depending on your git setup.

@itemize
@item
@strong{Master branch}: (easy option)

If you added your patch to @code{master}, then:

@example
git pull -r
git-cl upload origin/master
@end example

@c Mention staging here?
If you have git push ability, make sure that you @emph{remove}
your patch (with @command{git rebase} or @command{git reset})
before pushing other stuff.

@c don't make this one an @example; we don't want to make it easy
@c for people to use this accidently
Notifications of patches are automatically added to our issue
tracker to reduce the chance of patches getting lost.  To suppress
this (not recommended), add the @code{-n / --no-code-issue}
option.

@item
@strong{Separate branch}: (complicated option)

Ensure your changes are committed in a separate branch, which should
differ from the reference branch to be used (usually
@code{origin/master}) by just the changes to be uploaded.  Checkout the
branch with the changes:

@example
git checkout some-branch-with-changes
@end example

If the reference branch is to be @code{origin/master}, ensure that the
branch containing the changes is up-to-date with it.  Use
@command{git rebase} or @command{git pull -r} to rebase the branch to
the head of @code{origin/master}.  For example:

@example
git pull -r origin master
@end example

Finally, start the upload by entering:

@example
git-cl upload <reference SHA1 ID>
@end example

@noindent
where <reference SHA1 ID> is the SHA1 ID of the commit to be used
as a reference source for the patch.  Generally, this will be the
SHA1 ID of origin/master, and in that case you can just use the command:

@example
git-cl upload origin/master
@end example

@end itemize

First you will see a terminal editor where you can edit the
message that will accompany your patch. @command{git-cl} will
respect the @env{EDITOR} environment variable if defined,
otherwise it will use @command{vi} as the default editor.

After prompting for your Google email address and password, the
patch set will be posted to Rietveld, and you will be given a URL
for your patch.

@warning{Some installations of git-cl fail when uploading a patch
with certain filename extensions.  If this happens, it can
generally be fixed by editing the list of exceptions at top of
@file{git-cl.py}.}

@subsubheading Announcing your patch set

You should then announce the patch by logging into the code review
issue webpage and using @qq{Publish + Mail Comments} to add a
(mostly bogus) comment to your issue.  The text of your comment
will be sent to our developer mailing list.

@warning{There is no automatic notification of a new patch; you
must add a comment yourself.}

@subsubheading Revisions

As revisions are made in response to comments, successive patch sets
for the same issue can be uploaded by reissuing the git-cl command
with the modified branch checked out.

Sometimes in response to comments on revisions, the best way to
work may require creation of a new branch in git.  In order to
associate the new branch with an existing Rietveld issue,
the following command can be used:

@example
git-cl issue issue-number
@end example

@noindent
where @code{issue-number} is the number of the existing Rietveld
issue.

@subsubheading Resetting git-cl

If @command{git-cl} becomes confused, you can @qq{reset} it by
running:

@example
git-cl issue 0
@end example


@node The patch review cycle
@subsection The patch review cycle

Your patch will be available for reviews for the next few hours or
days.  Three times a week, patches with no known problems are
gathered into a @qq{patch countdown} and their status changed to
@code{patch-countdown}.  The countdown is a 48-hour waiting period
in which any final reviews or complaints should be made.

During the countdown, your patch may be set to
@code{patch-needs_work}, indicating that you should fix something
(or at least discuss why the patch needs no modification).  If no
problems are found, the patch will be set to @code{patch-push}.

Once a patch has @code{patch-push}, it should be sent to your
mentor for uploading.  If you have git push ability, look at
@ref{Pushing to staging}.

@itemize

@item
Patches get added to the tracker and to Rietveld by the @qq{git-cl} tool, with
a status of @qq{patch-new}.

@item
The automated tester, Patchy, verifies that the patch can be applied
to current master.  By default, it checks that the patch allows @code{make}
and @code{make test} to complete successfully.  It can also be configured to
check that @code{make doc} is successful. If it passes, Patchy changes the
status to @qq{patch-review} and emails the developer list.  If the patch
fails, Patchy sets it to @qq{patch-needs_work} and notifies the developer list.

@item
The Patch Meister reviews the tracker periodically, to list patches
which have been on review for at least 24 hours. The list is found at

@smallexample
@uref{http://code.google.com/p/lilypond/issues/list?can=2&q=label:patch%20patch=review&sort=modified+patch&colspec=ID%20Type%20Status%20Priority%20Owner%20Patch%20Summary%20Modified}
@end smallexample

@item
For each patch, the Handler reviews any discussion on the tracker
and on Rietveld, to determine whether the patch can go forward.  If
there is any indication that a developer thinks the patch is not
ready, the Handler marks it @qq{patch-needs_work} and makes a comment
regarding the reason, referring to the Rietveld item if needed.

@item
Patches with explicit approval, or at least no negative comment, can
be updated to @qq{patch-countdown}.  When saving the tracker item,
clear the @qq{send email} box to prevent sending notification for
each patch.

@item
The Patch Meister sends an email to the developer list, with a fixed
subject line, to enable filtering by email clients:

@example
PATCH: Countdown to 20130113
@end example

The text of the email sets the deadline for this countdown batch.  At
present, batches are done on Tuesday, Thursday and Sunday evenings.

To create the countdown announcement, use the
@code{make-countdown-announcement.sh} script, which takes the
deadline date, and optionally your name.  Follow the instructions
provided:

@example
cd $LILYPOND_GIT
scripts/auxiliar/make-countdown-announcement.sh "Jan 1, 2001" James
@end example

The script produces an announcement that is easily readable in all
email clients.  Also, whenever a new contributor submits a patch,
you will be prompted to add the new username and author name to
the script itself, and then commit those changes to the main git
repository.


@item
On the scheduled countdown day, the Patch Meister reviews the
previous list of patches on countdown, with the same procedure and
criteria as before.  Patches with no controversy can be set to
@qq{patch-push} with a courtesy message added to the comment block.

@item
Roughly at six month intervals, the Patch Meister can list the
patches which have been set to @qq{patch-needs-work} and send the
results to the developer list for review.  In most cases, these
patches should be marked @qq{patch-abandoned} but this should come
from the developer if possible.

@item
As in most organisations of unpaid volunteers, fixed procedures are
useful in as much as they get the job done.  In our community, there
is room for senior developers to bypass normal patch handling flows,
particularly now that the testing of patches is largely automated.
Similarly, the minimum age of 24 hours can reasonably be waived if
the patch is minor and from an experienced developer.


@end itemize

@ignore
There is a single Patch Meister, and a number of Patch Helpers
(rename this?).  The list of known patches awaiting review is:

@example
@uref{http://code.google.com/p/lilypond/issues/list?can=2&q=label:patch&sort=patch}
@end example


@subheading Helpers: adding patches

The primary duty is to add patches to the google tracker; we have
a bad track record of losing patches in email.  Patches generally
come to the @code{lilypond-devel} mailing list, but are sometimes
sent to @code{bug-lilypond}, @code{lilypond-users}, or
@code{frogs} mailing list instead.

@itemize
@item
Unless a patch is clearly in response to an existing issue, add a
new issue with the @code{Patch-new} label and a link to the patch
(either on the mailing list archives or the codereview url).

Issue numbers are cheap; losing developers because they got fed up
with us losing their hard work is expensive.


@c if we enter patches immediately, I don't think this is relevant.

@item
Before adding a patch-reminder issue, do a quick check to see if
it was pushed without sending any email.  This can be checked for
searching for relevant terms (from the patch subject or commit
message) on the webgit page:

@example
@uref{http://git.savannah.gnu.org/gitweb/?p=lilypond.git}
@end example


@item
If the patch is clearly in response to an existing issue, then
update that issue with the @code{Patch-new} label and a link to
the patch (either on the mailing list archives or the codereview
url).

@item
After adding the issue, please send a response email to the same
group(s) that the initial patch was sent to.

If the initial email was sent to multiple mailing lists (such as
both @code{bugs} and @code{devel}), then reply to all those
mailing lists as well.  The email should contain a link to the
issue you just added.

@end itemize

@subheading Helpers: @code{Patch-review} label

The secondary duty is to do make sure that every issue in the
tracker with a @code{Patch-review} label has passed these
@qq{obvious} tests:

@itemize
@item
Applies automatically to git master.

It's ok to have offsets, but not conflicts.

@item
Regtest comparison looks ok; no unexpected changes.

@item
Descriptive subject line.

Avoid subjects like @qq{fixes 123}; instead write @qq{Doc: discuss
stacking-dir for BassFigureAlignment (fix 123)}.

@item
Compiles docs from scratch.  Only check this if you have reason to
suspect it might not work.

@item
(maybe)

Check code indentation and style.  This should be easier post-GOP
when we have a better-defined code style.

@end itemize


@subheading Patch Meister

The Patch Meister will:

@itemize

@item
send @qq{countdown} emails to
@code{lilypond-devel} when patches appear to be ready.

@item
send general requests to review patches, or even nasty requests to
review patches.

@item
downgrade patches from @code{Patch-review} to
@code{Patch-needs_work} as appropriate.

@item
downgrade patches from @code{Patch-needs_work} to
@code{Patch-abandoned} if no actions have been taken in four
weeks.

@end itemize

@end ignore


@node Advanced Git procedures
@section Advanced Git procedures


@warning{This section is not necessary for normal contributors;
these commands are presented for information for people interested
in learning more about git.}

It is possible to work with several branches on the same local Git
repository; this is especially useful for translators who may have
to deal with both @code{translation} and a stable branch,
e.g. @code{stable/2.12}.

Some Git commands are introduced first, then a workflow with
several Git branches of LilyPond source code is presented.


@menu
* Merge conflicts::
* Advanced Git concepts::
* Resolving conflicts::
* Reverting all local changes::
* Working with remote branches::
* Git log::
* Applying remote patches::
* Cleaning up multiple patches::
* Commit access::
* Pushing to staging::
@end menu


@node Merge conflicts
@subsection Merge conflicts

To be filled in later, and/or moved to a different section.  I
just wanted to make sure that I had a stub ready somewhere.


@node Advanced Git concepts
@subsection Advanced Git concepts


A bit of Git vocabulary will be explained below.  The following is
only introductory; for a better understanding of Git concepts, you
may wish to read @ref{Other Git documentation}.

The @code{git@tie{}pull@tie{}origin} command above is just a
shortcut for this command:

@example
git pull git://git.sv.gnu.org/lilypond.git/ @var{branch}:origin/@var{branch}
@end example

@noindent
where @code{@var{branch}} is typically @code{master} or
@code{translation}; if you do not know or remember, see
@ref{Downloading remote branches} to remember which commands you
issued or which source code you wanted to get.

A @emph{commit} is a set of changes made to the sources; it also
includes the committish of the parent commit, the name and e-mail
of the @emph{author} (the person who wrote the changes), the name
and e-mail of the @emph{committer} (the person who brings these
changes into the Git repository), and a commit message.

A @emph{committish} is the SHA1 checksum of a commit, a number
made of 40 hexadecimal digits, which acts as the internal unique
identifier for this commit.  To refer to a particular revision,
don't use vague references like the (approximative) date, simply
copy and paste the committish.

A @emph{branch} is nothing more than a pointer to a particular
commit, which is called the @emph{head} of the branch; when
referring to a branch, one often actually thinks about its head
and the ancestor commits of the head.

Now we will explain the two last commands you used to get the
source code from Git---see @ref{Downloading individual branches}.

@example
git remote add -ft @var{branch} -m @var{branch} \
  origin git://git.sv.gnu.org/lilypond.git/

git checkout -b @var{branch} origin/@var{branch}
@end example

The @command{git@tie{}remote} has created a branch called
@code{origin/@var{branch}} in your local Git repository.  As this
branch is a copy of the remote branch web from git.sv.gnu.org
LilyPond repository, it is called a @emph{remote branch}, and is
meant to track the changes on the branch from git.sv.gnu.org: it
will be updated every time you run
@command{git@tie{}pull@tie{}origin} or
@command{git@tie{}fetch@tie{}origin}.

The @command{git@tie{}checkout} command has created a branch named
@code{@var{branch}}.  At the beginning, this branch is identical
to @code{origin/@var{branch}}, but it will differ as soon as you
make changes, e.g. adding newly translated pages or editing some
documentation or code source file.  Whenever you pull, you merge
the changes from @code{origin/@var{branch}} and
@code{@var{branch}} since the last pulling.  If you do not have
push (i.e. @qq{write}) access on git.sv.gnu.org, your
@code{@var{branch}} will always differ from
@code{origin/@var{branch}}.  In this case, remember that other
people working like you with the remote branch @code{@var{branch}}
of git://git.sv.gnu.org/lilypond.git/ (called
@code{origin/@var{branch}} on your local repository) know nothing
about your own @code{@var{branch}}: this means that whenever you
use a committish or make a patch, others expect you to take the
latest commit of @code{origin/@var{branch}} as a reference.

Finally, please remember to read the man page of every Git command
you will find in this manual in case you want to discover
alternate methods or just understand how it works.


@node Resolving conflicts
@subsection Resolving conflicts


Occasionally an update may result in conflicts -- this happens
when you and somebody else have modified the same part of the same
file and git cannot figure out how to merge the two versions
together.  When this happens, you must manually merge the two
versions.

If you need some documentation to understand and resolve
conflicts, see paragraphs @emph{How conflicts are presented} and
@emph{How to resolve conflicts} in @command{git merge} man page.

If all else fails, you can follow the instructions in
@ref{Reverting all local changes}.  Be aware that this eliminates
any changes you have made!


@node Reverting all local changes
@subsection Reverting all local changes

Sometimes git will become hopelessly confused, and you just want
to get back to a known, stable state.  This command destroys any
local changes you have made in the currently checked-out branch,
but at least you get back to the current online version:

@example
git reset --hard origin/master
@end example


@node Working with remote branches
@subsection Working with remote branches


@subsubheading Fetching new branches from git.sv.gnu.org

To fetch and check out a new branch named @code{@var{branch}} on
git.sv.gnu.org, run from top of the Git repository

@example
git config --add remote.origin.fetch \
  +refs/heads/@var{branch}:refs/remotes/origin/@var{branch}

git checkout --track -b @var{branch} origin/@var{branch}
@end example

After this, you can pull @code{@var{branch}} from git.sv.gnu.org
with:

@example
git pull
@end example

Note that this command generally fetches all branches you added
with @command{git@tie{}remote@tie{}add} (when you initialized the
repository) or @command{git@tie{}config@tie{}--add}, i.e. it
updates all remote branches from remote @code{origin}, then it
merges the remote branch tracked by the current branch into the
current branch.  For example, if your current branch is
@code{master}, @code{origin/master} will be merged into
@code{master}.


@subsubheading Local clones, or having several working trees

If you play with several Git branches, e.g. @code{master},
@code{translation}, @code{stable/2.12}), you may want to
have one source and build tree for each branch; this is possible
with subdirectories of your local Git repository, used as local
cloned subrepositories.  To create a local clone for the branch
named @code{@var{branch}}, run

@example
git checkout @var{branch}
git clone -lsn . @var{subdir}
cd @var{subdir}
git reset --hard
@end example

Note that @code{@var{subdir}} must be a directory name which does
not already exist.  In @code{@var{subdir}}, you can use all Git
commands to browse revisions history, commit and uncommit changes;
to update the cloned subrepository with changes made on the main
repository, cd into @code{@var{subdir}} and run
@command{git@tie{}pull}; to send changes made on the subrepository
back to the main repository, run @command{git@tie{}push} from
@code{@var{subdir}}.  Note that only one branch (the currently
checked out branch) is created in the subrepository by default; it
is possible to have several branches in a subrepository and do
usual operations (checkout, merge, create, delete...) on these
branches, but this possibility is not detailed here.

When you push @code{@var{branch}} from @code{@var{subdir}} to the
main repository, and @code{@var{branch}} is checked out in the
main repository, you must save uncommitted changes (see
@command{git@tie{}stash}) and do
@command{git@tie{}reset@tie{}--hard} in the main repository in
order to apply pushed changes in the working tree of the main
repository.


@node Git log
@subsection Git log


The commands above don't only bring you the latest version of the
sources, but also the full history of revisions (revisions, also
called commits, are changes made to the sources), stored in the
@file{.git} directory.  You can browse this history with

@example
git log     # only shows the logs (author, committish and commit message)
git log -p  # also shows diffs
gitk        # shows history graphically
@end example

@warning{The @code{gitk} command may require a separate
@code{gitk} package, available in the appropriate distribution's
repositories.}


@node Applying remote patches
@subsection Applying remote patches


TODO: Explain how to determine if a patch was created with
@code{git@tie{}format-patch}.

Well-formed git patches created with @code{git@tie{}format-patch}
should be committed with the following command:

@example
git am @var{patch}
@end example

Patches created without @code{git@tie{}format-patch} can be
applied in two steps.  The first step is to apply the patch to the
working tree and the index:

@example
git apply --index @var{patch}
@end example

@noindent
The second step is to commit the changes and give credit to the
author of the patch.  This can be done with the following command:

@example
git commit --author="@var{John Smith} <@var{john@@example.com}>"
@end example

Please note that using the @code{--index} option for patching is quite
important here and @emph{cannot} reliably be replaced by using the
@code{-a} option when committing: that would only commit files from the
working tree that are already registered with git, so every file that
the patch actually @emph{adds}, like a regtest for a fixed bug, would
get lost.  For the same reason, you should not use the git-independent
@samp{patch} program for applying patches.



@node Cleaning up multiple patches
@subsection Cleaning up multiple patches

If you have been developing on your own branch for a while, you
may have more commmits than is really sensible.  To revise your
work and condense commits, use:

@example
git rebase origin/master
git rebase -i origin/master
@end example

@warning{Be a bit cautious -- if you completely remove commits
during the interactive session, you will... err... completely
remove those commits.}


@node Commit access
@subsection Commit access

Most contributors are not able to commit patches directly to the
main repository---only members of the LilyPond development team
have @emph{commit access}.  If you are a contributor and are
interested in joining the development team, contact the Project
Manager through the mailing list
(@email{lilypond-devel@@gnu.org}).  Generally, only contributors
who have already provided a number of patches which have been
pushed to the main repository will be considered for membership.

If you have been approved by the Project Manager, use the
following procedure to obtain commit access:

@enumerate
@item
If you don't already have one, set up a Savannah user account at
@uref{https://savannah.gnu.org/account/register.php}.  If your web
browser responds with an @qq{untrusted connection} message when
you visit the link, follow the steps for including the CAcert root
certificate in your browser, given at
@uref{http://savannah.gnu.org/tls/tutorial/}.

@warning{Savannah will silently put your username in lower-case --
do not try to use capital letters.}


@item
After registering, if you are not logged in automatically, login
at @uref{https://savannah.gnu.org/account/login.php}---this should
take you to your @qq{my} page
(@uref{https://savannah.gnu.org/my/}).


@item
Click on the @qq{My Groups} link to access the @qq{My Group
Membership} page.  From there, find the @qq{Request for Inclusion}
box and search for @qq{LilyPond}.  Among the search results, check
the box labeled @qq{GNU LilyPond Music Typesetter} and write a
brief (required) message for the Project Manager (@qq{Hey it's
me!} should be fine).

Note that you will not have commit access until the Project
Manager activates your membership.  Once your membership is
activated, LilyPond should appear under the heading @qq{Groups I'm
Contributor of} on your @qq{My Group Membership} page.


@item
Generate an SSH @q{rsa} key pair.  Enter the following at the
command prompt:

@example
ssh-keygen -t rsa
@end example

When prompted for a location to save the key, press <ENTER> to
accept the default location (@file{~/.ssh/id_rsa}).

Next you are asked to enter an optional passphrase.  On most
systems, if you use a passphrase, you will likely be prompted for
it every time you use @command{git@tie{}push} or
@command{git@tie{}pull}.  You may prefer this since it can protect
you from your own mistakes (like pushing when you mean to pull),
though you may find it tedious to keep re-entering it.

You can change/enable/disable your passphrase at any time with:

@example
ssh-keygen -f ~/.ssh/id_rsa -p
@end example

Note that the GNOME desktop has a feature which stores your
passphrase for you for an entire GNOME session.  If you use a
passphrase to @qq{protect you from yourself}, you will want to
disable this feature, since you'll only be prompted once.  Run the
following command, then logout of GNOME and log back in:

@example
gconftool-2 --set -t bool \
  /apps/gnome-keyring/daemon-components/ssh false
@end example

After setting up your passphrase, your private key is saved as
@file{~/.ssh/id_rsa} and your public key is saved as
@file{~/.ssh/id_rsa.pub}.


@item
Register your public SSH @q{rsa} key with Savannah.  From the
@qq{My Account Configuration} page, click on @qq{Edit SSH Keys},
then paste the contents of your @file{~/.ssh/id_rsa.pub} file into
one of the @qq{Authorized keys} text fields, and click
@qq{Update}.

Savannah should respond with something like:

@example
Success: Key #1 seen Keys registered
@end example


@item
Configure Git to use the SSH protocol (instead of the GIT
protocol).  From your local Git repository, enter:

@example
git config remote.origin.url \
  ssh://@var{user}@@git.sv.gnu.org/srv/git/lilypond.git
@end example

@noindent
replacing @var{user} with your Savannah username.


@item
After your membership has been activated and you've configured Git
to use SSH, test the connection with:

@example
git pull --verbose
@end example

SSH should issue the following warning:

@example
The authenticity of host 'git.sv.gnu.org (140.186.70.72)' can't
be established.
RSA key fingerprint is
80:5a:b0:0c:ec:93:66:29:49:7e:04:2b:fd:ba:2c:d5.
Are you sure you want to continue connecting (yes/no)?
@end example

Make sure the RSA key fingerprint displayed matches the one above.
If it doesn't, respond @qq{no} and check that you configured Git
properly in the previous step.  If it does match, respond
@qq{yes}.  SSH should then issue another warning:

@example
Warning: Permanently added 'git.sv.gnu.org,140.186.70.72' (RSA) to
the list of known hosts.
@end example

The list of known hosts is stored in the file
@file{~/.ssh/known_hosts}.

At this point, you are prompted for your passphrase if you have
one, then Git will attempt a pull.

If @command{git@tie{}pull@tie{}--verbose} fails, you should see
error messages like these:

@example
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
@end example

If you get the above error, you may have made a mistake when
registering your SSH key at Savannah.  If the key is properly
registered, you probably just need to wait for the Savannah server
to activate it.  It usually takes a few minutes for the key to be
active after registering it, but if it still doesn't work after an
hour, ask for help on the mailing list.

If @command{git@tie{}pull@tie{}--verbose} succeeds, the output
will include a @q{From} line that shows @q{ssh} as the protocol:

@example
From ssh://git.sv.gnu.org/srv/git/lilypond
@end example

If the protocol shown is not @q{ssh}, check that you configured
Git properly in the previous step.


@item
Test your commit access with a dry run:

@warning{Do not push directly to master; instead, push to staging.
See @ref{Pushing to staging}.}

@example
git push --dry-run --verbose
@end example

Note that recent versions of Git (Git 1.6.3 or later) will issue a
big warning if the above command is used.  The simplest solution
is to tell Git to push all matching branches by default:

@example
git config push.default matching
@end example

@noindent
Then @code{git@tie{}push} should work as before.  For more
details, consult the @code{git@tie{}push} man page.


@item
Repeat the steps from generating an RSA key through to testing
your commit access, for each machine from which you will be
making commits, or you may simply copy the files from your
local @file{~/.ssh} folder to the same folder on the other
machine.

@end enumerate

@subsubheading Technical details

@itemize
@item
On Firefox, to view or remove the CAcert root certificate, go to:
Edit > Preferences > Advanced > Encryption > View Certificates >
Authorities > Certificate Name > Root CA > CA Cert Signing
Authority.

@item
The @command{git@tie{}config} commands above should modify your
local repository's @file{.git/config} file.  These lines:

@example
[remote "origin"]
        url = git://git.sv.gnu.org/lilypond.git/
@end example

@noindent
should now be changed to:

@example
[remote "origin"]
        url = ssh://@var{user}@@git.sv.gnu.org/srv/git/lilypond.git
@end example

@noindent
where @var{user} is your login name on Savannah.

@item
Similarly, the
@command{git@tie{}config@tie{}push.default@tie{}matching} command
should add these lines to @file{.git/config}:

@example
[push]
        default = matching
@end example
@end itemize

@knownissues
Encryption protocols, including ssh, generally do not permit packet
fragmentation to avoid introducing a point of insecurity.  This
means that the maximum packet size must not exceed the smallest
MTU (Maximum Transmission Unit) set in the routers along the path.
This smallest MTU is determined by a procedure during call set-up
which relies on the transmission over the path of ICMP packets.
If any of the routers in the path block ICMP packets this mechanism
fails, resulting in the possibility of packets being transmitted
which exceed the MTU of one of the routers.  If this happens the
packet is discarded, causing the ssh session to hang, timeout or
terminate with the error message

@example
ssh: connect to host <host ip addr> port 22: Bad file number
fatal: The remote end hung up unexpectedly
@end example

depending on precisely when in the proceedings the first large
packet is transmitted.  Most routers on the internet have MTU
set to 1500, but routers installed in homes to connect via
broadband may use a slightly smaller MTU for efficient transmission
over ATM.  If this problem is encountered a possible work-around is
to set the MTU in the local router to 1500.


@node Pushing to staging
@subsection Pushing to staging

Do not push directly to the git @code{master} branch.  Instead,
push to @code{staging}.

You will not see your patch on @code{origin/master} until some
automatic tests have been run.  These tests are run every couple
of hours; please wait at least 12 hours before wondering if your
patch has been lost.  Note that you can check the commits on
@code{origin/staging} by looking at the git web interface on
savannah.

It may happen occasionally that the staging branch breaks automated
testing.  In this case the automatic move of staging material to
master gets halted in order to avoid broken material entering master.
This is a safety net.  Please do not try breaking out from it by
adding fixes on top of staging: in that case the whole sequence will
end up in master after all, defeating the purpose of the system.  The
proper fix usually involves rewriting the staging branch and is best
left to core developers after discussion on the developer list.

Before pushing to staging it is a good practice to check whether
staging is ahead of master, and if so, wait until master has caught up
with staging before pushing.  This simplifies things if changes to
staging have to be backed out for some reason.  To check whether
master has caught up with staging you can look at the git web interface
on savannah, or do:

@example
git fetch
gitk
@end example

and check that @code{origin/master} is at the same commit as
@code{origin/staging}.  Another option is to see if any commits are
listed when you do:

@example
git fetch
git log origin/master..origin/staging
@end example

@subsubheading If your work is in a patch file

Assuming that your patch is in a file called
@file{0001-my-patch.patch} (see @ref{Patches}), and you are currently
on git master, do:

@example
git checkout staging
git pull -r
git am 0001-my-patch.patch
gitk
git push origin staging
git checkout master
@end example

@warning{Do not skip the @command{gitk} step; a quick 5-second
check of the visual history can save a great deal of frustration
later on.  You should only see that @command{staging} is only 1
commit ahead of @code{origin/staging}.}

@subsubheading If your work is in a branch

If you are working on branches and your work is in
@code{my_branch_name}, then do:

@example
git checkout my_branch_name
git pull -r origin staging
@end example

This will rebase your branch on @code{origin/staging}.  At this point
git will let you know if there are any conflicts.  If so, resolve them
before continuing:

@example
gitk
git push origin HEAD:staging
@end example

@warning{Do not skip the @command{gitk} step; a quick 5-second
check of the visual history can save a great deal of frustration
later on.  You should see that @code{my_branch_name} is only ahead of
@code{origin/staging} by the commits from your branch.}



@node Git on Windows
@section Git on Windows

@warning{We heavily recommend that development be done with our
virtual machine @ref{LilyDev}.}

@c Some of this may duplicate stuff in other sections
@c But it is probably best for windows users to have it all together
@c If necessary, clear this up later  -td

TODO: Decide what to do with this...  Pare it down?  Move
paragraphs next to analogous Unix instructions? -mp

@subsection Background to nomenclature

Git is a system for tracking the changes made to source files by a
distributed set of editors.  It is designed to work without a
master repository, but we have chosen to have a master repository
for LilyPond files.  Editors hold a local copy of the master
repository together with any changes they have made locally.
Local changes are held in a local @q{branch}, of which there may
be several, but these instructions assume you are using just one.
The files visible in the local repository always correspond to
those on the currently @q{checked out} local branch.

Files are edited on a local branch, and in that state the changes
are said to be @q{unstaged}.  When editing is complete, the
changes are moved to being @q{staged for commit}, and finally the
changes are @q{committed} to the local branch.  Once committed,
the changes (called a @q{commit}) are given a unique 40-digit
hexadecimal reference number called the @q{Committish} or @q{SHA1
ID} which identifies the commit to Git.  Such committed changes
can be sent to the master repository by @q{pushing} them (if you
have write permission) or by sending them by email to someone who
has, either as a complete file or as a @q{diff} or @q{patch}
(which send just the differences from the master repository).

@subsection Installing git

Obtain Git from
@uref{https://git-for-windows.github.io/}.

Note that most users will not need to install SSH.  That is not
required until you have been granted direct push permissions to
the master git repository.

Start Git by clicking on the desktop icon.  This will bring up a
command line bash shell.  This may be unfamiliar to Windows users.
If so, follow these instructions carefully.  Commands are entered
at a $ prompt and are terminated by keying a newline.

@subsection Initialising Git

Decide where you wish to place your local Git repository, creating
the folders in Windows as necessary.  Here we call the folder to
contain the repository @code{[path]/Git}, but if you intend using
Git for other projects a directory name like @code{lilypond-git}
might be better.  You will need to have space for around
100Mbytes.

Start the Git bash shell by clicking on the desk-top icon
installed with Git and type

@example
cd [path]/Git
@end example

to position the shell at your new Git repository.

Note: if [path] contains folders with names containing spaces use

@example
cd "[path]/Git"
@end example

Then type

@example
git init
@end example

to initialize your Git repository.

Then type (all on one line; the shell will wrap automatically)

@example
git remote add -ft master origin git://git.sv.gnu.org/lilypond.git
@end example

to download the lilypond master files.

@warning{Be patient!  Even on a broadband connection this can take
10 minutes or more.  Wait for lots of [new tag] messages and the $
prompt.}

We now need to generate a local copy of the downloaded files in a
new local branch.  Your local branch needs to have a name.  It is
usual to call it @q{master} and we shall do that here.

To do this, type

@example
git checkout -b master origin/master
@end example

This creates a second branch called @q{master}.  You will see two
warnings (ignore these), and a message advising you that your
local branch @q{master} has been set up to track the remote
branch.  You now have two branches, a local branch called
@q{master}, and a tracking branch called @q{origin/master}, which
is a shortened form of @q{remotes/origin/master}.

Return to Windows Explorer and look in your Git repository.  You
should see lots of folders.  For example, the LilyPond
documentation can be found in [path]/Git/Documentation/.

The Git bash shell is terminated by typing @code{exit} or by
clicking on the usual Windows close-window widget.

@subsection Git GUI

Almost all subsequent work will use the Git Graphical User
Interface, which avoids having to type command line commands.  To
start Git GUI first start the Git bash shell by clicking on the
desktop icon, and type

@example
cd [path]/Git
git gui
@end example

The Git GUI will open in a new window.  It contains four panels
and 7 pull-down menus.  At this stage do not use any of the
commands under Branch, Commit, Merge or Remote.  These will be
explained later.

The top panel on the left contains the names of files which you
are in the process of editing (Unstaged Changes), and the lower
panel on the left contains the names of files you have finished
editing and have staged ready for committing (Staged Changes).  At
present, these panels will be empty as you have not yet made any
changes to any file.  After a file has been edited and saved the
top panel on the right will display the differences between the
edited file selected in one of the panels on the left and the last
version committed on the current branch.

The panel at bottom right is used to enter a descriptive message
about the change before committing it.

The Git GUI is terminated by entering CNTL-Q while it is the
active window or by clicking on the usual Windows close-window
widget.

@subsection Personalising your local git repository

Open the Git GUI, click on

@example
Edit -> Options
@end example

and enter your name and email address in the left-hand (Git
Repository) panel.  Leave everything else unchanged and save it.

Note that Windows users must leave the default setting for line
endings unchanged.  All files in a git repository must have lines
terminated by just a LF, as this is required for Merge to work,
but Windows files are terminated by CRLF by default.  The git
default setting causes the line endings of files in a Windows git
repository to be flipped automatically between LF and CRLF as
required.  This enables files to be edited by any Windows editor
without causing problems in the git repository.

@subsection Checking out a branch

At this stage you have two branches in your local repository,
both identical.  To see them click on

@example
Branch -> Checkout
@end example

You should have one local branch called @q{master} and one
tracking branch called @q{origin/master}.  The latter is your
local copy of the @q{remotes/origin/master} branch in the master
LilyPond repository.  The local @q{master} branch is where you
will make your local changes.

When a particular branch is selected, i.e., checked out, the files
visible in your repository are changed to reflect the state of the
files on that branch.

@subsection Updating files from @q{remote/origin/master}

Before starting the editing of a file, ensure your local
repository contains the latest version of the files in the remote
repository by first clicking

@example
Remote -> Fetch from -> origin
@end example

@noindent
in the Git GUI.

This will place the latest version of every file, including all
the changes made by others, into the @q{origin/master} branch of
the tracking branches in your git repository.  You can see these
files by checking out this branch, but you must @emph{never} edit
any files while this branch is checked out.  Check out your local
@q{master} branch again.

You then need to merge these fetched files into your local
@q{master} branch by clicking on

@example
Merge -> Local Merge
@end example

@noindent
and if necessary select the local @q{master} branch.

Note that a merge cannot be completed if you have made any local
changes which have not yet been committed.

This merge will update all the files in the @q{master} branch to
reflect the current state of the @q{origin/master} branch.  If any
of the changes conflict with changes you have made yourself
recently you will be notified of the conflict (see below).

@subsection Editing files

First ensure your @q{master} branch is checked out, then simply
edit the files in your local Git repository with your favourite
editor and save them back there.  If any file contains non-ASCII
characters ensure you save it in UTF-8 format.  Git will detect
any changes whenever you restart Git GUI and the file names will
then be listed in the Unstaged Changes panel.  Or you can click
the Rescan button to refresh the panel contents at any time.  You
may break off and resume editing any time.

The changes you have made may be displayed in diff form in the top
right-hand panel of Git GUI by clicking on the file name shown in
one of the left panels.

When your editing is complete, move the files from being Unstaged
to Staged by clicking the document symbol to the left of each
name.  If you change your mind it can be moved back by clicking on
the ticked box to the left of the name.

Finally the changes you have made may be committed to your
@q{master} branch by entering a brief message in the Commit
Message box and clicking the Commit button.

If you wish to amend your changes after a commit has been made,
the original version and the changes you made in that commit may
be recovered by selecting

@example
Commit -> Amend Last Commit
@end example

@noindent
or by checking the Amend Last Commit radio button at bottom right.
This will return the changes to the Staged state, so further
editing made be carried out within that commit.  This must only be
done @emph{before} the changes have been Pushed or sent to your
mentor for Pushing - after that it is too late and corrections
have to be made as a separate commit.


@subsection Sending changes to @q{remotes/origin/master}

If you do not have write access to @q{remotes/origin/master} you
will need to send your changes by email to someone who does.

First you need to create a diff or patch file containing your
changes.  To create this, the file must first be committed.  Then
terminate the Git GUI.  In the git bash shell first cd to your Git
repository with

@example
cd [path]/Git
@end example

if necessary, then produce the patch with

@example
git format-patch origin
@end example

This will create a patch file for all the locally committed files
which differ from @q{origin/master}.  The patch file can be found
in [path]/Git and will have a name formed from the commit message.

@subsection Resolving merge conflicts

As soon as you have committed a changed file your local
@code{master} branch has diverged from @code{origin/master}, and
will remain diverged until your changes have been committed in
@code{remotes/origin/master} and Fetched back into your
@code{origin/master} branch.  Similarly, if a new commit has been
made to @code{remotes/origin/master} by someone else and Fetched,
your local @code{master} branch is divergent.  You can detect a
divergent branch by clicking on

@example
Repository -> Visualise all branch history
@end example

This opens up a very useful new window called @q{gitk}.  Use this
to browse all the commits made by yourself and others.

If the diagram at top left of the resulting window does not show
your @code{master} tag on the same node as the
@code{remotes/origin/master} tag your branch has diverged from
@code{origin/master}.  This is quite normal if files you have
modified yourself have not yet been Pushed to
@code{remotes/origin/master} and Fetched, or if files modified and
committed by others have been Fetched since you last Merged
@code{origin/master} into your local @code{master} branch.

If a file being merged from @code{origin/master} differs from one
you have modified in a way that cannot be resolved automatically
by git, Merge will report a Conflict which you must resolve by
editing the file to create the version you wish to keep.

This could happen if the person updating
@code{remotes/origin/master} for you has added some changes of his
own before committing your changes to
@code{remotes/origin/master}, or if someone else has changed the
same file since you last fetched the file from
@code{remotes/origin/master}.

Open the file in your editor and look for sections which are
delimited with ...

[to be completed when I next have a merge conflict to be sure I
give the right instructions  -td]


@subsection Other actions

The instructions above describe the simplest way of using git on
Windows.  Other git facilities which may usefully supplement these
include

@itemize
@item Using multiple local branches (Create, Rename, Delete)
@item Resetting branches
@item Cherry-picking commits
@item Pushing commits to @w{remote/origin/master}
@item Using gitk to review history
@end itemize

Once familiarity with using git on Windows has been gained the
standard git manuals can be used to learn about these.


@node Repository directory structure
@section Repository directory structure


@c TODO: integrate the roadmap better
@verbatiminclude ROADMAP


@node Other Git documentation
@section Other Git documentation

@itemize
@item
Official git man pages:
@uref{http://www.kernel.org/pub/software/scm/git/docs/}

@item
More in-depth tutorials: @uref{http://git-scm.com/documentation}

@item
Book about git: @uref{http://progit.org/,Pro Git}

@item
Github help: @uref{http://help.github.com/}
(very highly recommended by Graham)

@end itemize