New upstream version 2.19.80

[lilypond.git] / Documentation / contributor / release-work.itexi

@c -*- coding: utf-8; mode: texinfo; -*-
@node Release work
@chapter Release work

@menu
* Development phases::
* Minor release checklist::
* Major release checklist::
* Release extra notes::
* Notes on builds with GUB::
@end menu


@node Development phases
@section Development phases

There are 2 states of development on @code{master}:

@enumerate

@item @strong{Normal development}:
Any commits are fine.

@item @strong{Build-frozen}:
Do not require any additional or updated libraries or make
non-trivial changes to the build process.  Any such patch (or
branch) may not be merged with master during this period.

This should occur approximately 1 month before any alpha version
of the next stable release, and ends when the next unstable branch
begins.

@end enumerate


After announcing a beta release, branch @code{stable/2.x}.  There
are 2 states of development for this branch:

@enumerate
@item @strong{Normal maintenance}:
The following patches @strong{MAY NOT} be merged with this branch:

@itemize
@item Any change to the input syntax.  If a file compiled with a
previous @code{2.x} (beta) version, then it must compile in the
new version.

Exception: any bugfix to a Critical issue.

@item New features with new syntax @emph{may be committed},
although once committed that syntax cannot change during the
remainder of the stable phase.

@item Any change to the build dependencies (including programming
libraries, documentation process programs, or python modules used
in the buildscripts).  If a contributor could compile a previous
lilypond @code{2.x}, then he must be able to compile the new
version.

@end itemize

@item @strong{Release prep}:
Only translation updates and important bugfixes are allowed.

@end enumerate


@node Minor release checklist
@section Minor release checklist

A @qq{minor release} means an update of @code{y} in @code{2.x.y}.

@subheading Pre-release

@enumerate

@item
Don't forget to prepare the GUB build machine by deleting and moving unneeded
files: see @qq{Subsequent builds} in @ref{Notes on builds with GUB}.

@item
Using any system with git pull access (not necessarily the GUB
build machine), use the commands below to do the following:

@itemize
@item
switch to the release branch

@item
update the release branch from origin/master

@item
update the translation files

@item
create the release announcement

@item
update the build versions.

@itemize
@item
VERSION_DEVEL = the current development version (previous VERSION_DEVEL + 0.01)

@item
VERSION_STABLE = the current stable version (probably no change here)

@end itemize

@item
update the @qq{Welcome to LilyPond} version numbers to the version about to be
released

@end itemize

This requires a system which has the release/unstable
branch.  If you get a warning saying you are in @code{detached HEAD}
state, then you should create a release/unstable branch with
@code{git checkout release/unstable}.

Check the environment variables are set as in
@ref{Environment variables}.

You need to ensure you have a clean index and work tree.  If the
checkout displays modified files, you might want to run
@code{git reset --hard} before continuing.

@example
git fetch
git checkout release/unstable
git merge origin/master
make -C $LILYPOND_BUILD_DIR po-replace
mv $LILYPOND_BUILD_DIR/po/lilypond.pot po/
gedit Documentation/web/news-new.itexi Documentation/web/news-old.itexi
gedit Documentation/web/news-headlines.itexi
gedit VERSION
gedit ly/Wel*.ly
@end example

Editing the @file{news-headlines.itexi} file is a bit tricky, since it contains
URLs with escaped characters.  An example of what is needed is that releasing
@code{2.19.50} after the release of @code{2.19.49} needed the line:

@example
@@uref@{news.html#LilyPond-2_002e19_002e49-released-October-16_002c-2016,
  LilyPond 2.19.49 released - @@emph@{October 16, 2016@}@}
@end example

to be changed to:

@example
@@uref@{news.html#LilyPond-2_002e19_002e50-released-November-6_002c-2016,
  LilyPond 2.19.50 released - @@emph@{November 6, 2016@}@}
@end example

Don't forget to update the entry above that line to show the latest release
version.

@item
Commit, push, switch back to master (or wherever else):

@example
git commit -m "Release: bump VERSION_DEVEL." VERSION
git commit -m "PO: update template." po/lilypond.pot
git commit -m "Release: update news." Documentation/web/
git commit -m "Release: bump Welcome versions." ly/Wel*.ly
git push origin HEAD:release/unstable
git checkout master
@end example

@item
If you do not have the previous release test-output tarball, download
it and put it in @code{regtests/}

@item Prepare GUB environment by running:

@example
### my-gub.sh
# special terminal, and default PATH environment.
# import these special environment vars:
#   HOME, HTTP_PROXY, TERM
env -i \
  HOME=$HOME \
  HTTP_PROXY=$HTTP_PROXY \
  bash --rcfile my-bashrc
@end example

@example
### my-bashrc
export PS1="\[\e[1;33mGUB-ENV \w\]$ \[\e[0m\]"
export PATH=$PATH
export TERM=xterm
@end example


@item Build release on GUB by running:

@example
make LILYPOND_BRANCH=release/unstable lilypond
@end example

@noindent
or something like:

@example
make LILYPOND_BRANCH=stable/2.16 lilypond
@end example

@item
Check the regtest comparison in @file{uploads/webtest/} for
any unintentional breakage.  More info in
@ref{Precompiled regression tests}.

@item
If any work was done on GUB since the last release, upload
binaries to a temporary location, ask for feedback, and wait a day
or two in case there's any major problems.

@warning{Always do this for a stable release.}

@end enumerate


@subheading Actual release

@enumerate

@item If you're not the right user on the webserver, remove the
@code{t} from the rsync command in:

@example
test-lily/rsync-lily-doc.py
test-lily/rsync-test.py
@end example

@item Upload GUB by running:

@example
make lilypond-upload \
  LILYPOND_REPO_URL=git://git.sv.gnu.org/lilypond.git \
  LILYPOND_BRANCH=release/unstable
@end example

@noindent
or something like:

@example
make lilypond-upload \
  LILYPOND_REPO_URL=git://git.sv.gnu.org/lilypond.git \
  LILYPOND_BRANCH=stable/2.12
@end example

@end enumerate


@subheading Post release

@enumerate

@item
Update the current staging branch with the current news:

@example
git fetch
git checkout origin/staging
git merge origin/release/unstable
@end example

@item
Update @file{VERSION} in lilypond git and upload changes:

@example
gedit VERSION
@end example

@itemize
@item
VERSION = what you just did +0.0.1
@end itemize

@example
git commit -m "Release: bump VERSION." VERSION
git push origin HEAD:staging
@end example

If the push fails with a message like

@example
 ! [rejected]        HEAD -> staging (non-fast-forward)
@end example

@noindent
it means that somebody else updated the staging branch while you were
preparing your change.  In that case, you need to restart the Post
Release process.  Otherwise, proceed:

@item Wait a few hours for the website to update.

@item Email release notice to @code{info-lilypond}

@end enumerate



@node Major release checklist
@section Major release checklist

A @qq{major release} means an update of @code{x} in @code{2.x.0}.

@subheading Main requirements

These are the current official guidelines.

@itemize
@item
0 Critical issues for two weeks (14 days) after the latest release
candidate.

@end itemize


@subheading Potential requirements

These might become official guidelines in the future.

@itemize
@item
Check reg test

@item
Check all 2ly scripts

@item
Check for emergencies the docs:

@example
grep FIXME --exclude "misc/*" --exclude "*GNUmakefile" \
  --exclude "snippets/*" ????*/*
@end example

@item
Check for altered regtests, and document as necessary:

@example
git diff -u -r release/2.@var{FIRST-CURRENT-STABLE} \
  -r release/2.@var{LAST-CURRENT-DEVELOPMENT} input/regression/
@end example

@end itemize


@subheading Housekeeping requirements

Before the release:

@itemize
@item
write release notes.  note: stringent size requirements for
various websites, so be brief.

@item
Run convert-ly on all files, bump parser minimum version.

@item
Update lilypond.pot:

@example
make -C $LILYPOND_BUILD_DIR po-replace
mv $LILYPOND_BUILD_DIR/po/lilypond.pot po/
@end example

@item
Make directories on lilypond.org:

@example
~/download/sources/v2.@var{NEW-STABLE}
~/download/sources/v2.@var{NEW-DEVELOPMENT}
@end example

@item
Shortly after the release, move all current contributors to
previous contributors in
@file{Documentation/included/authors.itexi}.

@item
Delete old material in @file{Documentation/changes.tely}, but
don't forget to check it still compiles!  Also update the version
numbers:

@example
@@node Top
@@top New features in 2.@var{NEW-STABLE} since 2.@var{OLD-STABLE}
@end example

@item
Website:

@itemize
@item
make a link from the old unstable to the next stable in
lilypond.org's @file{/doc/} dir.  Keep all previous unstable->stable
doc symlinks.

Also, make the old docs self-contained -- if there's a redirect in
@file{/doc/v2.@var{OLD-STABLE}/Documentation/index.html} , replace it with the
@file{index.html.old-2.@var{OLD-STABLE}} files.

The post-2.13 docs will need another way of handling the
self-containment.  It won't be hard to whip up a python script
that changes the link to @file{../../../../manuals.html} to
@file{../website/manuals.html}, but it's still a 30-minute task that
needs to be done before 2.16.

@item
doc auto redirects to @code{v2.@var{NEW-STABLE}}

@item
add these two lines to @file{Documentation/web/server/robots.txt}:

@example
Disallow: /doc/v2.@var{OLD-STABLE}/
Disallow: /doc/v2.@var{NEW-DEVELOPMENT}/
@end example

@end itemize

@end itemize

@subheading Unsorted

@itemize
@item
submit po template for translation: send url of tarball to
@email{coordinator@@translationproject.org}, mentioning
lilypond-VERSION.pot

@item
update links to distros providing lilypond packages?  link in:
@file{Documentation/web/download.itexi}

This has nothing to do with the release, but it's a @qq{periodic
maintenance} task that might make sense to include with releases.

@item
Send announcements to...

News:

@example
comp.music.research
comp.os.linux.announce

comp.text.tex
rec.music.compose
@end example

Mail:

@example
info-lilypond@@gnu.org

linux-audio-announce@@lists.linuxaudio.org
linux-audio-user@@lists.linuxaudio.org
linux-audio-dev@@lists.linuxaudio.org

tex-music@@icking-music-archive.org

--- non-existant?
abcusers@@blackmill.net

rosegarden-user@@lists.sourceforge.net
info-gnu@@gnu.org
noteedit-user@@berlios.de

gmane.comp.audio.fomus.devel
gmane.linux.audio.users
gmane.linux.audio.announce
gmane.comp.audio.rosegarden.devel
@end example

Web:

@example
lilypond.org
freshmeat.net
linuxfr.com
http://www.apple.com/downloads
harmony-central.com (news@@harmony-central.com)
versiontracker.com [auto]
hitsquad.com [auto]
https://savannah.gnu.org/news/submit.php?group_id=1673  @c => planet.gnu.org
@end example

@end itemize


@node Release extra notes
@section Release extra notes

@subsubheading Regenerating regression tests

Regenerating regtests (if the lilypond-book naming has changed):

@itemize

@item
git checkout release/lilypond-X.Y.Z-A

@item
take lilypond-book and any related makefile updates from the
latest git.

@item
configure; make; make test

@item
tar -cjf lilypond-X.Y.Z-A.test-output.tar.bz2 input/regression/out-test/

@item
mv lilypond-X.Y.Z-A.test-output.tar.bz2 ../gub/regtests/

@item
cd ../gub/regtests/

@item
make lilypond

@end itemize


@subsubheading stable/2.12

If releasing stable/2.12, then:

@itemize

@item
apply doc patch: patches/rsync-lily.patch  (or something like
that)

@item
change infodir in gub/specs/lilypond-doc.py from "lilypond.info"
to "lilypond-web.info"
@end itemize

@subsubheading Updating a release (changing a in x.y.z-a)

Really tentative instructions, almost certainly can be done
better.

@enumerate

@item
change the VERSION back to release you want.  push change.
(hopefully you'll have forgotten to update it when you made your
last release)

@item
make sure that there aren't any lilypond files floating around in
target/  (like usr/bin/lilypond).

@item
build the specific package(s) you want, i.e.

@example
bin/gub mingw::lilypond-installer
make LILYPOND_BRANCH=stable/2.12 -f lilypond.make doc
bin/gub --platform=darwin-x86 \
  'git://git.sv.gnu.org/lilypond-doc.git?branch=stable/2.12'
@end example

or

build everything with the normal "make lilypond", then (maybe)
manually delete stuff you don't want to upload.

@item
manually upload them.  good luck figuring out the rsync
command(s).  Hints are in test-lily/

or

run the normal lilypond-upload command, and (maybe) manually
delete stuff you didn't want to upload from the server.

@end enumerate

@node Notes on builds with GUB
@section Notes on builds with GUB

@subsubheading Building GUB

GUB - the Grand Unified Builder - is used to build the release
versions of LilyPond.  For background information, see
@ref{Grand Unified Builder (GUB)}.  The simplest way to set up a
GUB build environment is to use a virtual machine with LilyDev
(@ref{LilyDev}).  Follow the instructions on that page to set this
up.  Make sure that your virtual machine has enough disk space -
a GUB installation takes over 30 GBytes of disk space, and if you
allocate too little, it will fail during the setting up stage and
you will have to start again.  64 GBytes should be sufficient.

While GUB is being built, any interruptions are likely to make it
almost impossible to restart.  If at all possible, leave the build
to continue uninterrupted.

Download GUB and start the set up:

@example
git clone git://github.com/gperciva/gub.git
cd gub
make bootstrap
@end example

This will take a very long time, even on a very fast computer.
You will need to be patient.  It's also liable to fail - it
downloads a number of tools, and some will have moved and others
won't respond to the network.  For example, the perl archive.
If this happens, download it from
@uref{http://www.cpan.org/src/5.0/perl-5.10.0.tar.gz}, saving the
archive to @file{gub/downloads/perl/}.  Continue the set up with:

@example
make bootstrap
@end example

Once this has completed successfully, you can build the LilyPond
release package.  However, this uses an archived version of the
regression tests, so it is better to download this first.
Download the test output from lilypond.org (you will need to
replace @code{2.15.33-1} with the latest build):

@smallexample
@uref{http://lilypond.org/downloads/binaries/test-output/lilypond-2.15.33-1.test-output.tar.bz2}
@end smallexample

Copy the tarball into @file{regtests/}, and tell the build system that
you have done this:

@example
touch regtests/ignore
@end example

Now start the GUB build:

@example
make lilypond
@end example

That's it.  This will build LilyPond from current master.  To build
the current unstable release, run:

@example
make LILYPOND_BRANCH=release/unstable lilypond
@end example

The first time you do this, it will take a very long time.

Assuming the build has gone well, it can be uploaded using:

@example
make lilypond-upload
  LILYPOND_BRANCH=release/unstable
  LILYPOND_REPO_URL=git://git.sv.gnu.org/lilypond.git
@end example

@subsubheading Output files

GUB builds the files it needs into the directory
@code{gub/target/}.  As a general rule, these don't need to be
touched unless there is a problem building GUB (see below).
The files to be uploaded are in @code{gub/uploads/}.  Once the
build has completed successfully, there should be 8
installation files and 3 archives, totalling about 600MB.
There are also 4 directories:

@example
gub/signatures
gub/localdoc
gub/webdoc
gub/webtest
@end example

@code{signatures} contains files that are used to track whether
some of the archives have already been built.  Don't touch
these.

@code{localdoc} probably contains local copies of the
documentation.

@code{webdoc} contains the documentation to be uploaded.

@code{webtest} contains the regtest comparison, which should
be checked before upload, and is also uploaded for subsequent
checking.

The total upload is about 700 MB in total, and on an ADSL
connection will take about 4 hours to upload.

@subsubheading Subsequent builds

In principle, building the next release of LilyPond requires
no action other then following the instructions in
@ref{Minor release checklist}.  Because much of the
infrastructure has already been built, it will take much less
time - about an hour on a fast computer.

Continuing to build LilyPond without any other
archiving/deletion of previous builds is likely to be successful,
but will take up a fair amount of disk space (around 2GB per
build) which may be a problem with a Virtual Machine.  It's
therefore recommended to move (not copy) @code{gub/uploads} to
another machine/disk after each build, if space is at a premium.

However, if a significant change has been made to the LilyPond
source (e.g. added source files) the build may fail if tried on
top of a previous build.  If this happens, be sure to
move/delete @code{gub/uploads} and all mentions of LilyPond
in @code{gub/target}.  The latter can be achieved with this
command:

@example
rm -rf target/*/*/*lilypond*
@end example

Be @emph{very} careful with this command.  Typing it wrongly
could wipe your disk completely.

@subsubheading Updating the web site

The @code{make lilypond-upload} command updates the documentation
on the LilyPond web site.  However, it does @emph{not} update
any part of the site that is not part of the documentation - for
example, the front page (@code{index.html}).  The website is
updated by 2 cron jobs running on the web server.  One of these
pulls git master to the web server, and the other makes the
website with the standard @code{make website} command.  They run
hourly, 30 minutes apart.  So - to update the front page of the
website, it's necessary to update @code{VERSION} and
@code{news-headlines.itexi} in master and then wait for the cron
jobs to run.  (N.B. - this is done by pushing the changes to
staging and letting patchy do its checks before it pushes to
master).