Update CG with website build and bug squad

[lilypond.git] / Documentation / contributor / build-notes.itexi

@c -*- coding: utf-8; mode: texinfo; -*-


@node Build system notes
@chapter Build system notes

@warning{This chapter is in high flux, and is being run in a
@qq{wiki-like} fashion.  Do not trust anything you read in this
chapter.}

@menu
* Build system overview::
* Tips for working on the build system::
* General build system notes::
* Doc build::
* Website build::
* Building an Ubuntu distro::
@end menu


@node Build system overview
@section Build system overview

Build system is currently GNU make, with an extra "stepmake" layer
on top.  Look at files in @file{make/} and @file{stepmake/} and
all @file{GNUmakefile}s.

There is wide-spread dissatisfaction with this system, and we are
considering changing.  This would be a huge undertaking (estimated
200+ hours).  This change will probably involve not using GNU make
any more -- but a discussion about the precise build system will
have to wait.  Before we reach that point, we need to figure out
(at least approximately) what the current build system does.

Fundamentally, a build system does two things:

@enumerate
@item
Constructs command-line commands, for example:

@example
lilypond-book \
  --tons --of --options \
  pitches.itely
texi2pdf \
  --more --imperial --and --metric --tons --of --options \
  pitches.texi
@end example

@item
If there was a previous build, it decides which parts of the
system need to be rebuilt.

@end enumerate

When I try to do anything in the build system, it helps to remind
myself of this.  The "end result" is just a series of command-line
commands.  All the black magick is just an attempt to construct
those commands.

@node Tips for working on the build system
@section Tips for working on the build system

@itemize
@item
Add:

@example
echo "aaa"

echo "bbb"
@end example

to the build system files in various places.  This will let you
track where the program is, in various points of the build.

PH note.  There are lots of places where Make doesn't let you put
echo commands.  My top tip for tracing how make runs is to put

@example
$(error Some Text to display)
@end example

This will stop make running and print the text @code{Some Text to
display}.

End PH note.

@item
First task: understand how @code{make website} works,
@emph{without} the translations.  Looking at the english-only
website is the best introduction to the build system... it only
covers about 5% of the whole thing, but even that will likely take
10 hours or more.

@end itemize


@node General build system notes
@section General build system notes

@menu
* How stepmake works::
@end menu

@node How stepmake works
@subsection How stepmake works

Typing make website runs the file @file{GNUmakefile} from the
build directory.  This only contains 3 lines:

@example
depth = .
include config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/GNUmakefile.in
@end example

The variable @code{depth} is used throughout the make system to
track how far down the directory structure the make is.  The first
include sets lots of variables but doesn't "do" anything.  The
second runs the file @file{GNUmakefile.in} from the top level
source directory.

This sets another load of variables, and then includes (i.e.
immediately runs) @file{stepmake.make} from the @file{make}
subdirectory.  This sets a load of other variables, does some
testing to see if SCONS (another build tool?) is being used, and
then runs @file{make/config.make} - which doesn't seem to exist...

GP: scons is indeed a different build tool; I think that Jan
experimented with it 5 years ago or something.  It seems like we
still have bits and pieces of it floating around.

Next, it runs @file{make/toplevel-version.make}, which sets the
version variables for major, minor, patch, stable, development and
mypatchlevel (which seems to be used for patch numbers for
non-stable versions only?).

Next - @file{make/local.make}, which doesn't exist.

Then a few more variable and the interesting comment:

@example
# Don't try to outsmart us, you puny computer!
# Well, UGH.  This only removes builtin rules from
@end example

and then tests to see whether BUILTINS_REMOVED is defined.  It
appears to be when I run make, and so
@file{stepmake/stepmake/no-builtin-rules.make} is run.  The
comment at the head of this file says:

@example
# UGH.  GNU make comes with implicit rules.
# We don't want any of them, and can't force users to run
# --no-builtin-rules
@end example

I've not studied that file at length, but assume it removes all
make's build-in rules (e.g. @file{*.c} files are run through the
GNU C compiler) - there's a lot of them in here, and a lot of
comments, and I'd guess most of it isn't needed.

We return to @file{stepmake.make}, where we hit the make rule all:
The first line of this is:

@smallexample
-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end smallexample

which, when the variables are substituted, gives:

@example
./make/generic-inclusions.make
./make/lilypond-inclusions.make.
@end example

(Note - according to the make documentation, -include is only
different from include in that it doesn't produce any kind of
error message when the included file doesn't exist).

And the first file doesn't exist.  Nor the second.  Next:

@smallexample
-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
@end smallexample

which expands to the following files:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
@end example

One little feature to notice here - these are all absolute file
locations - the line prior to this used relative locations.  And
none of these files exist, either.  (Further note - I'm assuming
all these lines of make I'm following are autogenerated, but
that'll be something else to discover.)

Next in @file{stepmake.make}:

@smallexample
include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
@end smallexample

which expands to:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
/home/phil/lilypond-git/stepmake/stepmake/po-vars.make
/home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
@end example

Woo.  They all exist (they should as there's no - in front of the
include).  @file{generic-vars.make} sets loads of variables
(funnily enough).  @file{toplevel-vars.make} is very short - one
line commented as @code{# override Generic_vars.make:} and 2 as
follows:

@example
# urg?
include $(stepdir)/documentation-vars.make
@end example

I assume the urg comment refers to the fact that this should
really just create more variables, but it actually sends us off to
@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.

That file is a 3 line variable setting one.

@file{po-vars.make} has the one-line comment @code{# empty}, as
does @file{install-vars.make}.

So now we're back to @file{stepmake.make}.

The next lines are
:
@smallexample
# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end smallexample

and the include expands to:

@example
include ./make/generic-vars.make ./make/lilypond-vars.make.
@end example

These again set variables, and in some cases export them to allow
child @code{make} processes to use them.

The final 4 lines of @file{stepmake.make} are:

@smallexample
include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
@end smallexample

which expand as follows:

@example
include ./make/generic-rules.make ./make/lilypond-rules.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
include ./make/generic-targets.make ./make/lilypond-targets.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
@end example

@file{lilypond-rules.make} is @code{#empty}

@file{generic-rules.make} does seem to have 2 rules in it.  They
are:

@example
$(outdir)/%.ly: %.lym4
        $(M4) $< | sed "s/\`/,/g" > $@

$(outdir)/%: %.in
        rm -f $@
        cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
@end example

I believe the first rule is for *.ly files, and has a prerequisite
that *.lym4 files must be built first.  The recipe is @code{m4  |
sed "s/\`/,/g" >}.  Perhaps someone with more Unix/make knowledge
can comment on exactly what the rules mean/do.

@file{toplevel-rules.make} is @code{#empty}

@file{po-rules.make} is @code{#empty}

@file{install-rules.make} is @code{#empty}

@file{generic-targets.make} contains 2 lines of comments.

@file{lilypond-targets.make} contains only:

@example
## TODO: fail dist or web if no \version present.
check-version:
        grep -L version $(LY_FILES)
@end example

@file{stepmake/generic-targets.make} contains lots of rules - too
many to list here - it seems to be the main file for rules. (FWIW
I haven't actually found a rule for website: anywhere, although
it clearly exists.  I have also found that you can display a rule
in the terminal by typing, say @code{make -n website}.  This is
probably common knowledge.

@file{stepmake/toplevel-targets.make} adds a load of other (and
occasionally the same) rules to the gernric-targets.

@file{stepmake/po-targets.make} is rules for po* makes.

@file{stepmake/install-targets.make} has rules for local-install*.

And that's the end of stepmake.make.  Back to
@file{GNUmakefile.in}.

A bit more info from 27 March.  I've put some error traces into
@code{GNUmakefile} in the build directory, and it looks like the
following lines actually cause the make to run (putting an error
call above them - no make; below them - make):

@smallexample
ifeq ($(out),www)
# All web targets, except info image symlinks and info docs are
# installed in non-recursing target from TOP-SRC-DIR
install-WWW:
        -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
        rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
        $(MAKE) -C Documentation omf-local-install
@end smallexample

I don't currently understand the @code{ifeq}, since @code{$(out)}
is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
translates to:

@example
-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
  -c -m 755 -d /usr/local/share/doc/lilypond/html
@end example

End of work for Sunday 27th.

Another alterative approach to understanding the website build
would be to redirect @code{make -n website} and @code{make website}
to a text file and work through a) what it does and b) where the
errors are occurring.

GP: wow, all the above is much more complicated than I've ever
looked at stuff -- I tend to do a "back first" approach (where I
begin from the command-line that I want to modify, figure out
where it's generated, and then figure out how to change the
generated command-line), rather than a "front first" (where you
begin from the "make" command).


@node Doc build
@section Doc build

@menu
* The function of make doc::
* Building a bibliography::
@end menu

@node The function of make doc
@subsection The function of make doc

The following is a set of notes on how make doc functions.

Preliminary question to be answered some time: where do all the
GNUmakefiles come from.  They're in the build directory, but this
is not part of source.  Must be the configure script.  And it
looks like this comes from autogen.sh.  Must at some point kill
the whole git directory, repull and see what is created when.

Anyway, here's how make doc progresses:

This is the build dependency tree from
@file{stepmake/stepmake/generic-targets.make}:

@example
doc: doc-stage-1
  doc-stage-1:
        $(MAKE) -C $(depth)/scripts/build out=
        $(MAKE) out=www WWW-1
          WWW-1: local-WWW-1
        $(LOOP)
          $(MAKE) out=www WWW-2
          WWW-2: local-WWW-2
            $(LOOP)
          $(MAKE) out=www WWW-post
@end example

@example
MAKE = make --no-builtin-rules
-C = Change to directory before make
@end example

doc-stage-1 does lots of opening and looking in files, but no
processing.

@example
Variable LOOP =

+ make PACKAGE=LILYPOND package=lilypond -C python
&& make PACKAGE=LILYPOND package=lilypond -C scripts
&&  make PACKAGE=LILYPOND package=lilypond -C flower
&&  make PACKAGE=LILYPOND package=lilypond -C lily
&&  make PACKAGE=LILYPOND package=lilypond -C mf
&&  make PACKAGE=LILYPOND package=lilypond -C ly
&&  make PACKAGE=LILYPOND package=lilypond -C tex
&&  make PACKAGE=LILYPOND package=lilypond -C ps
&&  make PACKAGE=LILYPOND package=lilypond -C scm
&&  make PACKAGE=LILYPOND package=lilypond -C po
&&  make PACKAGE=LILYPOND package=lilypond -C make
&&  make PACKAGE=LILYPOND package=lilypond -C elisp
&&  make PACKAGE=LILYPOND package=lilypond -C vim
&&  make PACKAGE=LILYPOND package=lilypond -C input
&&  make PACKAGE=LILYPOND package=lilypond -C stepmake
&&  make PACKAGE=LILYPOND package=lilypond -C Documentation
&& true
@end example

From git grep:

stepmake/stepmake/generic-vars.make has this:

@smallexample
LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true
@end smallexample

$@ is the name of the target - WWW-1 in this case.

In GNUmakefile.in we find:

@example
SUBDIRS = python scripts \
        flower lily \
        mf ly \
        tex ps scm \
        po make \
        elisp vim \
        input \
        stepmake $(documentation-dir)
@end example

So that's how we get the main make loop...

That loop expands like this:

@example
make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
true
@end example

The directories up to and including vim produce no effect with
make in non-debug mode, although debug does show lots of action.

@file{git/build/input/GNUmakefile} is:

@example
depth=../
include $(depth)/config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/./input/GNUmakefile
MODULE_INCLUDES += $(src-dir)/$(outbase)
@end example

The first include is:

@example
..//config.make
@end example

(note the // which is strictly wrong)

which has lots of variables to set, but no action occurs.

The second is:

@example
lilypond-git/./input/GNUmakefile
@end example

which similarly doesn't create any actual action.

An error message at the end of build/input/GNUmakefile stops
make processing before it moves on to regression - so where does
that come from?

And the answer is - make processes all directories in the
directory it's entered (with some exceptions like out and out-www)
and so it changes to /regression.

It then seems to consider whether it needs to make/remake loads of
makefiles.  Don't understand this yet.  Possibly these are all the
makefiles it's processing, and it always checks they're up to date
before processing other files?

Could be correct - some of this output is:

@example
Must remake target `../../make/ly-inclusions.make'.
Failed to remake target file `../../make/ly-inclusions.make'.
@end example

Having decided that, it then leaves the directory and re-executes:

@example
make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
@end example

The top of this make is:

@example
This program built for i486-pc-linux-gnu
Reading makefiles...
Reading makefile `GNUmakefile'...
Reading makefile `../..//config.make' (search path) (no ~ expansion)...
@end example

which looks like it's re-reading all its known makefiles to check
they're up to date.

(From the make manual:

To this end, after reading in all makefiles, make will consider each as a goal target and
attempt to update it. If a makefile has a rule which says how to update it (found either
in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10
[Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have
been checked, if any have actually been changed, make starts with a clean slate and reads
all the makefiles over again. (It will also attempt to update each of them over again, but
normally this will not change them again, since they are already up to date.)

So my assumption seems correct)

There appear to be about 74 of them.  After all the makefile
checking, we get this:

@smallexample
Updating goal targets....
Considering target file `WWW-1'.
File `WWW-1' does not exist.
Considering target file `local-WWW-1'.
File `local-WWW-1' does not exist.
Considering target file `out-www/collated-files.texi'.
File `out-www/collated-files.texi' does not exist.
Looking for an implicit rule for `out-www/collated-files.texi'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `collated-files.tely'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `out-www/collated-files.tely'.
Trying rule prerequisite `out-www/version.itexi'.
Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
@end smallexample

grep finds this if searching for local-WWW-1:

@example
make/lysdoc-targets.make:
  local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
@end example

which means that local-WWW-1 depends on coll*.texi and coll*.pdf
and so these will need to be checked to see if they're up to date.
So make needs to find rules for both of those and (as it says) it
certainly needs to make coll*.texi, since it doesn't exist.

In ly-rules.make we have:

@example
.SUFFIXES: .doc .tely .texi .ly
@end example

which I'll work out at some point, and also this rule:

@smallexample
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
@end smallexample

Note that the recipe is a very long line - it could probably
benefit from splitting.  The same makefile also has:

@smallexample
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
@end smallexample

@noindent
which seems to be an almost exact duplicate.  Whatever, the first
one is executed first.  Have not checked if the second executes.

The first recipe translates as this:

@example
LILYPOND_VERSION=2.15.0 /usr/bin/python   --process=' ' \
  --output=./out-www --format= --lily-output-dir \
    /home/phil/lilypond-git/build/out/lybook-db
@end example

@noindent
if we stop the build with an $(error), but I think this is because
we need to allow it to process the dependencies first.  It looks
like foo.texi is shown as being dependent on foo.tely, plus a load
of other files.

@example
DOCUMENTATION_LOCALE_TARGET is blank
INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
  /home/phil/lilypond-git/scm/autochange.scm
@end example

plus 10s (100s?) of other .scm files.

@example
SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
  /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
@end example

ditto .ly files.  This does seem a teency bit wrong - it looks like
the .ly and .scm files have been interchanged.  ly-vars.make has
these 2 lines:

@example
INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
@end example

Looks like a bug.....

So it now works its way through all these files, checking if they
need to be remade.  This is 100s of lines of the debug listing,
although none in the normal list.  Clearly none has to be made
since they're source files.  It concludes:

@example
Must remake target `out-www/collated-files.tely'
@end example

@file{lysdoc-rules.make} has this:

@smallexample
$(outdir)/collated-files.tely: $(COLLATED_FILES)
        $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
@end smallexample

@file{lysdoc-vars.make} has:

@example
COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
@end example

We find that:

@example
TEXINFO_SOURCES = AAA-intro-regression.tely
OUT_LY_FILES is empty
@end example

so LY_FILES has the big long list of all the .ly files in the
regression directory.

This kicks off

@example
/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
@end example

with a list of all the files in the regression test directory. This
should (I believe) create the file collated-files.tely.

So the next rule in make is for @file{version.itexi}, and make duly
checks this.  There's a rule in @file{doc-i18n-root-rules.make} that this
depends on @file{git/VERSION}:

@smallexample
$(outdir)/version.%: $(top-src-dir)/VERSION
        $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
@end smallexample

This causes create-version-itexi.py to run and create
version.itexi.

Once that's done, all the other *.scm and *.ly files are checked
and since they have no rules associated, they aren't remade (just
as well for source files, really).  Since version.itexi was remade
make concludes that collated-files.texi must be remade.  To do
this, it runs lilypond-book.py on collated-files.tely, as below:

@example
LILYPOND_VERSION=2.15.0
  /usr/bin/python
  /home/phil/lilypond-git/scripts/lilypond-book.py
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    --process='/home/phil/lilypond-git/build/out/bin/lilypond
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www
    -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    -dbackend=eps
    --formats=ps,png,pdf
    -dinclude-eps-fonts
    -dgs-load-fonts
    --header=doctitle
    --header=doctitlecs
    --header=doctitlede
    --header=doctitlees
    --header=doctitlefr
    --header=doctitlehu
    --header=doctitleit
    --header=doctitleja
    --header=doctitlenl
    --header=doctitlezh
    --header=texidoc
    --header=texidoccs
    --header=texidocde
    --header=texidoces
    --header=texidocfr
    --header=texidochu
    --header=texidocit
    --header=texidocja
    --header=texidocnl
    --header=texidoczh
    -dcheck-internal-types
    -ddump-signatures
    -danti-alias-factor=2'
    --output=./out-www
    --format=texi-html
    --verbose
    --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
    out-www/collated-files.tely
@end example

So - lilypond-book runs on:

@example
input/regression/out-www/collated-files.tely
@end example


Note the --verbose flag - this is from the make variable
LILYPOND_BOOK_VERBOSE which is added to the make variable
LILYPOND_BOOK_FLAGS.

Now found the invocation to write some of the image files.  It's
like this:

@example
/home/phil/lilypond-git/build/out/bin/lilypond
  -I /home/phil/lilypond-git/input/regression/
  -I ./out-www -I /home/phil/lilypond-git/input
  -I /home/phil/lilypond-git/Documentation
  -I /home/phil/lilypond-git/Documentation/snippets
  -I /home/phil/lilypond-git/input/regression/
  -I /home/phil/lilypond-git/Documentation/included/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/Documentation/pictures
  -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
  -dbackend=eps
  --formats=ps,png,pdf
  -dinclude-eps-fonts
  -dgs-load-fonts
  --header=doctitle
  --header=doctitlecs
  --header=doctitlede
  --header=doctitlees
  --header=doctitlefr
  --header=doctitlehu
  --header=doctitleit
  --header=doctitleja
  --header=doctitlenl
  --header=doctitlezh
  --header=texidoc
  --header=texidoccs
  --header=texidocde
  --header=texidoces
  --header=texidocfr
  --header=texidochu
  --header=texidocit
  --header=texidocja
  --header=texidocnl
  --header=texidoczh
  -dcheck-internal-types
  -ddump-signatures
  -danti-alias-factor=2
  -I  "/home/phil/lilypond-git/build/out/lybook-db"
  -I  "/home/phil/lilypond-git/build/input/regression"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/build/input/regression/out-www"
  -I  "/home/phil/lilypond-git/input"
  -I  "/home/phil/lilypond-git/Documentation"
  -I  "/home/phil/lilypond-git/Documentation/snippets"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/Documentation/included"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/Documentation/pictures"
  -I  "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
  --formats=eps
  --verbose
  -deps-box-padding=3.000000
  -dread-file-list
  -dno-strip-output-dir
  "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
@end example

Note the --verbose.  This causes 100s of lines of Lily debug output.
But at present I can't work out where the flag comes from. Later.


@node Building a bibliography
@subsection Building a bibliography

Bibliography files contain a list of citations, like this:

@example
@@Book@{vinci,
  author = @{Vinci, Albert C.@},
  title = @{Fundamentals of Traditional Music Notation@},
  publisher = @{Kent State University Press@},
  year = @{1989@}
@}
@end example

There are a variety of types of citation (e.g. Book (as above),
article, publication).  Each cited publication has a list of
entries that can be used to identify the publication.
Bibliograpies are normally stored as files with a .bib
extension.  One part of the doc-build process is transforming the
bibliography information into @code{texinfo} files.  The commands
to do this are in the @file{GNUmakefile} in the
@file{Documentation} directory.

A typical line of the makefile to translate a single bibliography
is:

@example
$(outdir)/colorado.itexi:
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
                -s $(top-src-dir)/Documentation/lily-bib \
                -o $(outdir)/colorado.itexi \
                $(src-dir)/essay/colorado.bib
@end example

Line by line:

@example
$(outdir)/colorado.itexi:
@end example

We're making the file @file{colorado.itexi} and so this is the
make instruction.

@example
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
@end example

It's in the @file{essay} directory and we want to run the
bib2texi.py script against it.

@example
                -s $(top-src-dir)/Documentation/lily-bib \
@end example

The style template is @file{lily-bib.bst} and is found in the
@file{Documentation} directory.

@example
                -o $(outdir)/colorado.itexi \
@end example

The output file in @file{colorado.itexi}.

@example
                $(src-dir)/essay/colorado.bib
@end example

The input file is @file{colorado.bib} in the @file{essay}
directory.

The @code{bib2texi} Python script used to be used with a variety
of options, but now is always called using the same options, as
above.  Its job is to create the file containing the options for
@code{bibtex} (the program that actually does the translation),
run bibtex, and then clean up some temporary files.  Its main
"value add" is the creation of the options file, using this code:

@example
open (tmpfile + '.aux', 'w').write (r'''
\relax
\citation@{*@}
\bibstyle@{%(style)s@}
\bibdata@{%(files)s@}''' % vars ())
@end example

The key items are the style file (now always lily-bib for us) and
the input file.

The style file is written in its own specialised language,
described to some extent at

@example
@uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
@end example

The file @file{lily-bib.bst} also has fairly extensive commenting.

@node Website build
@section Website build

@warning{This information applies only to the standard @code{make
website} from the normal build directory.  The process is
different for @code{dev/website-build}.}

The rule for make website is found in GNUmakefile.in:

@example
website:
$(MAKE) config_make=$(config_make) \
        top-src-dir=$(top-src-dir) \
        -f $(top-src-dir)/make/website.make \
        website
@end example

This translates as:

@example
make --no-builtin-rules config_make=./config.make \
                top-src-dir=/home/phil/lilypond-git \
                -f /home/phil/lilypond-git/make/website.make \
                website
@end example

which has the effect of setting the variables @code{config_make}
and @code{top-src-dir} and then processing the file
@code{git/make/website.make} with the target of website.

@code{website.make} starts with the following:

@example
ifeq (@$(WEBSITE_ONLY_BUILD),1)
@end example

which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
was set to one on the command line.  This is only done for
standalone website builds, not in the normal case.  The result of
the test determines the value of some variables that are set.  A
number of other variables are set, in order to establish locations
of various files.  An example is:

@example
CREATE_VERSION=python @$(script-dir)/create-version-itexi.py
@end example

The rule for website is:

@smallexample
website: website-texinfo website-css website-pictures website-examples web-post
        cp @$(SERVER_FILES)/favicon.ico @$(OUT)/website
        cp @$(SERVER_FILES)/robots.txt @$(OUT)/website
        cp @$(top-htaccess) @$(OUT)/.htaccess
        cp @$(dir-htaccess) @$(OUT)/website/.htaccess
@end smallexample

so we see that this starts by running the rules for 5 other
targets, then finishes by copying some files.  We'll cover that
later - first @code{website-texinfo}.  That rule is:

@example
website-texinfo: website-version website-xrefs website-bibs
        for l in '' @$(WEB_LANGS); do \
                if test -n "@$@$l"; then \
                        langopt=--lang="@$@$l"; \
                        langsuf=.@$@$l; \
                fi; \
                @$(TEXI2HTML) --prefix=index \
                        --split=section \
                        --I=@$(top-src-dir)/Documentation/"@$@$l" \
                        --I=@$(top-src-dir)/Documentation \
                        --I=@$(OUT) \
                        @$@$langopt \
                        --init-file=@$(texi2html-init-file) \
                        -D web_version \
                        --output=@$(OUT)/"@$@$l" \
                        @$(top-src-dir)/Documentation/"@$@$l"/web.texi ; \
                ls @$(OUT)/@$@$l/*.html | xargs grep -L \
                        'UNTRANSLATED NODE: IGNORE ME' | \
                        sed 's!@$(OUT)/'@$@$l'/!!g' | xargs \
                        @$(MASS_LINK) --prepend-suffix="@$@$langsuf" \
                        hard @$(OUT)/@$@$l/ @$(OUT)/website/ ; \
        done
@end example

which therefore depends on @code{website-version},
@code{website-xrefs} and @code{website-bibs}.

@example
website-version:
        mkdir -p @$(OUT)
        @$(CREATE_VERSION) @$(top-src-dir) > @$(OUT)/version.itexi
        @$(CREATE_WEBLINKS) @$(top-src-dir) > @$(OUT)/weblinks.itexi
@end example

which translates as:

@example
mkdir -p out-website
python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
        /home/phil/lilypond-git > out-website/version.itexi
python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
        /home/phil/lilypond-git > out-website/weblinks.itexi
@end example

So, we make out-website then send the output of
@code{create-version-itexi.py} to @code{out-website/version.itexi}
and @code{create-weblinks-itexi.py} to
@code{out-website/weblinks.itexi}.

@code{create-version-itexi.py} parses the file @code{VERSION} in
the top source dir.  It contains:

@example
PACKAGE_NAME=LilyPond
MAJOR_VERSION=2
MINOR_VERSION=15
PATCH_LEVEL=13
MY_PATCH_LEVEL=
VERSION_STABLE=2.14.2
VERSION_DEVEL=2.15.12
@end example

currently.  @code{c-v-i.py} parses this to:

@example
@@c ************************ Version numbers ************
@@macro version
2.15.13
@@end macro

@@macro versionStable
2.14.2
@@end macro

@@macro versionDevel
2.15.12
@@end macro
@end example

@code{create-weblinks-itexi.py} creates a load of texi macros (of
the order of 1000) similar to:

@example
@@macro manualStableGlossaryPdf
@@uref{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf}
@@end macro.
@end example

It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:

@code{langdefs.py: warning: lilypond-doc gettext domain not found.}

Next:

@example
website-xrefs: website-version
        for l in '' @$(WEB_LANGS); do \
@end example

is the start of the rule, truncated for brevity.  This loops
through the languages to be used on the website, processing some
variables which I don't fully understand, to run this command:

@smallexample
python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
        -I /home/phil/lilypond-git/Documentation \
        -I /home/phil/lilypond-git/Documentation/"@$l" \
        -I out-website -o out-website --split=node \
        --known-missing-files= \
                 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
        -q \
        /home/phil/lilypond-git/Documentation/"@$l"/web.texi ;\
@end smallexample

There's a good description of what
@code{extract_texi_filenames.py} does at the top of the script,
but a shortened version is:

@code{If this script is run on a file texifile.texi, it produces
a file texifile[.LANG].xref-map with tab-separated entries
of the form NODE\tFILENAME\tANCHOR.}

An example from
@code{web.nl.xref-map} is:

@example
Inleiding        Introduction        Introduction
@end example

@code{e-t-f.py} follows the includes from document to document.
We know some have not been created yet, and
@code{known-missing-files} option tells @code{e-t-f.py} which
these are.

It then does this:

@example
for m in @$(MANUALS); do \
@end example

to run @code{e-t-f.py} against all of the manuals, in each
language. Next:

@example
website-bibs: website-version
        BSTINPUTS=@$(top-src-dir)/Documentation/web \
                @$(WEB_BIBS) -s web \
                -s @$(top-src-dir)/Documentation/lily-bib \
                -o @$(OUT)/others-did.itexi \
                @$(quiet-flag) \
                @$(top-src-dir)/Documentation/web/others-did.bib
@end example

This is half the command.  It runs @code{bib2texi.py} on 2
@code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
This converts bibliography files into texi files with
@code{bibtex}.

Next the commands in the @code{website-texinfo} rule are run:

@example
for l in '' @$(WEB_LANGS); do \
@end example

run @code{texi2html}.  This is the program that outputs the
progress message (found in
@code{Documentation/lilypond-texi2html.init}):

@code{Processing web site: []}

It also outputs warning messages like:

@code{WARNING: Unable to find node 'Řešení potíží' in book usage.}

@example
website-css:
        cp @$(top-src-dir)/Documentation/css/*.css @$(OUT)/website
@end example

Copies 3 css files to out-website/website.  Then:

@example
website-pictures:
        mkdir -p @$(OUT)/website/pictures
        if [ -d @$(PICTURES) ]; \
        then \
                cp @$(PICTURES)/* @$(OUT)/website/pictures ; \
                ln -sf website/pictures @$(OUT)/pictures  ;\
        fi
@end example

which translates as:

@smallexample
if [ -d Documentation/pictures/out-www ]; \
    then \
        cp Documentation/pictures/out-www/* out-website/website/pictures ; \
        ln -sf website/pictures out-website/pictures  ;\
    fi
@end smallexample

i.e. it copies the contents of
@code{build/Documentation/pictures/out-www/*} to
@code{out-website/website/pictures}.  Unfortunately, the pictures
are only created once @code{make doc} has been run, so an initial
run of @code{make website} copies nothing, and the pictures on the
website (e.g. the logo) do not exist.  Next:

@example
website-examples:
        mkdir -p @$(OUT)/website/ly-examples
        if [ -d @$(EXAMPLES) ]; \
        then \
                cp @$(EXAMPLES)/* @$(OUT)/website/ly-examples ; \
        fi
@end example

translates to:

@smallexample
mkdir -p out-website/website/ly-examples
if [ -d Documentation/web/ly-examples/out-www ]; \
    then \
        cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
    fi
@end smallexample

This does the same with the LilyPond examples (found at
@uref{http://lilypond.org/examples.html}).  Again, these are
actually only created by @code{make doc} (and since they are
generated from LilyPond source files, require a working LilyPond
@code{exe} made with @code{make}).  So this does nothing
initially.  Then:

@example
web-post:
        @$(WEB_POST) @$(OUT)/website
@end example

which is:

@smallexample
python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
@end smallexample

which describes itself as:

@code{This is web_post.py. This script deals with translations
in the "make website" target.}

It also does a number of other things, including adding the Google
tracker code and the language selection footer.  We're now at
the end of our story.  The final 4 lines of the recipe for website
are:

@example
cp @$(SERVER_FILES)/favicon.ico @$(OUT)/website
cp @$(SERVER_FILES)/robots.txt @$(OUT)/website
cp @$(top-htaccess) @$(OUT)/.htaccess
cp @$(dir-htaccess) @$(OUT)/website/.htaccess
@end example

The first translates as:

@smallexample
cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
@end smallexample

so we see these are just copying the support files for the web
server.

@subsubheading website.make summary

Recipes in @file{website.make}:

@itemize

@item
@code{website:}
this is the "master" rule.  It calls the other rules in order,
then copies some extra files around - see below for further
of the process it produces.

@item
@code{website-version}:
this calls the python scripts below:
@itemize
@item
@example
scripts/build/create-version-itexi.py
@end example

This writes a @@version, @@versionStable, and @@versionDevel based
on the top-level VERSIONS file, to
@code{out-website/version.itexi}

@item
@example
scripts/build/create-weblinks-itexi.py
@end example

This creates a ton of macros in @code{out-website/weblinks.itexi}.
Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.

It's quite monstrous because it deals with combinations of
stable/devel, source/docs, lang/lang/lang*10, etc.

@end itemize

@item
@code{website-xrefs:}
creates files used for complicated "out-of-build" references to
@code{out-website/*.xref-map}

If you just write @@ref@{@}, then all's groovy and we wouldn't
need this.  But if you write @@rlearning@{@}, then our custom
texi2html init file needs to know about our custom xref file
format, which tells our custom texi2html init file how to create
the link.

GP: we should have a separate @@node to discuss xrefs.  Also, take a
quick look at a generated xref file -- it's basically just a list
of @@node's [sic teenager pluralization rule] from the file.

@item
@code{website-bib:}
generates the bibliography texinfo files from the .bib files - in
the case of the website build these are @file{others-did.bib} and
@file{we-wrote.bib}.

@item
@code{website-texinfo:}
this is the main part; it calles texi2html to generate the actual
html.  It also has a ton of options to texi2html to pass info to
our custom init file.

The file actually built is called @file{web.texi}, and is either
in the @file{Documentation} directory, or a sub-directory specific
to the language.

The options file is @file{/Documentation/lilypond-texi2html.init}.
This contains *lots* of option and configuration stuff, and also
includes the line:

@smallexample
print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
@end smallexample

This is where one of the console messages is generated.

We have somewhere between 2-4 different ways "to pass info to our
custom init file".  This is highly Not Good (tm), but that's how
things work at the moment.

After texi2html, it does some black magick to deal with
untranslated nodes in the translations.  Despite writing that
part, I can't remember how it works.  But in theory, you could
figure it out by copy&pasting each part of the command (by "part",
I mean "stuff before each | pipe"), substituting the variables,
then looking at the text that's output.  For example,

@example
  ls $(OUT)/$$l/*.html
@end example

is going to print a list of all html files, in all languages, in
the build directory.  Then more stuff happens to each of those
files (that's what xargs does).

@item
@code{website-css:}
just copies files to the build dir.

@item
@code{website-pictures, website-examples:}
more file copies, with an if statement to handle if you don't have
any generated pictures/examples.

@item
@code{web-post:}
runs:

@example
scripts/build/website_post.py
@end example

which, it adds the "this page is translated in klingon" to the
bottom of html pages, and adds the google analytics javascript.
It also has hard-coded lilypond version numbers, which is Bad
(tm).

@end itemize

Here's a summary of what gets called, in what order, when we run
@code{make website}

@example
website:
  website-texinfo:
    website-version:
      creates version.itexi and weblinks.itexi
    website-xrefs:
      runs extract_texi_filenames.py
    website-bibs:
      creates bibliography files, described above
  website-css:
    copies css files
  website-pictures:
    copies pictures
  website-examples:
    copies examples
  web-post:
    runs website_post.py
  Then some file copying
@end example

@node Building an Ubuntu distro
@section Building an Ubuntu distro


Here's the short instruction on how to create lilybuntu iso image
(Jonathan Kulp did this on a spare drive,
but he supposes it can be done in a VM too):

@enumerate

@item
Install ubuntu, reboot.
@item
Run all updates, reboot if asked.
@item
Enable src repos, refresh package lists.
@item
Install LilyPond build deps:
@example
sudo apt-get build-dep lilypond
@end example
@item
Install git and autoconf:
@example
sudo apt-get install git-core gitk autoconf
@end example

@item
Test to see whether everything works fine now:
@enumerate
@item
use @command{lily-git.tcl} to grab source files
@item
go to source dir and do
@example
"./autogen.sh" ; make ; make doc
@end example
@item
if all compiles, move on to iso creation...
@end enumerate

@item
Download & install "remastersys":
@uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
@item
Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
@item
Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
default live session username, etc).
@item
Remove non-essential desktop software as desired.
@item
Create iso:
@example
sudo remastersys dist
@end example
New iso is in @file{/home/remastersys/remastersys/}.
@item
Test iso by installing in VM and repeating steps above for
getting source files and building lp and docs.

@end enumerate