Imported Upstream version 2.14.2

[lilypond.git] / Documentation / contributor / build-notes.itexi

@c -*- coding: utf-8; mode: texinfo; -*-


@node Build system notes
@chapter Build system notes

@warning{This chapter is in high flux, and is being run in a
@qq{wiki-like} fashion.  Do not trust anything you read in this
chapter.}

@menu
* Build system overview::
* Tips for working on the build system::
* General build system notes::
* Doc build::
* Website build::
* Building an Ubuntu distro::
@end menu


@node Build system overview
@section Build system overview

Build system is currently GNU make, with an extra "stepmake" layer
on top.  Look at files in @file{make/} and @file{stepmake/} and
all @file{GNUmakefile}s.

There is wide-spread dissatisfaction with this system, and we are
considering changing.  This would be a huge undertaking (estimated
200+ hours).  This change will probably involve not using GNU make
any more -- but a discussion about the precise build system will
have to wait.  Before we reach that point, we need to figure out
(at least approximately) what the current build system does.

Fundamentally, a build system does two things:

@enumerate
@item
Constructs command-line commands, for example:

@example
lilypond-book \
  --tons --of --options \
  pitches.itely
texi2pdf \
  --more --imperial --and --metric --tons --of --options \
  pitches.texi
@end example

@item
If there was a previous build, it decides which parts of the
system need to be rebuilt.

@end enumerate

When I try to do anything in the build system, it helps to remind
myself of this.  The "end result" is just a series of command-line
commands.  All the black magick is just an attempt to construct
those commands.

@node Tips for working on the build system
@section Tips for working on the build system

@itemize
@item
Add:

@example
echo "aaa"

echo "bbb"
@end example

to the build system files in various places.  This will let you
track where the program is, in various points of the build.

PH note.  There are lots of places where Make doesn't let you put
echo commands.  My top tip for tracing how make runs is to put

@example
$(error Some Text to display)
@end example

This will stop make running and print the text @code{Some Text to
display}.

End PH note.

@item
First task: understand how @code{make website} works,
@emph{without} the translations.  Looking at the english-only
website is the best introduction to the build system... it only
covers about 5% of the whole thing, but even that will likely take
10 hours or more.

@end itemize


@node General build system notes
@section General build system notes

@menu
* How stepmake works::
@end menu

@node How stepmake works
@subsection How stepmake works

Typing make website runs the file @file{GNUmakefile} from the
build directory.  This only contains 3 lines:

@example
depth = .
include config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/GNUmakefile.in
@end example

The variable @code{depth} is used throughout the make system to
track how far down the directory structure the make is.  The first
include sets lots of variables but doesn't "do" anything.  The
second runs the file @file{GNUmakefile.in} from the top level
source directory.

This sets another load of variables, and then includes (i.e.
immediately runs) @file{stepmake.make} from the @file{make}
subdirectory.  This sets a load of other variables, does some
testing to see if SCONS (another build tool?) is being used, and
then runs @file{make/config.make} - which doesn't seem to exist...

GP: scons is indeed a different build tool; I think that Jan
experimented with it 5 years ago or something.  It seems like we
still have bits and pieces of it floating around.

Next, it runs @file{make/toplevel-version.make}, which sets the
version variables for major, minor, patch, stable, development and
mypatchlevel (which seems to be used for patch numbers for
non-stable versions only?).

Next - @file{make/local.make}, which doesn't exist.

Then a few more variable and the interesting comment:

@example
# Don't try to outsmart us, you puny computer!
# Well, UGH.  This only removes builtin rules from
@end example

and then tests to see whether BUILTINS_REMOVED is defined.  It
appears to be when I run make, and so
@file{stepmake/stepmake/no-builtin-rules.make} is run.  The
comment at the head of this file says:

@example
# UGH.  GNU make comes with implicit rules.
# We don't want any of them, and can't force users to run
# --no-builtin-rules
@end example

I've not studied that file at length, but assume it removes all
make's build-in rules (e.g. @file{*.c} files are run through the
GNU C compiler) - there's a lot of them in here, and a lot of
comments, and I'd guess most of it isn't needed.

We return to @file{stepmake.make}, where we hit the make rule all:
The first line of this is:

@example
-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end example

which, when the variables are substituted, gives:

@example
./make/generic-inclusions.make
./make/lilypond-inclusions.make.
@end example

(Note - according to the make documentation, -include is only
different from include in that it doesn't produce any kind of
error message when the included file doesn't exist).

And the first file doesn't exist.  Nor the second.  Next:

@example
-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
@end example

which expands to the following files:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
@end example

One little feature to notice here - these are all absolute file
locations - the line prior to this used relative locations.  And
none of these files exist, either.  (Further note - I'm assuming
all these lines of make I'm following are autogenerated, but
that'll be something else to discover.)

Next in @file{stepmake.make}:

@example
include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
@end example

which expands to:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
/home/phil/lilypond-git/stepmake/stepmake/po-vars.make
/home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
@end example

Woo.  They all exist (they should as there's no - in front of the
include).  @file{generic-vars.make} sets loads of variables
(funnily enough).  @file{toplevel-vars.make} is very short - one
line commented as @code{# override Generic_vars.make:} and 2 as
follows:

@example
# urg?
include $(stepdir)/documentation-vars.make
@end example

I assume the urg comment refers to the fact that this should
really just create more variables, but it actually sends us off to
@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.

That file is a 3 line variable setting one.

@file{po-vars.make} has the one-line comment @code{# empty}, as
does @file{install-vars.make}.

So now we're back to @file{stepmake.make}.

The next lines are
:
@example
# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end example

and the include expands to:

@example
include ./make/generic-vars.make ./make/lilypond-vars.make.
@end example

These again set variables, and in some cases export them to allow
child @code{make} processes to use them.

The final 4 lines of @file{stepmake.make} are:

@example
include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
@end example

which expand as follows:

@example
include ./make/generic-rules.make ./make/lilypond-rules.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
include ./make/generic-targets.make ./make/lilypond-targets.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
@end example

@file{lilypond-rules.make} is @code{#empty}

@file{generic-rules.make} does seem to have 2 rules in it.  They
are:

@example
$(outdir)/%.ly: %.lym4
        $(M4) $< | sed "s/\`/,/g" > $@

$(outdir)/%: %.in
        rm -f $@
        cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
@end example

I believe the first rule is for *.ly files, and has a prerequisite
that *.lym4 files must be built first.  The recipe is @code{m4  |
sed "s/\`/,/g" >}.  Perhaps someone with more Unix/make knowledge
can comment on exactly what the rules mean/do.

@file{toplevel-rules.make} is @code{#empty}

@file{po-rules.make} is @code{#empty}

@file{install-rules.make} is @code{#empty}

@file{generic-targets.make} contains 2 lines of comments.

@file{lilypond-targets.make} contains only:

@example
## TODO: fail dist or web if no \version present.
check-version:
        grep -L version $(LY_FILES)
@end example

@file{stepmake/generic-targets.make} contains lots of rules - too
many to list here - it seems to be the main file for rules. (FWIW
I haven't actually found a rule for website: anywhere, although
it clearly exists.  I have also found that you can display a rule
in the terminal by typing, say @code{make -n website}.  This is
probably common knowledge.

@file{stepmake/toplevel-targets.make} adds a load of other (and
occasionally the same) rules to the gernric-targets.

@file{stepmake/po-targets.make} is rules for po* makes.

@file{stepmake/install-targets.make} has rules for local-install*.

And that's the end of stepmake.make.  Back to
@file{GNUmakefile.in}.

A bit more info from 27 March.  I've put some error traces into
@code{GNUmakefile} in the build directory, and it looks like the
following lines actually cause the make to run (putting an error
call above them - no make; below them - make):

@example
ifeq ($(out),www)
# All web targets, except info image symlinks and info docs are
# installed in non-recursing target from TOP-SRC-DIR
install-WWW:
        -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
        rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
        $(MAKE) -C Documentation omf-local-install
@end example

I don't currently understand the @code{ifeq}, since @code{$(out)}
is empty at this point, but the line starting @code{-$(INSTALL)}
translates to:

@example
-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py -c -m 755 -d /usr/local/share/doc/lilypond/html
@end example

End of work for Sunday 27th.

Another alterative approach to understanding the website build
would be to redirect @code{make -n website} and @code{make website}
to a text file and work through a) what it does and b) where the
errors are occurring.

GP: wow, all the above is much more complicated than I've ever
looked at stuff -- I tend to do a "back first" approach (where I
begin from the command-line that I want to modify, figure out
where it's generated, and then figure out how to change the
generated command-line), rather than a "front first" (where you
begin from the "make" command).


@node Doc build
@section Doc build

@menu
* Building a bibliography::
@end menu

@node Building a bibliography
@subsection Building a bibliography

Bibliography files contain a list of citations, like this:

@example
@@Book@{vinci,
  author = @{Vinci, Albert C.@},
  title = @{Fundamentals of Traditional Music Notation@},
  publisher = @{Kent State University Press@},
  year = @{1989@}
@}
@end example

There are a variety of types of citation (e.g. Book (as above),
article, publication).  Each cited publication has a list of
entries that can be used to identify the publication.
Bibliograpies are normally stored as files with a .bib
extension.  One part of the doc-build process is transforming the
bibliography information into @code{texinfo} files.  The commands
to do this are in the @file{GNUmakefile} in the
@file{Documentation} directory.

A typical line of the makefile to translate a single bibliography
is:

@example
$(outdir)/colorado.itexi:
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
                -s $(top-src-dir)/Documentation/lily-bib \
                -o $(outdir)/colorado.itexi \
                $(src-dir)/essay/colorado.bib
@end example

Line by line:

@example
$(outdir)/colorado.itexi:
@end example

We're making the file @file{colorado.itexi} and so this is the
make instruction.

@example
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
@end example

It's in the @file{essay} directory and we want to run the
bib2texi.py script against it.

@example
                -s $(top-src-dir)/Documentation/lily-bib \
@end example

The style template is @file{lily-bib.bst} and is found in the
@file{Documentation} directory.

@example
                -o $(outdir)/colorado.itexi \
@end example

The output file in @file{colorado.itexi}.

@example
                $(src-dir)/essay/colorado.bib
@end example

The input file is @file{colorado.bib} in the @file{essay}
directory.

The @code{bib2texi} Python script used to be used with a variety
of options, but now is always called using the same options, as
above.  Its job is to create the file containing the options for
@code{bibtex} (the program that actually does the translation),
run bibtex, and then clean up some temporary files.  Its main
"value add" is the creation of the options file, using this code:

@example
open (tmpfile + '.aux', 'w').write (r'''
\relax
\citation@{*@}
\bibstyle@{%(style)s@}
\bibdata@{%(files)s@}''' % vars ())
@end example

The key items are the style file (now always lily-bib for us) and
the input file.

The style file is written in its own specialised language,
described to some extent at

@example
@uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
@end example

The file @file{lily-bib.bst} also has fairly extensive commenting.

@node Website build
@section Website build

Start here: @file{make/website.make}

The overall build system begins with @ref{How stepmake works}.
Summary: when you type @code{make website} this ends up running
@file{GNUmakefile.in} in the @file{git} directory.  Right at the
bottom, this has the lines:

@example
# we want this separate for security; see CG 4.2.  -gp
website:
        $(MAKE) config_make=$(config_make) \
                top-src-dir=$(top-src-dir) \
                -f $(top-src-dir)/make/website.make \
                website
@end example

On my system this expands to:

@example
make --no-builtin-rules config_make=./config.make \
                top-src-dir=/home/phil/lilypond-git \
                -f /home/phil/lilypond-git/make/website.make \
                website
@end example

We see that the @code{$(MAKE)} expands to
@code{make --no-builtin-rules} which is how @code{MAKE} is
defined higher up the makefile.  The -f switch defines the
makefile to be used - in this case
@file{git/make/website.make}.  That's where all the action
happens.

We believe that note that *none* of the variables that
are loaded (from depth to version numbers to whatever) are used in
@file{website.make}.  Instead, @file{website.make} sets up its own
variables at the top of the file.  If you're wondering if there's
some smart reason for this, then the answer is "no".  It's because
I (GP) didn't know/trust the original variables when I was writing
that file.

Website build includes @ref{Building a bibliography}.

@subsubheading Output from @code{make -n website}

Sorry, including this output directly produces problems in the
build system.  Please run:

@example
make -n website &> my-file.txt
@end example

to see the full output from the make.

@subsubheading website.make variables

The file begins by setting up some variables.  These
may/might/probably mirror existing variables, but lacking any docs
about those variables, I thought it would be simpler to keep
everything in the same file.

Note that for security reasons, we @strong{don't} call scripts in
the git dir when building on the web server.  See @ref{Uploading
and security}.  So we definitely want to keep those definitions
for the WEBSITE_ONLY_BUILD.

After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
there's another bunch of lines setting up generic variables.

@subsubheading website.make building parts

Parts of @file{website.make}:

@itemize

@item
@code{website:}
this is the "master" rule.  It calls the other rules in order,
then copies some extra files around - see below for further
of the process it produces.

@item
@code{website-version}:
this calls the python scripts below:

@itemize
@item
@example
scripts/build/create-version-itexi.py
@end example

This writes a @@version, @@versionStable, and @@versionDevel based
on the top-level VERSIONS file, to
@code{out-website/version.itexi}

@item
@example
scripts/build/create-weblinks-itexi.py
@end example

This creates a ton of macros in @code{out-website/weblinks.itexi}.
Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.

It's quite monstrous because it deals with combinations of
stable/devel, source/docs, lang/lang/lang*10, etc.

@end itemize


@item
@code{website-xrefs:}
creates files used for complicated "out-of-build" references to
@code{out-website/*.xref-map}

If you just write @@ref@{@}, then all's groovy and we wouldn't
need this.  But if you write @@rlearning@{@}, then our custom
texi2html init file needs to know about our custom xref file
format, which tells our custom texi2html init file how to create
the link.

GP: we should have a separate @@node to discuss xrefs.  Also, take a
quick look at a generated xref file -- it's basically just a list
of @@node's [sic teenager pluralization rule] from the file.

@item
@code{website-bib:}
generates the bibliography texinfo files from the .bib files - in
the case of the website build these are @file{others-did.bib} and
@file{we-wrote.bib}.

@item
@code{website-texinfo:}
this is the main part; it calles texi2html to generate the actual
html.  It also has a ton of options to texi2html to pass info to
our custom init file.

We have somewhere between 2-4 different ways "to pass info to our
custom init file".  This is highly Not Good (tm), but that's how
things work at the moment.

After texi2html, it does some black magick to deal with
untranslated nodes in the translations.  Despite writing that
part, I can't remember how it works.  But in theory, you could
figure it out by copy&pasting each part of the command (by "part",
I mean "stuff before each | pipe"), substituting the variables,
then looking at the text that's output.  For example,

@example
  ls $(OUT)/$$l/*.html
@end example

is going to print a list of all html files, in all languages, in
the build directory.  Then more stuff happens to each of those
files (that's what xargs does).

@item
@code{website-css:}
just copies files to the build dir.

@item
@code{website-pictures, website-examples:}
more file copies, with an if statement to handle if you don't have
any generated pictures/examples.

@item
@code{web-post:}
runs:

@example
scripts/build/website_post.py
@end example

which, it adds the "this page is translated in klingon" to the
bottom of html pages, and adds the google analytics javascript.
It also has hard-coded lilypond version numbers, which is Bad
(tm).

@end itemize

Here's a summary of what gets called, in what order, when we run
@code{make website}

@example
website:
  website-texinfo:
    website-version:
      creates version.itexi and weblinks.itexi
    website-xrefs:
      runs extract_texi_filenames.py
    website-bibs:
      creates bibliography files, described above
  website-css:
    copies css files
  website-pictures:
    copies pictures
  website-examples:
    copies examples
  web-post:
    runs website_post.py
  Then some file copying
@end example

@node Building an Ubuntu distro
@section Building an Ubuntu distro

@enumerate
@item
Install ubuntu, reboot
@item
Run all updates, reboot if asked
@item
Enable src repos, refresh package lists
@item
Install LilyPond build deps:
@example
  sudo apt-get build-dep lilypond
@end example
@item
Install git and autoconf:
@example
  sudo apt-get install git-core gitk autoconf
@end example

@item
TEST TO SEE WHETHER EVERYTHING WORKS NOW:
@enumerate
@item
Use lily-git.tcl to grab source files
@item
Go to source dir and do "./autogen.sh" ; make ; make doc
@item
If all compiles, move on to iso creation...

@end enumerate

@item
Download & install "remastersys":
@example
  http://sourceforge.net/projects/remastersys/
@end example
@item
Copy lily-git.tcl script file into /etc/skel/
@item
Modify /etc/remastersys.conf as desired (change .iso name, default
live session username, etc)
@item
Remove non-essential desktop software as desired
@item
Create iso:  sudo remastersys dist
@item
New iso is in /home/remastersys/remastersys/
@item
Test iso by installing in VM and repeating steps above for
getting source files and building lp and docs
@end enumerate