CG: move generic stepmake docs out of web build.

[lilypond.git] / Documentation / contributor / build-notes.itexi

@c -*- coding: utf-8; mode: texinfo; -*-


@node Build system notes
@chapter Build system notes

@warning{This chapter is in high flux, and is being run in a
@qq{wiki-like} fashion.  Do not trust anything you read in this
chapter.}

@menu
* Build system overview::
* Tips for working on the build system::
* General build system notes::
* Doc build::
* Website build::
* Building an Ubuntu distro::
@end menu


@node Build system overview
@section Build system overview

Build system is currently GNU make, with an extra "stepmake" layer
on top.  Look at files in @file{make/} and @file{stepmake/} and
all @file{GNUmakefile}s.

There is wide-spread dissatisfaction with this system, and we are
considering changing.  This would be a huge undertaking (estimated
200+ hours).  This change will probably involve not using GNU make
any more -- but a discussion about the precise build system will
have to wait.  Before we reach that point, we need to figure out
(at least approximately) what the current build system does.

Fundamentally, a build system does two things:

@enumerate
@item
Constructs command-line commands, for example:

@example
lilypond-book \
  --tons --of --options \
  pitches.itely
texi2pdf \
  --more --imperial --and --metric --tons --of --options \
  pitches.texi
@end example

@item
If there was a previous build, it decides which parts of the
system need to be rebuilt.

@end enumerate

When I try to do anything in the build system, it helps to remind
myself of this.  The "end result" is just a series of command-line
commands.  All the black magick is just an attempt to construct
those commands.

@node Tips for working on the build system
@section Tips for working on the build system

@itemize
@item
Add:

@example
echo "aaa"

echo "bbb"
@end example

to the build system files in various places.  This will let you
track where the program is, in various points of the build.

@item
First task: understand how @code{make website} works,
@emph{without} the translations.  Looking at the english-only
website is the best introduction to the build system... it only
covers about 5% of the whole thing, but even that will likely take
10 hours or more.

@end itemize


@node General build system notes
@section General build system notes

@menu
* How stepmake works::
@end menu

@node How stepmake works
@subsection How stepmake works

Typing make website runs the file @file{GNUmakefile} from the
build directory.  This only contains 3 lines:

@example
depth = .
include config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/GNUmakefile.in
@end example

The variable @code{depth} is used throughout the make system to
track how far down the directory structure the make is.  The first
include sets lots of variables but doesn't "do" anything.  The
second runs the file @file{GNUmakefile.in} from the top level
source directory.

This sets another load of variables, and then includes (i.e.
immediately runs) @file{stepmake.make} from the @file{make}
subdirectory.  This sets a load of other variables, does some
testing to see if SCONS (another build tool?) is being used, and
then runs @file{make/config.make} - which doesn't seem to exist...

GP: scons is indeed a different build tool; I think that Jan
experimented with it 5 years ago or something.  It seems like we
still have bits and pieces of it floating around.

Next, it runs @file{make/toplevel-version.make}, which sets the
version variables for major, minor, patch, stable, development and
mypatchlevel (which seems to be used for patch numbers for
non-stable versions only?).

Next - @file{make/local.make}, which doesn't exist.

Then a few more variable and the interesting comment:

@example
# Don't try to outsmart us, you puny computer!
# Well, UGH.  This only removes builtin rules from
@end example

and then tests to see whether BUILTINS_REMOVED is defined.  It
appears to be when I run make, and so
@file{stepmake/stepmake/no-builtin-rules.make} is run.  The
comment at the head of this file says:

@example
# UGH.  GNU make comes with implicit rules.
# We don't want any of them, and can't force users to run
# --no-builtin-rules
@end example

I've not studied that file at length, but assume it removes all
make's build-in rules (e.g. @file{*.c} files are run through the
GNU C compiler) - there's a lot of them in here, and a lot of
comments, and I'd guess most of it isn't needed.

We return to @file{stepmake.make}, where we hit the make rule all:
The first line of this is:

@example
-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end example

which, when the variables are substituted, gives:

@example
./make/generic-inclusions.make
./make/lilypond-inclusions.make.
@end example

(Note - according to the make documentation, -include is only
different from include in that it doesn't produce any kind of
error message when the included file doesn't exist).

And the first file doesn't exist.  Nor the second.  Next:

@example
-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
@end example

which expands to the following files:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
@end example

One little feature to notice here - these are all absolute file
locations - the line prior to this used relative locations.  And
none of these files exist, either.  (Further note - I'm assuming
all these lines of make I'm following are autogenerated, but
that'll be something else to discover.)

Next in @file{stepmake.make}:

@example
include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
@end example

which expands to:

@example
/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
/home/phil/lilypond-git/stepmake/stepmake/po-vars.make
/home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
@end example

Woo.  They all exist (they should as there's no - in front of the
include).  @file{generic-vars.make} sets loads of variables
(funnily enough).  @file{toplevel-vars.make} is very short - one
line commented as @code{# override Generic_vars.make:} and 2 as
follows:

@example
# urg?
include $(stepdir)/documentation-vars.make
@end example

I assume the urg comment refers to the fact that this should
really just create more variables, but it actually sends us off to
@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.

That file is a 3 line variable setting one.

@file{po-vars.make} has the one-line comment @code{# empty}, as
does @file{install-vars.make}.

So now we're back to @file{stepmake.make}.

The next lines are
:
@example
# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
@end example

and the include expands to:

@example
include ./make/generic-vars.make ./make/lilypond-vars.make.
@end example

These again set variables, and in some cases export them to allow
child @code{make} processes to use them.

The final 4 lines of @file{stepmake.make} are:

@example
include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
@end example

which expand as follows:

@example
include ./make/generic-rules.make ./make/lilypond-rules.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
  /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
include ./make/generic-targets.make ./make/lilypond-targets.make
include
  /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
  /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
@end example

@file{lilypond-rules.make} is @code{#empty}
@file{generic-rules.make} does seem to have 2 rules in it.  They
are:

@example
$(outdir)/%.ly: %.lym4
        $(M4) $< | sed "s/\`/,/g" > $@

$(outdir)/%: %.in
        rm -f $@
        cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
@end example

I believe the first rule is for *.ly files, and has a prerequisite
that *.lym4 files must be built first.  The recipe is @code{m4  |
sed "s/\`/,/g" >}.  Perhaps someone with more Unix/make knowledge
can comment on exactly what the rules mean/do.

@file{toplevel-rules.make} is @code{#empty}
@file{po-rules.make} is @code{#empty}
@file{install-rules.make} is @code{#empty}
@file{generic-targets.make} contains 2 lines of comments.
@file{lilypond-targets.make} contains only:

@example
## TODO: fail dist or web if no \version present.
check-version:
        grep -L version $(LY_FILES)
@end example

@file{stepmake/generic-targets.make} contains lots of rules - too
many to list here - it seems to be the main file for rules. (FWIW
I haven't actually found a rule for website: anywhere, although
it clearly exists.  I have also found that you can display a rule
in the terminal by typing, say @code{make -n website}.  This is
probably common knowledge.

@file{stepmake/toplevel-targets.make} adds a load of other (and
occasionally the same) rules to the gernric-targets.

@file{stepmake/po-targets.make} is rules for po* makes.

@file{stepmake/install-targets.make} has rules for local-install*.

And that's the end of stepmake.make.  Back to
@file{GNUmakefile.in}.  More some other time.

Another alterative approach to understanding the website build
would be to redirect @code{make -n website} and @code{make website}
to a text file and work through a) what it does and b) where the
errors are occurring.

GP: wow, all the above is much more complicated than I've ever
looked at stuff -- I tend to do a "back first" approach (where I
begin from the command-line that I want to modify, figure out
where it's generated, and then figure out how to change the
generated command-line), rather than a "front first" (where you
begin from the "make" command).


@node Doc build
@section Doc build

@menu
* Building a bibliography::
@end menu

@node Building a bibliography
@subsection Building a bibliography

Bibliography files contain a list of citations, like this:

@example
@@Book@{vinci,
  author = @{Vinci, Albert C.@},
  title = @{Fundamentals of Traditional Music Notation@},
  publisher = @{Kent State University Press@},
  year = @{1989@}
@}
@end example

There are a variety of types of citation (e.g. Book (as above),
article, publication).  Each cited publication has a list of
entries that can be used to identify the publication.
Bibliograpies are normally stored as files with a .bib
extension.  One part of the doc-build process is transforming the
bibliography information into @code{texinfo} files.  The commands
to do this are in the @file{GNUmakefile} in the
@file{Documentation} directory.

A typical line of the makefile to translate a single bibliography
is:

@example
$(outdir)/colorado.itexi:
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
                -s $(top-src-dir)/Documentation/lily-bib \
                -o $(outdir)/colorado.itexi \
                $(src-dir)/essay/colorado.bib
@end example

Line by line:

@example
$(outdir)/colorado.itexi:
@end example

We're making the file @file{colorado.itexi} and so this is the
make instruction.

@example
        BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
@end example

It's in the @file{essay} directory and we want to run the
bib2texi.py script against it.

@example
                -s $(top-src-dir)/Documentation/lily-bib \
@end example

The style template is @file{lily-bib.bst} and is found in the
@file{Documentation} directory.

@example
                -o $(outdir)/colorado.itexi \
@end example

The output file in @file{colorado.itexi}.

@example
                $(src-dir)/essay/colorado.bib
@end example

The input file is @file{colorado.bib} in the @file{essay}
directory.

The @code{bib2texi} Python script used to be used with a variety
of options, but now is always called using the same options, as
above.  Its job is to create the file containing the options for
@code{bibtex} (the program that actually does the translation),
run bibtex, and then clean up some temporary files.  Its main
"value add" is the creation of the options file, using this code:

@example
open (tmpfile + '.aux', 'w').write (r'''
\relax
\citation@{*@}
\bibstyle@{%(style)s@}
\bibdata@{%(files)s@}''' % vars ())
@end example

The key items are the style file (now always lily-bib for us) and
the input file.

The style file is written in its own specialised language,
described to some extent at

@example
@uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
@end example

The file @file{lily-bib.bst} also has fairly extensive commenting.

@node Website build
@section Website build

Start here: @file{make/website.make}

The overall build system begins with @ref{How stepmake works}.

However, we do believe that note that *none* of the variables that
are loaded (from depth to version numbers to whatever) are used in
@file{website.make}.  Instead, @file{website.make} sets up its own
variables at the top of the file.  If you're wondering if there's
some smart reason for this, then the answer is "no".  It's because
I didn't know/trust the original variables when I was writing that
file.


Website build includes @ref{Building a bibliography}.


@subsubheading website.make variables

The file begins by setting up some variables.  These
may/might/probably mirror existing variables, but lacking any docs
about those variables, I thought it would be simpler to keep
everything in the same file.

Note that for security reasons, we *don't* call scripts in the git
dir when building on the web server.  See @ref{Uploading and
security}.  So we definitely want to keep those definitions for
the WEBSITE_ONLY_BUILD.

After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
there's another bunch of lines setting up generic variables.

@subsubheading website.make building parts

The website-version rule creates files that define macros for
commands like @@stableDocsNotationPdf@{@}, by calling python scripts.
Those scripts are scripts/build/create-version-itexi.py and
scripts/build/create-weblinks-itexi.py.

website-xrefs: creates files used for complicated "out-of-build"
references.  If you just write @@ref@{@}, then all's groovy.  But
if you do @@rlearning@{@}, then our custom texi2html init file
needs to know about our custom xref file format, which tells our
custom texi2html init file how to create the link.

We should have a separate @@node to discuss xrefs.  Also, take a
quick look at a generated xref file -- it's basically just a list
of @@node's [sic teenager pluralization rule] from the file.

website-bib: generates the bibliography texinfo files from the
.bib files.

website-texinfo: this is the money shot; it calles texi2html to
generate the actual html.  It also has a ton of options to
texi2html to pass info to our custom init file.

After texi2html, it does some black magick to deal with
untranslated nodes in the translations.  Despite writing that
part, I can't remember how it works.  But in theory, you could
figure it out by copy&pasting each part of the command (by "part",
I mean "stuff before each | pipe"), substituting the variables,
then looking at the text that's output.  For example,
  ls $(OUT)/$$l/*.html
is going to print a list of all html files, in all languages, in
the build directory.  Then more stuff happens to each of those
files (that's what xargs does).

website-css: just copies files to the build dir.

website-pictures, website-examples: more file copies, with an if
statement to handle if you don't have any generated
pictures/examples.

web-post: buggered if I know.  Apparently it runs the
scripts/build/website_post.py  file, which despite writing, I
can't remember what it does.

ok, it adds the "this page is translated in klingon" to the bottom
of html pages, and adds the google analytics javascript.  It also
has hard-coded lilypond version numbers.

website: this is the "master" rule.  It calls the bits and pieces
in order, then copies some extra files around.


@node Building an Ubuntu distro
@section Building an Ubuntu distro

@enumerate
@item
Install ubuntu, reboot
@item
Run all updates, reboot if asked
@item
Enable src repos, refresh package lists
@item
Install LilyPond build deps:
@example
  sudo apt-get build-dep lilypond
@end example
@item
Install git and autoconf:
@example
  sudo apt-get install git-core gitk autoconf
@end example

@item
TEST TO SEE WHETHER EVERYTHING WORKS NOW:
@enumerate
@item
Use lily-git.tcl to grab source files
@item
Go to source dir and do "./autogen.sh" ; make ; make doc
@item
If all compiles, move on to iso creation...

@end enumerate

@item
Download & install "remastersys":
@example
  http://sourceforge.net/projects/remastersys/
@end example
@item
Copy lily-git.tcl script file into /etc/skel/
@item
Modify /etc/remastersys.conf as desired (change .iso name, default
live session username, etc)
@item
Remove non-essential desktop software as desired
@item
Create iso:  sudo remastersys dist
@item
New iso is in /home/remastersys/remastersys/
@item
Test iso by installing in VM and repeating steps above for
getting source files and building lp and docs
@end enumerate