@menu
* Build system overview::
+* Tips for working on the build system::
+* General build system notes::
* Doc build::
* Website build::
+* Building an Ubuntu distro::
@end menu
commands. All the black magick is just an attempt to construct
those commands.
+@node Tips for working on the build system
+@section Tips for working on the build system
+
+@itemize
+@item
+Add:
+
+@example
+echo "aaa"
+
+echo "bbb"
+@end example
+
+to the build system files in various places. This will let you
+track where the program is, in various points of the build.
+
+PH note. There are lots of places where Make doesn't let you put
+echo commands. My top tip for tracing how make runs is to put
+
+@example
+$(error Some Text to display)
+@end example
+
+This will stop make running and print the text @code{Some Text to
+display}.
+
+End PH note.
+
+@item
+First task: understand how @code{make website} works,
+@emph{without} the translations. Looking at the english-only
+website is the best introduction to the build system... it only
+covers about 5% of the whole thing, but even that will likely take
+10 hours or more.
+
+@end itemize
+
+
+@node General build system notes
+@section General build system notes
+
+@menu
+* How stepmake works::
+@end menu
+
+@node How stepmake works
+@subsection How stepmake works
+
+Typing make website runs the file @file{GNUmakefile} from the
+build directory. This only contains 3 lines:
+
+@example
+depth = .
+include config$(if $(conf),-$(conf),).make
+include $(configure-srcdir)/GNUmakefile.in
+@end example
+
+The variable @code{depth} is used throughout the make system to
+track how far down the directory structure the make is. The first
+include sets lots of variables but doesn't "do" anything. The
+second runs the file @file{GNUmakefile.in} from the top level
+source directory.
+
+This sets another load of variables, and then includes (i.e.
+immediately runs) @file{stepmake.make} from the @file{make}
+subdirectory. This sets a load of other variables, does some
+testing to see if SCONS (another build tool?) is being used, and
+then runs @file{make/config.make} - which doesn't seem to exist...
+
+GP: scons is indeed a different build tool; I think that Jan
+experimented with it 5 years ago or something. It seems like we
+still have bits and pieces of it floating around.
+
+Next, it runs @file{make/toplevel-version.make}, which sets the
+version variables for major, minor, patch, stable, development and
+mypatchlevel (which seems to be used for patch numbers for
+non-stable versions only?).
+
+Next - @file{make/local.make}, which doesn't exist.
+
+Then a few more variable and the interesting comment:
+
+@example
+# Don't try to outsmart us, you puny computer!
+# Well, UGH. This only removes builtin rules from
+@end example
+
+and then tests to see whether BUILTINS_REMOVED is defined. It
+appears to be when I run make, and so
+@file{stepmake/stepmake/no-builtin-rules.make} is run. The
+comment at the head of this file says:
+
+@example
+# UGH. GNU make comes with implicit rules.
+# We don't want any of them, and can't force users to run
+# --no-builtin-rules
+@end example
+
+I've not studied that file at length, but assume it removes all
+make's build-in rules (e.g. @file{*.c} files are run through the
+GNU C compiler) - there's a lot of them in here, and a lot of
+comments, and I'd guess most of it isn't needed.
+
+We return to @file{stepmake.make}, where we hit the make rule all:
+The first line of this is:
+
+@smallexample
+-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
+@end smallexample
+
+which, when the variables are substituted, gives:
+
+@example
+./make/generic-inclusions.make
+./make/lilypond-inclusions.make.
+@end example
+
+(Note - according to the make documentation, -include is only
+different from include in that it doesn't produce any kind of
+error message when the included file doesn't exist).
+
+And the first file doesn't exist. Nor the second. Next:
+
+@smallexample
+-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
+@end smallexample
+
+which expands to the following files:
+
+@example
+/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
+/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
+/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
+/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
+@end example
+
+One little feature to notice here - these are all absolute file
+locations - the line prior to this used relative locations. And
+none of these files exist, either. (Further note - I'm assuming
+all these lines of make I'm following are autogenerated, but
+that'll be something else to discover.)
+
+Next in @file{stepmake.make}:
+
+@smallexample
+include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
+@end smallexample
+
+which expands to:
+
+@example
+/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
+/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
+/home/phil/lilypond-git/stepmake/stepmake/po-vars.make
+/home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
+@end example
+
+Woo. They all exist (they should as there's no - in front of the
+include). @file{generic-vars.make} sets loads of variables
+(funnily enough). @file{toplevel-vars.make} is very short - one
+line commented as @code{# override Generic_vars.make:} and 2 as
+follows:
+
+@example
+# urg?
+include $(stepdir)/documentation-vars.make
+@end example
+
+I assume the urg comment refers to the fact that this should
+really just create more variables, but it actually sends us off to
+@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
+
+That file is a 3 line variable setting one.
+
+@file{po-vars.make} has the one-line comment @code{# empty}, as
+does @file{install-vars.make}.
+
+So now we're back to @file{stepmake.make}.
+
+The next lines are
+:
+@smallexample
+# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
+include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
+@end smallexample
+
+and the include expands to:
+
+@example
+include ./make/generic-vars.make ./make/lilypond-vars.make.
+@end example
+
+These again set variables, and in some cases export them to allow
+child @code{make} processes to use them.
+
+The final 4 lines of @file{stepmake.make} are:
+
+@smallexample
+include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
+include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
+include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
+include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
+@end smallexample
+
+which expand as follows:
+
+@example
+include ./make/generic-rules.make ./make/lilypond-rules.make
+include
+ /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
+ /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
+ /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
+ /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
+include ./make/generic-targets.make ./make/lilypond-targets.make
+include
+ /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
+ /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
+ /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
+ /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
+@end example
+
+@file{lilypond-rules.make} is @code{#empty}
+
+@file{generic-rules.make} does seem to have 2 rules in it. They
+are:
+
+@example
+$(outdir)/%.ly: %.lym4
+ $(M4) $< | sed "s/\`/,/g" > $@
+
+$(outdir)/%: %.in
+ rm -f $@
+ cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
+@end example
+
+I believe the first rule is for *.ly files, and has a prerequisite
+that *.lym4 files must be built first. The recipe is @code{m4 |
+sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
+can comment on exactly what the rules mean/do.
+
+@file{toplevel-rules.make} is @code{#empty}
+
+@file{po-rules.make} is @code{#empty}
+
+@file{install-rules.make} is @code{#empty}
+
+@file{generic-targets.make} contains 2 lines of comments.
+
+@file{lilypond-targets.make} contains only:
+
+@example
+## TODO: fail dist or web if no \version present.
+check-version:
+ grep -L version $(LY_FILES)
+@end example
+
+@file{stepmake/generic-targets.make} contains lots of rules - too
+many to list here - it seems to be the main file for rules. (FWIW
+I haven't actually found a rule for website: anywhere, although
+it clearly exists. I have also found that you can display a rule
+in the terminal by typing, say @code{make -n website}. This is
+probably common knowledge.
+
+@file{stepmake/toplevel-targets.make} adds a load of other (and
+occasionally the same) rules to the gernric-targets.
+
+@file{stepmake/po-targets.make} is rules for po* makes.
+
+@file{stepmake/install-targets.make} has rules for local-install*.
+
+And that's the end of stepmake.make. Back to
+@file{GNUmakefile.in}.
+
+A bit more info from 27 March. I've put some error traces into
+@code{GNUmakefile} in the build directory, and it looks like the
+following lines actually cause the make to run (putting an error
+call above them - no make; below them - make):
+
+@smallexample
+ifeq ($(out),www)
+# All web targets, except info image symlinks and info docs are
+# installed in non-recursing target from TOP-SRC-DIR
+install-WWW:
+ -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
+ rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
+ $(MAKE) -C Documentation omf-local-install
+@end smallexample
+
+I don't currently understand the @code{ifeq}, since @code{$(out)}
+is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
+translates to:
+
+@example
+-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
+ -c -m 755 -d /usr/local/share/doc/lilypond/html
+@end example
+
+End of work for Sunday 27th.
+
+Another alterative approach to understanding the website build
+would be to redirect @code{make -n website} and @code{make website}
+to a text file and work through a) what it does and b) where the
+errors are occurring.
+
+GP: wow, all the above is much more complicated than I've ever
+looked at stuff -- I tend to do a "back first" approach (where I
+begin from the command-line that I want to modify, figure out
+where it's generated, and then figure out how to change the
+generated command-line), rather than a "front first" (where you
+begin from the "make" command).
+
@node Doc build
@section Doc build
@menu
+* The function of make doc::
* Building a bibliography::
@end menu
+@node The function of make doc
+@subsection The function of make doc
+
+The following is a set of notes on how make doc functions.
+
+Preliminary question to be answered some time: where do all the
+GNUmakefiles come from. They're in the build directory, but this
+is not part of source. Must be the configure script. And it
+looks like this comes from autogen.sh. Must at some point kill
+the whole git directory, repull and see what is created when.
+
+Anyway, here's how make doc progresses:
+
+This is the build dependency tree from
+@file{stepmake/stepmake/generic-targets.make}:
+
+@example
+doc: doc-stage-1
+ doc-stage-1:
+ $(MAKE) -C $(depth)/scripts/build out=
+ $(MAKE) out=www WWW-1
+ WWW-1: local-WWW-1
+ $(LOOP)
+ $(MAKE) out=www WWW-2
+ WWW-2: local-WWW-2
+ $(LOOP)
+ $(MAKE) out=www WWW-post
+@end example
+
+@example
+MAKE = make --no-builtin-rules
+-C = Change to directory before make
+@end example
+
+doc-stage-1 does lots of opening and looking in files, but no
+processing.
+
+@example
+Variable LOOP =
+
++ make PACKAGE=LILYPOND package=lilypond -C python
+&& make PACKAGE=LILYPOND package=lilypond -C scripts
+&& make PACKAGE=LILYPOND package=lilypond -C flower
+&& make PACKAGE=LILYPOND package=lilypond -C lily
+&& make PACKAGE=LILYPOND package=lilypond -C mf
+&& make PACKAGE=LILYPOND package=lilypond -C ly
+&& make PACKAGE=LILYPOND package=lilypond -C tex
+&& make PACKAGE=LILYPOND package=lilypond -C ps
+&& make PACKAGE=LILYPOND package=lilypond -C scm
+&& make PACKAGE=LILYPOND package=lilypond -C po
+&& make PACKAGE=LILYPOND package=lilypond -C make
+&& make PACKAGE=LILYPOND package=lilypond -C elisp
+&& make PACKAGE=LILYPOND package=lilypond -C vim
+&& make PACKAGE=LILYPOND package=lilypond -C input
+&& make PACKAGE=LILYPOND package=lilypond -C stepmake
+&& make PACKAGE=LILYPOND package=lilypond -C Documentation
+&& true
+@end example
+
+From git grep:
+
+stepmake/stepmake/generic-vars.make has this:
+
+@smallexample
+LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true
+@end smallexample
+
+$@ is the name of the target - WWW-1 in this case.
+
+In GNUmakefile.in we find:
+
+@example
+SUBDIRS = python scripts \
+ flower lily \
+ mf ly \
+ tex ps scm \
+ po make \
+ elisp vim \
+ input \
+ stepmake $(documentation-dir)
+@end example
+
+So that's how we get the main make loop...
+
+That loop expands like this:
+
+@example
+make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
+make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
+true
+@end example
+
+The directories up to and including vim produce no effect with
+make in non-debug mode, although debug does show lots of action.
+
+@file{git/build/input/GNUmakefile} is:
+
+@example
+depth=../
+include $(depth)/config$(if $(conf),-$(conf),).make
+include $(configure-srcdir)/./input/GNUmakefile
+MODULE_INCLUDES += $(src-dir)/$(outbase)
+@end example
+
+The first include is:
+
+@example
+..//config.make
+@end example
+
+(note the // which is strictly wrong)
+
+which has lots of variables to set, but no action occurs.
+
+The second is:
+
+@example
+lilypond-git/./input/GNUmakefile
+@end example
+
+which similarly doesn't create any actual action.
+
+An error message at the end of build/input/GNUmakefile stops
+make processing before it moves on to regression - so where does
+that come from?
+
+And the answer is - make processes all directories in the
+directory it's entered (with some exceptions like out and out-www)
+and so it changes to /regression.
+
+It then seems to consider whether it needs to make/remake loads of
+makefiles. Don't understand this yet. Possibly these are all the
+makefiles it's processing, and it always checks they're up to date
+before processing other files?
+
+Could be correct - some of this output is:
+
+@example
+Must remake target `../../make/ly-inclusions.make'.
+Failed to remake target file `../../make/ly-inclusions.make'.
+@end example
+
+Having decided that, it then leaves the directory and re-executes:
+
+@example
+make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
+@end example
+
+The top of this make is:
+
+@example
+This program built for i486-pc-linux-gnu
+Reading makefiles...
+Reading makefile `GNUmakefile'...
+Reading makefile `../..//config.make' (search path) (no ~ expansion)...
+@end example
+
+which looks like it's re-reading all its known makefiles to check
+they're up to date.
+
+(From the make manual:
+
+To this end, after reading in all makefiles, make will consider each as a goal target and
+attempt to update it. If a makefile has a rule which says how to update it (found either
+in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10
+[Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have
+been checked, if any have actually been changed, make starts with a clean slate and reads
+all the makefiles over again. (It will also attempt to update each of them over again, but
+normally this will not change them again, since they are already up to date.)
+
+So my assumption seems correct)
+
+There appear to be about 74 of them. After all the makefile
+checking, we get this:
+
+@smallexample
+Updating goal targets....
+Considering target file `WWW-1'.
+File `WWW-1' does not exist.
+Considering target file `local-WWW-1'.
+File `local-WWW-1' does not exist.
+Considering target file `out-www/collated-files.texi'.
+File `out-www/collated-files.texi' does not exist.
+Looking for an implicit rule for `out-www/collated-files.texi'.
+Trying pattern rule with stem `collated-files.texi'.
+Trying implicit prerequisite `collated-files.texi.in'.
+Trying pattern rule with stem `collated-files.texi'.
+Trying implicit prerequisite `collated-files.texi.in'.
+Trying pattern rule with stem `collated-files'.
+Trying implicit prerequisite `collated-files.tely'.
+Trying pattern rule with stem `collated-files'.
+Trying implicit prerequisite `out-www/collated-files.tely'.
+Trying rule prerequisite `out-www/version.itexi'.
+Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
+@end smallexample
+
+grep finds this if searching for local-WWW-1:
+
+@example
+make/lysdoc-targets.make:
+ local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
+@end example
+
+which means that local-WWW-1 depends on coll*.texi and coll*.pdf
+and so these will need to be checked to see if they're up to date.
+So make needs to find rules for both of those and (as it says) it
+certainly needs to make coll*.texi, since it doesn't exist.
+
+In ly-rules.make we have:
+
+@example
+.SUFFIXES: .doc .tely .texi .ly
+@end example
+
+which I'll work out at some point, and also this rule:
+
+@smallexample
+$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
+ LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
+@end smallexample
+
+Note that the recipe is a very long line - it could probably
+benefit from splitting. The same makefile also has:
+
+@smallexample
+$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
+ LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
+@end smallexample
+
+@noindent
+which seems to be an almost exact duplicate. Whatever, the first
+one is executed first. Have not checked if the second executes.
+
+The first recipe translates as this:
+
+@example
+LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
+ --output=./out-www --format= --lily-output-dir \
+ /home/phil/lilypond-git/build/out/lybook-db
+@end example
+
+@noindent
+if we stop the build with an $(error), but I think this is because
+we need to allow it to process the dependencies first. It looks
+like foo.texi is shown as being dependent on foo.tely, plus a load
+of other files.
+
+@example
+DOCUMENTATION_LOCALE_TARGET is blank
+INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
+ /home/phil/lilypond-git/scm/autochange.scm
+@end example
+
+plus 10s (100s?) of other .scm files.
+
+@example
+SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
+ /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
+@end example
+
+ditto .ly files. This does seem a teency bit wrong - it looks like
+the .ly and .scm files have been interchanged. ly-vars.make has
+these 2 lines:
+
+@example
+INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
+SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
+@end example
+
+Looks like a bug.....
+
+So it now works its way through all these files, checking if they
+need to be remade. This is 100s of lines of the debug listing,
+although none in the normal list. Clearly none has to be made
+since they're source files. It concludes:
+
+@example
+Must remake target `out-www/collated-files.tely'
+@end example
+
+@file{lysdoc-rules.make} has this:
+
+@smallexample
+$(outdir)/collated-files.tely: $(COLLATED_FILES)
+ $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
+@end smallexample
+
+@file{lysdoc-vars.make} has:
+
+@example
+COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
+@end example
+
+We find that:
+
+@example
+TEXINFO_SOURCES = AAA-intro-regression.tely
+OUT_LY_FILES is empty
+@end example
+
+so LY_FILES has the big long list of all the .ly files in the
+regression directory.
+
+This kicks off
+
+@example
+/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
+@end example
+
+with a list of all the files in the regression test directory. This
+should (I believe) create the file collated-files.tely.
+
+So the next rule in make is for @file{version.itexi}, and make duly
+checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
+depends on @file{git/VERSION}:
+
+@smallexample
+$(outdir)/version.%: $(top-src-dir)/VERSION
+ $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
+@end smallexample
+
+This causes create-version-itexi.py to run and create
+version.itexi.
+
+Once that's done, all the other *.scm and *.ly files are checked
+and since they have no rules associated, they aren't remade (just
+as well for source files, really). Since version.itexi was remade
+make concludes that collated-files.texi must be remade. To do
+this, it runs lilypond-book.py on collated-files.tely, as below:
+
+@example
+LILYPOND_VERSION=2.15.0
+ /usr/bin/python
+ /home/phil/lilypond-git/scripts/lilypond-book.py
+ -I /home/phil/lilypond-git/input/regression/
+ -I ./out-www -I /home/phil/lilypond-git/input
+ -I /home/phil/lilypond-git/Documentation
+ -I /home/phil/lilypond-git/Documentation/snippets
+ -I /home/phil/lilypond-git/input/regression/
+ -I /home/phil/lilypond-git/Documentation/included/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/Documentation/pictures
+ -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
+ --process='/home/phil/lilypond-git/build/out/bin/lilypond
+ -I /home/phil/lilypond-git/input/regression/
+ -I ./out-www
+ -I /home/phil/lilypond-git/input
+ -I /home/phil/lilypond-git/Documentation
+ -I /home/phil/lilypond-git/Documentation/snippets
+ -I /home/phil/lilypond-git/input/regression/
+ -I /home/phil/lilypond-git/Documentation/included/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/Documentation/pictures
+ -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
+ -dbackend=eps
+ --formats=ps,png,pdf
+ -dinclude-eps-fonts
+ -dgs-load-fonts
+ --header=doctitle
+ --header=doctitlecs
+ --header=doctitlede
+ --header=doctitlees
+ --header=doctitlefr
+ --header=doctitlehu
+ --header=doctitleit
+ --header=doctitleja
+ --header=doctitlenl
+ --header=doctitlezh
+ --header=texidoc
+ --header=texidoccs
+ --header=texidocde
+ --header=texidoces
+ --header=texidocfr
+ --header=texidochu
+ --header=texidocit
+ --header=texidocja
+ --header=texidocnl
+ --header=texidoczh
+ -dcheck-internal-types
+ -ddump-signatures
+ -danti-alias-factor=2'
+ --output=./out-www
+ --format=texi-html
+ --verbose
+ --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
+ out-www/collated-files.tely
+@end example
+
+So - lilypond-book runs on:
+
+@example
+input/regression/out-www/collated-files.tely
+@end example
+
+
+Note the --verbose flag - this is from the make variable
+LILYPOND_BOOK_VERBOSE which is added to the make variable
+LILYPOND_BOOK_FLAGS.
+
+Now found the invocation to write some of the image files. It's
+like this:
+
+@example
+/home/phil/lilypond-git/build/out/bin/lilypond
+ -I /home/phil/lilypond-git/input/regression/
+ -I ./out-www -I /home/phil/lilypond-git/input
+ -I /home/phil/lilypond-git/Documentation
+ -I /home/phil/lilypond-git/Documentation/snippets
+ -I /home/phil/lilypond-git/input/regression/
+ -I /home/phil/lilypond-git/Documentation/included/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/build/mf/out/
+ -I /home/phil/lilypond-git/Documentation/pictures
+ -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
+ -dbackend=eps
+ --formats=ps,png,pdf
+ -dinclude-eps-fonts
+ -dgs-load-fonts
+ --header=doctitle
+ --header=doctitlecs
+ --header=doctitlede
+ --header=doctitlees
+ --header=doctitlefr
+ --header=doctitlehu
+ --header=doctitleit
+ --header=doctitleja
+ --header=doctitlenl
+ --header=doctitlezh
+ --header=texidoc
+ --header=texidoccs
+ --header=texidocde
+ --header=texidoces
+ --header=texidocfr
+ --header=texidochu
+ --header=texidocit
+ --header=texidocja
+ --header=texidocnl
+ --header=texidoczh
+ -dcheck-internal-types
+ -ddump-signatures
+ -danti-alias-factor=2
+ -I "/home/phil/lilypond-git/build/out/lybook-db"
+ -I "/home/phil/lilypond-git/build/input/regression"
+ -I "/home/phil/lilypond-git/input/regression"
+ -I "/home/phil/lilypond-git/build/input/regression/out-www"
+ -I "/home/phil/lilypond-git/input"
+ -I "/home/phil/lilypond-git/Documentation"
+ -I "/home/phil/lilypond-git/Documentation/snippets"
+ -I "/home/phil/lilypond-git/input/regression"
+ -I "/home/phil/lilypond-git/Documentation/included"
+ -I "/home/phil/lilypond-git/build/mf/out"
+ -I "/home/phil/lilypond-git/build/mf/out"
+ -I "/home/phil/lilypond-git/Documentation/pictures"
+ -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
+ --formats=eps
+ --verbose
+ -deps-box-padding=3.000000
+ -dread-file-list
+ -dno-strip-output-dir
+ "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
+@end example
+
+Note the --verbose. This causes 100s of lines of Lily debug output.
+But at present I can't work out where the flag comes from. Later.
+
+
@node Building a bibliography
@subsection Building a bibliography
+Bibliography files contain a list of citations, like this:
+
+@example
+@@Book@{vinci,
+ author = @{Vinci, Albert C.@},
+ title = @{Fundamentals of Traditional Music Notation@},
+ publisher = @{Kent State University Press@},
+ year = @{1989@}
+@}
+@end example
+
+There are a variety of types of citation (e.g. Book (as above),
+article, publication). Each cited publication has a list of
+entries that can be used to identify the publication.
+Bibliograpies are normally stored as files with a .bib
+extension. One part of the doc-build process is transforming the
+bibliography information into @code{texinfo} files. The commands
+to do this are in the @file{GNUmakefile} in the
+@file{Documentation} directory.
+
+A typical line of the makefile to translate a single bibliography
+is:
+
+@example
+$(outdir)/colorado.itexi:
+ BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
+ -s $(top-src-dir)/Documentation/lily-bib \
+ -o $(outdir)/colorado.itexi \
+ $(src-dir)/essay/colorado.bib
+@end example
+
+Line by line:
+
+@example
+$(outdir)/colorado.itexi:
+@end example
+
+We're making the file @file{colorado.itexi} and so this is the
+make instruction.
+
+@example
+ BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
+@end example
+
+It's in the @file{essay} directory and we want to run the
+bib2texi.py script against it.
+
+@example
+ -s $(top-src-dir)/Documentation/lily-bib \
+@end example
+
+The style template is @file{lily-bib.bst} and is found in the
+@file{Documentation} directory.
+@example
+ -o $(outdir)/colorado.itexi \
+@end example
+
+The output file in @file{colorado.itexi}.
+
+@example
+ $(src-dir)/essay/colorado.bib
+@end example
+
+The input file is @file{colorado.bib} in the @file{essay}
+directory.
+
+The @code{bib2texi} Python script used to be used with a variety
+of options, but now is always called using the same options, as
+above. Its job is to create the file containing the options for
+@code{bibtex} (the program that actually does the translation),
+run bibtex, and then clean up some temporary files. Its main
+"value add" is the creation of the options file, using this code:
+
+@example
+open (tmpfile + '.aux', 'w').write (r'''
+\relax
+\citation@{*@}
+\bibstyle@{%(style)s@}
+\bibdata@{%(files)s@}''' % vars ())
+@end example
+
+The key items are the style file (now always lily-bib for us) and
+the input file.
+
+The style file is written in its own specialised language,
+described to some extent at
+
+@example
+@uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
+@end example
+
+The file @file{lily-bib.bst} also has fairly extensive commenting.
@node Website build
@section Website build
Start here: @file{make/website.make}
+The overall build system begins with @ref{How stepmake works}.
+Summary: when you type @code{make website} this ends up running
+@file{GNUmakefile.in} in the @file{git} directory. Right at the
+bottom, this has the lines:
+
+@example
+# we want this separate for security; see CG 4.2. -gp
+website:
+ $(MAKE) config_make=$(config_make) \
+ top-src-dir=$(top-src-dir) \
+ -f $(top-src-dir)/make/website.make \
+ website
+@end example
+
+On my system this expands to:
+
+@example
+make --no-builtin-rules config_make=./config.make \
+ top-src-dir=/home/phil/lilypond-git \
+ -f /home/phil/lilypond-git/make/website.make \
+ website
+@end example
+
+We see that the @code{$(MAKE)} expands to
+@code{make --no-builtin-rules} which is how @code{MAKE} is
+defined higher up the makefile. The -f switch defines the
+makefile to be used - in this case
+@file{git/make/website.make}. That's where all the action
+happens.
+
+We believe that note that *none* of the variables that
+are loaded (from depth to version numbers to whatever) are used in
+@file{website.make}. Instead, @file{website.make} sets up its own
+variables at the top of the file. If you're wondering if there's
+some smart reason for this, then the answer is "no". It's because
+I (GP) didn't know/trust the original variables when I was writing
+that file.
+
Website build includes @ref{Building a bibliography}.
+@subsubheading Output from @code{make -n website}
+
+Sorry, including this output directly produces problems in the
+build system. Please run:
+
+@example
+make -n website &> my-file.txt
+@end example
+
+to see the full output from the make.
+
+@subsubheading website.make variables
+
+The file begins by setting up some variables. These
+may/might/probably mirror existing variables, but lacking any docs
+about those variables, I thought it would be simpler to keep
+everything in the same file.
+
+Note that for security reasons, we @strong{don't} call scripts in
+the git dir when building on the web server. See @ref{Uploading
+and security}. So we definitely want to keep those definitions
+for the WEBSITE_ONLY_BUILD.
+
+After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
+there's another bunch of lines setting up generic variables.
+
+@subsubheading website.make building parts
+Parts of @file{website.make}:
+@itemize
+@item
+@code{website:}
+this is the "master" rule. It calls the other rules in order,
+then copies some extra files around - see below for further
+of the process it produces.
+
+@item
+@code{website-version}:
+this calls the python scripts below:
+
+@itemize
+@item
+@example
+scripts/build/create-version-itexi.py
+@end example
+
+This writes a @@version, @@versionStable, and @@versionDevel based
+on the top-level VERSIONS file, to
+@code{out-website/version.itexi}
+
+@item
+@example
+scripts/build/create-weblinks-itexi.py
+@end example
+
+This creates a ton of macros in @code{out-website/weblinks.itexi}.
+Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
+@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
+
+It's quite monstrous because it deals with combinations of
+stable/devel, source/docs, lang/lang/lang*10, etc.
+
+@end itemize
+
+
+@item
+@code{website-xrefs:}
+creates files used for complicated "out-of-build" references to
+@code{out-website/*.xref-map}
+
+If you just write @@ref@{@}, then all's groovy and we wouldn't
+need this. But if you write @@rlearning@{@}, then our custom
+texi2html init file needs to know about our custom xref file
+format, which tells our custom texi2html init file how to create
+the link.
+
+GP: we should have a separate @@node to discuss xrefs. Also, take a
+quick look at a generated xref file -- it's basically just a list
+of @@node's [sic teenager pluralization rule] from the file.
+
+@item
+@code{website-bib:}
+generates the bibliography texinfo files from the .bib files - in
+the case of the website build these are @file{others-did.bib} and
+@file{we-wrote.bib}.
+
+@item
+@code{website-texinfo:}
+this is the main part; it calles texi2html to generate the actual
+html. It also has a ton of options to texi2html to pass info to
+our custom init file.
+
+The file actually built is called @file{web.texi}, and is either
+in the @file{Documentation} directory, or a sub-directory specific
+to the language.
+
+The options file is @file{/Documentation/lilypond-texi2html.init}.
+This contains *lots* of option and configuration stuff, and also
+includes the line:
+
+@smallexample
+print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
+@end smallexample
+
+This is where one of the console messages is generated.
+
+We have somewhere between 2-4 different ways "to pass info to our
+custom init file". This is highly Not Good (tm), but that's how
+things work at the moment.
+
+After texi2html, it does some black magick to deal with
+untranslated nodes in the translations. Despite writing that
+part, I can't remember how it works. But in theory, you could
+figure it out by copy&pasting each part of the command (by "part",
+I mean "stuff before each | pipe"), substituting the variables,
+then looking at the text that's output. For example,
+
+@example
+ ls $(OUT)/$$l/*.html
+@end example
+
+is going to print a list of all html files, in all languages, in
+the build directory. Then more stuff happens to each of those
+files (that's what xargs does).
+
+@item
+@code{website-css:}
+just copies files to the build dir.
+
+@item
+@code{website-pictures, website-examples:}
+more file copies, with an if statement to handle if you don't have
+any generated pictures/examples.
+
+@item
+@code{web-post:}
+runs:
+
+@example
+scripts/build/website_post.py
+@end example
+
+which, it adds the "this page is translated in klingon" to the
+bottom of html pages, and adds the google analytics javascript.
+It also has hard-coded lilypond version numbers, which is Bad
+(tm).
+
+@end itemize
+
+Here's a summary of what gets called, in what order, when we run
+@code{make website}
+
+@example
+website:
+ website-texinfo:
+ website-version:
+ creates version.itexi and weblinks.itexi
+ website-xrefs:
+ runs extract_texi_filenames.py
+ website-bibs:
+ creates bibliography files, described above
+ website-css:
+ copies css files
+ website-pictures:
+ copies pictures
+ website-examples:
+ copies examples
+ web-post:
+ runs website_post.py
+ Then some file copying
+@end example
+
+@node Building an Ubuntu distro
+@section Building an Ubuntu distro
+
+
+Here's the short instruction on how to create lilybuntu iso image
+(Jonathan Kulp did this on a spare drive,
+but he supposes it can be done in a VM too):
+
+@enumerate
+
+@item
+Install ubuntu, reboot.
+@item
+Run all updates, reboot if asked.
+@item
+Enable src repos, refresh package lists.
+@item
+Install LilyPond build deps:
+@example
+sudo apt-get build-dep lilypond
+@end example
+@item
+Install git and autoconf:
+@example
+sudo apt-get install git-core gitk autoconf
+@end example
+
+@item
+Test to see whether everything works fine now:
+@enumerate
+@item
+use @command{lily-git.tcl} to grab source files
+@item
+go to source dir and do
+@example
+"./autogen.sh" ; make ; make doc
+@end example
+@item
+if all compiles, move on to iso creation...
+@end enumerate
+
+@item
+Download & install "remastersys":
+@uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
+@item
+Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
+@item
+Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
+default live session username, etc).
+@item
+Remove non-essential desktop software as desired.
+@item
+Create iso:
+@example
+sudo remastersys dist
+@end example
+New iso is in @file{/home/remastersys/remastersys/}.
+@item
+Test iso by installing in VM and repeating steps above for
+getting source files and building lp and docs.
+
+@end enumerate
projects / lilypond.git / blobdiff