X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;ds=sidebyside;f=Documentation%2Fcontributor%2Fbuild-notes.itexi;h=a5f8c2ccc0b00acc39df6983675bc03c57db2f54;hb=4478a8dc0ac8de7e2596f01e7b212290d05d637c;hp=6bea5628c83cc0885dfa72065a04adbb3c22fb4c;hpb=321beac428ad24824bbd35cc2421a3ee12c630ba;p=lilypond.git diff --git a/Documentation/contributor/build-notes.itexi b/Documentation/contributor/build-notes.itexi index 6bea5628c8..a5f8c2ccc0 100644 --- a/Documentation/contributor/build-notes.itexi +++ b/Documentation/contributor/build-notes.itexi @@ -11,6 +11,7 @@ chapter.} @menu * Build system overview:: * Tips for working on the build system:: +* General build system notes:: * Doc build:: * Website build:: @end menu @@ -72,6 +73,18 @@ echo "bbb" to the build system files in various places. This will let you track where the program is, in various points of the build. +PH note. There are lots of places where Make doesn't let you put +echo commands. My top tip for tracing how make runs is to put + +@example +$(error Some Text to display) +@end example + +This will stop make running and print the text @code{Some Text to +display}. + +End PH note. + @item First task: understand how @code{make website} works, @emph{without} the translations. Looking at the english-only @@ -82,14 +95,781 @@ covers about 5% of the whole thing, but even that will likely take @end itemize +@node General build system notes +@section General build system notes + +@menu +* How stepmake works:: +@end menu + +@node How stepmake works +@subsection How stepmake works + +Typing make website runs the file @file{GNUmakefile} from the +build directory. This only contains 3 lines: + +@example +depth = . +include config$(if $(conf),-$(conf),).make +include $(configure-srcdir)/GNUmakefile.in +@end example + +The variable @code{depth} is used throughout the make system to +track how far down the directory structure the make is. The first +include sets lots of variables but doesn't "do" anything. Default +values for these variables are automatically detected at the +./configure step, which creates the file @file{config.make}. +The second include runs the file @file{GNUmakefile.in} from +the top level source directory. + +This sets another load of variables, and then includes (i.e. +immediately runs) @file{stepmake.make} from the @file{make} +subdirectory. This sets a load of other variables, does some +testing to see if SCONS (another build tool?) is being used, and +then runs @file{make/config.make} - which doesn't seem to exist... + +GP: scons is indeed a different build tool; I think that Jan +experimented with it 5 years ago or something. It seems like we +still have bits and pieces of it floating around. + +Next, it runs @file{make/toplevel-version.make}, which sets the +version variables for major, minor, patch, stable, development and +mypatchlevel (which seems to be used for patch numbers for +non-stable versions only?). + +Next - @file{make/local.make}, which doesn't exist. + +Then a few more variable and the interesting comment: + +@example +# Don't try to outsmart us, you puny computer! +# Well, UGH. This only removes builtin rules from +@end example + +and then tests to see whether BUILTINS_REMOVED is defined. It +appears to be when I run make, and so +@file{stepmake/stepmake/no-builtin-rules.make} is run. The +comment at the head of this file says: + +@example +# UGH. GNU make comes with implicit rules. +# We don't want any of them, and can't force users to run +# --no-builtin-rules +@end example + +I've not studied that file at length, but assume it removes all +make's build-in rules (e.g. @file{*.c} files are run through the +GNU C compiler) - there's a lot of them in here, and a lot of +comments, and I'd guess most of it isn't needed. + +We return to @file{stepmake.make}, where we hit the make rule all: +The first line of this is: + +@smallexample +-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end smallexample + +which, when the variables are substituted, gives: + +@example +./make/generic-inclusions.make +./make/lilypond-inclusions.make. +@end example + +(Note - according to the make documentation, -include is only +different from include in that it doesn't produce any kind of +error message when the included file doesn't exist). + +And the first file doesn't exist. Nor the second. Next: + +@smallexample +-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expands to the following files: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make. +@end example + +One little feature to notice here - these are all absolute file +locations - the line prior to this used relative locations. And +none of these files exist, either. + +(Further note - I'm assuming all these lines of make I'm following are +autogenerated, but that'll be something else to discover.) + +JM: @emph{``No, these lines are not useful in LilyPond (this is why +you think they are autogenerated), but they are part of StepMake, +which was meant to be a package to be installed as a build system over +autoconf/make in software project source trees.''} + +Next in @file{stepmake.make}: + +@smallexample +include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expands to: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make +/home/phil/lilypond-git/stepmake/stepmake/po-vars.make +/home/phil/lilypond-git/stepmake/stepmake/install-vars.make. +@end example + +Woo. They all exist (they should as there's no - in front of the +include). @file{generic-vars.make} sets loads of variables +(funnily enough). @file{toplevel-vars.make} is very short - one +line commented as @code{# override Generic_vars.make:} and 2 as +follows: + +@example +# urg? +include $(stepdir)/documentation-vars.make +@end example + +I assume the urg comment refers to the fact that this should +really just create more variables, but it actually sends us off to +@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}. + +That file is a 3 line variable setting one. + +@file{po-vars.make} has the one-line comment @code{# empty}, as +does @file{install-vars.make}. + +So now we're back to @file{stepmake.make}. + +The next lines are +: +@smallexample +# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH) +include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end smallexample + +and the include expands to: + +@example +include ./make/generic-vars.make ./make/lilypond-vars.make. +@end example + +These again set variables, and in some cases export them to allow +child @code{make} processes to use them. + +The final 4 lines of @file{stepmake.make} are: + +@smallexample +include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES))) +include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expand as follows: + +@example +include ./make/generic-rules.make ./make/lilypond-rules.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make + /home/phil/lilypond-git/stepmake/stepmake/po-rules.make + /home/phil/lilypond-git/stepmake/stepmake/install-rules.make +include ./make/generic-targets.make ./make/lilypond-targets.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make + /home/phil/lilypond-git/stepmake/stepmake/po-targets.make + /home/phil/lilypond-git/stepmake/stepmake/install-targets.make +@end example + +@file{lilypond-rules.make} is @code{#empty} + +@file{generic-rules.make} does seem to have 2 rules in it. They +are: + +@example +$(outdir)/%.ly: %.lym4 + $(M4) $< | sed "s/\`/,/g" > $@@ + +$(outdir)/%: %.in + rm -f $@@ + cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@@ +@end example + +I believe the first rule is for *.ly files, and has a prerequisite +that *.lym4 files must be built first. The recipe is @code{m4 | +sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge +can comment on exactly what the rules mean/do. + +@file{toplevel-rules.make} is @code{#empty} + +@file{po-rules.make} is @code{#empty} + +@file{install-rules.make} is @code{#empty} + +@file{generic-targets.make} contains 2 lines of comments. + +@file{lilypond-targets.make} contains only: + +@example +## TODO: fail dist or web if no \version present. +check-version: + grep -L version $(LY_FILES) +@end example + +@file{stepmake/generic-targets.make} contains lots of rules - too +many to list here - it seems to be the main file for rules. (FWIW +I haven't actually found a rule for website: anywhere, although +it clearly exists. I have also found that you can display a rule +in the terminal by typing, say @code{make -n website}. This is +probably common knowledge. + +@file{stepmake/toplevel-targets.make} adds a load of other (and +occasionally the same) rules to the gernric-targets. + +@file{stepmake/po-targets.make} is rules for po* makes. + +@file{stepmake/install-targets.make} has rules for local-install*. + +And that's the end of stepmake.make. Back to +@file{GNUmakefile.in}. + +A bit more info from 27 March. I've put some error traces into +@code{GNUmakefile} in the build directory, and it looks like the +following lines actually cause the make to run (putting an error +call above them - no make; below them - make): + +@smallexample +ifeq ($(out),www) +# All web targets, except info image symlinks and info docs are +# installed in non-recursing target from TOP-SRC-DIR +install-WWW: + -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir) + rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir) + $(MAKE) -C Documentation omf-local-install +@end smallexample + +I don't currently understand the @code{ifeq}, since @code{$(out)} +is empty at this point, but the line starting @w{@code{-$(INSTALL)}} +translates to: + +@example +-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \ + -c -m 755 -d /usr/local/share/doc/lilypond/html +@end example + +End of work for Sunday 27th. + +Another alterative approach to understanding the website build +would be to redirect @code{make -n website} and @code{make website} +to a text file and work through a) what it does and b) where the +errors are occurring. + +GP: wow, all the above is much more complicated than I've ever +looked at stuff -- I tend to do a "back first" approach (where I +begin from the command-line that I want to modify, figure out +where it's generated, and then figure out how to change the +generated command-line), rather than a "front first" (where you +begin from the "make" command). + @node Doc build @section Doc build @menu +* The function of make doc:: * Building a bibliography:: @end menu +@node The function of make doc +@subsection The function of make doc + +The following is a set of notes on how make doc functions. + +Preliminary question to be answered some time: where do all the +GNUmakefiles come from. They're in the build directory, but this +is not part of source. Must be the configure script. And it +looks like this comes from autogen.sh. Must at some point kill +the whole git directory, repull and see what is created when. + +Anyway, here's how make doc progresses: + +This is the build dependency tree from +@file{stepmake/stepmake/generic-targets.make}: + +@example +doc: doc-stage-1 + doc-stage-1: + $(MAKE) -C $(depth)/scripts/build out= + $(MAKE) out=www WWW-1 + WWW-1: local-WWW-1 + $(LOOP) + $(MAKE) out=www WWW-2 + WWW-2: local-WWW-2 + $(LOOP) + $(MAKE) out=www WWW-post +@end example + +@example +MAKE = make --no-builtin-rules +-C = Change to directory before make +@end example + +doc-stage-1 does lots of opening and looking in files, but no +processing. + +@example +Variable LOOP = + ++ make PACKAGE=LILYPOND package=lilypond -C python +&& make PACKAGE=LILYPOND package=lilypond -C scripts +&& make PACKAGE=LILYPOND package=lilypond -C flower +&& make PACKAGE=LILYPOND package=lilypond -C lily +&& make PACKAGE=LILYPOND package=lilypond -C mf +&& make PACKAGE=LILYPOND package=lilypond -C ly +&& make PACKAGE=LILYPOND package=lilypond -C tex +&& make PACKAGE=LILYPOND package=lilypond -C ps +&& make PACKAGE=LILYPOND package=lilypond -C scm +&& make PACKAGE=LILYPOND package=lilypond -C po +&& make PACKAGE=LILYPOND package=lilypond -C make +&& make PACKAGE=LILYPOND package=lilypond -C elisp +&& make PACKAGE=LILYPOND package=lilypond -C vim +&& make PACKAGE=LILYPOND package=lilypond -C input +&& make PACKAGE=LILYPOND package=lilypond -C stepmake +&& make PACKAGE=LILYPOND package=lilypond -C Documentation +&& true +@end example + +From git grep: + +stepmake/stepmake/generic-vars.make has this: + +@smallexample +LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@@ &&) true +@end smallexample + +$@@ is the name of the target - WWW-1 in this case. + +In GNUmakefile.in we find: + +@example +SUBDIRS = python scripts \ + flower lily \ + mf ly \ + tex ps scm \ + po make \ + elisp vim \ + input \ + stepmake $(documentation-dir) +@end example + +So that's how we get the main make loop... + +That loop expands like this: + +@example +make PACKAGE=LILYPOND package=lilypond -C python WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C po WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C make WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C input WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 && +true +@end example + +The directories up to and including vim produce no effect with +make in non-debug mode, although debug does show lots of action. + +@file{git/build/input/GNUmakefile} is: + +@example +depth=../ +include $(depth)/config$(if $(conf),-$(conf),).make +include $(configure-srcdir)/./input/GNUmakefile +MODULE_INCLUDES += $(src-dir)/$(outbase) +@end example + +The first include is: + +@example +..//config.make +@end example + +(note the // which is strictly wrong) + +which has lots of variables to set, but no action occurs. + +The second is: + +@example +lilypond-git/./input/GNUmakefile +@end example + +which similarly doesn't create any actual action. + +An error message at the end of build/input/GNUmakefile stops +make processing before it moves on to regression - so where does +that come from? + +And the answer is - make processes all directories in the +directory it's entered (with some exceptions like out and out-www) +and so it changes to /regression. + +It then seems to consider whether it needs to make/remake loads of +makefiles. Don't understand this yet. Possibly these are all the +makefiles it's processing, and it always checks they're up to date +before processing other files? + +Could be correct - some of this output is: + +@example +Must remake target `../../make/ly-inclusions.make'. +Failed to remake target file `../../make/ly-inclusions.make'. +@end example + +Having decided that, it then leaves the directory and re-executes: + +@example +make PACKAGE=LILYPOND package=lilypond -C regression WWW-1 +@end example + +The top of this make is: + +@example +This program built for i486-pc-linux-gnu +Reading makefiles... +Reading makefile `GNUmakefile'... +Reading makefile `../..//config.make' (search path) (no ~ expansion)... +@end example + +which looks like it's re-reading all its known makefiles to check +they're up to date. + +(From the make manual: + +To this end, after reading in all makefiles, make will consider each +as a goal target and attempt to update it. If a makefile has a rule +which says how to update it (found either in that very makefile or in +another one) or if an implicit rule applies to it (see Chapter 10 +[Using Implicit Rules], page 103), it will be updated if +necessary. After all makefiles have been checked, if any have actually +been changed, make starts with a clean slate and reads all the +makefiles over again. (It will also attempt to update each of them +over again, but normally this will not change them again, since they +are already up to date.) + +So my assumption seems correct) + +There appear to be about 74 of them. After all the makefile +checking, we get this: + +@smallexample +Updating goal targets.... +Considering target file `WWW-1'. +File `WWW-1' does not exist. +Considering target file `local-WWW-1'. +File `local-WWW-1' does not exist. +Considering target file `out-www/collated-files.texi'. +File `out-www/collated-files.texi' does not exist. +Looking for an implicit rule for `out-www/collated-files.texi'. +Trying pattern rule with stem `collated-files.texi'. +Trying implicit prerequisite `collated-files.texi.in'. +Trying pattern rule with stem `collated-files.texi'. +Trying implicit prerequisite `collated-files.texi.in'. +Trying pattern rule with stem `collated-files'. +Trying implicit prerequisite `collated-files.tely'. +Trying pattern rule with stem `collated-files'. +Trying implicit prerequisite `out-www/collated-files.tely'. +Trying rule prerequisite `out-www/version.itexi'. +Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi' +@end smallexample + +grep finds this if searching for local-WWW-1: + +@example +make/lysdoc-targets.make: + local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf +@end example + +which means that local-WWW-1 depends on coll*.texi and coll*.pdf +and so these will need to be checked to see if they're up to date. +So make needs to find rules for both of those and (as it says) it +certainly needs to make coll*.texi, since it doesn't exist. + +In ly-rules.make we have: + +@example +.SUFFIXES: .doc .tely .texi .ly +@end example + +which I'll work out at some point, and also this rule: + +@smallexample +$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES) + LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $< +@end smallexample + +Note that the recipe is a very long line - it could probably +benefit from splitting. The same makefile also has: + +@smallexample +$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES) + LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $< +@end smallexample + +@noindent +which seems to be an almost exact duplicate. Whatever, the first +one is executed first. Have not checked if the second executes. + +The first recipe translates as this: + +@example +LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \ + --output=./out-www --format= --lily-output-dir \ + /home/phil/lilypond-git/build/out/lybook-db +@end example + +@noindent +if we stop the build with an $(error), but I think this is because +we need to allow it to process the dependencies first. It looks +like foo.texi is shown as being dependent on foo.tely, plus a load +of other files. + +@example +DOCUMENTATION_LOCALE_TARGET is blank +INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \ + /home/phil/lilypond-git/scm/autochange.scm +@end example + +plus 10s (100s?) of other .scm files. + +@example +SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \ + /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly +@end example + +ditto .ly files. This does seem a teency bit wrong - it looks like +the .ly and .scm files have been interchanged. ly-vars.make has +these 2 lines: + +@example +INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm) +SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly) +@end example + +Looks like a bug..... + +So it now works its way through all these files, checking if they +need to be remade. This is 100s of lines of the debug listing, +although none in the normal list. Clearly none has to be made +since they're source files. It concludes: + +@example +Must remake target `out-www/collated-files.tely' +@end example + +@file{lysdoc-rules.make} has this: + +@smallexample +$(outdir)/collated-files.tely: $(COLLATED_FILES) + $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^ +@end smallexample + +@file{lysdoc-vars.make} has: + +@example +COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) ) +@end example + +We find that: + +@example +TEXINFO_SOURCES = AAA-intro-regression.tely +OUT_LY_FILES is empty +@end example + +so LY_FILES has the big long list of all the .ly files in the +regression directory. + +This kicks off + +@example +/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely +@end example + +with a list of all the files in the regression test directory. This +should (I believe) create the file collated-files.tely. + +So the next rule in make is for @file{version.itexi}, and make duly +checks this. There's a rule in @file{doc-i18n-root-rules.make} that this +depends on @file{git/VERSION}: + +@smallexample +$(outdir)/version.%: $(top-src-dir)/VERSION + $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@ +@end smallexample + +This causes create-version-itexi.py to run and create +version.itexi. + +Once that's done, all the other *.scm and *.ly files are checked +and since they have no rules associated, they aren't remade (just +as well for source files, really). Since version.itexi was remade +make concludes that collated-files.texi must be remade. To do +this, it runs lilypond-book.py on collated-files.tely, as below: + +@example +LILYPOND_VERSION=2.15.0 + /usr/bin/python + /home/phil/lilypond-git/scripts/lilypond-book.py + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + --process='/home/phil/lilypond-git/build/out/bin/lilypond + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www + -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + -dbackend=eps + --formats=ps,png,pdf + -dinclude-eps-fonts + -dgs-load-fonts + --header=doctitle + --header=doctitlecs + --header=doctitlede + --header=doctitlees + --header=doctitlefr + --header=doctitlehu + --header=doctitleit + --header=doctitleja + --header=doctitlenl + --header=doctitlezh + --header=texidoc + --header=texidoccs + --header=texidocde + --header=texidoces + --header=texidocfr + --header=texidochu + --header=texidocit + --header=texidocja + --header=texidocnl + --header=texidoczh + -dcheck-internal-types + -ddump-signatures + -danti-alias-factor=2' + --output=./out-www + --format=texi-html + --verbose + --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db + out-www/collated-files.tely +@end example + +So - lilypond-book runs on: + +@example +input/regression/out-www/collated-files.tely +@end example + + +Note the --verbose flag - this is from the make variable +LILYPOND_BOOK_VERBOSE which is added to the make variable +LILYPOND_BOOK_FLAGS. + +Now found the invocation to write some of the image files. It's +like this: + +@example +/home/phil/lilypond-git/build/out/bin/lilypond + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + -dbackend=eps + --formats=ps,png,pdf + -dinclude-eps-fonts + -dgs-load-fonts + --header=doctitle + --header=doctitlecs + --header=doctitlede + --header=doctitlees + --header=doctitlefr + --header=doctitlehu + --header=doctitleit + --header=doctitleja + --header=doctitlenl + --header=doctitlezh + --header=texidoc + --header=texidoccs + --header=texidocde + --header=texidoces + --header=texidocfr + --header=texidochu + --header=texidocit + --header=texidocja + --header=texidocnl + --header=texidoczh + -dcheck-internal-types + -ddump-signatures + -danti-alias-factor=2 + -I "/home/phil/lilypond-git/build/out/lybook-db" + -I "/home/phil/lilypond-git/build/input/regression" + -I "/home/phil/lilypond-git/input/regression" + -I "/home/phil/lilypond-git/build/input/regression/out-www" + -I "/home/phil/lilypond-git/input" + -I "/home/phil/lilypond-git/Documentation" + -I "/home/phil/lilypond-git/Documentation/snippets" + -I "/home/phil/lilypond-git/input/regression" + -I "/home/phil/lilypond-git/Documentation/included" + -I "/home/phil/lilypond-git/build/mf/out" + -I "/home/phil/lilypond-git/build/mf/out" + -I "/home/phil/lilypond-git/Documentation/pictures" + -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www" + --formats=eps + --verbose + -deps-box-padding=3.000000 + -dread-file-list + -dno-strip-output-dir + "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"' +@end example + +Note the --verbose. This causes 100s of lines of Lily debug output. +But at present I can't work out where the flag comes from. Later. + + @node Building a bibliography @subsection Building a bibliography @@ -118,10 +898,10 @@ is: @example $(outdir)/colorado.itexi: - BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \ - -s $(top-src-dir)/Documentation/lily-bib \ - -o $(outdir)/colorado.itexi \ - $(src-dir)/essay/colorado.bib + BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \ + -s $(top-src-dir)/Documentation/lily-bib \ + -o $(outdir)/colorado.itexi \ + $(src-dir)/essay/colorado.bib @end example Line by line: @@ -134,27 +914,27 @@ We're making the file @file{colorado.itexi} and so this is the make instruction. @example - BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \ + BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \ @end example It's in the @file{essay} directory and we want to run the bib2texi.py script against it. @example - -s $(top-src-dir)/Documentation/lily-bib \ + -s $(top-src-dir)/Documentation/lily-bib \ @end example The style template is @file{lily-bib.bst} and is found in the @file{Documentation} directory. @example - -o $(outdir)/colorado.itexi \ + -o $(outdir)/colorado.itexi \ @end example The output file in @file{colorado.itexi}. @example - $(src-dir)/essay/colorado.bib + $(src-dir)/essay/colorado.bib @end example The input file is @file{colorado.bib} in the @file{essay} @@ -187,124 +967,482 @@ described to some extent at The file @file{lily-bib.bst} also has fairly extensive commenting. + @node Website build @section Website build -Start here: @file{make/website.make} +@warning{This information applies only to the standard @code{make +website} from the normal build directory. The process is +different for @code{dev/website-build}.} -Typing make website runs the file @file{GNUmakefile} from the -build directory. This only contains 3 lines: +The rule for make website is found in GNUmakefile.in: @example -depth = . -include config$(if $(conf),-$(conf),).make -include $(configure-srcdir)/GNUmakefile.in +website: +$(MAKE) config_make=$(config_make) \ + top-src-dir=$(top-src-dir) \ + -f $(top-src-dir)/make/website.make \ + website @end example -The variable @code{depth} is used throughout the make system to -track how far down the directory structure the make is. The first -include sets lots of variables but doesn't "do" anything. The -second runs the file @file{GNUmakefile.in} from the top level -source directory. - -This sets another load of variables, and then includes (i.e. -immediately runs) @file{stepmake.make} from the @file{make} -subdirectory. This sets a load of other variables, does some -testing to see if SCONS (another build tool?) is being used, and -then runs @file{make/config.make} - which doesn't seem to exist... +This translates as: -Next, it runs @file{make/toplevel-version.make}, which sets the -version variables for major, minor, patch, stable, development and -mypatchlevel (which seems to be used for patch numbers for -non-stable versions only?). +@example +make --no-builtin-rules config_make=./config.make \ + top-src-dir=/home/phil/lilypond-git \ + -f /home/phil/lilypond-git/make/website.make \ + website +@end example -Next - @file{make/local.make}, which doesn't exist. +which has the effect of setting the variables @code{config_make} +and @code{top-src-dir} and then processing the file +@code{git/make/website.make} with the target of website. -Then a few more variable and the interesting comment: +@code{website.make} starts with the following: @example -# Don't try to outsmart us, you puny computer! -# Well, UGH. This only removes builtin rules from +ifeq ($(WEBSITE_ONLY_BUILD),1) @end example -and then tests to see whether BUILTINS_REMOVED is defined. It -appears to be when I run make, and so -@file{stepmake/stepmake/no-builtin-rules.make} is run. The -comment at the head of this file says: +which checks to see whether the variable @code{WEBSITE_ONLY_BUILD} +was set to one on the command line. This is only done for +standalone website builds, not in the normal case. The result of +the test determines the value of some variables that are set. A +number of other variables are set, in order to establish locations +of various files. An example is: @example -# UGH. GNU make comes with implicit rules. -# We don't want any of them, and can't force users to run -# --no-builtin-rules +CREATE_VERSION=python $(script-dir)/create-version-itexi.py @end example -I've not studied that file at length, but assume it removes all -make's build-in rules (e.g. @file{*.c} files are run through the -GNU C compiler) - there's a lot of them in here, and a lot of -comments, and I'd guess most of it isn't needed. +The rule for website is: -We return to @file{stepmake.make}, where we hit the make rule all: -The first line of this is: +@smallexample +website: website-texinfo website-css website-pictures website-examples web-post + cp $(SERVER_FILES)/favicon.ico $(OUT)/website + cp $(SERVER_FILES)/robots.txt $(OUT)/website + cp $(top-htaccess) $(OUT)/.htaccess + cp $(dir-htaccess) $(OUT)/website/.htaccess +@end smallexample + +so we see that this starts by running the rules for 5 other +targets, then finishes by copying some files. We'll cover that +later - first @code{website-texinfo}. That rule is: @example --include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES))) +website-texinfo: website-version website-xrefs website-bibs + for l in '' $(WEB_LANGS); do \ + if test -n "$$l"; then \ + langopt=--lang="$$l"; \ + langsuf=.$$l; \ + fi; \ + $(TEXI2HTML) --prefix=index \ + --split=section \ + --I=$(top-src-dir)/Documentation/"$$l" \ + --I=$(top-src-dir)/Documentation \ + --I=$(OUT) \ + $$langopt \ + --init-file=$(texi2html-init-file) \ + -D web_version \ + --output=$(OUT)/"$$l" \ + $(top-src-dir)/Documentation/"$$l"/web.texi ; \ + ls $(OUT)/$$l/*.html | xargs grep -L \ + 'UNTRANSLATED NODE: IGNORE ME' | \ + sed 's!$(OUT)/'$$l'/!!g' | xargs \ + $(MASS_LINK) --prepend-suffix="$$langsuf" \ + hard $(OUT)/$$l/ $(OUT)/website/ ; \ + done @end example -which, when the variables are substituted, gives: +which therefore depends on @code{website-version}, +@code{website-xrefs} and @code{website-bibs}. @example -./make/generic-inclusions.make -./make/lilypond-inclusions.make. +website-version: + mkdir -p $(OUT) + $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi + $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi @end example -(Note - according to the make documentation, -include is only -different from include in that it doesn't produce any kind of -error message when the included file doesn't exist). +which translates as: -And the first file doesn't exist. Nor the second. Next: +@example +mkdir -p out-website +python /home/phil/lilypond-git/scripts/build/create-version-itexi.py + /home/phil/lilypond-git > out-website/version.itexi +python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py + /home/phil/lilypond-git > out-website/weblinks.itexi +@end example + +So, we make out-website then send the output of +@code{create-version-itexi.py} to @code{out-website/version.itexi} +and @code{create-weblinks-itexi.py} to +@code{out-website/weblinks.itexi}. + +@code{create-version-itexi.py} parses the file @code{VERSION} in +the top source dir. It contains: @example --include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES))) +PACKAGE_NAME=LilyPond +MAJOR_VERSION=2 +MINOR_VERSION=15 +PATCH_LEVEL=13 +MY_PATCH_LEVEL= +VERSION_STABLE=2.14.2 +VERSION_DEVEL=2.15.12 @end example -which expands to the following files: +currently. @code{c-v-i.py} parses this to: @example -/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make. +@@c ************************ Version numbers ************ +@@macro version +2.15.13 +@@end macro + +@@macro versionStable +2.14.2 +@@end macro + +@@macro versionDevel +2.15.12 +@@end macro @end example -One little feature to notice here - these are all absolute file -locations - the line prior to this used relative locations. And -none of these files exist, either. (Further note - I'm assuming -all these lines of make I'm following are autogenerated, but -that'll be something else to discover.) +@code{create-weblinks-itexi.py} creates a load of texi macros (of +the order of 1000) similar to: + +@example +@@macro manualStableGlossaryPdf +@@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@} +@@end macro. +@end example + +It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning: + +@code{langdefs.py: warning: lilypond-doc gettext domain not found.} Next: @example -include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES))) +website-xrefs: website-version + for l in '' $(WEB_LANGS); do \ @end example -which expands to: +is the start of the rule, truncated for brevity. This loops +through the languages to be used on the website, processing some +variables which I don't fully understand, to run this command: + +@smallexample +python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \ + -I /home/phil/lilypond-git/Documentation \ + -I /home/phil/lilypond-git/Documentation/"$l" \ + -I out-website -o out-website --split=node \ + --known-missing-files= \ + /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \ + -q \ + /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\ +@end smallexample + +There's a good description of what +@code{extract_texi_filenames.py} does at the top of the script, +but a shortened version is: + +@code{If this script is run on a file texifile.texi, it produces +a file texifile[.LANG].xref-map with tab-separated entries +of the form NODE\tFILENAME\tANCHOR.} + +An example from +@code{web.nl.xref-map} is: @example -/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make -/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make -/home/phil/lilypond-git/stepmake/stepmake/po-vars.make -/home/phil/lilypond-git/stepmake/stepmake/install-vars.make. +Inleiding Introduction Introduction +@end example + +@code{e-t-f.py} follows the includes from document to document. +We know some have not been created yet, and +@code{known-missing-files} option tells @code{e-t-f.py} which +these are. + +It then does this: + +@example +for m in $(MANUALS); do \ @end example -Woo. They all exist (they should - no - in front of the include). +to run @code{e-t-f.py} against all of the manuals, in each +language. Next: + +@example +website-bibs: website-version + BSTINPUTS=$(top-src-dir)/Documentation/web \ + $(WEB_BIBS) -s web \ + -s $(top-src-dir)/Documentation/lily-bib \ + -o $(OUT)/others-did.itexi \ + $(quiet-flag) \ + $(top-src-dir)/Documentation/web/others-did.bib +@end example -Next step will be to work out what these do. +This is half the command. It runs @code{bib2texi.py} on 2 +@code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}. +This converts bibliography files into texi files with +@code{bibtex}. -That's my 30 minutes for Monday. +Next the commands in the @code{website-texinfo} rule are run: + +@example +for l in '' $(WEB_LANGS); do \ +@end example -Website build includes @ref{Building a bibliography}. +run @code{texi2html}. This is the program that outputs the +progress message (found in +@code{Documentation/lilypond-texi2html.init}): +@code{Processing web site: []} +It also outputs warning messages like: + +@code{WARNING: Unable to find node 'Řešení potíží' in book usage.} + +@example +website-css: + cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website +@end example + +Copies 3 css files to out-website/website. Then: + +@example +website-pictures: + mkdir -p $(OUT)/website/pictures + if [ -d $(PICTURES) ]; \ + then \ + cp $(PICTURES)/* $(OUT)/website/pictures ; \ + ln -sf website/pictures $(OUT)/pictures ;\ + fi +@end example + +which translates as: + +@smallexample +if [ -d Documentation/pictures/out-www ]; \ + then \ + cp Documentation/pictures/out-www/* out-website/website/pictures ; \ + ln -sf website/pictures out-website/pictures ;\ + fi +@end smallexample + +i.e. it copies the contents of +@code{build/Documentation/pictures/out-www/*} to +@code{out-website/website/pictures}. Unfortunately, the pictures +are only created once @code{make doc} has been run, so an initial +run of @code{make website} copies nothing, and the pictures on the +website (e.g. the logo) do not exist. Next: + +@example +website-examples: + mkdir -p $(OUT)/website/ly-examples + if [ -d $(EXAMPLES) ]; \ + then \ + cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \ + fi +@end example + +translates to: + +@smallexample +mkdir -p out-website/website/ly-examples +if [ -d Documentation/web/ly-examples/out-www ]; \ + then \ + cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \ + fi +@end smallexample + +This does the same with the LilyPond examples (found at +@uref{http://lilypond.org/examples.html}). Again, these are +actually only created by @code{make doc} (and since they are +generated from LilyPond source files, require a working LilyPond +@code{exe} made with @code{make}). So this does nothing +initially. Then: + +@example +web-post: + $(WEB_POST) $(OUT)/website +@end example + +which is: + +@smallexample +python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website +@end smallexample + +which describes itself as: + +@code{This is web_post.py. This script deals with translations +in the "make website" target.} + +It also does a number of other things, including adding the Google +tracker code and the language selection footer. We're now at +the end of our story. The final 4 lines of the recipe for website +are: + +@example +cp $(SERVER_FILES)/favicon.ico $(OUT)/website +cp $(SERVER_FILES)/robots.txt $(OUT)/website +cp $(top-htaccess) $(OUT)/.htaccess +cp $(dir-htaccess) $(OUT)/website/.htaccess +@end example +The first translates as: +@smallexample +cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website +@end smallexample + +so we see these are just copying the support files for the web +server. + +@subsubheading website.make summary + +Recipes in @file{website.make}: + +@itemize + +@item +@code{website:} +this is the "master" rule. It calls the other rules in order, +then copies some extra files around - see below for further +of the process it produces. + +@item +@code{website-version}: +this calls the python scripts below: +@itemize +@item +@example +scripts/build/create-version-itexi.py +@end example + +This writes a @@version, @@versionStable, and @@versionDevel based +on the top-level VERSIONS file, to +@code{out-website/version.itexi} + +@item +@example +scripts/build/create-weblinks-itexi.py +@end example + +This creates a ton of macros in @code{out-website/weblinks.itexi}. +Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows, +@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh. + +It's quite monstrous because it deals with combinations of +stable/devel, source/docs, lang/lang/lang*10, etc. + +@end itemize + +@item +@code{website-xrefs:} +creates files used for complicated "out-of-build" references to +@code{out-website/*.xref-map} + +If you just write @@ref@{@}, then all's groovy and we wouldn't +need this. But if you write @@rlearning@{@}, then our custom +texi2html init file needs to know about our custom xref file +format, which tells our custom texi2html init file how to create +the link. + +GP: we should have a separate @@node to discuss xrefs. Also, take a +quick look at a generated xref file -- it's basically just a list +of @@node's [sic teenager pluralization rule] from the file. + +@item +@code{website-bib:} +generates the bibliography texinfo files from the .bib files - in +the case of the website build these are @file{others-did.bib} and +@file{we-wrote.bib}. + +@item +@code{website-texinfo:} +this is the main part; it calles texi2html to generate the actual +html. It also has a ton of options to texi2html to pass info to +our custom init file. + +The file actually built is called @file{web.texi}, and is either +in the @file{Documentation} directory, or a sub-directory specific +to the language. + +The options file is @file{/Documentation/lilypond-texi2html.init}. +This contains *lots* of option and configuration stuff, and also +includes the line: + +@smallexample +print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n"; +@end smallexample + +This is where one of the console messages is generated. + +We have somewhere between 2-4 different ways "to pass info to our +custom init file". This is highly Not Good (tm), but that's how +things work at the moment. + +After texi2html, it does some black magick to deal with +untranslated nodes in the translations. Despite writing that +part, I can't remember how it works. But in theory, you could +figure it out by copy&pasting each part of the command (by "part", +I mean "stuff before each | pipe"), substituting the variables, +then looking at the text that's output. For example, + +@example + ls $(OUT)/$$l/*.html +@end example + +is going to print a list of all html files, in all languages, in +the build directory. Then more stuff happens to each of those +files (that's what xargs does). + +@item +@code{website-css:} +just copies files to the build dir. + +@item +@code{website-pictures, website-examples:} +more file copies, with an if statement to handle if you don't have +any generated pictures/examples. + +@item +@code{web-post:} +runs: + +@example +scripts/build/website_post.py +@end example + +which, it adds the "this page is translated in klingon" to the +bottom of html pages, and adds the google analytics javascript. +It also has hard-coded lilypond version numbers, which is Bad +(tm). + +@end itemize + +Here's a summary of what gets called, in what order, when we run +@code{make website} + +@example +website: + website-texinfo: + website-version: + creates version.itexi and weblinks.itexi + website-xrefs: + runs extract_texi_filenames.py + website-bibs: + creates bibliography files, described above + website-css: + copies css files + website-pictures: + copies pictures + website-examples: + copies examples + web-post: + runs website_post.py + Then some file copying +@end example