X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;f=Documentation%2Fcontributor%2Fbuild-notes.itexi;h=b84bdbef6067dbae72fd48c29e0ba2ab54a88c99;hb=a066a93ee74edebb9d238a1bac93c3bc7e8e6e4a;hp=6bea5628c83cc0885dfa72065a04adbb3c22fb4c;hpb=321beac428ad24824bbd35cc2421a3ee12c630ba;p=lilypond.git diff --git a/Documentation/contributor/build-notes.itexi b/Documentation/contributor/build-notes.itexi index 6bea5628c8..b84bdbef60 100644 --- a/Documentation/contributor/build-notes.itexi +++ b/Documentation/contributor/build-notes.itexi @@ -11,8 +11,10 @@ chapter.} @menu * Build system overview:: * Tips for working on the build system:: +* General build system notes:: * Doc build:: * Website build:: +* Building an Ubuntu distro:: @end menu @@ -72,6 +74,18 @@ echo "bbb" to the build system files in various places. This will let you track where the program is, in various points of the build. +PH note. There are lots of places where Make doesn't let you put +echo commands. My top tip for tracing how make runs is to put + +@example +$(error Some Text to display) +@end example + +This will stop make running and print the text @code{Some Text to +display}. + +End PH note. + @item First task: understand how @code{make website} works, @emph{without} the translations. Looking at the english-only @@ -82,14 +96,770 @@ covers about 5% of the whole thing, but even that will likely take @end itemize +@node General build system notes +@section General build system notes + +@menu +* How stepmake works:: +@end menu + +@node How stepmake works +@subsection How stepmake works + +Typing make website runs the file @file{GNUmakefile} from the +build directory. This only contains 3 lines: + +@example +depth = . +include config$(if $(conf),-$(conf),).make +include $(configure-srcdir)/GNUmakefile.in +@end example + +The variable @code{depth} is used throughout the make system to +track how far down the directory structure the make is. The first +include sets lots of variables but doesn't "do" anything. The +second runs the file @file{GNUmakefile.in} from the top level +source directory. + +This sets another load of variables, and then includes (i.e. +immediately runs) @file{stepmake.make} from the @file{make} +subdirectory. This sets a load of other variables, does some +testing to see if SCONS (another build tool?) is being used, and +then runs @file{make/config.make} - which doesn't seem to exist... + +GP: scons is indeed a different build tool; I think that Jan +experimented with it 5 years ago or something. It seems like we +still have bits and pieces of it floating around. + +Next, it runs @file{make/toplevel-version.make}, which sets the +version variables for major, minor, patch, stable, development and +mypatchlevel (which seems to be used for patch numbers for +non-stable versions only?). + +Next - @file{make/local.make}, which doesn't exist. + +Then a few more variable and the interesting comment: + +@example +# Don't try to outsmart us, you puny computer! +# Well, UGH. This only removes builtin rules from +@end example + +and then tests to see whether BUILTINS_REMOVED is defined. It +appears to be when I run make, and so +@file{stepmake/stepmake/no-builtin-rules.make} is run. The +comment at the head of this file says: + +@example +# UGH. GNU make comes with implicit rules. +# We don't want any of them, and can't force users to run +# --no-builtin-rules +@end example + +I've not studied that file at length, but assume it removes all +make's build-in rules (e.g. @file{*.c} files are run through the +GNU C compiler) - there's a lot of them in here, and a lot of +comments, and I'd guess most of it isn't needed. + +We return to @file{stepmake.make}, where we hit the make rule all: +The first line of this is: + +@smallexample +-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end smallexample + +which, when the variables are substituted, gives: + +@example +./make/generic-inclusions.make +./make/lilypond-inclusions.make. +@end example + +(Note - according to the make documentation, -include is only +different from include in that it doesn't produce any kind of +error message when the included file doesn't exist). + +And the first file doesn't exist. Nor the second. Next: + +@smallexample +-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expands to the following files: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make. +@end example + +One little feature to notice here - these are all absolute file +locations - the line prior to this used relative locations. And +none of these files exist, either. (Further note - I'm assuming +all these lines of make I'm following are autogenerated, but +that'll be something else to discover.) + +Next in @file{stepmake.make}: + +@smallexample +include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expands to: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make +/home/phil/lilypond-git/stepmake/stepmake/po-vars.make +/home/phil/lilypond-git/stepmake/stepmake/install-vars.make. +@end example + +Woo. They all exist (they should as there's no - in front of the +include). @file{generic-vars.make} sets loads of variables +(funnily enough). @file{toplevel-vars.make} is very short - one +line commented as @code{# override Generic_vars.make:} and 2 as +follows: + +@example +# urg? +include $(stepdir)/documentation-vars.make +@end example + +I assume the urg comment refers to the fact that this should +really just create more variables, but it actually sends us off to +@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}. + +That file is a 3 line variable setting one. + +@file{po-vars.make} has the one-line comment @code{# empty}, as +does @file{install-vars.make}. + +So now we're back to @file{stepmake.make}. + +The next lines are +: +@smallexample +# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH) +include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end smallexample + +and the include expands to: + +@example +include ./make/generic-vars.make ./make/lilypond-vars.make. +@end example + +These again set variables, and in some cases export them to allow +child @code{make} processes to use them. + +The final 4 lines of @file{stepmake.make} are: + +@smallexample +include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES))) +include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES))) +@end smallexample + +which expand as follows: + +@example +include ./make/generic-rules.make ./make/lilypond-rules.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make + /home/phil/lilypond-git/stepmake/stepmake/po-rules.make + /home/phil/lilypond-git/stepmake/stepmake/install-rules.make +include ./make/generic-targets.make ./make/lilypond-targets.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make + /home/phil/lilypond-git/stepmake/stepmake/po-targets.make + /home/phil/lilypond-git/stepmake/stepmake/install-targets.make +@end example + +@file{lilypond-rules.make} is @code{#empty} + +@file{generic-rules.make} does seem to have 2 rules in it. They +are: + +@example +$(outdir)/%.ly: %.lym4 + $(M4) $< | sed "s/\`/,/g" > $@ + +$(outdir)/%: %.in + rm -f $@ + cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@ +@end example + +I believe the first rule is for *.ly files, and has a prerequisite +that *.lym4 files must be built first. The recipe is @code{m4 | +sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge +can comment on exactly what the rules mean/do. + +@file{toplevel-rules.make} is @code{#empty} + +@file{po-rules.make} is @code{#empty} + +@file{install-rules.make} is @code{#empty} + +@file{generic-targets.make} contains 2 lines of comments. + +@file{lilypond-targets.make} contains only: + +@example +## TODO: fail dist or web if no \version present. +check-version: + grep -L version $(LY_FILES) +@end example + +@file{stepmake/generic-targets.make} contains lots of rules - too +many to list here - it seems to be the main file for rules. (FWIW +I haven't actually found a rule for website: anywhere, although +it clearly exists. I have also found that you can display a rule +in the terminal by typing, say @code{make -n website}. This is +probably common knowledge. + +@file{stepmake/toplevel-targets.make} adds a load of other (and +occasionally the same) rules to the gernric-targets. + +@file{stepmake/po-targets.make} is rules for po* makes. + +@file{stepmake/install-targets.make} has rules for local-install*. + +And that's the end of stepmake.make. Back to +@file{GNUmakefile.in}. + +A bit more info from 27 March. I've put some error traces into +@code{GNUmakefile} in the build directory, and it looks like the +following lines actually cause the make to run (putting an error +call above them - no make; below them - make): + +@smallexample +ifeq ($(out),www) +# All web targets, except info image symlinks and info docs are +# installed in non-recursing target from TOP-SRC-DIR +install-WWW: + -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir) + rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir) + $(MAKE) -C Documentation omf-local-install +@end smallexample + +I don't currently understand the @code{ifeq}, since @code{$(out)} +is empty at this point, but the line starting @w{@code{-$(INSTALL)}} +translates to: + +@example +-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \ + -c -m 755 -d /usr/local/share/doc/lilypond/html +@end example + +End of work for Sunday 27th. + +Another alterative approach to understanding the website build +would be to redirect @code{make -n website} and @code{make website} +to a text file and work through a) what it does and b) where the +errors are occurring. + +GP: wow, all the above is much more complicated than I've ever +looked at stuff -- I tend to do a "back first" approach (where I +begin from the command-line that I want to modify, figure out +where it's generated, and then figure out how to change the +generated command-line), rather than a "front first" (where you +begin from the "make" command). + @node Doc build @section Doc build @menu +* The function of make doc:: * Building a bibliography:: @end menu +@node The function of make doc +@subsection The function of make doc + +The following is a set of notes on how make doc functions. + +Preliminary question to be answered some time: where do all the +GNUmakefiles come from. They're in the build directory, but this +is not part of source. Must be the configure script. And it +looks like this comes from autogen.sh. Must at some point kill +the whole git directory, repull and see what is created when. + +Anyway, here's how make doc progresses: + +This is the build dependency tree from +@file{stepmake/stepmake/generic-targets.make}: + +@example +doc: doc-stage-1 + doc-stage-1: + $(MAKE) -C $(depth)/scripts/build out= + $(MAKE) out=www WWW-1 + WWW-1: local-WWW-1 + $(LOOP) + $(MAKE) out=www WWW-2 + WWW-2: local-WWW-2 + $(LOOP) + $(MAKE) out=www WWW-post +@end example + +@example +MAKE = make --no-builtin-rules +-C = Change to directory before make +@end example + +doc-stage-1 does lots of opening and looking in files, but no +processing. + +@example +Variable LOOP = + ++ make PACKAGE=LILYPOND package=lilypond -C python +&& make PACKAGE=LILYPOND package=lilypond -C scripts +&& make PACKAGE=LILYPOND package=lilypond -C flower +&& make PACKAGE=LILYPOND package=lilypond -C lily +&& make PACKAGE=LILYPOND package=lilypond -C mf +&& make PACKAGE=LILYPOND package=lilypond -C ly +&& make PACKAGE=LILYPOND package=lilypond -C tex +&& make PACKAGE=LILYPOND package=lilypond -C ps +&& make PACKAGE=LILYPOND package=lilypond -C scm +&& make PACKAGE=LILYPOND package=lilypond -C po +&& make PACKAGE=LILYPOND package=lilypond -C make +&& make PACKAGE=LILYPOND package=lilypond -C elisp +&& make PACKAGE=LILYPOND package=lilypond -C vim +&& make PACKAGE=LILYPOND package=lilypond -C input +&& make PACKAGE=LILYPOND package=lilypond -C stepmake +&& make PACKAGE=LILYPOND package=lilypond -C Documentation +&& true +@end example + +From git grep: + +stepmake/stepmake/generic-vars.make has this: + +@smallexample +LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true +@end smallexample + +$@ is the name of the target - WWW-1 in this case. + +In GNUmakefile.in we find: + +@example +SUBDIRS = python scripts \ + flower lily \ + mf ly \ + tex ps scm \ + po make \ + elisp vim \ + input \ + stepmake $(documentation-dir) +@end example + +So that's how we get the main make loop... + +That loop expands like this: + +@example +make PACKAGE=LILYPOND package=lilypond -C python WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C po WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C make WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C input WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 && +make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 && +true +@end example + +The directories up to and including vim produce no effect with +make in non-debug mode, although debug does show lots of action. + +@file{git/build/input/GNUmakefile} is: + +@example +depth=../ +include $(depth)/config$(if $(conf),-$(conf),).make +include $(configure-srcdir)/./input/GNUmakefile +MODULE_INCLUDES += $(src-dir)/$(outbase) +@end example + +The first include is: + +@example +..//config.make +@end example + +(note the // which is strictly wrong) + +which has lots of variables to set, but no action occurs. + +The second is: + +@example +lilypond-git/./input/GNUmakefile +@end example + +which similarly doesn't create any actual action. + +An error message at the end of build/input/GNUmakefile stops +make processing before it moves on to regression - so where does +that come from? + +And the answer is - make processes all directories in the +directory it's entered (with some exceptions like out and out-www) +and so it changes to /regression. + +It then seems to consider whether it needs to make/remake loads of +makefiles. Don't understand this yet. Possibly these are all the +makefiles it's processing, and it always checks they're up to date +before processing other files? + +Could be correct - some of this output is: + +@example +Must remake target `../../make/ly-inclusions.make'. +Failed to remake target file `../../make/ly-inclusions.make'. +@end example + +Having decided that, it then leaves the directory and re-executes: + +@example +make PACKAGE=LILYPOND package=lilypond -C regression WWW-1 +@end example + +The top of this make is: + +@example +This program built for i486-pc-linux-gnu +Reading makefiles... +Reading makefile `GNUmakefile'... +Reading makefile `../..//config.make' (search path) (no ~ expansion)... +@end example + +which looks like it's re-reading all its known makefiles to check +they're up to date. + +(From the make manual: + +To this end, after reading in all makefiles, make will consider each as a goal target and +attempt to update it. If a makefile has a rule which says how to update it (found either +in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10 +[Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have +been checked, if any have actually been changed, make starts with a clean slate and reads +all the makefiles over again. (It will also attempt to update each of them over again, but +normally this will not change them again, since they are already up to date.) + +So my assumption seems correct) + +There appear to be about 74 of them. After all the makefile +checking, we get this: + +@smallexample +Updating goal targets.... +Considering target file `WWW-1'. +File `WWW-1' does not exist. +Considering target file `local-WWW-1'. +File `local-WWW-1' does not exist. +Considering target file `out-www/collated-files.texi'. +File `out-www/collated-files.texi' does not exist. +Looking for an implicit rule for `out-www/collated-files.texi'. +Trying pattern rule with stem `collated-files.texi'. +Trying implicit prerequisite `collated-files.texi.in'. +Trying pattern rule with stem `collated-files.texi'. +Trying implicit prerequisite `collated-files.texi.in'. +Trying pattern rule with stem `collated-files'. +Trying implicit prerequisite `collated-files.tely'. +Trying pattern rule with stem `collated-files'. +Trying implicit prerequisite `out-www/collated-files.tely'. +Trying rule prerequisite `out-www/version.itexi'. +Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi' +@end smallexample + +grep finds this if searching for local-WWW-1: + +@example +make/lysdoc-targets.make: + local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf +@end example + +which means that local-WWW-1 depends on coll*.texi and coll*.pdf +and so these will need to be checked to see if they're up to date. +So make needs to find rules for both of those and (as it says) it +certainly needs to make coll*.texi, since it doesn't exist. + +In ly-rules.make we have: + +@example +.SUFFIXES: .doc .tely .texi .ly +@end example + +which I'll work out at some point, and also this rule: + +@smallexample +$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES) + LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $< +@end smallexample + +Note that the recipe is a very long line - it could probably +benefit from splitting. The same makefile also has: + +@smallexample +$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES) + LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $< +@end smallexample + +@noindent +which seems to be an almost exact duplicate. Whatever, the first +one is executed first. Have not checked if the second executes. + +The first recipe translates as this: + +@example +LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \ + --output=./out-www --format= --lily-output-dir \ + /home/phil/lilypond-git/build/out/lybook-db +@end example + +@noindent +if we stop the build with an $(error), but I think this is because +we need to allow it to process the dependencies first. It looks +like foo.texi is shown as being dependent on foo.tely, plus a load +of other files. + +@example +DOCUMENTATION_LOCALE_TARGET is blank +INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \ + /home/phil/lilypond-git/scm/autochange.scm +@end example + +plus 10s (100s?) of other .scm files. + +@example +SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \ + /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly +@end example + +ditto .ly files. This does seem a teency bit wrong - it looks like +the .ly and .scm files have been interchanged. ly-vars.make has +these 2 lines: + +@example +INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm) +SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly) +@end example + +Looks like a bug..... + +So it now works its way through all these files, checking if they +need to be remade. This is 100s of lines of the debug listing, +although none in the normal list. Clearly none has to be made +since they're source files. It concludes: + +@example +Must remake target `out-www/collated-files.tely' +@end example + +@file{lysdoc-rules.make} has this: + +@smallexample +$(outdir)/collated-files.tely: $(COLLATED_FILES) + $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^ +@end smallexample + +@file{lysdoc-vars.make} has: + +@example +COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) ) +@end example + +We find that: + +@example +TEXINFO_SOURCES = AAA-intro-regression.tely +OUT_LY_FILES is empty +@end example + +so LY_FILES has the big long list of all the .ly files in the +regression directory. + +This kicks off + +@example +/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely +@end example + +with a list of all the files in the regression test directory. This +should (I believe) create the file collated-files.tely. + +So the next rule in make is for @file{version.itexi}, and make duly +checks this. There's a rule in @file{doc-i18n-root-rules.make} that this +depends on @file{git/VERSION}: + +@smallexample +$(outdir)/version.%: $(top-src-dir)/VERSION + $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@ +@end smallexample + +This causes create-version-itexi.py to run and create +version.itexi. + +Once that's done, all the other *.scm and *.ly files are checked +and since they have no rules associated, they aren't remade (just +as well for source files, really). Since version.itexi was remade +make concludes that collated-files.texi must be remade. To do +this, it runs lilypond-book.py on collated-files.tely, as below: + +@example +LILYPOND_VERSION=2.15.0 + /usr/bin/python + /home/phil/lilypond-git/scripts/lilypond-book.py + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + --process='/home/phil/lilypond-git/build/out/bin/lilypond + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www + -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + -dbackend=eps + --formats=ps,png,pdf + -dinclude-eps-fonts + -dgs-load-fonts + --header=doctitle + --header=doctitlecs + --header=doctitlede + --header=doctitlees + --header=doctitlefr + --header=doctitlehu + --header=doctitleit + --header=doctitleja + --header=doctitlenl + --header=doctitlezh + --header=texidoc + --header=texidoccs + --header=texidocde + --header=texidoces + --header=texidocfr + --header=texidochu + --header=texidocit + --header=texidocja + --header=texidocnl + --header=texidoczh + -dcheck-internal-types + -ddump-signatures + -danti-alias-factor=2' + --output=./out-www + --format=texi-html + --verbose + --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db + out-www/collated-files.tely +@end example + +So - lilypond-book runs on: + +@example +input/regression/out-www/collated-files.tely +@end example + + +Note the --verbose flag - this is from the make variable +LILYPOND_BOOK_VERBOSE which is added to the make variable +LILYPOND_BOOK_FLAGS. + +Now found the invocation to write some of the image files. It's +like this: + +@example +/home/phil/lilypond-git/build/out/bin/lilypond + -I /home/phil/lilypond-git/input/regression/ + -I ./out-www -I /home/phil/lilypond-git/input + -I /home/phil/lilypond-git/Documentation + -I /home/phil/lilypond-git/Documentation/snippets + -I /home/phil/lilypond-git/input/regression/ + -I /home/phil/lilypond-git/Documentation/included/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/build/mf/out/ + -I /home/phil/lilypond-git/Documentation/pictures + -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www + -dbackend=eps + --formats=ps,png,pdf + -dinclude-eps-fonts + -dgs-load-fonts + --header=doctitle + --header=doctitlecs + --header=doctitlede + --header=doctitlees + --header=doctitlefr + --header=doctitlehu + --header=doctitleit + --header=doctitleja + --header=doctitlenl + --header=doctitlezh + --header=texidoc + --header=texidoccs + --header=texidocde + --header=texidoces + --header=texidocfr + --header=texidochu + --header=texidocit + --header=texidocja + --header=texidocnl + --header=texidoczh + -dcheck-internal-types + -ddump-signatures + -danti-alias-factor=2 + -I "/home/phil/lilypond-git/build/out/lybook-db" + -I "/home/phil/lilypond-git/build/input/regression" + -I "/home/phil/lilypond-git/input/regression" + -I "/home/phil/lilypond-git/build/input/regression/out-www" + -I "/home/phil/lilypond-git/input" + -I "/home/phil/lilypond-git/Documentation" + -I "/home/phil/lilypond-git/Documentation/snippets" + -I "/home/phil/lilypond-git/input/regression" + -I "/home/phil/lilypond-git/Documentation/included" + -I "/home/phil/lilypond-git/build/mf/out" + -I "/home/phil/lilypond-git/build/mf/out" + -I "/home/phil/lilypond-git/Documentation/pictures" + -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www" + --formats=eps + --verbose + -deps-box-padding=3.000000 + -dread-file-list + -dno-strip-output-dir + "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"' +@end example + +Note the --verbose. This causes 100s of lines of Lily debug output. +But at present I can't work out where the flag comes from. Later. + + @node Building a bibliography @subsection Building a bibliography @@ -192,119 +962,279 @@ The file @file{lily-bib.bst} also has fairly extensive commenting. Start here: @file{make/website.make} -Typing make website runs the file @file{GNUmakefile} from the -build directory. This only contains 3 lines: +The overall build system begins with @ref{How stepmake works}. +Summary: when you type @code{make website} this ends up running +@file{GNUmakefile.in} in the @file{git} directory. Right at the +bottom, this has the lines: @example -depth = . -include config$(if $(conf),-$(conf),).make -include $(configure-srcdir)/GNUmakefile.in +# we want this separate for security; see CG 4.2. -gp +website: + $(MAKE) config_make=$(config_make) \ + top-src-dir=$(top-src-dir) \ + -f $(top-src-dir)/make/website.make \ + website @end example -The variable @code{depth} is used throughout the make system to -track how far down the directory structure the make is. The first -include sets lots of variables but doesn't "do" anything. The -second runs the file @file{GNUmakefile.in} from the top level -source directory. +On my system this expands to: -This sets another load of variables, and then includes (i.e. -immediately runs) @file{stepmake.make} from the @file{make} -subdirectory. This sets a load of other variables, does some -testing to see if SCONS (another build tool?) is being used, and -then runs @file{make/config.make} - which doesn't seem to exist... +@example +make --no-builtin-rules config_make=./config.make \ + top-src-dir=/home/phil/lilypond-git \ + -f /home/phil/lilypond-git/make/website.make \ + website +@end example -Next, it runs @file{make/toplevel-version.make}, which sets the -version variables for major, minor, patch, stable, development and -mypatchlevel (which seems to be used for patch numbers for -non-stable versions only?). +We see that the @code{$(MAKE)} expands to +@code{make --no-builtin-rules} which is how @code{MAKE} is +defined higher up the makefile. The -f switch defines the +makefile to be used - in this case +@file{git/make/website.make}. That's where all the action +happens. + +We believe that note that *none* of the variables that +are loaded (from depth to version numbers to whatever) are used in +@file{website.make}. Instead, @file{website.make} sets up its own +variables at the top of the file. If you're wondering if there's +some smart reason for this, then the answer is "no". It's because +I (GP) didn't know/trust the original variables when I was writing +that file. -Next - @file{make/local.make}, which doesn't exist. +Website build includes @ref{Building a bibliography}. -Then a few more variable and the interesting comment: +@subsubheading Output from @code{make -n website} + +Sorry, including this output directly produces problems in the +build system. Please run: @example -# Don't try to outsmart us, you puny computer! -# Well, UGH. This only removes builtin rules from +make -n website &> my-file.txt @end example -and then tests to see whether BUILTINS_REMOVED is defined. It -appears to be when I run make, and so -@file{stepmake/stepmake/no-builtin-rules.make} is run. The -comment at the head of this file says: +to see the full output from the make. -@example -# UGH. GNU make comes with implicit rules. -# We don't want any of them, and can't force users to run -# --no-builtin-rules -@end example +@subsubheading website.make variables -I've not studied that file at length, but assume it removes all -make's build-in rules (e.g. @file{*.c} files are run through the -GNU C compiler) - there's a lot of them in here, and a lot of -comments, and I'd guess most of it isn't needed. +The file begins by setting up some variables. These +may/might/probably mirror existing variables, but lacking any docs +about those variables, I thought it would be simpler to keep +everything in the same file. -We return to @file{stepmake.make}, where we hit the make rule all: -The first line of this is: +Note that for security reasons, we @strong{don't} call scripts in +the git dir when building on the web server. See @ref{Uploading +and security}. So we definitely want to keep those definitions +for the WEBSITE_ONLY_BUILD. + +After some split WEBSITE_ONLY_BUILD vs. normal build definitions, +there's another bunch of lines setting up generic variables. + +@subsubheading website.make building parts + +Parts of @file{website.make}: +@itemize + +@item +@code{website:} +this is the "master" rule. It calls the other rules in order, +then copies some extra files around - see below for further +of the process it produces. + +@item +@code{website-version}: +this calls the python scripts below: + +@itemize +@item @example --include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES))) +scripts/build/create-version-itexi.py @end example -which, when the variables are substituted, gives: +This writes a @@version, @@versionStable, and @@versionDevel based +on the top-level VERSIONS file, to +@code{out-website/version.itexi} +@item @example -./make/generic-inclusions.make -./make/lilypond-inclusions.make. +scripts/build/create-weblinks-itexi.py @end example -(Note - according to the make documentation, -include is only -different from include in that it doesn't produce any kind of -error message when the included file doesn't exist). +This creates a ton of macros in @code{out-website/weblinks.itexi}. +Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows, +@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh. -And the first file doesn't exist. Nor the second. Next: +It's quite monstrous because it deals with combinations of +stable/devel, source/docs, lang/lang/lang*10, etc. -@example --include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES))) -@end example +@end itemize -which expands to the following files: + +@item +@code{website-xrefs:} +creates files used for complicated "out-of-build" references to +@code{out-website/*.xref-map} + +If you just write @@ref@{@}, then all's groovy and we wouldn't +need this. But if you write @@rlearning@{@}, then our custom +texi2html init file needs to know about our custom xref file +format, which tells our custom texi2html init file how to create +the link. + +GP: we should have a separate @@node to discuss xrefs. Also, take a +quick look at a generated xref file -- it's basically just a list +of @@node's [sic teenager pluralization rule] from the file. + +@item +@code{website-bib:} +generates the bibliography texinfo files from the .bib files - in +the case of the website build these are @file{others-did.bib} and +@file{we-wrote.bib}. + +@item +@code{website-texinfo:} +this is the main part; it calles texi2html to generate the actual +html. It also has a ton of options to texi2html to pass info to +our custom init file. + +The file actually built is called @file{web.texi}, and is either +in the @file{Documentation} directory, or a sub-directory specific +to the language. + +The options file is @file{/Documentation/lilypond-texi2html.init}. +This contains *lots* of option and configuration stuff, and also +includes the line: + +@smallexample +print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n"; +@end smallexample + +This is where one of the console messages is generated. + +We have somewhere between 2-4 different ways "to pass info to our +custom init file". This is highly Not Good (tm), but that's how +things work at the moment. + +After texi2html, it does some black magick to deal with +untranslated nodes in the translations. Despite writing that +part, I can't remember how it works. But in theory, you could +figure it out by copy&pasting each part of the command (by "part", +I mean "stuff before each | pipe"), substituting the variables, +then looking at the text that's output. For example, @example -/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make -/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make. + ls $(OUT)/$$l/*.html @end example -One little feature to notice here - these are all absolute file -locations - the line prior to this used relative locations. And -none of these files exist, either. (Further note - I'm assuming -all these lines of make I'm following are autogenerated, but -that'll be something else to discover.) +is going to print a list of all html files, in all languages, in +the build directory. Then more stuff happens to each of those +files (that's what xargs does). + +@item +@code{website-css:} +just copies files to the build dir. + +@item +@code{website-pictures, website-examples:} +more file copies, with an if statement to handle if you don't have +any generated pictures/examples. -Next: +@item +@code{web-post:} +runs: @example -include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES))) +scripts/build/website_post.py @end example -which expands to: +which, it adds the "this page is translated in klingon" to the +bottom of html pages, and adds the google analytics javascript. +It also has hard-coded lilypond version numbers, which is Bad +(tm). + +@end itemize + +Here's a summary of what gets called, in what order, when we run +@code{make website} @example -/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make -/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make -/home/phil/lilypond-git/stepmake/stepmake/po-vars.make -/home/phil/lilypond-git/stepmake/stepmake/install-vars.make. +website: + website-texinfo: + website-version: + creates version.itexi and weblinks.itexi + website-xrefs: + runs extract_texi_filenames.py + website-bibs: + creates bibliography files, described above + website-css: + copies css files + website-pictures: + copies pictures + website-examples: + copies examples + web-post: + runs website_post.py + Then some file copying @end example -Woo. They all exist (they should - no - in front of the include). +@node Building an Ubuntu distro +@section Building an Ubuntu distro -Next step will be to work out what these do. -That's my 30 minutes for Monday. +Here's the short instruction on how to create lilybuntu iso image +(Jonathan Kulp did this on a spare drive, +but he supposes it can be done in a VM too): -Website build includes @ref{Building a bibliography}. +@enumerate +@item +Install ubuntu, reboot. +@item +Run all updates, reboot if asked. +@item +Enable src repos, refresh package lists. +@item +Install LilyPond build deps: +@example +sudo apt-get build-dep lilypond +@end example +@item +Install git and autoconf: +@example +sudo apt-get install git-core gitk autoconf +@end example +@item +Test to see whether everything works fine now: +@enumerate +@item +use @command{lily-git.tcl} to grab source files +@item +go to source dir and do +@example +"./autogen.sh" ; make ; make doc +@end example +@item +if all compiles, move on to iso creation... +@end enumerate +@item +Download & install "remastersys": +@uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/} +@item +Copy @command{lily-git.tcl} script file into @file{/etc/skel/}. +@item +Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name, +default live session username, etc). +@item +Remove non-essential desktop software as desired. +@item +Create iso: +@example +sudo remastersys dist +@end example +New iso is in @file{/home/remastersys/remastersys/}. +@item +Test iso by installing in VM and repeating steps above for +getting source files and building lp and docs. +@end enumerate