1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
20 @node Build system overview
21 @section Build system overview
23 Build system is currently GNU make, with an extra "stepmake" layer
24 on top. Look at files in @file{make/} and @file{stepmake/} and
25 all @file{GNUmakefile}s.
27 There is wide-spread dissatisfaction with this system, and we are
28 considering changing. This would be a huge undertaking (estimated
29 200+ hours). This change will probably involve not using GNU make
30 any more -- but a discussion about the precise build system will
31 have to wait. Before we reach that point, we need to figure out
32 (at least approximately) what the current build system does.
34 Fundamentally, a build system does two things:
38 Constructs command-line commands, for example:
42 --tons --of --options \
45 --more --imperial --and --metric --tons --of --options \
50 If there was a previous build, it decides which parts of the
51 system need to be rebuilt.
55 When I try to do anything in the build system, it helps to remind
56 myself of this. The "end result" is just a series of command-line
57 commands. All the black magick is just an attempt to construct
60 @node Tips for working on the build system
61 @section Tips for working on the build system
73 to the build system files in various places. This will let you
74 track where the program is, in various points of the build.
76 PH note. There are lots of places where Make doesn't let you put
77 echo commands. My top tip for tracing how make runs is to put
80 $(error Some Text to display)
83 This will stop make running and print the text @code{Some Text to
89 First task: understand how @code{make website} works,
90 @emph{without} the translations. Looking at the english-only
91 website is the best introduction to the build system... it only
92 covers about 5% of the whole thing, but even that will likely take
98 @node General build system notes
99 @section General build system notes
102 * How stepmake works::
105 @node How stepmake works
106 @subsection How stepmake works
108 Typing make website runs the file @file{GNUmakefile} from the
109 build directory. This only contains 3 lines:
113 include config$(if $(conf),-$(conf),).make
114 include $(configure-srcdir)/GNUmakefile.in
117 The variable @code{depth} is used throughout the make system to
118 track how far down the directory structure the make is. The first
119 include sets lots of variables but doesn't "do" anything. Default
120 values for these variables are automatically detected at the
121 ./configure step, which creates the file @file{config.make}.
122 The second include runs the file @file{GNUmakefile.in} from
123 the top level source directory.
125 This sets another load of variables, and then includes (i.e.
126 immediately runs) @file{stepmake.make} from the @file{make}
127 subdirectory. This sets a load of other variables, does some
128 testing to see if SCONS (another build tool?) is being used, and
129 then runs @file{make/config.make} - which doesn't seem to exist...
131 GP: scons is indeed a different build tool; I think that Jan
132 experimented with it 5 years ago or something. It seems like we
133 still have bits and pieces of it floating around.
135 Next, it runs @file{make/toplevel-version.make}, which sets the
136 version variables for major, minor, patch, stable, development and
137 mypatchlevel (which seems to be used for patch numbers for
138 non-stable versions only?).
140 Next - @file{make/local.make}, which doesn't exist.
142 Then a few more variable and the interesting comment:
145 # Don't try to outsmart us, you puny computer!
146 # Well, UGH. This only removes builtin rules from
149 and then tests to see whether BUILTINS_REMOVED is defined. It
150 appears to be when I run make, and so
151 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
152 comment at the head of this file says:
155 # UGH. GNU make comes with implicit rules.
156 # We don't want any of them, and can't force users to run
160 I've not studied that file at length, but assume it removes all
161 make's build-in rules (e.g. @file{*.c} files are run through the
162 GNU C compiler) - there's a lot of them in here, and a lot of
163 comments, and I'd guess most of it isn't needed.
165 We return to @file{stepmake.make}, where we hit the make rule all:
166 The first line of this is:
169 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
172 which, when the variables are substituted, gives:
175 ./make/generic-inclusions.make
176 ./make/lilypond-inclusions.make.
179 (Note - according to the make documentation, -include is only
180 different from include in that it doesn't produce any kind of
181 error message when the included file doesn't exist).
183 And the first file doesn't exist. Nor the second. Next:
186 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
189 which expands to the following files:
192 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
193 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
195 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
198 One little feature to notice here - these are all absolute file
199 locations - the line prior to this used relative locations. And
200 none of these files exist, either.
202 (Further note - I'm assuming all these lines of make I'm following are
203 autogenerated, but that'll be something else to discover.)
205 JM: @emph{``No, these lines are not useful in LilyPond (this is why
206 you think they are autogenerated), but they are part of StepMake,
207 which was meant to be a package to be installed as a build system over
208 autoconf/make in software project source trees.''}
210 Next in @file{stepmake.make}:
213 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
219 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
220 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
221 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
222 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
225 Woo. They all exist (they should as there's no - in front of the
226 include). @file{generic-vars.make} sets loads of variables
227 (funnily enough). @file{toplevel-vars.make} is very short - one
228 line commented as @code{# override Generic_vars.make:} and 2 as
233 include $(stepdir)/documentation-vars.make
236 I assume the urg comment refers to the fact that this should
237 really just create more variables, but it actually sends us off to
238 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
240 That file is a 3 line variable setting one.
242 @file{po-vars.make} has the one-line comment @code{# empty}, as
243 does @file{install-vars.make}.
245 So now we're back to @file{stepmake.make}.
250 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
251 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
254 and the include expands to:
257 include ./make/generic-vars.make ./make/lilypond-vars.make.
260 These again set variables, and in some cases export them to allow
261 child @code{make} processes to use them.
263 The final 4 lines of @file{stepmake.make} are:
266 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
267 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
268 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
269 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
272 which expand as follows:
275 include ./make/generic-rules.make ./make/lilypond-rules.make
277 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
278 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
279 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
280 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
281 include ./make/generic-targets.make ./make/lilypond-targets.make
283 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
284 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
285 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
286 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
289 @file{lilypond-rules.make} is @code{#empty}
291 @file{generic-rules.make} does seem to have 2 rules in it. They
295 $(outdir)/%.ly: %.lym4
296 $(M4) $< | sed "s/\`/,/g" > $@@
300 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@@
303 I believe the first rule is for *.ly files, and has a prerequisite
304 that *.lym4 files must be built first. The recipe is @code{m4 |
305 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
306 can comment on exactly what the rules mean/do.
308 @file{toplevel-rules.make} is @code{#empty}
310 @file{po-rules.make} is @code{#empty}
312 @file{install-rules.make} is @code{#empty}
314 @file{generic-targets.make} contains 2 lines of comments.
316 @file{lilypond-targets.make} contains only:
319 ## TODO: fail dist or web if no \version present.
321 grep -L version $(LY_FILES)
324 @file{stepmake/generic-targets.make} contains lots of rules - too
325 many to list here - it seems to be the main file for rules. (FWIW
326 I haven't actually found a rule for website: anywhere, although
327 it clearly exists. I have also found that you can display a rule
328 in the terminal by typing, say @code{make -n website}. This is
329 probably common knowledge.
331 @file{stepmake/toplevel-targets.make} adds a load of other (and
332 occasionally the same) rules to the gernric-targets.
334 @file{stepmake/po-targets.make} is rules for po* makes.
336 @file{stepmake/install-targets.make} has rules for local-install*.
338 And that's the end of stepmake.make. Back to
339 @file{GNUmakefile.in}.
341 A bit more info from 27 March. I've put some error traces into
342 @code{GNUmakefile} in the build directory, and it looks like the
343 following lines actually cause the make to run (putting an error
344 call above them - no make; below them - make):
348 # All web targets, except info image symlinks and info docs are
349 # installed in non-recursing target from TOP-SRC-DIR
351 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
352 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
353 $(MAKE) -C Documentation omf-local-install
356 I don't currently understand the @code{ifeq}, since @code{$(out)}
357 is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
361 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
362 -c -m 755 -d /usr/local/share/doc/lilypond/html
365 End of work for Sunday 27th.
367 Another alterative approach to understanding the website build
368 would be to redirect @code{make -n website} and @code{make website}
369 to a text file and work through a) what it does and b) where the
370 errors are occurring.
372 GP: wow, all the above is much more complicated than I've ever
373 looked at stuff -- I tend to do a "back first" approach (where I
374 begin from the command-line that I want to modify, figure out
375 where it's generated, and then figure out how to change the
376 generated command-line), rather than a "front first" (where you
377 begin from the "make" command).
384 * The function of make doc::
385 * Building a bibliography::
388 @node The function of make doc
389 @subsection The function of make doc
391 The following is a set of notes on how make doc functions.
393 Preliminary question to be answered some time: where do all the
394 GNUmakefiles come from. They're in the build directory, but this
395 is not part of source. Must be the configure script. And it
396 looks like this comes from autogen.sh. Must at some point kill
397 the whole git directory, repull and see what is created when.
399 Anyway, here's how make doc progresses:
401 This is the build dependency tree from
402 @file{stepmake/stepmake/generic-targets.make}:
407 $(MAKE) -C $(depth)/scripts/build out=
408 $(MAKE) out=www WWW-1
411 $(MAKE) out=www WWW-2
414 $(MAKE) out=www WWW-post
418 MAKE = make --no-builtin-rules
419 -C = Change to directory before make
422 doc-stage-1 does lots of opening and looking in files, but no
428 + make PACKAGE=LILYPOND package=lilypond -C python
429 && make PACKAGE=LILYPOND package=lilypond -C scripts
430 && make PACKAGE=LILYPOND package=lilypond -C flower
431 && make PACKAGE=LILYPOND package=lilypond -C lily
432 && make PACKAGE=LILYPOND package=lilypond -C mf
433 && make PACKAGE=LILYPOND package=lilypond -C ly
434 && make PACKAGE=LILYPOND package=lilypond -C tex
435 && make PACKAGE=LILYPOND package=lilypond -C ps
436 && make PACKAGE=LILYPOND package=lilypond -C scm
437 && make PACKAGE=LILYPOND package=lilypond -C po
438 && make PACKAGE=LILYPOND package=lilypond -C make
439 && make PACKAGE=LILYPOND package=lilypond -C elisp
440 && make PACKAGE=LILYPOND package=lilypond -C vim
441 && make PACKAGE=LILYPOND package=lilypond -C input
442 && make PACKAGE=LILYPOND package=lilypond -C stepmake
443 && make PACKAGE=LILYPOND package=lilypond -C Documentation
449 stepmake/stepmake/generic-vars.make has this:
452 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@@ &&) true
455 $@@ is the name of the target - WWW-1 in this case.
457 In GNUmakefile.in we find:
460 SUBDIRS = python scripts \
467 stepmake $(documentation-dir)
470 So that's how we get the main make loop...
472 That loop expands like this:
475 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
476 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
477 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
484 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
485 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
486 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
487 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
488 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
489 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
490 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
494 The directories up to and including vim produce no effect with
495 make in non-debug mode, although debug does show lots of action.
497 @file{git/build/input/GNUmakefile} is:
501 include $(depth)/config$(if $(conf),-$(conf),).make
502 include $(configure-srcdir)/./input/GNUmakefile
503 MODULE_INCLUDES += $(src-dir)/$(outbase)
506 The first include is:
512 (note the // which is strictly wrong)
514 which has lots of variables to set, but no action occurs.
519 lilypond-git/./input/GNUmakefile
522 which similarly doesn't create any actual action.
524 An error message at the end of build/input/GNUmakefile stops
525 make processing before it moves on to regression - so where does
528 And the answer is - make processes all directories in the
529 directory it's entered (with some exceptions like out and out-www)
530 and so it changes to /regression.
532 It then seems to consider whether it needs to make/remake loads of
533 makefiles. Don't understand this yet. Possibly these are all the
534 makefiles it's processing, and it always checks they're up to date
535 before processing other files?
537 Could be correct - some of this output is:
540 Must remake target `../../make/ly-inclusions.make'.
541 Failed to remake target file `../../make/ly-inclusions.make'.
544 Having decided that, it then leaves the directory and re-executes:
547 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
550 The top of this make is:
553 This program built for i486-pc-linux-gnu
555 Reading makefile `GNUmakefile'...
556 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
559 which looks like it's re-reading all its known makefiles to check
562 (From the make manual:
564 To this end, after reading in all makefiles, make will consider each
565 as a goal target and attempt to update it. If a makefile has a rule
566 which says how to update it (found either in that very makefile or in
567 another one) or if an implicit rule applies to it (see Chapter 10
568 [Using Implicit Rules], page 103), it will be updated if
569 necessary. After all makefiles have been checked, if any have actually
570 been changed, make starts with a clean slate and reads all the
571 makefiles over again. (It will also attempt to update each of them
572 over again, but normally this will not change them again, since they
573 are already up to date.)
575 So my assumption seems correct)
577 There appear to be about 74 of them. After all the makefile
578 checking, we get this:
581 Updating goal targets....
582 Considering target file `WWW-1'.
583 File `WWW-1' does not exist.
584 Considering target file `local-WWW-1'.
585 File `local-WWW-1' does not exist.
586 Considering target file `out-www/collated-files.texi'.
587 File `out-www/collated-files.texi' does not exist.
588 Looking for an implicit rule for `out-www/collated-files.texi'.
589 Trying pattern rule with stem `collated-files.texi'.
590 Trying implicit prerequisite `collated-files.texi.in'.
591 Trying pattern rule with stem `collated-files.texi'.
592 Trying implicit prerequisite `collated-files.texi.in'.
593 Trying pattern rule with stem `collated-files'.
594 Trying implicit prerequisite `collated-files.tely'.
595 Trying pattern rule with stem `collated-files'.
596 Trying implicit prerequisite `out-www/collated-files.tely'.
597 Trying rule prerequisite `out-www/version.itexi'.
598 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
601 grep finds this if searching for local-WWW-1:
604 make/lysdoc-targets.make:
605 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
608 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
609 and so these will need to be checked to see if they're up to date.
610 So make needs to find rules for both of those and (as it says) it
611 certainly needs to make coll*.texi, since it doesn't exist.
613 In ly-rules.make we have:
616 .SUFFIXES: .doc .tely .texi .ly
619 which I'll work out at some point, and also this rule:
622 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
623 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
626 Note that the recipe is a very long line - it could probably
627 benefit from splitting. The same makefile also has:
630 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
631 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
635 which seems to be an almost exact duplicate. Whatever, the first
636 one is executed first. Have not checked if the second executes.
638 The first recipe translates as this:
641 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
642 --output=./out-www --format= --lily-output-dir \
643 /home/phil/lilypond-git/build/out/lybook-db
647 if we stop the build with an $(error), but I think this is because
648 we need to allow it to process the dependencies first. It looks
649 like foo.texi is shown as being dependent on foo.tely, plus a load
653 DOCUMENTATION_LOCALE_TARGET is blank
654 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
655 /home/phil/lilypond-git/scm/autochange.scm
658 plus 10s (100s?) of other .scm files.
661 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
662 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
665 ditto .ly files. This does seem a teency bit wrong - it looks like
666 the .ly and .scm files have been interchanged. ly-vars.make has
670 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
671 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
674 Looks like a bug.....
676 So it now works its way through all these files, checking if they
677 need to be remade. This is 100s of lines of the debug listing,
678 although none in the normal list. Clearly none has to be made
679 since they're source files. It concludes:
682 Must remake target `out-www/collated-files.tely'
685 @file{lysdoc-rules.make} has this:
688 $(outdir)/collated-files.tely: $(COLLATED_FILES)
689 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
692 @file{lysdoc-vars.make} has:
695 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
701 TEXINFO_SOURCES = AAA-intro-regression.tely
702 OUT_LY_FILES is empty
705 so LY_FILES has the big long list of all the .ly files in the
706 regression directory.
711 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
714 with a list of all the files in the regression test directory. This
715 should (I believe) create the file collated-files.tely.
717 So the next rule in make is for @file{version.itexi}, and make duly
718 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
719 depends on @file{git/VERSION}:
722 $(outdir)/version.%: $(top-src-dir)/VERSION
723 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
726 This causes create-version-itexi.py to run and create
729 Once that's done, all the other *.scm and *.ly files are checked
730 and since they have no rules associated, they aren't remade (just
731 as well for source files, really). Since version.itexi was remade
732 make concludes that collated-files.texi must be remade. To do
733 this, it runs lilypond-book.py on collated-files.tely, as below:
736 LILYPOND_VERSION=2.15.0
738 /home/phil/lilypond-git/scripts/lilypond-book.py
739 -I /home/phil/lilypond-git/input/regression/
740 -I ./out-www -I /home/phil/lilypond-git/input
741 -I /home/phil/lilypond-git/Documentation
742 -I /home/phil/lilypond-git/Documentation/snippets
743 -I /home/phil/lilypond-git/input/regression/
744 -I /home/phil/lilypond-git/Documentation/included/
745 -I /home/phil/lilypond-git/build/mf/out/
746 -I /home/phil/lilypond-git/build/mf/out/
747 -I /home/phil/lilypond-git/Documentation/pictures
748 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
749 --process='/home/phil/lilypond-git/build/out/bin/lilypond
750 -I /home/phil/lilypond-git/input/regression/
752 -I /home/phil/lilypond-git/input
753 -I /home/phil/lilypond-git/Documentation
754 -I /home/phil/lilypond-git/Documentation/snippets
755 -I /home/phil/lilypond-git/input/regression/
756 -I /home/phil/lilypond-git/Documentation/included/
757 -I /home/phil/lilypond-git/build/mf/out/
758 -I /home/phil/lilypond-git/build/mf/out/
759 -I /home/phil/lilypond-git/Documentation/pictures
760 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
785 -dcheck-internal-types
787 -danti-alias-factor=2'
791 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
792 out-www/collated-files.tely
795 So - lilypond-book runs on:
798 input/regression/out-www/collated-files.tely
802 Note the --verbose flag - this is from the make variable
803 LILYPOND_BOOK_VERBOSE which is added to the make variable
806 Now found the invocation to write some of the image files. It's
810 /home/phil/lilypond-git/build/out/bin/lilypond
811 -I /home/phil/lilypond-git/input/regression/
812 -I ./out-www -I /home/phil/lilypond-git/input
813 -I /home/phil/lilypond-git/Documentation
814 -I /home/phil/lilypond-git/Documentation/snippets
815 -I /home/phil/lilypond-git/input/regression/
816 -I /home/phil/lilypond-git/Documentation/included/
817 -I /home/phil/lilypond-git/build/mf/out/
818 -I /home/phil/lilypond-git/build/mf/out/
819 -I /home/phil/lilypond-git/Documentation/pictures
820 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
845 -dcheck-internal-types
847 -danti-alias-factor=2
848 -I "/home/phil/lilypond-git/build/out/lybook-db"
849 -I "/home/phil/lilypond-git/build/input/regression"
850 -I "/home/phil/lilypond-git/input/regression"
851 -I "/home/phil/lilypond-git/build/input/regression/out-www"
852 -I "/home/phil/lilypond-git/input"
853 -I "/home/phil/lilypond-git/Documentation"
854 -I "/home/phil/lilypond-git/Documentation/snippets"
855 -I "/home/phil/lilypond-git/input/regression"
856 -I "/home/phil/lilypond-git/Documentation/included"
857 -I "/home/phil/lilypond-git/build/mf/out"
858 -I "/home/phil/lilypond-git/build/mf/out"
859 -I "/home/phil/lilypond-git/Documentation/pictures"
860 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
863 -deps-box-padding=3.000000
865 -dno-strip-output-dir
866 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
869 Note the --verbose. This causes 100s of lines of Lily debug output.
870 But at present I can't work out where the flag comes from. Later.
873 @node Building a bibliography
874 @subsection Building a bibliography
876 Bibliography files contain a list of citations, like this:
880 author = @{Vinci, Albert C.@},
881 title = @{Fundamentals of Traditional Music Notation@},
882 publisher = @{Kent State University Press@},
887 There are a variety of types of citation (e.g. Book (as above),
888 article, publication). Each cited publication has a list of
889 entries that can be used to identify the publication.
890 Bibliograpies are normally stored as files with a .bib
891 extension. One part of the doc-build process is transforming the
892 bibliography information into @code{texinfo} files. The commands
893 to do this are in the @file{GNUmakefile} in the
894 @file{Documentation} directory.
896 A typical line of the makefile to translate a single bibliography
900 $(outdir)/colorado.itexi:
901 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
902 -s $(top-src-dir)/Documentation/lily-bib \
903 -o $(outdir)/colorado.itexi \
904 $(src-dir)/essay/colorado.bib
910 $(outdir)/colorado.itexi:
913 We're making the file @file{colorado.itexi} and so this is the
917 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
920 It's in the @file{essay} directory and we want to run the
921 bib2texi.py script against it.
924 -s $(top-src-dir)/Documentation/lily-bib \
927 The style template is @file{lily-bib.bst} and is found in the
928 @file{Documentation} directory.
931 -o $(outdir)/colorado.itexi \
934 The output file in @file{colorado.itexi}.
937 $(src-dir)/essay/colorado.bib
940 The input file is @file{colorado.bib} in the @file{essay}
943 The @code{bib2texi} Python script used to be used with a variety
944 of options, but now is always called using the same options, as
945 above. Its job is to create the file containing the options for
946 @code{bibtex} (the program that actually does the translation),
947 run bibtex, and then clean up some temporary files. Its main
948 "value add" is the creation of the options file, using this code:
951 open (tmpfile + '.aux', 'w').write (r'''
954 \bibstyle@{%(style)s@}
955 \bibdata@{%(files)s@}''' % vars ())
958 The key items are the style file (now always lily-bib for us) and
961 The style file is written in its own specialised language,
962 described to some extent at
965 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
968 The file @file{lily-bib.bst} also has fairly extensive commenting.
972 @section Website build
974 @warning{This information applies only to the standard @code{make
975 website} from the normal build directory. The process is
976 different for @code{dev/website-build}.}
978 The rule for make website is found in GNUmakefile.in:
982 $(MAKE) config_make=$(config_make) \
983 top-src-dir=$(top-src-dir) \
984 -f $(top-src-dir)/make/website.make \
991 make --no-builtin-rules config_make=./config.make \
992 top-src-dir=/home/phil/lilypond-git \
993 -f /home/phil/lilypond-git/make/website.make \
997 which has the effect of setting the variables @code{config_make}
998 and @code{top-src-dir} and then processing the file
999 @code{git/make/website.make} with the target of website.
1001 @code{website.make} starts with the following:
1004 ifeq ($(WEBSITE_ONLY_BUILD),1)
1007 which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
1008 was set to one on the command line. This is only done for
1009 standalone website builds, not in the normal case. The result of
1010 the test determines the value of some variables that are set. A
1011 number of other variables are set, in order to establish locations
1012 of various files. An example is:
1015 CREATE_VERSION=python $(script-dir)/create-version-itexi.py
1018 The rule for website is:
1021 website: website-texinfo website-css website-pictures website-examples web-post
1022 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1023 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1024 cp $(top-htaccess) $(OUT)/.htaccess
1025 cp $(dir-htaccess) $(OUT)/website/.htaccess
1028 so we see that this starts by running the rules for 5 other
1029 targets, then finishes by copying some files. We'll cover that
1030 later - first @code{website-texinfo}. That rule is:
1033 website-texinfo: website-version website-xrefs website-bibs
1034 for l in '' $(WEB_LANGS); do \
1035 if test -n "$$l"; then \
1036 langopt=--lang="$$l"; \
1039 $(TEXI2HTML) --prefix=index \
1041 --I=$(top-src-dir)/Documentation/"$$l" \
1042 --I=$(top-src-dir)/Documentation \
1045 --init-file=$(texi2html-init-file) \
1047 --output=$(OUT)/"$$l" \
1048 $(top-src-dir)/Documentation/"$$l"/web.texi ; \
1049 ls $(OUT)/$$l/*.html | xargs grep -L \
1050 'UNTRANSLATED NODE: IGNORE ME' | \
1051 sed 's!$(OUT)/'$$l'/!!g' | xargs \
1052 $(MASS_LINK) --prepend-suffix="$$langsuf" \
1053 hard $(OUT)/$$l/ $(OUT)/website/ ; \
1057 which therefore depends on @code{website-version},
1058 @code{website-xrefs} and @code{website-bibs}.
1063 $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
1064 $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
1067 which translates as:
1070 mkdir -p out-website
1071 python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
1072 /home/phil/lilypond-git > out-website/version.itexi
1073 python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
1074 /home/phil/lilypond-git > out-website/weblinks.itexi
1077 So, we make out-website then send the output of
1078 @code{create-version-itexi.py} to @code{out-website/version.itexi}
1079 and @code{create-weblinks-itexi.py} to
1080 @code{out-website/weblinks.itexi}.
1082 @code{create-version-itexi.py} parses the file @code{VERSION} in
1083 the top source dir. It contains:
1086 PACKAGE_NAME=LilyPond
1091 VERSION_STABLE=2.14.2
1092 VERSION_DEVEL=2.15.12
1095 currently. @code{c-v-i.py} parses this to:
1098 @@c ************************ Version numbers ************
1103 @@macro versionStable
1107 @@macro versionDevel
1112 @code{create-weblinks-itexi.py} creates a load of texi macros (of
1113 the order of 1000) similar to:
1116 @@macro manualStableGlossaryPdf
1117 @@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@}
1121 It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:
1123 @code{langdefs.py: warning: lilypond-doc gettext domain not found.}
1128 website-xrefs: website-version
1129 for l in '' $(WEB_LANGS); do \
1132 is the start of the rule, truncated for brevity. This loops
1133 through the languages to be used on the website, processing some
1134 variables which I don't fully understand, to run this command:
1137 python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
1138 -I /home/phil/lilypond-git/Documentation \
1139 -I /home/phil/lilypond-git/Documentation/"$l" \
1140 -I out-website -o out-website --split=node \
1141 --known-missing-files= \
1142 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
1144 /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
1147 There's a good description of what
1148 @code{extract_texi_filenames.py} does at the top of the script,
1149 but a shortened version is:
1151 @code{If this script is run on a file texifile.texi, it produces
1152 a file texifile[.LANG].xref-map with tab-separated entries
1153 of the form NODE\tFILENAME\tANCHOR.}
1156 @code{web.nl.xref-map} is:
1159 Inleiding Introduction Introduction
1162 @code{e-t-f.py} follows the includes from document to document.
1163 We know some have not been created yet, and
1164 @code{known-missing-files} option tells @code{e-t-f.py} which
1170 for m in $(MANUALS); do \
1173 to run @code{e-t-f.py} against all of the manuals, in each
1177 website-bibs: website-version
1178 BSTINPUTS=$(top-src-dir)/Documentation/web \
1179 $(WEB_BIBS) -s web \
1180 -s $(top-src-dir)/Documentation/lily-bib \
1181 -o $(OUT)/others-did.itexi \
1183 $(top-src-dir)/Documentation/web/others-did.bib
1186 This is half the command. It runs @code{bib2texi.py} on 2
1187 @code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
1188 This converts bibliography files into texi files with
1191 Next the commands in the @code{website-texinfo} rule are run:
1194 for l in '' $(WEB_LANGS); do \
1197 run @code{texi2html}. This is the program that outputs the
1198 progress message (found in
1199 @code{Documentation/lilypond-texi2html.init}):
1201 @code{Processing web site: []}
1203 It also outputs warning messages like:
1205 @code{WARNING: Unable to find node 'ŘeÅ¡enà potÞÃ' in book usage.}
1209 cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
1212 Copies 3 css files to out-website/website. Then:
1216 mkdir -p $(OUT)/website/pictures
1217 if [ -d $(PICTURES) ]; \
1219 cp $(PICTURES)/* $(OUT)/website/pictures ; \
1220 ln -sf website/pictures $(OUT)/pictures ;\
1224 which translates as:
1227 if [ -d Documentation/pictures/out-www ]; \
1229 cp Documentation/pictures/out-www/* out-website/website/pictures ; \
1230 ln -sf website/pictures out-website/pictures ;\
1234 i.e. it copies the contents of
1235 @code{build/Documentation/pictures/out-www/*} to
1236 @code{out-website/website/pictures}. Unfortunately, the pictures
1237 are only created once @code{make doc} has been run, so an initial
1238 run of @code{make website} copies nothing, and the pictures on the
1239 website (e.g. the logo) do not exist. Next:
1243 mkdir -p $(OUT)/website/ly-examples
1244 if [ -d $(EXAMPLES) ]; \
1246 cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
1253 mkdir -p out-website/website/ly-examples
1254 if [ -d Documentation/web/ly-examples/out-www ]; \
1256 cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
1260 This does the same with the LilyPond examples (found at
1261 @uref{http://lilypond.org/examples.html}). Again, these are
1262 actually only created by @code{make doc} (and since they are
1263 generated from LilyPond source files, require a working LilyPond
1264 @code{exe} made with @code{make}). So this does nothing
1269 $(WEB_POST) $(OUT)/website
1275 python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
1278 which describes itself as:
1280 @code{This is web_post.py. This script deals with translations
1281 in the "make website" target.}
1283 It also does a number of other things, including adding the Google
1284 tracker code and the language selection footer. We're now at
1285 the end of our story. The final 4 lines of the recipe for website
1289 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1290 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1291 cp $(top-htaccess) $(OUT)/.htaccess
1292 cp $(dir-htaccess) $(OUT)/website/.htaccess
1295 The first translates as:
1298 cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
1301 so we see these are just copying the support files for the web
1304 @subsubheading website.make summary
1306 Recipes in @file{website.make}:
1312 this is the "master" rule. It calls the other rules in order,
1313 then copies some extra files around - see below for further
1314 of the process it produces.
1317 @code{website-version}:
1318 this calls the python scripts below:
1322 scripts/build/create-version-itexi.py
1325 This writes a @@version, @@versionStable, and @@versionDevel based
1326 on the top-level VERSIONS file, to
1327 @code{out-website/version.itexi}
1331 scripts/build/create-weblinks-itexi.py
1334 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1335 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1336 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1338 It's quite monstrous because it deals with combinations of
1339 stable/devel, source/docs, lang/lang/lang*10, etc.
1344 @code{website-xrefs:}
1345 creates files used for complicated "out-of-build" references to
1346 @code{out-website/*.xref-map}
1348 If you just write @@ref@{@}, then all's groovy and we wouldn't
1349 need this. But if you write @@rlearning@{@}, then our custom
1350 texi2html init file needs to know about our custom xref file
1351 format, which tells our custom texi2html init file how to create
1354 GP: we should have a separate @@node to discuss xrefs. Also, take a
1355 quick look at a generated xref file -- it's basically just a list
1356 of @@node's [sic teenager pluralization rule] from the file.
1360 generates the bibliography texinfo files from the .bib files - in
1361 the case of the website build these are @file{others-did.bib} and
1362 @file{we-wrote.bib}.
1365 @code{website-texinfo:}
1366 this is the main part; it calles texi2html to generate the actual
1367 html. It also has a ton of options to texi2html to pass info to
1368 our custom init file.
1370 The file actually built is called @file{web.texi}, and is either
1371 in the @file{Documentation} directory, or a sub-directory specific
1374 The options file is @file{/Documentation/lilypond-texi2html.init}.
1375 This contains *lots* of option and configuration stuff, and also
1379 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1382 This is where one of the console messages is generated.
1384 We have somewhere between 2-4 different ways "to pass info to our
1385 custom init file". This is highly Not Good (tm), but that's how
1386 things work at the moment.
1388 After texi2html, it does some black magick to deal with
1389 untranslated nodes in the translations. Despite writing that
1390 part, I can't remember how it works. But in theory, you could
1391 figure it out by copy&pasting each part of the command (by "part",
1392 I mean "stuff before each | pipe"), substituting the variables,
1393 then looking at the text that's output. For example,
1396 ls $(OUT)/$$l/*.html
1399 is going to print a list of all html files, in all languages, in
1400 the build directory. Then more stuff happens to each of those
1401 files (that's what xargs does).
1405 just copies files to the build dir.
1408 @code{website-pictures, website-examples:}
1409 more file copies, with an if statement to handle if you don't have
1410 any generated pictures/examples.
1417 scripts/build/website_post.py
1420 which, it adds the "this page is translated in klingon" to the
1421 bottom of html pages, and adds the google analytics javascript.
1422 It also has hard-coded lilypond version numbers, which is Bad
1427 Here's a summary of what gets called, in what order, when we run
1434 creates version.itexi and weblinks.itexi
1436 runs extract_texi_filenames.py
1438 creates bibliography files, described above
1446 runs website_post.py
1447 Then some file copying