1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
22 @node Build system overview
23 @section Build system overview
25 Build system is currently GNU make, with an extra "stepmake" layer
26 on top. Look at files in @file{make/} and @file{stepmake/} and
27 all @file{GNUmakefile}s.
29 There is wide-spread dissatisfaction with this system, and we are
30 considering changing. This would be a huge undertaking (estimated
31 200+ hours). This change will probably involve not using GNU make
32 any more -- but a discussion about the precise build system will
33 have to wait. Before we reach that point, we need to figure out
34 (at least approximately) what the current build system does.
36 Fundamentally, a build system does two things:
40 Constructs command-line commands, for example:
44 --tons --of --options \
47 --more --imperial --and --metric --tons --of --options \
52 If there was a previous build, it decides which parts of the
53 system need to be rebuilt.
57 When I try to do anything in the build system, it helps to remind
58 myself of this. The "end result" is just a series of command-line
59 commands. All the black magick is just an attempt to construct
62 @node Tips for working on the build system
63 @section Tips for working on the build system
75 to the build system files in various places. This will let you
76 track where the program is, in various points of the build.
78 PH note. There are lots of places where Make doesn't let you put
79 echo commands. My top tip for tracing how make runs is to put
82 $(error Some Text to display)
85 This will stop make running and print the text @code{Some Text to
91 First task: understand how @code{make website} works,
92 @emph{without} the translations. Looking at the english-only
93 website is the best introduction to the build system... it only
94 covers about 5% of the whole thing, but even that will likely take
100 @node General build system notes
101 @section General build system notes
104 * How stepmake works::
107 @node How stepmake works
108 @subsection How stepmake works
110 Typing make website runs the file @file{GNUmakefile} from the
111 build directory. This only contains 3 lines:
115 include config$(if $(conf),-$(conf),).make
116 include $(configure-srcdir)/GNUmakefile.in
119 The variable @code{depth} is used throughout the make system to
120 track how far down the directory structure the make is. The first
121 include sets lots of variables but doesn't "do" anything. Default
122 values for these variables are automatically detected at the
123 ./configure step, which creates the file @file{config.make}.
124 The second include runs the file @file{GNUmakefile.in} from
125 the top level source directory.
127 This sets another load of variables, and then includes (i.e.
128 immediately runs) @file{stepmake.make} from the @file{make}
129 subdirectory. This sets a load of other variables, does some
130 testing to see if SCONS (another build tool?) is being used, and
131 then runs @file{make/config.make} - which doesn't seem to exist...
133 GP: scons is indeed a different build tool; I think that Jan
134 experimented with it 5 years ago or something. It seems like we
135 still have bits and pieces of it floating around.
137 Next, it runs @file{make/toplevel-version.make}, which sets the
138 version variables for major, minor, patch, stable, development and
139 mypatchlevel (which seems to be used for patch numbers for
140 non-stable versions only?).
142 Next - @file{make/local.make}, which doesn't exist.
144 Then a few more variable and the interesting comment:
147 # Don't try to outsmart us, you puny computer!
148 # Well, UGH. This only removes builtin rules from
151 and then tests to see whether BUILTINS_REMOVED is defined. It
152 appears to be when I run make, and so
153 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
154 comment at the head of this file says:
157 # UGH. GNU make comes with implicit rules.
158 # We don't want any of them, and can't force users to run
162 I've not studied that file at length, but assume it removes all
163 make's build-in rules (e.g. @file{*.c} files are run through the
164 GNU C compiler) - there's a lot of them in here, and a lot of
165 comments, and I'd guess most of it isn't needed.
167 We return to @file{stepmake.make}, where we hit the make rule all:
168 The first line of this is:
171 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
174 which, when the variables are substituted, gives:
177 ./make/generic-inclusions.make
178 ./make/lilypond-inclusions.make.
181 (Note - according to the make documentation, -include is only
182 different from include in that it doesn't produce any kind of
183 error message when the included file doesn't exist).
185 And the first file doesn't exist. Nor the second. Next:
188 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
191 which expands to the following files:
194 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
195 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
196 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
197 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
200 One little feature to notice here - these are all absolute file
201 locations - the line prior to this used relative locations. And
202 none of these files exist, either.
204 (Further note - I'm assuming all these lines of make I'm following are
205 autogenerated, but that'll be something else to discover.)
207 JM: @emph{``No, these lines are not useful in LilyPond (this is why
208 you think they are autogenerated), but they are part of StepMake,
209 which was meant to be a package to be installed as a build system over
210 autoconf/make in software project source trees.''}
212 Next in @file{stepmake.make}:
215 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
221 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
222 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
223 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
224 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
227 Woo. They all exist (they should as there's no - in front of the
228 include). @file{generic-vars.make} sets loads of variables
229 (funnily enough). @file{toplevel-vars.make} is very short - one
230 line commented as @code{# override Generic_vars.make:} and 2 as
235 include $(stepdir)/documentation-vars.make
238 I assume the urg comment refers to the fact that this should
239 really just create more variables, but it actually sends us off to
240 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
242 That file is a 3 line variable setting one.
244 @file{po-vars.make} has the one-line comment @code{# empty}, as
245 does @file{install-vars.make}.
247 So now we're back to @file{stepmake.make}.
252 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
253 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
256 and the include expands to:
259 include ./make/generic-vars.make ./make/lilypond-vars.make.
262 These again set variables, and in some cases export them to allow
263 child @code{make} processes to use them.
265 The final 4 lines of @file{stepmake.make} are:
268 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
269 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
270 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
271 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
274 which expand as follows:
277 include ./make/generic-rules.make ./make/lilypond-rules.make
279 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
280 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
281 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
282 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
283 include ./make/generic-targets.make ./make/lilypond-targets.make
285 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
286 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
287 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
288 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
291 @file{lilypond-rules.make} is @code{#empty}
293 @file{generic-rules.make} does seem to have 2 rules in it. They
297 $(outdir)/%.ly: %.lym4
298 $(M4) $< | sed "s/\`/,/g" > $@@
302 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@@
305 I believe the first rule is for *.ly files, and has a prerequisite
306 that *.lym4 files must be built first. The recipe is @code{m4 |
307 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
308 can comment on exactly what the rules mean/do.
310 @file{toplevel-rules.make} is @code{#empty}
312 @file{po-rules.make} is @code{#empty}
314 @file{install-rules.make} is @code{#empty}
316 @file{generic-targets.make} contains 2 lines of comments.
318 @file{lilypond-targets.make} contains only:
321 ## TODO: fail dist or web if no \version present.
323 grep -L version $(LY_FILES)
326 @file{stepmake/generic-targets.make} contains lots of rules - too
327 many to list here - it seems to be the main file for rules. (FWIW
328 I haven't actually found a rule for website: anywhere, although
329 it clearly exists. I have also found that you can display a rule
330 in the terminal by typing, say @code{make -n website}. This is
331 probably common knowledge.
333 @file{stepmake/toplevel-targets.make} adds a load of other (and
334 occasionally the same) rules to the gernric-targets.
336 @file{stepmake/po-targets.make} is rules for po* makes.
338 @file{stepmake/install-targets.make} has rules for local-install*.
340 And that's the end of stepmake.make. Back to
341 @file{GNUmakefile.in}.
343 A bit more info from 27 March. I've put some error traces into
344 @code{GNUmakefile} in the build directory, and it looks like the
345 following lines actually cause the make to run (putting an error
346 call above them - no make; below them - make):
350 # All web targets, except info image symlinks and info docs are
351 # installed in non-recursing target from TOP-SRC-DIR
353 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
354 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
355 $(MAKE) -C Documentation omf-local-install
358 I don't currently understand the @code{ifeq}, since @code{$(out)}
359 is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
363 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
364 -c -m 755 -d /usr/local/share/doc/lilypond/html
367 End of work for Sunday 27th.
369 Another alterative approach to understanding the website build
370 would be to redirect @code{make -n website} and @code{make website}
371 to a text file and work through a) what it does and b) where the
372 errors are occurring.
374 GP: wow, all the above is much more complicated than I've ever
375 looked at stuff -- I tend to do a "back first" approach (where I
376 begin from the command-line that I want to modify, figure out
377 where it's generated, and then figure out how to change the
378 generated command-line), rather than a "front first" (where you
379 begin from the "make" command).
386 * The function of make doc::
387 * Building a bibliography::
390 @node The function of make doc
391 @subsection The function of make doc
393 The following is a set of notes on how make doc functions.
395 Preliminary question to be answered some time: where do all the
396 GNUmakefiles come from. They're in the build directory, but this
397 is not part of source. Must be the configure script. And it
398 looks like this comes from autogen.sh. Must at some point kill
399 the whole git directory, repull and see what is created when.
401 Anyway, here's how make doc progresses:
403 This is the build dependency tree from
404 @file{stepmake/stepmake/generic-targets.make}:
409 $(MAKE) -C $(depth)/scripts/build out=
410 $(MAKE) out=www WWW-1
413 $(MAKE) out=www WWW-2
416 $(MAKE) out=www WWW-post
420 MAKE = make --no-builtin-rules
421 -C = Change to directory before make
424 doc-stage-1 does lots of opening and looking in files, but no
430 + make PACKAGE=LILYPOND package=lilypond -C python
431 && make PACKAGE=LILYPOND package=lilypond -C scripts
432 && make PACKAGE=LILYPOND package=lilypond -C flower
433 && make PACKAGE=LILYPOND package=lilypond -C lily
434 && make PACKAGE=LILYPOND package=lilypond -C mf
435 && make PACKAGE=LILYPOND package=lilypond -C ly
436 && make PACKAGE=LILYPOND package=lilypond -C tex
437 && make PACKAGE=LILYPOND package=lilypond -C ps
438 && make PACKAGE=LILYPOND package=lilypond -C scm
439 && make PACKAGE=LILYPOND package=lilypond -C po
440 && make PACKAGE=LILYPOND package=lilypond -C make
441 && make PACKAGE=LILYPOND package=lilypond -C elisp
442 && make PACKAGE=LILYPOND package=lilypond -C vim
443 && make PACKAGE=LILYPOND package=lilypond -C input
444 && make PACKAGE=LILYPOND package=lilypond -C stepmake
445 && make PACKAGE=LILYPOND package=lilypond -C Documentation
451 stepmake/stepmake/generic-vars.make has this:
454 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@@ &&) true
457 $@@ is the name of the target - WWW-1 in this case.
459 In GNUmakefile.in we find:
462 SUBDIRS = python scripts \
469 stepmake $(documentation-dir)
472 So that's how we get the main make loop...
474 That loop expands like this:
477 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
484 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
485 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
486 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
487 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
488 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
489 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
490 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
491 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
492 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
496 The directories up to and including vim produce no effect with
497 make in non-debug mode, although debug does show lots of action.
499 @file{git/build/input/GNUmakefile} is:
503 include $(depth)/config$(if $(conf),-$(conf),).make
504 include $(configure-srcdir)/./input/GNUmakefile
505 MODULE_INCLUDES += $(src-dir)/$(outbase)
508 The first include is:
514 (note the // which is strictly wrong)
516 which has lots of variables to set, but no action occurs.
521 lilypond-git/./input/GNUmakefile
524 which similarly doesn't create any actual action.
526 An error message at the end of build/input/GNUmakefile stops
527 make processing before it moves on to regression - so where does
530 And the answer is - make processes all directories in the
531 directory it's entered (with some exceptions like out and out-www)
532 and so it changes to /regression.
534 It then seems to consider whether it needs to make/remake loads of
535 makefiles. Don't understand this yet. Possibly these are all the
536 makefiles it's processing, and it always checks they're up to date
537 before processing other files?
539 Could be correct - some of this output is:
542 Must remake target `../../make/ly-inclusions.make'.
543 Failed to remake target file `../../make/ly-inclusions.make'.
546 Having decided that, it then leaves the directory and re-executes:
549 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
552 The top of this make is:
555 This program built for i486-pc-linux-gnu
557 Reading makefile `GNUmakefile'...
558 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
561 which looks like it's re-reading all its known makefiles to check
564 (From the make manual:
566 To this end, after reading in all makefiles, make will consider each
567 as a goal target and attempt to update it. If a makefile has a rule
568 which says how to update it (found either in that very makefile or in
569 another one) or if an implicit rule applies to it (see Chapter 10
570 [Using Implicit Rules], page 103), it will be updated if
571 necessary. After all makefiles have been checked, if any have actually
572 been changed, make starts with a clean slate and reads all the
573 makefiles over again. (It will also attempt to update each of them
574 over again, but normally this will not change them again, since they
575 are already up to date.)
577 So my assumption seems correct)
579 There appear to be about 74 of them. After all the makefile
580 checking, we get this:
583 Updating goal targets....
584 Considering target file `WWW-1'.
585 File `WWW-1' does not exist.
586 Considering target file `local-WWW-1'.
587 File `local-WWW-1' does not exist.
588 Considering target file `out-www/collated-files.texi'.
589 File `out-www/collated-files.texi' does not exist.
590 Looking for an implicit rule for `out-www/collated-files.texi'.
591 Trying pattern rule with stem `collated-files.texi'.
592 Trying implicit prerequisite `collated-files.texi.in'.
593 Trying pattern rule with stem `collated-files.texi'.
594 Trying implicit prerequisite `collated-files.texi.in'.
595 Trying pattern rule with stem `collated-files'.
596 Trying implicit prerequisite `collated-files.tely'.
597 Trying pattern rule with stem `collated-files'.
598 Trying implicit prerequisite `out-www/collated-files.tely'.
599 Trying rule prerequisite `out-www/version.itexi'.
600 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
603 grep finds this if searching for local-WWW-1:
606 make/lysdoc-targets.make:
607 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
610 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
611 and so these will need to be checked to see if they're up to date.
612 So make needs to find rules for both of those and (as it says) it
613 certainly needs to make coll*.texi, since it doesn't exist.
615 In ly-rules.make we have:
618 .SUFFIXES: .doc .tely .texi .ly
621 which I'll work out at some point, and also this rule:
624 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
625 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
628 Note that the recipe is a very long line - it could probably
629 benefit from splitting. The same makefile also has:
632 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
633 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
637 which seems to be an almost exact duplicate. Whatever, the first
638 one is executed first. Have not checked if the second executes.
640 The first recipe translates as this:
643 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
644 --output=./out-www --format= --lily-output-dir \
645 /home/phil/lilypond-git/build/out/lybook-db
649 if we stop the build with an $(error), but I think this is because
650 we need to allow it to process the dependencies first. It looks
651 like foo.texi is shown as being dependent on foo.tely, plus a load
655 DOCUMENTATION_LOCALE_TARGET is blank
656 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
657 /home/phil/lilypond-git/scm/autochange.scm
660 plus 10s (100s?) of other .scm files.
663 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
664 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
667 ditto .ly files. This does seem a teency bit wrong - it looks like
668 the .ly and .scm files have been interchanged. ly-vars.make has
672 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
673 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
676 Looks like a bug.....
678 So it now works its way through all these files, checking if they
679 need to be remade. This is 100s of lines of the debug listing,
680 although none in the normal list. Clearly none has to be made
681 since they're source files. It concludes:
684 Must remake target `out-www/collated-files.tely'
687 @file{lysdoc-rules.make} has this:
690 $(outdir)/collated-files.tely: $(COLLATED_FILES)
691 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
694 @file{lysdoc-vars.make} has:
697 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
703 TEXINFO_SOURCES = AAA-intro-regression.tely
704 OUT_LY_FILES is empty
707 so LY_FILES has the big long list of all the .ly files in the
708 regression directory.
713 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
716 with a list of all the files in the regression test directory. This
717 should (I believe) create the file collated-files.tely.
719 So the next rule in make is for @file{version.itexi}, and make duly
720 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
721 depends on @file{git/VERSION}:
724 $(outdir)/version.%: $(top-src-dir)/VERSION
725 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
728 This causes create-version-itexi.py to run and create
731 Once that's done, all the other *.scm and *.ly files are checked
732 and since they have no rules associated, they aren't remade (just
733 as well for source files, really). Since version.itexi was remade
734 make concludes that collated-files.texi must be remade. To do
735 this, it runs lilypond-book.py on collated-files.tely, as below:
738 LILYPOND_VERSION=2.15.0
740 /home/phil/lilypond-git/scripts/lilypond-book.py
741 -I /home/phil/lilypond-git/input/regression/
742 -I ./out-www -I /home/phil/lilypond-git/input
743 -I /home/phil/lilypond-git/Documentation
744 -I /home/phil/lilypond-git/Documentation/snippets
745 -I /home/phil/lilypond-git/input/regression/
746 -I /home/phil/lilypond-git/Documentation/included/
747 -I /home/phil/lilypond-git/build/mf/out/
748 -I /home/phil/lilypond-git/build/mf/out/
749 -I /home/phil/lilypond-git/Documentation/pictures
750 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
751 --process='/home/phil/lilypond-git/build/out/bin/lilypond
752 -I /home/phil/lilypond-git/input/regression/
754 -I /home/phil/lilypond-git/input
755 -I /home/phil/lilypond-git/Documentation
756 -I /home/phil/lilypond-git/Documentation/snippets
757 -I /home/phil/lilypond-git/input/regression/
758 -I /home/phil/lilypond-git/Documentation/included/
759 -I /home/phil/lilypond-git/build/mf/out/
760 -I /home/phil/lilypond-git/build/mf/out/
761 -I /home/phil/lilypond-git/Documentation/pictures
762 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
787 -dcheck-internal-types
789 -danti-alias-factor=2'
793 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
794 out-www/collated-files.tely
797 So - lilypond-book runs on:
800 input/regression/out-www/collated-files.tely
804 Note the --verbose flag - this is from the make variable
805 LILYPOND_BOOK_VERBOSE which is added to the make variable
808 Now found the invocation to write some of the image files. It's
812 /home/phil/lilypond-git/build/out/bin/lilypond
813 -I /home/phil/lilypond-git/input/regression/
814 -I ./out-www -I /home/phil/lilypond-git/input
815 -I /home/phil/lilypond-git/Documentation
816 -I /home/phil/lilypond-git/Documentation/snippets
817 -I /home/phil/lilypond-git/input/regression/
818 -I /home/phil/lilypond-git/Documentation/included/
819 -I /home/phil/lilypond-git/build/mf/out/
820 -I /home/phil/lilypond-git/build/mf/out/
821 -I /home/phil/lilypond-git/Documentation/pictures
822 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
847 -dcheck-internal-types
849 -danti-alias-factor=2
850 -I "/home/phil/lilypond-git/build/out/lybook-db"
851 -I "/home/phil/lilypond-git/build/input/regression"
852 -I "/home/phil/lilypond-git/input/regression"
853 -I "/home/phil/lilypond-git/build/input/regression/out-www"
854 -I "/home/phil/lilypond-git/input"
855 -I "/home/phil/lilypond-git/Documentation"
856 -I "/home/phil/lilypond-git/Documentation/snippets"
857 -I "/home/phil/lilypond-git/input/regression"
858 -I "/home/phil/lilypond-git/Documentation/included"
859 -I "/home/phil/lilypond-git/build/mf/out"
860 -I "/home/phil/lilypond-git/build/mf/out"
861 -I "/home/phil/lilypond-git/Documentation/pictures"
862 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
865 -deps-box-padding=3.000000
867 -dno-strip-output-dir
868 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
871 Note the --verbose. This causes 100s of lines of Lily debug output.
872 But at present I can't work out where the flag comes from. Later.
875 @node Building a bibliography
876 @subsection Building a bibliography
878 Bibliography files contain a list of citations, like this:
882 author = @{Vinci, Albert C.@},
883 title = @{Fundamentals of Traditional Music Notation@},
884 publisher = @{Kent State University Press@},
889 There are a variety of types of citation (e.g. Book (as above),
890 article, publication). Each cited publication has a list of
891 entries that can be used to identify the publication.
892 Bibliograpies are normally stored as files with a .bib
893 extension. One part of the doc-build process is transforming the
894 bibliography information into @code{texinfo} files. The commands
895 to do this are in the @file{GNUmakefile} in the
896 @file{Documentation} directory.
898 A typical line of the makefile to translate a single bibliography
902 $(outdir)/colorado.itexi:
903 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
904 -s $(top-src-dir)/Documentation/lily-bib \
905 -o $(outdir)/colorado.itexi \
906 $(src-dir)/essay/colorado.bib
912 $(outdir)/colorado.itexi:
915 We're making the file @file{colorado.itexi} and so this is the
919 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
922 It's in the @file{essay} directory and we want to run the
923 bib2texi.py script against it.
926 -s $(top-src-dir)/Documentation/lily-bib \
929 The style template is @file{lily-bib.bst} and is found in the
930 @file{Documentation} directory.
933 -o $(outdir)/colorado.itexi \
936 The output file in @file{colorado.itexi}.
939 $(src-dir)/essay/colorado.bib
942 The input file is @file{colorado.bib} in the @file{essay}
945 The @code{bib2texi} Python script used to be used with a variety
946 of options, but now is always called using the same options, as
947 above. Its job is to create the file containing the options for
948 @code{bibtex} (the program that actually does the translation),
949 run bibtex, and then clean up some temporary files. Its main
950 "value add" is the creation of the options file, using this code:
953 open (tmpfile + '.aux', 'w').write (r'''
956 \bibstyle@{%(style)s@}
957 \bibdata@{%(files)s@}''' % vars ())
960 The key items are the style file (now always lily-bib for us) and
963 The style file is written in its own specialised language,
964 described to some extent at
967 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
970 The file @file{lily-bib.bst} also has fairly extensive commenting.
974 @section Website build
976 @warning{This information applies only to the standard @code{make
977 website} from the normal build directory. The process is
978 different for @code{dev/website-build}.}
980 The rule for make website is found in GNUmakefile.in:
984 $(MAKE) config_make=$(config_make) \
985 top-src-dir=$(top-src-dir) \
986 -f $(top-src-dir)/make/website.make \
993 make --no-builtin-rules config_make=./config.make \
994 top-src-dir=/home/phil/lilypond-git \
995 -f /home/phil/lilypond-git/make/website.make \
999 which has the effect of setting the variables @code{config_make}
1000 and @code{top-src-dir} and then processing the file
1001 @code{git/make/website.make} with the target of website.
1003 @code{website.make} starts with the following:
1006 ifeq ($(WEBSITE_ONLY_BUILD),1)
1009 which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
1010 was set to one on the command line. This is only done for
1011 standalone website builds, not in the normal case. The result of
1012 the test determines the value of some variables that are set. A
1013 number of other variables are set, in order to establish locations
1014 of various files. An example is:
1017 CREATE_VERSION=python $(script-dir)/create-version-itexi.py
1020 The rule for website is:
1023 website: website-texinfo website-css website-pictures website-examples web-post
1024 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1025 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1026 cp $(top-htaccess) $(OUT)/.htaccess
1027 cp $(dir-htaccess) $(OUT)/website/.htaccess
1030 so we see that this starts by running the rules for 5 other
1031 targets, then finishes by copying some files. We'll cover that
1032 later - first @code{website-texinfo}. That rule is:
1035 website-texinfo: website-version website-xrefs website-bibs
1036 for l in '' $(WEB_LANGS); do \
1037 if test -n "$$l"; then \
1038 langopt=--lang="$$l"; \
1041 $(TEXI2HTML) --prefix=index \
1043 --I=$(top-src-dir)/Documentation/"$$l" \
1044 --I=$(top-src-dir)/Documentation \
1047 --init-file=$(texi2html-init-file) \
1049 --output=$(OUT)/"$$l" \
1050 $(top-src-dir)/Documentation/"$$l"/web.texi ; \
1051 ls $(OUT)/$$l/*.html | xargs grep -L \
1052 'UNTRANSLATED NODE: IGNORE ME' | \
1053 sed 's!$(OUT)/'$$l'/!!g' | xargs \
1054 $(MASS_LINK) --prepend-suffix="$$langsuf" \
1055 hard $(OUT)/$$l/ $(OUT)/website/ ; \
1059 which therefore depends on @code{website-version},
1060 @code{website-xrefs} and @code{website-bibs}.
1065 $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
1066 $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
1069 which translates as:
1072 mkdir -p out-website
1073 python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
1074 /home/phil/lilypond-git > out-website/version.itexi
1075 python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
1076 /home/phil/lilypond-git > out-website/weblinks.itexi
1079 So, we make out-website then send the output of
1080 @code{create-version-itexi.py} to @code{out-website/version.itexi}
1081 and @code{create-weblinks-itexi.py} to
1082 @code{out-website/weblinks.itexi}.
1084 @code{create-version-itexi.py} parses the file @code{VERSION} in
1085 the top source dir. It contains:
1088 PACKAGE_NAME=LilyPond
1093 VERSION_STABLE=2.14.2
1094 VERSION_DEVEL=2.15.12
1097 currently. @code{c-v-i.py} parses this to:
1100 @@c ************************ Version numbers ************
1105 @@macro versionStable
1109 @@macro versionDevel
1114 @code{create-weblinks-itexi.py} creates a load of texi macros (of
1115 the order of 1000) similar to:
1118 @@macro manualStableGlossaryPdf
1119 @@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@}
1123 It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:
1125 @code{langdefs.py: warning: lilypond-doc gettext domain not found.}
1130 website-xrefs: website-version
1131 for l in '' $(WEB_LANGS); do \
1134 is the start of the rule, truncated for brevity. This loops
1135 through the languages to be used on the website, processing some
1136 variables which I don't fully understand, to run this command:
1139 python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
1140 -I /home/phil/lilypond-git/Documentation \
1141 -I /home/phil/lilypond-git/Documentation/"$l" \
1142 -I out-website -o out-website --split=node \
1143 --known-missing-files= \
1144 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
1146 /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
1149 There's a good description of what
1150 @code{extract_texi_filenames.py} does at the top of the script,
1151 but a shortened version is:
1153 @code{If this script is run on a file texifile.texi, it produces
1154 a file texifile[.LANG].xref-map with tab-separated entries
1155 of the form NODE\tFILENAME\tANCHOR.}
1158 @code{web.nl.xref-map} is:
1161 Inleiding Introduction Introduction
1164 @code{e-t-f.py} follows the includes from document to document.
1165 We know some have not been created yet, and
1166 @code{known-missing-files} option tells @code{e-t-f.py} which
1172 for m in $(MANUALS); do \
1175 to run @code{e-t-f.py} against all of the manuals, in each
1179 website-bibs: website-version
1180 BSTINPUTS=$(top-src-dir)/Documentation/web \
1181 $(WEB_BIBS) -s web \
1182 -s $(top-src-dir)/Documentation/lily-bib \
1183 -o $(OUT)/others-did.itexi \
1185 $(top-src-dir)/Documentation/web/others-did.bib
1188 This is half the command. It runs @code{bib2texi.py} on 2
1189 @code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
1190 This converts bibliography files into texi files with
1193 Next the commands in the @code{website-texinfo} rule are run:
1196 for l in '' $(WEB_LANGS); do \
1199 run @code{texi2html}. This is the program that outputs the
1200 progress message (found in
1201 @code{Documentation/lilypond-texi2html.init}):
1203 @code{Processing web site: []}
1205 It also outputs warning messages like:
1207 @code{WARNING: Unable to find node 'ŘeÅ¡enà potÞÃ' in book usage.}
1211 cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
1214 Copies 3 css files to out-website/website. Then:
1218 mkdir -p $(OUT)/website/pictures
1219 if [ -d $(PICTURES) ]; \
1221 cp $(PICTURES)/* $(OUT)/website/pictures ; \
1222 ln -sf website/pictures $(OUT)/pictures ;\
1226 which translates as:
1229 if [ -d Documentation/pictures/out-www ]; \
1231 cp Documentation/pictures/out-www/* out-website/website/pictures ; \
1232 ln -sf website/pictures out-website/pictures ;\
1236 i.e. it copies the contents of
1237 @code{build/Documentation/pictures/out-www/*} to
1238 @code{out-website/website/pictures}. Unfortunately, the pictures
1239 are only created once @code{make doc} has been run, so an initial
1240 run of @code{make website} copies nothing, and the pictures on the
1241 website (e.g. the logo) do not exist. Next:
1245 mkdir -p $(OUT)/website/ly-examples
1246 if [ -d $(EXAMPLES) ]; \
1248 cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
1255 mkdir -p out-website/website/ly-examples
1256 if [ -d Documentation/web/ly-examples/out-www ]; \
1258 cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
1262 This does the same with the LilyPond examples (found at
1263 @uref{http://lilypond.org/examples.html}). Again, these are
1264 actually only created by @code{make doc} (and since they are
1265 generated from LilyPond source files, require a working LilyPond
1266 @code{exe} made with @code{make}). So this does nothing
1271 $(WEB_POST) $(OUT)/website
1277 python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
1280 which describes itself as:
1282 @code{This is web_post.py. This script deals with translations
1283 in the "make website" target.}
1285 It also does a number of other things, including adding the Google
1286 tracker code and the language selection footer. We're now at
1287 the end of our story. The final 4 lines of the recipe for website
1291 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1292 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1293 cp $(top-htaccess) $(OUT)/.htaccess
1294 cp $(dir-htaccess) $(OUT)/website/.htaccess
1297 The first translates as:
1300 cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
1303 so we see these are just copying the support files for the web
1306 @subsubheading website.make summary
1308 Recipes in @file{website.make}:
1314 this is the "master" rule. It calls the other rules in order,
1315 then copies some extra files around - see below for further
1316 of the process it produces.
1319 @code{website-version}:
1320 this calls the python scripts below:
1324 scripts/build/create-version-itexi.py
1327 This writes a @@version, @@versionStable, and @@versionDevel based
1328 on the top-level VERSIONS file, to
1329 @code{out-website/version.itexi}
1333 scripts/build/create-weblinks-itexi.py
1336 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1337 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1338 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1340 It's quite monstrous because it deals with combinations of
1341 stable/devel, source/docs, lang/lang/lang*10, etc.
1346 @code{website-xrefs:}
1347 creates files used for complicated "out-of-build" references to
1348 @code{out-website/*.xref-map}
1350 If you just write @@ref@{@}, then all's groovy and we wouldn't
1351 need this. But if you write @@rlearning@{@}, then our custom
1352 texi2html init file needs to know about our custom xref file
1353 format, which tells our custom texi2html init file how to create
1356 GP: we should have a separate @@node to discuss xrefs. Also, take a
1357 quick look at a generated xref file -- it's basically just a list
1358 of @@node's [sic teenager pluralization rule] from the file.
1362 generates the bibliography texinfo files from the .bib files - in
1363 the case of the website build these are @file{others-did.bib} and
1364 @file{we-wrote.bib}.
1367 @code{website-texinfo:}
1368 this is the main part; it calles texi2html to generate the actual
1369 html. It also has a ton of options to texi2html to pass info to
1370 our custom init file.
1372 The file actually built is called @file{web.texi}, and is either
1373 in the @file{Documentation} directory, or a sub-directory specific
1376 The options file is @file{/Documentation/lilypond-texi2html.init}.
1377 This contains *lots* of option and configuration stuff, and also
1381 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1384 This is where one of the console messages is generated.
1386 We have somewhere between 2-4 different ways "to pass info to our
1387 custom init file". This is highly Not Good (tm), but that's how
1388 things work at the moment.
1390 After texi2html, it does some black magick to deal with
1391 untranslated nodes in the translations. Despite writing that
1392 part, I can't remember how it works. But in theory, you could
1393 figure it out by copy&pasting each part of the command (by "part",
1394 I mean "stuff before each | pipe"), substituting the variables,
1395 then looking at the text that's output. For example,
1398 ls $(OUT)/$$l/*.html
1401 is going to print a list of all html files, in all languages, in
1402 the build directory. Then more stuff happens to each of those
1403 files (that's what xargs does).
1407 just copies files to the build dir.
1410 @code{website-pictures, website-examples:}
1411 more file copies, with an if statement to handle if you don't have
1412 any generated pictures/examples.
1419 scripts/build/website_post.py
1422 which, it adds the "this page is translated in klingon" to the
1423 bottom of html pages, and adds the google analytics javascript.
1424 It also has hard-coded lilypond version numbers, which is Bad
1429 Here's a summary of what gets called, in what order, when we run
1436 creates version.itexi and weblinks.itexi
1438 runs extract_texi_filenames.py
1440 creates bibliography files, described above
1448 runs website_post.py
1449 Then some file copying
1452 @node Building an Ubuntu distro
1453 @section Building an Ubuntu distro
1456 Here's the short instruction on how to create lilybuntu iso image
1457 (Jonathan Kulp did this on a spare drive,
1458 but he supposes it can be done in a VM too):
1463 Install ubuntu, reboot.
1465 Run all updates, reboot if asked.
1467 Enable src repos, refresh package lists.
1469 Install LilyPond build deps:
1471 sudo apt-get build-dep lilypond
1474 Install git and autoconf:
1476 sudo apt-get install git-core gitk autoconf
1480 Test to see whether everything works fine now:
1483 use @command{lily-git.tcl} to grab source files
1485 go to source dir and do
1487 "./autogen.sh" ; make ; make doc
1490 if all compiles, move on to iso creation...
1494 Download & install "remastersys":
1495 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
1497 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
1499 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
1500 default live session username, etc).
1502 Remove non-essential desktop software as desired.
1506 sudo remastersys dist
1508 New iso is in @file{/home/remastersys/remastersys/}.
1510 Test iso by installing in VM and repeating steps above for
1511 getting source files and building lp and docs.
1516 @section Building GUB
1518 GUB - the Grand Unified Builder - is used to build the release
1519 versions of LilyPond. For background information, see
1520 @ref{Grand Unified Builder (GUB)}. The simplest way to set up a
1521 GUB build environment is to use a virtual machine with LilyDev
1522 (@ref{LilyDev}). Follow the instructions on that page to set this
1523 up. Make sure that your virtual machine has enough disk space -
1524 a GUB installation takes over 30 GBytes of disk space, and if you
1525 allocate too little, it will fail during the setting up stage and
1526 you will have to start again. 64 GBytes should be sufficient.
1528 While GUB is being built, any interruptions are likely to make it
1529 almost impossible to restart. If at all possible, leave the build
1530 to continue uniterrupted.
1532 Download GUB and start the set up:
1535 git clone git://github.com/gperciva/gub/gub.git
1540 This downloads and installs a number of packages. You may find
1541 some fail during download and you will need to download them
1542 manually. For example, the perl archive. If this happens,
1544 @uref{http://www.cpan.org/src/5.0/perl-5.10.0.tar.gz}, saving the
1545 archive to @file{gub/downloads/perl/}. Continue the set up with:
1551 Once this has completed successfully, you can build the LilyPond
1552 release package. However, this uses an archived version of the
1553 regression tests, so it is better to download this first.
1554 Download the test output from lilypond.org:
1557 @uref{http://lilypond.org/download/binaries/test-output/lilypond-2.15.33-1.test-output.tar.bz2}
1560 Copy the tarball into @file{gub/regtests/}, and tell the build
1561 system that you have done this:
1564 touch regtests/ignore
1567 Now start the GUB build:
1573 That's it. This will build LilyPond from current master. To build
1574 the current unstable release, run:
1577 make LILYPOND_BRANCH=release/unstable lilypond