1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
22 @node Build system overview
23 @section Build system overview
25 Build system is currently GNU make, with an extra "stepmake" layer
26 on top. Look at files in @file{make/} and @file{stepmake/} and
27 all @file{GNUmakefile}s.
29 There is wide-spread dissatisfaction with this system, and we are
30 considering changing. This would be a huge undertaking (estimated
31 200+ hours). This change will probably involve not using GNU make
32 any more -- but a discussion about the precise build system will
33 have to wait. Before we reach that point, we need to figure out
34 (at least approximately) what the current build system does.
36 Fundamentally, a build system does two things:
40 Constructs command-line commands, for example:
44 --tons --of --options \
47 --more --imperial --and --metric --tons --of --options \
52 If there was a previous build, it decides which parts of the
53 system need to be rebuilt.
57 When I try to do anything in the build system, it helps to remind
58 myself of this. The "end result" is just a series of command-line
59 commands. All the black magick is just an attempt to construct
62 @node Tips for working on the build system
63 @section Tips for working on the build system
75 to the build system files in various places. This will let you
76 track where the program is, in various points of the build.
78 PH note. There are lots of places where Make doesn't let you put
79 echo commands. My top tip for tracing how make runs is to put
82 $(error Some Text to display)
85 This will stop make running and print the text @code{Some Text to
91 First task: understand how @code{make website} works,
92 @emph{without} the translations. Looking at the english-only
93 website is the best introduction to the build system... it only
94 covers about 5% of the whole thing, but even that will likely take
100 @node General build system notes
101 @section General build system notes
104 * How stepmake works::
107 @node How stepmake works
108 @subsection How stepmake works
110 Typing make website runs the file @file{GNUmakefile} from the
111 build directory. This only contains 3 lines:
115 include config$(if $(conf),-$(conf),).make
116 include $(configure-srcdir)/GNUmakefile.in
119 The variable @code{depth} is used throughout the make system to
120 track how far down the directory structure the make is. The first
121 include sets lots of variables but doesn't "do" anything. Default
122 values for these variables are automatically detected at the
123 ./configure step, which creates the file @file{config.make}.
124 The second include runs the file @file{GNUmakefile.in} from
125 the top level source directory.
127 This sets another load of variables, and then includes (i.e.
128 immediately runs) @file{stepmake.make} from the @file{make}
129 subdirectory. This sets a load of other variables, does some
130 testing to see if SCONS (another build tool?) is being used, and
131 then runs @file{make/config.make} - which doesn't seem to exist...
133 GP: scons is indeed a different build tool; I think that Jan
134 experimented with it 5 years ago or something. It seems like we
135 still have bits and pieces of it floating around.
137 Next, it runs @file{make/toplevel-version.make}, which sets the
138 version variables for major, minor, patch, stable, development and
139 mypatchlevel (which seems to be used for patch numbers for
140 non-stable versions only?).
142 Next - @file{make/local.make}, which doesn't exist.
144 Then a few more variable and the interesting comment:
147 # Don't try to outsmart us, you puny computer!
148 # Well, UGH. This only removes builtin rules from
151 and then tests to see whether BUILTINS_REMOVED is defined. It
152 appears to be when I run make, and so
153 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
154 comment at the head of this file says:
157 # UGH. GNU make comes with implicit rules.
158 # We don't want any of them, and can't force users to run
162 I've not studied that file at length, but assume it removes all
163 make's build-in rules (e.g. @file{*.c} files are run through the
164 GNU C compiler) - there's a lot of them in here, and a lot of
165 comments, and I'd guess most of it isn't needed.
167 We return to @file{stepmake.make}, where we hit the make rule all:
168 The first line of this is:
171 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
174 which, when the variables are substituted, gives:
177 ./make/generic-inclusions.make
178 ./make/lilypond-inclusions.make.
181 (Note - according to the make documentation, -include is only
182 different from include in that it doesn't produce any kind of
183 error message when the included file doesn't exist).
185 And the first file doesn't exist. Nor the second. Next:
188 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
191 which expands to the following files:
194 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
195 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
196 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
197 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
200 One little feature to notice here - these are all absolute file
201 locations - the line prior to this used relative locations. And
202 none of these files exist, either. (Further note - I'm assuming
203 all these lines of make I'm following are autogenerated, but
204 that'll be something else to discover.)
206 Next in @file{stepmake.make}:
209 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
215 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
216 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
217 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
218 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
221 Woo. They all exist (they should as there's no - in front of the
222 include). @file{generic-vars.make} sets loads of variables
223 (funnily enough). @file{toplevel-vars.make} is very short - one
224 line commented as @code{# override Generic_vars.make:} and 2 as
229 include $(stepdir)/documentation-vars.make
232 I assume the urg comment refers to the fact that this should
233 really just create more variables, but it actually sends us off to
234 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
236 That file is a 3 line variable setting one.
238 @file{po-vars.make} has the one-line comment @code{# empty}, as
239 does @file{install-vars.make}.
241 So now we're back to @file{stepmake.make}.
246 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
247 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
250 and the include expands to:
253 include ./make/generic-vars.make ./make/lilypond-vars.make.
256 These again set variables, and in some cases export them to allow
257 child @code{make} processes to use them.
259 The final 4 lines of @file{stepmake.make} are:
262 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
263 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
264 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
265 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
268 which expand as follows:
271 include ./make/generic-rules.make ./make/lilypond-rules.make
273 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
274 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
275 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
276 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
277 include ./make/generic-targets.make ./make/lilypond-targets.make
279 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
280 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
281 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
282 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
285 @file{lilypond-rules.make} is @code{#empty}
287 @file{generic-rules.make} does seem to have 2 rules in it. They
291 $(outdir)/%.ly: %.lym4
292 $(M4) $< | sed "s/\`/,/g" > $@
296 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
299 I believe the first rule is for *.ly files, and has a prerequisite
300 that *.lym4 files must be built first. The recipe is @code{m4 |
301 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
302 can comment on exactly what the rules mean/do.
304 @file{toplevel-rules.make} is @code{#empty}
306 @file{po-rules.make} is @code{#empty}
308 @file{install-rules.make} is @code{#empty}
310 @file{generic-targets.make} contains 2 lines of comments.
312 @file{lilypond-targets.make} contains only:
315 ## TODO: fail dist or web if no \version present.
317 grep -L version $(LY_FILES)
320 @file{stepmake/generic-targets.make} contains lots of rules - too
321 many to list here - it seems to be the main file for rules. (FWIW
322 I haven't actually found a rule for website: anywhere, although
323 it clearly exists. I have also found that you can display a rule
324 in the terminal by typing, say @code{make -n website}. This is
325 probably common knowledge.
327 @file{stepmake/toplevel-targets.make} adds a load of other (and
328 occasionally the same) rules to the gernric-targets.
330 @file{stepmake/po-targets.make} is rules for po* makes.
332 @file{stepmake/install-targets.make} has rules for local-install*.
334 And that's the end of stepmake.make. Back to
335 @file{GNUmakefile.in}.
337 A bit more info from 27 March. I've put some error traces into
338 @code{GNUmakefile} in the build directory, and it looks like the
339 following lines actually cause the make to run (putting an error
340 call above them - no make; below them - make):
344 # All web targets, except info image symlinks and info docs are
345 # installed in non-recursing target from TOP-SRC-DIR
347 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
348 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
349 $(MAKE) -C Documentation omf-local-install
352 I don't currently understand the @code{ifeq}, since @code{$(out)}
353 is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
357 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
358 -c -m 755 -d /usr/local/share/doc/lilypond/html
361 End of work for Sunday 27th.
363 Another alterative approach to understanding the website build
364 would be to redirect @code{make -n website} and @code{make website}
365 to a text file and work through a) what it does and b) where the
366 errors are occurring.
368 GP: wow, all the above is much more complicated than I've ever
369 looked at stuff -- I tend to do a "back first" approach (where I
370 begin from the command-line that I want to modify, figure out
371 where it's generated, and then figure out how to change the
372 generated command-line), rather than a "front first" (where you
373 begin from the "make" command).
380 * The function of make doc::
381 * Building a bibliography::
384 @node The function of make doc
385 @subsection The function of make doc
387 The following is a set of notes on how make doc functions.
389 Preliminary question to be answered some time: where do all the
390 GNUmakefiles come from. They're in the build directory, but this
391 is not part of source. Must be the configure script. And it
392 looks like this comes from autogen.sh. Must at some point kill
393 the whole git directory, repull and see what is created when.
395 Anyway, here's how make doc progresses:
397 This is the build dependency tree from
398 @file{stepmake/stepmake/generic-targets.make}:
403 $(MAKE) -C $(depth)/scripts/build out=
404 $(MAKE) out=www WWW-1
407 $(MAKE) out=www WWW-2
410 $(MAKE) out=www WWW-post
414 MAKE = make --no-builtin-rules
415 -C = Change to directory before make
418 doc-stage-1 does lots of opening and looking in files, but no
424 + make PACKAGE=LILYPOND package=lilypond -C python
425 && make PACKAGE=LILYPOND package=lilypond -C scripts
426 && make PACKAGE=LILYPOND package=lilypond -C flower
427 && make PACKAGE=LILYPOND package=lilypond -C lily
428 && make PACKAGE=LILYPOND package=lilypond -C mf
429 && make PACKAGE=LILYPOND package=lilypond -C ly
430 && make PACKAGE=LILYPOND package=lilypond -C tex
431 && make PACKAGE=LILYPOND package=lilypond -C ps
432 && make PACKAGE=LILYPOND package=lilypond -C scm
433 && make PACKAGE=LILYPOND package=lilypond -C po
434 && make PACKAGE=LILYPOND package=lilypond -C make
435 && make PACKAGE=LILYPOND package=lilypond -C elisp
436 && make PACKAGE=LILYPOND package=lilypond -C vim
437 && make PACKAGE=LILYPOND package=lilypond -C input
438 && make PACKAGE=LILYPOND package=lilypond -C stepmake
439 && make PACKAGE=LILYPOND package=lilypond -C Documentation
445 stepmake/stepmake/generic-vars.make has this:
448 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true
451 $@ is the name of the target - WWW-1 in this case.
453 In GNUmakefile.in we find:
456 SUBDIRS = python scripts \
463 stepmake $(documentation-dir)
466 So that's how we get the main make loop...
468 That loop expands like this:
471 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
472 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
473 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
474 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
475 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
476 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
477 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
484 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
485 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
486 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
490 The directories up to and including vim produce no effect with
491 make in non-debug mode, although debug does show lots of action.
493 @file{git/build/input/GNUmakefile} is:
497 include $(depth)/config$(if $(conf),-$(conf),).make
498 include $(configure-srcdir)/./input/GNUmakefile
499 MODULE_INCLUDES += $(src-dir)/$(outbase)
502 The first include is:
508 (note the // which is strictly wrong)
510 which has lots of variables to set, but no action occurs.
515 lilypond-git/./input/GNUmakefile
518 which similarly doesn't create any actual action.
520 An error message at the end of build/input/GNUmakefile stops
521 make processing before it moves on to regression - so where does
524 And the answer is - make processes all directories in the
525 directory it's entered (with some exceptions like out and out-www)
526 and so it changes to /regression.
528 It then seems to consider whether it needs to make/remake loads of
529 makefiles. Don't understand this yet. Possibly these are all the
530 makefiles it's processing, and it always checks they're up to date
531 before processing other files?
533 Could be correct - some of this output is:
536 Must remake target `../../make/ly-inclusions.make'.
537 Failed to remake target file `../../make/ly-inclusions.make'.
540 Having decided that, it then leaves the directory and re-executes:
543 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
546 The top of this make is:
549 This program built for i486-pc-linux-gnu
551 Reading makefile `GNUmakefile'...
552 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
555 which looks like it's re-reading all its known makefiles to check
558 (From the make manual:
560 To this end, after reading in all makefiles, make will consider each as a goal target and
561 attempt to update it. If a makefile has a rule which says how to update it (found either
562 in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10
563 [Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have
564 been checked, if any have actually been changed, make starts with a clean slate and reads
565 all the makefiles over again. (It will also attempt to update each of them over again, but
566 normally this will not change them again, since they are already up to date.)
568 So my assumption seems correct)
570 There appear to be about 74 of them. After all the makefile
571 checking, we get this:
574 Updating goal targets....
575 Considering target file `WWW-1'.
576 File `WWW-1' does not exist.
577 Considering target file `local-WWW-1'.
578 File `local-WWW-1' does not exist.
579 Considering target file `out-www/collated-files.texi'.
580 File `out-www/collated-files.texi' does not exist.
581 Looking for an implicit rule for `out-www/collated-files.texi'.
582 Trying pattern rule with stem `collated-files.texi'.
583 Trying implicit prerequisite `collated-files.texi.in'.
584 Trying pattern rule with stem `collated-files.texi'.
585 Trying implicit prerequisite `collated-files.texi.in'.
586 Trying pattern rule with stem `collated-files'.
587 Trying implicit prerequisite `collated-files.tely'.
588 Trying pattern rule with stem `collated-files'.
589 Trying implicit prerequisite `out-www/collated-files.tely'.
590 Trying rule prerequisite `out-www/version.itexi'.
591 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
594 grep finds this if searching for local-WWW-1:
597 make/lysdoc-targets.make:
598 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
601 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
602 and so these will need to be checked to see if they're up to date.
603 So make needs to find rules for both of those and (as it says) it
604 certainly needs to make coll*.texi, since it doesn't exist.
606 In ly-rules.make we have:
609 .SUFFIXES: .doc .tely .texi .ly
612 which I'll work out at some point, and also this rule:
615 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
616 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
619 Note that the recipe is a very long line - it could probably
620 benefit from splitting. The same makefile also has:
623 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
624 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
628 which seems to be an almost exact duplicate. Whatever, the first
629 one is executed first. Have not checked if the second executes.
631 The first recipe translates as this:
634 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
635 --output=./out-www --format= --lily-output-dir \
636 /home/phil/lilypond-git/build/out/lybook-db
640 if we stop the build with an $(error), but I think this is because
641 we need to allow it to process the dependencies first. It looks
642 like foo.texi is shown as being dependent on foo.tely, plus a load
646 DOCUMENTATION_LOCALE_TARGET is blank
647 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
648 /home/phil/lilypond-git/scm/autochange.scm
651 plus 10s (100s?) of other .scm files.
654 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
655 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
658 ditto .ly files. This does seem a teency bit wrong - it looks like
659 the .ly and .scm files have been interchanged. ly-vars.make has
663 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
664 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
667 Looks like a bug.....
669 So it now works its way through all these files, checking if they
670 need to be remade. This is 100s of lines of the debug listing,
671 although none in the normal list. Clearly none has to be made
672 since they're source files. It concludes:
675 Must remake target `out-www/collated-files.tely'
678 @file{lysdoc-rules.make} has this:
681 $(outdir)/collated-files.tely: $(COLLATED_FILES)
682 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
685 @file{lysdoc-vars.make} has:
688 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
694 TEXINFO_SOURCES = AAA-intro-regression.tely
695 OUT_LY_FILES is empty
698 so LY_FILES has the big long list of all the .ly files in the
699 regression directory.
704 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
707 with a list of all the files in the regression test directory. This
708 should (I believe) create the file collated-files.tely.
710 So the next rule in make is for @file{version.itexi}, and make duly
711 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
712 depends on @file{git/VERSION}:
715 $(outdir)/version.%: $(top-src-dir)/VERSION
716 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
719 This causes create-version-itexi.py to run and create
722 Once that's done, all the other *.scm and *.ly files are checked
723 and since they have no rules associated, they aren't remade (just
724 as well for source files, really). Since version.itexi was remade
725 make concludes that collated-files.texi must be remade. To do
726 this, it runs lilypond-book.py on collated-files.tely, as below:
729 LILYPOND_VERSION=2.15.0
731 /home/phil/lilypond-git/scripts/lilypond-book.py
732 -I /home/phil/lilypond-git/input/regression/
733 -I ./out-www -I /home/phil/lilypond-git/input
734 -I /home/phil/lilypond-git/Documentation
735 -I /home/phil/lilypond-git/Documentation/snippets
736 -I /home/phil/lilypond-git/input/regression/
737 -I /home/phil/lilypond-git/Documentation/included/
738 -I /home/phil/lilypond-git/build/mf/out/
739 -I /home/phil/lilypond-git/build/mf/out/
740 -I /home/phil/lilypond-git/Documentation/pictures
741 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
742 --process='/home/phil/lilypond-git/build/out/bin/lilypond
743 -I /home/phil/lilypond-git/input/regression/
745 -I /home/phil/lilypond-git/input
746 -I /home/phil/lilypond-git/Documentation
747 -I /home/phil/lilypond-git/Documentation/snippets
748 -I /home/phil/lilypond-git/input/regression/
749 -I /home/phil/lilypond-git/Documentation/included/
750 -I /home/phil/lilypond-git/build/mf/out/
751 -I /home/phil/lilypond-git/build/mf/out/
752 -I /home/phil/lilypond-git/Documentation/pictures
753 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
778 -dcheck-internal-types
780 -danti-alias-factor=2'
784 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
785 out-www/collated-files.tely
788 So - lilypond-book runs on:
791 input/regression/out-www/collated-files.tely
795 Note the --verbose flag - this is from the make variable
796 LILYPOND_BOOK_VERBOSE which is added to the make variable
799 Now found the invocation to write some of the image files. It's
803 /home/phil/lilypond-git/build/out/bin/lilypond
804 -I /home/phil/lilypond-git/input/regression/
805 -I ./out-www -I /home/phil/lilypond-git/input
806 -I /home/phil/lilypond-git/Documentation
807 -I /home/phil/lilypond-git/Documentation/snippets
808 -I /home/phil/lilypond-git/input/regression/
809 -I /home/phil/lilypond-git/Documentation/included/
810 -I /home/phil/lilypond-git/build/mf/out/
811 -I /home/phil/lilypond-git/build/mf/out/
812 -I /home/phil/lilypond-git/Documentation/pictures
813 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
838 -dcheck-internal-types
840 -danti-alias-factor=2
841 -I "/home/phil/lilypond-git/build/out/lybook-db"
842 -I "/home/phil/lilypond-git/build/input/regression"
843 -I "/home/phil/lilypond-git/input/regression"
844 -I "/home/phil/lilypond-git/build/input/regression/out-www"
845 -I "/home/phil/lilypond-git/input"
846 -I "/home/phil/lilypond-git/Documentation"
847 -I "/home/phil/lilypond-git/Documentation/snippets"
848 -I "/home/phil/lilypond-git/input/regression"
849 -I "/home/phil/lilypond-git/Documentation/included"
850 -I "/home/phil/lilypond-git/build/mf/out"
851 -I "/home/phil/lilypond-git/build/mf/out"
852 -I "/home/phil/lilypond-git/Documentation/pictures"
853 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
856 -deps-box-padding=3.000000
858 -dno-strip-output-dir
859 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
862 Note the --verbose. This causes 100s of lines of Lily debug output.
863 But at present I can't work out where the flag comes from. Later.
866 @node Building a bibliography
867 @subsection Building a bibliography
869 Bibliography files contain a list of citations, like this:
873 author = @{Vinci, Albert C.@},
874 title = @{Fundamentals of Traditional Music Notation@},
875 publisher = @{Kent State University Press@},
880 There are a variety of types of citation (e.g. Book (as above),
881 article, publication). Each cited publication has a list of
882 entries that can be used to identify the publication.
883 Bibliograpies are normally stored as files with a .bib
884 extension. One part of the doc-build process is transforming the
885 bibliography information into @code{texinfo} files. The commands
886 to do this are in the @file{GNUmakefile} in the
887 @file{Documentation} directory.
889 A typical line of the makefile to translate a single bibliography
893 $(outdir)/colorado.itexi:
894 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
895 -s $(top-src-dir)/Documentation/lily-bib \
896 -o $(outdir)/colorado.itexi \
897 $(src-dir)/essay/colorado.bib
903 $(outdir)/colorado.itexi:
906 We're making the file @file{colorado.itexi} and so this is the
910 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
913 It's in the @file{essay} directory and we want to run the
914 bib2texi.py script against it.
917 -s $(top-src-dir)/Documentation/lily-bib \
920 The style template is @file{lily-bib.bst} and is found in the
921 @file{Documentation} directory.
924 -o $(outdir)/colorado.itexi \
927 The output file in @file{colorado.itexi}.
930 $(src-dir)/essay/colorado.bib
933 The input file is @file{colorado.bib} in the @file{essay}
936 The @code{bib2texi} Python script used to be used with a variety
937 of options, but now is always called using the same options, as
938 above. Its job is to create the file containing the options for
939 @code{bibtex} (the program that actually does the translation),
940 run bibtex, and then clean up some temporary files. Its main
941 "value add" is the creation of the options file, using this code:
944 open (tmpfile + '.aux', 'w').write (r'''
947 \bibstyle@{%(style)s@}
948 \bibdata@{%(files)s@}''' % vars ())
951 The key items are the style file (now always lily-bib for us) and
954 The style file is written in its own specialised language,
955 described to some extent at
958 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
961 The file @file{lily-bib.bst} also has fairly extensive commenting.
965 @section Website build
967 @warning{This information applies only to the standard @code{make
968 website} from the normal build directory. The process is
969 different for @code{dev/website-build}.}
971 The rule for make website is found in GNUmakefile.in:
975 $(MAKE) config_make=$(config_make) \
976 top-src-dir=$(top-src-dir) \
977 -f $(top-src-dir)/make/website.make \
984 make --no-builtin-rules config_make=./config.make \
985 top-src-dir=/home/phil/lilypond-git \
986 -f /home/phil/lilypond-git/make/website.make \
990 which has the effect of setting the variables @code{config_make}
991 and @code{top-src-dir} and then processing the file
992 @code{git/make/website.make} with the target of website.
994 @code{website.make} starts with the following:
997 ifeq ($(WEBSITE_ONLY_BUILD),1)
1000 which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
1001 was set to one on the command line. This is only done for
1002 standalone website builds, not in the normal case. The result of
1003 the test determines the value of some variables that are set. A
1004 number of other variables are set, in order to establish locations
1005 of various files. An example is:
1008 CREATE_VERSION=python $(script-dir)/create-version-itexi.py
1011 The rule for website is:
1014 website: website-texinfo website-css website-pictures website-examples web-post
1015 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1016 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1017 cp $(top-htaccess) $(OUT)/.htaccess
1018 cp $(dir-htaccess) $(OUT)/website/.htaccess
1021 so we see that this starts by running the rules for 5 other
1022 targets, then finishes by copying some files. We'll cover that
1023 later - first @code{website-texinfo}. That rule is:
1026 website-texinfo: website-version website-xrefs website-bibs
1027 for l in '' $(WEB_LANGS); do \
1028 if test -n "$$l"; then \
1029 langopt=--lang="$$l"; \
1032 $(TEXI2HTML) --prefix=index \
1034 --I=$(top-src-dir)/Documentation/"$$l" \
1035 --I=$(top-src-dir)/Documentation \
1038 --init-file=$(texi2html-init-file) \
1040 --output=$(OUT)/"$$l" \
1041 $(top-src-dir)/Documentation/"$$l"/web.texi ; \
1042 ls $(OUT)/$$l/*.html | xargs grep -L \
1043 'UNTRANSLATED NODE: IGNORE ME' | \
1044 sed 's!$(OUT)/'$$l'/!!g' | xargs \
1045 $(MASS_LINK) --prepend-suffix="$$langsuf" \
1046 hard $(OUT)/$$l/ $(OUT)/website/ ; \
1050 which therefore depends on @code{website-version},
1051 @code{website-xrefs} and @code{website-bibs}.
1056 $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
1057 $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
1060 which translates as:
1063 mkdir -p out-website
1064 python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
1065 /home/phil/lilypond-git > out-website/version.itexi
1066 python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
1067 /home/phil/lilypond-git > out-website/weblinks.itexi
1070 So, we make out-website then send the output of
1071 @code{create-version-itexi.py} to @code{out-website/version.itexi}
1072 and @code{create-weblinks-itexi.py} to
1073 @code{out-website/weblinks.itexi}.
1075 @code{create-version-itexi.py} parses the file @code{VERSION} in
1076 the top source dir. It contains:
1079 PACKAGE_NAME=LilyPond
1084 VERSION_STABLE=2.14.2
1085 VERSION_DEVEL=2.15.12
1088 currently. @code{c-v-i.py} parses this to:
1091 @@c ************************ Version numbers ************
1096 @@macro versionStable
1100 @@macro versionDevel
1105 @code{create-weblinks-itexi.py} creates a load of texi macros (of
1106 the order of 1000) similar to:
1109 @@macro manualStableGlossaryPdf
1110 @@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@}
1114 It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:
1116 @code{langdefs.py: warning: lilypond-doc gettext domain not found.}
1121 website-xrefs: website-version
1122 for l in '' $(WEB_LANGS); do \
1125 is the start of the rule, truncated for brevity. This loops
1126 through the languages to be used on the website, processing some
1127 variables which I don't fully understand, to run this command:
1130 python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
1131 -I /home/phil/lilypond-git/Documentation \
1132 -I /home/phil/lilypond-git/Documentation/"$l" \
1133 -I out-website -o out-website --split=node \
1134 --known-missing-files= \
1135 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
1137 /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
1140 There's a good description of what
1141 @code{extract_texi_filenames.py} does at the top of the script,
1142 but a shortened version is:
1144 @code{If this script is run on a file texifile.texi, it produces
1145 a file texifile[.LANG].xref-map with tab-separated entries
1146 of the form NODE\tFILENAME\tANCHOR.}
1149 @code{web.nl.xref-map} is:
1152 Inleiding Introduction Introduction
1155 @code{e-t-f.py} follows the includes from document to document.
1156 We know some have not been created yet, and
1157 @code{known-missing-files} option tells @code{e-t-f.py} which
1163 for m in $(MANUALS); do \
1166 to run @code{e-t-f.py} against all of the manuals, in each
1170 website-bibs: website-version
1171 BSTINPUTS=$(top-src-dir)/Documentation/web \
1172 $(WEB_BIBS) -s web \
1173 -s $(top-src-dir)/Documentation/lily-bib \
1174 -o $(OUT)/others-did.itexi \
1176 $(top-src-dir)/Documentation/web/others-did.bib
1179 This is half the command. It runs @code{bib2texi.py} on 2
1180 @code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
1181 This converts bibliography files into texi files with
1184 Next the commands in the @code{website-texinfo} rule are run:
1187 for l in '' $(WEB_LANGS); do \
1190 run @code{texi2html}. This is the program that outputs the
1191 progress message (found in
1192 @code{Documentation/lilypond-texi2html.init}):
1194 @code{Processing web site: []}
1196 It also outputs warning messages like:
1198 @code{WARNING: Unable to find node 'ŘeÅ¡enà potÞÃ' in book usage.}
1202 cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
1205 Copies 3 css files to out-website/website. Then:
1209 mkdir -p $(OUT)/website/pictures
1210 if [ -d $(PICTURES) ]; \
1212 cp $(PICTURES)/* $(OUT)/website/pictures ; \
1213 ln -sf website/pictures $(OUT)/pictures ;\
1217 which translates as:
1220 if [ -d Documentation/pictures/out-www ]; \
1222 cp Documentation/pictures/out-www/* out-website/website/pictures ; \
1223 ln -sf website/pictures out-website/pictures ;\
1227 i.e. it copies the contents of
1228 @code{build/Documentation/pictures/out-www/*} to
1229 @code{out-website/website/pictures}. Unfortunately, the pictures
1230 are only created once @code{make doc} has been run, so an initial
1231 run of @code{make website} copies nothing, and the pictures on the
1232 website (e.g. the logo) do not exist. Next:
1236 mkdir -p $(OUT)/website/ly-examples
1237 if [ -d $(EXAMPLES) ]; \
1239 cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
1246 mkdir -p out-website/website/ly-examples
1247 if [ -d Documentation/web/ly-examples/out-www ]; \
1249 cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
1253 This does the same with the LilyPond examples (found at
1254 @uref{http://lilypond.org/examples.html}). Again, these are
1255 actually only created by @code{make doc} (and since they are
1256 generated from LilyPond source files, require a working LilyPond
1257 @code{exe} made with @code{make}). So this does nothing
1262 $(WEB_POST) $(OUT)/website
1268 python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
1271 which describes itself as:
1273 @code{This is web_post.py. This script deals with translations
1274 in the "make website" target.}
1276 It also does a number of other things, including adding the Google
1277 tracker code and the language selection footer. We're now at
1278 the end of our story. The final 4 lines of the recipe for website
1282 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1283 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1284 cp $(top-htaccess) $(OUT)/.htaccess
1285 cp $(dir-htaccess) $(OUT)/website/.htaccess
1288 The first translates as:
1291 cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
1294 so we see these are just copying the support files for the web
1297 @subsubheading website.make summary
1299 Recipes in @file{website.make}:
1305 this is the "master" rule. It calls the other rules in order,
1306 then copies some extra files around - see below for further
1307 of the process it produces.
1310 @code{website-version}:
1311 this calls the python scripts below:
1315 scripts/build/create-version-itexi.py
1318 This writes a @@version, @@versionStable, and @@versionDevel based
1319 on the top-level VERSIONS file, to
1320 @code{out-website/version.itexi}
1324 scripts/build/create-weblinks-itexi.py
1327 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1328 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1329 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1331 It's quite monstrous because it deals with combinations of
1332 stable/devel, source/docs, lang/lang/lang*10, etc.
1337 @code{website-xrefs:}
1338 creates files used for complicated "out-of-build" references to
1339 @code{out-website/*.xref-map}
1341 If you just write @@ref@{@}, then all's groovy and we wouldn't
1342 need this. But if you write @@rlearning@{@}, then our custom
1343 texi2html init file needs to know about our custom xref file
1344 format, which tells our custom texi2html init file how to create
1347 GP: we should have a separate @@node to discuss xrefs. Also, take a
1348 quick look at a generated xref file -- it's basically just a list
1349 of @@node's [sic teenager pluralization rule] from the file.
1353 generates the bibliography texinfo files from the .bib files - in
1354 the case of the website build these are @file{others-did.bib} and
1355 @file{we-wrote.bib}.
1358 @code{website-texinfo:}
1359 this is the main part; it calles texi2html to generate the actual
1360 html. It also has a ton of options to texi2html to pass info to
1361 our custom init file.
1363 The file actually built is called @file{web.texi}, and is either
1364 in the @file{Documentation} directory, or a sub-directory specific
1367 The options file is @file{/Documentation/lilypond-texi2html.init}.
1368 This contains *lots* of option and configuration stuff, and also
1372 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1375 This is where one of the console messages is generated.
1377 We have somewhere between 2-4 different ways "to pass info to our
1378 custom init file". This is highly Not Good (tm), but that's how
1379 things work at the moment.
1381 After texi2html, it does some black magick to deal with
1382 untranslated nodes in the translations. Despite writing that
1383 part, I can't remember how it works. But in theory, you could
1384 figure it out by copy&pasting each part of the command (by "part",
1385 I mean "stuff before each | pipe"), substituting the variables,
1386 then looking at the text that's output. For example,
1389 ls $(OUT)/$$l/*.html
1392 is going to print a list of all html files, in all languages, in
1393 the build directory. Then more stuff happens to each of those
1394 files (that's what xargs does).
1398 just copies files to the build dir.
1401 @code{website-pictures, website-examples:}
1402 more file copies, with an if statement to handle if you don't have
1403 any generated pictures/examples.
1410 scripts/build/website_post.py
1413 which, it adds the "this page is translated in klingon" to the
1414 bottom of html pages, and adds the google analytics javascript.
1415 It also has hard-coded lilypond version numbers, which is Bad
1420 Here's a summary of what gets called, in what order, when we run
1427 creates version.itexi and weblinks.itexi
1429 runs extract_texi_filenames.py
1431 creates bibliography files, described above
1439 runs website_post.py
1440 Then some file copying
1443 @node Building an Ubuntu distro
1444 @section Building an Ubuntu distro
1447 Here's the short instruction on how to create lilybuntu iso image
1448 (Jonathan Kulp did this on a spare drive,
1449 but he supposes it can be done in a VM too):
1454 Install ubuntu, reboot.
1456 Run all updates, reboot if asked.
1458 Enable src repos, refresh package lists.
1460 Install LilyPond build deps:
1462 sudo apt-get build-dep lilypond
1465 Install git and autoconf:
1467 sudo apt-get install git-core gitk autoconf
1471 Test to see whether everything works fine now:
1474 use @command{lily-git.tcl} to grab source files
1476 go to source dir and do
1478 "./autogen.sh" ; make ; make doc
1481 if all compiles, move on to iso creation...
1485 Download & install "remastersys":
1486 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
1488 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
1490 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
1491 default live session username, etc).
1493 Remove non-essential desktop software as desired.
1497 sudo remastersys dist
1499 New iso is in @file{/home/remastersys/remastersys/}.
1501 Test iso by installing in VM and repeating steps above for
1502 getting source files and building lp and docs.
1507 @section Building GUB
1509 GUB - the Grand Unified Builder - is used to build the release
1510 versions of LilyPond. For background information, see
1511 @ref{Grand Unified Builder (GUB)}. The simplest way to set up a
1512 GUB build environment is to use a virtual machine with LilyDev
1513 (@ref{LilyDev}). Follow the instructions on that page to set this
1514 up. Make sure that your virtual machine has enough disk space -
1515 a GUB installation takes over 30 GBytes of disk space, and if you
1516 allocate too little, it will fail during the setting up stage and
1517 you will have to start again. 64 GBytes should be sufficient.
1519 While GUB is being built, any interruptions are likely to make it
1520 almost impossible to restart. If at all possible, leave the build
1521 to continue uniterrupted.
1523 Download GUB and start the set up:
1526 git clone git://github.com/gperciva/gub/gub.git
1531 This downloads and installs a number of packages. You may find
1532 some fail during download and you will need to download them
1533 manually. For example, the perl archive. If this happens,
1535 @uref{http://www.cpan.org/src/5.0/perl-5.10.0.tar.gz}, saving the
1536 archive to @file{gub/downloads/perl/}. Continue the set up with:
1542 Once this has completed successfully, you can build the LilyPond
1543 release package. However, this uses an archived version of the
1544 regression tests, so it is better to download this first.
1545 Download the test output from lilypond.org:
1548 @uref{http://lilypond.org/download/binaries/test-output/lilypond-2.15.33-1.test-output.tar.bz2}
1551 Copy the tarball into @file{gub/regtests/}, and tell the build
1552 system that you have done this:
1555 touch regtests/ignore
1558 Now start the GUB build:
1564 That's it. This will build LilyPond from current master. To build
1565 the current unstable release, run:
1568 make LILYPOND_BRANCH=release/unstable lilypond