1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
21 @node Build system overview
22 @section Build system overview
24 Build system is currently GNU make, with an extra "stepmake" layer
25 on top. Look at files in @file{make/} and @file{stepmake/} and
26 all @file{GNUmakefile}s.
28 There is wide-spread dissatisfaction with this system, and we are
29 considering changing. This would be a huge undertaking (estimated
30 200+ hours). This change will probably involve not using GNU make
31 any more -- but a discussion about the precise build system will
32 have to wait. Before we reach that point, we need to figure out
33 (at least approximately) what the current build system does.
35 Fundamentally, a build system does two things:
39 Constructs command-line commands, for example:
43 --tons --of --options \
46 --more --imperial --and --metric --tons --of --options \
51 If there was a previous build, it decides which parts of the
52 system need to be rebuilt.
56 When I try to do anything in the build system, it helps to remind
57 myself of this. The "end result" is just a series of command-line
58 commands. All the black magick is just an attempt to construct
61 @node Tips for working on the build system
62 @section Tips for working on the build system
74 to the build system files in various places. This will let you
75 track where the program is, in various points of the build.
77 PH note. There are lots of places where Make doesn't let you put
78 echo commands. My top tip for tracing how make runs is to put
81 $(error Some Text to display)
84 This will stop make running and print the text @code{Some Text to
90 First task: understand how @code{make website} works,
91 @emph{without} the translations. Looking at the english-only
92 website is the best introduction to the build system... it only
93 covers about 5% of the whole thing, but even that will likely take
99 @node General build system notes
100 @section General build system notes
103 * How stepmake works::
106 @node How stepmake works
107 @subsection How stepmake works
109 Typing make website runs the file @file{GNUmakefile} from the
110 build directory. This only contains 3 lines:
114 include config$(if $(conf),-$(conf),).make
115 include $(configure-srcdir)/GNUmakefile.in
118 The variable @code{depth} is used throughout the make system to
119 track how far down the directory structure the make is. The first
120 include sets lots of variables but doesn't "do" anything. Default
121 values for these variables are automatically detected at the
122 ./configure step, which creates the file @file{config.make}.
123 The second include runs the file @file{GNUmakefile.in} from
124 the top level source directory.
126 This sets another load of variables, and then includes (i.e.
127 immediately runs) @file{stepmake.make} from the @file{make}
128 subdirectory. This sets a load of other variables, does some
129 testing to see if SCONS (another build tool?) is being used, and
130 then runs @file{make/config.make} - which doesn't seem to exist...
132 GP: scons is indeed a different build tool; I think that Jan
133 experimented with it 5 years ago or something. It seems like we
134 still have bits and pieces of it floating around.
136 Next, it runs @file{make/toplevel-version.make}, which sets the
137 version variables for major, minor, patch, stable, development and
138 mypatchlevel (which seems to be used for patch numbers for
139 non-stable versions only?).
141 Next - @file{make/local.make}, which doesn't exist.
143 Then a few more variable and the interesting comment:
146 # Don't try to outsmart us, you puny computer!
147 # Well, UGH. This only removes builtin rules from
150 and then tests to see whether BUILTINS_REMOVED is defined. It
151 appears to be when I run make, and so
152 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
153 comment at the head of this file says:
156 # UGH. GNU make comes with implicit rules.
157 # We don't want any of them, and can't force users to run
161 I've not studied that file at length, but assume it removes all
162 make's build-in rules (e.g. @file{*.c} files are run through the
163 GNU C compiler) - there's a lot of them in here, and a lot of
164 comments, and I'd guess most of it isn't needed.
166 We return to @file{stepmake.make}, where we hit the make rule all:
167 The first line of this is:
170 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
173 which, when the variables are substituted, gives:
176 ./make/generic-inclusions.make
177 ./make/lilypond-inclusions.make.
180 (Note - according to the make documentation, -include is only
181 different from include in that it doesn't produce any kind of
182 error message when the included file doesn't exist).
184 And the first file doesn't exist. Nor the second. Next:
187 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
190 which expands to the following files:
193 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
195 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
196 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
199 One little feature to notice here - these are all absolute file
200 locations - the line prior to this used relative locations. And
201 none of these files exist, either. (Further note - I'm assuming
202 all these lines of make I'm following are autogenerated, but
203 that'll be something else to discover.)
205 Next in @file{stepmake.make}:
208 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
214 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
215 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
216 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
217 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
220 Woo. They all exist (they should as there's no - in front of the
221 include). @file{generic-vars.make} sets loads of variables
222 (funnily enough). @file{toplevel-vars.make} is very short - one
223 line commented as @code{# override Generic_vars.make:} and 2 as
228 include $(stepdir)/documentation-vars.make
231 I assume the urg comment refers to the fact that this should
232 really just create more variables, but it actually sends us off to
233 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
235 That file is a 3 line variable setting one.
237 @file{po-vars.make} has the one-line comment @code{# empty}, as
238 does @file{install-vars.make}.
240 So now we're back to @file{stepmake.make}.
245 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
246 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
249 and the include expands to:
252 include ./make/generic-vars.make ./make/lilypond-vars.make.
255 These again set variables, and in some cases export them to allow
256 child @code{make} processes to use them.
258 The final 4 lines of @file{stepmake.make} are:
261 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
262 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
263 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
264 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
267 which expand as follows:
270 include ./make/generic-rules.make ./make/lilypond-rules.make
272 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
273 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
274 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
275 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
276 include ./make/generic-targets.make ./make/lilypond-targets.make
278 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
279 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
280 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
281 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
284 @file{lilypond-rules.make} is @code{#empty}
286 @file{generic-rules.make} does seem to have 2 rules in it. They
290 $(outdir)/%.ly: %.lym4
291 $(M4) $< | sed "s/\`/,/g" > $@
295 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
298 I believe the first rule is for *.ly files, and has a prerequisite
299 that *.lym4 files must be built first. The recipe is @code{m4 |
300 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
301 can comment on exactly what the rules mean/do.
303 @file{toplevel-rules.make} is @code{#empty}
305 @file{po-rules.make} is @code{#empty}
307 @file{install-rules.make} is @code{#empty}
309 @file{generic-targets.make} contains 2 lines of comments.
311 @file{lilypond-targets.make} contains only:
314 ## TODO: fail dist or web if no \version present.
316 grep -L version $(LY_FILES)
319 @file{stepmake/generic-targets.make} contains lots of rules - too
320 many to list here - it seems to be the main file for rules. (FWIW
321 I haven't actually found a rule for website: anywhere, although
322 it clearly exists. I have also found that you can display a rule
323 in the terminal by typing, say @code{make -n website}. This is
324 probably common knowledge.
326 @file{stepmake/toplevel-targets.make} adds a load of other (and
327 occasionally the same) rules to the gernric-targets.
329 @file{stepmake/po-targets.make} is rules for po* makes.
331 @file{stepmake/install-targets.make} has rules for local-install*.
333 And that's the end of stepmake.make. Back to
334 @file{GNUmakefile.in}.
336 A bit more info from 27 March. I've put some error traces into
337 @code{GNUmakefile} in the build directory, and it looks like the
338 following lines actually cause the make to run (putting an error
339 call above them - no make; below them - make):
343 # All web targets, except info image symlinks and info docs are
344 # installed in non-recursing target from TOP-SRC-DIR
346 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
347 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
348 $(MAKE) -C Documentation omf-local-install
351 I don't currently understand the @code{ifeq}, since @code{$(out)}
352 is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
356 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
357 -c -m 755 -d /usr/local/share/doc/lilypond/html
360 End of work for Sunday 27th.
362 Another alterative approach to understanding the website build
363 would be to redirect @code{make -n website} and @code{make website}
364 to a text file and work through a) what it does and b) where the
365 errors are occurring.
367 GP: wow, all the above is much more complicated than I've ever
368 looked at stuff -- I tend to do a "back first" approach (where I
369 begin from the command-line that I want to modify, figure out
370 where it's generated, and then figure out how to change the
371 generated command-line), rather than a "front first" (where you
372 begin from the "make" command).
379 * The function of make doc::
380 * Building a bibliography::
383 @node The function of make doc
384 @subsection The function of make doc
386 The following is a set of notes on how make doc functions.
388 Preliminary question to be answered some time: where do all the
389 GNUmakefiles come from. They're in the build directory, but this
390 is not part of source. Must be the configure script. And it
391 looks like this comes from autogen.sh. Must at some point kill
392 the whole git directory, repull and see what is created when.
394 Anyway, here's how make doc progresses:
396 This is the build dependency tree from
397 @file{stepmake/stepmake/generic-targets.make}:
402 $(MAKE) -C $(depth)/scripts/build out=
403 $(MAKE) out=www WWW-1
406 $(MAKE) out=www WWW-2
409 $(MAKE) out=www WWW-post
413 MAKE = make --no-builtin-rules
414 -C = Change to directory before make
417 doc-stage-1 does lots of opening and looking in files, but no
423 + make PACKAGE=LILYPOND package=lilypond -C python
424 && make PACKAGE=LILYPOND package=lilypond -C scripts
425 && make PACKAGE=LILYPOND package=lilypond -C flower
426 && make PACKAGE=LILYPOND package=lilypond -C lily
427 && make PACKAGE=LILYPOND package=lilypond -C mf
428 && make PACKAGE=LILYPOND package=lilypond -C ly
429 && make PACKAGE=LILYPOND package=lilypond -C tex
430 && make PACKAGE=LILYPOND package=lilypond -C ps
431 && make PACKAGE=LILYPOND package=lilypond -C scm
432 && make PACKAGE=LILYPOND package=lilypond -C po
433 && make PACKAGE=LILYPOND package=lilypond -C make
434 && make PACKAGE=LILYPOND package=lilypond -C elisp
435 && make PACKAGE=LILYPOND package=lilypond -C vim
436 && make PACKAGE=LILYPOND package=lilypond -C input
437 && make PACKAGE=LILYPOND package=lilypond -C stepmake
438 && make PACKAGE=LILYPOND package=lilypond -C Documentation
444 stepmake/stepmake/generic-vars.make has this:
447 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true
450 $@ is the name of the target - WWW-1 in this case.
452 In GNUmakefile.in we find:
455 SUBDIRS = python scripts \
462 stepmake $(documentation-dir)
465 So that's how we get the main make loop...
467 That loop expands like this:
470 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
471 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
472 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
473 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
474 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
475 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
476 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
477 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
484 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
485 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
489 The directories up to and including vim produce no effect with
490 make in non-debug mode, although debug does show lots of action.
492 @file{git/build/input/GNUmakefile} is:
496 include $(depth)/config$(if $(conf),-$(conf),).make
497 include $(configure-srcdir)/./input/GNUmakefile
498 MODULE_INCLUDES += $(src-dir)/$(outbase)
501 The first include is:
507 (note the // which is strictly wrong)
509 which has lots of variables to set, but no action occurs.
514 lilypond-git/./input/GNUmakefile
517 which similarly doesn't create any actual action.
519 An error message at the end of build/input/GNUmakefile stops
520 make processing before it moves on to regression - so where does
523 And the answer is - make processes all directories in the
524 directory it's entered (with some exceptions like out and out-www)
525 and so it changes to /regression.
527 It then seems to consider whether it needs to make/remake loads of
528 makefiles. Don't understand this yet. Possibly these are all the
529 makefiles it's processing, and it always checks they're up to date
530 before processing other files?
532 Could be correct - some of this output is:
535 Must remake target `../../make/ly-inclusions.make'.
536 Failed to remake target file `../../make/ly-inclusions.make'.
539 Having decided that, it then leaves the directory and re-executes:
542 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
545 The top of this make is:
548 This program built for i486-pc-linux-gnu
550 Reading makefile `GNUmakefile'...
551 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
554 which looks like it's re-reading all its known makefiles to check
557 (From the make manual:
559 To this end, after reading in all makefiles, make will consider each as a goal target and
560 attempt to update it. If a makefile has a rule which says how to update it (found either
561 in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10
562 [Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have
563 been checked, if any have actually been changed, make starts with a clean slate and reads
564 all the makefiles over again. (It will also attempt to update each of them over again, but
565 normally this will not change them again, since they are already up to date.)
567 So my assumption seems correct)
569 There appear to be about 74 of them. After all the makefile
570 checking, we get this:
573 Updating goal targets....
574 Considering target file `WWW-1'.
575 File `WWW-1' does not exist.
576 Considering target file `local-WWW-1'.
577 File `local-WWW-1' does not exist.
578 Considering target file `out-www/collated-files.texi'.
579 File `out-www/collated-files.texi' does not exist.
580 Looking for an implicit rule for `out-www/collated-files.texi'.
581 Trying pattern rule with stem `collated-files.texi'.
582 Trying implicit prerequisite `collated-files.texi.in'.
583 Trying pattern rule with stem `collated-files.texi'.
584 Trying implicit prerequisite `collated-files.texi.in'.
585 Trying pattern rule with stem `collated-files'.
586 Trying implicit prerequisite `collated-files.tely'.
587 Trying pattern rule with stem `collated-files'.
588 Trying implicit prerequisite `out-www/collated-files.tely'.
589 Trying rule prerequisite `out-www/version.itexi'.
590 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
593 grep finds this if searching for local-WWW-1:
596 make/lysdoc-targets.make:
597 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
600 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
601 and so these will need to be checked to see if they're up to date.
602 So make needs to find rules for both of those and (as it says) it
603 certainly needs to make coll*.texi, since it doesn't exist.
605 In ly-rules.make we have:
608 .SUFFIXES: .doc .tely .texi .ly
611 which I'll work out at some point, and also this rule:
614 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
615 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
618 Note that the recipe is a very long line - it could probably
619 benefit from splitting. The same makefile also has:
622 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
623 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
627 which seems to be an almost exact duplicate. Whatever, the first
628 one is executed first. Have not checked if the second executes.
630 The first recipe translates as this:
633 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
634 --output=./out-www --format= --lily-output-dir \
635 /home/phil/lilypond-git/build/out/lybook-db
639 if we stop the build with an $(error), but I think this is because
640 we need to allow it to process the dependencies first. It looks
641 like foo.texi is shown as being dependent on foo.tely, plus a load
645 DOCUMENTATION_LOCALE_TARGET is blank
646 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
647 /home/phil/lilypond-git/scm/autochange.scm
650 plus 10s (100s?) of other .scm files.
653 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
654 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
657 ditto .ly files. This does seem a teency bit wrong - it looks like
658 the .ly and .scm files have been interchanged. ly-vars.make has
662 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
663 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
666 Looks like a bug.....
668 So it now works its way through all these files, checking if they
669 need to be remade. This is 100s of lines of the debug listing,
670 although none in the normal list. Clearly none has to be made
671 since they're source files. It concludes:
674 Must remake target `out-www/collated-files.tely'
677 @file{lysdoc-rules.make} has this:
680 $(outdir)/collated-files.tely: $(COLLATED_FILES)
681 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
684 @file{lysdoc-vars.make} has:
687 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
693 TEXINFO_SOURCES = AAA-intro-regression.tely
694 OUT_LY_FILES is empty
697 so LY_FILES has the big long list of all the .ly files in the
698 regression directory.
703 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
706 with a list of all the files in the regression test directory. This
707 should (I believe) create the file collated-files.tely.
709 So the next rule in make is for @file{version.itexi}, and make duly
710 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
711 depends on @file{git/VERSION}:
714 $(outdir)/version.%: $(top-src-dir)/VERSION
715 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
718 This causes create-version-itexi.py to run and create
721 Once that's done, all the other *.scm and *.ly files are checked
722 and since they have no rules associated, they aren't remade (just
723 as well for source files, really). Since version.itexi was remade
724 make concludes that collated-files.texi must be remade. To do
725 this, it runs lilypond-book.py on collated-files.tely, as below:
728 LILYPOND_VERSION=2.15.0
730 /home/phil/lilypond-git/scripts/lilypond-book.py
731 -I /home/phil/lilypond-git/input/regression/
732 -I ./out-www -I /home/phil/lilypond-git/input
733 -I /home/phil/lilypond-git/Documentation
734 -I /home/phil/lilypond-git/Documentation/snippets
735 -I /home/phil/lilypond-git/input/regression/
736 -I /home/phil/lilypond-git/Documentation/included/
737 -I /home/phil/lilypond-git/build/mf/out/
738 -I /home/phil/lilypond-git/build/mf/out/
739 -I /home/phil/lilypond-git/Documentation/pictures
740 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
741 --process='/home/phil/lilypond-git/build/out/bin/lilypond
742 -I /home/phil/lilypond-git/input/regression/
744 -I /home/phil/lilypond-git/input
745 -I /home/phil/lilypond-git/Documentation
746 -I /home/phil/lilypond-git/Documentation/snippets
747 -I /home/phil/lilypond-git/input/regression/
748 -I /home/phil/lilypond-git/Documentation/included/
749 -I /home/phil/lilypond-git/build/mf/out/
750 -I /home/phil/lilypond-git/build/mf/out/
751 -I /home/phil/lilypond-git/Documentation/pictures
752 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
777 -dcheck-internal-types
779 -danti-alias-factor=2'
783 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
784 out-www/collated-files.tely
787 So - lilypond-book runs on:
790 input/regression/out-www/collated-files.tely
794 Note the --verbose flag - this is from the make variable
795 LILYPOND_BOOK_VERBOSE which is added to the make variable
798 Now found the invocation to write some of the image files. It's
802 /home/phil/lilypond-git/build/out/bin/lilypond
803 -I /home/phil/lilypond-git/input/regression/
804 -I ./out-www -I /home/phil/lilypond-git/input
805 -I /home/phil/lilypond-git/Documentation
806 -I /home/phil/lilypond-git/Documentation/snippets
807 -I /home/phil/lilypond-git/input/regression/
808 -I /home/phil/lilypond-git/Documentation/included/
809 -I /home/phil/lilypond-git/build/mf/out/
810 -I /home/phil/lilypond-git/build/mf/out/
811 -I /home/phil/lilypond-git/Documentation/pictures
812 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
837 -dcheck-internal-types
839 -danti-alias-factor=2
840 -I "/home/phil/lilypond-git/build/out/lybook-db"
841 -I "/home/phil/lilypond-git/build/input/regression"
842 -I "/home/phil/lilypond-git/input/regression"
843 -I "/home/phil/lilypond-git/build/input/regression/out-www"
844 -I "/home/phil/lilypond-git/input"
845 -I "/home/phil/lilypond-git/Documentation"
846 -I "/home/phil/lilypond-git/Documentation/snippets"
847 -I "/home/phil/lilypond-git/input/regression"
848 -I "/home/phil/lilypond-git/Documentation/included"
849 -I "/home/phil/lilypond-git/build/mf/out"
850 -I "/home/phil/lilypond-git/build/mf/out"
851 -I "/home/phil/lilypond-git/Documentation/pictures"
852 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
855 -deps-box-padding=3.000000
857 -dno-strip-output-dir
858 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
861 Note the --verbose. This causes 100s of lines of Lily debug output.
862 But at present I can't work out where the flag comes from. Later.
865 @node Building a bibliography
866 @subsection Building a bibliography
868 Bibliography files contain a list of citations, like this:
872 author = @{Vinci, Albert C.@},
873 title = @{Fundamentals of Traditional Music Notation@},
874 publisher = @{Kent State University Press@},
879 There are a variety of types of citation (e.g. Book (as above),
880 article, publication). Each cited publication has a list of
881 entries that can be used to identify the publication.
882 Bibliograpies are normally stored as files with a .bib
883 extension. One part of the doc-build process is transforming the
884 bibliography information into @code{texinfo} files. The commands
885 to do this are in the @file{GNUmakefile} in the
886 @file{Documentation} directory.
888 A typical line of the makefile to translate a single bibliography
892 $(outdir)/colorado.itexi:
893 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
894 -s $(top-src-dir)/Documentation/lily-bib \
895 -o $(outdir)/colorado.itexi \
896 $(src-dir)/essay/colorado.bib
902 $(outdir)/colorado.itexi:
905 We're making the file @file{colorado.itexi} and so this is the
909 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
912 It's in the @file{essay} directory and we want to run the
913 bib2texi.py script against it.
916 -s $(top-src-dir)/Documentation/lily-bib \
919 The style template is @file{lily-bib.bst} and is found in the
920 @file{Documentation} directory.
923 -o $(outdir)/colorado.itexi \
926 The output file in @file{colorado.itexi}.
929 $(src-dir)/essay/colorado.bib
932 The input file is @file{colorado.bib} in the @file{essay}
935 The @code{bib2texi} Python script used to be used with a variety
936 of options, but now is always called using the same options, as
937 above. Its job is to create the file containing the options for
938 @code{bibtex} (the program that actually does the translation),
939 run bibtex, and then clean up some temporary files. Its main
940 "value add" is the creation of the options file, using this code:
943 open (tmpfile + '.aux', 'w').write (r'''
946 \bibstyle@{%(style)s@}
947 \bibdata@{%(files)s@}''' % vars ())
950 The key items are the style file (now always lily-bib for us) and
953 The style file is written in its own specialised language,
954 described to some extent at
957 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
960 The file @file{lily-bib.bst} also has fairly extensive commenting.
964 @section Website build
966 @warning{This information applies only to the standard @code{make
967 website} from the normal build directory. The process is
968 different for @code{dev/website-build}.}
970 The rule for make website is found in GNUmakefile.in:
974 $(MAKE) config_make=$(config_make) \
975 top-src-dir=$(top-src-dir) \
976 -f $(top-src-dir)/make/website.make \
983 make --no-builtin-rules config_make=./config.make \
984 top-src-dir=/home/phil/lilypond-git \
985 -f /home/phil/lilypond-git/make/website.make \
989 which has the effect of setting the variables @code{config_make}
990 and @code{top-src-dir} and then processing the file
991 @code{git/make/website.make} with the target of website.
993 @code{website.make} starts with the following:
996 ifeq ($(WEBSITE_ONLY_BUILD),1)
999 which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
1000 was set to one on the command line. This is only done for
1001 standalone website builds, not in the normal case. The result of
1002 the test determines the value of some variables that are set. A
1003 number of other variables are set, in order to establish locations
1004 of various files. An example is:
1007 CREATE_VERSION=python $(script-dir)/create-version-itexi.py
1010 The rule for website is:
1013 website: website-texinfo website-css website-pictures website-examples web-post
1014 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1015 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1016 cp $(top-htaccess) $(OUT)/.htaccess
1017 cp $(dir-htaccess) $(OUT)/website/.htaccess
1020 so we see that this starts by running the rules for 5 other
1021 targets, then finishes by copying some files. We'll cover that
1022 later - first @code{website-texinfo}. That rule is:
1025 website-texinfo: website-version website-xrefs website-bibs
1026 for l in '' $(WEB_LANGS); do \
1027 if test -n "$$l"; then \
1028 langopt=--lang="$$l"; \
1031 $(TEXI2HTML) --prefix=index \
1033 --I=$(top-src-dir)/Documentation/"$$l" \
1034 --I=$(top-src-dir)/Documentation \
1037 --init-file=$(texi2html-init-file) \
1039 --output=$(OUT)/"$$l" \
1040 $(top-src-dir)/Documentation/"$$l"/web.texi ; \
1041 ls $(OUT)/$$l/*.html | xargs grep -L \
1042 'UNTRANSLATED NODE: IGNORE ME' | \
1043 sed 's!$(OUT)/'$$l'/!!g' | xargs \
1044 $(MASS_LINK) --prepend-suffix="$$langsuf" \
1045 hard $(OUT)/$$l/ $(OUT)/website/ ; \
1049 which therefore depends on @code{website-version},
1050 @code{website-xrefs} and @code{website-bibs}.
1055 $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
1056 $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
1059 which translates as:
1062 mkdir -p out-website
1063 python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
1064 /home/phil/lilypond-git > out-website/version.itexi
1065 python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
1066 /home/phil/lilypond-git > out-website/weblinks.itexi
1069 So, we make out-website then send the output of
1070 @code{create-version-itexi.py} to @code{out-website/version.itexi}
1071 and @code{create-weblinks-itexi.py} to
1072 @code{out-website/weblinks.itexi}.
1074 @code{create-version-itexi.py} parses the file @code{VERSION} in
1075 the top source dir. It contains:
1078 PACKAGE_NAME=LilyPond
1083 VERSION_STABLE=2.14.2
1084 VERSION_DEVEL=2.15.12
1087 currently. @code{c-v-i.py} parses this to:
1090 @@c ************************ Version numbers ************
1095 @@macro versionStable
1099 @@macro versionDevel
1104 @code{create-weblinks-itexi.py} creates a load of texi macros (of
1105 the order of 1000) similar to:
1108 @@macro manualStableGlossaryPdf
1109 @@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@}
1113 It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:
1115 @code{langdefs.py: warning: lilypond-doc gettext domain not found.}
1120 website-xrefs: website-version
1121 for l in '' $(WEB_LANGS); do \
1124 is the start of the rule, truncated for brevity. This loops
1125 through the languages to be used on the website, processing some
1126 variables which I don't fully understand, to run this command:
1129 python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
1130 -I /home/phil/lilypond-git/Documentation \
1131 -I /home/phil/lilypond-git/Documentation/"$l" \
1132 -I out-website -o out-website --split=node \
1133 --known-missing-files= \
1134 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
1136 /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
1139 There's a good description of what
1140 @code{extract_texi_filenames.py} does at the top of the script,
1141 but a shortened version is:
1143 @code{If this script is run on a file texifile.texi, it produces
1144 a file texifile[.LANG].xref-map with tab-separated entries
1145 of the form NODE\tFILENAME\tANCHOR.}
1148 @code{web.nl.xref-map} is:
1151 Inleiding Introduction Introduction
1154 @code{e-t-f.py} follows the includes from document to document.
1155 We know some have not been created yet, and
1156 @code{known-missing-files} option tells @code{e-t-f.py} which
1162 for m in $(MANUALS); do \
1165 to run @code{e-t-f.py} against all of the manuals, in each
1169 website-bibs: website-version
1170 BSTINPUTS=$(top-src-dir)/Documentation/web \
1171 $(WEB_BIBS) -s web \
1172 -s $(top-src-dir)/Documentation/lily-bib \
1173 -o $(OUT)/others-did.itexi \
1175 $(top-src-dir)/Documentation/web/others-did.bib
1178 This is half the command. It runs @code{bib2texi.py} on 2
1179 @code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
1180 This converts bibliography files into texi files with
1183 Next the commands in the @code{website-texinfo} rule are run:
1186 for l in '' $(WEB_LANGS); do \
1189 run @code{texi2html}. This is the program that outputs the
1190 progress message (found in
1191 @code{Documentation/lilypond-texi2html.init}):
1193 @code{Processing web site: []}
1195 It also outputs warning messages like:
1197 @code{WARNING: Unable to find node 'ŘeÅ¡enà potÞÃ' in book usage.}
1201 cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
1204 Copies 3 css files to out-website/website. Then:
1208 mkdir -p $(OUT)/website/pictures
1209 if [ -d $(PICTURES) ]; \
1211 cp $(PICTURES)/* $(OUT)/website/pictures ; \
1212 ln -sf website/pictures $(OUT)/pictures ;\
1216 which translates as:
1219 if [ -d Documentation/pictures/out-www ]; \
1221 cp Documentation/pictures/out-www/* out-website/website/pictures ; \
1222 ln -sf website/pictures out-website/pictures ;\
1226 i.e. it copies the contents of
1227 @code{build/Documentation/pictures/out-www/*} to
1228 @code{out-website/website/pictures}. Unfortunately, the pictures
1229 are only created once @code{make doc} has been run, so an initial
1230 run of @code{make website} copies nothing, and the pictures on the
1231 website (e.g. the logo) do not exist. Next:
1235 mkdir -p $(OUT)/website/ly-examples
1236 if [ -d $(EXAMPLES) ]; \
1238 cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
1245 mkdir -p out-website/website/ly-examples
1246 if [ -d Documentation/web/ly-examples/out-www ]; \
1248 cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
1252 This does the same with the LilyPond examples (found at
1253 @uref{http://lilypond.org/examples.html}). Again, these are
1254 actually only created by @code{make doc} (and since they are
1255 generated from LilyPond source files, require a working LilyPond
1256 @code{exe} made with @code{make}). So this does nothing
1261 $(WEB_POST) $(OUT)/website
1267 python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
1270 which describes itself as:
1272 @code{This is web_post.py. This script deals with translations
1273 in the "make website" target.}
1275 It also does a number of other things, including adding the Google
1276 tracker code and the language selection footer. We're now at
1277 the end of our story. The final 4 lines of the recipe for website
1281 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1282 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1283 cp $(top-htaccess) $(OUT)/.htaccess
1284 cp $(dir-htaccess) $(OUT)/website/.htaccess
1287 The first translates as:
1290 cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
1293 so we see these are just copying the support files for the web
1296 @subsubheading website.make summary
1298 Recipes in @file{website.make}:
1304 this is the "master" rule. It calls the other rules in order,
1305 then copies some extra files around - see below for further
1306 of the process it produces.
1309 @code{website-version}:
1310 this calls the python scripts below:
1314 scripts/build/create-version-itexi.py
1317 This writes a @@version, @@versionStable, and @@versionDevel based
1318 on the top-level VERSIONS file, to
1319 @code{out-website/version.itexi}
1323 scripts/build/create-weblinks-itexi.py
1326 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1327 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1328 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1330 It's quite monstrous because it deals with combinations of
1331 stable/devel, source/docs, lang/lang/lang*10, etc.
1336 @code{website-xrefs:}
1337 creates files used for complicated "out-of-build" references to
1338 @code{out-website/*.xref-map}
1340 If you just write @@ref@{@}, then all's groovy and we wouldn't
1341 need this. But if you write @@rlearning@{@}, then our custom
1342 texi2html init file needs to know about our custom xref file
1343 format, which tells our custom texi2html init file how to create
1346 GP: we should have a separate @@node to discuss xrefs. Also, take a
1347 quick look at a generated xref file -- it's basically just a list
1348 of @@node's [sic teenager pluralization rule] from the file.
1352 generates the bibliography texinfo files from the .bib files - in
1353 the case of the website build these are @file{others-did.bib} and
1354 @file{we-wrote.bib}.
1357 @code{website-texinfo:}
1358 this is the main part; it calles texi2html to generate the actual
1359 html. It also has a ton of options to texi2html to pass info to
1360 our custom init file.
1362 The file actually built is called @file{web.texi}, and is either
1363 in the @file{Documentation} directory, or a sub-directory specific
1366 The options file is @file{/Documentation/lilypond-texi2html.init}.
1367 This contains *lots* of option and configuration stuff, and also
1371 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1374 This is where one of the console messages is generated.
1376 We have somewhere between 2-4 different ways "to pass info to our
1377 custom init file". This is highly Not Good (tm), but that's how
1378 things work at the moment.
1380 After texi2html, it does some black magick to deal with
1381 untranslated nodes in the translations. Despite writing that
1382 part, I can't remember how it works. But in theory, you could
1383 figure it out by copy&pasting each part of the command (by "part",
1384 I mean "stuff before each | pipe"), substituting the variables,
1385 then looking at the text that's output. For example,
1388 ls $(OUT)/$$l/*.html
1391 is going to print a list of all html files, in all languages, in
1392 the build directory. Then more stuff happens to each of those
1393 files (that's what xargs does).
1397 just copies files to the build dir.
1400 @code{website-pictures, website-examples:}
1401 more file copies, with an if statement to handle if you don't have
1402 any generated pictures/examples.
1409 scripts/build/website_post.py
1412 which, it adds the "this page is translated in klingon" to the
1413 bottom of html pages, and adds the google analytics javascript.
1414 It also has hard-coded lilypond version numbers, which is Bad
1419 Here's a summary of what gets called, in what order, when we run
1426 creates version.itexi and weblinks.itexi
1428 runs extract_texi_filenames.py
1430 creates bibliography files, described above
1438 runs website_post.py
1439 Then some file copying
1442 @node Building an Ubuntu distro
1443 @section Building an Ubuntu distro
1446 Here's the short instruction on how to create lilybuntu iso image
1447 (Jonathan Kulp did this on a spare drive,
1448 but he supposes it can be done in a VM too):
1453 Install ubuntu, reboot.
1455 Run all updates, reboot if asked.
1457 Enable src repos, refresh package lists.
1459 Install LilyPond build deps:
1461 sudo apt-get build-dep lilypond
1464 Install git and autoconf:
1466 sudo apt-get install git-core gitk autoconf
1470 Test to see whether everything works fine now:
1473 use @command{lily-git.tcl} to grab source files
1475 go to source dir and do
1477 "./autogen.sh" ; make ; make doc
1480 if all compiles, move on to iso creation...
1484 Download & install "remastersys":
1485 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
1487 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
1489 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
1490 default live session username, etc).
1492 Remove non-essential desktop software as desired.
1496 sudo remastersys dist
1498 New iso is in @file{/home/remastersys/remastersys/}.
1500 Test iso by installing in VM and repeating steps above for
1501 getting source files and building lp and docs.