1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
21 @node Build system overview
22 @section Build system overview
24 Build system is currently GNU make, with an extra "stepmake" layer
25 on top. Look at files in @file{make/} and @file{stepmake/} and
26 all @file{GNUmakefile}s.
28 There is wide-spread dissatisfaction with this system, and we are
29 considering changing. This would be a huge undertaking (estimated
30 200+ hours). This change will probably involve not using GNU make
31 any more -- but a discussion about the precise build system will
32 have to wait. Before we reach that point, we need to figure out
33 (at least approximately) what the current build system does.
35 Fundamentally, a build system does two things:
39 Constructs command-line commands, for example:
43 --tons --of --options \
46 --more --imperial --and --metric --tons --of --options \
51 If there was a previous build, it decides which parts of the
52 system need to be rebuilt.
56 When I try to do anything in the build system, it helps to remind
57 myself of this. The "end result" is just a series of command-line
58 commands. All the black magick is just an attempt to construct
61 @node Tips for working on the build system
62 @section Tips for working on the build system
74 to the build system files in various places. This will let you
75 track where the program is, in various points of the build.
77 PH note. There are lots of places where Make doesn't let you put
78 echo commands. My top tip for tracing how make runs is to put
81 $(error Some Text to display)
84 This will stop make running and print the text @code{Some Text to
90 First task: understand how @code{make website} works,
91 @emph{without} the translations. Looking at the english-only
92 website is the best introduction to the build system... it only
93 covers about 5% of the whole thing, but even that will likely take
99 @node General build system notes
100 @section General build system notes
103 * How stepmake works::
106 @node How stepmake works
107 @subsection How stepmake works
109 Typing make website runs the file @file{GNUmakefile} from the
110 build directory. This only contains 3 lines:
114 include config$(if $(conf),-$(conf),).make
115 include $(configure-srcdir)/GNUmakefile.in
118 The variable @code{depth} is used throughout the make system to
119 track how far down the directory structure the make is. The first
120 include sets lots of variables but doesn't "do" anything. Default
121 values for these variables are automatically detected at the
122 ./configure step, which creates the file @file{config.make}.
123 The second include runs the file @file{GNUmakefile.in} from
124 the top level source directory.
126 This sets another load of variables, and then includes (i.e.
127 immediately runs) @file{stepmake.make} from the @file{make}
128 subdirectory. This sets a load of other variables, does some
129 testing to see if SCONS (another build tool?) is being used, and
130 then runs @file{make/config.make} - which doesn't seem to exist...
132 GP: scons is indeed a different build tool; I think that Jan
133 experimented with it 5 years ago or something. It seems like we
134 still have bits and pieces of it floating around.
136 Next, it runs @file{make/toplevel-version.make}, which sets the
137 version variables for major, minor, patch, stable, development and
138 mypatchlevel (which seems to be used for patch numbers for
139 non-stable versions only?).
141 Next - @file{make/local.make}, which doesn't exist.
143 Then a few more variable and the interesting comment:
146 # Don't try to outsmart us, you puny computer!
147 # Well, UGH. This only removes builtin rules from
150 and then tests to see whether BUILTINS_REMOVED is defined. It
151 appears to be when I run make, and so
152 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
153 comment at the head of this file says:
156 # UGH. GNU make comes with implicit rules.
157 # We don't want any of them, and can't force users to run
161 I've not studied that file at length, but assume it removes all
162 make's build-in rules (e.g. @file{*.c} files are run through the
163 GNU C compiler) - there's a lot of them in here, and a lot of
164 comments, and I'd guess most of it isn't needed.
166 We return to @file{stepmake.make}, where we hit the make rule all:
167 The first line of this is:
170 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
173 which, when the variables are substituted, gives:
176 ./make/generic-inclusions.make
177 ./make/lilypond-inclusions.make.
180 (Note - according to the make documentation, -include is only
181 different from include in that it doesn't produce any kind of
182 error message when the included file doesn't exist).
184 And the first file doesn't exist. Nor the second. Next:
187 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
190 which expands to the following files:
193 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
195 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
196 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
199 One little feature to notice here - these are all absolute file
200 locations - the line prior to this used relative locations. And
201 none of these files exist, either.
203 (Further note - I'm assuming all these lines of make I'm following are
204 autogenerated, but that'll be something else to discover.)
206 JM: @emph{``No, these lines are not useful in LilyPond (this is why
207 you think they are autogenerated), but they are part of StepMake,
208 which was meant to be a package to be installed as a build system over
209 autoconf/make in software project source trees.''}
211 Next in @file{stepmake.make}:
214 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
220 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
221 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
222 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
223 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
226 Woo. They all exist (they should as there's no - in front of the
227 include). @file{generic-vars.make} sets loads of variables
228 (funnily enough). @file{toplevel-vars.make} is very short - one
229 line commented as @code{# override Generic_vars.make:} and 2 as
234 include $(stepdir)/documentation-vars.make
237 I assume the urg comment refers to the fact that this should
238 really just create more variables, but it actually sends us off to
239 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
241 That file is a 3 line variable setting one.
243 @file{po-vars.make} has the one-line comment @code{# empty}, as
244 does @file{install-vars.make}.
246 So now we're back to @file{stepmake.make}.
251 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
252 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
255 and the include expands to:
258 include ./make/generic-vars.make ./make/lilypond-vars.make.
261 These again set variables, and in some cases export them to allow
262 child @code{make} processes to use them.
264 The final 4 lines of @file{stepmake.make} are:
267 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
268 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
269 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
270 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
273 which expand as follows:
276 include ./make/generic-rules.make ./make/lilypond-rules.make
278 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
279 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
280 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
281 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
282 include ./make/generic-targets.make ./make/lilypond-targets.make
284 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
285 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
286 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
287 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
290 @file{lilypond-rules.make} is @code{#empty}
292 @file{generic-rules.make} does seem to have 2 rules in it. They
296 $(outdir)/%.ly: %.lym4
297 $(M4) $< | sed "s/\`/,/g" > $@@
301 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@@
304 I believe the first rule is for *.ly files, and has a prerequisite
305 that *.lym4 files must be built first. The recipe is @code{m4 |
306 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
307 can comment on exactly what the rules mean/do.
309 @file{toplevel-rules.make} is @code{#empty}
311 @file{po-rules.make} is @code{#empty}
313 @file{install-rules.make} is @code{#empty}
315 @file{generic-targets.make} contains 2 lines of comments.
317 @file{lilypond-targets.make} contains only:
320 ## TODO: fail dist or web if no \version present.
322 grep -L version $(LY_FILES)
325 @file{stepmake/generic-targets.make} contains lots of rules - too
326 many to list here - it seems to be the main file for rules. (FWIW
327 I haven't actually found a rule for website: anywhere, although
328 it clearly exists. I have also found that you can display a rule
329 in the terminal by typing, say @code{make -n website}. This is
330 probably common knowledge.
332 @file{stepmake/toplevel-targets.make} adds a load of other (and
333 occasionally the same) rules to the gernric-targets.
335 @file{stepmake/po-targets.make} is rules for po* makes.
337 @file{stepmake/install-targets.make} has rules for local-install*.
339 And that's the end of stepmake.make. Back to
340 @file{GNUmakefile.in}.
342 A bit more info from 27 March. I've put some error traces into
343 @code{GNUmakefile} in the build directory, and it looks like the
344 following lines actually cause the make to run (putting an error
345 call above them - no make; below them - make):
349 # All web targets, except info image symlinks and info docs are
350 # installed in non-recursing target from TOP-SRC-DIR
352 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
353 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
354 $(MAKE) -C Documentation omf-local-install
357 I don't currently understand the @code{ifeq}, since @code{$(out)}
358 is empty at this point, but the line starting @w{@code{-$(INSTALL)}}
362 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
363 -c -m 755 -d /usr/local/share/doc/lilypond/html
366 End of work for Sunday 27th.
368 Another alterative approach to understanding the website build
369 would be to redirect @code{make -n website} and @code{make website}
370 to a text file and work through a) what it does and b) where the
371 errors are occurring.
373 GP: wow, all the above is much more complicated than I've ever
374 looked at stuff -- I tend to do a "back first" approach (where I
375 begin from the command-line that I want to modify, figure out
376 where it's generated, and then figure out how to change the
377 generated command-line), rather than a "front first" (where you
378 begin from the "make" command).
385 * The function of make doc::
386 * Building a bibliography::
389 @node The function of make doc
390 @subsection The function of make doc
392 The following is a set of notes on how make doc functions.
394 Preliminary question to be answered some time: where do all the
395 GNUmakefiles come from. They're in the build directory, but this
396 is not part of source. Must be the configure script. And it
397 looks like this comes from autogen.sh. Must at some point kill
398 the whole git directory, repull and see what is created when.
400 Anyway, here's how make doc progresses:
402 This is the build dependency tree from
403 @file{stepmake/stepmake/generic-targets.make}:
408 $(MAKE) -C $(depth)/scripts/build out=
409 $(MAKE) out=www WWW-1
412 $(MAKE) out=www WWW-2
415 $(MAKE) out=www WWW-post
419 MAKE = make --no-builtin-rules
420 -C = Change to directory before make
423 doc-stage-1 does lots of opening and looking in files, but no
429 + make PACKAGE=LILYPOND package=lilypond -C python
430 && make PACKAGE=LILYPOND package=lilypond -C scripts
431 && make PACKAGE=LILYPOND package=lilypond -C flower
432 && make PACKAGE=LILYPOND package=lilypond -C lily
433 && make PACKAGE=LILYPOND package=lilypond -C mf
434 && make PACKAGE=LILYPOND package=lilypond -C ly
435 && make PACKAGE=LILYPOND package=lilypond -C tex
436 && make PACKAGE=LILYPOND package=lilypond -C ps
437 && make PACKAGE=LILYPOND package=lilypond -C scm
438 && make PACKAGE=LILYPOND package=lilypond -C po
439 && make PACKAGE=LILYPOND package=lilypond -C make
440 && make PACKAGE=LILYPOND package=lilypond -C elisp
441 && make PACKAGE=LILYPOND package=lilypond -C vim
442 && make PACKAGE=LILYPOND package=lilypond -C input
443 && make PACKAGE=LILYPOND package=lilypond -C stepmake
444 && make PACKAGE=LILYPOND package=lilypond -C Documentation
450 stepmake/stepmake/generic-vars.make has this:
453 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@@ &&) true
456 $@@ is the name of the target - WWW-1 in this case.
458 In GNUmakefile.in we find:
461 SUBDIRS = python scripts \
468 stepmake $(documentation-dir)
471 So that's how we get the main make loop...
473 That loop expands like this:
476 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
477 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
484 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
485 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
486 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
487 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
488 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
489 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
490 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
491 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
495 The directories up to and including vim produce no effect with
496 make in non-debug mode, although debug does show lots of action.
498 @file{git/build/input/GNUmakefile} is:
502 include $(depth)/config$(if $(conf),-$(conf),).make
503 include $(configure-srcdir)/./input/GNUmakefile
504 MODULE_INCLUDES += $(src-dir)/$(outbase)
507 The first include is:
513 (note the // which is strictly wrong)
515 which has lots of variables to set, but no action occurs.
520 lilypond-git/./input/GNUmakefile
523 which similarly doesn't create any actual action.
525 An error message at the end of build/input/GNUmakefile stops
526 make processing before it moves on to regression - so where does
529 And the answer is - make processes all directories in the
530 directory it's entered (with some exceptions like out and out-www)
531 and so it changes to /regression.
533 It then seems to consider whether it needs to make/remake loads of
534 makefiles. Don't understand this yet. Possibly these are all the
535 makefiles it's processing, and it always checks they're up to date
536 before processing other files?
538 Could be correct - some of this output is:
541 Must remake target `../../make/ly-inclusions.make'.
542 Failed to remake target file `../../make/ly-inclusions.make'.
545 Having decided that, it then leaves the directory and re-executes:
548 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
551 The top of this make is:
554 This program built for i486-pc-linux-gnu
556 Reading makefile `GNUmakefile'...
557 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
560 which looks like it's re-reading all its known makefiles to check
563 (From the make manual:
565 To this end, after reading in all makefiles, make will consider each
566 as a goal target and attempt to update it. If a makefile has a rule
567 which says how to update it (found either in that very makefile or in
568 another one) or if an implicit rule applies to it (see Chapter 10
569 [Using Implicit Rules], page 103), it will be updated if
570 necessary. After all makefiles have been checked, if any have actually
571 been changed, make starts with a clean slate and reads all the
572 makefiles over again. (It will also attempt to update each of them
573 over again, but normally this will not change them again, since they
574 are already up to date.)
576 So my assumption seems correct)
578 There appear to be about 74 of them. After all the makefile
579 checking, we get this:
582 Updating goal targets....
583 Considering target file `WWW-1'.
584 File `WWW-1' does not exist.
585 Considering target file `local-WWW-1'.
586 File `local-WWW-1' does not exist.
587 Considering target file `out-www/collated-files.texi'.
588 File `out-www/collated-files.texi' does not exist.
589 Looking for an implicit rule for `out-www/collated-files.texi'.
590 Trying pattern rule with stem `collated-files.texi'.
591 Trying implicit prerequisite `collated-files.texi.in'.
592 Trying pattern rule with stem `collated-files.texi'.
593 Trying implicit prerequisite `collated-files.texi.in'.
594 Trying pattern rule with stem `collated-files'.
595 Trying implicit prerequisite `collated-files.tely'.
596 Trying pattern rule with stem `collated-files'.
597 Trying implicit prerequisite `out-www/collated-files.tely'.
598 Trying rule prerequisite `out-www/version.itexi'.
599 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
602 grep finds this if searching for local-WWW-1:
605 make/lysdoc-targets.make:
606 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
609 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
610 and so these will need to be checked to see if they're up to date.
611 So make needs to find rules for both of those and (as it says) it
612 certainly needs to make coll*.texi, since it doesn't exist.
614 In ly-rules.make we have:
617 .SUFFIXES: .doc .tely .texi .ly
620 which I'll work out at some point, and also this rule:
623 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
624 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
627 Note that the recipe is a very long line - it could probably
628 benefit from splitting. The same makefile also has:
631 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
632 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
636 which seems to be an almost exact duplicate. Whatever, the first
637 one is executed first. Have not checked if the second executes.
639 The first recipe translates as this:
642 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
643 --output=./out-www --format= --lily-output-dir \
644 /home/phil/lilypond-git/build/out/lybook-db
648 if we stop the build with an $(error), but I think this is because
649 we need to allow it to process the dependencies first. It looks
650 like foo.texi is shown as being dependent on foo.tely, plus a load
654 DOCUMENTATION_LOCALE_TARGET is blank
655 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
656 /home/phil/lilypond-git/scm/autochange.scm
659 plus 10s (100s?) of other .scm files.
662 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
663 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
666 ditto .ly files. This does seem a teency bit wrong - it looks like
667 the .ly and .scm files have been interchanged. ly-vars.make has
671 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
672 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
675 Looks like a bug.....
677 So it now works its way through all these files, checking if they
678 need to be remade. This is 100s of lines of the debug listing,
679 although none in the normal list. Clearly none has to be made
680 since they're source files. It concludes:
683 Must remake target `out-www/collated-files.tely'
686 @file{lysdoc-rules.make} has this:
689 $(outdir)/collated-files.tely: $(COLLATED_FILES)
690 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
693 @file{lysdoc-vars.make} has:
696 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
702 TEXINFO_SOURCES = AAA-intro-regression.tely
703 OUT_LY_FILES is empty
706 so LY_FILES has the big long list of all the .ly files in the
707 regression directory.
712 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
715 with a list of all the files in the regression test directory. This
716 should (I believe) create the file collated-files.tely.
718 So the next rule in make is for @file{version.itexi}, and make duly
719 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
720 depends on @file{git/VERSION}:
723 $(outdir)/version.%: $(top-src-dir)/VERSION
724 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
727 This causes create-version-itexi.py to run and create
730 Once that's done, all the other *.scm and *.ly files are checked
731 and since they have no rules associated, they aren't remade (just
732 as well for source files, really). Since version.itexi was remade
733 make concludes that collated-files.texi must be remade. To do
734 this, it runs lilypond-book.py on collated-files.tely, as below:
737 LILYPOND_VERSION=2.15.0
739 /home/phil/lilypond-git/scripts/lilypond-book.py
740 -I /home/phil/lilypond-git/input/regression/
741 -I ./out-www -I /home/phil/lilypond-git/input
742 -I /home/phil/lilypond-git/Documentation
743 -I /home/phil/lilypond-git/Documentation/snippets
744 -I /home/phil/lilypond-git/input/regression/
745 -I /home/phil/lilypond-git/Documentation/included/
746 -I /home/phil/lilypond-git/build/mf/out/
747 -I /home/phil/lilypond-git/build/mf/out/
748 -I /home/phil/lilypond-git/Documentation/pictures
749 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
750 --process='/home/phil/lilypond-git/build/out/bin/lilypond
751 -I /home/phil/lilypond-git/input/regression/
753 -I /home/phil/lilypond-git/input
754 -I /home/phil/lilypond-git/Documentation
755 -I /home/phil/lilypond-git/Documentation/snippets
756 -I /home/phil/lilypond-git/input/regression/
757 -I /home/phil/lilypond-git/Documentation/included/
758 -I /home/phil/lilypond-git/build/mf/out/
759 -I /home/phil/lilypond-git/build/mf/out/
760 -I /home/phil/lilypond-git/Documentation/pictures
761 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
786 -dcheck-internal-types
788 -danti-alias-factor=2'
792 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
793 out-www/collated-files.tely
796 So - lilypond-book runs on:
799 input/regression/out-www/collated-files.tely
803 Note the --verbose flag - this is from the make variable
804 LILYPOND_BOOK_VERBOSE which is added to the make variable
807 Now found the invocation to write some of the image files. It's
811 /home/phil/lilypond-git/build/out/bin/lilypond
812 -I /home/phil/lilypond-git/input/regression/
813 -I ./out-www -I /home/phil/lilypond-git/input
814 -I /home/phil/lilypond-git/Documentation
815 -I /home/phil/lilypond-git/Documentation/snippets
816 -I /home/phil/lilypond-git/input/regression/
817 -I /home/phil/lilypond-git/Documentation/included/
818 -I /home/phil/lilypond-git/build/mf/out/
819 -I /home/phil/lilypond-git/build/mf/out/
820 -I /home/phil/lilypond-git/Documentation/pictures
821 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
846 -dcheck-internal-types
848 -danti-alias-factor=2
849 -I "/home/phil/lilypond-git/build/out/lybook-db"
850 -I "/home/phil/lilypond-git/build/input/regression"
851 -I "/home/phil/lilypond-git/input/regression"
852 -I "/home/phil/lilypond-git/build/input/regression/out-www"
853 -I "/home/phil/lilypond-git/input"
854 -I "/home/phil/lilypond-git/Documentation"
855 -I "/home/phil/lilypond-git/Documentation/snippets"
856 -I "/home/phil/lilypond-git/input/regression"
857 -I "/home/phil/lilypond-git/Documentation/included"
858 -I "/home/phil/lilypond-git/build/mf/out"
859 -I "/home/phil/lilypond-git/build/mf/out"
860 -I "/home/phil/lilypond-git/Documentation/pictures"
861 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
864 -deps-box-padding=3.000000
866 -dno-strip-output-dir
867 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
870 Note the --verbose. This causes 100s of lines of Lily debug output.
871 But at present I can't work out where the flag comes from. Later.
874 @node Building a bibliography
875 @subsection Building a bibliography
877 Bibliography files contain a list of citations, like this:
881 author = @{Vinci, Albert C.@},
882 title = @{Fundamentals of Traditional Music Notation@},
883 publisher = @{Kent State University Press@},
888 There are a variety of types of citation (e.g. Book (as above),
889 article, publication). Each cited publication has a list of
890 entries that can be used to identify the publication.
891 Bibliograpies are normally stored as files with a .bib
892 extension. One part of the doc-build process is transforming the
893 bibliography information into @code{texinfo} files. The commands
894 to do this are in the @file{GNUmakefile} in the
895 @file{Documentation} directory.
897 A typical line of the makefile to translate a single bibliography
901 $(outdir)/colorado.itexi:
902 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
903 -s $(top-src-dir)/Documentation/lily-bib \
904 -o $(outdir)/colorado.itexi \
905 $(src-dir)/essay/colorado.bib
911 $(outdir)/colorado.itexi:
914 We're making the file @file{colorado.itexi} and so this is the
918 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
921 It's in the @file{essay} directory and we want to run the
922 bib2texi.py script against it.
925 -s $(top-src-dir)/Documentation/lily-bib \
928 The style template is @file{lily-bib.bst} and is found in the
929 @file{Documentation} directory.
932 -o $(outdir)/colorado.itexi \
935 The output file in @file{colorado.itexi}.
938 $(src-dir)/essay/colorado.bib
941 The input file is @file{colorado.bib} in the @file{essay}
944 The @code{bib2texi} Python script used to be used with a variety
945 of options, but now is always called using the same options, as
946 above. Its job is to create the file containing the options for
947 @code{bibtex} (the program that actually does the translation),
948 run bibtex, and then clean up some temporary files. Its main
949 "value add" is the creation of the options file, using this code:
952 open (tmpfile + '.aux', 'w').write (r'''
955 \bibstyle@{%(style)s@}
956 \bibdata@{%(files)s@}''' % vars ())
959 The key items are the style file (now always lily-bib for us) and
962 The style file is written in its own specialised language,
963 described to some extent at
966 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
969 The file @file{lily-bib.bst} also has fairly extensive commenting.
973 @section Website build
975 @warning{This information applies only to the standard @code{make
976 website} from the normal build directory. The process is
977 different for @code{dev/website-build}.}
979 The rule for make website is found in GNUmakefile.in:
983 $(MAKE) config_make=$(config_make) \
984 top-src-dir=$(top-src-dir) \
985 -f $(top-src-dir)/make/website.make \
992 make --no-builtin-rules config_make=./config.make \
993 top-src-dir=/home/phil/lilypond-git \
994 -f /home/phil/lilypond-git/make/website.make \
998 which has the effect of setting the variables @code{config_make}
999 and @code{top-src-dir} and then processing the file
1000 @code{git/make/website.make} with the target of website.
1002 @code{website.make} starts with the following:
1005 ifeq ($(WEBSITE_ONLY_BUILD),1)
1008 which checks to see whether the variable @code{WEBSITE_ONLY_BUILD}
1009 was set to one on the command line. This is only done for
1010 standalone website builds, not in the normal case. The result of
1011 the test determines the value of some variables that are set. A
1012 number of other variables are set, in order to establish locations
1013 of various files. An example is:
1016 CREATE_VERSION=python $(script-dir)/create-version-itexi.py
1019 The rule for website is:
1022 website: website-texinfo website-css website-pictures website-examples web-post
1023 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1024 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1025 cp $(top-htaccess) $(OUT)/.htaccess
1026 cp $(dir-htaccess) $(OUT)/website/.htaccess
1029 so we see that this starts by running the rules for 5 other
1030 targets, then finishes by copying some files. We'll cover that
1031 later - first @code{website-texinfo}. That rule is:
1034 website-texinfo: website-version website-xrefs website-bibs
1035 for l in '' $(WEB_LANGS); do \
1036 if test -n "$$l"; then \
1037 langopt=--lang="$$l"; \
1040 $(TEXI2HTML) --prefix=index \
1042 --I=$(top-src-dir)/Documentation/"$$l" \
1043 --I=$(top-src-dir)/Documentation \
1046 --init-file=$(texi2html-init-file) \
1048 --output=$(OUT)/"$$l" \
1049 $(top-src-dir)/Documentation/"$$l"/web.texi ; \
1050 ls $(OUT)/$$l/*.html | xargs grep -L \
1051 'UNTRANSLATED NODE: IGNORE ME' | \
1052 sed 's!$(OUT)/'$$l'/!!g' | xargs \
1053 $(MASS_LINK) --prepend-suffix="$$langsuf" \
1054 hard $(OUT)/$$l/ $(OUT)/website/ ; \
1058 which therefore depends on @code{website-version},
1059 @code{website-xrefs} and @code{website-bibs}.
1064 $(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
1065 $(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
1068 which translates as:
1071 mkdir -p out-website
1072 python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
1073 /home/phil/lilypond-git > out-website/version.itexi
1074 python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
1075 /home/phil/lilypond-git > out-website/weblinks.itexi
1078 So, we make out-website then send the output of
1079 @code{create-version-itexi.py} to @code{out-website/version.itexi}
1080 and @code{create-weblinks-itexi.py} to
1081 @code{out-website/weblinks.itexi}.
1083 @code{create-version-itexi.py} parses the file @code{VERSION} in
1084 the top source dir. It contains:
1087 PACKAGE_NAME=LilyPond
1092 VERSION_STABLE=2.14.2
1093 VERSION_DEVEL=2.15.12
1096 currently. @code{c-v-i.py} parses this to:
1099 @@c ************************ Version numbers ************
1104 @@macro versionStable
1108 @@macro versionDevel
1113 @code{create-weblinks-itexi.py} creates a load of texi macros (of
1114 the order of 1000) similar to:
1117 @@macro manualStableGlossaryPdf
1118 @@uref@{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf@}
1122 It loads its languages from langdefs.py, and therefore outputs the following unhelpful warning:
1124 @code{langdefs.py: warning: lilypond-doc gettext domain not found.}
1129 website-xrefs: website-version
1130 for l in '' $(WEB_LANGS); do \
1133 is the start of the rule, truncated for brevity. This loops
1134 through the languages to be used on the website, processing some
1135 variables which I don't fully understand, to run this command:
1138 python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
1139 -I /home/phil/lilypond-git/Documentation \
1140 -I /home/phil/lilypond-git/Documentation/"$l" \
1141 -I out-website -o out-website --split=node \
1142 --known-missing-files= \
1143 /home/phil/lilypond-git/scripts/build/website-known-missing-files.txt \
1145 /home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
1148 There's a good description of what
1149 @code{extract_texi_filenames.py} does at the top of the script,
1150 but a shortened version is:
1152 @code{If this script is run on a file texifile.texi, it produces
1153 a file texifile[.LANG].xref-map with tab-separated entries
1154 of the form NODE\tFILENAME\tANCHOR.}
1157 @code{web.nl.xref-map} is:
1160 Inleiding Introduction Introduction
1163 @code{e-t-f.py} follows the includes from document to document.
1164 We know some have not been created yet, and
1165 @code{known-missing-files} option tells @code{e-t-f.py} which
1171 for m in $(MANUALS); do \
1174 to run @code{e-t-f.py} against all of the manuals, in each
1178 website-bibs: website-version
1179 BSTINPUTS=$(top-src-dir)/Documentation/web \
1180 $(WEB_BIBS) -s web \
1181 -s $(top-src-dir)/Documentation/lily-bib \
1182 -o $(OUT)/others-did.itexi \
1184 $(top-src-dir)/Documentation/web/others-did.bib
1187 This is half the command. It runs @code{bib2texi.py} on 2
1188 @code{.bib} files - @code{others-did.bib} and @code{we-wrote.bib}.
1189 This converts bibliography files into texi files with
1192 Next the commands in the @code{website-texinfo} rule are run:
1195 for l in '' $(WEB_LANGS); do \
1198 run @code{texi2html}. This is the program that outputs the
1199 progress message (found in
1200 @code{Documentation/lilypond-texi2html.init}):
1202 @code{Processing web site: []}
1204 It also outputs warning messages like:
1206 @code{WARNING: Unable to find node 'ŘeÅ¡enà potÞÃ' in book usage.}
1210 cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
1213 Copies 3 css files to out-website/website. Then:
1217 mkdir -p $(OUT)/website/pictures
1218 if [ -d $(PICTURES) ]; \
1220 cp $(PICTURES)/* $(OUT)/website/pictures ; \
1221 ln -sf website/pictures $(OUT)/pictures ;\
1225 which translates as:
1228 if [ -d Documentation/pictures/out-www ]; \
1230 cp Documentation/pictures/out-www/* out-website/website/pictures ; \
1231 ln -sf website/pictures out-website/pictures ;\
1235 i.e. it copies the contents of
1236 @code{build/Documentation/pictures/out-www/*} to
1237 @code{out-website/website/pictures}. Unfortunately, the pictures
1238 are only created once @code{make doc} has been run, so an initial
1239 run of @code{make website} copies nothing, and the pictures on the
1240 website (e.g. the logo) do not exist. Next:
1244 mkdir -p $(OUT)/website/ly-examples
1245 if [ -d $(EXAMPLES) ]; \
1247 cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
1254 mkdir -p out-website/website/ly-examples
1255 if [ -d Documentation/web/ly-examples/out-www ]; \
1257 cp Documentation/web/ly-examples/out-www/* out-website/website/ly-examples ; \
1261 This does the same with the LilyPond examples (found at
1262 @uref{http://lilypond.org/examples.html}). Again, these are
1263 actually only created by @code{make doc} (and since they are
1264 generated from LilyPond source files, require a working LilyPond
1265 @code{exe} made with @code{make}). So this does nothing
1270 $(WEB_POST) $(OUT)/website
1276 python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
1279 which describes itself as:
1281 @code{This is web_post.py. This script deals with translations
1282 in the "make website" target.}
1284 It also does a number of other things, including adding the Google
1285 tracker code and the language selection footer. We're now at
1286 the end of our story. The final 4 lines of the recipe for website
1290 cp $(SERVER_FILES)/favicon.ico $(OUT)/website
1291 cp $(SERVER_FILES)/robots.txt $(OUT)/website
1292 cp $(top-htaccess) $(OUT)/.htaccess
1293 cp $(dir-htaccess) $(OUT)/website/.htaccess
1296 The first translates as:
1299 cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico out-website/website
1302 so we see these are just copying the support files for the web
1305 @subsubheading website.make summary
1307 Recipes in @file{website.make}:
1313 this is the "master" rule. It calls the other rules in order,
1314 then copies some extra files around - see below for further
1315 of the process it produces.
1318 @code{website-version}:
1319 this calls the python scripts below:
1323 scripts/build/create-version-itexi.py
1326 This writes a @@version, @@versionStable, and @@versionDevel based
1327 on the top-level VERSIONS file, to
1328 @code{out-website/version.itexi}
1332 scripts/build/create-weblinks-itexi.py
1335 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1336 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1337 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1339 It's quite monstrous because it deals with combinations of
1340 stable/devel, source/docs, lang/lang/lang*10, etc.
1345 @code{website-xrefs:}
1346 creates files used for complicated "out-of-build" references to
1347 @code{out-website/*.xref-map}
1349 If you just write @@ref@{@}, then all's groovy and we wouldn't
1350 need this. But if you write @@rlearning@{@}, then our custom
1351 texi2html init file needs to know about our custom xref file
1352 format, which tells our custom texi2html init file how to create
1355 GP: we should have a separate @@node to discuss xrefs. Also, take a
1356 quick look at a generated xref file -- it's basically just a list
1357 of @@node's [sic teenager pluralization rule] from the file.
1361 generates the bibliography texinfo files from the .bib files - in
1362 the case of the website build these are @file{others-did.bib} and
1363 @file{we-wrote.bib}.
1366 @code{website-texinfo:}
1367 this is the main part; it calles texi2html to generate the actual
1368 html. It also has a ton of options to texi2html to pass info to
1369 our custom init file.
1371 The file actually built is called @file{web.texi}, and is either
1372 in the @file{Documentation} directory, or a sub-directory specific
1375 The options file is @file{/Documentation/lilypond-texi2html.init}.
1376 This contains *lots* of option and configuration stuff, and also
1380 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1383 This is where one of the console messages is generated.
1385 We have somewhere between 2-4 different ways "to pass info to our
1386 custom init file". This is highly Not Good (tm), but that's how
1387 things work at the moment.
1389 After texi2html, it does some black magick to deal with
1390 untranslated nodes in the translations. Despite writing that
1391 part, I can't remember how it works. But in theory, you could
1392 figure it out by copy&pasting each part of the command (by "part",
1393 I mean "stuff before each | pipe"), substituting the variables,
1394 then looking at the text that's output. For example,
1397 ls $(OUT)/$$l/*.html
1400 is going to print a list of all html files, in all languages, in
1401 the build directory. Then more stuff happens to each of those
1402 files (that's what xargs does).
1406 just copies files to the build dir.
1409 @code{website-pictures, website-examples:}
1410 more file copies, with an if statement to handle if you don't have
1411 any generated pictures/examples.
1418 scripts/build/website_post.py
1421 which, it adds the "this page is translated in klingon" to the
1422 bottom of html pages, and adds the google analytics javascript.
1423 It also has hard-coded lilypond version numbers, which is Bad
1428 Here's a summary of what gets called, in what order, when we run
1435 creates version.itexi and weblinks.itexi
1437 runs extract_texi_filenames.py
1439 creates bibliography files, described above
1447 runs website_post.py
1448 Then some file copying
1451 @node Building an Ubuntu distro
1452 @section Building an Ubuntu distro
1455 Here's the short instruction on how to create lilybuntu iso image
1456 (Jonathan Kulp did this on a spare drive,
1457 but he supposes it can be done in a VM too):
1462 Install ubuntu, reboot.
1464 Run all updates, reboot if asked.
1466 Enable src repos, refresh package lists.
1468 Install LilyPond build deps:
1470 sudo apt-get build-dep lilypond
1473 Install git and autoconf:
1475 sudo apt-get install git-core gitk autoconf
1479 Test to see whether everything works fine now:
1482 use @command{lily-git.tcl} to grab source files
1484 go to source dir and do
1486 "./autogen.sh" ; make ; make doc
1489 if all compiles, move on to iso creation...
1493 Download & install "remastersys":
1494 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
1496 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
1498 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
1499 default live session username, etc).
1501 Remove non-essential desktop software as desired.
1505 sudo remastersys dist
1507 New iso is in @file{/home/remastersys/remastersys/}.
1509 Test iso by installing in VM and repeating steps above for
1510 getting source files and building lp and docs.