1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
21 @node Build system overview
22 @section Build system overview
24 Build system is currently GNU make, with an extra "stepmake" layer
25 on top. Look at files in @file{make/} and @file{stepmake/} and
26 all @file{GNUmakefile}s.
28 There is wide-spread dissatisfaction with this system, and we are
29 considering changing. This would be a huge undertaking (estimated
30 200+ hours). This change will probably involve not using GNU make
31 any more -- but a discussion about the precise build system will
32 have to wait. Before we reach that point, we need to figure out
33 (at least approximately) what the current build system does.
35 Fundamentally, a build system does two things:
39 Constructs command-line commands, for example:
43 --tons --of --options \
46 --more --imperial --and --metric --tons --of --options \
51 If there was a previous build, it decides which parts of the
52 system need to be rebuilt.
56 When I try to do anything in the build system, it helps to remind
57 myself of this. The "end result" is just a series of command-line
58 commands. All the black magick is just an attempt to construct
61 @node Tips for working on the build system
62 @section Tips for working on the build system
74 to the build system files in various places. This will let you
75 track where the program is, in various points of the build.
77 PH note. There are lots of places where Make doesn't let you put
78 echo commands. My top tip for tracing how make runs is to put
81 $(error Some Text to display)
84 This will stop make running and print the text @code{Some Text to
90 First task: understand how @code{make website} works,
91 @emph{without} the translations. Looking at the english-only
92 website is the best introduction to the build system... it only
93 covers about 5% of the whole thing, but even that will likely take
99 @node General build system notes
100 @section General build system notes
103 * How stepmake works::
106 @node How stepmake works
107 @subsection How stepmake works
109 Typing make website runs the file @file{GNUmakefile} from the
110 build directory. This only contains 3 lines:
114 include config$(if $(conf),-$(conf),).make
115 include $(configure-srcdir)/GNUmakefile.in
118 The variable @code{depth} is used throughout the make system to
119 track how far down the directory structure the make is. The first
120 include sets lots of variables but doesn't "do" anything. The
121 second runs the file @file{GNUmakefile.in} from the top level
124 This sets another load of variables, and then includes (i.e.
125 immediately runs) @file{stepmake.make} from the @file{make}
126 subdirectory. This sets a load of other variables, does some
127 testing to see if SCONS (another build tool?) is being used, and
128 then runs @file{make/config.make} - which doesn't seem to exist...
130 GP: scons is indeed a different build tool; I think that Jan
131 experimented with it 5 years ago or something. It seems like we
132 still have bits and pieces of it floating around.
134 Next, it runs @file{make/toplevel-version.make}, which sets the
135 version variables for major, minor, patch, stable, development and
136 mypatchlevel (which seems to be used for patch numbers for
137 non-stable versions only?).
139 Next - @file{make/local.make}, which doesn't exist.
141 Then a few more variable and the interesting comment:
144 # Don't try to outsmart us, you puny computer!
145 # Well, UGH. This only removes builtin rules from
148 and then tests to see whether BUILTINS_REMOVED is defined. It
149 appears to be when I run make, and so
150 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
151 comment at the head of this file says:
154 # UGH. GNU make comes with implicit rules.
155 # We don't want any of them, and can't force users to run
159 I've not studied that file at length, but assume it removes all
160 make's build-in rules (e.g. @file{*.c} files are run through the
161 GNU C compiler) - there's a lot of them in here, and a lot of
162 comments, and I'd guess most of it isn't needed.
164 We return to @file{stepmake.make}, where we hit the make rule all:
165 The first line of this is:
168 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
171 which, when the variables are substituted, gives:
174 ./make/generic-inclusions.make
175 ./make/lilypond-inclusions.make.
178 (Note - according to the make documentation, -include is only
179 different from include in that it doesn't produce any kind of
180 error message when the included file doesn't exist).
182 And the first file doesn't exist. Nor the second. Next:
185 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
188 which expands to the following files:
191 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
192 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
193 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
197 One little feature to notice here - these are all absolute file
198 locations - the line prior to this used relative locations. And
199 none of these files exist, either. (Further note - I'm assuming
200 all these lines of make I'm following are autogenerated, but
201 that'll be something else to discover.)
203 Next in @file{stepmake.make}:
206 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
212 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
213 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
214 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
215 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
218 Woo. They all exist (they should as there's no - in front of the
219 include). @file{generic-vars.make} sets loads of variables
220 (funnily enough). @file{toplevel-vars.make} is very short - one
221 line commented as @code{# override Generic_vars.make:} and 2 as
226 include $(stepdir)/documentation-vars.make
229 I assume the urg comment refers to the fact that this should
230 really just create more variables, but it actually sends us off to
231 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
233 That file is a 3 line variable setting one.
235 @file{po-vars.make} has the one-line comment @code{# empty}, as
236 does @file{install-vars.make}.
238 So now we're back to @file{stepmake.make}.
243 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
244 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
247 and the include expands to:
250 include ./make/generic-vars.make ./make/lilypond-vars.make.
253 These again set variables, and in some cases export them to allow
254 child @code{make} processes to use them.
256 The final 4 lines of @file{stepmake.make} are:
259 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
260 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
261 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
262 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
265 which expand as follows:
268 include ./make/generic-rules.make ./make/lilypond-rules.make
270 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
271 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
272 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
273 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
274 include ./make/generic-targets.make ./make/lilypond-targets.make
276 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
277 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
278 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
279 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
282 @file{lilypond-rules.make} is @code{#empty}
284 @file{generic-rules.make} does seem to have 2 rules in it. They
288 $(outdir)/%.ly: %.lym4
289 $(M4) $< | sed "s/\`/,/g" > $@
293 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
296 I believe the first rule is for *.ly files, and has a prerequisite
297 that *.lym4 files must be built first. The recipe is @code{m4 |
298 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
299 can comment on exactly what the rules mean/do.
301 @file{toplevel-rules.make} is @code{#empty}
303 @file{po-rules.make} is @code{#empty}
305 @file{install-rules.make} is @code{#empty}
307 @file{generic-targets.make} contains 2 lines of comments.
309 @file{lilypond-targets.make} contains only:
312 ## TODO: fail dist or web if no \version present.
314 grep -L version $(LY_FILES)
317 @file{stepmake/generic-targets.make} contains lots of rules - too
318 many to list here - it seems to be the main file for rules. (FWIW
319 I haven't actually found a rule for website: anywhere, although
320 it clearly exists. I have also found that you can display a rule
321 in the terminal by typing, say @code{make -n website}. This is
322 probably common knowledge.
324 @file{stepmake/toplevel-targets.make} adds a load of other (and
325 occasionally the same) rules to the gernric-targets.
327 @file{stepmake/po-targets.make} is rules for po* makes.
329 @file{stepmake/install-targets.make} has rules for local-install*.
331 And that's the end of stepmake.make. Back to
332 @file{GNUmakefile.in}.
334 A bit more info from 27 March. I've put some error traces into
335 @code{GNUmakefile} in the build directory, and it looks like the
336 following lines actually cause the make to run (putting an error
337 call above them - no make; below them - make):
341 # All web targets, except info image symlinks and info docs are
342 # installed in non-recursing target from TOP-SRC-DIR
344 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
345 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
346 $(MAKE) -C Documentation omf-local-install
349 I don't currently understand the @code{ifeq}, since @code{$(out)}
350 is empty at this point, but the line starting @code{-$(INSTALL)}
354 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py \
355 -c -m 755 -d /usr/local/share/doc/lilypond/html
358 End of work for Sunday 27th.
360 Another alterative approach to understanding the website build
361 would be to redirect @code{make -n website} and @code{make website}
362 to a text file and work through a) what it does and b) where the
363 errors are occurring.
365 GP: wow, all the above is much more complicated than I've ever
366 looked at stuff -- I tend to do a "back first" approach (where I
367 begin from the command-line that I want to modify, figure out
368 where it's generated, and then figure out how to change the
369 generated command-line), rather than a "front first" (where you
370 begin from the "make" command).
377 * The function of make doc::
378 * Building a bibliography::
381 @node The function of make doc
382 @subsection The function of make doc
384 The following is a set of notes on how make doc functions.
386 Preliminary question to be answered some time: where do all the
387 GNUmakefiles come from. They're in the build directory, but this
388 is not part of source. Must be the configure script. And it
389 looks like this comes from autogen.sh. Must at some point kill
390 the whole git directory, repull and see what is created when.
392 Anyway, here's how make doc progresses:
394 This is the build dependency tree from
395 @file{stepmake/stepmake/generic-targets.make}:
400 $(MAKE) -C $(depth)/scripts/build out=
401 $(MAKE) out=www WWW-1
404 $(MAKE) out=www WWW-2
407 $(MAKE) out=www WWW-post
411 MAKE = make --no-builtin-rules
412 -C = Change to directory before make
415 doc-stage-1 does lots of opening and looking in files, but no
421 + make PACKAGE=LILYPOND package=lilypond -C python
422 && make PACKAGE=LILYPOND package=lilypond -C scripts
423 && make PACKAGE=LILYPOND package=lilypond -C flower
424 && make PACKAGE=LILYPOND package=lilypond -C lily
425 && make PACKAGE=LILYPOND package=lilypond -C mf
426 && make PACKAGE=LILYPOND package=lilypond -C ly
427 && make PACKAGE=LILYPOND package=lilypond -C tex
428 && make PACKAGE=LILYPOND package=lilypond -C ps
429 && make PACKAGE=LILYPOND package=lilypond -C scm
430 && make PACKAGE=LILYPOND package=lilypond -C po
431 && make PACKAGE=LILYPOND package=lilypond -C make
432 && make PACKAGE=LILYPOND package=lilypond -C elisp
433 && make PACKAGE=LILYPOND package=lilypond -C vim
434 && make PACKAGE=LILYPOND package=lilypond -C input
435 && make PACKAGE=LILYPOND package=lilypond -C stepmake
436 && make PACKAGE=LILYPOND package=lilypond -C Documentation
442 stepmake/stepmake/generic-vars.make has this:
445 LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true
448 $@ is the name of the target - WWW-1 in this case.
450 In GNUmakefile.in we find:
453 SUBDIRS = python scripts \
460 stepmake $(documentation-dir)
463 So that's how we get the main make loop...
465 That loop expands like this:
468 make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
469 make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
470 make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
471 make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
472 make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
473 make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
474 make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
475 make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
476 make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
477 make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
478 make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
479 make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
480 make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
481 make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
482 make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
483 make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
487 The directories up to and including vim produce no effect with
488 make in non-debug mode, although debug does show lots of action.
490 @file{git/build/input/GNUmakefile} is:
494 include $(depth)/config$(if $(conf),-$(conf),).make
495 include $(configure-srcdir)/./input/GNUmakefile
496 MODULE_INCLUDES += $(src-dir)/$(outbase)
499 The first include is:
505 (note the // which is strictly wrong)
507 which has lots of variables to set, but no action occurs.
512 lilypond-git/./input/GNUmakefile
515 which similarly doesn't create any actual action.
517 An error message at the end of build/input/GNUmakefile stops
518 make processing before it moves on to regression - so where does
521 And the answer is - make processes all directories in the
522 directory it's entered (with some exceptions like out and out-www)
523 and so it changes to /regression.
525 It then seems to consider whether it needs to make/remake loads of
526 makefiles. Don't understand this yet. Possibly these are all the
527 makefiles it's processing, and it always checks they're up to date
528 before processing other files?
530 Could be correct - some of this output is:
533 Must remake target `../../make/ly-inclusions.make'.
534 Failed to remake target file `../../make/ly-inclusions.make'.
537 Having decided that, it then leaves the directory and re-executes:
540 make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
543 The top of this make is:
546 This program built for i486-pc-linux-gnu
548 Reading makefile `GNUmakefile'...
549 Reading makefile `../..//config.make' (search path) (no ~ expansion)...
552 which looks like it's re-reading all its known makefiles to check
555 (From the make manual:
557 To this end, after reading in all makefiles, make will consider each as a goal target and
558 attempt to update it. If a makefile has a rule which says how to update it (found either
559 in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10
560 [Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have
561 been checked, if any have actually been changed, make starts with a clean slate and reads
562 all the makefiles over again. (It will also attempt to update each of them over again, but
563 normally this will not change them again, since they are already up to date.)
565 So my assumption seems correct)
567 There appear to be about 74 of them. After all the makefile
568 checking, we get this:
571 Updating goal targets....
572 Considering target file `WWW-1'.
573 File `WWW-1' does not exist.
574 Considering target file `local-WWW-1'.
575 File `local-WWW-1' does not exist.
576 Considering target file `out-www/collated-files.texi'.
577 File `out-www/collated-files.texi' does not exist.
578 Looking for an implicit rule for `out-www/collated-files.texi'.
579 Trying pattern rule with stem `collated-files.texi'.
580 Trying implicit prerequisite `collated-files.texi.in'.
581 Trying pattern rule with stem `collated-files.texi'.
582 Trying implicit prerequisite `collated-files.texi.in'.
583 Trying pattern rule with stem `collated-files'.
584 Trying implicit prerequisite `collated-files.tely'.
585 Trying pattern rule with stem `collated-files'.
586 Trying implicit prerequisite `out-www/collated-files.tely'.
587 Trying rule prerequisite `out-www/version.itexi'.
588 Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'
591 grep finds this if searching for local-WWW-1:
594 make/lysdoc-targets.make:
595 local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
598 which means that local-WWW-1 depends on coll*.texi and coll*.pdf
599 and so these will need to be checked to see if they're up to date.
600 So make needs to find rules for both of those and (as it says) it
601 certainly needs to make coll*.texi, since it doesn't exist.
603 In ly-rules.make we have:
606 .SUFFIXES: .doc .tely .texi .ly
609 which I'll work out at some point, and also this rule:
612 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
613 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
616 Note that the recipe is a very long line - it could probably
617 benefit from splitting. The same makefile also has:
620 $(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
621 LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
625 which seems to be an almost exact duplicate. Whatever, the first
626 one is executed first. Have not checked if the second executes.
628 The first recipe translates as this:
631 LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' \
632 --output=./out-www --format= --lily-output-dir \
633 /home/phil/lilypond-git/build/out/lybook-db
637 if we stop the build with an $(error), but I think this is because
638 we need to allow it to process the dependencies first. It looks
639 like foo.texi is shown as being dependent on foo.tely, plus a load
643 DOCUMENTATION_LOCALE_TARGET is blank
644 INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
645 /home/phil/lilypond-git/scm/autochange.scm
648 plus 10s (100s?) of other .scm files.
651 SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
652 /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
655 ditto .ly files. This does seem a teency bit wrong - it looks like
656 the .ly and .scm files have been interchanged. ly-vars.make has
660 INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
661 SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
664 Looks like a bug.....
666 So it now works its way through all these files, checking if they
667 need to be remade. This is 100s of lines of the debug listing,
668 although none in the normal list. Clearly none has to be made
669 since they're source files. It concludes:
672 Must remake target `out-www/collated-files.tely'
675 @file{lysdoc-rules.make} has this:
678 $(outdir)/collated-files.tely: $(COLLATED_FILES)
679 $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^
682 @file{lysdoc-vars.make} has:
685 COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
691 TEXINFO_SOURCES = AAA-intro-regression.tely
692 OUT_LY_FILES is empty
695 so LY_FILES has the big long list of all the .ly files in the
696 regression directory.
701 /home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
704 with a list of all the files in the regression test directory. This
705 should (I believe) create the file collated-files.tely.
707 So the next rule in make is for @file{version.itexi}, and make duly
708 checks this. There's a rule in @file{doc-i18n-root-rules.make} that this
709 depends on @file{git/VERSION}:
712 $(outdir)/version.%: $(top-src-dir)/VERSION
713 $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
716 This causes create-version-itexi.py to run and create
719 Once that's done, all the other *.scm and *.ly files are checked
720 and since they have no rules associated, they aren't remade (just
721 as well for source files, really). Since version.itexi was remade
722 make concludes that collated-files.texi must be remade. To do
723 this, it runs lilypond-book.py on collated-files.tely, as below:
726 LILYPOND_VERSION=2.15.0
728 /home/phil/lilypond-git/scripts/lilypond-book.py
729 -I /home/phil/lilypond-git/input/regression/
730 -I ./out-www -I /home/phil/lilypond-git/input
731 -I /home/phil/lilypond-git/Documentation
732 -I /home/phil/lilypond-git/Documentation/snippets
733 -I /home/phil/lilypond-git/input/regression/
734 -I /home/phil/lilypond-git/Documentation/included/
735 -I /home/phil/lilypond-git/build/mf/out/
736 -I /home/phil/lilypond-git/build/mf/out/
737 -I /home/phil/lilypond-git/Documentation/pictures
738 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
739 --process='/home/phil/lilypond-git/build/out/bin/lilypond
740 -I /home/phil/lilypond-git/input/regression/
742 -I /home/phil/lilypond-git/input
743 -I /home/phil/lilypond-git/Documentation
744 -I /home/phil/lilypond-git/Documentation/snippets
745 -I /home/phil/lilypond-git/input/regression/
746 -I /home/phil/lilypond-git/Documentation/included/
747 -I /home/phil/lilypond-git/build/mf/out/
748 -I /home/phil/lilypond-git/build/mf/out/
749 -I /home/phil/lilypond-git/Documentation/pictures
750 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
775 -dcheck-internal-types
777 -danti-alias-factor=2'
781 --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
782 out-www/collated-files.tely
785 So - lilypond-book runs on:
788 input/regression/out-www/collated-files.tely
792 Note the --verbose flag - this is from the make variable
793 LILYPOND_BOOK_VERBOSE which is added to the make variable
796 Now found the invocation to write some of the image files. It's
800 /home/phil/lilypond-git/build/out/bin/lilypond
801 -I /home/phil/lilypond-git/input/regression/
802 -I ./out-www -I /home/phil/lilypond-git/input
803 -I /home/phil/lilypond-git/Documentation
804 -I /home/phil/lilypond-git/Documentation/snippets
805 -I /home/phil/lilypond-git/input/regression/
806 -I /home/phil/lilypond-git/Documentation/included/
807 -I /home/phil/lilypond-git/build/mf/out/
808 -I /home/phil/lilypond-git/build/mf/out/
809 -I /home/phil/lilypond-git/Documentation/pictures
810 -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
835 -dcheck-internal-types
837 -danti-alias-factor=2
838 -I "/home/phil/lilypond-git/build/out/lybook-db"
839 -I "/home/phil/lilypond-git/build/input/regression"
840 -I "/home/phil/lilypond-git/input/regression"
841 -I "/home/phil/lilypond-git/build/input/regression/out-www"
842 -I "/home/phil/lilypond-git/input"
843 -I "/home/phil/lilypond-git/Documentation"
844 -I "/home/phil/lilypond-git/Documentation/snippets"
845 -I "/home/phil/lilypond-git/input/regression"
846 -I "/home/phil/lilypond-git/Documentation/included"
847 -I "/home/phil/lilypond-git/build/mf/out"
848 -I "/home/phil/lilypond-git/build/mf/out"
849 -I "/home/phil/lilypond-git/Documentation/pictures"
850 -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
853 -deps-box-padding=3.000000
855 -dno-strip-output-dir
856 "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
859 Note the --verbose. This causes 100s of lines of Lily debug output.
860 But at present I can't work out where the flag comes from. Later.
863 @node Building a bibliography
864 @subsection Building a bibliography
866 Bibliography files contain a list of citations, like this:
870 author = @{Vinci, Albert C.@},
871 title = @{Fundamentals of Traditional Music Notation@},
872 publisher = @{Kent State University Press@},
877 There are a variety of types of citation (e.g. Book (as above),
878 article, publication). Each cited publication has a list of
879 entries that can be used to identify the publication.
880 Bibliograpies are normally stored as files with a .bib
881 extension. One part of the doc-build process is transforming the
882 bibliography information into @code{texinfo} files. The commands
883 to do this are in the @file{GNUmakefile} in the
884 @file{Documentation} directory.
886 A typical line of the makefile to translate a single bibliography
890 $(outdir)/colorado.itexi:
891 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
892 -s $(top-src-dir)/Documentation/lily-bib \
893 -o $(outdir)/colorado.itexi \
894 $(src-dir)/essay/colorado.bib
900 $(outdir)/colorado.itexi:
903 We're making the file @file{colorado.itexi} and so this is the
907 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
910 It's in the @file{essay} directory and we want to run the
911 bib2texi.py script against it.
914 -s $(top-src-dir)/Documentation/lily-bib \
917 The style template is @file{lily-bib.bst} and is found in the
918 @file{Documentation} directory.
921 -o $(outdir)/colorado.itexi \
924 The output file in @file{colorado.itexi}.
927 $(src-dir)/essay/colorado.bib
930 The input file is @file{colorado.bib} in the @file{essay}
933 The @code{bib2texi} Python script used to be used with a variety
934 of options, but now is always called using the same options, as
935 above. Its job is to create the file containing the options for
936 @code{bibtex} (the program that actually does the translation),
937 run bibtex, and then clean up some temporary files. Its main
938 "value add" is the creation of the options file, using this code:
941 open (tmpfile + '.aux', 'w').write (r'''
944 \bibstyle@{%(style)s@}
945 \bibdata@{%(files)s@}''' % vars ())
948 The key items are the style file (now always lily-bib for us) and
951 The style file is written in its own specialised language,
952 described to some extent at
955 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
958 The file @file{lily-bib.bst} also has fairly extensive commenting.
961 @section Website build
963 Start here: @file{make/website.make}
965 The overall build system begins with @ref{How stepmake works}.
966 Summary: when you type @code{make website} this ends up running
967 @file{GNUmakefile.in} in the @file{git} directory. Right at the
968 bottom, this has the lines:
971 # we want this separate for security; see CG 4.2. -gp
973 $(MAKE) config_make=$(config_make) \
974 top-src-dir=$(top-src-dir) \
975 -f $(top-src-dir)/make/website.make \
979 On my system this expands to:
982 make --no-builtin-rules config_make=./config.make \
983 top-src-dir=/home/phil/lilypond-git \
984 -f /home/phil/lilypond-git/make/website.make \
988 We see that the @code{$(MAKE)} expands to
989 @code{make --no-builtin-rules} which is how @code{MAKE} is
990 defined higher up the makefile. The -f switch defines the
991 makefile to be used - in this case
992 @file{git/make/website.make}. That's where all the action
995 We believe that note that *none* of the variables that
996 are loaded (from depth to version numbers to whatever) are used in
997 @file{website.make}. Instead, @file{website.make} sets up its own
998 variables at the top of the file. If you're wondering if there's
999 some smart reason for this, then the answer is "no". It's because
1000 I (GP) didn't know/trust the original variables when I was writing
1003 Website build includes @ref{Building a bibliography}.
1005 @subsubheading Output from @code{make -n website}
1007 Sorry, including this output directly produces problems in the
1008 build system. Please run:
1011 make -n website &> my-file.txt
1014 to see the full output from the make.
1016 @subsubheading website.make variables
1018 The file begins by setting up some variables. These
1019 may/might/probably mirror existing variables, but lacking any docs
1020 about those variables, I thought it would be simpler to keep
1021 everything in the same file.
1023 Note that for security reasons, we @strong{don't} call scripts in
1024 the git dir when building on the web server. See @ref{Uploading
1025 and security}. So we definitely want to keep those definitions
1026 for the WEBSITE_ONLY_BUILD.
1028 After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
1029 there's another bunch of lines setting up generic variables.
1031 @subsubheading website.make building parts
1033 Parts of @file{website.make}:
1039 this is the "master" rule. It calls the other rules in order,
1040 then copies some extra files around - see below for further
1041 of the process it produces.
1044 @code{website-version}:
1045 this calls the python scripts below:
1050 scripts/build/create-version-itexi.py
1053 This writes a @@version, @@versionStable, and @@versionDevel based
1054 on the top-level VERSIONS file, to
1055 @code{out-website/version.itexi}
1059 scripts/build/create-weblinks-itexi.py
1062 This creates a ton of macros in @code{out-website/weblinks.itexi}.
1063 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
1064 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
1066 It's quite monstrous because it deals with combinations of
1067 stable/devel, source/docs, lang/lang/lang*10, etc.
1073 @code{website-xrefs:}
1074 creates files used for complicated "out-of-build" references to
1075 @code{out-website/*.xref-map}
1077 If you just write @@ref@{@}, then all's groovy and we wouldn't
1078 need this. But if you write @@rlearning@{@}, then our custom
1079 texi2html init file needs to know about our custom xref file
1080 format, which tells our custom texi2html init file how to create
1083 GP: we should have a separate @@node to discuss xrefs. Also, take a
1084 quick look at a generated xref file -- it's basically just a list
1085 of @@node's [sic teenager pluralization rule] from the file.
1089 generates the bibliography texinfo files from the .bib files - in
1090 the case of the website build these are @file{others-did.bib} and
1091 @file{we-wrote.bib}.
1094 @code{website-texinfo:}
1095 this is the main part; it calles texi2html to generate the actual
1096 html. It also has a ton of options to texi2html to pass info to
1097 our custom init file.
1099 The file actually built is called @file{web.texi}, and is either
1100 in the @file{Documentation} directory, or a sub-directory specific
1103 The options file is @file{/Documentation/lilypond-texi2html.init}.
1104 This contains *lots* of option and configuration stuff, and also
1108 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
1111 This is where one of the console messages is generated.
1113 We have somewhere between 2-4 different ways "to pass info to our
1114 custom init file". This is highly Not Good (tm), but that's how
1115 things work at the moment.
1117 After texi2html, it does some black magick to deal with
1118 untranslated nodes in the translations. Despite writing that
1119 part, I can't remember how it works. But in theory, you could
1120 figure it out by copy&pasting each part of the command (by "part",
1121 I mean "stuff before each | pipe"), substituting the variables,
1122 then looking at the text that's output. For example,
1125 ls $(OUT)/$$l/*.html
1128 is going to print a list of all html files, in all languages, in
1129 the build directory. Then more stuff happens to each of those
1130 files (that's what xargs does).
1134 just copies files to the build dir.
1137 @code{website-pictures, website-examples:}
1138 more file copies, with an if statement to handle if you don't have
1139 any generated pictures/examples.
1146 scripts/build/website_post.py
1149 which, it adds the "this page is translated in klingon" to the
1150 bottom of html pages, and adds the google analytics javascript.
1151 It also has hard-coded lilypond version numbers, which is Bad
1156 Here's a summary of what gets called, in what order, when we run
1163 creates version.itexi and weblinks.itexi
1165 runs extract_texi_filenames.py
1167 creates bibliography files, described above
1175 runs website_post.py
1176 Then some file copying
1179 @node Building an Ubuntu distro
1180 @section Building an Ubuntu distro
1183 Here's the short instruction on how to create lilybuntu iso image
1184 (Jonathan Kulp did this on a spare drive,
1185 but he supposes it can be done in a VM too):
1190 Install ubuntu, reboot.
1192 Run all updates, reboot if asked.
1194 Enable src repos, refresh package lists.
1196 Install LilyPond build deps:
1198 sudo apt-get build-dep lilypond
1201 Install git and autoconf:
1203 sudo apt-get install git-core gitk autoconf
1207 Test to see whether everything works fine now:
1210 use @command{lily-git.tcl} to grab source files
1212 go to source dir and do
1214 "./autogen.sh" ; make ; make doc
1217 if all compiles, move on to iso creation...
1221 Download & install "remastersys":
1222 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
1224 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
1226 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
1227 default live session username, etc).
1229 Remove non-essential desktop software as desired.
1233 sudo remastersys dist
1235 New iso is in @file{/home/remastersys/remastersys/}.
1237 Test iso by installing in VM and repeating steps above for
1238 getting source files and building lp and docs.