1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
16 * Building an Ubuntu distro::
20 @node Build system overview
21 @section Build system overview
23 Build system is currently GNU make, with an extra "stepmake" layer
24 on top. Look at files in @file{make/} and @file{stepmake/} and
25 all @file{GNUmakefile}s.
27 There is wide-spread dissatisfaction with this system, and we are
28 considering changing. This would be a huge undertaking (estimated
29 200+ hours). This change will probably involve not using GNU make
30 any more -- but a discussion about the precise build system will
31 have to wait. Before we reach that point, we need to figure out
32 (at least approximately) what the current build system does.
34 Fundamentally, a build system does two things:
38 Constructs command-line commands, for example:
42 --tons --of --options \
45 --more --imperial --and --metric --tons --of --options \
50 If there was a previous build, it decides which parts of the
51 system need to be rebuilt.
55 When I try to do anything in the build system, it helps to remind
56 myself of this. The "end result" is just a series of command-line
57 commands. All the black magick is just an attempt to construct
60 @node Tips for working on the build system
61 @section Tips for working on the build system
73 to the build system files in various places. This will let you
74 track where the program is, in various points of the build.
77 First task: understand how @code{make website} works,
78 @emph{without} the translations. Looking at the english-only
79 website is the best introduction to the build system... it only
80 covers about 5% of the whole thing, but even that will likely take
91 * Building a bibliography::
94 @node Building a bibliography
95 @subsection Building a bibliography
97 Bibliography files contain a list of citations, like this:
101 author = @{Vinci, Albert C.@},
102 title = @{Fundamentals of Traditional Music Notation@},
103 publisher = @{Kent State University Press@},
108 There are a variety of types of citation (e.g. Book (as above),
109 article, publication). Each cited publication has a list of
110 entries that can be used to identify the publication.
111 Bibliograpies are normally stored as files with a .bib
112 extension. One part of the doc-build process is transforming the
113 bibliography information into @code{texinfo} files. The commands
114 to do this are in the @file{GNUmakefile} in the
115 @file{Documentation} directory.
117 A typical line of the makefile to translate a single bibliography
121 $(outdir)/colorado.itexi:
122 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
123 -s $(top-src-dir)/Documentation/lily-bib \
124 -o $(outdir)/colorado.itexi \
125 $(src-dir)/essay/colorado.bib
131 $(outdir)/colorado.itexi:
134 We're making the file @file{colorado.itexi} and so this is the
138 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
141 It's in the @file{essay} directory and we want to run the
142 bib2texi.py script against it.
145 -s $(top-src-dir)/Documentation/lily-bib \
148 The style template is @file{lily-bib.bst} and is found in the
149 @file{Documentation} directory.
152 -o $(outdir)/colorado.itexi \
155 The output file in @file{colorado.itexi}.
158 $(src-dir)/essay/colorado.bib
161 The input file is @file{colorado.bib} in the @file{essay}
164 The @code{bib2texi} Python script used to be used with a variety
165 of options, but now is always called using the same options, as
166 above. Its job is to create the file containing the options for
167 @code{bibtex} (the program that actually does the translation),
168 run bibtex, and then clean up some temporary files. Its main
169 "value add" is the creation of the options file, using this code:
172 open (tmpfile + '.aux', 'w').write (r'''
175 \bibstyle@{%(style)s@}
176 \bibdata@{%(files)s@}''' % vars ())
179 The key items are the style file (now always lily-bib for us) and
182 The style file is written in its own specialised language,
183 described to some extent at
186 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
189 The file @file{lily-bib.bst} also has fairly extensive commenting.
192 @section Website build
194 Start here: @file{make/website.make}
196 Typing make website runs the file @file{GNUmakefile} from the
197 build directory. This only contains 3 lines:
201 include config$(if $(conf),-$(conf),).make
202 include $(configure-srcdir)/GNUmakefile.in
205 The variable @code{depth} is used throughout the make system to
206 track how far down the directory structure the make is. The first
207 include sets lots of variables but doesn't "do" anything. The
208 second runs the file @file{GNUmakefile.in} from the top level
211 This sets another load of variables, and then includes (i.e.
212 immediately runs) @file{stepmake.make} from the @file{make}
213 subdirectory. This sets a load of other variables, does some
214 testing to see if SCONS (another build tool?) is being used, and
215 then runs @file{make/config.make} - which doesn't seem to exist...
217 GP: scons is indeed a different build tool; I think that Jan
218 experimented with it 5 years ago or something. It seems like we
219 still have bits and pieces of it floating around.
221 GP: note that *none* of the variables that are loaded (from depth
222 to version numbers to whatever) are used in website.make.
223 Instead, website.make sets up its own variables at the top of the
224 file. If you're wondering if there's some smart reason for this,
225 then the answer is "no". It's because I didn't know/trust the
226 original variables when I was writing that file.
228 Next, it runs @file{make/toplevel-version.make}, which sets the
229 version variables for major, minor, patch, stable, development and
230 mypatchlevel (which seems to be used for patch numbers for
231 non-stable versions only?).
233 Next - @file{make/local.make}, which doesn't exist.
235 Then a few more variable and the interesting comment:
238 # Don't try to outsmart us, you puny computer!
239 # Well, UGH. This only removes builtin rules from
242 and then tests to see whether BUILTINS_REMOVED is defined. It
243 appears to be when I run make, and so
244 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
245 comment at the head of this file says:
248 # UGH. GNU make comes with implicit rules.
249 # We don't want any of them, and can't force users to run
253 I've not studied that file at length, but assume it removes all
254 make's build-in rules (e.g. @file{*.c} files are run through the
255 GNU C compiler) - there's a lot of them in here, and a lot of
256 comments, and I'd guess most of it isn't needed.
258 We return to @file{stepmake.make}, where we hit the make rule all:
259 The first line of this is:
262 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
265 which, when the variables are substituted, gives:
268 ./make/generic-inclusions.make
269 ./make/lilypond-inclusions.make.
272 (Note - according to the make documentation, -include is only
273 different from include in that it doesn't produce any kind of
274 error message when the included file doesn't exist).
276 And the first file doesn't exist. Nor the second. Next:
279 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
282 which expands to the following files:
285 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
286 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
287 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
288 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
291 One little feature to notice here - these are all absolute file
292 locations - the line prior to this used relative locations. And
293 none of these files exist, either. (Further note - I'm assuming
294 all these lines of make I'm following are autogenerated, but
295 that'll be something else to discover.)
297 Next in @file{stepmake.make}:
300 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
306 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
307 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
308 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
309 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
312 Woo. They all exist (they should as there's no - in front of the
313 include). @file{generic-vars.make} sets loads of variables
314 (funnily enough). @file{toplevel-vars.make} is very short - one
315 line commented as @code{# override Generic_vars.make:} and 2 as
320 include $(stepdir)/documentation-vars.make
323 I assume the urg comment refers to the fact that this should
324 really just create more variables, but it actually sends us off to
325 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
327 That file is a 3 line variable setting one.
329 @file{po-vars.make} has the one-line comment @code{# empty}, as
330 does @file{install-vars.make}.
332 So now we're back to @file{stepmake.make}.
337 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
338 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
341 and the include expands to:
344 include ./make/generic-vars.make ./make/lilypond-vars.make.
347 These again set variables, and in some cases export them to allow
348 child @code{make} processes to use them.
350 The final 4 lines of @file{stepmake.make} are:
353 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
354 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
355 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
356 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
359 which expand as follows:
362 include ./make/generic-rules.make ./make/lilypond-rules.make
364 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
365 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
366 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
367 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
368 include ./make/generic-targets.make ./make/lilypond-targets.make
370 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
371 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
372 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
373 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
376 @file{lilypond-rules.make} is @code{#empty}
377 @file{generic-rules.make} does seem to have 2 rules in it. They
381 $(outdir)/%.ly: %.lym4
382 $(M4) $< | sed "s/\`/,/g" > $@
386 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
389 I believe the first rule is for *.ly files, and has a prerequisite
390 that *.lym4 files must be built first. The recipe is @code{m4 |
391 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
392 can comment on exactly what the rules mean/do.
394 @file{toplevel-rules.make} is @code{#empty}
395 @file{po-rules.make} is @code{#empty}
396 @file{install-rules.make} is @code{#empty}
397 @file{generic-targets.make} contains 2 lines of comments.
398 @file{lilypond-targets.make} contains only:
401 ## TODO: fail dist or web if no \version present.
403 grep -L version $(LY_FILES)
406 @file{stepmake/generic-targets.make} contains lots of rules - too
407 many to list here - it seems to be the main file for rules. (FWIW
408 I haven't actually found a rule for website: anywhere, although
409 it clearly exists. I have also found that you can display a rule
410 in the terminal by typing, say @code{make -n website}. This is
411 probably common knowledge.
413 @file{stepmake/toplevel-targets.make} adds a load of other (and
414 occasionally the same) rules to the gernric-targets.
416 @file{stepmake/po-targets.make} is rules for po* makes.
418 @file{stepmake/install-targets.make} has rules for local-install*.
420 And that's the end of stepmake.make. Back to
421 @file{GNUmakefile.in}. More some other time.
423 Another alterative approach to understanding the website build
424 would be to redirect @code{make -n website} and @code{make website}
425 to a text file and work through a) what it does and b) where the
426 errors are occurring.
428 GP: wow, all the above is much more complicated than I've ever
429 looked at stuff -- I tend to do a "back first" approach (where I
430 begin from the command-line that I want to modify, figure out
431 where it's generated, and then figure out how to change the
432 generated command-line), rather than a "front first" (where you
433 begin from the "make" command). In the long term, we should
434 probably move it to a general "how stepmake works" section,
435 because it's not particularly relevant to the website build.
437 Website build includes @ref{Building a bibliography}.
440 @subsubheading website.make variables
442 The file begins by setting up some variables. These
443 may/might/probably mirror existing variables, but lacking any docs
444 about those variables, I thought it would be simpler to keep
445 everything in the same file.
447 Note that for security reasons, we *don't* call scripts in the git
448 dir when building on the web server. See @ref{Uploading and
449 security}. So we definitely want to keep those definitions for
450 the WEBSITE_ONLY_BUILD.
452 After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
453 there's another bunch of lines setting up generic variables.
455 @subsubheading website.make building parts
457 The website-version rule creates files that define macros for
458 commands like @@stableDocsNotationPdf@{@}, by calling python scripts.
459 Those scripts are scripts/build/create-version-itexi.py and
460 scripts/build/create-weblinks-itexi.py.
462 website-xrefs: creates files used for complicated "out-of-build"
463 references. If you just write @@ref@{@}, then all's groovy. But
464 if you do @@rlearning@{@}, then our custom texi2html init file
465 needs to know about our custom xref file format, which tells our
466 custom texi2html init file how to create the link.
468 We should have a separate @@node to discuss xrefs. Also, take a
469 quick look at a generated xref file -- it's basically just a list
470 of @@node's [sic teenager pluralization rule] from the file.
472 website-bib: generates the bibliography texinfo files from the
475 website-texinfo: this is the money shot; it calles texi2html to
476 generate the actual html. It also has a ton of options to
477 texi2html to pass info to our custom init file.
479 After texi2html, it does some black magick to deal with
480 untranslated nodes in the translations. Despite writing that
481 part, I can't remember how it works. But in theory, you could
482 figure it out by copy&pasting each part of the command (by "part",
483 I mean "stuff before each | pipe"), substituting the variables,
484 then looking at the text that's output. For example,
486 is going to print a list of all html files, in all languages, in
487 the build directory. Then more stuff happens to each of those
488 files (that's what xargs does).
490 website-css: just copies files to the build dir.
492 website-pictures, website-examples: more file copies, with an if
493 statement to handle if you don't have any generated
496 web-post: buggered if I know. Apparently it runs the
497 scripts/build/website_post.py file, which despite writing, I
498 can't remember what it does.
500 ok, it adds the "this page is translated in klingon" to the bottom
501 of html pages, and adds the google analytics javascript. It also
502 has hard-coded lilypond version numbers.
504 website: this is the "master" rule. It calls the bits and pieces
505 in order, then copies some extra files around.
508 @node Building an Ubuntu distro
509 @section Building an Ubuntu distro
513 Install ubuntu, reboot
515 Run all updates, reboot if asked
517 Enable src repos, refresh package lists
519 Install LilyPond build deps:
521 sudo apt-get build-dep lilypond
524 Install git and autoconf:
526 sudo apt-get install git-core gitk autoconf
530 TEST TO SEE WHETHER EVERYTHING WORKS NOW:
533 Use lily-git.tcl to grab source files
535 Go to source dir and do "./autogen.sh" ; make ; make doc
537 If all compiles, move on to iso creation...
542 Download & install "remastersys":
544 http://sourceforge.net/projects/remastersys/
547 Copy lily-git.tcl script file into /etc/skel/
549 Modify /etc/remastersys.conf as desired (change .iso name, default
550 live session username, etc)
552 Remove non-essential desktop software as desired
554 Create iso: sudo remastersys dist
556 New iso is in /home/remastersys/remastersys/
558 Test iso by installing in VM and repeating steps above for
559 getting source files and building lp and docs