1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
21 @node Build system overview
22 @section Build system overview
24 Build system is currently GNU make, with an extra "stepmake" layer
25 on top. Look at files in @file{make/} and @file{stepmake/} and
26 all @file{GNUmakefile}s.
28 There is wide-spread dissatisfaction with this system, and we are
29 considering changing. This would be a huge undertaking (estimated
30 200+ hours). This change will probably involve not using GNU make
31 any more -- but a discussion about the precise build system will
32 have to wait. Before we reach that point, we need to figure out
33 (at least approximately) what the current build system does.
35 Fundamentally, a build system does two things:
39 Constructs command-line commands, for example:
43 --tons --of --options \
46 --more --imperial --and --metric --tons --of --options \
51 If there was a previous build, it decides which parts of the
52 system need to be rebuilt.
56 When I try to do anything in the build system, it helps to remind
57 myself of this. The "end result" is just a series of command-line
58 commands. All the black magick is just an attempt to construct
61 @node Tips for working on the build system
62 @section Tips for working on the build system
74 to the build system files in various places. This will let you
75 track where the program is, in various points of the build.
77 PH note. There are lots of places where Make doesn't let you put
78 echo commands. My top tip for tracing how make runs is to put
81 $(error Some Text to display)
84 This will stop make running and print the text @code{Some Text to
90 First task: understand how @code{make website} works,
91 @emph{without} the translations. Looking at the english-only
92 website is the best introduction to the build system... it only
93 covers about 5% of the whole thing, but even that will likely take
99 @node General build system notes
100 @section General build system notes
103 * How stepmake works::
106 @node How stepmake works
107 @subsection How stepmake works
109 Typing make website runs the file @file{GNUmakefile} from the
110 build directory. This only contains 3 lines:
114 include config$(if $(conf),-$(conf),).make
115 include $(configure-srcdir)/GNUmakefile.in
118 The variable @code{depth} is used throughout the make system to
119 track how far down the directory structure the make is. The first
120 include sets lots of variables but doesn't "do" anything. The
121 second runs the file @file{GNUmakefile.in} from the top level
124 This sets another load of variables, and then includes (i.e.
125 immediately runs) @file{stepmake.make} from the @file{make}
126 subdirectory. This sets a load of other variables, does some
127 testing to see if SCONS (another build tool?) is being used, and
128 then runs @file{make/config.make} - which doesn't seem to exist...
130 GP: scons is indeed a different build tool; I think that Jan
131 experimented with it 5 years ago or something. It seems like we
132 still have bits and pieces of it floating around.
134 Next, it runs @file{make/toplevel-version.make}, which sets the
135 version variables for major, minor, patch, stable, development and
136 mypatchlevel (which seems to be used for patch numbers for
137 non-stable versions only?).
139 Next - @file{make/local.make}, which doesn't exist.
141 Then a few more variable and the interesting comment:
144 # Don't try to outsmart us, you puny computer!
145 # Well, UGH. This only removes builtin rules from
148 and then tests to see whether BUILTINS_REMOVED is defined. It
149 appears to be when I run make, and so
150 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
151 comment at the head of this file says:
154 # UGH. GNU make comes with implicit rules.
155 # We don't want any of them, and can't force users to run
159 I've not studied that file at length, but assume it removes all
160 make's build-in rules (e.g. @file{*.c} files are run through the
161 GNU C compiler) - there's a lot of them in here, and a lot of
162 comments, and I'd guess most of it isn't needed.
164 We return to @file{stepmake.make}, where we hit the make rule all:
165 The first line of this is:
168 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
171 which, when the variables are substituted, gives:
174 ./make/generic-inclusions.make
175 ./make/lilypond-inclusions.make.
178 (Note - according to the make documentation, -include is only
179 different from include in that it doesn't produce any kind of
180 error message when the included file doesn't exist).
182 And the first file doesn't exist. Nor the second. Next:
185 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
188 which expands to the following files:
191 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
192 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
193 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
197 One little feature to notice here - these are all absolute file
198 locations - the line prior to this used relative locations. And
199 none of these files exist, either. (Further note - I'm assuming
200 all these lines of make I'm following are autogenerated, but
201 that'll be something else to discover.)
203 Next in @file{stepmake.make}:
206 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
212 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
213 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
214 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
215 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
218 Woo. They all exist (they should as there's no - in front of the
219 include). @file{generic-vars.make} sets loads of variables
220 (funnily enough). @file{toplevel-vars.make} is very short - one
221 line commented as @code{# override Generic_vars.make:} and 2 as
226 include $(stepdir)/documentation-vars.make
229 I assume the urg comment refers to the fact that this should
230 really just create more variables, but it actually sends us off to
231 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
233 That file is a 3 line variable setting one.
235 @file{po-vars.make} has the one-line comment @code{# empty}, as
236 does @file{install-vars.make}.
238 So now we're back to @file{stepmake.make}.
243 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
244 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
247 and the include expands to:
250 include ./make/generic-vars.make ./make/lilypond-vars.make.
253 These again set variables, and in some cases export them to allow
254 child @code{make} processes to use them.
256 The final 4 lines of @file{stepmake.make} are:
259 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
260 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
261 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
262 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
265 which expand as follows:
268 include ./make/generic-rules.make ./make/lilypond-rules.make
270 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
271 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
272 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
273 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
274 include ./make/generic-targets.make ./make/lilypond-targets.make
276 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
277 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
278 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
279 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
282 @file{lilypond-rules.make} is @code{#empty}
284 @file{generic-rules.make} does seem to have 2 rules in it. They
288 $(outdir)/%.ly: %.lym4
289 $(M4) $< | sed "s/\`/,/g" > $@
293 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
296 I believe the first rule is for *.ly files, and has a prerequisite
297 that *.lym4 files must be built first. The recipe is @code{m4 |
298 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
299 can comment on exactly what the rules mean/do.
301 @file{toplevel-rules.make} is @code{#empty}
303 @file{po-rules.make} is @code{#empty}
305 @file{install-rules.make} is @code{#empty}
307 @file{generic-targets.make} contains 2 lines of comments.
309 @file{lilypond-targets.make} contains only:
312 ## TODO: fail dist or web if no \version present.
314 grep -L version $(LY_FILES)
317 @file{stepmake/generic-targets.make} contains lots of rules - too
318 many to list here - it seems to be the main file for rules. (FWIW
319 I haven't actually found a rule for website: anywhere, although
320 it clearly exists. I have also found that you can display a rule
321 in the terminal by typing, say @code{make -n website}. This is
322 probably common knowledge.
324 @file{stepmake/toplevel-targets.make} adds a load of other (and
325 occasionally the same) rules to the gernric-targets.
327 @file{stepmake/po-targets.make} is rules for po* makes.
329 @file{stepmake/install-targets.make} has rules for local-install*.
331 And that's the end of stepmake.make. Back to
332 @file{GNUmakefile.in}.
334 A bit more info from 27 March. I've put some error traces into
335 @code{GNUmakefile} in the build directory, and it looks like the
336 following lines actually cause the make to run (putting an error
337 call above them - no make; below them - make):
341 # All web targets, except info image symlinks and info docs are
342 # installed in non-recursing target from TOP-SRC-DIR
344 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
345 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
346 $(MAKE) -C Documentation omf-local-install
349 I don't currently understand the @code{ifeq}, since @code{$(out)}
350 is empty at this point, but the line starting @code{-$(INSTALL)}
354 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py -c -m 755 -d /usr/local/share/doc/lilypond/html
357 End of work for Sunday 27th.
359 Another alterative approach to understanding the website build
360 would be to redirect @code{make -n website} and @code{make website}
361 to a text file and work through a) what it does and b) where the
362 errors are occurring.
364 GP: wow, all the above is much more complicated than I've ever
365 looked at stuff -- I tend to do a "back first" approach (where I
366 begin from the command-line that I want to modify, figure out
367 where it's generated, and then figure out how to change the
368 generated command-line), rather than a "front first" (where you
369 begin from the "make" command).
376 * Building a bibliography::
379 @node Building a bibliography
380 @subsection Building a bibliography
382 Bibliography files contain a list of citations, like this:
386 author = @{Vinci, Albert C.@},
387 title = @{Fundamentals of Traditional Music Notation@},
388 publisher = @{Kent State University Press@},
393 There are a variety of types of citation (e.g. Book (as above),
394 article, publication). Each cited publication has a list of
395 entries that can be used to identify the publication.
396 Bibliograpies are normally stored as files with a .bib
397 extension. One part of the doc-build process is transforming the
398 bibliography information into @code{texinfo} files. The commands
399 to do this are in the @file{GNUmakefile} in the
400 @file{Documentation} directory.
402 A typical line of the makefile to translate a single bibliography
406 $(outdir)/colorado.itexi:
407 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
408 -s $(top-src-dir)/Documentation/lily-bib \
409 -o $(outdir)/colorado.itexi \
410 $(src-dir)/essay/colorado.bib
416 $(outdir)/colorado.itexi:
419 We're making the file @file{colorado.itexi} and so this is the
423 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
426 It's in the @file{essay} directory and we want to run the
427 bib2texi.py script against it.
430 -s $(top-src-dir)/Documentation/lily-bib \
433 The style template is @file{lily-bib.bst} and is found in the
434 @file{Documentation} directory.
437 -o $(outdir)/colorado.itexi \
440 The output file in @file{colorado.itexi}.
443 $(src-dir)/essay/colorado.bib
446 The input file is @file{colorado.bib} in the @file{essay}
449 The @code{bib2texi} Python script used to be used with a variety
450 of options, but now is always called using the same options, as
451 above. Its job is to create the file containing the options for
452 @code{bibtex} (the program that actually does the translation),
453 run bibtex, and then clean up some temporary files. Its main
454 "value add" is the creation of the options file, using this code:
457 open (tmpfile + '.aux', 'w').write (r'''
460 \bibstyle@{%(style)s@}
461 \bibdata@{%(files)s@}''' % vars ())
464 The key items are the style file (now always lily-bib for us) and
467 The style file is written in its own specialised language,
468 described to some extent at
471 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
474 The file @file{lily-bib.bst} also has fairly extensive commenting.
477 @section Website build
479 Start here: @file{make/website.make}
481 The overall build system begins with @ref{How stepmake works}.
482 Summary: when you type @code{make website} this ends up running
483 @file{GNUmakefile.in} in the @file{git} directory. Right at the
484 bottom, this has the lines:
487 # we want this separate for security; see CG 4.2. -gp
489 $(MAKE) config_make=$(config_make) \
490 top-src-dir=$(top-src-dir) \
491 -f $(top-src-dir)/make/website.make \
495 On my system this expands to:
498 make --no-builtin-rules config_make=./config.make \
499 top-src-dir=/home/phil/lilypond-git \
500 -f /home/phil/lilypond-git/make/website.make \
504 We see that the @code{$(MAKE)} expands to
505 @code{make --no-builtin-rules} which is how @code{MAKE} is
506 defined higher up the makefile. The -f switch defines the
507 makefile to be used - in this case
508 @file{git/make/website.make}. That's where all the action
511 We believe that note that *none* of the variables that
512 are loaded (from depth to version numbers to whatever) are used in
513 @file{website.make}. Instead, @file{website.make} sets up its own
514 variables at the top of the file. If you're wondering if there's
515 some smart reason for this, then the answer is "no". It's because
516 I (GP) didn't know/trust the original variables when I was writing
519 Website build includes @ref{Building a bibliography}.
521 @subsubheading Output from @code{make -n website}
523 Sorry, including this output directly produces problems in the
524 build system. Please run:
527 make -n website &> my-file.txt
530 to see the full output from the make.
532 @subsubheading website.make variables
534 The file begins by setting up some variables. These
535 may/might/probably mirror existing variables, but lacking any docs
536 about those variables, I thought it would be simpler to keep
537 everything in the same file.
539 Note that for security reasons, we @strong{don't} call scripts in
540 the git dir when building on the web server. See @ref{Uploading
541 and security}. So we definitely want to keep those definitions
542 for the WEBSITE_ONLY_BUILD.
544 After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
545 there's another bunch of lines setting up generic variables.
547 @subsubheading website.make building parts
549 Parts of @file{website.make}:
555 this is the "master" rule. It calls the other rules in order,
556 then copies some extra files around - see below for further
557 of the process it produces.
560 @code{website-version}:
561 this calls the python scripts below:
566 scripts/build/create-version-itexi.py
569 This writes a @@version, @@versionStable, and @@versionDevel based
570 on the top-level VERSIONS file, to
571 @code{out-website/version.itexi}
575 scripts/build/create-weblinks-itexi.py
578 This creates a ton of macros in @code{out-website/weblinks.itexi}.
579 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
580 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
582 It's quite monstrous because it deals with combinations of
583 stable/devel, source/docs, lang/lang/lang*10, etc.
589 @code{website-xrefs:}
590 creates files used for complicated "out-of-build" references to
591 @code{out-website/*.xref-map}
593 If you just write @@ref@{@}, then all's groovy and we wouldn't
594 need this. But if you write @@rlearning@{@}, then our custom
595 texi2html init file needs to know about our custom xref file
596 format, which tells our custom texi2html init file how to create
599 GP: we should have a separate @@node to discuss xrefs. Also, take a
600 quick look at a generated xref file -- it's basically just a list
601 of @@node's [sic teenager pluralization rule] from the file.
605 generates the bibliography texinfo files from the .bib files - in
606 the case of the website build these are @file{others-did.bib} and
610 @code{website-texinfo:}
611 this is the main part; it calles texi2html to generate the actual
612 html. It also has a ton of options to texi2html to pass info to
613 our custom init file.
615 The file actually built is called @file{web.texi}, and is either
616 in the @file{Documentation} directory, or a sub-directory specific
619 The options file is @file{/Documentation/lilypond-texi2html.init}.
620 This contains *lots* of option and configuration stuff, and also
624 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
627 This is where one of the console messages is generated.
629 We have somewhere between 2-4 different ways "to pass info to our
630 custom init file". This is highly Not Good (tm), but that's how
631 things work at the moment.
633 After texi2html, it does some black magick to deal with
634 untranslated nodes in the translations. Despite writing that
635 part, I can't remember how it works. But in theory, you could
636 figure it out by copy&pasting each part of the command (by "part",
637 I mean "stuff before each | pipe"), substituting the variables,
638 then looking at the text that's output. For example,
644 is going to print a list of all html files, in all languages, in
645 the build directory. Then more stuff happens to each of those
646 files (that's what xargs does).
650 just copies files to the build dir.
653 @code{website-pictures, website-examples:}
654 more file copies, with an if statement to handle if you don't have
655 any generated pictures/examples.
662 scripts/build/website_post.py
665 which, it adds the "this page is translated in klingon" to the
666 bottom of html pages, and adds the google analytics javascript.
667 It also has hard-coded lilypond version numbers, which is Bad
672 Here's a summary of what gets called, in what order, when we run
679 creates version.itexi and weblinks.itexi
681 runs extract_texi_filenames.py
683 creates bibliography files, described above
692 Then some file copying
695 @node Building an Ubuntu distro
696 @section Building an Ubuntu distro
699 Here's the short instruction on how to create lilybuntu iso image
700 (Jonathan Kulp did this on a spare drive,
701 but he supposes it can be done in a VM too):
706 Install ubuntu, reboot.
708 Run all updates, reboot if asked.
710 Enable src repos, refresh package lists.
712 Install LilyPond build deps:
714 sudo apt-get build-dep lilypond
717 Install git and autoconf:
719 sudo apt-get install git-core gitk autoconf
723 Test to see whether everything works fine now:
726 use @command{lily-git.tcl} to grab source files
728 go to source dir and do
730 "./autogen.sh" ; make ; make doc
733 if all compiles, move on to iso creation...
737 Download & install "remastersys":
738 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
740 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
742 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
743 default live session username, etc).
745 Remove non-essential desktop software as desired.
749 sudo remastersys dist
751 New iso is in @file{/home/remastersys/remastersys/}.
753 Test iso by installing in VM and repeating steps above for
754 getting source files and building lp and docs.