1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
14 * General build system notes::
17 * Building an Ubuntu distro::
21 @node Build system overview
22 @section Build system overview
24 Build system is currently GNU make, with an extra "stepmake" layer
25 on top. Look at files in @file{make/} and @file{stepmake/} and
26 all @file{GNUmakefile}s.
28 There is wide-spread dissatisfaction with this system, and we are
29 considering changing. This would be a huge undertaking (estimated
30 200+ hours). This change will probably involve not using GNU make
31 any more -- but a discussion about the precise build system will
32 have to wait. Before we reach that point, we need to figure out
33 (at least approximately) what the current build system does.
35 Fundamentally, a build system does two things:
39 Constructs command-line commands, for example:
43 --tons --of --options \
46 --more --imperial --and --metric --tons --of --options \
51 If there was a previous build, it decides which parts of the
52 system need to be rebuilt.
56 When I try to do anything in the build system, it helps to remind
57 myself of this. The "end result" is just a series of command-line
58 commands. All the black magick is just an attempt to construct
61 @node Tips for working on the build system
62 @section Tips for working on the build system
74 to the build system files in various places. This will let you
75 track where the program is, in various points of the build.
77 PH note. There are lots of places where Make doesn't let you put
78 echo commands. My top tip for tracing how make runs is to put
81 $(error Some Text to display)
84 This will stop make running and print the text @code{Some Text to
90 First task: understand how @code{make website} works,
91 @emph{without} the translations. Looking at the english-only
92 website is the best introduction to the build system... it only
93 covers about 5% of the whole thing, but even that will likely take
99 @node General build system notes
100 @section General build system notes
103 * How stepmake works::
106 @node How stepmake works
107 @subsection How stepmake works
109 Typing make website runs the file @file{GNUmakefile} from the
110 build directory. This only contains 3 lines:
114 include config$(if $(conf),-$(conf),).make
115 include $(configure-srcdir)/GNUmakefile.in
118 The variable @code{depth} is used throughout the make system to
119 track how far down the directory structure the make is. The first
120 include sets lots of variables but doesn't "do" anything. The
121 second runs the file @file{GNUmakefile.in} from the top level
124 This sets another load of variables, and then includes (i.e.
125 immediately runs) @file{stepmake.make} from the @file{make}
126 subdirectory. This sets a load of other variables, does some
127 testing to see if SCONS (another build tool?) is being used, and
128 then runs @file{make/config.make} - which doesn't seem to exist...
130 GP: scons is indeed a different build tool; I think that Jan
131 experimented with it 5 years ago or something. It seems like we
132 still have bits and pieces of it floating around.
134 Next, it runs @file{make/toplevel-version.make}, which sets the
135 version variables for major, minor, patch, stable, development and
136 mypatchlevel (which seems to be used for patch numbers for
137 non-stable versions only?).
139 Next - @file{make/local.make}, which doesn't exist.
141 Then a few more variable and the interesting comment:
144 # Don't try to outsmart us, you puny computer!
145 # Well, UGH. This only removes builtin rules from
148 and then tests to see whether BUILTINS_REMOVED is defined. It
149 appears to be when I run make, and so
150 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
151 comment at the head of this file says:
154 # UGH. GNU make comes with implicit rules.
155 # We don't want any of them, and can't force users to run
159 I've not studied that file at length, but assume it removes all
160 make's build-in rules (e.g. @file{*.c} files are run through the
161 GNU C compiler) - there's a lot of them in here, and a lot of
162 comments, and I'd guess most of it isn't needed.
164 We return to @file{stepmake.make}, where we hit the make rule all:
165 The first line of this is:
168 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
171 which, when the variables are substituted, gives:
174 ./make/generic-inclusions.make
175 ./make/lilypond-inclusions.make.
178 (Note - according to the make documentation, -include is only
179 different from include in that it doesn't produce any kind of
180 error message when the included file doesn't exist).
182 And the first file doesn't exist. Nor the second. Next:
185 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
188 which expands to the following files:
191 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
192 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
193 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
194 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
197 One little feature to notice here - these are all absolute file
198 locations - the line prior to this used relative locations. And
199 none of these files exist, either. (Further note - I'm assuming
200 all these lines of make I'm following are autogenerated, but
201 that'll be something else to discover.)
203 Next in @file{stepmake.make}:
206 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
212 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
213 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
214 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
215 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
218 Woo. They all exist (they should as there's no - in front of the
219 include). @file{generic-vars.make} sets loads of variables
220 (funnily enough). @file{toplevel-vars.make} is very short - one
221 line commented as @code{# override Generic_vars.make:} and 2 as
226 include $(stepdir)/documentation-vars.make
229 I assume the urg comment refers to the fact that this should
230 really just create more variables, but it actually sends us off to
231 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
233 That file is a 3 line variable setting one.
235 @file{po-vars.make} has the one-line comment @code{# empty}, as
236 does @file{install-vars.make}.
238 So now we're back to @file{stepmake.make}.
243 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
244 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
247 and the include expands to:
250 include ./make/generic-vars.make ./make/lilypond-vars.make.
253 These again set variables, and in some cases export them to allow
254 child @code{make} processes to use them.
256 The final 4 lines of @file{stepmake.make} are:
259 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
260 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
261 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
262 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
265 which expand as follows:
268 include ./make/generic-rules.make ./make/lilypond-rules.make
270 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
271 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
272 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
273 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
274 include ./make/generic-targets.make ./make/lilypond-targets.make
276 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
277 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
278 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
279 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
282 @file{lilypond-rules.make} is @code{#empty}
284 @file{generic-rules.make} does seem to have 2 rules in it. They
288 $(outdir)/%.ly: %.lym4
289 $(M4) $< | sed "s/\`/,/g" > $@
293 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
296 I believe the first rule is for *.ly files, and has a prerequisite
297 that *.lym4 files must be built first. The recipe is @code{m4 |
298 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
299 can comment on exactly what the rules mean/do.
301 @file{toplevel-rules.make} is @code{#empty}
303 @file{po-rules.make} is @code{#empty}
305 @file{install-rules.make} is @code{#empty}
307 @file{generic-targets.make} contains 2 lines of comments.
309 @file{lilypond-targets.make} contains only:
312 ## TODO: fail dist or web if no \version present.
314 grep -L version $(LY_FILES)
317 @file{stepmake/generic-targets.make} contains lots of rules - too
318 many to list here - it seems to be the main file for rules. (FWIW
319 I haven't actually found a rule for website: anywhere, although
320 it clearly exists. I have also found that you can display a rule
321 in the terminal by typing, say @code{make -n website}. This is
322 probably common knowledge.
324 @file{stepmake/toplevel-targets.make} adds a load of other (and
325 occasionally the same) rules to the gernric-targets.
327 @file{stepmake/po-targets.make} is rules for po* makes.
329 @file{stepmake/install-targets.make} has rules for local-install*.
331 And that's the end of stepmake.make. Back to
332 @file{GNUmakefile.in}.
334 A bit more info from 27 March. I've put some error traces into
335 @code{GNUmakefile} in the build directory, and it looks like the
336 following lines actually cause the make to run (putting an error
337 call above them - no make; below them - make):
341 # All web targets, except info image symlinks and info docs are
342 # installed in non-recursing target from TOP-SRC-DIR
344 -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
345 rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir)
346 $(MAKE) -C Documentation omf-local-install
349 I don't currently understand the @code{ifeq}, since @code{$(out)}
350 is empty at this point, but the line starting @code{-$(INSTALL)}
354 -/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py -c -m 755 -d /usr/local/share/doc/lilypond/html
357 End of work for Sunday 27th.
359 Another alterative approach to understanding the website build
360 would be to redirect @code{make -n website} and @code{make website}
361 to a text file and work through a) what it does and b) where the
362 errors are occurring.
364 GP: wow, all the above is much more complicated than I've ever
365 looked at stuff -- I tend to do a "back first" approach (where I
366 begin from the command-line that I want to modify, figure out
367 where it's generated, and then figure out how to change the
368 generated command-line), rather than a "front first" (where you
369 begin from the "make" command).
376 * Building a bibliography::
379 @node Building a bibliography
380 @subsection Building a bibliography
382 Bibliography files contain a list of citations, like this:
386 author = @{Vinci, Albert C.@},
387 title = @{Fundamentals of Traditional Music Notation@},
388 publisher = @{Kent State University Press@},
393 There are a variety of types of citation (e.g. Book (as above),
394 article, publication). Each cited publication has a list of
395 entries that can be used to identify the publication.
396 Bibliograpies are normally stored as files with a .bib
397 extension. One part of the doc-build process is transforming the
398 bibliography information into @code{texinfo} files. The commands
399 to do this are in the @file{GNUmakefile} in the
400 @file{Documentation} directory.
402 A typical line of the makefile to translate a single bibliography
406 $(outdir)/colorado.itexi:
407 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
408 -s $(top-src-dir)/Documentation/lily-bib \
409 -o $(outdir)/colorado.itexi \
410 $(src-dir)/essay/colorado.bib
416 $(outdir)/colorado.itexi:
419 We're making the file @file{colorado.itexi} and so this is the
423 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
426 It's in the @file{essay} directory and we want to run the
427 bib2texi.py script against it.
430 -s $(top-src-dir)/Documentation/lily-bib \
433 The style template is @file{lily-bib.bst} and is found in the
434 @file{Documentation} directory.
437 -o $(outdir)/colorado.itexi \
440 The output file in @file{colorado.itexi}.
443 $(src-dir)/essay/colorado.bib
446 The input file is @file{colorado.bib} in the @file{essay}
449 The @code{bib2texi} Python script used to be used with a variety
450 of options, but now is always called using the same options, as
451 above. Its job is to create the file containing the options for
452 @code{bibtex} (the program that actually does the translation),
453 run bibtex, and then clean up some temporary files. Its main
454 "value add" is the creation of the options file, using this code:
457 open (tmpfile + '.aux', 'w').write (r'''
460 \bibstyle@{%(style)s@}
461 \bibdata@{%(files)s@}''' % vars ())
464 The key items are the style file (now always lily-bib for us) and
467 The style file is written in its own specialised language,
468 described to some extent at
471 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
474 The file @file{lily-bib.bst} also has fairly extensive commenting.
477 @section Website build
479 Start here: @file{make/website.make}
481 The overall build system begins with @ref{How stepmake works}.
483 However, we do believe that note that *none* of the variables that
484 are loaded (from depth to version numbers to whatever) are used in
485 @file{website.make}. Instead, @file{website.make} sets up its own
486 variables at the top of the file. If you're wondering if there's
487 some smart reason for this, then the answer is "no". It's because
488 I didn't know/trust the original variables when I was writing that
492 Website build includes @ref{Building a bibliography}.
494 @subsubheading Output from @code{make -n website}
496 Sorry, including this output directly produces problems in the
497 build system. Please run:
500 make -n website &> my-file.txt
504 @subsubheading website.make variables
506 The file begins by setting up some variables. These
507 may/might/probably mirror existing variables, but lacking any docs
508 about those variables, I thought it would be simpler to keep
509 everything in the same file.
511 Note that for security reasons, we @strong{don't} call scripts in
512 the git dir when building on the web server. See @ref{Uploading
513 and security}. So we definitely want to keep those definitions
514 for the WEBSITE_ONLY_BUILD.
516 After some split WEBSITE_ONLY_BUILD vs. normal build definitions,
517 there's another bunch of lines setting up generic variables.
519 @subsubheading website.make building parts
521 Parts of @file{website.make}:
526 @code{website-version}:
527 this calls python scripts to define teinxfo macros.
532 scripts/build/create-version-itexi.py
535 This writes a @@version, @@versionStable, and @@versionDevel based
536 on the top-level VERSIONS file, to
537 @code{out-website/version.itexi}
541 scripts/build/create-weblinks-itexi.py
544 This creates a ton of macros in @code{out-website/weblinks.itexi}.
545 Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows,
546 @code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh.
548 It's quite monstrous because it deals with combinations of
549 stable/devel, source/docs, lang/lang/lang*10, etc.
555 @code{website-xrefs:}
556 creates files used for complicated "out-of-build" references to
557 @code{out-website/*.xref-map}
559 If you just write @@ref@{@}, then all's groovy and we wouldn't
560 need this. But if you write @@rlearning@{@}, then our custom
561 texi2html init file needs to know about our custom xref file
562 format, which tells our custom texi2html init file how to create
565 GP: we should have a separate @@node to discuss xrefs. Also, take a
566 quick look at a generated xref file -- it's basically just a list
567 of @@node's [sic teenager pluralization rule] from the file.
571 generates the bibliography texinfo files from the .bib files.
574 @code{website-texinfo:}
575 this is the main part; it calles texi2html to generate the actual
576 html. It also has a ton of options to texi2html to pass info to
577 our custom init file.
579 The file actually built is called @file{web.texi}, and is either
580 in the @file{Documentation} directory, or a sub-directory specific
583 The options file is @file{/Documentation/lilypond-texi2html.init}.
584 This contains *lots* of option and configuration stuff, and also
588 print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n";
591 This is where one of the console messages is generated.
593 We have somewhere between 2-4 different ways "to pass info to our
594 custom init file". This is highly Not Good (tm), but that's how
595 things work at the moment.
597 After texi2html, it does some black magick to deal with
598 untranslated nodes in the translations. Despite writing that
599 part, I can't remember how it works. But in theory, you could
600 figure it out by copy&pasting each part of the command (by "part",
601 I mean "stuff before each | pipe"), substituting the variables,
602 then looking at the text that's output. For example,
608 is going to print a list of all html files, in all languages, in
609 the build directory. Then more stuff happens to each of those
610 files (that's what xargs does).
614 just copies files to the build dir.
617 @code{website-pictures, website-examples:}
618 more file copies, with an if statement to handle if you don't have
619 any generated pictures/examples.
626 scripts/build/website_post.py
629 which, it adds the "this page is translated in klingon" to the
630 bottom of html pages, and adds the google analytics javascript.
631 It also has hard-coded lilypond version numbers, which is Bad
636 this is the "master" rule. It calls the bits and pieces in order,
637 then copies some extra files around.
642 @node Building an Ubuntu distro
643 @section Building an Ubuntu distro
646 Here's the short instruction on how to create lilybuntu iso image
647 (Jonathan Kulp did this on a spare drive,
648 but he supposes it can be done in a VM too):
653 Install ubuntu, reboot.
655 Run all updates, reboot if asked.
657 Enable src repos, refresh package lists.
659 Install LilyPond build deps:
661 sudo apt-get build-dep lilypond
664 Install git and autoconf:
666 sudo apt-get install git-core gitk autoconf
670 Test to see whether everything works fine now:
673 use @command{lily-git.tcl} to grab source files
675 go to source dir and do
677 "./autogen.sh" ; make ; make doc
680 if all compiles, move on to iso creation...
684 Download & install "remastersys":
685 @uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/}
687 Copy @command{lily-git.tcl} script file into @file{/etc/skel/}.
689 Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name,
690 default live session username, etc).
692 Remove non-essential desktop software as desired.
696 sudo remastersys dist
698 New iso is in @file{/home/remastersys/remastersys/}.
700 Test iso by installing in VM and repeating steps above for
701 getting source files and building lp and docs.