X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;f=Documentation%2Fcontributor%2Fbuild-notes.itexi;h=dcfb147ec64fba7004360f0d1a131ebc3516c9ed;hb=84cdc498410193c81778d4364796ef0ac86530de;hp=e7e0ba09cc2394e74b7dcdb4e3f7fe5b0b99a1af;hpb=5a9fa23f94887e1896bec7e31a57bdb9bdcde0ad;p=lilypond.git diff --git a/Documentation/contributor/build-notes.itexi b/Documentation/contributor/build-notes.itexi index e7e0ba09cc..dcfb147ec6 100644 --- a/Documentation/contributor/build-notes.itexi +++ b/Documentation/contributor/build-notes.itexi @@ -11,8 +11,10 @@ chapter.} @menu * Build system overview:: * Tips for working on the build system:: +* General build system notes:: * Doc build:: * Website build:: +* Building an Ubuntu distro:: @end menu @@ -72,6 +74,18 @@ echo "bbb" to the build system files in various places. This will let you track where the program is, in various points of the build. +PH note. There are lots of places where Make doesn't let you put +echo commands. My top tip for tracing how make runs is to put + +@example +$(error Some Text to display) +@end example + +This will stop make running and print the text @code{Some Text to +display}. + +End PH note. + @item First task: understand how @code{make website} works, @emph{without} the translations. Looking at the english-only @@ -82,6 +96,278 @@ covers about 5% of the whole thing, but even that will likely take @end itemize +@node General build system notes +@section General build system notes + +@menu +* How stepmake works:: +@end menu + +@node How stepmake works +@subsection How stepmake works + +Typing make website runs the file @file{GNUmakefile} from the +build directory. This only contains 3 lines: + +@example +depth = . +include config$(if $(conf),-$(conf),).make +include $(configure-srcdir)/GNUmakefile.in +@end example + +The variable @code{depth} is used throughout the make system to +track how far down the directory structure the make is. The first +include sets lots of variables but doesn't "do" anything. The +second runs the file @file{GNUmakefile.in} from the top level +source directory. + +This sets another load of variables, and then includes (i.e. +immediately runs) @file{stepmake.make} from the @file{make} +subdirectory. This sets a load of other variables, does some +testing to see if SCONS (another build tool?) is being used, and +then runs @file{make/config.make} - which doesn't seem to exist... + +GP: scons is indeed a different build tool; I think that Jan +experimented with it 5 years ago or something. It seems like we +still have bits and pieces of it floating around. + +Next, it runs @file{make/toplevel-version.make}, which sets the +version variables for major, minor, patch, stable, development and +mypatchlevel (which seems to be used for patch numbers for +non-stable versions only?). + +Next - @file{make/local.make}, which doesn't exist. + +Then a few more variable and the interesting comment: + +@example +# Don't try to outsmart us, you puny computer! +# Well, UGH. This only removes builtin rules from +@end example + +and then tests to see whether BUILTINS_REMOVED is defined. It +appears to be when I run make, and so +@file{stepmake/stepmake/no-builtin-rules.make} is run. The +comment at the head of this file says: + +@example +# UGH. GNU make comes with implicit rules. +# We don't want any of them, and can't force users to run +# --no-builtin-rules +@end example + +I've not studied that file at length, but assume it removes all +make's build-in rules (e.g. @file{*.c} files are run through the +GNU C compiler) - there's a lot of them in here, and a lot of +comments, and I'd guess most of it isn't needed. + +We return to @file{stepmake.make}, where we hit the make rule all: +The first line of this is: + +@example +-include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end example + +which, when the variables are substituted, gives: + +@example +./make/generic-inclusions.make +./make/lilypond-inclusions.make. +@end example + +(Note - according to the make documentation, -include is only +different from include in that it doesn't produce any kind of +error message when the included file doesn't exist). + +And the first file doesn't exist. Nor the second. Next: + +@example +-include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES))) +@end example + +which expands to the following files: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make +/home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make. +@end example + +One little feature to notice here - these are all absolute file +locations - the line prior to this used relative locations. And +none of these files exist, either. (Further note - I'm assuming +all these lines of make I'm following are autogenerated, but +that'll be something else to discover.) + +Next in @file{stepmake.make}: + +@example +include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES))) +@end example + +which expands to: + +@example +/home/phil/lilypond-git/stepmake/stepmake/generic-vars.make +/home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make +/home/phil/lilypond-git/stepmake/stepmake/po-vars.make +/home/phil/lilypond-git/stepmake/stepmake/install-vars.make. +@end example + +Woo. They all exist (they should as there's no - in front of the +include). @file{generic-vars.make} sets loads of variables +(funnily enough). @file{toplevel-vars.make} is very short - one +line commented as @code{# override Generic_vars.make:} and 2 as +follows: + +@example +# urg? +include $(stepdir)/documentation-vars.make +@end example + +I assume the urg comment refers to the fact that this should +really just create more variables, but it actually sends us off to +@file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}. + +That file is a 3 line variable setting one. + +@file{po-vars.make} has the one-line comment @code{# empty}, as +does @file{install-vars.make}. + +So now we're back to @file{stepmake.make}. + +The next lines are +: +@example +# ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH) +include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES))) +@end example + +and the include expands to: + +@example +include ./make/generic-vars.make ./make/lilypond-vars.make. +@end example + +These again set variables, and in some cases export them to allow +child @code{make} processes to use them. + +The final 4 lines of @file{stepmake.make} are: + +@example +include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES))) +include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES))) +include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES))) +@end example + +which expand as follows: + +@example +include ./make/generic-rules.make ./make/lilypond-rules.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make + /home/phil/lilypond-git/stepmake/stepmake/po-rules.make + /home/phil/lilypond-git/stepmake/stepmake/install-rules.make +include ./make/generic-targets.make ./make/lilypond-targets.make +include + /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make + /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make + /home/phil/lilypond-git/stepmake/stepmake/po-targets.make + /home/phil/lilypond-git/stepmake/stepmake/install-targets.make +@end example + +@file{lilypond-rules.make} is @code{#empty} + +@file{generic-rules.make} does seem to have 2 rules in it. They +are: + +@example +$(outdir)/%.ly: %.lym4 + $(M4) $< | sed "s/\`/,/g" > $@ + +$(outdir)/%: %.in + rm -f $@ + cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@ +@end example + +I believe the first rule is for *.ly files, and has a prerequisite +that *.lym4 files must be built first. The recipe is @code{m4 | +sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge +can comment on exactly what the rules mean/do. + +@file{toplevel-rules.make} is @code{#empty} + +@file{po-rules.make} is @code{#empty} + +@file{install-rules.make} is @code{#empty} + +@file{generic-targets.make} contains 2 lines of comments. + +@file{lilypond-targets.make} contains only: + +@example +## TODO: fail dist or web if no \version present. +check-version: + grep -L version $(LY_FILES) +@end example + +@file{stepmake/generic-targets.make} contains lots of rules - too +many to list here - it seems to be the main file for rules. (FWIW +I haven't actually found a rule for website: anywhere, although +it clearly exists. I have also found that you can display a rule +in the terminal by typing, say @code{make -n website}. This is +probably common knowledge. + +@file{stepmake/toplevel-targets.make} adds a load of other (and +occasionally the same) rules to the gernric-targets. + +@file{stepmake/po-targets.make} is rules for po* makes. + +@file{stepmake/install-targets.make} has rules for local-install*. + +And that's the end of stepmake.make. Back to +@file{GNUmakefile.in}. + +A bit more info from 27 March. I've put some error traces into +@code{GNUmakefile} in the build directory, and it looks like the +following lines actually cause the make to run (putting an error +call above them - no make; below them - make): + +@example +ifeq ($(out),www) +# All web targets, except info image symlinks and info docs are +# installed in non-recursing target from TOP-SRC-DIR +install-WWW: + -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir) + rsync -rl --exclude='*.signature' $(outdir)/offline-root $(DESTDIR)$(webdir) + $(MAKE) -C Documentation omf-local-install +@end example + +I don't currently understand the @code{ifeq}, since @code{$(out)} +is empty at this point, but the line starting @code{-$(INSTALL)} +translates to: + +@example +-/usr/bin/python /home/phil/lilypond-git/stepmake/bin/install.py -c -m 755 -d /usr/local/share/doc/lilypond/html +@end example + +End of work for Sunday 27th. + +Another alterative approach to understanding the website build +would be to redirect @code{make -n website} and @code{make website} +to a text file and work through a) what it does and b) where the +errors are occurring. + +GP: wow, all the above is much more complicated than I've ever +looked at stuff -- I tend to do a "back first" approach (where I +begin from the command-line that I want to modify, figure out +where it's generated, and then figure out how to change the +generated command-line), rather than a "front first" (where you +begin from the "make" command). + @node Doc build @section Doc build @@ -192,8 +478,279 @@ The file @file{lily-bib.bst} also has fairly extensive commenting. Start here: @file{make/website.make} +The overall build system begins with @ref{How stepmake works}. +Summary: when you type @code{make website} this ends up running +@file{GNUmakefile.in} in the @file{git} directory. Right at the +bottom, this has the lines: + +@example +# we want this separate for security; see CG 4.2. -gp +website: + $(MAKE) config_make=$(config_make) \ + top-src-dir=$(top-src-dir) \ + -f $(top-src-dir)/make/website.make \ + website +@end example + +On my system this expands to: + +@example +make --no-builtin-rules config_make=./config.make \ + top-src-dir=/home/phil/lilypond-git \ + -f /home/phil/lilypond-git/make/website.make \ + website +@end example + +We see that the @code{$(MAKE)} expands to +@code{make --no-builtin-rules} which is how @code{MAKE} is +defined higher up the makefile. The -f switch defines the +makefile to be used - in this case +@file{git/make/website.make}. That's where all the action +happens. + +We believe that note that *none* of the variables that +are loaded (from depth to version numbers to whatever) are used in +@file{website.make}. Instead, @file{website.make} sets up its own +variables at the top of the file. If you're wondering if there's +some smart reason for this, then the answer is "no". It's because +I (GP) didn't know/trust the original variables when I was writing +that file. + Website build includes @ref{Building a bibliography}. +@subsubheading Output from @code{make -n website} + +Sorry, including this output directly produces problems in the +build system. Please run: + +@example +make -n website &> my-file.txt +@end example + +to see the full output from the make. + +@subsubheading website.make variables + +The file begins by setting up some variables. These +may/might/probably mirror existing variables, but lacking any docs +about those variables, I thought it would be simpler to keep +everything in the same file. + +Note that for security reasons, we @strong{don't} call scripts in +the git dir when building on the web server. See @ref{Uploading +and security}. So we definitely want to keep those definitions +for the WEBSITE_ONLY_BUILD. + +After some split WEBSITE_ONLY_BUILD vs. normal build definitions, +there's another bunch of lines setting up generic variables. + +@subsubheading website.make building parts + +Parts of @file{website.make}: + +@itemize + +@item +@code{website:} +this is the "master" rule. It calls the other rules in order, +then copies some extra files around - see below for further +of the process it produces. + +@item +@code{website-version}: +this calls the python scripts below: + +@itemize +@item +@example +scripts/build/create-version-itexi.py +@end example + +This writes a @@version, @@versionStable, and @@versionDevel based +on the top-level VERSIONS file, to +@code{out-website/version.itexi} + +@item +@example +scripts/build/create-weblinks-itexi.py +@end example + +This creates a ton of macros in @code{out-website/weblinks.itexi}. +Stuff like @@downloadStableLinuxNormal, @@downloadStableWidows, +@code{@@stableDocsNotationPdf@{@}}, @@downloadDevelSourch-zh. + +It's quite monstrous because it deals with combinations of +stable/devel, source/docs, lang/lang/lang*10, etc. + +@end itemize + + +@item +@code{website-xrefs:} +creates files used for complicated "out-of-build" references to +@code{out-website/*.xref-map} + +If you just write @@ref@{@}, then all's groovy and we wouldn't +need this. But if you write @@rlearning@{@}, then our custom +texi2html init file needs to know about our custom xref file +format, which tells our custom texi2html init file how to create +the link. + +GP: we should have a separate @@node to discuss xrefs. Also, take a +quick look at a generated xref file -- it's basically just a list +of @@node's [sic teenager pluralization rule] from the file. + +@item +@code{website-bib:} +generates the bibliography texinfo files from the .bib files - in +the case of the website build these are @file{others-did.bib} and +@file{we-wrote.bib}. + +@item +@code{website-texinfo:} +this is the main part; it calles texi2html to generate the actual +html. It also has a ton of options to texi2html to pass info to +our custom init file. + +The file actually built is called @file{web.texi}, and is either +in the @file{Documentation} directory, or a sub-directory specific +to the language. + +The options file is @file{/Documentation/lilypond-texi2html.init}. +This contains *lots* of option and configuration stuff, and also +includes the line: + +@example +print STDERR "Initializing settings for web site: [$Texi2HTML::THISDOC@{current_lang@}]\n"; +@end example + +This is where one of the console messages is generated. +We have somewhere between 2-4 different ways "to pass info to our +custom init file". This is highly Not Good (tm), but that's how +things work at the moment. +After texi2html, it does some black magick to deal with +untranslated nodes in the translations. Despite writing that +part, I can't remember how it works. But in theory, you could +figure it out by copy&pasting each part of the command (by "part", +I mean "stuff before each | pipe"), substituting the variables, +then looking at the text that's output. For example, +@example + ls $(OUT)/$$l/*.html +@end example + +is going to print a list of all html files, in all languages, in +the build directory. Then more stuff happens to each of those +files (that's what xargs does). + +@item +@code{website-css:} +just copies files to the build dir. + +@item +@code{website-pictures, website-examples:} +more file copies, with an if statement to handle if you don't have +any generated pictures/examples. + +@item +@code{web-post:} +runs: + +@example +scripts/build/website_post.py +@end example + +which, it adds the "this page is translated in klingon" to the +bottom of html pages, and adds the google analytics javascript. +It also has hard-coded lilypond version numbers, which is Bad +(tm). + +@end itemize + +Here's a summary of what gets called, in what order, when we run +@code{make website} + +@example +website: + website-texinfo: + website-version: + creates version.itexi and weblinks.itexi + website-xrefs: + runs extract_texi_filenames.py + website-bibs: + creates bibliography files, described above + website-css: + copies css files + website-pictures: + copies pictures + website-examples: + copies examples + web-post: + runs website_post.py + Then some file copying +@end example + +@node Building an Ubuntu distro +@section Building an Ubuntu distro + + +Here's the short instruction on how to create lilybuntu iso image +(Jonathan Kulp did this on a spare drive, +but he supposes it can be done in a VM too): + +@enumerate + +@item +Install ubuntu, reboot. +@item +Run all updates, reboot if asked. +@item +Enable src repos, refresh package lists. +@item +Install LilyPond build deps: +@example +sudo apt-get build-dep lilypond +@end example +@item +Install git and autoconf: +@example +sudo apt-get install git-core gitk autoconf +@end example + +@item +Test to see whether everything works fine now: +@enumerate +@item +use @command{lily-git.tcl} to grab source files +@item +go to source dir and do +@example +"./autogen.sh" ; make ; make doc +@end example +@item +if all compiles, move on to iso creation... +@end enumerate + +@item +Download & install "remastersys": +@uref{http://sourceforge.net/projects/remastersys/, http://sourceforge.net/projects/remastersys/} +@item +Copy @command{lily-git.tcl} script file into @file{/etc/skel/}. +@item +Modify @file{/etc/remastersys.conf} as desired (change @code{.iso} name, +default live session username, etc). +@item +Remove non-essential desktop software as desired. +@item +Create iso: +@example +sudo remastersys dist +@end example +New iso is in @file{/home/remastersys/remastersys/}. +@item +Test iso by installing in VM and repeating steps above for +getting source files and building lp and docs. + +@end enumerate