1 @c -*- coding: utf-8; mode: texinfo; -*-
3 @chapter Programming work
6 * Overview of LilyPond architecture::
7 * LilyPond programming languages::
8 * Programming without compiling::
11 * Debugging LilyPond::
12 * Adding or modifying features::
17 * LilyPond miscellany::
20 @node Overview of LilyPond architecture
21 @section Overview of LilyPond architecture
23 LilyPond processes the input file into graphical and musical output in a
24 number of stages. This process, along with the types of routines that
25 accomplish the various stages of the process, is described in this section. A
26 more complete description of the LilyPond architecture and internal program
27 execution is found in Erik Sandberg's
28 @uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's
31 The first stage of LilyPond processing is @emph{parsing}. In the parsing
32 process, music expressions in LilyPond input format are converted to music
33 expressions in Scheme format. In Scheme format, a music expression is a list
34 in tree form, with nodes that indicate the relationships between various music
35 events. The LilyPond parser is written in Bison.
37 The second stage of LilyPond processing is @emph{iterating}. Iterating
38 assigns each music event to a context, which is the environment in which the
39 music will be finally engraved. The context is responsible for all further
40 processing of the music. It is during the iteration stage that contexts are
41 created as necessary to ensure that every note has a Voice type context (e.g.
42 Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice,
43 GregorianTranscriptionVoice), that the Voice type contexts exist in
44 appropriate Staff type contexts, and that parallel Staff type contexts exist
45 in StaffGroup type contexts. In addition, during the iteration stage each
46 music event is assigned a moment, or a time in the music when the event
49 Each type of music event has an associated iterator. Iterators are defined in
50 *-iterator.cc. During iteration, an
51 event's iterator is called to deliver that music event to the appropriate
54 The final stage of LilyPond processing is @emph{translation}. During
55 translation, music events are prepared for graphical or midi output. The
56 translation step is accomplished by the polymorphic base class Translator
57 through its two derived classes: Engraver (for graphical output) and
58 Performer (for midi output).
60 Translators are defined in C++ files named *-engraver.cc and *-performer.cc.
61 Much of the work of translating is handled by Scheme functions,
62 which is one of the keys to LilyPond's exceptional flexibility.
64 @sourceimage{architecture-diagram,,,png}
67 @node LilyPond programming languages
68 @section LilyPond programming languages
70 Programming in LilyPond is done in a variety of programming languages. Each
71 language is used for a specific purpose or purposes. This section describes
72 the languages used and provides links to reference manuals and tutorials for
73 the relevant language.
77 The core functionality of LilyPond is implemented in C++.
79 C++ is so ubiquitous that it is difficult to identify either a reference
80 manual or a tutorial. Programmers unfamiliar with C++ will need to spend some
81 time to learn the language before attempting to modify the C++ code.
83 The C++ code calls Scheme/GUILE through the GUILE interface, which is
85 @uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE
90 The LilyPond lexer is implemented in Flex, an implementation of the Unix lex
91 lexical analyser generator. Resources for Flex can be found
92 @uref{http://flex.sourceforge.net/, here}.
96 The LilyPond parser is implemented in Bison, a GNU parser generator. The
97 Bison homepage is found at @uref{http://www.gnu.org/software/bison/,
98 gnu.org}. The manual (which includes both a reference and tutorial) is
99 @uref{http://www.gnu.org/software/bison/manual/index.html, available} in a
104 GNU Make is used to control the compiling process and to build the
105 documentation and the website. GNU Make documentation is available at
106 @uref{http://www.gnu.org/software/make/manual/, the GNU website}.
108 @subsection GUILE or Scheme
110 GUILE is the dialect of Scheme that is used as LilyPond's extension language.
111 Many extensions to LilyPond are written entirely in GUILE. The
112 @uref{http://www.gnu.org/software/guile/manual/html_node/index.html,
113 GUILE Reference Manual} is available online.
115 @uref{http://mitpress.mit.edu/sicp/full-text/book/book.html, Structure and
116 Interpretation of Computer Programs}, a popular textbook used to teach
117 programming in Scheme is available in its entirety online.
119 An introduction to Guile/Scheme as used in LilyPond can be found in the
120 @rextend{Scheme tutorial}.
124 MetaFont is used to create the music fonts used by LilyPond. A MetaFont
125 tutorial is available at @uref{http://metafont.tutorial.free.fr/, the
126 METAFONT tutorial page}.
128 @subsection PostScript
130 PostScript is used to generate graphical output. A brief PostScript tutorial
131 is @uref{http://local.wasp.uwa.edu.au/~pbourke/dataformats/postscript/,
132 available online}. The
133 @uref{http://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf, PostScript Lanugage
134 Reference} is available online in PDF format.
138 Python is used for XML2ly and is used for buillding the documentation and the
141 Python documentation is available at @uref{http://www.python.org/doc/,
144 @node Programming without compiling
145 @section Programming without compiling
147 Much of the development work in LilyPond takes place by changing *.ly or
148 *.scm files. These changes can be made without compiling LilyPond. Such
149 changes are described in this section.
152 @subsection Modifying distribution files
154 Much of LilyPond is written in Scheme or LilyPond input files. These
155 files are interpreted when the program is run, rather than being compiled
156 when the program is built, and are present in all LilyPond distributions.
157 You will find .ly files in the ly/ directory and the Scheme files in the
158 scm/ directory. Both Scheme files and .ly files can be modified and
159 saved with any text editor. It's probably wise to make a backup copy of
160 your files before you modify them, although you can reinstall if the
161 files become corrupted.
163 Once you've modified the files, you can test the changes just by running
164 LilyPond on some input file. It's a good idea to create a file that
165 demonstrates the feature you're trying to add. This file will eventually
166 become a regression test and will be part of the LilyPond distribution.
168 @subsection Desired file formatting
170 Files that are part of the LilyPond distribution have Unix-style line
171 endings (LF), rather than DOS (CR+LF) or MacOS 9 and earlier (CR). Make
172 sure you use the necessary tools to ensure that Unix-style line endings are
173 preserved in the patches you create.
175 Tab characters should not be included in files for distribution. All
176 indentation should be done with spaces. Most editors have settings to
177 allow the setting of tab stops and ensuring that no tab characters are
178 included in the file.
180 Scheme files and LilyPond files should be written according to standard
181 style guidelines. Scheme file guidelines can be found at
182 @uref{http://community.schemewiki.org/?scheme-style}. Following these
183 guidelines will make your code easier to read. Both you and others that
184 work on your code will be glad you followed these guidelines.
186 For LilyPond files, you should follow the guidelines for LilyPond snippets
187 in the documentation. You can find these guidelines at
188 @ref{Texinfo introduction and usage policy}.
190 @node Finding functions
191 @section Finding functions
193 When making changes or fixing bugs in LilyPond, one of the initial
194 challenges is finding out where in the code tree the functions to
195 be modified live. With nearly 3000 files in the source tree,
196 trial-and-error searching is generally ineffective. This section
197 describes a process for finding interesting code.
199 @subsection Using the ROADMAP
201 The file ROADMAP is located in the main directory of the lilypond source.
202 ROADMAP lists all of the directories in the LilPond source tree, along
203 with a brief description of the kind of files found in each directory.
204 This can be a very helpful tool for deciding which directories to search
205 when looking for a function.
208 @subsection Using grep to search
210 Having identified a likely subdirectory to search, the grep utility can
211 be used to search for a function name. The format of the grep command is
214 grep -i functionName subdirectory/*
217 This command will search all the contents of the directory subdirectory/
218 and display every line in any of the files that contains
219 functionName. The @code{-i} option makes @command{grep} ignore
220 case -- this can be very useful if you are not yet familiar with
221 our capitalization conventions.
223 The most likely directories to grep for function names are scm/ for
224 scheme files, ly/ for lilypond input (*.ly) files, and lily/ for C++
228 @subsection Using git grep to search
230 If you have used git to obtain the source, you have access to a
231 powerful tool to search for functions. The command:
234 git grep functionName
237 will search through all of the files that are present in the git
238 repository looking for functionName. It also presents the results
239 of the search using @code{less}, so the results are displayed one page
242 @subsection Searching on the git repository at Savannah
244 You can also use the equivalent of git grep on the Savannah server.
249 Go to http://git.sv.gnu.org/gitweb/?p=lilypond.git
252 In the pulldown box that says commit, select grep.
255 Type functionName in the search box, and hit enter/return
259 This will initiate a search of the remote git repository.
268 @subsection Handling errors
270 As a general rule, you should always try to continue computations,
271 even if there is some kind of error. When the program stops, it
272 is often very hard for a user to pinpoint what part of the input
273 causes an error. Finding the culprit is much easier if there is
274 some viewable output.
276 So functions and methods do not return errorcodes, they never
277 crash, but report a programming_error and try to carry on.
279 @subsection Languages
281 C++ and Python are preferred. Python code should use PEP 8.
283 @subsection Filenames
285 Definitions of classes that are only accessed via pointers (*) or
286 references (&) shall not be included as include files.
292 ".cc" Implementation files
293 ".icc" Inline definition files
294 ".tcc" non inline Template defs
298 (setq auto-mode-alist
299 (append '(("\\.make$" . makefile-mode)
300 ("\\.cc$" . c++-mode)
301 ("\\.icc$" . c++-mode)
302 ("\\.tcc$" . c++-mode)
303 ("\\.hh$" . c++-mode)
304 ("\\.pod$" . text-mode)
309 The class Class_name is coded in @q{class-name.*}
311 @subsection Indentation
313 Standard GNU coding style is used. In emacs:
316 (add-hook 'c++-mode-hook
317 '(lambda() (c-set-style "gnu")
321 If you like using font-lock, you can also add this to your
325 (setq font-lock-maximum-decoration t)
326 (setq c++-font-lock-keywords-3
328 c++-font-lock-keywords-3
329 '(("\\b\\(a-zA-Z_?+_\\)\\b" 1 font-lock-variable-name-face) ("\\b\\(A-Z?+a-z_?+\\)\\b" 1 font-lock-type-face))
333 Some source files may not currently have proper indenting. If this
334 is the case, it is desirable to fix the improper indenting when the
335 file is modified, with the hope of continually improving the code.
337 @subsection Indenting files with emacs in script mode
339 @c email to wl@gnu.org when I get here.
341 @warning{this is pending some confirmation on -devel. July 2009 -gp}
343 Command-line script to format stuff with emacs:
347 emacs $1 -batch --eval '(indent-region (point-min) (point-max) nil)' -f save-buffer
350 (that's all on one line)
352 Save it as a shell script, then run on the file(s) you modified.
354 @subsection Indenting with vim
356 Although emacs indentation is the LilyPond standard, acceptable
357 indentation can usually be accomplished with vim. Some hints for
369 filetype plugin indent on
371 set ignorecase smartcase
374 set statusline=%F%m%r%h%w\ %{&ff}\ %Y\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ %04l,%04v\ %p%%\ [LEN=%L]
377 " Remove trailing whitespace on write
378 autocmd BufWritePre * :%s/\s\+$//e
381 With this .vimrc, files can be reindented automatically by highlihting
382 the lines to be indented in visual mode (use V to enter visual mode)
385 A scheme.vim file will help improve the indentation. This one
386 was suggested by Patrick McCarty. It should be saved in
387 ~/.vim/after/syntax/scheme.vim.
390 " Additional Guile-specific 'forms'
391 syn keyword schemeSyntax define-public define* define-safe-public
392 syn keyword schemeSyntax use-modules define-module
393 syn keyword schemeSyntax defmacro-public define-macro
394 syn keyword schemeSyntax define-markup-command
395 syn keyword schemeSyntax define-markup-list-command
396 syn keyword schemeSyntax let-keywords* lambda* define*-public
397 syn keyword schemeSyntax defmacro* defmacro*-public
399 " All of the above should influence indenting too
400 set lw+=define-public,define*,define-safe-public,use-modules,define-module
401 set lw+=defmacro-public,define-macro
402 set lw+=define-markup-command,define-markup-list-command
403 set lw+=let-keywords*,lambda*,define*-public,defmacro*,defmacro*-public
405 " These forms should not influence indenting
409 " Try to highlight all ly: procedures
410 syn match schemeFunc "ly:[^) ]\+"
413 @subsection Classes and Types
422 Member variable names end with an underscore:
431 Macro names should be written in uppercase completely.
434 @subsection Broken code
436 Do not write broken code. This includes hardwired dependencies,
437 hardwired constants, slow algorithms and obvious limitations. If
438 you can not avoid it, mark the place clearly, and add a comment
439 explaining shortcomings of the code.
441 We reject broken-in-advance on principle.
445 Variable names should be complete words, rather than abbreviations.
446 For example, it is preferred to use @code{thickness} rather than
447 @code{th} or @code{t}.
449 Multi-word variable names in C++ should have the words separated
450 by the underscore character (@q{_}).
452 Multi-word variable names in Scheme should have the words separated
457 Comments may not be needed if descriptive variable names are used
458 in the code and the logic is straightforward. However, if the
459 logic is difficult to follow, and particularly if non-obvious
460 code has been included to resolve a bug, a comment describing
461 the logic and/or the need for the non-obvious code should be included.
463 There are instances where the current code could be commented better.
464 If significant time is required to understand the code as part of
465 preparing a patch, it would be wise to add comments reflecting your
466 understanding to make future work easier.
470 Messages need to follow Localization.
473 @subsection Localization
475 This document provides some guidelines for programmers write user
476 messages. To help translations, user messages must follow
477 uniform conventions. Follow these rules when coding for LilyPond.
478 Hopefully, this can be replaced by general GNU guidelines in the
479 future. Even better would be to have an English (en_BR, en_AM)
480 guide helping programmers writing consistent messages for all GNU
483 Non-preferred messages are marked with `+'. By convention,
484 ungrammatical examples are marked with `*'. However, such ungrammatical
485 examples may still be preferred.
490 Every message to the user should be localized (and thus be marked
491 for localization). This includes warning and error messages.
494 Do not localize/gettextify:
498 `programming_error ()'s
501 `programming_warning ()'s
507 output strings (PostScript, TeX, etc.)
512 Messages to be localized must be encapsulated in `_ (STRING)' or
513 `_f (FORMAT, ...)'. E.g.:
516 warning (_ ("need music in a score"));
517 error (_f ("cannot open file: `%s'", file_name));
520 In some rare cases you may need to call `gettext ()' by hand. This
521 happens when you pre-define (a list of) string constants for later
522 use. In that case, you'll probably also need to mark these string
523 constants for translation, using `_i (STRING)'. The `_i' macro is
524 a no-op, it only serves as a marker for `xgettext'.
527 char const* messages[] = @{
528 _i ("enable debugging output"),
529 _i ("ignore lilypond version"),
536 puts (gettext (messages i));
540 See also `flower/getopt-long.cc' and `lily/main.cc'.
543 Do not use leading or trailing whitespace in messages. If you need
544 whitespace to be printed, prepend or append it to the translated
548 message ("Calculating line breaks..." + " ");
552 Error or warning messages displayed with a file name and line
553 number never start with a capital, eg,
556 foo.ly: 12: not a duration: 3
559 Messages containing a final verb, or a gerund (`-ing'-form) always
560 start with a capital. Other (simpler) messages start with a
566 Not declaring: `foo'.
570 Avoid abbreviations or short forms, use `cannot' and `do not'
571 rather than `can't' or `don't'
572 To avoid having a number of different messages for the same
573 situation, well will use quoting like this `"message: `%s'"' for all
574 strings. Numbers are not quoted:
577 _f ("cannot open file: `%s'", name_str)
578 _f ("cannot find character number: %d", i)
582 Think about translation issues. In a lot of cases, it is better to
583 translate a whole message. The english grammar must not be imposed
584 on the translator. So, instead of
587 stem at + moment.str () + does not fit in beam
593 _f ("stem at %s does not fit in beam", moment.str ())
597 Split up multi-sentence messages, whenever possible. Instead of
600 warning (_f ("out of tune! Can't find: `%s'", "Key_engraver"));
601 warning (_f ("cannot find font `%s', loading default", font_name));
607 warning (_ ("out of tune:"));
608 warning (_f ("cannot find: `%s', "Key_engraver"));
609 warning (_f ("cannot find font: `%s', font_name));
610 warning (_f ("Loading default font"));
614 If you must have multiple-sentence messages, use full punctuation.
615 Use two spaces after end of sentence punctuation. No punctuation
616 (esp. period) is used at the end of simple messages.
619 _f ("Non-matching braces in text `%s', adding braces", text)
620 _ ("Debug output disabled. Compiled with NPRINT.")
621 _f ("Huh? Not a Request: `%s'. Ignoring.", request)
625 Do not modularize too much; words frequently cannot be translated
626 without context. It is probably safe to treat most occurences of
627 words like stem, beam, crescendo as separately translatable words.
630 When translating, it is preferable to put interesting information
631 at the end of the message, rather than embedded in the middle.
632 This especially applies to frequently used messages, even if this
633 would mean sacrificing a bit of eloquency. This holds for original
634 messages too, of course.
637 en: cannot open: `foo.ly'
638 + nl: kan `foo.ly' niet openen (1)
639 kan niet openen: `foo.ly'* (2)
640 niet te openen: `foo.ly'* (3)
644 The first nl message, although grammatically and stylistically
645 correct, is not friendly for parsing by humans (even if they speak
646 dutch). I guess we would prefer something like (2) or (3).
649 Do not run make po/po-update with GNU gettext < 0.10.35
655 @node Debugging LilyPond
656 @section Debugging LilyPond
658 The most commonly used tool for debugging LilyPond is the GNU
659 debugger gdb. The gdb tool is used for investigating and debugging
660 core Lilypond code written in C++. Another tool is available for
661 debugging Scheme code using the Guile debugger. This section
662 describes how to use both gdb and the Guile Debugger.
665 * Debugging overview::
666 * Debugging C++ code::
667 * Debugging Scheme code::
670 @node Debugging overview
671 @subsection Debugging overview
673 Using a debugger simplifies troubleshooting in at least two ways.
675 First, breakpoints can be set to pause execution at any desired point.
676 Then, when execution has paused, debugger commands can be issued to
677 explore the values of various variables or to execute functions.
679 Second, the debugger can display a stack trace, which shows the
680 sequence in which functions have been called and the arguments
681 passed to the called functions.
683 @node Debugging C++ code
684 @subsection Debugging C++ code
686 The GNU debugger, gdb, is the principal tool for debugging C++ code.
688 @unnumberedsubsubsec Compiling LilyPond for use with gdb
690 In order to use gdb with LilyPond, it is necessary to compile
691 LilyPond with debugging information. This is accomplished by running
692 the following commands in the main LilyPond source directory.
695 ./configure --disable-optimising
699 This will create a version of LilyPond containing debugging
700 information that will allow the debugger to tie the source code
701 to the compiled code.
703 You should not do @var{make install} if you want to use a debugger
704 with LilyPond. The @var{make install} command will strip debugging
705 information from the LilyPond binary.
707 @unnumberedsubsubsec Typical gdb usage
709 Once you have compiled the Lilypond image with the necessary
710 debugging information it will have been written to a location in a
711 subfolder of your current working directory:
717 This is important as you will need to let gdb know where to find the
718 image containing the symbol tables. You can invoke gdb from the
725 This loads the LilyPond symbol tables into gdb. Then, to run
726 LilyPond on @code{test.ly} under the debugger, enter
734 As an alternative to running gdb at the command line you may try
735 a graphical interface to gdb such as ddd
741 You can also use sets of standard gdb commands stored in a .gdbinit
742 file (see next section).
744 @unnumberedsubsubsec Typical .gdbinit files
746 The behavior of gdb can be readily customized through the use of a
747 @var{.gdbinit} file. A @var{.gdbinit} file is a file named
748 @var{.gdbinit} (notice the @qq{.} at the beginning of the file name)
749 that is placed in a user's home directory.
751 The @var{.gdbinit} file below is from Han-Wen. It sets breakpoints
752 for all errors and defines functions for displaying scheme objects
753 (ps), grobs (pgrob), and parsed music expressions (pmusic).
756 file lily/out/lilypond
758 b Grob::programming_error
761 print ly_display_scm($arg0)
764 print ly_display_scm($arg0->self_scm_)
765 print ly_display_scm($arg0->mutable_property_alist_)
766 print ly_display_scm($arg0->immutable_property_alist_)
767 print ly_display_scm($arg0->object_alist_)
770 print ly_display_scm($arg0->self_scm_)
771 print ly_display_scm($arg0->mutable_property_alist_)
772 print ly_display_scm($arg0->immutable_property_alist_)
776 @node Debugging Scheme code
777 @subsection Debugging Scheme code
779 Scheme code can be developed using the Guile command line
780 interpreter @code{top-repl}. You can either investigate
781 interactively using just Guile or you can use the debugging
782 tools available within Guile.
784 @unnumberedsubsubsec Using Guile interactively with LilyPond
786 In order to experiment with Scheme programming in the LilyPond
787 environment, it is necessary to have a Guile interpreter that
788 has all the LilyPond modules loaded. This requires the following
791 First, define a Scheme symbol for the active module in the .ly file:
794 #(module-define! (resolve-module '(guile-user))
795 'lilypond-module (current-module))
798 Now place a Scheme function in the .ly file that gives an
799 interactive Guile prompt:
805 When the .ly file is compiled, this causes the compilation to be
806 interrupted and an interactive guile prompt to appear. Once the
807 guile prompt appears, the LilyPond active module must be set as the
808 current guile module:
811 guile> (set-current-module lilypond-module)
814 You can demonstrate these commands are operating properly by typing the name
815 of a LilyPond public scheme function to check it has been defined:
818 guile> fret-diagram-verbose-markup
819 #<procedure fret-diagram-verbose-markup (layout props marking-list)>
822 If the LilyPond module has not been correctly loaded, an error
823 message will be generated:
826 guile> fret-diagram-verbose-markup
827 ERROR: Unbound variable: fret-diagram-verbose-markup
828 ABORT: (unbound-variable)
831 Once the module is properly loaded, any valid LilyPond Scheme
832 expression can be entered at the interactive prompt.
834 After the investigation is complete, the interactive guile
835 interpreter can be exited:
841 The compilation of the .ly file will then continue.
843 @unnumberedsubsubsec Using the Guile debugger
845 To set breakpoints and/or enable tracing in Scheme functions, put
848 \include "guile-debugger.ly"
851 in your input file after any scheme procedures you have defined in
852 that file. This will invoke the Guile command-line after having set
853 up the environment for the debug command-line. When your input file
854 is processed, a guile prompt will be displayed. You may now enter
855 commands to set up breakpoints and enable tracing by the Guile debugger.
857 @unnumberedsubsubsec Using breakpoints
859 At the guile prompt, you can set breakpoints with
860 the @code{set-break!} procedure:
863 guile> (set-break! my-scheme-procedure)
866 Once you have set the desired breakpoints, you exit the guile repl frame
873 Then, when one of the scheme routines for which you have set
874 breakpoints is entered, guile will interrupt execution in a debug
875 frame. At this point you will have access to Guile debugging
876 commands. For a listing of these commands, type:
882 Alternatively you may code the breakpoints in your Lilypond source
883 file using a command such as:
886 #(set-break! my-scheme-procedure)
889 immediately after the @code{\include} statement. In this case the
890 breakpoint will be set straight after you enter the @code{(quit)}
891 command at the guile prompt.
893 Embedding breakpoint commands like this is particularly useful if
894 you want to look at how the Scheme procedures in the @var{.scm}
895 files supplied with LilyPond work. To do this, edit the file in
896 the relevant directory to add this line near the top:
899 (use-modules (scm guile-debugger))
902 Now you can set a breakpoint after the procedure you are interested
903 in has been declared. For example, if you are working on routines
904 called by @var{print-book-with} in @var{lily-library.scm}:
907 (define (print-book-with parser book process-procedure)
908 (let* ((paper (ly:parser-lookup parser '$defaultpaper))
909 (layout (ly:parser-lookup parser '$defaultlayout))
910 (outfile-name (get-outfile-name parser)))
911 (process-procedure book paper layout outfile-name)))
913 (define-public (print-book-with-defaults parser book)
914 (print-book-with parser book ly:book-process))
916 (define-public (print-book-with-defaults-as-systems parser book)
917 (print-book-with parser book ly:book-process-to-systems))
921 At this point in the code you could add this to set a breakpoint at
925 (set-break! print-book-with)
928 @unnumberedsubsubsec Tracing procedure calls and evaluator steps
930 Two forms of trace are available:
933 (set-trace-call! my-scheme-procedure)
939 (set-trace-subtree! my-scheme-procedure)
942 @code{set-trace-call!} causes Scheme to log a line to the standard
943 output to show when the procedure is called and when it exits.
945 @code{set-trace-subtree!} traces every step the Scheme evaluator
946 performs in evaluating the procedure.
948 @node Adding or modifying features
949 @section Adding or modifying features
951 When a new feature is to be added to LilyPond, it is necessary to
952 ensure that the feature is properly integrated to maintain
953 its long-term support. This section describes the steps necessary
954 for feature addition and modification.
956 @subsection Write the code
958 You should probably create a new git branch for writing the code, as that
959 will separate it from the master branch and allow you to continue
960 to work on small projects related to master.
962 Please be sure to follow the rules for programming style discussed
963 earlier in this chapter.
965 @subsection Write regression tests
967 In order to demonstrate that the code works properly, you will
968 need to write one or more regression tests. These tests are
969 typically .ly files that are found in input/regression.
971 Regression tests should be as brief as possible to demonstrate the
972 functionality of the code.
974 Regression tests should generally cover one issue per test. Several
975 short, single-issue regression tests are preferred to a single, long,
976 multiple-issue regression test.
978 Use existing regression tests as templates to demonstrate the type of
979 header information that should be included in a regression test.
981 @subsection Write convert-ly rule
983 If the modification changes the input syntax, a convert-ly rule
984 should be written to automatically update input files from older
987 convert-ly rules are found in python/convertrules.py
989 If possible, the convert-ly rule should allow automatic updating
990 of the file. In some cases, this will not be possible, so the
991 rule will simply point out to the user that the feature needs
994 @subsection Automatically update documentation, snippets, and regtests
996 convert-ly should be used to update the documentation, the snippets,
997 and the regression tests. This not only makes the necessary syntax
998 changes, it also tests the convert-ly rules.
1000 The automatic updating is a three step process. First, be sure you
1001 are in the top-level source directory. Then, for the
1005 find Documentation/ -name '*.itely' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
1009 where @var{X.Y.Z} is the version number of the last released development
1012 Next, for the snippets, do:
1015 find Documentation/snippets/ -name '*.ly' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
1018 Finally, for the regression tests, do:
1021 find input/regression/ -name '*.ly' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
1025 @subsection Manually update documentation, snippets, and regtests
1027 Where the convert-ly rule is not able to automatically update the inline
1028 lilypond code in the documentation (i.e. if a NOT_SMART rule is used), the
1029 documentation must be manually updated. The inline snippets that require
1030 changing must be changed in the English version of the docs and all
1031 translated versions. If the inline code is not changed in the
1032 translated documentation, the old snippets will show up in the
1033 English version of the documentation.
1035 Where the convert-ly rule is not able to automatically update snippets
1036 in Documentation/snippets/, those snippets must be manually updated.
1037 Those snippets should be copied to Documentation/snippets/new. The
1038 comments at the top of the snippet describing its automatice generation
1039 should be removed. All translated texidoc strings should be removed.
1040 The comment @qq{% begin verbatim} should be removed. The syntax of
1041 the snippet should then be manually edited.
1043 Where snippets in Documentation/snippets are made obsolete, the snippet
1044 should be copied to Documentation/snippets/new. The comments and
1045 texidoc strings should be removed as described above. Then the body
1046 of the snippet should be changed to:
1050 This snippet is deprecated as of version X.Y.Z and
1051 will be removed from the documentation.
1056 where X.Y.Z is the version number for which the convert-ly rule was
1059 Update the snippet files by running:
1062 scripts/auxiliar/makelsr.py
1065 Where the convert-ly rule is not able to automatically update regression
1066 tests, the regression tests in input/regression should be manually
1069 Although it is not required, it is helpful if the developer
1070 can write relevant material for inclusion in the Notation
1071 Reference. If the developer does not feel qualified to write
1072 the documentation, a documentation editor will be able to
1073 write it from the regression tests. The text that is added to
1074 or removed from the documentation should be changed only in
1075 the English version.
1077 @subsection Edit changes.tely
1079 An entry should be added to Documentation/changes.tely to describe
1080 the feature changes to be implemented. This is especially important
1081 for changes that change input file syntax.
1083 Hints for changes.tely entries are given at the top of the file.
1085 New entries in changes.tely go at the top of the file.
1087 The changes.tely entry should be written to show how the new change
1088 improves LilyPond, if possible.
1090 @subsection Verify successful build
1092 When the changes have been made, successful completion must be
1100 When these commands complete without error, the patch is
1101 considered to function successfully.
1103 Developers on Windows who are unable to build LilyPond should
1104 get help from a Linux or OSX developer to do the make tests.
1106 @subsection Verify regression test
1108 In order to avoid breaking LilyPond, it is important to verify that
1109 the regression tests all succeed. This process is described in
1110 @ref{Regression tests}.
1112 @subsection Post patch for comments
1114 For any change other than a minor change, a patch set should be
1115 posted on @uref{http://codereview.appspot.com/, Rietveld} for comment.
1116 This requires the use of an external package, git-cl, and an email
1119 git-cl is installed by:
1122 git clone git://neugierig.org/git-cl.git
1125 Then, add the git-cl directory to your PATH, or create a
1126 symbolic link to the git-cl and upload.py in one of your
1127 PATH directories (like usr/bin). git-cl is then
1128 configured by entering the command
1135 in the LilyPond git directory and answering the questions that
1136 are asked. If you do not understand the question answer with just
1139 The patch set is posted to Rietveld as follows. Ensure your changes
1140 are committed in a separate branch, which should differ from the
1141 reference branch to be used by just the changes to be uploaded.
1142 If the reference branch is to be origin/master, ensure this is
1143 up-to-date. If necessary, use git rebase to rebase the branch
1144 containing the changes to the head of origin/master. Finally,
1145 check out branch with the changes and enter the command:
1148 git cl upload <reference SHA1 ID>
1152 where <reference SHA1 ID> is the SHA1 ID of the commit to be used
1153 as a reference source for the patch. Generally, this will be the
1154 SHA1 ID of origin/master, and in that case the command
1157 git cl upload origin/master
1163 After prompting for your Google email address and password, the
1164 patch set will be posted to Rietveld.
1166 You should then announce the patch by sending
1167 an email to lilypond-devel, with a subject line
1168 starting with PATCH:, asking for comments on the patch.
1170 As revisions are made in response to comments, successive patch sets
1171 for the same issue can be uploaded by reissuing the git-cl command
1172 with the modified branch checked out.
1174 @subsection Push patch
1176 Once all the comments have been addressed, the patch can be pushed.
1178 If the author has push privileges, the author will push the patch.
1179 Otherwise, a developer with push privileges will push the patch.
1181 @subsection Closing the issues
1183 Once the patch has been pushed, all the relevant issues should be
1186 On Rietveld, the author should log in an close the issue either by
1187 using the @q{Edit Issue} link, or by clicking the circled x icon
1188 to the left of the issue name.
1190 If the changes were in response to a feature request on the Google
1191 issue tracker for LilyPond, the author should change the status to
1192 Fixed and a tag @q{fixed_x_y_z} should be added, where the patch was
1193 fixed in version x.y.z. If
1194 the author does not have privileges to change the status, an email
1195 should be sent to bug-lilypond requesting the BugMeister to change
1198 @node Iterator tutorial
1199 @section Iterator tutorial
1201 FIXME -- this is a placeholder for a tutorial on iterators
1203 Iterators are routines written in C++ that process music expressions
1204 and sent the music events to the appropriate engravers and/or
1207 @node Engraver tutorial
1208 @section Engraver tutorial
1210 FIXME -- This is a placeholder for a tutorial on how engravers work.
1212 Engravers are C++ classes that catch music events and
1213 create the appropriate grobs for display on the page. Though the
1214 majority of engravers are responsible for the creation of a single grob,
1215 in some cases (e.g. @code{New_fingering_engraver}), several different grobs
1218 @subsection Useful methods for information processing
1220 An engraver inherits the following public methods from the Translator
1221 base class, which can be used to process listened events and acknowledged
1225 @item @code{virtual void initialize ()}
1226 @item @code{void start_translation_timestep ()}
1227 @item @code{void process_music ()}
1228 @item @code{void process_acknowledged ()}
1229 @item @code{void stop_translation_timestep ()}
1230 @item @code{virtual void finalize ()}
1233 These methods are listed in order of translation time, with
1234 @code{initialize ()} and @code{finalize ()} bookending the whole
1235 process. @code{initialize ()} can be used for one-time initialization
1236 of context properties before translation starts, whereas
1237 @code{finalize ()} is often used to tie up loose ends at the end of
1238 translation: for example, an unterminated spanner might be completed
1239 automatically or reported with a warning message.
1241 @subsection Translation process
1243 At each timestep in the music, translation proceeds by calling the
1244 following methods in turn:
1246 @code{start_translation_timestep ()} is called before any user information enters
1247 the translators, i.e., no property operations (\set, \override, etc.) or events
1248 have been processed yet.
1250 @code{process_music ()} and @code{process_acknowledged ()} are called after events
1251 have been heard, or grobs have been acknowledged. The latter tends to be used
1252 exclusively with engravers which only acknowledge grobs, whereas the former is
1253 the default method for main processing within engravers.
1255 @code{stop_translation_timestep ()} is called after all user information has been
1256 processed prior to beginning the translation for the next timestep.
1258 @subsection Preventing garbage collection for SCM member variables
1260 In certain cases, an engraver might need to ensure private Scheme variables
1261 (with type SCM) do not get swept away by Guile's garbage collector: for example,
1262 a cache of the previous key signature which must persist persist between timesteps.
1263 The method @code{virtual derived_mark () const} can be used in such cases to mark
1264 such objects as follows:
1267 Engraver_name::derived_mark ()
1269 scm_gc_mark (private_scm_member_)
1274 @subsection Listening to music events
1276 External interfaces to to the engraver are implemented by protected
1277 macros including one or more of the following:
1280 @item @code{DECLARE_TRANSLATOR_LISTENER (event_name)}
1281 @item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)}
1285 where @var{event_name} is the type of event required to provide the
1286 input the engraver needs and @var{Engraver_name} is the name of the
1289 Following declaration of a listener, the method is implemented as follows:
1292 IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)
1294 Engraver_name::listen_event_name (Stream event *event)
1296 ...body of listener method...
1300 @subsection Acknowledging grobs
1302 Some engravers also need information from grobs as they are created
1303 and as they terminate. The mechanism and methods to obtain this
1304 information are set up by the macros:
1307 @item @code{DECLARE_ACKNOWLEDGER (grob_interface)}
1308 @item @code{DECLARE_END_ACKNOWLEDGER (grob_interface)}
1311 where @var{grob_interface} is an interface supported by the
1312 grob(s) which should be acknowledged. For example, the following
1313 code would declare acknowledgers for a @code{NoteHead} grob (via the
1314 @code{note-head-interface}) and any grobs which support the
1315 @code{side-position-interface}:
1318 @code{DECLARE_ACKNOWLEDGER (note_head)}
1319 @code{DECLARE_ACKNOWLEDGER (side_position)}
1322 The @code{DECLARE_END_ACKNOWLEDGER ()} macro sets up a spanner-specific
1323 acknowledger which will be called whenever a spanner ends.
1325 Following declaration of an acknowledger, the method is coded as follows:
1329 Engraver_name::acknowledge_interface_name (Grob_info info)
1331 ...body of acknowledger method...
1335 @subsection Engraver declaration/documentation
1337 An engraver must have a public macro
1340 @item @code{TRANSLATOR_DECLARATIONS (Engraver_name)}
1344 where @code{Engraver_name} is the name of the engraver. This
1345 defines the common variables and methods used by every engraver.
1347 At the end of the engraver file, one or both of the following
1348 macros are generally called to document the engraver in the
1349 Internals Reference:
1352 @item @code{ADD_ACKNOWLEDGER (Engraver_name, grob_interface)}
1353 @item @code{ADD_TRANSLATOR (Engraver_name, Engraver_doc,
1354 Engraver_creates, Engraver_reads, Engraver_writes)}
1358 where @code{Engraver_name} is the name of the engraver, @code{grob_interface}
1359 is the name of the interface that will be acknowledged,
1360 @code{Engraver_doc} is a docstring for the engraver,
1361 @code{Engraver_creates} is the set of grobs created by the engraver,
1362 @code{Engraver_reads} is the set of properties read by the engraver,
1363 and @code{Engraver_writes} is the set of properties written by
1366 @node Callback tutorial
1367 @section Callback tutorial
1369 FIXME -- This is a placeholder for a tutorial on callback functions.
1371 @node LilyPond scoping
1372 @section LilyPond scoping
1374 The Lilypond language has a concept of scoping, ie you can do
1380 (display (+ foo 2)))
1383 @noindent with @code{\paper}, @code{\midi} and @code{\header} being
1384 nested scope inside the @file{.ly} file-level scope. @w{@code{foo = 1}}
1385 is translated in to a scheme variable definition.
1387 This implemented using modules, with each scope being an anonymous
1388 module that imports its enclosing scope's module.
1390 Lilypond's core, loaded from @file{.scm} files, is usually placed in the
1391 @code{lily} module, outside the @file{.ly} level. In the case of
1398 we want to reuse the built-in definitions, without changes effected in
1399 user-level @file{a.ly} leaking into the processing of @file{b.ly}.
1401 The user-accessible definition commands have to take care to avoid
1402 memory leaks that could occur when running multiple files. All
1403 information belonging to user-defined commands and markups is stored in
1404 a manner that allows it to be garbage-collected when the module is
1405 dispersed, either by being stored module-locally, or in weak hash
1408 @node LilyPond miscellany
1409 @section LilyPond miscellany
1411 This is a place to dump information that may be of use to developers
1412 but doesn't yet have a proper home. Ideally, the length of this section
1413 would become zero as items are moved to other homes.
1415 @subsection Info from Han-Wen Email
1417 In 2004, Douglas Linhardt decided to try starting a document that would
1418 explain LilyPond architecture and design principles. The material below
1419 is extracted from that email, which can be found at
1420 @uref{http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/2992}.
1421 The headings reflect questions from Doug or comments from Han-Wen;
1422 the body text are Han-Wen's answers.
1424 @unnumberedsubsubsec Figuring out how things work.
1426 I must admit that when I want to know how a program works, I use grep
1427 and emacs and dive into the source code. The comments and the code
1428 itself are usually more revealing than technical documents.
1430 @unnumberedsubsubsec What's a grob, and how is one used?
1432 Graphical object - they are created from within engravers, either as
1433 Spanners (derived class) -slurs, beams- or Items (also a derived
1434 class) -notes, clefs, etc.
1436 There are two other derived classes System (derived from Spanner,
1437 contaning a "line of music") and Paper_column (derived from Item, it
1438 contains all items that happen at the same moment). They are separate
1439 classes because they play a special role in the linebreaking process.
1441 @unnumberedsubsubsec What's a smob, and how is one used?
1443 A C(++) object that is encapsulated so it can be used as a Scheme
1444 object. See GUILE info, "19.3 Defining New Types (Smobs)"
1446 @unnumberedsubsubsec When is each C++ class constructed and used
1453 In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME().
1458 Constructed during "interpreting" phase.
1463 Executive branch of Contexts, plugins that create grobs, usually one
1464 engraver per grob type. Created together with context.
1474 These are not C++ classes per se. The idea of a Grob interface hasn't
1475 crystallized well. ATM, an interface is a symbol, with a bunch of grob
1476 properties. They are not objects that are created or destroyed.
1481 Objects that walk through different music classes, and deliver events
1482 in a synchronized way, so that notes that play together are processed
1483 at the same moment and (as a result) end up on the same horizontal position.
1485 Created during interpreting phase.
1487 BTW, the entry point for interpreting is ly:run-translator
1488 (ly_run_translator on the C++ side)
1492 @unnumberedsubsubsec Can you get to Context properties from a Music object?
1494 You can create music object with a Scheme function that reads context
1495 properties (the \applycontext syntax). However, that function is
1496 executed during Interpreting, so you can not really get Context
1497 properties from Music objects, since music objects are not directly
1498 connected to Contexts. That connection is made by the Music_iterators
1500 @unnumberedsubsubsec Can you get to Music properties from a Context object?
1502 Yes, if you are given the music object within a Context
1503 object. Normally, the music objects enter Contexts in synchronized
1504 fashion, and the synchronization is done by Music_iterators.
1506 @unnumberedsubsubsec What is the relationship between C++ classes and Scheme objects?
1508 Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are
1509 manipulated from C++ as well using the GUILE C function interface
1512 @unnumberedsubsubsec How do Scheme procedures get called from C++ functions?
1514 scm_call_*, where * is an integer from 0 to 4.
1515 Also scm_c_eval_string (), scm_eval ()
1517 @unnumberedsubsubsec How do C++ functions get called from Scheme procedures?
1519 Export a C++ function to Scheme with LY_DEFINE.
1521 @unnumberedsubsubsec What is the flow of control in the program?
1523 Good question. Things used to be clear-cut, but we have Scheme
1524 and SMOBs now, which means that interactions do not follow a very
1525 rigid format anymore. See below for an overview, though.
1527 @unnumberedsubsubsec Does the parser make Scheme procedure calls or C++ function calls?
1529 Both. And the Scheme calls can call C++ and vice versa. It's nested,
1530 with the SCM datatype as lubrication between the interactions
1532 (I think the word "lubrication" describes the process better than the
1533 traditional word "glue")
1535 @unnumberedsubsubsec How do the front-end and back-end get started?
1537 Front-end: a file is parsed, the rest follows from that. Specifically,
1539 Parsing leads to a Music + Music_output_def object (see parser.yy,
1540 definition of toplevel_expression )
1542 A Music + Music_output_def object leads to a Global_context object (see
1543 ly_run_translator ())
1545 During interpreting, Global_context + Music leads to a bunch of
1546 Contexts. (see Global_translator::run_iterator_on_me () )
1548 After interpreting, Global_context contains a Score_context (which
1549 contains staves, lyrics etc.) as a child. Score_context::get_output ()
1550 spews a Music_output object (either a Paper_score object for notation
1551 or Performance object for MIDI).
1553 The Music_output object is the entry point for the backend. (see
1554 ly_render_output () )
1556 The main steps of the backend itself are in
1561 paper-score.cc , Paper_score::process_
1564 system.cc , System::get_lines()
1567 The step, where things go from grobs to output, is in
1568 System::get_line(): each grob delivers a Stencil (a Device
1569 independent output description), which is interpreted by our
1570 outputting backends (scm/output-tex.scm and scm/output-ps.scm)
1571 to produce TeX and PS.
1575 Interactions between grobs and putting things into .tex and .ps files
1576 have gotten a little more complex lately. Jan has implemented
1577 page-breaking, so now the backend also involves Paper_book,
1578 Paper_lines and other things. This area is still heavily in flux, and
1579 perhaps not something you should want to look at.
1581 @unnumberedsubsubsec How do the front-end and back-end communicate?
1583 There is no communication from backend to front-end. From front-end to
1584 backend is simply the program flow: music + definitions gives
1585 contexts, contexts yield output, after processing, output is written
1588 @unnumberedsubsubsec Where is the functionality associated with KEYWORDs?
1590 See my-lily-lexer.cc (keywords, there aren't that many) and ly/*.ly
1591 (most of the other backslashed \words are identifiers)
1593 @unnumberedsubsubsec What Contexts/Properties/Music/etc. are available when they are processed?
1595 What do you mean exactly with this question?
1597 See ly/engraver-init.ly for contexts, see scm/define-*.scm for other
1600 @unnumberedsubsubsec How do you decide if something is a Music, Context, or Grob property?
1601 Why is part-combine-status a Music property when it seems (IMO)
1602 to be related to the Staff context?
1604 The Music_iterators and Context communicate through two channels
1606 Music_iterators can set and read context properties, idem for
1607 Engravers and Contexts
1609 Music_iterators can send "synthetic" music events (which aren't in
1610 the input) to a context. These are caught by Engravers. This is
1611 mostly a one way communication channel.
1613 part-combine-status is part of such a synthetic event, used by
1614 Part_combine_iterator to communicate with Part_combine_engraver.
1617 @unnumberedsubsubsec I'm adding a property to affect how \autochange works. It seems to
1618 me that it should be a context property, but the Scheme autochange
1619 procecure has a Music argument. Does this mean I should use
1622 \autochange is one of these extra strange beasts: it requires
1623 look-ahead to decide when to change staves. This is achieved by
1624 running the interpreting step twice (see scm/part-combiner.scm , at
1625 the bottom), and storing the result of the first step (where to switch
1626 staves) in a Music property. Since you want to influence that
1627 where-to-switch list, your must affect the code in
1628 make-autochange-music (scm/part-combiner.scm). That code is called
1629 directly from the parser and there are no official "parsing
1630 properties" yet, so there is no generic way to tune \autochange. We
1631 would have to invent something new for this, or add a separate
1635 \autochange #around-central-C ..music..
1639 where around-central-C is some function that is called from
1640 make-autochange-music.
1642 @unnumberedsubsubsec I get lost figuring out what environment the code I'm looking at is in when it executes.
1643 I found both the C++ and Scheme autochange code. Then I was
1644 trying to figure out where the code got called from. I finally figured out that
1645 the Scheme procedure was called before the C++ iterator code, but it took me a
1646 while to figure that out, and I still didn't know who did the calling in the
1647 first place. I only know a little bit about Flex and Bison, so reading those
1648 files helped only a little bit.
1650 @emph{Han-Wen:} GDB can be of help here. Set a breakpoint in C++, and run. When you
1651 hit the breakpoint, do a backtrace. You can inspect Scheme objects
1652 along the way by doing
1655 p ly_display_scm(obj)
1658 this will display OBJ through GUILE.
1660 @subsection Music functions and GUILE debugging
1662 Ian Hulin was trying to do some debugging in music functions, and
1663 came up with the following question
1666 I'm working on the Guile Debugger Stuff, and would like to try
1667 debugging a music function definition such as:
1670 conditionalMark = #(define-music-function (parser location) ()
1671 #@{ \tag #'instrumental-part @{\mark \default@} #@} )
1674 It appears conditionalMark does not get set up as an
1675 equivalent of a Scheme
1678 (define conditionalMark = define-music-function(parser location () ...
1682 although something gets defined because Scheme apparently recognizes
1685 #(set-break! conditionalMark)
1689 later on in the file without signalling any Guile errors.
1691 However the breakpoint trap is never encountered as
1692 define-music-function passed things on to ly:make-music-function,
1693 which is really C++ code ly_make_music_function, so Guile never
1694 finds out about the breakpoint.
1696 Han-Wen answered as follows:
1698 You can see the defintion by doing
1701 #(display conditionalMark)
1705 inside the .ly file.
1707 The breakpoint failing may have to do with the call sequence. See
1708 parser.yy, run_music_function(). The function is called directly from
1709 C++, without going through the GUILE evaluator, so I think that is why
1710 there is no debugger trap.