1 @c -*- coding: us-ascii; mode: texinfo; -*-
3 @chapter Programming work
6 * Overview of LilyPond architecture::
7 * LilyPond programming languages::
8 * Programming without compiling::
11 * Debugging LilyPond::
12 * Adding or modifying features::
15 @node Overview of LilyPond architecture
16 @section Overview of LilyPond architecture
18 LilyPond processes the input file into graphical and musical output in a
19 number of stages. This process, along with the types of routines that
20 accomplish the various stages of the process, is described in this section. A
21 more complete description of the LilyPond architecture and internal program
22 execution is found in Erik Sandberg's
23 @uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's
27 The first stage of LilyPond processing is @emph{parsing}. In the parsing
28 process, music expressions in LilyPond input format are converted to music
29 expressions in Scheme format. In Scheme format, a music expression is a list
30 in tree form, with nodes that indicate the relationships between various music
31 events. The LilyPond parser is written in Bison.
33 The second stage of LilyPond processing is @emph{iterating}. Iterating
34 assigns each music event to a context, which is the environment in which the
35 music will be finally engraved. The context is responsible for all further
36 processing of the music. It is during the iteration stage that contexts are
37 created as necessary to ensure that every note has a Voice type context (e.g.
38 Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice,
39 GregorianTranscriptionVoice), that the Voice type contexts exist in
40 appropriate Staff type contexts, and that parallel Staff type contexts exist
41 in StaffGroup type contexts. In addition, during the iteration stage each
42 music event is assigned a moment, or a time in the music when the event
45 Each type of music event has an associated iterator. Iterators are defined in
46 *-iterator.cc. During iteration, an
47 event's iterator is called to deliver that music event to the appropriate
50 The final stage of LilyPond processing is @emph{translation}. During
51 translation, music events are prepared for graphical or midi output. The
52 translation step is accomplished by translators or engravers (the distinction
55 Translators are defined in C++ files named *-engraver.cc. In *-engraver.cc, a
56 C++ class of Engraver type is created. The Engraver is also declared as a
57 translator. Much of the work of translating is handled by Scheme functions,
58 which is one of the keys to LilyPond's exceptional flexibility.
61 @node LilyPond programming languages
62 @section LilyPond programming languages
64 Programming in LilyPond is done in a variety of programming languages. Each
65 language is used for a specific purpose or purposes. This section describes
66 the languages used and provides links to reference manuals and tutorials for
67 the relevant language.
71 The core functionality of LilyPond is implemented in C++.
73 C++ is so ubiquitous that it is difficult to identify either a reference
74 manual or a tutorial. Programmers unfamiliar with C++ will need to spend some
75 time to learn the language before attempting to modify the C++ code.
77 The C++ code calls Scheme/GUILE through the GUILE interface, which is
79 @uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE
84 The LilyPond parser is implemented in Bison, a GNU parser generator. The
85 Bison homepage is found at @uref{http://www.gnu.org/software/bison/,
86 gnu.org}. The manual (which includes both a reference and tutorial) is
87 @uref{http://www.gnu.org/software/bison/manual/index.html, available} in a
92 GNU Make is used to control the compiling process and to build the
93 documentation and the website. GNU Make documentation is available at
94 @uref{http://www.gnu.org/software/make/manual/, the GNU website}.
96 @subsection GUILE or Scheme
98 GUILE is the dialect of Scheme that is used as LilyPond's extension language. Many extensions to LilyPond are written entirely in GUILE. The
99 @uref{http://www.gnu.org/software/guile/manual/html_node/index.html,
100 GUILE Reference Manual} is available online.
102 @uref{http://mitpress.mit.edu/sicp/full-text/book/book.html, Structure and
103 Interpretation of Computer Programs}, a popular textbook used to teach
104 programming in Scheme is available in its entirety online.
106 An introduction to Guile/Scheme as used in LilyPond can be found in the
107 Learning Manual, see @rlearning{Scheme tutorial}.
111 MetaFont is used to create the music fonts used by LilyPond. A MetaFont
112 tutorial is available at @uref{http://metafont.tutorial.free.fr/, the
113 METAFONT tutorial page}.
115 @subsection PostScript
117 PostScript is used to generate graphical output. A brief PostScript tutorial
118 is @uref{http://local.wasp.uwa.edu.au/~pbourke/dataformats/postscript/,
119 available online}. The
120 @uref{http://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf, PostScript Lanugage
121 Reference} is available online in PDF format.
125 Python is used for XML2ly and is used for buillding the documentation and the
128 Python documentation is available at @uref{http://www.python.org/doc/,
131 @node Programming without compiling
132 @section Programming without compiling
134 Much of the development work in LilyPond takes place by changing *.ly or
135 *.scm files. These changes can be made without compiling LilyPond. Such
136 changes are described in this section.
139 @subsection Modifying distribution files
141 Much of LilyPond is written in Scheme or LilyPond input files. These
142 files are interpreted when the program is run, rather than being compiled
143 when the program is built, and are present in all LilyPond distributions.
144 You will find .ly files in the ly/ directory and the Scheme files in the
145 scm/ directory. Both Scheme files and .ly files can be modified and
146 saved with any text editor. It's probably wise to make a backup copy of
147 your files before you modify them, although you can reinstall if the
148 files become corrupted.
150 Once you've modified the files, you can test the changes just by running
151 LilyPond on some input file. It's a good idea to create a file that
152 demonstrates the feature you're trying to add. This file will eventually
153 become a regression test and will be part of the LilyPond distribution.
155 @subsection Desired file formatting
157 Files that are part of the LilyPond distribution have Unix-style line
158 endings (LF), rather than DOS (CR+LF) or MacOS 9 and earlier (CR). Make
159 sure you use the necessary tools to ensure that Unix-style line endings are
160 preserved in the patches you create.
162 Tab characters should not be included in files for distribution. All
163 indentation should be done with spaces. Most editors have settings to
164 allow the setting of tab stops and ensuring that no tab characters are
165 included in the file.
167 Scheme files and LilyPond files should be written according to standard
168 style guidelines. Scheme file guidelines can be found at
169 @uref{http://community.schemewiki.org/?scheme-style}. Following these
170 guidelines will make your code easier to read. Both you and others that
171 work on your code will be glad you followed these guidelines.
173 For LilyPond files, you should follow the guidelines for LilyPond snippets
174 in the documentation. You can find these guidelines at
175 @ref{Texinfo introduction and usage policy}.
177 @node Finding functions
178 @section Finding functions
180 When making changes or fixing bugs in LilyPond, one of the initial
181 challenges is finding out where in the code tree the functions to
182 be modified live. With nearly 3000 files in the source tree,
183 trial-and-error searching is generally ineffective. This section
184 describes a process for finding interesting code.
186 @subsection Using the ROADMAP
188 The file ROADMAP is located in the main directory of the lilypond source.
189 ROADMAP lists all of the directories in the LilPond source tree, along
190 with a brief description of the kind of files found in each directory.
191 This can be a very helpful tool for deciding which directories to search
192 when looking for a function.
195 @subsection Using grep to search
197 Having identified a likely subdirectory to search, the grep utility can
198 be used to search for a function name. The format of the grep command is
201 grep -i functionName subdirectory/*
204 This command will search all the contents of the directory subdirectory/
205 and display every line in any of the files that contains
206 functionName. The @code{-i} option makes @command{grep} ignore
207 case -- this can be very useful if you are not yet familiar with
208 our capitalization conventions.
210 The most likely directories to grep for function names are scm/ for
211 scheme files, ly/ for lilypond input (*.ly) files, and lily/ for C++
215 @subsection Using git grep to search
217 If you have used git to obtain the source, you have access to a
218 powerful tool to search for functions. The command:
221 git grep functionName
224 will search through all of the files that are present in the git
225 repository looking for functionName. It also presents the results
226 of the search using @code{less}, so the results are displayed one page
229 @subsection Searching on the git repository at Savannah
231 You can also use the equivalent of git grep on the Savannah server.
236 Go to http://git.sv.gnu.org/gitweb/?p=lilypond.git
239 In the pulldown box that says commit, select grep.
242 Type functionName in the search box, and hit enter/return
246 This will initiate a search of the remote git repository.
251 @c email to wl@gnu.org when I get here.
253 @warning{this is pending some confirmation on -devel. July 2009 -gp}
255 Command-line script to format stuff with emacs:
259 emacs $1 -batch --eval '(indent-region (point-min) (point-max) nil)' -f save-buffer
262 (that's all on one line)
264 Save it as a shell script, then run on the file(s) you modified.
269 @subsection Handling errors
271 As a general rule, you should always try to continue computations,
272 even if there is some kind of error. When the program stops, it
273 is often very hard for a user to pinpoint what part of the input
274 causes an error. Finding the culprit is much easier if there is
275 some viewable output.
277 So functions and methods do not return errorcodes, they never
278 crash, but report a programming_error and try to carry on.
280 @subsection Languages
282 C++ and Python are preferred. Python code should use PEP 8.
284 @subsection Filenames
286 Definitions of classes that are only accessed via pointers (*) or
287 references (&) shall not be included as include files.
293 ".cc" Implementation files
294 ".icc" Inline definition files
295 ".tcc" non inline Template defs
299 (setq auto-mode-alist
300 (append '(("\\.make$" . makefile-mode)
301 ("\\.cc$" . c++-mode)
302 ("\\.icc$" . c++-mode)
303 ("\\.tcc$" . c++-mode)
304 ("\\.hh$" . c++-mode)
305 ("\\.pod$" . text-mode)
310 The class Class_name is coded in @q{class-name.*}
312 @subsection Indentation
314 Standard GNU coding style is used. In emacs:
317 (add-hook 'c++-mode-hook
318 '(lambda() (c-set-style "gnu")
322 If you like using font-lock, you can also add this to your
326 (setq font-lock-maximum-decoration t)
327 (setq c++-font-lock-keywords-3
329 c++-font-lock-keywords-3
330 '(("\\b\\(a-zA-Z_?+_\\)\\b" 1 font-lock-variable-name-face) ("\\b\\(A-Z?+a-z_?+\\)\\b" 1 font-lock-type-face))
335 @subsection Classes and Types
344 Member variable names end with an underscore:
353 Macro names should be written in uppercase completely.
356 @subsection Broken code
358 Do not write broken code. This includes hardwired dependencies,
359 hardwired constants, slow algorithms and obvious limitations. If
360 you can not avoid it, mark the place clearly, and add a comment
361 explaining shortcomings of the code.
363 We reject broken-in-advance on principle.
370 Messages need to follow Localization.
373 @subsection Localization
375 This document provides some guidelines for programmers write user
376 messages. To help translations, user messages must follow
377 uniform conventions. Follow these rules when coding for LilyPond.
378 Hopefully, this can be replaced by general GNU guidelines in the
379 future. Even better would be to have an English (en_BR, en_AM)
380 guide helping programmers writing consistent messages for all GNU
383 Non-preferred messages are marked with `+'. By convention,
384 ungrammatical examples are marked with `*'. However, such ungrammatical
385 examples may still be preferred.
390 Every message to the user should be localized (and thus be marked
391 for localization). This includes warning and error messages.
394 Don't localize/gettextify:
398 `programming_error ()'s
401 `programming_warning ()'s
407 output strings (PostScript, TeX, etc.)
412 Messages to be localised must be encapsulated in `_ (STRING)' or
413 `_f (FORMAT, ...)'. E.g.:
416 warning (_ ("need music in a score"));
417 error (_f ("cannot open file: `%s'", file_name));
420 In some rare cases you may need to call `gettext ()' by hand. This
421 happens when you pre-define (a list of) string constants for later
422 use. In that case, you'll probably also need to mark these string
423 constants for translation, using `_i (STRING)'. The `_i' macro is
424 a no-op, it only serves as a marker for `xgettext'.
427 char const* messages[] = @{
428 _i ("enable debugging output"),
429 _i ("ignore lilypond version"),
436 puts (gettext (messages i));
440 See also `flower/getopt-long.cc' and `lily/main.cc'.
443 Do not use leading or trailing whitespace in messages. If you need
444 whitespace to be printed, prepend or append it to the translated
448 message ("Calculating line breaks..." + " ");
452 Error or warning messages displayed with a file name and line
453 number never start with a capital, eg,
456 foo.ly: 12: not a duration: 3
459 Messages containing a final verb, or a gerund (`-ing'-form) always
460 start with a capital. Other (simpler) messages start with a
466 Not declaring: `foo'.
470 Avoid abbreviations or short forms, use `cannot' and `do not'
471 rather than `can't' or `don't'
472 To avoid having a number of different messages for the same
473 situation, well will use quoting like this `"message: `%s'"' for all
474 strings. Numbers are not quoted:
477 _f ("cannot open file: `%s'", name_str)
478 _f ("cannot find character number: %d", i)
482 Think about translation issues. In a lot of cases, it is better to
483 translate a whole message. The english grammar must not be imposed
484 on the translator. So, instead of
487 stem at + moment.str () + does not fit in beam
493 _f ("stem at %s does not fit in beam", moment.str ())
497 Split up multi-sentence messages, whenever possible. Instead of
500 warning (_f ("out of tune! Can't find: `%s'", "Key_engraver"));
501 warning (_f ("cannot find font `%s', loading default", font_name));
507 warning (_ ("out of tune:"));
508 warning (_f ("cannot find: `%s', "Key_engraver"));
509 warning (_f ("cannot find font: `%s', font_name));
510 warning (_f ("Loading default font"));
514 If you must have multiple-sentence messages, use full punctuation.
515 Use two spaces after end of sentence punctuation. No punctuation
516 (esp. period) is used at the end of simple messages.
519 _f ("Non-matching braces in text `%s', adding braces", text)
520 _ ("Debug output disabled. Compiled with NPRINT.")
521 _f ("Huh? Not a Request: `%s'. Ignoring.", request)
525 Do not modularise too much; words frequently cannot be translated
526 without context. It is probably safe to treat most occurences of
527 words like stem, beam, crescendo as separately translatable words.
530 When translating, it is preferable to put interesting information
531 at the end of the message, rather than embedded in the middle.
532 This especially applies to frequently used messages, even if this
533 would mean sacrificing a bit of eloquency. This holds for original
534 messages too, of course.
537 en: cannot open: `foo.ly'
538 + nl: kan `foo.ly' niet openen (1)
539 kan niet openen: `foo.ly'* (2)
540 niet te openen: `foo.ly'* (3)
544 The first nl message, although grammatically and stylistically
545 correct, is not friendly for parsing by humans (even if they speak
546 dutch). I guess we would prefer something like (2) or (3).
549 Do not run make po/po-update with GNU gettext < 0.10.35
555 @node Debugging LilyPond
556 @section Debugging LilyPond
558 The most commonly used tool for debugging LilyPond is the GNU debugger
559 gdb. Use of gdb is described in this section.
561 @subsection Debugging overview
563 Using a debugger simplifies troubleshooting in at least two ways.
565 First, breakpoints can be set to pause execution at any desired point.
566 Then, when execution has paused, debugger commands can be issued to
567 explore the values of various variables or to execute functions.
569 Second, the debugger allows the display of a stack trace, which shows
570 the sequence in which functions are called and the arguments to the
571 various function calls.
574 @subsection Compiling with debugging information
576 In order to use a debugger with LilyPond, it is necessary to compile
577 LilyPond with debugging information. This is accomplished by running
578 the following commands in the main LilyPond source directory.
581 ./configure --disable-optimising
586 This will create a version of LilyPond that contains the debugging
587 information that will allow the debugger to tie the source code
588 to the compiled code.
590 You should not do @var{make install} if you want to use a debugger
591 with LilyPond. @var{make install} will strip the debugging information
592 from the LilyPond binary.
594 To set breakpoints in Scheme functions, put
597 \include "guile-debugger.ly"
600 in your input file after any scheme procedures you have defined in
601 that file. When your input file is processed, a guile prompt
602 will be displayed. At the guile prompt, you can set breakpoints with
603 the @code{break!} procedure:
606 guile> (break! my-scheme-procedure)
609 Once you have set the desired breakpoints, you exit the guile repl frame
616 When one of the scheme routines for which you have set breakpoints is
617 entered, guile will interrupt execution in a debug frame. At this point,
618 you will have access to guile debugging commands. For a listing of these
625 @subsection Typical gdb usage
627 @subsection Typical .gdbinit files
629 The behavior of gdb can be readily customized through the use of
630 @var{.gdbinit} files. A @var{.gdbinit} file is a file named
631 @var{.gdbinit} (notice the @qq{.} at the beginning of the file name)
632 that is placed in a user's home directory.
634 The @var{.gdbinit} file below is from Han-Wen. It sets breakpoints
635 for all errors and defines functions for displaying scheme objects
636 (ps), grobs (pgrob), and parsed music expressions (pmusic).
639 file lily/out/lilypond
641 b Grob::programming_error
644 print ly_display_scm($arg0)
647 print ly_display_scm($arg0->self_scm_)
648 print ly_display_scm($arg0->mutable_property_alist_)
649 print ly_display_scm($arg0->immutable_property_alist_)
650 print ly_display_scm($arg0->object_alist_)
653 print ly_display_scm($arg0->self_scm_)
654 print ly_display_scm($arg0->mutable_property_alist_)
655 print ly_display_scm($arg0->immutable_property_alist_)
659 @subsection Using Guile interactively with LilyPond
661 In order to experiment with Scheme programming in the LilyPond
662 environment, it is convenient to have a Guile interpreter that
663 has all the LilyPond modules loaded. This requires the following
666 First, define a Scheme symbol for the active module
670 #(module-define! (resolve-module '(guile-user))
671 'lilypond-module (current-module))
674 Second, place a Scheme function in the .ly file that gives an interactive Guile
681 When the .ly file is compiled, this causes the compilation to be interrupted
682 and an interactive guile prompt to appear. When the guile prompt appears,
683 the LilyPond active module must be set as the current guile module:
686 guile> (set-current-module lilypond-module)
689 Proper operation of these commands can be demonstrated by typing the name
690 of a LilyPond public scheme function to see if it's properly defined:
693 guile> fret-diagram-verbose-markup
694 #<procedure fret-diagram-verbose-markup (layout props marking-list)>
697 If the LilyPond module has not been correctly loaded, an error
698 message will be generated:
701 guile> fret-diagram-verbose-markup
702 ERROR: Unbound variable: fret-diagram-verbose-markup
703 ABORT: (unbound-variable)
706 Once the module is properly loaded, any valid LilyPond Scheme expression
707 can be entered at the interactive prompt.
709 After the investigation is complete, the interactive guile interpreter
716 The compilation of the .ly file will then continue.
718 @node Adding or modifying features
719 @section Adding or modifying features
721 When a new feature is to be added to LilyPond, it is necessary to
722 ensure that the feature is properly integrated to maintain
723 its long-term support. This section describes the steps necessary
724 for feature addition.
726 @subsection Write the code
728 You should create a new git branch for writing the code, as that
729 will separate it from the master branch and allow you to continue
730 to work on small projects related to master.
732 Please be sure to follow the rules for programming style discussed
733 earlier in this chapter.
735 @subsection Write regression tests
737 In order to demonstrate that the code works properly, you will
738 need to write one or more regression tests. These tests are
739 typically .ly files that are found in input/regression.
741 Regression tests should be as brief as possible to demonstrate the
742 functionality of the code.
744 Regression tests should generally cover one issue per test. Several
745 short, single-issue regression tests are preferred to a single, long,
746 multiple-issue regression test.
748 Use existing regression tests as templates to demonstrate the type of
749 header information that should be included in a regression test.
751 @subsection Write documentation
753 Although it is not required, it is helpful if the developer can
754 write relevant material for inclusion in the Notation Reference.
755 If the developer does not feel qualified to write the documentation,
756 a documentation editor will be able to write it from the regression
759 If the modification changes the input syntax so that inline snippets in
760 the documentation need to be changed, you will need to change the
761 snippets in both the english version of the documentation and any
762 translated versions. If you do not change the snippets in all
763 translations, older versions of the snippet may be included
764 when the documentation is built.
766 If lsr snippets need to be changed, the snippet should be copied to
767 @file{Documentation/snippets/new} and modified there. The portions of
768 the snippet that are added by makelsr.py should be removed. The changed
769 snippet will then be included in all versions of the documentation.
771 If non-snippet text is changed in the english documentation, no
772 corresponding changes should be made in the translated documentation.
774 @subsection Write convert-ly rule
776 If the modification changes the input syntax, a convert-ly rule
777 should be written to automatically update input files from older
780 convert-ly rules are found in python/convertrules.py
782 If possible, the convert-ly rule should allow automatic updating
783 of the file. In some cases, this will not be possible, so the
784 rule will simply point out to the user that the feature needs
787 @subsection Write NEWS entry
789 An entry should be added to the NEWS file to describe the feature
790 changes to be implemented. This is especially important for changes
791 that change input file syntax.
793 Hints for NEWS file entries are given at the top of the NEWS file.
795 New entries in NEWS go at the top of the file.
797 The NEWS entry should be written to show how the new change
798 improves LilyPond, if possible.
800 @subsection Verify regression test
802 In order to avoid breaking LilyPond, it is important to verify that
803 the regression tests all succeed. This process is described in
804 @ref{Regression tests}.
806 @subsection Post patch for comments
808 For any change other than a minor change, a patch set should be
809 posted on Rietveld for comment.
811 The patch set is posted by issuing the following command, after
812 first committing all changes:
815 git-cl upload <reference SHA1 ID>
819 where <reference SHA1 ID> is the SHA1 ID of the commit to be used
820 as a reference source for the patch (generally, this will be the
821 SHA1 ID of origin/master).
823 After prompting for an email and a password, the patch set will be
826 An email should then be sent to lilypond-devel, with a subject line
827 starting with PATCH:, asking for comments on the patch.
829 As revisions are made in response to comments, successive patch sets
830 for the same issue can be uploaded by reissuing the git-cl command.
832 @subsection Push patch
834 Once all the comments have been addressed, the patch can be pushed.
836 If the author has push privileges, the author will push the patch.
837 Otherwise, a developer with push privileges will push the patch.
839 @subsection Closing the issues
841 Once the patch has been pushed, all the relevant issues should be
844 On Rietveld, the author should log in an close the issue either by
845 using the @q{Edit Issue} link, or by clicking the circled x icon
846 to the left of the issue name.
848 If the changes were in response to a feature request on the Google
849 issue tracker for LilyPond, the author should change the status to
850 @q{Fixed_x_y_z} where the patch was fixed in version x.y.z. If
851 the author does not have privileges to change the status, an email
852 should be sent to bug-lilypond requesting the BugMeister to change