@c -*- coding: us-ascii; mode: texinfo; -*- @node Programming work @chapter Programming work @menu * Overview of LilyPond architecture:: * LilyPond programming languages:: * Programming without compiling:: * Finding functions:: * Code style:: * Debugging LilyPond:: * Adding or modifying features:: @end menu @node Overview of LilyPond architecture @section Overview of LilyPond architecture LilyPond processes the input file into graphical and musical output in a number of stages. This process, along with the types of routines that accomplish the various stages of the process, is described in this section. A more complete description of the LilyPond architecture and internal program execution is found in Erik Sandberg's @uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's thesis}. The first stage of LilyPond processing is @emph{parsing}. In the parsing process, music expressions in LilyPond input format are converted to music expressions in Scheme format. In Scheme format, a music expression is a list in tree form, with nodes that indicate the relationships between various music events. The LilyPond parser is written in Bison. The second stage of LilyPond processing is @emph{iterating}. Iterating assigns each music event to a context, which is the environment in which the music will be finally engraved. The context is responsible for all further processing of the music. It is during the iteration stage that contexts are created as necessary to ensure that every note has a Voice type context (e.g. Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice, GregorianTranscriptionVoice), that the Voice type contexts exist in appropriate Staff type contexts, and that parallel Staff type contexts exist in StaffGroup type contexts. In addition, during the iteration stage each music event is assigned a moment, or a time in the music when the event begins. Each type of music event has an associated iterator. Iterators are defined in *-iterator.cc. During iteration, an event's iterator is called to deliver that music event to the appropriate context(s). The final stage of LilyPond processing is @emph{translation}. During translation, music events are prepared for graphical or midi output. The translation step is accomplished by translators or engravers (the distinction is unclear). Translators are defined in C++ files named *-engraver.cc. In *-engraver.cc, a C++ class of Engraver type is created. The Engraver is also declared as a translator. Much of the work of translating is handled by Scheme functions, which is one of the keys to LilyPond's exceptional flexibility. @node LilyPond programming languages @section LilyPond programming languages Programming in LilyPond is done in a variety of programming languages. Each language is used for a specific purpose or purposes. This section describes the languages used and provides links to reference manuals and tutorials for the relevant language. @subsection C++ The core functionality of LilyPond is implemented in C++. C++ is so ubiquitous that it is difficult to identify either a reference manual or a tutorial. Programmers unfamiliar with C++ will need to spend some time to learn the language before attempting to modify the C++ code. The C++ code calls Scheme/GUILE through the GUILE interface, which is documented in the @uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE Reference Manual}. @subsection GNU Bison The LilyPond parser is implemented in Bison, a GNU parser generator. The Bison homepage is found at @uref{http://www.gnu.org/software/bison/, gnu.org}. The manual (which includes both a reference and tutorial) is @uref{http://www.gnu.org/software/bison/manual/index.html, available} in a variety of formats. @subsection GNU Make GNU Make is used to control the compiling process and to build the documentation and the website. GNU Make documentation is available at @uref{http://www.gnu.org/software/make/manual/, the GNU website}. @subsection GUILE or Scheme GUILE is the dialect of Scheme that is used as LilyPond's extension language. Many extensions to LilyPond are written entirely in GUILE. The @uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE Reference Manual} is available online. @uref{http://mitpress.mit.edu/sicp/full-text/book/book.html, Structure and Interpretation of Computer Programs}, a popular textbook used to teach programming in Scheme is available in its entirety online. An introduction to Guile/Scheme as used in LilyPond can be found in the Learning Manual, see @rlearning{Scheme tutorial}. @subsection MetaFont MetaFont is used to create the music fonts used by LilyPond. A MetaFont tutorial is available at @uref{http://metafont.tutorial.free.fr/, the METAFONT tutorial page}. @subsection PostScript PostScript is used to generate graphical output. A brief PostScript tutorial is @uref{http://local.wasp.uwa.edu.au/~pbourke/dataformats/postscript/, available online}. The @uref{http://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf, PostScript Lanugage Reference} is available online in PDF format. @subsection Python Python is used for XML2ly and is used for buillding the documentation and the website. Python documentation is available at @uref{http://www.python.org/doc/, python.org}. @node Programming without compiling @section Programming without compiling Much of the development work in LilyPond takes place by changing *.ly or *.scm files. These changes can be made without compiling LilyPond. Such changes are described in this section. @subsection Modifying distribution files Much of LilyPond is written in Scheme or LilyPond input files. These files are interpreted when the program is run, rather than being compiled when the program is built, and are present in all LilyPond distributions. You will find .ly files in the ly/ directory and the Scheme files in the scm/ directory. Both Scheme files and .ly files can be modified and saved with any text editor. It's probably wise to make a backup copy of your files before you modify them, although you can reinstall if the files become corrupted. Once you've modified the files, you can test the changes just by running LilyPond on some input file. It's a good idea to create a file that demonstrates the feature you're trying to add. This file will eventually become a regression test and will be part of the LilyPond distribution. @subsection Desired file formatting Files that are part of the LilyPond distribution have Unix-style line endings (LF), rather than DOS (CR+LF) or MacOS 9 and earlier (CR). Make sure you use the necessary tools to ensure that Unix-style line endings are preserved in the patches you create. Tab characters should not be included in files for distribution. All indentation should be done with spaces. Most editors have settings to allow the setting of tab stops and ensuring that no tab characters are included in the file. Scheme files and LilyPond files should be written according to standard style guidelines. Scheme file guidelines can be found at @uref{http://community.schemewiki.org/?scheme-style}. Following these guidelines will make your code easier to read. Both you and others that work on your code will be glad you followed these guidelines. For LilyPond files, you should follow the guidelines for LilyPond snippets in the documentation. You can find these guidelines at @ref{Texinfo introduction and usage policy}. @node Finding functions @section Finding functions When making changes or fixing bugs in LilyPond, one of the initial challenges is finding out where in the code tree the functions to be modified live. With nearly 3000 files in the source tree, trial-and-error searching is generally ineffective. This section describes a process for finding interesting code. @subsection Using the ROADMAP The file ROADMAP is located in the main directory of the lilypond source. ROADMAP lists all of the directories in the LilPond source tree, along with a brief description of the kind of files found in each directory. This can be a very helpful tool for deciding which directories to search when looking for a function. @subsection Using grep to search Having identified a likely subdirectory to search, the grep utility can be used to search for a function name. The format of the grep command is @example grep -i functionName subdirectory/* @end example This command will search all the contents of the directory subdirectory/ and display every line in any of the files that contains functionName. The @code{-i} option makes @command{grep} ignore case -- this can be very useful if you are not yet familiar with our capitalization conventions. The most likely directories to grep for function names are scm/ for scheme files, ly/ for lilypond input (*.ly) files, and lily/ for C++ files. @subsection Using git grep to search If you have used git to obtain the source, you have access to a powerful tool to search for functions. The command: @example git grep functionName @end example will search through all of the files that are present in the git repository looking for functionName. It also presents the results of the search using @code{less}, so the results are displayed one page at a time. @subsection Searching on the git repository at Savannah You can also use the equivalent of git grep on the Savannah server. @itemize @item Go to http://git.sv.gnu.org/gitweb/?p=lilypond.git @item In the pulldown box that says commit, select grep. @item Type functionName in the search box, and hit enter/return @end itemize This will initiate a search of the remote git repository. @node Code style @section Code style @c email to wl@gnu.org when I get here. @menu @end menu @subsection Handling errors As a general rule, you should always try to continue computations, even if there is some kind of error. When the program stops, it is often very hard for a user to pinpoint what part of the input causes an error. Finding the culprit is much easier if there is some viewable output. So functions and methods do not return errorcodes, they never crash, but report a programming_error and try to carry on. @subsection Languages C++ and Python are preferred. Python code should use PEP 8. @subsection Filenames Definitions of classes that are only accessed via pointers (*) or references (&) shall not be included as include files. @verbatim filenames ".hh" Include files ".cc" Implementation files ".icc" Inline definition files ".tcc" non inline Template defs in emacs: (setq auto-mode-alist (append '(("\\.make$" . makefile-mode) ("\\.cc$" . c++-mode) ("\\.icc$" . c++-mode) ("\\.tcc$" . c++-mode) ("\\.hh$" . c++-mode) ("\\.pod$" . text-mode) ) auto-mode-alist)) @end verbatim The class Class_name is coded in @q{class-name.*} @subsection Indentation Standard GNU coding style is used. In emacs: @verbatim (add-hook 'c++-mode-hook '(lambda() (c-set-style "gnu") )) @end verbatim If you like using font-lock, you can also add this to your @q{.emacs}: @verbatim (setq font-lock-maximum-decoration t) (setq c++-font-lock-keywords-3 (append c++-font-lock-keywords-3 '(("\\b\\(a-zA-Z_?+_\\)\\b" 1 font-lock-variable-name-face) ("\\b\\(A-Z?+a-z_?+\\)\\b" 1 font-lock-type-face)) )) @end verbatim @subsection Classes and Types @verbatim This_is_a_class @end verbatim @subsection Members Member variable names end with an underscore: @verbatim Type Class::member_ @end verbatim @subsection Macros Macro names should be written in uppercase completely. @subsection Broken code Do not write broken code. This includes hardwired dependencies, hardwired constants, slow algorithms and obvious limitations. If you can not avoid it, mark the place clearly, and add a comment explaining shortcomings of the code. We reject broken-in-advance on principle. @subsection Naming @subsection Messages Messages need to follow Localization. @subsection Localization This document provides some guidelines for programmers write user messages. To help translations, user messages must follow uniform conventions. Follow these rules when coding for LilyPond. Hopefully, this can be replaced by general GNU guidelines in the future. Even better would be to have an English (en_BR, en_AM) guide helping programmers writing consistent messages for all GNU programs. Non-preferred messages are marked with `+'. By convention, ungrammatical examples are marked with `*'. However, such ungrammatical examples may still be preferred. @itemize @item Every message to the user should be localized (and thus be marked for localization). This includes warning and error messages. @item Don't localize/gettextify: @itemize @item `programming_error ()'s @item `programming_warning ()'s @item debug strings @item output strings (PostScript, TeX, etc.) @end itemize @item Messages to be localised must be encapsulated in `_ (STRING)' or `_f (FORMAT, ...)'. E.g.: @example warning (_ ("need music in a score")); error (_f ("cannot open file: `%s'", file_name)); @end example In some rare cases you may need to call `gettext ()' by hand. This happens when you pre-define (a list of) string constants for later use. In that case, you'll probably also need to mark these string constants for translation, using `_i (STRING)'. The `_i' macro is a no-op, it only serves as a marker for `xgettext'. @example char const* messages[] = @{ _i ("enable debugging output"), _i ("ignore lilypond version"), 0 @}; void foo (int i) @{ puts (gettext (messages i)); @} @end example See also `flower/getopt-long.cc' and `lily/main.cc'. @item Do not use leading or trailing whitespace in messages. If you need whitespace to be printed, prepend or append it to the translated message @example message ("Calculating line breaks..." + " "); @end example @item Error or warning messages displayed with a file name and line number never start with a capital, eg, @example foo.ly: 12: not a duration: 3 @end example Messages containing a final verb, or a gerund (`-ing'-form) always start with a capital. Other (simpler) messages start with a lowercase letter @example Processing foo.ly... `foo': not declared. Not declaring: `foo'. @end example @item Avoid abbreviations or short forms, use `cannot' and `do not' rather than `can't' or `don't' To avoid having a number of different messages for the same situation, well will use quoting like this `"message: `%s'"' for all strings. Numbers are not quoted: @example _f ("cannot open file: `%s'", name_str) _f ("cannot find character number: %d", i) @end example @item Think about translation issues. In a lot of cases, it is better to translate a whole message. The english grammar must not be imposed on the translator. So, instead of @example stem at + moment.str () + does not fit in beam @end example have @example _f ("stem at %s does not fit in beam", moment.str ()) @end example @item Split up multi-sentence messages, whenever possible. Instead of @example warning (_f ("out of tune! Can't find: `%s'", "Key_engraver")); warning (_f ("cannot find font `%s', loading default", font_name)); @end example rather say: @example warning (_ ("out of tune:")); warning (_f ("cannot find: `%s', "Key_engraver")); warning (_f ("cannot find font: `%s', font_name)); warning (_f ("Loading default font")); @end example @item If you must have multiple-sentence messages, use full punctuation. Use two spaces after end of sentence punctuation. No punctuation (esp. period) is used at the end of simple messages. @example _f ("Non-matching braces in text `%s', adding braces", text) _ ("Debug output disabled. Compiled with NPRINT.") _f ("Huh? Not a Request: `%s'. Ignoring.", request) @end example @item Do not modularise too much; words frequently cannot be translated without context. It is probably safe to treat most occurences of words like stem, beam, crescendo as separately translatable words. @item When translating, it is preferable to put interesting information at the end of the message, rather than embedded in the middle. This especially applies to frequently used messages, even if this would mean sacrificing a bit of eloquency. This holds for original messages too, of course. @example en: cannot open: `foo.ly' + nl: kan `foo.ly' niet openen (1) kan niet openen: `foo.ly'* (2) niet te openen: `foo.ly'* (3) @end example The first nl message, although grammatically and stylistically correct, is not friendly for parsing by humans (even if they speak dutch). I guess we would prefer something like (2) or (3). @item Do not run make po/po-update with GNU gettext < 0.10.35 @end itemize @node Debugging LilyPond @section Debugging LilyPond The most commonly used tool for debugging LilyPond is the GNU debugger gdb. Use of gdb is described in this section. @subsection Debugging overview Using a debugger simplifies troubleshooting in at least two ways. First, breakpoints can be set to pause execution at any desired point. Then, when execution has paused, debugger commands can be issued to explore the values of various variables or to execute functions. Second, the debugger allows the display of a stack trace, which shows the sequence in which functions are called and the arguments to the various function calls. @subsection Compiling with debugging information In order to use a debugger with LilyPond, it is necessary to compile LilyPond with debugging information. This is accomplished by running the following commands in the main LilyPond source directory. @example ./configure --disable-optimising make @end example This will create a version of LilyPond that contains the debugging information that will allow the debugger to tie the source code to the compiled code. You should not do @var{make install} if you want to use a debugger with LilyPond. @var{make install} will strip the debugging information from the LilyPond binary. To set breakpoints in Scheme functions, put @example \include "guile-debugger.ly" @end example in your input file after any scheme procedures you have defined in that file. When your input file is processed, a guile prompt will be displayed. At the guile prompt, you can set breakpoints with the @code{break!} procedure: @example guile> (break! my-scheme-procedure) @end example Once you have set the desired breakpoints, you exit the guile repl frame by typing: @example guile> (quit) @end example When one of the scheme routines for which you have set breakpoints is entered, guile will interrupt execution in a debug frame. At this point, you will have access to guile debugging commands. For a listing of these commands, type: @example debug> help @end example @subsection Typical gdb usage @subsection Typical .gdbinit files The behavior of gdb can be readily customized through the use of @var{.gdbinit} files. A @var{.gdbinit} file is a file named @var{.gdbinit} (notice the @qq{.} at the beginning of the file name) that is placed in a user's home directory. The @var{.gdbinit} file below is from Han-Wen. It sets breakpoints for all errors and defines functions for displaying scheme objects (ps), grobs (pgrob), and parsed music expressions (pmusic). @example file lily/out/lilypond b programming_error b Grob::programming_error define ps print ly_display_scm($arg0) end define pgrob print ly_display_scm($arg0->self_scm_) print ly_display_scm($arg0->mutable_property_alist_) print ly_display_scm($arg0->immutable_property_alist_) print ly_display_scm($arg0->object_alist_) end define pmusic print ly_display_scm($arg0->self_scm_) print ly_display_scm($arg0->mutable_property_alist_) print ly_display_scm($arg0->immutable_property_alist_) end @end example @subsection Using Guile interactively with LilyPond In order to experiment with Scheme programming in the LilyPond environment, it is convenient to have a Guile interpreter that has all the LilyPond modules loaded. This requires the following steps. First, define a Scheme symbol for the active module in the .ly file: @example #(module-define! (resolve-module '(guile-user)) 'lilypond-module (current-module)) @end example Second, place a Scheme function in the .ly file that gives an interactive Guile prompt: @example #(top-repl) @end example When the .ly file is compiled, this causes the compilation to be interrupted and an interactive guile prompt to appear. When the guile prompt appears, the LilyPond active module must be set as the current guile module: @example guile> (set-current-module lilypond-module) @end example Proper operation of these commands can be demonstrated by typing the name of a LilyPond public scheme function to see if it's properly defined: @example guile> fret-diagram-verbose-markup # @end example If the LilyPond module has not been correctly loaded, an error message will be generated: @example guile> fret-diagram-verbose-markup ERROR: Unbound variable: fret-diagram-verbose-markup ABORT: (unbound-variable) @end example Once the module is properly loaded, any valid LilyPond Scheme expression can be entered at the interactive prompt. After the investigation is complete, the interactive guile interpreter can be exited: @example guile> (quit) @end example The compilation of the .ly file will then continue. @node Adding or modifying features @section Adding or modifying features When a new feature is to be added to LilyPond, it is necessary to ensure that the feature is properly integrated to maintain its long-term support. This section describes the steps necessary for feature addition. @subsection Write the code You should create a new git branch for writing the code, as that will separate it from the master branch and allow you to continue to work on small projects related to master. Please be sure to follow the rules for programming style discussed earlier in this chapter. @subsection Write regression tests In order to demonstrate that the code works properly, you will need to write one or more regression tests. These tests are typically .ly files that are found in input/regression. Regression tests should be as brief as possible to demonstrate the functionality of the code. Regression tests should generally cover one issue per test. Several short, single-issue regression tests are preferred to a single, long, multiple-issue regression test. Use existing regression tests as templates to demonstrate the type of header information that should be included in a regression test. @subsection Write documentation Although it is not required, it is helpful if the developer can write relevant material for inclusion in the Notation Reference. If the developer does not feel qualified to write the documentation, a documentation editor will be able to write it from the regression tests. If the modification changes the input syntax so that inline snippets in the documentation need to be changed, you will need to change the snippets in both the english version of the documentation and any translated versions. If you do not change the snippets in all translations, older versions of the snippet may be included when the documentation is built. If lsr snippets need to be changed, the snippet should be copied to input/new and modified there. The portions of the snippet that are added by makelsr.py should be removed. The changed snippet will then be included in all versions of the documentation. If non-snippet text is changed in the english documentation, no corresponding changes should be made in the translated documentation. @subsection Write convert-ly rule If the modification changes the input syntax, a convert-ly rule should be written to automatically update input files from older versions. convert-ly rules are found in python/convertrules.py If possible, the convert-ly rule should allow automatic updating of the file. In some cases, this will not be possible, so the rule will simply point out to the user that the feature needs manual correction. @subsection Write NEWS entry An entry should be added to the NEWS file to describe the feature changes to be implemented. This is especially important for changes that change input file syntax. Hints for NEWS file entries are given at the top of the NEWS file. New entries in NEWS go at the top of the file. The NEWS entry should be written to show how the new change improves LilyPond, if possible. @subsection Verify regression test In order to avoid breaking LilyPond, it is important to verify that the regression tests all succeed. This process is described in @ref{Regression tests}. @subsection Post patch for comments For any change other than a minor change, a patch set should be posted on Rietveld for comment. The patch set is posted by issuing the following command, after first committing all changes: @example git-cl upload @end example @noindent where is the SHA1 ID of the commit to be used as a reference source for the patch (generally, this will be the SHA1 ID of origin/master). After prompting for an email and a password, the patch set will be posted to Rietveld. An email should then be sent to lilypond-devel, with a subject line starting with PATCH:, asking for comments on the patch. As revisions are made in response to comments, successive patch sets for the same issue can be uploaded by reissuing the git-cl command. @subsection Push patch Once all the comments have been addressed, the patch can be pushed. If the author has push privileges, the author will push the patch. Otherwise, a developer with push privileges will push the patch. @subsection Closing the issues Once the patch has been pushed, all the relevant issues should be closed. On Rietveld, the author should log in an close the issue either by using the @q{Edit Issue} link, or by clicking the circled x icon to the left of the issue name. If the changes were in response to a feature request on the Google issue tracker for LilyPond, the author should change the status to @q{Fixed_x_y_z} where the patch was fixed in version x.y.z. If the author does not have privileges to change the status, an email should be sent to bug-lilypond requesting the BugMeister to change the status.