X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;f=Documentation%2Fcontributor%2Fprogramming-work.itexi;fp=Documentation%2Fcontributor%2Fprogramming-work.itexi;h=6f3bbb824a6b4e505b57d51d1dea693069b55d21;hb=941dff9d2a67080e0dd8474f1e70f0c72ace6424;hp=0000000000000000000000000000000000000000;hpb=5a22d6233a39d3164e1ca043244794c268be4ad0;p=lilypond.git diff --git a/Documentation/contributor/programming-work.itexi b/Documentation/contributor/programming-work.itexi new file mode 100644 index 0000000000..6f3bbb824a --- /dev/null +++ b/Documentation/contributor/programming-work.itexi @@ -0,0 +1,2064 @@ +@c -*- coding: utf-8; mode: texinfo; -*- +@node Programming work +@chapter Programming work + +@menu +* Overview of LilyPond architecture:: +* LilyPond programming languages:: +* Programming without compiling:: +* Finding functions:: +* Code style:: +* Debugging LilyPond:: +* Tracing object relationships:: +* Adding or modifying features:: +* Iterator tutorial:: +* Engraver tutorial:: +* Callback tutorial:: +* LilyPond scoping:: +* LilyPond miscellany:: +@end menu + +@node Overview of LilyPond architecture +@section Overview of LilyPond architecture + +LilyPond processes the input file into graphical and musical output in a +number of stages. This process, along with the types of routines that +accomplish the various stages of the process, is described in this section. A +more complete description of the LilyPond architecture and internal program +execution is found in Erik Sandberg's +@uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's +thesis}. + +The first stage of LilyPond processing is @emph{parsing}. In the parsing +process, music expressions in LilyPond input format are converted to music +expressions in Scheme format. In Scheme format, a music expression is a list +in tree form, with nodes that indicate the relationships between various music +events. The LilyPond parser is written in Bison. + +The second stage of LilyPond processing is @emph{iterating}. Iterating +assigns each music event to a context, which is the environment in which the +music will be finally engraved. The context is responsible for all further +processing of the music. It is during the iteration stage that contexts are +created as necessary to ensure that every note has a Voice type context (e.g. +Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice, +GregorianTranscriptionVoice), that the Voice type contexts exist in +appropriate Staff type contexts, and that parallel Staff type contexts exist +in StaffGroup type contexts. In addition, during the iteration stage each +music event is assigned a moment, or a time in the music when the event +begins. + +Each type of music event has an associated iterator. Iterators are defined in +@file{*-iterator.cc}. During iteration, an +event's iterator is called to deliver that music event to the appropriate +context(s). + +The final stage of LilyPond processing is @emph{translation}. During +translation, music events are prepared for graphical or midi output. The +translation step is accomplished by the polymorphic base class Translator +through its two derived classes: Engraver (for graphical output) and +Performer (for midi output). + +Translators are defined in C++ files named @file{*-engraver.cc} +and @file{*-performer.cc}. +Much of the work of translating is handled by Scheme functions, +which is one of the keys to LilyPond's exceptional flexibility. + +@sourceimage{architecture-diagram,,,png} + + +@node LilyPond programming languages +@section LilyPond programming languages + +Programming in LilyPond is done in a variety of programming languages. Each +language is used for a specific purpose or purposes. This section describes +the languages used and provides links to reference manuals and tutorials for +the relevant language. + +@subsection C++ + +The core functionality of LilyPond is implemented in C++. + +C++ is so ubiquitous that it is difficult to identify either a reference +manual or a tutorial. Programmers unfamiliar with C++ will need to spend some +time to learn the language before attempting to modify the C++ code. + +The C++ code calls Scheme/GUILE through the GUILE interface, which is +documented in the +@uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE + Reference Manual}. + +@subsection Flex + +The LilyPond lexer is implemented in Flex, an implementation of the Unix lex +lexical analyser generator. Resources for Flex can be found +@uref{http://flex.sourceforge.net/, here}. + +@subsection GNU Bison + +The LilyPond parser is implemented in Bison, a GNU parser generator. The +Bison homepage is found at @uref{http://www.gnu.org/software/bison/, +gnu.org}. The manual (which includes both a reference and tutorial) is +@uref{http://www.gnu.org/software/bison/manual/index.html, available} in a +variety of formats. + +@subsection GNU Make + +GNU Make is used to control the compiling process and to build the +documentation and the website. GNU Make documentation is available at +@uref{http://www.gnu.org/software/make/manual/, the GNU website}. + +@subsection GUILE or Scheme + +GUILE is the dialect of Scheme that is used as LilyPond's extension language. +Many extensions to LilyPond are written entirely in GUILE. The +@uref{http://www.gnu.org/software/guile/manual/html_node/index.html, +GUILE Reference Manual} is available online. + +@uref{http://mitpress.mit.edu/sicp/full-text/book/book.html, Structure and +Interpretation of Computer Programs}, a popular textbook used to teach +programming in Scheme is available in its entirety online. + +An introduction to Guile/Scheme as used in LilyPond can be found in the +@rextend{Scheme tutorial}. + +@subsection MetaFont + +MetaFont is used to create the music fonts used by LilyPond. A MetaFont +tutorial is available at @uref{http://metafont.tutorial.free.fr/, the +METAFONT tutorial page}. + +@subsection PostScript + +PostScript is used to generate graphical output. A brief PostScript tutorial +is @uref{http://local.wasp.uwa.edu.au/~pbourke/dataformats/postscript/, +available online}. The +@uref{http://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf, PostScript Language +Reference} is available online in PDF format. + +@subsection Python + +Python is used for XML2ly and is used for building the documentation and the +website. + +Python documentation is available at @uref{http://www.python.org/doc/, +python.org}. + +@node Programming without compiling +@section Programming without compiling + +Much of the development work in LilyPond takes place by changing @file{*.ly} or +@file{*.scm} files. These changes can be made without compiling LilyPond. Such +changes are described in this section. + + +@subsection Modifying distribution files + +Much of LilyPond is written in Scheme or LilyPond input files. These +files are interpreted when the program is run, rather than being compiled +when the program is built, and are present in all LilyPond distributions. +You will find @file{.ly} files in the @file{ly/} directory and the Scheme files in the +@file{scm/} directory. Both Scheme files and @file{.ly} files can be modified and +saved with any text editor. It's probably wise to make a backup copy of +your files before you modify them, although you can reinstall if the +files become corrupted. + +Once you've modified the files, you can test the changes just by running +LilyPond on some input file. It's a good idea to create a file that +demonstrates the feature you're trying to add. This file will eventually +become a regression test and will be part of the LilyPond distribution. + +@subsection Desired file formatting + +Files that are part of the LilyPond distribution have Unix-style line +endings (LF), rather than DOS (CR+LF) or MacOS 9 and earlier (CR). Make +sure you use the necessary tools to ensure that Unix-style line endings are +preserved in the patches you create. + +Tab characters should not be included in files for distribution. All +indentation should be done with spaces. Most editors have settings to +allow the setting of tab stops and ensuring that no tab characters are +included in the file. + +Scheme files and LilyPond files should be written according to standard +style guidelines. Scheme file guidelines can be found at +@uref{http://community.schemewiki.org/?scheme-style}. Following these +guidelines will make your code easier to read. Both you and others that +work on your code will be glad you followed these guidelines. + +For LilyPond files, you should follow the guidelines for LilyPond snippets +in the documentation. You can find these guidelines at +@ref{Texinfo introduction and usage policy}. + +@node Finding functions +@section Finding functions + +When making changes or fixing bugs in LilyPond, one of the initial +challenges is finding out where in the code tree the functions to +be modified live. With nearly 3000 files in the source tree, +trial-and-error searching is generally ineffective. This section +describes a process for finding interesting code. + +@subsection Using the ROADMAP + +The file ROADMAP is located in the main directory of the lilypond source. +ROADMAP lists all of the directories in the LilyPond source tree, along +with a brief description of the kind of files found in each directory. +This can be a very helpful tool for deciding which directories to search +when looking for a function. + + +@subsection Using grep to search + +Having identified a likely subdirectory to search, the grep utility can +be used to search for a function name. The format of the grep command is + +@example +grep -i functionName subdirectory/* +@end example + +This command will search all the contents of the directory subdirectory/ +and display every line in any of the files that contains +functionName. The @code{-i} option makes @command{grep} ignore +case -- this can be very useful if you are not yet familiar with +our capitalization conventions. + +The most likely directories to grep for function names are @file{scm/} for +scheme files, ly/ for lilypond input (@file{*.ly}) files, and @file{lily/} for C++ +files. + + +@subsection Using git grep to search + +If you have used git to obtain the source, you have access to a +powerful tool to search for functions. The command: + +@example +git grep functionName +@end example + +will search through all of the files that are present in the git +repository looking for functionName. It also presents the results +of the search using @code{less}, so the results are displayed one page +at a time. + +@subsection Searching on the git repository at Savannah + +You can also use the equivalent of git grep on the Savannah server. + +@itemize + +@item +Go to http://git.sv.gnu.org/gitweb/?p=lilypond.git + +@item +In the pulldown box that says commit, select grep. + +@item +Type functionName in the search box, and hit enter/return + +@end itemize + +This will initiate a search of the remote git repository. + + +@node Code style +@section Code style + +This section describes style guidelines for LilyPond +source code. + +@menu +* Languages:: +* Filenames:: +* Indentation:: +* Naming conventions:: +* Broken code:: +* Code comments:: +* Handling errors:: +* Localization:: +@end menu + + +@node Languages +@subsection Languages + +C++ and Python are preferred. Python code should use PEP 8. + + +@node Filenames +@subsection Filenames + +Definitions of classes that are only accessed via pointers (*) or +references (&) shall not be included as include files. + +@verbatim + filenames + + ".hh" Include files + ".cc" Implementation files + ".icc" Inline definition files + ".tcc" non inline Template defs + + in emacs: + + (setq auto-mode-alist + (append '(("\\.make$" . makefile-mode) + ("\\.cc$" . c++-mode) + ("\\.icc$" . c++-mode) + ("\\.tcc$" . c++-mode) + ("\\.hh$" . c++-mode) + ("\\.pod$" . text-mode) + ) + auto-mode-alist)) +@end verbatim + +The class Class_name is coded in @q{class-name.*} + + +@node Indentation +@subsection Indentation + +Standard GNU coding style is used. In emacs: + +@verbatim + (add-hook 'c++-mode-hook + '(lambda() (c-set-style "gnu") + )) +@end verbatim + +If you like using font-lock, you can also add this to your +@q{.emacs}: + +@verbatim + (setq font-lock-maximum-decoration t) + (setq c++-font-lock-keywords-3 + (append + c++-font-lock-keywords-3 + '(("\\b\\(a-zA-Z_?+_\\)\\b" 1 font-lock-variable-name-face) ("\\b\\(A-Z?+a-z_?+\\)\\b" 1 font-lock-type-face)) + )) +@end verbatim + +Some source files may not currently have proper indenting. If this +is the case, it is desirable to fix the improper indenting when the +file is modified, with the hope of continually improving the code. + + +@subheading Indenting files with fixcc.py + +LilyPond provides a python script that will correct the indentation +on a c++ file: + +@example +scripts/auxiliar/fixcc.py lily/my-test-file.cc +@end example + +Be sure you replace @file{my-test-file.cc} with the name of the file +that you edited. + +If you are editing a file that contains an ADD_TRANSLATOR or ADD_INTERFACE +macro, the fixcc.py script will move the final parenthesis up one line +from where it should be. Please check the end of the file before +you run fixcc.py, and then put the final parenthesis and semicolon +back on a line by themselves. + + +@subheading Indenting files with emacs in script mode + +@c email to wl@gnu.org when I get here. + +@warning{this is pending some confirmation on -devel. July 2009 -gp} + +Command-line script to format stuff with emacs: + +@example +#!/bin/sh +emacs $1 -batch --eval '(indent-region (point-min) (point-max) nil)' -f save-buffer +@end example + +(that's all on one line) + +Save it as a shell script, then run on the file(s) you modified. + + +@subheading Indenting with vim + +Although emacs indentation is the LilyPond standard, acceptable +indentation can usually be accomplished with vim. Some hints for +vim are as follows: + +A workable .vimrc: + +@verbatim +set cindent +set smartindent +set autoindent +set expandtab +set softtabstop=2 +set shiftwidth=2 +filetype plugin indent on +set incsearch +set ignorecase smartcase +set hlsearch +set confirm +set statusline=%F%m%r%h%w\ %{&ff}\ %Y\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ %04l,%04v\ %p%%\ [LEN=%L] +set laststatus=2 +set number +" Remove trailing whitespace on write +autocmd BufWritePre * :%s/\s\+$//e +@end verbatim + +With this .vimrc, files can be reindented automatically by highlighting +the lines to be indented in visual mode (use V to enter visual mode) +and pressing =. + +A scheme.vim file will help improve the indentation. This one +was suggested by Patrick McCarty. It should be saved in +~/.vim/after/syntax/scheme.vim. + +@verbatim +" Additional Guile-specific 'forms' +syn keyword schemeSyntax define-public define*-public +syn keyword schemeSyntax define* lambda* let-keywords* +syn keyword schemeSyntax defmacro defmacro* define-macro +syn keyword schemeSyntax defmacro-public defmacro*-public +syn keyword schemeSyntax use-modules define-module +syn keyword schemeSyntax define-method define-class + +" Additional LilyPond-specific 'forms' +syn keyword schemeSyntax define-markup-command define-markup-list-command +syn keyword schemeSyntax define-safe-public define-music-function +syn keyword schemeSyntax def-grace-function + +" All of the above should influence indenting too +set lw+=define-public,define*-public +set lw+=define*,lambda*,let-keywords* +set lw+=defmacro,defmacro*,define-macro +set lw+=defmacro-public,defmacro*-public +set lw+=use-modules,define-module +set lw+=define-method,define-class +set lw+=define-markup-command,define-markup-list-command +set lw+=define-safe-public,define-music-function +set lw+=def-grace-function + +" These forms should not influence indenting +set lw-=if +set lw-=set! + +" Try to highlight all ly: procedures +syn match schemeFunc "ly:[^) ]\+" +@end verbatim + + +@node Naming conventions +@subsection Naming Conventions + +Naming conventions have been established for LilyPond +source code. + +@subheading Classes and Types + +Classes begin with an uppercase letter, and words +in class names are separated with @code{_}: + +@verbatim +This_is_a_class +@end verbatim + +@subheading Members + +Member variable names end with an underscore: + +@verbatim +Type Class::member_ +@end verbatim + +@subheading Macros + +Macro names should be written in uppercase completely, +with words separated by @code{_}: + +@verbatim +THIS_IS_A_MACRO +@end verbatim + +@subheading Variables + +Variable names should be complete words, rather than abbreviations. +For example, it is preferred to use @code{thickness} rather than +@code{th} or @code{t}. + +Multi-word variable names in C++ should have the words separated +by the underscore character (@q{_}): + +@verbatim +cxx_multiword_variable +@end verbatim + +Multi-word variable names in Scheme should have the words separated +by a hyphen (@q{-}): + +@verbatim +scheme-multiword-variable +@end verbatim + +@node Broken code +@subsection Broken code + +Do not write broken code. This includes hardwired dependencies, +hardwired constants, slow algorithms and obvious limitations. If +you can not avoid it, mark the place clearly, and add a comment +explaining shortcomings of the code. + +Ideally, the comment marking the shortcoming would include +TODO, so that it is marked for future fixing. + +We reject broken-in-advance on principle. + + +@node Code comments +@subsection Code comments + +Comments may not be needed if descriptive variable names are used +in the code and the logic is straightforward. However, if the +logic is difficult to follow, and particularly if non-obvious +code has been included to resolve a bug, a comment describing +the logic and/or the need for the non-obvious code should be included. + +There are instances where the current code could be commented better. +If significant time is required to understand the code as part of +preparing a patch, it would be wise to add comments reflecting your +understanding to make future work easier. + + +@node Handling errors +@subsection Handling errors + +As a general rule, you should always try to continue computations, +even if there is some kind of error. When the program stops, it +is often very hard for a user to pinpoint what part of the input +causes an error. Finding the culprit is much easier if there is +some viewable output. + +So functions and methods do not return errorcodes, they never +crash, but report a programming_error and try to carry on. + +Error and warning messages need to be localized. + + +@node Localization +@subsection Localization + +This document provides some guidelines to help programmers write +proper user +messages. To help translations, user messages must follow +uniform conventions. Follow these rules when coding for LilyPond. +Hopefully, this can be replaced by general GNU guidelines in the +future. Even better would be to have an English (en_BR, en_AM) +guide helping programmers writing consistent messages for all GNU +programs. + +Non-preferred messages are marked with `+'. By convention, +ungrammatical examples are marked with `*'. However, such ungrammatical +examples may still be preferred. + +@itemize + +@item +Every message to the user should be localized (and thus be marked +for localization). This includes warning and error messages. + +@item +Do not localize/gettextify: + +@itemize +@item +`programming_error ()'s + +@item +`programming_warning ()'s + +@item +debug strings + +@item +output strings (PostScript, TeX, etc.) + +@end itemize + +@item +Messages to be localized must be encapsulated in `_ (STRING)' or +`_f (FORMAT, ...)'. E.g.: + +@example +warning (_ ("need music in a score")); +error (_f ("cannot open file: `%s'", file_name)); +@end example + +In some rare cases you may need to call `gettext ()' by hand. This +happens when you pre-define (a list of) string constants for later +use. In that case, you'll probably also need to mark these string +constants for translation, using `_i (STRING)'. The `_i' macro is +a no-op, it only serves as a marker for `xgettext'. + +@example +char const* messages[] = @{ + _i ("enable debugging output"), + _i ("ignore lilypond version"), + 0 +@}; + +void +foo (int i) +@{ + puts (gettext (messages i)); +@} +@end example + +See also @file{flower/getopt-long.cc} and @file{lily/main.cc}. + +@item +Do not use leading or trailing whitespace in messages. If you need +whitespace to be printed, prepend or append it to the translated +message + +@example +message ("Calculating line breaks..." + " "); +@end example + +@item +Error or warning messages displayed with a file name and line +number never start with a capital, eg, + +@example +foo.ly: 12: not a duration: 3 +@end example + +Messages containing a final verb, or a gerund (`-ing'-form) always +start with a capital. Other (simpler) messages start with a +lowercase letter + +@example +Processing foo.ly... +`foo': not declared. +Not declaring: `foo'. +@end example + +@item +Avoid abbreviations or short forms, use `cannot' and `do not' +rather than `can't' or `don't' +To avoid having a number of different messages for the same +situation, well will use quoting like this `"message: `%s'"' for all +strings. Numbers are not quoted: + +@example +_f ("cannot open file: `%s'", name_str) +_f ("cannot find character number: %d", i) +@end example + +@item +Think about translation issues. In a lot of cases, it is better to +translate a whole message. English grammar must not be imposed on the +translator. So, instead of + +@example +stem at + moment.str () + does not fit in beam +@end example + +have + +@example +_f ("stem at %s does not fit in beam", moment.str ()) +@end example + +@item +Split up multi-sentence messages, whenever possible. Instead of + +@example +warning (_f ("out of tune! Can't find: `%s'", "Key_engraver")); +warning (_f ("cannot find font `%s', loading default", font_name)); +@end example + +rather say: + +@example +warning (_ ("out of tune:")); +warning (_f ("cannot find: `%s', "Key_engraver")); +warning (_f ("cannot find font: `%s', font_name)); +warning (_f ("Loading default font")); +@end example + +@item +If you must have multiple-sentence messages, use full punctuation. +Use two spaces after end of sentence punctuation. No punctuation +(esp. period) is used at the end of simple messages. + +@example +_f ("Non-matching braces in text `%s', adding braces", text) +_ ("Debug output disabled. Compiled with NPRINT.") +_f ("Huh? Not a Request: `%s'. Ignoring.", request) +@end example + +@item +Do not modularize too much; words frequently cannot be translated +without context. It is probably safe to treat most occurrences of +words like stem, beam, crescendo as separately translatable words. + +@item +When translating, it is preferable to put interesting information +at the end of the message, rather than embedded in the middle. +This especially applies to frequently used messages, even if this +would mean sacrificing a bit of eloquency. This holds for original +messages too, of course. + +@example +en: cannot open: `foo.ly' ++ nl: kan `foo.ly' niet openen (1) +kan niet openen: `foo.ly'* (2) +niet te openen: `foo.ly'* (3) +@end example + + +The first nl message, although grammatically and stylistically +correct, is not friendly for parsing by humans (even if they speak +dutch). I guess we would prefer something like (2) or (3). + +@item +Do not run make po/po-update with GNU gettext < 0.10.35 + +@end itemize + + + +@node Debugging LilyPond +@section Debugging LilyPond + +The most commonly used tool for debugging LilyPond is the GNU +debugger gdb. The gdb tool is used for investigating and debugging +core Lilypond code written in C++. Another tool is available for +debugging Scheme code using the Guile debugger. This section +describes how to use both gdb and the Guile Debugger. + +@menu +* Debugging overview:: +* Debugging C++ code:: +* Debugging Scheme code:: +@end menu + +@node Debugging overview +@subsection Debugging overview + +Using a debugger simplifies troubleshooting in at least two ways. + +First, breakpoints can be set to pause execution at any desired point. +Then, when execution has paused, debugger commands can be issued to +explore the values of various variables or to execute functions. + +Second, the debugger can display a stack trace, which shows the +sequence in which functions have been called and the arguments +passed to the called functions. + +@node Debugging C++ code +@subsection Debugging C++ code + +The GNU debugger, gdb, is the principal tool for debugging C++ code. + +@subheading Compiling LilyPond for use with gdb + +In order to use gdb with LilyPond, it is necessary to compile +LilyPond with debugging information. This is accomplished by running +the following commands in the main LilyPond source directory. + +@example +./configure --disable-optimising +make +@end example + +This will create a version of LilyPond containing debugging +information that will allow the debugger to tie the source code +to the compiled code. + +You should not do @var{make install} if you want to use a debugger +with LilyPond. The @var{make install} command will strip debugging +information from the LilyPond binary. + +@subheading Typical gdb usage + +Once you have compiled the Lilypond image with the necessary +debugging information it will have been written to a location in a +subfolder of your current working directory: + +@example +out/bin/lilypond +@end example + +This is important as you will need to let gdb know where to find the +image containing the symbol tables. You can invoke gdb from the +command line using the following: + +@example +gdb out/bin/lilypond +@end example +@noindent +This loads the LilyPond symbol tables into gdb. Then, to run +LilyPond on @file{test.ly} under the debugger, enter the following: + +@example +run test.ly +@end example + +@noindent +at the gdb prompt. + +As an alternative to running gdb at the command line you may try +a graphical interface to gdb such as ddd: + +@example +ddd out/bin/lilypond +@end example + +You can also use sets of standard gdb commands stored in a .gdbinit +file (see next section). + +@subheading Typical .gdbinit files + +The behavior of gdb can be readily customized through the use of a +@var{.gdbinit} file. A @var{.gdbinit} file is a file named +@var{.gdbinit} (notice the @qq{.} at the beginning of the file name) +that is placed in a user's home directory. + +The @var{.gdbinit} file below is from Han-Wen. It sets breakpoints +for all errors and defines functions for displaying scheme objects +(ps), grobs (pgrob), and parsed music expressions (pmusic). + +@example +file lily/out/lilypond +b programming_error +b Grob::programming_error + +define ps + print ly_display_scm($arg0) +end +define pgrob + print ly_display_scm($arg0->self_scm_) + print ly_display_scm($arg0->mutable_property_alist_) + print ly_display_scm($arg0->immutable_property_alist_) + print ly_display_scm($arg0->object_alist_) +end +define pmusic + print ly_display_scm($arg0->self_scm_) + print ly_display_scm($arg0->mutable_property_alist_) + print ly_display_scm($arg0->immutable_property_alist_) +end +@end example + +@node Debugging Scheme code +@subsection Debugging Scheme code + +Scheme code can be developed using the Guile command line +interpreter @code{top-repl}. You can either investigate +interactively using just Guile or you can use the debugging +tools available within Guile. + +@subheading Using Guile interactively with LilyPond + +In order to experiment with Scheme programming in the LilyPond +environment, it is necessary to have a Guile interpreter that +has all the LilyPond modules loaded. This requires the following +steps. + +First, define a Scheme symbol for the active module in the @file{.ly} file: + +@example +#(module-define! (resolve-module '(guile-user)) + 'lilypond-module (current-module)) +@end example + +Now place a Scheme function in the @file{.ly} file that gives an +interactive Guile prompt: + +@example +#(top-repl) +@end example + +When the @file{.ly} file is compiled, this causes the compilation to be +interrupted and an interactive guile prompt to appear. Once the +guile prompt appears, the LilyPond active module must be set as the +current guile module: + +@example +guile> (set-current-module lilypond-module) +@end example + +You can demonstrate these commands are operating properly by typing the name +of a LilyPond public scheme function to check it has been defined: + +@example +guile> fret-diagram-verbose-markup +# +@end example + +If the LilyPond module has not been correctly loaded, an error +message will be generated: + +@example +guile> fret-diagram-verbose-markup +ERROR: Unbound variable: fret-diagram-verbose-markup +ABORT: (unbound-variable) +@end example + +Once the module is properly loaded, any valid LilyPond Scheme +expression can be entered at the interactive prompt. + +After the investigation is complete, the interactive guile +interpreter can be exited: + +@example +guile> (quit) +@end example + +The compilation of the @file{.ly} file will then continue. + +@subheading Using the Guile debugger + +To set breakpoints and/or enable tracing in Scheme functions, put + +@example +\include "guile-debugger.ly" +@end example + +in your input file after any scheme procedures you have defined in +that file. This will invoke the Guile command-line after having set +up the environment for the debug command-line. When your input file +is processed, a guile prompt will be displayed. You may now enter +commands to set up breakpoints and enable tracing by the Guile debugger. + +@subheading Using breakpoints + +At the guile prompt, you can set breakpoints with +the @code{set-break!} procedure: + +@example +guile> (set-break! my-scheme-procedure) +@end example + +Once you have set the desired breakpoints, you exit the guile repl frame +by typing: + +@example +guile> (quit) +@end example + +Then, when one of the scheme routines for which you have set +breakpoints is entered, guile will interrupt execution in a debug +frame. At this point you will have access to Guile debugging +commands. For a listing of these commands, type: + +@example +debug> help +@end example + +Alternatively you may code the breakpoints in your Lilypond source +file using a command such as: + +@example +#(set-break! my-scheme-procedure) +@end example + +immediately after the @code{\include} statement. In this case the +breakpoint will be set straight after you enter the @code{(quit)} +command at the guile prompt. + +Embedding breakpoint commands like this is particularly useful if +you want to look at how the Scheme procedures in the @file{.scm} +files supplied with LilyPond work. To do this, edit the file in +the relevant directory to add this line near the top: + +@example +(use-modules (scm guile-debugger)) +@end example + +Now you can set a breakpoint after the procedure you are interested +in has been declared. For example, if you are working on routines +called by @var{print-book-with} in @file{lily-library.scm}: + +@example +(define (print-book-with parser book process-procedure) + (let* ((paper (ly:parser-lookup parser '$defaultpaper)) + (layout (ly:parser-lookup parser '$defaultlayout)) + (outfile-name (get-outfile-name parser))) + (process-procedure book paper layout outfile-name))) + +(define-public (print-book-with-defaults parser book) + (print-book-with parser book ly:book-process)) + +(define-public (print-book-with-defaults-as-systems parser book) + (print-book-with parser book ly:book-process-to-systems)) + +@end example + +At this point in the code you could add this to set a breakpoint at +print-book-with: + +@example +(set-break! print-book-with) +@end example + +@subheading Tracing procedure calls and evaluator steps + +Two forms of trace are available: + +@example +(set-trace-call! my-scheme-procedure) +@end example + +and + +@example +(set-trace-subtree! my-scheme-procedure) +@end example + +@code{set-trace-call!} causes Scheme to log a line to the standard +output to show when the procedure is called and when it exits. + +@code{set-trace-subtree!} traces every step the Scheme evaluator +performs in evaluating the procedure. + +@node Tracing object relationships +@section Tracing object relationships + +Understanding the LilyPond source often boils down to figuring out what +is happening to the Grobs. Where (and why) are they being created, +modified and destroyed? Tracing Lily through a debugger in order to +identify these relationships can be time-consuming and tedious. + +In order to simplify this process, a facility has been added to +display the grobs that are created and the properties that are set +and modified. Although it can be complex to get set up, once set up +it easily provides detailed information about the life of grobs +in the form of a network graph. + +Each of the steps necessary to use the graphviz utility +is described below. + +@enumerate + +@item Installing graphviz + +In order to create the graph of the object relationships, it is +first necessary to install Graphviz. graphviz is available for a +number of different platforms: + +@example +@uref{http://www.graphviz.org/Download..php} +@end example + +@item Modifying config.make + +In order for the Graphviz tool to work, config.make must be modified. +It is probably a good idea to first save a copy of config.make under +a different name. Then, edit config.make by removing every occurrence +of @code{-DNDEBUG}. + +@item Rebuilding LilyPond + +The executable code of LilyPond must be rebuilt from scratch: + +@example +make -C lily clean && make -C lily +@end example + +@item Create a graphviz-compatible @file{.ly} file + +In order to use the graphviz utility, the @file{.ly} file must include +@file{ly/graphviz-init.ly}, and should then specify the +grobs and symbols that should be tracked. An example of this +is found in @file{input/regression/graphviz.ly}. + +@item Run lilypond with output sent to a log file + +The Graphviz data is sent to stderr by lilypond, so it is +necessary to redirect stderr to a logfile: + +@example +lilypond graphviz.ly 2> graphviz.log +@end example + +@item Edit the logfile + +The logfile has standard lilypond output, as well as the Graphviz +output data. Delete everything from the beginning of the file +up to but not including the first occurrence of @code{digraph}. + +Also, delete the final liypond message about successs from the end +of the file. + +@item Process the logfile with @code{dot} + +The directed graph is created from the log file with the program +@code{dot}: + +@example +dot -Tpdf graphviz.log > graphviz.pdf +@end example + +@end enumerate + +The pdf file can then be viewed with any pdf viewer. + +When compiled without @code{-DNDEBUG}, lilypond may run slower +than normal. The original configuration can be restored by either +renaming the saved copy of @code{config.make} or rerunning +@code{configure}. Then rebuild lilypond with + +@example +make -C lily clean && make -C lily +@end example + + +@node Adding or modifying features +@section Adding or modifying features + +When a new feature is to be added to LilyPond, it is necessary to +ensure that the feature is properly integrated to maintain +its long-term support. This section describes the steps necessary +for feature addition and modification. + + +@menu +* Write the code:: +* Write regression tests:: +* Write convert-ly rule:: +* Automatically update documentation:: +* Manually update documentation:: +* Edit changes.tely:: +* Verify successful build:: +* Verify regression tests:: +* Post patch for comments:: +* Push patch:: +* Closing the issues:: +@end menu + +@node Write the code +@subsection Write the code + +You should probably create a new git branch for writing the code, as that +will separate it from the master branch and allow you to continue +to work on small projects related to master. + +Please be sure to follow the rules for programming style discussed +earlier in this chapter. + + +@node Write regression tests +@subsection Write regression tests + +In order to demonstrate that the code works properly, you will +need to write one or more regression tests. These tests are +typically @file{.ly} files that are found in @file{input/regression}. + +Regression tests should be as brief as possible to demonstrate the +functionality of the code. + +Regression tests should generally cover one issue per test. Several +short, single-issue regression tests are preferred to a single, long, +multiple-issue regression test. + +Use existing regression tests as templates to demonstrate the type of +header information that should be included in a regression test. + + +@node Write convert-ly rule +@subsection Write convert-ly rule + +If the modification changes the input syntax, a convert-ly rule +should be written to automatically update input files from older +versions. + +convert-ly rules are found in python/convertrules.py + +If possible, the convert-ly rule should allow automatic updating +of the file. In some cases, this will not be possible, so the +rule will simply point out to the user that the feature needs +manual correction. + +@subsubheading Updating version numbers + +If a development release occurs between you writing your patch and +having it approved+pushed, you will need to update the version +numbers in your tree. This can be done with: + +@example +scripts/auxiliar/update-patch-version old.version.number new.version.number +@end example + +It will change all files in git, so use with caution and examine +the resulting diff. + + +@node Automatically update documentation +@subsection Automatically update documentation + +@command{convert-ly} should be used to update the documentation, +the snippets, and the regression tests. This not only makes the +necessary syntax changes, it also tests the @command{convert-ly} +rules. + +The automatic updating is performed by moving to the top-level +source directory, then running: + +@example +scripts/auxiliar/update-with-convert-ly.sh +@end example + +If you did an out-of-tree build, pass in the relative path: + +@example +BUILD_DIR=../build-lilypond/ scripts/auxiliar/update-with-convert-ly.sh +@end example + + +@node Manually update documentation +@subsection Manually update documentation + +Where the convert-ly rule is not able to automatically update the inline +lilypond code in the documentation (i.e. if a NOT_SMART rule is used), the +documentation must be manually updated. The inline snippets that require +changing must be changed in the English version of the docs and all +translated versions. If the inline code is not changed in the +translated documentation, the old snippets will show up in the +English version of the documentation. + +Where the convert-ly rule is not able to automatically update snippets +in Documentation/snippets/, those snippets must be manually updated. +Those snippets should be copied to Documentation/snippets/new. The +comments at the top of the snippet describing its automatic generation +should be removed. All translated texidoc strings should be removed. +The comment @qq{% begin verbatim} should be removed. The syntax of +the snippet should then be manually edited. + +Where snippets in Documentation/snippets are made obsolete, the snippet +should be copied to Documentation/snippets/new. The comments and +texidoc strings should be removed as described above. Then the body +of the snippet should be changed to: + +@example +\markup @{ + This snippet is deprecated as of version X.Y.Z and + will be removed from the documentation. +@} +@end example + +@noindent +where X.Y.Z is the version number for which the convert-ly rule was +written. + +Update the snippet files by running: + +@example +scripts/auxiliar/makelsr.py +@end example + +Where the convert-ly rule is not able to automatically update regression +tests, the regression tests in input/regression should be manually +edited. + +Although it is not required, it is helpful if the developer +can write relevant material for inclusion in the Notation +Reference. If the developer does not feel qualified to write +the documentation, a documentation editor will be able to +write it from the regression tests. The text that is added to +or removed from the documentation should be changed only in +the English version. + + +@node Edit changes.tely +@subsection Edit changes.tely + +An entry should be added to Documentation/changes.tely to describe +the feature changes to be implemented. This is especially important +for changes that change input file syntax. + +Hints for changes.tely entries are given at the top of the file. + +New entries in changes.tely go at the top of the file. + +The changes.tely entry should be written to show how the new change +improves LilyPond, if possible. + + +@node Verify successful build +@subsection Verify successful build + +When the changes have been made, successful completion must be +verified by doing + +@example +make all +make doc +@end example + +When these commands complete without error, the patch is +considered to function successfully. + +Developers on Windows who are unable to build LilyPond should +get help from a Linux or OSX developer to do the make tests. + + +@node Verify regression tests +@subsection Verify regression tests + +In order to avoid breaking LilyPond, it is important to verify that +the regression tests succeed, and that no unwanted changes are +introduced into the output. This process is described in +@ref{Regtest comparison}. + +@subheading Typical developer's edit/compile/test cycle + +TODO: is @code{[-j@var{X} CPU_COUNT=@var{X}]} useful for +@code{test-baseline}, @code{check}, @code{clean}, +@code{test-redo}? Neil Puttock says it is useful for +everything but @code{clean}, which is disk-limited. +Need to check formally. + +@itemize +@item +Initial test: + +@example +make [-j@var{X}] +make test-baseline +make [-j@var{X} CPU_COUNT=@var{X}] check +@end example + +@item +Edit/compile/test cycle: + +@example +@emph{## edit source files, then...} + +make clean @emph{## only if needed (see below)} +make [-j@var{X}] @emph{## only if needed (see below)} +make test-redo @emph{## redo files differing from baseline} +make [-j@var{X} CPU_COUNT=@var{X}] check @emph{## CPU_COUNT here?} +@end example + +@item +Reset: + +@example +make test-clean +@end example +@end itemize + +If you modify any source files that have to be compiled (such as +@file{.cc} or @file{.hh} files in @file{flower/} or @file{lily/}), +then you must run @command{make} before @command{make test-redo}, +so @command{make} can compile the modified files and relink all +the object files. If you only modify files which are interpreted, +like those in the @file{scm/} and @file{ly/} directories, then +@command{make} is not needed before @command{make test-redo}. + +TODO: Fix the following paragraph. You can do @command{rm mf/out/*} +instead of make clean, and you can probably do +@command{make -C mf/ clean} as well, but I haven't checked it -- cds + +Also, if you modify any font definitions in the @file{mf/} +directory then you must run @command{make clean} and +@command{make} before running @command{make test-redo}. This will +recompile everything, whether modified or not, and takes a lot +longer. + +Running @command{make@tie{}check} will leave an HTML page +@file{out/test-results/index.html}. This page shows all the +important differences that your change introduced, whether in the +layout, MIDI, performance or error reporting. + + + + +@node Post patch for comments +@subsection Post patch for comments + +See @ref{Uploading a patch for review}. + + +@node Push patch +@subsection Push patch + +Once all the comments have been addressed, the patch can be pushed. + +If the author has push privileges, the author will push the patch. +Otherwise, a developer with push privileges will push the patch. + + +@node Closing the issues +@subsection Closing the issues + +Once the patch has been pushed, all the relevant issues should be +closed. + +On Rietveld, the author should log in an close the issue either by +using the @q{Edit Issue} link, or by clicking the circled x icon +to the left of the issue name. + +If the changes were in response to a feature request on the Google +issue tracker for LilyPond, the author should change the status to +Fixed and a tag @q{fixed_x_y_z} should be added, where the patch was +fixed in version x.y.z. If +the author does not have privileges to change the status, an email +should be sent to bug-lilypond requesting the BugMeister to change +the status. + + +@node Iterator tutorial +@section Iterator tutorial + +TODO -- this is a placeholder for a tutorial on iterators + +Iterators are routines written in C++ that process music expressions +and sent the music events to the appropriate engravers and/or +performers. + + +@node Engraver tutorial +@section Engraver tutorial + +Engravers are C++ classes that catch music events and +create the appropriate grobs for display on the page. Though the +majority of engravers are responsible for the creation of a single grob, +in some cases (e.g. @code{New_fingering_engraver}), several different grobs +may be created. + +Engravers listen for events and acknowledge grobs. Events are passed to +the engraver in time-step order during the iteration phase. Grobs are +made available to the engraver when they are created by other engravers +during the iteration phase. + + +@menu +* Useful methods for information processing:: +* Translation process:: +* Preventing garbage collection for SCM member variables:: +* Listening to music events:: +* Acknowledging grobs:: +* Engraver declaration/documentation:: +@end menu + +@node Useful methods for information processing +@subsection Useful methods for information processing + +An engraver inherits the following public methods from the Translator +base class, which can be used to process listened events and acknowledged +grobs: + +@itemize +@item @code{virtual void initialize ()} +@item @code{void start_translation_timestep ()} +@item @code{void process_music ()} +@item @code{void process_acknowledged ()} +@item @code{void stop_translation_timestep ()} +@item @code{virtual void finalize ()} +@end itemize + +These methods are listed in order of translation time, with +@code{initialize ()} and @code{finalize ()} bookending the whole +process. @code{initialize ()} can be used for one-time initialization +of context properties before translation starts, whereas +@code{finalize ()} is often used to tie up loose ends at the end of +translation: for example, an unterminated spanner might be completed +automatically or reported with a warning message. + + +@node Translation process +@subsection Translation process + +At each timestep in the music, translation proceeds by calling the +following methods in turn: + +@code{start_translation_timestep ()} is called before any user +information enters the translators, i.e., no property operations +(\set, \override, etc.) or events have been processed yet. + +@code{process_music ()} and @code{process_acknowledged ()} are called +after all events in the current time step have been heard, or all +grobs in the current time step have been acknowledged. The latter +tends to be used exclusively with engravers which only acknowledge +grobs, whereas the former is the default method for main processing +within engravers. + +@code{stop_translation_timestep ()} is called after all user +information has been processed prior to beginning the translation for +the next timestep. + + +@node Preventing garbage collection for SCM member variables +@subsection Preventing garbage collection for SCM member variables + +In certain cases, an engraver might need to ensure private Scheme +variables (with type SCM) do not get swept away by Guile's garbage +collector: for example, a cache of the previous key signature which +must persist between timesteps. The method +@code{virtual derived_mark () const} can be used in such cases: + +@example +Engraver_name::derived_mark () +@{ + scm_gc_mark (private_scm_member_) +@} +@end example + + +@node Listening to music events +@subsection Listening to music events + +External interfaces to the engraver are implemented by protected +macros including one or more of the following: + +@itemize +@item @code{DECLARE_TRANSLATOR_LISTENER (event_name)} +@item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)} +@end itemize + +@noindent +where @var{event_name} is the type of event required to provide the +input the engraver needs and @var{Engraver_name} is the name of the +engraver. + +Following declaration of a listener, the method is implemented as follows: + +@example +IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name) +void +Engraver_name::listen_event_name (Stream event *event) +@{ + ...body of listener method... +@} +@end example + + +@node Acknowledging grobs +@subsection Acknowledging grobs + +Some engravers also need information from grobs as they are created +and as they terminate. The mechanism and methods to obtain this +information are set up by the macros: + +@itemize +@item @code{DECLARE_ACKNOWLEDGER (grob_interface)} +@item @code{DECLARE_END_ACKNOWLEDGER (grob_interface)} +@end itemize + +where @var{grob_interface} is an interface supported by the +grob(s) which should be acknowledged. For example, the following +code would declare acknowledgers for a @code{NoteHead} grob (via the +@code{note-head-interface}) and any grobs which support the +@code{side-position-interface}: + +@example +@code{DECLARE_ACKNOWLEDGER (note_head)} +@code{DECLARE_ACKNOWLEDGER (side_position)} +@end example + +The @code{DECLARE_END_ACKNOWLEDGER ()} macro sets up a spanner-specific +acknowledger which will be called whenever a spanner ends. + +Following declaration of an acknowledger, the method is coded as follows: + +@example +void +Engraver_name::acknowledge_interface_name (Grob_info info) +@{ + ...body of acknowledger method... +@} +@end example + + +@node Engraver declaration/documentation +@subsection Engraver declaration/documentation + +An engraver must have a public macro + +@itemize +@item @code{TRANSLATOR_DECLARATIONS (Engraver_name)} +@end itemize + +@noindent +where @code{Engraver_name} is the name of the engraver. This +defines the common variables and methods used by every engraver. + +At the end of the engraver file, one or both of the following +macros are generally called to document the engraver in the +Internals Reference: + +@itemize +@item @code{ADD_ACKNOWLEDGER (Engraver_name, grob_interface)} +@item @code{ADD_TRANSLATOR (Engraver_name, Engraver_doc, + Engraver_creates, Engraver_reads, Engraver_writes)} +@end itemize + +@noindent +where @code{Engraver_name} is the name of the engraver, @code{grob_interface} +is the name of the interface that will be acknowledged, +@code{Engraver_doc} is a docstring for the engraver, +@code{Engraver_creates} is the set of grobs created by the engraver, +@code{Engraver_reads} is the set of properties read by the engraver, +and @code{Engraver_writes} is the set of properties written by +the engraver. + +The @code{ADD_ACKNOWLEDGER} and @code{ADD_TRANSLATOR} macros use a +non-standard indentation system. Each interface, grob, read property, +and write property is on its own line, and the closing parenthesis +and semicolon for the macro all occupy a separate line beneath the final +interface or write property. See existing engraver files for more +information. + + +@node Callback tutorial +@section Callback tutorial + +TODO -- This is a placeholder for a tutorial on callback functions. + +@node LilyPond scoping +@section LilyPond scoping + +The Lilypond language has a concept of scoping, i.e. you can do + +@example +foo = 1 + +#(begin + (display (+ foo 2))) +@end example + +@noindent with @code{\paper}, @code{\midi} and @code{\header} being +nested scope inside the @file{.ly} file-level scope. @w{@code{foo = 1}} +is translated in to a scheme variable definition. + +This implemented using modules, with each scope being an anonymous +module that imports its enclosing scope's module. + +Lilypond's core, loaded from @file{.scm} files, is usually placed in the +@code{lily} module, outside the @file{.ly} level. In the case of + +@example +lilypond a.ly b.ly +@end example + +@noindent +we want to reuse the built-in definitions, without changes effected in +user-level @file{a.ly} leaking into the processing of @file{b.ly}. + +The user-accessible definition commands have to take care to avoid +memory leaks that could occur when running multiple files. All +information belonging to user-defined commands and markups is stored in +a manner that allows it to be garbage-collected when the module is +dispersed, either by being stored module-locally, or in weak hash +tables. + +@node LilyPond miscellany +@section LilyPond miscellany + +This is a place to dump information that may be of use to developers +but doesn't yet have a proper home. Ideally, the length of this section +would become zero as items are moved to other homes. + + +@menu +* Spacing algorithms:: +* Info from Han-Wen email:: +* Music functions and GUILE debugging:: +@end menu + +@node Spacing algorithms +@subsection Spacing algorithms + +Here is information from an email exchange about spacing algorithms. + +On Thu, 2010-02-04 at 15:33 -0500, Boris Shingarov wrote: +I am experimenting with some modifications to the line breaking code, +and I am stuck trying to understand how some of it works. So far my +understanding is that Simple_spacer operates on a vector of Grobs, and +it is a well-known Constrained-QP problem (rods = constraints, springs += quadratic function to minimize). What I don't understand is, if the +spacer operates at the level of Grobs, which are built at an earlier +stage in the pipeline, how are the changes necessitated by differences +in line breaking, taken into account? in other words, if I take the +last measure of a line and place it on the next line, it is not just a +matter of literally moving that graphic to where the start of the next +line is, but I also need to draw a clef, key signature, and possibly +other fundamental things -- but at that stage in the rendering +pipeline, is it not too late?? + +Joe Neeman answered: + +We create lots of extra grobs (eg. a BarNumber at every bar line) but +most of them are not drawn. See the break-visibility property in +item-interface. + + +@node Info from Han-Wen email +@subsection Info from Han-Wen email + +In 2004, Douglas Linhardt decided to try starting a document that would +explain LilyPond architecture and design principles. The material below +is extracted from that email, which can be found at +@uref{http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/2992}. +The headings reflect questions from Doug or comments from Han-Wen; +the body text are Han-Wen's answers. + +@subheading Figuring out how things work. + +I must admit that when I want to know how a program works, I use grep +and emacs and dive into the source code. The comments and the code +itself are usually more revealing than technical documents. + +@subheading What's a grob, and how is one used? + +Graphical object - they are created from within engravers, either as +Spanners (derived class) -slurs, beams- or Items (also a derived +class) -notes, clefs, etc. + +There are two other derived classes System (derived from Spanner, +containing a "line of music") and Paper_column (derived from Item, it +contains all items that happen at the same moment). They are separate +classes because they play a special role in the linebreaking process. + +@subheading What's a smob, and how is one used? + +A C(++) object that is encapsulated so it can be used as a Scheme +object. See GUILE info, "19.3 Defining New Types (Smobs)" + +@@subheading When is each C++ class constructed and used + +@itemize + +@item +Music classes + +In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME(). + +@item +Contexts + +Constructed during "interpreting" phase. + +@item +Engravers + +Executive branch of Contexts, plugins that create grobs, usually one +engraver per grob type. Created together with context. + +@item +Layout Objects + += grobs + +@item +Grob Interfaces + +These are not C++ classes per se. The idea of a Grob interface hasn't +crystallized well. ATM, an interface is a symbol, with a bunch of grob +properties. They are not objects that are created or destroyed. + +@item +Iterators + +Objects that walk through different music classes, and deliver events +in a synchronized way, so that notes that play together are processed +at the same moment and (as a result) end up on the same horizontal position. + +Created during interpreting phase. + +BTW, the entry point for interpreting is ly:run-translator +(ly_run_translator on the C++ side) + +@end itemize + +@subheading Can you get to Context properties from a Music object? + +You can create music object with a Scheme function that reads context +properties (the \applycontext syntax). However, that function is +executed during Interpreting, so you can not really get Context +properties from Music objects, since music objects are not directly +connected to Contexts. That connection is made by the Music_iterators + +@subheading Can you get to Music properties from a Context object? + +Yes, if you are given the music object within a Context +object. Normally, the music objects enter Contexts in synchronized +fashion, and the synchronization is done by Music_iterators. + +@subheading What is the relationship between C++ classes and Scheme objects? + +Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are +manipulated from C++ as well using the GUILE C function interface +(prefix: scm_) + +@subheading How do Scheme procedures get called from C++ functions? + +scm_call_*, where * is an integer from 0 to 4. +Also scm_c_eval_string (), scm_eval () + +@subheading How do C++ functions get called from Scheme procedures? + +Export a C++ function to Scheme with LY_DEFINE. + +@subheading What is the flow of control in the program? + +Good question. Things used to be clear-cut, but we have Scheme +and SMOBs now, which means that interactions do not follow a very +rigid format anymore. See below for an overview, though. + +@subheading Does the parser make Scheme procedure calls or C++ function calls? + +Both. And the Scheme calls can call C++ and vice versa. It's nested, +with the SCM datatype as lubrication between the interactions + +(I think the word "lubrication" describes the process better than the +traditional word "glue") + +@subheading How do the front-end and back-end get started? + +Front-end: a file is parsed, the rest follows from that. Specifically, + +Parsing leads to a Music + Music_output_def object (see parser.yy, +definition of toplevel_expression ) + +A Music + Music_output_def object leads to a Global_context object (see +ly_run_translator ()) + +During interpreting, Global_context + Music leads to a bunch of +Contexts (see Global_translator::run_iterator_on_me ()). + +After interpreting, Global_context contains a Score_context (which +contains staves, lyrics etc.) as a child. Score_context::get_output () +spews a Music_output object (either a Paper_score object for notation +or Performance object for MIDI). + +The Music_output object is the entry point for the backend (see +ly_render_output ()). + +The main steps of the backend itself are in + +@itemize + +@item +@file{paper-score.cc} , Paper_score::process_ + +@item +@file{system.cc} , System::get_lines() + +@item +The step, where things go from grobs to output, is in +System::get_line(): each grob delivers a Stencil (a Device +independent output description), which is interpreted by our +outputting backends (@file{scm/output-tex.scm} and +@file{scm/output-ps.scm}) to produce TeX and PS. + +@end itemize + +Interactions between grobs and putting things into .tex and .ps files +have gotten a little more complex lately. Jan has implemented +page-breaking, so now the backend also involves Paper_book, +Paper_lines and other things. This area is still heavily in flux, and +perhaps not something you should want to look at. + +@subheading How do the front-end and back-end communicate? + +There is no communication from backend to front-end. From front-end to +backend is simply the program flow: music + definitions gives +contexts, contexts yield output, after processing, output is written +to disk. + +@subheading Where is the functionality associated with KEYWORDs? + +See @file{my-lily-lexer.cc} (keywords, there aren't that many) +and @file{ly/*.ly} (most of the other backslashed @code{/\words} are identifiers) + +@subheading What Contexts/Properties/Music/etc. are available when they are processed? + +What do you mean exactly with this question? + +See @file{ly/engraver-init.ly} for contexts, +see @file{scm/define-*.scm} for other objects. + +@subheading How do you decide if something is a Music, Context, or Grob property? +Why is part-combine-status a Music property when it seems (IMO) +to be related to the Staff context? + +The Music_iterators and Context communicate through two channels + +Music_iterators can set and read context properties, idem for +Engravers and Contexts + +Music_iterators can send "synthetic" music events (which aren't in +the input) to a context. These are caught by Engravers. This is +mostly a one way communication channel. + +part-combine-status is part of such a synthetic event, used by +Part_combine_iterator to communicate with Part_combine_engraver. + + +@subheading Deciding between context and music properties + +I'm adding a property to affect how \autochange works. It seems to +me that it should be a context property, but the Scheme autochange +procedure has a Music argument. Does this mean I should use +a Music property? + +\autochange is one of these extra strange beasts: it requires +look-ahead to decide when to change staves. This is achieved by +running the interpreting step twice (see +@file{scm/part-combiner.scm} , at the bottom), and +storing the result of the first step (where to switch +staves) in a Music property. Since you want to influence that +where-to-switch list, your must affect the code in +make-autochange-music (@file{scm/part-combiner.scm}). +That code is called directly from the parser and there are no +official "parsing properties" yet, so there is no generic way +to tune \autochange. We would have to invent something new +for this, or add a separate argument, + +@example + \autochange #around-central-C ..music.. +@end example + +@noindent +where around-central-C is some function that is called from +make-autochange-music. + +@subheading More on context and music properties + +From Neil Puttock, in response to a question about transposition: + +Context properties (using \set & \unset) are tied to engravers: they +provide information relevant to the generation of graphical objects. + +Since transposition occurs at the music interpretation stage, it has +no direct connection with engravers: the pitch of a note is fixed +before a notehead is created. Consider the following minimal snippet: + +@example +@{ c' @} +@end example + +This generates (simplified) a NoteEvent, with its pitch and duration +as event properties, + +@example +(make-music + 'NoteEvent + 'duration + (ly:make-duration 2 0 1 1) + 'pitch + (ly:make-pitch 0 0 0) +@end example + +which the Note_heads_engraver hears. It passes this information on to +the NoteHead grob it creates from the event, so the head's correct +position and duration-log can be determined once it's ready for +printing. + +If we transpose the snippet, + +@example +\transpose c d @{ c' @} +@end example + +the pitch is changed before it reaches the engraver (in fact, it +happens just after the parsing stage with the creation of a +TransposedMusic music object): + +@example +(make-music + 'NoteEvent + 'duration + (ly:make-duration 2 0 1 1) + 'pitch + (ly:make-pitch 0 1 0) +@end example + +You can see an example of a music property relevant to transposition: +untransposable. + +@example +\transpose c d @{ c'2 \withMusicProperty #'untransposable ##t c' @} +@end example + +-> the second c' remains untransposed. + +Take a look at @file{lily/music.cc} to see where the transposition takes place. + + +@subheading How do I tell about the execution environment? + +I get lost figuring out what environment the code I'm looking at is in when it +executes. I found both the C++ and Scheme autochange code. Then I was trying +to figure out where the code got called from. I finally figured out that the +Scheme procedure was called before the C++ iterator code, but it took me a +while to figure that out, and I still didn't know who did the calling in the +first place. I only know a little bit about Flex and Bison, so reading those +files helped only a little bit. + +@emph{Han-Wen:} GDB can be of help here. Set a breakpoint in C++, and run. When you +hit the breakpoint, do a backtrace. You can inspect Scheme objects +along the way by doing + +@example +p ly_display_scm(obj) +@end example + +this will display OBJ through GUILE. + +@node Music functions and GUILE debugging +@subsection Music functions and GUILE debugging + +Ian Hulin was trying to do some debugging in music functions, and +came up with the following question + +HI all, +I'm working on the Guile Debugger Stuff, and would like to try +debugging a music function definition such as: + +@example +conditionalMark = #(define-music-function (parser location) () +    #@{ \tag #'instrumental-part @{\mark \default@}  #@} ) +@end example + +It appears conditionalMark does not get set up as an +equivalent of a Scheme + +@example +(define conditionalMark = define-music-function(parser location () ... +@end example + +@noindent +although something gets defined because Scheme apparently recognizes + +@example +#(set-break! conditionalMark) +@end example + +@noindent +later on in the file without signalling any Guile errors. + +However the breakpoint trap is never encountered as +define-music-function passed things on to ly:make-music-function, +which is really C++ code ly_make_music_function, so Guile never +finds out about the breakpoint. + +Han-Wen answered as follows: + +You can see the definition by doing + +@example +#(display conditionalMark) +@end example + +noindent +inside the @file{.ly} file. + +The breakpoint failing may have to do with the call sequence. See +@file{parser.yy}, run_music_function(). The function is called directly from +C++, without going through the GUILE evaluator, so I think that is why +there is no debugger trap.