-@c -*- coding: us-ascii; mode: texinfo; -*-
+@c -*- coding: utf-8; mode: texinfo; -*-
@node Programming work
@chapter Programming work
* Finding functions::
* Code style::
* Debugging LilyPond::
+* Tracing object relationships::
* Adding or modifying features::
* Iterator tutorial::
* Engraver tutorial::
* Callback tutorial::
* LilyPond scoping::
+* LilyPond miscellany::
@end menu
@node Overview of LilyPond architecture
@uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's
thesis}.
-
The first stage of LilyPond processing is @emph{parsing}. In the parsing
process, music expressions in LilyPond input format are converted to music
expressions in Scheme format. In Scheme format, a music expression is a list
The final stage of LilyPond processing is @emph{translation}. During
translation, music events are prepared for graphical or midi output. The
-translation step is accomplished by translators or engravers (the distinction
-is unclear).
+translation step is accomplished by the polymorphic base class Translator
+through its two derived classes: Engraver (for graphical output) and
+Performer (for midi output).
-Translators are defined in C++ files named *-engraver.cc. In *-engraver.cc, a
-C++ class of Engraver type is created. The Engraver is also declared as a
-translator. Much of the work of translating is handled by Scheme functions,
+Translators are defined in C++ files named *-engraver.cc and *-performer.cc.
+Much of the work of translating is handled by Scheme functions,
which is one of the keys to LilyPond's exceptional flexibility.
+@sourceimage{architecture-diagram,,,png}
+
@node LilyPond programming languages
@section LilyPond programming languages
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE
Reference Manual}.
+@subsection Flex
+
+The LilyPond lexer is implemented in Flex, an implementation of the Unix lex
+lexical analyser generator. Resources for Flex can be found
+@uref{http://flex.sourceforge.net/, here}.
+
@subsection GNU Bison
The LilyPond parser is implemented in Bison, a GNU parser generator. The
@subsection GUILE or Scheme
-GUILE is the dialect of Scheme that is used as LilyPond's extension language. Many extensions to LilyPond are written entirely in GUILE. The
+GUILE is the dialect of Scheme that is used as LilyPond's extension language.
+Many extensions to LilyPond are written entirely in GUILE. The
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html,
GUILE Reference Manual} is available online.
programming in Scheme is available in its entirety online.
An introduction to Guile/Scheme as used in LilyPond can be found in the
-Learning Manual, see @rlearning{Scheme tutorial}.
+@rextend{Scheme tutorial}.
@subsection MetaFont
syn keyword schemeSyntax define-public define* define-safe-public
syn keyword schemeSyntax use-modules define-module
syn keyword schemeSyntax defmacro-public define-macro
-syn keyword schemeSyntax define-builtin-markup-command
syn keyword schemeSyntax define-markup-command
-syn keyword schemeSyntax define-builtin-markup-list-command
+syn keyword schemeSyntax define-markup-list-command
syn keyword schemeSyntax let-keywords* lambda* define*-public
syn keyword schemeSyntax defmacro* defmacro*-public
" All of the above should influence indenting too
set lw+=define-public,define*,define-safe-public,use-modules,define-module
-set lw+=defmacro-public,define-macro,define-builtin-markup-command
-set lw+=define-markup-command,define-builtin-markup-list-command
+set lw+=defmacro-public,define-macro
+set lw+=define-markup-command,define-markup-list-command
set lw+=let-keywords*,lambda*,define*-public,defmacro*,defmacro*-public
" These forms should not influence indenting
@node Debugging LilyPond
@section Debugging LilyPond
-The most commonly used tool for debugging LilyPond is the GNU debugger
-gdb. Use of gdb is described in this section.
+The most commonly used tool for debugging LilyPond is the GNU
+debugger gdb. The gdb tool is used for investigating and debugging
+core Lilypond code written in C++. Another tool is available for
+debugging Scheme code using the Guile debugger. This section
+describes how to use both gdb and the Guile Debugger.
+@menu
+* Debugging overview::
+* Debugging C++ code::
+* Debugging Scheme code::
+@end menu
+
+@node Debugging overview
@subsection Debugging overview
Using a debugger simplifies troubleshooting in at least two ways.
Then, when execution has paused, debugger commands can be issued to
explore the values of various variables or to execute functions.
-Second, the debugger allows the display of a stack trace, which shows
-the sequence in which functions are called and the arguments to the
-various function calls.
+Second, the debugger can display a stack trace, which shows the
+sequence in which functions have been called and the arguments
+passed to the called functions.
+@node Debugging C++ code
+@subsection Debugging C++ code
-@subsection Compiling with debugging information
+The GNU debugger, gdb, is the principal tool for debugging C++ code.
-In order to use a debugger with LilyPond, it is necessary to compile
+@unnumberedsubsubsec Compiling LilyPond for use with gdb
+
+In order to use gdb with LilyPond, it is necessary to compile
LilyPond with debugging information. This is accomplished by running
the following commands in the main LilyPond source directory.
@example
./configure --disable-optimising
-
make
@end example
-This will create a version of LilyPond that contains the debugging
+This will create a version of LilyPond containing debugging
information that will allow the debugger to tie the source code
to the compiled code.
You should not do @var{make install} if you want to use a debugger
-with LilyPond. @var{make install} will strip the debugging information
-from the LilyPond binary.
+with LilyPond. The @var{make install} command will strip debugging
+information from the LilyPond binary.
-To set breakpoints in Scheme functions, put
+@unnumberedsubsubsec Typical gdb usage
+
+Once you have compiled the Lilypond image with the necessary
+debugging information it will have been written to a location in a
+subfolder of your current working directory:
@example
-\include "guile-debugger.ly"
+out/bin/lilypond
@end example
-in your input file after any scheme procedures you have defined in
-that file. When your input file is processed, a guile prompt
-will be displayed. At the guile prompt, you can set breakpoints with
-the @code{break!} procedure:
+This is important as you will need to let gdb know where to find the
+image containing the symbol tables. You can invoke gdb from the
+command line using
@example
-guile> (break! my-scheme-procedure)
+gdb out/bin/lilypond
@end example
-
-Once you have set the desired breakpoints, you exit the guile repl frame
-by typing:
+@noindent
+This loads the LilyPond symbol tables into gdb. Then, to run
+LilyPond on @code{test.ly} under the debugger, enter
@example
-guile> (quit)
+run test.ly
@end example
+@noindent
+at the gdb prompt.
-When one of the scheme routines for which you have set breakpoints is
-entered, guile will interrupt execution in a debug frame. At this point,
-you will have access to guile debugging commands. For a listing of these
-commands, type:
+As an alternative to running gdb at the command line you may try
+a graphical interface to gdb such as ddd
@example
-debug> help
+ddd out/bin/lilypond
@end example
-@subsection Typical gdb usage
+You can also use sets of standard gdb commands stored in a .gdbinit
+file (see next section).
-@subsection Typical .gdbinit files
+@unnumberedsubsubsec Typical .gdbinit files
-The behavior of gdb can be readily customized through the use of
-@var{.gdbinit} files. A @var{.gdbinit} file is a file named
+The behavior of gdb can be readily customized through the use of a
+@var{.gdbinit} file. A @var{.gdbinit} file is a file named
@var{.gdbinit} (notice the @qq{.} at the beginning of the file name)
that is placed in a user's home directory.
end
@end example
-@subsection Using Guile interactively with LilyPond
+@node Debugging Scheme code
+@subsection Debugging Scheme code
+
+Scheme code can be developed using the Guile command line
+interpreter @code{top-repl}. You can either investigate
+interactively using just Guile or you can use the debugging
+tools available within Guile.
+
+@unnumberedsubsubsec Using Guile interactively with LilyPond
In order to experiment with Scheme programming in the LilyPond
-environment, it is convenient to have a Guile interpreter that
+environment, it is necessary to have a Guile interpreter that
has all the LilyPond modules loaded. This requires the following
steps.
-First, define a Scheme symbol for the active module
-in the .ly file:
+First, define a Scheme symbol for the active module in the .ly file:
@example
#(module-define! (resolve-module '(guile-user))
- 'lilypond-module (current-module))
+ 'lilypond-module (current-module))
@end example
-Second, place a Scheme function in the .ly file that gives an interactive Guile
-prompt:
+Now place a Scheme function in the .ly file that gives an
+interactive Guile prompt:
@example
#(top-repl)
@end example
-When the .ly file is compiled, this causes the compilation to be interrupted
-and an interactive guile prompt to appear. When the guile prompt appears,
-the LilyPond active module must be set as the current guile module:
+When the .ly file is compiled, this causes the compilation to be
+interrupted and an interactive guile prompt to appear. Once the
+guile prompt appears, the LilyPond active module must be set as the
+current guile module:
@example
guile> (set-current-module lilypond-module)
@end example
-Proper operation of these commands can be demonstrated by typing the name
-of a LilyPond public scheme function to see if it's properly defined:
+You can demonstrate these commands are operating properly by typing the name
+of a LilyPond public scheme function to check it has been defined:
@example
guile> fret-diagram-verbose-markup
ABORT: (unbound-variable)
@end example
-Once the module is properly loaded, any valid LilyPond Scheme expression
-can be entered at the interactive prompt.
+Once the module is properly loaded, any valid LilyPond Scheme
+expression can be entered at the interactive prompt.
-After the investigation is complete, the interactive guile interpreter
-can be exited:
+After the investigation is complete, the interactive guile
+interpreter can be exited:
@example
guile> (quit)
The compilation of the .ly file will then continue.
+@unnumberedsubsubsec Using the Guile debugger
+
+To set breakpoints and/or enable tracing in Scheme functions, put
+
+@example
+\include "guile-debugger.ly"
+@end example
+
+in your input file after any scheme procedures you have defined in
+that file. This will invoke the Guile command-line after having set
+up the environment for the debug command-line. When your input file
+is processed, a guile prompt will be displayed. You may now enter
+commands to set up breakpoints and enable tracing by the Guile debugger.
+
+@unnumberedsubsubsec Using breakpoints
+
+At the guile prompt, you can set breakpoints with
+the @code{set-break!} procedure:
+
+@example
+guile> (set-break! my-scheme-procedure)
+@end example
+
+Once you have set the desired breakpoints, you exit the guile repl frame
+by typing:
+
+@example
+guile> (quit)
+@end example
+
+Then, when one of the scheme routines for which you have set
+breakpoints is entered, guile will interrupt execution in a debug
+frame. At this point you will have access to Guile debugging
+commands. For a listing of these commands, type:
+
+@example
+debug> help
+@end example
+
+Alternatively you may code the breakpoints in your Lilypond source
+file using a command such as:
+
+@example
+#(set-break! my-scheme-procedure)
+@end example
+
+immediately after the @code{\include} statement. In this case the
+breakpoint will be set straight after you enter the @code{(quit)}
+command at the guile prompt.
+
+Embedding breakpoint commands like this is particularly useful if
+you want to look at how the Scheme procedures in the @var{.scm}
+files supplied with LilyPond work. To do this, edit the file in
+the relevant directory to add this line near the top:
+
+@example
+(use-modules (scm guile-debugger))
+@end example
+
+Now you can set a breakpoint after the procedure you are interested
+in has been declared. For example, if you are working on routines
+called by @var{print-book-with} in @var{lily-library.scm}:
+
+@example
+(define (print-book-with parser book process-procedure)
+ (let* ((paper (ly:parser-lookup parser '$defaultpaper))
+ (layout (ly:parser-lookup parser '$defaultlayout))
+ (outfile-name (get-outfile-name parser)))
+ (process-procedure book paper layout outfile-name)))
+
+(define-public (print-book-with-defaults parser book)
+ (print-book-with parser book ly:book-process))
+
+(define-public (print-book-with-defaults-as-systems parser book)
+ (print-book-with parser book ly:book-process-to-systems))
+
+@end example
+
+At this point in the code you could add this to set a breakpoint at
+print-book-with:
+
+@example
+(set-break! print-book-with)
+@end example
+
+@unnumberedsubsubsec Tracing procedure calls and evaluator steps
+
+Two forms of trace are available:
+
+@example
+(set-trace-call! my-scheme-procedure)
+@end example
+
+and
+
+@example
+(set-trace-subtree! my-scheme-procedure)
+@end example
+
+@code{set-trace-call!} causes Scheme to log a line to the standard
+output to show when the procedure is called and when it exits.
+
+@code{set-trace-subtree!} traces every step the Scheme evaluator
+performs in evaluating the procedure.
+
+@node Tracing object relationships
+@section Tracing object relationships
+
+Understanding the LilyPond source often boils down to figuring out what
+is happening to the Grobs. Where (and why) are they being created,
+modified and destroyed? Tracing Lily through a debugger in order to
+identify these relationships can be time-consuming and tedious.
+
+In order to simplify this process, a facility has been added to
+display the grobs that are created and the properties that are set
+and modified. Although it can be complex to get set up, once set up
+it easily provides detailed information about the life of grobs
+in the form of a network graph.
+
+Each of the steps necessary to use the graphviz utility
+is described below.
+
+@enumerate
+
+@item Installing graphviz
+
+In order to create the graph of the object relationships, it is
+first necessary to install Graphviz. graphviz is available for a
+number of different platforms:
+
+@example
+@uref{http://www.graphviz.org/Download..php}
+@end example
+
+@item Modifying config.make
+
+In order for the Graphviz tool to work, config.make must be modified.
+It is probably a good idea to first save a copy of config.make under
+a different name. Then, edit config.make by removing every occurence
+of @code{-DNDEBUG}.
+
+@item Rebuilding LilyPond
+
+The executable code of LilyPond must be rebuilt from scratch:
+
+@example
+make -C lily clean && make -C lily
+@end example
+
+@item Create a graphviz-compatible .ly file
+
+In order to use the graphviz utility, the .ly file must include
+@file{ly/graphviz-init.ly}, and should then specify the
+grobs and symbols that should be tracked. An example of this
+is found in @file{input/regression/graphviz.ly}.
+
+@item Run lilypond with output sent to a log file
+
+The Graphviz data is sent to stderr by lilypond, so it is
+necessary to redirect stderr to a logfile:
+
+@example
+lilypond graphviz.ly 2> graphviz.log
+@end example
+
+@item Edit the logfile
+
+The logfile has standard lilypond output, as well as the Graphviz
+output data. Delete everything from the beginning of the file
+up to but not including the first occurence of @code{digraph}.
+
+@item Process the logfile with @code{dot}
+
+The directed graph is created from the log file with the program
+@code{dot}:
+
+@example
+dot -Tpdf graphviz.log > graphviz.pdf
+@end example
+
+@end enumerate
+
+The pdf file can then be viewed with any pdf viewer.
+
+When compiled without @code{-DNDEBUG}, lilypond may run slower
+than normal. The original configuration can be restored by either
+renaming the saved copy of @code{config.make} or rerunning
+@code{configure}. Then rebuild lilypond with
+
+@example
+make -C lily clean && make -C lily
+@end example
+
+
@node Adding or modifying features
@section Adding or modifying features
@example
\markup @{
- "This snippet is deprecated as of version X.Y.Z and
- will be removed from the documentation."
+ This snippet is deprecated as of version X.Y.Z and
+ will be removed from the documentation.
@}
@end example
Update the snippet files by running:
@example
-scripts\auxiliar\makelsr.py
+scripts/auxiliar/makelsr.py
@end example
Where the convert-ly rule is not able to automatically update regression
the feature changes to be implemented. This is especially important
for changes that change input file syntax.
-Hints for changes.tely entries are given at the top changes.tely.
+Hints for changes.tely entries are given at the top of the file.
New entries in changes.tely go at the top of the file.
@subsection Post patch for comments
For any change other than a minor change, a patch set should be
-posted on Rietveld for comment. This requires the use of an
-external package, git-cl, and an email account on Google.
+posted on @uref{http://codereview.appspot.com/, Rietveld} for comment.
+This requires the use of an external package, git-cl, and an email
+account on Google.
git-cl is installed by:
configured by entering the command
@example
-git-cl config
+git cl config
@end example
@noindent
check out branch with the changes and enter the command:
@example
-git-cl upload <reference SHA1 ID>
+git cl upload <reference SHA1 ID>
@end example
@noindent
SHA1 ID of origin/master, and in that case the command
@example
-git-cl upload origin/master
+git cl upload origin/master
@end example
@noindent
If the changes were in response to a feature request on the Google
issue tracker for LilyPond, the author should change the status to
-@q{Fixed_x_y_z} where the patch was fixed in version x.y.z. If
+Fixed and a tag @q{fixed_x_y_z} should be added, where the patch was
+fixed in version x.y.z. If
the author does not have privileges to change the status, an email
should be sent to bug-lilypond requesting the BugMeister to change
the status.
@node Iterator tutorial
@section Iterator tutorial
-FIXME -- this is a placeholder for a tutorial on iterators
+TODO -- this is a placeholder for a tutorial on iterators
Iterators are routines written in C++ that process music expressions
and sent the music events to the appropriate engravers and/or
@node Engraver tutorial
@section Engraver tutorial
-FIXME -- This is a placeholder for a tutorial on how engravers work.
-
Engravers are C++ classes that catch music events and
-create the appropriate grobs for display on the page. Each different
-type of grob has its own engraver.
+create the appropriate grobs for display on the page. Though the
+majority of engravers are responsible for the creation of a single grob,
+in some cases (e.g. @code{New_fingering_engraver}), several different grobs
+may be created.
-A typical engraver has protected functions including some or all
-of the following:
+Engravers listen for events and acknowledge grobs. Events are passed to
+the engraver in time-step order during the iteration phase. Grobs are
+made available to the engraver when they are created by other engravers
+during the iteration phase.
+
+@subsection Useful methods for information processing
+
+An engraver inherits the following public methods from the Translator
+base class, which can be used to process listened events and acknowledged
+grobs:
@itemize
-@item @code{start_translation_timestep ()}
-@item @code{process_music ()}
-@item @code{stop_translation_timestep ()}
-@item @code{derived_mark ()}
-@item @code{try_music ()}
-@item @code{finalize ()}
+@item @code{virtual void initialize ()}
+@item @code{void start_translation_timestep ()}
+@item @code{void process_music ()}
+@item @code{void process_acknowledged ()}
+@item @code{void stop_translation_timestep ()}
+@item @code{virtual void finalize ()}
@end itemize
-There are also protected functions that are specific to particular
-engraver, as needed by the engraving process.
+These methods are listed in order of translation time, with
+@code{initialize ()} and @code{finalize ()} bookending the whole
+process. @code{initialize ()} can be used for one-time initialization
+of context properties before translation starts, whereas
+@code{finalize ()} is often used to tie up loose ends at the end of
+translation: for example, an unterminated spanner might be completed
+automatically or reported with a warning message.
+
+@subsection Translation process
+
+At each timestep in the music, translation proceeds by calling the
+following methods in turn:
-External interfaces to to the engraver are implemented by protected
+@code{start_translation_timestep ()} is called before any user
+information enters the translators, i.e., no property operations
+(\set, \override, etc.) or events have been processed yet.
+
+@code{process_music ()} and @code{process_acknowledged ()} are called
+after all events in the current time step have been heard, or all
+grobs in the current time step have been acknowledged. The latter
+tends to be used exclusively with engravers which only acknowledge
+grobs, whereas the former is the default method for main processing
+within engravers.
+
+@code{stop_translation_timestep ()} is called after all user
+information has been processed prior to beginning the translation for
+the next timestep.
+
+@subsection Preventing garbage collection for SCM member variables
+
+In certain cases, an engraver might need to ensure private Scheme
+variables (with type SCM) do not get swept away by Guile's garbage
+collector: for example, a cache of the previous key signature which
+must persist between timesteps. The method
+@code{virtual derived_mark () const} can be used in such cases:
+
+@example
+Engraver_name::derived_mark ()
+@{
+ scm_gc_mark (private_scm_member_)
+@}
+@end example
+
+
+@subsection Listening to music events
+
+External interfaces to the engraver are implemented by protected
macros including one or more of the following:
@itemize
-@item @code{DECLARE_TRANSLATOR_LISTENER (event)}
-@item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event)}
+@item @code{DECLARE_TRANSLATOR_LISTENER (event_name)}
+@item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)}
@end itemize
@noindent
-where @var{event} is the type of event required to provide the input
-the engraver needs and @code{Engraver_name} is the name of the
-engraver. These macros set up the mechanism and declare the
-methods for passing the information in events of this kind to the
+where @var{event_name} is the type of event required to provide the
+input the engraver needs and @var{Engraver_name} is the name of the
engraver.
+Following declaration of a listener, the method is implemented as follows:
+
+@example
+IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)
+void
+Engraver_name::listen_event_name (Stream event *event)
+@{
+ ...body of listener method...
+@}
+@end example
+
+@subsection Acknowledging grobs
+
Some engravers also need information from grobs as they are created
and as they terminate. The mechanism and methods to obtain this
information are set up by the macros:
@itemize
-@item @code{DECLARE_ACKNOWLEDGER (grob)}
-@item @code{DECLARE_END_ACKNOWLEDGER (grob)}
+@item @code{DECLARE_ACKNOWLEDGER (grob_interface)}
+@item @code{DECLARE_END_ACKNOWLEDGER (grob_interface)}
@end itemize
-where @var{grob} is the type of grob of interest.
+where @var{grob_interface} is an interface supported by the
+grob(s) which should be acknowledged. For example, the following
+code would declare acknowledgers for a @code{NoteHead} grob (via the
+@code{note-head-interface}) and any grobs which support the
+@code{side-position-interface}:
+
+@example
+@code{DECLARE_ACKNOWLEDGER (note_head)}
+@code{DECLARE_ACKNOWLEDGER (side_position)}
+@end example
+
+The @code{DECLARE_END_ACKNOWLEDGER ()} macro sets up a spanner-specific
+acknowledger which will be called whenever a spanner ends.
+
+Following declaration of an acknowledger, the method is coded as follows:
+
+@example
+void
+Engraver_name::acknowledge_interface_name (Grob_info info)
+@{
+ ...body of acknowledger method...
+@}
+@end example
-An engraver will also generally have a public macro
+@subsection Engraver declaration/documentation
+
+An engraver must have a public macro
@itemize
@item @code{TRANSLATOR_DECLARATIONS (Engraver_name)}
Internals Reference:
@itemize
-@item @code{ADD_ACKNOWLEDGER (Engraver_name, grob)}
+@item @code{ADD_ACKNOWLEDGER (Engraver_name, grob_interface)}
@item @code{ADD_TRANSLATOR (Engraver_name, Engraver_doc,
Engraver_creates, Engraver_reads, Engraver_writes)}
@end itemize
@noindent
-where @code{Engraver_name} is the name of the engraver, @code{grob}
-is the name of those grobs that will be acknowledged,
+where @code{Engraver_name} is the name of the engraver, @code{grob_interface}
+is the name of the interface that will be acknowledged,
@code{Engraver_doc} is a docstring for the engraver,
-@code{Engraver_creates} is the grob created by the engraver,
+@code{Engraver_creates} is the set of grobs created by the engraver,
@code{Engraver_reads} is the set of properties read by the engraver,
and @code{Engraver_writes} is the set of properties written by
the engraver.
@node Callback tutorial
@section Callback tutorial
-FIXME -- This is a placeholder for a tutorial on callback functions.
+TODO -- This is a placeholder for a tutorial on callback functions.
@node LilyPond scoping
@section LilyPond scoping
@end example
@noindent with @code{\paper}, @code{\midi} and @code{\header} being
-nested scope inside the .ly file-level scope. @w{@code{foo = 1}} is
-translated in to a scheme variable definition.
+nested scope inside the @file{.ly} file-level scope. @w{@code{foo = 1}}
+is translated in to a scheme variable definition.
This implemented using modules, with each scope being an anonymous
module that imports its enclosing scope's module.
-The reason to put some functions (@qq{builtin}) outside the .ly level,
-is that in case of
+Lilypond's core, loaded from @file{.scm} files, is usually placed in the
+@code{lily} module, outside the @file{.ly} level. In the case of
@example
lilypond a.ly b.ly
@end example
@noindent
-we want to reuse the built-in definitions, without changes
-effected in a.ly leaking into the processing of b.ly.
+we want to reuse the built-in definitions, without changes effected in
+user-level @file{a.ly} leaking into the processing of @file{b.ly}.
+
+The user-accessible definition commands have to take care to avoid
+memory leaks that could occur when running multiple files. All
+information belonging to user-defined commands and markups is stored in
+a manner that allows it to be garbage-collected when the module is
+dispersed, either by being stored module-locally, or in weak hash
+tables.
+
+@node LilyPond miscellany
+@section LilyPond miscellany
+
+This is a place to dump information that may be of use to developers
+but doesn't yet have a proper home. Ideally, the length of this section
+would become zero as items are moved to other homes.
+
+@subsection Spacing algorithms
+
+Here is information from an email exchange about spacing algorithms.
+
+On Thu, 2010-02-04 at 15:33 -0500, Boris Shingarov wrote:
+I am experimenting with some modifications to the line breaking code,
+and I am stuck trying to understand how some of it works. So far my
+understanding is that Simple_spacer operates on a vector of Grobs, and
+it is a well-known Constrained-QP problem (rods = constraints, springs
+= quadratic function to minimize). What I don't understand is, if the
+spacer operates at the level of Grobs, which are built at an earlier
+stage in the pipeline, how are the changes necessitated by differences
+in line breaking, taken into account? in other words, if I take the
+last measure of a line and place it on the next line, it is not just a
+matter of literally moving that graphic to where the start of the next
+line is, but I also need to draw a clef, key signature, and possibly
+other fundamental things -- but at that stage in the rendering
+pipeline, is it not too late??
+
+Joe Neeman answered:
+
+We create lots of extra grobs (eg. a BarNumber at every bar line) but
+most of them are not drawn. See the break-visibility property in
+item-interface.
+
+@subsection Info from Han-Wen Email
+
+In 2004, Douglas Linhardt decided to try starting a document that would
+explain LilyPond architecture and design principles. The material below
+is extracted from that email, which can be found at
+@uref{http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/2992}.
+The headings reflect questions from Doug or comments from Han-Wen;
+the body text are Han-Wen's answers.
+
+@unnumberedsubsubsec Figuring out how things work.
+
+I must admit that when I want to know how a program works, I use grep
+and emacs and dive into the source code. The comments and the code
+itself are usually more revealing than technical documents.
+
+@unnumberedsubsubsec What's a grob, and how is one used?
+
+Graphical object - they are created from within engravers, either as
+Spanners (derived class) -slurs, beams- or Items (also a derived
+class) -notes, clefs, etc.
+
+There are two other derived classes System (derived from Spanner,
+contaning a "line of music") and Paper_column (derived from Item, it
+contains all items that happen at the same moment). They are separate
+classes because they play a special role in the linebreaking process.
+
+@unnumberedsubsubsec What's a smob, and how is one used?
+
+A C(++) object that is encapsulated so it can be used as a Scheme
+object. See GUILE info, "19.3 Defining New Types (Smobs)"
+
+@unnumberedsubsubsec When is each C++ class constructed and used
+
+@itemize
+
+@item
+Music classes
+
+In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME().
+
+@item
+Contexts
+
+Constructed during "interpreting" phase.
+
+@item
+Engravers
+
+Executive branch of Contexts, plugins that create grobs, usually one
+engraver per grob type. Created together with context.
+
+@item
+Layout Objects
+
+= grobs
+
+@item
+Grob Interfaces
+
+These are not C++ classes per se. The idea of a Grob interface hasn't
+crystallized well. ATM, an interface is a symbol, with a bunch of grob
+properties. They are not objects that are created or destroyed.
+
+@item
+Iterators
+
+Objects that walk through different music classes, and deliver events
+in a synchronized way, so that notes that play together are processed
+at the same moment and (as a result) end up on the same horizontal position.
+
+Created during interpreting phase.
+
+BTW, the entry point for interpreting is ly:run-translator
+(ly_run_translator on the C++ side)
+
+@end itemize
+
+@unnumberedsubsubsec Can you get to Context properties from a Music object?
+
+You can create music object with a Scheme function that reads context
+properties (the \applycontext syntax). However, that function is
+executed during Interpreting, so you can not really get Context
+properties from Music objects, since music objects are not directly
+connected to Contexts. That connection is made by the Music_iterators
+
+@unnumberedsubsubsec Can you get to Music properties from a Context object?
+
+Yes, if you are given the music object within a Context
+object. Normally, the music objects enter Contexts in synchronized
+fashion, and the synchronization is done by Music_iterators.
+
+@unnumberedsubsubsec What is the relationship between C++ classes and Scheme objects?
+
+Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are
+manipulated from C++ as well using the GUILE C function interface
+(prefix: scm_)
+
+@unnumberedsubsubsec How do Scheme procedures get called from C++ functions?
+
+scm_call_*, where * is an integer from 0 to 4.
+Also scm_c_eval_string (), scm_eval ()
+
+@unnumberedsubsubsec How do C++ functions get called from Scheme procedures?
+
+Export a C++ function to Scheme with LY_DEFINE.
+
+@unnumberedsubsubsec What is the flow of control in the program?
+
+Good question. Things used to be clear-cut, but we have Scheme
+and SMOBs now, which means that interactions do not follow a very
+rigid format anymore. See below for an overview, though.
-Maintaining this scoping when one .ly file can be included in another
-.ly file can be challenging. A @code{define-public-toplevel} macro
-has been created in order to handle a difficulty caused by the modules
-being not the same when a .ly file is included into another.
-This provided a way to define all markup commands in the same module.
-At this time, we have found no easier way to define a function in a given
-module (not the current one) than to define this macro.
+@unnumberedsubsubsec Does the parser make Scheme procedure calls or C++ function calls?
-With this architecture, the guile module system is not bypassed:
-module-define!, module-export! and module-ref are all guile module
-primitives.
+Both. And the Scheme calls can call C++ and vice versa. It's nested,
+with the SCM datatype as lubrication between the interactions
-A second reason for using this current architecture is to avoid memory
-leaks that could occur when running multiple files if toplevel
-functions were registered permanently.
+(I think the word "lubrication" describes the process better than the
+traditional word "glue")
+
+@unnumberedsubsubsec How do the front-end and back-end get started?
+
+Front-end: a file is parsed, the rest follows from that. Specifically,
+
+Parsing leads to a Music + Music_output_def object (see parser.yy,
+definition of toplevel_expression )
+
+A Music + Music_output_def object leads to a Global_context object (see
+ly_run_translator ())
+
+During interpreting, Global_context + Music leads to a bunch of
+Contexts. (see Global_translator::run_iterator_on_me () )
+
+After interpreting, Global_context contains a Score_context (which
+contains staves, lyrics etc.) as a child. Score_context::get_output ()
+spews a Music_output object (either a Paper_score object for notation
+or Performance object for MIDI).
+
+The Music_output object is the entry point for the backend. (see
+ly_render_output () )
+
+The main steps of the backend itself are in
+
+@itemize
+
+@item
+paper-score.cc , Paper_score::process_
+
+@item
+system.cc , System::get_lines()
+
+@item
+The step, where things go from grobs to output, is in
+System::get_line(): each grob delivers a Stencil (a Device
+independent output description), which is interpreted by our
+outputting backends (scm/output-tex.scm and scm/output-ps.scm)
+to produce TeX and PS.
+
+@end itemize
+
+Interactions between grobs and putting things into .tex and .ps files
+have gotten a little more complex lately. Jan has implemented
+page-breaking, so now the backend also involves Paper_book,
+Paper_lines and other things. This area is still heavily in flux, and
+perhaps not something you should want to look at.
+
+@unnumberedsubsubsec How do the front-end and back-end communicate?
+
+There is no communication from backend to front-end. From front-end to
+backend is simply the program flow: music + definitions gives
+contexts, contexts yield output, after processing, output is written
+to disk.
+
+@unnumberedsubsubsec Where is the functionality associated with KEYWORDs?
+
+See my-lily-lexer.cc (keywords, there aren't that many) and ly/*.ly
+(most of the other backslashed \words are identifiers)
+
+@unnumberedsubsubsec What Contexts/Properties/Music/etc. are available when they are processed?
+
+What do you mean exactly with this question?
+
+See ly/engraver-init.ly for contexts, see scm/define-*.scm for other
+objects.
+
+@unnumberedsubsubsec How do you decide if something is a Music, Context, or Grob property?
+Why is part-combine-status a Music property when it seems (IMO)
+to be related to the Staff context?
+
+The Music_iterators and Context communicate through two channels
+
+Music_iterators can set and read context properties, idem for
+Engravers and Contexts
+
+Music_iterators can send "synthetic" music events (which aren't in
+the input) to a context. These are caught by Engravers. This is
+mostly a one way communication channel.
+
+part-combine-status is part of such a synthetic event, used by
+Part_combine_iterator to communicate with Part_combine_engraver.
+
+
+@unnumberedsubsubsec I'm adding a property to affect how \autochange works. It seems to
+me that it should be a context property, but the Scheme autochange
+procecure has a Music argument. Does this mean I should use
+a Music property?
+
+\autochange is one of these extra strange beasts: it requires
+look-ahead to decide when to change staves. This is achieved by
+running the interpreting step twice (see scm/part-combiner.scm , at
+the bottom), and storing the result of the first step (where to switch
+staves) in a Music property. Since you want to influence that
+where-to-switch list, your must affect the code in
+make-autochange-music (scm/part-combiner.scm). That code is called
+directly from the parser and there are no official "parsing
+properties" yet, so there is no generic way to tune \autochange. We
+would have to invent something new for this, or add a separate
+argument,
+
+@example
+ \autochange #around-central-C ..music..
+@end example
+
+@noindent
+where around-central-C is some function that is called from
+make-autochange-music.
+
+@unnumberedsubsubsec I get lost figuring out what environment the code I'm looking at is in when it executes.
+I found both the C++ and Scheme autochange code. Then I was
+trying to figure out where the code got called from. I finally figured out that
+the Scheme procedure was called before the C++ iterator code, but it took me a
+while to figure that out, and I still didn't know who did the calling in the
+first place. I only know a little bit about Flex and Bison, so reading those
+files helped only a little bit.
+
+@emph{Han-Wen:} GDB can be of help here. Set a breakpoint in C++, and run. When you
+hit the breakpoint, do a backtrace. You can inspect Scheme objects
+along the way by doing
+
+@example
+p ly_display_scm(obj)
+@end example
+
+this will display OBJ through GUILE.
+
+@subsection Music functions and GUILE debugging
+
+Ian Hulin was trying to do some debugging in music functions, and
+came up with the following question
+
+HI all,
+I'm working on the Guile Debugger Stuff, and would like to try
+debugging a music function definition such as:
+
+@example
+conditionalMark = #(define-music-function (parser location) ()
+ #@{ \tag #'instrumental-part @{\mark \default@} #@} )
+@end example
+
+It appears conditionalMark does not get set up as an
+equivalent of a Scheme
+
+@example
+(define conditionalMark = define-music-function(parser location () ...
+@end example
+
+@noindent
+although something gets defined because Scheme apparently recognizes
+
+@example
+#(set-break! conditionalMark)
+@end example
+
+@noindent
+later on in the file without signalling any Guile errors.
+
+However the breakpoint trap is never encountered as
+define-music-function passed things on to ly:make-music-function,
+which is really C++ code ly_make_music_function, so Guile never
+finds out about the breakpoint.
+
+Han-Wen answered as follows:
+
+You can see the defintion by doing
+
+@example
+#(display conditionalMark)
+@end example
+noindent
+inside the .ly file.
+The breakpoint failing may have to do with the call sequence. See
+parser.yy, run_music_function(). The function is called directly from
+C++, without going through the GUILE evaluator, so I think that is why
+there is no debugger trap.