* Code style::
* Debugging LilyPond::
* Adding or modifying features::
+* Iterator tutorial::
+* Engraver tutorial::
+* Callback tutorial::
+* LilyPond scoping::
@end menu
@node Overview of LilyPond architecture
The final stage of LilyPond processing is @emph{translation}. During
translation, music events are prepared for graphical or midi output. The
-translation step is accomplished by translators or engravers (the distinction
-is unclear).
+translation step is accomplished by the polymorphic base class Translator
+through its two derived classes: Engraver (for graphical output) and
+Performer (for midi output).
-Translators are defined in C++ files named *-engraver.cc. In *-engraver.cc, a
-C++ class of Engraver type is created. The Engraver is also declared as a
-translator. Much of the work of translating is handled by Scheme functions,
+Translators are defined in C++ files named *-engraver.cc and *-performer.cc.
+Much of the work of translating is handled by Scheme functions,
which is one of the keys to LilyPond's exceptional flexibility.
+@sourceimage{architecture-diagram,,,png}
+
@node LilyPond programming languages
@section LilyPond programming languages
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE
Reference Manual}.
+@subsection Flex
+
+The LilyPond lexer is implemented in Flex, an implementation of the Unix lex
+lexical analyser generator. Resources for Flex can be found
+@uref{http://flex.sourceforge.net/, here}.
+
@subsection GNU Bison
The LilyPond parser is implemented in Bison, a GNU parser generator. The
@subsection GUILE or Scheme
-GUILE is the dialect of Scheme that is used as LilyPond's extension language. Many extensions to LilyPond are written entirely in GUILE. The
+GUILE is the dialect of Scheme that is used as LilyPond's extension language.
+Many extensions to LilyPond are written entirely in GUILE. The
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html,
GUILE Reference Manual} is available online.
@node Code style
@section Code style
-@c email to wl@gnu.org when I get here.
-
-@warning{this is pending some confirmation on -devel. July 2009 -gp}
-
-Command-line script to format stuff with emacs:
-
-@example
-#!/bin/sh
-emacs $1 -batch --eval '(indent-region (point-min) (point-max) nil)' -f save-buffer
-@end example
-
-(that's all on one line)
-
-Save it as a shell script, then run on the file(s) you modified.
@menu
@end menu
))
@end verbatim
+Some source files may not currently have proper indenting. If this
+is the case, it is desirable to fix the improper indenting when the
+file is modified, with the hope of continually improving the code.
+
+@subsection Indenting files with emacs in script mode
+
+@c email to wl@gnu.org when I get here.
+
+@warning{this is pending some confirmation on -devel. July 2009 -gp}
+
+Command-line script to format stuff with emacs:
+
+@example
+#!/bin/sh
+emacs $1 -batch --eval '(indent-region (point-min) (point-max) nil)' -f save-buffer
+@end example
+
+(that's all on one line)
+
+Save it as a shell script, then run on the file(s) you modified.
+
+@subsection Indenting with vim
+
+Although emacs indentation is the LilyPond standard, acceptable
+indentation can usually be accomplished with vim. Some hints for
+vim are as follows:
+
+A workable .vimrc:
+
+@verbatim
+set cindent
+set smartindent
+set autoindent
+set expandtab
+set softtabstop=2
+set shiftwidth=2
+filetype plugin indent on
+set incsearch
+set ignorecase smartcase
+set hlsearch
+set confirm
+set statusline=%F%m%r%h%w\ %{&ff}\ %Y\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ %04l,%04v\ %p%%\ [LEN=%L]
+set laststatus=2
+set number
+" Remove trailing whitespace on write
+autocmd BufWritePre * :%s/\s\+$//e
+@end verbatim
+
+With this .vimrc, files can be reindented automatically by highlihting
+the lines to be indented in visual mode (use V to enter visual mode)
+and pressing =.
+
+A scheme.vim file will help improve the indentation. This one
+was suggested by Patrick McCarty. It should be saved in
+~/.vim/after/syntax/scheme.vim.
+
+@verbatim
+" Additional Guile-specific 'forms'
+syn keyword schemeSyntax define-public define* define-safe-public
+syn keyword schemeSyntax use-modules define-module
+syn keyword schemeSyntax defmacro-public define-macro
+syn keyword schemeSyntax define-markup-command
+syn keyword schemeSyntax define-markup-list-command
+syn keyword schemeSyntax let-keywords* lambda* define*-public
+syn keyword schemeSyntax defmacro* defmacro*-public
+
+" All of the above should influence indenting too
+set lw+=define-public,define*,define-safe-public,use-modules,define-module
+set lw+=defmacro-public,define-macro
+set lw+=define-markup-command,define-markup-list-command
+set lw+=let-keywords*,lambda*,define*-public,defmacro*,defmacro*-public
+
+" These forms should not influence indenting
+set lw-=if
+set lw-=set!
+
+" Try to highlight all ly: procedures
+syn match schemeFunc "ly:[^) ]\+"
+@end verbatim
@subsection Classes and Types
@subsection Naming
+Variable names should be complete words, rather than abbreviations.
+For example, it is preferred to use @code{thickness} rather than
+@code{th} or @code{t}.
+
+Multi-word variable names in C++ should have the words separated
+by the underscore character (@q{_}).
+
+Multi-word variable names in Scheme should have the words separated
+by a hyphen (@q{-}).
+
+@subsection Comments
+
+Comments may not be needed if descriptive variable names are used
+in the code and the logic is straightforward. However, if the
+logic is difficult to follow, and particularly if non-obvious
+code has been included to resolve a bug, a comment describing
+the logic and/or the need for the non-obvious code should be included.
+
+There are instances where the current code could be commented better.
+If significant time is required to understand the code as part of
+preparing a patch, it would be wise to add comments reflecting your
+understanding to make future work easier.
@subsection Messages
for localization). This includes warning and error messages.
@item
-Don't localize/gettextify:
+Do not localize/gettextify:
@itemize
@item
@end itemize
@item
-Messages to be localised must be encapsulated in `_ (STRING)' or
+Messages to be localized must be encapsulated in `_ (STRING)' or
`_f (FORMAT, ...)'. E.g.:
@example
@end example
@item
-Do not modularise too much; words frequently cannot be translated
+Do not modularize too much; words frequently cannot be translated
without context. It is probably safe to treat most occurences of
words like stem, beam, crescendo as separately translatable words.
@example
#(module-define! (resolve-module '(guile-user))
- 'lilypond-module (current-module))
+ 'lilypond-module (current-module))
@end example
Second, place a Scheme function in the .ly file that gives an interactive Guile
When a new feature is to be added to LilyPond, it is necessary to
ensure that the feature is properly integrated to maintain
its long-term support. This section describes the steps necessary
-for feature addition.
+for feature addition and modification.
@subsection Write the code
-You should create a new git branch for writing the code, as that
+You should probably create a new git branch for writing the code, as that
will separate it from the master branch and allow you to continue
to work on small projects related to master.
Use existing regression tests as templates to demonstrate the type of
header information that should be included in a regression test.
-@subsection Write documentation
-
-Although it is not required, it is helpful if the developer can
-write relevant material for inclusion in the Notation Reference.
-If the developer does not feel qualified to write the documentation,
-a documentation editor will be able to write it from the regression
-tests.
-
-If the modification changes the input syntax so that inline snippets in
-the documentation need to be changed, you will need to change the
-snippets in both the english version of the documentation and any
-translated versions. If you do not change the snippets in all
-translations, older versions of the snippet may be included
-when the documentation is built.
-
-If lsr snippets need to be changed, the snippet should be copied to
-@file{Documentation/snippets/new} and modified there. The portions of
-the snippet that are added by makelsr.py should be removed. The changed
-snippet will then be included in all versions of the documentation.
-
-If non-snippet text is changed in the english documentation, no
-corresponding changes should be made in the translated documentation.
-
@subsection Write convert-ly rule
If the modification changes the input syntax, a convert-ly rule
rule will simply point out to the user that the feature needs
manual correction.
-@subsection Write NEWS entry
+@subsection Automatically update documentation, snippets, and regtests
+
+convert-ly should be used to update the documentation, the snippets,
+and the regression tests. This not only makes the necessary syntax
+changes, it also tests the convert-ly rules.
+
+The automatic updating is a three step process. First, be sure you
+are in the top-level source directory. Then, for the
+documentation, do:
+
+@example
+find Documentation/ -name '*.itely' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
+@end example
+
+@noindent
+where @var{X.Y.Z} is the version number of the last released development
+version.
+
+Next, for the snippets, do:
+
+@example
+find Documentation/snippets/ -name '*.ly' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
+@end example
+
+Finally, for the regression tests, do:
-An entry should be added to the NEWS file to describe the feature
-changes to be implemented. This is especially important for changes
-that change input file syntax.
+@example
+find input/regression/ -name '*.ly' | xargs convert-ly -e --from @qq{@var{X.Y.Z}}
+
+@end example
-Hints for NEWS file entries are given at the top of the NEWS file.
+@subsection Manually update documentation, snippets, and regtests
+
+Where the convert-ly rule is not able to automatically update the inline
+lilypond code in the documentation (i.e. if a NOT_SMART rule is used), the
+documentation must be manually updated. The inline snippets that require
+changing must be changed in the English version of the docs and all
+translated versions. If the inline code is not changed in the
+translated documentation, the old snippets will show up in the
+English version of the documentation.
+
+Where the convert-ly rule is not able to automatically update snippets
+in Documentation/snippets/, those snippets must be manually updated.
+Those snippets should be copied to Documentation/snippets/new. The
+comments at the top of the snippet describing its automatice generation
+should be removed. All translated texidoc strings should be removed.
+The comment @qq{% begin verbatim} should be removed. The syntax of
+the snippet should then be manually edited.
+
+Where snippets in Documentation/snippets are made obsolete, the snippet
+should be copied to Documentation/snippets/new. The comments and
+texidoc strings should be removed as described above. Then the body
+of the snippet should be changed to:
-New entries in NEWS go at the top of the file.
+@example
+\markup @{
+ This snippet is deprecated as of version X.Y.Z and
+ will be removed from the documentation.
+@}
+@end example
-The NEWS entry should be written to show how the new change
+@noindent
+where X.Y.Z is the version number for which the convert-ly rule was
+written.
+
+Update the snippet files by running:
+
+@example
+scripts/auxiliar/makelsr.py
+@end example
+
+Where the convert-ly rule is not able to automatically update regression
+tests, the regression tests in input/regression should be manually
+edited.
+
+Although it is not required, it is helpful if the developer
+can write relevant material for inclusion in the Notation
+Reference. If the developer does not feel qualified to write
+the documentation, a documentation editor will be able to
+write it from the regression tests. The text that is added to
+or removed from the documentation should be changed only in
+the English version.
+
+@subsection Edit changes.tely
+
+An entry should be added to Documentation/changes.tely to describe
+the feature changes to be implemented. This is especially important
+for changes that change input file syntax.
+
+Hints for changes.tely entries are given at the top of the file.
+
+New entries in changes.tely go at the top of the file.
+
+The changes.tely entry should be written to show how the new change
improves LilyPond, if possible.
+@subsection Verify successful build
+
+When the changes have been made, successful completion must be
+verified by doing
+
+@example
+make all
+make doc
+@end example
+
+When these commands complete without error, the patch is
+considered to function successfully.
+
+Developers on Windows who are unable to build LilyPond should
+get help from a Linux or OSX developer to do the make tests.
+
@subsection Verify regression test
In order to avoid breaking LilyPond, it is important to verify that
@subsection Post patch for comments
For any change other than a minor change, a patch set should be
-posted on Rietveld for comment.
+posted on @uref{http://codereview.appspot.com/, Rietveld} for comment.
+This requires the use of an external package, git-cl, and an email
+account on Google.
+
+git-cl is installed by:
+
+@example
+git clone git://neugierig.org/git-cl.git
+@end example
-The patch set is posted by issuing the following command, after
-first committing all changes:
+Then, add the git-cl directory to your PATH, or create a
+symbolic link to the git-cl and upload.py in one of your
+PATH directories (like usr/bin). git-cl is then
+configured by entering the command
@example
-git-cl upload <reference SHA1 ID>
+git cl config
+@end example
+
+@noindent
+in the LilyPond git directory and answering the questions that
+are asked. If you do not understand the question answer with just
+a newline (CR).
+
+The patch set is posted to Rietveld as follows. Ensure your changes
+are committed in a separate branch, which should differ from the
+reference branch to be used by just the changes to be uploaded.
+If the reference branch is to be origin/master, ensure this is
+up-to-date. If necessary, use git rebase to rebase the branch
+containing the changes to the head of origin/master. Finally,
+check out branch with the changes and enter the command:
+
+@example
+git cl upload <reference SHA1 ID>
@end example
@noindent
where <reference SHA1 ID> is the SHA1 ID of the commit to be used
-as a reference source for the patch (generally, this will be the
-SHA1 ID of origin/master).
+as a reference source for the patch. Generally, this will be the
+SHA1 ID of origin/master, and in that case the command
+
+@example
+git cl upload origin/master
+@end example
+
+@noindent
+can be used.
-After prompting for an email and a password, the patch set will be
-posted to Rietveld.
+After prompting for your Google email address and password, the
+patch set will be posted to Rietveld.
-An email should then be sent to lilypond-devel, with a subject line
+You should then announce the patch by sending
+an email to lilypond-devel, with a subject line
starting with PATCH:, asking for comments on the patch.
As revisions are made in response to comments, successive patch sets
-for the same issue can be uploaded by reissuing the git-cl command.
+for the same issue can be uploaded by reissuing the git-cl command
+with the modified branch checked out.
@subsection Push patch
If the changes were in response to a feature request on the Google
issue tracker for LilyPond, the author should change the status to
-@q{Fixed_x_y_z} where the patch was fixed in version x.y.z. If
+Fixed and a tag @q{fixed_x_y_z} should be added, where the patch was
+fixed in version x.y.z. If
the author does not have privileges to change the status, an email
should be sent to bug-lilypond requesting the BugMeister to change
the status.
+
+@node Iterator tutorial
+@section Iterator tutorial
+
+FIXME -- this is a placeholder for a tutorial on iterators
+
+Iterators are routines written in C++ that process music expressions
+and sent the music events to the appropriate engravers and/or
+performers.
+
+@node Engraver tutorial
+@section Engraver tutorial
+
+FIXME -- This is a placeholder for a tutorial on how engravers work.
+
+Engravers are C++ classes that catch music events and
+create the appropriate grobs for display on the page. Though the
+majority of engravers are responsible for the creation of a single grob,
+in some cases (e.g. @code{New_fingering_engraver}), several different grobs
+may be created.
+
+@subsection Useful methods for information processing
+
+An engraver inherits the following public methods from the Translator
+base class, which can be used to process listened events and acknowledged
+grobs:
+
+@itemize
+@item @code{virtual void initialize ()}
+@item @code{void start_translation_timestep ()}
+@item @code{void process_music ()}
+@item @code{void process_acknowledged ()}
+@item @code{void stop_translation_timestep ()}
+@item @code{virtual void finalize ()}
+@end itemize
+
+These methods are listed in order of translation time, with
+@code{initialize ()} and @code{finalize ()} bookending the whole
+process. @code{initialize ()} can be used for one-time initialization
+of context properties before translation starts, whereas
+@code{finalize ()} is often used to tie up loose ends at the end of
+translation: for example, an unterminated spanner might be completed
+automatically or reported with a warning message.
+
+@subsection Translation process
+
+At each timestep in the music, translation proceeds by calling the
+following methods in turn:
+
+@code{start_translation_timestep ()} is called before any user information enters
+the translators, i.e., no property operations (\set, \override, etc.) or events
+have been processed yet.
+
+@code{process_music ()} and @code{process_acknowledged ()} are called after events
+have been heard, or grobs have been acknowledged. The latter tends to be used
+exclusively with engravers which only acknowledge grobs, whereas the former is
+the default method for main processing within engravers.
+
+@code{stop_translation_timestep ()} is called after all user information has been
+processed prior to beginning the translation for the next timestep.
+
+@subsection Preventing garbage collection for SCM member variables
+
+In certain cases, an engraver might need to ensure private Scheme variables
+(with type SCM) do not get swept away by Guile's garbage collector: for example,
+a cache of the previous key signature which must persist persist between timesteps.
+The method @code{virtual derived_mark () const} can be used in such cases to mark
+such objects as follows:
+
+@example
+Engraver_name::derived_mark ()
+@{
+ scm_gc_mark (private_scm_member_)
+@}
+@end example
+
+
+@subsection Listening to music events
+
+External interfaces to to the engraver are implemented by protected
+macros including one or more of the following:
+
+@itemize
+@item @code{DECLARE_TRANSLATOR_LISTENER (event_name)}
+@item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)}
+@end itemize
+
+@noindent
+where @var{event_name} is the type of event required to provide the
+input the engraver needs and @var{Engraver_name} is the name of the
+engraver.
+
+Following declaration of a listener, the method is implemented as follows:
+
+@example
+IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)
+void
+Engraver_name::listen_event_name (Stream event *event)
+@{
+ ...body of listener method...
+@}
+@end example
+
+@subsection Acknowledging grobs
+
+Some engravers also need information from grobs as they are created
+and as they terminate. The mechanism and methods to obtain this
+information are set up by the macros:
+
+@itemize
+@item @code{DECLARE_ACKNOWLEDGER (grob_interface)}
+@item @code{DECLARE_END_ACKNOWLEDGER (grob_interface)}
+@end itemize
+
+where @var{grob_interface} is an interface supported by the
+grob(s) which should be acknowledged. For example, the following
+code would declare acknowledgers for a @code{NoteHead} grob (via the
+@code{note-head-interface}) and any grobs which support the
+@code{side-position-interface}:
+
+@example
+@code{DECLARE_ACKNOWLEDGER (note_head)}
+@code{DECLARE_ACKNOWLEDGER (side_position)}
+@end example
+
+The @code{DECLARE_END_ACKNOWLEDGER ()} macro sets up a spanner-specific
+acknowledger which will be called whenever a spanner ends.
+
+Following declaration of an acknowledger, the method is coded as follows:
+
+@example
+void
+Engraver_name::acknowledge_interface_name (Grob_info info)
+@{
+ ...body of acknowledger method...
+@}
+@end example
+
+@subsection Engraver declaration/documentation
+
+An engraver must have a public macro
+
+@itemize
+@item @code{TRANSLATOR_DECLARATIONS (Engraver_name)}
+@end itemize
+
+@noindent
+where @code{Engraver_name} is the name of the engraver. This
+defines the common variables and methods used by every engraver.
+
+At the end of the engraver file, one or both of the following
+macros are generally called to document the engraver in the
+Internals Reference:
+
+@itemize
+@item @code{ADD_ACKNOWLEDGER (Engraver_name, grob_interface)}
+@item @code{ADD_TRANSLATOR (Engraver_name, Engraver_doc,
+ Engraver_creates, Engraver_reads, Engraver_writes)}
+@end itemize
+
+@noindent
+where @code{Engraver_name} is the name of the engraver, @code{grob_interface}
+is the name of the interface that will be acknowledged,
+@code{Engraver_doc} is a docstring for the engraver,
+@code{Engraver_creates} is the set of grobs created by the engraver,
+@code{Engraver_reads} is the set of properties read by the engraver,
+and @code{Engraver_writes} is the set of properties written by
+the engraver.
+
+@node Callback tutorial
+@section Callback tutorial
+
+FIXME -- This is a placeholder for a tutorial on callback functions.
+
+@node LilyPond scoping
+@section LilyPond scoping
+
+The Lilypond language has a concept of scoping, ie you can do
+
+@example
+foo = 1
+
+#(begin
+ (display (+ foo 2)))
+@end example
+
+@noindent with @code{\paper}, @code{\midi} and @code{\header} being
+nested scope inside the @file{.ly} file-level scope. @w{@code{foo = 1}}
+is translated in to a scheme variable definition.
+
+This implemented using modules, with each scope being an anonymous
+module that imports its enclosing scope's module.
+
+Lilypond's core, loaded from @file{.scm} files, is usually placed in the
+@code{lily} module, outside the @file{.ly} level. In the case of
+
+@example
+lilypond a.ly b.ly
+@end example
+
+@noindent
+we want to reuse the built-in definitions, without changes effected in
+user-level @file{a.ly} leaking into the processing of @file{b.ly}.
+
+The user-accessible definition commands have to take care to avoid
+memory leaks that could occur when running multiple files. All
+information belonging to user-defined commands and markups is stored in
+a manner that allows it to be garbage-collected when the module is
+dispersed, either by being stored module-locally, or in weak hash
+tables.
projects / lilypond.git / blobdiff