+
+@node Iterator tutorial
+@section Iterator tutorial
+
+FIXME -- this is a placeholder for a tutorial on iterators
+
+Iterators are routines written in C++ that process music expressions
+and sent the music events to the appropriate engravers and/or
+performers.
+
+@node Engraver tutorial
+@section Engraver tutorial
+
+FIXME -- This is a placeholder for a tutorial on how engravers work.
+
+Engravers are C++ classes that catch music events and
+create the appropriate grobs for display on the page. Though the
+majority of engravers are responsible for the creation of a single grob,
+in some cases (e.g. @code{New_fingering_engraver}), several different grobs
+may be created.
+
+@subsection Useful methods for information processing
+
+An engraver inherits the following public methods from the Translator
+base class, which can be used to process listened events and acknowledged
+grobs:
+
+@itemize
+@item @code{virtual void initialize ()}
+@item @code{void start_translation_timestep ()}
+@item @code{void process_music ()}
+@item @code{void process_acknowledged ()}
+@item @code{void stop_translation_timestep ()}
+@item @code{virtual void finalize ()}
+@end itemize
+
+These methods are listed in order of translation time, with
+@code{initialize ()} and @code{finalize ()} bookending the whole
+process. @code{initialize ()} can be used for one-time initialization
+of context properties before translation starts, whereas
+@code{finalize ()} is often used to tie up loose ends at the end of
+translation: for example, an unterminated spanner might be completed
+automatically or reported with a warning message.
+
+@subsection Translation process
+
+At each timestep in the music, translation proceeds by calling the
+following methods in turn:
+
+@code{start_translation_timestep ()} is called before any user information enters
+the translators, i.e., no property operations (\set, \override, etc.) or events
+have been processed yet.
+
+@code{process_music ()} and @code{process_acknowledged ()} are called after events
+have been heard, or grobs have been acknowledged. The latter tends to be used
+exclusively with engravers which only acknowledge grobs, whereas the former is
+the default method for main processing within engravers.
+
+@code{stop_translation_timestep ()} is called after all user information has been
+processed prior to beginning the translation for the next timestep.
+
+@subsection Preventing garbage collection for SCM member variables
+
+In certain cases, an engraver might need to ensure private Scheme variables
+(with type SCM) do not get swept away by Guile's garbage collector: for example,
+a cache of the previous key signature which must persist persist between timesteps.
+The method @code{virtual derived_mark () const} can be used in such cases to mark
+such objects as follows:
+
+@example
+Engraver_name::derived_mark ()
+@{
+ scm_gc_mark (private_scm_member_)
+@}
+@end example
+
+
+@subsection Listening to music events
+
+External interfaces to to the engraver are implemented by protected
+macros including one or more of the following:
+
+@itemize
+@item @code{DECLARE_TRANSLATOR_LISTENER (event_name)}
+@item @code{IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)}
+@end itemize
+
+@noindent
+where @var{event_name} is the type of event required to provide the
+input the engraver needs and @var{Engraver_name} is the name of the
+engraver.
+
+Following declaration of a listener, the method is implemented as follows:
+
+@example
+IMPLEMENT_TRANSLATOR_LISTENER (Engraver_name, event_name)
+void
+Engraver_name::listen_event_name (Stream event *event)
+@{
+ ...body of listener method...
+@}
+@end example
+
+@subsection Acknowledging grobs
+
+Some engravers also need information from grobs as they are created
+and as they terminate. The mechanism and methods to obtain this
+information are set up by the macros:
+
+@itemize
+@item @code{DECLARE_ACKNOWLEDGER (grob_interface)}
+@item @code{DECLARE_END_ACKNOWLEDGER (grob_interface)}
+@end itemize
+
+where @var{grob_interface} is an interface supported by the
+grob(s) which should be acknowledged. For example, the following
+code would declare acknowledgers for a @code{NoteHead} grob (via the
+@code{note-head-interface}) and any grobs which support the
+@code{side-position-interface}:
+
+@example
+@code{DECLARE_ACKNOWLEDGER (note_head)}
+@code{DECLARE_ACKNOWLEDGER (side_position)}
+@end example
+
+The @code{DECLARE_END_ACKNOWLEDGER ()} macro sets up a spanner-specific
+acknowledger which will be called whenever a spanner ends.
+
+Following declaration of an acknowledger, the method is coded as follows:
+
+@example
+void
+Engraver_name::acknowledge_interface_name (Grob_info info)
+@{
+ ...body of acknowledger method...
+@}
+@end example
+
+@subsection Engraver declaration/documentation
+
+An engraver must have a public macro
+
+@itemize
+@item @code{TRANSLATOR_DECLARATIONS (Engraver_name)}
+@end itemize
+
+@noindent
+where @code{Engraver_name} is the name of the engraver. This
+defines the common variables and methods used by every engraver.
+
+At the end of the engraver file, one or both of the following
+macros are generally called to document the engraver in the
+Internals Reference:
+
+@itemize
+@item @code{ADD_ACKNOWLEDGER (Engraver_name, grob_interface)}
+@item @code{ADD_TRANSLATOR (Engraver_name, Engraver_doc,
+ Engraver_creates, Engraver_reads, Engraver_writes)}
+@end itemize
+
+@noindent
+where @code{Engraver_name} is the name of the engraver, @code{grob_interface}
+is the name of the interface that will be acknowledged,
+@code{Engraver_doc} is a docstring for the engraver,
+@code{Engraver_creates} is the set of grobs created by the engraver,
+@code{Engraver_reads} is the set of properties read by the engraver,
+and @code{Engraver_writes} is the set of properties written by
+the engraver.
+
+@node Callback tutorial
+@section Callback tutorial
+
+FIXME -- This is a placeholder for a tutorial on callback functions.
+
+@node LilyPond scoping
+@section LilyPond scoping
+
+The Lilypond language has a concept of scoping, ie you can do
+
+@example
+foo = 1
+
+#(begin
+ (display (+ foo 2)))
+@end example
+
+@noindent with @code{\paper}, @code{\midi} and @code{\header} being
+nested scope inside the @file{.ly} file-level scope. @w{@code{foo = 1}}
+is translated in to a scheme variable definition.
+
+This implemented using modules, with each scope being an anonymous
+module that imports its enclosing scope's module.
+
+Lilypond's core, loaded from @file{.scm} files, is usually placed in the
+@code{lily} module, outside the @file{.ly} level. In the case of
+
+@example
+lilypond a.ly b.ly
+@end example
+
+@noindent
+we want to reuse the built-in definitions, without changes effected in
+user-level @file{a.ly} leaking into the processing of @file{b.ly}.
+
+The user-accessible definition commands have to take care to avoid
+memory leaks that could occur when running multiple files. All
+information belonging to user-defined commands and markups is stored in
+a manner that allows it to be garbage-collected when the module is
+dispersed, either by being stored module-locally, or in weak hash
+tables.
+
+@node LilyPond miscellany
+@section LilyPond miscellany
+
+This is a place to dump information that may be of use to developers
+but doesn't yet have a proper home. Ideally, the length of this section
+would become zero as items are moved to other homes.
+
+@subsection Info from Han-Wen Email
+
+In 2004, Douglas Linhardt decided to try starting a document that would
+explain LilyPond architecture and design principles. The material below
+is extracted from that email, which can be found at
+@uref{http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/2992}.
+The headings reflect questions from Doug or comments from Han-Wen;
+the body text are Han-Wen's answers.
+
+@unnumberedsubsubsec Figuring out how things work.
+
+I must admit that when I want to know how a program works, I use grep
+and emacs and dive into the source code. The comments and the code
+itself are usually more revealing than technical documents.
+
+@unnumberedsubsubsec What's a grob, and how is one used?
+
+Graphical object - they are created from within engravers, either as
+Spanners (derived class) -slurs, beams- or Items (also a derived
+class) -notes, clefs, etc.
+
+There are two other derived classes System (derived from Spanner,
+contaning a "line of music") and Paper_column (derived from Item, it
+contains all items that happen at the same moment). They are separate
+classes because they play a special role in the linebreaking process.
+
+@unnumberedsubsubsec What's a smob, and how is one used?
+
+A C(++) object that is encapsulated so it can be used as a Scheme
+object. See GUILE info, "19.3 Defining New Types (Smobs)"
+
+@unnumberedsubsubsec When is each C++ class constructed and used
+
+@itemize
+
+@item
+Music classes
+
+In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME().
+
+@item
+Contexts
+
+Constructed during "interpreting" phase.
+
+@item
+Engravers
+
+Executive branch of Contexts, plugins that create grobs, usually one
+engraver per grob type. Created together with context.
+
+@item
+Layout Objects
+
+= grobs
+
+@item
+Grob Interfaces
+
+These are not C++ classes per se. The idea of a Grob interface hasn't
+crystallized well. ATM, an interface is a symbol, with a bunch of grob
+properties. They are not objects that are created or destroyed.
+
+@item
+Iterators
+
+Objects that walk through different music classes, and deliver events
+in a synchronized way, so that notes that play together are processed
+at the same moment and (as a result) end up on the same horizontal position.
+
+Created during interpreting phase.
+
+BTW, the entry point for interpreting is ly:run-translator
+(ly_run_translator on the C++ side)
+
+@end itemize
+
+@unnumberedsubsubsec Can you get to Context properties from a Music object?
+
+You can create music object with a Scheme function that reads context
+properties (the \applycontext syntax). However, that function is
+executed during Interpreting, so you can not really get Context
+properties from Music objects, since music objects are not directly
+connected to Contexts. That connection is made by the Music_iterators
+
+@unnumberedsubsubsec Can you get to Music properties from a Context object?
+
+Yes, if you are given the music object within a Context
+object. Normally, the music objects enter Contexts in synchronized
+fashion, and the synchronization is done by Music_iterators.
+
+@unnumberedsubsubsec What is the relationship between C++ classes and Scheme objects?
+
+Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are
+manipulated from C++ as well using the GUILE C function interface
+(prefix: scm_)
+
+@unnumberedsubsubsec How do Scheme procedures get called from C++ functions?
+
+scm_call_*, where * is an integer from 0 to 4.
+Also scm_c_eval_string (), scm_eval ()
+
+@unnumberedsubsubsec How do C++ functions get called from Scheme procedures?
+
+Export a C++ function to Scheme with LY_DEFINE.
+
+@unnumberedsubsubsec What is the flow of control in the program?
+
+Good question. Things used to be clear-cut, but we have Scheme
+and SMOBs now, which means that interactions do not follow a very
+rigid format anymore. See below for an overview, though.
+
+@unnumberedsubsubsec Does the parser make Scheme procedure calls or C++ function calls?
+
+Both. And the Scheme calls can call C++ and vice versa. It's nested,
+with the SCM datatype as lubrication between the interactions
+
+(I think the word "lubrication" describes the process better than the
+traditional word "glue")
+
+@unnumberedsubsubsec How do the front-end and back-end get started?
+
+Front-end: a file is parsed, the rest follows from that. Specifically,
+
+Parsing leads to a Music + Music_output_def object (see parser.yy,
+definition of toplevel_expression )
+
+A Music + Music_output_def object leads to a Global_context object (see
+ly_run_translator ())
+
+During interpreting, Global_context + Music leads to a bunch of
+Contexts. (see Global_translator::run_iterator_on_me () )
+
+After interpreting, Global_context contains a Score_context (which
+contains staves, lyrics etc.) as a child. Score_context::get_output ()
+spews a Music_output object (either a Paper_score object for notation
+or Performance object for MIDI).
+
+The Music_output object is the entry point for the backend. (see
+ly_render_output () )
+
+The main steps of the backend itself are in
+
+@itemize
+
+@item
+paper-score.cc , Paper_score::process_
+
+@item
+system.cc , System::get_lines()
+
+@item
+The step, where things go from grobs to output, is in
+System::get_line(): each grob delivers a Stencil (a Device
+independent output description), which is interpreted by our
+outputting backends (scm/output-tex.scm and scm/output-ps.scm)
+to produce TeX and PS.
+
+@end itemize
+
+Interactions between grobs and putting things into .tex and .ps files
+have gotten a little more complex lately. Jan has implemented
+page-breaking, so now the backend also involves Paper_book,
+Paper_lines and other things. This area is still heavily in flux, and
+perhaps not something you should want to look at.
+
+@unnumberedsubsubsec How do the front-end and back-end communicate?
+
+There is no communication from backend to front-end. From front-end to
+backend is simply the program flow: music + definitions gives
+contexts, contexts yield output, after processing, output is written
+to disk.
+
+@unnumberedsubsubsec Where is the functionality associated with KEYWORDs?
+
+See my-lily-lexer.cc (keywords, there aren't that many) and ly/*.ly
+(most of the other backslashed \words are identifiers)
+
+@unnumberedsubsubsec What Contexts/Properties/Music/etc. are available when they are processed?
+
+What do you mean exactly with this question?
+
+See ly/engraver-init.ly for contexts, see scm/define-*.scm for other
+objects.
+
+@unnumberedsubsubsec How do you decide if something is a Music, Context, or Grob property?
+Why is part-combine-status a Music property when it seems (IMO)
+to be related to the Staff context?
+
+The Music_iterators and Context communicate through two channels
+
+Music_iterators can set and read context properties, idem for
+Engravers and Contexts
+
+Music_iterators can send "synthetic" music events (which aren't in
+the input) to a context. These are caught by Engravers. This is
+mostly a one way communication channel.
+
+part-combine-status is part of such a synthetic event, used by
+Part_combine_iterator to communicate with Part_combine_engraver.
+
+
+@unnumberedsubsubsec I'm adding a property to affect how \autochange works. It seems to
+me that it should be a context property, but the Scheme autochange
+procecure has a Music argument. Does this mean I should use
+a Music property?
+
+\autochange is one of these extra strange beasts: it requires
+look-ahead to decide when to change staves. This is achieved by
+running the interpreting step twice (see scm/part-combiner.scm , at
+the bottom), and storing the result of the first step (where to switch
+staves) in a Music property. Since you want to influence that
+where-to-switch list, your must affect the code in
+make-autochange-music (scm/part-combiner.scm). That code is called
+directly from the parser and there are no official "parsing
+properties" yet, so there is no generic way to tune \autochange. We
+would have to invent something new for this, or add a separate
+argument,
+
+@example
+ \autochange #around-central-C ..music..
+@end example
+
+@noindent
+where around-central-C is some function that is called from
+make-autochange-music.
+
+@unnumberedsubsubsec I get lost figuring out what environment the code I'm looking at is in when it executes.
+I found both the C++ and Scheme autochange code. Then I was
+trying to figure out where the code got called from. I finally figured out that
+the Scheme procedure was called before the C++ iterator code, but it took me a
+while to figure that out, and I still didn't know who did the calling in the
+first place. I only know a little bit about Flex and Bison, so reading those
+files helped only a little bit.
+
+@emph{Han-Wen:} GDB can be of help here. Set a breakpoint in C++, and run. When you
+hit the breakpoint, do a backtrace. You can inspect Scheme objects
+along the way by doing
+
+@example
+p ly_display_scm(obj)
+@end example
+
+this will display OBJ through GUILE.
+
+@subsection Music functions and GUILE debugging
+
+Ian Hulin was trying to do some debugging in music functions, and
+came up with the following question
+
+HI all,
+I'm working on the Guile Debugger Stuff, and would like to try
+debugging a music function definition such as:
+
+@example
+conditionalMark = #(define-music-function (parser location) ()
+ #@{ \tag #'instrumental-part @{\mark \default@} #@} )
+@end example
+
+It appears conditionalMark does not get set up as an
+equivalent of a Scheme
+
+@example
+(define conditionalMark = define-music-function(parser location () ...
+@end example
+
+@noindent
+although something gets defined because Scheme apparently recognizes
+
+@example
+#(set-break! conditionalMark)
+@end example
+
+@noindent
+later on in the file without signalling any Guile errors.
+
+However the breakpoint trap is never encountered as
+define-music-function passed things on to ly:make-music-function,
+which is really C++ code ly_make_music_function, so Guile never
+finds out about the breakpoint.
+
+Han-Wen answered as follows:
+
+You can see the defintion by doing
+
+@example
+#(display conditionalMark)
+@end example
+
+noindent
+inside the .ly file.
+
+The breakpoint failing may have to do with the call sequence. See
+parser.yy, run_music_function(). The function is called directly from
+C++, without going through the GUILE evaluator, so I think that is why
+there is no debugger trap.