Massive changes - see ChangeLog.

[lilypond.git] / Documentation / user / internals.itely

@c -*-texinfo-*-
@c Note:
@c
@c A menu is needed before every deeper *section nesting of @nodes
@c Run M-x texinfo-all-menus-update
@c to automagically fill in these menus
@c before saving changes


@node Technical manual
@chapter Technical manual


When LilyPond is run, it reads an input file which is parsed.  During
parsing, Music objects are created. This music is interpreted, which
is done by contexts, that produce graphical objects.  This section
discusses details of these three concepts, and how they are glued
together with the embedded Scheme interpreter.

@menu
* Interpretation context::      
* Scheme integration::          
* Music storage format::        
* Lexical details::             
* Output details::              
@end menu


@node Interpretation context
@section Interpretation context

@menu
* Creating contexts::           
* Default contexts::            
* Context properties::          
* Context evaluation::          
* Defining contexts::           
* Engravers and performers::    
* Defining new contexts::       
@end menu


Interpretation contexts are objects that only exist during program
run.  During the interpretation phase (when @code{interpreting music}
is printed on the standard output), the music expression in a
@code{\score} block is interpreted in time order, the same order in
which we hear and play the music.  During this phase, the interpretation
context holds the state for the current point within the music, for
example:
@itemize @bullet
@item What notes are playing at this point?

@item What symbols will be printed at this point?

@item What is the current key signature, time signature, point within
the measure, etc.?
@end itemize

Contexts are grouped hierarchically: A @internalsref{Voice} context is
contained in a @internalsref{Staff} context (because a staff can contain
multiple voices at any point), a @internalsref{Staff} context is contained in
@internalsref{Score}, @internalsref{StaffGroup}, or
@internalsref{ChoirStaff} context.

Contexts associated with sheet music output are called @emph{notation
contexts}, those for sound output are called @emph{performance
contexts}.  The default definitions of the standard notation and
performance contexts can be found in @file{ly/engraver-init.ly} and
@file{ly/performer-init.ly}, respectively.


@node Creating contexts
@subsection Creating contexts
@cindex @code{\context}
@cindex context selection

Contexts for a music expression can be selected manually, using the
following music expression:

@example
\context @var{contexttype} [= @var{contextname}] @var{musicexpr}
@end example

@noindent
This means that @var{musicexpr} should be interpreted within a context
of type @var{contexttype} (with name @var{contextname} if specified).
If no such context exists, it will be created:

@lilypond[verbatim,singleline]
\score {
  \notes \relative c'' {
    c4 <d4 \context Staff = "another" e4> f
  }
}
@end lilypond

@noindent
In this example, the @code{c} and @code{d} are printed on the default
staff.  For the @code{e}, a context @code{Staff} called @code{another}
is specified; since that does not exist, a new context is created.
Within @code{another}, a (default) Voice context is created for the
@code{e4}.  A context is ended when when all music referring it has
finished, so after the third quarter, @code{another} is removed.


@node Default contexts
@subsection Default contexts

Every top level music is interpreted by the @code{Score} context; in
other words, you may think of @code{\score} working like

@example
\score @{
  \context Score @var{music}
@}
@end example

Music expressions  inherit their context from the enclosing music
expression. Hence, it is not necessary to explicitly specify
@code{\context} for most expressions.  In
the following example, only the sequential expression has an explicit
context. The notes contained therein inherit the @code{goUp} context
from the enclosing music expression.

@lilypond[verbatim,singleline]
  \notes \context Voice = goUp { c'4 d' e' }
@end lilypond


Second, contexts are created automatically to be able to interpret the
music expressions.  Consider the following example:

@lilypond[verbatim, singleline]
  \score { \notes { c'4-( d' e'-) } }
@end lilypond

@noindent
The sequential music is interpreted by the Score context initially,
but when a note is encountered, contexts are setup to accept that
note.  In this case, a @code{Thread}, @code{Voice}, and @code{Staff}
context are created.  The rest of the sequential music is also
interpreted with the same @code{Thread}, @code{Voice}, and
@code{Staff} context, putting the notes on the same staff, in the same
voice.

@node Context properties
@subsection Context properties

Contexts have properties.  These properties are set from the @file{.ly}
file using the following expression:
@cindex @code{\property}
@cindex context properties
@cindex properties, context

@example
\property @var{contextname}.@var{propname} = @var{value}
@end example

@noindent
Sets the @var{propname} property of the context @var{contextname} to
the specified Scheme expression @var{value}.  Both @var{propname} and
@var{contextname} are strings, which can often be written unquoted.

@cindex inheriting
Properties that are set in one context are inherited by all of the
contained contexts.  This means that a property valid for the
@internalsref{Voice} context can be set in the @internalsref{Score} context
(for example) and thus take effect in all @internalsref{Voice} contexts.

@cindex @code{Current}
If you do not wish to specify the name of the context in the
@code{\property}-expression itself, you can refer to the abstract context
name, @code{Current}.  The @code{Current} context is the latest
used context.  This will typically mean the @internalsref{Thread} context, 
but you can force another context with the
@code{\property}-command.  Hence the expressions

@example
\property @var{contextname}.@var{propname} = @var{value}
@end example

@noindent
and

@example
\context @var{contextname}
\property Current.@var{propname} = @var{value}
@end example

@noindent
do the same thing.  The main use for this is in predefined variables.
This construction allows the specification of a property-setting
without restriction to a specific context.

Properties can be unset using the following statement.
@example
\property @var{contextname}.@var{propname} \unset
@end example

@cindex properties, unsetting
@cindex @code{\unset} 

@noindent
This removes the definition of @var{propname} in @var{contextname}.  If
@var{propname} was not defined in @var{contextname} (but was inherited
from a higher context), then this has no effect.

@refbugs

The syntax of @code{\unset} is asymmetric: @code{\property \unset} is not
the inverse of @code{\property \set}.


@node Context evaluation
@subsection Context evaluation

Contexts can be modified during interpretation with Scheme code. The
syntax for this is
@example
  \applycontext @var{function}
@end example

@var{function} should be a Scheme function taking a single argument,
being the context to apply it to. The following code will print the
current bar number on the standard output during the compile:

@example
    \applycontext
      #(lambda (x)
         (format #t "\nWe were called in barnumber ~a.\n"
          (ly:get-context-property x 'currentBarNumber)))
@end example



@node Defining contexts
@subsection Defining contexts

@cindex context definition
@cindex translator definition

The most common way to create a new context definition is by extending
an existing one.  An existing context from the paper block is copied
by referencing a context identifier:

@example
\paper @{
  \translator @{
    @var{context-identifier}
  @}
@}
@end example

@noindent
Every predefined context has a standard identifier. For example, the
@code{Staff} context can be referred to as @code{\StaffContext}.

The context can then be modified by setting or changing properties,
e.g.
@example
\translator @{
  \StaffContext
  Stem \set #'thickness = #2.0
  defaultBarType = #"||"
@}
@end example
These assignments happen before interpretation starts, so a @code{\property}
command will override any predefined settings.

@cindex engraver

@refbugs

It is not possible to collect multiple property assignments in a
variable, and apply to one @code{\translator} definition by
referencing that variable.

@node Engravers and performers
@subsection  Engravers and performers


Each context is composed of a number of building blocks, or plug-ins
called engravers.  An engraver is a specialized C++ class that is
compiled into the executable. Typically, an engraver is responsible
for one function: the @code{Slur_engraver} creates only @code{Slur}
objects, and the @code{Skip_event_swallow_translator} only swallows
(silently gobbles) @code{SkipEvent}s.



@cindex engraver
@cindex plug-in

An existing context definition can be changed by adding or removing an
engraver. The syntax for these operations is 
@example
\consists @var{engravername}
\remove @var{engravername}
@end example

@cindex \consists
@cindex \remove

@noindent
Here @var{engravername} is a string, the name of an engraver in the
system. In the following example, the @code{Clef_engraver} is removed
from the Staff context. The result is a staff without a clef, where
the central C is at its default position, the center line:

@lilypond[verbatim,singleline]
\score {
  \notes {
    c'4 f'4
  }
  \paper {
    \translator {
      \StaffContext
      \remove Clef_engraver
    }
  }
}
@end lilypond

A list of all engravers is in the internal documentation,
see @internalsref{All engravers}.

@node Defining new contexts
@subsection Defining new contexts


It is also possible to define new contexts from scratch.  To do this,
you must define give the new context a name.  In the following
example, a very simple Staff context is created: one that will put
note heads on a staff symbol.

@example
\translator @{
  \type "Engraver_group_engraver"
  \name "SimpleStaff"
  \alias "Staff"
  \consists "Staff_symbol_engraver"
  \consists "Note_head_engraver"
  \consistsend "Axis_group_engraver"
@}
@end example

@noindent
The argument of @code{\type} is the name for a special engraver that
handles cooperation between simple engravers such as
@code{Note_head_engraver} and @code{Staff_symbol_engraver}.  This
should always be  @code{Engraver_group_engraver} (unless you are
defining a Score context from scratch, in which case
@code{Score_engraver}   must be used).

The complete list of context  modifiers is the following:
@itemize @bullet
@item @code{\alias} @var{alternate-name}:
This specifies a different name.  In the above example,
@code{\property Staff.X = Y} will also work on @code{SimpleStaff}s.

@item @code{\consistsend} @var{engravername}:
Analogous to @code{\consists}, but makes sure that
@var{engravername} is always added to the end of the list of
engravers.

Engravers that group context objects into axis groups or alignments
need to be at the end of the list. @code{\consistsend} insures that
engravers stay at the end even if a user adds or removes engravers.
    
@item @code{\accepts} @var{contextname}:
This context can contains @var{contextname} contexts.  The first
@code{\accepts} is created as a default context when events (e.g. notes
or rests) are encountered.

@item @code{\denies}:
The opposite of @code{\accepts}.

@item @code{\name} @var{contextname}:
This sets the type name of the context, e.g. @code{Staff},
@code{Voice}.  If the name is not specified, the translator will not
do anything.
@end itemize


@node Scheme integration
@section Scheme integration

@cindex Scheme
@cindex GUILE
@cindex Scheme, in-line code
@cindex accessing Scheme
@cindex evaluating Scheme
@cindex LISP

LilyPond internally uses GUILE, a Scheme-interpreter, to represent
data throughout the whole program, and glue together different program
modules. For advanced usage, it is sometimes necessary to access and
program the Scheme interpreter.

Scheme is a full-blown programming language, from the LISP
family. and a full discussion is outside the scope of this document.
Interested readers are referred to the website
@uref{http://www.schemers.org/} for more information on Scheme.

The GUILE library for extension is documented at
@uref{http://www.gnu.org/software/guile}.
@ifinfo
When it is installed, the following link should take you to its manual
@ref{(guile.info)guile}
@end ifinfo

@menu
* Inline Scheme::               
* Input variables and Scheme::  
* Scheme datatypes::            
* Assignments::                 
@end menu

@node Inline Scheme
@subsection Inline Scheme

Scheme expressions can be entered in the input file by entering a
hash-sign (@code{#}).  The expression following the hash-sign is
evaluated as Scheme. For example, the boolean value @var{true} is
@code{#t} in Scheme, so for LilyPond @var{true} looks like @code{##t},
and can be used in property assignments:
@example
  \property Staff.autoBeaming = ##f
@end example


@node Input variables and Scheme
@subsection Input variables and Scheme


The input format supports the notion of variable: in the following
example, a music expression is assigned to a variable with the name
@code{traLaLa}.
@example
  traLaLa = \notes @{ c'4 d'4 @}
@end example

@noindent

There is also a form of scoping: in the following example, the
@code{\paper} block also contains a @code{traLaLa} variable, which is
independent of the outer @code{\traLaLa}.
@example
  traLaLa = \notes @{ c'4 d'4 @}
  \paper @{ traLaLa = 1.0 @}
@end example
@c
In effect, each input file is a scope, and all @code{\header},
@code{\midi} and @code{\paper} blocks are scopes nested inside that
toplevel scope.

Both variables and scoping are implemented in the GUILE module system.
An anonymous Scheme module is attached to each scope. An assignment of
the form
@example
 traLaLa = \notes @{ c'4 d'4 @} 
@end example

@noindent
is internally converted to a Scheme definition
@example
 (define traLaLa @var{Scheme value of ``@code{\notes ... }''})
@end example

This means that input variables and Scheme variables may be freely
mixed.  In the following example, a music fragment is stored in the
variable @code{traLaLa}, and duplicated using Scheme. The result is
imported in a @code{\score} by means of a second variable
@code{twice}:
@example
  traLaLa = \notes @{ c'4 d'4 @}
  
  #(define newLa (map ly:music-deep-copy
    (list traLaLa traLaLa)))
  #(define twice
    (make-sequential-music newLa))

  \score @{ \twice @}
@end example

In the above example, music expressions can be `exported' from the
input to the Scheme interpreter. The opposite is also possible. By
wrapping a Scheme value in the function @code{ly:export}, a Scheme
value is interpreted as if it were entered in LilyPond syntax: instead
of defining @code{\twice}, the example above could also have been
written as
@example
  @dots{}
  \score @{ #(ly:export (make-sequential-music newLa)) @}
@end example





@node Scheme datatypes
@subsection Scheme datatypes

Scheme is used to glue together different program modules. To aid this
glue function, many LilyPond specific object types can be passed as
Scheme value. 

The following list are all LilyPond specific types, that
can exist during parsing:
@table @code
@item Duration
@item Input
@item Moment
@item Music
@item Event
In C++ terms, an @code{Event} is a subtype of @code{Music}. However,
both have different functions in the syntax.
@item Music_output_def
@item Pitch
@item Score
@item Translator_def
@end table


During a run, transient objects are also created and destroyed.

@table @code
@item Grob: short for `Graphical object'.
@item Scheme_hash_table 
@item Music_iterator

@item Molecule: Device-independent page output object,
including dimensions.  

@item Syllable_group

@item Spring_smob

@item Translator: An object that produces audio objects or Grobs.
It may be accessed with @code{\applyoutput}.

@item Font_metric: An object representing a font.
@end table

Many functions are defined to manipulate these data structures. They
are all listed and documented in the internals manual, see
@internalsref{All scheme functions}.


@node Assignments
@subsection Assignments
@cindex Assignments

Variables allow objects to be assigned to names during the parse
stage.  To assign a variable, use
@example
@var{name}@code{=}@var{value}
@end example
To refer to a variable, precede its name with a backslash:
`@code{\}@var{name}'.  @var{value} is any valid Scheme value or any of
the input-types listed above.  Variable assignments can appear at top
level in the LilyPond file, but also in @code{\paper} blocks.

A variable can be created with any string for its name, but for
accessing it in the LilyPond syntax, its name must consist of
alphabetic characters only, and may not be a keyword of the syntax.
There are no restrictions for naming and accessing variables in the
Scheme interpreter,

The right hand side of a variable assignment is parsed completely
before the assignment is done, so variables may be  redefined in terms
of its old value, e.g.
@c
@example
foo = \foo * 2.0
@end example

When a variable is referenced in LilyPond syntax, the information it
points to is copied.  For this reason, an variable reference must
always be the first item in a block.

@example
\paper @{
  foo = 1.0
  \paperIdent % wrong and invalid
@}
@end example

@example
\paper @{
  \paperIdent % correct
  foo = 1.0
@}
@end example

      

@node Music storage format
@section Music storage format

Music in LilyPond is entered as music expressions. This section
discusses different types of music expressions, and explains how
information is stored internally. This internal storage is accessible
through the Scheme interpreter, so music expressions may be
manipulated using Scheme functions. 

@menu
* Music expressions::           
* Internal music representation::  
* Manipulating music expressions::  
@end menu

@node Music expressions
@subsection Music expressions
@cindex music expressions

Notes, rests, lyric syllables are music expressions.  Small music
expressions may be combined to form larger ones, for example, by
enclosing a list of expressions in @code{\sequential @{ @}} or @code{<
>}.  In the following example, a compound expression is formed out of
the quarter note @code{c} and a quarter note @code{d}:

@example 
\sequential @{ c4 d4 @} 
@end example 

@cindex Sequential music
@cindex @code{\sequential}
@cindex sequential music
@cindex @code{<}
@cindex @code{>}
@cindex Simultaneous music
@cindex @code{\simultaneous}

The two basic compound music expressions are simultaneous and
sequential music:

@example
\sequential @code{@{} @var{musicexprlist} @code{@}}
\simultaneous @code{@{} @var{musicexprlist} @code{@}}
@end example

For both, there is a shorthand:

@example
@code{@{} @var{musicexprlist} @code{@}}
@end example

@noindent
for sequential and

@example
@code{<} @var{musicexprlist} @code{>}
@end example

@noindent
for simultaneous music.
In principle, the way in which you nest sequential and simultaneous to
produce music is not relevant.  In the following example, three chords
are expressed in two different ways:

@lilypond[fragment,verbatim,center,quote]
\notes \context Voice {
  <a c'> <b d'> <c' e'>
  < { a b c' } { c' d' e' } >
}
@end lilypond
However, using @code{<} and @code{>} for entering chords leads to
various peculiarities. For this reason, a special syntax
for chords was introduced in version 1.7: @code{<< >>}.





Other compound music expressions include:
@example
\repeat @var{expr}
\transpose @var{from} @var{to} @var{expr}
\apply @var{func} @var{expr}
\context @var{type} = @var{id} @var{expr}
\times @var{fraction} @var{expr}
@end example

@node Internal music representation
@subsection Internal music representation






When a music expression is parsed, it is converted into a set of
Scheme music objects. The defining property of a music object is that
it takes up time. Time is a rational number that measures the length
of a piece of music, in whole notes.

A music object has three kinds of types:
@itemize @bullet
@item
  music name: Each music expression has a name, for example, a note
leads to a @internalsref{NoteEvent}, and @code{\simultaneous} leads to
a @internalsref{SimultaneousMusic}. A list of all expressions
available is in the internals manual, under @internalsref{Music
expressions}.

@item
  `type' or interface: Each music name has several `types' or interface,
  for example, a note is an @code{event}, but it is also a @code{note-event}, 
  a @code{rhythmic-event} and a @code{melodic-event}.

  All classes of music are listed in the internals manual, under
  @internalsref{Music classes}. 
@item
C++ object: Each music object is represented by a C++ object. For technical
reasons, different music objects may be represented by different C++
object types. For example, a note is @code{Event} object, while
@code{\grace} creates a @code{Grace_music} object.

We expect that distinctions between different C++ types will disappear
in the future.
@end itemize

The actual information of a music expression is stored in properties.
For example, a @internalsref{NoteEvent} has @code{pitch} and
@code{duration} properties that store the pitch and duration of that
note.  A list of all properties available is in the internals manual,
under @internalsref{Music properties}.

A compound music expression is a music object that contains other
music objects in its properties. A list of objects can be stored in
the @code{elements} property of a music object, or a single `child'
music object in the @code{element} object. For example,
@internalsref{SequentialMusic} has its children in @code{elements},
and @internalsref{GraceMusic} has its single argument in
@code{element}. The body of a repeat is in @code{element} property of
@internalsref{RepeatedMusic}, and the alternatives in @code{elements}.

@node Manipulating music expressions
@subsection Manipulating music expressions

Music objects and their properties can be accessed and manipulated
directly, through the @code{\apply} mechanism.  
The syntax for @code{\apply} is 
@example
\apply #@var{func} @var{music}
@end example

@noindent
This means that the scheme function @var{func} is called with
@var{music} as its argument.  The return value of @var{func} is the
result of the entire expression.  @var{func} may read and write music
properties using the functions @code{ly:get-mus-property} and
@code{ly:set-mus-property!}.

An example is a function that reverses the order of elements in
its argument:
@lilypond[verbatim,singleline]
  #(define (rev-music-1 m)
     (ly:set-mus-property! m 'elements (reverse
       (ly:get-mus-property m 'elements)))
     m)
  \score {  \notes \apply #rev-music-1 { c4 d4 } }
@end lilypond

The use of such a function is very limited. The effect of this
function is void when applied to an argument which is does not have
multiple children.  The following function application has no effect:

@example
  \apply #rev-music-1 \grace @{ c4 d4 @}
@end example

@noindent
In this case, @code{\grace} is stored as @internalsref{GraceMusic}, which has no
@code{elements}, only a single @code{element}. Every generally
applicable function for @code{\apply} must -- like music expressions
themselves -- be recursive.

The following example is such a recursive function: It first extracts
the @code{elements} of an expression, reverses them and puts them
back. Then it recurses, both on @code{elements} and @code{element}
children.
@example
#(define (reverse-music music)
  (let* ((elements (ly:get-mus-property music 'elements))
         (child (ly:get-mus-property music 'element))
         (reversed (reverse elements)))

    ; set children
    (ly:set-mus-property! music 'elements reversed)

    ; recurse
    (if (ly:music? child) (reverse-music child))
    (map reverse-music reversed)
    
    music))
@end example

A slightly more elaborate example is in
@inputfileref{input/test,reverse-music.ly}.

Some of the input syntax is also implemented as recursive music
functions. For example, the syntax for polyphony
@example
  < a \\ b>
@end example

@noindent
is actually  implemented as a recursive function that replaces the
above by the internal equivalent of
@example
  < \context Voice = "1" @{ \voiceOne a @}
    \context Voice = "2" @{ \voiceTwo b @} >
@end example

Other applications of @code{\apply} are writing out repeats
automatically (@inputfileref{input/test,unfold-all-repeats.ly}),
saving keystrokes (@inputfileref{input/test,music-box.ly}) and
exporting
LilyPond input to other formats  (@inputfileref{input/test,to-xml.ly})

@seealso

@file{scm/music-functions.scm}, @file{scm/music-types.scm},
@inputfileref{input/test,add-staccato.ly},
@inputfileref{input/test,unfold-all-repeats.ly}, and
@inputfileref{input/test,music-box.ly}.

@node Lexical details
@section Lexical details


@cindex string
@cindex concatenate

Begins and ends with the @code{"} character.  To include a @code{"}
character in a string write @code{\"}.  Various other backslash
sequences have special interpretations as in the C language.  A string
that contains no spaces can be written without the quotes.  Strings can
be concatenated with the @code{+} operator.


@node Output details
@section Output details

LilyPond's default output format is @TeX{}.  Using the option @option{-f}
(or @option{--format}) other output formats can be selected also, but
currently none of them work reliably.

At the beginning of the output file, various global parameters are defined.
It also contains a large @code{\special} call to define PostScript routines
to draw items not representable with @TeX{}, mainly slurs and ties.  A DVI
driver must be able to understand such embedded PostScript, or the output
will be rendered incompletely.

Then the file @file{lilyponddefs.tex} is loaded to define the macros used
in the code which follows.  @file{lilyponddefs.tex} includes various other
files, partially depending on the global parameters.

Now the music is output system by system (a `system' consists of all
staves belonging together).  From @TeX{}'s point of view, a system is an
@code{\hbox} which contains a lowered @code{\vbox} so that it is centered
vertically on the baseline of the text.  Between systems,
@code{\interscoreline} is inserted vertically to have stretchable space.
The horizontal dimension of the @code{\hbox} is given by the
@code{linewidth} parameter from LilyPond's @code{\paper} block.


After the last system LilyPond emits a stronger variant of
@code{\interscoreline} only if the macro
@code{\lilypondpaperlastpagefill} is not defined (flushing the systems
to the top of the page).  You can avoid that by setting the variable
@code{lastpagefill} in LilyPond's @code{\paper} block.

It is possible to fine-tune the vertical offset further by defining the
macro @code{\lilypondscoreshift}:

@example
\def\lilypondscoreshift@{0.25\baselineskip@}
@end example

@noindent
where @code{\baselineskip} is the distance from one text line to the next.

The code produced by LilyPond should be run through La@TeX{}, not
plain @TeX{}.

Here an example how to embed a small LilyPond file @code{foo.ly} into
running La@TeX{} text without using the @code{lilypond-book} script
(@pxref{lilypond-book manual}):

@example
\documentclass@{article@}

\def\lilypondpaperlastpagefill@{@}
\lineskip 5pt
\def\lilypondscoreshift@{0.25\baselineskip@}

\begin@{document@}
This is running text which includes an example music file
\input@{foo.tex@}
right here.
\end@{document@}
@end example

The file @file{foo.tex} has been simply produced with

@example
lilypond foo.ly
@end example

It is important to set the @code{indent} parameter to zero in the
@code{\paper} block of @file{foo.ly}.

The call to @code{\lineskip} assures that there is enough vertical space
between the LilyPond box and the surrounding text lines.

@c EOF