* Documentation/user/refman.itely: revise

[lilypond.git] / Documentation / user / internals.itely

@c -*-texinfo-*-
@c Note:
@c
@c A menu is needed before every deeper *section nesting of @nodes
@c Run M-x texinfo-all-menus-update
@c to automagically fill in these menus
@c before saving changes


@node Advanced Topics
@chapter Advanced Topics


When LilyPond is run, it reads an input file.  During parsing, Music
objects are created. This music is interpreted, which is done by
contexts, that produce graphical objects.  This section discusses
details of these three concepts, and how they are glued together with
the embedded Scheme interpreter. 

@menu
* Interpretation context::      
* Scheme integration::          
* Music storage format::        
* Syntactic details::           
* Lexical details::             
* Output details::              
@end menu


@node Interpretation context
@section Interpretation context

@menu
* Creating contexts::           
* Default contexts::            
* Context evaluation::          
* Context properties::          
* Engravers and performers::    
* Changing context definitions::  
* Defining new contexts::       
@end menu


Interpretation contexts are objects that only exist during program
run.  During the interpretation phase (when @code{interpreting music}
is interpreted to standard output), the music expression in a
@code{\score} block is interpreted in time order, the same order in
which hear and play the music.  During this phase, the interpretation
context holds the state for the current point within the music, for example
@itemize @bullet
@item What notes are playing at this point?

@item What symbols will be printed at this point?

@item What is the current key signature, time signature, point within
the measure, etc.?
@end itemize

Contexts are grouped hierarchically: A @internalsref{Voice} context is
contained in a @internalsref{Staff} context (because a staff can contain
multiple voices at any point), a @internalsref{Staff} context is contained in
@internalsref{Score}, @internalsref{StaffGroup}, or
@internalsref{ChoirStaff} context.

Contexts associated with sheet music output are called @emph{notation
contexts}, those for sound output are called @emph{performance
contexts}.  The default definitions of the standard notation and
performance contexts can be found in @file{ly/engraver-init.ly} and
@file{ly/performer-init.ly}, respectively.


@node Creating contexts
@subsection Creating contexts
@cindex @code{\context}
@cindex context selection

Contexts for a music expression can be selected manually, using the
following music expression.

@example
\context @var{contexttype} [= @var{contextname}] @var{musicexpr}
@end example

@noindent
This means that @var{musicexpr} should be interpreted within a context
of type @var{contexttype} (with name @var{contextname} if specified).
If no such context exists, it will be created.

@lilypond[verbatim,singleline]
\score {
  \notes \relative c'' {
    c4 <d4 \context Staff = "another" e4> f
  }
}
@end lilypond

@noindent
In this example, the @code{c} and @code{d} are printed on the
default staff.  For the @code{e}, a context Staff called
@code{another} is specified; since that does not exist, a new
context is created.  Within @code{another}, a (default) Voice context
is created for the @code{e4}.  When all music referring to a
context is finished, the context is ended as well.  So after the
third quarter, @code{another} is removed.


@node Default contexts
@subsection Default contexts

Most music expressions do not need an explicit @code{\context}
declaration: they inherit the notation context from their parent.  In
the following example, only the sequential expression has an explicit
context. The notes contained therein inherit the @code{goUp} context
from the enclosing music expression.

@lilypond[verbatim,singleline]
  \notes \context Voice = goUp { c'4 d' e' }
@end lilypond

There are some quirks that you must keep in mind when dealing with
defaults:

First, every top level music is interpreted by the Score context; in other
words, you may think of @code{\score} working like

@example
\score @{
  \context Score @var{music}
@}
@end example

Second, contexts are created automatically to be able to interpret the
music expressions.  Consider the following example.

@lilypond[verbatim, singleline]
  \score { \notes { c'4-( d' e'-) } }
@end lilypond

@noindent
The sequential music is interpreted by the Score context initially
(notice that the @code{\context} specification is redundant), but when a
note is encountered, contexts are setup to accept that note.  In this
case, a Thread, Voice, and Staff context are created.  The rest of the
sequential music is also interpreted with the same Thread, Voice, and
Staff context, putting the notes on the same staff, in the same voice.

@node Context evaluation
@subsection Context evaluation

Scheme code can be used to modify contexts. The syntax for this is

@example
  \applycontext @var{function}
@end example

@var{function} should be a Scheme function taking a single argument,
being the context to apply it with. The following code will print the
current bar number on the standard output during the compile.

@example
    \applycontext
        #(lambda (x)
         (format #t "\nWe were called in barnumber ~a.\n"
          (ly:get-context-property x 'currentBarNumber)))
@end example


@node Context properties
@subsection Context properties


Notation contexts have properties.  These properties are from
the @file{.ly} file using the following expression:
@cindex @code{\property}
@cindex context properties
@cindex properties, context

@example
\property @var{contextname}.@var{propname} = @var{value}
@end example

@noindent
Sets the @var{propname} property of the context @var{contextname} to
the specified Scheme expression @var{value}.  Both @var{propname} and
@var{contextname} are strings, which can often be written unquoted.

@cindex inheriting
Properties that are set in one context are inherited by all of the
contained contexts.  This means that a property valid for the
@internalsref{Voice} context can be set in the @internalsref{Score} context
(for example) and thus take effect in all @internalsref{Voice} contexts.

@cindex @code{Current}
If you do not wish to specify the name of the context in the
@code{\property}-expression itself, you can refer to the abstract context
name, @code{Current}.  The @code{Current} context is the latest
used context.  This will typically mean the @internalsref{Thread}
context, but you can force another context with the
@code{\property}-command.  Hence the expressions

@example
\property @var{contextname}.@var{propname} = @var{value}
@end example

@noindent
and

@example
\context @var{contextname}
\property Current.@var{propname} = @var{value}
@end example

@noindent
do the same thing.  The main use for this is in predefined variables.
This construction allows the specification of a property-setting
without restriction to a specific context.

Properties can be unset using the following statement.
@example
\property @var{contextname}.@var{propname} \unset
@end example

@cindex properties, unsetting
@cindex @code{\unset} 

@noindent
This removes the definition of @var{propname} in @var{contextname}.  If
@var{propname} was not defined in @var{contextname} (but was inherited
from a higher context), then this has no effect.

@refbugs

The syntax of @code{\unset} is asymmetric: @code{\property \unset} is not
the inverse of @code{\property \set}.


@node Engravers and performers
@subsection Engravers and performers

@node Changing context definitions
@subsection Changing context definitions
@cindex context definition
@cindex translator definition

The most common way to define a context is by extending an existing
context.  You can change an existing context from the paper block by
first initializing a translator with an existing context identifier:

@example
\paper @{
  \translator @{
    @var{context-identifier}
  @}
@}
@end example

@noindent
Every predefined context has a standard identifier. For example, the
@code{Staff} context can be referred to as @code{\StaffContext}.

@noindent
The context can then be modified by setting or changing properties,
e.g.
@example
\translator @{
  \StaffContext
  Stem \set #'thickness = #1.2
  defaultBarType = #"||"
@}
@end example
These assignments happen before interpretation starts, so a @code{\property}
command will override any predefined settings.

@cindex engraver

@refbugs

It is not possible to collect multiple property assignments in a
variable, and apply to one @code{\translator} definition by
referencing that variable.

@node Defining new contexts
@subsection Defining new contexts

@cindex engraver
@cindex plug-in

Each context is composed of a number of building blocks, or plug-ins
called engravers.  An engraver is a specialized C++ class that is
compiled into the executable. Typically, an engraver is responsible
for one function: the @code{Slur_engraver} creates only @code{Slur}
objects, and the @code{Skip_event_swallow_translator} only swallows
(silently gobbles) @code{SkipEvent}s.

An existing context definition can be changed by adding or removing an
engraver. The syntax for these operations is 
@example
\consists @var{engravername}
\remove @var{engravername}
@end example

@cindex \consists
@cindex \remove

@noindent
Here @var{engravername} is a string, the name of an engraver in the
system. In the following example, the @code{Clef_engraver} is removed
from the Staff context. The result is a staff without a clef, where
the central C is at its default position, the center line.

@lilypond[verbatim,singleline]
\score {
  \notes {
    c'4 f'4
  }
  \paper {
    \translator {
      \StaffContext
      \remove Clef_engraver
    }
  }
}
@end lilypond

A list of all engravers is in the internal documentation,
@internalsref{All engravers}.

It is also possible to define new contexts from scratch.  To do this,
you must define give the new context a name.  In the following
example, a very simple Staff context is created: one that will put
note heads on a staff symbol.

@example
\translator @code{
  \type "Engraver_group_engraver"
  \name "SimpleStaff"
  \alias "Staff"
  \consists "Staff_symbol_engraver"
  \consists "Note_head_engraver"
  \consistsend "Axis_group_engraver"
}@
@end example

@noindent
The argument of @code{\type} is the name for a special engraver that
handles cooperation between simple engravers such as
@code{Note_head_engraver} and @code{Staff_symbol_engraver}.  This
should always be  @code{Engraver_group_engraver} (unless you are
defining a Score context from scratch, in which case
@code{Score_engraver}   must be used).

The complete list of context  modifiers is as follows:
@itemize @bullet
@item @code{\alias} @var{alternate-name}:
This specifies a different name.  In the above example,
@code{\property Staff.X = Y} will also work on @code{SimpleStaff}s

@item @code{\consistsend} @var{engravername}:
Analogous to @code{\consists}, but makes sure that
@var{engravername} is always added to the end of the list of
engravers.

Engravers that group context objects into axis groups or alignments
need to be at the end of the list. @code{\consistsend} insures that
engravers stay at the end even if a user adds or removes engravers.
    
@item @code{\accepts} @var{contextname}:
This context can contains @var{contextname} contexts.  The first
@code{\accepts} is created as a default context when events (eg. notes
or rests) are encountered.

@item @code{\denies}:
The opposite of @code{\accepts}.

@item @code{\name} @var{contextname}:
This sets the type name of the context, e.g. @code{Staff},
@code{Voice}.  If the name is not specified, the translator will not
do anything.
@end itemize


@node Scheme integration
@section Scheme integration

@cindex Scheme
@cindex GUILE
@cindex Scheme, in-line code
@cindex accessing Scheme
@cindex evaluating Scheme
@cindex LISP

LilyPond internally uses GUILE, a Scheme-interpreter.  Scheme is a
language from the LISP family.  You can learn more about Scheme at
@uref{http://www.scheme.org}.  It is used to represent data throughout
the whole program.

In some places of the input file, Scheme expressions also form valid
syntax: wherever it is allowed, GUILE can be accessed directly by
entering a hash-sign (@code{#}).  The expression following the
hash-sign is evaluated as Scheme. For example,  the boolean value @var{true} is
@code{#t} in Scheme, so for LilyPond @var{true} looks like @code{##t},
and can be used in property assignments:
@example
  \property Staff.autoBeaming = ##f
@end example

Scheme is a full-blown programming language, and a full discussion is
outside the scope of this document.  Interested readers are referred to
the website @uref{http://www.schemers.org/} for more information on
Scheme.



@menu
* Input variables and Scheme::  
* Scheme datatypes::            
* Assignments::                 
@end menu

@node Input variables and Scheme
@subsection Input variables and Scheme


The input format supports the notion of variable: in the following
example, a music expression is assigned to a variable with the name
@code{traLaLa}.
@example
  traLaLa = \notes @{ c'4 d'4 @}
@end example

@noindent

There is also a form of scoping: in the following example, the
@code{\paper} block also contains a @code{traLaLa} variable, which is
independent of the outer \traLaLa.
@example
  traLaLa = \notes @{ c'4 d'4 @}
  \paper @{ traLaLa = 1.0 @}
@end example
@c
In effect, each input file is a scope, and all @code{\header},
@code{\midi} and @code{\paper} blocks are scopes nested inside that
toplevel scope.

Both variables and scoping are implemented in the GUILE module system.
A anonymous Scheme module is attached to each scope. An assignment of
the form
@example
 traLaLa = \notes @{ c'4 d'4 @} 
@end example

@noindent
is internally converted to a Scheme definition
@example
 (define traLaLa @var{Scheme value of @code{\notes ... }})
@end example

This means that input variables and Scheme variables may be freely
mixed.  In the following example, a music fragment is stored in the
variable @code{traLaLa}, and duplicated using Scheme. The result is
imported in a @code{\score} by means of a second variable
@code{twice}
@example
  traLaLa = \notes @{ c'4 d'4 @}
  
  #(define newLa (map ly:music-deep-copy
    (list traLaLa traLaLa)))
  #(define twice
    (make-sequential-music newLa))

  \score @{ \twice @}
@end example

In the above example, music expressions can be `exported' from the
input to the Scheme interpreter. The opposite is also possible. By
wrapping a Scheme value in the function @code{ly:export}, a Scheme
value is interpreted as if it were entered in LilyPond syntax: instead
of defining @code{\twice}, the example above could also have been
written as
@example
  @dots{}
  \score @{ #(ly:export (make-sequential-music newLa)) @}
@end example





@node Scheme datatypes
@subsection Scheme datatypes

Scheme is used to glue together different program modules. To aid this
glue function, many lilypond specific object types can be passed as
Scheme value. 

The following list are all lilypond specific types, that
can exist during parsing:
@itemize @bullet
@item Duration
@item Identifier
@item Input
@item Moment
@item Music
@item Music_output_def
@item Pitch
@item Score
@item Translator_def
@end itemize


During a run, transient objects are also created and destroyed.

@itemize @bullet
@item Grob: short for `Graphical object'.
@item Scheme_hash_table 
@item Music_iterator

@item Molecule: Device-independent page output object,
including dimensions.  

@item Syllable_group

@item Spring_smob

@item Translator: An object that produces audio objects or Grobs.  This is
not yet user-accessible.

@item Font_metric: An object representing a font.
@end itemize

Many functions are defined to manipulate these data structures. They
are all listed and documented in @internalsref{All scheme functions}.

@c .   {Assignments}
@node Assignments
@subsection Assignments
@cindex Assignments

Identifiers allow objects to be assigned to names during the parse
stage.  To assign an identifier, use @var{name}@code{=}@var{value}.
To refer to an identifier, precede its name with a backslash:
`@code{\}@var{name}'.  @var{value} is any valid Scheme value or any of
the input-types listed above.  Identifier assignments can appear at top
level in the LilyPond file, but also in @code{\paper} blocks.

An identifier can be created with any string for its name, but you will
only be able to refer to identifiers whose names begin with a letter,
being entirely alphabetical.  It is impossible to refer to an identifier
whose name is the same as the name of a keyword.

The right hand side of an identifier assignment is parsed completely
before the assignment is done, so it is allowed to redefine an
identifier in terms of its old value, e.g.

@example
foo = \foo * 2.0
@end example

When an identifier is referenced, the information it points to is
copied.  For this reason, an identifier reference must always be the
first item in a block.

@example
\paper @{
  foo = 1.0
  \paperIdent % wrong and invalid
@}
@end example

@example
\paper @{
  \paperIdent % correct
  foo = 1.0
@}
@end example

      

@node Music storage format
@section Music storage format

@menu
* Music expressions::           
* Manipulating music expressions::  
@end menu

@node Music expressions
@subsection Music expressions
@cindex music expressions

Music in LilyPond is entered as a music expression.  Notes, rests, lyric
syllables are music expressions, and you can combine music expressions
to form new ones, for example by enclosing a list of expressions in
@code{\sequential @{ @}} or @code{< >}.  In the following example, a
compound expression is formed out of the quarter note @code{c} and a
quarter note @code{d}:

@example 
\sequential @{ c4 d4 @} 
@end example 

@cindex Sequential music
@cindex @code{\sequential}
@cindex sequential music
@cindex @code{<}
@cindex @code{>}
@cindex Simultaneous music
@cindex @code{\simultaneous}

The two basic compound music expressions are simultaneous and
sequential music.

@example
\sequential @code{@{} @var{musicexprlist} @code{@}}
\simultaneous @code{@{} @var{musicexprlist} @code{@}}
@end example

For both, there is a shorthand:

@example
@code{@{} @var{musicexprlist} @code{@}}
@end example

@noindent
for sequential and

@example
@code{<} @var{musicexprlist} @code{>}
@end example

@noindent
for simultaneous music.
In principle, the way in which you nest sequential and simultaneous to
produce music is not relevant.  In the following example, three chords
are expressed in two different ways:

@lilypond[fragment,verbatim,center,quote]
\notes \context Voice {
  <a c'> <b d'> <c' e'>
  < { a b c' } { c' d' e' } >
}
@end lilypond
However, using @code{<} and @code{>} for entering chords leads to
various peculiarities. For this reason, a special syntax
for chords was introduced in version 1.7: @code{<< >>}.





Other compound music expressions include
@example
\repeat @var{expr}
\transpose @var{from} @var{to} @var{expr}
\apply @var{func} @var{expr}
\context @var{type} = @var{id} @var{expr}
\times @var{fraction} @var{expr}
@end example


@c . {Manipulating music expressions}
@node Manipulating music expressions
@subsection Manipulating music expressions

The @code{\apply} mechanism gives you access to the internal
representation of music.  You can write Scheme-functions that operate
directly on it.  The syntax is 

@example
\apply #@var{func} @var{music}
@end example

@noindent
This means that @var{func} is applied to @var{music}.  The function
@var{func} should return a music expression.

This example replaces the text string of a script.  It also shows a dump
of the music it processes, which is useful if you want to know more
about how music is stored.

@lilypond[verbatim,singleline]
#(define (testfunc x)
  (if (equal? (ly:get-mus-property x 'text) (make-simple-markup "foo"))
      (ly:set-mus-property! x 'text (make-simple-markup "bar")))
  ;; recurse
  (ly:set-mus-property! x 'elements
    (map testfunc (ly:get-mus-property x 'elements)))
  (display x)
  x)

\score {
  \notes
  \apply #testfunc { c'4_"foo" }
} 
@end lilypond

A final example is a function that reverses a piece of music in time:

@lilypond[verbatim,singleline]
#(define (reverse-music music)
  (let* ((elements (ly:get-mus-property music 'elements))
         (reversed (reverse elements))
         (span-dir (ly:get-mus-property music 'span-direction)))
    (ly:set-mus-property! music 'elements reversed)
    (if (ly:dir? span-dir)
        (ly:set-mus-property! music 'span-direction (- span-dir)))
    (map reverse-music reversed)
    music))

music = \notes { c'4 d'4( e'4 f'4 }

\score {
  \context Voice {
    \music
    \apply #reverse-music \music
  }
}
@end lilypond

@seealso

More examples are given in the distributed example files in
@code{input/test/}.


@refbugs

Directly accessing internal representations is dangerous: The
implementation is subject to changes, so you should avoid this feature
if possible.


@node Syntactic details
@section Syntactic details
@cindex Syntactic details

This section describes details that were too boring to be put elsewhere.

@menu
* Lexical modes::               
* Ambiguities::                 
@end menu


@c .  {Lexical modes}
@node Lexical modes
@subsection Lexical modes
@cindex Lexical modes
@cindex input mode
@cindex mode, input 
@cindex @code{\notes}
@cindex @code{\chords}
@cindex @code{\lyrics}

To simplify entering notes, lyrics, and chords, LilyPond has three
special input modes in addition to the default mode: note, lyrics, and
chords mode.  These input modes change the way that normal, unquoted
words are interpreted: For example, the word @code{cis} may be
interpreted as a C-sharp, as a lyric syllable `cis' or as a C-sharp
major triad respectively.

A mode switch is entered as a compound music expression

@example
@code{\notes} @var{musicexpr}
@code{\chords} @var{musicexpr}
@code{\lyrics} @var{musicexpr}
@code{\figures} @var{musicexpr}
@code{\markup} @var{markupexpr}
@end example

@noindent
In each of these cases, these expressions do not add anything to the
meaning of their arguments.  They just instruct the parser in what mode
to parse their arguments.

Different input modes for music expressions may be nested.


@c .  {Ambiguities}
@node Ambiguities
@subsection Ambiguities
@cindex ambiguities
@cindex grammar

The grammar contains two ambiguities.

@itemize @bullet
@item The assignment

@example 
foo = bar 
@end example 

@noindent
is interpreted as the string identifier assignment.  However,
it can also be interpreted as making a string identifier @code{\foo}
containing @code{"bar"}, or a music identifier @code{\foo} containing
the syllable `bar'.  The former interpretation is chosen.

@item If you do a nested repeat like

@example 
\repeat @dots{}
\repeat @dots{}
\alternative 
@end example 

@noindent
then it is ambiguous to which @code{\repeat} the
@code{\alternative} belongs.  This is the classic if-then-else
dilemma.  It may be solved by using braces.
@end itemize


@c .  {Lexical details}
@node Lexical details
@section Lexical details

@menu
* Strings::                     
@end menu

@node Strings
@subsection Strings
@cindex string
@cindex concatenate

Begins and ends with the @code{"} character.  To include a @code{"}
character in a string write @code{\"}.  Various other backslash
sequences have special interpretations as in the C language.  A string
that contains no spaces can be written without the quotes.  Strings can
be concatenated with the @code{+} operator.


@c .  {Output details}
@node Output details
@section Output details

LilyPond's default output format is @TeX{}.  Using the option @option{-f}
(or @option{--format}) other output formats can be selected also, but
currently none of them reliably work.

At the beginning of the output file, various global parameters are defined.
It also contains a large @code{\special} call to define PostScript routines
to draw items not representable with @TeX{}, mainly slurs and ties.  A DVI
driver must be able to understand such embedded PostScript, or the output
will be rendered incompletely.

Then the file @file{lilyponddefs.tex} is loaded to define the macros used
in the code which follows.  @file{lilyponddefs.tex} includes various other
files, partially depending on the global parameters.

Now the music is output system by system (a `system' consists of all
staves belonging together).  From @TeX{}'s point of view, a system is an
@code{\hbox} which contains a lowered @code{\vbox} so that it is centered
vertically on the baseline of the text.  Between systems,
@code{\interscoreline} is inserted vertically to have stretchable space.
The horizontal dimension of the @code{\hbox} is given by the
@code{linewidth} parameter from LilyPond's @code{\paper} block.


After the last system LilyPond emits a stronger variant of
@code{\interscoreline} only if the macro
@code{\lilypondpaperlastpagefill} is not defined (flushing the systems
to the top of the page).  You can avoid that by setting the variable
@code{lastpagefill} in LilyPond's @code{\paper} block.

It is possible to fine-tune the vertical offset further by defining the
macro @code{\lilypondscoreshift}.  Example:

@example
\def\lilypondscoreshift@{0.25\baselineskip@}
@end example

@noindent
@code{\baselineskip} is the distance from one text line to the next.

The code produced by LilyPond should be run through La@TeX{}, not
plain @TeX{}.

Here an example how to embed a small LilyPond file @code{foo.ly} into
running La@TeX{} text without using the @code{lilypond-book} script
(@pxref{Integrating text and music with lilypond-book}).

@example
\documentclass@{article@}

\def\lilypondpaperlastpagefill@{@}
\lineskip 5pt
\def\lilypondscoreshift@{0.25\baselineskip@}

\begin@{document@}
This is running text which includes an example music file
\input@{foo.tex@}
right here.
\end@{document@}
@end example

The file @file{foo.tex} has been simply produced with

@example
lilypond foo.ly
@end example

It is important to set the @code{indent} parameter to zero in the
@code{\paper} block of @file{foo.ly}.

The call to @code{\lineskip} assures that there is enough vertical space
between the LilyPond box and the surrounding text lines.

@c EOF