Run scripts/auxiliar/update-with-convert-ly.sh

[lilypond.git] / Documentation / essay / engraving.itely

@c -*- coding: utf-8; mode: texinfo; -*-

@ignore
    Translation of GIT committish: FILL-IN-HEAD-COMMITTISH

    When revising a translation, copy the HEAD committish of the
    version that you are working on.  For details, see the Contributors'
    Guide, node Updating translation committishes..
@end ignore

@c \version "2.16.0"

@node Music engraving
@chapter Music engraving

This essay describes why LilyPond was created and how it can produce
such beautiful sheet music.

@c TODO:
@c remove 3mm eps bounding box left padding for Sarabande (This will
@c     require adding a new snippet option to lilypond-book.py
@c check formatting of HTML output

@menu
* The LilyPond story::
* Engraving details::
* Automated engraving::
* Building software::
* Putting LilyPond to work::
* Engraved examples (BWV 861)::
@end menu

@node The LilyPond story
@section The LilyPond story

Long before LilyPond had been used to engrave beautiful performance
scores, before it could create university course notes or even simple
melodies, before there was a community of users around the world or even
an essay on music engraving, LilyPond began with a question:

@quotation
Why does most computer output fail to achieve the beauty and balance of
a hand-engraved score?
@end quotation

@noindent
Some of the answers can be found by examining the two scores
@ifnottex
below.
@end ifnottex
@iftex
on the following pages.
@end iftex
The first score is a beautiful hand-engraved score from 1950 and the
second is a modern, computer-engraved edition.

@ifnottex
@quotation
@noindent
Bärenreiter BA 320, @copyright{}1950:

@sourceimage{baer-suite1-fullpage,,,png}
@end quotation

@quotation
@noindent
Henle no. 666, @copyright{}2000:

@sourceimage{henle-suite1-fullpage,,,png}
@end quotation
@end ifnottex

The notes here are identical, taken from Bach's first Suite for solo
cello, but the appearance is different, especially if you print them out
and view them from a distance.
@ifnottex
(The PDF version of this manual has high-resolution images suitable for
printing.)
@end ifnottex
Try reading or playing from each of the scores and you will find that
the hand-engraved score is more enjoyable to use.  It has flowing lines
and movement, and it feels like a living, breathing piece of music,
while the newer edition seems cold and mechanical.

It is hard to immediately see what makes the difference with the newer
edition.  Everything looks neat and tidy, possibly even @qq{better}
because it looks more computerized and uniform.  This really puzzled us
for quite a while.  We wanted to improve computer notation, but we first
had to figure out what was wrong with it.

The answer lies in the precise, mathematical uniformity of the newer
edition.  Find the bar line in the middle of each line: in the
hand-engraved score the position of these bar lines has some
natural variation, while in the newer version they line up almost
perfectly.  This is shown in these simplified page layout diagrams,
traced from the hand-engraved (left) and computer-generated music
(right):

@quotation
@iftex
@sourceimage{pdf/page-layout-comparison,,,}
@end iftex
@ifnottex
@sourceimage{page-layout-comparison,,,png}
@end ifnottex
@end quotation
@noindent

In the computer-generated output, even the individual note heads are
aligned in vertical columns, making the contour of the melody disappear
into a rigid grid of musical markings.

There are other differences as well: in the hand-engraved edition the
vertical lines are all stronger, the slurs lie closer to the note heads,
and there is more variety in the slopes of the beams.  Although such
details may seem like nitpicking, the result is a score that is easier
to read.  In the computer-generated output, each line is nearly identical
and if the musician looks away for a moment she will be lost on the
page.

LilyPond was designed to solve the problems we found in existing
software and to create beautiful music that mimics the finest
hand-engraved scores.

@iftex
@page
@noindent
Bärenreiter BA 320, @copyright{}1950:

@sourceimage{baer-suite1-fullpage-hires,16cm,,}
@page
@noindent
Henle no. 666, @copyright{}2000:
@sp 3
@sourceimage{henle-suite1-fullpage-hires,16cm,,}
@page
@end iftex

@node Engraving details
@section Engraving details

@cindex engraving
@cindex typography, music
@cindex music typography
@cindex plate engraving
@cindex music engraving

The art of music typography is called @emph{(plate) engraving}, a term
that derives from the manual process of music printing@footnote{Early
European printers explored several processes, including hand-carved
wooden blocks, movable type, and engraved sheets of thin metal.
Typesetting had the advantage of being more easily corrected and
facilitating the inclusion of text and lyrics, but only engraving
offered the ability to do unimpeded layout and unanticipated notation.
In the end, hand-engraved scores became the standard for all printed
music, with the exception of some hymnals and songbooks where
typesetting was justified by its ease and economy, even into the
twentieth century.}.  Just a few decades ago, sheet music was made by
cutting and stamping the music into a zinc or pewter plate in mirror
image.  The plate would be inked, and the depressions caused by the
cutting and stamping would hold ink.  An image was formed by pressing
paper to the plate.  The stamping and cutting was done completely by hand
and making a correction was cumbersome, so the engraving had to be
nearly perfect in one go.  Engraving was a highly specialized skill; a
craftsman had to complete around five years of training before earning
the title of master engraver, and another five years of experience were
necessary to become truly skilled.

@quotation
@iftex
@sourceimage{hader-slaan,,7cm,}
@end iftex
@ifnottex
@sourceimage{hader-slaan,,,jpg}
@end ifnottex
@end quotation

LilyPond is inspired by traditional manual engravings published by
European music publishers in and towards the end of the first half of
the twentieth century, including Bärenreiter, Duhem, Durand,
Hofmeister, Peters, and Schott.  This is sometimes regarded as the peak
of traditional musical engraving practice.  As we have studied these
editions we have learned a great deal about what goes into a
well-engraved score, and the aspects that we wanted to imitate in
LilyPond.

@c Now all newly printed music is produced with computers.  This has
@c obvious advantages: prints are cheaper to make, editorial work can be
@c delivered by email, and the original data can be easily stored.
@c Unfortunately, computer-generated scores rarely match the quality of
@c hand-engraved scores.  Instead, computer printouts have a bland,
@c mechanical look, which makes them unpleasant to play from.

@menu
* Music fonts::
* Optical spacing::
* Ledger lines::
* Optical sizing::
* Why work so hard?::
@end menu

@node Music fonts
@unnumberedsubsec Music fonts

The images below illustrate some differences between traditional
engraving and typical computer output.  The left picture shows a scan of
a flat symbol from a hand-engraved Bärenreiter edition, while the right
picture depicts a symbol from an edition of the same music published in
2000.  Although both images are printed in the same shade of ink, the
earlier version looks darker: the staff lines are heavier, and the
Bärenreiter flat has a bold, almost voluptuous rounded look.  The right
scan, on the other hand, has thinner lines and a straight layout with
sharp corners.

@multitable @columnfractions .25 .25 .25 .25
@item @tab
@ifnotinfo
@iftex
@sourceimage{baer-flat-gray,,4cm,}
@end iftex
@ifnottex
@sourceimage{baer-flat-gray,,,png}
@end ifnottex

@tab
@iftex
@sourceimage{henle-flat-gray,,4cm,}
@end iftex
@ifnottex
@sourceimage{henle-flat-gray,,,png}
@end ifnottex

@end ifnotinfo
@ifinfo
@sourceimage{henle-flat-bw,,,png}
@sourceimage{baer-flat-bw,,,png}
@sourceimage{lily-flat-bw,,,png}
@end ifinfo


@item @tab
Bärenreiter (1950)
@tab
Henle (2000)

@end multitable

@cindex musical symbols
@cindex font
@cindex blackness
@cindex balance

When we wanted to write a computer program to create music typography,
there were no musical fonts freely available that could match the
elegance of our favorite scores.  Undeterred, we created a font of
musical symbols, relying on nice printouts of hand-engraved music.  The
experience helped develop a typographical taste, and it made us
appreciate subtle design details.  Without that experience, we would not
have realized how ugly the fonts were that we admired at first.

Below is a sample of two music fonts: the upper set is the default font
in the Sibelius software (the @emph{Opus} font), and the lower set is
our own LilyPond font.

@quotation
@iftex
@sourceimage{pdf/OpusAndFeta,,,}
@end iftex
@ifnottex
@sourceimage{OpusAndFeta,,,png}
@end ifnottex
@end quotation

The LilyPond symbols are heavier and their weight is more consistent,
which makes them easier to read.  Fine endings, such as the ones on the
sides of the quarter rest, should not end in sharp points, but rather
in rounded shapes.  This is because sharp corners of the punching dies
are fragile and quickly wear out when stamping in metal.  Taken together,
the blackness of the font must be carefully tuned together with the
thickness of lines, beams and slurs to give a strong yet balanced
overall impression.

Also, notice that our half-note head is not elliptic but slightly diamond
shaped.  The vertical stem of a flat symbol is slightly brushed, becoming
wider at the top.  The sharp and the natural are easier to distinguish
from a distance because their angled lines have different slopes and the
vertical strokes are heavier.

@node Optical spacing
@unnumberedsubsec Optical spacing

In spacing, the distribution of space should reflect the durations
between notes.  However, as we saw in the Bach Suite above, many modern
scores adhere to the durations with mathematical precision, which leads
to poor results.  In the next example a motif is printed twice: the
first time using exact mathematical spacing, and the second with
corrections.  Which do you prefer?

@cindex optical spacing

@lilypond
\paper {
  ragged-right = ##t
  indent = #0.0
}

music = {
   c'4 e''4 e'4 b'4
   \stemDown
   b'8[ e'' a' e'']
   \stemNeutral
   e'8[ e'8 e'8 e'8]
}
\score
{
  \music
  \layout {
    \context {
      \Staff
      \override NoteSpacing #'stem-spacing-correction = #0.0
      \override NoteSpacing #'same-direction-correction = #0.0
      \override StaffSpacing #'stem-spacing-correction = #0.0
    }
  }
}
@end lilypond

@lilypond
\paper {
  ragged-right = ##t
  indent = #0.0
}

music = {
   c'4 e''4 e'4 b'4 |
   \stemDown
   b'8[ e'' a' e'']
   \stemNeutral
   e'8[ e'8 e'8 e'8]
}

\score
{
  \music
  \layout {
    \context {
      \Staff
      \override NoteSpacing #'stem-spacing-correction = #0.6
    }
  }
}
@end lilypond

@cindex regular rhythms
@cindex regular spacing
@cindex spacing, regular

Each bar in the fragment only uses notes that are played in a constant
rhythm.  The spacing should reflect that.  Unfortunately, the eye deceives
us a little; not only does it notice the distance between note heads, it
also takes into account the distance between consecutive stems.  As a
result, the notes of an up-stem/@/down-stem combination should be put
farther apart, and the notes of a down-stem/@/up-stem combination should
be put closer together, all depending on the combined vertical positions
of the notes.  The lower two measures are printed with this correction,
the upper two measures, however, form down-stem/@/up-stem clumps of
notes.  A master engraver would adjust the spacing as needed to please
the eye.

The spacing algorithms in LilyPond even take the barlines into account,
which is why the final up-stem in the properly spaced example has been
given a little more space before the barline to keep it from looking
crowded.  A down-stem would not need this adjustment.

@node Ledger lines
@unnumberedsubsec Ledger lines

@cindex ledger lines
@cindex collisions

Ledger lines present a typographical challenge: they make it more
difficult to space musical symbols close together and they must be clear
enough to identify the pitch at a glance.  In the example below, we see
that ledger lines should be thicker than normal staff lines and that an
expert engraver will shorten a ledger line to allow closer spacing with
accidentals.  We have included this feature in LilyPond's engraving.

@multitable @columnfractions .25 .25 .25 .25
@item @tab

@iftex
@sourceimage{baer-ledger,3cm,,}
@end iftex
@ifnottex
@sourceimage{baer-ledger,,,png}
@end ifnottex

@tab

@iftex
@sourceimage{lily-ledger,3cm,,}
@end iftex
@ifnottex
@sourceimage{lily-ledger,,,png}
@end ifnottex

@end multitable


@node Optical sizing
@unnumberedsubsec Optical sizing

Music may need to be printed in a range of sizes.  Originally, this was
accomplished by creating punching dies in each of the required sizes,
which meant that each die was designed to look its best at that size.
With the advent of digital fonts, a single outline can be mathematically
scaled to any size, which is very convenient, but at the smaller sizes
the glyphs will appear very light.

In LilyPond, we have created fonts in a range of weights, corresponding
to a range of music sizes.  This is a LilyPond engraving at staff size
26:

@quotation
@iftex
@sourceimage{pdf/size26,,23mm,}
@end iftex
@ifnottex
@sourceimage{size26,,,png}
@end ifnottex
@end quotation

@noindent
and this is the same engraving set at staff size 11, then
magnified by 236% to print at the same size as the previous example:

@quotation
@iftex
@sourceimage{pdf/size11,,23mm,}
@end iftex
@ifnottex
@sourceimage{size11,,,png}
@end ifnottex
@end quotation

@noindent
At smaller sizes, LilyPond uses proportionally heavier lines so the
music will still read well.

@ignore
This also allows staves of different sizes to coexist peacefully when
used together on the same page:

@c TODO: are the stems in this example the right thickness? How should
@c line weights be scaled for small staves?

@c Grieg's Violin Sonata Op. 45
@lilypond[indent=1.5cm]
global = {
  \time 6/8
  \key c \minor
}

\score {
  <<
    \new Staff \with {
      fontSize = #-4
      \override StaffSymbol #'staff-space = #(magstep -4)
      \override StaffSymbol #'thickness = #(magstep -3)
    }
    \relative c' {
      \global
      \set Staff.instrumentName = #"Violin"
      c8.(\f^> b16 c d) ees8.(^> d16 c b)
      g8.(^> b16 c ees) g8-.^> r r
      R2.
    }
    \new PianoStaff <<
      \set PianoStaff.instrumentName = #"Piano"
      \new Staff \relative c' {
        \global
        s2.
        s4. s8 r8 r16 <c f aes c>
        <c f aes c>4.^> <c ees g>8 r r
      }
      \new Staff \relative c {
        \global
        \clef "bass"
        <<
        {
          \once \override DynamicText #'X-offset = #-3
          <ees g c>2.~->^\f
          <ees g c>4.~ <ees g c>8
        } \\ {
          <c g c,>2.~
          <c g c,>4.~ <c g c,>8
        }
        >>
        r8 r16 <f, c' aes'>16
        <f c' aes'>4.-> <c' g'>8 r r
      }
    >>
  >>
}
@end lilypond
@end ignore

@node Why work so hard?
@unnumberedsubsec Why work so hard?

Musicians are usually more absorbed with performing than with studying
the looks of a piece of music, so nitpicking typographical details may
seem academic.  But it is not.  Sheet music is performance material:
everything is done to aid the musician in letting her perform better,
and anything that is unclear or unpleasant to read is a hindrance.

Traditionally engraved music uses bold symbols on heavy staff to create
a strong, well-balanced look that stands out well when the music is far
away from the reader: for example, if it is on a music stand.  A careful
distribution of white space allows music to be set very tightly without
crowding symbols together.  The result minimizes the number of page
turns, which is a great advantage.

This is a common characteristic of typography.  Layout should be pretty,
not only for its own sake, but especially because it helps the reader in
his task.  For sheet music this is of double importance because musicians
have a limited amount of attention.  The less attention they need for
reading, the more they can focus on playing the music.  In other words,
better typography translates to better performances.

These examples demonstrate that music typography is an art that is
subtle and complex, and that producing it requires considerable
expertise, which musicians usually do not have.  LilyPond is our
effort to bring the graphical excellence of hand-engraved music to
the computer age, and make it available to normal musicians.  We
have tuned our algorithms, font-designs, and program settings to
produce prints that match the quality of the old editions we love
to see and love to play from.


@node Automated engraving
@section Automated engraving

@cindex engraving, automated
@cindex automated engraving

Here we describe what is required to create software that can mimic the
layout of engraved scores: a method of describing good layouts to the
computer and a lot of detailed comparisons with real engravings.

@menu
* Beauty contests::
* Improvement by benchmarking::
* Getting things right::
@end menu

@node Beauty contests
@unnumberedsubsec Beauty contests

How do we actually make formatting decisions?  In other words, which
of the three configurations should we choose for the following slur?

@lilypond
\relative c {
  \clef bass
  \once \override Slur #'positions = #'(1.5 . 1)
  e8[( f] g[ a b d,)] r4
  \once \override Slur #'positions = #'(2 . 3)
  e8[( f] g[ a b d,)] r4
  e8[( f] g[ a b d,)] r4
}
@end lilypond

There are a few books on the art of music engraving
available.  Unfortunately, they contain simple rules of thumb and some
examples.  Such rules can be instructive, but they are a far cry from
an algorithm that we could readily implement in a computer.  Following
the instructions from literature leads to algorithms with lots of
hand-coded exceptions.  Doing all this case analysis is a lot of work,
and often not all cases are covered completely:

@quotation
@iftex
@sourceimage{ross-beam-scan,7cm,,}
@end iftex
@ifnottex
@sourceimage{ross-beam-scan,,,.jpg}
@end ifnottex
@end quotation

(Image source: Ted Ross, @emph{The Art of Music Engraving})

Rather than trying to write detailed layout rules for every possible
scenario, we only have to describe the objectives well enough that
LilyPond can judge the attractiveness of several alternatives.  Then,
for each possible configuration we compute an ugliness score and we
choose the least ugly configuration.

For example, here are three possible slur configurations, and LilyPond
has given each one a score in @q{ugly points}.  The first example gets
15.39 points for grazing one of the noteheads:

@lilypond
\relative c {
  \clef bass
  \once \override Slur #'positions = #'(1.5 . 1)
  e8[(_"15.39" f] g[ a b d,)] r4
}
@end lilypond

The
second one is nicer, but the slur doesn't start or end on the note heads.
It gets 1.71 points for the left side and 9.37 points for the right
side, plus another 2 points because the slur ascends while the melody
descends for a total of 13.08 ugly points:

@lilypond
\relative c {
  \clef bass
  \once \override Slur #'positions = #'(2 . 3)
  e8[(_"13.08" f] g[ a b d,)] r4
}
@end lilypond

The final slur gets 10.04
points for the gap on the right and 2 points for the upward slope, but
it is the most attractive of the three configurations, so LilyPond
selects this one:

@lilypond
\relative c {
  \clef bass
  e8[(_"12.04" f] g[ a b d,)] r4
}
@end lilypond

This technique is quite general, and is used to make optimal decisions
for beam configurations, ties and dots in chords, line breaks, and page
breaks.  The results of these decisions can be judged by comparison to
real engravings.

@node Improvement by benchmarking
@unnumberedsubsec Improvement by benchmarking

LilyPond's output has improved gradually over time, and it continues to
improve by comparing its output to hand-engraved scores.

For example, here is one line of a benchmark piece from a
hand-engraved edition (Bärenreiter BA320):

@iftex
@sourceimage{baer-sarabande-hires,16cm,,}
@end iftex
@ifnottex
@sourceimage{baer-sarabande,,,png}
@end ifnottex

@noindent
and the same quotation as engraved by a very old version of LilyPond
(version 1.4, May 2001):

@iftex
@sourceimage{pdf/lily14-sarabande,16cm,,}
@end iftex
@ifnottex
@sourceimage{lily14-sarabande,,,png}
@end ifnottex

@noindent The LilyPond 1.4 output is certainly readable, but close
comparison with the hand-engraved score showed a lot of errors in the
formatting details:

@iftex
@sourceimage{lily14-sarabande-annotated-hires,16cm,,}
@end iftex
@ifnottex
@sourceimage{lily14-sarabande-annotated,,,png}
@end ifnottex

@itemize @bullet
@item there is too much space before the time signature
@item the stems of the beamed notes are too long
@item the second and fourth measures are too narrow
@item the slur is awkward-looking
@item the trill marks are too big
@item the stems are too thin
@end itemize

@noindent
(There were also two missing note heads, several missing editorial
annotations, and an incorrect pitch!)

By adjusting the layout rules and font design, the output has improved
considerably.  Compare the same reference score and the output
from the current version of LilyPond (@version{}):

@iftex
@sourceimage{baer-sarabande-hires,16cm,,}
@end iftex
@ifnottex
@sourceimage{baer-sarabande,,,png}
@end ifnottex

@lilypond[staffsize=17.5,line-width=15.9\cm]
\relative c {
  \clef "bass"
  \key d \minor
  \time 3/4
  \mergeDifferentlyDottedOn
  <<
    { \slurDashed d8.-\flageolet( e16) e4.-\trill( d16 e) }
     \\
    { d4_2 a2 }
  >>
  \slurDashed
  <f' a, d,>4. e8( d c)
  \slurSolid
  bes8 g' f e16( f g_1 a_2 bes_3 d,_2)
  \slurDashed
  cis4.-\trill b8_3( a g)
  <<
    { \slurDashed d'8.( e16) e4.-\trill( d16 e) }
     \\
    { <f, a>4 a2 }
  >>
}
@end lilypond

@noindent
The current output is not a clone of the reference edition, but it is
much closer to publication quality that the earlier output.

@node Getting things right
@unnumberedsubsec Getting things right

We can also measure LilyPond's ability to make music engraving decisions
automatically by comparing LilyPond's output to the output of a
commercial software product.  In this case we have chosen Finale 2008,
which is one of the most popular commercial score writers, particularly
in North America.  Sibelius is its major rival and appears to be
especially strong in the European market.

For our comparison we selected Bach's Fugue in G minor from the
Well-Tempered Clavier, Book I, BWV 861, whose opening subject is

@lilypond
\relative c' {
  \key g \minor
  \clef "treble_8"
  r8 d ees g, fis4 g
  r8 a16 bes c8 bes16 a bes8
}
@end lilypond

@noindent

We made our comparison by engraving the last seven measures of the piece
(28--34) in Finale and LilyPond.  This is the point in the piece where
the subject returns in a three-part stretto and leads into the closing
section.  In the Finale version, we have resisted the temptation to make
any adjustments to the default output because we are trying to show the
things that each software package gets right without assistance.  The
only major edits that we made were adjusting the page size to match this
essay and forcing the music onto two systems to make the comparison
easier.  By default Finale would have engraved two systems of three
measures each and a final, full-width system containing a single
measure.

Many of the differences between the two engravings are visible in
measures 28--29, as shown here with Finale first and LilyPond second:

@iftex
@sourceimage{pdf/bwv861mm28-29,14cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861mm28-29,,,png}
@end ifnottex

@lilypond[staffsize=19.5,line-width=14\cm]
global = { \key g \minor }

partI = \relative c' {
  \voiceOne
  fis8 d' ees g, fis4 g
  r8 a16 bes c8 bes16 a d8 r r4
}

partII = \relative c' {
  \voiceTwo
  d4 r4 r8 d'16 c bes8 c16 d
  ees8 d c ees a, r r4
}

partIII = \relative c' {
  \voiceOne
  r2 r8 d ees g, fis4 g r8 a16 bes c8 bes16 a
}

partIV = \relative c {
  \voiceTwo
  d4 r r2
  r8 d ees g, fis4 a
}

\score {
  <<
    % \set Score.barNumberVisibility = #all-bar-numbers-visible
    % required in 2.13
    \set Score.currentBarNumber = #28
    \bar ""
    \new PianoStaff <<
      \new Staff = "RH" <<
        \global
        \new Voice = "voiceI" { \partI }
        \new Voice = "voiceII" { \partII }
      >>
      \new Staff = "LH"
      <<
        \clef "bass"
        \global
        \new Voice = "voiceIII" { \partIII }
        \new Voice = "voiceIV" { \partIV }
      >>
    >>
  >>
  \layout {
    \context {
      \Staff
      \remove "Time_signature_engraver"
    }
    \context {
      \PianoStaff
      \override StaffGrouper #'staff-staff-spacing #'padding = #1
    }
  }
}
@end lilypond

Some shortcomings in the unedited Finale output include:
@itemize @bullet
@item Most of the beams extend too far off the staff.  A beam that points
towards the center of the staff should have a length of about one
octave, but engravers shorten this when the beam points away from the
staff in multi-voice music.  The Finale beaming can easily be improved
with their Patterson Beams plug-in, but we elected to skip that step for
this example.
@item Finale doesn't adjust the positions of interlocking note heads,
which makes the music extremely difficult to read when the upper and
lower voices exchange positions temporarily:

@lilypond
collide = \once \override NoteColumn #'force-hshift = #0

\score {
  <<
    \new Voice = "sample" \relative c''{
      \key g \minor
      <<
        { \voiceOne g4 \collide g4 }
        \new Voice { \voiceTwo bes \collide bes }
      >>
    }
    \new Lyrics \lyricsto "sample" \lyricmode { "good " " bad" }
  >>
}
@end lilypond

@item Finale has placed all of the rests at fixed heights on the staff.
The user is free to adjust them as needed, but the software makes no
attempt to consider the content of the other voice.  As luck would have
it, there are no true collisions between notes and rests in this example,
but that has more to do with the positions of the notes than the rest.
In other words, Bach deserves more credit for avoiding a complete
collision than Finale does.

@end itemize

This example is not intended to suggest that Finale cannot be used to
produce publication-quality output.  On the contrary, in the hands of a
skilled user it can and does, but it requires skill and time.  One of the
fundamental differences between LilyPond and commercial scorewriters is
that LilyPond hopes to reduce the amount of human intervention to an
absolute minimum, while other packages try to provide an attractive
interface in which to make these types of edits.

One particularly glaring omission we found from Finale is a missing flat
in measure 33:

@quotation
@iftex
@sourceimage{pdf/bwv861mm33-34-annotate,7.93cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861mm33-34-annotate,,,png}
@end ifnottex
@end quotation

@noindent
The flat symbol is required to cancel out the natural in the same
measure, but Finale misses it because it occurred in a different voice.
So in addition to running a beaming plug-in and checking the spacing on
the noteheads and rests, the user must also check each measure for
cross-voice accidentals to avoid interrupting a rehearsal over an
engraving error.

If you are interested in examining these examples in more detail, the
full seven-measure excerpt can be found at the end of this essay along
with four different published engravings.  Close examination reveals that
there is some acceptable variation among the hand-engravings, but that
LilyPond compares reasonably well to that acceptable range.  There are
still some shortcomings in the LilyPond output, for example, it appears
a bit too aggressive in shortening some of the stems, so there is room
for further development and fine-tuning.

Of course, typography relies on human judgment of appearance, so people
cannot be replaced completely.  However, much of the dull work can be
automated.  If LilyPond solves most of the common situations correctly,
this will be a huge improvement over existing software.  Over the course
of years, the software can be refined to do more and more things
automatically, so manual overrides are less and less necessary.  Where
manual adjustments are needed, LilyPond's structure has been designed
with that flexibility in mind.

@node Building software
@section Building software

This section describes some of the programming decisions that we made
when designing LilyPond.

@menu
* Music representation::
* What symbols to engrave?::
* Flexible architecture::
@end menu


@node Music representation
@unnumberedsubsec Music representation

@cindex syntax
@cindex recursive structures

Ideally, the input format for any high-level formatting system is
an abstract description of the content.  In this case, that would
be the music itself.  This poses a formidable problem: how can we
define what music really is? Instead of trying to find an answer,
we have reversed the question.  We write a program capable of
producing sheet music, and adjust the format to be as lean as
possible.  When the format can no longer be trimmed down, by
definition we are left with content itself.  Our program serves as
a formal definition of a music document.

The syntax is also the user-interface for LilyPond, hence it is
easy to type:

@example
@{
  c'4 d'8
@}
@end example

@noindent
to create a quarter note on middle C (C1) and an eighth note on
the D above middle C (D1).

@lilypond[quote]
{
  c'4 d'8
}
@end lilypond

On a microscopic scale, such syntax is easy to use.  On a larger
scale, syntax also needs structure.  How else can you enter
complex pieces like symphonies and operas?  The structure is
formed by the concept of music expressions: by combining small
fragments of music into larger ones, more complex music can be
expressed.  For example

@lilypond[quote,verbatim,fragment,relative=1]
f4
@end lilypond

@noindent
Simultaneous notes can be constructed by enclosing them with
@code{<<} and @code{>>}:

@example
<<c4 d4 e4>>
@end example

@lilypond[quote,fragment,relative=1]
\new Voice { <<c4 d4 e>> }
@end lilypond

@noindent
This expression is put in sequence by enclosing it in curly braces
@code{@{@tie{}@dots{}@tie{}@}}:

@example
@{ f4 <<c4 d4 e4>> @}
@end example

@lilypond[quote,relative=1,fragment]
{ f4 <<c d e4>> }
@end lilypond

@noindent
The above is also an expression, and so it may be combined again
with another simultaneous expression (a half note) using
@code{<<}, @code{\\}, and @code{>>}:

@example
<< g2 \\ @{ f4 <<c4 d4 e4>> @} >>
@end example

@lilypond[quote,fragment,relative=2]
\new Voice { << g2 \\ { f4 <<c d e>> } >> }
@end lilypond

Such recursive structures can be specified neatly and formally in
a context-free grammar.  The parsing code is also generated from
this grammar.  In other words, the syntax of LilyPond is clearly
and unambiguously defined.

User-interfaces and syntax are what people see and deal with most.
They are partly a matter of taste, and also the subject of much
discussion.  Although discussions on taste do have their merit,
they are not very productive.  In the larger picture of LilyPond,
the importance of input syntax is small: inventing neat syntax is
easy, while writing decent formatting code is much harder.  This
is also illustrated by the line-counts for the respective
components: parsing and representation take up less than 10% of
the source code.

When designing the structures used in LilyPond, we made some different
decisions than are apparent in other software.  Consider the hierarchical
nature of music notation:

@lilypond[quote,fragment]
<<
  \new Staff \relative c'' {
    \key g \major
    \time 3/4
    d4 g,8 a b c d4 g, g
  }
  \new Staff \relative c' {
    \clef "bass"
    \key g \major
    <g b d>2 a4 b2.
  }
>>
@end lilypond

In this case, there are pitches grouped into chords that belong to
measures, which belong to staves.  This resembles a tidy structure of
nested boxes:

@quotation
@iftex
@sourceimage{pdf/nestedboxes,,4cm,}
@end iftex
@ifnottex
@sourceimage{nestedboxes,,,png}
@end ifnottex
@end quotation

Unfortunately, the structure is tidy because it is based on some
excessively restrictive assumptions.  This becomes apparent if we
consider a more complicated musical example:

@lilypond[quote]
\layout {
  \context {
    \Score
    \remove "Timing_translator"
    \remove "Default_bar_line_engraver"
  }
  \context {
    \Staff
    \consists "Timing_translator"
    \consists "Default_bar_line_engraver"
  }
}

\new PianoStaff <<
  \new Staff = "RH" <<
    \new Voice = "I" \relative c''' {
      \time 3/4
      \voiceOne
      \times 6/7 { g8 g g g g g g }
      \oneVoice
      r4 <b,, fis' g bes> r4\fermata
    }
    \new Voice = "II" \relative c' {
      \voiceTwo
      c4
      \times 4/5 {
        <c ees>8 f g
        \change Staff = "LH" \oneVoice
        \stemUp g,( c}
      r4
      \override Stem #'cross-staff = ##t
      \override Stem #'length = #12
      <fis, b>) r\fermata
    }
  >>
  \new Staff = "LH" <<
    \new Voice = "III" \relative c' {
      \time 2/4
      \clef "bass"
      g4 \stopStaff s
      \startStaff s2*2
    }
  >>
>>
@end lilypond

In this example, staves start and stop at will, voices jump around
between staves, and the staves have different time signatures.  Many
software packages would struggle with reproducing this example because
they are built on the nested box structure.  With LilyPond, on the other
hand, we have tried to keep the input format and the structure as
flexible as possible.

@node What symbols to engrave?
@unnumberedsubsec What symbols to engrave?

@cindex engraving
@cindex typography
@cindex engraver
@cindex plug-in

The formatting process decides where to place symbols.  However,
this can only be done once it is decided @emph{what} symbols
should be printed -- in other words, what notation to use.

Common music notation is a system of recording music that has
evolved over the past 1000 years.  The form that is now in common
use dates from the early Renaissance.  Although the basic form
(i.e., note heads on a 5-line staff) has not changed, the details
still evolve to express the innovations of contemporary notation.
Hence, common music notation encompasses some 500 years of music.
Its applications range from monophonic melodies to monstrous
counterpoints for a large orchestra.

How can we get a grip on such a seven-headed beast, and force it
into the confines of a computer program?  Our solution is to break
up the problem of notation (as opposed to engraving, i.e.,
typography) into digestible and programmable chunks: every type of
symbol is handled by a separate module, a so-called plug-in.  Each
plug-in is completely modular and independent, so each can be
developed and improved separately.  Such plug-ins are called
@code{engraver}s, by analogy with craftsmen who translate musical
ideas to graphic symbols.

In the following example, we start out with a plug-in for note
heads, the @code{Note_heads_engraver}.

@lilypond[quote,ragged-right]
\include "engraver-example.ily"

\score {
  \topVoice
  \layout {
    \context {
      \Voice
      \remove "Stem_engraver"
      \remove "Phrasing_slur_engraver"
      \remove "Slur_engraver"
      \remove "Script_engraver"
      \remove "Beam_engraver"
      \remove "Auto_beam_engraver"
    }
    \context {
      \Staff
      \remove "Accidental_engraver"
      \remove "Key_engraver"
      \remove "Clef_engraver"
      \remove "Bar_engraver"
      \remove "Time_signature_engraver"
      \remove "Staff_symbol_engraver"
      \consists "Pitch_squash_engraver"
    }
  }
}
@end lilypond

@noindent
Then a @code{Staff_symbol_engraver} adds the staff,

@lilypond[quote,ragged-right]
\include "engraver-example.ily"

\score {
  \topVoice
  \layout {
    \context {
      \Voice
      \remove "Stem_engraver"
      \remove "Phrasing_slur_engraver"
      \remove "Slur_engraver"
      \remove "Script_engraver"
      \remove "Beam_engraver"
      \remove "Auto_beam_engraver"
    }
    \context {
      \Staff
      \remove "Accidental_engraver"
      \remove "Key_engraver"
      \remove "Clef_engraver"
      \remove "Bar_engraver"
      \consists "Pitch_squash_engraver"
      \remove "Time_signature_engraver"
    }
  }
}
@end lilypond

@noindent
the @code{Clef_engraver} defines a reference point for the staff,

@lilypond[quote,ragged-right]
\include "engraver-example.ily"

\score {
  \topVoice
  \layout {
    \context {
      \Voice
      \remove "Stem_engraver"
      \remove "Phrasing_slur_engraver"
      \remove "Slur_engraver"
      \remove "Script_engraver"
      \remove "Beam_engraver"
      \remove "Auto_beam_engraver"
    }
    \context {
      \Staff
      \remove "Accidental_engraver"
      \remove "Key_engraver"
      \remove "Bar_engraver"
      \remove "Time_signature_engraver"
    }
  }
}
@end lilypond

@noindent
and the @code{Stem_engraver} adds stems.

@lilypond[quote,ragged-right]
\include "engraver-example.ily"

\score {
  \topVoice
  \layout {
    \context {
      \Voice
      \remove "Phrasing_slur_engraver"
      \remove "Slur_engraver"
      \remove "Script_engraver"
      \remove "Beam_engraver"
      \remove "Auto_beam_engraver"
    }
    \context {
      \Staff
      \remove "Accidental_engraver"
      \remove "Key_engraver"
      \remove "Bar_engraver"
      \remove "Time_signature_engraver"
    }
  }
}
@end lilypond

@noindent
The @code{Stem_engraver} is notified of any note head coming
along.  Every time one (or more, for a chord) note head is seen, a
stem object is created and connected to the note head.  By adding
engravers for beams, slurs, accents, accidentals, bar lines, time
signature, and key signature, we get a complete piece of notation.

@lilypond[quote,ragged-right]
\include "engraver-example.ily"
\score { \topVoice }
@end lilypond

@cindex polyphony
@cindex engraving multiple voices
@cindex contexts

This system works well for monophonic music, but what about
polyphony?  In polyphonic notation, many voices can share a staff.

@lilypond[quote,ragged-right]
\include "engraver-example.ily"
\new Staff << \topVoice \\ \botVoice >>
@end lilypond

In this situation, the accidentals and staff are shared, but the
stems, slurs, beams, etc., are private to each voice.  Hence,
engravers should be grouped.  The engravers for note heads, stems,
slurs, etc., go into a group called @q{Voice context}, while the
engravers for key, accidental, bar, etc., go into a group called
@q{Staff context}.  In the case of polyphony, a single Staff
context contains more than one Voice context.  Similarly, multiple
Staff contexts can be put into a single Score context.  The Score
context is the top level notation context.

@lilypond[quote,ragged-right]
\include "engraver-example.ily"
\score {
   <<
      \new Staff << \topVoice \\ \botVoice >>
      \new Staff << \pah \\ \hoom >>
   >>
}
@end lilypond

@seealso
Internals Reference: @rinternals{Contexts}.

@node Flexible architecture
@unnumberedsubsec Flexible architecture

When we started, we wrote the LilyPond program entirely in the C++
programming language; the program's functionality was set in stone
by the developers.  That proved to be unsatisfactory for a number
of reasons:

@itemize

@item When LilyPond makes mistakes, users need to override
formatting decisions.  Therefore, the user must have access to the
formatting engine.  Hence, rules and settings cannot be fixed by
us at compile-time but must be accessible for users at run-time.

@item Engraving is a matter of visual judgment, and therefore a
matter of taste.  As knowledgeable as we are, users can disagree
with our personal decisions.  Therefore, the definitions of
typographical style must also be accessible to the user.

@item Finally, we continually refine the formatting algorithms, so
we need a flexible approach to rules.  The C++ language forces a
certain method of grouping rules that cannot readily be applied to
formatting music notation.

@end itemize

@cindex Scheme programming language

These problems have been addressed by integrating an interpreter
for the Scheme programming language and rewriting parts of
LilyPond in Scheme.  The current formatting architecture is built
around the notion of graphical objects, described by Scheme
variables and functions.  This architecture encompasses formatting
rules, typographical style and individual formatting decisions.
The user has direct access to most of these controls.

Scheme variables control layout decisions.  For example, many
graphical objects have a direction variable that encodes the
choice between up and down (or left and right).  Here you see two
chords, with accents and arpeggios.  In the first chord, the
graphical objects have all directions down (or left).  The second
chord has all directions up (right).

@lilypond[quote,ragged-right]
\score {
  \relative c' {
    \stemDown <e g b>4_>-\arpeggio
    \override Arpeggio #'direction = #RIGHT
    \stemUp <e g b>4^>-\arpeggio
  }
  \layout {
    \context {
      \Score
      \override SpacingSpanner #'spacing-increment = #3
      \override TimeSignature #'transparent = ##t
    }
  }
}
@end lilypond

@cindex score formatting
@cindex formatting a score
@cindex formatting rules

@noindent
The process of formatting a score consists of reading and writing
the variables of graphical objects.  Some variables have a preset
value.  For example, the thickness of many lines -- a
characteristic of typographical style -- is a variable with a
preset value.  You are free to alter this value, giving your score
a different typographical impression.

@lilypond[quote,ragged-right]
fragment = {
   \clef bass f8 as8
   c'4-~ c'16 as g f e16 g bes c' des'4
}
<<
   \new Staff \fragment
   \new Staff \with {
      \override Beam #'beam-thickness = #0.3
      \override Stem #'thickness = #0.5
      \override Bar #'thickness = #3.6
      \override Tie #'thickness = #2.2
      \override StaffSymbol #'thickness = #3.0
      \override Tie #'extra-offset = #'(0 .  0.3)
      }
      \fragment
>>
@end lilypond

Formatting rules are also preset variables: each object has
variables containing procedures.  These procedures perform the
actual formatting, and by substituting different ones, we can
change the appearance of objects.  In the following example, the
rule governing which note head objects are used to produce the
note head symbol is changed during the music fragment.

@lilypond[quote,ragged-right]
#(set-global-staff-size 30)

#(define (mc-squared grob orig current)
  (let* ((interfaces (ly:grob-interfaces grob))
         (pos (ly:grob-property grob 'staff-position)))
    (if (memq 'note-head-interface interfaces)
        (begin
          (ly:grob-set-property! grob 'stencil
            (grob-interpret-markup grob
              (make-lower-markup 0.5
                (case pos
                  ((-5) "m")
                  ((-3) "c ")
                  ((-2) (make-smaller-markup (make-bold-markup "2")))
                  (else "bla")))))))))

\new Voice \relative c' {
  \stemUp
  \set autoBeaming = ##f
  \time 2/4
  <d f g>4
  \once \override NoteHead #'stencil = #note-head::brew-ez-stencil
  \once \override NoteHead #'font-size = #-7
  \once \override NoteHead #'font-family = #'sans
  \once \override NoteHead #'font-series = #'bold
  <d f g>4
  \once \override NoteHead #'style = #'cross
  <d f g>4
  \applyOutput #'Voice #mc-squared
  <d f g>4
  <<
    { d8[ es-( fis^^ g] fis2-) }
    \repeat unfold 5 { \applyOutput #'Voice #mc-squared s8 }
  >>
}
@end lilypond



@node Putting LilyPond to work
@section Putting LilyPond to work

@cindex simple examples
@cindex examples, simple

We have written LilyPond as an experiment of how to condense the
art of music engraving into a computer program.  Thanks to all
that hard work, the program can now be used to perform useful
tasks.  The simplest application is printing notes.

@lilypond[quote,relative=1]
{
  \time 2/4
  c4 c g'4 g a4 a g2
}
@end lilypond

@noindent
By adding chord names and lyrics we obtain a lead sheet.

@lilypond[quote,ragged-right]
<<
   \chords { c2 c f2 c }
   \new Staff
   \relative c' {
     \time 2/4
     c4 c g' g a a g2
   }
   \addlyrics { twin -- kle twin -- kle lit -- tle star }
>>
@end lilypond

Polyphonic notation and piano music can also be printed.  The
following example combines some more exotic constructs.

@lilypond[quote,line-width=15.9\cm]
\header {
  title = "Screech and boink"
  subtitle = "Random complex notation"
  composer = "Han-Wen Nienhuys"
}

\score {
  \context PianoStaff <<
    \new Staff = "up" {
      \time 4/8
      \key c \minor
      << {
        \revert Stem #'direction
        \change Staff = down
        \set subdivideBeams = ##t
        g16.[
          \change Staff = up
          c'''32
          \change Staff = down
          g32
          \change Staff = up
          c'''32
          \change Staff = down
          g16]
        \change Staff = up
        \stemUp
        \set followVoice = ##t
        c'''32([ b''16 a''16 gis''16 g''32)]
      } \\ {
        s4 \times 2/3 { d'16[ f' g'] } as'32[ b''32 e'' d'']
      } \\ {
        s4 \autoBeamOff d''8.. f''32
      } \\ {
        s4 es''4
      } >>
    }

    \new Staff = "down" {
      \clef bass
      \key c \minor
      \set subdivideBeams = ##f
      \override Stem  #'french-beaming = ##t
      \override Beam  #'beam-thickness = #0.3
      \override Stem  #'thickness = #4.0
      g'16[ b16 fis16 g16]
      << \makeClusters {
        as16 <as b>
        <g b>
        <g cis>
      } \\ {
        \override Staff.Arpeggio  #'arpeggio-direction =#down
        <cis, e, gis, b, cis>4\arpeggio
      }
    >> }
  >>
  \midi {
    \tempo 8 = 60
  }
  \layout {
    \context {
      \Staff
      \consists Horizontal_bracket_engraver
    }
  }
}
@end lilypond

The fragments shown above have all been written by hand, but that
is not a requirement.  Since the formatting engine is mostly
automatic, it can serve as an output means for other programs that
manipulate music.  For example, it can also be used to convert
databases of musical fragments to images for use on websites and
multimedia presentations.

This manual also shows an application: the input format is text, and can
therefore be easily embedded in other text-based formats such as
@LaTeX{}, HTML, or in the case of this manual, Texinfo.  Using the
@command{lilypond-book} program, included with LilyPond, the input
fragments can be replaced by music images in the resulting PDF or HTML
output files.  Another example is the third-party OOoLilyPond extension
for OpenOffice.org, which makes it extremely easy to embed musical
examples in documents.

For more examples of LilyPond in action, full documentation, and the
software itself, see our main website: www.lilypond.org.

@page
@node Engraved examples (BWV 861)
@section Engraved examples (BWV 861)

This section contains four reference engravings and two
software-engraved versions of Bach's Fugue in G minor from the
Well-Tempered Clavier, Book I, BWV 861 (the last seven measures).

@noindent
Bärenreiter BA5070 (Neue Ausgabe Sämtlicher Werke, Serie V, Band 6.1,
1989):

@iftex
@sourceimage{bwv861-baer,16cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861-baer-small,,,png}
@end ifnottex

@noindent
Bärenreiter BA5070 (Neue Ausgabe Sämtlicher Werke, Serie V, Band 6.1,
1989), an alternate musical source.  Aside from the textual differences,
this demonstrates slight variations in the engraving decisions, even
from the same publisher and edition:

@iftex
@sourceimage{bwv861-baer-alt,16cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861-baer-alt-small,,,png}
@end ifnottex

@noindent
Breitkopf & Härtel, edited by Ferruccio Busoni (Wiesbaden, 1894), also
available from the Petrucci Music Library (IMSLP #22081).  The editorial
markings (fingerings, articulations, etc.) have been removed for clearer
comparison with the other editions here:

@iftex
@sourceimage{bwv861-breitkopf,16cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861-breitkopf-small,,,png}
@end ifnottex

@noindent
Bach-Gesellschaft edition (Leipzig, 1866), available from the Petrucci
Music Library (IMSPL #02221):

@iftex
@sourceimage{bwv861-gessellschaft,16cm,,}
@end iftex
@ifnottex
@sourceimage{bwv861-gessellschaft-small,,,png}
@end ifnottex

@noindent
Finale 2008:

@iftex
@sourceimage{pdf/bwv861-finale2008a,,,}
@end iftex
@ifnottex
@sourceimage{bwv861-finale2008a,,,png}
@end ifnottex

@sp 4
@noindent
LilyPond, version @version{}:

@lilypond[staffsize=14.3,line-width=15.9\cm]
global = {\key g \minor}

partI = \relative c' {
  \voiceOne
  fis8 d' ees g, fis4 g
  r8 a16 bes c8 bes16 a d8 r r4
  r2 r8 d16 ees f8 ees16 d
  ees4 ~ ees16 d c bes a4 r8 ees'16 d
  c8 d16 ees d8 e16 fis g8 fis16 g a4 ~
  a8 d, g f ees d c bes
  a2 g\fermata \bar "|."
}

partII = \relative c' {
  \voiceTwo
  d4 r4 r8 d'16 c bes8 c16 d
  ees8 d c ees a, r r4
  r8 fis16 g a8 g16 fis g2 ~
  g2 r8 d' ees g,
  fis4 g r8 a16 bes c8 bes16 a
  bes4. <g b>8 <a c> r <d, g> r
  <ees g>4 <d fis> d2
}
partIII = \relative c' {
  \voiceOne
  r2 r8 d ees g, fis4 g r8 a16 bes c8 bes16 a
  bes2 ~ bes8 b16 a g8 a16 b
  c4 r r2
  R1
  r8 d ees g, fis4 g
  r8 a16 bes c8 bes16 a b2
}
partIV = \relative c {
  \voiceTwo
  d4 r r2
  r8 d ees g, fis4 a
  d,8 d'16 c bes8 c16 d ees2 ~
  ees8 ees16 d c8 d16 ees fis,8 a16 g fis8 g16 a
  d,8 d'16 c bes8 c16 d ees8 c a fis'
  g f ees d c bes a g
  c a d d, g2\fermata
}

\score {
  <<
    % \set Score.barNumberVisibility = #all-bar-numbers-visible
    % required in 2.13
    \set Score.currentBarNumber = #28
    \bar ""
    \new PianoStaff <<
      \new Staff = "RH" <<
        \global
        \new Voice = "voiceI" { \partI }
        \new Voice = "voiceII" { \partII }
      >>

      \new Staff = "LH" <<
        \clef "bass"
        \global
        \new Voice = "voiceIII" { \partIII }
        \new Voice = "voiceIV" { \partIV }
      >>
    >>
  >>
  \layout {
    \context {
      \Staff
      \remove "Time_signature_engraver"
    }
    \context {
      \PianoStaff
      \override StaffGrouper #'staff-staff-spacing #'padding = #1
    }
  }
}
@end lilypond