Docs: CG 3.3.6 Tidy syntax survey; add @uref

[lilypond.git] / Documentation / devel / programming-work.itexi

@c -*- coding: us-ascii; mode: texinfo; -*-
@node Programming work
@chapter Programming work

@menu
* Overview of LilyPond architecture::
* LilyPond programming languages::
* Programming without compiling::  
* Finding functions::           
* Code style::                  
* Debugging LilyPond::          
@end menu

@node Overview of LilyPond architecture
@section Overview of LilyPond architecture

LilyPond processes the input file into graphical and musical output in a
number of stages.  This process, along with the types of routines that
accomplish the various stages of the process, is described in this section.  A
more complete description of the LilyPond architecture and internal program
execution is found in Erik Sandberg's
@uref{http://lilypond.org/web/images/thesis-erik-sandberg.pdf, master's
thesis}.


The first stage of LilyPond processing is @emph{parsing}.  In the parsing
process, music expressions in LilyPond input format are converted to music
expressions in Scheme format.  In Scheme format, a music expression is a list
in tree form, with nodes that indicate the relationships between various music
events.  The LilyPond parser is written in Bison.

The second stage of LilyPond processing is @emph{iterating}.  Iterating
assigns each music event to a context, which is the environment in which the
music will be finally engraved.  The context is responsible for all further
processing of the music.  It is during the iteration stage that contexts are
created as necessary to ensure that every note has a Voice type context (e.g.
Voice, TabVoice, DrumVoice, CueVoice, MensuralVoice, VaticanaVoice,
GregorianTranscriptionVoice), that the Voice type contexts exist in
appropriate Staff type contexts, and that parallel Staff type contexts exist
in StaffGroup type contexts.  In addition, during the iteration stage each
music event is assigned a moment, or a time in the music when the event
begins.

Each type of music event has an associated iterator.  Iterators are defined in
*-iterator.cc. During iteration, an
event's iterator is called to deliver that music event to the appropriate
context(s).

The final stage of LilyPond processing is @emph{translation}.  During
translation, music events are prepared for graphical or midi output.  The
translation step is accomplished by translators or engravers (the distinction
is unclear).

Translators are defined in C++ files named *-engraver.cc.  In *-engraver.cc, a
C++ class of Engraver type is created.  The Engraver is also declared as a
translator.  Much of the work of translating is handled by Scheme functions,
which is one of the keys to LilyPond's exceptional flexibility.


@node LilyPond programming languages
@section LilyPond programming languages

Programming in LilyPond is done in a variety of programming languages.  Each
language is used for a specific purpose or purposes.  This section describes
the languages used and provides links to reference manuals and tutorials for
the relevant language.

@subsection C++

The core functionality of LilyPond is implemented in C++.

C++ is so ubiquitous that it is difficult to identify either a reference
manual or a tutorial.  Programmers unfamiliar with C++ will need to spend some
time to learn the language before attempting to modify the C++ code.

The C++ code calls Scheme/GUILE through the GUILE interface, which is
documented in the
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html, GUILE
  Reference Manual}.

@subsection GNU Bison

The LilyPond parser is implemented in Bison, a GNU parser generator.  The
Bison homepage is found at @uref{http://www.gnu.org/software/bison/,
gnu.org}.  The manual (which includes both a reference and tutorial) is
@uref{http://www.gnu.org/software/bison/manual/index.html, available} in a
variety of formats.

@subsection GNU Make

GNU Make is used to control the compiling process and to build the
documentation and the website.  GNU Make documentation is available at
@uref{http://www.gnu.org/software/make/manual/, the GNU website}.

@subsection GUILE or Scheme

GUILE is the dialect of Scheme that is used as LilyPond's extension language.  Many extensions to LilyPond are written entirely in GUILE.  The
@uref{http://www.gnu.org/software/guile/manual/html_node/index.html,
GUILE Reference Manual} is available online.

@uref{http://mitpress.mit.edu/sicp/full-text/book/book.html, Structure and
Interpretation of Computer Programs}, a popular textbook used to teach
programming in Scheme is available in its entirety online.

@subsection MetaFont

MetaFont is used to create the music fonts used by LilyPond.  A MetaFont
tutorial is available at @uref{http://metafont.tutorial.free.fr/, the
METAFONT tutorial page}.

@subsection PostScript

PostScript is used to generate graphical output.  A brief PostScript tutorial
is @uref{http://local.wasp.uwa.edu.au/~pbourke/dataformats/postscript/,
available online}.  The
@uref{http://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf, PostScript Lanugage
Reference} is available online in PDF format.

@subsection Python

Python is used for XML2ly and is used for buillding the documentation and the
website.

Python documentation is available at @uref{http://www.python.org/doc/,
python.org}.

@node Programming without compiling
@section Programming without compiling

Much of the development work in LilyPond takes place by changing *.ly or
*.scm files.  These changes can be made without compiling LilyPond.  Such
changes are described in this section.


@subsection Modifying distribution files

Much of LilyPond is written in Scheme or LilyPond input files.  These
files are interpreted when the program is run, rather than being compiled
when the program is built, and are present in all LilyPond distributions.
You will find .ly files in the ly/ directory and the Scheme files in the
scm/ directory.  Both Scheme files and .ly files can be modified and
saved with any text editor.  It's probably wise to make a backup copy of
your files before you modify them, although you can reinstall if the
files become corrupted.

Once you've modified the files, you can test the changes just by running
LilyPond on some input file.  It's a good idea to create a file that
demonstrates the feature you're trying to add.  This file will eventually
become a regression test and will be part of the LilyPond distribution.

@subsection Desired file formatting

Files that are part of the LilyPond distribution have Unix-style line
endings (LF), rather than DOS (CR+LF) or MacOS 9 and earlier (CR).  Make
sure you use the necessary tools to ensure that Unix-style line endings are
preserved in the patches you create.

Tab characters should not be included in files for distribution.  All
indentation should be done with spaces.  Most editors have settings to
allow the setting of tab stops and ensuring that no tab characters are
included in the file.

Scheme files and LilyPond files should be written according to standard
style guidelines.  Scheme file guidelines can be found at
@uref{http://community.schemewiki.org/?scheme-style}.  Following these
guidelines will make your code easier to read.  Both you and others that
work on your code will be glad you followed these guidelines.

For LilyPond files, you should follow the guidelines for LilyPond snippets
in the documentation.  You can find these guidelines at
@ref{Texinfo introduction and usage policy}.

@node Finding functions
@section Finding functions

When making changes or fixing bugs in LilyPond, one of the initial
challenges is finding out where in the code tree the functions to
be modified live.  With nearly 3000 files in the source tree,
trial-and-error searching is generally ineffective. This section
describes a process for finding interesting code.

@subsection Using the ROADMAP

The file ROADMAP is located in the main directory of the lilypond source.
ROADMAP lists all of the directories in the LilPond source tree, along
with a brief description of the kind of files found in each directory.
This can be a very helpful tool for deciding which directories to search
when looking for a function.


@subsection Using grep to search

Having identified a likely subdirectory to search, the grep utility can
be used to search for a function name.  The format of the grep command is

@example
grep -i functionName subdirectory/*
@end example

This command will search all the contents of the directory subdirectory/
and display every line in any of the files that contains
functionName.  The @code{-i} option makes @command{grep} ignore
case -- this can be very useful if you are not yet familiar with
our capitalization conventions.

The most likely directories to grep for function names are scm/ for
scheme files, ly/ for lilypond input (*.ly) files, and lily/ for C++
files.


@subsection Using git grep to search

If you have used git to obtain the source, you have access to a
powerful tool to search for functions.  The command:

@example
git grep functionName
@end example

will search through all of the files that are present in the git
repository looking for functionName.  It also presents the results
of the search using @code{less}, so the results are displayed one page
at a time.

@subsection Searching on the git repository at Savannah

You can also use the equivalent of git grep on the Savannah server.

@itemize

@item
Go to http://git.sv.gnu.org/gitweb/?p=lilypond.git

@item
In the pulldown box that says commit, select grep.

@item
Type functionName in the search box, and hit enter/return

@end itemize

This will initiate a search of the remote git repository.


@node Code style
@section Code style
@c email to wl@gnu.org when I get here.

@menu
@end menu

@subsection Handling errors

As a general rule, you should always try to continue computations,
even if there is some kind of error.  When the program stops, it
is often very hard for a user to pinpoint what part of the input
causes an error.  Finding the culprit is much easier if there is
some viewable output.

So functions and methods do not return errorcodes, they never
crash, but report a programming_error and try to carry on.

@subsection Languages

C++ and Python are preferred.  Python code should use PEP 8.

@subsection Filenames

Definitions of classes that are only accessed via pointers (*) or
references (&) shall not be included as include files.

@verbatim
   filenames

        ".hh"   Include files
             ".cc"      Implementation files
             ".icc"     Inline definition files
             ".tcc"     non inline Template defs

   in emacs:

             (setq auto-mode-alist
                   (append '(("\\.make$" . makefile-mode)
                        ("\\.cc$" . c++-mode)
                        ("\\.icc$" . c++-mode)
                        ("\\.tcc$" . c++-mode)
                        ("\\.hh$" . c++-mode)
                        ("\\.pod$" . text-mode)
                        )
                      auto-mode-alist))
@end verbatim

The class Class_name is coded in @q{class-name.*}

@subsection Indentation

Standard GNU coding style is used. In emacs:

@verbatim
             (add-hook 'c++-mode-hook
                  '(lambda() (c-set-style "gnu")
                     ))
@end verbatim

If you like using font-lock, you can also add this to your
@q{.emacs}:

@verbatim
             (setq font-lock-maximum-decoration t)
             (setq c++-font-lock-keywords-3
                   (append
                    c++-font-lock-keywords-3
                    '(("\\b\\(a-zA-Z_?+_\\)\\b" 1 font-lock-variable-name-face) ("\\b\\(A-Z?+a-z_?+\\)\\b" 1 font-lock-type-face))
                    ))
@end verbatim


@subsection Classes and Types

@verbatim
This_is_a_class
@end verbatim


@subsection Members

Member variable names end with an underscore:

@verbatim
Type Class::member_
@end verbatim


@subsection Macros

Macro names should be written in uppercase completely.


@subsection Broken code

Do not write broken code.  This includes hardwired dependencies,
hardwired constants, slow algorithms and obvious limitations.  If
you can not avoid it, mark the place clearly, and add a comment
explaining shortcomings of the code.

We reject broken-in-advance on principle.

@subsection Naming


@subsection Messages

Messages need to follow Localization.


@subsection Localization

This document provides some guidelines for programmers write user
messages.  To help translations, user messages must follow
uniform conventions.  Follow these rules when coding for LilyPond.
Hopefully, this can be replaced by general GNU guidelines in the
future.  Even better would be to have an English (en_BR, en_AM)
guide helping programmers writing consistent messages for all GNU
programs.

Non-preferred messages are marked with `+'. By convention,
ungrammatical examples are marked with `*'.  However, such ungrammatical
examples may still be preferred.

@itemize

@item
Every message to the user should be localized (and thus be marked
for localization). This includes warning and error messages.

@item
Don't localize/gettextify:

@itemize
@item
`programming_error ()'s

@item
`programming_warning ()'s

@item
debug strings

@item
output strings (PostScript, TeX, etc.)

@end itemize

@item
Messages to be localised must be encapsulated in `_ (STRING)' or
`_f (FORMAT, ...)'. E.g.:

@example
warning (_ ("need music in a score"));
error (_f ("cannot open file: `%s'", file_name));
@end example

In some rare cases you may need to call `gettext ()' by hand. This
happens when you pre-define (a list of) string constants for later
use. In that case, you'll probably also need to mark these string
constants for translation, using `_i (STRING)'. The `_i' macro is
a no-op, it only serves as a marker for `xgettext'.

@example
char const* messages[] = @{
  _i ("enable debugging output"),
  _i ("ignore lilypond version"),
  0
@};

void
foo (int i)
@{
  puts (gettext (messages i));
@}
@end example

See also `flower/getopt-long.cc' and `lily/main.cc'.

@item
Do not use leading or trailing whitespace in messages. If you need
whitespace to be printed, prepend or append it to the translated
message

@example
message ("Calculating line breaks..." + " ");
@end example

@item
Error or warning messages displayed with a file name and line
number never start with a capital, eg,

@example
foo.ly: 12: not a duration: 3
@end example

Messages containing a final verb, or a gerund (`-ing'-form) always
start with a capital. Other (simpler) messages start with a
lowercase letter

@example
Processing foo.ly...
`foo': not declared.
Not declaring: `foo'.
@end example

@item
Avoid abbreviations or short forms, use `cannot' and `do not'
rather than `can't' or `don't'
To avoid having a number of different messages for the same
situation, well will use quoting like this `"message: `%s'"' for all
strings. Numbers are not quoted:

@example
_f ("cannot open file: `%s'", name_str)
_f ("cannot find character number: %d", i)
@end example

@item
Think about translation issues. In a lot of cases, it is better to
translate a whole message. The english grammar must not be imposed
on the translator. So, instead of

@example
stem at  + moment.str () +  does not fit in beam
@end example

have

@example
_f ("stem at %s does not fit in beam", moment.str ())
@end example

@item
Split up multi-sentence messages, whenever possible. Instead of

@example
warning (_f ("out of tune!  Can't find: `%s'", "Key_engraver"));
warning (_f ("cannot find font `%s', loading default", font_name));
@end example

rather say:

@example
warning (_ ("out of tune:"));
warning (_f ("cannot find: `%s', "Key_engraver"));
warning (_f ("cannot find font: `%s', font_name));
warning (_f ("Loading default font"));
@end example

@item
If you must have multiple-sentence messages, use full punctuation.
Use two spaces after end of sentence punctuation. No punctuation
(esp. period) is used at the end of simple messages.

@example
_f ("Non-matching braces in text `%s', adding braces", text)
_ ("Debug output disabled.  Compiled with NPRINT.")
_f ("Huh?  Not a Request: `%s'.  Ignoring.", request)
@end example

@item
Do not modularise too much; words frequently cannot be translated
without context. It is probably safe to treat most occurences of
words like stem, beam, crescendo as separately translatable words.

@item
When translating, it is preferable to put interesting information
at the end of the message, rather than embedded in the middle.
This especially applies to frequently used messages, even if this
would mean sacrificing a bit of eloquency. This holds for original
messages too, of course.

@example
en: cannot open: `foo.ly'
+   nl: kan `foo.ly' niet openen (1)
kan niet openen: `foo.ly'*   (2)
niet te openen: `foo.ly'*    (3)
@end example


The first nl message, although grammatically and stylistically
correct, is not friendly for parsing by humans (even if they speak
dutch). I guess we would prefer something like (2) or (3).

@item
Do not run make po/po-update with GNU gettext < 0.10.35

@end itemize



@node Debugging LilyPond
@section Debugging LilyPond

The most commonly used tool for debugging LilyPond is the GNU debugger
gdb.  Use of gdb is described in this section.

@subsection Debugging overview

Using a debugger simplifies troubleshooting in at least two ways.

First, breakpoints can be set to pause execution at any desired point.
Then, when execution has paused, debugger commands can be issued to
explore the values of various variables or to execute functions.

Second, the debugger allows the display of a stack trace, which shows
the sequence in which functions are called and the arguments to the
various function calls.


@subsection Compiling with debugging information

In order to use a debugger with LilyPond, it is necessary to compile
LilyPond with debugging information.  This is accomplished by ...

TODO -- get good description here, or perhaps add debugging compile
to AU1.1 as it comes to CG and just use a reference here.

TODO -- Test the following to make sure it is true.

If you want to be able to set breakpoints in Scheme functions, it is
necessary to compile guile with debugging information.  This is done
by ...

TODO -- get compiling description for guile here.

@subsection Typical gdb usage

@subsection Typical .gdbinit files

The behavior of gdb can be readily customized through the use of
@var{.gdbinit} files.  A @var{.gdbinit} file is a file named
@var{.gdbinit} (notice the @qq{.} at the beginning of the file name)
  that is placed in a user's home directory.

The @var{.gdbinit} file below is from Han-Wen.  It sets breakpoints
for all errors and defines functions for displaying scheme objects
(ps), grobs (pgrob), and parsed music expressions (pmusic).

@example
file lily/out/lilypond
b scm_error
b programming_error
b Grob::programming_error

define ps
   print ly_display_scm($arg0)
  end
  define pgrob
     print ly_display_scm($arg0->self_scm_)
     print ly_display_scm($arg0->mutable_property_alist_)
     print ly_display_scm($arg0->immutable_property_alist_)
     print ly_display_scm($arg0->object_alist_)
  end
  define pmusic
     print ly_display_scm($arg0->self_scm_)
     print ly_display_scm($arg0->mutable_property_alist_)
     print ly_display_scm($arg0->immutable_property_alist_)
  end
@end example

@subsection Using Guile interactively with LilyPond

In order to experiment with Scheme programming in the LilyPond
environment, it is convenient to have a Guile interpreter that
has all the LilyPond modules loaded.  This requires the following
steps.

First, define a Scheme symbol for the active module
in the .ly file:

@example
#(module-define! (resolve-module '(guile-user))
    'lilypond-module (current-module))
@end example

Second, place a Scheme function in the .ly file that gives an interactive Guile
prompt:

@example
#(top-repl)
@end example

When the .ly file is compiled, this causes the compilation to be interrupted
and an interactive guile prompt to appear.  When the guile prompt appears,
the LilyPond active module must be set as the current guile module:

@example
guile> (set-current-module lilypond-module)
@end example

Proper operation of these commands can be demonstrated by typing the name
of a LilyPond public scheme function to see if it's properly defined:

@example
guile> fret-diagram-verbose-markup
#<procedure fret-diagram-verbose-markup (layout props marking-list)>
@end example

If the LilyPond module has not been correctly loaded, an error
message will be generated:

@example
guile> fret-diagram-verbose-markup
ERROR: Unbound variable: fret-diagram-verbose-markup
ABORT: (unbound-variable)
@end example

Once the module is properly loaded, any valid LilyPond Scheme expression
can be entered at the interactive prompt.

After the investigation is complete, the interactive guile interpreter
can be exited:

@example
guile> (quit)
@end example

The compilation of the .ly file will then continue.