convert-ly the user manual

[lilypond.git] / Documentation / user / invoking.itely

@c -*- coding: utf-8; mode: texinfo; -*-
@c This file is part of lilypond.tely

@node Running LilyPond
@chapter Running LilyPond

This chapter details the technicalities of running LilyPond.

Some of these commands are run from the command-line.  By
``command-line'', we mean the command
line in the operating system.  Windows users
might be more familiar with the terms ``DOS shell'' or
``command shell''; OSX users might be more familiar with the
terms ``terminal'' or ``console''.  OSX users should also
consult @ref{Notes for the MacOS X app}.

Describing how to use
this part of an operating system is outside the scope of this
manual; please consult other documentation on this topic if
you are unfamiliar with the command-line.

@menu
* Invoking lilypond::           
* Notes for the MacOS X app::   
* Updating files with convert-ly::  
* Reporting bugs::              
* Error messages::              
* Editor support::              
* Point and click::             
@end menu

@node Invoking lilypond
@section Invoking lilypond
@cindex Invoking LilyPond
@cindex command line options
@cindex options, command line
@cindex switches


The @code{lilypond} executable may be called as follows from the command line.

@example
lilypond [@var{option}]@dots{} @var{file}@dots{}
@end example


When invoked with a filename that has no extension, the @file{.ly}
extension is tried first.  To read input from stdin, use a
dash (@code{-}) for @var{file}.

When @file{filename.ly} is processed it will produce
@file{filename.tex} as output (or @file{filename.ps} for PostScript
output).  If @file{filename.ly} contains more than one @code{\score}
block, then the rest of the scores will be output in numbered files,
starting with @file{filename-1.tex}.  Several files can be specified;
they will each be processed independently.  @footnote{The status of
GUILE is not reset after processing a @code{.ly} file, so be careful
not to change any system defaults from within Scheme.}


@subsection Command line options

The following options are supported:

@table @code

@item -e,--evaluate=@var{expr}
Evaluate the Scheme @var{expr} before parsing any @file{.ly} files.
Multiple @code{-e} options may be given, they will be evaluated
sequentially.

The expression will be evaluated in the @code{guile-user} module, so
if you want to use definitions in @var{expr}, use

@example
lilypond -e '(define-public a 42)'
@end example

@noindent
on the command-line, and include

@example
#(use-modules (guile-user))
@end example

@noindent
at the top of the @code{.ly} file.

@item -f,--format=@var{format}
which formats should be written.  Choices are @code{svg}, @code{ps},
@code{pdf}, @code{png}, @code{tex}, @code{dvi}.

@item -b,--backend=@var{format}
the output format to use for the back-end.  Choices are
@table @code
@item tex
for @TeX{} output, to be processed with La@TeX{}.  If present, the file
@file{file.textmetrics} is read to determine text extents.
@item texstr
dump text strings to @file{.texstr} file, which can be run through
(La)@TeX{}, resulting in a @code{.textmetrics} file, which contains the
extents of strings of text.
@item ps
 for PostScript.
@cindex PostScript output

  Postscript files include TTF, Type1 and OTF fonts.  No subsetting of
  these fonts is done.  When using oriental character sets, this can
  lead to huge files.

@item eps
 for encapsulated PostScript.  This dumps every page (system) as a separate
@file{EPS} file, without fonts, and as one collated @file{EPS} file with
all pages (systems) including fonts.

This mode is used by default by lilypond-book.

@item svg
 for SVG (Scalable Vector Graphics).  This dumps every page as a separate
@file{SVG} file, with embedded fonts.
@cindex SVG (Scalable Vector Graphics)
  You need a SVG viewer which supports embedded fonts, or a SVG
  viewer which is able to replace the embedded fonts with OTF fonts.
  Under Unix, you may use @uref{http://www.inkscape.org,Inkscape}
  (version 0.42 or later), after copying the OTF fonts in directory
  @file{PATH/TO/share/lilypond/VERSION/fonts/otf/} to @file{~/.fonts/}.
@item scm
 for a dump of the raw, internal Scheme-based drawing commands.
@cindex Scheme dump
@end table

@cindex output format, setting

@item -d,--define-default=@var{var}=@var{val}
This sets the internal program option @var{var} to the Scheme value
@var{val}. If @var{val} is not supplied, then @var{#t} is used. To
switch off an option, @code{no-} may be prefixed to @var{var}, e.g.

@cindex point and click, command line

@example
-dno-point-and-click
@end example

@noindent
is the same as
@example
-dpoint-and-click='#f'
@end example

Another notable option is

@example
-dpaper-size=\"letter\"
@end example

@noindent
Note that the string must be enclosed in escaped quotes ( @code{\"} ).

Setting the @code{-dhelp} option will print a summary of the options
available, and exit.

@item -h,--help
Show a summary of usage.

@item -H,--header=FIELD
Dump a header field to file BASENAME.FIELD

@item --include, -I=@var{directory}
Add @var{directory} to the search path for input files.
@cindex file searching
@cindex search path

@item -i,--init=@var{file}
Set init file to @var{file} (default: @file{init.ly}).

@item -o,--output=@var{FILE}
Set the default output file to @var{FILE}.  The appropriate
suffix will be added (ie @code{.pdf} for pdf, @code{.tex}
for tex, etc).

@item --ps
Generate PostScript.

@item --dvi
Generate DVI files.  In this case, the @TeX{} backend should be
specified, i.e., @code{-b tex}.

@item --png
Generate pictures of each page, in PNG format.  This implies
@code{--ps}.  The resolution in DPI of the image may be set with
@example
-dresolution=110
@end example

@item --pdf
Generate PDF.  This implies @code{--ps}.

@item --preview
Generate an output file containing the titles and the first system

@item --no-pages
Do not generate the full pages.  Useful in combination with
@code{--preview}.

@item -s,--safe
Do not trust the @code{.ly} input.

When LilyPond formatting is available through a web server, either the
@code{--safe} or the @code{--jail} option @b{MUST} be passed.  The
@code{--safe} option will prevent inline Scheme code from wreaking
havoc, for example

@quotation
@verbatim
#(system "rm -rf /")
{
  c4^#(ly:export (ly:gulp-file "/etc/passwd"))
}
@end verbatim
@end quotation

The @code{--safe} option works by evaluating in-line Scheme
expressions in a special safe module.  This safe module is derived from
GUILE @file{safe-r5rs} module, but adds a number of functions of the
LilyPond API.  These functions are listed in @file{scm/@/safe@/-lily@/.scm}.

In addition, @code{--safe} disallows @code{\include} directives and
disables the use of backslashes in @TeX{} strings.

In @code{--safe} mode, it is not possible to import LilyPond variables
into Scheme.

@code{--safe} does @emph{not} detect resource overuse.  It is still
possible to make the program hang indefinitely, for example by feeding
cyclic data structures into the backend.  Therefore, if using LilyPond
on a publicly accessible webserver, the process should be limited in
both CPU and memory usage.

Note that @code{--safe} will prevent many useful LilyPond snippets from
being compiled.  For a softer but secure alternative you can use the
@code{--jail} option.


@item -j,--jail=@var{user},@var{group},@var{jail},@var{dir}
Run LilyPond in a chroot jail.

The @code{--jail} option provides a more flexible alternative to
@code{--safe} when LilyPond formatting is available through a web
server or whenever LilyPond executes externally provided
sources.

The @code{--jail} option works by changing the root of LilyPond to
@var{jail} just before starting the actual compilation process.  The user
and group are then changed to match those provided, and the current
directory is changed to @var{dir}.  This setup guarantees that it is not
possible (at least in theory) to escape from the jail.  Note that for
@code{--jail} to work LilyPond must be run as root, which is usually
accomplished in a safe way using @command{sudo}.

Setting up a jail is a slightly delicate matter, as we must be sure that
LilyPond is able to find whatever it needs to compile the source
@emph{inside the jail}. A typical setup comprises the following items:

@table @asis
@item Setting up a separate filesystem
A separate filesystem should be created for LilyPond, so that it can be
mounted with safe options such as @code{noexec}, @code{nodev}, and
@code{nosuid}.  In this way, it is impossible to run executables or to
write directly to a device from LilyPond.  If you do not want to create a
separate partition, just create a file of reasonable size and use it to
mount a loop device.  A separate filesystem also guarantees that LilyPond
cannot write more space than it is allowed.

@item Setting up a separate user
A separate user and group (say, @samp{lily}/@samp{lily}) with low
privileges should be used to run LilyPond inside the jail.  There should
be a single directory writable by this user, which should be passed in
@var{dir}.

@item Preparing the jail
LilyPond needs to read a number of files while running.  All these files
are to be copied into the jail, under the same path they appear in the
real root filesystem.  The entire content of the LilyPond installation
(e.g., @file{/usr/share/lilypond})
should be copied.

If problems arise, the simplest way to trace them down is to run
LilyPond using @command{strace}, which will allow you to determine which
files are missing.

@item Running LilyPond
In a jail mounted with @code{noexec} it is impossible to execute any external
program.  Therefore LilyPond must be run with a backend that does not
require any such program.  As we already mentioned, it must be also run
with superuser privileges (which, of course, it will lose immediately),
possibly using @command{sudo}.  It is a good idea to limit the number of
seconds of CPU time LilyPond can use (e.g., using @command{ulimit
-t}), and, if your operating system supports it, the amount of memory
that can be allocated.
@end table


@item -v,--version
Show version information.

@item -V,--verbose
Be verbose: show full paths of all files read, and give timing
information.

@item -w,--warranty
Show the warranty with which GNU LilyPond comes. (It comes with
@strong{NO WARRANTY}!)
@end table


@subsection Environment variables


@cindex LANG
@cindex LILYPONDPREFIX

@code{Lilypond} recognizes the following environment variables:
@table @code
@item LILYPONDPREFIX
This specifies a directory where locale messages and
data files will be looked up by default.  The directory should contain
subdirectories called @file{ly/}, @file{ps/}, @file{tex/}, etc.

@item LANG
This selects the language for the warning messages.

@item LILYPOND_GC_YIELD
With this variable the memory footprint and performance can be
adjusted. It is a percentage tunes memory management behavior. With
higher values, the program uses more memory, with smaller values, it
uses more CPU time. The default value is @code{70}.

@end table


@node Notes for the MacOS X app
@section Notes for the MacOS X app

The scripts (such as lilypond-book, convert-ly, abc2ly, etc.) are also
included inside MacOS X .app. They can be run from the command line by
invoking them directly, e.g.

@example
@var{path/to}/LilyPond.app/Contents/Resources/bin/convert-ly
@end example

Alternatively, you may add this directory to your path.  Modify (or create)
a file called @code{.profile} in your home directory such that it contains

@example
export PATH=$PATH:@var{path/to}/LilyPond.app/Contents/Resources/bin
@end example

@noindent
This file should end with a blank line.

Note that @var{path/to} will generally be @code{/Applications/}.


@node Updating files with convert-ly
@section Updating with @command{convert-ly}

@cindex Updating a LilyPond file
@funindex convert-ly

The LilyPond input syntax is routinely changed to simplify it or improve
it in different ways.  As a side effect of this, the LilyPond interpreter
often is no longer compatible with older input files.  To remedy this,
the program @command{convert-ly} can be used to deal with most of the
syntax changes between LilyPond versions.

It uses @code{\version} statements in the input files to detect the
old version number.  In most cases, to upgrade your input file it is
sufficient to run@footnote{MacOS X users may execute this command
under the menu entry @samp{Compile > Update syntax}.}

@example
convert-ly -e myfile.ly
@end example

If there are no changes to myfile.ly and file called myfile.ly.NEW
is created, then myfile.ly is already updated.

@command{convert-ly} always converts up to the last syntax change handled by
it.  This means that the @code{\version} number left in the file is
usually lower than the version of @command{convert-ly} itself.

To upgrade LilyPond fragments in texinfo files, use

@example
convert-ly --from=... --to=... --no-version *.itely
@end example

To upgrade many files at once, combine @code{convert-ly} with
standard unix commands.  This example will upgrade all @code{.ly}
files in the current directory

@example
for f in *.ly; do convert-ly -e $f; done;
@end example

In general, the program is invoked as follows:

@example
convert-ly [@var{option}]@dots{} @var{file}@dots{}
@end example


The following options can be given:

@table @code
@item -e,--edit
Do an inline edit of the input file.  Overrides @code{--output}.

@item -f,--from=@var{from-patchlevel}
Set the version to convert from.  If this is not set, @command{convert-ly}
will guess this, on the basis of @code{\version} strings in the file.

@item -n,--no-version
Normally, @command{convert-ly} adds a @code{\version} indicator
to the output.  Specifying this option suppresses this.

@item -s, --show-rules
Show all known conversions and exit.

@item --to=@var{to-patchlevel}
Set the goal version of the conversion.  It defaults to the latest
available version.

@item -h, --help
Print usage help.
@end table


@refbugs

Not all language changes are handled.  Only one output option can be
specified.  Automatically updating scheme and lilypond scheme
interfaces is quite unlikely; be prepared to tweak scheme code
manually.


@c  We might want to make this a completely new section, along with more
@c  info about how to upgrade old input files.  -gp

@ignore
Copy and paste from CVS, last updated
Aug 18, 2005

http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/lilypond/lily-bugs/bugs/
convert-ly.txt?rev=HEAD&content-type=text/plain

NEW: not exactly copied; this list has been modified.  Since we're
changing the bug system, it doesn't make sense to copy from
the bug CVS any more.  I'll figure out something else.  -gp
@end ignore
@verbatim

There are a few things that the convert-ly cannot handle. Here's a list of
limitations
that the community has complained about.

This bug report structure has been chosen because convert-ly has a structure
that doesn't
allow to smoothly implement all needed changes. Thus this is just a wishlist,
placed
here for reference.

1.6->2.0:
 Doesn't always convert figured bass correctly, specifically things like {<
>}. Mats' comment on working around this:
   To be able to run convert-ly
   on it, I first replaced all occurencies of '{<' to some dummy like '{#'
   and similarly I replaced '>}' with '&}'. After the conversion, I could
   then change back from '{ #' to '{ <' and from '& }' to '> }'.
 Doesn't convert all text markup correctly. In the old markup syntax,
 it was possible to group a number of markup commands together within
parentheses, e.g.
   -#'((bold italic) "string")
   This will incorrectly be converted into
   -\markup{{\bold italic} "string"}
   instead of the correct
   -\markup{\bold \italic "string"}
2.0->2.2:
 Doesn't handle \partcombine
 Doesn't do \addlyrics => \lyricsto, this breaks some scores with multiple
stanzas.
2.0->2.4:
 \magnify isn't changed to \fontsize.
    - \magnify #m => \fontsize #f, where f = 6ln(m)/ln(2)
 remove-tag isn't changed.
    - \applyMusic #(remove-tag '. . .) => \keepWithTag #'. . .
 first-page-number isn't changed.
    - first-page-number no => printfirst-page-number = ##f
 Line breaks in header strings aren't converted.
    - \\\\  as line break in \header strings => \markup \center-align <
      "First Line" "Second Line" >
 Crescendo and decrescendo terminators aren't converted.
    - \rced => \!
    - \rc => \!
2.2->2.4:
 \turnOff (used in \set Staff.VoltaBracket = \turnOff) is not properly
converted.
2.4.2->2.5.9
 \markup{ \center-align <{ ... }> } should be converted to:
 \markup{ \center-align {\line { ... }} }
 but now, \line is missing.
2.4->2.6
 Special LaTeX characters such as $~$ in text are not converted to UTF8.
2.8
 \score{} must now begin with a music expression.  Anything else
 (particularly \header{}) must come after the music.
@end verbatim


@node Reporting bugs
@section Reporting bugs

@cindex bugs
@cindex reporting bugs

If you have input that results in a crash or an erroneous output, then
that is a bug.  We try to respond to bug-reports promptly, and fix them as
soon as possible.  Help us by sending a defective input file, so we can
reproduce the problem. Send the report via:

@example
@uref{http://post.gmane.org/post.php?group=gmane.comp.gnu.lilypond.bugs}
@end example

A few tips:
@itemize @bullet

@item Try to produce a very small input file which demonstrates the problem;
one or two bars is often sufficient to reproduce a bug. The smaller the
input file is, the easier it is for us to debug the problem.

@item Don't forget to tell which version of LilyPond you use!

@item If possible, use @code{ragged-right} in your example. This makes sure
that the bug can be reproduced in all paper sizes.
@end itemize

@ignore
@c the bug database is not up to date enough.

When you've found a bug, have a look at our
@uref{http://@/lilypond@/.org/@/bugs/@/v2.8/@/,bug database} to see if
it has already been reported.  You could also try to do a few searches
on the mailing list for the bug.  Sometimes the bug will have already
been reported and a fix or workaround is already known.
@end ignore

Here is an example of a good bug report:

@verbatim
It seems that placement of accidentals is broken.  In the
following example, the accidental touches the note head.

Using Mac OSX 10.3.7, lilypond 2.7.32

\version "2.9.13"
\layout { ragged-right = ##t }
\relative c'' {
   a4 b cis d
}
@end verbatim

@lilypond[quote]
\layout { ragged-right = ##t }
\relative c''{
  \override Accidental #'extra-offset = #'(1.0 . 0)
  a4 b cis d
}
@end lilypond

@node Error messages
@section Error messages

@cindex error messages
Different error messages can appear while compiling a file:

@table @emph
@cindex warning

@item Warning
Something looks suspect.  If you are requesting something out of the
ordinary then you will understand the message, and can ignore it.
However, warnings usually indicate that something is wrong with the
input file.

@item Error
Something is definitely wrong.  The current processing step (parsing,
interpreting, or formatting) will be finished, but the next step will
be skipped.

@cindex error
@cindex fatal error
@item Fatal error
Something is definitely wrong, and LilyPond cannot continue.  This
happens rarely.  The most usual cause is misinstalled fonts.

@cindex trace, Scheme
@cindex call trace
@cindex Scheme error
@item Scheme error
Errors that occur while executing Scheme code are caught by the Scheme
interpreter.  If running with the verbose option (@code{-V} or
@code{--verbose}) then a call trace of the offending
function call is printed.

@cindex Programming error
@item Programming error
There was some internal inconsistency.  These error messages are
intended to help the programmers and debuggers.  Usually, they can be
ignored.  Sometimes, they come in such big quantities that they obscure
other output.  In this case, file a bug-report.

@item Aborted (core dumped)
This signals a serious programming error that caused the program to
crash.  Such errors are considered critical.  If you stumble on one,
send a bug-report.


@end table

@cindex errors, message format
If warnings and errors can
be linked to some part of the input file, then error messages have the
following form

@example
@var{filename}:@var{lineno}:@var{columnno}: @var{message}
@var{offending input line}
@end example

A line-break is inserted in the offending line to indicate the column
where the error was found.  For example,

@example
test.ly:2:19: error: not a duration: 5:
  @{ c'4 e'5
             g' @}
@end example

These locations are LilyPond's best guess about where the warning or
error occurred, but (by their very nature) warnings and errors occur
when something unexpected happens.  If you can't see an error in the
indicated line of your input file, try checking one or two lines
above the indicated position.


@node Editor support
@section Editor support

@cindex editors
@cindex vim
@cindex emacs
@cindex modes, editor
@cindex syntax coloring
@cindex coloring, syntax

There is support from different editors for LilyPond.

@table @asis
@item Emacs
Emacs has a @file{lilypond-mode}, which provides keyword
autocompletion, indentation, LilyPond specific parenthesis matching
and syntax coloring, handy compile short-cuts and reading LilyPond
manuals using Info.  If @file{lilypond-mode} is not installed on your
platform, then read the
@ifhtml
@uref{source/Documentation/topdocs/INSTALL.html,installation instructions}.
@end ifhtml
@ifnothtml
installation instructions.
@end ifnothtml

@item VIM

For @uref{http://@/www@/.vim@/.org,VIM}, a @file{vimrc} is supplied, along
with syntax coloring tools.  For more information, refer to the
@ifhtml
@uref{source/Documentation/topdocs/INSTALL.html,installation instructions}.
@end ifhtml
@ifnothtml
installation instructions.
@end ifnothtml


@item JEdit

The @uref{http://@/www@/.jedit@/.org@/,jEdit} editor has a LilyPond plugin.
This plugin includes a DVI viewer, integrated help and viewing via
GhostScript.  It can be installed by doing @key{Plugins > Plugin
Manager}, and selecting @code{LilyTool} from the @key{Install} tab.

@end table

All these editors can be made to jump into the input file to the source
of a symbol in the graphical output.  See @ref{Point and click}.


@node Point and click
@section Point and click
@cindex point and click


Point and click lets you find notes in the input by clicking on them
in the PDF viewer.  This makes it easier to find input that causes
some error in the sheet music.

When this functionality is active, LilyPond adds hyperlinks to the PDF
file. These hyperlinks are sent to the web-browser, which opens a
text-editor with the cursor in the right place.

To make this chain work, you should configure your PDF viewer to
follow hyperlinks using the @file{lilypond-invoke-editor} script
supplied with LilyPond.

For Xpdf on Unix, the following should be present in
@file{xpdfrc}@footnote{On unix, this file is found either in
@file{/etc/xpdfrc} or as @file{.xpdfrc} in your home directory.}

@example
urlCommand     "lilypond-invoke-editor %s"
@end example

The program @file{lilypond-invoke-editor} is a small helper
program. It will invoke an editor for the special @code{textedit}
URIs, and run a web browser for others.  It tests the environment
variable @code{EDITOR} for the following patterns,

@table @code
@item emacs
  this will invoke
@example
emacsclient --no-wait +@var{line}:@var{column} @var{file}
@end example
@item vim
  this will invoke
@example
gvim --remote +:@var{line}:norm@var{char} @var{file}
@end example

@item nedit
this will invoke
@example
  nc -noask +@var{line} @var{file}'
@end example
@end table

The environment variable @code{LYEDITOR} is used to override this. It
contains the command line to start the editor, where @code{%(file)s},
@code{%(column)s}, @code{%(line)s} is replaced with the file, column
and line respectively. The  setting

@example
emacsclient --no-wait +%(line)s:%(column)s %(file)s
@end example

@noindent
for @code{LYEDITOR} is equivalent to the standard emacsclient
invocation.


@cindex file size, output

The point and click links enlarge the output files significantly. For
reducing the size of PDF and PS files, point and click may be switched
off by issuing

@example
#(ly:set-option 'point-and-click #f)
@end example

@noindent
in a @file{.ly} file.  Alternately, you may pass this as an command-line
option

@example
lilypond -dno-point-and-click file.ly
@end example