Doc: Usage - update Convert-ly chapters

[lilypond.git] / Documentation / usage / updating.itely

@c -*- coding: utf-8; mode: texinfo; -*-

@ignore
    Translation of GIT committish: FILL-IN-HEAD-COMMITTISH

    When revising a translation, copy the HEAD committish of the
    version that you are working on.  For details, see the Contributors'
    Guide, node Updating translation committishes..
@end ignore

@c \version "2.16.0"


@node Updating files with convert-ly
@chapter Updating files with @command{convert-ly}

@cindex Updating a LilyPond file
@cindex convert-ly

As LilyPond is improved, the syntax (input language) of some commands
and functions can change.  This can result in unexpected errors,
warnings or even output when input files, previously created with older
versions of LilyPond are then used with later versions.

To help with this the @command{convert-ly} command can be used to
upgrade these older input files to use the newer syntax.

@menu
* Why does the syntax change?::
* Invoking convert-ly::
* Command line options for convert-ly::
* Problems running convert-ly::
* Manual conversions::
@end menu


@node Why does the syntax change?
@section Why does the syntax change?

@cindex convert-ly
@cindex updating old input files

Often, syntax changes are made to make the input simpler to both read
and write, but occasionally the changes are made to accommodate new
features or enhancements to existing functions.

To illustrate this here is a real example:

All @code{\paper} and @code{\layout} property names were supposed to be
written in the form @code{first-second-third}.  However, in LilyPond
version 2.11.60, it was noticed that the @code{printallheaders} property
did not follow this convention.  Should this property be left alone
(confusing new users with an inconsistent format)? Or should it be
changed (annoying old users with existing LilyPond input files)?

The decision was made to change the name of the property to
@code{print-all-headers}, and by using the @command{convert-ly} command
the old users had a way to automatically update their existing input
files.

However, the @command{convert-ly} command cannot always be used to
manage all syntax changes.  In versions of LilyPond before 2.4.2,
accents and non-English characters were entered using standard LaTeX
notation. For example the French word for @q{Christmas} was entered as
@code{No\"el}.  But in LilyPond 2.6 onwards, the special @code{ë} must
be entered directly as a UTF-8 character.  The @command{convert-ly}
command cannot change LaTeX special characters into UTF-8 characters, so
older LilyPond input files have to edited manually.

The conversion rules of the @command{convert-ly} command work using text
pattern-matching and replacement (rather than @q{understanding} the
context of what it is changing within a given input file).  This has
several consequences:

@itemize @bullet
@item
The reliability of the conversion depends on the quality of each applied
rule set and on the complexity of the respective change.  Sometimes
conversions may require additional, manual fixes, so the original input
files should be kept for comparison just in case.

@item
Only conversions to newer syntax changes are possible: there are no rule
sets to go back to older versions of LilyPond.  So the input file
should only be upgraded when older versions of LilyPond are no longer
being maintained.  Again, the original input files should be kept just
in case; perhaps using version control systems (i.e. Git) to help with
maintaining multiple versions of your input files.

@item
LilyPond is quite robust when processing @q{creatively} placed or
omitted whitespace, but the rules used by @command{convert-ly} often
make some stylistic assumptions.  Therefore following the input style as
used in the LilyPond manuals is advised for painless upgrades,
particularly as the examples in the manuals themselves are all upgraded
using the @command{convert-ly} command.
@end itemize


@node Invoking convert-ly
@section Invoking @command{convert-ly}

The @command{convert-ly} command uses the @code{\version} number in
the input file to detect older versions.  In most cases, to upgrade your
input file it is sufficient just to run;

@example
convert-ly -e myfile.ly
@end example

@noindent
in the directory containing the input file.  This will upgrade
@file{myfile.ly} in-place and preserve the original file by renaming it
@file{myfile.ly~}.  The @code{\version} number in the upgraded input
file, along with any required syntax updates, is also changed.

When run, the @command{convert-ly} command will output the version
numbers of which conversions have been made to.  If no version numbers
are listed in the output for the file, it is already up to date and
using the latest LilyPond syntax.

@warning{For each new version of LilyPond, a new @command{convert-ly}
command is created, however not every version of LilyPond will need
syntax changes for its input files from the version before.  This means
that the @command{convert-ly} command will only convert input files up
to the latest syntax change it has and this, in turn, may mean that the
@code{\version} number left in the upgraded input file is sometimes
earlier than the version of @command{convert-ly} command itself.}

To convert all input files in a single directory use;

@example
convert-ly -e *.ly
@end example

Linux and MacOS@tie{}X users can both use the appropriate terminal
application, but MacOS@tie{}X users can also execute this command
directly under the menu entry @code{Compile > Update syntax}.

A Windows user would run the command;

@example
convert-ly.py -e *.ly
@end example

@noindent
entering these commands in a @code{command prompt} usually found under
@code{Start > Accessories > Command Prompt} or for version 8 users,
by typing in the search window @q{command prompt}.

To convert all input files that reside in different sets of
subdirectories;

@example
find . -name '*.ly' -exec convert-ly -e '@{@}' \;
@end example

This example searches and converts all input files in the current
directory and all directories below it recursively.  The converted files
will be located in the same directory along with their renamed
originals.  This should also work for MacOS@tie{}X users, although only
via the terminal app.

Windows user would use;

@example
forfiles /s /M *.ly /c "cmd /c convert-ly.py -e @@file"
@end example

Alternatively, an explicit path to the top-level of your folder
containing all the sub-folders that have input files in them can be
stated using the @code{/p} option;

@example
forfiles /s /p C:\Documents\MyScores /M *.ly /c "cmd /c convert-ly.py -e @@file"
@end example

If there are spaces in the path to the top-level folder, then the whole
path needs to be inside double quotes;

@example
forfiles /s /p "C:\Documents\My Scores" /M *.ly /c "cmd /c convert-ly.py -e @@file"
@end example



@node Command line options for convert-ly
@section Command line options for @command{convert-ly}

The program is invoked as follows:

@example
convert-ly [@var{option}]@dots{} @var{filename}@dots{}
@end example

The following options can be given:

@table @code
@item -d, --diff-version-update
increase the @code{\version} string only if the file has actually
been changed.  In that case, the version header will correspond to
the version after the last actual change.  An unstable version
number will be rounded up to the next stable version number unless
that would exceed the target version number.  Without this option,
the version will instead reflect the last @emph{attempted}
conversion.

@item -e, --edit
Apply the conversions direct to the input file, modifying it
in-place.  The original file is renamed as @file{myfile.ly~}.  This
backup file may be a hidden file on some operating systems.
Alternatively, if you want to specify a different name for the
upgraded file without using the @code{-e} options default @code{~}
appended to the old input file, the output can be redirected instead;

@example
convert-ly myfile.ly > mynewfile.ly
@end example

Windows user would use;

@example
convert-ly.py myfile.ly > mynewfile.ly
@end example

@item -b, --backup-numbered
When used with the @samp{-e} option, number the backup files so that
no previous version is overwritten.  The backup files may be hidden
on some operating systems.

@item -f, --from=@var{from-patchlevel}
Set the version to convert from.  If this is not set,
@command{convert-ly} will guess this, on the basis of @code{\version}
strings in the file. E.g. @option{--from=2.10.25}

@item -h, --help
Print usage help.

@item -l @var{loglevel}, --loglevel=@var{loglevel}
Set the output verbosity to @var{loglevel}. Possible values, in upper
case, are @code{PROGRESS} (the default), @code{NONE}, @code{WARNING},
@code{ERROR} and @code{DEBUG}.

@item -n, --no-version
Normally, @command{convert-ly} adds a @code{\version} indicator
to the output.  Specifying this option suppresses this.

@item -s, --show-rules
Show all known conversions and exit.

@item -t, --to=@var{to-patchlevel}
Explicitly set which @code{\version} to convert to, otherwise the
default is the most current value.  It must be higher than the
starting version.

@example
convert-ly --to=2.14.1 myfile.ly
@end example

@end table

To upgrade LilyPond fragments in texinfo files, use

@example
convert-ly --from=@dots{} --to=@dots{} --no-version *.itely
@end example

To see the changes in the LilyPond syntax between two versions, use

@example
convert-ly --from=@dots{} --to=@dots{} -s
@end example


@node Problems running convert-ly
@section Problems running @code{convert-ly}

When running convert-ly in a Command Prompt window under Windows
on a file which has spaces in the filename or in the path to it,
it is necessary to surround the entire input file name with three
(!) sets of double quotes:

@example
convert-ly """D:/My Scores/Ode.ly""" > "D:/My Scores/new Ode.ly"
@end example

If the simple @command{convert-ly -e *.ly} command fails because the
expanded command line becomes too long, the @command{convert-ly}
command may be placed in a loop instead.  This example for UNIX
will upgrade all @file{.ly} files in the current directory

@example
for f in *.ly; do convert-ly -e $f; done;
@end example

In the Windows Command Prompt window the corresponding command is

@example
for %x in (*.ly) do convert-ly -e """%x"""
@end example

Not all language changes are handled.  Only one output option can be
specified.  Automatically updating scheme and LilyPond scheme
interfaces is quite unlikely; be prepared to tweak scheme code
manually.


@node Manual conversions
@section Manual conversions

In theory, a program like @command{convert-ly} could handle any
syntax change.  After all, a computer program interprets the old
version and the new version, so another computer program can
translate one file into another@footnote{At least, this is
possible in any LilyPond file which does not contain scheme.  If
there is scheme in the file, then the LilyPond file contains a
Turing-complete language, and we run into problems with the famous
@qq{Halting Problem} in computer science.}.

However, the LilyPond project has limited resources: not all
conversions are performed automatically.  Below is a list of known
problems.


@verbatim
1.6->2.0:
 Doesn't always convert figured bass correctly, specifically things like {<
>}.  Mats' comment on working around this:
   To be able to run convert-ly
   on it, I first replaced all occurrences of '{<' to some dummy like '{#'
   and similarly I replaced '>}' with '&}'.  After the conversion, I could
   then change back from '{ #' to '{ <' and from '& }' to '> }'.
 Doesn't convert all text markup correctly.  In the old markup syntax,
 it was possible to group a number of markup commands together within
parentheses, e.g.
   -#'((bold italic) "string")
   This will incorrectly be converted into
   -\markup{{\bold italic} "string"}
   instead of the correct
   -\markup{\bold \italic "string"}
2.0->2.2:
 Doesn't handle \partcombine
 Doesn't do \addlyrics => \lyricsto, this breaks some scores with multiple
stanzas.
2.0->2.4:
 \magnify isn't changed to \fontsize.
    - \magnify #m => \fontsize #f, where f = 6ln(m)/ln(2)
 remove-tag isn't changed.
    - \applyMusic #(remove-tag '. . .) => \keepWithTag #'. . .
 first-page-number isn't changed.
    - first-page-number no => print-first-page-number = ##f
 Line breaks in header strings aren't converted.
    - \\\\  as line break in \header strings => \markup \center-align <
      "First Line" "Second Line" >
 Crescendo and decrescendo terminators aren't converted.
    - \rced => \!
    - \rc => \!
2.2->2.4:
 \turnOff (used in \set Staff.VoltaBracket = \turnOff) is not properly
converted.
2.4.2->2.5.9
 \markup{ \center-align <{ ... }> } should be converted to:
 \markup{ \center-align {\line { ... }} }
 but now, \line is missing.
2.4->2.6
 Special LaTeX characters such as $~$ in text are not converted to UTF8.
2.8
 \score{} must now begin with a music expression.  Anything else
 (particularly \header{}) must come after the music.
@end verbatim