Clean up buildscripts

[lilypond.git] / Documentation / TRANSLATION

LILYPOND DOCUMENTATION TRANSLATION

TABLE OF CONTENTS

SOURCES
GIT
REQUIREMENTS
WHICH DOCUMENTATION CAN BE TRANSLATED
STARTING A TRANSLATION IN A NEW LANGUAGE
FILES TO BE TRANSLATED
TRANSLATION DETAILED INSTRUCTIONS
* LEARNING MANUAL AND OTHER TEXINFO DOCUMENTATION
* REFERENCE NOTATION AND PROGRAM USAGE MANUAL
* DOCUMENTATION INDEX index.html.in
CHECK STATE OF TRANSLATION
UPDATE A TRANSLATION
POLICY DURING GDP PROCESS
MANAGING TRANSLATIONS WITH GIT
SOME GIT TIPS
DEALING WITH SEVERAL GIT BRANCHES
GIT PUSH ACCESS
TECHNICAL BACKGROUND


SOURCES

The sources live in a GIT repository.  Git 1.5.x is required, and
latest version available on your platform is always recommended.  To
get a fresh version of LilyPond sources run

    mkdir lily ; cd lily ; git init-db ; mkdir .git/remotes

then write the two following lines to a text file named .git/remotes/trans

URL: git://git.sv.gnu.org/lilypond.git/
Pull: lilypond/translation:refs/remotes/origin/lilypond/translation

then run

    git fetch trans
    git checkout -b lilypond/translation origin/lilypond/translation


GIT

The reader is supposed to be familiar with Git, for example by
having experience from lilypond.org translation; see
http://git.sv.gnu.org/gitweb/?p=lilypond.git;a=blob_plain;f=README;hb=web/master

If you do not have this experience, you may want to read the first two
chapters of Git User's Manual at
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html


REQUIREMENTS

Working on LilyPond documentation translations requires:

    * python
    * make
    * gettext


WHICH DOCUMENTATION CAN BE TRANSLATED

The makfiles and scripts infrastructure currently supports translation
of the following documentation:

    * documentation index (HTML)
    * user manual and program usage -- Texinfo source, PDF and HTML
output; Info output might be added if there is enough demand for it.


STARTING A TRANSLATION IN A NEW LANGUAGE

At top of the source directory, do

    ./autogen.sh

or (if you want to install your self-compiled LilyPond locally):

    ./autogen.sh --prefix=$HOME

If you want to compile LilyPond -- which is almost required to build
the docs, but is not required to do translation only -- fix all
dependencies and rerun ./configure (with the same options as for
autogen).

Cd into Documentation and run:

    make ISOLANG=<MY-LANGUAGE> new-lang

where <MY-LANGUAGE> is the ISO 639 language code.

Add a language definition for your language in
python/langdefs.py.

See next section about what files to translate and the following
detailed instructions after the next section.


FILES TO BE TRANSLATED

All the following files are in Documentation/
Translation of Documentation/foo/bar should be
Documentation/<LANG>/foo/bar.
Unmentioned files should not be translated.

Priorities:
  1. delivery
  2. 3. 4. 5. later
  6. optional

Files marked with priority 3, 4 or 5 may be submitted individually.
Word counts (excluding lilypond snippets) are given for each file.

-1- Documentation index and Tutorial
429   user/lilypond-learning.tely
6365  user/tutorial.itely
23    user/dedication.itely
423   user/macros.itexi
171   index.html.in
6346  po/lilypond-doc.pot (translate to po/<MY_LANGUAGE>.po)
---   ../lilypond-texi2html.init (section TRANSLATIONS)
13757 total

-2- Introduction and beginning of Application Usage
411   user/preface.itely
3855  user/introduction.itely
407   user/lilypond-program.tely
1930  user/install.itely (partial translation)
1149  user/setup.itely
2827  user/running.itely
10579 total

-3- Learning manual
10318 user/fundamental.itely -- Fundamental concepts
14647 user/tweaks.itely -- Tweaking output
3007  user/working.itely -- Working on LilyPond files
483   user/templates.itely -- Templates
28455 total

-4- Notation reference
695   user/lilypond.tely
91    user/notation.itely -- Musical notation
3123  user/pitches.itely
5013  user/rhythms.itely
1146  user/expressive.itely
555   user/repeats.itely
1455  user/simultaneous.itely
1701  user/staff.itely
895   user/editorial.itely
2286  user/text.itely
76    user/specialist.itely -- Specialist notation
2670  user/vocal.itely
1464  user/chords.itely
702   user/piano.itely
810   user/percussion.itely
826   user/guitar.itely
66    user/strings.itely
242   user/bagpipes.itely
4487  user/ancient.itely
5805  user/input.itely -- Input syntax
2164  user/non-music.itely -- Non-musical notation
8451  user/spacing.itely -- Spacing issues
11391 user/changing-defaults.itely -- Changing defaults
5202  user/programming-interface.itely -- Interfaces for programmers
1190  user/notation-appendices.itely -- Notation manual tables
250   user/cheatsheet.itely -- Cheat sheet
62756 total

-5- Application usage
3248  user/lilypond-book.itely -- LilyPond-book
1171  user/converters.itely -- Converting from other formats
4419  total

-6- Appendices whose translation is optional
310   user/literature.itely
960   user/scheme-tutorial.itely (needs to be revised first)
1270  total


TRANSLATION DETAILED INSTRUCTIONS

Please follow all these instructions with care to ensure quality work.

All files should be encoded in UTF-8.

* LEARNING MANUAL AND OTHER TEXINFO DOCUMENTATION

Any title which comes with one of the following commands must not be
translated directly in the Texinfo source

@node                                                   @majorheading
@chapter       @unnumbered          @appendix           @chapheading
@section       @unnumberedsec       @appendixsec        @heading
@subsection    @unnumberedsubsec    @appendixsubsec     @subheading
@subsubsection @unnumberedsubsubsec @appendixsubsubsec  @subsubheading
@ref           @rglos

As a notable exception, the second argument 'Bar baz' of
@ref{Foo,'Bar baz',,info-file} should be translated.

@uref's names are to be translated.

In any section which looks like

@menu
* node1:: thing1
* node2:: thing2
...
@end menu

the node names (nodeN) are NOT to be translated, whereas extra title
information (thingN) is.

Every node name or section title must from now on be translated
separately in a .po file (just as well as LilyPond output messages).
This .po file should be in Documentation/po.


Take care of using typographic rules for your language, especially in
user/macros.itexi.


Please keep verbatim copies of music snippets (in @lilypond blocs).
However, some music snippets containing text that shows in the
rendered music, and sometimes translating this text really helps the
user to understand the documentation; in this case, and only in this
case, you may as an exception translate text in the music snippet, and
then you must add a line immediately before the @lilypond block,
beginning with

@c KEEP LY

Otherwise the music snippet would be reset to the same content as the
English version at next 'make snippet-update' run (see UPDATING A
TRANSLATION below).

When you encounter

  @lilypondfile[<number of fragment options>,texidoc]{FILENAME.ly}

in the source, open input/lsr/FILENAME.ly, translate the `texidoc'
header field it contains, enclose it with 'texidoc<MY-LANGUAGE> = "'
and '"', and write it into input/texidocs/FILENAME.texidoc -- please
keep possibly existing translations in other languages!
Additionnally, you may translate the snippet's title in `doctitle'
header field, in case `doctitle' is a fragment option used in
@lilypondfile; you can do this exactly the same way as `texidoc'.  For
instance, input/texidocs/FILENAME.texidoc may contain

doctitlees = "Spanish title baz"
texidoces = "
Spanish translation blah
"
doctitlede = "German title bar"
texidocde = "German translation foo
"


@example blocs need not be verbatim copies, e.g. variable names,
file names and comments should be translated.

Index entries (@cindex and so on) should be translated.

Carefully apply every rule exposed in Documentation/README.txt.  If
one of these rules conflicts with a rule specific to your language,
please ask the Translation meister and/or the Documentation Editor on
lilypond-devel@gnu.org.


* REFERENCE NOTATION AND PROGRAM USAGE MANUAL

Copy user/lilypond.tely (or user/lilypond-program.tely, respectively)
into <MY-LANGUAGE>/user, then translate this file and run
skeleton-update (see UPDATE A TRANSLATION below).  Your are now ready
to translate notation reference (program usage manual, respectively)
exactly like the learning manual.


* DOCUMENTATION INDEX index.html.in

Unlike almost all HTML pages in this documentation, links in this page
are not tweaked by add_html_footer.py, so links should be manually
edited to link to existing translations.


CHECK STATE OF TRANSLATION

First pull, then cd into Documentation (or at top of the source tree,
replace 'make' with 'make -C Documentation') and run

    make ISOLANG=<MY_LANGUAGE> check-translation

This presents a diff of the original files since the most recent
revision of the translation.  To check a single file, cd into
Documentation and run

    make CHECKED_FILES=<MY-LANGUAGE>/user/foo.itely check-translation


Small tip: to see only which files need to be updated, do

    make ISOLANG=<MY_LANGUAGE> check-translation | grep 'diff --git'

To avoid printing terminal colors control characters, which is often
desirable when you redirect output to a file, run

    make ISOLANG=<MY_LANGUAGE> NO_COLOR=1 check-translation


Global state of the translation is recorded in
Documentation/translations.html.in, which is used to generate
Translations status page.  To update that page, do from Documentation/

    make translation-status

This will also leave out/translations-status.txt, which contains
up-to-dateness percentages for each translated file, and update word
counts of documentation files in this file.


UPDATE A TRANSLATION

Instead of running check-translation, you can run update-translation,
which will run your favorite text editor to update files.  First, make
sure environment variable EDITOR is set to a text editor command, then
run from Documentation

    make ISOLANG=<MY-LANGUAGE> update-translation

or to update a single file

    make CHECKED_FILES=<MY-LANGUAGE>/user/foo.itely update-translation

For each file to be udpated, update-translation will open your text
editor with this file and a diff of the file in English; if the diff
cannot be generated or is bigger than the file in English itself, the
full file in English will be opened instead.

Texinfo skeleton files, i.e. .itely files not yet translated,
containing only the Texinfo structure can be updated automatically:
whenever 'make check-translation' shows that such files should be
updated, run from Documentation

    make ISOLANG=<MY_LANGUAGE> skeleton-update

.po message catalogs in Documentation/po may be updated with (from
Documentation or Documentation/po)

    make po-update

WARNING: if you run po-update and somebody else does the same and
pushes before you push or send a patch to be applied, there will be a
conflict when you pull.  Therefore, it is better that only the
Translation meister runs this command.

Updating music snippets can quickly become cumbersome, as most
snippets should be identical in all languages.  Fortunately, there is
a script that can do this odd job for you (run from Documentation):

    make ISOLANG=<MY_LANGUAGE> snippet-update

This script overwrites music snippets in <MY_LANGUAGE>/user/every.itely
with music snippets from user/every.itely.  It ignores skeleton files,
and keeps intact music snippets preceded with a line starting with '@c
KEEP LY'; it reports an error for each .itely that has not the same
music snippet count in both languages.  Always use this script with a
lot of care, i.e. run it on a clean Git working tree, and check the
changes it made with "git diff" before committing; if you don't do so,
some @lilypond snippets might be broken or make no sense in their
context.

Finally, a command runs the three update processes above for all
enabled languages (from Documentation):

    make all-translations-update

Use this command with caution, and keep in mind it will not be really
useful until translations are stabilized after the end of GDP.


POLICY DURING GDP PROCESS
or "How to maintain translations without updating them"

During GDP progress, documentation changes so much, and translators are
often involved in GDP too, so keeping translations up to date is very
difficult.  However, it is possible -- and even recommended -- to
perform some maintaining that keeps translated documentation usable
and eases future translation updating.  The rationale below the tasks
list motivates this plan.

The following tasks are listed in decreasing priority order.

1) Update macros.itexi.  For each obsolete macro definition, if it is
possible to update macro usage in documentation with an automatic text
or regexp substitution, do it and delete the macro definition from
macros.itexi; otherwise, mark this macro definition as obsolete with a
comment, and keep it in macros.itexi until the documentation
translation has been updated and no longer uses this macro.

2) Update *.tely files completely with make check-translation -- you
may want to redirect ouptput to a file because of overwhelming output,
or call check-translation.py on individual files, see CHECK STATE OF
TRANSLATION.

3) in .itelys, match sections and .itely file names with those from
English docs, which possibly involves moving nodes contents in block
between files, without updating contents itself.  In other words, the
game is catching where has gone each section.  In Learning manual, and
in Notation Reference sections which have been revised in GDP, there
may be completely new sections: in this case, copy @node and
@section-command from English docs, and add the marker for
untranslated status '@untranslated' on a single line.  Note that it is
not possible to exactly match subsections or subsubsections of
documentation in English, when contents has been deeply revised; in
this case, keep obsolete (sub)subsections in the translation, marking
them with a line '@c obsolete' just before the node.

4) update sections finished in GDP; check sections status at GDP website.


* Hints for Emacs users

Emacs with Texinfo mode makes this step easier:

- without Emacs AucTeX installed, C-c C-s shows structure of current
Texinfo file in a new buffer *Occur*; to show structure of two files 
simultaneously, first split Emacs window in 4 tiles (with C-x 1 and 
C-x 2), press C-c C-s to show structure of one file (e.g. the translated
file), copy *Occur* contents into *Scratch*, then press C-c C-s for the 
other file.
If you happen to have installed AucTeX, you can either call the macro
by doing M-x texinfo-show-structure or create a key binding in your
~/.emacs, by adding the four following lines:
        (add-hook 'Texinfo-mode-hook
                  '(lambda ()
                     (define-key Texinfo-mode-map "\C-cs"
                      'texinfo-show-structure)))
and then obtain the structure in the *Occur* buffer with C-c s


- Do not bother updating @menus when all menu entries are in the same
file ; make sure there is at least a (possibly empty) @menu block
everywhere it is needed, then do C-c C-u C-a ("update all menus") when
you have updated all the rest of the file.

- Moving to next or previous node: press C-s and type node (or C-s
@node if the text contains the word 'node') then press C-s to move to
next node or C-r to move to previous node.  Similar operation can be
used to move to the next/previous section.

- Moving a whole node (or even a sequence of nodes): jump to beginning
of the node (quit incremental search by pressing an arrow), press
C-SPACE, press C-s node and repeat C-s until you have selected enough
text, cut it with C-w or C-x, jump to the right place (moving between
nodes with the previous hint is often useful) and paste with C-y or
C-v.


4) update documentation PO.  Unless you have special interest in
having all titles translated in the next development release, it is
better to wait until step 3) has been completed, to avoid doing the
work more than once.

5) Fix broken cross-references by running (from Documentation/)

  make ISOLANG=<YOUR-LANGUAGE> fix-xrefs

This step requires a sucessful documentation build (with 'make web').
Some cross-references are broken because they point to a node that
exists in the documentation in English, which has not been added to
the translation; in this case, do not fix the cross-reference but keep
it "broken", so that the resulting HTML link will point to an existing
page of documentation in English.

Rationale

You may wonder if it would not be better to leave translations as-is
until you can really start updating translations.  There are several
reasons to do these maintenance tasks right now.

- This will have to be done sooner or later anyway, before updating
translation of documentation contents, and this can already be done
without needing to be redone later, as sections of documentation in
English are mostly revised once.  However, note that not all
documentation sectioning has been revised in one go, so all this
maintenance plan has to be repeated whenever a big reorganization is
made. Currently (in May 2008), only chapters 3-7 in Notation Reference
and Application Usage have not been reorganized yet.

- This just makes translated documentation take advantage of the new
organization, which is far better than the old one.

- Moving and renaming sections to match sectioning of documentation in
English simplify future updating work: it allows updating the
translation by side-by-side comparison, without bothering whether
cross-reference names already exist in the translation.

- Each maintenance task (except 4) updating PO files) can be done by
the same person for all languages, which saves overall time spent by
translators to achieve this task: the node names and section titles
are in English, so you can do.  It is important to take advantage of
this now, as it will be more complicated (but still possible) to do
step 3) in all languages when documentation is compiled with
texi2html and node names are directly translated in source files.


MANAGING TRANSLATIONS WITH GIT

This policy explains how to manage Git branches and commit
translations to Git.

* Translation changes matching master branch are preferably made on
lilypond/translation branch; they may be pushed directly on master
only if they do not break compilation of LilyPond and its
documentation, and in this case they should be pushed to
lilypond/translation too.  Similarly, changes matching stable/X.Y are
preferably made on lilypond/X.Ytranslation.

* lilypond/translation Git branch may be merged into master only if
LilyPond ('make all') and documentation ('make web') compile
succesfully.

* master Git branch may be merged into lilypond/translation whenever
'make' and 'make web' are succesful (in order to ease documentation
compilation by translators), or when significant changes had been made
in documentation in English in master branch.

* General maintenance may be done by anybody who knows what he does
in documentation in all languages, without informing translators
first.  General maintenance include simple text substitutions
(e.g. automated by sed), compilation fixes, updating Texinfo or
lilypond-book commands, updating macros, updating ly code, fixing
cross-references, and operations described in 'POLICY DURING GDP
PROCESS'.


SOME GIT TIPS

* Saving uncommited changes in the working tree:

    git diff > foo.diff

This does not save untracked or ignored files.  If you prefer to
include changes added to the index with 'git add', replace 'git diff'
with 'git diff HEAD'.
Then, you may try to apply foo.diff on a source tree with

    patch -p1 < foo.diff


DEALING WITH SEVERAL GIT BRANCHES

* It is possible to work with several branches on the same local Git
repository; this is especially useful for translators who may have to
deal with both lilypond/translation and a stable branch
(e.g. stable/2.12 or lilypond/translation-2.12).  To fetch and check
out a new branch named BRANCH on git.sv.gnu.org, write the two
following lines to a text file named .git/remotes/SHORTHAND --
SHORTHAND is the name of the remote file, i.e. whatever easy-to-type
name you would like to use when pulling or pushing BRANCH, and usually
SHORTHAND is an abbreviation of BRANCH without slashes

URL: git://git.sv.gnu.org/lilypond.git/
Pull: BRANCH:refs/remotes/origin/BRANCH

Then, run

    git fetch SHORTHAND
    git checkout -b BRANCH origin/BRANCH

After this, you are able to pull BRANCH from git.sv.gnu.org with

    git pull SHORTHAND

You can check out another branch OTHER_BRANCH, i.e. check out
OTHER_BRANCH to the working tree, with

    git checkout OTHER_BRANCH

E.g. lilypond/translation, which you still have in your local Git
repository but is no longer checked out since you have created the new
branch BRANCH.

Note that it is possible to check out another branch while having
uncommitted changes, but it is not recommended unless you know what
you are doing; it is recommended to run 'git status' to check this
kind of issue before checking out another branch.

When pulling using SHORTHAND, do not forget to check first that the
right branch is checked out, i.e. the branch named A in the first part
of the "A:B" refspec in .git/remotes/SHORTHAND: as a matter of fact,
when you pull using A:B refspec, Git fetch A on the server as B remote
branch on your local repository, then tries to merge B into the
currently checked out branch.

To remember which branch is currently checked out, run 'git branch',
which will list all branches and mark the currently checked out branch
with a star, or 'git status'.


* To merge branch FOO into branch BAR, i.e. to "add" all changes made
in branch FOO to branch BAR, run

    git checkout BAR
    git merge FOO

If any conflict happens, please carefully follow the instructions
given by 'git merge' -- you usually must resolve conflicts with a text
editor by merging pieces of files marked with "<<<" "===" and ">>>",
removing these 3 kinds of conflict marks, then commit the result
exactly like a usual commit.

For example, as a translator, you will often want to merge master into
lilypond/translation; on the other hand, the Translations meister
wants to merge lilypond/translation into master whenever he has
checked that lilypond/translation builds successfully.


* If you play with several Git branches (e.g. master,
lilypond/translation, stable/2.12), you may want to have one source
and build tree for each branch; this is possible with subdirectories
of your local Git repository, used as local cloned subrepositories.
To create a local clone for the branch named BRANCH, run

    git checkout BRANCH
    git clone -l -s -n . SUBDIR
    cd SUBDIR
    git reset --hard

Note that SUBDIR must be a directory name which does not already
exist.  In SUBDIR, you can use all Git commands to browse revisions
history, commit and uncommit changes; to update the cloned
subrepository with changes made on the main repository, cd into SUBDIR
and run 'git pull'; to send changes made on the subrepository back to
the main repository, run 'git push' from SUBDIR.  Note that only one
branch (the currently checked out branch) is created in the
subrepository by default; it is possible to have several branches in a
subrepository and do usual operations (checkout, merge, create,
delete...) on these branches, but this possibility is not detailed
here.

Note that when you push BRANCH from SUBDIR to the main repository,
and BRANCH is checked out in the main repository, you must save
uncommitted changes (see SOME GIT TIPS) and do 'git reset --hard' in
the main repository in order to apply pushed changes in the working
tree of the main repository.


GIT PUSH ACCESS

If you have permission to push to Git with login USER, please start a
new Git repository from scratch to avoid polluting history with
duplicate commits; follow the usual instructions, except that every
file you write in .git/remotes should contain instead

URL: ssh://USER@git.sv.gnu.org/srv/git/lilypond.git
Push: BRANCH:refs/heads/BRANCH
Pull: BRANCH:refs/remotes/origin/BRANCH

Then, you can use .git/remotes/NAME to push BRANCH with

    git push NAME

which works regardless of the branch checked out.


TECHNICAL BACKGROUND

A number of Python scripts handle a part of the documentation
translation process.
All scripts used to maintain the translations
are located in scripts/aux/:

* check_translation.py  -- show diff to update a translation
* texi-langutils.py  -- quickly and dirtily parse Texinfo files to
make message catalogs and Texinfo skeleton files
* texi-skeleton-update.py -- update Texinfo skeleton files
* update-snippets.py -- synchronize ly snippets with those from
English docs
* translations-status.py -- update translations status pages and word
counts in the file you are reading.
* tely-gettext.py -- gettext node names, section titles and references
in the sources; WARNING only use this script when support for
"makeinfo --html" has been dropped.

Other scripts are used in the build process, in scripts/build/:
* html-gettext.py -- translate node names, section titles and cross
references in HTML files generated by makeinfo
* texi-gettext.py -- gettext node names, section titles and references
before calling texi2pdf
* mass-link.py -- link or symlink files between English documentation
and documentation in other languages

Python modules used by scripts in scripts/aux/ or scripts/build/ (but
not by installed Python scripts) are located in python/aux/:
* manuals_definitions.py -- define manual names and name of
cross-reference Texinfo macros
* buildlib.py -- common functions (read piped output
of a shell command, use Git)
* postprocess_html.py (module imported by www_post.py) -- add footer and
tweak links in HTML pages

And finally
* python/langdefs.py  -- language definitions module