Merge branch 'master' into lilypond/translation

[lilypond.git] / Documentation / contributor / lsr-work.itexi

@c -*- coding: utf-8; mode: texinfo; -*-
@node LSR work
@chapter LSR work

@menu
* Introduction to LSR::
* Adding and editing snippets::
* Approving snippets::
* LSR to Git::
* Fixing snippets in LilyPond sources::
* Renaming a snippet::
* Updating LSR to a new version::
@end menu


@node Introduction to LSR
@section Introduction to LSR

The
@uref{http://lsr.dsi.unimi.it/, LilyPond Snippet Repository (LSR)}
is a collection of lilypond examples.  A subset of these examples
are automatically imported into the documentation, making it easy
for users to contribute to the docs without learning Git and
Texinfo.


@node Adding and editing snippets
@section Adding and editing snippets

@subheading General guidelines

When you create (or find!) a nice snippet, if it supported by LilyPond
version running on LSR, please add it to LSR.  Go to
@uref{http://lsr.dsi.unimi.it/, LSR} and log in -- if you haven't
already, create an account.  Follow the instructions on the website.
These instructions also explain how to modify existing snippets.

If you think the snippet is particularly informative and you think it
should be included in the documentation, tag it with @qq{docs} and one
or more other categories, or ask somebody who has editing permissions to
do it on the development list.

Please make sure that the lilypond code follows the guidelines in
@ref{LilyPond formatting}.

If a new snippet created for documentation purposes compiles with
LilyPond version currently on LSR, it should be added to LSR, and a
reference to the snippet should be added to the documentation.

If the new snippet uses new features that are not available in the
current LSR version, the snippet should be added to
@file{Documentation/snippets/new} and a reference should be added to the
manual.

Snippets created or updated in @file{Documentation/snippets/new} should
be copied to @file{Documentation/snippets} by invoking at top of the
source tree

@example
scripts/auxiliar/makelsr.py
@end example

@noindent
This also copies translated texidoc fields and snippet titles into
snippets in @file{Documentation/snippets}.  Note: this, in turn, could
make the translated texidoc fields to appear as out of sync when you
run @code{make check-translation}, if the originals changed from the
last translation update, even if the translations are also updated;
see @ref{Documentation translation maintenance} for details about
updating the docs; in particular, see @ref{Updating translation
committishes} to learn how to mark these translated fields as fully
updated.

Be sure that @command{make doc} runs successfully before submitting a
patch, to prevent breaking compilation.

@subheading Formatting snippets in @file{Documentation/snippets/new}

When adding a file to this directory, please start the file with

@example
\version "2.x.y"
\header @{
% Use existing LSR tags other than 'docs'; see makelsr.py for
% the list of tags used to sort snippets.  E.g.:
  lsrtags = "rhythms,expressive-marks"
% This texidoc string will be formatted by Texinfo
  texidoc = "
This code demonstrates ...
"
% Please put doctitle last so that the '% begin verbatim'
% mark will be added correctly by makelsr.py.
  doctitle = "Snippet title"
@}
@end example

@noindent
and name the file @file{snippet-title.ly}.


@node Approving snippets
@section Approving snippets

The main task of LSR editors is approving snippets.  To find a list of
unapproved snippets, log into @uref{http://lsr.dsi.unimi.it/, LSR} and
select @qq{No} from the dropdown menu to the right of the word
@qq{Approved} at the bottom of the interface, then click
@qq{Enable filter}.

Check each snippet:

@enumerate

@item
Does the snippet make sense and does what the author claims that
it does?  If you think the snippet is particularly helpful, add
the @qq{docs} tag and at least one other tag.

@item
If the snippet is tagged with @qq{docs}, check to see if it
matches our guidelines for @ref{LilyPond formatting}.

Also, snippets tagged with @qq{docs} should not be explaining
(replicating) existing material in the docs.  They should not
refer to the docs; the docs should refer to them.

@item
If the snippet uses scheme, check that everything looks good and
there are no security risks.

@warning{Somebody could sneak a @code{#'(system "rm -rf /")}
command into our source tree if you do not do this!  Take this
step @strong{VERY SERIOUSLY}.}

@end enumerate


@node LSR to Git
@section LSR to Git

@enumerate

@item
Make sure that @command{convert-ly} and @command{lilypond} are a
bleeding edge version -- the latest release or even better a fresh
snapshot from Git master.

@item
Start by creating a list of updated snippets from your local
repository.  From the top source directory, run:

@example
scripts/auxiliar/makelsr.py
@end example

Commit the changes and make a patch.  Check the patch has nothing
other than minor changes - in particular changes to the commitish
for translations.  If all is good and you're confident in what
you've done, this can be pushed directly to staging.

@item
Next, download the updated snippets and run makelsr against them.
From the top source directory, run:

@smallexample
wget http://lsr.dsi.unimi.it/download/lsr-snippets-docs-@var{YYYY-MM-DD}.tar.gz
tar -xzf lsr-snippets-docs-@var{YYYY-MM-DD}.tar.gz
scripts/auxiliar/makelsr.py lsr-snippets-docs-@var{YYYY-MM-DD}
@end smallexample

@noindent
where @var{YYYY-MM-DD} is the current date, e.g. 2011-12-25.

@item
Follow the instructions printed on the console to manually check for
unsafe files.  These are:

@example
Unsafe files printed in lsr-unsafe.txt: CHECK MANUALLY!
  git add Documentation/snippets/*.ly
  xargs git diff HEAD < lsr-unsafe.txt
@end example

First, it's important to check for any added files and add them to
the files git is tracking.  Run @code{git status} and look
carefully to see if files have been added.  If so, add them with
@code{git add}.

As the console says, makelsr creates a list of possibly unsafe
files in @file{lsr-unsafe.txt} by running @code{lilypond} against each
snippet using the @code{-dsafe} switch.  This list can be quite
long.  However, by using the command @code{xargs git diff HEAD < lsr-unsafe.txt}
git will take that list and check whether any of the snippets are
different from the snippet already in master.  If any is different
it must be checked manually VERY CAREFULLY.

@warning{Somebody could sneak a @code{#'(system "rm -rf /")}
command into our source tree if you do not do this!  Take this
step @strong{VERY SERIOUSLY}.}

If there is any doubt about any of the files, you are strongly
advised to run a review on Rietveld.

@item
If a Review is not needed, commit the changes and push to staging.

@end enumerate

Note that whenever there is one snippet from
@file{Documentation/snippets/new} and the other from LSR with the same
file name, the one from @file{Documentation/snippets/new} will be copied
by @command{makelsr.py}.


@node Fixing snippets in LilyPond sources
@section Fixing snippets in LilyPond sources

In case some snippet from @file{Documentation/snippets} causes the
documentation compilation to fail, the following steps should be
followed to fix it reliably.

@enumerate

@item
Look up the snippet filename @file{@var{foo}.ly} in the error output
or log, then fix the file @file{Documentation/snippets/@var{foo}.ly} to make the
documentation build successfully.

@item
Determine where it comes from by looking at its first line, e.g. run

@example
head -1 Documentation/snippets/@var{foo}.ly
@end example

@item
@strong{In case the snippet comes from LSR}, apply the fix to the
snippet in LSR and send a notification email to a LSR editor with CC to
the development list -- see @ref{Adding and editing snippets}.  The
failure may sometimes not be caused by the snippet in LSR but by the
syntax conversion made by @command{convert-ly}; in this case, try to fix
@command{convert-ly} or report the problem on the development list, then
run @command{makelsr.py} again, see @ref{LSR to Git}.  In some cases,
when some features has been introduced or vastly changed so it requires
(or takes significant advantage of) important changes in the snippet, it
is simpler and recommended to write a new version of the snippet in
@file{Documentation/snippets/new}, then run @command{makelsr.py}.

@item
@strong{In case the snippet comes from}
@file{Documentation/snippets/new}, apply in
@file{Documentation/snippets/new/@var{foo}.ly} the same fix you did in
@file{Documentation/snippets/@var{foo}.ly}.  In case the build failure
was caused by a translation string, you may have to fix
@file{input/texidocs/@var{foo}.texidoc} instead.

@item
In any case, commit all changes to Git.

@end enumerate


@node Renaming a snippet
@section Renaming a snippet

Due to the potential duality of snippets (i.e. they may exist both
in the LSR database, and in @code{Documentation/snippets/new/}),
this process is a bit more involved than we might like.

@enumerate
@item
Send an email LSR editor, requesting the renaming.

@item
The LSR editor does the renaming (or debates the topic with you),
then warns the LSR-to-git person (wanted: better title) about the
renaming.

@item
LSR-to-git person does his normal job, but then also renames any
copies of the snippets in @code{Documentation/snippets/new/}, and
any instances of the snippet name in the documentation.

@code{git grep} is highly recommended for this task.

@end enumerate


@node Updating LSR to a new version
@section Updating LSR to a new version

To update LSR, perform the following steps:

@enumerate

@item
Download the latest snippet tarball, extract it, and run
@code{convert-ly} on all files using the command-line option
@option{--to=@var{version}} to ensure snippets are updated to the
correct stable version.

@item
Copy relevant snippets (i.e., snippets whose version is equal to or less
than the new version of LilyPond) from
@file{Documentation/snippets/new/} into the tarball.

You must not rename any files during this, or the next, stage.

@item
Verify that all files compile with the new version of LilyPond,
ideally without any warnings or errors.  To ease the process,
you may use the shell script that appears after this list.

Due to the workload involved, we @emph{do not} require that you
verify that all snippets produce the expected output.  If you
happen to notice any such snippets and can fix them, great; but as
long as all snippets compile, don't delay this step due to some
weird output.  If a snippet is broken, the hordes of willing
web-2.0 volunteers will fix it.  It's not our problem.

@item
Create a tarball and send it back to Sebastiano.

@item
When LSR has been updated, download another snippet tarball, verify that
the relevant snippets from @file{Documentation/snippets/new/} were
included, then delete those snippets from
@file{Documentation/snippets/new/}.

@end enumerate


Here is a shell script to run all @file{.ly} files in a directory
and redirect terminal output to text files, which are then
searched for the word "failed" to see which snippets do not compile.

@smallexample
#!/bin/bash

for LILYFILE in *.ly
do
  STEM=$(basename "$LILYFILE" .ly)
  echo "running $LILYFILE..."
  lilypond --format=png -ddelete-intermediate-files "$LILYFILE" >& "$STEM".txt
done

grep failed *.txt
@end smallexample