@c -*- coding: utf-8; mode: texinfo; -*-
@node Website work
@chapter Website work

@menu
* Introduction to website work::
* Uploading and security::
* Debugging website and docs locally::
* Translating the website::
@end menu


@node Introduction to website work
@section Introduction to website work

The website is @emph{not} written directly in HTML;
instead it is autogenerated along with the documentation through a
sophisticated setup, using Texinfo source files.  Texinfo is the
standard for documentation of GNU software and allows generating
output in HTML, PDF, and Info formats, which drastically reduces
maintenance effort and ensures that the website content is
consistent with the rest of the documentation.  This makes the
environment for improving the website rather different from common
web development.

If you have not contributed to LilyPond before, a good starting
point might be incremental changes to the CSS file, to be found at
@uref{http://lilypond.org/css/lilypond-website.css} or in the
LilyPond source code at @file{./Documentation/css/lilypond-website.css}.

Large scale structural changes tend to require familiarity with
the project in general, a track record in working on LilyPond
documentation as well as a prospect of long-term commitment.

The Texinfo source file for generating HTML are to be found in

@example
Documentation/web.texi
Documentation/web/*.texi
@end example

Unless otherwise specified, follow the instructions and policies
given in @ref{Documentation work}.  That chapter also contains a
quick introduction to Texinfo; consulting an external Texinfo
manual should be not necessary.

@subheading Exceptions to the documentation policies

@itemize

@item
Sectioning: the website only uses chapters and sections; no
subsections or subsubsections.

@item
@@ref@{@}s to other manuals (@@ruser, @@rlearning, etc): you can't
link to any pieces of automatically generated documentation, like
the IR or certain NR appendices.

@item
The bibliography in Community->Publications is generated automatically
from @file{.bib} files; formatting is done automatically by
@file{texi-web.bst}.

@item
@dots{}

@item
For anything not listed here, just follow the same style as the
existing website texinfo files.

@end itemize


@node Uploading and security
@section Uploading and security

@subheading Overall idea

To reduce the CPU burden on the shared host (as well as some
security concerns), we do not compile all of LilyPond.  The
website build process runs @code{texi2html}, but all media files
(be they graphical @code{lilypond} output, photos of people, or
pdfs) are copied from the @code{$LILYPOND_WEB_MEDIA_GIT}
repository.

All scripts and makefiles used for the website build are run from
a @qq{trusted} copy.  Any modification to those files in git needs
a human to review the changes (after they have been made in git)
before they are used on the server.


@subheading Building the website (quick local)

Initial setup: make sure that you have the environment variables
@code{$LILYPOND_GIT}, @code{$LILYPOND_BUILD_DIR} and
@code{$LILYPOND_WEB_MEDIA_GIT} set up correctly.  For more
information, see @ref{Environment variables}.

Once that is done,

@example
cd $LILYPOND_BUILD_DIR
make website
@end example

The website is in @file{out-website/website/index.html}.


@subheading Building the website (exactly as on the server)

@subsubheading Setting up (exactly as on the server)

Initial setup: you still need @code{$LILYPOND_GIT} and
@code{$LILYPOND_WEB_MEDIA_GIT}.

Once that's done, create:

@example
mkdir -p $HOME/lilypond/
mkdir -p $HOME/lilypond/bin/
mkdir -p $HOME/lilypond/cron/
mkdir -p $HOME/lilypond/trusted-scripts/
@end example

The add these files to @file{$HOME/lilypond/bin/}:

Update git repositories:

@smallexample
@verbatim
### update-git.sh
#!/bin/sh
cd $LILYPOND_GIT
git fetch origin
git merge origin/master
cd $LILYPOND_WEB_MEDIA_GIT
git fetch origin
git merge origin/master
@end verbatim
@end smallexample

Check for any updates to trusted scripts / files:

@smallexample
@verbatim
### check-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
diff -u $DEST/website.make \
  $LILYPOND_GIT/make/website.make
diff -u $DEST/lilypond-texi2html.init \
  $LILYPOND_GIT/Documentation/lilypond-texi2html.init
diff -u $DEST/extract_texi_filenames.py \
  $LILYPOND_GIT/scripts/build/extract_texi_filenames.py
diff -u $DEST/create-version-itexi.py \
  $LILYPOND_GIT/scripts/build/create-version-itexi.py
diff -u $DEST/create-weblinks-itexi.py \
  $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py
diff -u $DEST/mass-link.py \
  $LILYPOND_GIT/scripts/build/mass-link.py
diff -u $DEST/website_post.py \
  $LILYPOND_GIT/scripts/build/website_post.py
diff -u $DEST/bib2texi.py \
  $LILYPOND_GIT/scripts/build/bib2texi.py
diff -u $DEST/langdefs.py \
  $LILYPOND_GIT/python/langdefs.py
diff -u $DEST/lilypond.org.htaccess \
  $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess
diff -u $DEST/website-dir.htaccess \
  $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess
@end verbatim
@end smallexample

If the changes look ok, make them trusted:

@smallexample
@verbatim
### copy-from-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
cp $LILYPOND_GIT/make/website.make \
  $DEST/website.make
cp $LILYPOND_GIT/Documentation/lilypond-texi2html.init \
  $DEST/lilypond-texi2html.init
cp $LILYPOND_GIT/scripts/build/extract_texi_filenames.py \
  $DEST/extract_texi_filenames.py
cp $LILYPOND_GIT/scripts/build/create-version-itexi.py \
  $DEST/create-version-itexi.py
cp $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py \
  $DEST/create-weblinks-itexi.py
cp $LILYPOND_GIT/scripts/build/mass-link.py \
  $DEST/mass-link.py
cp $LILYPOND_GIT/scripts/build/website_post.py \
  $DEST/website_post.py
cp $LILYPOND_GIT/scripts/build/bib2texi.py \
  $DEST/bib2texi.py
cp $LILYPOND_GIT/python/langdefs.py \
  $DEST/langdefs.py
cp $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess \
  $DEST/lilypond.org.htaccess
cp $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess \
  $DEST/website-dir.htaccess
@end verbatim
@end smallexample

Build the website:

@smallexample
@verbatim
### make-website.sh
#!/bin/sh
DEST=$HOME/web/
BUILD=$HOME/lilypond/build-website
mkdir -p $BUILD
cd $BUILD
cp $HOME/lilypond/trusted-scripts/website.make .

make -f website.make WEBSITE_ONLY_BUILD=1 website
rsync -raO $BUILD/out-website/website/ $DEST/website/
cp $BUILD/out-website/pictures $DEST
cp $BUILD/out-website/.htaccess $DEST
@end verbatim
@end smallexample

Then in the @file{cronjob/} directory, put the cronjob to automate
the trusted portions:

@warning{@code{cron} will not inherit environment variables from
your main setup, so you must re-define any variables inside your
@code{crontab}.}

@smallexample
@verbatim
# website-rebuild.cron
LILYPOND_GIT=   ... fill this in
LILYPOND_WEB_MEDIA_GIT=   ... fill this in

11 * * * * $HOME/lilypond/trusted-scripts/update-git.sh >/dev/null 2>&1
22 * * * * $HOME/lilypond/trusted-scripts/make-website.sh >/dev/null 2>&1
@end verbatim
@end smallexample

As the final stage of the setup, run your @code{copy-from-git.sh}
script, assuming that you trust the current state of scripts in
lilypond git.

@subsubheading Normal maintenance

When there is a change to the build scripts and/or website
makefile, log in to the server (or your own home machine if you're
testing this there), and do

@example
update-git.sh
check-git.sh
@end example

After reviewing the changes carefully, you can update the trusted
scripts with @code{copy-from-git.sh}.


@subsubheading Building the website (exactly as on the server)

Run @code{make-website.sh}; the final version ends up in
@file{$HOME/web/}.

On the actual server, the website is generated hourly by user
@code{graham} the host @code{lilypond.org}.  You can set up the
cronjob by doing:

@example
crontab $HOME/lilypond/website-rebuild.cron
@end example


@subheading Initial setup for new users on actual serve

You should symlink your own @file{~/lilypond/} to
@file{~graham/lilypond/}

If this directory does not exist, make it.  Git master should go
in @file{~/lilypond/lilypond-git/} but make sure you enable:

@example
git config core.filemode false
@end example

If you have created any files in @file{~graham/lilypond/} then
please run:

@example
chgrp lilypond ~graham/lilypond/ -R
chmod 775 ~graham/lilypond/ -R
@end example



@subsubheading Additional information

Some information about the website is stored in
@file{~graham/lilypond/*.txt}; this information should not be
shared with people without trusted access to the server.


@node Debugging website and docs locally
@section Debugging website and docs locally

@itemize
@item
Install Apache (you can use version 2, but keep in mind that the server
hosting lilypond.org runs version 1.3).  These instructions
assume that you also enable @code{mod_userdir}, and use
@code{$HOME/public_html} as DocumentRoot (i.e. the root directory of
the web server).

@item
Build the online docs and website:

@example
make WEB_TARGETS="offline online" doc
make website
@end example

This will make all the language variants of the website.  To save
a little time, just the English version can be made with the
command @code{make WEB_LANGS='' website} or the English and (for
example) the French with @code{make WEB_LANGS='fr' website}.

@item
Choose the web directory where to copy the built stuff.
If you already have other web projects in your DocumentRoot and don't
need to test the @file{.htaccess} file, you can copy to
@code{~/public_html/lilypond.org}.  Otherwise you'd better copy
to @code{~/public_html}.
It's highly recommended to have your build dir and web dir on the
same partition.

@item
Add the directory for the online documentation:

@example
mkdir -p ~/public_html/doc/v2.19/
@end example

You may want to add also the stable documentation in
@code{~/public_html/doc/v2.18/}, extracting the contents of the html
directory present in the tarball available in @rweb{All}. Just in case
you want to test the redirects to the stable documentation.

@item
Copy the files with rsync:

@example
rsync -av --delete out-website/website ~/public_html/
cp out-website/.htaccess ~/public_html
rsync -av --delete out-www/online-root/ ~/public_html/doc/v2.19/
@end example

@end itemize


@node Translating the website
@section Translating the website

As it has much more audience, the website should be translated before
the documentation; see @ref{Translating the documentation}.

In addition to the normal documentation translation practices,
there are a few additional things to note:

@itemize

@item
Build the website with:

@example
make website
@end example

@noindent
however, please note that this command is not designed for being
run multiple times.  If you see unexpected output (mainly the page
footers getting all messed up), then delete your
@file{out-website} directory and run @code{make website} again.

@item
Some of the translation infrastructure is defined in python files;
you must look at the @code{### translation data} sections in:

@example
scripts/build/create-weblinks-itexi.py
scripts/build/website_post.py
@end example

@item
Translations are not included by default in @code{make website}.
To test your translation, edit the @code{WEB_LANGUAGES =} line in
@file{python/langdefs.py}.  You will need to copy this updated
script to @code{$LILYPOND_BUILD_DIR/python/out}.

Do not submit a patch to add your language to this file unless
@code{make website} completes with fewer than 5 warnings.

@item
Links to manuals are done with macros like
@code{@@manualDevelLearningSplit}.  To get translated links, you
must change that to @code{@@manualDevelLearningSplit-es} (for
es/Spanish translations, for example).

@end itemize