Doc: CG: explain website+LILYPOND_WEB_MEDIA_GIT

This work was sponsored by David Kastrup.

diff --git a/Documentation/contributor/source-code.itexi b/Documentation/contributor/source-code.itexi
index 75363f4d40be7d54eb615b0c66f1fd2e837d8d14..9322e766da9edb243e5fb6f08f95c7fa436c71c2 100644 (file)
--- a/Documentation/contributor/source-code.itexi
+++ b/Documentation/contributor/source-code.itexi
@@ -112,6 +112,7 @@ multiple projects concurrently.
 @menu
 * Setting up::
 * Git for the impatient::
+* Other repositories::
 * Downloading remote branches::
 @end menu
 
@@ -522,9 +523,35 @@ your branch (safe)}, but replace the @code{-d} on the final line with
 a @code{-D}.
 
 
+@node Other repositories
+@subsection Other repositories
+
+There is a separate repository for general administrative scripts,
+as well as pictures and media files for the website.  People
+interested in working on the website should download this
+repository, and set their @code{$LILYPOND_WEB_MEDIA_GIT}
+environment variable to point to that repository.
+
+@example
+@uref{https://github.com/gperciva/lilypond-extra}
+@end example
+
+To configure an environment variable in bash (the default for most
+Linux distributions),
+
+@example
+export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/
+@end example
+
+
+There are a few other repositories floating around, which will
+hopefully be documented in the near future.
+
+
 @node Downloading remote branches
 @subsection Downloading remote branches
 
+@warning{contains obsolete + misleading info}
 
 @menu
 * Organization of remote branches::
@@ -589,6 +616,8 @@ only be used as a last resort.
 @node Downloading individual branches
 @unnumberedsubsubsec Downloading individual branches
 
+@warning{obsolete, should be deleted!}
+
 
 Once you have initialized an empty Git repository on your system
 (see @ref{Initializing a repository}), you can download a remote

diff --git a/Documentation/contributor/website-work.itexi b/Documentation/contributor/website-work.itexi
index aaced80cff446d2845649a332f49209d3d8c9839..33b13464b64bff7296ccb808f1e178169024fef4 100644 (file)
--- a/Documentation/contributor/website-work.itexi
+++ b/Documentation/contributor/website-work.itexi
@@ -58,206 +58,137 @@ existing website texinfo files.
 @node Uploading and security
 @section Uploading and security
 
-@subheading Building the website from scratch for local checking
-
-Initial setup:
-
-Create directories:
-
-@example
-mkdir $HOME/lilypond/
-mkdir $HOME/lilypond/media/
-mkdir $HOME/lilypond/trusted-scripts/
-@end example
+@subheading Overall idea
 
 To reduce the CPU burden on the shared host (as well as some
-security concerns), the 'Documentation/pictures/' and
-'Documentation/ly-examples/' directories are **not** compiled.
-We will do this ourselves right now.
-
-Go to your lilypond build directory.  make doc.
+security concerns), we do not compile all of LilyPond.  The
+website build process runs @code{texi2html}, but all media files
+(be they graphical @code{lilypond} output, photos of people, or
+pdfs) are copied from the @code{$LILYPOND_WEB_MEDIA_GIT}
+repository.
 
-Set up some variables (you'll only do this once:)
+All scripts and makefiles used for the website build are run from
+a @qq{trusted} copy.  Any modification to those files in git needs
+a human to review the changes (after they have been made in git)
+before they are used on the server.
 
-@example
-BUILD_DIR=$HOME/lilypond-git
-PICS=$BUILD_DIR/Documentation/pictures/
-EXAMPLES=$BUILD_DIR/Documentation/ly-examples/
-@end example
 
-Copy files over:
+@subheading Building the website (quick local)
 
-@example
-cp -r $PICS $HOME/lilypond/media/pictures
-cp -r $EXAMPLES $HOME/lilypond/media/ly-examples
-@end example
+Initial setup: make sure that you have the environment variables
+@code{$LILYPOND_GIT} and @code{$LILYPOND_WEB_MEDIA_GIT} set up
+correctly.  For more information, see @ref{Other repositories}.
 
-Get the scripts you need.  First define these variables:
+Once that is done,
 
 @example
-GIT=$HOME/lilypond-git
-DEST=$HOME/lilypond/trusted-scripts
-@end example
-
-Then do the copying:
-
-@smallexample
-cp $GIT/make/website.make $DEST/website.make
-cp $GIT/Documentation/lilypond-texi2html.init $DEST/lilypond-texi2html.init
-cp $GIT/scripts/build/extract_texi_filenames.py  $DEST/extract_texi_filenames.py
-cp $GIT/scripts/build/create-version-itexi.py $DEST/create-version-itexi.py
-cp $GIT/scripts/build/create-weblinks-itexi.py $DEST/create-weblinks-itexi.py
-cp $GIT/scripts/build/mass-link.py $DEST/mass-link.py
-cp $GIT/scripts/build/website_post.py $DEST/website_post.py
-cp $GIT/scripts/build/bib2texi.py $DEST/bib2texi.py
-cp $GIT/Documentation/web/server/lilypond.org.htaccess $DEST/lilypond.org.htaccess
-cp $GIT/Documentation/web/server/website-dir.htaccess $DEST/website-dir.htaccess
-@end smallexample
-
-For a complete build you will need a copy of @code{lilypond-extra} git repository.
-You can checkout a fresh copy easily:
-
-@example
-export LILYPOND_WEB_MEDIA_GIT=$HOME/lilypond-extra
-git clone git://github.com/gperciva/lilypond-extra.git $LILYPOND_WEB_MEDIA_GIT
-@end example
-
-Just note that the example above expects a bash environment. If you are using another shell
-you might need to use a different keyword, other than @code{export}.
-
-Delete your build directory (or maybe just rename your build
-directory to build-old).
-
-@example
-cd $HOME/lilypond
-@end example
-
-Run
-
-@example
-make -f ../lilypond-git/make/website.make WEBSITE_ONLY_BUILD=1 \
-  TOP_SRC_DIR=$HOME/lilypond-git/ \
-  PYTHONPATH=$HOME/lilypond-git/python \
-  TEXI2HTML_PROGRAM=texi2html \
-  website
+cd $LILYPOND_GIT/build/
+make website
 @end example
 
-The website should be at:
+The website is in @file{out-website/website/index.html}.
 
-@example
-$HOME/lilypond/out-website/website/index.html
-@end example
 
-@subheading Building the online website
+@subheading Building the website (exactly as on the server)
 
-The website is generated hourly by user @code{graham} the host
-@code{lilypond.org}.  For security reasons, we do not use the
-makefiles and scripts directly from git; copies of the relevant
-scripts are examined and copied to
-@code{~graham/lilypond/trusted-scripts/}
+@subsubheading Setting up (exactly as on the server)
 
-@subheading Initial setup
-
-You should symlink your own @file{~/lilypond/} to
-@file{~graham/lilypond/}
-
-If this directory does not exist, make it.  Git master should go
-in @file{~/lilypond/lilypond-git/} but make sure you enable:
-
-@example
-git config core.filemode false
-@end example
+Initial setup: you still need @code{$LILYPOND_GIT} and
+@code{$LILYPOND_WEB_MEDIA_GIT}.
 
-If you have created any files in @file{~graham/lilypond/} then
-please run:
+Once that's done, create:
 
 @example
-chgrp lilypond ~graham/lilypond/ -R
-chmod 775 ~graham/lilypond/ -R
+mkdir -p $HOME/lilypond/
+mkdir -p $HOME/lilypond/bin/
+mkdir -p $HOME/lilypond/cron/
+mkdir -p $HOME/lilypond/trusted-scripts/
 @end example
 
-To reduce the CPU burden on the shared host (as well as some
-security concerns), the @file{Documentation/pictures/} and
-@file{Documentation/ly-examples/} directories are
-@strong{not} compiled.  You need to upload them, and if they ever
-change, a user in the @code{lilypond} group must upload them to
-@file{~graham/lilypond/media} on the host.
-
-Upload latest pictures/ and ly-examples/ (local script):
+The add these files to @file{$HOME/lilypond/bin/}:
 
-@warning{You may need to change a number of items in the below
-script.}
-
-@verbatim
-### upload-lily-web-media.sh
-#!/bin/sh
-BUILD_DIR=$HOME/src/build-lilypond
-
-PICS=$BUILD_DIR/Documentation/pictures/out-www/
-EXAMPLES=$BUILD_DIR/Documentation/ly-examples/out-www/
-
-cd $BUILD_DIR
-rsync -a $PICS graham@lilypond.org:lilypond/media/pictures
-rsync -a $EXAMPLES graham@lilypond.org:lilypond/media/ly-examples
-@end verbatim
-
-
-
-@subheading Normal maintenance
-
-Get latest source code:
+Update git repositories:
 
+@smallexample
 @verbatim
 ### update-git.sh
 #!/bin/sh
-cd $HOME/lilypond/lilypond-git
+cd $LILYPOND_GIT
+git fetch origin
+git merge origin/master
+cd $LILYPOND_WEB_MEDIA_GIT
 git fetch origin
 git merge origin/master
 @end verbatim
+@end smallexample
 
 Check for any updates to trusted scripts / files:
+
 @smallexample
 @verbatim
 ### check-git.sh
 #!/bin/sh
-GIT=$HOME/lilypond/lilypond-git
 DEST=$HOME/lilypond/trusted-scripts
-diff -u $DEST/website.make $GIT/make/website.make
-diff -u $DEST/lilypond-texi2html.init $GIT/Documentation/lilypond-texi2html.init
-diff -u $DEST/extract_texi_filenames.py $GIT/scripts/build/extract_texi_filenames.py
-diff -u $DEST/create-version-itexi.py $GIT/scripts/build/create-version-itexi.py
-diff -u $DEST/create-weblinks-itexi.py $GIT/scripts/build/create-weblinks-itexi.py
-diff -u $DEST/mass-link.py $GIT/scripts/build/mass-link.py
-diff -u $DEST/website_post.py $GIT/scripts/build/website_post.py
-diff -u $DEST/bib2texi.py $GIT/scripts/build/bib2texi.py
-diff -u $DEST/langdefs.py $GIT/python/langdefs.py
-diff -u $DEST/lilypond.org.htaccess $GIT/Documentation/web/server/lilypond.org.htaccess
-diff -u $DEST/website-dir.htaccess $GIT/Documentation/web/server/website-dir.htaccess
+diff -u $DEST/website.make \
+  $LILYPOND_GIT/make/website.make
+diff -u $DEST/lilypond-texi2html.init \
+  $LILYPOND_GIT/Documentation/lilypond-texi2html.init
+diff -u $DEST/extract_texi_filenames.py \
+  $LILYPOND_GIT/scripts/build/extract_texi_filenames.py
+diff -u $DEST/create-version-itexi.py \
+  $LILYPOND_GIT/scripts/build/create-version-itexi.py
+diff -u $DEST/create-weblinks-itexi.py \
+  $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py
+diff -u $DEST/mass-link.py \
+  $LILYPOND_GIT/scripts/build/mass-link.py
+diff -u $DEST/website_post.py \
+  $LILYPOND_GIT/scripts/build/website_post.py
+diff -u $DEST/bib2texi.py \
+  $LILYPOND_GIT/scripts/build/bib2texi.py
+diff -u $DEST/langdefs.py \
+  $LILYPOND_GIT/python/langdefs.py
+diff -u $DEST/lilypond.org.htaccess \
+  $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess
+diff -u $DEST/website-dir.htaccess \
+  $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess
 @end verbatim
+@end smallexample
 
 If the changes look ok, make them trusted:
 
+@smallexample
 @verbatim
 ### copy-from-git.sh
 #!/bin/sh
-GIT=$HOME/lilypond/lilypond-git
 DEST=$HOME/lilypond/trusted-scripts
-cp $GIT/make/website.make $DEST/website.make
-cp $GIT/Documentation/lilypond-texi2html.init $DEST/lilypond-texi2html.init
-cp $GIT/scripts/build/extract_texi_filenames.py $DEST/extract_texi_filenames.py
-cp $GIT/scripts/build/create-version-itexi.py $DEST/create-version-itexi.py
-cp $GIT/scripts/build/create-weblinks-itexi.py $DEST/create-weblinks-itexi.py
-cp $GIT/scripts/build/mass-link.py $DEST/mass-link.py
-cp $GIT/scripts/build/website_post.py $DEST/website_post.py
-cp $GIT/scripts/build/bib2texi.py $DEST/bib2texi.py
-cp $GIT/python/langdefs.py $DEST/langdefs.py
-cp $GIT/Documentation/web/server/lilypond.org.htaccess $DEST/lilypond.org.htaccess
-cp $GIT/Documentation/web/server/website-dir.htaccess $DEST/website-dir.htaccess
+cp $LILYPOND_GIT/make/website.make \
+  $DEST/website.make
+cp $LILYPOND_GIT/Documentation/lilypond-texi2html.init \
+  $DEST/lilypond-texi2html.init
+cp $LILYPOND_GIT/scripts/build/extract_texi_filenames.py \
+  $DEST/extract_texi_filenames.py
+cp $LILYPOND_GIT/scripts/build/create-version-itexi.py \
+  $DEST/create-version-itexi.py
+cp $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py \
+  $DEST/create-weblinks-itexi.py
+cp $LILYPOND_GIT/scripts/build/mass-link.py \
+  $DEST/mass-link.py
+cp $LILYPOND_GIT/scripts/build/website_post.py \
+  $DEST/website_post.py
+cp $LILYPOND_GIT/scripts/build/bib2texi.py \
+  $DEST/bib2texi.py
+cp $LILYPOND_GIT/python/langdefs.py \
+  $DEST/langdefs.py
+cp $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess \
+  $DEST/lilypond.org.htaccess
+cp $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess \
+  $DEST/website-dir.htaccess
 @end verbatim
 @end smallexample
 
 Build the website:
 
+@smallexample
 @verbatim
 ### make-website.sh
 #!/bin/sh
@@ -267,19 +198,84 @@ mkdir -p $BUILD
 cd $BUILD
 cp $HOME/lilypond/trusted-scripts/website.make .
 
-make -f website.make WEBSITE_ONLY_BUILD=1 -B website
+make -f website.make WEBSITE_ONLY_BUILD=1 website
 rsync -raO $BUILD/out-website/website/ $DEST/website/
 cp $BUILD/out-website/pictures $DEST
 cp $BUILD/out-website/.htaccess $DEST
 @end verbatim
+@end smallexample
 
-Cronjob to automate the trusted portions:
+Then in the @file{cronjob/} directory, put the cronjob to automate
+the trusted portions:
 
+@warning{@code{cron} will not inherit environment variables from
+your main setup, so you must re-define any variables inside your
+@code{crontab}.}
+
+@smallexample
 @verbatim
 # website-rebuild.cron
+LILYPOND_GIT=   ... fill this in
+LILYPOND_WEB_MEDIA_GIT=   ... fill this in
+
 11 * * * * $HOME/lilypond/trusted-scripts/update-git.sh >/dev/null 2>&1
 22 * * * * $HOME/lilypond/trusted-scripts/make-website.sh >/dev/null 2>&1
 @end verbatim
+@end smallexample
+
+As the final stage of the setup, run your @code{copy-from-git.sh}
+script, assuming that you trust the current state of scripts in
+lilypond git.
+
+@subsubheading Normal maintenance
+
+When there is a change to the build scripts and/or website
+makefile, log in to the server (or your own home machine if you're
+testing this there), and do
+
+@example
+update-git.sh
+check-git.sh
+@end example
+
+After reviewing the changes carefully, you can update the trusted
+scripts with @code{copy-from-git.sh}.
+
+
+@subsubheading Building the website (exactly as on the server)
+
+Run @code{make-website.sh}; the final version ends up in
+@file{$HOME/web/}.
+
+On the actual server, the website is generated hourly by user
+@code{graham} the host @code{lilypond.org}.  You can set up the
+cronjob by doing:
+
+@example
+crontab $HOME/lilypond/website-rebuild.cron
+@end example
+
+
+@subheading Initial setup for new users on actual serve
+
+You should symlink your own @file{~/lilypond/} to
+@file{~graham/lilypond/}
+
+If this directory does not exist, make it.  Git master should go
+in @file{~/lilypond/lilypond-git/} but make sure you enable:
+
+@example
+git config core.filemode false
+@end example
+
+If you have created any files in @file{~graham/lilypond/} then
+please run:
+
+@example
+chgrp lilypond ~graham/lilypond/ -R
+chmod 775 ~graham/lilypond/ -R
+@end example
+
 
 
 @subsubheading Additional information