From 1a06fafe6e9b5b6d0a131adef382ff80ed2df56c Mon Sep 17 00:00:00 2001 From: Graham Percival Date: Sat, 4 Feb 2012 21:19:26 +0000 Subject: [PATCH] Doc: CG: explain website+LILYPOND_WEB_MEDIA_GIT This work was sponsored by David Kastrup. --- Documentation/contributor/source-code.itexi | 29 ++ Documentation/contributor/website-work.itexi | 304 +++++++++---------- 2 files changed, 179 insertions(+), 154 deletions(-) diff --git a/Documentation/contributor/source-code.itexi b/Documentation/contributor/source-code.itexi index 75363f4d40..9322e766da 100644 --- a/Documentation/contributor/source-code.itexi +++ b/Documentation/contributor/source-code.itexi @@ -112,6 +112,7 @@ multiple projects concurrently. @menu * Setting up:: * Git for the impatient:: +* Other repositories:: * Downloading remote branches:: @end menu @@ -522,9 +523,35 @@ your branch (safe)}, but replace the @code{-d} on the final line with a @code{-D}. +@node Other repositories +@subsection Other repositories + +There is a separate repository for general administrative scripts, +as well as pictures and media files for the website. People +interested in working on the website should download this +repository, and set their @code{$LILYPOND_WEB_MEDIA_GIT} +environment variable to point to that repository. + +@example +@uref{https://github.com/gperciva/lilypond-extra} +@end example + +To configure an environment variable in bash (the default for most +Linux distributions), + +@example +export LILYPOND_WEB_MEDIA_GIT=$HOME/dir/of/lilypond-extra/ +@end example + + +There are a few other repositories floating around, which will +hopefully be documented in the near future. + + @node Downloading remote branches @subsection Downloading remote branches +@warning{contains obsolete + misleading info} @menu * Organization of remote branches:: @@ -589,6 +616,8 @@ only be used as a last resort. @node Downloading individual branches @unnumberedsubsubsec Downloading individual branches +@warning{obsolete, should be deleted!} + Once you have initialized an empty Git repository on your system (see @ref{Initializing a repository}), you can download a remote diff --git a/Documentation/contributor/website-work.itexi b/Documentation/contributor/website-work.itexi index aaced80cff..33b13464b6 100644 --- a/Documentation/contributor/website-work.itexi +++ b/Documentation/contributor/website-work.itexi @@ -58,206 +58,137 @@ existing website texinfo files. @node Uploading and security @section Uploading and security -@subheading Building the website from scratch for local checking - -Initial setup: - -Create directories: - -@example -mkdir $HOME/lilypond/ -mkdir $HOME/lilypond/media/ -mkdir $HOME/lilypond/trusted-scripts/ -@end example +@subheading Overall idea To reduce the CPU burden on the shared host (as well as some -security concerns), the 'Documentation/pictures/' and -'Documentation/ly-examples/' directories are **not** compiled. -We will do this ourselves right now. - -Go to your lilypond build directory. make doc. +security concerns), we do not compile all of LilyPond. The +website build process runs @code{texi2html}, but all media files +(be they graphical @code{lilypond} output, photos of people, or +pdfs) are copied from the @code{$LILYPOND_WEB_MEDIA_GIT} +repository. -Set up some variables (you'll only do this once:) +All scripts and makefiles used for the website build are run from +a @qq{trusted} copy. Any modification to those files in git needs +a human to review the changes (after they have been made in git) +before they are used on the server. -@example -BUILD_DIR=$HOME/lilypond-git -PICS=$BUILD_DIR/Documentation/pictures/ -EXAMPLES=$BUILD_DIR/Documentation/ly-examples/ -@end example -Copy files over: +@subheading Building the website (quick local) -@example -cp -r $PICS $HOME/lilypond/media/pictures -cp -r $EXAMPLES $HOME/lilypond/media/ly-examples -@end example +Initial setup: make sure that you have the environment variables +@code{$LILYPOND_GIT} and @code{$LILYPOND_WEB_MEDIA_GIT} set up +correctly. For more information, see @ref{Other repositories}. -Get the scripts you need. First define these variables: +Once that is done, @example -GIT=$HOME/lilypond-git -DEST=$HOME/lilypond/trusted-scripts -@end example - -Then do the copying: - -@smallexample -cp $GIT/make/website.make $DEST/website.make -cp $GIT/Documentation/lilypond-texi2html.init $DEST/lilypond-texi2html.init -cp $GIT/scripts/build/extract_texi_filenames.py $DEST/extract_texi_filenames.py -cp $GIT/scripts/build/create-version-itexi.py $DEST/create-version-itexi.py -cp $GIT/scripts/build/create-weblinks-itexi.py $DEST/create-weblinks-itexi.py -cp $GIT/scripts/build/mass-link.py $DEST/mass-link.py -cp $GIT/scripts/build/website_post.py $DEST/website_post.py -cp $GIT/scripts/build/bib2texi.py $DEST/bib2texi.py -cp $GIT/Documentation/web/server/lilypond.org.htaccess $DEST/lilypond.org.htaccess -cp $GIT/Documentation/web/server/website-dir.htaccess $DEST/website-dir.htaccess -@end smallexample - -For a complete build you will need a copy of @code{lilypond-extra} git repository. -You can checkout a fresh copy easily: - -@example -export LILYPOND_WEB_MEDIA_GIT=$HOME/lilypond-extra -git clone git://github.com/gperciva/lilypond-extra.git $LILYPOND_WEB_MEDIA_GIT -@end example - -Just note that the example above expects a bash environment. If you are using another shell -you might need to use a different keyword, other than @code{export}. - -Delete your build directory (or maybe just rename your build -directory to build-old). - -@example -cd $HOME/lilypond -@end example - -Run - -@example -make -f ../lilypond-git/make/website.make WEBSITE_ONLY_BUILD=1 \ - TOP_SRC_DIR=$HOME/lilypond-git/ \ - PYTHONPATH=$HOME/lilypond-git/python \ - TEXI2HTML_PROGRAM=texi2html \ - website +cd $LILYPOND_GIT/build/ +make website @end example -The website should be at: +The website is in @file{out-website/website/index.html}. -@example -$HOME/lilypond/out-website/website/index.html -@end example -@subheading Building the online website +@subheading Building the website (exactly as on the server) -The website is generated hourly by user @code{graham} the host -@code{lilypond.org}. For security reasons, we do not use the -makefiles and scripts directly from git; copies of the relevant -scripts are examined and copied to -@code{~graham/lilypond/trusted-scripts/} +@subsubheading Setting up (exactly as on the server) -@subheading Initial setup - -You should symlink your own @file{~/lilypond/} to -@file{~graham/lilypond/} - -If this directory does not exist, make it. Git master should go -in @file{~/lilypond/lilypond-git/} but make sure you enable: - -@example -git config core.filemode false -@end example +Initial setup: you still need @code{$LILYPOND_GIT} and +@code{$LILYPOND_WEB_MEDIA_GIT}. -If you have created any files in @file{~graham/lilypond/} then -please run: +Once that's done, create: @example -chgrp lilypond ~graham/lilypond/ -R -chmod 775 ~graham/lilypond/ -R +mkdir -p $HOME/lilypond/ +mkdir -p $HOME/lilypond/bin/ +mkdir -p $HOME/lilypond/cron/ +mkdir -p $HOME/lilypond/trusted-scripts/ @end example -To reduce the CPU burden on the shared host (as well as some -security concerns), the @file{Documentation/pictures/} and -@file{Documentation/ly-examples/} directories are -@strong{not} compiled. You need to upload them, and if they ever -change, a user in the @code{lilypond} group must upload them to -@file{~graham/lilypond/media} on the host. - -Upload latest pictures/ and ly-examples/ (local script): +The add these files to @file{$HOME/lilypond/bin/}: -@warning{You may need to change a number of items in the below -script.} - -@verbatim -### upload-lily-web-media.sh -#!/bin/sh -BUILD_DIR=$HOME/src/build-lilypond - -PICS=$BUILD_DIR/Documentation/pictures/out-www/ -EXAMPLES=$BUILD_DIR/Documentation/ly-examples/out-www/ - -cd $BUILD_DIR -rsync -a $PICS graham@lilypond.org:lilypond/media/pictures -rsync -a $EXAMPLES graham@lilypond.org:lilypond/media/ly-examples -@end verbatim - - - -@subheading Normal maintenance - -Get latest source code: +Update git repositories: +@smallexample @verbatim ### update-git.sh #!/bin/sh -cd $HOME/lilypond/lilypond-git +cd $LILYPOND_GIT +git fetch origin +git merge origin/master +cd $LILYPOND_WEB_MEDIA_GIT git fetch origin git merge origin/master @end verbatim +@end smallexample Check for any updates to trusted scripts / files: + @smallexample @verbatim ### check-git.sh #!/bin/sh -GIT=$HOME/lilypond/lilypond-git DEST=$HOME/lilypond/trusted-scripts -diff -u $DEST/website.make $GIT/make/website.make -diff -u $DEST/lilypond-texi2html.init $GIT/Documentation/lilypond-texi2html.init -diff -u $DEST/extract_texi_filenames.py $GIT/scripts/build/extract_texi_filenames.py -diff -u $DEST/create-version-itexi.py $GIT/scripts/build/create-version-itexi.py -diff -u $DEST/create-weblinks-itexi.py $GIT/scripts/build/create-weblinks-itexi.py -diff -u $DEST/mass-link.py $GIT/scripts/build/mass-link.py -diff -u $DEST/website_post.py $GIT/scripts/build/website_post.py -diff -u $DEST/bib2texi.py $GIT/scripts/build/bib2texi.py -diff -u $DEST/langdefs.py $GIT/python/langdefs.py -diff -u $DEST/lilypond.org.htaccess $GIT/Documentation/web/server/lilypond.org.htaccess -diff -u $DEST/website-dir.htaccess $GIT/Documentation/web/server/website-dir.htaccess +diff -u $DEST/website.make \ + $LILYPOND_GIT/make/website.make +diff -u $DEST/lilypond-texi2html.init \ + $LILYPOND_GIT/Documentation/lilypond-texi2html.init +diff -u $DEST/extract_texi_filenames.py \ + $LILYPOND_GIT/scripts/build/extract_texi_filenames.py +diff -u $DEST/create-version-itexi.py \ + $LILYPOND_GIT/scripts/build/create-version-itexi.py +diff -u $DEST/create-weblinks-itexi.py \ + $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py +diff -u $DEST/mass-link.py \ + $LILYPOND_GIT/scripts/build/mass-link.py +diff -u $DEST/website_post.py \ + $LILYPOND_GIT/scripts/build/website_post.py +diff -u $DEST/bib2texi.py \ + $LILYPOND_GIT/scripts/build/bib2texi.py +diff -u $DEST/langdefs.py \ + $LILYPOND_GIT/python/langdefs.py +diff -u $DEST/lilypond.org.htaccess \ + $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess +diff -u $DEST/website-dir.htaccess \ + $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess @end verbatim +@end smallexample If the changes look ok, make them trusted: +@smallexample @verbatim ### copy-from-git.sh #!/bin/sh -GIT=$HOME/lilypond/lilypond-git DEST=$HOME/lilypond/trusted-scripts -cp $GIT/make/website.make $DEST/website.make -cp $GIT/Documentation/lilypond-texi2html.init $DEST/lilypond-texi2html.init -cp $GIT/scripts/build/extract_texi_filenames.py $DEST/extract_texi_filenames.py -cp $GIT/scripts/build/create-version-itexi.py $DEST/create-version-itexi.py -cp $GIT/scripts/build/create-weblinks-itexi.py $DEST/create-weblinks-itexi.py -cp $GIT/scripts/build/mass-link.py $DEST/mass-link.py -cp $GIT/scripts/build/website_post.py $DEST/website_post.py -cp $GIT/scripts/build/bib2texi.py $DEST/bib2texi.py -cp $GIT/python/langdefs.py $DEST/langdefs.py -cp $GIT/Documentation/web/server/lilypond.org.htaccess $DEST/lilypond.org.htaccess -cp $GIT/Documentation/web/server/website-dir.htaccess $DEST/website-dir.htaccess +cp $LILYPOND_GIT/make/website.make \ + $DEST/website.make +cp $LILYPOND_GIT/Documentation/lilypond-texi2html.init \ + $DEST/lilypond-texi2html.init +cp $LILYPOND_GIT/scripts/build/extract_texi_filenames.py \ + $DEST/extract_texi_filenames.py +cp $LILYPOND_GIT/scripts/build/create-version-itexi.py \ + $DEST/create-version-itexi.py +cp $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py \ + $DEST/create-weblinks-itexi.py +cp $LILYPOND_GIT/scripts/build/mass-link.py \ + $DEST/mass-link.py +cp $LILYPOND_GIT/scripts/build/website_post.py \ + $DEST/website_post.py +cp $LILYPOND_GIT/scripts/build/bib2texi.py \ + $DEST/bib2texi.py +cp $LILYPOND_GIT/python/langdefs.py \ + $DEST/langdefs.py +cp $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess \ + $DEST/lilypond.org.htaccess +cp $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess \ + $DEST/website-dir.htaccess @end verbatim @end smallexample Build the website: +@smallexample @verbatim ### make-website.sh #!/bin/sh @@ -267,19 +198,84 @@ mkdir -p $BUILD cd $BUILD cp $HOME/lilypond/trusted-scripts/website.make . -make -f website.make WEBSITE_ONLY_BUILD=1 -B website +make -f website.make WEBSITE_ONLY_BUILD=1 website rsync -raO $BUILD/out-website/website/ $DEST/website/ cp $BUILD/out-website/pictures $DEST cp $BUILD/out-website/.htaccess $DEST @end verbatim +@end smallexample -Cronjob to automate the trusted portions: +Then in the @file{cronjob/} directory, put the cronjob to automate +the trusted portions: +@warning{@code{cron} will not inherit environment variables from +your main setup, so you must re-define any variables inside your +@code{crontab}.} + +@smallexample @verbatim # website-rebuild.cron +LILYPOND_GIT= ... fill this in +LILYPOND_WEB_MEDIA_GIT= ... fill this in + 11 * * * * $HOME/lilypond/trusted-scripts/update-git.sh >/dev/null 2>&1 22 * * * * $HOME/lilypond/trusted-scripts/make-website.sh >/dev/null 2>&1 @end verbatim +@end smallexample + +As the final stage of the setup, run your @code{copy-from-git.sh} +script, assuming that you trust the current state of scripts in +lilypond git. + +@subsubheading Normal maintenance + +When there is a change to the build scripts and/or website +makefile, log in to the server (or your own home machine if you're +testing this there), and do + +@example +update-git.sh +check-git.sh +@end example + +After reviewing the changes carefully, you can update the trusted +scripts with @code{copy-from-git.sh}. + + +@subsubheading Building the website (exactly as on the server) + +Run @code{make-website.sh}; the final version ends up in +@file{$HOME/web/}. + +On the actual server, the website is generated hourly by user +@code{graham} the host @code{lilypond.org}. You can set up the +cronjob by doing: + +@example +crontab $HOME/lilypond/website-rebuild.cron +@end example + + +@subheading Initial setup for new users on actual serve + +You should symlink your own @file{~/lilypond/} to +@file{~graham/lilypond/} + +If this directory does not exist, make it. Git master should go +in @file{~/lilypond/lilypond-git/} but make sure you enable: + +@example +git config core.filemode false +@end example + +If you have created any files in @file{~graham/lilypond/} then +please run: + +@example +chgrp lilypond ~graham/lilypond/ -R +chmod 775 ~graham/lilypond/ -R +@end example + @subsubheading Additional information -- 2.39.2