small updates, new URL for apt-get

[cran2deb.git] / papers / useR2009 / cran2deb.tex

%% add 'handout' option for handouts, and pgfpages for 2-on-1
\documentclass[smaller,compress]{beamer}   
%\usepackage{pgfpages}
%\pgfpagesuselayout{2 on 1}[letterpaper,border shrink=5mm]
%\pgfpagesuselayout{4 on 1}[letterpaper,border shrink=5mm]
%\pgfpagesuselayout{2 on 1}[a4,border shrink=5mm]

\include{setup}  %% has all definitions etc

%\title[cran2deb: Automated CRAN to Debian packages generation]{cran2deb: A
%  system to automatically provide 1700+ CRAN packages as Debian binaries} 
\title[cran2deb: CRAN to Debian packages]{cran2deb:  A fully automated CRAN to \\
  Debian package generation system} 
\subtitle{\textsl{UseR! 2009 Presentation}}
\subject{UseR! 2009 Presentation}
\author[Charles Blundell \and Dirk Eddelbuettel]{Charles Blundell\inst{1} \and Dirk Eddelbuettel\inst{2}}
\institute[Gatsby \and Debian]{\inst{1}Gatsby Computational Neuroscience Unit
  \\ University College London, UK \and \inst{2}Debian and R Projects \\ Chicago,
IL, USA}
%\date[UseR! 2009]{Presentation at UseR! 2009 \\ Rennnes, France \\ July 2009}
\date[UseR! 2009 Presentation]{Universit\'{e} Rennes II, Agrocampus Ouest \\ Laboratoire de
  Math\'{e}matiques Appliqu\'{e}es \\ 8-10 July 2009}


\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\begin{frame}
  \frametitle{Overview}
  \tableofcontents
\end{frame}

\section[Why]{Why: Background and Motivation}
\begin{frame}
  \frametitle{About R -- and its repositories}
  \framesubtitle{An open statistical language / environment -- with lots of
    excellent code contributions}

  A few key facts that are non-controversial at a \textsl{useR!} conference:
  \begin{itemize} 
  \item \R\ is now a standard for statistical applications and research
  \item \textit{``Success has many fathers''}: several key drivers can
    be identified as to why \R has done so well 
  \item We would like to stress \textsl{repositories} and available packages here:
    CRAN, as well as BioConductor and Omegahat.
  \item CRAN has been one of the drivers: an open yet rigorously QA'ed
    repository which has experienced tremendous growth
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{CRAN Packages} %% NB Or shall we merge this with the preceding slide?
  \framesubtitle{Exponential Growth}

  \begin{columns}
    \begin{column}{3in}
      \begin{figure}
        \includegraphics[height=6cm,transparent]{figures/Packages}

        \begin{scriptsize}
          Source: Fox (2008, 2009), our calculations
        \end{scriptsize}
      \end{figure}
    \end{column}
    \begin{column}{2in}
      \begin{itemize} 
        \item CRAN archive network growing by 40\% p.a., now at around 1750 packages

        \item John Fox provided this chart in an invited lecture at the last
        \emph{useR!} meetings.
      \end{itemize}
    \end{column}
    \begin{column}{0.25in}
      \phantom{XX}
    \end{column}
  \end{columns}  
\end{frame}

\begin{frame}
  \frametitle{Debian and Ubuntu} % NB Maybe skip this slide?
  \framesubtitle{Open Linux distributions}

  A few key points:
  \begin{itemize} 
  \item Debian is \textsl{the} community-driven Linux distribution where
    numerous volunteers provide over twenty-thousand packages for around
    a dozen architectures.
  \item Packages and package management ``just work'': with arguably the most
    advanced and robust package management system, and a tremendous
    build and test infrastructure.
  \item Ubuntu has taken Debian, added a fair amount of spit and polish, as
    well as regular bi-annual releases, and has rapidly gained mind- and
    well as market-share as the Linux distribution to beat.
  \item We also note that the CRAN backend is implemented on Debian.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Why build Debian R packages?}
  \framesubtitle{Combining R and Debian}
  Bates, Eddelbuettel and Gebhard (UseR! 2004) listed a number of reason
  that still hold:
  \begin{itemize} 
  \item \textbf{Dependencies} are resolved automatically: \textsl{it just
      works}
  \item \textbf{Convenience} of installing binary packages via
    \texttt{apt-get} %is
    %easier than building from source
  \item \textbf{Quality control} as build daemons, automated rebuilds,
    porting, ... all ensure that everything is pretty much buildable all the
    time
  \item \textbf{Scalability} as building one binary package and scripting
    installation on a cluster beats doing lots of manual installations
  \item \textbf{Common platform} as Debian forms the base for Ubuntu and
    several other derivative or single-focus distributions
  \item \textbf{Different architectures} ranging from small arm or MIPS based
    systems to amd64, sparc64, hppa or even s390 mainframes
  \item \textbf{Audience} given the reach of Debian and Ubuntu, large number
    of users can be reached with little effort
  \end{itemize}

\end{frame}

%\section{What is behind it?}
%\begin{frame}
%  \frametitle{So what is a Debian package?} % NB Maybe skip this?
%  \framesubtitle{And how do I build it?}
%
%  Building a Debian package is similar to using \texttt{R
%    CMD binary} etc:
%  \begin{itemize} 
%  \item Reads meta-information is read from the files in the debian/ directory
%    \begin{itemize} 
%    \item debian/control (similar to R's DESCRIPTION) lists names,
%      maintainers, build- and run-time dependencies
%    \item debian/copyright lists all author, license holders and copyright
%      statements 
%    \item debian/changelog provides current and past version numbers with a
%      list of  all changes in chronological fashion
%    \item debian/rules is a Makefile containing all steps to configure,
%      build, install, package-create and clean
%    \end{itemize}
%  \item Employs a number of external tools scripts and tools, can be used
%    interactively or in batch mode in chroot'ed 'clean rooms'
%  \end{itemize}
%\end{frame}


\section[How]{How: Key aspects of the approach and implementation}
\begin{frame}
  \frametitle{Comparing two approaches}
  \framesubtitle{What have we learned?}

  Eddelbuettel, Vernazobres, Gebhard and M\"{o}ller (UseR 2007) implemented a
  system which provides a basis for comparison:

  \MedSkip

  \begin{columns}
    \begin{column}{2in}
      \textsl{Then}
      \begin{itemize}
      \item Top-down approach 
      \item Monolithic and large Perl program 
      \item Meta-information encode directly as Perl hashes in program
      \item Re-implementing chunks of what \R does in parsing archives
      \item Not very robust
      \end{itemize}
    \end{column}      

    \begin{column}{2in}
      \textsl{Now}
      \begin{itemize}
      \item Bottom-up approach
      \item Collection of \R and shell scripts, also lots of SQL
      \item Re-using \R internal infrastructure as much as possible
      \item Influenced by %Eddelbuettel's
        \href{http://dirk.eddelbuettel.com/cranberries/}{CRANberries} and its
        200 lines of \R code to monitor and summarize CRAN changes
      \end{itemize}
    \end{column}      
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Technology Overview: Big Picture}
  \framesubtitle{Key components}

  Our cran2deb system is implemented as a collection of small tools:
  \begin{itemize}
  \item cran2deb itself is a wrapper script calling out to about twenty other
    'worker' scripts implementing the principal commands
    \begin{itemize}
    \item 'worker' scripts are written in \R (for littler), Korn/Bash shell,
      and in the Plan9 shell rc
    \item these scripts are small: the largest is 4 kb and only seven
      are larger than 1 kb
    \item this is recursive: 'help' is one of these scripts scanning for
      doc-strings in the other scripts
    \end{itemize}
  \item cran2deb is also an R package that is being called by some of the R
    scripts; the R package has just over 1500 lines of code, and it calls out
    to R functionality from package utils and tools.
  \end{itemize}
\end{frame}  

\begin{frame}
  \frametitle{Technology Overview} 
  \framesubtitle{A walk through: some details}

  What does cran2deb do:
  \begin{itemize}
  \item pulls new meta-data from CRAN via \texttt{available.packages()}
  \item detects new (or changed) packages and builds each one via:
    \begin{itemize}
    \item map declared \R dependencies onto cran2deb packages
    \item map free-form SystemRequirements onto Debian packages
      \begin{itemize}
      \item Rules for this shared among packages---many packages ``just work''.
      \end{itemize}
    \item add any undeclared dependencies (this applies to just 36 packages
      and often entails only loading, say, MASS).
    \item build each package in its own isolated, clean, fresh, up to date
      build environment via pbuilder: this looks like a fresh install of
      Debian and ensures correctness of dependencies. 
    \end{itemize}
  \item checks package quality via Debian's lintian.
  \end{itemize}
\end{frame}  

\begin{frame}
  \frametitle{Technology Overview} 
  \framesubtitle{A walk through: some more details}

  What does cran2deb do (cont.):
  \begin{itemize}
  \item uses RSQLite backend for cran2deb state: everything from package
    meta-information, blacklist of bad packages, to build logs. 
  \item checks for a free license of a package before its built:
    \begin{itemize}
        \item initially: handcrafted regular expressions to match
                licenses.
        \item some packages ignore ``Writing R extensions'' guidelines
          concerning the License: field: how many ways to write GPL? 
            \begin{itemize}
                \item initialised vs. its expansion (GPL vs. GNU general public license)
                \item license vs. licence
                \item see \texttt{http://www.gnu.org/GPL}
                \item (v, version) (2.0, 2) or (higher, later, newer, greater, above)
                \item typos of the above
                \item file LICENSE: contents reformatted in arbitrary ways
            \end{itemize}
        \item now: strip white space and perform other harmless transforms
                and match SHA1 checksums to determine license;  likewise for contents of LICENSE
                file.
    \end{itemize}
  \end{itemize}
\end{frame}  

\begin{frame}
  \frametitle{Technology Overview} 
  \framesubtitle{Continued}

  Re-use, re-duce, re-cycle:

  \begin{itemize}
  \item \R's infrastructure is used to obtain the \R view of the world:
    what packages and where, first approximation to dependencies.
  \item All this uses the Debian build infrastructure, notably the
    pbuilder chroot environment and the package management system
  \item cran2deb sets the build environment up by invoking the proper Debian
    scripts 
  \item the `production line' of packages is fully automated via cron and report status
    summaries by email
  \item per-package patches are allowed (currently eleven packages have
    mostly trivial patches)
  \item source code is available via the r-forge subversion repository and archive
  \end{itemize}

\end{frame}

\section[Status]{Status: Where are we now?}

\begin{frame}
  \frametitle{Building 1700+ package}
  \framesubtitle{Summary from a package views}

  It's easy: basically \textsl{everything} builds and is available as a
  Debian package (complete with full dependencies) --- apart from:

  \begin{itemize}
  \item 17 packages that are \textsl{not free enough}:\footnote{Generally these
do not allow commercial use, modification and/or distribution with the
exception of ConvCalendar which gives no modification or distribution rights.}
    mclust, mclust02, ConvCalendar, SDDA, conf.design, isa2, optmatch,
    rankreg, realized, rngwell19937, tnet, spatialkernel, Bhat, PTAk,
    PredictiveRegression, RLadyBug, mapproj 
  \item 1 package that is obsolete: xgobi
  \item 2 package that break building packages via cran2deb:\footnote{They
      take down the cronjob; we are stumped as to why.} dprep, EngrExpt
  \item 1 package that is not built for 'other' reasons:\footnote{It contains
      binary code.} sabreR
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Building 1700+ package}
  \framesubtitle{Continued}

  \begin{itemize}
  \item 47 packages that have \textsl{unsatisfied
      dependencies}:\footnote{Some require other commercial software, some
      require software we classified\newline as non-free, some require BioConductor packages.}
    ROracle, Rlsf, Rsge, CarbonEL, VhayuR, gputools, klaR, wgaim, svGUI,
    RScaLAPACK, caMassClass, Rcplex, ADaCGH, DAAGbio, GFMaps, GOSim,
    Metabonomic, classGraph, gcExplorer, logilasso, pcalg, celsius, multtest,
    hopach, GExMap, LMGene, PCS, SubpathwayMiner, gene2pathway, PhViD,
    SNPMaP, qdg, lsa, mpm, sisus, metaMA, clustTool, clustvarsel,
    SpectralGEM, bayesCGH, crosshybDetector  
  \item 8 package that (as of end of June) fail for unclassified reasons:
    IDPmisc, Rsymphony, SuppDists, aroma.apd, aroma.core, aroma.affymetrix, cmprskContin, mvgraph
  \end{itemize}

  \MedSkip
  \textsl{But everything else}---currently 1770 packages---builds and is
  available via \texttt{apt-get} and other package management frontends!
\end{frame}  

\begin{frame}
  \frametitle{Status and credits}
  \framesubtitle{Ready for wider deployment and testing}

  Who do we owe, and where is it at: 

  \begin{itemize}
  \item The ground-work was provided during Google Summer of Code (GSoC) 2008 under the
    umbrella of the Debian project. We thank Google for the GSoC support.
  \item Currently we are using a (small) Xen-instance on a server at WU Wien to host
    two Debian pbuilder chroots and an archive. We thank WU Wien/CRAN for
    hosting and cpu cycles.
  \item 1700+ packages for i386 and amd64 on Debian testing
  \item In daily use for the last few weeks!
  \end{itemize}

  \MedSkip
  So just add one of these URLs:\newline
  { \scriptsize

    \texttt{deb http://debian.cran.r-project.org/cran2deb/debian-i386 testing/}

    \texttt{deb http://debian.cran.r-project.org/cran2deb/debian-amd64 testing/}
  }
\end{frame}

\section[Next]{Next: Open Issues}
\begin{frame}
  \frametitle{Question to be addressed}
  \framesubtitle{For cran2deb to migrate out of beta testing}

  %Things that may need to be sorted out:
  \begin{itemize}
  \item \textbf{Licenses:}
    \begin{itemize}
    \item  What can or cannot be (re-)distributed by CRAN and its mirrors?
    \item What can or cannot be used (and/or modified) by all users?
    \end{itemize}
  \item \textbf{Externtal dependencies} % Remaining external dependencies: 
    \begin{itemize}
    \item BioConductor is the single largest source: BioBase, RGraphviz, etc
    \item Other external libraries or tools not in Debian 
    \item Commercial external dependencies: SGE, LSF, Oracle, Vhayu
    \end{itemize}
  \item \textbf{Scope} 
    \begin{itemize}
    \item Builds for other architectures?
    \item Builds for other Debian flavours such as Ubuntu?
    \item Builds of other repositories: BioConductor? R-Forge?
    \end{itemize}
  \end{itemize}
\end{frame}

\end{document}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: t
%%% End: