From: Michael Hanke Date: Fri, 9 Mar 2012 15:31:09 +0000 (+0100) Subject: Blog post on using FSL with Condor. X-Git-Url: https://git.donarmstrong.com/?a=commitdiff_plain;h=430b1ea2a637f988e1069e9b97b0e5888f89ab14;p=neurodebian.git Blog post on using FSL with Condor. --- diff --git a/sphinx/blog/2012/2012-03-09_parallelize_fsl_with_condor.rst b/sphinx/blog/2012/2012-03-09_parallelize_fsl_with_condor.rst new file mode 100644 index 0000000..a1550f6 --- /dev/null +++ b/sphinx/blog/2012/2012-03-09_parallelize_fsl_with_condor.rst @@ -0,0 +1,182 @@ +:date: 2012-03-09 18:00:00 +:tags: neuroscience, software, imaging, fsl, condor, distributed computing + +.. _chap_parallelize_fsl_with_condor: + +Parallelizing FSL -- without the pain +===================================== + +FSL_ is a very popular analysis suite for neuroimaging data. Unfortunately, +right out of the box FSL's routines do not make good use of today's multi-core +machines. Consequently, researchers using FSL on individual workstations have +difficulties harnessing the computing power of their equipment. + +.. _FSL: http://www.fmrib.ox.ac.uk/fsl/ + +However, it is not the case that FSL doesn't support modern computing +environments. It is actually one of very few packages that come with built-in +support for running computations on a cluster back-end. Unfortunately, it relies +on the GridEngine_ software (formerly Sun, now from Oracle) to be installed +locally, and configured properly. This is a task that is known to be +non-trivial and can cause serious headache in a neuroscientist. While FSL has +been ported to use other batch systems, such as Torque_, none of them is much +easier to deploy. + +.. _GridEngine: http://www.oracle.com/us/products/tools/oracle-grid-engine-075549.html +.. _Torque: http://www.adaptivecomputing.com/products/torque.php + +What would be helpful is a batch system that can be easily installed without +complicated configuration procedures. Preferably, such a system is powerful +enough to scale across multiple machines, whenever the demand for computing +power increases. At NeuroDebian_ we looked at a number of projects and found +Condor_ to be the most versatile tools, yet still straightforward to use in a +single-machine scenario. + +.. _NeuroDebian: http://neuro.debian.net +.. _Condor: http://research.cs.wisc.edu/condor/ + +Over the past two years we have been working on getting Condor packaged for +Debian. We are now very proud to announce that, a few days ago, Condor has been +accepted as an `official Debian package`_ (in Debian experimental for the time +being). NeuroDebian, as usual, `provides backports`_ of Condor for numerous +Debian and Ubuntu releases. + +.. _official Debian package: http://packages.debian.org/condor +.. _provides backports: http://neuro.debian.net/pkgs/condor.html + +With the availability of Condor packages, parallelizing FSL on a local +workstation is now possible by following a simple four-step procedure. + + +1. Configure your machine to use the NeuroDebian repository +----------------------------------------------------------- + +[ if you are already a NeuroDebian user you can skip this step ] + +Go to http://neuro.debian.net and follow the setup instructions given on the +front-page. This means selecting your Debian or Ubuntu release, select a package +repository mirror close to you, and run the little shell script snippet that is +presented to you. After having updated you package lists you are ready to go. + +If you are not running Debian or Ubuntu on your machine, you can also install +the `NeuroDebian virtual machine`_ which is able to utilize multiple CPU-core. +The remaining setup steps apply to VM users as well. + +.. _NeuroDebian virtual machine: http://neuro.debian.net/vm.html + + +2. Install FSL +-------------- + +[ if you already have the latest FSL package version, you can skip this too ] + +Open a terminal and install FSL being running the command:: + + % sudo apt-get install fsl + +Of course you can also use any graphical package manager front-end (e.g. +software center). The terminal commands are simply used for clarity and +simplicity. + +Check what version of FSL got installed. Any version from 4.1.9-2 onwards is +fine. If you got an older version, you either haven't succeeded doing step 1, +or you are running a version of Debian or Ubuntu that is too old, and you would +need to upgrade it first. + + +3. Install Condor +----------------- + +Install Condor with:: + + % sudo apt-get install condor + +Any Condor package version from 7.7.4-1 onwards comes with all necessary +features. During installation a few question will be asked. You should enable +the Debconf-based configuration and ask for a "Personal Condor" installation +(see screenshots). + +.. image:: pics/blog/condor_install1.png + +.. image:: pics/blog/condor_install2.png + + +Now you have a fully functional Condor installation that advertises one compute +slot per detected CPU-core on your machine. You can display the current status +with the ``condor_status`` command. On a dual-core machine this could look like +this:: + + % condor_status + + Name OpSys Arch State Activity LoadAv Mem ActvtyTime + + slot1@meiner LINUX INTEL Owner Idle 0.320 1972 0+00:00:04 + slot2@meiner LINUX INTEL Owner Idle 0.000 1972 0+00:00:05 + Total Owner Claimed Unclaimed Matched Preempting Backfill + + INTEL/LINUX 2 2 0 0 0 0 0 + + Total 2 2 0 0 0 0 0 + +By default, Condor will only run jobs when the machine is idle. If you want to +run jobs immediately when they are submitted you need to change the +configuration slightly. To do this run:: + + % sudo dpkg-reconfigure condor + +This will again ask a number of questions. Leave everything on the respective +defaults, but enable running jobs regardless of other machine activity (see +screen shot below). + +.. image:: pics/blog/condor_install3.png + + +4. Enjoy parallelized FSL +------------------------- + +Now you are ready to make use of Condor with FSL. Source the configuration +from ``/etc/fsl/fsl.sh`` as usual:: + + % . /etc/fsl/fsl.sh + +Whenever you want FSL to parallelize jobs through Condor, simply set the +:envvar:`FSLPARALLEL` environment variable to ``condor`` and run FSL as usual:: + + % export FSLPARALLEL=condor + +You can check whether the setup is working by running ``condor_q`` to see the +list of submitted/running jobs. Running the FEAT part of the FSL-FEEDS test +suite could look like this:: + + % condor_q + + -- Submitter: meiner : <127.0.0.1:52379> : meiner + ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD + 1.0 michael 3/9 14:47 0+00:00:07 R 0 0.7 zsh -c /usr/share/ + 2.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/ + 3.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster2_sentinel. + 4.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/ + 5.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster4_sentinel. + 6.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/ + 7.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster6_sentinel. + 8.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/ + 9.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster8_sentinel. + 10.0 michael 3/9 14:47 0+00:00:00 H 0 0.7 zsh -c /usr/share/ + 11.0 michael 3/9 14:47 0+00:00:01 R 0 0.0 cluster10_sentinel + + 11 jobs; 0 completed, 0 removed, 0 idle, 6 running, 5 held, 0 suspended + + +Done +---- + +There is obviously much more that could be said about other exciting features +of Condor, but we will leave this for another blog post in the future. As a +little teaser we could mention that the Debian Condor package comes with an +emulator of SGE's ``qsub``. Through this adaptor many software that can make use +of SGE can also be used with Condor, for example NiPyPE_... + +.. _NiPyPE: http://nipy.sourceforge.net/nipype/ + +If you are now keen on exploring the possibilities of Condor, take a look at the +manual that comes in the ``condor-doc`` package. diff --git a/sphinx/blog/2012/index.rst b/sphinx/blog/2012/index.rst new file mode 100644 index 0000000..0e33650 --- /dev/null +++ b/sphinx/blog/2012/index.rst @@ -0,0 +1,9 @@ +2011 +==== + +.. toctree:: + :maxdepth: 1 + :glob: + + 2012* + diff --git a/sphinx/blog/2012/pics b/sphinx/blog/2012/pics new file mode 120000 index 0000000..1bbd53b --- /dev/null +++ b/sphinx/blog/2012/pics @@ -0,0 +1 @@ +../../pics \ No newline at end of file diff --git a/sphinx/blog/index.rst b/sphinx/blog/index.rst index f545a0e..48c6262 100644 --- a/sphinx/blog/index.rst +++ b/sphinx/blog/index.rst @@ -24,5 +24,6 @@ Archive :maxdepth: 2 :glob: + 2012/index 2011/index 2010/index diff --git a/sphinx/pics/blog/condor_install1.png b/sphinx/pics/blog/condor_install1.png new file mode 100644 index 0000000..b9d78aa Binary files /dev/null and b/sphinx/pics/blog/condor_install1.png differ diff --git a/sphinx/pics/blog/condor_install2.png b/sphinx/pics/blog/condor_install2.png new file mode 100644 index 0000000..787348a Binary files /dev/null and b/sphinx/pics/blog/condor_install2.png differ diff --git a/sphinx/pics/blog/condor_install3.png b/sphinx/pics/blog/condor_install3.png new file mode 100644 index 0000000..ad91156 Binary files /dev/null and b/sphinx/pics/blog/condor_install3.png differ