Mirroring LCG/gLite Software

Here are some instructions for one example of setting up a YUM-based mirror of the current gLite and CA software, circa May 2006. This is only interesting if you are already using yum to manage your operating system software. Note that CERN is moving from apt to yum with SLC 4:

http://linux.web.cern.ch/linux/updates/slc4.shtml
In order to unify software update mechanism accross all the supported platforms and also accross all Scientific Linux based distributions, the default update system on SLC4 has been switched to yum as of 13.02.2006. The APT based update system remains functional but is deprecated and may stop working in the future.

Setting up a mirror:

set aside a largish directory for use; in the example Makefile it is called /ks/mirror/
give ownership of this directory to some unprivileged user; in this example we have an unprivileged user named 'mirror'
Put the Makefile in the top level of your mirror; thus in this example: /ks/mirror/Makefile
Invoke the attached script, which simply runs the necessary make commands. The first time you run it will take a while to populate the directories.
It would be nice to also verify the signatures of the RPMs in the script; unfortunately there is no policy on RPM-signing in LCG/gLite software.

For an example of setting up an apt-based repository see the wiki entry at goc.grid.sinica.edu.tw.

Notes about the Makefile

You will see that in some parts of the makefile I run 'yum-arch' and/or 'createrepo' -- this is because traditionally CERN did not support the newer version of yum that comes with SL 4. Indeed, for the recently-released gLite 3.0 trees they are currently not even creating the traditional yum headers of SL 3. Note that the 'createrepo' SL4 RPM can be installed onto an SL 3 system if your mirror is running SL3.

I also use wget extensively - this is somewhat historical from mirroring LCG software before an ftp-based software tree was available. Note that with both linuxsoft.cern.ch and glitesoft.cern.ch (an alias for linuxsoft) ftp is also allowed. I will post soon some example uses of 'lftp' that provide the same functionality as wget (I want to test if there is any difference in performance). The short-comings of both wget and lftp are that they do not allow you (AFAIK) to return a listing of what they would do, without actually doing it. Even though linuxsoft allows rsync too, it is not currently configured to push out anything but SLC4 (to see, do: rsync rsync://glitesoft.cern.ch )

Once a day I update my local mirror via a crontab entry using the linked script. The output is mailed to me, and I can see if there are any new changes.

Two-stage Mirroring

For maximum flexibility and control we have implemented a two-stage mirror - that is, we automatically mirror the software to the first stage mirror. From the first-stage mirror we push changes manually using the Makefile to the second-stage mirror. The advantage is that we control tightly what ends up on the second-stage mirror. The disadvantage is needing sufficient diskspace on 2 nodes to do this.

Both the script and the Makefile make use of 2-stage mirroring. Thus you will need to edit both the Makefile and the update script to remove the sections that use a second stage mirror. All that is reallly needed are the sections get-lcg_ca, get-glite_3_x and if you are still also using lcg 2.7, then get-lcg_2_7_x_sl3. All other references to test_* and do_* can be removed.

Example yum.conf stanza

Here is an example /etc/yum.conf file that contains references to the LCG/gLite repositories glite-3.0, glite-3.0_ext, glite-3.0__updates and lcg-CA. I have had some problems with the so-called glite externals repository (clashes with mysql), so locally I exclude a few of the troublesome RPMs from that repository.