# Migrating info.cern.ch to a multiple-host platform

## Abstract

The main World-Wide Web server at CERN had become too overloaded to provide an acceptable service. To solve this, Its services were split up onto several machines in the W3 Organization's name domain. Further, the http service was put on two machines, serving files stored with AFS, in the method pioneered by NCSA. Initial performance shows a marked improvement, and future expansion should be much easier.

## The Problem

The computer popularly known as `info.cern.ch` is a very well-known machine.

In the early days of the World-Wide-Web project, `info` was the original NeXT machine sitting on Tim Berners-Lee's desk. It was the central repository of all WWW code, documentation, and "promotional material" (such as the Virtual Library). It also served as an FTP server for people climbing onto the web, and it offered a service where people could telnet to the machine to use the simple line-mode browser. It hosted an HTTP-to-WAIS Gateway for early web clients which did not support WAIS directly. It hosted several mailing lists for people involved in the project.

There were several results of this. There were many URLs with the hostname `info.cern.ch` created, all over the web. Early clients had the address of this machine hardcoded in as the location of the WAIS gateway. The line-mode browser had a hardcoded into it a default initial URL pointing to that machine. The standard bootstrapping instructions for people new to the Web were,

> "Telnet to `info.cern.ch` and go from there."

As the web exploded in popularity, this NeXT machine became extremely overloaded. In November 1993, a new Sun, named `www0.cern.ch` was purchased as a replacement and the name `info.cern.ch` became an alias for that machine. This machine was still the central repository and server, and it additionally became an NFS server for other local development machines.

By Summer 1994, `info.cern.ch` was again overloaded. During peak load times, roughly corresponding to normal business hours in America, the peak average request rate would approach 5 Hz, resulting in a data flow of 50 kilobytes per second. The load average on `www0.cern.ch` would climb to anywhere between 50 and 100, NFS would shut down, and the machine became unusable. Not only did this impede code development at CERN, having an unresponsive `info.cern.ch` meant that a large central portion of the web would be down.

## Initial decision

It was decided that the most cost-effective way of solving this overload problem was to split the services onto more than one machine.

### *.w3.org

One easy, and rather common, solution is to separate the services onto differently named machines, i.e., one name for telnet, one for ftp, one for http, etc. These names can be created as aliases of one or more machines, and can be moved around transparently as the load dictates. This solution was adopted. Because of naming restrictions at CERN, and because the responsibility for these core services was at this time moving to the new W3 Organization, the new names were created in the Organization's DNS domain, `w3.org`.

The names chosen were:

           Service     Name
           -------     ----
           http       www.w3.org
           ftp        ftp.w3.org
           gopher     gopher.w3.org
           WAIS g/w   www.w3.org
           telnet     telnet.w3.org
           mail       mail.w3.org

It was accepted that the name `info.cern.ch` would have to continue to exist. There was some debate over whether it should be synonymous with `www.w3.org`, to preserve all of the URLs, or `telnet.w3.org`, to preserve the bootstrap instructions. It was decided administratively that `info.cern.ch` should be the same as `www.w3.org`.

### AFS and round-robin DNS

As most of the load on `info.cern.ch` was http traffic, and as this traffic continues to grow exponentially, it was decided to offer the http service from multiple platforms. A method for doing this was pioneered at NCSA; it was decided to implement the same solution here. Basically, the solution consists of making the filsystem of information available over AFS to more than one identically-configured server machines, and using multiple DNS address records to point to the set of machines.

An additional benefit of this approach is that when the W3O gets going more fully, we can provide multiple identical servers at members' sites scattered around the world. With the new multihome response-seeking code added to the library, this should take advantage of the AFS cache to provide a rapid delivery information, while keeping coordination tasks simple.

## Hardware

The old single-point host, `www0.cern.ch`, is a Sun SparcServer 10 (sun4m sparc) running SunOS 5.3, with 128 Mb of memory. This machine is (at the time of writing) the host for `telnet.w3.org`, `gopher.w3.org`, `ftp.w3.org`, and `mail.w3.org`. There are two machines, canonically named `www6.cern.ch` and `www7.cern.ch`, providing the `www.w3.org` (a.k.a `info.cern.ch`) service. They are both Sun SparcStation 20s (sun4m sparc) running SunOS 5.3, with 128 Mb and 96 Mb of memory, respectively. The machines' disks were reorganized, to provide for 400 Mb of AFS cache each.

## Implementation

### AFS

AFS is a supported service at CERN. This made the setup of the W3O filespace very simple. In particular, there is a locally written command, `sysctl`, which takes simplifies most administrative tasks. The commands given below are `sysctl` commands.

W3O was allocated 2 Gb on each of two CERN AFS servers. The W3O's AFS filespace is globally accessible at `/afs/cern.ch/w3.org/`. Under this directory one finds the directories `ftp`, `w3`, and `www`. These correspond to the anonymous ftp directory, `http://info.cern.ch/`, and `http://info.cern.ch/hypertext/WWW`. The latter split reflects the original partition of the URL space onto `www0`'s two 1Gb disks. There is a stub `var` directory, and softlinks `WWW-keyser -> www/www-name/WWW-keyser`, `WWW -> www`, and `W3 -> w3`. The first was a result of a slight rationalization of the directory structure; the latter two are necessary because many pathnames refer to the capitalized directory names, but a restriction requires that all AFS volume mount points begin with a lowercase letter.

After analyzing the existing structure of the directories, it was decided to place the `WWW`, `WWW-name`, `Administration`, and `History` directories on one server; and the `ftp`, `W3/hypertext`, and `DataSources` directories on the other. The commands used to create the system follow:

    www6% /afs/usr/local/bin/sysctl -h afs4 -s
    Sysctld (Version 1) on afs4.cern.ch
    sysctl> help

    Primitives:
            vos-addsite
            vos-create
            vos-move
            vos-release      or
            vos-rename
            vos-remove

            fs-mkmount
            fs-rmmount
            fs-setacl
            fs-setquota

    Project volumes:
            pv-list [{-s | -a}]

    sysctl> pv-list -s
    ACL        Mount point / Pattern                    Server(s)
    ===        ===== =====   =======                    =========
    w3         /afs/.cern.ch/w3.org/[a-z]               afs3/j afs4/j
             - p\.w3\.

    sysctl> vos-create afs4 j p.w3.org
    sysctl> fs-mkmount /afs/.cern.ch/w3.org p.w3.org
    sysctl> fs-setquota p.w3.org 5000

    sysctl> vos-create afs4 j p.w3.ftp
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/ftp p.w3.ftp
    sysctl> fs-setquota p.w3.ftp 500000

    sysctl> vos-create afs4 j p.w3.hypertext
    sysctl> !mkdir /afs/cern.ch/w3.org/w3
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/w3/hypertext p.w3.hypertext
    sysctl> fs-setquota p.w3.hypertext 600000

    sysctl> vos-create afs4 j p.w3.DataSources
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/w3/hypertext/DataSources p.w3.DataSources
    sysctl> fs-setquota p.w3.DataSources 200000

    sysctl> vos-create afs3 j p.w3.WWW
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/www p.w3.WWW
    sysctl> fs-setquota p.w3.WWW 850000

    sysctl> vos-create afs3 j p.w3.WWW-name
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/www-name p.w3.WWW-name
    sysctl> fs-setquota p.w3.WWW-name 350000

    sysctl> vos-create afs3 j p.w3.Administration
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/Administration p.w3.Administration
    sysctl> fs-setquota p.w3.Administration 500000

    sysctl> vos-create afs3 j p.w3.History
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/History p.w3.History
    sysctl> fs-setquota p.w3.History 500000

    sysctl> vos-create afs3 j p.w3.var.adm
    sysctl> !mkdir /afs/cern.ch/w3.org/var
    sysctl> fs-mkmount /afs/.cern.ch/w3.org/var/adm p.w3.var.adm
    sysctl> fs-setquota p.w3.var.adm 400000

    sysctl> exit
    www6%

Put in the ACL stuff here...

### Data Transfer

Data was transferred from the disks of `www0` to the AFS volumes using `tar` and `rcp`. The files were read as `root`, to overcome permissions; they were created as the new AFS user `wwwteam`. Unfortunately, `chown` and `chgrp` are not supported under AFS, and as a result `wwwteam` ended up owning everything. However, all team members, as well as `wwwteam`, are full members of the `wwwteam:users` group and have full permissions on these files.

The initial data copy took approximately two days (part of the delay was a result of the standard overload on `www0.cern.ch`). No attempt was made at this time to ensure data consistancy; this was deliberately merely a snapshot.

### Service and testing

At this point, `http` daemons were installed on `www6.cern.ch` and `www7.cern.ch`; with identical configurations based on the `www0.cern.ch` original. Some sub-webs on `info.cern.ch` are actually NFS-mounted from other machines (notably, `dxcern.cern.ch`); arrangements had to be made to automount these directories.

As the name `info.cern.ch` will point to these new machines, but the non-http services will not be supported on them, pseudo-servers were installed on the telnet, ftp, and gopher ports. These "servers" merely consist of the following entries in `/etc/inetd.conf`:

    ftp     stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/ftp.message
    telnet  stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/telnet.message
    70      stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/gopher.message

where the message files contain pointers to the new service names.

There are a few web services on `info.cern.ch` which accept input and modify files (notably WIT and the New Server Registration Form). The CGI scripts implementing these services had to be modified to authenticate themselves; the modifications are described in the CERN AFS User's Guide. Another AFS account, `wwwcron`, was created for these automated scripts: this account only has the bare AFS permissions needed for the scripts to run.

One concern was whether this AFS-based service could keep up with the request rate: it would be expected if file access over AFS were slower than access to local disks; the question was whether or not the overhead would significantly impact performance. Therefore, a small multithreaded program was created which would read one of `www0.cern.ch`'s own `httpd` logfiles, and rapidly fire off requests to the new servers simulating actual usage patterns. The number of threads could be chosen; varying this number varied the number of concurrent requests and allowed this variable to be factored out of the results. This program, with a couple dozen threads, could run through the logfile at about five times the actual logged rate. After going over the new servers' logfiles, it was concluded that AFS did not create a bottleneck that would slow the new machines below the response time of the original.

A request was made to the CERN firewall folks to allow `http` access to the two new machines; to do this, the machines had to be renumbered. They are now 128.141.202.119 and 128.141.202.120.

### Final copy and official switchover

Following these tests, a find was done on the entire file hierarchy of `www0.cern.ch`, to find files modified since the time they were transferred to AFS. A moratorium was placed on modifiying files, these files were moved across, and then the AFS versions were officially announced as being the official masters. Development continued using AFS, greatly relieving the problems created when `www0.cern.ch` was too loaded to serve NFS requests.

At this time, `http` requests to `info.cern.ch` were of course still going to `www0.cern.ch`. The `http` server on `www0.cern.ch` was re-configured to use the AFS versions. For historical reasons, `www0.cern.ch` did not have the full AFS software installed upon it; rather, it used an NFS/AFS gateway service. This created two problems: the NFS/AFS gateway only provides read-only access, breaking the CGI scripts mentioned above; second, the NFS client software on `www0.cern.ch` could not stand up to the intense load, and the machine repeatedly crashed. We relented, and reverted to using the local disks.

### DNS change

The next step was to change the DNS address records to point to the new machines. Typical procedure is merely to create multiple address records in the DNS configuration files, like this:

#### Before

    $ORIGIN cern.ch.
    www0             IN      A       128.141.201.214
    info             IN      CNAME   www0.cern.ch.

#### After

    $ORIGIN cern.ch.
    www0             IN      A       128.141.201.214
    www6             IN      A       128.141.202.119
    www7             IN      A       128.141.202.120
    info             IN      A       128.141.202.119
                     IN      A       128.141.202.120
    $ORIGIN w3.org.
    telnet           IN      CNAME   www0.cern.ch.
    gopher           IN      CNAME   www0.cern.ch.
    ftp              IN      CNAME   www0.cern.ch.
    mail             IN      CNAME   www0.cern.ch.
    www              IN      A       128.141.202.119
                     IN      A       128.141.202.120

Note: the `w3.org` domain is actually maintained at MIT.

Unfortunately, things aren't so simple at CERN. There is a large Oracle database listing every computer on site, with all kinds of data: its hardware address, its physical location, the people responsible for it, its name, what networks it is on, what networking protocols it supports, and its network addresses. When this database was created, a constraint was placed on it, requiring a one-to-one mapping between machine names and IP addresses. Operations diagnostic tools rely on these constraints, so they can't merely be lifted. It was estimated by the responsible people that it would take a couple weeks to lift this restriction (which had already been requested by other people, as well, for similar purposes). One possibility we proposed was to alias `info.cern.ch` to the name `www.w3.org`; the `w3.org` domain is maintained by MIT, and does not suffer this limitation. However, after we suggested this possibility, a work-around was found. This work-around will bridge the gap until the modified database system is ready.

The change was made the afternoon of Monday, the 17th of October. Initially the load to the two new machines was light, as the new DNS information had not yet propagated to out to refill the caches. However, by Wednesday virtually all requests were being sent to the new servers.

The initial response has been very good. The load average of all three machines has stayed quite low, and neither machine serving http has crashed. Response time seems to be better, although there are anomalous failures tenatively connected with touching a file not in the AFS cache.

## Next steps

There are still a few things that should be attended to:

- The AFS usage should be monitored and tuned,
- The httpd server response should be monitored, both to examine this possible first-time-touched problem, and to predict when more machines will need to be added to the service,
- The other services should be monitored, and either split or restricted as needed to maintain usable services.

## Conclusions

The services once all hosted on `info.cern.ch` have been successfully separated onto multiple machines. The method used allows further splits between services, if needed, and establishes a parallel, distributed `http` service that can be easily extended in the future.

The immediate problems of overload have been alleviated. Also, the main filesystem has been moved to a separate service, making it more independant of our overloaded machines. While we will probably always be overloaded, we believe this change will reduce the resultant problems.