Server Migration

It’s been three years since we upgraded the hardware that hosts our various sites. I contacted my provider (Crucial Paradigm) and got some competitive offers. Stefan, my friend in Berlin that I split the server with, and I agreed on the following specs:

• Athlon 64 x2 4000 (2 cores @ 2.1GHz, 512K L2 each)
• 4GB RAM, 160GB SATA, 100Mbps
• CentOS 5
• Apache, BIND, MySQL, Postifx, Spam Assassin, ClamAV, Cyrus IMAP
• SELinux enforcing

Once again this is going to be a dedicated, remote, hosted server. A few days later and they contacted me with the login information. I’m going to describe the move from a high-level. I’m not going to go through the individual config file modifications or how to dump a Cyrus database.

It only took me about a day to prep the new system for the move. Stefan was gone for a week so I could move pretty much all of my stuff but I would wait until he got back to move the shared mail services. It is rather difficult to incrementally move users from one system to another using Cyrus so I just did a test migration of the IMAP spools and databases but waited on actually updating the DNS MX records.

First I updated the glue records and name servers at my registrar. I made the new system master and the old system I just modified to run as a slave. I also had Josh fix his config since he is my DNS buddy (I highly recommend using the DNS buddy system). With DNS up and running I added entries for the new server.

Then I moved my website. I just dumped my databases and pulled them into the new system. A few Apache tweaks and everything was good to go. I was pulling my hair out at one point trying to figure out why the CentOS default page was appearing but I finally tracked it down to a welcome.conf configuration file. I removed that and found I could debug Apache config problems more easily.

I decided to tackle all of the SSL stuff at once. I setup Apache, Cyrus, and Postfix w/ certs and CAs. I tested each service to ensure I had the directory permissions correct and had the authentication mechanisms properly configured. It is at this point that I tested a Cyrus migration knowing that I would be repeating the process again later.

Then I moved on to configuring the rest of the Postfix chain using Amavis, Spam Assassin, and ClamAV. There was plenty of documentation on this process available online. Note that ClamAV (and maybe a few of the others) are only available in the rpmforge yum repos. These programs aren’t officially part of the CentOS/RHEL distributions but are commonly used by the user communities. As a result their configuration files don’t mesh precisely with the rest of the services, for example storing the virus database in /var/clamav by default instead of /var/lib/clamav. I fixed these discrepancies and would recommend you do the same. It did take me a while to track all of these down. freshclam (the ClamAV updater) was running from a Cron job. It was trying to write to “/var/clamav” which didn’t exist. As a result the virus signatures weren’t updating. I discovered this only through reviewing the logs.

As soon as Stefan got back he moved all of his services and data. Finally we cut the mail over to the new system by fixing the MX records and using the Postfix transport map feature on the old system to force it to relay to the new server. That completed the migration.

SELinux
Upon login I discovered SELinux was disabled despite my specific request that it be enabled. I decided to fix it by hand instead of letting Brickwall fix it automatically to show that going from disabled to permissive to enforcing isn’t bad and shouldn’t be scary. It should be done regardless of the presence of a management tool like Brickwall. I changed the flag in /etc/selinux/config to enforcing and ran # touch /.autorelabel.

I also added enforcing=0 to the ends of the kernel command lines in /boot/grub/menu.lst.

Tangent – I chose this route, modifying the grub file, for going into permissive mode during configuration and testing instead of setting the flag in /etc/selinux/config to permissive. Applications, such as those built into Red Hat and CentOS as well as third party applications like Brickwall, modify the settings in /etc/selinux/config. It is possible for me to inadvertently set the system to boot into enforcing before I am ready using these applications. Since this is a remote server and the configuration is in flux I want to make sure a quick remote reboot request gets me back in to my system. It is the exact same as using iptables remotely; you don’t load the firewall rules automatically on boot until you’re sure you can get back in through SSH. As soon as I’m confident that I typed in the SSH network settings properly the system boots into enforcing mode.

After setup is complete remember to remove the kernel command line flag – always going into permissive mode on reboot, much like not auto-loading firewall rules, is a patentable bad idea on a production system.

I disabled all unnecessary services (everything except SSH at this point) and rebooted. The system relabeled and came up in permissive mode. Everything was working fine and running in the proper SELinux domains according to # ps axZ. I ran # setenforce 1 to go into enforcing mode for the first time.

Nothing bad happened. The world didn’t end. The system didn’t stop responding. I’m also assuming the rampant rumor that SELinux kills Giraffes in enforcing mode is false because I watched the news and saw nothing Giraffe related. This SELinux stuff really isn’t as hard, or as mean as people make it sound. BTW Brickwall would have taken care of all of this editing of SELinux config files and relabeling stuff for me if I wasn’t such a control freak.

Well turning it on is one thing. Really using it is a whole ‘nother thing right? That is where Brickwall comes into play (disclaimer [1]). I mentioned it before but think “SELinux Management” and there are free versions for Fedora and demos for RHEL and CentOS. Because I’m cool I get to use the Enterprise Edition. Mere mortals may have to run Professional or Standard via “ssh -X”. There is no difference in configurability. Both use the same configuration GUI. The only policy difference is the policy for the remote daemon – not really needed if the remote enterprise management daemon isn’t installed. One has centralized/remote management, the other does not.

Brickwall Enterprise Edition has two components, the centralized manager application with configuration editor and the remote daemon. The centralized management application takes different plug-ins – I’m only going to be using Brickwall plug-in that supports general SELinux configuration. I installed the Enterprise Manager with the Brickwall Enterprise component on my desktop RHEL 5 system. This installation process generated the remote daemon RPMs for me. These packages include SSL keys tying the remote daemon to this specific Enterprise Manager install. The SSL keys are used to encrypt the network traffic between the manager and the client daemons.

I installed the remote daemon on the new system. It is just a little daemon that facilitates remote management of SELinux policy. I added one of the IPs of this system to a group in my Enterprise Manager. The system had to be “activated” which means Brickwall had to switch from the standard targeted policy to a Brickwall policy. There aren’t really any functional differences between the two policies – a default Brickwall policy is semantically equivalent to a targeted policy as shipped by Red Hat/CentOS. But the Brickwall policy contains the structure we need in-place to customize the policy later.

Continuing w/ Brickwall I started restricting network settings for the services I would be running (listed above). I restricted things like the spam, virus, mysql, and other mail filtering services to local host only. I restricted service ports to meet my requirements. I applied the configuration changes and rebooted and went into enforcing to verify services came up in the proper domain after a reboot.

All of the SELinux domain names show exactly what processes they target so run`ps axZ`. All of you key services should be running as “*:*:servicename_t”, such as “mysqld_t” or “httpd_t” or “postfix_smtp_t”.

The only thing that should be “unconfined_t” is user processes or custom services that aren’t targeted by SELinux. Note that there are some 200 odd applications or services covered by SELinux so there is a good chance your “custom service” is covered.

[1] Disclaimer: I work for the company that makes Brickwall. Since I’m only an amateur blogger (not commerically endorsed, still eligible for the Bloglympics) and since this doesn’t appear on tresys.com I’m not going to be writing a review or praise the software, I’m just going to run through using it to batten down the hatches on my system.