Continue reading »]]> A few things all of us are probably interested in:

• SSD Write Amplification:
  
  
  http://www.extremetech.com/article2/0,2845,2329594,00.asp
  
  
• Journaling and write performance in ext4:
  
  
  http://thunk.org/tytso/blog/2009/03/01/ssds-journaling-and-noatimerelatime/

The bottom-line as far as wear and tear: vfat is basically the same as a journaled FS from a wear-leveling standpoint, both are worst case. Thankfully vfat, being the worst case, forced the manufacturers to deal with the problem early. So all those outlandish claims I made about the world being flat and NTFS on thumbdrives are unfounded. Ted’s evidence does not speak to the amount of effort it takes to create an NTFS thumbdrive in Windows, however

Continue reading »]]> It’s been three years since we upgraded the hardware that hosts our various sites. I contacted my provider (Crucial Paradigm) and got some competitive offers. Stefan, my friend in Berlin that I split the server with, and I agreed on the following specs:

• Athlon 64 x2 4000 (2 cores @ 2.1GHz, 512K L2 each)
• 4GB RAM, 160GB SATA, 100Mbps
• CentOS 5
• Apache, BIND, MySQL, Postifx, Spam Assassin, ClamAV, Cyrus IMAP
• SELinux enforcing

Once again this is going to be a dedicated, remote, hosted server. A few days later and they contacted me with the login information. I’m going to describe the move from a high-level. I’m not going to go through the individual config file modifications or how to dump a Cyrus database.

It only took me about a day to prep the new system for the move. Stefan was gone for a week so I could move pretty much all of my stuff but I would wait until he got back to move the shared mail services. It is rather difficult to incrementally move users from one system to another using Cyrus so I just did a test migration of the IMAP spools and databases but waited on actually updating the DNS MX records.

First I updated the glue records and name servers at my registrar. I made the new system master and the old system I just modified to run as a slave. I also had Josh fix his config since he is my DNS buddy (I highly recommend using the DNS buddy system). With DNS up and running I added entries for the new server.

Then I moved my website. I just dumped my databases and pulled them into the new system. A few Apache tweaks and everything was good to go. I was pulling my hair out at one point trying to figure out why the CentOS default page was appearing but I finally tracked it down to a welcome.conf configuration file. I removed that and found I could debug Apache config problems more easily.

I decided to tackle all of the SSL stuff at once. I setup Apache, Cyrus, and Postfix w/ certs and CAs. I tested each service to ensure I had the directory permissions correct and had the authentication mechanisms properly configured. It is at this point that I tested a Cyrus migration knowing that I would be repeating the process again later.

Then I moved on to configuring the rest of the Postfix chain using Amavis, Spam Assassin, and ClamAV. There was plenty of documentation on this process available online. Note that ClamAV (and maybe a few of the others) are only available in the rpmforge yum repos. These programs aren’t officially part of the CentOS/RHEL distributions but are commonly used by the user communities. As a result their configuration files don’t mesh precisely with the rest of the services, for example storing the virus database in /var/clamav by default instead of /var/lib/clamav. I fixed these discrepancies and would recommend you do the same. It did take me a while to track all of these down. freshclam (the ClamAV updater) was running from a Cron job. It was trying to write to “/var/clamav” which didn’t exist. As a result the virus signatures weren’t updating. I discovered this only through reviewing the logs.

As soon as Stefan got back he moved all of his services and data. Finally we cut the mail over to the new system by fixing the MX records and using the Postfix transport map feature on the old system to force it to relay to the new server. That completed the migration.

SELinux
Upon login I discovered SELinux was disabled despite my specific request that it be enabled. I decided to fix it by hand instead of letting Brickwall fix it automatically to show that going from disabled to permissive to enforcing isn’t bad and shouldn’t be scary. It should be done regardless of the presence of a management tool like Brickwall. I changed the flag in /etc/selinux/config to enforcing and ran # touch /.autorelabel.

I also added enforcing=0 to the ends of the kernel command lines in /boot/grub/menu.lst.

Tangent – I chose this route, modifying the grub file, for going into permissive mode during configuration and testing instead of setting the flag in /etc/selinux/config to permissive. Applications, such as those built into Red Hat and CentOS as well as third party applications like Brickwall, modify the settings in /etc/selinux/config. It is possible for me to inadvertently set the system to boot into enforcing before I am ready using these applications. Since this is a remote server and the configuration is in flux I want to make sure a quick remote reboot request gets me back in to my system. It is the exact same as using iptables remotely; you don’t load the firewall rules automatically on boot until you’re sure you can get back in through SSH. As soon as I’m confident that I typed in the SSH network settings properly the system boots into enforcing mode.

After setup is complete remember to remove the kernel command line flag – always going into permissive mode on reboot, much like not auto-loading firewall rules, is a patentable bad idea on a production system.

I disabled all unnecessary services (everything except SSH at this point) and rebooted. The system relabeled and came up in permissive mode. Everything was working fine and running in the proper SELinux domains according to # ps axZ. I ran # setenforce 1 to go into enforcing mode for the first time.

Nothing bad happened. The world didn’t end. The system didn’t stop responding. I’m also assuming the rampant rumor that SELinux kills Giraffes in enforcing mode is false because I watched the news and saw nothing Giraffe related. This SELinux stuff really isn’t as hard, or as mean as people make it sound. BTW Brickwall would have taken care of all of this editing of SELinux config files and relabeling stuff for me if I wasn’t such a control freak.

Well turning it on is one thing. Really using it is a whole ‘nother thing right? That is where Brickwall comes into play (disclaimer [1]). I mentioned it before but think “SELinux Management” and there are free versions for Fedora and demos for RHEL and CentOS. Because I’m cool I get to use the Enterprise Edition. Mere mortals may have to run Professional or Standard via “ssh -X”. There is no difference in configurability. Both use the same configuration GUI. The only policy difference is the policy for the remote daemon – not really needed if the remote enterprise management daemon isn’t installed. One has centralized/remote management, the other does not.

Brickwall Enterprise Edition has two components, the centralized manager application with configuration editor and the remote daemon. The centralized management application takes different plug-ins – I’m only going to be using Brickwall plug-in that supports general SELinux configuration. I installed the Enterprise Manager with the Brickwall Enterprise component on my desktop RHEL 5 system. This installation process generated the remote daemon RPMs for me. These packages include SSL keys tying the remote daemon to this specific Enterprise Manager install. The SSL keys are used to encrypt the network traffic between the manager and the client daemons.

I installed the remote daemon on the new system. It is just a little daemon that facilitates remote management of SELinux policy. I added one of the IPs of this system to a group in my Enterprise Manager. The system had to be “activated” which means Brickwall had to switch from the standard targeted policy to a Brickwall policy. There aren’t really any functional differences between the two policies – a default Brickwall policy is semantically equivalent to a targeted policy as shipped by Red Hat/CentOS. But the Brickwall policy contains the structure we need in-place to customize the policy later.

Continuing w/ Brickwall I started restricting network settings for the services I would be running (listed above). I restricted things like the spam, virus, mysql, and other mail filtering services to local host only. I restricted service ports to meet my requirements. I applied the configuration changes and rebooted and went into enforcing to verify services came up in the proper domain after a reboot.

All of the SELinux domain names show exactly what processes they target so run`ps axZ`. All of you key services should be running as “*:*:servicename_t”, such as “mysqld_t” or “httpd_t” or “postfix_smtp_t”.

The only thing that should be “unconfined_t” is user processes or custom services that aren’t targeted by SELinux. Note that there are some 200 odd applications or services covered by SELinux so there is a good chance your “custom service” is covered.

[1] Disclaimer: I work for the company that makes Brickwall. Since I’m only an amateur blogger (not commerically endorsed, still eligible for the Bloglympics) and since this doesn’t appear on tresys.com I’m not going to be writing a review or praise the software, I’m just going to run through using it to batten down the hatches on my system.

Continue reading »]]> I’ve been thinking of researching the PS3 hypervisor. Mainly from a security perspective but this led me to thinking. You know what would be cool? Micro-partioning a PS3. Obviously IBM has experience with micro-partitioning on the PPC and the Linux distros are already tried and tested in these exact environments. You could use the system as a Linux server for your house (SMB, mail, et al), play games/fold proteins at the same time, and even run multiple guests if you desire.

The PS3 already has a hypervisor that allocates a subset of system resources to the “otheros” kernel. One example, the hypervisor only exposes either a 10G or a 50G virtual disk (/dev/sda) instead of the 50/10 split physical disk that actually exists (/dev/sda1, /dev/sda2). I can only assume that this is to protect proprietary/otherwise protected information on the game OS partition. A second example of the hypervisor partitioning resources: the guest OS can only access a subset of the total cell processor cores available. Another measure in controlling access to proprietary information?

Granted this hypervisor is not nearly as advanced as those found in other IBM PPC platforms. Still, I might have found a use for virtualization in my home – running games and running services at the same time. Yippee? A second use, the real dork in me wants to run other Linux OSs at the same time as a Fedora OS. Mainly because I want to experiment on other Linux distros but run Fedora to aid in my research mentioned above.

Continue reading »]]> I’ve been having tons of wireless network issues lately. At work, which is not surprising being I work at tech company – only God knows who is running what service on their laptop or who is running the microwave in the kitchen across the hall. More worrisome, I’ve been having issues at home too.

So I get home last night and settle down write some documentation in DocBook format. I’ve been writing a custom XSLT in a valiant attempt to unify the documentation process at work. xsltproc is called to perform the transformation on the XML data and create HTML and PDFs as output. Fairly straight forward, right?

Wrong…

I start streaming from iTunes to my Airport Express. Currently, I’m using an Airport Extreme running 802.11N at 5GHz only and an Airport Express running 802.11G for my iPhone and visitors without 802.11N compatible laptops. The Express is wired to a LAN port on the Extreme and is connected via a TIP fiber cable to my stereo.

All of the sudden VPN and VNC sessions start dropping like hell. wtf mate? Long story short my xsltproc process went from 30 seconds to 5 minutes because of remote URLs being referenced as DTDs. I know, I know. My night was pretty damn exciting.

Of course I jump to conclusions and start blaming Leopard. Well after punching the wall a few times I sat down. It turns out it only happens when I have my Airport Extreme in 802.11n 5GHz mode. As a rudimentary example, my download speeds form kernel.org went from 30KB to about 700 KB when I went to 802.11n 2.4GHz only. Not sure what is wrong but I blame Apple Just kidding. If you look at this image you will see a dip in the graph in the center when the Extreme reboots to go from 2.4 GHz to 5 GHz. When it comes back online notice the distinct dip that is in the signal, communication quality, and signal to noise ratio. Not sure if the issue is hardware or interference related but given a similar amount of noise is seen at 2.4 GHz I’m guessing it is hardware related.