Continue reading »]]> If you have ever tried to use a new Fedora or RHEL to build packages for older systems you may have run into errors at installation time. For example, if you were to roll a package on Fedora >=10 or RHEL >=6 and tried to install it on RHEL 5 you will get rpmlib(FileDigests) errors. This is because newer versions of RPM use new compression and hashing algorithms. To force an RPM to be build using the older algorithms add these lines to your spec file:
%define _binary_payload w9.gzdio
%define _binary_filedigest_algorithm 1
%define _source_filedigest_algorithm 1

You can wrap them in a conditional check if you only want to use the older algorithms when targeted older versions of RHEL/Fedora but they are forward compatible, ie you can install a package compiled with these flags on RHEL 6 without a problem.

Continue reading »]]> If you want to run sudo from any part of a cron job make sure you comment out the

Defaults requiretty

from /etc/sudoers. Bit me twice now so hopefully I’ll remember next time.

Ed: Apologies to those who have read this before but I had to restore from backup and my last post wasn’t in that backup.

Continue reading »]]> A few things all of us are probably interested in:

• SSD Write Amplification:
  
  
  http://www.extremetech.com/article2/0,2845,2329594,00.asp
  
  
• Journaling and write performance in ext4:
  
  
  http://thunk.org/tytso/blog/2009/03/01/ssds-journaling-and-noatimerelatime/

The bottom-line as far as wear and tear: vfat is basically the same as a journaled FS from a wear-leveling standpoint, both are worst case. Thankfully vfat, being the worst case, forced the manufacturers to deal with the problem early. So all those outlandish claims I made about the world being flat and NTFS on thumbdrives are unfounded. Ted’s evidence does not speak to the amount of effort it takes to create an NTFS thumbdrive in Windows, however

Continue reading »]]> My favorite BASH shell completion of all time is hostname completion for all of my favorite commands:

SSH_COMPLETE=( $(cat ~/.ssh/known_hosts | \
cut -f 1 -d ‘ ‘ | \
sed -e s/,.*//g | \
uniq | \
egrep -v [0123456789]) )

complete -o default -W “${SSH_COMPLETE[*]}” ssh scp sftp rsync nmap traceroute ping nslookup dig host nmap nc

I find most of the hosts I ssh into I use for basic network diagnostics. So performing completion for traceroute, ping, etc based on the contents of known_hosts works great. Just add it to your .bashrc.

Update: 2008/11/04

I’ve modified it to support completion of SVN commands leveraging the hostnames expanded during the SSH completion above:

SVN_COMPLETE=( $(svn -h|grep -e '^ '|awk '{ print $1; }') $SSH_COMPLETE )
complete -o default -W "${SVN_COMPLETE[*]} ${SSH_COMPLETE[*]}" svn

Adding these may lengthen your shell exec tasks.

I test at work. I work from home. But I will not test from home. Fix it yourself. (trademark pending).

Continue reading »]]> I was trying to connect to RHN from Yum/up2date in Red Hat Enterprise Linux 5. I kept getting fatal invalid SSL cert errors. The strange part – out of all of the machines I tested it was only occuring on a single RHEL 5 laptop. The really strange part – it was happening on the host as well as inside a guest in a VM running in VMware Player.

If you get this:


[root@rhel5-vm ~]# yum search openoffice
Loading "rhnplugin" plugin
Loading "installonlyn" plugin
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.main(sys.argv[1:])
File "/usr/share/yum-cli/yummain.py", line 85, in main
base.getOptionsConfig(args)
File "/usr/share/yum-cli/cli.py", line 199, in getOptionsConfig
errorlevel=opts.errorlevel)
File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 134, in doConfigSetup
self.plugins.run('init')
File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 153, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/lib/yum-plugins/rhnplugin.py", line 88, in init_hook
login_info = up2dateAuth.getLoginInfo()
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 139, in getLoginInfo
login()
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 98, in login
li = server.up2date.login(systemId)
File "/usr/share/rhn/up2date_client/rhnserver.py", line 64, in __call__
raise up2dateErrors.SSLCertificateVerifyFailedError()
up2date_client.up2dateErrors.SSLCertificateVerifyFailedError: The SSL certificate failed verification.


Check the date/time on the machine. Mine was a VM syncing w/ a host clock with a bad CMOS battery. Caused it to think it was 2005 and the cert wasn’t valid yet. Also explains why it happened in both host and guest.

Continue reading »]]> Ran into a problem with Exchange. I created a server-side rule to place spam messages in a folder that didn’t exist. All spam was instead going to the root folder, the folder above my inbox. Well if you use Entourage to access Exchange you will not be able to access this folder. If you go into Outlook Web Access (OWA) you can navigate to the root user folder by clicking on “Folders”.

Unfortunately by this point I had like thousands and thousands of messages. There is no “Delete All” option in OWA. Craptacular. So moving on I mounted the server via WebDAV in finder (cmd+k). The URL to mount will look something like:
http://exchange.servername.com/exchange/username

If this folder is large navigating to it in Finder will be a horrific experience. Open Terminal.app and navigate to the mountpoint.

~/> cd /Volumes/username

Once again, if the folder is large rm *.EML will not work as the wildcard expansion done by bash will exceed the length of the command-line itself (32K by default IIRC). Try this:


/Volumes/username> SRC=./*.EML
/Volumes/username> for i in $SRC; do rm "$i"; done


This will cleanup most, if not all, of the mess. Some files will not be removed due to escape characters and escaped escaped characters etc. Open the folder in Finder and delete the rest. There are def. more elegant ways to handle this from a scripting standpoint but this was quick and worked.

Continue reading »]]> It’s been three years since we upgraded the hardware that hosts our various sites. I contacted my provider (Crucial Paradigm) and got some competitive offers. Stefan, my friend in Berlin that I split the server with, and I agreed on the following specs:

• Athlon 64 x2 4000 (2 cores @ 2.1GHz, 512K L2 each)
• 4GB RAM, 160GB SATA, 100Mbps
• CentOS 5
• Apache, BIND, MySQL, Postifx, Spam Assassin, ClamAV, Cyrus IMAP
• SELinux enforcing

Once again this is going to be a dedicated, remote, hosted server. A few days later and they contacted me with the login information. I’m going to describe the move from a high-level. I’m not going to go through the individual config file modifications or how to dump a Cyrus database.

It only took me about a day to prep the new system for the move. Stefan was gone for a week so I could move pretty much all of my stuff but I would wait until he got back to move the shared mail services. It is rather difficult to incrementally move users from one system to another using Cyrus so I just did a test migration of the IMAP spools and databases but waited on actually updating the DNS MX records.

First I updated the glue records and name servers at my registrar. I made the new system master and the old system I just modified to run as a slave. I also had Josh fix his config since he is my DNS buddy (I highly recommend using the DNS buddy system). With DNS up and running I added entries for the new server.

Then I moved my website. I just dumped my databases and pulled them into the new system. A few Apache tweaks and everything was good to go. I was pulling my hair out at one point trying to figure out why the CentOS default page was appearing but I finally tracked it down to a welcome.conf configuration file. I removed that and found I could debug Apache config problems more easily.

I decided to tackle all of the SSL stuff at once. I setup Apache, Cyrus, and Postfix w/ certs and CAs. I tested each service to ensure I had the directory permissions correct and had the authentication mechanisms properly configured. It is at this point that I tested a Cyrus migration knowing that I would be repeating the process again later.

Then I moved on to configuring the rest of the Postfix chain using Amavis, Spam Assassin, and ClamAV. There was plenty of documentation on this process available online. Note that ClamAV (and maybe a few of the others) are only available in the rpmforge yum repos. These programs aren’t officially part of the CentOS/RHEL distributions but are commonly used by the user communities. As a result their configuration files don’t mesh precisely with the rest of the services, for example storing the virus database in /var/clamav by default instead of /var/lib/clamav. I fixed these discrepancies and would recommend you do the same. It did take me a while to track all of these down. freshclam (the ClamAV updater) was running from a Cron job. It was trying to write to “/var/clamav” which didn’t exist. As a result the virus signatures weren’t updating. I discovered this only through reviewing the logs.

As soon as Stefan got back he moved all of his services and data. Finally we cut the mail over to the new system by fixing the MX records and using the Postfix transport map feature on the old system to force it to relay to the new server. That completed the migration.

SELinux
Upon login I discovered SELinux was disabled despite my specific request that it be enabled. I decided to fix it by hand instead of letting Brickwall fix it automatically to show that going from disabled to permissive to enforcing isn’t bad and shouldn’t be scary. It should be done regardless of the presence of a management tool like Brickwall. I changed the flag in /etc/selinux/config to enforcing and ran # touch /.autorelabel.

I also added enforcing=0 to the ends of the kernel command lines in /boot/grub/menu.lst.

Tangent – I chose this route, modifying the grub file, for going into permissive mode during configuration and testing instead of setting the flag in /etc/selinux/config to permissive. Applications, such as those built into Red Hat and CentOS as well as third party applications like Brickwall, modify the settings in /etc/selinux/config. It is possible for me to inadvertently set the system to boot into enforcing before I am ready using these applications. Since this is a remote server and the configuration is in flux I want to make sure a quick remote reboot request gets me back in to my system. It is the exact same as using iptables remotely; you don’t load the firewall rules automatically on boot until you’re sure you can get back in through SSH. As soon as I’m confident that I typed in the SSH network settings properly the system boots into enforcing mode.

After setup is complete remember to remove the kernel command line flag – always going into permissive mode on reboot, much like not auto-loading firewall rules, is a patentable bad idea on a production system.

I disabled all unnecessary services (everything except SSH at this point) and rebooted. The system relabeled and came up in permissive mode. Everything was working fine and running in the proper SELinux domains according to # ps axZ. I ran # setenforce 1 to go into enforcing mode for the first time.

Nothing bad happened. The world didn’t end. The system didn’t stop responding. I’m also assuming the rampant rumor that SELinux kills Giraffes in enforcing mode is false because I watched the news and saw nothing Giraffe related. This SELinux stuff really isn’t as hard, or as mean as people make it sound. BTW Brickwall would have taken care of all of this editing of SELinux config files and relabeling stuff for me if I wasn’t such a control freak.

Well turning it on is one thing. Really using it is a whole ‘nother thing right? That is where Brickwall comes into play (disclaimer [1]). I mentioned it before but think “SELinux Management” and there are free versions for Fedora and demos for RHEL and CentOS. Because I’m cool I get to use the Enterprise Edition. Mere mortals may have to run Professional or Standard via “ssh -X”. There is no difference in configurability. Both use the same configuration GUI. The only policy difference is the policy for the remote daemon – not really needed if the remote enterprise management daemon isn’t installed. One has centralized/remote management, the other does not.

Brickwall Enterprise Edition has two components, the centralized manager application with configuration editor and the remote daemon. The centralized management application takes different plug-ins – I’m only going to be using Brickwall plug-in that supports general SELinux configuration. I installed the Enterprise Manager with the Brickwall Enterprise component on my desktop RHEL 5 system. This installation process generated the remote daemon RPMs for me. These packages include SSL keys tying the remote daemon to this specific Enterprise Manager install. The SSL keys are used to encrypt the network traffic between the manager and the client daemons.

I installed the remote daemon on the new system. It is just a little daemon that facilitates remote management of SELinux policy. I added one of the IPs of this system to a group in my Enterprise Manager. The system had to be “activated” which means Brickwall had to switch from the standard targeted policy to a Brickwall policy. There aren’t really any functional differences between the two policies – a default Brickwall policy is semantically equivalent to a targeted policy as shipped by Red Hat/CentOS. But the Brickwall policy contains the structure we need in-place to customize the policy later.

Continuing w/ Brickwall I started restricting network settings for the services I would be running (listed above). I restricted things like the spam, virus, mysql, and other mail filtering services to local host only. I restricted service ports to meet my requirements. I applied the configuration changes and rebooted and went into enforcing to verify services came up in the proper domain after a reboot.

All of the SELinux domain names show exactly what processes they target so run`ps axZ`. All of you key services should be running as “*:*:servicename_t”, such as “mysqld_t” or “httpd_t” or “postfix_smtp_t”.

The only thing that should be “unconfined_t” is user processes or custom services that aren’t targeted by SELinux. Note that there are some 200 odd applications or services covered by SELinux so there is a good chance your “custom service” is covered.

[1] Disclaimer: I work for the company that makes Brickwall. Since I’m only an amateur blogger (not commerically endorsed, still eligible for the Bloglympics) and since this doesn’t appear on tresys.com I’m not going to be writing a review or praise the software, I’m just going to run through using it to batten down the hatches on my system.

Continue reading »]]> I’ve been thinking of researching the PS3 hypervisor. Mainly from a security perspective but this led me to thinking. You know what would be cool? Micro-partioning a PS3. Obviously IBM has experience with micro-partitioning on the PPC and the Linux distros are already tried and tested in these exact environments. You could use the system as a Linux server for your house (SMB, mail, et al), play games/fold proteins at the same time, and even run multiple guests if you desire.

The PS3 already has a hypervisor that allocates a subset of system resources to the “otheros” kernel. One example, the hypervisor only exposes either a 10G or a 50G virtual disk (/dev/sda) instead of the 50/10 split physical disk that actually exists (/dev/sda1, /dev/sda2). I can only assume that this is to protect proprietary/otherwise protected information on the game OS partition. A second example of the hypervisor partitioning resources: the guest OS can only access a subset of the total cell processor cores available. Another measure in controlling access to proprietary information?

Granted this hypervisor is not nearly as advanced as those found in other IBM PPC platforms. Still, I might have found a use for virtualization in my home – running games and running services at the same time. Yippee? A second use, the real dork in me wants to run other Linux OSs at the same time as a Fedora OS. Mainly because I want to experiment on other Linux distros but run Fedora to aid in my research mentioned above.