Royksopp – A Higher Place

I recently moved house, to move in with my girlfriend. To make things simple, I sold my old HTPC (a first gen Mac Mini, slightly warmed over and with a 1TB external drive) and home theater to the mates whose flat I was departing.

This leaves me with nowhere to store the copious amounts of porn… errr… linux iso’s that I download. I couldn’t be bothered building something from scratch, and really wanted something that was relatively simple for me to administrate and with low power consumption. Granted I could have chucked a spare Via Epia motherboard with a SATA card into a spare case and I’d have been pretty much done with it, or resurrected my Shuttle. Instead, I got a suave looking Acer Easystore H340 with 2x 1TB drives:

The first problem I have with it is that it comes with Windows Home Server, and I’m thinking of petitioning Acer for a cashback on that. Windows is dandy for gaming but for anything else… no thanks. Plus being a Linux admin I simply can’t and won’t allow that shit in my house (except, of course, for the woeful Vista that I tolerate on the girlfriend’s laptop). And on top of that you need to install management software on a Windows box to set it up, something my girlfriend quite fiercely would not allow. Windows, LOL:

So as a BSD guy at heart I checked out FreeNAS, and decided it was probably mismatched given the hardware specs, though ZFS capability is appealing (albeit basically useless in my case, with my 32bit restriction). Openfiler is much of the same. What I was really after was something that I could pretty much replace WHS with, while tying in with my current BSD and Linux work, as well as being able to hook in with my homebrew modified Wii. Then I found Amahi. It’s beta, it’s not perfect (i.e. I don’t agree with the use of MySQL in the Greyhole subsystem, I’d prefer Postgres for anything where security of data is involved) but it’s pretty damned promising.

Ok, so here’s how I installed it. Windows heads at this point need to realise that Linux has a hidden strength – the ability to move a boot drive from PC to PC, and provided the hardware is friendly – i.e. the same architecture, it’ll just work ™. The same with Windows will tend to screw with the HAL and you’ll get BSOD’s.

First, I removed the boot drive and chucked it into a spare box, an Iwill XP4 Evo with a SATA card. As the specs of the Acer are conservative (Intel Atom, 2G of memory), I went with the 32bit version of Amahi. (Also note: At the end of the post I show how you can modify a PCI-E x16 graphics card to run at PCI-E x1. You could just do that and install Amahi straight on to the A340. The instructions I give won’t work, but they’ll give you a guideline. Standard disclaimers, YMMV’s etc apply)

Then I followed the instructions written by a no doubt devilishly handsome fellow on the Amahi forums. (Hint: it was me!)

Then, with Amahi booted and running, I issued the following command:
rm /etc/udev/rules.d/70-persistent-net.rules

Then I vi’d /etc/sysconfig/network-scripts/ifcfg-eth0 and removed the HWADDRESS line, and edited ONBOOT=no to read ONBOOT=yes.

We do the above so that when Amahi next boots, it will pick up the first ethernet interface and assign it the device name “eth0″. If you’re after a more descriptive explanation, look up udev in google. Then I issued a halt, waited for the Iwill box to poweroff, then I plugged the boot drive into the Acer box, which I then fired up. Et voila!

Now for some advanced tips:
1) You’ll notice the led’s aren’t right. Do this (adjusting to suit, e.g. get the latest version from here):

yum -y install gcc-c++ libudev-devel
cd /tmp
wget http://bitbucket.org/adaptation/mediasmartserverd/get/5654cec4f4d1.zip
unzip 5654cec4f4d1.zip
cd mediasmartserverd/ && make
mv mediasmartserverd /opt
chmod 755 /opt/mediasmartserverd
echo "# start our led daemon" >> /etc/rc.local
echo "/opt/mediasmartserverd -D" >> /etc/rc.local
./opt/mediasmartserverd -D

2) to add another drive, use cfdisk followed by mkfs.ext4 /dev/sdb1, followed by hda-diskmount. This may require some prereq installs:

yum -y install pmount fuse fuse-libs ntfs-3g

3) … probably more to come!

Now, this wasn’t without its issues. But only because of my own stupid fault – I thought I’d shorted the debug jumper but I’d actually shorted the CMOS clear jumpers. FAIL. This set the BIOS date back to something like 2007, Fedora was then complaining about file timestamps being way out of whack and it was subsequently demanding a fsck.

A permanent fix may exist in Network Console on Acid, but for now I had to get the headless Easystore some VGA capability. Balking at the $200 cost and lead time of a debug card, instead I went to a local PC store and petitioned them for any cheap/faulty PCI-E video cards they might have. They sold me a GeForce 8500GT with a dodgy HSF for NZD$30, little did they know that I had a plan.

A couple of drops of sewing machine oil in the bearings sorted out the HSF. I then used my hand nibbler and cut it physically to x4, but that didn’t work. So with some electrical tape I knocked it back to 1x, and that did work. Along with a USB keyboard, I was then able to see that the drives needed a fsck and sorted that out. In the future I’ll pick up a cheap low profile card and make this a permanent addition to the box.

So, that’s it for now. I’ll no doubt update this post and any subsequent ones, but hopefully this helps, and good luck if you decide to try out Amahi

And for good measure I’ll say it again: ALL STANDARD DISCLAIMERS APPLY! I’m all care, no responsibility.

The Dandy Warhols – You were the last high

So I upgraded my Eee-PC to Ubuntu 9.10 Karmic Koala and it hasn’t been without its problems. It seems that the best way to upgrade Ubuntu is, as always, back everything up, install anew and restore your data.

The main issue I had was that the screen would randomly go black, audio would still work but everything else was unresponsive. Switching to another terminal (e.g. ctrl+alt+F2) resulted in a screen with a full height cursor that was a couple of pixels wide… typing returned vertical white lines.

The fix is to edit /etc/X11/xorg.conf (e.g. sudo gedit /etc/X11/xorg.conf)

Section "Device"
Identifier "Configured Video Device"
EndSection

Becomes:

Section "Device"
Identifier "Configured Video Device"
Driver "intel"
Option "ForceEnablePipeA" "true"
Option "FramebufferCompression" "off"
EndSection

I’ve lost my other terminals, and ACPI is still broken – it won’t wake up after you’ve closed the lid – but apart from that it’s working a treat.

Moby – Disco Lies

Flax search is a search engine based on the powerful Xapian search engine. Basically it’s Xapian pre-configured with a nice UI and some extra features, and their business model is custom configuration and support, while offering Flax Basic as a taster. At the moment Flax Basic is developed primarily for Windows as a desktop search tool (a’la Google Desktop) but it can also be used as an intranet search engine, and it can be installed on Linux, like so. As root:

apt-get install python-cherrypy3 python-processing html2text
cd /opt
wget http://flaxcode.googlecode.com/files/flax-source-1.0.0.tgz && tar xvf flax-source-1.0.0.tgz
mv flax-source-1.0.0 flax
wget http://flaxcode.googlecode.com/files/HTMLTemplate-1.4.2.tar.gz && tar xvf HTMLTemplate-1.4.2.tar.gz
http://xappy.googlecode.com/files/xappy-0.5.tar.gz && tar xvf xappy-0.5.tar.gz
cd HTMLTemplate-1.4.2/ && python setup.py install
cd ../xappy-0.5 && python setup.py install
cd ../flax/src && python startflax.py --set-admin-password

They recommend starting startflax.py with --conf-dir=. to get it reading conf files held within the same directory, the python scripts seem to look in data/conf/ so simply issue:
mv *.conf data/conf/ (or cp *.conf data/conf/, or cd into data/conf/ and make some symlinks… your choice)

Then finally:
python startflax.py

Browse to http://localhost:8090/admin and login with the username ‘admin’ and the password that you configured, et voila! It’s all yours.

These instructions assume you have all the dependencies and tools necessary, the first apt-get should get you mostly sorted though. YMMV!

1 Giant Leap – Nothing Wrong With Me

I was having a hard time figuring out how to migrate Cacti from one box to another, or more specifically I could do the migration by the book but nothing was working, so after much head scratching and googling I’m going to collate my notes here. This assumes Debian to Debian, and we’ll call the boxes oldcacti and newcacti.

Step 1 is to setup Cacti, RRDTool, SNMP etc on newcacti. This is outside the scope of this post.

Step 2 is to migrate your database, so dump it, scp it to newcacti:

oldcacti:/etc/cacti# mysqldump -u cacti -p > cacti.sql
Enter password:
oldcacti:/etc/cacti# scp cacti.sql root@newcacti:/etc/cacti/
root@newcacti's password:
cacti.sql 100% 463KB 462.5KB/s 00:00
oldcacti:/etc/cacti#

and restore it:

newcacti:/etc/cacti# mysql -u cacti -p < cacti.sql
Enter password:
newcacti:/etc/cacti#

Step 3 All of your settings should now be across, but you'll likely want to keep your RRD history too. Unfortunately you cannot just scp the files across as you'll likely get complaints about the RRD files being created on a different arch:

COMMENT:From 2008/06/18 14:05:32 To 2008/06/19 14:05:32\c
ERROR: Garbage ':05:32 To 2008/06/19 14:05:32\c' after command:
COMMENT:From 2008/06/18 14:05:32 To 2008/06/19 14:05:32\c
ERROR: This RRD was created on other architecture
ERROR: This RRD was created on other architecture
ERROR: This RRD was created on other architecture
ERROR: This RRD was created on other architecture
ERROR: Garbage ':05:41 To 2008/06/19 14:05:41\c' after command:
COMMENT:From 2008/06/18 14:05:41 To 2008/06/19 14:05:41\c
ERROR: Garbage ':05:41 To 2008/06/19 14:05:41\c' after command:

So we need to export the files to xml and then reimport them:

rrdtool dump filename.rrd > filename.xml
rrdtool restore filename.xml > filename.rrd

But if you’ve got dozens, hundreds or even thousands of RRD files, doing them one by one is going to get tired, very quickly. So here we go:

oldcacti:/usr/share/cacti/site/rra/# ls -1 *.rrd | awk '{print "rrdtool dump "$1" > "$1".xml"}' | sh -x
scp *.xml root@newcacti:/usr/share/cacti/site/rra/

followed by:

newcacti:/usr/share/cacti/site/rra/# rm *.rrd
newcacti:/usr/share/cacti/site/rra/# ls -1 *.rrd.xml | sed 's/\.xml//' | awk '{print "rrdtool restore "$1".xml "$1}' | sh -x
newcacti:/usr/share/cacti/site/rra/# rm *.xml
newcacti:/usr/share/cacti/site/rra/# chown www-data:www-data *

That should be it, the final chown ensures that Apache can actually open the files and present them. Assuming Cacti et al is configured right on newcacti, you should be seeing some pretty graphs

Jake Shimabukuro – Time after Time

So yesterday I was bored and contemplating some discussions we’ve been having around our team at work. After some complicated backhistory, our office had replaced its perfectly fine Linux based firewall with a string of Juniper Netscreens, which can be best described as prohibitively complicated and increasingly useless. We cannot do any decent logging, graphing or statistics. We cannot easily put in a VPN to a client. We cannot maintain reliability of our own VPN services. We are tied to a platform that simply isn’t working, is wasting a lot of time to maintain and simply providing no business value. And for what? The illusion that hey, we get support from Juniper, and we pay money so it must be better! etc

Now, geeks and tinkerers will all yell out loud about how you can just get commodity PC hardware from a pile of decommissioned junk and throw in some cheap $5 NICs and install TEH LUNIX! Linux will save the day, linux will feed your cat and pleasure your wife in ways the Kama Sutra could only dream about, blah blah blahnix, while I work in my basement getting a realtek driver ported to an older kernel so that I can get 0.0001% performance out of this 486! IN LUNIX!

And they’re right, you can recycle an old box with a couple of NICs and make yourself a very powerful router and firewall using something like Smoothwall, or, if you have a bit more grunt, Clarkconnect or Untangle.

But we’re talking corporate level stuff here. All the advantages of no vendor lock-in with all the performance of corporate level gear. And a Celeron with a few realtek cards from Dick Smith simply won’t cut it. Especially when you’re talking multiple gigabit ethernet connections which will completely flood an ancient PCI bus.

We specifically have a need for some 18 ports of routing, some of which can get by with plain old 10/100, but most if not all should be GbE if possible. So, you’re looking at a PCIe bus and maybe these, Intel Pro/1000 PT Quad PCIe cards. Also for the kind of theoretical maximum throughput, you’re looking at a CPU over 3GHz preferably.

Anyone got a 3GHz+ box with multiple PCIe slots just lying about? Didn’t think so.

So I was looking around for specific products that achieved this; looking specifically at SBC, PC/104, m-ITX etc with a scope for scalability, rack-mountability, and the ability to be used for other tasks such as graphing, logging, SNMP, DHCP, DNS caching and Transparent Squid proxying. I was disheartened to find no such devices at the easy end of a google search and contemplated rolling my own solution and on-selling it. There was plenty with lots of 10/100 ports, a few with GbE but only a PCI slot… none with quite the right combination.

Then I found this:

It’s almost perfect. Using a mix of PCI and PCIe, you could max that out with 10x GbE ports and 8x 10/100 ports. Throw in a mini-PCI VPN accelerator, a hard drive for logs and caching, then either hand craft OpenBSD or install pfsense. If you need more ports or redundancy, configure another one and link the two together using CARP.

I would also recommend maxing out the CPU at the fastest that the board can take, as well as maxing out the memory with a decent brand (crucial, mushkin). Sure, you could spec it lower and upgrade further down the track, but on the other hand will the components still be around when it comes time to upgrade? Max it out now and you should get a considerable life-time out of the device.

I’m still waiting on a local distributor to get back to me with a price, but it’s promising.

Pendulum – 9000 Miles

So yesterday at work while sipping away at my godawful morning coffee, a ticket came into our fault logging system. Reverse DNS lookups, it said, were a little on the slow side. Oh great, I thought, children dying of starvation in third world countries, I’m all out of whiskey, and all this user can obsess about is DNS response times.

So I replicated the fault. Getting more interested, I jumped onto our primary DNS server and checked top and ps, nothing seemed out of the ordinary, so I restarted bind and watched what happened. For about ten seconds, everything behaved properly, then the lookups slowed back down again.

Double Ewe Tee Eff? rm /home/user/rawiri/sarcasm, now I’m interested.

So I tail -f‘ed /var/log/bind9-query.log and I was surprised to see that our ticketing system server was hammering DNS, trying to lookup for the same IP (192.168.0.15 – important to the story) a few thousand times per second. So I jumped onto the ticketing server, which also doubles as a syslog server, and issued a grep -r 192.168.0.15 /var/log/*

This spat up thousands upon thousands of results, from logs stored there by our firewall, sourced from an IP range assigned to our office in Jakarta, trying to get in touch with 192.168.0.15 on port 2048. Something in Jakarta was hammering our firewall, which was logging this on our ticketing server, which was hammering our DNS server trying to figure out who or what 192.168.0.15 is. Cool.

So I jumped onto the two boxes in our Jakarta office (at about 3am their time) and noticed that our Jakartan IT colleague had setup a peered/sibling Squid proxy configuration, which was very cool, however netstat -ap | grep 2048 on both boxes revealed that Squid was doing the dirty on this port. So I went into /etc/squid/ and issued a grep 192.168.0.15 *, the results were amusing:

wccp_router = 192.168.0.15

Back in the day, we had a cisco router on that IP address, and our NZ Squid server was configured to point to it on the wccp protocol to offer some transparency. That router has been gone for several months now, but the squid.conf was never updated to match this. Our Jakartan colleague had inherited our squid.conf file to setup his first proxy, and that was trying to poll the old cisco router. The wccp heartbeats fell into the background chatter of the firewall logs, and it wasn’t until the peered proxy configuration was put in place that things really started hammering away: the secondary proxy would try to heartbeat via the primary proxy, which would also try to heartbeat, resulting in an avalanche of wccp traffic slamming into our firewall.

Disabling that setting on two proxies in Jakarta, and two proxies in NZ, before restarting squid on the lot and voila! DNS in NZ started behaving again.

That was a pretty impressive catch, well I thought at least. But today I caught out a three-way IP conflict that was preventing a VPN from coming up, as well as an nfsmapid issue that was affecting all of our Solaris boxes, all by watching the appropriate logs. Three big catches, two days.

With these outstanding issues sorted, we were able to hammer away at dependant faults and got a lot of stuff resolved and out of our too-hard basket.

Related: To anyone wanting to try out the Octopussy log frontend on Debian – beware! It’s really designed to be a standalone product. If you install it on a server that has Apache configured for multiple services already, you might find that it will break stuff, remove the default site and rewrite your conf files.

Concord Dawn – Bitch Killer

I have a HP G2 DL380 which I got to replace a handful of rack servers with one box, and compared to its predecessor, the G2 hardware is absolutely sublime. There was something not quite right about beige era Compaq equipment, and so the G1 came off as simply godawful. Anyway, the other day I got my G2 out of storage so that I could set it up for some testing and development work.

Unfortunately the DL380 sounds like an air raid siren, and doesn’t abide by standard ACPI/IPMI, so there was little chance of getting it to shut up in FreeBSD, unless I ran FreeBSD-5.x or maybe 6 plus taking my chances with compat5x.

Instead I’ve thrown on Debian Etch, grabbed the appropriate .deb from HP – you’re looking for hpasm, but their other packages might be of interest for array diagnostics etc. I dropped this into /root and issued the following:

apt-get update
apt-get install snmpd libstdc++2.10-glibc2.2
cd /root
dpkg -i hpasm-7.8.0-100.etch26.i386.deb
hpasm activate

It will fire up and take you through a scripted setup, simply answer as best you can and it will slow those fans down to a pleasant hum. Now you can go on using your DL380 without fearing attack from the Japanese

You may also need to add
/opt/compaq/hpasmd/bin/hpasmd activate
to /etc/rc.local or similar to get this to automatically fire up on boot. YMMV.

Billboards Top 100 – Sandpipers – Guantanamera

I was enjoying a cruisy day at work, browsing some airline sites to try and find a cheap flight to Auckland – after a 40 hour epic trip back from Belgium, I have zero interest in being in a car for 10 hours, I’ll take the one hour flight thanks – anyway, I got a text from Tamati reading Could not start kstartupconfig. Check your installation. WTF?.

oh.yay.

The short of it is that he had created a seperate account for his girlfriend and since then things had been severely bung. As he’s on Kubuntu, I figured I’d just upgrade him from Feisty to Gutsy and that would be that. CTRL+ALT+F2, logged in as him and issued the following:

sudo vi /etc/apt/sources.list
:%s/feisty/gutsy/g
:wq!
sudo apt-get update && apt-get dist-upgrade

After a couple of hours of waiting, alas the problem remained. Nothing in the logs, nothing in any tails. I figured I’d try to install gnome and see if that was any different

sudo apt-get install ubuntu-desktop

After some more waiting, a nice Gnome login screen appeared, the login attempt failed albeit differently. Fortunately this time, gtk and X were a lot more forthcoming with useful information in the logs. Essentially Tamati’s profile was trying to build itself within /home instead of /home/tamati where it should be. So something like this:

cat /etc/passwd | grep tamati
tamati:*:1001:1001:Tamati Blundell:/home:/bin/bash


Well then, I simply had to adjust the line to read tamati:*:1001:1001:Tamati Blundell:/home/tamati:/bin/bash

And we were away… roughly. There’s still a few loose ends to tie up with the interface, but both Gnome and KDE are now working… sorta.