Sting & Eric Clapton – It’s Probably Me

There are times when working in a server room where there is an equipment need that you simply cannot justify in either a business or common sense way. We recently had a project to move all of our fibre optic based equipment to the same rack cabinet, where we could seperate and secure the delicate fibre optic from the network, management network, and power cables.

Unfortunately, most equipment out there is for splicing, which is not what we want. We want a simple way to secure slack and ensure correct bend radius specs are adhered to. We went through a few potential vendors of cable management gear trying to find the best fibre management equipment for our realistically simple needs, and each of them had products, but in the few hundreds of US dollars. To make sure some fibre optic cable bends correctly. Yeah, just let that sink in for a while.

The other problem is that the non-technical people with control of the company purse strings did not consider this a critical expenditure, no matter how often we pointed out that fibre is extremely delicate, very expensive and also the most used equipment at present. And, should it fail, productivity will come to a screaming halt. Subsequently we won’t put out products on time, and people won’t get paid. “It is not a critical expense. My shoes/affair/drug-habit is a critical expense!” *sigh*

Today I went to my favourite hardware store: Mitre 10 Mega, and while buying other stuff for the new house I browsed around to see if anything inspired me. There’s actually a wide variety of options to suit this requirement, including large door handles, surface drainage, pulley wheels etc I eventually settled on some plain wheels, 150mm with a 38mm hub width.

Cutting off the rubber wheels reveals some concave hubs – perfect for keeping fibre in place, and at about the perfect size for a gentle bend radius that is well beyond the minimum spec, while not being too big for the rack cabinet for which it is intended. And the best thing: NZD$22 for two. If your mounting method allowed, you could potentially buy one wheel, cut it in half on a bandsaw (or similar) and use it like that to manage the bends.

My plan is to ream out the holes that I have drilled in the spokes, and use them to pass through some velcro cable ties to make sure that the fibre stays on the spools. I’m still undecided on how to affix it to the rack cabinet – either with a nut and bolt or with velcro, of which we have a large boxful.

Expect an update when this is put into production

What cost-saving common-sense solutions have you readers ever put in place in a production environment?

Pendulum – 9000 Miles

So yesterday at work while sipping away at my godawful morning coffee, a ticket came into our fault logging system. Reverse DNS lookups, it said, were a little on the slow side. Oh great, I thought, children dying of starvation in third world countries, I’m all out of whiskey, and all this user can obsess about is DNS response times.

So I replicated the fault. Getting more interested, I jumped onto our primary DNS server and checked top and ps, nothing seemed out of the ordinary, so I restarted bind and watched what happened. For about ten seconds, everything behaved properly, then the lookups slowed back down again.

Double Ewe Tee Eff? rm /home/user/rawiri/sarcasm, now I’m interested.

So I tail -f‘ed /var/log/bind9-query.log and I was surprised to see that our ticketing system server was hammering DNS, trying to lookup for the same IP (192.168.0.15 – important to the story) a few thousand times per second. So I jumped onto the ticketing server, which also doubles as a syslog server, and issued a grep -r 192.168.0.15 /var/log/*

This spat up thousands upon thousands of results, from logs stored there by our firewall, sourced from an IP range assigned to our office in Jakarta, trying to get in touch with 192.168.0.15 on port 2048. Something in Jakarta was hammering our firewall, which was logging this on our ticketing server, which was hammering our DNS server trying to figure out who or what 192.168.0.15 is. Cool.

So I jumped onto the two boxes in our Jakarta office (at about 3am their time) and noticed that our Jakartan IT colleague had setup a peered/sibling Squid proxy configuration, which was very cool, however netstat -ap | grep 2048 on both boxes revealed that Squid was doing the dirty on this port. So I went into /etc/squid/ and issued a grep 192.168.0.15 *, the results were amusing:

wccp_router = 192.168.0.15

Back in the day, we had a cisco router on that IP address, and our NZ Squid server was configured to point to it on the wccp protocol to offer some transparency. That router has been gone for several months now, but the squid.conf was never updated to match this. Our Jakartan colleague had inherited our squid.conf file to setup his first proxy, and that was trying to poll the old cisco router. The wccp heartbeats fell into the background chatter of the firewall logs, and it wasn’t until the peered proxy configuration was put in place that things really started hammering away: the secondary proxy would try to heartbeat via the primary proxy, which would also try to heartbeat, resulting in an avalanche of wccp traffic slamming into our firewall.

Disabling that setting on two proxies in Jakarta, and two proxies in NZ, before restarting squid on the lot and voila! DNS in NZ started behaving again.

That was a pretty impressive catch, well I thought at least. But today I caught out a three-way IP conflict that was preventing a VPN from coming up, as well as an nfsmapid issue that was affecting all of our Solaris boxes, all by watching the appropriate logs. Three big catches, two days.

With these outstanding issues sorted, we were able to hammer away at dependant faults and got a lot of stuff resolved and out of our too-hard basket.

Related: To anyone wanting to try out the Octopussy log frontend on Debian – beware! It’s really designed to be a standalone product. If you install it on a server that has Apache configured for multiple services already, you might find that it will break stuff, remove the default site and rewrite your conf files.

Pendulum – 9000 Miles

Today was a pretty good day at work, with a lot of smaller jobs out of the way and a whole new feel around the office.

I got home to find out that my counter-culture superhero, George Carlin, had died at the age of 71.

Well, that sucks. He was one of my favourite comedians: I have an extensive library of his work, and I was hoping to see him perform on stage before he passed, but alas, it looks like that’s not going to happen now. Yes, I would have even put up with America’s airport security to see George Carlin at work.

George, if I may call you George, sir, this whiskey’s for you.

Dirty Vegas – Days Go By

Weight on a chemists scales on the Champs-Elysées in Paris: 118kg
Weight this morning on my calibrated scales: 93.9kg
Average loss: roughly 3kg a month or so.

Diet: Still as awful as ever, with fast food and junk food, however with an increased metabolism I’m eating smaller servings. Ok, I’ll be honest, I have been having something for breakfast every day, no matter what – even McDonald’s, but usually flavoured oatmeal. The point behind breakfast is getting your body chemistry, blood sugar, protein etc levels up and ready for the day. This causes you to average out your intake over the course of the day, or at least it should.

I’ve also been having slightly better dinners and far less alcohol consumption. So I’m not packing my gut with an over-abundance of starch, sugars, empty carbs/cals or fat.

Exercise: Same as always; the walking during my commute to and from work everyday, so.. maybe 3-4km every day. I’m now even more motivated to get back on the bike though, even taking this morning’s unfortunate news into account.

Weight Retention: Seems to be stable, judging by the other readings on my fangdangled scales. This means I can jump on my bike and shed some kilos quite easily in a short time. This, by the way, is the secret behind the claims on those infomercial exercise machines:

lose a billion pounds in 12 days or your money back!

Yeah… if you start exercising, one of the first things to go is water retention – two to three weeks of rapid shedding followed by a steep drop off in loss, which I think is often a big demotivator – because people see diminishing returns and don’t understand why.

Method:
Step One: Learn and embrace techniques to increase your metabolism.

Ministry Of Sound – Greg, Jeroenski & Roog – My Mind is Twisted

Well moving house took no time at all – a lot of stuff went to the Salvation Army across the street, I still have a bunch of stuff in the garage to sift through, but on the whole it went like clockwork.

Except, of course, for the internet connection. We got the WiMax antenna down, but the ability to move the ladder from the old place to the new place eludes us, so I have not been able to get onto the new roof and set it up. I’m posting this from work, because there’s a lull in my otherwise flat-out schedule.

So now I’m waiting on my older brother to get his van back from the shop, because I have to move a rack cabinet as well. Then we can get the ladder from the old place, then I can find a sunny moment to jump on the roof and setup the intergeek, and then all will be well in the world.

I have noticed though that I’m not really internet addicted, life goes on without chatting to people in DC or IM. What does suck though is if I’m potting around the house and I have an idea that I’d like to research – it sucks not being able to sit down and google away. I’ve also noticed that I’m a hypocrite: where workmates bemoaning not being able to access work from home would make me roll my eyes and suggest they get a life, I too have noticed that it’s a lot harder to clear out the night-time emails from my northern hemispherean colleagues.

So, hopefully I’ll get the internet connection back up and running sometime this week, then I can do some updates here

Concord Dawn – Bitch Killer

I have a HP G2 DL380 which I got to replace a handful of rack servers with one box, and compared to its predecessor, the G2 hardware is absolutely sublime. There was something not quite right about beige era Compaq equipment, and so the G1 came off as simply godawful. Anyway, the other day I got my G2 out of storage so that I could set it up for some testing and development work.

Unfortunately the DL380 sounds like an air raid siren, and doesn’t abide by standard ACPI/IPMI, so there was little chance of getting it to shut up in FreeBSD, unless I ran FreeBSD-5.x or maybe 6 plus taking my chances with compat5x.

Instead I’ve thrown on Debian Etch, grabbed the appropriate .deb from HP – you’re looking for hpasm, but their other packages might be of interest for array diagnostics etc. I dropped this into /root and issued the following:

apt-get update
apt-get install snmpd libstdc++2.10-glibc2.2
cd /root
dpkg -i hpasm-7.8.0-100.etch26.i386.deb
hpasm activate

It will fire up and take you through a scripted setup, simply answer as best you can and it will slow those fans down to a pleasant hum. Now you can go on using your DL380 without fearing attack from the Japanese

You may also need to add
/opt/compaq/hpasmd/bin/hpasmd activate
to /etc/rc.local or similar to get this to automatically fire up on boot. YMMV.