Partial Downtime. TAO.

Everbody’s favourite Linux server Tao was down for ten minutes today between 13:40 and 13:50.
Websites and Email on that domain were effected. It’s back up now, and we’re investigating the cause of the problem.

Datacentre Downtime

Made Media customers experienced 50 minutes of unplanned downtime today between 1pm and 2pm. This affected all websites and email hosted by us. We’re always making a big deal about how reliable our hosting service is, so an explanation is due.

Ironically, the power outage was caused by a fault in the Uninterruptible Power Supply at our datacentre. This is the bit of kit that is supposed to ensure that we keep serving in the case of a power outage. Engineers at the data centre have switched to standard mains power whilst they investigate the issue and bring the UPS back online. They are not expecting any further power outages when they bring it back into the loop.

The issue affected the whole of our floor at the data centre. Happily once the power was restored all of our servers all just woke up and started serving again. We’ve checked for database corruption on all domains and everything is A OK.

We apologise for any inconvenience caused—that’s the first 50 minutes of unplanned downtime in more than twelve months.

iPhone announced – for real

The Apple iPhone has finally been announced and, incredibly, it looks like it might just live up to the [massive] hype.

Engadget has a pretty comprehensive write up but, in short, this looks like a fantastic device. In addition to a terrific looking phone interface, it's also an iPod with an on-screen controller. Running OSX, it includes iTunes, Safari, a mail client that looks just like Mail on my Powerbook. Add to that a gesture-based touchscreen, bluetooth, wifi and half-decent battery life then it looks like a winner.

Sadly we'll have to wait until 'Quarter 4' for a European launch but you can count me in for an order as soon as they'll take my money.

UPDATE: The BBC puts edited highlights up for viewing within minutes. This is pretty impressive as it's online before Apple themselves have made the keynote generally available. When they do (soon) it'll be here.

Has Telewest bricked 45,000 TV Drives? – UPDATE: No!

UPDATE: I'm thrilled to say that service was restored at 8.33pm. Others are not yet so lucky but things are looking good. The Digital Spy Discussion forums have the latest info.

At around 2pm today my TV Drive suddenly took away Soccer Saturday and replaced it with a screen saying "Important: Do not switch off your set top box as it is being updated".

Two hours laters when nothing had changed, it was clear something had gone horribly wrong. After a 40 minute phone queue wait, I was told that this affects EVERY TV Drive user in the country and there is no estimated time to fix. They hope that they can put things right within 24 hours but no promises.

Of course, if they've borked the firmware with a bug-ridden update - and pushed that to every TV Drive in the country then the fix might take a lot longer and be much more expensive...

For updates check out Telewest/Blueyonder status (next update 9pm is the message at the time of writing) and the Digital Spy Discussion forum.

Wonder how long I'll have to wait until I can watch TV again?

mmNet. Now with less spam

You can’t have failed to notice the meteoric rise in spam in recent months. Many commentators characterise the fluctuations in spam rates as an arms race. Spammers come up with new techniques, the anti-spam houses respond with new rules. Just recently the spammers have started winning the race again. The worst of the new breed of spam are image-based, stock scams because they are not susceptible to most common spam-catching techniques. There has also been a rise in advertising for – ahem – ‘upsizing’ products as well as the usual pharmaceuticals.

We know that some of our customers have been struggling under the load recently, and we have been working to improve the situation. We’ve finally brought it under control, to the point where we are now seeing just one or two unwanted emails slipping through the net per day.

Here’s what we’re doing:

1. We’re now checking all incoming mail against the SpamHaus SBL and XBL list, and rejecting mail from IP addresses listed in their database. Using IP blacklists can be controversial, but SpamHaus make a real effort to be fair and avoid blacklisting innocent IPs. This gets rid of about 50% of spam before it even gets near your email account.

1. We’ve tweaked the SpamAssassin rules on our server so that they are now much more accurate. We’ve even managed to successfully detect image spam. We’re seeing close to zero false-positives though, so we’re not just blocking images indiscriminately.

   What you should do:

2. Set up a filter rule. Whenever we detect a message that we suspect is spam we mark the subject line, so that it begins: [spam?] . Most email clients will allow you to easily create a folder/mailbox called ‘spam’ and set up a rule to divert all mail marked [spam?] to that folder. You can then review the folder periodically to check that no legitimate mail has slipped in. Doing this should leave you with next to no spam in your in-box. If you’re lucky enough to have an e-mail client with Bayes spam filtering built in then it should begin locking onto the word [spam?] automatically, so you may not even need to add a rule.

1. Switch off catchalls. If you are still using a catch-all email address (an address which picks up mail for unknown names at your domain) now is the time to switch it off. Spammers often just guess at the first part of email addresses, so catch-all addresses are a magnet for spam. It’s much better to bounce email to unknown addresses. That way, even if someone does spell your name wrong, they will know about it.

   What if you’re still getting spam?

   If you’re still receiving unmarked spam, just forward it on to: spamcop@mmnet.co.uk. We’ll take a look and see if we can tweak the rules to prevent mail like that from slipping through in the future. At the very least it will be added to our Bayes database.

   What if a genuine email gets marked [spam?]

   Forward it on to spamcop@mmnet.co.uk. We’ll recognise that the email has been marked incorrectly, and add it to the ‘good’ Bayes database . This will help to prevent similar mis-identification in the future.

   What about viruses?

   Happily the SpamHaus SBL/XBL list is catching most viruses and phishing schemes before they get near your email account. If one does slip through, the Dr Web service kicks in and isolates the email.

   In the spam arms race we’ve just taken the advantage back. Hopefully our clients can breathe a sigh of relief until the spammers come up with something new. We’re also working on some black secret technology to provide the same relief to people who don’t even host their email with us. Watch this space.

Apple hosts our Jobs Board Digest widget

We submitted our new Jobs Board Digest widget to Apple just yesterday. We assumed it would take a while to process but no. Browsing our referrers this morning I spotted this.

That's impressively efficient on Apple's part.

Jobs Board Digest – now with more jobs

Jobs Board Digest continues to grow. We’ve just added a few extra job boards and given the design a much-needed refresh. We’re now providing a combined listing and rss feed for the job boards from:

• 37 Signals
• Authentic Jobs
• Boxes and Arrows
• CocoaDev
• Joel On Software
• Krop
• Think Vitamin

And, for those of you who appreciate such things there is now a Dashboard Widget too.

Free ISP-in-a-box ISO

If you need to manage more than a handful of domains on your own box, and you’re not man enough to do everything the raw way (I’m not), you probably use some kind of virtual host control panel software. At Made Media, we use Plesk. It has its quirks, but it’s one of the best. It’s not cheap though, and you could argue it’s a little over-complex.

But we didn’t get into Open Source to pay for stuff did we?

A bit of aimless web surfing led me to a post on Scripting News, where Dave Weiner reveals that he picked up a Cobalt Cube from eBay for $125. This got me googling for Cobalt stuff.

For the uninitiated, Cobalt were the first company to really push the idea of Linux-based Internet Appliances. While the Cube was a kind of all-in-one Linux-based Exchange equivalent for the tech savvy office, the RaQ was a Web Hosting service appliance. It was pretty-much the fore-bearer of the explosion of data-centre ‘blade’ servers, and also the grand-daddy of hosting control panels like Plesk, Ensim, Cpanel et al. With it you could manage up to 100 websites on a tiny purple box, with a web-based control panel that made it easy to configure domains, and even pass a level of control out to your clients.

Sun snapped up Cobalt in 2000 but didn’t develop the products a whole lot further before discontinuing the line in 2003. By that time the rest of the web-hosting scene had caught onto the idea and were developing similar systems on top of white-box hardware.

And there the story ended. Or so I thought, until I did a bit of googling last night.

It turns out that in a typical exmple of doing the right thing, Sun open-sourced the Cobalt software. A crack team of Japanese hackers have been working on the code ever since, resulting in an Open Source version, Blue Quartz.

Even better, some bright people have got it working on top of CentOS, which is a hardened, community-supported clone of Red Hat Enterprise Linux (which we use on our main webservers at Made). The great advantage of this is that you can use the latest apache, mysql and PHP, and security fixes are a simple ‘yum update’ away.

But ultimately, the thing that really tweaked my interest is that one enterprising Linux Hacker has produced an all-in-one, free combined ISO installer that will install the whole lot on pretty-much any old PC in around 20 minutes.

So from reading Dave Wiener’s post, to having a fully-running, control panelled, reseller web-host box in my spare room took about two-hours all-in. The computer was an ancient 500mhz celeron machine that really needs freecycling. As it happens the fan-noise is irritating so this little project will probably go no further. But for charities, companies on a budget, or just garage enthusiasts this offers a great opportunity to run their own budget web-server in the spirit of open source.

Take a look!

Server Lover – Apply Within

Due to continued expansion of our technical offerings, we are in desperate need of a full time Network Support Manager to join our small, but perfectly-formed team in Birmingham, England. You will keep our network running at 99.9% availability, applying patches, maintaining backup procedures, negotiating bandwidth, doing fresh server installs, and supporting the development team. You will also provide first-line support to our clients, both for hosting services and for our own web applications. You possess the patience of a saint, the organisational skills of a librarian, enough charm and telephone manner to soothe the most irate, confused client, and experience with some combination of the following:

Linux, Apache, MySQL, Shell Scripting, Plesk, OS X, Windows, Subversion, Ruby-on-Rails, SpamAssassin, Hardware, RAID, M0n0wall, Cisco Routers, Rsync, DNS

Our systems are well documented, and we can provide training on most aspects of the job, so we’ll consider candidates from a wide range of commercial backgrounds. Salary will be dependent upon your level of experience.

Please send your CV and a covering email to tim.kitchen@mademedia.co.uk. Sorry – no agencies please.

Big in Poland

A small spike in Polish traffic to our blog, lead me to this article, discussing my MyActiveRecord MySQL/Object Mapping class.

Jana, who is after all a Slav, had a go at translating. She assures me that the post signs off:

This is one of the best PHP Classes I've ever seen. It just works!

I guess that's better than:

This is one of the worst PHP Classes I've ever seen. It doesn't work!

But perhaps the wife is just trying to bolster my ego...

• apple (4)
• christmas (7)
• culture (35)
• design (21)
• diarised (8)
• events (1)
• javascript (1)
• life (43)
• linux (6)
• local (37)
• news (83)
• php (6)
• projects (48)
• technology (27)

• January 2012
• December 2011
• November 2011
• October 2011
• September 2011
• August 2011
• July 2011
• June 2011
• May 2011
• March 2011
• January 2011
• December 2010
• November 2010
• October 2010
• July 2010
• April 2010
• February 2010
• January 2010
• December 2009
• October 2009
• September 2009
• August 2009
• July 2009
• April 2009
• March 2009
• February 2009
• December 2008
• November 2008
• September 2008
• July 2008
• June 2008
• April 2008
• March 2008
• February 2008
• January 2008
• December 2007
• October 2007
• September 2007
• August 2007
• June 2007
• May 2007
• April 2007
• March 2007
• February 2007
• January 2007
• December 2006
• November 2006
• October 2006
• September 2006
• June 2006
• March 2006
• February 2006
• March 2005
• October 2004