The issue described in the previous post has now been resolved. No more NFS, no more ugly errors its malfunction ensued. And as a side-benefit, pages now load quicker. Horray!
(We promise the next post would be less geeky)
At around 7:00pm GMT November 21, our website became inaccessible for 1 hour. Again, we are deeply sorry for the downtime. What happened was that our Network File system became unmountable and needed a reboot.The majority of data are stored in the database. The rest are caches of temporary data in the form of files.When you have one web server, you can put all those files into that server’s local file system. When you have more than one, you need to put the files where all the web servers can access.One of the easier ways is to use a Network File system (NFS). With NFS, the web servers will the folders on the shared file server as if they are local.The biggest drawback of NFS is that, when something goes wrong, and the shared file server became unmountable, the web servers will stop, as if there’s something wrong with their “real” local file systems. In short, it drags everyone down.To make things worse, with this type of error, we cannot display anything friendlier than the discomfortingly-worded “Service unavailable : Internal error : 132″ page.We have been moving away from NFS recently but there are still some older stuffs that we haven’t converted. Rest assured we’ll be speeding up to achieve independence from NFS these couple days.
At around 5:50pm GMT November 14, our website became inaccessible for 8 hours. We are deeply sorry for the downtime.
Here’s what happened:
What should be a minor issue turned into a major downtime (8 long hours!) simply because we were not alerted to it quickly enough. Definitely not something to be proud of. We have now wised up and included this type of issues into our monitoring and alert system.
We’ve always taken extra care to protect your data and make sure the hardware is always-on. We are using this incident as a wake-up call to ensure our software gets the same level of care as well.
Whenever I visit someone’s shelf, the first thing I look is how much our taste overlaps. This is what aNobii used to show:
Easy to understand but sometimes too simplistic to be helpful. Perhaps I’ve read a book three times while the other person got it as a gift and left it to gather dust ever since. Perhaps I found a book a total waste of time while the person hailed it as the best thing written ever. Not to forget that the absence of the same book could sometimes speak louder than its presence.
So we’ve spiced things up by including a slew of different ingredients into cooking up the “taste compatibility” score. Here’s the new look (heavily influenced by Last.fm’s take):
We’ve tested around before settling with the current formula but it’s by no means final. Let us know what you think of it!
Blog