A New Day At ArtSites.com

[Aug 7, 2006] ArtSites.com has recently recovered from a disastrous server crash. Most of our content is quite old because recent backups of the website were also lost due to other issues. The crash affected all ArtSites.com clients, causing their websites to be down for several weeks in some cases, and their now-restored sites are also not up to date in many cases. Given the hardship this issue has caused so many, I would not blame any of our clients if they left ArtSites.com to seek Web services elsewhere. There certainly are plenty of other Web developers, website designers, hosting providers and SEO professionals to choose from--and yet, not a single client has chosen to leave us. In most cases, they have even been highly supportive--disappointed by the downtime, to be sure, but more than willing to stick it out with us.

Why have our clients chosen to stay? For one thing, there is surely an element of loyalty involved. I think more important, however, the Web services that we provide to independent artists are uniquely targeted to their needs and desires. The site administration pages that we provide to our clients, for example, enable them to manage the contents of their websites to their heart's content--from adding and editing copy to tweaking the finest details of the images they upload. And, considering the freedom and power that we offer our clients to control their websites' contents, our services are very economical and simple to use. There is more to it than that, of course, but only clients who have actually used our services can fully appreciate all of the benefits that we offer to them.

The recent server debacle, however, has been an unacceptable exception. So I think a little explanation is in order as to what happened and why it won't happen again.

In this day and age it seems inexcusable for a website to go down--and to stay down for as long as several weeks! That's what I would think, anyway. Before the crash, our dedicated server had been up for over 440 days straight; 100% uptime for well over a year. That's enough to become quite confident in a service and to even become rather blasť about it. Before the crash, I would have felt mostly contempt for any would-be Web host who had my story to tell, but what I didn't know was a lot! And I can tell you this: I now have a whole new outlook on disaster recovery strategies. We all know how important it is to backup computer data for when it is needed for disaster recovery--and we all know that such a disaster should be counted as an inevitability. Anyone who has messed around with computers for long enough has learned this lesson first hand. Keeping current backups is a routine imperative for everyone who keeps data on a computer at home or in the office--and especially on a Web server. Such disasters have stricken me more than once, locally, so I certainly know better than to be nonchalant about backup.

In fact, ArtSites.com had dual backup strategies before the crash--and let this be a lesson to any other Web developers who happen to stop by here. One backup was an automatic daily process that stored the backup on a remote server. The other was only a manual backup that I would perform occasionally, compressing and downloading a host of files and database records to a local hard drive. Both backup systems failed. Ultimately, the failures were due to my own inattention, of course. Since implementing them when the old server had first been set up, I hadn't consistently tested them. That is, I hadn't unpacked them and put them on a server to actually see them work. I was content that the daily backups were there, and if that failed for some remote reason there would be the manual backup--allowing an "acceptable" save, even if it had been a month since I'd thought to do it.

As it happens, the manual backups that I made were corrupted. I learned this after initially becoming bewildered that the automatic backups were not recoverable, either. After wasting too much time on fruitless efforts of recovering the backups, I still don't have all the answers to what went wrong, but I do have some good clues--and now I really know better how to properly prepare for disaster recovery in the future.

Ultimately, the recent recovery of all ArtSites.com client sites required resorting to local files that had been kept since performing major upgrades to everyone's sites last year(!)--and from even the initial uploads for newer clients--synchronizing these as best as possible what database records were also stored locally. It was a crazily grueling weeks-long task of piecing together files that produced inferior results in the end when it should have taken only a few seconds and little more effort than typing "gunzip" to achieve perfection. Actually, it needn't have happened at all had we had collocated servers. I've dreamt of colloated servers for a long time, but they just seemed way too expensive. Now it doesn't. Unfortunately, as the case was, the resources that I finally had to resort to did not include any content and images that clients had added to their sites in the meantime. To get the sites as up to date as possible, I was reduced to turning to such resources as archive.org and Google caches.

It was while looking at the Google caches of some client Show Calendar pages that I got a major clue of what may have happened to the automatic backups. The way the Show Calendar works, by default, it displays only current and future shows on an artist's schedule. To keep the calendar current, past shows are not displayed to website visitors. And yet recent Google caches of various client Show Calendars included shows dating back to 2002! Besides offering a clue to what happened to the backups, I have to add that seeing those caches was somewhat embarrassing because, in several cases, they included even the first show entry that I typically make in a new implementation of the Show Calendar in order to test the calendar program on a new site. These test entries often include a "show description" that is actually basic instructions for the client about how to manage show listings...and, maybe, a personal comment. In a few cases, for clients with whom I am particularly familiar, I've also included a not necessarily appropriate image to demonstrate the Show Calendar's capacity to include a picture with each listing. When making these test/instructional entries I always use a past date so they can only be seen from the client's administration pages and are not publicly viewable. But in Google's caches, there they were!

That's when the realization about the backups came. The clock on the server must have been off at the time Google had crawled those calendars. The most likely way for that to happen would be if the clock battery were dead when the server had to be restarted. Without a live battery, the server's default date upon restarting would be January 1, 1970 and run up from there. The automatic backups with such old dates were deleted by default as being too old to be worth keeping. That's my partial guess at the moment, anyway. It still leaves questions unanswered, and logs and scripts from the old server are no longer accessible. In any case, the old backup plan was clearly too marginal. There is much more to the whole episode, of course, but who really wants to hear more? Besides, just like backing up computer data, everyone knows better than to work through a lightning storm, even if your back is against a wall, right?

The good news is that, wow, this whole server debacle has been brilliantly educational! It has shed so much should-have-known-better light on many subjects. Not all of them--heck--not even the most important ones pertaining simply to running a fast, secure and reliable Web server. Now that the nightmare is over, I have found great broadly-reaching inspiration in it. Tomorrow, for example, I'll tell you about ArtSites.com new disaster recovery strategy--a shining example of such, I think.

--Patrick


Other ArtSites News:


ArtSites.com Domain Name for Sale

[Apr 9, 2016] ArtSites.com, ArtSites.net and ArtSites.org domain names are for sale. ...read more

Artist Web Services: Major Update for Artists

[Nov 15, 2006] If you are an artist who is serious about representing your art on the Internet, we're trying to make it as obvious as possible for you to consider ArtSites as your first choice in a Web Developer. ...read more

Upgrade to XHTML Strict

[Sep 25, 2006] Returning visitors may notice some recent changes in the ArtSites.com website. Most noticeable might be the look of the site, but the new graphic design is a tiny part of what's new. The biggest upgrade, by far, is that the entire site has been rewritten in XHTML. ...read more

ArtSites Client Backup Program

[Sep 5, 2006] ArtSites has added a new and powerful feature to its offerings. All websites develop by ArtSites now include a Client Backup Program. With a single click, you can back-up your entire site. ...read more

ArtSites Disaster Recovery Strategy

[Aug 8, 2006] The menu name for this news article is Aster because a proper disaster recovery strategy makes even that which most might call a disaster into an event that is easily taken in stride, removing the 'dis' from disaster, leaving only the aster--a shining star. ...read more

A New Day At ArtSites.com

[Aug 7, 2006] ArtSites.com has recently recovered from a disastrous server crash. Most of our content is quite old because recent backups of the website were also lost due to other issues. ...read more

The Cobbler's Kids Get Shoes

[Dec 29, 2004] For the past several years, ArtSites has focused all of its attention on developing some darn nice websites for independent artists. And all of our efforts have gone into that. The benefit to our clients is that ArtSites can now deliver a very powerful Web presence to artists, and we can do it with great economy. ...read more


Featured Artist:

Pottery and ShoholaBells
by David Greenbaum

Pottery and ShoholaBells by
David Greenbaum