Rodent Regatta

This long article is going to call into question the issue of weblogs and scalability. It’s based on personal experience and flawed information flows. Yet, it’s going to raise the specter that shared hosting, years of writing and tools that backfire combine to make for a very fragile blogosphere.

Scalability is going to be an issue. A boom in weblogs began in 2002. Now, many weblogs have large numbers of entries, posts or articles. This begs questions of disk capacity, software performance, server performance and bandwidth impacts. In other words, questions of I.T. architecture, information architecture and user habits need to be understood and communicated effectively.

Instapundit has over 23,000 24,000 articles. This tiny weblog has between 4000 and 5000 articles. Together we have tens of thousands of readers. grins How do we optimize without stifling the desire to write? How do we write without bringing down a host? The Questions and Issues

1. [An urgent request]Does anyone know how to set up an archive page using Textpattern so that the web host doesn’t get hammered each time the archive page is refreshed?
2. Is it the number of articles in a weblog or the total amount of text in the weblog that is most likely to put an undue load on a host?
3. Does anyone know which (pre-release) version of Textpattern is better to use to minimize the impact on the host?
4. Is there a way to know in advance how many inquiries to a database a weblog might make, or must you run your own server to get those test results?
5. Standards-based web design is important, but it isn’t sufficient. What are the essential pieces of knowledge required to be a writer on the web?
6. I’ve only spent $600 on a lifetime of hosting with TextDrive. I spent some more commissioning a designer. Now, what are the roles & responsibilities of the host?
7. What metrics should I be watching and where are they?
8. What is the role and what are the responsibilities of the customer? Who is the customer? What does he know? What does he need to know?
9. What are the responsibilities of plugins and plugin authors?
10. Was the situation you’re about to read about really a silver bullet or am I merely someone who know longer starts their air conditioner at 5:00p.m. thereby lowering the load on the generating plant?
11. When will Textpattern 1.0 be finished? When will the TextPanel control panel for TextDrive be available? [Hint: “soon” is not an answer!]

Current Condition

This weblog has been offline. It had caused service disruptions at the web host, TextDrive. Rejoining the blogosphere without an archive page may prevent further problems, but it’s very difficult to be sure.

I’ve been told that ignorance of all things XHTML, CSS, hosting, plugins, templates, forms, sections, articles—and that list goes on and on—is no excuse for having harmed the web host and its customers. However, no one can (or is willing to) take the time to spell out the minimum body of knowledge required to write a weblog, have it hosted and be a good (technical) citizen on the web.

The Background

This particular problem began back in February when I returned to the web with a new design I had commissioned using Textpattern. However, the problem was called to my attention on June 4, 2005. Here’s the email I received from TextDrive’s team:

6/4/2005 Steve: Please have a quick look at [a link went here] [a text file name went here] and let us know how you’re going to resolve that issue where you’re running 7,401,285 queries in less than 24 hours (Yes, that’s 7.4 Million). And #2 looks to be yours as well. More details to follow from one of the other staff, but whatever it is needs to stop. [name withheld]

Eight minutes after receiving that email, I responded this way:

6/4/2005 [name withheld]: I’m totally clueless. Obviously, I see my name in lines one and two of that text file, but I don’t have a clue what they mean. Is my weblog doing something behind the scenes that is causing that? Is there some setting in Textpattern that I’ve got wrong. Be assured I’m ready and willing to be cooperative, but I don’t know what to look at to fix the problem. If you can offer any further advice or pointers, I’ll take care of this immediately. Steve

A bit of explanation might help. I commissioned my present design. Perhaps every other weblog author has intimate knowledge of every detail of every feature within the weblog tools and templates they use. I do not.

About 18 minutes later, I got this reply from TextDrive:

6/4/2005 Steve: We’ll take a bit deeper look, and sorry if I was presumptuous, we may have pegged you for “Steve” and have the wrong one. Not sure one way or the other yet, but the top 2 listing a steve of some sort made me want to fire out a quick email just to see if you were aware of something, or had some plugins or such going amiss. We’ll see how things shake out and be in touch. [name withheld]

Then, roughly 9 minutes later, this came in from TextDrive:

6/4/2005 The query is 7401285 select realname from txp_users where name=’Steve’ [name withheld]

I took an hour. I dug around in my weblog, but wasn’t at all sure what I was looking for. Frankly, I was under the impression that something had changed in the last day or two (i.e. early June) that might have created the problem. I had no idea that the problem they were talking to me about had probably been going on since February. Here’s what I said to them:

6/4/2005 [name withheld]: Other than posting to Rodent Regatta (rodentregatta.com), I’ve not made any changes to it in a couple of months. No new versions of Txp. No new plugins. I use CuteFTP to access TextDrive and frequently update to the latest (SVN) version of Textpattern on rodentregatta.org. However, there are no plugins or anything else running there.

[Name withheld], I want nothing but the best for TextDrive, and if I’m doing something on my end to harm the business, please let me know. I assure you it will be out of ignorance and not out of any malicious intent. I go by “Steve.” My name is Steve Pilgrim and I have used spilgrim as email names and such. However, for logging into TextDrive, I think I’m logging in as [user id withheld].

I’ve just checked everything that is loading when my laptop boots up. I’ve run antivirus, spybot search & destroy and adaware. Clearly, I’m showing my ignorance, but I can’t find a thing that involves MySQL other than my weblogs. Tonight is a good night for me to work with anyone you suggest to get to the bottom of all this. So, let me know how you want to proceed. Steve

My goal was to be cooperative and disclose fully what I knew. I couldn’t imagine what had started causing the problem. I was looking any and everywhere. In hindsight—and by inference—I’ve been accused of saying I wasn’t running plugins or that I didn’t mention all of the plugins. Two things: 1) I said there were no plugins running at the dot org domain; 2) my colophon lists every plugin that was being used.

About three hours after the original June 4 email, I got this email:

6/4/2005 Steve: Like I somewhat suspected after hearing from you the first time, we really don’t think it’s you, as the query is looking for name=Steve, and yours is name=[user id withheld] and realname=Steve. I’ll keep you posted nonetheless, and don’t sweat it, we’ll track it down, and it doesn’t look like you. Sorry if I was a bit brusk in my initial conversation – no harm intended. That said, does rodentregatta really get traffic to do that #2 query 132K times in a day? [name withheld]

By the sixth of June, I had inquired within the support forums.

Perhaps I didn’t ask the right questions. Perhaps I didn’t understand the answers I was getting. Whatever the case, I heard nothing more and it concerned me. Attempting to be a “good shared host customer” I initiated additional discussion on June 14, 2005. Here’s what I said:

6/14/2005 at 11:02a.m. [name withheld]: I haven’t heard any more about this. I’m just curious as to whether the TextDrive staff has been able to find out what I might be doing that puts undue stress on the TextDrive servers. My situation is this: I use Textpattern to publish my weblog. I use FeedDemon to read RSS feeds. I have no scripts or custom code that I’m aware of. Is there something I need to do to stop “abusing” TextDrive? Am I still considered a “problem” customer? Steve

By late afternoon on June 14th, I received this response:

6/14/2005 at 4:17p.m. Steve: It was you and your archive page. We turned my attention to the other Steve after you said “However, there are no plugins or anything else running there” and I took your word for it. Throttling the other Steve didn’t do anything, so we ran a complete a complete trace and it was your rss_superarchive.

Your archive page was taking the normal 300,000 queries an hour to nearly 3 million and that was only with a few hits on it. And continuous blocks of [code blocks you’ve already seen] in your archive Page was just cooking the mysql server.

Now besides costing me thousands of dollars in direct man hours (because even today I had two working on it for the last 8 hours), the additional time supporting mysql problems on barclay, having lost a couple of clients and having things about our “mysql problems” out there … I don’t even want to think about it.

Regardless, it can’t happen again, and while not-knowing-better isn’t an excuse, I’ll go ahead and take it as one. You can read some numbers on the forum about it, it won’t identify you but I have to discuss it. The general rules apply even to plugins: if you’re experimenting, then do it somewhere else and push things that are solid up to the server. [name withheld]

Needless to say, I was charmed. The rest of the emails are mostly defensiveness on my part concerning the misreading of the dot org vs. dot com domain names. I won’t repeat them here. However, I never intended for TextDrive users to experience another 10 days of trouble after TextDrive’s initial email to me.

It also came as a surprise that TextDrive would use backdoor techniques to log into my copy of Textpattern, set themselves up as administrative users, rename my archive page and delete a plugin that was doing the harm. Please understand I had/have nothing to hide. I’m thrilled that they found the problem. I would have gladly done exactly what they asked of me, but they didn’t ask me to alter anything.

The Conclusion

If you’ve read this far, I want to be perfectly clear. I admire the initiative, determination and technical savvy of the folks at TextDrive. I’ve wrestled mightily with myself in the last week regarding whether or not I’m personally qualified to own a TextDrive account and be their customer. I continue to want only the best for the organization. They are young, aggressive, effective and will polish their skills over time.

What would be ideal at this point is to have some way to guage what we need to know to be effective customers of TextDrive or any other web host. Until that is specified, count on others to do things that are as harmful as what I did to TextDrive. It’s time to prevent those things!