Home Page

Watching the World Wide Web

Web Growing Pains: The Price of Success

Maintaining a successful Web site can consume increasing amounts of money and resources. Make sure you budget for both.

Many people judge the success of their Web site by how much traffic it generates. But it may not be obvious that the more traffic you generate, the more money it costs to keep your site current and your customers happy. In cyberspace, this is the price of success.

More traffic generally forces you to use more people to run your site and answer the questions you get from your customers; more horsepower for your server to ensure that your site runs fast enough to keep up with the traffic; and a faster communications line to ensure that customers can get through. If your site is generating traffic but not a lot of revenue, selling these additional expenses to management can be a difficult task, especially if those expenses aren't budgeted.

When our site went live in January 1995, we had no idea how much traffic we'd generate. When we were averaging 13,000 hits per day within two weeks of going live, we were flabbergasted. Traffic was much higher than we expected.

However, sales were much lower. It seemed that people were visiting our site, finding the information they wanted, and then returning to traditional channels (bookstores) to buy the products. In itself, this was fine with us; like other publishers, we like to promote our resellers. But it didn't resolve our basic conflict. More cyber-traffic means more questions from more readers and, hence, more people to answer those questions in a timely manner. Our policy is to answer questions within 24 hours; cyber-customers are not a patient lot.

Upgrading on the Fly

By June, we were averaging more than 30,000 hits per day. Through careful load measurement and system performance statistics, we knew that it was time to upgrade our server, a single-processor Sparc 20. We added a second processor and doubled the RAM to 128MB. We also increased hard disk capacity from 5GB to more than 10GB.

Obviously, you can't upgrade your server in the middle of the day, because traffic is too high. That's the equivalent of closing a restaurant at lunchtime. It's better to upgrade in the middle of the night on a weekend, even though your system administrators might not like it. And when you upgrade, rather than confusing customers by being completely off-line, use a spare server, which could be as simple as a 486 running Linux. Our spare server, a 486, broadcasts a message that says the site is currently down for upgrades and asks you to visit again in a few hours.

The upgrade process could take only a few minutes, but it could take hours if things don't go smoothly. Our preferred upgrade time is 2 a.m. on Sundays. In the interest of the sanity of our system administration staff, however, we tend to perform minor upgrades at 1 a.m. in the middle of the week and save the middle of the night weekend chores for major upgrades that bring the system down for long periods of time.

Hiring Mode

Increasing horsepower also required us to upgrade our staff. We hired two more people in July to keep up with the traffic and updating the site to meet the demand. The site's success made this a fairly easy sell to upper management.

It's important to continually update the site, or traffic will not return and will not increase. If you're experiencing steady or declining traffic over a long period of time, chances are you're not updating the material on your Web site often enough.

By August, our traffic had ballooned to 100,000 hits per day. Because we still didn't generate much revenue, we created a frequently asked questions list to help users get answers online, without having to submit a question. This measure saved us from having to hire an additional technical support person.

We redesigned our site in September to accommodate the addition of several divisions of our company. We ran a test site in a hidden location on our main server for several weeks, adding material and tweaking the new site. When it came time to take the new site live, we had to change the URL of the test site to the URL of our main site and take the old site offline. We weren't sure how long this would take, but it required the services of a programmer on the West Coast working with our system administrator. The programmer agreed to come into his office at 6 a.m. Pacific time. To our surprise, everything went smoothly, taking a total of only eight minutes.

By November, traffic had increased to 200,000 hits per day. Support is still an issue, as is maintaining more than 20,000 pages of information about more than 6,000 books and software products. To make our server run more smoothly, we upgraded the operating system from Solaris 2.3 to Solaris 2.4.

We also added two secondary servers to handle traffic that puts unusually heavy loads on our server. This includes majordomo requests, which generate a lot of system overhead due to the fact that we have more than 40 mailing lists on our server and high-bandwidth data, such as documents using Adobe's PDF format and our experiments with new technologies such as Virtual Reality Markup Language (VRML) and Java.

More, More, More

Traffic, by the way, has not increased at a constant rate. In the beginning, there were periods of growth followed by periods of level or somewhat lower traffic. I likened it to a stairstep approach. However, starting in June, traffic began to increase dramatically and showed no sign of leveling off until the 200,000 hit mark in October. Our system administrators constantly monitor the site for performance to identify when the server and/or the communications line needs to be upgraded.

Because the cost difference wasn't too great, we installed a T-1 line from the beginning. A T-1 works at 1.544 megabits per second at a cost of less than $2,000 per month. Our T-1 is currently at no more than 50 percent of capacity through the day, so we feel that it can last for another few months of traffic increases (traffic is currently increasing by about 40 percent per month) before we have to add another, which we've budgeted for this year.

For next year, we've also budgeted an upgrade to a new server with faster processors, using the Sparc 20 as a secondary server for more complicated applications. And we're going to increase the size of our staff to help us maintain our site.

I hope our experience demonstrates that it's crucial to put the costs of a successful Web site into your budget. Otherwise, you could be setting yourself up for failure.

Jordan Gold is publisher of online services for Macmillan Digital USA, the multimedia arm of Macmillan Publishing, in Indianapolis, IN. He can be reached at jgold@mcp.com. The Web address is http://www.mcp.com.