Traffic, Data, Files, Oh My! Doing the Schedulefly Server Dance

We made a quick trip to our data center in Raleigh to slide a new database server into our rack and I thought I’d take a few minutes to blog about what we’ve built so far. Most people don’t know what’s going on behind the scenes of a database intensive web application like ours and in most cases they should not. Like a car engine, to them it’s a black box with lots of moving parts inside and it should just work. I agree, it should. And that’s simple to manage right? If more people come to the web site it should always serve them right? How hard can it be? Funny – I think this is the reason I still have friends who ask me “Is Schedulefly really your full time job?”. Too funny. If they only knew. I guess they assume, since it’s simple little “web site”, that you just need to set it up and hand it to someone to host. Then create a Google ad words account so millions of people can find it and buy it while you sail away into the sunset watching your bank account get fatter. Man, if only!

To handle our customers and our steady growth, Schedulefly now has 5 servers (and some other equipment to support them). This was our plan 3 years ago – to eventually get to 5 servers and the setup we now have. We have a primary web server, a backup web server, a database server, a backup database server and a development server. The primary web server handles all of our web traffic – from our web site and our apps. The backup server is currently idle and waiting to take over if needed – in case the primary machine dies before we expected it to. Our next move is bring the 2nd web server online to help share the load. For now – one web server is getting it done. We have a big beefy Dell server where our SQL database lives. The coolest, geekiest part of our data center setup is the 2nd database server. It is literally a real-time mirrored image of our live server. To the millisecond, it is kept in synch with our primary database and is ready to hop into action if the primary server has issues. To help give our database a bit more breathing room we are currently working on a caching technique that will greatly reduce the number of round-trips that our web server makes to query our database. This is critical for us going forward.

So we are doing the best we can to have redundancy and up-time at each level – power, internet, firewall, router, web server and database server. It’s really cool to see and I am thrilled we are scaling this ourselves as we grow. While we do spend a bit more to purchase hardware and manage it, we rest easier knowing we have a network dedicated to our growing business (that we have built along the way). There is also a bit of comfort knowing we can visit the data center and put our hands on our servers. We like the “clear box”, instead of the black one you can’t see inside. We are fortunate to also have an IT leader who is second to none in scaling a database intensive application like Schedulefly. Most small businesses don’t have this kind of technical person in-house, so they outsource it. Both ways work well. You choose one and go.

So for now we are cooking with gas. We continue to monitor all kinds of things like memory usage, disk space, backups and bandwidth usage. All of this goes up every week – every day really. Our database grows, our traffic increases, our log files get bigger, more people visit and upload photos and documents and our bandwidth usage climbs too. Heck, man handling the backups we take for our customers is getting tough so we have to routinely look at the backup routines we have in place and consider re-engineering them. It’s an evolving, living, breathing system for sure.

I tell my team that our system, like any web based database service that keeps growing, is like a ticking time-bomb. If you ignore it for even just a few days and don’t move and shake and tweak and do a little server dance – it will explode. I promise . It will happen. There is no “super computer” that we can install that will handle our customers forever. I think cloud computing may be trying to create the illusion of this for businesses – but at the end of the day – it’s still humans behind the scenes doing the server dance to help a web based business scale. In our case, we want to be the people dancing at Schedulefly.

Wes