We’ve completed CodeRage 2007 online virtual conference day one. After arriving at work in Scotts Valley at 4:30am (pdt) to finish preparations for the day, we started the conference with the opening keynote. More than 800 developers connected (or tried to connect) in the first fifteen minutes of the conference. Little did we know that an “Internet storm” was brewing. We don’t have much visibility into the bandwidth load and utilization in Scotts Valley. As we started the keynote all was well until Michael Swindell, VP of Products and Strategy, started his portion of the opening. We started to get reports, in the chat room, of audio drops and then we heard, on our end, very strange vocal effects including a pitch lowering and slowing in Michael’s voice on the stream.
As I was asking Michael to slow down his presentation, the problem kept getting progressively worse. What was happening? First, we have more than 1900 registrations for the week like online conference. It was great that so many of our customers wanted to be a part of the event (and more are still arriving). We called our John in IT and he took at look at the Internet pipe report. He told us that we had maxed out the bandwidth. John Kaster took a look at the streaming servers, and they were at less than 10% CPU and memory load. What was going on? We then put 2+2 together and realized that we had not throttled back the Turbo Explorer Editions download servers and Bittorrents. I also remembered an email message from Sunday night about a new field test download being available. The Turbo downloads and the field test download were eating up more than 60% of the total bandwidth. It was a perfect storm, a trifecta of too many Turbo downloads, a major field test download, and a ravenous developer community logging in to our streaming servers.
What to do? Several things: 1) Throttle back the download and ftp servers, 2) redirect the streams over to servers outside our firewall, and 3) provide slide downloads and continue with audio streaming on our servers. Did we learn valuable lessons about more visibility into where all the bandwidth utilization was happening? Yes we did. Did we put our contingency plans, for other servers outside the firewall, into action? Yes we did. Did we ask for patience (and forgiveness) from our community members? Yes we did. Thank you attendees for giving us some time to calm down the perfect storm. We cleared up most of the problems and got the streams back up in less than an hour after the storm first hit. The rest of today went smoothly and we expect smooth sailing for the next four days of technical sessions.
It was great to see so many developers interested in our new Delphi 2007 for Win32, Delphi for PHP, and JBuilder 2007. We also gave everyone a chance to hear directly from members of our R&D teams and to ask questions direct to the developers. While we received many thanks and congratulations throughout and after the first day, we also received some criticisms and negative comments. We listened and we solved the problems.
For those who worried that the start up problems on day one might be indicative of our ability to produce quality products, I can only say that bandwidth and pipe management are completely different issues from producing complex developer tools. But I also know that we have to prove that CodeGear is different, and better, every day and in everything we do. I can’t wait for the next four days of technical sessions to show what we can do with our products and to help our customers be super heroes in your organizations and for your customers. In hindsight, we can all learn lessons from this “champagne“ problem. I just hope we don’t have a major earthquake sometime this week (like we did on October 17, 1989, right when the Quattro team was signing off a major release).
I’m sure that you will enjoy the next four days. At the end of the week we will put up a conference evaluation form where all of the almost 2000 registered attendees will tell us what they thought of the entire conference (and not just the first hour or so).
I will blog each day with news and thoughts about our first online virtual conference. We’ll see you all online throughout the week.
You can keep up to date on everything that is happening at CodeRage 2007 at http://www.thecoderage.com/. Daily updated information is also available at the CodeRage Daily Shout-OUT!
{ 5 } Comments
The professionalism of the CodeGear team most certainly was allowed to shine today - not that it doesn’t get to everyday … but today we got the chance to see that it really is everybody on the team that has this attitude - people we don’t get to see/hear about all that often. It was great to see.
… allow me to "sum up"
CodeGear [the team] ROCKS!
David:
Don’t you think it’s rather silly writing about getting up at 4 am to prepare for such a great event when reality is you haven’t done any homework in figuring out what the ISP is able to provide in terms of bandwidth?
I wouldn’t be showing such inflated ego when you know a whole bunch of people were really dissapointed when they can’t connect!
JC
I’m looking forward to the rest of the conference, and while it was unfortunate that Day 1 had its glitches, like DaveK I think your attitude was commendable - and I’m glad for CodeGear that it suffered from the problem of "too much interest".
I’ll be placing my order for D2007 at the end of the week. With SA, natch…
At least from my part, David, it went perfect. I had direct contact with many developers around the world, I could get answers from the "V.I.P." lounge about questions I wanted to know… Bandwich problems.. ok.. but from my point of view you made a F.R. (Fast Response. Keep on goin’. Kudos from my part.
To give credit, where credit is due, I am overwhelmed with admiration for the CodeRage team who were able to set-up the Interwise solution at such short notice.
Because of that I was able to enjoy the presentations by Chris Hesik and Charlie Caro yesterday with sound and video, and found them both extremely useful.
Congratulations to the whole team!
Post a Comment