Overloaded Servers Crash For Hours Ahead Of iRacing's Rolex 24

Illustration for article titled Overloaded Servers Crash For Hours Ahead Of iRacing's Rolex 24
Photo: No Brakes Racing / Twitter (Getty Images)

I’ve been listening to my husband scramble to find a group of fellow racers to take part in iRacing’s virtual Rolex 24 at Daytona for months. I mean months. This man repaired his whole entire broken sim rig specifically for this race. And then the servers crashed.

The virtual Rolex 24 has attracted race fans across the globe interested in trying their hand at running a full-day endurance race with some of their closest sim racing pals. I’ve watched people sweat over their liveries for weeks to create the best possible car. There have been practice sessions lined up for this event to give people the time to practice. And if you’re as organized as my husband and his team, you’ve gathered a team of six drivers, accounted for all of their various schedules and preferences, and scheduled out driving time based on skill level and experience.

Advertisement

If you’re my husband and his team, you also spent upwards of five hours waiting for the server to open.

This morning’s race was scheduled to start at about 8 am ET. But when I went to talk to my husband a little before 10, everyone was still waiting around just to get in.

“I don’t know what’s going on,” my husband said. “The whole thing just imploded.”

If you’re not familiar with the way iRacing organizes its sim events, the service usually offers several different time slots for big, official events that count towards your license rank and your safety rating. When people log on to race, the service has to create multiple different race servers in order to accommodate for them. If 100 people register for a race, iRacing evaluates the safety ratings and licenses of the people who entered, then segments them off into different slots for the actual race. That way, you’ll have a standard, 30-ish car field where everyone will be on a relatively similar skill level.

Advertisement

For the Daytona 24, one time slot kicked off at 8pm ET on Friday, January 22. The second—the most popular event—was scheduled for 8am on Saturday.

Advertisement

As it turned out, there were so many sim racers—at least 15,000—who all attempted to log in at the same time for that time slot. An explanation as to the problem on the iRacing forums read as follows:

The scheduler had issues bringing new race servers online fast enough. Teams were waiting to be assigned to their practice servers after registering because there were no free slots. They then needed to wait for a new server to spin up and be ready to be assigned.

When the launch came for the session we didn’t receive any feedback from the scheduler. We could see that it was alive and scheduling other sessions.

Looking at the logs, we could see that the scheduler determined it was taking too long at one point and timed out.

Advertisement

Basically, the scheduler did the same thing your internet browser does when it takes way too long to load: it just times out. It’s too much all at once.

iRacing basically had to override that function, to demand that it populate different race servers no matter how long it took. And it took one hell of a long time. My husband and his crew got into a server just before 1pm ET, but there were still others waiting by that time.

Advertisement

It took hours for iRacing to rectify the problem, at which point it offered a $5 credit for every driver that had entered the race. As the iRacing crew noted, “this is only a small token for your time and frustration however we hope that it is an olive branch showing our dedication to improving these events moving forward.”

This has been a consistent problem with large endurance events on iRacing. Servers ran slow during the 9am ET time slot for Le Mans on June 20, 2020, and other sim racers reported to me similar lags when loading in for 2020's Daytona 24 event, albeit to a lesser extent.

Advertisement

It’s understandable that there are issues with a rapidly growing service. Race fans desperate for a little motorsport action signed up to iRacing during the COVID-19 pandemic as a way to get their speed fix, and even professional drivers are hitting the service to turn laps and have a good time. While iRacing may have thought it was prepared to handle an influx of 15,000 people all trying to enter the same server, shit happens. You can’t really know until you try.

Advertisement

What many sim racers find unacceptable, however, is the lack of communication. The first iRacing tweet acknowledging the issue came at 9:14 am ET, over an hour after people started really struggling—and even then, it was a fairly vague message.

“Our scheduler experienced a technical issue that we are currently evaluating. This timeslot of the Daytona 24h will NOT have a race start,” a tweet read. “We are evaluating our options to see if we can put up another one in the next hour or so. Stay tuned for more information.”

Advertisement

It took several further hours—and countless tweets demanding to know what happened—before iRacing noted what the problem was. Sim racers scheduled to compete spent most of that time troubleshooting to see if the problem was on their end or just sitting on their hands, speculating in group chats. Even worse, iRacing shut down its user forums so fans had one fewer place left to vent.

Advertisement

It’s a shitty situation for everyone involved, iRacing users and staff combined. But after today’s chaos, it’ll be worth iRacing’s time and money to figure out how to combat these problems in the future.

Advertisement

And if you’re looking for some goofballs to watch, you can watch my husband’s team in car No. 6 here or their sister team, No. 90, here.

Weekends at Jalopnik. Managing editor at A Girl's Guide to Cars. Lead IndyCar writer and assistant editor at Frontstretch. Novelist. Motorsport fanatic.

Share This Story

Get our newsletter

DISCUSSION

So to specify a few things that I do at least have a little understanding about and clarify another thing:

1. iRacing mentioned it was investigating within minutes of the issue on the forum post. The initial issue appeared that the scheduler was functioning a “job”, essentially, it’s pulling all the data about 10,000 people, for however many thousands teams, and then it’s having to gather their irating, average it out for each team, then organize each team by that irating, then open up a server for the “batch” of drivers that add up to a server.

2. The issue was a timeout where this process + setting up a server was taking too long. The browser analogy is pretty solid!

3. This issue doesn’t happen every year. It’s a different issue every year. Last year was an anti-ddos system set too tight it was self ddosing... Lol.

4. The communication problem wasn’t a simply iRacing ignoring it. They believed the process was functioning but was in a hung state. It appears they didn’t know it was stuck until a deeper dive.

——

To add additional stuff: they probably shut down the data server to unstick the scheduler (eg: shutting the forum down, they rely on the same system), and while I understand people being upset about not being given information, it WAS given just in really stupid places. Drew Adamson(I think that’s his last name) was tweeting every 15-20 minutes what they were doing, eg: investigating a root cause and trying to get the system up to try again. But this should have 100% been on the official iRacing twitter and in the main ui.

Just to add too, unless something like this exact same issue came up before, there isn’t really a way to prepare for it. Like you could do stress tests, but like... How far do you really take that before it seems unreasonable?

Last year’s Daytona 24 was 18 splits and other than initial ddos issues causing lag, was fine. This one by all calculations is around 47 different servers of players.

Im still pissed. I’m supposed to be driving now but instead I’m driving in 4 hours. It sucks.

But, as a Technical Support Engineer, they did the troubleshooting fine, could’ve been way better, and speaking with a QA guy, both of us can’t be mad at them. Which really freaking sucks cause I’d love to just sit here and be mad at them but now instead I have to focus my anger in “educating” something folks don’t care about cause it’s not their problem to think about why iRacing was down.

Which my response is: no, you’re right. This shouldn’t happen. All I can do is feel for users and the devs alike. So be mad. Maybe they’ll look harder at preventative measures next time. But if this post gives you any additional information, or at least some ease of mind, that's all my intent was. Cause jfc they need to do better. I just... Am not sure how other than centralizing their communication a bit more.