24 Hour Test – Community Update

Hello Le Mans Ultimate fans!

A huge thank you to the 6,000+ players and 1,100+ teams that contributed to the ongoing team racing and driver swap tests with the 24 Hours of Le Mans test over the last weekend. It’s a huge commitment to be a part of this and we really appreciate you joining us in testing this game mode.

We started the driver swap test program in August and have been constantly hotfixing and patching the experience since then. It has been amazing to see (and race against) those of you that committed to helping us make this experience better. Your help and time has been invaluable and together we’re making something stronger.

Aside from automated testing, we’ve only been able to run three 24 Hour races internally with real drivers before this weekend, so with your help we now have invaluable data on more than 20 heavily populated ‘public’ race events.

As it is in the “real world”, Le Mans is the ultimate test of endurance. Despite the 200+ races and 40,000+ swaps performed in the driver swap test program since it started in August, we closely monitored the races and found some new issues not previously encountered. Whilst we are embarrassed some of you experienced problems, the point of the programme was to stress test for issues. Now we’re in the process of (a) error fixing software issues and (b) installing improved configuration scripts to ensure the cloud services we deploy to are capable of handling the kind of load and complexity experienced across these mass participation races.

Server Issues

1. Not being able to rejoin a server

Despite running successfully for the entire 24 hours, two servers lost their HTTP port bindings so players couldn’t rejoin if they dropped out of the server.

We are investigating ways to look at how we can prevent this from happening on future events.

2. Incorrect configuration for servers

Before the race started, a small number of splits didn’t receive the correct configuration (drivers, teams, track and other session information) meaning they couldn’t be joined and the race didn’t begin.

This is the same process as all other Daily/Weekly/Special events we run every day without issue so we’re looking for a pattern here.

We will also be adding additional management tools which will allow admins to recreate failed splits should anything go wrong with starting up future events.

3. Latency (lag)

We trialled three different server providers this weekend with differing results.

Given the scale of the 24hr events at Le Mans, data is exponentially higher with the number of cars and the total number of players connected at any one time. The total number of connected players in a 24 hour race was much higher than in the previous 4/6/8/12 hour test races and so presented a bigger challenge than previous events.

Now we have some real-world data on these providers with more players we can better tune which server providers we use for large team races.

4. Server crash

We experienced just one actual server crash in an area not seen before, 20 hours into one split’s race. We’ll be patching this as soon as possible. From our perspective, it’s one crash too many!

5. Server Termination

One server was terminated by our system as it stopped responding to health checks. After 5 minutes of it being non responsive to these checks, our system automatically marked it as unhealthy and initiated a termination process. We are investigating why this happened.

Performance

We are also investigating these issues:

• Reduced frame rate

• Disconnects during a driver swap

• Replay loading and general replay experience during live events

• Track limits being stricter after a driver swap

Summary

Whilst it is incredibly frustrating to experience these issues (for us as well as all of you who had signed up and practiced hard for this race), we believe these issues are all fixable.

We’re currently hoping to run another 24 Hour test race in January, depending on the progress of investigation and fixes. The driver swap feature will not be considered “full release” until we are satisfied with the performance and stability of a 24 hour race across multiple concurrent splits.

We still expect to run shorter team races in the meantime during December, which have been running very well for the last few months.

We know a lot of you are still looking for Engineer mode… keep an eye out for some news here coming very soon

Thank you again to everyone that participated in this valuable test and apologies to those of you that had your race experience compromised. We’re still working hard to make team endurance races on Le Mans Ultimate a genre-defining experience. Your input and help here is, as always, greatly appreciated.

Team S397