New World launched this week and, as our review-in-progress says, it’s complicated. MMO launches are notoriously difficult and almost every game has its issues, with New World being no exception.
But there are small problems and there are huge problems: New World suffered from some overlong queues and coping with server demand, but it didn’t collapse and now seems to be powering-through its teething struggles. PC Gamer spoke to Rich Lawrence, the head of developer Amazon Games (AG), about New World’s first days in the wild—and what was going on behind-the-scenes.
PC Gamer: What was the biggest blockage New World’s launch ran into? It seemed to be going well when it was just the EU servers.
Rich Lawrence: The only substantial issue we’ve had is load driven by demand. Each of the regions we targeted for launch opened on time in their respective areas. So it was less about EU going well, and more about just the very high number of players who showed up ready to play over time.
PCG: Did AG anticipate player behavior such as region-switching to avoid queues, and was this as big an issue as some thought?
RL: We believe in choice for players, so we deliberately provided the ability to change regions, and just supply the information of what that means in terms of latency impact. Gamers tend to be well informed and understand the consequences of higher latency, so it’s their choice. It wasn’t an issue.
PCG: There is a perception that this is simply a number-crunching problem: “oh why don’t they just turn more servers on.” Is that mistaken?
RL: I wouldn’t call it mistaken, but there is a lot of detail that can get lost there. Each server that a player sees is actually multiple machines configured to talk to each other—if we do our jobs right, that is not visible to the players, but it’s still a complication.
In the first day of operation we doubled the number of servers. That’s well over a thousand machines, hundreds of thousands of configurations and changes.
Rich Lawrence
The machines have different roles, and thus different configurations and data to work with, that needs to be copied in place. They have network rules to talk to each other and the players in order to be secure. They need to be plugged in to larger systems, like log-in, or customer service. And once all that is done you need to test them, because there’s always the small chance of bad hardware somewhere, and even automated systems sometimes break on copying a file, that kind of thing. And though we automate many steps, it always requires some humans—a server needs a name, decisions on whether it should be a specific language, where it goes, how to notify customers it’s available; including translating to local languages.
I completely understand the player perspective of “I need a server to play on, now!” We heard that loud and clear. In the first day of operation we doubled the number of servers. That’s well over a thousand machines, hundreds of thousands of configurations and changes. I describe this to illustrate that it has a time component. It’s time we are happy to spend of course, but not trivial and absolutely not something that we were taking casually. If we had been sitting around and just not turning on servers, I’d be furious with us as a team. That wasn’t the case; we worked frantically non-stop through the first 40 hours of operation and we’re keeping up that pace, with more servers pouring in. We will keep them coming until it’s not a problem.
PCG: Can you explain in layman’s terms the logistical challenge of launching a game that needs to handle almost a million players on day one?
RL: The question above hits on a good portion of it. Thousands of machines working together is not trivial, although a lot depends on the game type in the first place—MMOs are one of the most complex. Luckily we have [Amazon Web Services] AWS for many aspects, so actually getting a machine for instance is incredibly easy—tying a set of them together and making them into a New World server is most of the work. But to the core of your question, the primary challenge is coordination. Even if everything about the service side was a magic button that just said ‘run game,’ you would have a ton of coordination to track.
PCG: What was the biggest unforeseen problem?
RL: Absolutely it was the scale of demand. Which on the one hand, great! We’re enthused by player interest in the game. But we didn’t have anywhere near this strong an indication from pre-orders or beta—we were happy with both, to be clear, but the numbers grew so quickly and so close to launch it caught us by surprise, I have to admit.
PCG: Do you think it’s possible to launch an MMO ‘smoothly’ or is that just an illusion?
RL: There are approaches that reduce risk considerably, but with tradeoffs. You can focus on player connections […] and that is often beneficial for large scale—in those models you tend to think of hosting processes as a pool, and invoke them any time they are needed. Players often perceive this as “phasing”, “vertical phases”, “channels”, it goes by many different names. If you do a great job of that than arguably a “server” is just a label, and players can go anywhere.
The tradeoffs are it tends to mix up socialization—you don’t see the same people around you in game, or at least not as often. Which can be bad for both PvE and PvP gameplay, where a lot of the enjoyment can come from knowing your friend/opponent that you run across in the world.
“We had what looked right based on pre-order indicators, but we missed the mark on that, and didn’t give the players the experience we had worked so hard to provide them.”
This isn’t the only way, but if for example you took that approach, and had enough pool resources on day one, and no bugs, and had tested to scale—which is a huge challenge, since you can simulate a million players but bots just aren’t players—if all that happened you can have a smooth launch. For us that would have meant a very different game than New World though, without the seamless open world aspect, not as large in outdoor environments, and some other aspects.
PCG: If you were going to launch New World again, is there anything you’d do differently?
RL: This is probably very obvious, but I would have started with considerably more servers ready on day one. We had what looked right based on pre-order indicators, but we missed the mark on that, and didn’t give the players the experience we had worked so hard to provide them. I also would have had character transfers ready on day one—we knew they were important, but we chose gameplay features as a priority running up to launch. We’ll get transfers in soon; the feature is already well in-progress.