Question: Could you tell us about an interesting or hard to find or fix bug you've encountered since working on Entropia?

Wistrel

Elite
Joined
Mar 30, 2005
Posts
2,517
Location
The Arctic (GMT)
Society
Entropia Pioneers
Avatar Name
Wistrel (Wisty) Chianti
One for the Developers, and obviously not asking for anyone to spill anything "top secret" here, but there are quite a few people who would be interested in hearing about what it's like working with the Entropia code, especially with it having been around for so many years now. It must present quite a challenge at times I'd imagine and I'm sure the developers would love a chance to talk in detail about the sort of things they do.

Hopefully you'll consider this. I appreciate this might look like I'm asking for someone to "air their dirty laundry", but everyone makes mistakes and talking about them is educational to others who have an interest in programming/game development.

Thanks
Wistrel
 

My|MindArk

MindArk Official
MindArk Official
Joined
May 4, 2020
Posts
68
  • Hi Wistrel! And thank you for your question!

We mainly use c++ for developing Entropia and as you know it has been around for a long time now, which means that the code base and the systems have evolved over the years. We are constantly improving on the code base but we still have code that is old in some places and harder to maintain.

I love c++ but there are so many ways you can do it wrong, which I would say is its main disadvantage.
The hardest bugs to find are usually when you end up in Undefined Behavior, where anything can happen and maybe a bug appears once a year or once a decade.
You could classify the challenges into the following categories.

  • Resource Management
  • Uninitialized memory
  • Double deletion
  • Concurrency
  • Race conditions
  • Deadlocks
  • Bad computational throughput (waiting on locks)

Two recent problems we have looked into are Lag and the issue when people entering space always ended up in the Mother Ship Ark Royal.
The first one falls into the Concurrency category and the second one falls into Resource Management.

Lag



The top two clients have no issues at all. The one in the middle does, and that is because his traffic is going through a router where the bottom client is logging in using an authenticator, and while he is logging in, the client above him logged out. The middle client is now experiencing one of these freezes. Note that even though the client second to the top is on the same game server, he is not experiencing the same freeze.

We are aware that there are still some shorter freezes happening, and we are going to keep improving the networking, but the worst issues seem to be gone for now
We would like to thank you for sticking around, and if you didn’t, we would gladly welcome you back! And of course, we are sorry for any inconvenience this has caused you.




Ending up in Ark Royal
One of the more recent incidents we’ve had was a couple of weeks ago when we released 17.4.0, and it turned into a 10 hour downtime and full day of chaos.

We restarted the servers in test mode and began the final testing in the live environment, before opening it up to all players. Some of us were going to space to test the new space event. The first one started on Calypso, got into his Quad-Wing, and up, up, up, and out into space. But nooo, he did not end up in space at all, he ended up in the mothership Ark Royal! The second one tried going from the Calypso Space Station, but the same thing happened to him as well. Whatever we did, everyone always ended up in Ark Royal!

After a lot of debugging, we found the issue in a piece of code that hadn’t been changed for well over a year. It was the code that calculated the size of the sphere that is used to enter a mothership. Normally it looks like this:


Now, it was so big it contained all of space! No wonder you always ended up in Ark Royale, it was the first mothership with public access, and since you collided with the sphere, that part was technically correct.

The issue was that one of the values used in the sphere-calculation could sometimes be uninitialized. This meant that if you were lucky, it would be 0 and everything worked, if not it would be crazy numbers like 8.8345e+13, which would result in a huge sphere.

For some reason, after well over a year without issues, our luck had run dry. We are still not sure why the issue just popped up now, but at least it works as intended again!

Now, when we look back at it, this was a quite amusing/comical issue. During the downtime when we were trying to find it, not so much! 😄
- Magnus Ryme
 
Top