Note to the MA network coders

[smoothee]

First off, in certain locations and during certain times of the day there is some serious network lag and that's to be expected until this new version is a little more mature. However, I think you have a bug in your implementation and I'll explain:

Some background for the 99.99999% of people who are not network programmers. There are 2 main types of network packets:

1: TCP. TCP is a network type that is very slow but is guaranteed to be delivered and IN ORDER. This is how versions BELOW v10 transmitted most/all data.
2: UDP. This is very fast but is prone to huge errors due to little or no error checking. this is what most of v10 uses. (this is an educated guess. I've not set up any tests but it's painfully obvious this is the case. Testing this might also be a violation of the TOS so there we are...)

The problem is that when I'm hunting in a high lag area, the UDP packets come in OUT OF ORDER. So for example,

Packet 1 says: I'm hit for 115hp and my hp is now 100.
Packet 2 says: I'm hit for 75 and my hp is now 25.

If these 2 packets are reversed due to lag it will MAKE MY HEALTH BAR SHOW 100 and not 25. The server knows it's at 25 and when I'm hit for 26 I'm dead. Laying on the ground, dead, with my health bar showing I have another 75 hp is frustrating as hell.

Solution: NUMBER THE PACKETS

This is how every single other MMO works. PLEASE DO IT FOR UDP SO I CAN SEE MY ACTUAL HEALTH!


EDIT: a discussion about the introduction of TCP/UDP mix begins at the end of page 2. This is a better solution but will take them more time to implement and is certainly not as simple. I'll bet that a year from now this is how it will work but they need to get some kind of half-way solution to this in place now.

 

[SuperKanjo]

Is it really this simple ??

 

[smoothee]

Is it really this simple ??

To fix this issue, yes. Numbering UDP packets will fix this immediately.

 

[SuperKanjo]

To fix this issue, yes. Numbering UDP packets will fix this immediately.

Ok, but if it's this simple i think every network programmer (and also MA) should know this.

 

[Von]

It also happens in mining and is anoying as hell.

Von

 

[Rentonx]

Ok, but if it's this simple i think every network programmer (and also MA) should know this.

Keep in mind MA are composed of around 50 employees... hence they maybe a tad bit slow

 

[smoothee]

Ok, but if it's this simple i think every network programmer (and also MA) should know this.

One could only hope...

However, I'm assuming that with all the other details in v10, this one got skipped and is something that doesn't show up on testing unless they test during high-lag network conditions(0% of the time for them).

If MA is reading this, your game would be twice as fun if you use this during testing.:
http://wanem.sourceforge.net/

 

[kyleen]

wont numbering hte packets slow down the connection speed?

 

[smoothee]

Keep in mind MA are composed of around 50 employees... hence they maybe a tad bit slow

I understand. I'm simply bringing this up because I think it's a high-priority item and that not many players here can talk in network-speak fluently enough to explain it to the very special 1 or 2 of those 50 employees.

 

[Xzion]

Ok, but if it's this simple i think every network programmer (and also MA) should know this.

Aperently not

 

[smoothee]

wont numbering hte packets slow down the connection speed?

No, not one tiny tiny bit. It will only slow things down if we have to split the packets into 2 if they get too big. 4 bytes on the end of a 100 byte packet will NEVER split it. Therefore, speed will remain the same.

 

[Nadie]

Interesting

Is this also what could be causing the healing problems I get all the time where I obviously need to heal but the server doesn't think I do until the 4th attempt? Just curious as I don't know anything about this stuff

 

[smoothee]

Interesting

Is this also what could be causing the healing problems I get all the time where I obviously need to heal but the server doesn't think I do until the 4th attempt? Just curious as I don't know anything about this stuff

Nadie,
GREAT COMMENT. You have it exactly! This problem will go away when they number UDP packets.

 

[CrouchingTiger]

I dont know how many hosts are hosting their servers but wouldnt it be better lets say they had one host in europe and one in the very far west and we were given the option to choose what server we wanted to connect to upon login depending on your location.

So if your from the US you could use the very far west server covering the u.s side of players and people from the east and europe could use the european server .

Or it could automatically connect you to the appropriate server upon your location but then automatically never did work good for MA. So rather give us the choice of which one to connect to.

Would that not seriously decrease the current lag and connectivity issue we currently have majorly. As majority of the bugs seem to vanish on a good connection with less lag i noticed.

Regards
El Tigre

 

[Calin]

Have you sent a message about this to support?

 

[smoothee]

I dont know how many hosts are hosting their servers but wouldnt it be better lets say they had one host in europe and one in the very far west and we were given the option to choose what server we wanted to connect to upon login depending on your location.

So if your from the US you could use the very far west server covering the u.s side of players and people from the east and europe could use the european server .

Or it could automatically connect you to the appropriate server upon your location but then automatically never did work good for MA. So rather give us the choice of which one to connect to.

Would that not seriously decrease the current lag and connectivity issue we currently have majorly. As majority of the bugs seem to vanish on a good connection with less lag i noticed.

Regards
El Tigre

Indeed you're right. However, I *think* that this is how it works now. I've done zero testing on this but I remember a discussion about some US servers for MA. This means that the problem isn't in lag due to the packet on the wire but within the MA server park(s).

This makes it even more important to number UDP packets.

 

[sledge-]

Had it often with heals and hits the last days - during Sunday evening it was very obvious

But it can deliver fun moments too (not many but still):
I was shooting a mob and health went low so knowing that there is no tp chip to get me back i decided to heal ... ok fap out, heal and during fap reload the mob dies by a delayed hit

Had it the other way around too unfortunately - ok enough health for at least one hit and then dead because mob gave me damage twice in 1 sec

 

[smoothee]

Have you sent a message about this to support?

I have not. I'm gathering some screen shots for this now and will submit one in the morning. I'm looking for as much evidence on this issue as I can so it's tougher for them to simply ignore me.

See Nadie's post above for a good example of what I'll be submitting.

 

[Nadie]

Nadie,
GREAT COMMENT. You have it exactly! This problem will go away when they number UDP packets.

Ah great stuff thank you. Nice to know it's not just me with these issues, that there is an explanation for them, and that hopefully, something can be done to fix them.

Just so you know I have already submitted this myself and ofc got the standard response that it was one of the issues being looked in to. It would be great if more people would send in cases about these issues though so they realise how many people have these problems and it becomes high priority.

 

[aridash]

Is it really this simple ??

no. for a start old VU was UDP. all games and streaming in general is UDP since the time delay of TCP is not usually acceptable versus dropping the occasional packet then account for compensating for this. further evidence that its more complicated is that we all connect to one server, while as pointed out certain areas (behind that single point) have differences in lag effects. this suggest there are internal network and or server issues, anomolies such as loot lag prove this.

wont numbering hte packets slow down the connection speed?

of course it will: its how TCP confirms data transmission (with additional error checking too). really i dont see how this can possible avoid the scenario being described. if i have UDP+smoothee_inc packets numbers 2 to 42 received, i still have to wait for number 1 if im to strictly process them in order. or i can drop no.1 and either slot it in later or forget about it altogether... oh wait i've gained nothing.

as for multiple geographic servers, you cant do that with a single economic world as we currently have. you'd be having to refer back to the central server for loot lookups, auction, PvP trades or such, leading to internal system lag... oh wait...
However, there might be scope for this with new planets were much of these system might be more decoupled anyway.

 

[smoothee]

no. for a start old VU was UDP. all games and streaming in general is UDP since the time delay of TCP is not usually acceptable versus dropping the occasional packet then account for compensating for this. further evidence that its more complicated is that we all connect to one server, while as pointed out certain areas (behind that single point) have differences in lag effects. this suggest there are internal network and or server issues, anomolies such as loot lag prove this.



of course it will: its how TCP confirms data transmission (with additional error checking too). really i dont see how this can possible avoid the scenario being described. if i have UDP+smoothee_inc packets numbers 2 to 42 received, i still have to wait for number 1 if im to strictly process them in order. or i can drop no.1 and either slot it in later or forget about it altogether... oh wait i've gained nothing.

"no. for a start old VU was UDP."
-- I'll have to take your word for this. I didn't do any testing on pre v10 for TCP/UDP but the only real metric here is that this problem was not in pre-v10 versions. The point is that version 9 packets were displayed in order.

"or i can drop no.1 and either slot it in later or forget about it altogether"
-- you've got it exactly. how this normally works is you categorize packets as important or unimportant. You send important data (like tool use, key position points, etc) via TCP because you're 100% sure it will get there in order. Then you send all the other stuff that's not really important like intra-key position points via UDP. If you get a UDP packet out of sequence then you just toss it. You get the strengths of each without the weaknesses.

The real solution here is to use more TCP as well as UDP numbering. There are 10 years of industry whitepapers supporting this and I hope the fine folks at MA are at least aware of them. I only originally mentioned UDP numbering because it's something they can do right now, today, with very little work.

 

[Stutoman]

This is actually what is responsable for that bug where you lost connection and when you came back online you had both the loot and ammo in your inventory. Also why 15peds temporarily went missing from my ped card once.

This isn't a joke.

 

[Turnsleft]

Ok, but if it's this simple i think every network programmer (and also MA) should know this.

ROFL depends how much they pay and if the coder is any good, plenty of chancers out there..!!

Crafting is the same, I get a failed message as soon as I click, sometimes I get the hof/global sounds/lights before the click process has finished and the size has been displayed in the chat window, sometimes the other way round.

The only issue would be figuring out how many messages to buffer before killing them off, or heck throw back some response messages ... ooo that would be TCPish, far too sensible

 

[smoothee]

The only issue would be figuring out how many messages to buffer before killing them off, or heck throw back some response messages ... ooo that would be TCPish, far too sensible

Yup. TCP. It's the most common protocol for a reason.

See my post above but we can be pretty sure that we already have TCP and UDP connections set up. They just need to be a little smarter about how they apply TCP.

 

[aridash]

"no. for a start old VU was UDP." -- I'll have to take your word for this. I didn't do any testing on pre v10 for TCP/UDP but the only real metric here is that this problem was not in pre-v10 versions. The point is that version 9 packets were displayed in order.

eh? the problems you talk about have always been around and mentioned here from time to time: after certain VUs, during events, at weekends for some and not others etc. MA advise for firewalls hasnt changed as far as im aware with VU 10:
# 30584 (UDP inbound/outbound)
# 30583 (TCP Outbound)
# 554 (TCP Inbound)

so its a mix, probably as you suggest some important stuff is TCP. That TCP traffic is going to be prone to substantial lag, such as that Nadie described where it feels like theres no responce from the server.

 

[smoothee]

eh? the problems you talk about have always been around and mentioned here from time to time: after certain VUs, during events, at weekends for some and not others etc. MA advise for firewalls hasnt changed as far as im aware with VU 10:
# 30584 (UDP inbound/outbound)
# 30583 (TCP Outbound)
# 554 (TCP Inbound)

so its a mix, probably as you suggest some important stuff is TCP. That TCP traffic is going to be prone to substantial lag, such as that Nadie described where it feels like theres no responce from the server.

"That TCP traffic is going to be prone to substantial lag"
-- No, I must disagree strongly here. TCP cannot be out of order. It's one of the basic BASIC features of it. If messages are out of order then it MUST be UDP.

Think about it: you have 2 of the same type of messages like 2 fap messages as nadie shows us. these 2 messages will absolutely be using the same network protocol yet they are out of order. This simply cannot EVER be TCP.

 

[jdegre]

"That TCP traffic is going to be prone to substantial lag"
-- No, I must disagree strongly here. TCP cannot be out of order. It's one of the basic BASIC features of it. If messages are out of order then it MUST be UDP.

Think about it: you have 2 of the same type of messages like 2 fap messages as nadie shows us. these 2 messages will absolutely be using the same network protocol yet they are out of order. This simply cannot EVER be TCP.

smoothee, i think that what you're missing here is that both TCP and UDP use the same underlying mechanism: IP datagrams. so, the real problem comes when one of these IP datagrams is lost, or comes out of order due to a big delay in the path. this can happen in TCP exactly the same as in UDP.

what to do, in this case? if you're using TCP, your network stack will simply wait for the right IP datagram to come in order. if it does not come, it will be eventually retransmitted by the client, adding even more delay. the whole game would freeze waiting for ordered delivery of packets.

so, i'd tend to agree with aridash. i really doubt 9.4 used TCP, and i don't even think it is feasible to use it at all. when packets are lost (or delayed) it's simply a trade off between reliability and responsiveness. for an "action" game, you definitely need responsiveness.

as far as i can tell, 9.4 used UDP for "game" traffic, and only TCP for "participant content upload" and such.

 

[aridash]

"That TCP traffic is going to be prone to substantial lag"
-- No, I must disagree strongly here. TCP cannot be out of order. It's one of the basic BASIC features of it. If messages are out of order then it MUST be UDP.

i dont say this in a nasty way, but do you understand what you are talking about? TCP can be delivered out of order but the recipient will then have to wait for the complete set before proceding. that will cause lag, a perceptible delay until the data set is cmoplete, it may even wait for a retransmission. if its UDP you just push on and ignore the gap, or make it up or back fill (ie jumping about of health level).

 

[Arto]

If you haven't tested ....

Some observations I've made:
- PE/EU has used UDP for game traffic since I started ~3 years
- If you do stuff (shoot, run etc.) there are about 5 packets /s

Some thoughts.
- PE/EU isn't actually sending a lot of real time data, therefore packet sequence is unlikely to be too random / jitter prone.
- UDP is just fine when implemented well, TCP is NOT the answer for this type of real time traffic.
- If you were to decrypt / decode the UDP data content there is a high possibility you would find a simple sequencing data element.

What evidence is there to suggest Mindark don't use a packet sequence check?

 

[Metatron]

First off, in certain locations and during certain times of the day there is some serious network lag and that's to be expected until this new version is a little more mature. However, I think you have a bug in your implementation and I'll explain:

Some background for the 99.99999% of people who are not network programmers. There are 2 main types of network packets:

1: TCP. TCP is a network type that is very slow but is guaranteed to be delivered and IN ORDER. This is how versions BELOW v10 transmitted most/all data.
2: UDP. This is very fast but is prone to huge errors due to little or no error checking. this is what most of v10 uses. (this is an educated guess. I've not set up any tests but it's painfully obvious this is the case. Testing this might also be a violation of the TOS so there we are...)

The problem is that when I'm hunting in a high lag area, the UDP packets come in OUT OF ORDER. So for example,

Packet 1 says: I'm hit for 115hp and my hp is now 100.
Packet 2 says: I'm hit for 75 and my hp is now 25.

If these 2 packets are reversed due to lag it will MAKE MY HEALTH BAR SHOW 100 and not 25. The server knows it's at 25 and when I'm hit for 26 I'm dead. Laying on the ground, dead, with my health bar showing I have another 75 hp is frustrating as hell.

Solution: NUMBER THE PACKETS

This is how every single other MMO works. PLEASE DO IT FOR UDP SO I CAN SEE MY ACTUAL HEALTH!


EDIT: a discussion about the introduction of TCP/UDP mix begins at the end of page 2. This is a better solution but will take them more time to implement and is certainly not as simple. I'll bet that a year from now this is how it will work but they need to get some kind of half-way solution to this in place now.

Was planning on posting something about this. The amount of AVERAGE lag you get from a normal cable connection in the US is unacceptable in EU. I hate seeing 20% of my chat constantly spammed with yellow messages, and the rest being out of order and confusing.