Massive Lag Spikes/Disconnects

[Resolve]

Just about 20 minutes ago, most of my guildies including me got DC-ed from the server. I have been DC-ed previously for about 6x in the past hour. I encounter spikes every minute lasting from 2 sec to 10 sec. I used speedtest.net to test my latency and and I got:

For my local ISP:




For Dallas, TX:




The lag spikes' frequency has gone up so much so that Amistr is now unplayable to me for more than half of a day. The spikes happen in off-peak hours of the server, when there arr just about 60 or less online.

[Resolve]

Its getting worse. I got disconnected at least 20 times in the past 2 hours.

Tinker Bell and Headache~ have also reported spikes, and theyre in London UK.

[Koma]

I have been noticing something odd at night as well. I'm looking into it and also re-balancing the server this week. The firewall and database server have been reconfigured to utilize resources better also.

A few days ago, there was an outage at around 8AM (@time) caused by the map-server disconnecting from the char-server for about 5 minutes (since it recovers). This may be related.

[rephikul]

What kinda host doesnt have their own backup power supply???

Hosts and ISPs also tend to do their maintainance at night cause not many traffic at the time. My internet for instance go off for an hour or two at roughly 3apm every month.

[Koma]

Had nothing to do with power -- hence why the servers still have their uptime -- unless you mean their datacenter's machines. It's possible that it's routine maintenance but this is recurring too often.

[rephikul]

Had nothing to do with power.

Oh right I misread your post.

Here's what you wrote.

A few days ago, there was an outage at around 8AM (@time) caused by the map-server disconnecting from the char-server for about 5 minutes (since it recovers). This may be related.

Here's what I read

A few days ago, there was an outage at around 8AM (@time) caused the map-server disconnecting from the char-server for about 5 minutes (since it recovers). This may be related.

[Resolve]

Had nothing to do with power -- hence why the servers still have their uptime -- unless you mean their datacenter's machines. It's possible that it's routine maintenance but this is recurring too often.

I now have 7 chars in my merch fleet on vend, because i cannot do anything else. Setting them up took took me about 100 logins just because I dc so often. One sucessful login sometimes last for just 10 seconds.

Diablo and z e r o have also reported a similar case. The new guild "cry" have also been complaining about spikes. I just defended admin to the newbies by telling them that you are already fixin it. Good thing I was helping them start out so they were ok with it.

But this can only go so far, as we all know.

[Koma]

I've been talking with Resolve about this situation and it appears to be due to ISPs peering a bad route to the server. To be exact, it is the AS6453 (Tata Communications) line in his case which links Asia to the USA. Some Asians don't experience this issue as they are routed through NTT instead. He has tested the issue with other RO servers which use NTT and AS6453 and found that the servers with AS6453 routes had the same exact problem as here.

Person #1 -> ISP-> NTT -> USA -> Server (No Lag/Disconnections)
Person #2 -> ISP-> AS6453 -> USA -> Server (Lag/Disconnections) (Resolve's Path)

As far as us at Amistr's Administration goes, our end went through a better path to Resolve:

Server -> USA -> NTT -> ISP -> Resolve

There's not much we can specifically do in this situation except hope that those trans-Pacific lines get better OR the ISPs in question peer through NTT instead of AS6453 -- besides doing another server move which IS plausible but we'll have to see. From what I've discussed with him, it is similar to the issues Frantech experiences with Filipino customers since around July -- and we determined that they have the same route.

One other contributing factor to this is that peak load of that network line is causing slowdowns too. Originally mentioned was that this was happening during high network traffic business hours locally -- but also happened on the weekend.

This was a no-issue with the last datacenter our server was hosted at due to its location in which ISPs peered differently to it as well -- however the tradeoff was having other players connecting through slower routes (Europeans were travelling through CogentCo which caused a large deal of our issues back then).

These are just the details for any Asian lag. I have yet to investigate the root cause for European (German) lag.

[Resolve]

I did a tracert on amistr's server from my pc. It shows that my packets go through the TATA [as6453] communications cable. I asked z e r o [not lagging.dc-ing] , who also lives in the Philippines to do a tracert of his own and yet his packets go thru the NTT line instead. We have the same ISP and we share pretty much the same DSLAM.

My Route:


1 57 ms 56 ms 57 ms 112.***.***
2 57 ms 58 ms 60 ms 119.**.***.***
3 59 ms 57 ms 58 ms 58.71.0.144
4 223 ms 222 ms 222 ms if-12-0.mcore3.LAA-LosAngeles.as6453.net [209.58.85.49]
5 227 ms 225 ms 223 ms if-6-0-0-811.mcore4.LAA-LosAngeles.as6453.net [216.6.84.118]
6 256 ms 258 ms 258 ms if-1-0-0-701.core1.DTX-Dallas.as6453.net [216.6.53.45]
7 259 ms 259 ms 259 ms if-1-0-0-825.core2.DTX-Dallas.as6453.net [66.198.2.57]
8 262 ms 264 ms 260 ms Vlan31.icore2.DTX-Dallas.as6453.net [66.198.2.38]
9 258 ms 257 ms 256 ms 4.68.63.161
10 257 ms 258 ms 261 ms ae-4-90.edge3.Dallas1.Level3.net [4.69.145.200]
11 258 ms 258 ms 257 ms COLO4-DALLA.edge3.Dallas1.Level3.net [8.9.232.74]
12 * * * Request timed out.
13 * 259 ms 259 ms ns3.***.com



Trace complete.

z e r o's

1 7 ms 9 ms 8 ms 192.***.***.*
2 9 ms 24 ms 10 ms 121.**.**.**
3 12 ms 5 ms 7 ms 121.54.3.245
4 20 ms 12 ms 11 ms 203.87.132.101
5 16 ms 11 ms 10 ms 119.92.129.117.static.pldt.net [119.92.129.117]

6 9 ms 15 ms 5 ms 58.71.0.143
7 196 ms 196 ms 200 ms p16-0-1-3.r20.snjsca04.us.bb.gin.ntt.net [129.250.12.105]
8 198 ms 203 ms 194 ms ae-1.r21.plalca01.us.bb.gin.ntt.net [129.250.5.32]
9 * * * Request timed out.
10 223 ms 225 ms 219 ms hagg-03-ge-1-0-0-459.dlfw.twtelecom.net [64.129.234.20]
11 229 ms 223 ms 224 ms 206.123.64.21
12 * * * Request timed out.
13 239 ms 223 ms 225 ms ns3.************************

I realised [using IP locators] that the IP 58.71.0.143 is PLDT's central station or w/e its called so its pldt who decides where my packets go to. I have called PLDT and requested a rerouting and I am still waiting for a reply.

This issue, says the tech support guy, was probably cause by the massive packet rerouting which was performed when southeast asia's submarine APCN2 communications cable had a fault last august due to inclement weather. I was normally routed to NTT all the time, and now I am being routed to TATA because of that rerouting.

The Tech Supp guy has filed a request for the PLDT tech team to switch my ports or something, and im still waiting for approval. I hope it works.


P.S.
I have reason to believe that packet routing is the issue and not spyware because I just did a thorough scan of my system for spyware and the best that the 3 spyware detection software found is just McAfee.

[Resolve]

Is it possible Koma that you acquire another IP address that routes me via NTT?

Because *bleep*RO's servers are also in dallas[the 1st one that I mentioned which was working fine], and yet they get me routed via NTT.


The bottomline here is that most filipinos have dynamic IP's and so their connectivity to amistr will at be grossly unrealiable 50% of the time, as our packets go thru either TATA [VSNL Transpacific] which is problematic and NTT[Trans-Pacific Express] which is the workable line.