Associate
- Joined
- 2 Feb 2009
- Posts
- 842
for those TLDR; I think there is some strange backbone routing issue that started on 19th October and I have no idea who to report it to.
My assumptions; (and yes I understand assumptions can be dangerous!)
Given that it’s only 40% of our clients that are getting this issue, (and the same clients), we’re assuming that it’s not an issue our end. If it was our end we would expect to see the errors spread across all clients.
Given that even the 40% of the clients that are erroring sometimes have periods where the connection seems stable for short periods, we’re also assuming it isn’t client firewalls or some other blocking their end. If it was firewall or blocking then it would either connect or not? (yes we've had lots of times in the past where we've lost contact due to IT people changing firewall rules!)
Anyone any ideas? or any backbone network engineers on this forum?? anyone know who to even send this kind of stuff to? (we've already emailed out own hosting company, but they're being typically, we can't see any issues from here!)
Thanks.
R.
We are having a weird network related issue with one of our servers that's hosted in a Manchester data centre.
I will explain a little about the background of how we use and connect to that server to show that the issue is a recent one.
On that server is a web service connected to a database. we have 50 of our clients calling in every 30 seconds, and each time one calls, the web service records some information. This has been the case for the last few years without any major issues.
However, since around 10.40am on 19th October 2023 about 40% of them are getting intermittent connection errors. The other 60% are working as they have been for years.
No recent changes have been made to client or webservice back-end code or database.
Below is from two clients error stats broken down by day.
Client 1
Client 2
Around 40% of the 50 clients dialing into that server are very simliar to the client 2 stats above, with a sharp increase in the number of errors.
The errors the clients are reporting are very generic,
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ourIPAddress:443”
“The underlying connection was closed: An unexpected error occurred on a receive.”
"Unable to connect to the remote server"
I will explain a little about the background of how we use and connect to that server to show that the issue is a recent one.
On that server is a web service connected to a database. we have 50 of our clients calling in every 30 seconds, and each time one calls, the web service records some information. This has been the case for the last few years without any major issues.
However, since around 10.40am on 19th October 2023 about 40% of them are getting intermittent connection errors. The other 60% are working as they have been for years.
No recent changes have been made to client or webservice back-end code or database.
Below is from two clients error stats broken down by day.
Client 1
| Client15014 | 16/10/2023 | 0 |
| Client15014 | 17/10/2023 | 3 |
| Client15014 | 18/10/2023 | 0 |
| Client15014 | 19/10/2023 | 0 |
| Client15014 | 20/10/2023 | 1 |
| Client15014 | 21/10/2023 | 0 |
| Client15014 | 22/10/2023 | 0 |
| Client15014 | 23/10/2023 | 0 |
Client 2
| Client3553 | 16/10/2023 | 0 |
| Client3553 | 17/10/2023 | 3 |
| Client3553 | 18/10/2023 | 0 |
| Client3553 | 19/10/2023 | 127 |
| Client3553 | 20/10/2023 | 152 |
| Client3553 | 21/10/2023 | 238 |
| Client3553 | 22/10/2023 | 125 |
| Client3553 | 23/10/2023 | 114 |
Around 40% of the 50 clients dialing into that server are very simliar to the client 2 stats above, with a sharp increase in the number of errors.
The errors the clients are reporting are very generic,
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ourIPAddress:443”
“The underlying connection was closed: An unexpected error occurred on a receive.”
"Unable to connect to the remote server"
My assumptions; (and yes I understand assumptions can be dangerous!)
Given that it’s only 40% of our clients that are getting this issue, (and the same clients), we’re assuming that it’s not an issue our end. If it was our end we would expect to see the errors spread across all clients.
Given that even the 40% of the clients that are erroring sometimes have periods where the connection seems stable for short periods, we’re also assuming it isn’t client firewalls or some other blocking their end. If it was firewall or blocking then it would either connect or not? (yes we've had lots of times in the past where we've lost contact due to IT people changing firewall rules!)
Anyone any ideas? or any backbone network engineers on this forum?? anyone know who to even send this kind of stuff to? (we've already emailed out own hosting company, but they're being typically, we can't see any issues from here!)
Thanks.
R.
