[CentOS] 140,000 PPS Unicast Traffic - Finding the Cause

Redgie · 2 Dec 2011 at 11:21

Dear OcUK Linux Gods,

My dedicated server yesterday spiked up to 140k PPS of Unicast traffic. This was presumed to be part of a DDoS attack and the servers port was disabled.

I've been given remote KVM access but am finding it difficult to find the cause. The techs at the hosting company identified a number of shell scripts running and disabled them but won't re-open the port until I have found the cause.

The best way I have to find it is in the apache access_log but it's 6GB and without internet access I've no way of viewing it (Trying to view it over KVM just crashes it understandably).

Is there any common linux attacks I can look for? I can ask the company to sort it but the server is used for none profit purposes and already costs me money so the idea of paying an engineer £30 per hour isn't ideal for me.

Thanks in advance to anybody with any ideas.

memyselfandi · 3 Dec 2011 at 14:10

So your system was compromised in some way and there were processes running and sending out these packets.

- have they provided any information on these shell scripts which were running which they killed off.
- Given that the system was internet facing was it patched up to date? Did it have SELinux enabled? Was any other work done to secure the system beyond the default installation?

Personally I would be looking less at this stage at what was compromised and more at securing any data and configuration settings, both of which would need to be checked in detail to make sure they have not been compromised and that the latter is not opening any security holes before being reimplemented.

The system itself ... well you have no idea on how much it has been compromised so you have no way of knowing if any binary files have been replaced hence you don't know if any commands you run are showing the truth or not, for instance for all you know there could be a patched ps command binary which doesn't show processes with certain names. I'd be installing the OS and re-securing from the ground up.

Redgie · 3 Dec 2011 at 15:59

Reinstalling it and starting from scratch was my first though, but I'll have to pay to have it reinstalled and as someone not overly familiar with Linux it was a pain to set it up originally and I'd like to avoid that, though admittedly I see the security concerns.

Since the OP I've updated a considerable amount of packages including the important ones like Apache, MySQL and PHP, I'm currently running through Nessus vulnerability reports and fixing everything it's identified, I'll also be googling round for common ways to secure PHP and Apache.

Sadly like you said, I'll always be wondering if some hidden file is going to pop up and cause all hell.

KIA · 4 Dec 2011 at 12:35

Definitely re-install and keep critical packages up-to-date. Subscribe to your distro's security mailing list.

vidda · 4 Dec 2011 at 15:46

Which port did they disable? I assumed they turned the network port off but you said you updated packages.

Check to see if any users have been added on the system, if your server has been compromised it's quite common for user accounts to be added to run rogue software. At the same time check all home directories for anything suspicious.

Redgie · 4 Dec 2011 at 19:47

Alas, further scenarios occurred and it is being formatted

. Thanks guys, I knew you were right!

101001101 · 9 Dec 2011 at 23:39

What? re-install the entire server? why? What exactly will that achieve considering the OP wont be doing the re-install? the hosting company will and will clearly profit from it. Massive overkill.

I'm confused as to how all that unicast traffic made it through to your "server", most Cisco kit will drop unicast traffic.

You're being a bit blind here, what exactly does your "server" do in the DC?

This is why its better hosting your own kit co-lo, for the sake of £50 of so a month with full access to your own server, this is what I would always opt for, DC techies dont know wtf there doing mostly.

memyselfandi · 10 Dec 2011 at 15:29

101001101 said:
What? re-install the entire server? why? What exactly will that achieve considering the OP wont be doing the re-install? the hosting company will and will clearly profit from it. Massive overkill.

You'd trust binaries and existing configurations on a server which appears to have been compromised to an unknown extent ... more fool you frankly.

A reinstall is the safe option. I don't know what level of service the OP has from whoever he is with but assuming that they just slap on a base install. For the OP this should be followed by patching, properly hardening of the system and installation and configuration of only the required services in a secure manner. No configuration should be re-used unless it has been checked to not do something unexpected.