been hacked / rootkit

pallys

pallys

pallys

Associate
Joined
2 Jul 2004
Posts
1,430
My server has been compromised. Running FC4, Kernel 2.6.17. I logged in via terminal and noticed the last login was via an unknown IP to me so was suspicious.
Then I tried to shutdown machine and got the following:

Code: 
Broadcast message from root (pts/0) (Tue Dec  9 18:50:41 2008):

The system is going down for system halt NOW!
/dev/null
[COLOR="Yellow"]****[/COLOR]: Can't open /dev/kmem for read/write (2)
[root@linux ~]#

What can I do now? I was just thinking of upgrading inplace to FC10. Data wise I'm OK, but the server scripts have been customised a lot and I feel like I will lose too much custom setup if I have to re-format and go through it all again.

Is it possible to just clean and remove all the offending bits etc?

Many Thanks.

Output of Chkrootkit below:

Code: 
[root@linux chkrootkit-0.48]# ./chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `crontab'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not found
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... INFECTED
Checking `inetd'... not tested
Checking `inetdconf'... not found
Checking `identd'... not found
Checking `init'... not infected
Checking `killall'... not infected
Checking `ldsopreload'... not infected
Checking `login'... not infected
Checking `ls'... not infected
Checking `lsof'... not infected
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... INFECTED
Checking `named'... not infected
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... not infected
Checking `pstree'... INFECTED
Checking `rpcinfo'... not found
Checking `rlogind'... not found
Checking `rshd'... not found
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not infected
Checking `tcpdump'... not infected
Checking `top'... INFECTED
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `vdir'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'... /etc/ld.so.hash
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... Possible t0rn v8 \(or variation\) rootkit installed
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi/auto/Net/SSLeay/.packlist /usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist

Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... Warning: Possible Showtee Rootkit installed
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for LOC rootkit... nothing found
Searching for Romanian rootkit...  /usr/include/file.h /usr/include/proc.h
Searching for HKRK rootkit... nothing found
Searching for Suckit rootkit... Warning: /sbin/init INFECTED
Searching for Volc rootkit... nothing found
Searching for Gold2 rootkit... nothing found
Searching for TC2 Worm default files and dirs... nothing found
Searching for Anonoying rootkit default files and dirs... nothing found
Searching for ZK rootkit default files and dirs... nothing found
Searching for ShKit rootkit default files and dirs... Possible ShKit rootkit installed
Searching for AjaKit rootkit default files and dirs... nothing found
Searching for zaRwT rootkit default files and dirs... nothing found
Searching for Madalin rootkit default files... nothing found
Searching for Fu rootkit default files... nothing found
Searching for ESRK rootkit default files... nothing found
Searching for rootedoor... nothing found
Searching for ENYELKM rootkit default files... nothing found
Searching for common ssh-scanners default files... nothing found
Searching for suspect PHP files... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... not infected
Checking `lkm'... You have    14 process hidden for readdir command
You have    95 process hidden for ps command
chkproc: Warning: Possible LKM Trojan installed
chkdirs: nothing detected
Checking `rexedcs'... not found
Checking `sniffer'... eth0: PF_PACKET(/sbin/dhclient, /sbin/dhclient)
Checking `w55808'... not infected
Checking `wted'... chkwtmp: nothing deleted
Checking `scalper'... not infected
Checking `slapper'... not infected
Checking `z2'... chklastlog: nothing deleted
Checking `chkutmp'... chkutmp: nothing deleted
[root@linux chkrootkit-0.48]#
 
Firstly you probs wanna edit the sweary out :P

Secondly, once you get a rootkit you can no longer trust the systems integrity in any way.

IMO your only solution should be a complete reinstall from trusted media. There may be a way to remove the suckit rootkit but its not a wise choice really.

I would suggest finding out how you were hacked first, learning from it and make sure that same hole isn't open again after your reinstall. Did you have any poorly passworded accounts? Might be worth checking through /var/log for access attempts as theres a chance they haven't been edited.

Also I highly recommend you read this post: http://www.linuxquestions.org/questions/showthread.php?p=935904#post935904
 
Last edited:
tntcoder said:
Firstly you probs wanna edit the sweary out :P

Secondly, once you get a rootkit you can no longer trust the systems integrity in any way.

IMO your only solution should be a complete reinstall from trusted media. There may be a way to remove the suckit rootkit but its not a wise choice really.

I would suggest finding out how you were hacked first, learning from it and make sure that same hole isn't open again after your reinstall. Did you have any poorly passworded accounts? Might be worth checking through /var/log for access attempts as theres a chance they haven't been edited.

Also I highly recommend you read this post: http://www.linuxquestions.org/questions/showthread.php?p=935904#post935904
Click to expand...

In this situation all required wording would be needed to help solve the problem.

http://www.linuxquestions.org/questions/linux-security-4/unload-suckit-problem-506456/ - This is your only option **edit** the above given is also a must read.

http://seclists.org/incidents/2003/Oct/0072.html
 
Last edited:
Linux hacked? Surely not! :p

If your custom scripts are not too long, you could save them. When you reinstall, go over them with an electron microscope.

Did your server allow remote access ( ssh ) ?
 
OK thanks guys, will bite the bullet and do a reformat and reinstall fresh, and customise the box again.

eXor said:
Linux hacked? Surely not! :p

If your custom scripts are not too long, you could save them. When you reinstall, go over them with an electron microscope.

Did your server allow remote access ( ssh ) ?
Click to expand...
Yeah I needed it to have SSH access (with password). My only mistake I can think of is allowing Root login. I would never had expected that to be compromised, so we cannot even trust SSH anymore :confused:, behind router firewall, ports were blocked except 80, 21, 22, server had not run any new executables for the past 2 years.

Rich43 said:
FC4 is very old, i doubt its still supported. you got what you diserve.
Click to expand...
It was updated to FC7 via YUM, the Kernel is reasonably new 2.6.

Can someone recommend me a different server dist? (Main requirement is LAMP) Might be a good time to change now.

Never had this problem with my window boxes and there used for the majority of the time always doing 'new' things,

I can scarcely believe it :eek:, its supposed to be the other way around :(
 
deadite66 said:
move SSH to a non standard port or use RSA keys and use something like partimage to make a backup you can quickly restore the system with.
Click to expand...
I'm a bit of a linux n00b, but I'd like to to this also (minus the part image bit).

I'm guessing it's easy enough to get puTTY to connect to a non-standard port?

eg 10.2.5.1:187 for port 187?

Is there a tutorial on how to change SSH ports?
 
MikeHunt79 said:
I'm a bit of a linux n00b, but I'd like to to this also (minus the part image bit).

I'm guessing it's easy enough to get puTTY to connect to a non-standard port?

eg 10.2.5.1:187 for port 187?

Is there a tutorial on how to change SSH ports?
Click to expand...

Just edit /etc/ssh/sshd.conf and change the port in there. The best solution as said though is RSA keys.
 
Debian makes a nice LAMP server, once you figure out what the package names are.

Do a base install, avoid any servers running which you don't know about.
 
tntcoder said:
Just edit /etc/ssh/sshd.conf and change the port in there. The best solution as said though is RSA keys.
Click to expand...

Well, I did try that but /etc/ssh/sshd.conf does not exist - I found /etc/ssh/ssh_config tho, and found the whole thing was commented out, but I found the "port" part, so I changed it.

I then rebooted the machine, yet it still stays on port 22?

Here's be file:

Code: 
# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
    Port 123
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
#   MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no

I'm running Xubuntu, I was wondering if I'm doing this wrong?
 
You could do what I do when I do a complete distro upgrade on my home server and just pull out the OS disk, (backup all data first). Replace with a new disk and install the new OS version. Import your data disk structure back in, (admittedly I have no data on the same physical disk as the OS so I just re-import the mdraid array).

Then slap the old disk into a usb caddy, mount it (with the noexec option to preventing running anything which has been compromised) and compare scripts which you have modified and migrate any changes you want over.

With changing the SSH port for external connections you do not need to change the configuration of SSH via sshd_config at all. On the port forwarding on you router change the forwarding rule to route incoming packets on port, say, 42994 to port 22 on the server. SSH Connections from outside your network will have to come in on port 42994 but you don't have to change the configuration on the server from the standard.
 
BigglesPiP said:
Debian makes a nice LAMP server
Click to expand...

Another vote for Debian. They take security very seriously ( SSL fiasco notwithstanding :p ).

MikeHunt79 said:
minus the part image bit
Click to expand...

You need a good backup solution, especially if you have spent hours customizing.

tntcoder said:
Just edit /etc/ssh/sshd.conf and change the port in there. The best solution as said though is RSA keys.
Click to expand...

Changing the port will only slow down an attacker. They can still scan every port on the machine. Having the firewall drop unsolicited packets ( instead of rejecting them ) may put off an impatient attacker, but you can't rely on this.

Have a look at RSA keys and / or port knocking.
 
^ Well I'm going to change to a non-standard port for now... Once I understand how RSA keys work I'll add that also, but a non-standard port is better than nothing. :)
memyselfandi said:
With changing the SSH port for external connections you do not need to change the configuration of SSH via sshd_config at all. On the port forwarding on you router change the forwarding rule to route incoming packets on port, say, 42994 to port 22 on the server. SSH Connections from outside your network will have to come in on port 42994 but you don't have to change the configuration on the server from the standard.
Click to expand...
Ooo, nice idea, I think I'll give that a go first. Thanks. :)
 
Recommend a server distro? There is only one in my opinion, Centos (free RHEL) so updates for as long as RHEL is supported (ie a very long time)
 
eXor said:
Changing the port will only slow down an attacker. They can still scan every port on the machine.
Click to expand...
That is true but it' a good first line of defense - I learnt this ages ago as I was getting a lot of attempts on port 22, so I changed it to [ANOTHER] and never was troubled again.
Naughty people can't scan every port on all boxes in the world, it would take too long - that's why they (normally) pick the easy ports and just scan them.

Definitely do not allow access as Root, it's quick enough to su from an account that has no other rights than to have ssh access. Add RSA encryption and access from only specific IP addresses and you should be a lot more secure.

Another vote for CentOS (free)as a server distro, it is rolled from (99%) the same source as Red Hat Enterprise Linux so it's tested at enterprise level :)
 
Thanks for the tips guys. Went with Debian and it seems so much more organised than FC.

Have disabled root login, and using fail2ban (like DenyHosts but better).

Better make a backup image this time...
 
You must log in or register to reply here.
Back
Top Bottom