How did I find out ?
Well, the hosting companies have been kind and prompt enough to send letters, informing that 2 of my servers are busy hacking other public websites.
How did I stop it ?
So first I went fishing for foreign IPs that shouldn’t be connected to my machine:
netstat --tcp --numeric
I was fortunate enough to have the hacker hacking into a virtualization server, where no public IPs should be connected, making it easier to spot the intruder.
After identifying the IP, I cold also see the port in use so I searched for the process that was keeping this port busy:
netstat -anp|grep :81[[:blank:]]
I killed that process but the next second I saw another process started.
crontab -e has shown me the living heart of the hacking script:
* * * * * /usr/include/.kde/update >/dev/null 2>&1
This is how I got to find out that my machine was trying to hack quite a few other addresses, at the speeds of a 100 Mbs, the max bandwidth.
So what was in this misterious “system-like-looking” folder?
cd /usr/include/.kde ls -lart
By looking at the bottom of the list I could spot the recently added hacking scripts:
I’ve opened each file to see the scrips and so I discovered mech.session file that was looking interesting.
It looks like someone managed to hack into the system, and installed a bot to be controlled from IRC, that was used to hack other sites, under my umbrella. I have to admit it: it’s neat ! ( just not for me )
What lessons did I learn ?
- Be aware of the OpenSSL vulnerabilities.
- Be very stricts with firewall’s rules.
- Don’t be too lazy to move SSH to a different port, disable root login and remove password authentication.
- Don’t treat security review as a nice-to-have.
- Scan your systems frequently. This intruder checklist can give you a good start, as well as this discussion.
- Create failover and back-up plans for high availability.