Server support blog

When server is compromised via some PHP script security hole intruders are getting relatively low level of access to the server. Although its enough to install PHP shell script or start some backdoor network daemon or proxy server or network scanner.

Lets see if we can find them

First step should be to disable the security hole itself – but on massive hosting server it’s not always the simple task to find the problem. So the best approach should be to work in parallel –  locate the running backdoor or proxy and find out as much as possible about the way it was uploaded and started.

The best tool for finding open network ports is lsof – “LiSt Open Files”. In unix everything is a file. The syntax is

If you have web server running on port 80 it would help to filter out these connections

Now you can look for anomalies. Don’t get fooled by the program name – hackers would try to mask the process under legitimate name to “get lost in the crowd”, so it could be anything – apache, httpd, sendmail. It just the anomalies that give up the offender, for example

• apache or any other daemon has no business to listen on the high number port it was not configured on
• sendmail run by user apache or nobody is also very suspicious.
• perl process that is listening on some network port (that’s all time popular)
• some process with obscure name,run by apache user

Unfortunately I can not find infected system to give a real example. But lsof output for the healthy system should look somewhat like this

lsof provides enough information on processes listening on TCP/UDP ports to have a shot at finding running backdoor or proxy or vulnerabilities scanner, started by hacker on compromised server – process name, user account under which this process started, port on which the process is listening, list of open file descriptors (fd).

Another benefit of running lsof to get a snapshot of network I/O on your system is to find possible performance bottlenecks. For example – if you see a lot of connections to mysql from apache process then it looks like your PHP applications are running suboptimal mysql queries and that requires attention. Same goes for connections from apache to UDP port 53 – DNS could be a bottleneck.

A lot of ssh connections on the server with shell access disabled could indicate possible brute force attack, a lot of  sendmail processes running trying to connect to outside servers – your server is used by spamming either via uploaded malicious script or … customers who are violating your Terms of Service.

There is plenty of useful information provided by lsof. Sky is the limit – happy lsof-ing.