Today I’ve been fixing a problem with my mail server after someone “hacked” (cracked) it. I’m reconstructing the chain of events as best I can, but the causality wasn’t obvious at the time.
Background: this machine is running Windows Server 2003 SP1 with Exchange Server 2003 SP2.
A few weeks ago, someone gained unauthorised access to the server over the internet. I’m not sure how exactly, but I suspect that they used a buffer overflow in IIS (since the server runs OWA). The server is up to date with all the relevant security patches, so that shouldn’t be possible, but something obviously went wrong.
This enabled the attacker to install 3 new services, configured for Automatic startup:
Name: Desings
Path: c:\windows\system32\dllcache\NTNetworksrv.exe
Description: [blank]
Name: Network Manager
Path: C:\WINDOWS\system32\network\network.exe
Description: “If this Service will be stopped your TCP/IP Network won’t work correct.”
Name: NT NetworkService
Path: C:\WINDOWS\system32\dllcache\NTNetworksrv.exe
Description: “Provides Windows-Events log and allows Administrator to Access to this computer”
The “Network Manager” service didn’t run, because the path didn’t exist. The other 2 services were both trying to run the same file, but DEP (Data Execution Prevention) stopped them from running. I saw the error messages regarding this, and I did a Google search for the filename, but I couldn’t find any information about it. In the absence of any malware reports, I foolishly assumed that it was ok, so I added an exception, telling DEP that it was ok to trust this program.
On Saturday, I rebooted the server. I tested it afterwards, and everything was working fine. However, this reboot meant that Windows tried to start all the Automatic services, and this time it successfully ran the “Desings” service. (It couldn’t run “NT NetworkService” because the file was already in use.) This then opened up a “back door”, allowing the attacker extra access. I think this must be a manual attack, because everything was fine yesterday (Sunday); looking at the timestamps on files, I don’t think the attack started until late last night.
The attacker used the back door to log into the server, using the local Administrator account. They modified the shortcut for Terminal Services, and they changed the local firewall settings.
The way that Windows Firewall works is that you can set it up with exceptions. So, you either say “Port X is open” (they have a predefined list for common protocols like HTTP), or you add programs to a list. When the server receives traffic on (say) port 80, it can then work down the list and say “Best Practices Analyzer Tool, do you want to accept this? No? How about you, IIS? Yes? Ok, here you go then.” So, there was supposed to be an entry in there for Exchange, but the attacker removed it. After that, port 25 didn’t work (for incoming email), port 135 didn’t work (for Outlook to use RPC), and ports 80/443 didn’t work (for OWA).
The attacker also added a couple of new firewall exceptions for “File Transfer Program”, where the path was something like this:
C:\Documents and Settings\Administrator\Desktop\sp33dftpc.exe
Presumably this was a way to copy their programs (hacking tools) onto the server.
Finally, the attacker created a new folder on the desktop:
“@admin your box is vuln…pay me 50 $ and I tell you the vuln”
Sadly they neglected to leave me any contact details, so I wasn’t able to show my appreciation.
I discovered the problem this morning. Curiously, Outlook 2000 could connect to the server ok, although Outlook 97 and Outlook 2003 couldn’t, so presumably that’s using a different protocol. I then spent about 6 hours on the phone to Microsoft technical support trying to sort it out. They weren’t amazingly helpful, e.g. the first person wanted to email me a summary of the support case so that I could confirm it was correct. I had to explain that I wouldn’t be able to read his email message unless he sent it to an alternate address; I’d hoped that this would have been obvious. They were able to point me in the right direction, by identifying this as a firewall problem, but their proposed solution was to disable the server’s firewall altogether!
I found the 3 rogue services fairly quickly: I ran msconfig.exe, then went to the Services tab, and checked the “Hide All Microsoft Services” box. However, I didn’t immediately recognise them as malware. The poor spelling and grammar should have been a warning sign, but a lot of companies have outsourced tech support to foreign countries (e.g. India) where people don’t speak English as their first language, so I initially thought that it was just a recruitment problem.
Similarly, I did a Google search for “Desings” which found several companies with misspelt names (e.g. “Fountain Desings” rather than “Fountain Designs”) but I didn’t find anything about this application. I figured that if it was trying to run the same .exe file as a legitimate service then that was ok. This was again stupid of me, since I had my logic backwards.
More generally, this fits a pattern I’ve noticed on other infected machines – malware tries to hide by pretending to be legitimate. A common example is that you see “netlogon.exe” in Task Manager, and assume it’s ok – the real one lives in the “System32” subfolder, so anything in the main Windows folder is fake.
When I realised what was going on, I stopped those services and deleted the underlying files. Similarly, I removed the firewall exceptions for their FTP programs and deleted those files too. I recreated the exception for Microsoft Exchange, and then I was able to send/receive email again. I’ve also reset the password for the local Administrator account and restored the shortcut for Terminal Services.
I’m annoyed with myself for adding that DEP exception. However, in my defence, that file should never have been allowed onto my server in the first place, i.e. that was poor programming from Microsoft.
As a mitigating factor, my email should be safe, because it requires domain admin (rather than local admin) privileges to access it. More generally, I don’t think that the attacker was trying to spy on my email, since he stopped it from working; maybe he just wanted my server to join a bot network or something. To be honest, I think I got lucky with a relatively incompetent attacker, so this has served as a wake-up call without any data loss.
However, wishful thinking isn’t a great security policy. This is where the line from Aliens comes in: “I say we dust off, nuke it from orbit. It’s the only way to be sure.” In this case, that means backing up the data, reformatting the hard drives, and ensuring that I only load programs back on that are known to be good. The downside is that this will take a while, but I think it’s necessary.