Anton Security Tip of the Day #9: But He "Wasn't Logged!"

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #9: But He "Wasn't Logged!"

The idea for this tip originated when my presentation on log analysis was rejected by one of the high-profile security conferences on the grounds that "logs don't matter since advanced attackers never leave traces in logs [or erase them before anybody can get to them] ." Indeed, some of my security friends of a more "offensive orientation" :-) have long developed this snobbish attitude about logs.

So, imagine a network that has fallen victim to a 0day-wielding "uber-haxor" :-), who kicked the door open (or snuck in), grabbed the crown jewels and took off like a bat out of hell :-) When, much too late as usual, the "good guys" rushed in to pick up the pieces, only there was seemingly nothing much to pick: the server logs were erased and the network IDS didn't make a peep (and, no, Richard, NSM was not deployed). What do you do now?

So, let's list some uncommon (and some common, but often untapped for the task at hand!) sources of log data and provide a few log analysis tips:

• firewall logs: boring they might be, but it is less likely that those are erased by the attacker (even though firewall logging can sometimes be jammed). And it is hardly possible to "not be logged" by the firewall (the analyst's challenge is in analyzing the huge volume of data to find the attacker's traces and to tell them from normal traffic). What is critical in making these logs useful is logging allowed connections, not only blocked ones.
• router logs and netflow records: same as firewalls logs - they are less likely to be erased, but huge log volumes make their use problematic. And, last but not least, these types of logging are more likely to be turned off completely ("routers should route, not log," grumble many network types)
• application logs: was it a "legacy" network attack or something application-level? If the latter is true, there might be logs found in unusual places. Is there anything in the webserver logs? Websphere made a peep? MySAP recorded something? PHP app logged something fun?
• various client logs: in one old case FTP client logs were extremely helpful, even though the entire syslog directory was blown away and no remote logging was ongoing; look for other client logs (yeah, even a web browser history is kind of a log...) to look for things missed by the attacker
• other systems' logs: was there anything that the attacker did that affected other systems? Might have they logged it? Even an attempted SSH connection or a scan from a compromised machine has a high chance of leaving traces elsewhere - cleaning all of such traces is a major challenge for an attacker (in any case, much harder than erasing logs and stopping logging on the compromised host)
• any remote or centralized logging: was any log saved from destruction by being copied to another system?

To conclude, while there is no "logwatch" pattern for "advanced attacks," logs are still very useful in such circumstances if you prepare by setting up a broad scope of log collection (I suspect using a log management system will be your only choice as log volumes will be pretty bone-crashing) and then combing through the logs after the incident. And remember the less common sources of log data, such as database logs.

I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.