And The Logs Weren't There: Part I

Seeing this story (where logs weren't available when sorely needed) inspired me to finish my own version of that...

First, let me quote the conclusions from the above post.

• "If you got important data to protect, log everything you reasonably can. All the “security” in this scenario failed and failed to help reconstruct events.
• Do you have one of those semi hands-off security appliances that you presume is working fine because you can connect to the web admin portal? Make it forward logs to somewhere.
• Do you have workstations which touch sensitive data anytime? Yes. Then boost the priority to configure central logging, stop procrastinating, then take comfort you’ll be in better shape than the poor souls at this company fighting to salvage their pride, and maybe their jobs."

Darn right! There is no better way to say it. But - guess what - despite us saying this (and pointing possible mistakes), a lot of people still choose to ignore logs. What can we use to help them, even despite themselves? We will invite our friend, Mr Fud F. Scarecrow :-) to assist us with this affair. Yes, many proclaim that people need to be naturally drawn towards doing "the right thing" and scaring people into action is not that efficient (especially, if you do this one too many times...). However, this is the world we live in and in it, FUD works. FUD sells insurance as well as safety features in cars and other products, moves compliance solutions, makes people read and update their boring DR and BC plans, and causes a lot of good overall :-)

And, indeed, one can get desensitized if you hear that "sky is falling" too often, but here is the thing: I am willing to take the risk of such "desensitization" given that sky is indeed "not quite static" :-)

So, let's look at one such scenario in this post (which is, as they say, "inspired by a true story"). Imagine you have a Windows Active Directory (AD) server (or a domain controller) that holds all of the accounts for a good part of your organization. One notable morning you get calls from dozens of frantic users (yes, including your boss's boss :-() who are unable to login to their Windows systems. Their computers reject their apparently correct logon credentials. You check the account settings on your AD box and your face turn pale: there aren't any accounts. The term would be "mysteriously vanished" :-) Where do you go next? Windows event logs on the AD server, of course. Good thinking! However, AD servers and DCs are famous for being very chatty, often producing hundreds of event log records per second on busy networks. Thus, when you look at the logs you notice that the entries older than 8 hours got rotated into oblivion. And there is nothing that points at the account disappearance within the remaining log data. Argggh!

Now, what do you do next? Do you feel at least a little fear about your job, maybe also feeling slightly uncertain of what exactly happened with your server or even doubtful that you can prevent it, if it were to happen again. Maybe your IT team has an SLA to worry about? Which has your boss's bonus dependent on it? What if it happens again tomorrow (after you painstakingly restored all the accounts based on old records, incremental backups and the info from the users)? And why can't it? - you have no idea why it happened this time... Are you 0wned now - or was it a junior admin error? Or a new Windows bug? You truly have no way of knowing, which, as we all know, doesn't help you to feel brave, certain and doubtless... :-)

Now, if only you bought that log management solution and started collecting and analyzing logs before, and not after the incident (as it really happened in the above case) Imagine that!

Technorati tags: security, chuvakin, logs, log management