More on World's Ugliest Logs

To follow the old thread on ugliest logs, I just saw this log entry:

...done

(yes, that's it!)

If you are a developer who produced it, please die. Be done :-)

More on ugly logs in the future!

Cyberterrorism + Postmodernism = ?

I am reading a paper about connections between "Cyberterrorism" to "Postmodernism."

Seriously!!

:-)

Admins , Good Guys or "I am NOT an Idiot!"

This is a follow-up to this ("On Doomsaying (Terry Childs case)") and this ("So ... Am I? Maybe I Am!"), both related to Terry Child case, as well as a response to this post  by Paul Venezia ("The anti-admin stance and the Childs case").
First, let me disclose something - my frantic efforts with the Paint allow me to proudly proclaim: I am a certified, trusted "Good Guy":

Good guys, let me tell you, do not need any controls placed on them; they are "trusted." Don't you have to trust somebody? Why not trust a sysadmin, for example?
So, what about controls? Ah, glad that you asked! "Controls" are for the bad guys; they are in place to prevent the bad guys from doing "an unspeakable evil" (tm) :-) on you. On the other hand, good guys are doing "the right thing" every time - why monitor them? It goes without saying that nobody ever moves between these groups, especially, not from "good guys" to "bad guys."
As I am rambling about this, many of my security-minded readers are wondering "what is Anton up to? Isn't it kind of OBVIOUS that controls are for everybody?" Controls know no good/bad! For example, a network control, say a NIPS, will block malicious web access due to a typo in a URL (by - gasp! - a good guy) or due to determined malicious hacking.
I think a few of my readers have watched one too many "Batman" movies and have acquired the dark side of the "IT hero" mentality." How about getting an "IT employee" mentality? If your boss is an idiot (and Terry's managers definitely seem pretty far gone in that direction...), than your "heroic duty" is to let them impale themselves on a sword of their idiocy, not to commit crimes (even if cybercrimes) to prevent that idiocy. Really, go find another job if you do not like the environment; good admins are needed in many places. For example, if your boss insists on posting all VPN passwords for all users publicly out of his sheer and unfathomable stupidity, it is your duty to tell him that it is "a very bad idea" - and not to change all passwords and not let him see it. "Doing you job" despite your boss and despite the law just doesn't work...
In other words, I want a banker making policy decisions at a bank, not a sysadmin. If a banker makes a wrong decision, his will suffer. If he is an idiot, he will most likely make the wrong decision. However, it is NOT the admin's decision to make - he does not "own" the business.  BTW, the fact that it is a city, not a bank, and it is taxpayer funded, does not change it.
Am I "anti-admin" for saying that admins should not run the business?  Am I "anti-admin" for saying controls (at least logging/auditing) on administrator activities are needed?  You call it "anti-admin", I call it common sense!!  Pray tell me, what makes admins float above accountability, control and  IT governance?
Please also read what Randy Smith said about this issue; a lot of good thoughts that I agree with.
Now I would like to respond to specific comments from my readers:

"What rankles your readers is how blithely you imply this problem has a simple or effective solution. It doesn't, all the processes or tools you advocate can do is speed up the time it takes to detect the lock-out, but not actually prevent it - i.e. they are ineffective at tackling the primary problem."

That is correct; the rogue admin problem has NO simple solution. You might prevent some (few, really) things, you might log some of them and then figure what happened, but there is no simple solution (it goes without saying that "just trust them" is NOT a solution...)

"We all know companies run without sane risk management all the time and are rarely held accountable in America. What makes you think anyone is "screwed"?"

Well, this is a good point; maybe I let my idealistic side take over. But, come on, just the fact that bad IT governance is somewhat common, doesn't make it right!

"Now ask yourself who is "screwed" by one person at a small company having all access and no accountability on a network. That's how I run my home network. Big deal."

Nobody is. I addressed it here. The risk is acceptable for smaller environments, usually. I don't have an overseeing body set up to control my home passwords :-)

"You seem to forget that sometimes the management just has to trust somebody. "

Addressed above.

"Chuvakin, you're a tool. Given the recent idiocy of the releasing of the VPN names and codes, it obviously shows that any sort of detest that Childs had against his superiors at the city were justified."

The fact that his bosses are idiots (which seems fairly well established!) does not make him right!
Bad boss + admin out of control =/= right :-)

"This is not a private organization. His superiors don't own the company and are NOT entitled to the data. We are, the taxpayers. And as a California taxpayer I fully support someone with the paranoia and technical skill of Terry Childs over a group of bureaucrats who release secure information to the public."

Properly evaluating this statement requires a law degree. Thus, no comment. Bureaucrats suck, but rogue admins are not a solution to that. Really!

"The guy was doing his job and doing it incredibly well, and keeping it out of the hands of those who, given their most recent choices, would bring potential disaster to the city."

He was NOT, unless crime is part of his job :-) Also, see comments on "IT heroes" above. If your boss is an idiot AND you don't like it, quit.

"Anton Chuvakin seems to think that all admins should be kept underneath management's boot at all times. [...]  Managers can't and don't understand what we do, and thus eventually come to the conclusion that we can't be trusted with our own knowledge. [...] Perhaps it's human nature to fear what you don't know or understand -- and that's why management can develop a fear of their own employees."

You say 'fear of employees', I say "insider risk management." You say "trust employees", I say "trust but [be able to] verify (=log)"

"his blog leads the casual reader to infer that their businesses are in danger of being hijacked by disgruntled Sys Admins and that isn’t the case." (from here)

Eh, not all businesses, but some businesses - definitely (hmm, see Terry Childs story or other published insider attack cases, all the way back to Omega Engineering case and maybe all the way back to ancient history)

"I despise people like Terry Childs, but despise Chicken Little’s like Anton Chuvakin even more." (from here)

You say  I am 'chicken little', I say "if your boss ignores insider risk management, he is stupid and deserves his business to fail."  I also add "if you think admins are 'above the law', you have a good chance of 'turning rogue' yourself AND then ending in jail."
Finally, this and my other posts about the case are inspired by on the media reporting; I possess no "insider knowledge" on this case  whatsoever.

UPDATE: Terry Childs found guilty.

Possibly related posts:

• "On Doomsaying (Terry Childs case)"
• "So ... Am I? Maybe I Am!"

Log Management - Day 1

Inspired by this and this here (and this too). It started from Jeremiah saying this:

“You’re hired on at a new company placed in charge of securing their online business (websites). You know next to nothing about the technical details of the infrastructure other than they have no existing web/software security program and a significant portion of the organizations revenues are generated through their websites.

What is the very first thing do on day 1?”

At about the same time, I saw a message posted to one of the mailing lists where the poster wondered: "I’ve been asked to look into finding a replacement to our current log management/auditing system.  This is a field I haven’t even come close to touching before, and really don’t know the ideal things to look for (or ignore), etc. I’ve been searching through SANS site as well as googling, and I’m not coming up with a lot of great starter information. " And then he asks "Where should I start?"

This is indeed a really good question!  Let's rephrase the above for the case of logging:

"You’re hired on at a new company placed in charge of TAKING CONTROL OVER THE LOGS. You know next to nothing about the technical details of the infrastructure other than they have no existing LOG MANAGEMENT process and tools... What is the very first thing do on day 1?”

So the "Day 1" of log management project. What's up?!

The very first thought that should cross you mind before you even do whatever first thing you wanted to do is "WHY?" (don't people hate those 'Why?" questions - focusing on "What?" or "How?" is soooooooo much easier....)

"Log management" is a solution, not a problem. What is your problem that you now have a mandate to solve?

Logs don't just drop on people :-) Well, not often.

What is it that motivated your boss (or his boss, or whoever) to decide to "address this", to "take control over logs?" Was it a new compliance mandate, PCI perhaps? Was it a recent incident where investigation hit the wall due to utter lack of logs? Was it a new corporation-wide IT efficiency improvement project? Was it a lawsuit where an e-discovery request was not satisfied and thus fine was levied? Was it a hot IT project that is impossible to complete without having a tool to analyze logs?

This "need" is very important since logging is a huge realm and not focusing on the need is akin to starting a journey into a hostile wilderness without  a map - in other words, it might be fun for a while, but it can end badly :-)

Next, what do you actually do first? Figure out what logs are needed for this effort and what systems produce them (and who "owns" them!) Analyzing SAP logs for J-SOX is a VERY different effort from analyzing Cisco ASA logs for network troubleshooting.

Only at this point you can start thinking about "tools:" parsers, logs, databases, reports, alerts, indexing and other technical thingies as well as capacity planning, scalability, etc. This is the stage where you learn the lingo and learn to cut through marketing messaging to get to the actual tool capabilities.

So, remember: given mandate to "tame the logging monster", think "WHY?" first!

So ... Am I? Maybe I Am!

Now, a lot of people who work for small businesses called me an idiot for this.

And you know what? Maybe they are right :-)

When I was a sole sysadmin for a small ISP, I didn't share my passwords with management either. They never asked ... but that is not the point. I would not have passed "a bus test", which is "will a business still run if a sysadmin is hit by a bus" [or, "goes rogue", by whatever definition of "rogue"]

Keeping all this in mind, will you accept if you bank closes doors until they can figure out what the password is on their database? Didn't think so ...

So, my point was that, in my opinion, it is an unacceptable risk for all but the smallest organizations to have one person who have the power to control access to critical systems AND to place no controls (neither monitoring, auditing nor preventative) on his activity.

AND that is why, back in my ISP days, one day a boss came to me with an old ragged notebook and said "write down the passwords here." I did. The notebook went back into his pocket (and then, presumably, in some more "secure storage," like the back of his closet at home or something :-))

On Doomsaying (Terry Childs case)

Maybe I should call it "on stupidity" and add it to my "Nobody Is That Dumb... Oh Wait" series?
Really, when I've heard about it first, I was like "ah, come on, I am sure the journalists are just mis-reporting it; nobody is that dumb in their approach to system security."
Well, they really were that dumb.
Honestly, from the "blatant disregard of common sense", this is very, very high on the list (many in security agree, some in IT disagree). This is where the words "a huge data security risk" really sound like a mild understatement.
But you know what is the most scary about this case? The fact that there are MANY organizations who manage their networks the same way: one admin with ALL the access and NONE of the monitoring.
One person + ALL access + NO accountability = you are screwed!
Also, in light of this, do you still think that "insider attacks" is some kinda security vendor propaganda? Well, go tell Terry Childs that :-) Even though some people still think that he is a good guy (more on that on Slashdot)
What also caught my attention is that some retard called his bail ($5m) "ridiculously high." Well, if he was an outside hacker, say a Romanian script kiddie, jailed for hacking SF network, would they release him for $5m? Maybe not! Now, do you get that this case is actually MUCH WORSE! Hacker might have gained access to many assets; this guy did have access.
So, think, think, think: CAN YOUR SYSADMINS "0WN" YOUR BUSINESS? (BTW, some people think that IT "owns" you already!)
Are you OK with it?
If not, do something - start logging and monitoring (and then controlling) their actions! If you think you cannot control them, then just monitor; if you think you can neither control nor monitor, then at least log them so they will know that there will be enough good evidence to let them rot in jail for many years ... Or, if you prefer an easier alternative, stop calling your business YOUR business.

UPDATE: a bit more on this from me is here, I am working on an additional piece clarifying what I said as well. Stand by! Also, thanks for arguing with me here. I will argue back tomorrow!

UPDATE: Terry Childs found guilty.

Possibly related posts:

• "Protecting logs from admins"

More on Logging and Accountability

Here is a very fun web conference on logs and logging by ISC(2); all sections are interesting, here is what mine is about:

"There are many other mechanisms of accountability in an organization, but logs are the one that pervades all IT. And if your IT is not accountable, your business is neither. Thus, if you tend to not be serious about logs, be aware that you are not serious about accountability. Is that the message your organization wants to be sending? The presentation will cover how logs can be used organization-wide to establish accountability of users, power-users, other IT as well as partners and others accessing systems and using your information. How to you make sure your users are accountable for their actions? How can you track their activities, if needed? How can auditors review the audit trails of various activities? Broad organization-wide log collection and analysis is the way to solve these and other problems related to accountability."

Enjoy!

Which Blogs Do I Read?

Somebody asked me what blogs do I read? I figured I'd post my answer here:

1. First, a bunch of security blogs (actually, the amount did SHRINK a bit compared to before - security blogosphere is too darn noisy and the signal/noise ratio is dropping thru the floor ...): here is the link
2. Travel blogs: here
3. A few blogs on presenting and writing (and blogging): here
4. A few career blogs: here
5. Miscellaneous fun blogs: warfare, psywar, influence, etc
6. Some VC, product management and general business blogs: here

In any case, hope it was useful!

On DNS

I was preparing a long and thoughtful post on the "DNS issue" (mentioning this, this, this, etc), but it was all in vain.

This is the last and final word on it. Thanks Rich for the link (he understates and calls it just "genius" :-))

All bow to Hoff's wisdom - and poetic super-powers, of course :-)

At SANSFIRE 2008 in Washington, DC

I just landed at Washington, DC to speak at SANSFIRE tomorrow (my Lunch and Learn on "Log Management 'Worst Practices'" is on Wednesday, July 23rd - come over, it will be fun!)

LogLogic Lunch and Learn Presentation
- "Worst Practices" of Log Management
- Speaker: Dr. Anton Chuvakin, GCIA, GCIH, GCFA
- Wednesday, July 23rd, 2008 * 12:30pm - 1:15 pm

Want to learn all the embarrassing mistakes and pitfalls that await you on the path to log management nirvana? Attend "'Worst Practices' of Log Management" presentation by LogLogic's Logging Evangelist Dr Anton Chuvakin that covers all the things that can go wrong while planning, evaluating, deploying and running a log management solution. Insufficient planning, unrealistic expectations, choosing tools on price alone, lack of logging configuration guidance are among such "worst practices." Each common "worst practice" will be accompanied by suggestions to avoid the errors and do things correctly! Everybody touts "best practices", but this is the place to learn how to avoid the opposite - and have fun in the process.

if you want to meet, drop me an email/call or just show up for "lunch and learn." Unfortunately, I am going back right after my presentation tomorrow...

Fun Reading on Security - 5

Instead of my usual "blogging frenzy" machine gun blast of short posts, I will just combine them into my new blog series "Fun Reading on Security." Here is an issue #5, dated June 11, 2008.

1. Another fun (and horrible) laptop theft story, to be shown to those naive souls who say "ah, just stolen for hardware"
2. Very fun dailydave thread on security future (sad, of course :-)) - here is an excerpt: "The complexity in security is not from any complexity in technology but the complexity in motivating people to truly care about security and act accordingly."
3. Prediction markets for security? Fun idea!
4. "Elevator pitch for explaining security risks to executives" by Lenny Zeltser @ SANS.
5. "In Praise of the Information Security Checklist."
6. A great WAF battle rages on (here and in many other places). PCI + June 30 + 6.6 + WAF = BOOM!
7. How do you protect from IT admins "going bad?" Separate data and infrastructure (easier said than done, for sure). Another related one is "Staff more dangerous than hackers."
8. Curious about PCI DSS compliance outside the US? Read this and this. Yes, it is pretty bad.
9. "Terminating an employee with privileged access" from SANS (scroll to bottom)
10. An interesting view on sad state of academic research in information security.
11. Useful reminder to many people pushing silly/useless security solutions: while you are doing this, your organization is losing 6% of revenue to fraud. Today. Every day. Fraud checklist is linked there as well.
12. Rich on "consumerization" of IT. Good stuff. You are ready for it, aren't you? More on this subject.
13. Obviously, you are reading Mike R mid-year grades for his predictions.  One that failed in the most spectacular fashion (grade "D") is also an instructive read.
14. Really good post on security vs risk management. Just read it.
15. Matasano launches a GRC solution :-)
16. After "security idiot" became "an official meme", it didn't take long for SecurityIdiot.com to launch with much fanfare! If you are still wondering how to misspell "SOX" go there... the mystery is answered.

See you next time!

Fun AV Cartoon

Thanks to Dancho for the link. The cartoon is here.

Need MORE Proof That I am Popular in UK! :-)

As I said before, my blog was nominated for ComputerWeekly.com blog contest in UK. It looks like I made the final list, so feel free to vote for me in this final stage.

in Security Blogs

Pre-post on "Logging: Day 1"

Pre-announcing blog content is a shitty idea, in general. However, as I am writing this piece on "Log Management: Day 1" (obviously, inspired by this and this here), I hear that more and more people are thrust into a situation where my piece will be of huge value, thus this pre-announcement.

So, if you are given a task to "tackle logs" for your organization, where should you start? What should you do first? Even, how do you decide what to do first?

All this and more - in my upcoming piece next week.

Issue That Virtually Everybody and Their Dog Is Confused About

Here is an issue that everybody and their dog (and, likely, their dog's fleas :-)) is confused about:

What does PCI DSS Requirement 2.2.1 ("Implement only one primary function
per server (for example, web servers, database servers, and DNS should
be implemented on separate servers)") mean in virtualized environments?

Is it "one function per VM instance" or "one function per physical server?"

I've heard reports of QSA interpreting it either way, which means that millions of dollars of IT investments might be at stake.

Here are some arguments that I've heard about:

• "VM instance is NOT a server" - thus physical separation is required.
• "VM IS a different machine, might be different OS, etc" - thus it IS sufficient separation.
• "VM is like a VLAN" - thus VM separation IS adequate separation. Then again: some say VLANs are not sufficient separation either.

I hereby call upon the unholy wisdom of Hoff to answer this little bugger...

New Meme: "Security Idiot"

It is official! :-)

"Security idiot" is now an official security meme, as per SecMeme.com :-)

Misc Reading Related To Verizon Breach Report

All sort of fun stuff was unearthed, discussed and - sometimes -  made-up upon reading the Verizon Security Breach Investigations report. Here are some things from the pile which I found fun:

• Report itself [PDF] and brief on it from Verizon (and two fun follow-ups, this and this here)
• "90% of all statistics can be made to say anything… 50% of the time, aka my thoughts on the Verizon report"
• "Data Breach Post Mortem Offers Surprises" (well, to some people, they are surprises ...)
• "Insider Threat Exaggerated, Study Says" (not, it doesn't, BTW)
• "Verizon Business Report Speaks Volumes" (from Richard, thus a MUST read)

And of course, here is my favorite part: "In 82 percent of cases, our investigators noted that the victim possessed the ability to discover the breach had they had they been more diligent in monitoring and analyzing event-related information [AC - i.e. logs] available to them at the time of the incident." and this  "Furthermore, a crime scene devoid of any network and system logs, a key resource for computer forensics, is a disturbingly common occurrence."

What can I say? Back to battle stations for me - to fight the war of making logs more popular! :-)

On Logs and Breach Disclosure Laws

Check out my fun paper called "Where the truth is: Logs and breach-disclosure laws" at ComputerWorld. I personally find the premise that logs help with breach notification mandates to be a perfect no-brainer, but it looks like some people consider it to be deep insight.

And, let's leave it at that: deep insight it is :-)

Key point for the impatient bunch: "... logs are essential for compliance with breach-notification laws because you know who exactly to notify. Proper log-keeping will save massive amounts of money while complying with both the letter and the spirit of this law."

Monthly Blog Round-Up - June 2008

I saw this idea of a monthly blog round-up and I liked it. In general, blogs are a bit "stateless" and a lot of good content gets lost since many people, sadly, only pay attention to what they see today. This is what is driving an idiotic campaign of such "news" as "hackers increase hacking", "compliance is hard/easy/matters/doesn't" or "awareness of virtualization/SaaS/hacking/compliance grows."

So, here is my next monthly "Security Warrior" blog round-up of top 5 popular posts and topics.

1. Again this month, my logging polls took the #1 spot!  Poll #8 that covered context data for log analysis is analyzed here. Other popular polls include a controversial Windows Log Collection Poll (which is a poll #7)  and poll #6 about logs that people actually look and poll #5 about logging challenges. Next poll is coming soon.
2. Not entirely surprising, my post/rant called "You Are "A Security Idiot" If ..." takes the #2 spot after being live for only a few days. Yes, we all like to point out other people's problems, especially when they are epically huge :-)
3. Also not surprisingly, my post "11 Signs That Your SIEM Is A Dog or "Raffy, You Killed SIM!"" is on the Top list. It is both humorous and sadly true (and backed up by other sources)
4. A curious subject of DLP or "data leak prevention" (specifically, the post called "So, CAN We Have DLP?") also tops the charts. My previous post on data leak 'prevention' ("In Passing on DLP") is popular as well.
5. Again and again, people googling for "open source SIEM" have pushed this post (this tiny old pathetic blurb) to top5. This ancient post from years ago explains why an open source SIEM will NOT emerge soon, if ever.

See you in July!

Possibly related posts / past monthly popular blog round-ups:

• Monthly Blog Round-Up - May 2008  
• Monthly Blog Round-Up - April 2008  
• Monthly Blog Round-Up - March 2008  
• Monthly Blog Round-Up - February 2008  
• Monthly Blog Round-Up - January 2008  
• Monthly Blog Round-Up - December 2007  
• Monthly Blog Round-Up - November 2007  
• Monthly Blog Round-Up - October 2007  
• Monthly Blog Round-Up - September 2007
• Monthly Blog Round-Up - August 2007

 

Technorati tags: blog, security, loggings, monthly