Do You Enjoy Searching?

Now, do you enjoy typing stuff into Google and seeing results come up? I kinda doubt it - you probably need to "find something" rather than feel an unstoppable urge to "search for something." At this point some might say - ah, this is just semantics. I assure you - it is not.

Let's look at something like using Watson or Inxight vs just using Google. In the first case, you are given a spade (Google prompt) and are told to dig until you either a) unearth what you need or b) get tired and leave or c) find something else and get distracted (sounds sad, but ADD epidemic is rampant indeed). In the second case, a smart application (such as Watson) tell you "hey, look at this - this is relevant to what you are doing [and it is!]; check it out." You are given that something you'd dig for, not a digging tool. And note that it is given to you without any digging, which is cool.

So, why are we discussing it here? This discussion has a direct link to computer log analysis and log management. Lately, I've met a few deluded folks who equate "log analysis" with "log searching." At first glance, it sounds truly ridiculous: why would someone enjoy searching logs? :-) OK, a need to find something specific in logs does arise (e.g. to answer targeted question such as "what did this user do?", "what specific systems were attacked?", etc), but I still see log searching as a somewhat secondary and somewhat painful route to answers of your logging questions. And, to top it off, you need to analyze the results for relevance. Yes, Virginia, it means more work... for you!

What is the other way? For example, in one presentation on text mining ("Introduction to Text and Web Mining" by Je Wei Liang) we find this curious picture:

It applies to logs perfectly! I'd rather have questions such as "what do I need to know" (and more specifically, "what's wrong?", "what requires attention?", "what's not normal?", etc) discovered, answered and presented by an automated system, than go for a digging session... A nice dashboard with coffee in the morning containing the answers to the above questions for my environment beats any "log search" over the head with the big stick :-)

Technorati tags: log management, logging, search

How to Analyze a Trillion Log Messages?

Somebody posted a message to a loganalysis list seeking help with analyzing a trillion log messages. Yes, you've heard it right - a trillion. Apart from some naive folks suggesting totally unsuitable vendor solutions, there was one smart post from Jose Nasario (here), which implied that the original poster will need to write some code himself. Why?

Here is why (see also my post to the list):  assuming 1 trillions records of 200 bytes, which is a typical
PIX log message size (a bit optimistic, in fact), we are looking at roughly 180TB of uncompressed log data. And we need to analyze it (even if we are not exactly sure for what, hopefully the poster himself knows) ... not just to store.

Thus, I hate (ehh, make it "have" :-)) to admit that Jose is probably right:  writing purpose-specific code might be the only way out. About a year ago, there was a discussion titled "parsing logs ultra-fast inline" on firewall-wizards list about something very similar. We can look up some old posts by Marcus Ranum for useful tips on super-fast but purpose-specific log processing.

For example, here he suggests a few specific data structure to "handle truly ginormous amounts of log data quickly" and concludes that "this approach runs faster than hell on even low-end hardware and can crunch through a lot of logs extremely rapidly." One of the follow-ups really hits the point that I am making here and in my post:  "if you put some thought into figuring out what you want to get from your log analysis, you can do it at extremely high speeds." A few more useful tips are added here.

So, nothing much we can do here - you are writing some code here, buddy :-) And, as far as tips are concerned, here is the "strategy" to solve it:

1. figure out what you want to do

2. write the code to do it

3. run it and wait, wait, wait ... possibly for a long time :-)

Indeed, there are many great general purpose log management solution on the market. However, we all know that there is always that "ginormous" amount of data that calls for custom code, optimized for the task.

Technorati tags: log management, log analysis, logging

Security on a Card: Good, Bad or Not'n Special?

One of the innovative things (and, yes, my long and thoughtful post on security innovation is still upcoming ... and, no, security innovation is NOT dead!)  I noticed at RSA 2007 was a few security "security on a card" solutions. What do I mean by "security on a card" here? A dedicated pluggable (USB, PCMCIA or whatever) hardware that carries some or all of the host protection, policy enforcement and, possibly, management, access and connectivity functions. You can also think about as a "micro-appliance" in the pocket.

In fact, I was briefed by a few Lucent-Alcatel folks who are working on a really cool "security platform" on a wireless card (called Project Evros). Yoggie folks have another one of that type, with somewhat different focus and functionality and so does Seclarity. There are also a few "stealth" vendors working on such technologies, as I was told.

Let's quickly look at pros and cons, generalized to cover whatever solutions of this type that I saw.

Pros:

• Much harder to disable for host-based malware as well as attackers (I am stopping short of just flatly saying "more secure" here!)
• Harder for "legitimate" users to bypass
• No security software to update, manage and, in general, mess with
• No software conflicts between security and application software
• Predictable environment for security functions to run (some cards contain fairly robust Linux-based OSs with their own CPUs and even separate wireless network uplinks)
• Likely, easier to centrally manage and support
• Some provide host-independent, but detailed access audit (Hurrah, more quality logs to manage!)

Cons:

• Another piece of hardware to break or lose
• Extra hardware cost
• Possibly, risk of losing some or all protection if hardware is removed
• The opposite risk of not being able to bypass it and thus do any work in case of hardware failures - new "single point of failure"
• Possibly reduced protection from threats that entirely bypass the card (e.g. from virus infected CD or DVD)

Overall, it sounds pretty impressive and this "security on a card" space will definitely be interesting and exciting to watch! I am predicting that we will see more of this type of stuff in the coming years, since these technology help deal with today's as well as yesterday's (such as worms...) security concerns: data theft, compliance risks, etc. What about future risks? :-) Only future can tell...

Technorati tags: security, innovation

Look At Logs, If ONLY for Forensics and Incident Response

So you are a log-hater! It is fine with me, admit it. You hate logs and never ever look at them. In fact, you don't even know you have them. And you don't care if you don't, to add insult to injury. (Well, hopefully few of my readers are like that, but I suspect there are some :-))

How can I convince such a person to look at logs seriously? Well, that's actually easy: it is natural to want "all the logs" the moment something goes terribly bad in your IT infrastructure or somebody smart 0wns your network.

However, it is really unlikely that logs will automagically show up on your doorstep right when things go South. If you were ignoring logs before,  it is unlikely that you actually have them! And that is the whole point of this post: if you still insist on not looking at logs (as many organizations do out there...), I would suggest you invest in log management anyway. Huh? If you do that just for having logs in case of an incident or an investigation, you'd still be grateful you did. And, as ROI fans will probably say, "it will pay for itself."

What if you are a CSO with the above mindset? Then, run, not walk and get reeducated on the value of log management by the enlightened Mike Rothman (start here, for example). He closes with this: "Because your forensics guys will thank you that they've got such detailed information when they've got to solve the mystery of a security incident."

To conclude, as I hinted in my post "Logs Are Everywhere!", doing it earlier (like, today! :-)) is easier, since there will be more logs later.

Technorati tags: chuvakin, log management, logs, trends, incident response

On SSL

I haven't attacked anybody on my blog for a long time... too long some would say. And, along came anti-SSL post from Pete Lindstrom, which, in my world falls under, "come on, nobody is that dumb ... oh, wait" category.

Most points made there sound more like "SSL is not enough" and only one point that is a bit more anti-SSL. Here it is:

"2) The threat model for sensitive Web data has never been one of sniffing traffic. There are still way too many accessible websites for this to be the case."

Won't SSL be one of the reasons for this? I did say that not using encryption when it is easy and accepted is a mistake and SSL is a perfect example. Indeed, much more data is stolen while in stored state and in bulk, but I would venture a guess that almost nobody would sniff for credit card numbers after compromising a web server due to SSL being commonplace...

More fun discussion is here and here.

On " Vista: Where UNIX was in the 1980s"

Vista UAC? Access control "innovation"? "Next-generation" OS security?
Enjoy "Vista: Where UNIX was in the 1980s"

On Security vs Convenience vs Embarassment

I just love this quote from Alan Paller: "convenience trumps security, but embarrassment trumps convenience." Sounds like a new driving principle for security.

Besides the above quote, I was actually quite impressed that "log management was cited [as a funding priority] by 23 percent of respondents"

But all this murks in comparison to the good old some-say-myth-some-say-truth that "everybody is 0wned." Specifically, Alan Paller said: "Federal systems are deeply penetrated, by hackers and other hostile interests."

On "Know your Enemy: Web Application Threats"

A fun new paper from the Honeynet Project on honeynets vs. web hacking.

Future Innovations

Here is a fun doc [PDF] with some predictions of what technologies might will appear before the year 2050.

Anton Security Tip of the Day #9: But He "Wasn't Logged!"

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #9: But He "Wasn't Logged!"

The idea for this tip originated when my presentation on log analysis was rejected by one of the high-profile security conferences on the grounds that "logs don't matter since advanced attackers never leave traces in logs [or erase them before anybody can get to them] ." Indeed, some of my security friends of a more "offensive orientation" :-) have long developed this snobbish attitude about logs.

So, imagine a network that has fallen victim to a 0day-wielding "uber-haxor" :-), who kicked the door open (or snuck in), grabbed the crown jewels and took off like a bat out of hell :-) When, much too late as usual, the "good guys" rushed in to pick up the pieces, only there was seemingly nothing much to pick: the server logs were erased and the network IDS didn't make a peep (and, no, Richard, NSM was not deployed). What do you do now?

So, let's list some uncommon (and some common, but often untapped for the task at hand!) sources of log data and provide a few log analysis tips:

• firewall logs: boring they might be, but it is less likely that those are erased by the attacker (even though firewall logging can sometimes be jammed). And it is hardly possible to "not be logged" by the firewall (the analyst's challenge is in analyzing the huge volume of data to find the attacker's traces and to tell them from normal traffic). What is critical in making these logs useful is logging allowed connections, not only blocked ones.
• router logs and netflow records: same as firewalls logs - they are less likely to be erased, but huge log volumes make their use problematic. And, last but not least, these types of logging are more likely to be turned off completely ("routers should route, not log," grumble many network types)
• application logs: was it a "legacy" network attack or something application-level? If the latter is true, there might be logs found in unusual places. Is there anything in the webserver logs? Websphere made a peep? MySAP recorded something? PHP app logged something fun?
• various client logs: in one old case FTP client logs were extremely helpful, even though the entire syslog directory was blown away and no remote logging was ongoing; look for other client logs (yeah, even a web browser history is kind of a log...) to look for things missed by the attacker
• other systems' logs: was there anything that the attacker did that affected other systems? Might have they logged it? Even an attempted SSH connection or a scan from a compromised machine has a high chance of leaving traces elsewhere - cleaning all of such traces is a major challenge for an attacker (in any case, much harder than erasing logs and stopping logging on the compromised host)
• any remote or centralized logging: was any log saved from destruction by being copied to another system?

To conclude, while there is no "logwatch" pattern for "advanced attacks," logs are still very useful in such circumstances if you prepare by setting up a broad scope of log collection (I suspect using a log management system will be your only choice as log volumes will be pretty bone-crashing) and then combing through the logs after the incident. And remember the less common sources of log data, such as database logs.

I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

On "Top Ten Investigative Boo-Boos"

Oldie (eh, from February, 2007, that is :-)), but goldie... An interesting and enlightening read "Top Ten Investigative Boo-Boos" from SecurityMonkey.

Just one example:

"2) Investigators destroy evidence because they didn't follow their own procedures. [...]"

More Fun Stuff on Security vs Security

So, my little poll on security violations by security people is running nicely. I realized that I totally forgot to add an option of "NEVER did anything of this sort."

Check out the current results here and vote here.

So far, it seems like we do like to violate security rules which - I am guessing here - we perceive as 'stupid.' But how can we then complain that users break the rules that they think are stupid?

Something is deeply amiss here... help me out!

Review of my Database Logging Paper

Thanks to Paul Melson for a thoughtful review of my database logging paper. I enjoyed reading his review almost as much as I enjoyed writing the paper :-) The part I liked the most is where he goes over the categories of log use for change management, threat detection, etc (and disagrees with logs being the best tool for some of them ...)

A Fun Question to Ponder ...

A small trivia question: do you know what is "thermorectal cryptanalysis"? No points will be earned if you speak Russian :-)

Security vs Security: Who Wins?

Reading this story by Ira Winkler inspired me to do this little poll here.

So, my esteemed readers of security profession, have you ever knowingly circumvented a security measure?

PCI Book Out Soon!

I am sure that "everybody in the know" is already, well, in the know, but still - here it comes, the first book on PCI: '"PCI Compliance: Understand and Implement Effective PCI Data Security Standard Compliance"by Tony Bradley (Author), Anton Chuvakin (Author), Anatoly Elberg (Author), Brian J Koerner (Author)'

Again and Again on Compliance vs Security

I am going thru my "backblog" of items :-) to blog about - here is one fun item where Andy weigh in on compliance vs security with "Compliance rarely leads to good security but good security almost always leads to compliance." Indeed, claiming the opposite is silly, just look at all this FISMA stuff.

However, what is missed in the above adage are all those areas of compliance and IT governance that have little to do with security: proper change control (which admittedly helps security immensely), documenting controls, SLAs, and a bunch of other stuff that ITIL, COBIT and others are made of.

So, compliance might not lead to security, but security doesn't lead to [full] compliance as well ...

Dumb and Dumber ...

Mike says: "I write one thing about SIM and every friggin' vendor that I've never heard of has to reach out to me and tell me how great they are."

He continues: "1st generation SIM [you know who you are...] is pretty much dead, and those that want to survive need to either focus on log management or more real-time network detection (which means they need to bring in NBA data)"

And then closes with: "But of course, lots of folks were lining up to get briefing slots to tell me how great they are. Without even hearing their pitch, it's a load of crap."

Sorry for that much quoting, but there is just no better way to say it - some companies in this segment have certainly overstayed the market's welcome...

Doing a Media Interview for Security Vendors

Humorous it might be, but oh, so much truth in there ...

A quote:

"The Players

Interviewees generally come in threes. There’s the Main Guy, the Other Guy, and the PR Bodyguard."

"Main Guys can build a 20-slide Powerpoint around a single trivial feature upgrade that is entirely 100% identical to something their competitors introduced and briefed you on three weeks ago."

"...At this point, you need to show that you’re paying attention. So ask, “What are your target verticals?”. The answer is “healthcare and financial services”."

"The sixth slide has a diagram with some boxes on the left, some people typing on the right, and a cloud in the middle."

"The tenth slide has the name of an analyst you can call if you’re a fresh-out-of-college rookie or just totally fucking clueless and lost."

etc, etc, etc

Fun Read: "Bad News Is Good"

Here is an interesting high-level article "Bad News Is Good" with a somewhat duh! conclusion - if neglecting security will make you have to pay ... a lot, things will change in a flash: "But a punch in the wallet: Now that ought to focus an enterprise's attention."

Let's Play a Fun Game Here ... A Scary Game

So, let's suppose somebody who is involved with incident response at a typical US public University has collected a few recent malware samples from the compromised machines and then submitted all the samples to VirusTotal for scanning with pretty much ALL current anti-virus and anti-virus-like products.

What do you think the average detection rate (i.e. a malware sample was identified as "something bad") was?

Any guesses? Here are a few numbers to help you choose:

1. 100%
2. 94%
3. 90%
4. 70%
5. 50%
6. 33%
7. 22%
8. 14%
9. 2%
10. Something else?

Let the games begin!

UPDATE: answer posted

UPDATE2: after much deliberation, I finally replaced anti-virus on my own systems with another technology. Read the details here.

Build A Better ... Top Influencers List :-)

Improving upon something written by others does drive a lot of blogging. So, why not improve a list of top influencers in security? Indeed, Computer Defense list makes a bit more sense, as far as technical credibility is concerned.

Update: any "list watcher" :-) must read the comments over at Matasano and have fun while doing it. Ability to not take oneself too seriously is critical in our biz :-)

On Ego Feasts and Top Lists :-)

Labeled a "list of the most influential security experts of 2007 - from corporate tech officers and government security types, to white hat hackers and bloggers", the list certainly generated a lot of noise in the security blogosphere.

And why not? After all, why do we blog? I am thinking ...

• To share knowledge with the community
• Because it is easier than writing a book :-)
• To generate F&G
  
• Because we need to generate awareness about our favorite technologies
• To vent and rant
  
• Whatever other reasons ...
  

So, in my estimation "fame and glory" is one the more common drivers that apply to just above every security blogger I've met. And if it doesn't apply to you - you are in denial :-) Thus, combine such personalities with whatever "top list of ..." and you get a lot of interest ...

Finally, I am honored to occupy a spot #4 in the sub-list of "Chief Blogging Officers" (I never thought I am one, BTW)!

Discounted Passes for CSI NetSec 2007 Anyone?

If you are into that sort of thing :-), I can give you a deeply discounted pass for the upcoming CSI NetSec 07 conference to take place in June 11-13, 2007 in AZ.

Since I am speaking there on mistakes of log management, I can give out a few of those discount codes that reduce the conference admission price by almost a $1000: "The discount passes cover the entire two-and-half-day conference for only $795 until May 11th (after which the discounted passes will be $995)--a regular conference pass is $1,645. "

To make use of the pass code go to CSI NetSec 07 Registration.

Update: just as I suspected :-) No takers so far...

Discounted Passes for CSI NetSec 2007 Anyone?

If you are into that sort of thing :-), I can give you a deeply discounted pass for the upcoming CSI NetSec 07 conference to take place in June 11-13, 2007 in AZ.

Since I am speaking there on mistakes of log management, I can give out a few of those discount codes that reduce the conference admission price from by almost a $1000: "The discount passes cover the entire two-and-half-day conference for only $795 until May 11th (after which the discounted passes will be $995)--a regular conference pass is $1,645. "

To make use of the pass go to CSI NetSec 07 Registration.

Discounted Passes for CSI NetSec 2007 Anyone?

If you are into that sort of thing :-), I can give you a deeply discounted pass for the upcoming CSI NetSec 07 conference to take place in June 11-13, 2007 in AZ.

Since I am speaking there on mistakes of log management, I can give out a few of those discount codes that reduce the conference admission price from by almost a $1000: "The discount passes cover the entire two-and-half-day conference for only $795 until May 11th (after which the discounted passes will be $995)--a regular conference pass is $1,645. "

To make use of the pass go to CSI NetSec 07 Registration.

On Database Logging and Auditing (Teaser + NOW Full Paper)

Here is a excerpt from a fun paper on database logging that I just wrote:

"Database security have been capturing more and more attention in recent years, even though most of the security issues surrounding the databases existed since the first day commercial database systems were introduced in the market.

Nowadays, database security is often seen as containing the following principal components:
• access control to database software, structures and data
• database configuration hardening
• database data encryption
• database vulnerability scanning

It is interesting to see that logging and auditing underline all of the above domains of database security. Indeed, the only way to verify what access control decisions are being made and who views what data from the RDBMS is to look at the authentication logs. Database configuration hardening includes enabling and increasing the auditing levels. Similarly, data encryption might be verified by log and configuration review. And, vulnerability exploitation usually leaves traces in logs despite what some say (the challenge is more often with understanding what the log said and not with having the logs)

In recent years, insider attacks gathered more attention than periodic outbreaks of malware; and database logging happens to be in the forefront of this fight against insider attacks. Database systems are usually deployed deep inside the company network and thus insiders are usually has the easiest opportunity to attack and compromise them, and then steal (or “extrude” as some would say) the data..."

Read more here if you are a CSI Member. If you are not, the only way to get my paper is to ask me (sorry, copyrighted stuff)

UPDATE: another similar paper by me is posted here.

UPDATE2: full paper mention above is posted, finally! Enjoy my "Introduction to Database Log Management" at InfosecWriters!

So WHAT Is He Doing Right?!

From "The 50 Most Important People on the Web":

"43. Mikko H. HypponenDirector of antivirus research, F-Secure

F-Secure's security news blog, written by director of antivirus research Mikko H. Hypponen, is one of the Internet's go-to places for learning about the latest security threats."


Yes, I do get that Bruce Schneier is on the list (how can he NOT be?), but this puzzles me to no end. Well, I have much to learn about personal marketing, it seems...

On Temptation ...

Every time I see stuff like this (Lynn vs Cisco or Paget vs HID), I can't help but think this: every "whitehat" guy at least knows someone who knows a "blackhat" guy (I am guessing here, but you get the point ... if you think that the above is too extreme replace with "knows someone who knows someone who knows a "blackhat")

Don't companies that try to suppress security research understand that if you do this to a security researcher, even the most ethical guy in the world will be tempted to JUST LEAK IT.

If you corner a rat, it bites. Don't corner security researchers :-) they have bigger teeth... much bigger. You want to suppress the legitimate security research? You think you just did? Go suppress the entire underground now!

On " Encryption Mistakes, Masterpiece by Chuvakin"

I so totally like it when people call what I wrote "a masterpiece."