Tuesday, October 30, 2007
"keep good logs. Logs should show who is doing what to your data. In particular, if insiders use admin level access to change data or review users data."
Of course! While people continue the futile search for the ultimate anti-insider technology (free tip: it doesn't exist!), logs, which sit right under their noses, contain the records of all activities. Such info is extremely useful for investigating insider behavior today as well as - in the future, I hope - predicting their transgressions.
BTW, I wrote this great piece on using logs vs insiders (to be published in ISSA Journal in November - check it out when it hits the shelves)
Check out the picture there first.
And let's run a simple idiocy test: imagine you make a product "Anti-X" and sell it under the name of "Anti-X" (no surprises here). Then you OEM a piece called "Anti-Y" (which comes from a super-over-saturated market with all established players) - combine them together and - voila! - rebrand the combined package as "Anti-Y" (specifics here). The natural question arises: are you stupid? Sure seems like it - thus, a great motivational piece: if $108 million investment can be given to idiots, think how well a relatively smart person will do ...
So, the ZDNet blogger is right: The anti-spyware market that never existed is officially dead.
It had a horrible, confusing, abysmal name!
Generally, I hate the bandwagon-jumping. But, Chris, the reason that you "received about a dozen emails suggesting that Information Survivability just focuses on availability" is that the word "survivability" does bring that to mind. It really does! In fact, it is more about "scrambling, half-starved proto-mammals" (courtesy of Rich here) than about "process and risk management."
Now, the ideas are all gold! :-) I loved the, bookmarked them, del.icio.us'ed them, etc. Technical "anti-x-style" security does seem to miss a lot of what you are talking about in the piece. More risk thinking needs to be brought in (however hard it might be). Indeed, "7/10 information security programs are focused on compliance and managing threats and vulnerabilities - they don't holistically integrate and manage [business] risk. " (changing the latter is waaaaay easier said then done though) More fun work is ahead, that is for sure!
So, next time you come up with a name for a revolutionary concept, check out the Pirate's Ship Name generator. According to it, the pirates, ya know, would call a ship "The Rage of the Sargasso Sea" or "Satan's Horror" or something scary. They wouldn't call it "The Floating Mattress" or "Bathtub of Dirty Water" now, would they? :-)
Vote on! Results in about one week ...
See all my polls here.
Monday, October 29, 2007
- Authentication (What?!)
- Smart traffic sniffing (A yawn? Maybe)
- DLP (tooooo f late...)
- Behavioral anti-malware (one more? nooooo...)
- Identity risk-management (WTF?)
- Database security (well, maybe)
- Encryption (what a novel idea?)
- Code review (finally time?)
Friday, October 26, 2007
Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised "Top 11 Reasons to Secure and Protect Your Logs"
- Let's review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
- Oooh, logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
- A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
- PCI DSS just says so: "Secure audit trails so they cannot be altered." Wonna do it- or pay the fines?
- Do you protect financial records? Identity info? Passwords? Some of it ends up in logs - thus making them more sensitive. Secure the C-I-A of logs!
- Do you look at logs during incident investigation? Do you want them to be "true" or full of random (if creative...) cr*p, inserted by the guilty party? Secure the logs!
- Think that "attacks vs logging" are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
- Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
- Why change logs? No, really, why change logs? If you never change logs - and you never should - hash them right away after collection to make them immutable.
- Logs are backed up on tape - who will see them? Well, whoever restores the tape, that's who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
- Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!
Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information...
Possibly related posts:
Think about all the fun situations/headlines that can be imagined:
- "What is our malware availability strategy?"- asks the boss
- "Moving to malware-as-a-service for better scalability"
- "System administration 'best practices' for malware serving"
- "Load balancing your phishing servers for quality online victim experience"
- "IT person commended for saving a malware server from crashing"
- And of course: "Disabling logs for evidence avoidance for malware admins"
Thursday, October 25, 2007
So, this fun blurb has a few interesting thoughts on log management. And one on log management vs SIEM: "I like my logging solution to have plenty of evidence preservation goodness and I don’t want it muddied because a correlator had to normalize the data before it could parse, alarm on, or display the log data."
Some people share this view and some don't. How it will end up is anybody's guess - my crystal ball needs rebooting. Vista, you know :-)
So, my log collection poll results are in. Here is the link to the running total; the picture as of today is on the right.
I admit this poll was fun!
Let's review and discuss the findings after running it for slightly over a week.
First, which of my expectations were NOT met? Well, I did expect that firewalls will be #1, not Linux/Unix servers. Admittedly, the difference is not so big, but I am impressed: Unix syslog still rocks the logging world :-)
Second, the top source of collected logs is also the hardest to analyze due to its lack of structure. Nowadays I treat syslog from Unix/Linux as "broken English" and not as "data." It is a dog to parse (that is why we try to find something novel)
Third, I was amazed that database logs were THAT high on the list. Wow! All the evangelizing seems to have worked out :-)
Fourth, Windows server log collection is still in the dumps - but we need it! Go grab LASSO and dump those event logs into syslog without pesky agents. Easy!
Firth, other Unix logs - what are those? We might never know what the respondents meant: still, I think that these are binary audit logs and other fine-grained audit logging. Indeed, many people starting to look at BSM audits and other "ugly ducklings" of logging.
Sixth, web server logs are gold - everybody knows it. The poll confirms this as well: they are #2. Some fun analysis tips from me are coming soon.
Next poll coming soon! Thanks a lot for responding!
Possibly related posts:
"Security folks need to begin by aligning their investments with the same priorities the business is investing in. What you'll see very often is businesses spend at least 10 times more on application development than networking investment. And you'll see that security is the reverse of that. They spend 10 times as much on network security than application security."
Gunnar Peterson rocks! :-) You MUST go and read the whole interview here.
"On a desk sits a piece of paper exposing a single person’s non-public information: social security number, bank account numbers, etc. as well as name, address and phone number.
On the same desk sits an unencrypted tape with 10 million records whose data contains the name non-public information and data elements as the piece of paper."So what? Read on.
And, yes, it does have a few classic journalist distortions....
"The Best Security Books to have in your library" by SANS GIAC Advisory Board. "Security Warrior" is, of course, proudly featured among other good books, such as "Tao of NSM", "Security Metrics", " Hacker's Challenge" and many others. Check it out!
"Caution: Visit the Phoenix New Times online site and automatically become ensnared in an Arizona criminal probe.
That's because Arizona authorities are demanding the Phoenix New Times to hand over the names and internet protocol addresses of anybody who read its online news site since Jan. 1, 2004."Stuff like this gives logging a bad name in privacy circles ... And you know what? Maybe they are right... sometimes. Just reach for that "Delete" key....
Wednesday, October 24, 2007
Monday, October 22, 2007
Do read it; especially recommended for the folks in corporate security teams.
And now, back to perl - "Use of uninitialized value in join or string at textalog.pl line 306." Darn it! :-)
And I admit to it! There is science in it for sure, but the overall path logs -> human conclusions about what happened has too much Zen in it to be scientific...
The interesting parts of the answer are (full answer here):
- "Aside from legal requirements, companies must consider how far back an investigation might run and whether the logs will continue to have value after a time." This means that "PCI = 1 year retention" might not even be the minimum needed log retention.
- "As for using a central repository, a common standard is to move log files offline as quickly as practical." They are somewhat confused here, since they use "offline" to mean "off the production system"
- " Encrypting logs might also help repel tampering, but a more important control is to hash your log files. " Everybody who is somebody in log management space does it, of course. If you get your log tool from some 'monkey vendor', it would be prudent to check this somehow...
Book1 "The Long Run" (book) PDF RTF
Book 2 "The Last Dancer" (book) PDF RTF
I felt the same when I did when I was reading "Snowcrash"... OMG!
P.S. Thanks to Aitel and his team for hosting the books on their site.
Friday, October 19, 2007
Wednesday, October 17, 2007
- Windows NT events: "Security Event Descriptions"
- Windows 2000 security events: two articles
- Windows Server 2003 events at Technet Events & Errors Message Center and "complete list of WS03 security events: chapter 4 of the Windows Server 2003 Security Guide"
- "Windows Server 2008 events are self-documenting"
Loading Clusty Cloud ...
Revisiting this page periodically is guaranteed to product new insight on what's hot not vs yesterday. Admittedly, it is a trailing indicator, but a fun one at that.
Here is the link to me posting it the first time. Here it is again.
New concepts get added every few months; some die away ...
For example, "According to ESG Research, 44 percent of large organizations collect at least 1 terabyte of log file per month. Heck, 11 percent say that they capture more than 10 terabytes a month."
That's a lot of log! :-) And you know what? There will be more, MORE, M-O-R-E: "ESG Research reveals that large organizations plan to capture lots more log data from more devices for more analysis over the next few years."
Are you ready? Now it would have been appropriate for me to scream "buy LogLogic!", but I will resort to a milder "if you think you are ready, think again: are you ready to capture the data OR are you ready to MAKE SENSE of all this..."
Indeed, many people treat replacing their beloved "family-grown" log management system with a commercial logging solution similarly and thus experience intense anguish and other extreme feelings :-) Let's look at this syndrome in more details. As we all know, people love to write their log analysis scripts and little proggies (a hit of sourceforge confirms this, but there are great many more which are not shared with the world).
Some did more and built super-sophisticated log analysis systems. When I gave my Approach to Log Management presentation, I always related a story of a major bank which build an awesome system to analyze events which - gasp! - beat what most commercial players had on the market at the time (!). They used visualization, data mining, data profiling and a lot of other cool things. Others tried to beat major commercial vendors on scalability (ha!) and ended up creating only hilarity for everybody and a need to look for a new job for themselves (here).
So, what are the reasons why so many in-house log management projects fail miserably, often taking their creators down with them:
- Ongoing maintenance will kill you ☺ - logs formats and messages change ALL THE TIME. Will you take on keeping track of that? This is by far the #1 reason for such projects to end badly!
- No support, apart from you - you might like that, but will your boss? At some point, many creators of such apps start hating them due to the care and feeding they need...
- Does it pass the “bus test”? - Ah, the dreaded "bus test" - will the application still run if the chief/only developer gets hit by a bus? If you have 23,000 lines of Perl in your log analysis app, I can tell you the answer is NO!
- Handling production log volume - it works really well in the lab on your own system. But when that "M" becomes a "G" (megabytes into gigabytes) and maybe a "T" (in extreme cases - "P"), will it work? Probably not...
So, what should you do? For most companies, buying is the answer. Really, the common choices are:
- Build -> suffer -> buy
- Built a little -> plan what to buy based on that -> buy
Finally, of course there are situations when building is the only choice (e.g. here)
Tuesday, October 16, 2007
The veto message supposedly said "that it threatened to place burdensome costs on small businesses and attempted "to legislate in an area where the marketplace has already assigned responsibilities and liabilities that provide for the protection of consumers.""
Monday, October 15, 2007
A very insightful post from Anton Aylward called "Why I don’t see the need for elaborate Risk Analysis."
Fun quote from it: '“Standards” like a ISO-17799/27001, ITIL aren’t trying to do anything more than lead people though a process to make them deal with the basic good practices. When they talk of things like Risk Analysis they are trying to get people to think about risk and their risk posture, and that is, all to often, sadly, something most firms don’t seem to have got around to.'
In other words, if your password on a publicly exposed router is "password," please shut the trap up about "risk management!"
UPDATE: Yes, yes, yes! You guys are right: "the entire concept of “good practice” is simply a lazy man’s risk analysis." I was just venting about folks who spent time pontificating about risk management while being owned thru a router with a password of "password"...
- have huge vision,
- the ability to pull an all-nighter and crank out a rough prototype of the thing they are thinking about,
- have the unique ability to translate complex / abstract thoughts into simple English that a non-technical end-user can understand, and
- a willingness (or even desire) to get up in front of 1,000 people and talk about the latest greatest thing they are working on / thinking about."
UPDATE: also featured here with this fun quote: "Log analysis in particular has long been a thorn in IT's side. Either you tried hard to forget that terabyte or so of raw log data just sitting there, or you paid through the nose for a security information manager. Now, affordable log analyzers are available from companies like LogLogic"
Friday, October 12, 2007
But they will fail!
So we have a new result [i.e. Storm "worm"]: the enemy now has architects as good as our best."The post also explains why takedowns no longer work.
'"Safety" is not "security". Safety protects against ACCIDENTAL problems, security protects against INTENTIONAL problems.' It also reminds us that fake experts are out there ...
Think the above when designing your security countermeasures: do they protect against the a) benign but gnorant b) evil but dumb/lazy or c) evil, smart and willing? Data loss or data theft? Kiddies or cybermafia? CVE or 0d-day? You get the point...
Amen to that... but I will keep doing it, 'cause I think it is working :-)
Quote: "I am constantly amazed by the sheer lack of security in the average “enterprise” tool. I’ve looked at many over the years, and most are designed to the “soft squishy center” anti-security model. Typically:
- Accountability is simply missing. Yes, many systems have logs, but they are business irrelevant. My personal view is that if a business person doesn’t care about a log entry, it’s not worth collecting. Accountability is the key here, not 1 GB of logs per day"
Thursday, October 11, 2007
Tuesday, October 09, 2007
Monday, October 08, 2007
Maybe because of cases like this: "In proving liability, the industry did not have to demonstrate that the defendant's computer had a file-sharing program installed at the time that they inspected her hard drive. And the RIAA did not have to show that the defendant was at the keyboard when RIAA investigators accessed Thomas' share folder." Scary indeed!
Or maybe like this, which comes from my own recent experience? The other day I borrowed a DVD of a popular Russian series from my friend and intended to watch it in my DVD player. Surprise! The movie's region did not match the player. Now, obviously, I knew that DVDs are region-coded; I just never experienced it myself. As a quick test, I tried it in computers' DVD drives and, just as obviously, with no luck.
So, given that my intention was to watch a movie and not to start a security research project, I just called my friend and asked "Hey, how did you watch it? It seems to be of the 'wrong' region?" The friend said: "Oh, that's easy, just get the mplayer; I use on my Linux systems; I think they have a Windows version as well" (and they did - and the movie played just fine)
Now please somebody tell me: since when did watching a legally obtained movie have to involve "hacking"?
Maybe that is why MPAA and RIAA are hated with a deep passion: their security "inventions" defy common sense?
Do yours? :-) You will probably be hated too ... AND then your security program will fail.
UPDATE: another fun set of comments on this is here. The highlight: "There is no common sense when you put the legal system and technology together."
UPDATE2: obviously, the scumbags will lose, but how exactly will they meet their doom? Maybe like this?
Friday, October 05, 2007
CSI/FBI 2007 survey is here. We all know and love it as the best darn source of random numbers on this planet. Recent industry surveys prove that CSI/FBI survey numbers beat the quantum RNG systems at least 97.8% of the time and beat the laser-based random number generators approximately 77.72% of time, as long as the laser in question is not pointed at the Moon :-)
Still, I enjoy reading it every year! Yes, I am funny that way :-) So, this year's survey drops a BOMB (of sorts): malware is not longer the #1 cause of loss. Specifically, "Financial fraud overtook virus [A.C. - they really mean all malware here] attacks as the source of the greatest financial losses. Virus losses, which had been the leading cause of loss for seven straight years, fell to second place." Cool! But did it happen in reality, in respondents heads or not at all? You tell me!
To back up the above claim, the survey also discovers that "Insider abuse of network access or e-mail (such as trafficking in pornography or pirated software) edged out virus incidents as the most prevalent security problem, with 59 and 52 percent of respondents reporting each respectively."
Deloitte Global Security survey is here. It is also a fun read, if a bit dry. A scatter of insights (it is a loooooong document...):
- Lack of logging - is still one of the top 5 internal audit issue (30%)
- Believe it or not, IAM is not a top operational priority
- Unlike CSI/FBI above, viruses/malware still rule the top risks list (however, survey splits them into internal/external - and fraud shows up a bit near the top in the internal risks list)
- Many people start to feel that privacy regulations often contradict security regulations (49%) No kidding! I am predicting this one will blow up soon, but maybe in Europe first (since US privacy regulations are weak)
Another fun one - why stuff fails (of course, it is the dirty humans! :-):
Read the rest here!Technorati tags: security
Thursday, October 04, 2007
But then again, we all said it a dozen or so hundreds of times :-) What makes them act? Of course, good ole FUD or chanting the C-word ... :-)
The judges pointed out that in many cases it was simple to map an IP address to an identity with the help of 3rd parties, and declared that logging IP addresses was a "violation of the right to informational self-determination."I was tempted to put this into "Nobody is That Stupid ... Oh, Wait!", but, to be honest, I am not sure I fully believe it...
Wednesday, October 03, 2007
Tuesday, October 02, 2007
A follow-up to this. So, in search for some excitement, I loaded about 800MB of logs from a compromised Linux box (no, not this box - another one ...) into LogLogic and decided to get to the bottom of it. The box was deeply 0wned (like, exploit scanner and 'john' installed and running, etc)
So I run my totally-experimental super-secret log mining tools :-) on the whole set of logs and discovered that there is nothing that screams "0wned!" anywhere in the log set. Ok, so maybe it was owned before the log started, like the other one? Nope, the dates on most of the attacker software and other artifacts are newer than the beginning of the log set (which in fact goes all the way into 2006) - and it didn't bear any signs of date manipulation.
Are you getting it? Of course, it is about stolen access credentials! One of the accounts on the server was used by the attacker, who simply logged in to the server and started using it as his 0wn. Now, how do you detect that? No NIDS will trigger, NIPS will let is pass, no unusual types of log records might ever by produced (especially if only limited logging is enabled). And what raises the stakes is that this type of activity is not only about "hacked" accounts, but also about insider abuse of accounts.
However, there will likely be changes in how normal log records are produced.
Let's summarize some known methods for using a simple user "profile" to detect account theft aka account sharing aka user impersonation aka access with stolen/shared credentials. It implies that we've been collecting the logs before the incident and have a solid trail of normal users and legitimate account owners.
- Unusual login source IP (e.g. normal user always comes from 10.0.X.Y or 10.1.X.Y, but now we see a login from 172.16.0.Z) - will work in some cases, but not for the free-for-all servers such as at a University
- Unusual login time (e.g. normal user always comes from 9AM to 5PM, now we see 3AM) - will work in most cases, but will fail if the attacker happens to be in the same time zone
- Unusual login session length (e.g. normal user always stays logged in for 5-20 minutes, now we see a 10 hour-long session) - works only if logout is logged; might not catch a lot of realistic but malicious connections
- Unusual login frequency (e.g. legitimate user logs in once a day in the morning, now we see dozens of connections) - will work for some cases, but others will be missed
- Unusual login failure/success ratio before a successful login (e.g. normal user always types the password right the first time, now we see failures than successes) - not too reliable, obviously
- Unusual list of user actions performed (normal user only reads these files, but now we see writes to a very different set of files) - will work most frequently, but needs more granular logging of file access, object changes, etc (audit logging) [more on this in the near future!]
So, if you have logs of user activities, at the very least, logins and logouts ( but having records of more user activities is always better!), for the last few weeks or months, one can compute the above profiles using historical data and then compare them with current numbers (very similar to some of the methods from my classic log mining presentation).
The final missing bit is for how long to collect your normal user behaviors: I discovered that 1 week to 1 month works pretty well. Less time yields unstable results and more time necessitates much more data crunching without much gain.
In the above case, it turned out that the method #1 "Unusual login source IP" did the trick. .ro anybody? The question how they came upon the valid account credentials, however, remains... No obvious password guessing was seen in the logs.
However, there is this little tiny issue here: the above implies nicely "parsed" logs stored in a database, where one can always run a query (SELECT timestamp, username, source FROM logs WHERE <whatever> ORDER BY <whatever>) and then mangle the output (again, as I do here in many different ways).
Now, what if you want the above algorithms to run over all logs that contain the usernames and indicate user login/logout activities, not just the "known", parsed logs, neatly stored in a database? Yes, this is where is gets really, REALLY, R-E-A-L-L-Y fun! :-) But I will leave it for future discussion ...
As a final note, all this implies that your logging is on. Otherwise, see this ancient quote still rings mighty true: "In an incident, if you don't have good logs, you'd better have good luck."
Monday, October 01, 2007
I saw this idea of a monthly blog round-up and I liked it. In general, blogs are a bit "stateless" and a lot of good content gets lost since many people, sadly, only pay attention to what they see today. Yes, attention spans seem to be shrinking even among the security professionals ...
So, my 2nd monthly "Security Warrior" blog round-up, top posts and comments by topic.
- Same as last month, the "fallout" from being featured on a high-profile programming site continues to drive traffic. The topic that got such a huge boost was anti-virus efficiency. Thus, these posts were the most popular: Answer to My Antivirus Mystery Question and a "Fun" Story, More on Anti-virus and Anti-malware, Let's Play a Fun Game Here ... A Scary Game, as well as a final entry about my own switch away from AV: A Bit More on AV
- Interestingly, my small post on Once More on Failure of Academic Research in Security shows up in top content. If I see another paper on intrusion detection via neural networks, I will scream!
- No less interesting, Another Presentation: FINAL Full Log Mining Slides made it into Top5. It is indeed a very fun presentation, which summarizes a few years of my logging research.
- Due to a lot of inbound links, my Guide to Hating Competitors is popular. It sparked some fun follow-ups by Mike Rothman and others.
- In a bizarre twist of fate, the latest blurb in my humorous series Nobody Is That Dumb ... Oh, Wait VI made the top list. What gives? Is it the fact that it mentions Gartner? Or G-spot? :-)
See you in October :-)
Well, turns out they were dead serious :-) As I expressed my puzzlement to our resident PCI auditor, he explained that PoS logs and overall security of PoS devices are often "in-scope for PCI, but out of scope for a typical PCI audit." How bizarre is that? But let's start from the beginning.
First, WTF is a PoS? PoS, or Point-of-Sale terminal, is a machine that stores (and whoever else who takes credit cards) use to process credit card transactions (scan cards, communicate with verification server, print receipts, etc). It might be standalone or combined with a cash register. It might very very simple (just card reader + transaction unit in a single hardware unit) or as complex as a Windows PC with a cash drawer and no software updates (yuck!)
So, in the latter case, there are certainly logs involved. In fact, there are also PoS-specific application logs, such as this example below, coming from an IBM SurePoS device:
01/11 09:11 CC 5 W518 PROGRAM ADDDDDXUXDL HAS ENDED
B3/S111/E007 REASON=2 TYPE=3 RC=00000000
REASON: Application ended PROGRAM TYPE: Background
RC: No error
PoS devices might be configured to store credit card numbers locally (for backup) and also to offload them to a "branch server" (a store server or both a store server and a regional server). Are there logs of who accessed them on the local PoS system? Maybe. Are they looked at? Probably not. Maybe the logging is done better on the branch or store central server, but even this is not a certaintly
Overall, I am willing to bet a bottle of decent champagne that very few people, if anybody, in the whole world is regularly looking at PoS logs. At some happy point in the future, I predict they will start since the Beast of PCI will make them :-) When this happens, we will talk about PoS log analysis.
As of today, you would do comparatively well if you will collect and save them and thus will have a chance to review them for incident response for your next data theft case (or show them to an unusually nosey PCI auditor...)
More fun PoS security reading is here [PDF].
This post is obviously dedicated to the just-passed PCI DSS deadline ...
So, my entry on abandoning the "classic" signature-based anti-virus have generated mainly two types of responses:
- "What? You've been using AV all this time? Come on, everybody knows it is useless crap"
- "What? You abandon AV? How about defense in depth?"
And, BTW, Savant Protection does bundle a small signature-based AV engine (I think it is ClamAV), but it is not really essential for most of the protections and is probably only used to catch the truly stupid, obvious stuff.