Logging and Web Services

This post about the paper we wrote earlier this year (“Logging in the Age of Web Services” by Gunnar Petersen and Anton Chuvakin, published in “IEEE Security and Privacy”) will just prove how behind I am on blogging :-) Fortunately,  Gunnar announced this paper a long time ago; I just added it to “2blog” folder :-) The paper will be a useful read to those into either logging or web services or both. I do suspect that web services audit logging will be one of the unresolved mysteries when SaaS/PaaS/IaaS/cloud everything will enjoy further adoption…

Anton Security Tip of the Day #14: More access_log Fun: What Are You Not GETting?

Following the tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #14: More access_log Fun: What Are You Not GETting?

In this tip, we will look at some bizarre artifacts that show up in web server access logs today. Here we have a production log from an Apache web server that is full of interesting (and sometimes ominous!) little mysteries that we will investigate in order to determine their impact on security and operational health of the site.

Logs do contain more mysteries than we have time, so we will focus on a few of them: specifically, unusual web request methods. Let's see who is trying to POST or use some other method (OPTIONS, HEAD, PUT or something - see a list here) on our site, instead of just GET'ting the content (GET command is used by web browsers to retrieve the pages, while POST is used to upload content, press buttons, etc - at least in "web 1.0" land - see earlier tip #12 where POST request was found in proxy logs)

Here is one little artifact that attracted my attention due to a POST request vs a web forum as well as a battery of slashes (which actually increases in subsequent request - of which there were many)

10.10.102.250 - - [12/Feb/2008:16:10:50 -0500] "POST /phpBB3////ucp.php?mode=register&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0" 200 14397 http://www.example.com/phpBB3///ucp.php?mode=confirm&id=7640df05c7e24b7acf7a68800fe6dc59&type=1&sid=e5efaa77a777066c61f71808e9e57b19 "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126"

... more...

10.10.102.250 - - [12/Feb/2008:16:12:29 -0500] "POST /phpBB3///////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0" 200 9355 "http://www.example.com/phpBB3//////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19" "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126"

This one really is a mystery; what do we know about it? The server responded to the request OK (code 200), so the POST actually happened. The first request was a request to register with a web discussion board and the second was a request to login. Multiple slashes are actually ignored by the web server, so why put them in the request (no answer)? Also, I think that the User-Agent is spoofed ... do you know why? Finally, if I see something like that in my logs, I will definitely investigate it, primarily due to the fact that Apache responded with 200 OK code.

The next one is so classic it it dumb (and so dumb, it's a classic :-))

10.10.123.226 - - [12/Feb/2008:03:46:54 -0800] "POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1" 404 - "-" "MSFrontPage/6.0"

10.10.123.226 - - [12/Feb/2008:03:46:55 -0800] "OPTIONS / HTTP/1.1" 200 20210 "-" "Microsoft Data Access Internet Publishing Provider Protocol Discovery"

It is probably one of the ancient IIS attacks (check out this fun BlackHat preso on that, circa 2003) - why would someone probe for it now is beyond me. In any case, Apache on Linux and "*.exe" don't mix :-)

The final log record is also fun:

10.10.101.222 - - [12/Feb/2008:15:33:22 -0800] "PUT /zk.txt HTTP/1.0" 405 223 "-" "Microsoft Data Access Internet Publishing Provider DAV 1.1"

The above uses a PUT request which is pretty much deprecated now; the purpose of the above is clearly malicious. In fact, modern Apache shouldn't even allow it, thus it responds with code 405 "Method Not Allowed." Nothing to worry about (even though some poor critter got owned with that! BTW, if you follow that link, check out HTTP response code 201 - if you see it in your logs, run! :-))

Overall, this tip teaches to look for unusual request methods (POSTs to strange pages, all PUT, DELETE, OPTONS requests, etc) and then check the response codes to assess the impact. If your web server happily executed such strange request (code 200), that you'd need to dig further. And, you "lucky" :-) and you see the response code 201 "Run for the Hills" (in reality, it stands for "New File Created"), then you can go straight into incident response mode.

Another lesson to learn is that if you see too many POSTs or too many "GET then POST" sequences from the same IP in rapid succession, investigate it since no legitimate access should produce such a pattern...

As further reading, I heartily recommend this paper: "Detecting Attacks on Web Applications from Log Files"

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Fun Preso on Proxy Logs

I did a few insightful webcasts for LogLogic lately, here is one of them (webcast with voice, slides only), on analyzing and managing web proxy logs. It goes well with my logging tip #12, also on proxy logs.

Anton Security Tip of the Day #4: Code 200 = Code Red?

Not the wormy "CodeRed", but Red Alert, mind you.

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #4: Code 200: Good or Bad?

Now, this is somewhat related to my previous tip (#3), but as it applies to web server logs, such as IIS Extended W3C (or other) logs or Apache access_log logs. Unlike, say, databases, all webserves come with logging enabled, so web server log review is a common task for web server admins as well as for security teams. But what do you look for in normal web access logs, apart from web site access statistics and visitor trends?

Obviously, observing various access failures is important. All the pesky 404s, 401s as well as 50X response codes.

Here are a couple of examples from Apache web server access_log logs:

61.52.47.251 - - [26/Sep/2004:21:10:22 -0400] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 286 "-" "-"

or

67.170.226.184 - - [10/May/2004:13:18:09 -0400] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 400 293 "-" "-"

Is this ominous or what? :) Nah, it is just my Linux + Apache honeynet responding to some good ole IIS 4.X attack. Truly nothing to worry about.

How about this one though?

68.49.152.86 - - [22/Apr/2004:11:56:49 -0400] "GET /cgi-bin/awstats.pl HTTP/1.0" 200 2190 "-" "Mozilla/4.75 (Nikto/1.32 )"

Somebody using the web statistics CGI script? Benign, since the response code is 200 which means that the server served the page (or ran the script!) as expected? No, you are about to get 0wned thru an AWSTATS hole!

So, the gist of this tip is similar to the previous one: when monitoring web server logs, do not just focus on 404s, 401s, 50Xs and other "known bad" response codes; take a long hard look at the vanilla 200 response codes. "But how do I separate the "good" 200s from the "bad" 200s?" - you might ask! Well, that would be the subject of another tip in the future!

Just in case, here are the links to my previous tips: #3, #2, #1.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: security, tips, log analysis, log, web logs, web servers, log management