Anton Security Tip of the Day #6: The Other Web Log

Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to go along and join the initiative. One of the bloggers called it "pay it forward" to the community.

So, Anton Security Tip of the Day #6: The Other Web Log

We all know that Apache web server has its access_log log files while IIS has its w3c logs. When people think about web log analysis, they think about the above and only about the above logs, often calling them "the web server logs." However, the other web log - error_log (in case of Apache) - contain a lot of fun and useful info as well. Today's tip is to encourage the review and analysis of this treasure trough - pardon the idiom - of insight.

So, what goes into the error_log? Well, errors, duh! Why errors matter? 'Cause errors often present the only indication that a) you are being 0wned or, worse, that b) you've been 0wned by the attackers. Additionally, apart from the obvious security usage mentioned above, errors matter since they - or rather, some of them - require actions by the user and thus knowing about them is of huge importance. What is even more striking is that many error messages present an interesting tradeoff - do only a little now to correct the reasons (or, "the root cause" as network management people would say) for the error, or do a whole lot later when the system crashes, gets hacked or "misbehaves" in some other truly spectacular way. Now, a grizzled BOFH would surely say "heh, but who would want to do that! surely, dealing with a dramatic world-ending catastrophe later is more fun that making a minor config fix now." Well, I sincerely hope not all of my readers are such people :-)

Now, back to the Apache errors; we are going to show a few examples from a live phishing site, run by the attackers on a compromised server. We will start from the obvious - server restart, which actually happened as a result of an attack, in our case.

[Sun Mar 12 04:02:05 2006] [notice] SIGHUP received. Attempting to restart

Did YOU restart the server? No? Two choices then - someone else did (likely without permission!) or the server got restarted automatically for whatever strange reason. You certainly need to be at least somewhat concerned in both cases - thus: look for the above messages.

Here are a few more fun examples - the relevance of those for your business is left as an exercise for the readers, but all of the messages below

[Mon Mar 13 14:54:11 2006] [error] [client 10.10.10.10] Premature end of script headers: index.htm

Does "index.htm" sound like a script to you? It sure should not; and this error message indicates that somebody is trying to run it as a script - a clear indication of malicious behavior. The next message is also similar in this regard:

[Mon Mar 13 14:54:26 2006] [error] [client 10.10.10.10] attempt to invoke directory as script: /var/www/cgi-bin/tcpsupport/

and so is this one:

[Mon Mar 13 14:54:23 2006] [error] [client 10.10.10.10] (8)Exec format error: exec of '/var/www/cgi-bin/tcpsupport/main.htm' failed

Other interesting things spotted on the same server -this one looks like an overflow attack attempt (and, no, the NIDS did not make a squeak about it)

[Thu Mar 19 07:16:11 2006] [error] [client 10.10.10.10] request failed: URI too long (longer than 8190)

So, to conclude, the tips is: when doing web server log analysis, make sure you look at the error logs as well as the access logs.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.