Following the new "tradition" of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it "pay it forward" to the community.
So, Anton Security Tip of the Day #2: Watch those Pesky Log Rotation Routines
What is the biggest threat to your Linux logs, which reside happily at their home in /var/log? Is it crackers? Blackhats? Evil BOFH sysadmins? Dumb users? Malware? No, its the log rotation routine!
For example, many Linux distros such as RedHat, Fedora, etc use logrotate tool to get rid of "unwanted" logs. Other distros use other log file rotation tools that accomplish the same. However, many ship with default settings that are not optimized for log analysis and long term retention.
Upon deploying a Linux box one needs to look at /etc/logrotate.com file as well as /etc/logrotate.d/ directory to make sure that you are happy with retention settings (and you shouldn't be!)
One can safely increase the retention of most log files from a default of about 4 weeks (it differs by file type and distro) to at least 12 weeks (about 90 days, a common practice) or even more. While at it, one should also enable old log file compression since it is off by default (for whatever weird reason...) to save some disk space.
Note that if you are using a central log management system, this might not apply to you. But then again, in this case you are farther ahead of most system owners on the road to operational excellence.