Why Replace Your Baby?

Indeed, many people treat replacing their beloved "family-grown" log management system with a commercial logging solution similarly and thus experience intense anguish and other extreme feelings :-) Let's look at this syndrome  in more details. As we all know, people love to write their log analysis scripts and little proggies (a hit of sourceforge confirms this, but there are great many more which are not shared with the world).

Some did more and built super-sophisticated log analysis systems. When I gave my Approach to Log Management presentation, I always related a story of a major bank which build an awesome system to analyze events which - gasp! - beat what most commercial players had on the market at the time (!). They used visualization, data mining, data profiling and a lot of other cool things.  Others tried to beat major commercial vendors on scalability (ha!) and ended up creating only hilarity for everybody and a need to look for a new job for themselves (here).

So, what are the reasons why so many in-house log management projects fail miserably, often taking their creators down with them:

1. Ongoing maintenance will kill you ☺ - logs formats and messages change  ALL THE TIME. Will you take on keeping track of that? This is by far the #1 reason for such projects to end badly!
2. No support, apart from you - you might like that, but will your boss? At some point, many creators of such apps start hating them due to the care and feeding they need...
3. Does it pass the “bus test”? - Ah, the dreaded "bus test" - will the application still run if the chief/only developer gets hit by a bus? If you have 23,000 lines of Perl in your log analysis app, I can tell you the answer is NO!
4. Handling production log volume - it works really well in the lab on your own system. But when that "M" becomes a "G" (megabytes into gigabytes) and maybe a "T" (in extreme cases - "P"), will it work? Probably not...

So, what should you do? For most companies, buying is the answer. Really, the common choices are:

1. Build -> suffer -> buy
2. Buy
3. Built a little -> plan what to buy based on that -> buy

What is also exciting is that after you bought a log management platform, you can build some cool stuff on top, using available API and other things (e.g. here). More on that in the future!

Finally, of course there are situations when building is the only choice (e.g. here)

Technorati tags: log management, logging