System logs are one of the great untapped data resources of IT shops. They are analysed, usually in a silo context, often for a machine-specific task such as capacity management, performance analysis or used for post-exceptionaudits. Log files from different systems are not usually brought together and mined, or indexed, for post-hoc reporting, let alone real time analysis.
Security customers and vendors are one constituency that has naturally tended to appreciate the value of event logs. But usually this information is considered in quite a narrow context. Its all about correlating security and network data. That is not to say that companies like Tripwire or CyberTrust or QCC don’t provide a valuable service. But its clearly bounded, and concerns a particular set of risks.
Compliance to corporate or regulatory standards on the other hand is a much broader issue, which security incident management is just one element of. Compliance is a business process issue that goes far beyond tracking intrusion detection exceptions and patterns.
There are some vendors and technologies emerging that are looking to challenge the idea that log data is cheap, and should be written to background storage in case they are needed later.
Log management and analysis is not a subset of security incident management (SIM). In fact SIM is a subset of log management.
Log data can be tremendously valuable. I would point to LogLogic, LogRhythm and Splunk as some organisations to check out. You should check out the Splunk homepage, if only to see the rather amusing “What’s H0rked in your infrastructure today?” homepage. You can download Splunk and check it out. Get a feel for log analysis. Prism EventTracker and Network-Intelligence are another couple of firms in the space.
So what’s up with Log analysis? What is your approach? Are you using it in a compliance context? At RedMonk we have decided to track log management and analysis as a sector in its own right, as part of our compliance oriented architecture research.
Where does compliance meet log management and analysis? In reporting. Its all about reporting. How do you get IT talking to the business? Provide reports in a language they understand – like Sarbanes-Oxley or MiFID. If you’re going to pay a vendor money for a log analysis solution demand canned reports your business managers will appreciate. Expect them to drive business semantics into the software for you.
If you are a vendor of a log management tool, or a customer, please let us know – we would very much like to speak to you. Cote talked to our thinking here.
disclaimer: I recently met up with Andy Lark, chief marketing officer of LogLogic, a pal first and a RedMonk client second, and we drank some fine wine at Bedales and talked business. I also know Andy Grolnick , LogRhythm president, and would like to win the company as a client. Stephen O’Grady looked at Splunk, which is open source, and said it is interesting. I have friends at Tripwire and QCC. I had lunch with CyberTrust last week. I think that’s it.
Jaime Cardoso says:
March 22, 2006 at 3:56 pm
Congratulation, I had never seen an analyst covering the logging and reporting needs of an organization.
I feel these issues to be critical and, I had some wins because we could provide the customer with superior logging facilities (I even have a customer that set up SAS just for log analyses to better provide information to the business areas).
Most customers, unfortunatelly, don’t realise the gains in having the business side understanding what and to who the I.T. folks are servig their services so, most reporting capabilities are tied with the skills of the scripter, …
James Governor says:
March 22, 2006 at 5:13 pm
as usual you’re right on the money jaime. should i be thinking about SAS as a log management vendor- is their stuff packaged that way at all?
maybe we can do a briefing with you me and Cote and nail some thoughts down…
Jaime Cardoso says:
March 22, 2006 at 7:50 pm
I met SAS because of my friends that studied statistics. when that customer needed a solution to do complex analyses, predictions and all that stuff, I pointed them to SAS and to all the trainees I knew (at the time we were Hardware sellers and only got involved with it when it became time to tune the solution).
Putting a platform like SAS digging up correlations in log file (from applications, accesses and pretty much anything you can think of leads you to find pretty impressive stuff, stuff that can help your business (and do a much better acounting on the value of I.T.) but, Statistics apps vendors don’t seem to realise there is a market for with they already have products.
About the briefing, sure, call, email-me, whatever 🙂
James says:
March 23, 2006 at 1:00 pm
Log Management is important to enterprises in order to achieve compliance. I am struggling with whether log management is really isn’t own product space or a component of a larger problem-space of entitlements.
I have blogged on XACML and its potential usage to have a consistent entitlements model for the enterprise. Tools that support XACML could themselves handle log management and therefore shouldn’t be a distinct product line.
What am I missing?
james governor says:
March 23, 2006 at 3:27 pm
for one thing, James, not everyone supports XACML – so while I take your entitlements point, its really about sweating existing assets (logs), rather than introducing new structures, at least as i see it.
Cote' says:
March 24, 2006 at 5:53 am
The SAS connection is good. We should look into that more and perhaps talk with people, definitly to whoever it using SAS to do log management if we can.
Of course, when it comes to making sense of loosely/unstructured data, the search crowd comes in too: I’m having visions of text ads in log management tools: “why not call up Roy’s SysAdmin to fix this? Low hourly rates!” There’s a nice intersection of micro-formats and systems management too. Indeed, there’s long been many de facto standards in unix-land when it comes to logs.
Jaime Cardoso says:
March 24, 2006 at 5:44 pm
James: Log Management isn’t important to achieve compliance, from what I’m seeing, legislation is seen as a burden and very few organizations actualy take advantage of what was imposed by law.
If you talk with people in big comercial spaces, you’ll see that they spentd a lot of time and money traking customer to see the path they take on the corridors and stuff like that. That info is critical but, when you change the subject to their website, they know nothing about where the customers go, what they look into and how much time they spend on each page.
There is a lot of value in knowing what your systems are doing and, the is log analyses.
Cote: I kind of agree with you when you say that the searches crowd has something to say in this field but, truth is that the most interesting logging is made when you don’t actualy don’t know what you’re looking for.
If I know what I’m lookind, a log file and a shell is enough for pretty much anything (eat that you windows users). It’s when I don’t know what I’m looking for that it’s a problem.
There is a name for finding stuff that nobody was looking for (no, I’m not talking about Windows Vista): Statistics
People Over Process says:
March 31, 2006 at 11:14 pm
Systems Management in the Real World: erfwireless NOC
A couple weeks ago, I had the privilege to visit with one of my old friends, John Arley Burns, at his work, ERF Enterprise Network Services. He’s the President and COO which means, among many other concerns, that he…
People Over Process says:
March 31, 2006 at 11:16 pm
Systems Management in the Real World: ERF Enterprise Network Services
A couple weeks ago, I had the privilege to visit with one of my old friends, John Arley Burns, at his work, ERF Enterprise Network Services. He’s the President and COO which means, among many other concerns, that he…
Gal says:
June 14, 2007 at 4:00 pm
what about XpoLog at http://www.xpolog.com
it is a log analysis and management platform that supports both log search engine, anomaly detection and many more features that helps you get the best of your log file data.