Splunk Your Logs

[Gollum]

The only conversation I could find on the topic via Google is here:

https://www.raspberrypi.org/forums/viewtopic.php?t=14646&start=25

And some other talk about it here:

https://groups.google.com/forum/#!topic/loguino-users/WFDGp9oZgd4

 

I know Splunk won't be for everyone, especially as getting it set up is a bit of work, but that said it's certainly something anyone could follow along with a screen capture session to get started with. And I know we don't REALLY need another log viewing tool, but I'd argue that there's a lot of value in being able to bring some "big data analytics" to the table when it comes to digging into logs. While most log viewers are focused on drilling down into small point in time info, splunk would allow you to look at data correlations across large numbers of logs. I'm no splunk expert by any means, but hopefully I can continue building stuff useful for myself, while I share with the community.

 

Right now the log ingestion is a bit specific, and could easily break if anything doesn't mimick my setup. So some known basic requirements if you want this to work with my splunk app:

1) Logs should be stored in the default path of c:\users\$user\documents\tuner studio\$project name\Datalogs

  The above $user and $project name will have to be different than mine but it's important the layout match.

2) I'm using the stock log format output. No custom field, using Tunerstudio free edition (plan to purchase in the next week or so). The field layout might differ in your firmware and/or tunerstudio version. Any changes to this will break the data lookups, as I've had to hard code the field extraction. Once I figure out how to automate that field extraction from the header row (which isn't a real header row.... grrrrrrr) then this requirement won't be an issue. Also, once you have your field extractor working, the dashboards should work regardless of what data you're logging (meaning it's JUST the field extractor that's hard coded, there's a few things I rely on, like "RPM" but it should all just work).

 

As far as release, I'm not quite there yet. I'll start a github repo and post some youtube videos once I get there.

 

But as a tease:

 

That represents less than a day of actual work from start to finish. One of the things I like about splunk is that it's relatively easy to build visualizations once you know the basics.

[Highstakes72]

Gollum,

 

I work for Splunk and I am also a car guy. I use MS3 on my 1972 Turbo 4.8 Nova. Can you share your present work on this topic?

[ZGuy]

I'd better hurry up and get a MS controlled car running so I can participate too!  Just have to tell wife and daughter to stop having failures on their cars so I get greasy on what I want and not being their service tech!

 

I've started an IT job about 6 months ago and have recently been told I have to figure out how to setup SPLUNK to monitor our server logs.  I'm a total neophyte on SPLUNK, but it is an interesting tool for consuming and interpreting log files.

 

Cheers,

Tom

[Highstakes72]

Well, PM me. I can set you up with all you want to know. Prior to joining Splunk I ran a ~6TB/day platform for a fortune 50 company.

 

-Robbie

[Gollum]

Wow, I'd just about given up on this thread....

 

My current status isn't any different than it was when I last spent a day at it. The biggest and most obvious obstacle is transforming the data into something useful from a time-series perspective. Splunk like other similar tools really likes data to have a valid time series format so that a lot of the logic "just works". My issue is that MS logs don't work off of something like a GPS clock which is standardized. If you don't have a RTC you don't even get good NAMES for your logs to track down a valid way to transform data. If you could be 100% sure on file naming conventions and that name containing valid time references, you could transform the time column into valid time for splunk.

 

Once that's handled.... the rest is "easy" in that you transform/rename columns that you care about, and can also perform basic graphing, smoothing, plotting etc. I'm personally still changing my tune on an almost daily basis, so long-term data viewing isn't hugely relevant yet, but I should likely get back on this task soon. I've got a growing number of logs, and I could at least be using splunk to force my hand to name and organize them, and also have a way to easily recall data.

 

Log ingestion is only half the battle though. Someone needs to do some dashboarding to make the data useful. In what I'd consider an ideal world, you could have a splunk instance that people could toss logs at from all variety of makes/models and you could compare logs of similar or even indifferent setups. I should also be able to drill down to a single time under WOT for a single log as a "dryno view" dashboard, though that shouldn't be the point of this excersize. The real benefit here is to be able to look at dozens or hundreds of logs in a single pane of glass to view trends and anomalies.

Side note: Highstakes, if you ever work out of the SF office I'll have to stop by sometime. I've attended a couple of the monthly user group meetings, and though I don't work in SF at the moment I'm just north in San Rafael.

[Highstakes72]

Yeah, completely agree the default log structure is garbage. I am thinking of making a python "connector" that will take the initial datetime stamp and then create new composite timestamps using the "seconds" integer field. Then push that into Splunk in a serialized fashion. My group is based out of Tyson's Corner, VA but I am remote in north Texas as I cover some sites across the southwest.

[Gollum]

Yeah, I was doing essentially that via bash but that's not scale-able. If someone were doing this with a public service, you'd likely want to use a lambda function on upload of logs. I'm not sure I can quite afford that as a free service to people at the moment, but I can't think of a better way to convince people to donate logs (which is what I'm really after, ALL THE DATA!). Alternatively I could just run a small free license at home with a sftp site for people to drop logs and I could publish reports and take requests from people to show data. But I'd really rather have some help on the dash-boarding side of things as I don't have the time or the expertise to do it quickly/easily.

 

Well, if you're ever visiting home office and have time, I'll take ya for a ride in my ratty 280z.

[Highstakes72]

On the license piece, you can get a 10GB/day license if you sign up as a Splunk developer, which would be legitimate in this case. You have to renew the license every 6 months. On the SPL content side, that is easy enough to get help.

[Gollum]

Made a touch of headway on this today. I had some time, and decided to hack away to see how rough this would be with my existing knowledge of awk and the like. Here's my v0.1-0 data transformer:

https://github.com/nshobe/megalog

 

It works, at least for my logs. Uploading that to splunk I get 100% proper data ingestion with no post extraction required.

 

 

And of course all the fields populate automagically:

 

So yeah, just tossed the server instance up today, so no public user access yet. But that's not beyond possible. I'd like to have a proper log file uploader configured first though, so people can send their logs with the proper metadata.