Custom log processing/parsing

I have such log format:

[26830431.7966868][4][0.013590574264526367][30398][api][1374829886.320353][init] 
  GET /foo
  {"controller"=>"foo", "action"=>"index"}

[26830431.7966868][666][2.1876697540283203][30398][api][1374829888.4944339][request_end] 
  200 OK

The entry is constracted using such pattern:

[request_id][user_id][time_from_request_started][process_id][app][timestamp][tagline]
  payload

Durring request I have many point where I log something - app basically has complex behaviour. This helps me debug a lot the user behaviour.

The way I would like to parse it is that I would like to make have directory structure like this:

req_id
  |
  |----[time_from_request_started][process_id][timestamp][tagline]
  |
  etc

Basically each directory will have name based on req_id, with files wchich names are rest of tagline. These files will include payload.

And also I will have other directory, with users ids, which will contain symlinks to request done by this user.

First question: Is this structure correct? In my opinion it will make easy fast log access. The reason I want to use directories and files is that I like unix approach, and try it (feel by myself its drawbacks and advantages)

Second question: I will have no problem to use ruby for creating this. But I would like to learn some new tool, which is better suited for this. I am thinking about using just unix tools (pipe, awk etc) to achieve this, or write parser in golang which I am learning right now (even have time to implement simple map reduce). What tool is best suited for this?

I would not store logs in a directory to see how the users behave.

Depending on what behaviour you want to keep track of you could use different tools. One of these could be mixpanel or keen.io.

Instead of logging what the user did in a log file you would sent an event to either of those (they are pretty similar, pick the one you think has better docs / lib), then you would graph those events to better understand the behaviour of your users. I've done this a lot recently, to display data in a nice way I've used rickshaw.

The key point why I'm suggesting this is that if you go the file route you will still have to find a way to understand your data, something that graphs will help you a lot at. Also, visualization is something keen.io does by default, you may still want to do your graphs but it's a good start.

Hope this helped.

> Is this structure correct?

Only you can know that, it depends directly on how the data needs be accessed and used.

> What tool is best suited for this?

You could probably use UNIX tools to achieve this but it may as well be a good exercise to practice your Go skills by writing this. It would also be more extensible.