How to Best Structure your Logs: Log Analysis Tips and Best Practices with Gal Segal from eToro

 

eToro LogoThis week we have a guest blog post by Gal Segal. Gal is an engineer at eToro, the worlds Largest Social Trading & Investment Network. In this post Gal shares his thoughts on log analysis best practices including tips on how best to structure your logs as well as some useful patterns that can be applied within your log events. He also discusses how to use Logentries’ new log search functionality to more easily search your logs. Gal and his team at eToro use Logentries every day to analyze the mountains of server logs produced by their popular social investment network.

This post has been republished from Gal’s programming blog .

Just had a great talk with the guys from Logentries. We talked about some major improvements they’ve made and I wanted to share some thoughts. One thing that came up was the ability to make more complex searches on your logs.

Logentries Search Engine

Logentries search engine uses Reg-ex to filter data for the desired results. It is great and offers some nice flexibility, but in the past if you wanted to perform more complex queries, well, you had to dig deeply into proper Reg-ex syntax. I do many searches and always wanted some more power in my hands to get more accurate results. So if you are a log addict like me, you are gonna love these new Search features in Logentries!

Some Tips About Logging

Logging is good. Log everything.

There are 2 main things a log line should deliver:

    • Be readable (for humans or machines, depends on who or what will read it)
    • Be easy to search

Now, it is a good practice to adopt some patterns across your organization when it comes to logging.

These are the logging patterns I use:

  • Give each line a topic that suggests whats it about. Use brackets (like \[some term]\) to make searches more simple.
  • Use key value pairs inside the log line. You can use JSON objects or just key=value patterns.
  • Decide which keys are a most on all logs, such as “username”, “action”, “context”, “latency” and so on.
  • Add more data freely but in the same key value format.

For example, consider a log line like this one:

2013-06-21 09:31:30 [Stock] username=john action=buy stockName=goog sucess=true

This gives me a nicely structured log event that allows me to easily search and get great visibility into my apps. It can be particularly powerful if you can easily combine search terms to dig deeper into your events.

Introducing AND OR NOT Filters

Although Regex supports AND OR NOT filters, I couldn’t wrap my head around it and I found it difficult to use on Logentries when this was the only approach for performing more complex queries. Now, however, they have introduced AND, OR, and NOT keywords so we can easily use these to perform our more complex queries.

Lets say you want to search for entries that contains the terms a and b (**Make sure the conditions are capitalized.**) Then you would enter the following query:

a AND b

a or b:

a OR b

a  but not b:

a NOT b

You can also use multiple AND OR NOT in the search query.

Free Text Searches

Often times you need to search for partial matches, like free text search. For example, consider a log line like this one:

2013-06-21 09:31:30 [Stock] username=john action=buy stockName=goog sucess=true

If I would like to search all stock buy actions I can use this:

Stock AND /action=buy/

The / stands for partial search. Basically it means you treat each search term as a standalone search and combine it with the conditions.

Summary

Logging is one of the foundations of good programming. We need to do it well and have a good set of tools to analyze them. This improvement is pretty awesome IMHO and answers a real need in the developer’s life.

Happy logging!

This post has been republished from Gal’s programming blog .

Logentries_Try_It_Free_30_Days_Button_with_bar

Posted in Tips & Tricks

Leave a Reply