Extracting Key Values from Any Log Format Using Regex Named Capture Groups

Logs are event streams that are constantly spewed from every application, server instance, mobile and IOT device. They contain valuable information pertaining to application errors, system performance, security, feature usage and more.

Likely, the biggest issue with getting value from log data has been its structure, or lack of it… and while today many systems can produce JSON structured log data with keys and values that can be easily identified and analyzed by both human and machine, the vast majority of log data produced is in a semi- structured flat format e.g. syslog, apache, IIS events. – frank [10/Oct/2000:13:55:36 -0700] “GET /apache_pb.gif HTTP/1.0” 200 2326

Flat Apache Log Format

06:58:38.336 2015-01-07 11:58:38 at=info method=GET path=/images/sample.gif host=aaa.herokuapp.com fwd=” dyno=web.1 connect=1ms service=744ms status=200 bytes=12268

Heroku Key Value Pair Structure

The two examples above of the flat apache log vs. a Heroku key value pair (KVP) structured log highlight the value of adding some structure to your logs. The KVP log is more readable and easier to understand. Furthermore log management services will often identify the fields in KVP structured logs automatically so that you can easily extract and work with the values. For example, in the Heroku log above I can easily search only for logs where the value of the field ‘service’ is greater than 100. I can also then easily work with the results and visualize the average ‘service’ value over time.

service>100 calculate(AVERAGE:service)

Example Search Query for Heroku Log in Logentries

Screen Shot 2015-01-07 at 7.31.30 AM

Results of Query

This has always been harder to do because logs did not have any key names, and you cannot easily reference the values in your search query. Today, however at Logentries we have announced the ability to better define the structure of your logs through our RegEx field extraction using Named Capture Groups .

This capability allows you to define a field using RegEx such that you can more easily work with values in flat log events (think syslog, apache, windows event logs, mysql etc.). You can now define and work with field values in almost any log format such that you can start to perform queries and analytical functions on these logs as if they were in a structured format.

This can be particularly useful if you cannot change the configuration of your apache logs because they are produced in JSON or if your application produces some strange custom log format and therefore you do not have access to modify the log format.

Our new RegEx field extraction works by allowing you to identify values in your log events then allowing you to assign a name to these values – similar to having a Key value pair in your log events, using named capture groups. By assigning a name to the identified value(s), these values can be used with advanced search functions such as GroupBy or for calculating Counts, Sums, Averages or Unique Instance Counts. They can also be used for comparisons when creating alerts or tagging events.

Using the RegEx field extraction is pretty straightforward and in many cases you don’t need to be a RegEx expert (we’ve provided some nice common examples to help you out). To allow you to extract values from your logs we have followed the RE2 RegEx named capture groups concept. By using the RE2 regex named capture group syntax, you can identify a value and assign it a name which can be used in aggregation functions as in the following example:

12:12:14 new sales event – customer Tom – total sale 24.45 – item blanket

12:12:15 new sales event – customer Tom – total sale 100.45 – item jacket

12:12:16 new sales event – customer Tom – total sale 1000.33 – item computer

Sample Log events

 /total sale (?P<saleValue>\d*)/)

Sample regex to find the value following ‘total sale’ and assign this value to a named variable ‘saleValue’ 

Once you have identified and named the value you can then reuse the named capture group in a comparison with a search function.

/total sale (?P<saleValue>\d*)/ Calculate(Average:saleValue)

Sample Query to find the Average ‘saleValue’ including regex

KVP logs

Example of using RegEx Named Capture Group Field Extraction

Adding structure to a log of any format allows us to consider logs as a universal language of communication across teams and systems. We can now take logs in any format, identify the important fields in those logs and begin to analyze those fields. This is useful for both team and inter-system communication. In particular as today’s systems move to the cloud, logs are becoming even more valuable for  understanding system and application performance. Why? Well, it is difficult to instrument the cloud or cloud services and, thus, alternative approaches are required to give visibility into cloud-based components which otherwise can become black boxes from a performance or system monitoring perspective.

While it can be difficult to instrument cloud-based components, in general they tend to produce log data streams or provide access to APIs that can be polled to generate data streams. These data streams can be analyzed to give visibility into your systems for a range of different users and use cases, for example:

  • development: performance testing/optimization, error tracking
  • operations: monitoring performance metrics and server resource usage, identifying threshold breaches and anomalies, tracking trends
  • product management: understanding usage metrics and trends in how features are being used (note using logs for usage tracking means no need for complex or expensive BI tools – I like to call it ‘risk free analytics’)
  • marketing: tracking ad campaigns – e.g. via pixel tracking

Some examples of how logs can also be used as a universal language for intersystem communications:

  • For IT Dev and Ops Automation:
    • Tool integration: logs regularly act as the glue between different tools in your Ops toolkit. For example logs can be used to track errors, send a notification to a third party API (e.g. PagerDuty, Twillio, HipChat) such that it can be communicated with across your team. Similarly log events can be linked into bug tracking tools like Jira such that they provide relevant context to bugs for debugging and resolution.
    • Auto scaling: logs can contain information on server resource usage and are regularly used as a way to track trends in your server metrics and to trigger autoscaling of more servers where resource capacity is reached
  • For IOT:
    • It’s common for IOT devices to spew out streams of data in lots of different custom formats. Often a centralized controller or management portal needs to listen to this data to understand the network of devices and how they are behaving. Being able to translate any log format into meaningful metrics (e.g. using RegEx field extraction) opens up logs as an intermediate language for IOT system communication – expect logs to play a key role in IOT as it continues to grow.
  • For system integration:
    • Users of logentries regularly send events into our system, and then use our open API to extract key metrics or events that act as the input to other components in their system.

Logs have always captured important information, but due to their unstructured nature they have always been difficult to work with. At Logentries we think of them as a ‘linga franca.’ Linga franca is a Latin term to describe a situation where a third intermediate language is used to make communication possible between persons not sharing a native language. Lingua Franca was actually a mixed language that was used throughout the eastern Mediterranean as the language of commerce and diplomacy in and around the Renaissance era (14th-17th centuries) and was composed mostly of Italian with a broad vocabulary drawn from languages like French, Greek, Spanish, Portuguese and Arabic. If you think about it, logs are very similar insofar as they are a mix of information often pertaining to system errors, user actions, security related events, performance metrics, system load and more…Now with the ability to work with any log format, log management technologies are serving as a translation service allowing for communication through this lingua franca between team members and different systems and devices. Logs are thus fast becoming the universal language of communication for today’s systems.

Posted in Application Performance Monitoring, DevOps, Heroku, Log Management, Search

Leave a Reply