Heroku Labs, Heroku’s platform for providing “experimental features that are under consideration for inclusion into the Heroku platform”, recently launched
log-runtime-metrics for “enabling visibility into load and memory usage for running dynos”. With
log-runtime metrics, Heroku now inserts the following per-dyno stats directly into the log stream:
- Memory use
- Swap use
- Load average
log-runtime-metrics is currently free to use and enabling the feature is as easy as running the following command:
heroku labs:enable log-runtime-metrics.
Important! Don’t forget to restart your dynos after you’ve enabled
log-runtime metrics (this had me stuck for a good 10 minutes before realizing I forgot to restart).
The load and memory usage metrics appear as system logs in the Logplex log stream while per-dyno metrics are generated every 20 seconds.
You should note that adding
log-runtime-metrics can make the “dyno” key a bit confusing to use at first. Before adding
log-runtime-metrics, searching the
dyno keyword in a log management tool would have returned simple dyno names like
web.1 to help you identify individual dynos. When enabling
log-runtime-metrics, metrics use the key “source” to identify dynos by name in your dyno formation (e.g.
web.1) while the key “dyno” is used to identify the app id and a UUID to uniquely identify every distinct dyno on the Heroku platform. The purpose behind this is to keep the source value persistent while allowing you to have many different dyno values (which becomes useful when deploying or restart dynos). As you can see below, this change can make grouping by the key “dyno” a little misleading. I only have one dyno running, but the variation in how the Dyno key is used can make it appear that I have 2 Dynos running if I’m not paying attention.
From here, Heroku offers a number of simple keys to search to calculate different metrics. A full description of each metrics can be found at Heroku’s log-runtime-metrics article, but here’s a quick reference list:
|CPU load averages||Memory & swap|
memory_total represents the sum of resident, cache, and swap memory being used by the dyno, measured in megabytes.
log-runtime-metrics provides averages of CPU load at 1, 5 and 15 minute scales. These averages reflect the number of CPU tasks that are queued up. The dyno manager calculates this average by taking the count of runnable tasks for each dyno every 20 seconds.
For a full explanation of Heroku Lab’s new log-runtime-metrics, visit their article here.
To start analyzing your logs, create a free Logentries account in less than 60 seconds here.