Hutils – Explore your structured log data

Summary

Heroku’s internal components use logfmt to capture production activity, enabling retrospective request analysis, flexible trace queries, historical metrics, and incident investigation. The format can also become an unreadable 51-page wall of text, motivating hutils, a small suite of tools installable through Rubygems for exploring structured log data. Its ltap command extracts message streams from Splunk or Papertrail using a search query, while lviz parses messages into a tree that groups common key/value nodes and reduces duplicate data; the tools are designed to chain together. Additional commands include lcut for selecting fields and lfmt for colorized, whitespace-enhanced formatting, with both also supporting heroku logs --tail.

Takeaways

1. 1

   A standard build API fulfillment logfmt trace described in the post spans 51 pages, illustrating the format’s readability problem.

2. 2

   lviz groups the largest possible sets of common key/value nodes among descendant messages, pruning duplicate data in its terminal tree output.

3. 3

   ltap, lcut, and lfmt support workflows involving Splunk, Papertrail, and heroku logs --tail, and hutils tools can be chained together.