Heroku SignalFx Collector

A Golang Heroku app that collects metrics from apps and reports them to SignalFx. The Heroku SignalFx Collector collects the following metrics -

• Application metrics
• Heroku Dyno metrics
• Custom metrics

Note: Application and Heroku Dyno metrics will be prefixed by heroku. in SignalFx.

Deploy on Heroku

Deploy the SignalFx Collector on Heroku using:

Setup

At a high level, the collector runs as a HTTPS Log Drain app on Heroku. See here for more information.

Deploying Heroku SignalFx Collector

Clone this repo

git clone git@github.com:signalfx/heroku-signalfx-collector.git

Create a Heroku app

cd heroku-signalfx-collector/
heroku create

# Add Golang buildpack
heroku buildpacks:add heroku/go

Configure the collector

Use the following environment variables to configure the collector

Environment Variable Description Example SFX_TOKEN SignalFx access token of the org to which data needs to be sent (required) somevalidtoken SFX_INGEST_URL Ingest URL to which data needs to be sent (required if SFX_REALM is not set) https://ingest.us0.signalfx.com SFX_REALM SignalFx realm to which data is to be sent (required if SFX_INGEST_URL is not set) us0, us1, us2, eu0, ap0 SFX_METRICS_TO_EXCLUDE Comma separated metric names that the collector should not emit metric_name1,metric_name2,metric_name3 SFX_DIMENSION_PAIRS_TO_EXCLUDE Comma separated dimension key value pairs that the collector should not emit key1=val1,key2=val2 SFX_REPORTING_INTERVAL Reporting interval of the collector in seconds. Default value is 10 seconds 20 SFX_INTERNAL_METRICS Whether or not to report internal metrics (set to true by default) false

Configure Heroku App to send logs to SignalFx Collector

Enable Heroku log run-time metrics

heroku labs:enable log-runtime-metrics

See here for more information.

Configure Heroku App to expose Dyno metadata

heroku labs:enable runtime-dyno-metadata

This metadata is required by internal metrics to report accurate dimensions for fields like app_name and dyno_id.

See here for more information.

Follow the steps below so that an app sends logs to the collector

heroku drains:add "https://<SFX_COLLECTOR_APP_NAME>.herokuapp.com?app_name=<APP_NAME>"

Note: The above command expects SFX_COLLECTOR_APP_NAME to be the name of SignalFx Collector app and APP_NAME to be the Heroku app to monitored. Use heroku apps:info | grep '===' | cut -d' ' -f2 to set the app name. Passing in the app_name parameter is required. app_name is added as dimension to datapoints being by the collector.

To add custom dimensions to all datapoints being emitted by the collector, pass in query parameters while configuring the log drain.

heroku drains:add "https://<SFX_COLLECTOR_APP_NAME>.herokuapp.com?app_name=<APP_NAME>&dim1=key1&dim2=key2"

For example, configuring a log drain in the manner above will add dim1=key1 and dim2=key2 dimensions to all datapoints being emitted by the collector.

Deploy the collector to Heroku

git push heroku master
heroku ps:scale web=1

Sending in custom metrics

This app looks for logs of the following format. This is the standard format in which Heroku exposes default metrics. Custom metrics defined in the following format can be collected and emitted by the app.

\<(?P<pri>\d+)\>(?P<version>1) (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{6})?\+\d{2}:\d{2}) (?P<hostname>[a-z0-9\-\_\.]+) (?P<appname>[a-z0-9\.-]+) (?P<procid>[a-z0-9\-\_\.]+) (?P<msgid>\-) (?P<message>.*)$

IMPORTANT: All the fields in the above expect for the the last (message) field are generated by the Heroku platform. message field will contain application logs. This is where metrics and dimensions should be defined.

An example custom metric log that the collector can detect:

gauge#quota_used=20 cumulative#response_bytes=100 sfxdimension#service=backend sfxdimension#client=sfx_app

The above log message from an application would be interpreted as 2 datapoints, from quota_used (a gauge with value 20) and response_bytes (a cumulative counter with value 100) metrics, and both datapoints with service=backend and client_sfx_app dimensions.

Use the following keywords to (identify key-value pairs as metrics or dimensions) construct custom metrics

• gauge# - gauge type
• counter# - counter type
• cumulative# - cumulative counter type
• sfxdimension# - dimension

Internal Metrics

The collector reports internal metrics by default. Below is a list of internal metrics.

Metric Name Description sfx_heroku.total_drain_requests Number of drain requests received by the collector sfx_heroku.tracked_metrics Number of metrics collected per metric type. Metric types are determined by the dimension called type (i.e., cumulative_counter, counter, gauge).

Note: These metrics are collected by default and can be turned off by setting SFX_INTERNAL_METRICS to false.