Monitoring Docker Containers
with Splunk
Marc Chéné | Product Manager
Sept 27, 2017 | Washington, DC
Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
Who I am
Marc Chéné
• Product Manager, Engineer, APMer
• Dad/ super fan/ coach to 3, loves skiing, golfing,
music and a good drink
@marcchene
https://www.linkedin.com/in/marcchene
slack id: mchene
Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
Agenda
▶ Monitoring Options
▶ Analytical Insight – Tips & Tricks
▶ The World of Metrics
▶ (coming soon!) Docker Moby - v2 logging Plugin
Monitoring Options
logs, events and perf stats
Splunk and Docker – At A Glance
Visibility in your
Monitoring for your Delivering Splunk as
Container
Cloud Environments Containers
Environments
Splunk Logging Driver for Deep Visibility in Amazon Make getting Splunk as easy
Docker Web Services (AWS) and in as a single Docker pull
• Built into Docker – no EC2 Container Services command from the Docker
extra software required (ECS) Hub/Store
• Insight into container and
apps running in containers Splunk provides support for Forwarders and Splunk
Google Cloud Platform (GCP) Enterprise pre-configured to
Docker Universal Control collect machine data from
Plane Container Host and Docker
• Insight into administration, API
changes, and composition
Splunk Collection Options for Docker
• Docker Native Logging – Splunk logging driver, Syslog, JSON, AWS CloudWatch, etc.
• Forwarders – App Logs, Syslog UDP forwarding, Performance, etc.
• Logging libraries in .NET, Java and node.js
• Custom (e.g., Kafka with HTTP Event Collector)
• Cloud – AWS, GCP, Azure
Use the option that is right for you!
Log Streaming - Splunk Logging Driver for Docker
▶ Secure—supports TLS/SSL and
tokens
▶ Simple – config-based setup and
collect data
▶ Buffering
▶ Scale – Based on HTTP Data
Collector Based on Splunk HTTP
▶ Configurable - Supports
container labels, environment
variables
Log Streaming - Splunk Logging Driver for Docker
v1.13+
▶ Skip verification for the valid
splunk url
▶ Raw data collection from
the native log driver
▶ Embedded json format
support
▶ Performance Improvements
Docker Hub/Store
▶ Splunk container images available
• Splunk Enterprise 6.6.3
• Splunk Universal Forwarder 6.6.3
▶ Includes configuration and Docker Add-On for container
monitoring out-of-the-box
docker pull store/splunk/enterprise
docker pull store/splunk/universalforwarder:6.6.3
Deep Dive: What’s Do We Monitor?
▶ Docker Hub: https://hub.docker.com/r/splunk/universalforwarder/ tag: 6.5.3-
monitor
▶ GitHub: https://github.com/splunk/docker-itmonitoring
• Docker logs (ta-dockerlogs_fileinput) under “/host/containers/*/”
• [a-f0-9]+-json.log
• config.v2.json
• hostconfig.json
• hostname
• hosts
• resolv.conf
• Docker stats (ta-dockerstats)
• UCP logs (ta-ucplogs-sysloginput)
Demo Monitoring!
Analytical Insight –
Tips & Tricks
Analytical Insight – Tips & Tricks
▶ Sample Docker Compose file
▶ Correlations
• Docker SWARM mode
• Amazon Web Services (AWS)
▶ Log Options
• --log-opt tag="{{.Name}}/{{.FullID}}"
The World of Metrics
Terminology - What is a Measurement?
Treated natively as metrics, not log files
ABC.XYZ
Time Metric Name Measure Dimensions
Host (10.1.1.100,
system.cpu.idle numeric data point, web01.splunk.com)
different types such as Region (e.g., us-east-1, us-
count, gauge, timing, west-1, us-west-2, us-central1)
sample, etc IntanceTypes (e.g., t2.medium,
t2.large, m3.large)
© 2017 SPLUNK INC.
“Splunk provides ONE platform
to analyze and investigate
across both Logs and Metrics
Metrics Data Shape
Field Required Description
_time Y Microseconds since epoch
metric_name Y metric name
_value Y Value of the metric (numeric values only)
_dims Y Dimension names
host Y Origination Host
index Y Index to store the data
metric_type N Counter|Gauge – assume Gauge if not specified.
source N the source of the data
point, https://docs.splunk.com/Splexicon:Source
sourcetype Y Used for defining groupings of metrics and defining input time
rules
<fieldA>..<fieldZ> N Arbitrary number of dimensions
19
Key Features
SPL
Metric Store mstats Metrics Catalog
Ability to ingest and store tstats equivalent to REST APIs to query lists
metric measurements at query time series from of ingested metrics and
scale metrics indexes dimensions
Metrics Store
▶ Based on splunkd
▶ Dedicated Indexes for Metrics and Logs
▶ Full part of the platform
• RBAC
• Clustering
• Index Management
• Central Administration
▶ Optimized for fast time series queries and ingestion of metrics at scale
SPL: mstats
▶ mstats
• New SPL command
• Built off of tstats,
http://docs.splunk.com/Documentation/Splunk/6.6.1/SearchReference/Tstats
• Syntax
• | mstats <stats-fun>…
[WHERE index=<mymetricindex> metric_name=<metricname>…]
[BY <dimension-list> [span=<timespan>] ]
• Sample
• Stats:
| mstats avg(_value), count(_value)
WHERE metric_name="*.cpu.percent" by metric_name span=30s
• Time Series Visualization:
| mstats avg(_value), count(_value)
WHERE metric_name="*.cpu.percent" by metric_name span=30s
| timechart first(avg(_value)) as "avg" span=30s by metric_name
Metrics Catalog: Discovery & Search
▶ GET /services/catalog/metricstore/metrics • List all the dimension names that are
• List all metric names compatible with a given metric name
curl -k -u admin/pass "mem.free":
https://localhost:8089/services/catalog/metricstore/ curl -k -u admin/pass
metrics https://localhost:8089/services/catalog/metricstore/
dimensions?metric=mem.free
• List all metric names that apply to a given
dimension name "dc” • List all the dimension values for a given
curl -k -u admin/pass dimension name "dc”
https://localhost:8089/services/catalog/metricstore/ curl -k -u admin/pass
metrics?dimension=dc https://localhost:8089/services/catalog/metricstore/
dimensions/dc/values
▶ GET /services/catalog/metricstore/dimensions • List all the dimension values for a given
dimension name "dc" and metric name
• List all dimension names "mem.free”
curl -k -u admin/pass
curl -k -u admin/pass
https://localhost:8089/services/catalog/metricstore/
dimensions https://localhost:8089/services/catalog/metricstore/
dimensions/dc/values?metric=mem.free
23
GDI - Metric Ingestion Protocol: Collectd – Write
HTTP plugin
▶ Collectd, https://collectd.org - ~100 frontend plugins
▶ Scheduled push interval: 30secs
▶ # of metrics collected: ~350 (~1M measurements per day per server)
▶ Enabled plugins configurations, collectd.conf
1. csv 8. Logfile 15. tcpconns
2. cpu 9. memory 16. thermal
3. df 10. Network 17. ptime
4. disk 11. processes
5. Interface 12. protocols
6. irq 13. Syslog
7. load 14. swap
GDI: collectd write_http plugin
Sample write_http event
– {"values":[98.9363841194414],"dstypes":["derive"],"dsnames":["val
ue"],"time":1474401106.556,"interval":10.000,"host":"C5819124-
66AE-4B28-8E13-
914C3961E46C","plugin":"cpu","plugin_instance":"0","type":"cpu","
type_instance":"idle"}
Sample Result
– metric_name=cpu.idle.value
– _value=98.9363841194414
– Host=C5819124-66AE-4B28-8E13-914C3961E46C
GDI Deployment Options: Collectd & HEC
Cloud
HTTPS - HEC
Splunk CollectD Package
write_http
plugin
Server Farm
Splunk Indexing Tier
cAdvisor
▶ Provides container users an understanding of the resource usage and
performance characteristics of their running containers
▶ It is a running daemon that collects, aggregates, processes, and exports
information about running containers
27
DEMO Docker
Metrics!
Docker Moby - V2
Logging Plugin
Section subtitle goes here
Docker Moby - v2 logging Plugin
▶ Docker Hub: https://github.com/splunk/docker-logging-plugin
▶ Running the logging plugin
docker run --log-driver=splunk-log-driver:next \
--log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
--log-opt splunk-url=https://splunkhost:8088 \
--log-opt splunk-capath=/path/to/cert/cacert.pem \
--log-opt splunk-caname=SplunkServerDefaultCert \
--log-opt tag="{{.Name}}/{{.FullID}}" \
--log-opt labels=location \
--log-opt env=TEST \
--env "TEST=false" \
--label location=west \ your/application
Demo
© 2017 SPLUNK INC.
1. Docker Monitoring – You have options!
Key
Takeaways 2. Analytical Driven Insight
3. Metrics
4. Docker v2 logging API plugin
© 2017 SPLUNK INC.
Thank You
Don't forget to rate this session in the
.conf2017 mobile app
