Searching and Reporting with Splunk 4.
Document usage guidelines
Should be used only for enrolled students
Not meant to be a self-paced document
Not for distribution
07/12/2011
Operational Intelligence
Searching and Reporting with Splunk 4.2
Class Goals
Gain a deeper understanding of search and reporting concepts
Create efficient, well-formed searches
Perform calculations and evaluations on search results
Generate reports and charts
Correlate events with transactions
Create and use macros
Create and use lookups
Create and use summary indexes
Operational Intelligence
Searching and Reporting with Splunk 4.2
Course scenario
Examples used in this course are based on an online retail business
Searches and reports will be based on:
- Business analytics from the web access logs and lookups
- Internal operations information from mail and internal network data
Operational Intelligence
Searching and Reporting with Splunk 4.2
Data used in this course
Examples and labs in this course are based on a variety of data:
- Web application access logs
Events related to the online store web activity and sales
- Cisco mail logs
Events related to the internal email system
Operational Intelligence
Searching and Reporting with Splunk 4.2
Data used in this course (cont.)
- Cisco firewall logs
Events related to employee network usage
Youll become familiar with the fields and characteristics of these data
sources as the course progresses
Operational Intelligence
Searching and Reporting with Splunk 4.2
Course Outline
1. Search Fundamentals
2. Getting Statistics
3. Analyzing, Calculating, and Formatting
4. Creating Charts
5. Correlating Events
6. Enriching Data with Lookups
7. Summary Indexing
8. Creating and Using Macros
Operational Intelligence
Searching and Reporting with Splunk 4.2
Section 1:
Search Fundamentals
Operational Intelligence
Searching and Reporting with Splunk 4.2
Section objectives
Review basic search commands
Review general search practices
Examine the anatomy of a search
Understand search language syntax concepts
Review fields and use the fields command
Create a table
Extract fields
Examine multi-value fields
Operational Intelligence
Searching and Reporting with Splunk 4.2
Basic search review
Keywords
search for error!
Phrases
web error (different than web AND error)
Fields
search for status=404
Booleans
OR, NOT, ()s. ANDs are implied
Wildcards
status=40*, matching 401, etc.
Comparisons
=,!=,<,<=,>=,> (delay > 10)!
Operational Intelligence
10
Searching and Reporting with Splunk 4.2
General search practices
Time is the most efficient filter in Splunk narrowing by time is the most
effective thing you can do
The more you tell the search engine, the better shot you have at a good
result
- When applicable, searching for access
denied is always better than
searching for denied
If 90% of your events include the word 'error' but only 5% include the
word 'sshd', and your events have both of these words, including 'sshd' in
your search terms will help more than 'error alone
Operational Intelligence
11
Searching and Reporting with Splunk 4.2
General search practices (cont.)
Inclusion is generally better than exclusion
- Searching for "access
denied" will be faster than NOT "access
granted" !
Apply powerful filtering commands as early in your search as possible
- Filtering to one million events and then narrowing to ten events is much slower
than filtering to one thousand events and then ten events
Operational Intelligence
12
Searching and Reporting with Splunk 4.2
Anatomy of a Search
Disk
Intermediate
results table
Intermediate
results table
Final results
table
sourcetype=syslog ERROR | top user | fields - percent
Fetch events
from disk that
match
Operational Intelligence
Summarize
into table of
top 10 users
13
Remove
column
showing
percentage
Searching and Reporting with Splunk 4.2
Search language syntax concepts
Searches are made up of 5 basic components
Search terms what are we looking for?
- keywords, phrases, booleans, etc.
Commands what should we do with the results?
- Create a chart, compute statistics, evaluate and format, etc.
Functions how should we chart, compute or evaluate?
- Get a sum, get an average, transform the values, etc.
Arguments Are there variables we should apply to this function?
- Get an average value for a specific field, convert milliseconds to seconds, etc.
Clauses How should we group the results?
- Get the average of values for the price field grouped by product, etc.
Operational Intelligence
14
Searching and Reporting with Splunk 4.2
Search language example
This diagram represents a search, broken into the syntax components
Search for this
PIPE:
Take these events and
COMMAND:
Get some stats
PIPE:
Take these stats and
COMMAND:
Set values for the lost_revenue field
FUNCTION:
Get a sum
FUNCTION:
Create a string
ARGUMENT:
Get a sum of the price field
CLAUSE:
Call that sum lost_revenue
Operational Intelligence
15
ARGUMENT:
Format the string from values in the
lost_revenue field, insert commas
Searching and Reporting with Splunk 4.2
Fields review
Splunk automatically extracts important fields in these events
A few are highlighted below (there are many more!)
web access log
clientip
product_id
method
mail log
status
MID
mailfrom
network
proxy log
Operational Intelligence
cs_username
s_hostname
16
Searching and Reporting with Splunk 4.2
Fields command overview
Field extraction is one of the most costly parts of search
The fields command allows you to include or exclude specific fields in your
search or report
- Makes searching more efficient when traversing large amounts of data with
many fields
- Makes table-display cleaner by allowing you to remove unnecessary fields
Operational Intelligence
17
Searching and Reporting with Splunk 4.2
Include specific fields
Improves performance only the
fields you specify are extracted
Internal fields _raw and _time
are always included in output
- The fields command does not
remove internal fields unless
explicitly specified with the remove
fields command
fields - <field name>
Operational Intelligence
18
Searching and Reporting with Splunk 4.2
Remove specific fields
Use the remove fields
command to exclude specific
fields
As youll discover with
statistics and reporting
commands, removing fields
can be useful for display
purposes
Operational Intelligence
19
Searching and Reporting with Splunk 4.2
Create a table
The table command returns a
table formed by only the fields in
the arguments
Columns are displayed in the
same order of fields entered in the
command
- Column headers are field names
- Rows are field values
- Each row represents an event
Operational Intelligence
20
Searching and Reporting with Splunk 4.2
Rename fields
Use the rename command to
rename fields
Useful for giving fields more
meaningful names
Use quotes to rename to a phrase
- |
rename JSESSIONID as session ID!
Can be applied to wildcards to
rename multiple fields
- |
rename *ip as IP Address!
Operational Intelligence
21
Searching and Reporting with Splunk 4.2
Extract fields at search time
There may be times when a field you need for a particular search has not
been extracted
The rex command allows you to extract the field at search time
- Only persists for the duration of the search
- Good for rarely-used fields
- Save the search and use it as-needed, but the extraction wont persist in ALL
your searches
You can also use the rex command to create permanent extractions via
Splunk> Manager >> Fields >> Extractions
Learn more about the regex format and using
the rex command in [Link]/base/
Documentation
Operational Intelligence
22
Searching and Reporting with Splunk 4.2
Extract fields with rex
The network proxy data includes a user name for each event, but has not
been defined as a field
The rex command allows you to extract the field at search time
sourcetype="cisco_w*" | rex "\s+(?<username>[^-@]\w+)@" | top username!
!
For example purposes, the user name field has
been extracted as cs_username for this course
Operational Intelligence
23
Searching and Reporting with Splunk 4.2
Extract fields from a table-formatted event
Many data types are formatted as large single events, formatted as a table
Each event contains fields with multiple values
- Here, the first row represents the fields, all other rows represent values
Operational Intelligence
24
Searching and Reporting with Splunk 4.2
Extract fields from a table-formatted event (cont.)
The multikv command extracts
the fields you specify
- Field names are from the first row of
each event as displayed on
previous slide
The command creates a separate
event for each row
- This example also pipes to the
table command for better display
Operational Intelligence
25
Searching and Reporting with Splunk 4.2
Combine multi-valued fields
The mvcombine command combines events that have a single differing
field value into one result with a multi-value field of the differing field
- For each group of results that are identical except for the given field, combine
them into a single result where the given field is a multivalue field
-delim controls how values are combined
There are many commands for use with
multi-valued fields. The complete list can
be found at
[Link]/base/documentation
Operational Intelligence
26
Searching and Reporting with Splunk 4.2
Lab working with fields
To become familiar with the data used in this course, examine all three
sources of data
In the web access logs, search for all events of each data type
Create tables that includes specific fields
Challenge: In the mail logs, use the rex command to extract a new field
Operational Intelligence
27
Searching and Reporting with Splunk 4.2
Section 2:
Getting statistics
Operational Intelligence
28
Searching and Reporting with Splunk 4.2
Section Objectives
Understand the stats command
Preview reporting and charting commands
Display top and rare values for given fields
Use the stats command
Operational Intelligence
29
Searching and Reporting with Splunk 4.2
Reporting commands overview
The following basic reporting commands are covered in this section:
top display the most common values of a field
rare display the least common values of a field
stats calculate statistics on your events
eventstats attach summary statistics to each event
Operational Intelligence
30
Searching and Reporting with Splunk 4.2
Scenario callouts
Most of the examples for the remainder of this course relate to a specific
scenario
For each example, a question is posed that might be asked by a colleague
or manager in the company
- The answers live in the Splunk data!
- Example:
Who are the top site visitors?
Operational Intelligence
31
Searching and Reporting with Splunk 4.2
Getting top values
Who are the top site visitors?
The top command finds the
most common values of a given
field
- Returns top 10 results by default
Output is in table format
Automatically returns a count
and percentage
Adding limit=# after the top
command returns the specified
number of results
Operational Intelligence
32
Searching and Reporting with Splunk 4.2
Getting top values (cont.)
This example displays the top
10 from addresses found in
company email
Are we getting an unusual amount of mail from a few senders?
Clicking Show Report
allows quick creation of a
report suitable for a
dashboard
Operational Intelligence
33
Searching and Reporting with Splunk 4.2
Getting top values (cont.)
Using the fields command, you can remove the percent field from the
results
What are the top 5 sites being accessed through the firewall?
Operational Intelligence
34
Searching and Reporting with Splunk 4.2
Getting top values (cont.)
Adding the by clause to the top
command, we can view the top
sites and their associated
acceptable use category
What are the usage categories of the top sites?
sort -count ensures the table
is sorted by count in descending
order
Note: the usage field comes from an
automatic lookup, which is covered later in
this course.
Operational Intelligence
35
Searching and Reporting with Splunk 4.2
Getting rare values
The rare command returns the
least common field values of a
given field
What are the least frequently visited sites?
Options are identical to the top
command
Operational Intelligence
36
Searching and Reporting with Splunk 4.2
Getting statistics
The stats command allows you to get statistics on your data
You can apply different functions to the stats command, depending on the
type of statistics youre trying to achieve
Common functions include:
- count
returns the number of occurrences of a given field within the search
results!
- distinct_count, dc returns a count of unique values for a given field
- sum returns a sum of numeric values
- avg returns an average of numeric values
- list lists all values of a given field
- values lists unique values of a given field
Operational Intelligence
37
Searching and Reporting with Splunk 4.2
stats count
count returns the number of
occurrences of a given field
How many of each product was purchased?
The by clause returns a count
for each field value of a named
field
This example counts the
number of times each value of
product_id appears in the
events when the action field
contains purchase!
Operational Intelligence
38
Searching and Reporting with Splunk 4.2
stats count(field)
Adding a field as an argument to
the count function returns the
number of occurrences for that
field
How many sites of each usage type are accessed?
In this example, we count how
many websites (s_hostname)
that were accessed fell into each
usage category
Operational Intelligence
39
Searching and Reporting with Splunk 4.2
stats distinct count
distinct_count() or
dc() provides a count
of how many unique
values there are for a
given field
How many unique visitors to the online store?
This example counts how
many unique values exist
for clientip
Operational Intelligence
40
Searching and Reporting with Splunk 4.2
stats sum(field)
For fields with a numeric value,
you can sum the actual values of
that field
Whats the total time for all web requests per site visitor?
In this example, for each user web
session, we get a sum of the
values of the time_taken field
Note: the time_taken field values are in
milliseconds. Well learn how to convert to
seconds later in this course
Operational Intelligence
41
Searching and Reporting with Splunk 4.2
stats avg(field)
The avg function averages numeric values of a named field
This example averages all the values of the time_taken field, grouped by
the action field
Whats the average time it takes the application to perform an update vs purchase?
Operational Intelligence
42
Searching and Reporting with Splunk 4.2
addcoltotals
The addcoltotals
command computes the
sum of all numeric values for
a given field
How many items are being removed from carts before purchase?
- Adds the total to the bottom of
the column
Operational Intelligence
43
Searching and Reporting with Splunk 4.2
addtotals
The addtotals commands gets the
sum of numeric fields for each event
- Define a field name in which to place
the total value
- Optionally, specify only certain fields to
include in the sum
- In this example, the addtotals
command is applied to the rKB_PS
and wKB_PS fields
- By default, the command computes a
sum for all numeric fields in the event
Operational Intelligence
44
Searching and Reporting with Splunk 4.2
stats list(field)
What websites are the users accessing?
The list function lists all field
values for a given field
This example lists the websites
each user visits
- Since the security logs generate an
event for each network request, the
same hostname will appear multiple
times
- If you want a list of unique field
values, the list function isnt ideal
Operational Intelligence
45
Searching and Reporting with Splunk 4.2
stats values(field)
The values function creates a list
for each unique field value for a
given field
What websites are the users accessing?
Operational Intelligence
46
Searching and Reporting with Splunk 4.2
Getting summary stats for all events
The eventstats command provides summary statistics on all events in
the search results
Makes the statistics available as a field on each event
You can then run additional stats, calculations, or reports on the field
Operational Intelligence
47
Searching and Reporting with Splunk 4.2
eventstats
In this example, we get sum of the price field
for each product ID when a customer
removes the item from the cart
- Set a new field for that value lostSales!
Whats our revenue loss due to removal from
shopping cart?
Piping to eventstats, we create a new
field averageLosses!
- averageLosses contains the average of
the lostSales field value across ALL
events
- averageLosses value is added to each
event
Now we can add more commands to further
use the averageLosses field
Operational Intelligence
48
Searching and Reporting with Splunk 4.2
eventstats (cont.)
Whats our revenue loss due to removal from
shopping cart?
Now that we have a summary
value on each row, we can apply
further calculations across the data
- We use the eval command to
create a new field difference!
- Use the values of the
averageLosses and
lostSales fields to calculate the
difference
The eval command is covered
later in this course
Operational Intelligence
49
Searching and Reporting with Splunk 4.2
Create a chart
Running this in the Advanced Charting view creates a chart of the report
- Notice that all three statistics are included; lostSales, averageLosses, and
difference!
- Sales only wants to know how each product compares against an average
Operational Intelligence
50
Searching and Reporting with Splunk 4.2
Remove unwanted fields
Once all statistics are calculated,
you can remove fields from the
chart that are no longer needed
In this example we remove the
lostSales and
averageLosses fields
- This makes the chart cleaner by
only displaying the stats important
to you
Charting options are covered
later in this course
Operational Intelligence
51
Searching and Reporting with Splunk 4.2
Lab
Report on top and rare values
Use the stats command and associated functions
Use the eventstats command
Operational Intelligence
52
Searching and Reporting with Splunk 4.2
Section 3:
Formatting / Calculating
Operational Intelligence
53
Searching and Reporting with Splunk 4.2
Section Objectives
Understand the eval command
Perform calculations on values with eval!
Convert values
Round values
Format values
Use conditional statements
Further filter calculated results
Operational Intelligence
54
Searching and Reporting with Splunk 4.2
Eval command overview
eval allows you to calculate and manipulate field values
- Does not re-write event data
Supports a variety of functions
Results of eval are written to a specified field
- can be a new or existing field
- If the destination field exists, the values of the field are replaced by the results of
eval
Operational Intelligence
55
Searching and Reporting with Splunk 4.2
Convert values with eval
In this example, the report displays the average time it takes for the server
to process purchase and update http request actions
In the raw event, the time_taken field is stored in milliseconds
- Once the average is calculated, there are several decimal points as well
First, well use eval to convert the time_taken value into seconds
Operational Intelligence
56
Searching and Reporting with Splunk 4.2
Convert values with eval (cont.)
The results of eval must always be set to a new or existing field
In this example:
- Set a new field reqTime with the value we wish to chart and convert
- Indicate which field the eval results should populate
- Divide the reqTime field value by 1000 to
convert milliseconds to seconds
Whats the average amount of time it takes to perform an update vs purchase?
Set new field
Define a field to set the eval results to
Divide the field value by 1000
Operational Intelligence
57
Searching and Reporting with Splunk 4.2
Round values
But theres still too many pesky decimal points!
The round(field or number, decimals) function sets the value of a field to
the number of decimals you specify
- In this example, we divide the value of the reqTime field by 1000 then round to
2 decimal points.
Whats the average amount of time it takes to perform an update vs purchase?
Divide the value of reqTime by 1000, then round to 2 decimal points
Operational Intelligence
58
Searching and Reporting with Splunk 4.2
Compare values with eval
How are specific products performing against
the average?
You can perform mathematical
functions against fields with
numeric field values
This example compares the
average price across all events
when action=purchase
against the actual price in a given
event
- Subtract the value of price from
avgPrice!
Operational Intelligence
59
Searching and Reporting with Splunk 4.2
Format values with eval
eval can also be used to format values for a more meaningful display
Consider this search, that determines the lost revenue in the last 24 hours,
based on events that have a 503 status and the sum of the price field
values
How much money are we losing due to these server errors?
The result is somewhat ambiguous, especially if it will ultimately live on a
dashboard
Operational Intelligence
60
Searching and Reporting with Splunk 4.2
Format values with eval (cont.)
The tostring function can convert a numeric field value to a string
In this example, a $ is prepended to the value
Using commas as an argument, the function automatically applies commas
appropriately
How much money are we losing due to these server errors?
Apply to lost_revenue field
Prepend with a dollar sign
Convert the value of lost_reveue to a string
Auto-apply commas appropriately
Also works with boolean values , returning true or false
Operational Intelligence
61
Searching and Reporting with Splunk 4.2
tostring considerations
Once youve converted a numeric value to a string, the resulting value is no
longer considered numeric
- Sorting columns of numbers and timestamps may not work
- Automatic drilldown will not work with post-reporting conversions on charts
More information on formatting using the tostring
function can be found at [Link]/base/
documentation
Operational Intelligence
62
Searching and Reporting with Splunk 4.2
Using multiple eval commands
Remember that each new command
references the results of the previous
commands
In this example we:
- Set the sales field with total revenue for
each product ID
- Set a new field USD with the value of sales
and prepend with $
- Set a new field, GBP, with the exchange rate
(value of sales field * exchange rate)
- Prepend the GBP field with
- Remove the sales field from the final output
Operational Intelligence
63
Searching and Reporting with Splunk 4.2
Conditional statements
The if function takes three arguments
- The first argument is a Boolean expression
- If it evaluates to TRUE, the result is the second argument
- If it evaluates to FALSE, the result evaluates to the third argument
Otherwise, group everything else into a value called Other
If the value of the usage field is Business
Apply the results to a field called Business Usage
How much network bandwidth is going
toward non-business activity?
Operational Intelligence
64
Searching and Reporting with Splunk 4.2
Where command
Runs an eval expression to filter the results
- The result of the expression must be Boolean
- Keeps only the results for which the evaluation was successful and the Boolean
result was true
Useful in further filtering a search
Operational Intelligence
65
Searching and Reporting with Splunk 4.2
Where command example
This example counts the
occurrences of each hostname,
then only returns a result when the
count exceeds 50
Operational Intelligence
What sites were visited more than 50 times?
66
Searching and Reporting with Splunk 4.2
Lab
Convert field values
Round field values
Compare field values
Format field values
Use conditional statements
Filter results with the where command
Operational Intelligence
67
Searching and Reporting with Splunk 4.2
Section 4:
Charting
Operational Intelligence
68
Searching and Reporting with Splunk 4.2
Section Objectives
Identify chart types
Identify the chart command
Create a basic chart
Split values into multiple series
Define stacked mode
Omit null and other values from charts
Create a timechart
Chart multiple values on the same timeline
Format charts
Apply statistical functions
Group data with buckets
Create a rangemap
Operational Intelligence
69
Searching and Reporting with Splunk 4.2
Chart types
Column and bar charts
- Compare the frequency of field
values
Stacked column and bar charts
- All columns are segments of a
single column
Line chart
- Show trends over time or another
field
Operational Intelligence
70
Searching and Reporting with Splunk 4.2
Chart types (cont.)
Area chart
- Show trends either over time or in
comparison to another field value
Stacked area chart
- Show multiple series among the
trends in your data
Pie chart
- Show the relationship of parts of
your data to the entire set of data as
a whole
Operational Intelligence
71
Searching and Reporting with Splunk 4.2
Chart command overview
The chart command can display any series of data that you want to plot
You decide what field is tracked on the x-axis
- Where stats uses the by clause to group data, chart uses by or over to
determine which field takes the x-axis!
chart avg(bytes) over host by date_wday the host field is the x-axis and
the series is split by day of week
chart avg(bytes) by host the host field is the x-axis since theres no split for
the series
- Because the chart command is designed to return chartable results, the value of
the y-axis should always be numeric
Operational Intelligence
72
Searching and Reporting with Splunk 4.2
Basic chart example column
Note the following chart examples are created in
Advanced Charting view. Views > Advanced Charting
This example shows a basic chart
The count function counts the number of events for each product_name!
How many of each product are we selling?
Operational Intelligence
73
Searching and Reporting with Splunk 4.2
Chart split by
If you want to split the series, identify the x-axis field with over, then use by
to split the series by additional fields
In this example, we split the series by host!
Which hosts are the most active?
Operational Intelligence
74
Searching and Reporting with Splunk 4.2
Chart stacked mode
In stacked mode, all split-by values are displayed in a single column
Operational Intelligence
75
Searching and Reporting with Splunk 4.2
Chart stacked mode
Hovering over a segment in the column displays the statistic for that segment
Operational Intelligence
76
Searching and Reporting with Splunk 4.2
Omitting null and other
This example charts the file types that use a lot of bandwidth
Notice how the - and OTHER values skew the results
What file types use the most bandwidth?
Operational Intelligence
77
Searching and Reporting with Splunk 4.2
Omitting null and other (cont.)
Removing the - value in the initial search, then adding the options
useother=f and usenull=f removes the empty and other field values
from the results
What file types use the most bandwidth?
Operational Intelligence
78
Searching and Reporting with Splunk 4.2
Timechart
Time charts perform statistical aggregations against time
Trend behaviors and find anomalies over time
_time is always the x axis
You can optionally split data by another field
- each distinct value of the "split by" field is a separate series in the chart
Timecharts should typically be line or area charts
Operational Intelligence
79
Searching and Reporting with Splunk 4.2
Basic timechart
This basic timechart displays the number of usage violations over the last
24 hours
Note the same functions and arguments youve used with stats and chart
apply
How many usage violations in
the last 24 hours?
Operational Intelligence
80
Searching and Reporting with Splunk 4.2
Charting multiple values
This example displays the usage categories over a 1 hour period
Splitting by the usage field, each line represents a unique value of the field
The y-axis represents the count for each field value
Whats the overall usage trend
for the last 24 hours?
Operational Intelligence
81
Searching and Reporting with Splunk 4.2
Formatting options
With the multi-series mode set to combined, all fields share the y-axis
Operational Intelligence
82
Searching and Reporting with Splunk 4.2
Formatting options (cont.)
Setting the Multi-series mode to Split causes the y-axis to split for each field
value
The y-axis is divided into sections, each spanning the max and min count
Operational Intelligence
83
Searching and Reporting with Splunk 4.2
Applying statistical functions
As with the stats and chart commands, statistical functions can be
applied to the timechart command
How much revenue are we
getting for each product over
the last hour?
Operational Intelligence
84
Searching and Reporting with Splunk 4.2
Handling null values omit
Three options on handling null values in a timechart Omit, Connect, Treat
as Zero
The omit option displays gaps in the series
Operational Intelligence
85
Searching and Reporting with Splunk 4.2
Handling null values Connect
The connect option fills the gaps with a trendline
Operational Intelligence
86
Searching and Reporting with Splunk 4.2
Handling null values treat as zero
The treat as zero option flattens the trendline to the zero value
Operational Intelligence
87
Searching and Reporting with Splunk 4.2
Bucket
The bucket command
puts continuous numerical
values in fields into
discrete sets, or buckets
This example displays
search results over a 7
day time span, but
buckets the results into 1
day chunks
- The bucket command is
applied to the _time field
Operational Intelligence
88
Searching and Reporting with Splunk 4.2
Bucket (cont.)
This example displays web
requests that result in an http
status other than 200, grouped by
host.
The bucket command is applied to
the count field, and grouped into
intervals of 50.
Operational Intelligence
89
Searching and Reporting with Splunk 4.2
Rangemap
Rangemaps are a single-value
report useful on dashboards
To create a rangemap
- Craft and save a search using the
rangemap command
- Add the saved search to a
dashboard panel
Operational Intelligence
90
Searching and Reporting with Splunk 4.2
Rangemap (cont.)
Apply the rangemap to the
lostrevenue field
The rangemap command creates
a single-value result for a numeric
field value
Indicate a field to report on
Define conditions for the
rangemap display
- low displays green
- elevated displays yellow
What range of values are
noteable?
What range of values are
acceptable?
All other values, bad!
- severe displays red
Operational Intelligence
91
Searching and Reporting with Splunk 4.2
Rangemap (cont.)
Once you create and save the
search, create a panel for a
dashboard
Todays Lost Revenue
- Select Single Value for the panel
type
- Select the Saved search
- Add the panel
Operational Intelligence
92
Searching and Reporting with Splunk 4.2
Rangemap (cont.)
To display color based on your
specified ranges, you must add
the classField option to the
dashboard XML
- In the edit panel, click Edit name/
XML
The underlying XML displays
- Add
<option name="classField">range</option>
directly above the closing </single>
tag.
Operational Intelligence
93
Searching and Reporting with Splunk 4.2
Lab
Create a basic column chart
Create a multi-series chart
Work with formatting options
Create a timechart
Operational Intelligence
94
Searching and Reporting with Splunk 4.2
Section 5:
Correlating Events
Operational Intelligence
95
Searching and Reporting with Splunk 4.2
Section objectives
Identify transactions
Group events using fields
Group events using fields and time
Search with transactions
Report on transactions
Determine when to use transactions vs stats
Operational Intelligence
96
Searching and Reporting with Splunk 4.2
Transactions overview
A transaction is any group of conceptually related events that spans time
The events can come from multiple applications or hosts
- Events related to a single purchase from an online store can span across an
application server, database, e-commerce engine
- A single email message can create multiple events as it travels through various
queues
Operational Intelligence
97
Searching and Reporting with Splunk 4.2
Basic transaction example
Each event in the network web
traffic logs represents a single user
generating a single http request.
Visiting a single website normally
generates multiple http requests
- Html file(s)
- Images
- Javascript
- Css
-
Operational Intelligence
98
Searching and Reporting with Splunk 4.2
Group by fields
Using the transaction
command, you can group and
create a single event when
certain fields have the same
value
You can create transactions
across multiple tiers (i.e., web
server, application server,
media server) provided a
common field is shared
(accountID, sessionID)
Operational Intelligence
99
Searching and Reporting with Splunk 4.2
Group by fields (cont.)
In this example we group by the
following fields:
- s_hostname
[Link]
- cs_username the username
associated with the request
Operational Intelligence
100
Searching and Reporting with Splunk 4.2
Group by fields and time (cont.)
We also Indicate the maximum
time span and pause between
individual events
- maxspan=20m!
The maximum total time between the
earliest and latest events
If not specified, default is -1 (or no limit)
- maxpause=5m!
The maximum total time between
events
If not specified, default is -1 (or no limit)
Operational Intelligence
101
Searching and Reporting with Splunk 4.2
Examine a transaction
The example below is one of the transactions created by the search
The transaction
contains 7
individual events
Each event
shares the same
host and
username
Operational Intelligence
102
Searching and Reporting with Splunk 4.2
Group by startswith / endswith
You can use the startswith and endswith options to form transactions
based on terms, field values, or evaluations
In this example, the first event in the transaction includes *signon* and
the last event includes purchase
Operational Intelligence
103
Searching and Reporting with Splunk 4.2
Transaction-specific fields
The transaction command produces two additional fields, duration and
eventcount.
- The duration value is the difference between the timestamps for the first and last
event in the transaction.
- The eventcount value is the number of events in the transaction.
Operational Intelligence
104
Searching and Reporting with Splunk 4.2
Searching with transactions
Transactions can be useful when
a single event may not provide
enough information
This example searches for the
term REJECT in the email logs.
- The events that include the term
dont provide much information
about the rejection
Operational Intelligence
105
Searching and Reporting with Splunk 4.2
Searching with transactions (cont.)
By creating a transaction we
can then search and see
additional events related to
the rejection
- IP address of sender
- Reverse dns lookup results
- Action taken by the mail
system following the rejection
Operational Intelligence
106
Searching and Reporting with Splunk 4.2
Reporting on transactions
You can use the same statistics and reporting commands with transactions
This example takes advantage of the duration field, showing a trend of
the mail queue slowing down over 4 hours, then beginning to correct
- Adding events to the transaction from additional hosts or sources would
uncover the cause of the slowdown
Operational Intelligence
107
Searching and Reporting with Splunk 4.2
Transaction vs Stats
Use transaction when you need to see the events correlated together
Use stats when you just want to see the results of a calculation
Use transaction when you must define event grouping based on start/
end values
Use stats when you can group events based on a field value (eg by
src_ip)
When you can do it either way, choose stats because it is more efficient
- sourcetype=trade_entries
| transaction TradeID | table TradeID, eventcount
vs.
sourcetype=trade_entries | stats count by TradeID!
Operational Intelligence
108
Searching and Reporting with Splunk 4.2
Lab
Correlate events with transactions
Search and report on transactions
Operational Intelligence
109
Searching and Reporting with Splunk 4.2
Section 6:
Enriching Data with Lookups
Operational Intelligence
110
Searching and Reporting with Splunk 4.2
Section Objectives
Discover lookups
Examine a lookup file example
Create a lookup table
Define a lookup
Configure an automatic lookup
Use the lookup in searches and reports
Operational Intelligence
111
Searching and Reporting with Splunk 4.2
Lookups
Some data should not be indexed within Splunk
- Static or relatively unchanging data
Lookups allow you to add additional fields to your events:
- http status code descriptions (file not found, service unavailable)
- Descriptive text descriptions for errors or item IDs
- User names based on asset tags or static IP addresses
Two types of lookups
- File lookup
- Scripted lookup
Operational Intelligence
112
Searching and Reporting with Splunk 4.2
Defining a file lookup
1. Create the lookup table
2. Define the lookup
3. Configure the lookup to run
automatically
Operational Intelligence
113
Searching and Reporting with Splunk 4.2
Lookup file example
This example displays a lookup .csv
file used to associate a product name,
price, and competitors prices with an ID
The first row represents field
names (header)
The product_id field exists in the
access_combined events
- This is the input field
product_name, price,
flowermart, and flowersRUs fields
will be available to search once the
lookup is defined
- These are the output fields
Operational Intelligence
114
Searching and Reporting with Splunk 4.2
Create a lookup table
The first step in creating a
lookup is creating the lookup
table
Manager >> Lookups >>
Lookup table files lists all
existing lookup tables
- From this list you can edit,
change permissions, enable/
disable, and delete
Operational Intelligence
115
Searching and Reporting with Splunk 4.2
Create a lookup table (cont.)
1. Click new to display the
Add New form
2. Select a destination app
3. Browse and select the
.csv file to use for the
lookup table
4. Enter a name for the
lookup file
5. Save
Operational Intelligence
116
Searching and Reporting with Splunk 4.2
Define the lookup
Manager >> Lookups >>
Lookup definitions lists all
existing lookup definitions
- From this list you can edit,
change permissions,
enable/disable, delete and
clone
Operational Intelligence
117
Searching and Reporting with Splunk 4.2
Define the lookup (cont.)
1. Click new to display the
Add New form
2. Select a destination app
3. Enter a name for the lookup
definition
4. Select the .csv file to use
with the definition
Operational Intelligence
118
Searching and Reporting with Splunk 4.2
Time-based lookups
lookups can also be time-based if
a field in the lookup table that
represents a timestamp
- Specify the name of the time field
You can also specify a strptime
format and offsets for the time
matching
Operational Intelligence
119
Searching and Reporting with Splunk 4.2
Advanced Options
Under Advanced options, you
can specify:
- The minimum number of matches
for each input lookup value
- The maximum number of matches
for each input lookup value
- A default value to output if fewer
than the minimum number of
matches are present for a given
input
Operational Intelligence
120
Searching and Reporting with Splunk 4.2
Using a non-automatic lookup
If a lookup is not configured to run automatically, use the lookup command
in your search to use the lookup fields
Format is
| lookup <lookup_name> <input field> OUTPUT <output field>, <output field>!
This example calls a lookup that includes fields for a browser and os based
on the value of the useragent field
Operational Intelligence
121
Searching and Reporting with Splunk 4.2
Configure automatic lookup
Manager >> Lookups >>
Automatic lookups lists all
existing automatic lookup
configurations
- From this list you can edit
change permissions, enable/
disable, and clone
Operational Intelligence
122
Searching and Reporting with Splunk 4.2
Configure automatic lookup (cont.)
1. Select the destination app
2. Enter a name
3. Select the lookup table
prod_lookup
4. Choose a sourcetype, source or
host, and identify by name
Operational Intelligence
123
Searching and Reporting with Splunk 4.2
Configure automatic lookup (cont.)
5. Define the Lookup input fields
-
The field that exists in your events
that you are relating to the lookup
table
6. Define the Lookup output fields
-
-
The fields from your lookup table
that will be added to the events
Optionally, give the fields an
alternate name by typing a new
name after the = sign
Operational Intelligence
124
Searching and Reporting with Splunk 4.2
Using the automatic lookup
product_id field in event
To use an automatic lookup,
simply include the fields in
your search
Resulting report
Operational Intelligence
Fields from lookup in search
125
Searching and Reporting with Splunk 4.2
More lookup options
In addition to creating and using a file-based lookup you can also:
- Populate a lookup table with search results
- Set up a fields lookup based on an external command or script
- Set up a fields lookup based on an external database
More information on creating and configuring
lookups can be found at [Link]/base/
documentation
Operational Intelligence
126
Searching and Reporting with Splunk 4.2
Lab
Upload a lookup file
Create a lookup table
Use the lookup in a search
Configure the lookup to run automatically
Use the automatic lookup in a search
Operational Intelligence
127
Searching and Reporting with Splunk 4.2
Section 7:
Summary Indexing
Operational Intelligence
128
Searching and Reporting with Splunk 4.2
Section objectives
Define summary indexing
Create and schedule a summary search
Populate a summary index
Run searches against a summary index
Identify gaps and overlaps in the summary index
Correct gaps and overlaps in the summary index
Operational Intelligence
129
Searching and Reporting with Splunk 4.2
Summary Indexing Overview
Efficiently report on large volumes of data
Spread the cost of a computationally expensive report over time
Common use cases include:
- Run reports over long time ranges for large datasets more efficiently
Example: Show the number of page views and visitors each of your Web sites had over
the past 30 days, broken out by site
- Build a rolling report that shows aggregated statistics over long period of time
Example: A running count of downloads for a specific file on a website
Example: Calculate the average amount spent per purchase over a year.
Operational Intelligence
130
Searching and Reporting with Splunk 4.2
Without summary indexing.
In this example, were running a search for the top videos requested from a
website in the last 30 days
The site gets millions of hits per day
Searching the default index for a 30-day time span is not efficient as it scans
across the entire data set for the specified 30 day period
index
Operational Intelligence
131
Searching and Reporting with Splunk 4.2
With summary indexing
The pre-calculated statistics you want to report on is incrementally added to a
summary index via a saved, scheduled search
Data is stored in a special table format that optimizes efficiency.
- Each time the saved search runs, it appends the data in the summary index.
Searches or reports that may take several minutes or more to complete can be
generated quickly
Operational Intelligence
132
Searching and Reporting with Splunk 4.2
Summary indexing steps
1. Create a saved search using the si prefix that extracts broad statistics
you want to report on.
2. Schedule the search to run periodically over an appropriate time interval
(hourly, daily, every n-miuntes depending on your needs.
Each time the search runs, it saves the results since the last run into a summary
index that you designate.
3. Run searches and reports on this smaller, "faster" summary index
instead of working with the much larger dataset.
Operational Intelligence
133
Searching and Reporting with Splunk 4.2
Summary indexing flow
3.
sourcetype=access_combined | sistats count by file!
Run every hour, time range 1 hour
1.
2.
index
Operational Intelligence
Populating search
results are
formatted and
stored as statistics
134
Summary
index
Searching and Reporting with Splunk 4.2
1. Creating the summary search
Dont pipe other search operators
after the main summary indexing
reporting command
- Save that for the searches you run
against the summary indexes
Create a broad enough search to
capture the data you want
Use the proper reporting
command to get the statistics you
want for further processing later.
Operational Intelligence
135
Searching and Reporting with Splunk 4.2
1. Creating the summary search (cont.)
Make the time span of the search
smaller than the time span of the
reports you want to run against it
- To report on a month or a year,
make the summary search for -24h
hours, and run every day
- For a day -1h and run every hour.
Operational Intelligence
136
Searching and Reporting with Splunk 4.2
2. Schedule and save the search
If the time range you defined is 1
hour, schedule to run every hour.
- For 1 day, schedule 1 day, etc
Make sure to enable summary
indexing
Select the default summary
index, or a new index you created
- If selecting a new index, it must
be empty
Operational Intelligence
137
Searching and Reporting with Splunk 4.2
3. Run searches against summary
Youre limited to the initial reporting command you used to create the index,
but you can pipe to additional computations or operators.
- Saved search:
sourcetype = access_combined | sistats count by clientip!
- Search on index:
index=summary search_name=searchName| stats count by clientip | <additional commands>!
Operational Intelligence
138
Searching and Reporting with Splunk 4.2
Summary index gaps
Gaps are usually caused by two main factors:
A summary index initially only contains events from the point that you start
data collection
- There wont be data from before the summary index collection start date
- You can put it in there yourself with the backfill script.
splunkd outages
- If splunkd goes down for a significant amount of time, you could get gaps in your
summary data, depending on the populating search schedules.
Operational Intelligence
139
Searching and Reporting with Splunk 4.2
Identifying Gaps and overlaps
To identify gaps and overlaps in your data, run a search against the
summary index that uses the overlap command.
- Identify suspected gaps/overlaps in a search by specifying a start time
and
end time or a period and a saved search name, followed by the | overlap
command in the search string.
- Full details can be found at:
[Link]
Managesummaryindexgapsandoverlaps
Operational Intelligence
140
Searching and Reporting with Splunk 4.2
Correcting gaps
Use the backfill script fill_summary_index.py to correct gaps or backfill your
summary index
- Specify the App
- Specify a list of summary searches to backfill
- Or, specify to backfill all summary searches for the App
Use -dedup true to ensure the backfill script does not create duplicates
Operational Intelligence
141
Searching and Reporting with Splunk 4.2
Correcting gaps (cont.)
To backfill all summary index that already has data in it:
./splunk cmd python fill_summary_index.py -app flowerstore -name "*" -et mon@mon -lt @mon -dedup true -auth admin:changeme !
-dedup ensures the backfill script does not create an overlap
To backfill a summary index that does not have any data, dont use dedup
More information on backfilling a summary index can
be found at [Link]/base/documentation
Operational Intelligence
142
Searching and Reporting with Splunk 4.2
Summary index overlaps
Overlaps are events in a summary index (from the same index-populating
search) that share the same timestamp
Overlapping events skew reports and statistics created from summary
indexes
Overlaps can occur if you set the time range of a saved search to be longer
than the scheduled search interval
- For example, don't arrange for an hourly search to gather data for the past 90
minutes
Manually delete the overlaps from the summary index by using the search
language.
Operational Intelligence
143
Searching and Reporting with Splunk 4.2
Lab
Search a summary index
Create a report from a summary index
Operational Intelligence
144
Searching and Reporting with Splunk 4.2
Section 8:
Macros
Operational Intelligence
145
Searching and Reporting with Splunk 4.2
Section Objectives
Understand macros
Manage macros
Create a basic macro
Use a basic macro
Define arguments / variables for a macro
Use arguments with a macro
Add and use multiple arguments with a macro
Operational Intelligence
146
Searching and Reporting with Splunk 4.2
Macros overview
Useful when you frequently run searches or reports with similar search
syntax
Chunks of a search you can reuse in multiple places
Can be any chunk of your search string or search command pipeline
Allows you to define one or more arguments within the search segment
- Pass values to the argument when using the macro
Operational Intelligence
147
Searching and Reporting with Splunk 4.2
Managing Macros
As with other types of knowledge
objects, you can:
Create new
Edit
Set permissions
Enable/disable
Clone
Move
Delete
Operational Intelligence
148
Searching and Reporting with Splunk 4.2
Creating a basic macro
Manager >> Advanced search >>
Search Macros >> Add New
1. Select the destination app
2. Enter a name
-
If the macro includes arguments,
include the number of args in the
name
3. Type the search chunk
4. Save
Operational Intelligence
149
Searching and Reporting with Splunk 4.2
Using a basic macro
Type the macro name into the
search bar, or use it in a saved
search or report
Surround the macro name with
the left-quote (or grave accent)
character
sourcetype="cisco_esa" | transaction mid dcid icid!
- `macroname`
- Do not confuse with single-quote
character ()
Pipe to more commands, or
precede with search string
Operational Intelligence
150
Searching and Reporting with Splunk 4.2
Adding arguments
Change one or more variables of
the macro at search time
Include the number of arguments
in parenthesis after the macro
name
- performance(1)
Within the search definition use
$arg$!
- host=$host$!
In the arguments field, enter the
name of the argument
Operational Intelligence
151
Searching and Reporting with Splunk 4.2
Using arguments
When using a macro with
arguments, include the argument
in parenthesis following the macro
name.
Operational Intelligence
sourcetype="access_*" host=$host$
| chart avg(time_taken) as reqTime by action
| eval reqTime = round(reqTime/1000, 2)!
152
Searching and Reporting with Splunk 4.2
Adding multiple arguments
When creating multiple arguments
in the same macro:
Include the number of variables in
parenthesis in the macro name
- performance(2)
Enter the argument names,
separated by commas
- host, status
Operational Intelligence
153
Searching and Reporting with Splunk 4.2
Using multiple arguments
Include the arguments in
parenthesis, separated by a
comma
sourcetype="access_*" host=$host$ status=$status$
| chart avg(time_taken) as reqTime by action
| eval reqTime = round(reqTime/1000, 2)!
Operational Intelligence
154
Searching and Reporting with Splunk 4.2
Lab
Create a basic macro
Use the basic macro in a search
Create a macro with arguments
Use the macro and pass the arguments in a search
Operational Intelligence
155
Searching and Reporting with Splunk 4.2
Community based support
Splunk docs are constantly being updated and improved, so be sure
to select your version of Splunk to make sure the doc you are
reading applies to your version
[Link]
Splunk Answers: post specific questions and get them answered by
Splunk experts (also makes for great and informative reading)
[Link]
IRC Channel: Splunk maintains a channel #splunk on the EFNet
IRC server. Support engineers and many well-informed Splunk
users hang out there
Operational Intelligence
156
Searching and Reporting with Splunk 4.2
Splunk Support
Contact Splunk Support email: support@[Link]
File a case online
[Link]
24/7 phone depending on support contract
IRC #splunk on EFNET servers
Operational Intelligence
157
Searching and Reporting with Splunk 4.2
.conf2011:
nd
The
2
Annual
Splunk
Worldwide
Users
Conference
August
15-17,
2011:
The
WesFn
St.
Francis,
San
Francisco
54+
sessions
:
SplunkAnswers
Computer
Lab
:
2
General
Sessions
:
BoF
Meetups
Alumni
rate:
$595
Buy
3
get
1
free!
Use
your
Splunk
EducaFon
credits
for
Splunk
University!
Register
today:
[Link]/goto/conf2011
Searching and Reporting with Splunk 4.2
Operational Intelligence
Thank You
Please fill out the class survey
Operational Intelligence
159
Searching and Reporting with Splunk 4.2
