Search Head Clustering

Basics To Best Practices

Bharath Aleti | Product Manager, Splunk

Manu Jose | Sr. Software Engineer, Splunk

September 2017 | Washington, DC

 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 Agenda

▶︎ What is Search Head Clustering?

▶︎ Clustering Internals
▶︎ Distributed Scheduling
▶︎ Configuration Management
▶︎ Bundle Replication
▶︎ What’s New in SHC
 Search Head
Clustering Overview
What is Search Head Clustering?
 Search Head Clustering

Ability to group search heads into a cluster in order to provide

Highly Available and Scalable search services

MISSION
CRITICAL
ENTERPRISE
 Business Benefits of SHC

Horizontal Scaling Consistent User

Experience

Always-on Search Easy to add / manage

Services premium contents (apps)
Clustering
Internals
How does SHC work?
 SHC – How Does It Work?
1 2

1. Group search heads into a cluster (Horizontal scaling)

2. Captain gets elected dynamically (No Single point failure)
3. User created reports/dashboards automatically replicated to other search
heads (Consistent Configuration)
 Search Head Cluster Bring Up

1. Bootstrap captain
2. Bring-up members
captain
3. Captain establishes authority
4. Members join/register
members
5. CLI based cluster scale/shrink
...

config-log
{s1,s2, .., sn}
 Dynamic Captain & Auto Failover
members
Fixups
▶︎ Raft Consensus Protocol
new captain
from Stanford
• Diego Ongaro & John Osterhout

▶︎ SHC uses RAFT for LE and artifacts

Auto Failover running jobs
alerts, etc

...
search load

old captain
 Controlling Captaincy

▶︎ Captain Switching should be extremely rare

▶︎ Repair a problem by transfer captain without restarts!!!
▶︎ Rolling-restart from the captain maintains the node as captain after restarts
▶︎ Captain preference added for members
▶︎ Disaster Recovery using static captaincy
 Best Practices

▶︎ Add only fresh instances, if a node is re-purposed use “splunk clean all”
▶︎ High availability requires a minimum of 3 members
▶︎ All search heads on homogenous hardware and at same version
▶︎ Number of instances >= replication_factor
▶︎ Admin needs to manually do “splunk remove shcluster-member” on captain
to remove a dead node
▶︎ Multi-site clusters to have majority nodes at one site

12
 Distributed
Scheduling
How jobs are scheduled in SHC?
 Job Scheduling Orchestration

• Captain is job scheduler SUCC

• Eliminates need for a job-server
Search 1
• Job distribution based on round
robin or load-based heuristic

captain LOAD

search -3 load

...
search 2 balancer

FAIL
scheduler
 Job Scheduling

▶︎ Auto-failover – New captain becomes scheduler

▶︎ captain_is_adhoc_searchhead knob to reduce captain load
▶︎ Captain updates RA/DM summaries on indexers.
▶︎ Scheduler limits honored across the cluster
▶︎ Real time scheduled searches run one instance across cluster
▶︎ Centralized user quota Management*
 High Availability Of Search Results

▶︎ Artifacts are replicated across the SH members

▶︎ Adhoc searches are not replicated
▶︎ At least replication_factor number of nodes should be in UP state for
enforcing replication policy
▶︎ Replicated directory starts with “rsa_<sid>” in the dispatch directory
▶︎ Captain orchestrates reaping of search artifacts from dispatch directory
of all members
▶︎ An artifact is served based on availability from (1) itself, (2) search
originating node, (3) captain
 Centralized Cluster State

▶︎ Captain maintains a global view of alerts and suppressions and updates the
list to all members
▶︎ Captain registers all the adhoc searches run in the cluster
▶︎ Captain orchestrates reaping of search artifact replicas
▶︎ GET /services/search/jobs requests on any member will proxy to captain to
get complete jobs
 Configuration
Management
How are dynamic changes to SHC kept consistent?
 Configuration Files

▶︎ Goals
• Consistent user experience across all search heads
• Changes made on one member are reflected on all members
▶︎ Types of Configuration Files
• custom user content
• reports
• dashboards
• search-time knowledge
• field extractions
• event types
• macros
• system configurations
• inputs, forwarding, authentication
 Configuration Changes

▶︎ Users customize search and UI configurations via UI/CLI/REST

• save report
• add panel to dashboards
• create field extraction
▶︎ Administrators modify system configurations
– configure forwarding
– deploy centralized authentication (e.g. LDAP)
– install entirely new app or hand-edited configuration
 Search And UI Configurations

▶︎ Goal: Eventual Consistency

▶︎ Changes to search and UI configurations are replicated across the search
head cluster automatically
Conf Replication - Workflow

my_dashboard.xml
 Conf Replication – Progress Check

▶︎ captain keeps track of the conf replication progress of each SHC member

18bc830e3087301900bdf2a30dc1a67bf8
https://localhost:11089
318ced: Tue Jul 19 15:32:56 2016
18bc830e3087301900bdf2a30dc1a67bf8
https://localhost:8089
318ced: Tue Jul 19 15:32:52 2016
dc4a991d168ae746f27979212253d6fb95
https://localhost:8189
9fc92c: Fri Jul 1 13:51:05 2016
CaptainDummyOpId: Tue Jul 19
https://localhost:9089
15:32:09 2016
Bundle Replication
How are system-wide changes kept consistent?
 System Configurations

▶︎ Recall: only changes to search and UI configurations are replicated across the
search head cluster automatically
▶︎ Changes to system configurations are not replicated automatically because of
their high potential impact
▶︎ How are system configurations kept consistent, then?
 Configuration Deployment

▶︎ Deployer: a single, well-controlled instance outside of the cluster

▶︎ Configurations should be tested on dev/QA instances prior to deploy

D
 Bundle Push
1 2
/etc/shcluster/app1: No Changes
All apps and config are /etc/shcluster/app2: Updated Only updated apps and
shipped to the SHs in the /etc/shcluster/user: Updated updated user config is
initial deployer push pushed on subsequent
Deployer
bundle push
Bundle Push

Captain
3 4
A B C
App configuration User configuration is sent
is propagated to to the captain and then
all SHC members replicated to remaining
/etc/app2 /etc/app2: /etc/app2
SHC members
/etc/user
/etc/user /etc/user

5
Periodically, captain checks
for new bundles and
propagates the bundles to
the indexers
Idx1: Idx2: Idx3: Idx4:
KB cksum1 KB cksum1 KB cksum1 KB cksum1
KB cksum2 KB cksum2 KB cksum2 KB cksum2
 Bundle Replication
2
1 Captain SH periodically contacts CM to
Each bundle push A B C grab generation and peer set
includes a KB cksum, information. It tracks/reads
when it is sent to the async the latest common
indexers knowledge bundle across the
Idx1: cksum2 peers
Idx2: cksum2
Idx3: cksum3
3
Captains delegates a
scheduled search on 4 5
Search: cksum2
SH B SH B determines the If indexers do not have a common bundle
latest KB shared across • Best Effort Search uses common bundle
peers (cksum2) across the the largest subset of indexers
and excludes the other indexers
• Otherwise – a synchronous bundle
replication is kicked off prior to search

Idx3: KB cksum3 Idx4: KB cksum3

7
6
Indexers use the
Search request is knowledge bundle
issued with common (ckum2) included in
bundle checksum Idx1: Idx2: Idx3: Idx4: search request
(cksum2) KB cksum1 KB cksum1 KB cksum1 KB cksum1
KB cksum2 KB cksum2 KB cksum2 KB cksum2
 SH->SHC Migration
Single Search Head Deployer
/etc/app1/default/dashboard1.xml /etc/shcluster/app1/default/dashboard1 Deployer /shcluster/etc/app1
/etc/app1/local/dashboard2.xml /etc/shcluster/app1/local/dashboard2
Bundle Push

Captain

SHC Members A B C
/etc/app1/default/dashboard1.xml
/etc/app1/default/dashboard2.xml
/etc/app1 /etc/app1 /etc/app1

▶︎ Deployer merges default and local app configuration during migration

▶︎ Post migration, users cannot perform certain operations on app settings like
delete, move or unshare since default settings are immutable by a user
▶︎ Tip: Exclude default (ex: search) apps during migration to avoid overwrite.
Migrate any custom settings in default apps by moving them to a new app
Recent Additions
What’s New in SHC?
 SHC Health Checker
Goal: Improve diagnosability with actionable information
▶︎ High level cluster health assessment
▶︎ Display node status
• Captain/member
• Heartbeat status
• Uptime
• Local unpublished conf changes
▶︎ Determine conf replication baseline consistentcy
▶︎ Expose search concurrency limits (running/capacity)
Conf Replication - Health Check
 Resilient Conf Replication
▶︎ Higher resiliency to ensure continuous replication of knowledge objects across
the SHC members
• Conf replication failures when JSON string exceeds 512KB
• Long file path (>255 characters) leading to snapshot creation failure
• Large lookups files may block configuration push from the members
• Accelerated baseline match using bloom filters to find the common baseline

▶︎ Intelligent captain selection

• Prevent out-of-sync SHC member from becoming captain
 Bundle Push/Replication Improvements
▶︎ Delta bundle push to indexers on lookup deletes at runtime
• Trigger delta bundle replication when conf objects are deleted
▶︎ Deployer directs first bundle push to the Captain node
• Pushing to to captain enables faster bundle push down to the indexers
▶︎ Replicate option for lookup replication across SHC members
• replicate = true|false in transforms.conf
•True: lookup table is replicated to indexers,
•False: lookup table is only replicated within SHC and not to the indexers
• Avoids limitation of not replicating outputcsv (used to capture search results)
• Use outputlookup to create a new csv file and replicate to SH and indexers as needed
• Target usecase is ESTracker tables, that are replicated to only to SHC members
▶︎ Support MV fields in outputlookup
 SHC Manager UI
▶︎ New SHC UI available from any of the SHC members
▶︎ Enabled only in SHC environments
▶︎ Enables admins to run cluster operations (rolling restart, captain transfer)
▶︎ More functionality to come in upcoming releases

Actions Captain is Node2

 © 2017 SPLUNK INC.

1. SHC provides always-on search services

Key and consistent user experience
Takeaways 2. Enable SHC for horizontal scalability
3. Recent additions: SHC health check
(6.5), Increased conf replication resiliency
(6.6), SHC manager UI (6.6)
 © 2017 SPLUNK INC.

Thank You
Don't forget to rate this session in the
.conf2017 mobile app