Install and Configure Splunk Cluster(Distributed) Environment

Splunk cluster will be built with below server inventory

Server 1 Indexer1

Server 2 Indexer2

Server 3 Search head1

Server 4 Search head2

Server 5 Master Cluster, Deployment Server, License

Server and

Page | 1
Install enterprise Splunk software on, all the above servers.

How to install Splunk on Linux server

Server prerequisites:

1) /splunk mount point on server for installing splunk

2) splunk 6.2 rpm

We can use below commands to install splunk software on all the servers.
rpm -i <splunk software rpm name> --prefix=/splunk
Once installed go to home directory of splunk inside /bin type below command
./splunk enable boot-start
This will ask for a license agreement page
Select Y and splunk is configurednowstarted splunk services by below command
./splunk start
You may verify the same by below command
./splunk status

Steps to build / configure index cluster

Log in to the splunk instance that will act as master node of the cluster. This instance cannot be used as
an indexer.
http://ipofmaster:8000 you will get a home page like below.

Configuring Master Node

Page | 2
Go to Settings -> Indexer Clustering and enable indexer clustering

Select master node and click Next

Page | 3
Select Replication Factor as 2 and Search Factor as 2 and Security Key can be any unique key you can set.

Page | 4
This Security key used for adding additional Indexer and Search heads, so this needs to be stored
securely.
Here for demonstration “mykey” is used.
Click Enable Master role.

Restart splunk services once to complete configurations.

To restart the splunk go to the messages and click restart splunk from the manager

Page | 5
Master node (Master cluster) is configured Now we need to configure the peer nodes

Configure Peer Nodes

To configure peer nodes we need to follow below instructions on each of the indexers.
Log in to the web console to the indexer.

Go to Settings -> Indexer Clustering and enable indexer clustering

Page | 6
Select Peer node (indexer)

In the next screen – enter Master node IP (cluster master), Master port, Replication port and
Security key. Select Enable peer node.

Page | 7
Go to the server controls and restart the Splunk.

Repeat above steps and add indexer2

Adding Search head to cluster.
Log in to the web console of Search head 1

Go to Settings -> Indexer Clustering and enable indexer clustering

Page | 8
Select Search node and click next

Provide master node information, port details and keep security key, click next.

Page | 9
Click Enable Search head Node
Repeat the steps on the Search head 2.
Restart Splunk for completing the configurations.

Search head cluster Configuration

Search head cluster is configured using below command syntax.

Execute below command on search head 1 and restart Splunk service

- Here “mgmt._uri” is IP address of search head1
- “conf_deploy_fetch_url” is IP address of Master cluster node
- -auth – enter admin username and password

./splunk init shcluster-config -auth admin:admin@123 -mgmt_uri https://192.168.1..109:8089 -

replication_port 8081 -replication_factor 2 -conf_deploy_fetch_url https://192.168.1..106:8089

Execute below command on search head 2 and restart Splunk service.

Page | 10
 - Here “mgmt._uri” is IP address of search head1
- “conf_deploy_fetch_url” is IP address of Master cluster node
- -auth – enter admin username and password

./splunk init shcluster-config -auth admin:admin@123 -mgmt_uri https://192.168.1..109:8089 -

replication_port 8081 -replication_factor 2 -conf_deploy_fetch_url https://192.168.1..106:8089

With above steps search cluster is initiated. Now one of search should be promoted as Captain with
below command.
Execute below command on Search head 1.

- Here “-servers_list” enter IP address of Search head1 and Search head2

- If we execute this command on search head1, Search head 1 promoted as cluster captain.

./splunk bootstrap shcluster-captain -servers_list "https://192.168.1..109:8089,

https://192.168.1..108:8089" -auth admin:admin@123

Check cluster status using below command.

./splunk show shcluster-status

Page | 11
 Testing Search head Clustering Function

Create one user in search head 1.

Same user should be created automatically in Search head 2.

Apps Deployment to Search head in Cluster environment (distributed

environment).

Apps deployment to Search head in cluster environment follows different steps

compare to standalone installation.

Example if we want to create app called “myapp” below steps to be followed.

- Create app ( or install app) in Master node ( here apps deployment is managed by
deployer role, however we refer server as Master node)
- Copy apps from /splunk/splunk/etc/apps to /splunk/splunk/etc/shcluster/apps/

#cp –pr /splunk/splunk/etc/apps/myapp /splunk/splunk/etc/shcluster/apps/

- Execute below command on any of search head, and find out Captain search head.

#./splunk show shcluster-status

- Execute below command to distribute “myapp” from Master node to Captain Search
head.
./splunk apply shcluster-bundle -target “https://<captain search head ip:8089”

Now login to Search head1 and Search head 2 one by one, you should see “myapp” on
both Search head.

Now create some dashboard in side “myapp”, you should same dashboard replicated on
other Search head.

Page | 12
 Apps/Add-on Deployment to Indexer in Cluster environment (distributed
environment).

In the standalone splunk installation index on indexer is created using GUI.

But in the clustered environment we create index on indexer using Apps / Add-on.

Example if we wanted to create index called “myappindex” below steps needs to be

followed.

- Create app “myappaddon” in Master server.

- Move “myappaddon” folder in Master server to /splunk/splunk/etc/master-apps folder.

#mv /splunk/splunk/etc/apps/myappaddon /splunk/splunk/etc/master-apps/

- Create file called “indexes.conf” in /splunk/splunk/etc/master-apps/myappaddon/local
path.
- Add below entry to indexes.conf

[myappindex]
homePath = /splunk_index/myappindex/db
coldPath = /splunk_index/myappindex/coldb
thawedPath = /splunk_index/myappindex/thawed
maxTotalDataSizeMB = 10240
repFactor = auto

- Here “myappindex” is index ,we are creating on indexer

- Index data will be stored in indexer path /splunk_index/myappindex. This path should
be created manually on both indexer

#mkdir /splunk_index
#mkdir /splunk_index/myappindex
- Go to Master node – settings – indexer clustering ………………….
- Add some data to index “myappindex” on any of one indexer through “add data” tab on
splunk home page.
- Wait for some time, and you should find “myappindex” in cluster status page as below.

Page | 13