Splunk Enterprise 6.0.

3
Distributed Search
Generated: 4/22/2014 11:35 am

Copyright (c) 2014 Splunk Inc. All Rights Reserved

 Table of Contents
Overview...............................................................................................................1
About distributed search...........................................................................1
Some search scenarios.............................................................................2
What search heads send to search peers.................................................4
Licenses for distributed search..................................................................5
Version compatibility.................................................................................6

Configure distributed search..............................................................................8

Overview of configuration..........................................................................8
Designate the search head.......................................................................9
Add search peers....................................................................................10
Remove a search peer............................................................................14
Limit the knowledge bundle size.............................................................15
Manage distributed server names...........................................................16
Best practice: Forward search head data to the indexer layer................17

Mount the knowledge bundle...........................................................................19

About mounted bundles...........................................................................19
Configure mounted bundles....................................................................22
Use mounted bundles with search head pooling....................................25

Search head pooling..........................................................................................29

Overview of search head pooling............................................................29
Create a search head pool......................................................................31
Use a load balancer with the search head pool......................................35
Other pooling operations.........................................................................36
Manage configuration changes...............................................................37
Deployment server and search head pooling..........................................37
Select timing for configuration refresh.....................................................38

Distributed search in action..............................................................................39

How authorization works in distributed searches....................................39
How users can control distributed searches...........................................42

Troubleshoot distributed search......................................................................43

General configuration issues....................................................................43
Search head pooling configuration issues..............................................44
Distributed search error messages.........................................................47

i
Overview

About distributed search

Important: Before reading this manual, you should be familiar with the
fundamentals of Splunk Enterprise distributed deployment, as described in the
Distributed Deployment Manual.

In distributed search, a Splunk Enterprise instance called a search head sends

search requests to a group of indexers, which perform the actual searches on
their indexes. The search head then merges the results back to the user. In a
typical scenario, one search head manages searches on several indexers.

Use cases

These are some of the key use cases for distributed search:

• Horizontal scaling for enhanced performance. Distributed search

facilitates horizontal scaling by providing a way to distribute the indexing
and searching loads across multiple Splunk Enterprise instances, making
it possible to search and index large quantities of data.

• Access control. You can use distributed search to control access to

indexed data. In a typical situation, some users, such as security
personnel, might need access to data across the enterprise, while others
need access only to data in their functional area.

• Managing geo-dispersed data. Distributed search allows local offices to

access their own data, while maintaining centralized access at the
corporate level. Chicago and San Francisco can look just at their local
data; headquarters in New York can search its local data, as well as the
data in Chicago and San Francisco.

Distributed search components

The Splunk Enterprise instance that does the searching is referred to as the
search head. The indexers that participate in a distributed search are called
search peers or indexer nodes.

A search head by default runs its searches across all its search peers. You can

1
limit a search to one or more search peers by specifying the splunk_server field
in your query. See "Search across one or more distributed servers" in the Search
manual.

You can run multiple search heads across a set of search peers. To coordinate
the activity of multiple search heads (so that they share configuration settings,
search artifacts, and job management), you need to enable search head pooling.

Search heads and clusters

In index replication, clusters use search heads to search across the set of
indexers, or peer nodes. You deploy and configure search heads very differently
when they are part of a cluster. To learn more about search heads and clusters,
read "Configure the search head" in the Managing Indexers and Clusters
Manual.

Some search scenarios

This diagram shows a simple distributed search scenario for horizontal scaling,
with one search head searching across three peers:

2
In this diagram showing a distributed search scenario for access control, a
"security" department search head has visibility into all the indexing search
peers. Each search peer also has the ability to search its own data. In addition,
the department A search peer has access to both its data and the data of
department B:

Finally, this diagram shows load-balanced forwarders inputting data across the
set of indexers. There's a dedicated search head, as well as a search head on
each indexer. All the search heads can search across the entire set of indexers:

3
For more information on load balancing, see "Set up load balancing" in the
Forwarding Data manual.

For information on Splunk Enterprise distributed searches and capacity planning,

see "Dividing up indexing and searching" in the Installation manual.

What search heads send to search peers

When initiating a distributed search, the search head replicates and distributes its
knowledge objects to its search peers. Knowledge objects include saved
searches, event types, and other entities used in searching across indexes. The
search head needs to distribute this material to its search peers so that they can
properly execute queries on its behalf. The set of data that the search head
distributes is called the knowledge bundle.

What the knowledge bundle contains

The search peers use the search head's knowledge bundle to execute queries on
its behalf. When executing a distributed search, the peers are ignorant of any
local knowledge objects. They have access only to the objects in the search
head's knowledge bundle.

Bundles typically contain a subset of files (configuration files and assets) from
$SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps and

4
$SPLUNK_HOME/etc/users.

The process of distributing knowledge bundles means that peers by default

receive nearly the entire contents of the search head's apps. If an app contains
large binaries that do not need to be shared with the peers, you can reduce the
size of the bundle by means of the [replicationWhitelist] or
[replicationBlacklist] stanza in distsearch.conf. See "Limit the knowledge
bundle size" in this manual.

On the search head, the knowledge bundles resides under the

$SPLUNK_HOME/var/run directory. The bundles have the extension .bundle for full
bundles or .delta for delta bundles. They are tar files, so you can run tar tvf
against them to see the contents.

The knowledge bundle from the search head gets distributed to the
$SPLUNK_HOME/var/run/searchpeers directory on each search peer. Because the
knowledge bundle reside at a different location on the search peers than on the
search head, search scripts should not hardcode paths to resources.

Mount the knowledge bundle for greater efficiency

By default, the search head replicates and distributes the knowledge bundle to
each search peer. For greater efficiency, you can instead tell the search peers to
mount the knowledge bundle's directory location, eliminating the need for bundle
replication. When you mount a knowledge bundle, it's referred to as a mounted
bundle. To learn how to mount bundles, read "Mount the knowledge bundle".

User authorization

All authorization for a distributed search originates from the search head. At the
time it sends the search request to its search peers, the search head also
distributes the authorization information. It tells the search peers the name of the
user running the search, the user's role, and the location of the distributed
authorize.conf file containing the authorization information.

Licenses for distributed search

Each search peer in a distributed search deployment must have access to a
license pool.

5
Search heads performing no indexing or only summary indexing can use the
forwarder license. If the search head performs any other type of indexing, it must
have access to a license pool.

See "Licenses for search heads" in the Installation manual for a detailed
discussion of licensing issues.

Version compatibility
Important: This topic does not apply to clusters. All cluster nodes (master,
peers, and search heads) must run the same version of Splunk Enterprise. See
"Upgrade a cluster" in the Managing Indexers and Clusters manual.

Even with non-clustered deployments, it is best to upgrade search heads and

search peers at the same time, to take full advantage of the latest search
capabilities. If you cannot do so, read this topic.

Compatibility between search heads and search peers

6.x search heads are compatible with 6.x and 5.x search peers, in a
non-clustered environment. The search head must be at the same or higher level
than the search peers:

• A 6.x search head is compatible with a 5.x search peer.

• A 5.x search head is not compatible with a 6.x search peer.

Mixed-version distributed search compatibility

You can run a 6.x search head against 5.x search peers, but there are a few
compatibility issues to be aware of. To take full advantage of the 6.x feature set,
it is recommended that you upgrade both search head(s) and search peers at the
same time.

This section describes the compatibility issues.

6.x features in a mixed-version deployment

When running a 6.x search head against 5.x search peers, note the following:

• You can use data models on the search head but only without report
acceleration.

6
 • You can use Pivot on the search head.
• You can run predictive analytics (the predict command) on the search
head.

These features have been tested both with and without search head pooling.

Remote timeline feature can reduce performance

A 6.x search head by default asks its search peers to generate a remote timeline.
This isn't a problem with 6.x search peers, but 5.x search peers won't know how
to generate the timeline. As a result, searches can slow dramatically.

The workaround is to add the following attribute to limits.conf on the search

head :

[search]
remote_timeline_fetchall = false

After making this change, you must restart the search head.

Important: You should remove this attribute after all search peers have been
upgraded to 6.x.

7
Configure distributed search

Overview of configuration
The basic configuration to enable distributed search is simple. You just
designate a Splunk Enterprise instance as a search head and establish a
distributed search connection to a set of indexers.

There are a variety of other types of configuration that you can also perform.

Types of search heads

The typical distributed search deployment uses a dedicated search head; that is,
a search head dedicated to running searches. A dedicated search head does not
index external data.

You can, however, also designate one or more of your search peers (indexers)
as search heads. These dual function search heads can be in addition or instead
of a dedicated search head. See "Some search scenarios" for examples of
several distributed search topologies.

Configure distributed search

Setting up distributed search consists of three basic steps:

1. Designate a Splunk Enterprise instance as the search head. See "Designate

the search head".

2. Add search peers to the search head. See "Add search peers".

3. Add data inputs to the search peers. You add inputs the same as would for
any indexer, either directly on the search peer or through forwarders connecting
to the search peer. See the Getting Data In manual for information on data
inputs.

Other types of configuration

Other related types of configuration include:

• Removing a search peer.

8
 • Limiting the size of the knowledge bundle.
• Managing distributed server names.
• Mounting the knowledge bundle.
• Setting up a search head pool.
• Managing authorization.

Note: Splunk clusters also use search heads to search across their set of
indexers, or peer nodes. You deploy search heads very differently when they
are part of a cluster. To learn about deploying search heads in clusters, read
"Enable the search head" in the Managing Indexers and Clusters Manual.

Designate the search head

Distributed search is enabled by default on every Splunk Enterprise instance,
with the exception of forwarders. This means that every Splunk Enterprise server
can function as a search head to a specified group of indexers, referred to as
search peers.

In some cases, you might want a single instance to serve as both a search head
and a search peer. In other cases, however, you might need a dedicated search
head. A dedicated search head performs only searching; it does not index any
external data.

Install a dedicated search head

To install a dedicated search head, follow these steps:

1. Determine your hardware needs by reading this topic in the Installation

Manual.

2. Install Splunk Enterprise, as described in the topic in the Installation Manual

specific to your operating system.

3. Add the search head to your Enterprise license group, even though it's a
dedicated search head that's not expected to index any external data. For more
information, see "Types of Splunk Enterprise licenses".

4. Establish distributed search from the search head to all the indexers (search
peers), you want it to search. See "Add search peers" for how to do this.

9
5. Log in to the search head and perform a search that runs across all the search
peers, such as a search for *. Examine the splunk_server field in the results.
Verify that all the search peers are listed in that field.

6. See the Securing Splunk Enterprise manual for information on setting up

authentication.

Important: Do not configure the dedicated search head for indexing of external
data, since that will violate its license.

Configure a non-dedicated search head

If you want to use a single instance as both a search head and a search peer
(indexer), just install the search head as a regular Splunk Enterprise instance
with a normal license, as described in "About Splunk Enterprise licenses" in the
Installation manual. With a normal license, the instance can index external data.
You can also configure an existing indexer as a search head.

Once you have identified an instance to double as both search head and search
peer, add the search peers to the indexer. See "Add search peers".

Add search peers

Any Splunk Enterprise instance (except a forwarder) can function as a search
head to a specified group of indexers, referred to as search peers. This
distributed search capability is enabled by default.

To activate distributed search, you add search peers to a Splunk Enterprise

instance that you designate as a search head. You do this by specifying each
search peer manually. A number of other configuration options are available, but
you do not usually need to alter their default values.

When configuring instances as search heads or search peers, keep this key
distinction in mind:

• A search head must maintain a list of search peers, or it will have nothing
to search on. A dedicated search head does not external data inputs.
• A search peer must have specified external data inputs, or it will have
nothing to index.

10
These roles are not necessarily distinct. A Splunk Enterprise instance can
function simultaneously as both a search head and a search peer.

Important: Clusters also use search heads to search across the set of indexers,
or peer nodes. You deploy and configure search heads very differently when
they are part of a cluster. To learn more about configuring search heads in
clusters, read "Configure the search head" in the Managing Indexers and
Clusters Manual.

Configuration overview

You can set up distributed search on a search head using any of these
configuration methods:

• Splunk Web
• Splunk CLI
• The distsearch.conf configuration file

Splunk Web is the recommended method for most purposes.

You perform the configuration on the designated search head. The main step is
to specify the search head's search peers. The distributed search capability itself
is already enabled by default.

Important: Before an indexer can function as a search peer, you must change its
password from the default "changeme". Otherwise, the search head will not be
able to authenticate against it.

Aside from changing the password, no configuration is generally necessary on

the search peers. Access to the peers is controllable through public key
authentication. However, you do need to perform some configuration on the
search peers if you mount the knowledge bundles, so that the peers can
access them. See "Mount the knowledge bundle" for details.

Use Splunk Web

Specify the search peers

To specify the search peers:

1. Log into Splunk Web on the search head and click Settings at the top of the
page.

11
2. Click Distributed search in the Distributed Environment area.

3. Click Search peers.

4. On the Search peers page, select New.

5. Specify the search peer, along with any authentication settings.

6. Click Save.

7. Repeat for each of the search head's search peers.

Configure miscellaneous distributed search settings

To configure other settings:

1. Log into Splunk Web on the search head and click Settings at the top of the
page.

2. Click Distributed search in the Distributed Environment area.

3. Click Distributed search setup.

5. Change any settings as needed.

6. Click Save.

Use the CLI

To specify the search peers:

1. Navigate to the $SPLUNK_HOME/bin/ directory on the search head.

2. Invoke the splunk add search-server command for each search peer you
want to add.

Note the following:

• Use the -host flag to specify the IP address and management port for the
search peer.
• Provide credentials for both the local (search head) and remote (search
peer) instances. Use the -auth flag for the local credentials and the
-remoteUsername and -remotePassword flags for the remote credentials (in

12
 this example, for search peer 10.10.10.10). The remote credentials must
be for an admin-level user on the search peer.

For example:

splunk add search-server -host 10.10.10.10:8089 -auth admin:password

-remoteUsername admin -remotePassword passremote
A message indicates success, along with the need to restart the server.

Edit distsearch.conf

In most cases, the settings available through Splunk Web provide sufficient
options for configuring distributed search environments. Some advanced
configuration settings, however, are only available by directly editing
distsearch.conf. For information on the configuration options, see the
distsearch.conf spec file.

For more information on configuration files in general, see "About configuration

files".

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk Enterprise
automatically handles authentication. However, if you add peers by editing
distsearch.conf, you must distribute the key files manually.

After enabling distributed search on a Splunk Enterprise instance (and restarting

the instance), you will find the keys in this location:
$SPLUNK_HOME/etc/auth/distServerKeys/

Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem on the

search head to
$SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem on
the search peers.

Note: The <searchhead_name> is the search head's serverName, as described in

"Manage distributed server names".

Support for keys from multiple instances

Any number of search heads can have their certificates stored on search peers
for authentication. The search peers can store keys in

13
$SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem

For example, if you have search heads A and B and they both need to search the
search peer C, do the following:

1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/ and

etc/auth/distServerKeys/B/.

2. Copy A's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B's

trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart C.

Remove a search peer

You can remove a search peer from a search head through Splunk Web or the
CLI. As you might expect, doing so merely removes the search head's
knowledge of that search peer; it does not affect the peer itself.

Remove a search peer via Splunk Web

You can remove a search peer from a search head through the Distributed
search page on the search head's Splunk Web.

Note: This only removes the search peer entry from the search head; it does not
remove the search head key from the search peer. In most cases, this is not a
problem and no further action is needed.

Remove a search peer via the CLI

On the search head, run the splunk remove search-server command to remove
a search peer from the search head.

Note the following:

• Use the -auth flag to provide credentials for the search head only.
• Use the -url flag to specify the peer's location and splunkd management
port. By default, the management port is 8089, although it might be
different for your deployment.

This example removes the search peer 10.10.10.10:8089:

14
splunk remove search-server -auth admin:password -url 10.10.10.10:8089

A message indicating success appears after the peer is removed.

Disable the trust relationship

As an additional step, you can disable the trust relationship between the search
peer and the search head. To do this, delete the trusted.pem file from
$SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name> on the search peer.

Note: The <searchhead_name> is the search head's serverName, as described in

"Manage distributed server names".

This step is usually unnecessary.

Limit the knowledge bundle size

The knowledge bundle is the data that the search head replicates and
distributes to each search peer to enable its searches. For information on the
contents and purpose of this bundle, see "What search heads send to search
peers".

The knowledge bundle can grow quite large, because, by default, it includes
nearly the entire contents of all the search head's apps. To limit the size of the
bundle, you can create a replication whitelist. To do this, edit distsearch.conf
and specify a [replicationWhitelist] stanza:

[replicationWhitelist]
<name> = <whitelist_regex>
...

All files that satisfy the whitelist regex will be included in the bundle that the
search head distributes to its search peers. If multiple regex's are specified, the
bundle will include the union of those files.

In this example, the knowledge bundle will include all files with extensions of
either ".conf" or ".spec":

[replicationWhitelist]
allConf = *.conf

15
allSpec = *.spec

The names, such as allConf and allSpec, are used only for layering. That is, if
you have both a global and a local copy of distsearch.conf, the local copy can
be configured so that it overrides only one of the regex's. For instance, assume
that the example shown above is the global copy and that you then specify a
whitelist in your local copy like this:

[replicationWhitelist]
allConf = *.foo.conf

The two conf files will be layered, with the local copy taking precedence. Thus,
the search head will distribute only files that satisfy these two regex's:

allConf = *.foo.conf
allSpec = *.spec

For more information on attribute layering in configuration files, see "Attribute

precedence" in the Admin manual.

You can also create a replication blacklist, using the [replicationBlacklist]

stanza. The blacklist takes precedence over any whitelist. See distsearch.conf in
the Admin manual for details.

As an alternative to replicating and distributing a knowledge bundle, large or

small, to all search peers, you can mount the knowledge bundle on shared
storage. For more information, read "Mount the knowledge bundle".

Manage distributed server names

The name of each search head and search peer is determined by its serverName
attribute, specified in server.conf. The serverName attribute defaults to the
server's machine name.

In distributed search, all search heads and search peers in the group must have
unique names. The serverName has three specific uses in distributed search:

• For authenticating search heads. When search peers are authenticating

a search head, they look for the search head's key file in
/etc/auth/distServerKeys/<searchhead_name>/trusted.pem.

16
 • For identifying search peers in search queries. serverName is the value
of the splunk_server field that you specify when you want to query a
specific node. See "Retrieve events from indexes and distributed search
peers" in the Search manual.
• For identifying search peers in search results. serverName gets
reported back in the splunk_server field.

Note: serverName is not used when adding search peers to a search head. In
that case, you identify the search peers through their domain names or IP
addresses.

The only reason to change serverName is if you have multiple instances of Splunk
Enterprise residing on a single machine, and they're participating in the same
distributed search group. In that case, you'll need to change serverName to
distinguish them.

Best practice: Forward search head data to the

indexer layer
It is considered a best practice to forward all search head internal data to the
search peer (indexer) layer. This has several advantages:

• It accumulates all data in one place. This simplifies the process of

managing your data: You only need to manage your indexes and data at
one level, the indexer level.
• It enables diagnostics for the search head if it goes down. The data
leading up to the failure is accumulated on the indexers, where another
search head can later access it.
• By forwarding the results of summary index searches to the indexer level,
all search heads have access to them. Otherwise, they're only available to
the search head that generates them.

The preferred approach is to forward the data directly to the indexers, without
indexing separately on the search head. You do this by configuring the search
head as a forwarder. These are the main steps:

1. Make sure that all necessary indexes exist on the indexers. For example, the
SoS app uses a scripted input that it puts data into a custom index. If you install
SoS on the search head, you need to also install the SoS-TA add-on on the
indexers, to provide the indexers with the necessary index settings for the data
the app generates. On the other hand, since _audit and _internal exist on

17
indexers as well as search heads, you do not need to create separate versions of
those indexes to hold the corresponding search head data.

2. Configure the search head as a forwarder. Create an outputs.conf file on the

search head that configures the search head for load-balanced forwarding across
the set of search peers (indexers). For example:

[tcpout:my_search_peers]
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997
Note: Do not set indexAndForward=true in outputs.conf. If you do, the search
head will both retain the data locally and forward it to the search peers.

# Sample SEARCH HEAD outputs.conf

# TURN OFF INDEXING ON SEARCH HEAD

[indexAndForward]
index = false

[tcpout]
defaultGroup = primary_indexers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:primary_indexers]
server = <ip of indexer1>:9997,<ip of indexer2>:9997
autoLB = true

For details on configuring outputs.conf, read "Configure forwarders with

outputs.conf" in the Forwarding Data manual.

18
Mount the knowledge bundle

About mounted bundles

The set of data that a search head distributes to its search peers is called the
knowledge bundle. For information on the contents and purpose of this bundle,
see "What search heads send to search peers".

The bundle contents reside in the search head's

$SPLUNK_HOME/etc/{apps,users,system} subdirectories.

By default, the search head replicates and distributes the knowledge bundle to
each search peer. For greater efficiency, you can instead tell the search peers to
mount the knowledge bundle's directory location, eliminating the need for bundle
replication. When you mount a knowledge bundle on shared storage, it's referred
to as a mounted bundle.

Important: Most shared storage solutions don't work well across a WAN. Since
mounted bundles require shared storage, you generally should not implement
them across a WAN.

Why use mounted bundles

Mounted bundles are useful if you have large amounts of search-time data,
which could otherwise slow the replication process. One common cause of slow
bundle replication is large lookup tables.

A warning gets written to splunkd.log if bundle replication is taking a long time.

For example:

DistributedBundleReplicationManager - bundle replication to 2 peer(s)

took
too long (12943ms), bundle file size=296150KB ...

Mounted bundle architectures

Depending on your search head configuration, there are a number of ways to set
up mounted bundles. These are some of the typical ones:

19
• For a single search head. Mount the knowledge bundle on shared
storage. All the search peers then access the bundle to process search
requests. This diagram illustrates a single search head with a mounted
bundle on shared storage:

• For multiple pooled search heads. For multiple search heads, you can
combine mounted bundles with search head pooling. The pooled search
heads maintain one bundle on the shared storage, and all search peers
access that bundle. This diagram shows search head pooling with a
mounted bundle:

20
 • For multiple non-pooled search heads. Maintain the knowledge
bundle(s) on each search head's local storage. In this diagram, each
search head maintains its own bundle, which each search peer mounts
and accesses individually:

There are numerous other architectures you can design with mounted bundles.
You could, for example, use shared storage for multiple search heads, but
without search head pooling. On the shared storage, you would maintain
separate bundles for each search head. The search peers would need to access

21
each bundle individually.

In each case, the search peers need access to each search head's
$SPLUNK_HOME/etc/{apps,users,system} subdirectories. In the case of search
head pooling, the search peers need access to the pool's shared set of
subdirectories.

Important: The search peers use the mounted directories only when fulfilling the
search head's search requests. For indexing and other purposes not directly
related to distributed search, the search peers will use their own, local apps,
users, and system directories, the same as any other indexer.

Configure mounted bundles

To set up mounted bundles, you need to configure both the search head and its
search peers. The procedures described here assume the bundles are on shared
storage, but they do not need to be. They just need to be in some location that
both the search head and its search peers can access.

Note: It's best not to locate mounted bundles in the search head's local
$SPLUNK_HOME path.

These procedures also assume a single search head (no search head pooling).
For details on how to configure mounted bundles with search head pooling, see
"Use mounted bundles with search head pooling" below.

Configure the search head

Here are the steps you take on the search head:

1. Mount the bundle subdirectories ($SPLUNK_HOME/etc/{apps,users,system}) on

shared storage. The simplest way to do this is to mount the search head's entire
$SPLUNK_HOME/etc directory:

• On *nix platforms, set up an NFS mount.

• On Windows, set up a CIFS (SMB) share.

Important: The search head's Splunk user account needs read/write access to
the shared storage location. The search peers need read access to the bundle
subdirectories.

22
2. In the distsearch.conf file on the search head, set:

shareBundles=false

This stops the search head from replicating bundles to the search peers.

3. Restart the search head.

Configure the search peers

For each search peer, follow these steps to access the mounted bundle:

1. Mount the bundle directory on the search peer.

2. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the

search peer. For each search head that the peer is connected to, create a
[searchhead:<searchhead-splunk-server-name>] stanza, with these attributes:

[searchhead:<searchhead-splunk-server-name>]
mounted_bundles=true
bundles_location=<path_to_bundles>

Note the following:

• The search peer's configuration file must contain only the

[searchhead:<searchhead-splunk-server-name>] stanza(s). The other
stanzas in distsearch.conf are for search heads only.

• To identify the <searchhead-splunk-server-name>, run this command on

the search head:

splunk show servername

• The <path_to_bundles> needs to specify the mountpoint on the search

peer, not on the search head. For example, say $SPLUNK_HOME on your
search head is /opt/splunk, and you export /opt/splunk/etc via NFS.
Then, on the search peer, you mount that NFS share at
/mnt/splunk-head. The value of <path_to_bundles> should be
/mnt/splunk-head, not /opt/splunk.

23
Important: If multiple search heads will be distributing searches to this search
peer, you must create a separate stanza on the search peer for each of them.
This is necessary even if you're using search head pooling.

3. Restart the search peer.

Note: You can optionally set up symbolic links to the bundle subdirectories
(apps,users,system) to ensure that the search peer has access only to the
necessary subdirectories in the search head's /etc directory. See the following
example for details on how to do this.

Example configuration

Here's an example of how to set up mounted bundles on shared storage:

Search head

On a search head whose Splunk Enterprise server name is "searcher01":

1. Mount the search head's $SPLUNK_HOME/etc directory to shared storage with

read/write access.

2. In the distsearch.conf file on the search head, set:

[distributedSearch]
...
shareBundles = false

3. Restart the search head.

Search peers

For each search peer:

1. Mount the search head's $SPLUNK_HOME/etc directory on the search peer to:

/mnt/searcher01

2. (Optional.) Create a directory that consists of symbolic links to the bundle

subdirectories:

24
 /opt/shared_bundles/searcher01
/opt/shared_bundles/searcher01/system -> /mnt/searcher01/system
/opt/shared_bundles/searcher01/users -> /mnt/searcher01/users
/opt/shared_bundles/searcher01/apps -> /mnt/searcher01/apps

Note: This optional step is useful for ensuring that the peer has access only to
the necessary subdirectories.

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the

search peer, with this stanza:

[searchhead:searcher01]
mounted_bundles = true
bundles_location = /opt/shared_bundles/searcher01

4. Restart the search peer.

5. Repeat the process for each search peer.

Use mounted bundles with search head pooling

The process for configuring mounted bundles is basically no different if you're
using search head pooling to manage multiple search heads. A few things to
keep in mind:

• Use the same shared storage location for both the search head pool and
the mounted bundles. Search head pooling uses a subset of the
directories required for mounted bundles.
• Search head pooling itself only requires that you mount the
$SPLUNK_HOME/etc/{apps,users} directories. However, when using
mounted bundles, you must also provide a mounted
$SPLUNK_HOME/etc/system directory. This doesn't create any conflict
among the search heads, as they will always use their own versions of the
system directory and ignore the mounted version.
• The search peers must create separate stanzas in distsearch.conf for
each search head in the pool. The bundles_location in each of those
stanzas must be identical.

See "Configure search head pooling" for information on setting up a search head
pool.

25
Example configuration: Search head pooling with mounted
bundles

This example shows how to combine search head pooling and mounted bundles
in one system. There are two main sections to the example:

1. Set up a search head pool consisting of two search heads. In this part, you
also mount the bundles.

2. Set up the search peers so that they can access bundles from the search head
pool.

The example assumes you're using an NFS mount for the shared storage
location.

Part 1: Set up the search head pool

Before configuring the pool, perform these preliminary steps:

1. Enable two Splunk Enterprise instances as search heads. This example

assumes that the instances are named "searcher01" and "searcher02".

2. Set up a shared storage location accessible to each search head. This

example assumes that you set up an NFS mountpoint, specified on the search
heads as /mnt/search-head-pooling.

For detailed information on these steps, see "Create a pool of search heads".

Now, configure the search head pool:

1. On each search head, stop splunkd:

splunk stop splunkd

2. On each search head, enable search head pooling. In this example, you're
using an NFS mount of /mnt/search-head-pooling as your shared storage
location:

splunk pooling enable /mnt/search-head-pooling [--debug]

26
Among other things, this step creates empty /etc/apps and /etc/users
directories under /mnt/search-head-pooling. Step 3 uses those directories.

3. Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users

directories on one of the search heads into the /etc/apps and /etc/users
subdirectories under /mnt/search-head-pooling:

cp -r $SPLUNK_HOME/etc/apps/* /mnt/search-head-pooling/etc/apps

cp -r $SPLUNK_HOME/etc/users/* /mnt/search-head-pooling/etc/users

4. Copy one search head's $SPLUNK_HOME/etc/system directory to

/mnt/search-head-pooling/etc/system.

cp -r $SPLUNK_HOME/etc/system /mnt/search-head-pooling/etc/

5. Review the /mnt/search-head-pooling/etc/system/local/server.conf file for

a [pooling] stanza. If it exists, remove any entries.

6. On each search head, edit the distsearch.conf file to set shareBundles =

false:

[distributedSearch]
...
shareBundles = false

7. On each search head, start splunkd:

splunk start splunkd

Your search head pool should now be up and running.

Part 2: Mount bundles on the search peers

Now, mount the bundles on the search peers.

On each search peer, perform these steps:

1. Mount the shared storage location (the same location that was earlier set to
/mnt/search-head-pooling on the search heads) so that it appears as
/mnt/bundles on the peer.

27
2. Create a directory that consists of symbolic links to the bundle subdirectories:

/opt/shared_bundles/bundles/system -> /mnt/bundles/etc/system

/opt/shared_bundles/bundles/users -> /mnt/bundles/etc/users
/opt/shared_bundles/bundles/apps -> /mnt/bundles/etc/apps

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the

search peer, with stanzas for each of the two search heads:

[searchhead:searcher01]
mounted_bundles = true
bundles_location = /opt/shared_bundles/bundles

[searchhead:searcher02]
mounted_bundles = true
bundles_location = /opt/shared_bundles/bundles

4. Restart the search peer:

splunk restart splunkd

Repeat the process for each search peer.

28
Search head pooling

Overview of search head pooling

Important: Search head pooling is an advanced feature. It's recommended that
you contact the Splunk sales team to discuss your deployment before
attempting to implement it.
You can set up multiple search heads so that they share configuration and user
data. This is known as search head pooling. The main reason for having
multiple search heads is to facilitate horizontal scaling when you have large
numbers of users searching across the same data. Search head pooling can also
reduce the impact if a search head becomes unavailable. This diagram provides
an overview of a typical deployment with search head pooling:

You enable search head pooling on each search head that you want to be
included in the pool, so that they can share configuration and user data. Once
search head pooling has been enabled, these categories of objects will be
available as common resources across all search heads in the pool:

• configuration data -- configuration files containing settings for saved

searches and other knowledge objects.
• search artifacts, records of specific search runs.

29
 • scheduler state, so that only one search head in the pool runs a
particular scheduled search.

For example, if you create and save a search on one search head, all the other
search heads in the pool will automatically have access to it.

Search head pooling makes all files in $SPLUNK_HOME/etc/{apps,users} available

for sharing. This includes *.conf files, *.meta files, view files, search scripts,
lookup tables, etc.

Key implementation issues

Note the following:

• Most shared storage solutions don't perform well across a WAN. Since
search head pooling requires low-latency shared storage capable of
serving a high number of operations per second, implementing search
head pooling across a WAN is not supported.

• All search heads in a pool must be running the same version of Splunk
Enterprise. Be sure to upgrade all of them at once. See "Upgrade your
distributed deployment" in the Distributed Deployment Manual for details.

• The purpose of search head pooling is to simplify the management of

groups of dedicated search heads. Do not implement it on groups of
indexers doubling as search heads. That is an unsupported configuration.
Search head pooling has a significant effect on indexing performance.

• The search heads in a pool cannot be search peers of each other.

Search head pooling and knowledge bundles

The set of data that a search head distributes to its search peers is known as the
knowledge bundle. For details, see "What search heads send to search peers".

By default, only one search head in a search head pool sends the knowledge
bundle to the set of search peers. Also, if search heads in a pool are also search
peers of each other, they will not send bundles to each other, since they can
access the bundles in the pool.This is an optimization introduced in version 4.3.2
but made the default in version 5.0. It is controllable by means of the
useSHPBundleReplication attribute in distsearch.conf.

30
As a further optimization, you can mount knowledge bundles on shared storage,
as described in "About mounted bundles". By doing so, you eliminate the need to
distribute the bundle to the search peers. For information on how to combine
search head pooling with mounted knowledge bundles, read the section in that
topic called "Use mounted bundles with search head pooling".

For more information

See the other topics in this chapter for more information on search head pooling:

• "Create a search head pool"

• "Use a load balancer with the search head pool"
• "Other pooling operations"
• "Manage configuration changes"
• "Deployment server and search head pooling"
• "Select timing for configuration refresh"

Answers

Have questions? Visit Splunk Answers and see what questions and answers the
Splunk community has about search head pooling.

Create a search head pool

To create a pool of search heads, follow these steps:

1. Set up a shared storage location accessible to each search head.

2. Configure each individual search head.

3. Stop the search heads.

4. Enable pooling on each search head.

5. Copy user and app directories to the shared storage location.

6. Restart the search heads.

The steps are described below in detail:

31
1. Set up a shared storage location accessible to each search
head

So that each search head in a pool can share configurations and artifacts, they
need to access a common set of files via shared storage:

• On *nix platforms, set up an NFS mount.

• On Windows, set up a CIFS (SMB) share.

Important: The Splunk user account needs read/write access to the shared
storage location. When installing a search head on Windows, be sure to install it
as a user with read/write access to shared storage. The Local System user does
not have this access. For more information, see "Choose the user Splunk should
run as" in the Installation manual.

2. Configure each search head

a. Set up each search head individually, specifying the search peers in the usual
fashion. See "Add search peers".

b. Make sure that each search head has a unique serverName attribute,
configured in server.conf. See "Manage distributed server names" for detailed
information on this requirement. If the search head does not have a unique
serverName, a warning will be generated at start-up. See "Warning about unique
serverName attribute" for details.

c. Specify the necessary authentication. You have two choices:

• Specify user authentication on each search head separately. A valid user

on one search head is not automatically a user on another search head in
the pool. You can use LDAP to centrally manage user authentication, as
described in "Set up user authentication with LDAP".

• Place a common authentication configuration on shared storage, to be

used by all pool members. You must restart the pool members after any
change to the authentication.

Note: Any authentication change made on an individual pool member (for

example, via Splunk Web) overrides for that pool member only any configuration
on shared storage. You should, therefore, generally avoid making authentication
changes through Splunk Web if a common configuration already exists on shared

32
storage.

3. Stop the search heads

Before enabling pooling, you must stop splunkd. Do this for each search head in
the pool.

4. Enable pooling on each search head

Use the CLI command splunk pooling enable to enable pooling on a search
head. The command sets certain values in server.conf. It also creates
subdirectories within the shared storage location and validates that Splunk
Enterprise can create and move files within them.

Here's the command syntax:

splunk pooling enable <path_to_shared_storage> [--debug]

Note:

• On NFS, <path_to_shared_storage> should be the NFS's share

mountpoint.
• On Windows, <path_to_shared_storage> should be the UNC path of the
CIFS/SMB share.
• The --debug parameter causes the command to log additional information
to btool.log.

Execute this command on each search head in the pool.

The command sets values in the [pooling] stanza of the server.conf file in
$SPLUNK_HOME/etc/system/local.

You can also directly edit the [pooling] stanza of server.conf. For detailed
information on server.conf, look here.

Important: The [pooling] stanza must be placed in the server.conf file directly
under $SPLUNK_HOME/etc/system/local/. This means that you cannot deploy the
[pooling] stanza via an app, either on local disk or on shared storage. For
details see the server.conf spec file.

33
 5. Copy user and app directories to the shared storage
location

Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users

directories on an existing search head into the empty /etc/apps and /etc/users
directories in the shared storage location. Those directories were created in step
4 and reside under the <path_to_shared_storage> that you specified at that time.

For example, if your NFS mount is at /tmp/nfs, copy the apps subdirectories that
match this pattern:

$SPLUNK_HOME/etc/apps/*

into

/tmp/nfs/etc/apps

This results in a set of subdirectories like:

/tmp/nfs/etc/apps/search
/tmp/nfs/etc/apps/launcher
/tmp/nfs/etc/apps/unix
[...]

Similarly, copy the user subdirectories:

$SPLUNK_HOME/etc/users/*

into

/tmp/nfs/etc/users

Important: You can choose to copy over just a subset of apps and user
subdirectories; however, be sure to move them to the precise locations described
above.

6. Restart the search heads

After running the splunk pooling enable command, restart splunkd. Do this for
each search head in the pool.

34
Use a load balancer with the search head pool
You will probably want to run a load balancer in front of your search heads. That
way, users can access the pool of search heads through a single interface,
without needing to specify a particular one.

Another reason for using a load balancer is to ensure access to search artifacts
and results if one of the search heads goes down. Ordinarily, RSS and email
alerts provide links to the search head where the search originated. If that search
head goes down (and there's no load balancer), the artifacts and results become
inaccessible. However, if you've got a load balancer in front, you can set the
alerts so that they reference the load balancer instead of a particular search
head.

Configure the load balancer

There are a couple issues to note when selecting and configuring the load
balancer:

• The load balancer must employ layer-7 (application-level) processing.

• Configure the load balancer so that user sessions are "sticky" or

"persistent". This ensures that the user remains on a single search head
throughout their session.

Generate alert links to the load balancer

To generate alert links to the load balancer, you must edit alert_actions.conf:

1. Copy alert_actions.conf from a search head to the appropriate app directory

in the shared storage location. In most cases, this will be
/<path_to_shared_storage>/etc/apps/search/local.

2. Edit the hostname attribute to point to the load balancer:

hostname = <proxy host>:<port>

For details, see alert_actions.conf in the Admin manual.

The alert links should now point to the load balancer, not the individual search
heads.

35
Other pooling operations
Besides the splunk pooling enable CLI command, there are several other
commands that are important for managing search head pooling:

• splunk pooling validate

• splunk pooling disable
• splunk pooling display

You must stop splunkd before running splunk pooling enable or splunk
pooling disable. However, you can run splunk pooling validate and splunk
pooling display while splunkd is either stopped or running.

Validate that each search head has access to shared

resources

The splunk pooling enable command validates search head access when you
initially set up search head pooling. If you ever need to revalidate the search
head's access to shared resources (for example, if you change the NFS
configuration), you can run the splunk pooling validate CLI command:

splunk pooling validate [--debug]

Disable search head pooling

You can disable search head pooling with this CLI command:

splunk pooling disable [--debug]

Run this command for each search head that you need to disable.

Important: Before running the splunk pooling disable command, you must
stop splunkd. After running the command, you should restart splunkd.

Display pooling status

You can use the splunk pooling display CLI command to determine whether
pooling is enabled on a search head:

splunk pooling display

This example shows how the system response varies depending on whether
pooling is enabled:

36
$ splunk pooling enable /foo/bar
$ splunk pooling display
Search head pooling is enabled with shared storage at: /foo/bar
$ splunk pooling disable
$ splunk pooling display
Search head pooling is disabled

Manage configuration changes

Important: Once pooling is enabled on a search head, you must notify the
search head whenever you directly edit a configuration file.

Specifically, if you add a stanza to any configuration file in a local directory, you
must run the following command:

splunk btool fix-dangling

Note: This is not necessary if you make changes by means of Splunk Web or the
CLI.

Deployment server and search head pooling

With search head pooling, all search heads access a single set of configurations,
so you don't need to use a deployment server or a third party deployment
management tool like Puppet to push updates to multiple search heads.
However, you might still want to use a deployment tool with search head pooling,
in order to consolidate configuration operations across all Splunk Enterprise
instances.

If you want to use the deployment server to manage your search head
configuration, note the following:

1. Designate one of the search heads as a deployment client by creating a

deploymentclient.conf file in $SPLUNK_HOME/etc/system/local and specifying its
deployment server. You only need to designate one search head as a
deployment client.

2. In deploymentclient.conf, set the repositoryLocation attribute to the search

head's shared storage mountpoint. You must also set
serverRepositoryLocationPolicy=rejectAlways, so that the locally set

37
repositoryLocation gets used as the download location.

3. In serverclass.conf on the deployment server, define a server class for the

search head client.

For detailed information on the deployment server, see "About deployment

server" in the Updating Splunk Enterprise Instances manual.

Select timing for configuration refresh

In version 5.0.2 and earlier, the defaults for synchronizing from the storage
location were set to very frequent intervals. This could lead to excessive time
spent reading configuration changes from the pool, particularly in deployments
with large numbers of users (in the hundreds or thousands).

The default settings have been changed to less frequent intervals starting with
5.0.3. In server.conf, the following settings affect configuration refresh timing:

# 5.0.3 defaults
[pooling]
poll.interval.rebuild = 1m
poll.interval.check = 1m

The previous defaults for these settings were 2s and 5s, respectively.

With the old default values, a change made on one search head would become
available on another search head at most seven seconds later. There is usually
no need for updates to be propagated that quickly. By changing the settings to
values of one minute, the load on the shared storage system is greatly reduced.
Depending on your business needs, you might be able to set these values to
even longer intervals.

38
Distributed search in action

How authorization works in distributed searches

The authorization settings that a search peer uses when processing distributed
searches are different from those that it uses for its local activities, such as
administration and local search requests:

• When processing a distributed search, the search peer uses the settings
contained in the knowledge bundle that the search head distributes to all
the search peers when it sends them a search request. These settings are
created and managed on the search head.
• When performing local activities, the search peer uses the authorization
settings created and stored locally on the search peer itself.

When managing distributed searches, it is therefore important that you

distinguish between these two types of authorization. You need to be particularly
aware of how authorization settings get distributed through the knowledge bundle
when you're managing a system with search head pooling or mounted
bundles.

For background information, read about these key concepts:

• Splunk Enterprise authorization: The topic "About role-based user

access" in the Securing Splunk Enterprise manual
• Mounted bundles: The chapter "Mount the knowledge bundle" in this
manual
• Search head pooling: The chapter "Search head pooling" in this manual

Manage authorization for distributed searches

All authorization settings are stored in one or more authorize.conf files. This
includes settings configured through Splunk Web or the CLI. It is these
authorize.conf files that get distributed from the search head to the search
peers. On the knowledge bundle, the files are usually located in either
/etc/system/{local,default} and/or /etc/apps/<app-name>/{local,default}.

Since search peers automatically use the settings in the knowledge bundle,
things normally work fine. You configure roles for your users on the search head,
and the search head automatically distributes those configurations to the search
peers when it distributes the search itself.

39
With search head pooling, however, you must take care to ensure that the search
heads and the search peers all use the same set of authorize.conf file(s). For
this to happen, you must make sure:

• All search heads in the pool use the same set of authorize.conf files

• The set of authorize.conf files that the search heads use goes into the
knowledge bundle so that they get distributed to the search peers.

This topic describes the four main scenarios, based on whether or not you're
using search head pooling or mounted bundles. It describes the scenarios in
order from simple to complex.

Four scenarios

What you need to do with the distributed search authorize.conf files depends on
whether your deployment implements search head pooling or mounted bundles.
The four scenarios are:

• No search head pooling, no mounted bundles

• No search head pooling, mounted bundles
• Search head pooling, no mounted bundles
• Search head pooling, mounted bundles

The first two scenarios "just work" but the last two scenarios require careful
planning. For the sake of completeness, this section describes all four scenarios.

Note: These scenarios address authorization settings for distributed search only.
Local authorization settings function the same independent of your distributed
search deployment.

No search head pooling, no mounted bundles

Whatever authorization settings you have on the search head get automatically
distributed to its search peers as part of the replicated knowledge bundle that
they receive with distributed search requests.

No search head pooling, mounted bundles

Whatever authorization settings you have on the search head get automatically
placed in the mounted bundle and used by the search peers during distributed
search processing.

40
Search head pooling, no mounted bundles

The search heads in the pool share their /apps and /users directories but not
their /etc/system/local directories. Any authorize.conf file in an /apps
subdirectory will be automatically shared by all search heads and included in the
knowledge bundle when any of the search heads distributes a search request to
the search peers.

The problem arises because authorization changes can also get saved to an
authorize.conf file in a search head's /etc/system/local directory (for example,
if you update the search head's authorization settings via Splunk Web). This
directory does not get shared among the search heads in the pool, but it still gets
distributed to the search peers as part of the knowledge bundle. Because of how
the configuration system works, any copy of authorize.conf file in
/etc/system/local will have precedence over a copy in an /apps subdirectory.
(See "Configuration file precedence" in the Admin manual for details.)

Therefore, a copy of authorize.conf that gets distributed to the search peers

from a single search head's /etc/system/local directory has precedence over
any copies distributed from the search head pool's shared directory. Unless you
account for this situation, the search peers can end up using different
authorization settings for different searches, depending on which search head
distributed the search to them. For most situations, this is not what you want to
occur.

To avoid this problem, you need to make sure that any changes made to a
search head's /etc/system/local/authorize.conf file get propagated to all
search heads in the pool. One way to handle this is to move any changed
/etc/system/local/authorize.conf file into an app subdirectory, since all search
heads in the pool share the /apps directory.

Search head pooling, mounted bundles

This is similar to the previous scenario. The search heads in the pool share their
/apps and /users directories but not their /etc/system/local directories. Any
authorize.conf file in an /apps subdirectory will be automatically shared by all
search heads. It will also be included in the mounted bundle that the search
peers use when processing a search request from any of the search heads.

However, authorization changes can also wind up in an authorize.conf file in a

search head's /etc/system/local directory (for example, if you update the
search head's authorization settings via Splunk Web). This directory does not get
automatically shared among the search heads in the pool. It also does not get

41
automatically distributed to the mounted bundle that the search peers use.
Therefore, you must provide some mechanism that ensures that all the search
heads and all the search peers have access to that version of authorize.conf.

The simplest way to handle this is to move any changed

/etc/system/local/authorize.conf file into an app subdirectory, since both the
pooled search heads and all the search peers share the /apps directory.

How users can control distributed searches

From the user standpoint, specifying and running a distributed search is
essentially the same as running any other search. Behind the scenes, the search
head distributes the query to its search peers and consolidates the results when
presenting them to the user.

Users can limit the search peers that participate in a search. They also need to
be aware of the distributed search configuration to troubleshoot.

Perform distributed searches

In general, you specify a distributed search through the same set of commands
as for a local search. However, several additional commands and options are
available specifically to assist with controlling and limiting a distributed search.

A search head by default runs its searches across all search peers in its cluster.
You can limit a search to one or more search peers by specifying the
splunk_server field in your query. See "Retrieve events from indexes and
distributed search peers" in the Search manual.

The search command localop is also of use in defining distributed searches. It

enables you to limit the execution of subsequent commands to the search head.
See the description of localop in the Search Reference for details and an
example.

In addition, the lookup command provides a local argument for use with
distributed searches. If set to true, the lookup occurs only on the search head; if
false, the lookup occurs on the search peers as well. This is particularly useful
for scripted lookups, which replicate lookup tables. See the description of lookup
in the Search Reference for details and an example.

42
Troubleshoot distributed search

General configuration issues

Clock skew between search heads and search peers can
affect search behavior

It's important to keep the clocks on your search heads and search peers in sync,
via NTP (network time protocol) or some similar means. If the clocks are
out-of-sync by more than a few seconds, you can end up with search failures or
premature expiration of search artifacts.

Searches can fail if configurations have not yet been

replicated to search peers

Configuration changes can take a short time to propagate from search heads to
search peers. As a result, during the time between when configuration changes
are made on the search head and when they're replicated to the search peers
(typically, not more than a few minutes), distributed searches can either fail or
provide results based on the previous configuration.

Types of configuration changes that can cause search failures are those that
involve new apps or changes to authentication.conf or authorize.conf.
Examples include:

• changing the allowed indexes for a role and then running a search as a
user within that role
• creating a new app and then running a search from within that app

Any failures will be noted in messages on the search head.

Types of changes that can provide results based on the previous configuration
include changing a field extraction or a lookup table file.

To remediate, run the search again.

43
Search head pooling configuration issues
When implementing search head pooling, there are a few potential issues you
should be aware of, mainly having to do with coordination among search heads.

Authentication and authorization changes made in Splunk

Web apply only to a single search head

Authentication and authorization changes made through a search head's Splunk

Web apply only to that search head and not to other search heads in that pool.
Each member of the pool maintains its local configurations in
$SPLUNK_HOME/etc/system/local. To share configurations across the pool, set
them up in shared storage, as described in "Configure search head pooling".

Clock skew between search heads and shared storage can

affect search behavior

It's important to keep the clocks on your search heads and shared storage server
in sync, via NTP (network time protocol) or some similar means. If the clocks are
out-of-sync by more than a few seconds, you can end up with search failures or
premature expiration of search artifacts.

Permission problems on the shared storage server can cause

pooling failure

On each search head, the user account Splunk runs as must have read/write
permissions to the files on the shared storage server.

Performance analysis

A large percentage of search head pooling issues boil down to insufficient

performance.

When deploying or investigating a search head pooling environment, it's

important to consider these factors:

• Storage: The storage backing the pool must be able to handle a very high
number of IOPS. IOPS under 1000 will probably never work well.
• Network: The communication path between the backing store and the
search heads must be high bandwidth and extremely low latency. This
probably means your storage system should be on the same switch as

44
 your search heads. WAN links are not going to work.
• Server Parallelism: Because searching results in a large number of
processes requesting a large number of files, the parallelism in the system
must be high. This can require tuning the NFS server to handle a larger
number of requests in parallel.
• Client Parallelism: The client operating system must be able to handle a
significant number of requests at the same time.

To validate an environment, a typical approach would be:

• Use a storage benchmarking tool, such as Bonnie++, while the file store is
not in use to validate that the IOPS provided are robust.
• Use network testing methods to determine that the roundtrip time between
search heads and the storage system is on the order of 10ms.
• Perform known simple tasks such as creating a million files and then
deleting them.
• Assuming the above tests have not shown any weaknesses, perform
some IO load generation or run the actual Splunk Enterprise load while
gathering NFS stat data to see what's happening with the NFS requests.

NFS client concurrency limits can cause search timeouts or

slow search behavior

The search performance in a search head pool is a function of the throughput of

the shared storage and the search workload. The combined effect of concurrent
search users and concurrent scheduled searches running will yield a total IOPs
that the shared volume needs to support. IOP requirements will also vary by the
kind of searches run. To adequately provision a device to be shared between
search heads, you need to know the number of concurrent users submitting
searches and the number of jobs/apps that will be executed simultaneously.

If searches are timing out or running slowly, you might be exhausting the
maximum number of concurrent requests supported by the NFS client. To solve
this problem, increase your client concurrency limit. For example, on a Linux NFS
client, adjust the tcp_slot_table_entries setting.

NFS latency for large user count can incur configuration

access latency or slow dispatch reaping

Splunk Enterprise synchronizes the search head pool storage configuration state
with the in-memory state when it detects changes. Essentially, it reads the
configuration into memory when it detects updates. When dealing either with

45
overloaded search pool storage or with large numbers of users, apps, and
configuration files, this synchronization process can reduce performance. To
mitigate this, the minimum frequency of reading can be increased, as discussed
in "Select timing for configuration refresh".

Warning about unique serverName attribute

Each search head in the pool must have a unique serverName attribute. Splunk
Enterprise validates this condition when each search head starts. If it finds a
problem, it generates this error message:

serverName "<xxx>" has already been claimed by a member of this search

head pool
in <full path to pooling.ini on shared storage>
There was an error validating your search head pooling configuration.
For more
information, run 'splunk pooling validate'

The most common cause of this error is that another search head in the pool is
already using the current search head's serverName. To fix the problem, change
the current search head's serverName attribute in .system/local/server.conf.

There are a few other conditions that also can generate this error:

• The current search head's serverName has been changed.

• The current search head's GUID has been changed. This is usually due to
/etc/instance.cfg being deleted.

To fix these problems, run

splunk pooling replace-member

This updates the pooling.ini file with the current search head's
serverName->GUID mapping, overwriting any previous mapping.

Artifacts and incorrectly displayed items in Splunk Web after

upgrade

When upgrading pooled search heads, you must copy all updated apps - even
those that ship with Splunk Enterprise (such as the Search app and the data
preview feature, which is implemented as an app) - to the search head pool's
shared storage after the upgrade is complete. If you do not, you might see

46
artifacts or other incorrectly-displayed items in Splunk Web.

To fix the problem, copy all updated apps from an upgraded search head to the
shared storage for the search head pool, taking care to exclude the local
sub-directory of each app.

Important: Excluding the local sub-directory of each app from the copy process
prevents the overwriting of configuration files on the shared storage with local
copies of configuration files.

Once the apps have been copied, restart Splunk Enterprise on all search heads
in the pool.

Distributed search error messages

This table lists some of the more common search-time error messages
associated with distributed search:

Error message Meaning

status=down The specified remote peer is not available.
status=not a splunk The specified remote peer is not a Splunk
server Enterprise server.
The specified remote peer is using a duplicate
duplicate license
license.
certificate mismatch Authentication with the specified remote peer failed.

47