[[index-modules-allocation]]
== Index Shard Allocation

[float]
[[shard-allocation-filtering]]
=== Shard Allocation Filtering

Allows to control the allocation of indices on nodes based on include/exclude
filters. The filters can be set both on the index level and on the
cluster level. Lets start with an example of setting it on the cluster
level:

Lets say we have 4 nodes, each has specific attribute called `tag`
associated with it (the name of the attribute can be any name). Each
node has a specific value associated with `tag`. Node 1 has a setting
`node.tag: value1`, Node 2 a setting of `node.tag: value2`, and so on.

We can create an index that will only deploy on nodes that have `tag`
set to `value1` and `value2` by setting
`index.routing.allocation.include.tag` to `value1,value2`. For example:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.include.tag" : "value1,value2"
}'
--------------------------------------------------

On the other hand, we can create an index that will be deployed on all
nodes except for nodes with a `tag` of value `value3` by setting
`index.routing.allocation.exclude.tag` to `value3`. For example:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.exclude.tag" : "value3"
}'
--------------------------------------------------

`index.routing.allocation.require.*` can be used to
specify a number of rules, all of which MUST match in order for a shard
to be allocated to a node. This is in contrast to `include` which will
include a node if ANY rule matches.

The `include`, `exclude` and `require` values can have generic simple
matching wildcards, for example, `value1*`. Additionally, special attribute
names called `_ip`, `_name`, `_id` and `_host` can be used to match by node
ip address, name, id or host name, respectively.

Obviously a node can have several attributes associated with it, and
both the attribute name and value are controlled in the setting. For
example, here is a sample of several node configurations:

[source,js]
--------------------------------------------------
node.group1: group1_value1
node.group2: group2_value4
--------------------------------------------------

In the same manner, `include`, `exclude` and `require` can work against
several attributes, for example:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.include.group1" : "xxx"
    "index.routing.allocation.include.group2" : "yyy",
    "index.routing.allocation.exclude.group3" : "zzz",
    "index.routing.allocation.require.group4" : "aaa",
}'
--------------------------------------------------

The provided settings can also be updated in real time using the update
settings API, allowing to "move" indices (shards) around in realtime.

Cluster wide filtering can also be defined, and be updated in real time
using the cluster update settings API. This setting can come in handy
for things like decommissioning nodes (even if the replica count is set
to 0). Here is a sample of how to decommission a node based on `_ip`
address:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
    }
}'
--------------------------------------------------

[[delayed-allocation]]
=== Delaying allocation when a node leaves

When a node leaves the cluster for whatever reason, intentional or otherwise,
the master reacts by:

* Promoting a replica shard to primary to replace any primaries that were on the node.
* Allocating replica shards to replace the missing replicas (assuming there are enough nodes).
* Rebalancing shards evenly across the remaining nodes.

These actions are intended to protect the cluster against data loss by
ensuring that every shard is fully replicated as soon as possible.

Even though we throttle concurrent recoveries both at the
<<recovery,node level>> and at the <<shards-allocation,cluster level>>, this
``shard-shuffle'' can still put a lot of extra load on the cluster which
may not be necessary if the missing node is likely to return soon. Imagine
this scenario:

* Node 5 loses network connectivity.
* The master promotes a replica shard to primary for each primary that was on Node 5.
* The master allocates new replicas to other nodes in the cluster.
* Each new replica makes an entire copy of the primary shard across the network.
* More shards are moved to different nodes to rebalance the cluster.
* Node 5 returns after a few minutes.
* The master rebalances the cluster by allocating shards to Node 5.

If the master had just waited for a few minutes, then the missing shards could
have been re-allocated to Node 5 with the minimum of network traffic.  This
process would be even quicker for idle shards (shards not receiving indexing
requests) which have been automatically <<indices-synced-flush,sync-flushed>>.

The allocation of replica shards which become unassigned because a node has
left can be delayed with the `index.unassigned.node_left.delayed_timeout`
dynamic setting, which defaults to `1m`.

This setting can be updated on a live index (or on all indices):

[source,js]
------------------------------
PUT /_all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}
------------------------------
// AUTOSENSE

With delayed allocation enabled, the above scenario changes to look like this:

* Node 5 loses network connectivity.
* The master promotes a replica shard to primary for each primary that was on Node 5.
* The master logs a message that allocation of unassigned shards has been delayed, and for how long.
* The cluster remains yellow because there are unassigned replica shards.
* Node 5 returns after a few minutes, before the `timeout` expires.
* The missing replicas are re-allocated to Node 5 (and sync-flushed shards recover almost immediately).

NOTE: This setting will not affect the promotion of replicas to primaries, nor
will it affect the assignment of replicas that have not been assigned
previously.

[float]
==== Monitoring delayed unassigned shards

The number of shards whose allocation has been delayed by this timeout setting
can be viewed with the <<cluster-health,cluster health API>>:

[source,js]
------------------------------
GET _cluster/health <1>
------------------------------
<1> This request will return a `delayed_unassigned_shards` value.

[float]
==== Removing a node permanently

If a node is not going to return and you would like Elasticsearch to allocate
the missing shards immediately, just update the timeout to zero:


[source,js]
------------------------------
PUT /_all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "0"
  }
}
------------------------------
// AUTOSENSE

You can reset the timeout as soon as the missing shards have started to recover.

[[recovery-prioritization]]
=== Index recovery prioritization

Unallocated shards are recovered in order of priority, whenever possible.
Indices are sorted into priority order as follows:

* the optional `index.priority` setting (higher before lower)
* the index creation date (higher before lower)
* the index name (higher before lower)

This means that, by default, newer indices will be recovered before older indices.

Use the per-index dynamically updateable `index.priority` setting to customise
the index prioritization order.  For instance:

[source,json]
------------------------------
PUT index_1

PUT index_2

PUT index_3
{
  "settings": {
    "index.priority": 10
  }
}

PUT index_4
{
  "settings": {
    "index.priority": 5
  }
}
------------------------------
// AUTOSENSE

In the above example:

* `index_3` will be recovered first because it has the highest `index.priority`.
* `index_4` will be recovered next because it has the next highest priority.
* `index_2` will be recovered next because it was created more recently.
* `index_1` will be recovered last.

This setting accepts an integer, and can be updated on a live index with the
<<indices-update-settings,update index settings API>>:

[source,json]
------------------------------
PUT index_4/_settings
{
  "index.priority": 1
}
------------------------------
// AUTOSENSE

=== Total Shards Per Node

The `index.routing.allocation.total_shards_per_node` setting allows to
control how many total shards (replicas and primaries) for an index will be allocated per node.
It can be dynamically set on a live index using the update index
settings API.

[[disk]]
=== Disk-based Shard Allocation

Elasticsearch can be configured to prevent shard
allocation on nodes depending on disk usage for the node. This
functionality is enabled by default, and can be changed either in the
configuration file, or dynamically using:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'
--------------------------------------------------

Once enabled, Elasticsearch uses two watermarks to decide whether
shards should be allocated or can remain on the node.

`cluster.routing.allocation.disk.watermark.low` controls the low
watermark for disk usage. It defaults to 85%, meaning ES will not
allocate new shards to nodes once they have more than 85% disk
used. It can also be set to an absolute byte value (like 500mb) to
prevent ES from allocating shards if less than the configured amount
of space is available.

`cluster.routing.allocation.disk.watermark.high` controls the high
watermark. It defaults to 90%, meaning ES will attempt to relocate
shards to another node if the node disk usage rises above 90%. It can
also be set to an absolute byte value (similar to the low watermark)
to relocate shards once less than the configured amount of space is
available on the node.

NOTE: Percentage values refer to used disk space, while byte values refer to
free disk space. This can be confusing, since it flips the meaning of
high and low. For example, it makes sense to set the low watermark to 10gb
and the high watermark to 5gb, but not the other way around.

Both watermark settings can be changed dynamically using the cluster
settings API. By default, Elasticsearch will retrieve information
about the disk usage of the nodes every 30 seconds. This can also be
changed by setting the `cluster.info.update.interval` setting.

An example of updating the low watermark to no more than 80% of the disk size, a
high watermark of at least 50 gigabytes free, and updating the information about
the cluster every minute:

[source,js]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.watermark.low" : "80%",
        "cluster.routing.allocation.disk.watermark.high" : "50gb",
        "cluster.info.update.interval" : "1m"
    }
}'
--------------------------------------------------

By default, Elasticsearch will take into account shards that are currently being
relocated to the target node when computing a node's disk usage. This can be
changed by setting the `cluster.routing.allocation.disk.include_relocations`
setting to `false` (defaults to `true`). Taking relocating shards' sizes into
account may, however, mean that the disk usage for a node is incorrectly
estimated on the high side, since the relocation could be 90% complete and a
recently retrieved disk usage would include the total size of the relocating
shard as well as the space already used by the running relocation.
