sharding – ElasticSearch: Unassigned Shards, how to fix?

sharding – ElasticSearch: Unassigned Shards, how to fix?

By default, Elasticsearch will re-assign shards to nodes dynamically. However, if youve disabled shard allocation (perhaps you did a rolling restart and forgot to re-enable it), you can re-enable shard allocation.

# v0.90.x and earlier
curl -XPUT localhost:9200/_settings -d {
    index.routing.allocation.disable_allocation: false
}

# v1.0+
curl -XPUT localhost:9200/_cluster/settings -d {
    transient : {
        cluster.routing.allocation.enable : all
    }
}

Elasticsearch will then reassign shards as normal. This can be slow, consider raising indices.recovery.max_bytes_per_sec and cluster.routing.allocation.node_concurrent_recoveries to speed it up.

If youre still seeing issues, something else is probably wrong, so look in your Elasticsearch logs for errors. If you see EsRejectedExecutionException your thread pools may be too small.

Finally, you can explicitly reassign a shard to a node with the reroute API.

# Suppose shard 4 of index my-index is unassigned, so you want to
# assign it to node search03:
curl -XPOST localhost:9200/_cluster/reroute -d {
    commands: [{
        allocate: {
            index: my-index,
            shard: 4,
            node: search03,
            allow_primary: 1
        }
    }]
}

OK, Ive solved this with some help from ES support. Issue the following command to the API on all nodes (or the nodes you believe to be the cause of the problem):

curl -XPUT localhost:9200/<index>/_settings 
    -d {index.routing.allocation.disable_allocation: false}

where <index> is the index you believe to be the culprit. If you have no idea, just run this on all nodes:

curl -XPUT localhost:9200/_settings 
    -d {index.routing.allocation.disable_allocation: false}

I also added this line to my yaml config and since then, any restarts of the server/service have been problem free. The shards re-allocated back immediately.

FWIW, to answer an oft sought after question, set MAX_HEAP_SIZE to 30G unless your machine has less than 60G RAM, in which case set it to half the available memory.

References

sharding – ElasticSearch: Unassigned Shards, how to fix?

This little bash script will brute force reassign, you may lose data.

NODE=YOUR NODE NAME
IFS=$n
for line in $(curl -s localhost:9200/_cat/shards | fgrep UNASSIGNED); do
  INDEX=$(echo $line | (awk {print $1}))
  SHARD=$(echo $line | (awk {print $2}))

  curl -XPOST localhost:9200/_cluster/reroute -d {
     commands: [
        {
            allocate: {
                index: $INDEX,
                shard: $SHARD,
                node: $NODE,
                allow_primary: true
          }
        }
    ]
  }
done

Leave a Reply

Your email address will not be published.