Elasticsearch read_only_allow_delete auto setting
In an attempt to add a sprinkling of value to the accepted answer (and because i'll google this and come back in future), for my case the read_only_allow_delete flag was set because of the default settings for disk watermark being percentage based - which on my large disk did not make as much sense. So I changed these settings to be "size remaining" based as the documentation explains.
So before setting read_only_allow_delete back to false, I first set the watermark values based on disk space:
(using Kibana UI):
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "20gb",
"cluster.routing.allocation.disk.watermark.high": "15gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
}
}
PUT your_index_name/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
OR (through the terminal):
$ curl -XPUT -H "Content-Type: application/json" \
http://localhost:9200/_cluster/_settings \
-d '{"cluster.routing.allocation.disk.watermark.low": "20gb",
"cluster.routing.allocation.disk.watermark.high": "15gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "10gb"}'
$ curl -XPUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d '{"index.blocks.read_only_allow_delete": false}'
Background
We maintain a cluster where we have filebeat, metricbeat, packetbeat, etc. shippers pushing data into the cluster. Invariably some index would become hot and we'd want to either disable writing to it for a time or do clean up and reenable indices which had breached their low watermark thresholds and had automatically gone into read_only_allow_delete: true.
Bash Functions
To ease the management of our clusters for the rest of my team I wrote the following Bash functions to help perform these tasks without having to fumble around with curl or through Kibana's UI.
$ cat es_funcs.bash
### es wrapper cmd inventory
declare -A escmd
escmd[l]="./esl"
escmd[p]="./esp"
### es data node naming conventions
nodeBaseName="rdu-es-data-0"
declare -A esnode
esnode[l]="lab-${nodeBaseName}"
esnode[p]="${nodeBaseName}"
usage_chk1 () {
# usage msg for cmds w/ 1 arg
local env="$1"
[[ $env =~ [lp] ]] && return 0 || \
printf "\nUSAGE: ${FUNCNAME[1]} [l|p]\n\n" && return 1
}
enable_readonly_idxs () {
# set read_only_allow_delete flag
local env="$1"
usage_chk1 "$env" || return 1
DISALLOWDEL=$(cat <<-EOM
{
"index": {
"blocks": {
"read_only_allow_delete": "true"
}
}
}
EOM
)
${escmd[$env]} PUT '_all/_settings' -d "$DISALLOWDEL"
}
disable_readonly_idxs () {
# clear read_only_allow_delete flag
local env="$1"
usage_chk1 "$env" || return 1
ALLOWDEL=$(cat <<-EOM
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
EOM
)
${escmd[$env]} PUT '_all/_settings' -d "$ALLOWDEL"
}
Example Run
The above functions can be sourced in your shell like so:
$ . es_funcs.bash
NOTE: The arrays at the top of the file map short names for clusters if you happen to have multiple. We have 2, one for our lab and one for our production. So I represented those as l and p.
You can then run them like this to enable the read_only_allow_delete attribute (true) on your l cluster:
$ enable_readonly_idxs l
{"acknowledged":true}
or p:
$ enable_readonly_idxs p
{"acknowledged":true}
Helper Script Overview
There's one additional script that contains the curl commands which I use to interact with the clusters. This script is referenced in the escmd array at the top of the es_func.bash file. The array contains names of symlinks to a single shell script, escli.bash. The links are called esl and esp.
$ ll
-rw-r--r-- 1 smingolelli staff 9035 Apr 10 23:38 es_funcs.bash
-rwxr-xr-x 1 smingolelli staff 1626 Apr 10 23:02 escli.bash
-rw-r--r-- 1 smingolelli staff 338 Apr 5 00:27 escli.conf
lrwxr-xr-x 1 smingolelli staff 10 Jan 23 08:12 esl -> escli.bash
lrwxr-xr-x 1 smingolelli staff 10 Jan 23 08:12 esp -> escli.bash
The escli.bash script:
$ cat escli.bash
#!/bin/bash
#------------------------------------------------
# Detect how we were called [l|p]
#------------------------------------------------
[[ $(basename $0) == "esl" ]] && env="lab1" || env="rdu1"
#------------------------------------------------
# source escli.conf variables
#------------------------------------------------
# g* tools via brew install coreutils
[ $(uname) == "Darwin" ] && readlink=greadlink || readlink=readlink
. $(dirname $($readlink -f $0))/escli.conf
usage () {
cat <<-EOF
USAGE: $0 [HEAD|GET|PUT|POST] '...ES REST CALL...'
EXAMPLES:
$0 GET '_cat/shards?pretty'
$0 GET '_cat/indices?pretty&v&human'
$0 GET '_cat'
$0 GET ''
$0 PUT '_all/_settings' -d "\$DATA"
$0 POST '_cluster/reroute' -d "\$DATA"
EOF
exit 1
}
[ "$1" == "" ] && usage
#------------------------------------------------
# ...ways to call curl.....
#------------------------------------------------
if [ "${1}" == "HEAD" ]; then
curl -I -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
"${esBaseUrl}/$2"
elif [ "${1}" == "PUT" ]; then
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
elif [ "${1}" == "POST" ]; then
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
else
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 "${esBaseUrl}/$2" "$3" "$4" "$5"
fi
This script takes a single property file, escli.conf. In this file you specify the commands to retrieve your username + password from whereever, I use LastPass for that so retrieve them via lpass as well as setting the base URL to use for accessing your clusters REST API.
$ cat escli.conf
#################################################
### props used by escli.bash
#################################################
usernameCmd='lpass show --username somedom.com'
passwordCmd='lpass show --password somedom.com'
esBaseUrl="https://es-data-01a.${env}.somdom.com:9200"
contType="Content-Type: application/json"
I've put all this together in a Github repo (linked below) which also includes additional functions beyond the above 2 that I'm showing as examples for this question.
References
- https://github.com/slmingol/escli
Elasticsearch automatically sets "read_only_allow_delete": "true" when hard disk space is low.
Find the files which are filling up your storage and delete/move them. Once you have sufficient storage available run the following command through the Dev Tool in Kibana:
PUT your_index_name/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
OR (through the terminal):
$ curl -XPUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d '{"index.blocks.read_only_allow_delete": false}'
as mentioned in your question.