Elasticsearch未分配的碎片CircuitBreakingException [[parent]数据太大 [英] Elasticsearch unassigned shards CircuitBreakingException[[parent] Data too large
问题描述
我收到警告，指出elasticsearch有2个未分配的分片.我在下面的api调用中进行了收集，以收集更多细节.
I got alert stating elasticsearch has 2 unassigned shards. I made below api calls to gather more details.
curl -s http://localhost:9200/_cluster/allocation/explain | python -m json.tool
下面的输出
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"can_allocate": "no",
"current_state": "unassigned",
"index": "docs_0_1603929645264",
"node_allocation_decisions": [
{
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP}{IP:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested:
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
}
],
"node_decision": "no",
"node_id": "1XEXS92jTK-asdfasdfasdf",
"node_name": "elasticsearch-data-2",
"transport_address": "IP1:9300"
},
{
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested:
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
},
{
"decider": "same_shard",
"decision": "NO",
"explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[docs_0_1603929645264][0], node[fIaSuZsNTwODgZnt90f7kQ], [P], s[STARTED], a[id=stHnyqjLQ7OwFbaqs5vWqA]]"
}
],
"node_decision": "no",
"node_id": "fIaSuZsNTwODgZnt90f7kQ",
"node_name": "elasticsearch-data-1",
"transport_address": "IP1:9300"
},
{
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-10-30T06:10:16.305Z], failed_attempts[5], delayed=false, details[failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{Ip2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested:
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ], allocation_status[no_attempt]]]"
}
],
"node_decision": "no",
"node_id": "o_9jyrmOSca9T12J4bY0Nw",
"node_name": "elasticsearch-data-0",
"transport_address": "IP2:9300"
}
],
"primary": false,
"shard": 0,
"unassigned_info": {
"at": "2020-10-30T06:10:16.305Z",
"details": "failed shard on node [o_9jyrmOSca9T12J4bY0Nw]: failed recovery, failure RecoveryFailedException[[docs_0_1603929645264][0]: Recovery failed from {elasticsearch-data-1}{fIaSuZsNTwODgZnt90f7kQ}{Qxl9iPacQVS-tN_t4YJqrw}{IP1}{IP1:9300} into {elasticsearch-data-0}{o_9jyrmOSca9T12J4bY0Nw}{1w5mgwy0RYqBQ9c-qA_6Hw}{IP2}{IP2:9300}]; nested: RemoteTransportException[[elasticsearch-data-1][IP1:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [129] files with total size of [4.4gb]]; nested: RemoteTransportException[[elasticsearch-data-0][IP2:9300][internal:index/shard/recovery/file_chunk]]; nested:
CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [1972835086/1.8gb], which is larger than the limit of [1972122419/1.8gb], real usage: [1972833976/1.8gb], new bytes reserved: [1110/1kb]]; ",
"failed_allocation_attempts": 5,
"last_allocation_status": "no_attempt",
"reason": "ALLOCATION_FAILED"
}
}
我查询了断路器配置
curl -X GET "localhost:9200/_nodes/stats/breaker?pretty
并且可以看到3个节点(elasticsearch-data-0，elasticsearch-data-1和elasticsearch-data-2)的父limit_size_in_byes如下.
And can see that the parent limit_size_in_byes of 3 nodes (elasticsearch-data-0, elasticsearch-data-1 and elasticsearch-data-2) is as below.
"parent" : {
"limit_size_in_bytes" : 1972122419,
"limit_size" : "1.8gb",
"estimated_size_in_bytes" : 1648057776,
"estimated_size" : "1.5gb",
"overhead" : 1.0,
"tripped" : 139
}
我提到了此答案 https://stackoverflow.com/a/61954408 ，并计划增加其中一个的内存百分比断路器或整个JVM堆.
I referred to this answer https://stackoverflow.com/a/61954408 and planning to increase either the memory percentage of the circuit breaker or overall JVM heap.
这是一个k8s环境，elasticsearch-data部署为具有3个副本的有状态集.当我对状态集进行描述时，我可以看到下面定义的ENV变量
This is a k8s environment and the elasticsearch-data is deployed as a statefulset with 3 replicas. When I did a describe of the statefulset, I can see the below ENV variable defined
Containers:
elasticsearch:
Image: custom/elasticsearch-oss-s3:7.0.0
Port: 9300/TCP
Host Port: 0/TCP
Limits:
cpu: 10500m
memory: 21Gi
Requests:
cpu: 10
memory: 20Gi
Environment:
DISCOVERY_SERVICE: elasticsearch-discovery
NODE_MASTER: false
PROCESSORS: 11 (limits.cpu)
ES_JAVA_OPTS: -Djava.net.preferIPv4Stack=true -Xms2048m -Xmx2048m
按此，堆大小似乎是2048m
As per this, the heap size seems to be 2048m
我登录了elasticsearch-data窗格，并在弹性搜索配置目录下看到以下文件
I logged into a elasticsearch-data pod and under the elastic search config directory I see below files
elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties repository-s3
elasticsearch.yml没有任何堆配置.它只是具有主节点的名称等.
elasticsearch.yml doesn't have any heap config or as such. It just had the master nodes' name etc..
下面是jvm.options文件
Below is the jvm.options file
## JVM configuration
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms1g
-Xmx1g
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## DNS cache policy
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# explicitly set the stack size
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT
从上面看，似乎堆的总大小为1g.
From above, it seems the total heap size is 1g.
但是从此Pod的有状态集中定义的env变量来看，它似乎是2048m.
But from the env variable defined in the stateful set of this pod, it seems to be 2048m.
哪个是对的?
现在，通过下面的链接
可以使用以下设置来配置父级断路器:
The parent-level breaker can be configured with the following settings:
indices.breaker.total.use_real_memory (静态)确定父断路器应该考虑实际内存使用情况(是)(真)还是仅考虑子断路器保留的数量(假).默认为true.
indices.breaker.total.use_real_memory (Static) Determines whether the parent breaker should take real memory usage into account (true) or only consider the amount that is reserved by child circuit breakers (false). Defaults to true.
indices.breaker.total.limit (动态)整个父级断路器的启动限制.如果index.breaker.total.use_real_memory为false，则默认为JVM堆的70％.如果indexs.breaker.total.use_real_memory为true，则默认为JVM堆的95％.
indices.breaker.total.limit (Dynamic) Starting limit for overall parent breaker. Defaults to 70% of JVM heap if indices.breaker.total.use_real_memory is false. If indices.breaker.total.use_real_memory is true, defaults to 95% of the JVM heap.
但是错误和我查询的断路器统计信息中的极限值是1972122419字节(1.8G).这似乎不是2048M的95％或1g.
But the limit value in the error and in the breaker stats I queried is this - 1972122419 bytes (1.8G). This doesn't seem to be 95% of 2048M or 1g.
现在，如何增加断路器父级的堆或内存限制，以便摆脱该错误?
Now, how can I increase the heap or the memory limit of breaker parent so that I can get rid of this error?
推荐答案
这里有两件事，分片分配异常和断路器异常(看上去是嵌套异常).
two things here, shard allocation exception and circuit breaker exception(nested exception as it looks).
请使用以下命令在集群中重新触发分配，因为之前所有重试均失败，并且如果您仔细注意，则在异常消息中建议使用相同的命令.有关以下命令的更多信息，请参见相关的Github问题评论.
Please use the below command in your cluster to re-trigger the allocation as previous all retry was failed and the same is suggested in your exception message if you carefully notice. more info on below command is on this related Github issue comment.
curl -XPOST':9200/_cluster/reroute?retry_failed
curl -XPOST ':9200/_cluster/reroute?retry_failed
如果仍然无法运行，则必须修复父级断路器异常，应使用http://localhost:9200/_nodes/stats API知道ES节点的确切堆，并相应地增加它.
If still, it doesn't work, then you have to fix the parent circuit breaker exception, you should use the http://localhost:9200/_nodes/stats API to know the exact heap of your ES nodes, and accordingly increase it.
这篇关于Elasticsearch未分配的碎片CircuitBreakingException [[parent]数据太大的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
