Kubernetes 命名空间卡在终止状态 [英] Kubernetes Namespaces stuck in Terminating status

问题描述

我们有一个集群，其中的命名空间似乎永远不想完全删除，现在无法重新创建自定义指标命名空间，以便能够收集自定义指标以正确设置 HPA.我完全理解我可以使用所有自定义指标资源创建另一个命名空间，但考虑到命名空间陷入终止"状态，我有点担心集群的整体健康状况

$ kubectl get ns姓名 身份 年龄证书管理器 Active 14d自定义指标终止 7d默认活动 222dnfs-share Active 15d入口-nginx 活动 103dkube-public 活动 222dkube-system Active 222dlb 终止 4d监控 终止 6d生产活性 221d

我已经尝试将命名空间导出到 JSON、删除终结器并使用编辑过的 JSON 文件重新创建.还尝试 kubectl edit ns custom-metrics 并删除-kubernetes"终结器.一切都无济于事.

有没有人对我如何尝试销毁这些卡住"的命名空间有任何其他建议

curl 到 https://master-ip/api/v1/namespace/.../finalize 对我来说似乎不适用于 Google Kubernetes Engine，我假设这些操作在 GKE 集群上是不允许的

尝试类似的方法效果不佳:

$ kubectl delete ns custom-metrics --grace-period=0 --force

<块引用>

警告:立即删除不等待确认正在运行的资源已终止.资源可能会继续运行无限期地在集群上.服务器错误(冲突):操作无法在命名空间custom-metrics"上实现:系统是确保从此命名空间中删除所有内容.完成后，系统会自动清除此命名空间.

并且在此命名空间中根本没有列出任何资源:kubectl get all -n custom-metrics 或遍历此命名空间中的所有 api-resources 显示根本不存在资源:kubectl api-resources --namespaced=true -o name |xargs -n 1 kubectl get -n custom-metrics

解决方案

我做了一些类似于 rahul.tripathi 的事情，除了 curl 对我不起作用 - 我遵循了 https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e 执行以下操作:

NAMESPACE=kubectl 获取命名空间 $NAMESPACE -o json >$NAMESPACE.jsonsed -i -e 's/kubernetes"//' $NAMESPACE.jsonkubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize";-f ./$NAMESPACE.json

瞧！命名空间被删除

We have one cluster where it seems that namespaces never want to be deleted completely and now can't re-create custom-metrics namespace to be able to collect custom metrics to properly setup HPA. I fully understand that I can create another namespace with all custom-metrics resources, but a little concerned with the overall health of the cluster, given that the namespaces get stuck in "Terminating" state

$ kubectl get ns
NAME             STATUS        AGE
cert-manager     Active        14d
custom-metrics   Terminating   7d
default          Active        222d
nfs-share        Active        15d
ingress-nginx    Active        103d
kube-public      Active        222d
kube-system      Active        222d
lb               Terminating   4d
monitoring       Terminating   6d
production       Active        221d

I already tried to export namespaces to JSON, delete finalizers and re-create using edited JSON files. also tried to kubectl edit ns custom-metrics and delete "- kubernetes" finalizer. all to no avail.

does anyone have any other recommendations on how else I can try to destroy these "stuck" namespaces"

curl to https://master-ip/api/v1/namespace/...../finalize doesn't seem to work on Google Kubernetes Engine for me, I'm assuming these operations are not allowed on GKE cluster

Trying things like doesn't work as well:

$ kubectl delete ns custom-metrics --grace-period=0 --force

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "custom-metrics": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

and there're no resources listed in this namespaces at all: kubectl get all -n custom-metrics or looping through all api-resources in this namespace shows no resources exist at all: kubectl api-resources --namespaced=true -o name | xargs -n 1 kubectl get -n custom-metrics

解决方案

I did something similar to rahul.tripathi except the curl did not work for me - I followed https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e which does the following:

NAMESPACE=
kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

Voila! Namespace is deleted

这篇关于Kubernetes 命名空间卡在终止状态的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！