更新cronjob自动清理 Failed pod

增加飞书通知

当有 Failed pod 被清理时，飞书发送通知信息。

更新清理脚本 ConfigMap

首先将清理脚本存储为ConfigMap，便于维护和版本控制：

# cat cleaner-configmap-V2.yamlapiVersion: v1
kind: ConfigMap
metadata:name: failed-pod-cleaner-scriptnamespace: middle
data:clean_script.sh: |#!/bin/bashset -eo pipefail# ===== 配置部分 =====FEISHU_WEBHOOK_URL="https://open.feishu.cn/open-apis/bot/v2/hook/46e8888c-c004-4d6e-afb7-12345cd94b76"  # 请替换为真实URLCLEAN_THRESHOLD_MINUTES=30# ===== 函数定义 =====send_feishu_notification() {local namespace=$1local pod_name=$2local operation_time=$3local message=$(cat <<EOF{"msg_type": "text","content": {"text": "Kubernetes Failed Pod 清理通知\n------------------------\n操作类型: 自动清理\n命名空间: ${namespace}\nPod名称: ${pod_name}\n操作时间: ${operation_time}\n执行主机: $(hostname)"}}EOF)curl -X POST "$FEISHU_WEBHOOK_URL" \-H "Content-Type: application/json" \-d "$message" \--connect-timeout 5 \--silent > /dev/null}# ===== 主逻辑 =====TIMESTAMP=$(date -d "${CLEAN_THRESHOLD_MINUTES} minutes ago" -u +'%Y-%m-%dT%H:%M:%SZ')echo "[$(date)] 开始清理超过 ${CLEAN_THRESHOLD_MINUTES} 分钟的Failed Pod..."FAILED_PODS=$(kubectl get pods --all-namespaces \--field-selector=status.phase=Failed \--no-headers \-o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,TIME:.status.conditions[?(@.type=='Ready')].lastTransitionTime" \| awk -v limit="${TIMESTAMP}" '$3 < limit && $3 != "" {print $1,$2}')if [ -n "${FAILED_PODS}" ]; thenecho "[$(date)] 发现待清理Pod:"# 使用进程替换避免子shell问题while read -r ns pod; doecho " - 命名空间: $ns, Pod: $pod"start_time=$(date +%s.%N)if kubectl delete pod -n "$ns" "$pod" --grace-period=0; thenend_time=$(date +%s.%N)duration=$(printf "%.2f" $(echo "$end_time - $start_time" | bc))send_feishu_notification "$ns" "$pod" "$(date '+%Y-%m-%d %H:%M:%S')" "$duration" &elseecho "[错误] 删除失败: $ns/$pod" | tee -a /var/log/pod_cleaner.error.logfidone <<< "${FAILED_PODS}"  # Here-string输入wait # 等待所有后台通知完成elseecho "[$(date)] 未发现需要清理的Pod"fiecho "[$(date)] 清理操作完成"

更新镜像

安装curl，bc基础包

Dockerfile

FROM bitnami/kubectl:latest# 使用 UID 0 代替 USER root
USER 0# 安装所需工具
RUN apt-get update && \apt-get install -y bc curl && \rm -rf /var/lib/apt/lists/*# 切换回原始非 root 用户 (UID 1001)
USER 1001

打包镜像

docker build -t registry.cn-shanghai.aliyuncs.com/tengfeiwu/custom-kubectl:latest .

推送镜像

docker push registry.cn-shanghai.aliyuncs.com/tengfeiwu/custom-kubectl:latest

更新CronJob资源

创建具有适当RBAC权限的CronJob：

# cat cleaner-cronjob.yamlapiVersion: batch/v1
kind: CronJob
metadata:name: failed-pod-cleanernamespace: middle
spec:schedule: "0 * * * *"  # 每小时整点执行concurrencyPolicy: Forbid  # 禁止并发执行jobTemplate:spec:template:spec:serviceAccountName: pod-cleaner  # 使用专用ServiceAccountcontainers:- name: cleanerimage: registry.cn-shanghai.aliyuncs.com/tengfeiwu/custom-kubectl:latestcommand: ["/bin/bash", "/scripts/clean_script.sh"]volumeMounts:- name: cleanup-scriptmountPath: /scriptsrestartPolicy: OnFailurevolumes:- name: cleanup-scriptconfigMap:name: failed-pod-cleaner-scriptdefaultMode: 0744  # 赋予执行权限

更新服务

# 清理之前的cronjob
kubectl delete -f cleaner-configmap.yaml
kubectl delete -f cleaner-cronjob.yaml
# 创建最新的cronjob
kubectl apply -f cleaner-configmap-V2.yaml
kubectl apply -f cleaner-cronjob.yaml

测试及验证

创建测试pod

创建测试使用的 Failed pod

kubectl run test-failed --image=busybox --restart=Never -- /bin/false
kubectl run ceshi-failed --image=busybox --restart=Never -- /bin/sh -c "exit 1"

手动触发清理任务

kubectl -n middle create job --from=cronjob/failed-pod-cleaner manual-test

查看执行日志:

kubectl logs -n middle jobs/manual-test

输出内容：

[Thu Jul 31 06:08:15 UTC 2025] 开始清理超过 2 分钟的Failed Pod...
[Thu Jul 31 06:08:15 UTC 2025] 发现待清理Pod:- 命名空间: default, Pod: ceshi-failed
pod "ceshi-failed" deleted- 命名空间: default, Pod: test-failed
pod "test-failed" deleted
[Thu Jul 31 06:08:16 UTC 2025] 清理操作完成