zenjia

1.yml

@@ -725,4 +725,58 @@ pipeline {
     **告警描述:** 机器内存使用率超过设定阈值，触发告警
     {{if $event.RuleNote }}**规则说明:** {{$event.RuleNote}}{{end}}   
     {{if $event.Threshold}}**告警阈值:** {{$event.Threshold | humanizeSize}}{{end}}
-{{- end -}}
+{{- end -}}
+
+
+
+
+
+
+
+分析一下这个内存变化：(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi        15Gi       133Mi       1.0Mi       237Mi        98Mi
+Swap:          8.0Gi       8.0Gi       0.0Ki
+(base) [root@prod-lessie-server02 ~]#ps aux|grep 7001
+root      158009  0.0  0.0  35908 10420 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158011  0.8 90.3 22549000 14317120 ?   Sl   Oct21  24:39 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 117868 ?      Sl   Oct21   9:26 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 243708 ?      Sl   Oct21  12:08 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 260252 ?      Sl   Oct21  13:51 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39116 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404684 119708 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.1 2404716 31388 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028724  0.0  0.0   6408   384 pts/0    D+   18:37   0:00 grep --color=auto 7001
+(base) [root@prod-lessie-server02 ~]#ps aux|grep 7001
+root      158009  0.0  0.0  35908 11316 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 119916 ?      Sl   Oct21   9:27 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 246652 ?      Sl   Oct21  12:11 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 261532 ?      Sl   Oct21  13:53 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39244 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404684 119324 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.2 2404716 31772 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028770  0.0  0.0   6408  2176 pts/0    S+   18:39   0:00 grep --color=auto 7001
+(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi       2.0Gi        12Gi       1.0Mi       961Mi        13Gi
+Swap:          8.0Gi       3.9Gi       4.1Gi
+(base) [root@prod-lessie-server02 ~]#
+(base) [root@prod-lessie-server02 ~]#ps auxf|grep 7001
+root     1029136  0.0  0.0   6408  2304 pts/0    S+   18:39   0:00  |       |   \_ grep --color=auto 7001
+root      158009  0.0  0.0  35908 14004 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 121836 ?      Sl   Oct21   9:27  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39244 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 248700 ?      Sl   Oct21  12:11  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.2 2404716 31772 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 264348 ?      Sl   Oct21  13:53  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028807 50.5  3.4 3214636 553780 ?      Sl   18:39   0:08  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1029008  0.0  2.9 2408940 466648 ?      Sl   18:39   0:00      \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404652 120604 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+(base) [root@prod-lessie-server02 ~]#
+(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi       2.1Gi        12Gi       1.0Mi       1.0Gi        13Gi
+Swap:          8.0Gi       3.9Gi       4.1Gi   从工作进程上分析，从内存占用大到恢复是什么原因

proc_monitor.sh

@@ -0,0 +1,151 @@
+#!/bin/bash
+# 优化版：监控 gunicorn 主进程及其子进程资源变化（表格+彩色输出）
+
+PROC_NAME="gunicorn"
+LOG_DIR="/data/sh/monitor/logs"
+LOG_FILE="$LOG_DIR/proc_monitor.log"
+DATA_FILE="$LOG_DIR/last_snapshot.txt"
+
+mkdir -p "$LOG_DIR"
+
+# 颜色定义
+RED='\033[1;31m'
+GREEN='\033[1;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[1;34m'
+NC='\033[0m' # 无色
+
+# 获取系统内存概况
+get_sys_mem() {
+  free -h | awk '/Mem:/ {printf "系统内存概况：总 %s | 已用 %s | 空闲 %s | 可用 %s", $2, $3, $4, $7}'
+}
+
+# 获取时间
+format_time() {
+  date "+%Y-%m-%d %H:%M:%S"
+}
+
+# 获取 gunicorn 主进程和子进程信息
+get_proc_info() {
+  ps -eo pid,ppid,comm,%mem,%cpu,rss --no-headers | awk -v pname="$PROC_NAME" '$3 == pname {print $1, $2, $3, $4, $5, $6}'
+}
+
+# 保存当前状态快照
+save_snapshot() {
+  get_proc_info | awk '{print $1,$3,$4,$5}' > "$DATA_FILE"
+}
+
+# 打印表格头
+print_table_header() {
+  printf "%-10s %-12s %-12s %-12s %-12s\n" "PID" "进程名" "内存(Gi)" "CPU(%)" "备注"
+}
+
+# 打印分隔线
+print_line() {
+  printf "%s\n" "---------------------------------------------------------------"
+}
+
+# 获取主进程 PID
+MAIN_PID=$(pgrep -f "$PROC_NAME" | head -n 1)
+if [ -z "$MAIN_PID" ]; then
+  echo -e "${RED}未找到进程：$PROC_NAME${NC}"
+  exit 1
+fi
+
+TIMESTAMP=$(format_time)
+echo "==== $TIMESTAMP ====" >> "$LOG_FILE"
+echo "$(get_sys_mem)" >> "$LOG_FILE"
+echo "主进程名：$PROC_NAME | PID: $MAIN_PID" >> "$LOG_FILE"
+
+# 写表格头
+print_table_header >> "$LOG_FILE"
+print_line >> "$LOG_FILE"
+
+# 获取当前进程信息
+CUR_INFO=$(get_proc_info)
+
+echo "$CUR_INFO" | while read -r pid ppid name mem cpu rss; do
+  mem_gi=$(awk "BEGIN {printf \"%.2f\", $rss/1024/1024}") # KB -> Gi
+  printf "%-10s %-12s %-12s %-12s\n" "$pid" "$name" "${mem_gi}Gi" "${cpu}%" >> "$LOG_FILE"
+done
+
+print_line >> "$LOG_FILE"
+
+# 若无上次记录，跳过变化分析
+if [ ! -f "$DATA_FILE" ]; then
+  echo "暂无上次记录，跳过变化分析。" >> "$LOG_FILE"
+  save_snapshot
+  echo "==== $TIMESTAMP ====" >> "$LOG_FILE"
+  exit 0
+fi
+
+# 对比分析部分
+declare -A CUR_MEM CUR_CPU PRE_MEM PRE_CPU
+
+while read -r pid name mem cpu; do
+  CUR_MEM["$pid"]=$mem
+  CUR_CPU["$pid"]=$cpu
+done < <(get_proc_info | awk '{print $1,$3,$4,$5}')
+
+while read -r pid name mem cpu; do
+  PRE_MEM["$pid"]=$mem
+  PRE_CPU["$pid"]=$cpu
+done < "$DATA_FILE"
+
+CUR_PIDS=$(printf "%s\n" "${!CUR_MEM[@]}" | sort)
+PRE_PIDS=$(printf "%s\n" "${!PRE_MEM[@]}" | sort)
+
+NEW_PIDS=$(comm -23 <(echo "$CUR_PIDS") <(echo "$PRE_PIDS"))
+OLD_PIDS=$(comm -13 <(echo "$CUR_PIDS") <(echo "$PRE_PIDS"))
+
+echo "分析变化：" >> "$LOG_FILE"
+
+# 新旧进程分析
+if [ -n "$NEW_PIDS" ]; then
+  echo -e "  ${GREEN}新出现的子进程:${NC}" >> "$LOG_FILE"
+  for pid in $NEW_PIDS; do
+    echo "    $PROC_NAME (PID:$pid)" >> "$LOG_FILE"
+  done
+fi
+
+if [ -n "$OLD_PIDS" ]; then
+  echo -e "  ${YELLOW}已消失的子进程:${NC}" >> "$LOG_FILE"
+  for pid in $OLD_PIDS; do
+    echo "    $PROC_NAME (PID:$pid)" >> "$LOG_FILE"
+  done
+fi
+
+# 资源变化分析
+echo "  子进程资源变化：" >> "$LOG_FILE"
+for pid in $CUR_PIDS; do
+  if [ -n "${PRE_MEM[$pid]}" ]; then
+    cpu_diff=$(awk "BEGIN {printf \"%.1f\", ${CUR_CPU[$pid]} - ${PRE_CPU[$pid]}}")
+    mem_diff=$(awk "BEGIN {printf \"%.2f\", ${CUR_MEM[$pid]} - ${PRE_MEM[$pid]}}")
+
+    color_start=""
+    color_end="$NC"
+    if (( $(echo "$mem_diff > 0.2" | bc -l) )); then
+      color_start=$RED
+    elif (( $(echo "$mem_diff < -0.2" | bc -l) )); then
+      color_start=$GREEN
+    else
+      color_start=$YELLOW
+    fi
+
+    echo -e "    ${color_start}$PROC_NAME (PID:$pid): CPU变化 ${cpu_diff}% | 内存变化 ${mem_diff}%${color_end}" >> "$LOG_FILE"
+  fi
+done
+
+# 当前最高占用进程
+top_cpu_pid=$(ps -eo pid,comm,%cpu --no-headers | grep "$PROC_NAME" | sort -k3 -nr | head -n 1 | awk '{print $1}')
+top_mem_pid=$(ps -eo pid,comm,%mem --no-headers | grep "$PROC_NAME" | sort -k3 -nr | head -n 1 | awk '{print $1}')
+
+if [ -n "$top_cpu_pid" ]; then
+  echo -e "  ${BLUE}当前占用最高CPU的进程:${NC} $PROC_NAME (PID:$top_cpu_pid)" >> "$LOG_FILE"
+fi
+if [ -n "$top_mem_pid" ]; then
+  echo -e "  ${BLUE}当前占用最高内存的进程:${NC} $PROC_NAME (PID:$top_mem_pid)" >> "$LOG_FILE"
+fi
+
+save_snapshot
+echo "==== $TIMESTAMP ====" >> "$LOG_FILE"

问IA.md

@@ -230,4 +230,114 @@ python服务的nginx日志内容：
 客户端IP: 10.0.0.8 | 用户: - | 时间: 23/Sep/2025:20:20:40 +0800 | 请求方法和路径: "GET /internal/search/18d2fad9-12b6-4f91-80fb-7304adf0bcd7 HTTP/1.1" | 状态码: 200 | 响应大小: 868211 | XFF: "-" | 客户端UA: "Go-http-client/1.1" | 上游服务器: 10.0.0.13:7001 | 上游响应耗时: 0.103 | 请求总耗时: 0.103 | Host: 10.0.0.3
 客户端IP: 10.0.0.8 | 用户: - | 时间: 23/Sep/2025:20:20:40 +0800 | 请求方法和路径: "GET /internal/search/18d2fad9-12b6-4f91-80fb-7304adf0bcd7 HTTP/1.1" | 状态码: 200 | 响应大小: 868211 | XFF: "-" | 客户端UA: "Go-http-client/1.1" | 上游服务器: 10.0.0.13:7001 | 上游响应耗时: 0.107 | 请求总耗时: 0.107 | Host: 10.0.0.3
 客户端IP: 10.0.0.8 | 用户: - | 时间: 23/Sep/2025:20:20:41 +0800 | 请求方法和路径: "GET /internal/search/18d2fad9-12b6-4f91-80fb-7304adf0bcd7 HTTP/1.1" | 状态码: 200 | 响应大小: 868211 | XFF: "-" | 客户端UA: "Go-http-client/1.1" | 上游服务器: 10.0.0.13:7001 | 上游响应耗时: 0.084 | 请求总耗时: 0.084 | Host: 10.0.0.3
-客户端IP: 10.0.0.8 | 用户: - | 时间: 23/Sep/2025:20:20:41 +0800 | 请求方法和路径: "GET /internal/search/18d2fad9-12b6-4f91-80fb-7304adf0bcd7 HTTP/1.1" | 状态码: 200 | 响应大小: 868211 | XFF: "-" | 客户端UA: "Go-http-client/1.1" | 上游服务器: 10.0.0.13:7001 | 上游响应耗时: 0.293 | 请求总耗时: 0.294 | Host: 10.0.0.3
+客户端IP: 10.0.0.8 | 用户: - | 时间: 23/Sep/2025:20:20:41 +0800 | 请求方法和路径: "GET /internal/search/18d2fad9-12b6-4f91-80fb-7304adf0bcd7 HTTP/1.1" | 状态码: 200 | 响应大小: 868211 | XFF: "-" | 客户端UA: "Go-http-client/1.1" | 上游服务器: 10.0.0.13:7001 | 上游响应耗时: 0.293 | 请求总耗时: 0.294 | Host: 10.0.0.3
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+分析一下这个内存变化：(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi        15Gi       133Mi       1.0Mi       237Mi        98Mi
+Swap:          8.0Gi       8.0Gi       0.0Ki
+(base) [root@prod-lessie-server02 ~]#ps aux|grep 7001
+root      158009  0.0  0.0  35908 10420 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158011  0.8 90.3 22549000 14317120 ?   Sl   Oct21  24:39 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 117868 ?      Sl   Oct21   9:26 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 243708 ?      Sl   Oct21  12:08 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 260252 ?      Sl   Oct21  13:51 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39116 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404684 119708 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.1 2404716 31388 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028724  0.0  0.0   6408   384 pts/0    D+   18:37   0:00 grep --color=auto 7001
+(base) [root@prod-lessie-server02 ~]#ps aux|grep 7001
+root      158009  0.0  0.0  35908 11316 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 119916 ?      Sl   Oct21   9:27 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 246652 ?      Sl   Oct21  12:11 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 261532 ?      Sl   Oct21  13:53 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39244 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404684 119324 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.2 2404716 31772 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028770  0.0  0.0   6408  2176 pts/0    S+   18:39   0:00 grep --color=auto 7001
+(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi       2.0Gi        12Gi       1.0Mi       961Mi        13Gi
+Swap:          8.0Gi       3.9Gi       4.1Gi
+(base) [root@prod-lessie-server02 ~]#
+(base) [root@prod-lessie-server02 ~]#ps auxf|grep 7001
+root     1029136  0.0  0.0   6408  2304 pts/0    S+   18:39   0:00  |       |   \_ grep --color=auto 7001
+root      158009  0.0  0.0  35908 14004 ?        S    Oct21   0:12 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158012  0.3  0.7 4111768 121836 ?      Sl   Oct21   9:27  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158160  0.0  0.2 2404712 39244 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158013  0.4  1.5 5852256 248700 ?      Sl   Oct21  12:11  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158218  0.0  0.2 2404716 31772 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158014  0.5  1.6 6056408 264348 ?      Sl   Oct21  13:53  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158253  0.0  0.2 2406028 35400 ?       Sl   Oct21   0:06  |   \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1028807 50.5  3.4 3214636 553780 ?      Sl   18:39   0:08  \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root     1029008  0.0  2.9 2408940 466648 ?      Sl   18:39   0:00      \_ /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+root      158186  0.0  0.7 2404652 120604 ?      Sl   Oct21   0:06 /data/webapps/prod_lessie_sourcing_agents/venv/bin/python /data/webapps/prod_lessie_sourcing_agents/venv/bin/gunicorn -w 4 -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7001 --timeout 300 dialogue.app:app --max-requests 500 --max-requests-jitter 50
+(base) [root@prod-lessie-server02 ~]#
+(base) [root@prod-lessie-server02 ~]#free -h
+               total        used        free      shared  buff/cache   available
+Mem:            15Gi       2.1Gi        12Gi       1.0Mi       1.0Gi        13Gi
+Swap:          8.0Gi       3.9Gi       4.1Gi   从工作进程上分析，从内存占用大到恢复是什么原因
+
+
+
+
+给我一个sh脚本，每分钟记录这个项目的各个进程、子进程的内存cpu使用情况，并且将结果保存在文件中。
+要求：每次记录要时间，内存格式化成GB，cpu格式化成百分比，日志类似
+
+====2025-10-01 10:00:00 ====
+主进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+
+分析变化：
+1、哪个子进程使用了更多的内存，哪个进程使用了更多的CPU
+2、相比上次查询，哪个子进程使用了资源变化了多少？
+3、相比上次查询，哪个子进程没了，哪个子进程出现了
+====2025-10-01 10:00:00 ====
+
+====2025-10-01 10:01:00 ====
+主进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+  子进程名：xxx | PID: xxx | 内存使用情况：xxx | CPU使用情况：xxx
+
+分析变化：
+1、哪个子进程使用了更多的内存，哪个进程使用了更多的CPU
+2、相比上次查询，哪个子进程使用了资源变化了多少？
+3、相比上次查询，哪个子进程没了，哪个子进程出现了
+====2025-10-01 10:01:00 ====
+