dxin/Work-configuration-file
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Work-configuration-file/lessie.ai_error条目.md
dxin 0e593caf99 初始化提交
2025-10-07 15:58:15 +08:00

19 KiB
Raw Permalink Blame History

当前日志： 时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] 描述信息

时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] [event: xxx | status: xxx | msg: xxx] 描述信息

2025-05-25 21:23:20 - main - WARNING - [user_email: DXIN@qq.com] [conversation_id: ad945d8] [event: xxx | status: xxx | msg: xxx] 404 Error - Endpoint: /, Method: OPTIONS

2025-06-16 17:40:10 - dialogue.ai_service_call - INFO - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] 用户历史消息: 我是一条测试的日志

2025-06-16 17:40:28 - business - WARNING - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] [event: 达人搜素 | msg: 我是假的处理耗时过长,耗时: 64.96秒,匹配达人数: 219 | context: duration:64.96s,results_count:219]

背景：我是运维人员，将日志采集到es中，基于查询日志等级为error的日志进行查询告警。 当前日志格式：时间戳 - 方法名 - 日志等级 - [用户邮箱:用户邮箱] [会话id:会话id] 错误描述 or 错误堆栈信息 or 其它 如：2025-05-26 03:00:45 - main - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET 在日志采集filebesa采集的处理器设置进行采集并拆解了各个字段：

针对 influencer_5002.log 的 dissect（基础字段分解）

  • dissect: when: equals: log_type: influencer_5002.log tokenizer: '%{timestamp} - %{module} - %{level} - %{raw_tail}' field: "message" target_prefix: "mylog" ignore_missing: true overwrite_keys: true

从 message 中提取 user_email

  • dissect: when: equals: log_type: influencer_5002.log tokenizer: '[user_email: %{user_email}] %{tail}' field: "mylog.raw_tail" target_prefix: "mylog" ignore_missing: true overwrite_keys: true

提取 conversation_id

  • dissect: when: equals: log_type: influencer_5002.log tokenizer: '[conversation_id: %{conversation_id}] %{tail}' field: "mylog.tail" target_prefix: "mylog" ignore_missing: true overwrite_keys: true

当前这个方式可以做到日志告警基于日志等级是error进行查询告警，这里定义为"服务系统告警"。 现在需要添加需要“业务层面指标”、“技术层面指标”、“安全层面指标的告警”，在当前日志中不能直接体现，这需要更改日志结构或者在日志结构中添加业务字段或说明。 以下是未来需要增加的告警： 业务层面指标： 环节 监控事件 告警规则 达人检索 检索召回数量少 召回数量≤5 召回响应时间长 响应时间≥20s 任务创建 任务创建失败 用户走完流程，但任务未创建成功
邮件发送 任务未启动 任务创建后10min，状态仍为待启动 邮件发送速率低 邮件发送速度小于2min/封 任务中止 当日实发数<应发数，但超过2min未发送新邮件 任务发不满 任务状态已完成，但实发数<应发数 邮件回复率为0 任务完成后24h，回复率为0 退信率 整体退信率>5%

技术层面指标：

  1. 各接口请求成功率（尤其是在对话的时候，是否有顺利出AI回复）
  2. 并发用户数&任务数
  3. InsightIQ调用量

安全层面指标

  1. 异常登录监测
  2. 一定时间段内检索频次

怎么改造这个日志可以完成通过查询es可以做到监控告警，或则有别的方式时间吗？

1、请求端点404错误

2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET

2025-05-25 21:23:20 - __main__ - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 404 Error - Endpoint: /, Method: OPTIONS

2、请求语法错误

2025-05-26 02:27:44 - werkzeug - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 172.236.228.229 - - [26/May/2025 02:27:44] code 400, message Bad request syntax ('\x16\x03\x01\x01')

3、请求版本错误

2025-05-26 01:09:58 - werkzeug - ERROR - [user_email: yubeichuan@gmail.com] [conversation_id: 26a4c827-f83b-4af9-aa21-4f1e87b59c10] 3.143.33.63 - - [26/May/2025 01:09:58] code 400, message Bad request version ('À\x14À')

4、请求语法错误

2025-05-25 21:24:10 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:24:10] code 400, message Bad request syntax ('\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00bbbb0100000001')

5、无效的http请求类型

2025-05-25 21:23:45 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:23:45] code 400, message Bad HTTP/0.9 request type ('AMQP\x00\x00')

6、api请求失败

2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 详细错误信息: Traceback (most recent call last):
  File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 69, in _request
    response.raise_for_status()
  File "/root/miniconda3/envs/search/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error:  for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/webapps/influencer_search_agent/dialogue/influencer_5002.py", line 291, in search_single_card
    topic_list = insightiq_gateway.get_topic_list(query_text, channels)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/webapps/influencer_search_agent/gateway/insightiq_gateway.py", line 246, in get_topic_list
    response = self._request('GET', f'social/creators/dictionary/topics/', params={'work_platform_id': work_platform_id, 'identifier': query_text, 'limit': 6})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 86, in _request
    raise Exception(f"API请求失败: {str(e)}")
Exception: API请求失败: 429 Client Error:  for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6

2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 处理卡片 4 时出错: API请求失败: 429 Client Error:  for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6

2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] API请求异常: 429 Client Error:  for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6

7、参数错误

2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/

8、聊天处理错误

2025-05-25 19:45:06 - dialogue.ai_service_call - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: fef57c39-3e6c-4bbf-8805-c91d0636b1c0] process chat error: the JSON object must be str, bytes or bytearray, not NoneType
Traceback (most recent call last):
  File "/data/webapps/influencer_search_agent/dialogue/ai_service_call.py", line 859, in update_influencer_card_form
    last_history_message_list = json.loads(last_history_message_list_str)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/search/lib/python3.12/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

9、登录过期错误

2025-05-23 20:11:22 - memory.memory_slots - ERROR - [user_email: ] [conversation_id: 5d1614dd-9e92-4db6-97b6-83d38c94b1e3] 从远程数据库加载数据失败，状态码: 401，响应: {"code":401,"msg":"Your login status has expired, please log in again!","data":null}

在es里的摘要：

{
  "@timestamp": [
    "2025-05-25T19:00:50.099Z"
  ],
  "agent.ephemeral_id": [
    "0b9f9018-ed5f-4fd0-8ad2-554873ca870b"
  ],
  "agent.ephemeral_id.keyword": [
    "0b9f9018-ed5f-4fd0-8ad2-554873ca870b"
  ],
  "agent.id": [
    "d0764891-a93b-426f-9282-31c476153eb3"
  ],
  "agent.id.keyword": [
    "d0764891-a93b-426f-9282-31c476153eb3"
  ],
  "agent.name": [
    "VM-0-3-centos"
  ],
  "agent.name.keyword": [
    "VM-0-3-centos"
  ],
  "agent.type": [
    "filebeat"
  ],
  "agent.type.keyword": [
    "filebeat"
  ],
  "agent.version": [
    "8.17.0"
  ],
  "agent.version.keyword": [
    "8.17.0"
  ],
  "application": [
    "influencer_search_app.lessie.ai"
  ],
  "application.keyword": [
    "influencer_search_app.lessie.ai"
  ],
  "ecs.version": [
    "8.0.0"
  ],
  "ecs.version.keyword": [
    "8.0.0"
  ],
  "environment": [
    "app_lessie_ai"
  ],
  "environment.keyword": [
    "app_lessie_ai"
  ],
  "host.name": [
    "VM-0-3-centos"
  ],
  "input.type": [
    "log"
  ],
  "input.type.keyword": [
    "log"
  ],
  "log.file.path": [
    "/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log"
  ],
  "log.file.path.keyword": [
    "/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log"
  ],
  "log.offset": [
    60913844
  ],
  "log_type": [
    "influencer_5002.log"
  ],
  "log_type.keyword": [
    "influencer_5002.log"
  ],
  "message": [
    "2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
  ],
  "message.keyword": [
    "2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
  ],
  "mylog.level": [
    "ERROR"
  ],
  "mylog.level.keyword": [
    "ERROR"
  ],
  "mylog.message": [
    "[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
  ],
  "mylog.message.keyword": [
    "[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
  ],
  "mylog.module": [
    "__main__"
  ],
  "mylog.module.keyword": [
    "__main__"
  ],
  "mylog.timestamp": [
    "2025-05-26 03:00:45"
  ],
  "mylog.timestamp.keyword": [
    "2025-05-26 03:00:45"
  ],
  "_id": "AZcI0XdDvDmC7+Hb10iq",
  "_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002",
  "_score": null
}

2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 所有卡片搜索完成，总耗时: 8.42秒，找到总共 53 个结果 2025-05-27 00:19:10 - main - INFO - 卡片 3 (成分党护肤达人) 搜索完成，总耗时: 8.22秒，找到 10 个结果
2025-05-27 00:19:10 - main - INFO - 卡片 3 排序后的matched_ids_with_scores: [{'id': '6760880459083318277', 'score': 0.6418526}, {'id': '7200799333906154538', 'score': 0.6364443}, {'id': '13703370', 'score': 0.6362148}, {'id': '6756056737843987461', 'score': 0.6320581}, {'id': '6931172889514247174', 'score': 0.6319245}, {'id': '6586372536946245637', 'score': 0.6314599}, {'id': '6709610644901430278', 'score': 0.63083375}, {'id': '6526536534412235791', 'score': 0.6307928}, {'id': '7061495040306627631', 'score': 0.63024676}, {'id': '7029324212326630405', 'score': 0.6293421}]
2025-05-27 00:19:25 - dialogue.agent - INFO - [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 开始异步执行extract_slots和update_last_history_form 2025-05-27 00:19:25 - dialogue.ai_service_call - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] extract_slots language_prompt: [{'role': 'system', 'content': 'Hello, I need you to act as a language detection and translation assistant. Please follow these steps:\n\n1. First, analyze the provided user_message to detect its language.\n\nuser_message: 寻找适合推广卸妆水产品的美妆达人，要求粉丝量≥5000;\n\n2. Return two-character code of the language in lowercase, must not return anything else.\n\n For example:\n - If user_message is in Chinese, return "zh"\n - If user_message is in French, return "fr"\n - If user_message is in English, return "en"\n - If user_message is in Korean, return "ko"\n - If user_message is in Portuguese, return "pt"\n - If user_message is in Spanish, return "es"\n - If user_message is in German, return "de"\n - If user_message is in Italian, return "it"\n - If user_message is in Japanese, return "ja"\n\n Remember, your response should only contain the lowercase two-character code of the language, nothing else.\n'}]
2025-05-27 09:22:29 - utils.base_gateway - ERROR - [user_email: token_d98cf010...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/
2025-05-26 21:56:43 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: 4c97fc29-5d95-422e-9b9f-e206789119bd] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/
这样的日志格式，filebeat的处理器能拆分各个字段吗？好像info和error的日志日志结构不统一？可以按需匹配处理器吗？

{ "@timestamp": [ "2025-05-27T03:22:39.814Z" ], "message": [ "2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "message.keyword": [ "2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.level": [ "ERROR" ], "mylog.level.keyword": [ "ERROR" ], "mylog.module": [ "utils.base_gateway" ], "mylog.module.keyword": [ "utils.base_gateway" ], "mylog.raw_tail": [ "[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.raw_tail.keyword": [ "[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.tail": [ "HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.tail.keyword": [ "HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.timestamp": [ "2025-05-27 11:15:21" ], "mylog.timestamp.keyword": [ "2025-05-27 11:15:21" ], "mylog.user_email": [ "token_6fc20e11..." ], "mylog.user_email.keyword": [ "token_6fc20e11..." ], "_id": "AZcPw3dDvDmC7zSTrFPW", "_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002", "_score": null }

查询到的最新日志详情：

{ "_id": "AZcQbHdDvDmC71v4SVPH", "timestamp": "2", "module": "无", "level": "ERROR", "user_email": "无", "conversation_id": "无", "tail": "2025-05-27 12:48:54 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/..."}

[DEBUG] 飞书API完整响应：{"StatusCode": 0, "StatusMessage": "success", "code": 0, "data": {}, "msg": "success"} 告警已发送，发现30条错误日志

1、该脚本执行后发送到飞书机器人显示的时间错误，消息显示：触发时间：2025-05-27 01:10:57，实际时间是2025-05-27 09:10:54 2、帮我修改发送到飞书的消息的内容​：

触发时间：python脚本触发时间 错误数量：python脚本触发时查询到符合条件的条目数 最近一条日志详情 时间戳：kibana界面看到的 mylog.timestamp 字段 模块：kibana界面看到的 mylog.timestamp 字段 日志等级：kibana界面看到的 mylog.level 字段 用户邮箱：kibana界面看到的 mylog.user_email 字段 会话ID：kibana界面看到的 mylog.conversation_id 字段 日志信息：kibana界面看到的 mylog.tail 字段 @群成员 注意： 1、有些日志条目的“用户邮箱字段”或者“会话ID字段”，有时可能是空的，或者就没有该字段，当这两个字段没有值时显示“无” 2、日志信息字段其实就是错误堆栈信息，有时候可能会很长，需要截断，显示上限500字 3、脚本日志输出需要显示查询的具体信息。 4、@群成员：暂时保持@所有人

2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - main - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0