当前日志： 时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] 描述信息 时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] [event: xxx | status: xxx | msg: xxx] 描述信息 2025-05-25 21:23:20 - __main__ - WARNING - [user_email: DXIN@qq.com] [conversation_id: ad945d8] [event: xxx | status: xxx | msg: xxx] 404 Error - Endpoint: /, Method: OPTIONS 2025-06-16 17:40:10 - dialogue.ai_service_call - INFO - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] 用户历史消息: 我是一条测试的日志 2025-06-16 17:40:28 - business - WARNING - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] [event: 达人搜素 | msg: 我是假的处理耗时过长,耗时: 64.96秒,匹配达人数: 219 | context: duration:64.96s,results_count:219] 背景：我是运维人员，将日志采集到es中，基于查询日志等级为error的日志进行查询告警。 当前日志格式：时间戳 - 方法名 - 日志等级 - [用户邮箱:用户邮箱] [会话id:会话id] 错误描述 or 错误堆栈信息 or 其它 如：2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET 在日志采集filebesa采集的处理器设置进行采集并拆解了各个字段： # 针对 influencer_5002.log 的 dissect（基础字段分解） - dissect: when: equals: log_type: influencer_5002.log tokenizer: '%{timestamp} - %{module} - %{level} - %{raw_tail}' field: "message" target_prefix: "mylog" ignore_missing: true overwrite_keys: true # 从 message 中提取 user_email - dissect: when: equals: log_type: influencer_5002.log tokenizer: '[user_email: %{user_email}] %{tail}' field: "mylog.raw_tail" target_prefix: "mylog" ignore_missing: true overwrite_keys: true # 提取 conversation_id - dissect: when: equals: log_type: influencer_5002.log tokenizer: '[conversation_id: %{conversation_id}] %{tail}' field: "mylog.tail" target_prefix: "mylog" ignore_missing: true overwrite_keys: true 当前这个方式可以做到日志告警基于日志等级是error进行查询告警，这里定义为"服务系统告警"。 现在需要添加需要“业务层面指标”、“技术层面指标”、“安全层面指标的告警”，在当前日志中不能直接体现，这需要更改日志结构或者在日志结构中添加业务字段或说明。 以下是未来需要增加的告警： 业务层面指标： 环节 监控事件 告警规则 达人检索 检索召回数量少 召回数量≤5 召回响应时间长 响应时间≥20s 任务创建 任务创建失败 用户走完流程，但任务未创建成功 邮件发送 任务未启动 任务创建后10min，状态仍为待启动 邮件发送速率低 邮件发送速度小于2min/封 任务中止 当日实发数<应发数，但超过2min未发送新邮件 任务发不满 任务状态已完成，但实发数<应发数 邮件回复率为0 任务完成后24h，回复率为0 退信率 整体退信率>5% 技术层面指标： 1. 各接口请求成功率（尤其是在对话的时候，是否有顺利出AI回复） 2. 并发用户数&任务数 3. InsightIQ调用量 安全层面指标 1. 异常登录监测 2. 一定时间段内检索频次 怎么改造这个日志可以完成通过查询es可以做到监控告警，或则有别的方式时间吗？ ## 1、请求端点404错误 ``` 2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET ``` ``` 2025-05-25 21:23:20 - __main__ - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 404 Error - Endpoint: /, Method: OPTIONS ``` ## 2、请求语法错误 ``` 2025-05-26 02:27:44 - werkzeug - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 172.236.228.229 - - [26/May/2025 02:27:44] code 400, message Bad request syntax ('\x16\x03\x01\x01') ``` ## 3、请求版本错误 ``` 2025-05-26 01:09:58 - werkzeug - ERROR - [user_email: yubeichuan@gmail.com] [conversation_id: 26a4c827-f83b-4af9-aa21-4f1e87b59c10] 3.143.33.63 - - [26/May/2025 01:09:58] code 400, message Bad request version ('À\x14À') ``` ## 4、请求语法错误 ``` 2025-05-25 21:24:10 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:24:10] code 400, message Bad request syntax ('\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00bbbb0100000001') ``` ## 5、无效的http请求类型 ``` 2025-05-25 21:23:45 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:23:45] code 400, message Bad HTTP/0.9 request type ('AMQP\x00\x00') ``` ## 6、api请求失败 ``` 2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 详细错误信息: Traceback (most recent call last): File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 69, in _request response.raise_for_status() File "/root/miniconda3/envs/search/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/webapps/influencer_search_agent/dialogue/influencer_5002.py", line 291, in search_single_card topic_list = insightiq_gateway.get_topic_list(query_text, channels) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/webapps/influencer_search_agent/gateway/insightiq_gateway.py", line 246, in get_topic_list response = self._request('GET', f'social/creators/dictionary/topics/', params={'work_platform_id': work_platform_id, 'identifier': query_text, 'limit': 6}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 86, in _request raise Exception(f"API请求失败: {str(e)}") Exception: API请求失败: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6 ``` ``` 2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 处理卡片 4 时出错: API请求失败: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6 ``` ``` 2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] API请求异常: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6 ``` ## 7、参数错误 ``` 2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/ ``` ## 8、聊天处理错误 ``` 2025-05-25 19:45:06 - dialogue.ai_service_call - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: fef57c39-3e6c-4bbf-8805-c91d0636b1c0] process chat error: the JSON object must be str, bytes or bytearray, not NoneType Traceback (most recent call last): File "/data/webapps/influencer_search_agent/dialogue/ai_service_call.py", line 859, in update_influencer_card_form last_history_message_list = json.loads(last_history_message_list_str) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/search/lib/python3.12/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType ``` ## 9、登录过期错误 ``` 2025-05-23 20:11:22 - memory.memory_slots - ERROR - [user_email: ] [conversation_id: 5d1614dd-9e92-4db6-97b6-83d38c94b1e3] 从远程数据库加载数据失败，状态码: 401，响应: {"code":401,"msg":"Your login status has expired, please log in again!","data":null} ``` ## 在es里的摘要： ``` { "@timestamp": [ "2025-05-25T19:00:50.099Z" ], "agent.ephemeral_id": [ "0b9f9018-ed5f-4fd0-8ad2-554873ca870b" ], "agent.ephemeral_id.keyword": [ "0b9f9018-ed5f-4fd0-8ad2-554873ca870b" ], "agent.id": [ "d0764891-a93b-426f-9282-31c476153eb3" ], "agent.id.keyword": [ "d0764891-a93b-426f-9282-31c476153eb3" ], "agent.name": [ "VM-0-3-centos" ], "agent.name.keyword": [ "VM-0-3-centos" ], "agent.type": [ "filebeat" ], "agent.type.keyword": [ "filebeat" ], "agent.version": [ "8.17.0" ], "agent.version.keyword": [ "8.17.0" ], "application": [ "influencer_search_app.lessie.ai" ], "application.keyword": [ "influencer_search_app.lessie.ai" ], "ecs.version": [ "8.0.0" ], "ecs.version.keyword": [ "8.0.0" ], "environment": [ "app_lessie_ai" ], "environment.keyword": [ "app_lessie_ai" ], "host.name": [ "VM-0-3-centos" ], "input.type": [ "log" ], "input.type.keyword": [ "log" ], "log.file.path": [ "/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log" ], "log.file.path.keyword": [ "/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log" ], "log.offset": [ 60913844 ], "log_type": [ "influencer_5002.log" ], "log_type.keyword": [ "influencer_5002.log" ], "message": [ "2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET" ], "message.keyword": [ "2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET" ], "mylog.level": [ "ERROR" ], "mylog.level.keyword": [ "ERROR" ], "mylog.message": [ "[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET" ], "mylog.message.keyword": [ "[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET" ], "mylog.module": [ "__main__" ], "mylog.module.keyword": [ "__main__" ], "mylog.timestamp": [ "2025-05-26 03:00:45" ], "mylog.timestamp.keyword": [ "2025-05-26 03:00:45" ], "_id": "AZcI0XdDvDmC7+Hb10iq", "_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002", "_score": null } ``` 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 所有卡片搜索完成，总耗时: 8.42秒，找到总共 53 个结果 2025-05-27 00:19:10 - __main__ - INFO - 卡片 3 (成分党护肤达人) 搜索完成，总耗时: 8.22秒，找到 10 个结果 2025-05-27 00:19:10 - __main__ - INFO - 卡片 3 排序后的matched_ids_with_scores: [{'id': '6760880459083318277', 'score': 0.6418526}, {'id': '7200799333906154538', 'score': 0.6364443}, {'id': '13703370', 'score': 0.6362148}, {'id': '6756056737843987461', 'score': 0.6320581}, {'id': '6931172889514247174', 'score': 0.6319245}, {'id': '6586372536946245637', 'score': 0.6314599}, {'id': '6709610644901430278', 'score': 0.63083375}, {'id': '6526536534412235791', 'score': 0.6307928}, {'id': '7061495040306627631', 'score': 0.63024676}, {'id': '7029324212326630405', 'score': 0.6293421}] 2025-05-27 00:19:25 - dialogue.agent - INFO - [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 开始异步执行extract_slots和update_last_history_form 2025-05-27 00:19:25 - dialogue.ai_service_call - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] extract_slots language_prompt: [{'role': 'system', 'content': 'Hello, I need you to act as a language detection and translation assistant. Please follow these steps:\n\n1. First, analyze the provided user_message to detect its language.\n\nuser_message: 寻找适合推广卸妆水产品的美妆达人，要求粉丝量≥5000;\n\n2. Return two-character code of the language in lowercase, must not return anything else.\n\n For example:\n - If user_message is in Chinese, return "zh"\n - If user_message is in French, return "fr"\n - If user_message is in English, return "en"\n - If user_message is in Korean, return "ko"\n - If user_message is in Portuguese, return "pt"\n - If user_message is in Spanish, return "es"\n - If user_message is in German, return "de"\n - If user_message is in Italian, return "it"\n - If user_message is in Japanese, return "ja"\n\n Remember, your response should only contain the lowercase two-character code of the language, nothing else.\n'}] 2025-05-27 09:22:29 - utils.base_gateway - ERROR - [user_email: token_d98cf010...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/ 2025-05-26 21:56:43 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: 4c97fc29-5d95-422e-9b9f-e206789119bd] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/ 这样的日志格式，filebeat的处理器能拆分各个字段吗？好像info和error的日志日志结构不统一？可以按需匹配处理器吗？ { "@timestamp": [ "2025-05-27T03:22:39.814Z" ], "message": [ "2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "message.keyword": [ "2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.level": [ "ERROR" ], "mylog.level.keyword": [ "ERROR" ], "mylog.module": [ "utils.base_gateway" ], "mylog.module.keyword": [ "utils.base_gateway" ], "mylog.raw_tail": [ "[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.raw_tail.keyword": [ "[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.tail": [ "HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.tail.keyword": [ "HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/" ], "mylog.timestamp": [ "2025-05-27 11:15:21" ], "mylog.timestamp.keyword": [ "2025-05-27 11:15:21" ], "mylog.user_email": [ "token_6fc20e11..." ], "mylog.user_email.keyword": [ "token_6fc20e11..." ], "_id": "AZcPw3dDvDmC7zSTrFPW", "_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002", "_score": null } 查询到的最新日志详情： { "_id": "AZcQbHdDvDmC71v4SVPH", "timestamp": "2", "module": "无", "level": "ERROR", "user_email": "无", "conversation_id": "无", "tail": "2025-05-27 12:48:54 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/..."} [DEBUG] 飞书API完整响应：{"StatusCode": 0, "StatusMessage": "success", "code": 0, "data": {}, "msg": "success"} 告警已发送，发现30条错误日志 1、该脚本执行后发送到飞书机器人显示的时间错误，消息显示：触发时间：2025-05-27 01:10:57，实际时间是2025-05-27 09:10:54 2、帮我修改发送到飞书的消息的内容​： 触发时间：python脚本触发时间 错误数量：python脚本触发时查询到符合条件的条目数 最近一条日志详情 时间戳：kibana界面看到的 mylog.timestamp 字段 模块：kibana界面看到的 mylog.timestamp 字段 日志等级：kibana界面看到的 mylog.level 字段 用户邮箱：kibana界面看到的 mylog.user_email 字段 会话ID：kibana界面看到的 mylog.conversation_id 字段 日志信息：kibana界面看到的 mylog.tail 字段 @群成员 注意： 1、有些日志条目的“用户邮箱字段”或者“会话ID字段”，有时可能是空的，或者就没有该字段，当这两个字段没有值时显示“无” 2、日志信息字段其实就是错误堆栈信息，有时候可能会很长，需要截断，显示上限500字 3、脚本日志输出需要显示查询的具体信息。 4、@群成员：暂时保持@所有人 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0 2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0