Files
Work-configuration-file/lessie.ai_error条目.md
2025-10-07 15:58:15 +08:00

399 lines
19 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

当前日志:
时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] 描述信息
时间戳 - 方法名 - 日志等级 - [user_email: xxx] [conversation_id: xxx] [event: xxx | status: xxx | msg: xxx] 描述信息
2025-05-25 21:23:20 - __main__ - WARNING - [user_email: DXIN@qq.com] [conversation_id: ad945d8] [event: xxx | status: xxx | msg: xxx] 404 Error - Endpoint: /, Method: OPTIONS
2025-06-16 17:40:10 - dialogue.ai_service_call - INFO - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] 用户历史消息: 我是一条测试的日志
2025-06-16 17:40:28 - business - WARNING - [user_email: chendinxin@gmail.com] [conversation_id: 123456-1234-1234-1234-123456789] [event: 达人搜素 | msg: 我是假的处理耗时过长,耗时: 64.96秒,匹配达人数: 219 | context: duration:64.96s,results_count:219]
背景我是运维人员将日志采集到es中基于查询日志等级为error的日志进行查询告警。
当前日志格式:时间戳 - 方法名 - 日志等级 - [用户邮箱:用户邮箱] [会话id:会话id] 错误描述 or 错误堆栈信息 or 其它
2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET
在日志采集filebesa采集的处理器设置进行采集并拆解了各个字段
# 针对 influencer_5002.log 的 dissect基础字段分解
- dissect:
when:
equals:
log_type: influencer_5002.log
tokenizer: '%{timestamp} - %{module} - %{level} - %{raw_tail}'
field: "message"
target_prefix: "mylog"
ignore_missing: true
overwrite_keys: true
# 从 message 中提取 user_email
- dissect:
when:
equals:
log_type: influencer_5002.log
tokenizer: '[user_email: %{user_email}] %{tail}'
field: "mylog.raw_tail"
target_prefix: "mylog"
ignore_missing: true
overwrite_keys: true
# 提取 conversation_id
- dissect:
when:
equals:
log_type: influencer_5002.log
tokenizer: '[conversation_id: %{conversation_id}] %{tail}'
field: "mylog.tail"
target_prefix: "mylog"
ignore_missing: true
overwrite_keys: true
当前这个方式可以做到日志告警基于日志等级是error进行查询告警这里定义为"服务系统告警"。
现在需要添加需要“业务层面指标”、“技术层面指标”、“安全层面指标的告警”,在当前日志中不能直接体现,这需要更改日志结构或者在日志结构中添加业务字段或说明。
以下是未来需要增加的告警:
业务层面指标:
环节 监控事件 告警规则
达人检索 检索召回数量少 召回数量≤5
召回响应时间长 响应时间≥20s
任务创建 任务创建失败 用户走完流程,但任务未创建成功
邮件发送 任务未启动 任务创建后10min状态仍为待启动
邮件发送速率低 邮件发送速度小于2min/封
任务中止 当日实发数<应发数但超过2min未发送新邮件
任务发不满 任务状态已完成但实发数<应发数
邮件回复率为0 任务完成后24h回复率为0
退信率 整体退信率>5%
技术层面指标:
1. 各接口请求成功率尤其是在对话的时候是否有顺利出AI回复
2. 并发用户数&任务数
3. InsightIQ调用量
安全层面指标
1. 异常登录监测
2. 一定时间段内检索频次
怎么改造这个日志可以完成通过查询es可以做到监控告警或则有别的方式时间吗
## 1、请求端点404错误
```
2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET
```
```
2025-05-25 21:23:20 - __main__ - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 404 Error - Endpoint: /, Method: OPTIONS
```
## 2、请求语法错误
```
2025-05-26 02:27:44 - werkzeug - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 172.236.228.229 - - [26/May/2025 02:27:44] code 400, message Bad request syntax ('\x16\x03\x01\x01')
```
## 3、请求版本错误
```
2025-05-26 01:09:58 - werkzeug - ERROR - [user_email: yubeichuan@gmail.com] [conversation_id: 26a4c827-f83b-4af9-aa21-4f1e87b59c10] 3.143.33.63 - - [26/May/2025 01:09:58] code 400, message Bad request version ('À\x14À')
```
## 4、请求语法错误
```
2025-05-25 21:24:10 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:24:10] code 400, message Bad request syntax ('\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00bbbb0100000001')
```
## 5、无效的http请求类型
```
2025-05-25 21:23:45 - werkzeug - ERROR - [user_email: ] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 51.81.155.131 - - [25/May/2025 21:23:45] code 400, message Bad HTTP/0.9 request type ('AMQP\x00\x00')
```
## 6、api请求失败
```
2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 详细错误信息: Traceback (most recent call last):
File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 69, in _request
response.raise_for_status()
File "/root/miniconda3/envs/search/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/webapps/influencer_search_agent/dialogue/influencer_5002.py", line 291, in search_single_card
topic_list = insightiq_gateway.get_topic_list(query_text, channels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/webapps/influencer_search_agent/gateway/insightiq_gateway.py", line 246, in get_topic_list
response = self._request('GET', f'social/creators/dictionary/topics/', params={'work_platform_id': work_platform_id, 'identifier': query_text, 'limit': 6})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/webapps/influencer_search_agent/utils/base_gateway.py", line 86, in _request
raise Exception(f"API请求失败: {str(e)}")
Exception: API请求失败: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6
```
```
2025-05-25 20:35:38 - __main__ - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] 处理卡片 4 时出错: API请求失败: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6
```
```
2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] API请求异常: 429 Client Error: for url: https://api.insightiq.ai/v1/social/creators/dictionary/topics/?work_platform_id=14d9ddf5-51c6-415e-bde6-f8ed36ad7054&identifier=DIY+office&limit=6
```
## 7、参数错误
```
2025-05-25 20:35:38 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: ad945d82-5d7d-4874-bb9b-fbf7532ba4f9] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/
```
## 8、聊天处理错误
```
2025-05-25 19:45:06 - dialogue.ai_service_call - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: fef57c39-3e6c-4bbf-8805-c91d0636b1c0] process chat error: the JSON object must be str, bytes or bytearray, not NoneType
Traceback (most recent call last):
File "/data/webapps/influencer_search_agent/dialogue/ai_service_call.py", line 859, in update_influencer_card_form
last_history_message_list = json.loads(last_history_message_list_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/search/lib/python3.12/json/__init__.py", line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType
```
## 9、登录过期错误
```
2025-05-23 20:11:22 - memory.memory_slots - ERROR - [user_email: ] [conversation_id: 5d1614dd-9e92-4db6-97b6-83d38c94b1e3] 从远程数据库加载数据失败,状态码: 401响应: {"code":401,"msg":"Your login status has expired, please log in again!","data":null}
```
## 在es里的摘要
```
{
"@timestamp": [
"2025-05-25T19:00:50.099Z"
],
"agent.ephemeral_id": [
"0b9f9018-ed5f-4fd0-8ad2-554873ca870b"
],
"agent.ephemeral_id.keyword": [
"0b9f9018-ed5f-4fd0-8ad2-554873ca870b"
],
"agent.id": [
"d0764891-a93b-426f-9282-31c476153eb3"
],
"agent.id.keyword": [
"d0764891-a93b-426f-9282-31c476153eb3"
],
"agent.name": [
"VM-0-3-centos"
],
"agent.name.keyword": [
"VM-0-3-centos"
],
"agent.type": [
"filebeat"
],
"agent.type.keyword": [
"filebeat"
],
"agent.version": [
"8.17.0"
],
"agent.version.keyword": [
"8.17.0"
],
"application": [
"influencer_search_app.lessie.ai"
],
"application.keyword": [
"influencer_search_app.lessie.ai"
],
"ecs.version": [
"8.0.0"
],
"ecs.version.keyword": [
"8.0.0"
],
"environment": [
"app_lessie_ai"
],
"environment.keyword": [
"app_lessie_ai"
],
"host.name": [
"VM-0-3-centos"
],
"input.type": [
"log"
],
"input.type.keyword": [
"log"
],
"log.file.path": [
"/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log"
],
"log.file.path.keyword": [
"/data/webapps/influencer_search_agent/log/influencer_5002_20250523_193646.log"
],
"log.offset": [
60913844
],
"log_type": [
"influencer_5002.log"
],
"log_type.keyword": [
"influencer_5002.log"
],
"message": [
"2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
],
"message.keyword": [
"2025-05-26 03:00:45 - __main__ - ERROR - [user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
],
"mylog.level": [
"ERROR"
],
"mylog.level.keyword": [
"ERROR"
],
"mylog.message": [
"[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
],
"mylog.message.keyword": [
"[user_email: mumemen9@gmail.com] [conversation_id: cfe9f405-0919-40ea-9769-c75e3a1854ae] 404 Error - Endpoint: /, Method: GET"
],
"mylog.module": [
"__main__"
],
"mylog.module.keyword": [
"__main__"
],
"mylog.timestamp": [
"2025-05-26 03:00:45"
],
"mylog.timestamp.keyword": [
"2025-05-26 03:00:45"
],
"_id": "AZcI0XdDvDmC7+Hb10iq",
"_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002",
"_score": null
}
```
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 所有卡片搜索完成,总耗时: 8.42秒,找到总共 53 个结果
2025-05-27 00:19:10 - __main__ - INFO - 卡片 3 (成分党护肤达人) 搜索完成,总耗时: 8.22秒,找到 10 个结果
2025-05-27 00:19:10 - __main__ - INFO - 卡片 3 排序后的matched_ids_with_scores: [{'id': '6760880459083318277', 'score': 0.6418526}, {'id': '7200799333906154538', 'score': 0.6364443}, {'id': '13703370', 'score': 0.6362148}, {'id': '6756056737843987461', 'score': 0.6320581}, {'id': '6931172889514247174', 'score': 0.6319245}, {'id': '6586372536946245637', 'score': 0.6314599}, {'id': '6709610644901430278', 'score': 0.63083375}, {'id': '6526536534412235791', 'score': 0.6307928}, {'id': '7061495040306627631', 'score': 0.63024676}, {'id': '7029324212326630405', 'score': 0.6293421}]
2025-05-27 00:19:25 - dialogue.agent - INFO - [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 开始异步执行extract_slots和update_last_history_form
2025-05-27 00:19:25 - dialogue.ai_service_call - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] extract_slots language_prompt: [{'role': 'system', 'content': 'Hello, I need you to act as a language detection and translation assistant. Please follow these steps:\n\n1. First, analyze the provided user_message to detect its language.\n\nuser_message: 寻找适合推广卸妆水产品的美妆达人要求粉丝量≥5000;\n\n2. Return two-character code of the language in lowercase, must not return anything else.\n\n For example:\n - If user_message is in Chinese, return "zh"\n - If user_message is in French, return "fr"\n - If user_message is in English, return "en"\n - If user_message is in Korean, return "ko"\n - If user_message is in Portuguese, return "pt"\n - If user_message is in Spanish, return "es"\n - If user_message is in German, return "de"\n - If user_message is in Italian, return "it"\n - If user_message is in Japanese, return "ja"\n\n Remember, your response should only contain the lowercase two-character code of the language, nothing else.\n'}]
2025-05-27 09:22:29 - utils.base_gateway - ERROR - [user_email: token_d98cf010...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/
2025-05-26 21:56:43 - utils.base_gateway - ERROR - [user_email: jennie.christina.ai@gmail.com] [conversation_id: 4c97fc29-5d95-422e-9b9f-e206789119bd] Error param, https://api.insightiq.ai/v1/, social/creators/dictionary/topics/
这样的日志格式filebeat的处理器能拆分各个字段吗好像info和error的日志日志结构不统一可以按需匹配处理器吗
{
"@timestamp": [
"2025-05-27T03:22:39.814Z"
],
"message": [
"2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"message.keyword": [
"2025-05-27 11:15:21 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"mylog.level": [
"ERROR"
],
"mylog.level.keyword": [
"ERROR"
],
"mylog.module": [
"utils.base_gateway"
],
"mylog.module.keyword": [
"utils.base_gateway"
],
"mylog.raw_tail": [
"[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"mylog.raw_tail.keyword": [
"[user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"mylog.tail": [
"HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"mylog.tail.keyword": [
"HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/"
],
"mylog.timestamp": [
"2025-05-27 11:15:21"
],
"mylog.timestamp.keyword": [
"2025-05-27 11:15:21"
],
"mylog.user_email": [
"token_6fc20e11..."
],
"mylog.user_email.keyword": [
"token_6fc20e11..."
],
"_id": "AZcPw3dDvDmC7zSTrFPW",
"_index": ".ds-out-148-flymoonlog-app_lessie_ai-influencer_search_app.lessie.ai-2025.05-2025.05.21-000002",
"_score": null
}
查询到的最新日志详情:
{
"_id": "AZcQbHdDvDmC71v4SVPH",
"timestamp": "2",
"module": "无",
"level": "ERROR",
"user_email": "无",
"conversation_id": "无",
"tail": "2025-05-27 12:48:54 - utils.base_gateway - ERROR - [user_email: token_6fc20e11...] HTTP错误: 500 - 401 Client Error: for url: http://129.204.158.54:8070/prod-api/jenniefy/account/info/..."}
[DEBUG] 飞书API完整响应{"StatusCode": 0, "StatusMessage": "success", "code": 0, "data": {}, "msg": "success"}
告警已发送发现30条错误日志
1、该脚本执行后发送到飞书机器人显示的时间错误消息显示触发时间2025-05-27 01:10:57实际时间是2025-05-27 09:10:54
2、帮我修改发送到飞书的消息的内容
触发时间python脚本触发时间
错误数量python脚本触发时查询到符合条件的条目数
最近一条日志详情
时间戳kibana界面看到的 mylog.timestamp 字段
模块kibana界面看到的 mylog.timestamp 字段
日志等级kibana界面看到的 mylog.level 字段
用户邮箱kibana界面看到的 mylog.user_email 字段
会话IDkibana界面看到的 mylog.conversation_id 字段
日志信息kibana界面看到的 mylog.tail 字段
@群成员
注意:
1、有些日志条目的“用户邮箱字段”或者“会话ID字段”有时可能是空的或者就没有该字段当这两个字段没有值时显示“无”
2、日志信息字段其实就是错误堆栈信息有时候可能会很长需要截断显示上限500字
3、脚本日志输出需要显示查询的具体信息。
4、@群成员:暂时保持@所有人
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0
2025-05-27 00:19:10 - __main__ - INFO - [user_email: pengdachen1995@gmail.com] [conversation_id: 9ea1bef1-3121-484b-a239-5f413139d88c] 穿插排序后的结果数量: 0