删除气流任务日志 [英] Removing Airflow task logs
问题描述
我正在运行5个DAG,在过去的一个月中,它们在 base_log_folder
中总共生成了大约6GB的日志数据。我只是添加了 remote_base_log_folder
,但似乎并不排除登录到 base_log_folder
。
I'm running 5 DAG's which have generated a total of about 6GB of log data in the base_log_folder
over a months period. I just added a remote_base_log_folder
but it seems it does not exclude logging to the base_log_folder
.
有没有自动删除旧日志文件,旋转日志文件或强制气流不将其仅登录到远程存储中的磁盘(base_log_folder)?
Is there anyway to automatically remove old log files, rotate them or force airflow to not log on disk (base_log_folder) only in remote storage?
推荐答案
请参考 https://github.com/teamclairvoyant/ airflow-maintenance-dags
此插件具有DAG,可以杀死已暂停的任务并清除日志。
您可以掌握这些概念,并提出可以根据您的要求进行清理的新DAG。
This plugin has DAGs that can kill halted tasks and log-cleanups. You can grab the concepts and can come up with a new DAG that can cleanup as per your requirement.
这篇关于删除气流任务日志的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!