S3_delete_objects_operator 不删除bucket中的文件 [英] S3_delete_objects_operator does not delete files in the bucket

问题描述

我已经实现了 S3DeleteObjectsOperator，但即使任务显示密钥已成功删除，实际上该对象并未从 S3 存储桶中删除.

I have implemented S3DeleteObjectsOperator, but even though task shows key deleted successfully, in reality the object does not get deleted from the S3 bucket.

delete_s3bucket_files = S3DeleteObjectsOperator(
  task_id='delete_s3bucket_files',
  start_date=start_date,
  bucket='****',
  keys='******************',
  aws_conn_id='aws_default',
)

即使任务按时完成，它也不会删除存储桶中指定键内的对象.我可以看到下面的日志:

Even though the task is getting completed as passed, it does not delete objects inside the specified key in the bucket. I can see the logs below:

[2019-09-26 14:19:15,554] {base_task_runner.py:101}INFO - Job 1435: Subtask delete_s3bucket_files [2019-09-26 14:19:15,553] {cli.py:517} INFO - Running <TaskInstance: test_s3_delete.delete_s3bucket_files 2019-09-26T12:18:59.362470+00:00 [running]> on host Saurav-macbook.local
[2019-09-26 14:19:15,883] {s3_delete_objects_operator.py:83} INFO - Deleted: ['******************']

我如何了解任务正在执行什么以及为什么对象没有被删除?

How can I understand what the task is performing and why does the object does not get deleted?

推荐答案

我无法使用 S3DeleteObjectOperator 和 delete_objects() 方法删除对象在 S3Hook 中.所以我使用 boto3 delete() 方法来删​​除对象.

I couldn't get the object deletion working with S3DeleteObjectOperator as well as the delete_objects() method in S3Hook. So instead I used the boto3 delete() method to delete the object.

def delete_files():
  s3 = boto3.resource('s3', aws_access_key_id='****', aws_secret_access_key='******************')
  s3_bucket = s3.Bucket('****')
  s3_bucket.objects.all().delete()


delete_s3bucket_files = PythonOperator(
  task_id='delete_s3bucket_files',
  start_date=start_date,
  python_callable=delete_files,
  dag=dag
)

不确定这是否是正确的做法，但目前对我有用.

Not sure if this is the right way to do, but works for me as of now.

这篇关于S3_delete_objects_operator 不删除bucket中的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！