Skip to content

Upgrade botocore to attempt to fix recent failures in writing to S3

Brouberol requested to merge upgrade-botocore into main

DAG logs have been missing from S3 for a week in airflow-test-k8s, and we're seeing the following errors when tailing the task pod logs:

[2025-02-06T11:04:13.972+0000] {s3_task_handler.py:190} ERROR - Could not write logs to s3://logs.airflow-test-k8s.dse-k8s-eqiad/dag_id=produce_logs/run_id=manual__2025-02-06T11:03:06.958693+00:00/task_id=produce_logs/attempt=1.log
Traceback (most recent call last):
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py", line 179, in s3_write
    self.hook.load_string(
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 153, in wrapper
    return func(*bound_args.args, **bound_args.kwargs)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 126, in wrapper
    return func(*bound_args.args, **bound_args.kwargs)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 1198, in load_string
    self._upload_file_obj(f, key, bucket_name, replace, encrypt, acl_policy)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 1281, in _upload_file_obj
    client.upload_fileobj(
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/boto3/s3/inject.py", line 642, in upload_fileobj
    return future.result()
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/s3transfer/futures.py", line 264, in result
    raise self._exception
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/s3transfer/tasks.py", line 135, in __call__
    return self._execute_main(kwargs)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/s3transfer/tasks.py", line 158, in _execute_main
    return_value = self._main(**kwargs)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/s3transfer/upload.py", line 796, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/tmp/pyenv/versions/3.10.15/lib/python3.10/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: None

This seems to be related to a bad botocore release, cf https://github.com/fsspec/s3fs/issues/932#issuecomment-2603269485

Signed-off-by: Balthazar Rouberol brouberol@wikimedia.org

Merge request reports