Problem
When you try to run queries on tables from a catalog using a Hive Federation connection, you receive an error.
SparkException: Exception thrown in awaitResult: java.nio.file.AccessDeniedException: s3a://test-us-east-2/user/hive/warehouse/log.db/logs/_delta_log/00000000000000000000.json: open s3a://test-us-east-2/user/hive/warehouse/log.db/logs/_delta_log/00000000000000000000.json at 0 on s3a://test-us-east-2/user/hive/warehouse/log.db/logs/
File <command-431391062372215>, line 1
----> 1 get_ipython().run_cell_magic('sql', '', 'select * from catalog.schema.logs\n')
File /databricks/python/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py:1988, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1985 info = error_details_pb2.ErrorInfo()
1986 d.Unpack(info)
-> 1988 raise convert_exception(
1989 info,
1990 status.message,
1991 self._fetch_enriched_error(info),
1992 self._display_server_stack_trace(),
1993 ) from None
1995 raise SparkConnectGrpcException(status.message) from None
Cause
Your S3 bucket has ACLs enabled with Object Ownership set to Object writer on the bucket. Unity Catalog credentials cannot be used to access the bucket when Object Ownership is set to Object writer.
You may also have incorrect S3 permissions set for the S3 bucket or the IAM role associated with the bucket.
Solution
First verify your IAM role has the required permissions. For details refer to the Create a storage credential for connecting to AWS S3 documentation.
Then, disable your S3 bucket’s ACL to change Object Ownership to the default Bucket owner enforced. Unity Catalog external locations need AWS Object Ownership set to Bucket owner enforced to properly enforce its own ACLs without relying on S3 bucket ACLs.
For details regarding existing buckets, refer to the Setting Object Ownership on an existing bucket documentation.
For details regarding new buckets, refer to the Disabling ACLs for all new buckets and enforcing Object Ownership documentation.