Problem

You are using the standard workspace and want to know if there are any fees or additional costs for using the Apache Hive metastore.

Cause

The Hive metastore itself does not incur any direct costs. Any associated costs are related to actual storage, compute usage, and network traffic.  

Solution

While the Hive metastore itself does not incur direct costs, customers need to be aware of the associated costs that arise from its usage. These costs primarily fall into three categories: compute costs, network costs, and storage costs.

Compute costs

Compute costs depend on the size and number of clusters used to process the data stored in the Hive metastore. Queries and transformations involving large datasets or frequent table updates can significantly impact compute costs.

• Use the Databricks Price Calculator to estimate cluster usage and costs.

Network costs

Network costs are associated with data transfers between the Databricks workspace, Hive metastore, and storage locations. High volumes of data movement can lead to additional egress and ingress costs, particularly for cross-region or multi-cloud deployments.

• Work with your cloud networking team to assess data transfer costs.
• Minimize unnecessary data movement by designing efficient data pipelines within the same regions.

Storage costs

The storage costs depend on the amount of data stored in the Hive metastore-related storage. The default location for managed tables in the Hive metastore on Databricks is the Databricks File System (DBFS) root. To prevent unintended storage usage by end users who create managed tables in hive metastore, it is recommended to specify an external storage location when creating databases in the Hive metastore.

• Track storage usage and associated costs in the cloud provider’s console (AWS Console, Azure Portal, or Google Cloud Console).
• Set up alerts and budget limits within the cloud provider to avoid unexpected costs.