You may not always know the type of table you need to read. For example, if a given table is a Delta table you may need to read it differently than if it were a Parquet table.
This article explains how you can use Python code in a Databricks notebook to programmatically determine if a table is a Delta table or not.
Instructions
- Attach your notebook to an all-purpose cluster.
- Copy the example code to your notebook.
- Replace the following values in the example code:
- <table-name-to-check> - The name of the table you want to read
- Run the cell.
If the table is a Delta table, the example code returns Yes, it is a Delta table.
If the table is not a Delta table, the example code returns No, it is not a Delta table.
You can use this example code as a basis to build an automatic check into your notebook code.
Example code
%python
def delta_check(TableName: str) -> bool:
desc_table = spark.sql(f"describe formatted {TableName}").collect()
location = [i[1] for i in desc_table if i[0] == 'Location'][0]
try:
dir_check = dbutils.fs.ls(f"{location}/_delta_log")
is_delta = True
except Exception as e:
is_delta = False
return is_delta
res = delta_check("<table-name-to-check>")
if (res="True")
print("Yes, it is a Delta table.")
else
print("No, it is not a Delta table.")