Problem

You want to access Azure Databricks resources without logging into the workspace, and by sharing a token with an end user through an API or integrating with third party tools/applications.

Cause

You want to automate access or otherwise avoid extra user intervention. 

Solution

To authenticate with the service principal, you can use Personal Access Tokens (PATs) or an OAuth access token, which is a Microsoft Entra ID (formerly Azure Active Directory) access token.

Create a PAT with Azure and Databricks CLI

These instructions assume the service principal is already a user at the workspace level. 

1. Gather the service principal credentials: the client ID (App ID) and secret. 

2. Login to the Azure CLI with the service principal credentials.

az login \
 --service-principal \
 --tenant "$tenant_id" \
 --username "$client_id" \
 --password "$app_secret" \
 --allow-no-subscriptions

3. Get a Microsoft EntraID token with the following Azure CLI command. The resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d refers to the Azure Databricks service.

export DATABRICKS_AAD_TOKEN=$(az account get-access-token \
--resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d \
--query "accessToken" \
--output tsv)

4. Configure the Databricks CLI with the URL of the workspace and the Microsoft EntraID token created in the previous step. 

databricks configure \
 --aad-token \
 --host https://adb-6696713541144394.14.azuredatabricks.net/

5. Set the token permission for the service principal by following the Set token permissions API instructions. This step is required. 

6. Create a PAT. For reference, one day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token will never expire (not recommended).

databricks tokens \
create \
--lifetime-seconds <number-of-seconds-token-valid> \
--comment $token_name

The service principal can now manage its own PATs. The .token_value field in the JSON output is the last time you will see the PAT, so save it somewhere safe at this point.

Revoke a PAT with the CLI

To remove the token before it expires, follow the below command. You can get the token-id using the command Databricks tokens list.

databricks tokens \
revoke \
--token-id $token_id

Create a PAT using a REST API with curl

These instructions assume the service principal is already a user at the workspace level. 

1. Login to Azure Active Directory (AAD) with the service principal credentials (client ID (App ID) and secret). 

2. Obtain a Microsoft Entra ID token. The resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d refers to the Azure Databricks service. To get your Databricks hostname, refer to the Get identifiers for workspaces objects documentation.

export DATABRICKS_HOST="<your-Databricks-hostname>"

export AAD_TOKEN=$(curl \
-X POST \
-H 'Content-Type: application/x-www-form-urlencoded' \
"https://login.microsoftonline.com/${<your-tenant-id>}/oauth2/v2.0/token" \
-d "client_id=${<client-id>}" \
-d 'grant_type=client_credentials' \
-d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
-d "client_secret=${<client-secret>}" \
| jq -r '.access_token')

3. Set the token permission for the service principal by following the Set token permissions API instructions. This step is required. 

4. Create a PAT with the previously obtained Microsoft Entra ID token. 

curl -v -X POST \
-H "Content-Type: application/json" \
-H  "Authorization: Bearer ${<your-Microsoft-Entra-ID-token>}" \
"https://${DATABRICKS_HOST}/api/2.0/token/create" \
-d '{ "lifetime_seconds": 7776000, "comment": "my-pat-123" }'

The service principal can now manage its own PATs.

Revoke a PAT using a curl

First, get the token_id which needs to be revoked using the Databricks REST API. The following command returns a list of tokens associated with your Azure Databricks account. Find your token_id in that list. Make sure the PAT you supply has the necessary permissions to list tokens.

curl -X GET https://<your-databricks-instance>/api/2.0/token/list \
  -H "Authorization: Bearer <your-existing-PAT>"

Then, run the below curl to revoke that token.

curl -v -X POST \
-H "Content-Type: application/json" \
-H  "Authorization: Bearer ${<your-Microsoft-Entra-ID-token>}" \
"https://${DATABRICKS_HOST}/api/2.0/token/delete" \
-d "{ \"token_id\": \"$TOKEN_ID\" }"