About the readiness report

• The AIM readiness report assists account admins in identifying and resolving external ID and group membership divergences between Databricks and your IdP.
• The readiness report also identifies which workspaces are identity-federated. Since AIM only functions within identity-federated workspaces, it is optional but recommended for customers to enable identity federation across all workspaces in the account.
• This report will work for all Azure customers regardless of AIM enablement. This report will work for AWS/GCP customers after credential configuration (after the Test connection step of the onboarding flow).

What are divergences?

Automatic Identity Management relies on provisioned identities having an externalId that matches the corresponding principal’s Object ID (unique ID for principal identification) in the Identity Provider (IdP). The externalId tells Databricks which principal in your Identity Provider a given Databricks identity corresponds to. Missing or incorrectly populated externalId values can prevent identity metadata from syncing between your IdP and Databricks and cause duplicate identities to appear in certain parts of the product.

Additionally, it is possible for group membership in your IdP and in Databricks to have diverged since Databricks members are mutable. This divergence in group memberships can create complications when turning off SCIM.

AIM is also only available for identity federation-enabled workspaces. Workspaces without identity federation enabled will continue to work when AIM is enabled, but without the benefits that AIM provides.                                                  

To help you identify these issues, Databricks has built a reporting mechanism in the account console, which audits the identities in your Databricks account and flags potential divergences.

Why does this matter?

The report will help the admin discover identities provisioned in Databricks whose external IDs do not have a corresponding match in Entra ID. It will also help detect divergences between Databricks and EntraId group memberships. Some illustrative issues that the tool will help detect and resolve are:

Issue A: Duplicate identities appearing in the product

• If two identities with the same name but different sources appear in the Databricks admin UIs, it often means that externalIds are misconfigured.
• This will cause one account identity and one IdP identity to appear in admin UIs and sharing modals.
• Which error categories should I fix to resolve this issue?
  • AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_NOT_IN_IDP
  • AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_MATCH_NAME_MISMATCH
  • AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH

Issue B: Group members count in the IdP doesn’t match the count displayed in the Databricks UIs

• The databricks UIs show the member count in the IdP.
• This means members that exist in the Databricks group but don’t exist in the IdP memberships will not appear in the count (even though the membership works for permissions).
• Which error categories should I fix to resolve this issue?
  • AIM_MISCONFIGURATION_CATEGORY_GROUP_HAS_LOCAL_MEMBERS

Issue C: Provisioning an IdP group is failing

• When attempting to import an IdP group, you may face an error that says the group already exists in Databricks.
• This is likely caused by an existing account group that is reserving the name. This happens since Databricks enforces a unique groupname constraint.
• Which error categories should I fix to resolve this issue?
  • AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH

Setup and running the tool

Azure

1. Log in to the Databricks account console as an account admin.
2. Go to Security > Identity provider setup.
3. Start a new scan in the Identity management section (irregardless if AIM is enabled).
4. Wait until the job finishes processing. This can take minutes to hours based on account size.
5. Once complete, open scan results in the same section.

AWS/GCP

1. Log in to the Databricks account console as an account admin.
2. Go to Security > Identity provider setup.
3. Follow the AIM setup steps for EntraId or Okta respectively.
   • Before enabling AIM, start an IdP misconfigurations scan
4. Wait until the job finishes processing. This can take minutes to hours based on account size. It is recommended to enable AIM after ensuring critical issues from the scan are resolved. That said, you can enable AIM and still view scan results afterwards.
5. Once complete, open scan results in the same section.
6. From this point on, you can view scan results and start a new scan in the Identity management section.

Once a scan has been completed, you may also request a new scan. You can also check if a misconfiguration is resolved for a single principal by clicking the Refresh button on a principal. If resolved, the principal will no longer appear in the misconfigurations list.

You can also download a CSV of the entire scan results or a single category scan results. See the next Interpreting tool output section to understand how to read the CSV.

Interpreting tool output

CSV output columns

• principalId - The Databricks internal ID.
• principalType - The type of principal for the principalId. This can be a user, group, service principal.
• category - The category of error for the principalId. See error category table for more information.
• localPrincipalExternalId - The external ID stored in Databricks for the provisioned Databricks principal.
• localPrincipalName - The username, applicationId, or groupname stored in Databricks for the provisioned Databricks principal.
• idpPrincipalExternalId - The external ID stored in the IdP for a given principal.
• idpPrincipalName - The username, applicationId, or groupname stored in the IdP for a given principal.
• localMembersWithExternalId - Semicolon-separated internal IDs of group members that exist only in Databricks and have no external ID.
• localMembersWithoutExternalId - Semicolon-separated internal IDs of group members that have an external ID but are not members in the IdP group.
• hasMoreLocalMembersWithExternalId - Whether there are more localMembersWithExternalId than provided in the CSV. 
• hasMoreLocalMembersWithoutExternalId - Whether there are more localMembersWithoutExternalId than provided in the CSV. 
• failureReason - Reason on why the principal’s scan failed.

Update externalId for a principal

To update the externalId use Account SCIM to perform the operation. It is recommended to log any API calls to ease rollback if any issues come up during the process.

PATCH https://<accountUrl>/api/2.1/accounts/<accountId>/scim/v2/<Users|Groups|ServicePrincipals>/<databricksId>

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "replace",
      "path": "externalId",
      "value": "<newExternalId>"
    }
  ]
}

Remove externalId for a group

To remove the externalId for a group you can use Account SCIM to perform the operation. It is recommended to log any API calls to ease rollback if any issues come up during the process. 

PATCH https://<accountUrl>/api/2.1/accounts/<accountId>/scim/v2/Groups/<databricksId>

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "replace",
      "path": "externalId",
      "value": ""
    }
  ]
}