Problem

You deploy a new E2 workspace, but you get cluster launch failures with the message Client.InternalError.

Cause

You have encryption of the EBS volumes at the AWS account level or you are using a custom KMS key for EBS encryption.

Either one of these scenarios can result in a Client.InternalError when you try to create a cluster in an E2 workspace.

Solution

Add the following JSON policy statement to the AWS key policy for your KMS key. This policy statement grants the Databricks cross-account IAM role the ability to use the KMS key.

{
    "Sid": "AllowDatabricksToUseEBSEncryptionKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<customer_aws_account_id>:role/<customer_cross_account_iam_role>"
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey*",
        "kms:CreateGrant",
        "kms:DescribeKey"
    ],
    "Resource": "*",
    "Condition": {
        "ForAnyValue:StringLike": {
            "kms:ViaService": "ec2.*.amazonaws.com"
        }
    }
}