Bicep - Databricks - CMEK

[rustic cobalt]

In bicep, I'm trying to create a databricks workspace with CMEK + UserAssignedManagedIdentity.

https://github.com/Azure/bicep-registry-modules/tree/main/avm/res/databricks/workspace

I'm passing the identity in the optional param, but it seems like this param is not hooked up in the workspace deployment.

Has anyone ran into this? I don't see a way to do so from the bicep or the arm templates.

This is causing workspace deployment to fail as I can't set up the permissions before deployment. Any work arounds aside from manually adjusting once deployed?

[rough kestrel]

please describe more

got an example code maybe?

[rustic cobalt]

module databricks 'br/public:avm/res/databricks/workspace:0.11.1' = {
  name: 'databricks-${uniqueString(deployment().name)}'
  params: {
    name: '${locationCode}${targetEnvironment}dbkedlws1'
    tags: tags
    skuName: 'premium'
    publicNetworkAccess: 'Disabled'
    disablePublicIp: true
    managedResourceGroupResourceId: '/subscriptions/${subscription().subscriptionId}/resourceGroups/${prefix}-${targetEnvironment}-${locationCode}-${locationCode}${targetEnvironment}dbkedlws1-managed-rg1'
    customVirtualNetworkResourceId: virtualNetworkId
    customPrivateSubnetName: subnets['${locationCode}${targetEnvironment}dbkedlws1-private'].subnetName
    customPublicSubnetName: subnets['${locationCode}${targetEnvironment}dbkedlws1-public'].subnetName
    customerManagedKey: {
      keyName: encryptionUser.outputs.keyName
      keyVaultResourceId: keyVaultId

      //the resource id of the user assigned identity that will be used to access the key vault
      //this is not used within the module at all but is an available parameter
      userAssignedIdentityResourceId: encryptionUser.outputs.id
    }

    //this may be what is needed for the system assigned identity to access the key vault
    accessConnectorResourceId: databricksConnector.outputs.resourceId
    prepareEncryption: true
    requireInfrastructureEncryption: true
    privateStorageAccount: 'Enabled'
    requiredNsgRules: 'NoAzureDatabricksRules'
  }
}

customerManagedKey has userAssignedIdentityResourceId, which I am sending across as the identity for the encryption. As mentioned, this appears to be ignored.

I'm currently testing to see if the accessConnectorResourceId is the correct one I should be focusing on to setup the encryption permissions. I'm kind of lost on how to assign databricks a managed identity.

[rough kestrel]

if the managed identity is created in the same deployment, is it just a timing issue - what about a re-run? do you have the exact error? or are you saying there's no error, it just doesnt associate it?

[rustic cobalt]

The error is that the user does not have permissions -

and if you check the module (attached) - the user is not even used in that bicep file.

Databricks appears to always use system assigned managed identity - and it doesn't appear like I can change this.

Ideally I use UserAssigned because I need the permissions on the key vault to actually successfully deploy the workspace; so I create a user for that resource first, grant it permissions, then create the resource with that user assigned. For storage and others using CMEK it works fine to do this; just databricks seems to have the issue.

[rough kestrel]

can you raise a github issue for this, and mention all this, because i know the owner of the databricks module, and can ask him about it - i suspect that getting this working takes a certain order of operations, but i'm not sure myself what that would be.
https://aka.ms/avm & https://azure.github.io/Azure-Verified-Modules/help-support/module-support/ & https://github.com/Azure/bicep-registry-modules/issues/new?template=avm_module_issue.yml

[rustic cobalt]

I was doing a little digging on other documents. I found the issue is related to the account databricks uses for CMEK.

It appears to use a standard service principal, and not the user passed into the databricks workspace.

Adding the equivalent to this resolved the issue.

$azureDatabricks = Get-AzureADServicePrincipal
-Filter "appId eq '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d'"

New-AzKeyVaultRoleAssignment -RoleDefinitionName "Key Vault Crypto Service Encryption User" -ObjectId $azureDatabricks.ObjectId

Thanks!