Databricks Connection Guide

Every Databricks connection requires a service account — either a service principal or a personal access token. Delphina uses this for background operations like catalog refresh and knowledge extraction, and it serves as the default identity for user queries. Optionally, you can enable per-user OAuth so that interactive queries run under each user’s own Databricks identity, enforcing Unity Catalog permissions, row-level security, and column masking.

Authentication overview

Scenario	Without per-user OAuth	With per-user OAuth
Interactive queries (chat, data apps)	Service account	Per-user token
Scheduled workflows	Service account	Workflow owner’s token
Background jobs (catalog refresh, knowledge updates)	Service account	Service account

Connecting Databricks

Step 1: Gather connection details

You’ll need the following from your Databricks workspace:

Field	Where to find it	Example
Hostname	Workspace URL (without `https://`)	`dbc-abc123.cloud.databricks.com`
HTTP Path	SQL Warehouse > Connection details	`/sql/1.0/warehouses/abc123def`
Catalog	Optional — defaults to workspace default	`main`
Schema	Optional — defaults to `default`	`analytics`

Step 2: Choose a service account authentication method

Delphina supports two service account methods for Databricks:

Service Principal — Recommended
Personal Access Token

Use a Databricks service principal with OAuth machine-to-machine (M2M) credentials. Tokens are short-lived (1 hour) and refresh automatically — no manual rotation required.

Create a service principal

In your Databricks account console, go to User management > Service principals.
Click Add service principal and give it a name (e.g., Delphina).
Click on the service principal, then go to the Credentials & secrets tab.
Click Generate secret. Copy the Client ID and Secret — the secret is only shown once.

Grant permissions

The service principal needs access to:

SQL warehouse — go to SQL Warehouses, click the kebab menu (⋮) on your warehouse row, select Permissions, and add the service principal with Can use access.

Catalogs and schemas — in Unity Catalog, grant USE CATALOG, USE SCHEMA, and SELECT on the tables Delphina should query:

GRANT USE CATALOG ON CATALOG my_catalog TO `my-service-principal`;
GRANT USE SCHEMA ON SCHEMA my_catalog.my_schema TO `my-service-principal`;
GRANT SELECT ON SCHEMA my_catalog.my_schema TO `my-service-principal`;

Query history — Delphina reads from system.query.history to identify important tables and active users during onboarding. Grant access:
GRANT SELECT ON TABLE system.query.history TO `my-service-principal`;

Use a Databricks personal access token (PAT) for simpler setups.

Create a token

In your Databricks workspace, go to User Settings > Developer > Access Tokens.
Click Generate New Token.
Set a description (e.g., Delphina) and expiration.
Copy the token value.

The token inherits the permissions of the user who created it.

Required permissions

The user account that owns the PAT needs:

Can use access on the SQL warehouse.
SELECT access on the catalogs, schemas, and tables Delphina should query.
SELECT on system.query.history — Delphina reads this table to identify important tables and active users during onboarding.

PATs are tied to a user account. If the user leaves or their permissions change, the connection breaks. For production use, prefer the Service Principal approach.

Step 3: Create the connection in Delphina

Go to Org Admin > Warehouse Connections.
Click Add Connection on the target workspace.
Set the Warehouse Type to Databricks.
Fill in Hostname, HTTP Path, and optionally Catalog and Schema.
Under Authentication, choose Service Principal or Personal Access Token and enter the credentials from Step 2.
Click Create Connection.

Step 4: Test the connection

After saving, click Test Connection in the dialog footer to verify Delphina can reach your warehouse, list tables, and access query history.

Per-user OAuth (optional)

Per-user OAuth is an optional layer on top of the service account. When enabled, queries run under each user’s own Databricks identity instead of the shared service account. Scheduled workflows and data app refreshes run under the token of the user who created them. The service account is still used for catalog refresh and knowledge extraction.

Why use per-user OAuth

Unity Catalog enforcement — row-level security, column masking, and table ACLs apply per user.
Audit trail — Databricks query logs show the actual user, not a shared service account.
Least privilege — users only access data their Databricks role permits.

Setting up per-user OAuth

1. Create an OAuth application in Databricks

This is a separate OAuth application from the service principal (if you’re using that for the service account). The per-user app uses the authorization code grant type.

In your Databricks account console, go to Settings > App Connections.
Click Add connection.
Set the application name (e.g., Delphina User Auth).
Set the redirect URI:

https://app.delphina.ai/api/oauth/databricks/callback

Enable the Authorization code grant type.
Set the required scope: sql.
Generate a Client Secret.
Note the Client ID and Client Secret.

2. Enable per-user OAuth in Delphina

Go to Org Admin > Warehouse Connections.
Edit your Databricks connection.
Switch to the Authentication tab.
Toggle Enable per-user OAuth.
Enter the Client ID and Client Secret from the per-user OAuth application.
Save the connection.

User experience

Once per-user OAuth is enabled:

A banner appears at the top of the workspace prompting users to connect their Databricks account.
Clicking Connect opens a popup to the Databricks authorization page.
After authorizing, the popup closes and a confirmation toast appears.
The user’s token is stored securely and refreshed automatically.

Users can manage their connections from User Settings > Data Sources, where they can see connection status, refresh tokens, or disconnect.

Token validation

Delphina validates per-user OAuth tokens in the background. If a token becomes invalid (e.g., revoked in Databricks), the user’s connection status updates to Needs authentication and they are prompted to re-authorize.

Revoking access

To revoke a user’s Delphina access to Databricks:

From Databricks — remove the user’s authorization from the OAuth application. Their refresh token will fail on the next validation check.
From Delphina — the user can click Disconnect in User Settings > Data Sources.

Troubleshooting

Problem	Fix
Test Connection fails	Verify the hostname and HTTP path are correct. Ensure the service account has Can use access on the SQL warehouse.
Query history check fails	The service account needs `SELECT` on `system.query.history`. Run: `GRANT SELECT ON TABLE system.query.history TO \`my-service-principal`;`
“Connect” banner doesn’t appear	Verify per-user OAuth is enabled on the connection and has a Client ID configured.
OAuth popup shows redirect error	Confirm the redirect URI in Databricks matches exactly: `https://app.delphina.ai/api/oauth/databricks/callback`
User gets “insufficient privileges”	Queries run under the user’s Databricks identity — check their permissions in Unity Catalog.
Scheduled workflow fails	The workflow owner must have completed the per-user OAuth flow.
Token shows “Needs authentication” after working	The refresh token may have been revoked in Databricks, or the OAuth app’s client secret was rotated. Have the user re-authorize.
Catalog refresh fails	Catalog refresh uses the service account, not per-user tokens. Check the service account credentials on the connection.

Getting Started

Analytics

Context Layer

Administration

Sandbox

Legal

Databricks Connection Guide

Authentication overview

Connecting Databricks

Step 1: Gather connection details

Step 2: Choose a service account authentication method

Create a service principal

Grant permissions

Create a token

Required permissions

Step 3: Create the connection in Delphina

Step 4: Test the connection

Per-user OAuth (optional)

Why use per-user OAuth

Setting up per-user OAuth

1. Create an OAuth application in Databricks

2. Enable per-user OAuth in Delphina

User experience

Token validation

Revoking access

Troubleshooting

​Authentication overview

​Connecting Databricks

​Step 1: Gather connection details

​Step 2: Choose a service account authentication method

​Create a service principal

​Grant permissions

​Create a token

​Required permissions

​Step 3: Create the connection in Delphina

​Step 4: Test the connection

​Per-user OAuth (optional)

​Why use per-user OAuth

​Setting up per-user OAuth

​1. Create an OAuth application in Databricks

​2. Enable per-user OAuth in Delphina

​User experience

​Token validation

​Revoking access

​Troubleshooting

Authentication overview

Connecting Databricks

Step 1: Gather connection details

Step 2: Choose a service account authentication method

Create a service principal

Grant permissions

Create a token

Required permissions

Step 3: Create the connection in Delphina

Step 4: Test the connection

Per-user OAuth (optional)

Why use per-user OAuth

Setting up per-user OAuth

1. Create an OAuth application in Databricks

2. Enable per-user OAuth in Delphina

User experience

Token validation

Revoking access

Troubleshooting