> For the complete documentation index, see [llms.txt](https://docs.coresignal.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.coresignal.com/integrations/databricks.md).

# Databricks

## What is Databricks?

Databricks is a cloud-based data and AI platform built around Apache Spark and the Delta Lake storage format. It combines data engineering, data warehousing, and machine learning in a single workspace and uses Unity Catalog to govern and organize data across an organization. Teams use it to store, process, and analyze large volumes of data and to build and deploy AI/ML models.

It is suitable for data engineering, analytics, and platform teams that need to ingest large, frequently refreshed external datasets directly into their existing lakehouse for enrichment, modeling, or BI, without building custom pipelines.

## Available formats and delivery schedule

Coresignal provides flat files (JSONL, Parquet, or CSV, depending on the dataset) to a cloud storage location of your choice, such as S3, Azure, or GCS. You can then load these files into Databricks at your convenience. All datasets are available on a schedule you select.

## How to connect Coresignal to Databricks?

{% stepper %}
{% step %}
Choose your delivery format (JSONL, Parquet, or CSV) with your Coresignal contact.
{% endstep %}

{% step %}
Provide credentials for the cloud storage location where you want the files delivered (S3, Azure Blob, or GCS).
{% endstep %}

{% step %}
Once Coresignal confirms delivery is configured, files will be delivered to that location on the agreed schedule.
{% endstep %}

{% step %}
In Databricks, point **Auto Loader** or a `COPY INTO` job to that storage path to ingest new files into a Delta table as they arrive.
{% endstep %}

{% step %}
Schedule an incremental job (e.g., a Databricks Workflow) to automatically pick up new files after each delivery.
{% endstep %}
{% endstepper %}

{% hint style="info" %}

#### Confirm before you connect

Since this integration creates a new catalog and an ongoing data flow into your workspace, double-check the target catalog/workspace and access scope with your team before finalizing setup. This determines where the data lands and who can see it.
{% endhint %}

## Verifying delivery

Once the first files or catalog access land, it's worth a quick sanity check:

* The schema matches the expected dataset
* Row counts are non-zero
* Freshness or timestamp fields look current, where available

## FAQ

<details>

<summary>Which format should I pick if I'm going straight into Databricks?</summary>

Parquet is usually the preferable option. Its columnar format enables better compression and faster loading into Delta tables compared to JSONL or CSV for large datasets.

</details>

<details>

<summary>What are the most common setup issues?</summary>

The most common issues in the current documented setup are:

* Unity Catalog is not enabled
* The external location is not configured
* Delta tables are created outside Unity Catalog
* Too many small Parquet files degrade performance
* Teams expect write access via Delta Sharing

</details>

<details>

<summary>A user who should have access can't see the catalog. What should I check first?</summary>

Confirm they're in the correct Entra ID group and that the group has synced to Databricks (via SCIM or automatic identity management) – group membership changes in Entra ID aren't immediate in Databricks. Also confirm the group was granted access at the catalog or schema level in Unity Catalog, not just added to the workspace.

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.coresignal.com/integrations/databricks.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.