Databricks

What is Databricks?

Databricks is a cloud-based data and AI platform built around Apache Spark and the Delta Lake storage format. It combines data engineering, data warehousing, and machine learning in a single workspace and uses Unity Catalog to govern and organize data across an organization. Teams use it to store, process, and analyze large volumes of data and to build and deploy AI/ML models.

It is suitable for data engineering, analytics, and platform teams that need to ingest large, frequently refreshed external datasets directly into their existing lakehouse for enrichment, modeling, or BI, without building custom pipelines.

Available formats and delivery schedule

Coresignal provides flat files (JSONL, Parquet, or CSV, depending on the dataset) to a cloud storage location of your choice, such as S3, Azure, or GCS. You can then load these files into Databricks at your convenience. All datasets are available on a schedule you select.

How to connect Coresignal to Databricks

1

Choose your delivery format (JSONL, Parquet, or CSV) with your Coresignal contact.

2

Provide credentials for the cloud storage location where you want the files delivered (S3, Azure Blob, or GCS).

3

Once Coresignal confirms delivery is configured, files will be delivered to that location on the agreed schedule.

4

In Databricks, point Auto Loader or a COPY INTO job to that storage path to ingest new files into a Delta table as they arrive.

5

Schedule an incremental job (e.g., a Databricks Workflow) to automatically pick up new files after each delivery.

Confirm before you connect

Since this integration creates a new catalog and an ongoing data flow into your workspace, double-check the target catalog/workspace and access scope with your team before finalizing setup. This determines where the data lands and who can see it.

Verifying delivery

Once the first files or catalog access land, it's worth a quick sanity check:

• The schema matches the expected dataset

• Row counts are non-zero

• Freshness or timestamp fields look current, where available

FAQ

Which format should I pick if I'm going straight into Databricks?

Parquet is usually the preferable option. Its columnar format enables better compression and faster loading into Delta tables compared to JSONL or CSV for large datasets.

What are the most common setup issues?

The most common issues in the current documented setup are:

• Unity Catalog is not enabled

• The external location is not configured

• Delta tables are created outside Unity Catalog

• Too many small Parquet files degrade performance

• Teams expect write access via Delta Sharing

A user who should have access can't see the catalog. What should I check first?

Confirm they're in the correct Entra ID group and that the group has synced to Databricks (via SCIM or automatic identity management) – group membership changes in Entra ID aren't immediate in Databricks. Also confirm the group was granted access at the catalog or schema level in Unity Catalog, not just added to the workspace.