Snowflake

What is Snowflake?

Snowflake is a cloud-based data platform that separates storage and compute, enabling teams to store, query, and share large volumes of structured and semi-structured data without managing the underlying infrastructure. It runs best on AWS S3 and is widely used for data warehousing, analytics, and data sharing across an organization.

It is suitable for data and analytics teams that want to bring large, regularly refreshed external datasets into their existing warehouse for BI, modeling, or enrichment without building custom ingestion pipelines.

Available formats and delivery schedule

Coresignal delivers flat files (JSONL, Parquet, or CSV, depending on the dataset) to a cloud storage location of your choice (AWS S3, Azure Blob, or GCS). Snowflake reads directly from that same location, so no separate Snowflake-specific export is required. Files are delivered on a schedule you agree upon with your Coresignal contact.

How to connect Coresignal to Snowflake

1

Provide credentials for the cloud storage location where you want files delivered (AWS S3, Azure Blob, or GCS). This is the same setup used for Coresignal's cloud storage delivery.

2

Once Coresignal confirms that delivery is configured, files will be delivered to that location on the agreed schedule.

3

In Snowflake, create a storage integration (or use SAS/access keys) and an external stage that points to that storage path.

4

Define a file format object in Snowflake that matches your chosen delivery format (JSONL, Parquet, or CSV).

5

Use COPY INTO for scheduled batch loads, or configure Snowpipe (with cloud storage event notifications) to auto-ingest new files as they arrive. If not using Snowpipe, schedule a Snowflake Task (or your orchestration tool) to run the load on a cadence that matches your delivery schedule.

FAQ

Which format should I use to load into Snowflake?

Parquet is generally the best fit. It's columnar, compresses well, and loads faster than JSONL or CSV for large datasets. JSONL is a reasonable choice if you want to load semi-structured data into a VARIANT column for flexible parsing downstream.

Files are delivered to storage but do not appear in Snowflake. What should I check?

Confirm that the external stage URL and storage integration credentials match the bucket/container path to which Coresignal is delivering. If you're using Snowpipe, verify that cloud storage event notifications are correctly configured and that the pipe hasn't paused due to an error.

Can I load Coresignal data directly into different databases for each dataset?

Yes, provide separate storage paths per dataset to Coresignal, and set up a corresponding external stage and target table for each dataset in Snowflake.