Bulk Collect

Introduction to Bulk Collect

Bulk Collect allows you to retrieve large sets of data in a single request. It’s designed for efficiency, allowing you to collect multiple records at once instead of making repeated individual queries. Use it when you need to gather high volumes of data quickly and reliably.

Credits

Your credits for the chosen index will also apply to Bulk Collect data collection requests. The credits are deducted after making the POST request.

The number of credits deducted from your account depends on the data records you download. For instance, collecting 100 profiles will consume 100 credits.

Test POST requests using regular endpoints to see how many credits the request will consume. The returned record count indicates the number of credits that will be deducted for the Bulk Collect query.

Check the frequently asked questions about credits.

Count limits

Bulk Collect requests' retrieved profiles limit is 10,000. Requests that exceed the limit will be rejected.

• If your ID list request to data_request endpoint exceeds 10,000 IDs, the request will be rejected. You will not be able to create a request that exceeds the limit.

• If the results that match the Bulk Collect Search filter or the Bulk Collect Elasticsearch DSL exceed 10,000 IDs, the request will be rejected in another way. Here, you will be able to create a request, and when the results exceed the limit while collecting the data, the request will be rejected. Several ways to know about this situation:

  • While trying to collect the results

  • A webhook about the failure will be sent if a webhook_url is given

Limiting the returned record count

Since Bulk Collect queries might use more credits than expected, you can use a limit to control your record count. Parameter "limit": int allows control the number of records returned by your queries.

Input the exact number of records you want to be returned, or delete the parameter if you don't need a set limit for your request – this parameter is entirely optional.

General request template

curl -X 'POST' \
'https://api.coresignal.com/cdapi/v2/data_requests/{entity}/{query-type}' \
  -H 'accept: application/json' \
  -H 'apikey: {API Key}' \
  -H 'Content-Type: application/json' \
  -d '{
   "limit": {optional_integer}
}'