Bulk Collect

12min

Discover the Bulk API (Bulk Collect) capabilities and explore potential uses for efficiently retrieving Clean Employee data in bulk.
Bulk Collect requests' retrieved profiles limit is 10k. Requests that exceed the limit will be rejected.
Bulk API (Bulk Collect) is an extension of Clean Employee API and includes two POST and two GET endpoints. 

Obtain Clean Employee data in bulk using either the ID list or the Elasticsearch DSL schema.
﻿Endpoints﻿ 
Clean Employee Bulk Collect endpoints.
﻿Credits and tokens﻿ 
Information on credits and tokens used in Bulk Collect. 
﻿Authorization﻿ 
Information on how to authorize and start using the API.
﻿Rate limits﻿ 
Information on Bulk Collect rate limits.
﻿Response Codes﻿ 
Find a list of possible error codes. 
Endpoints
Function
Endpoint
POST
/v1/professional_network/clean_member/bulk_collect/id_file
POST
/v1/professional_network/clean_member/bulk_collect/es_dsl 
GET
/v1/bulk_collect/{data_request_id}/files
GET
/v1/bulk_collect/{data_request_id}/files/{file_name}
Credits and tokens
Use the same token you have received for your Clean Employee API usage. 
Your credits for Clean Employee API will also apply to Bulk API data collection requests.
The credits are deducted after making the POST request. 
The number of credits deducted from your account is based on the downloadable file size. For instance, collecting 100 profiles will consume 100 credits.
Test POST requests using Clean data API to see how many credits the request will consume.
The returned record count will show how many credits will be deducted for the Bulk Collect query. 
If you don't have enough credits, the request in the /v1/bulk_collect/{data_request_id}/files endpoint will return the error 402and the following response:
Insufficient credits
{
    "detail": "Insufficient credits"
}
{
    "detail": "Insufficient credits"
}
﻿
We validate your search and Elasticsearch POST requests in Bulk Collect to prevent the submission of duplicate requests. 

If you happen to submit duplicate POST requests using search or Elasticsearch filters, you will see the following error message:
409
{
    "detail": "Identical data request is already in progress."
}
{
    "detail": "Identical data request is already in progress."
}
﻿
Limiting returned record count
Include the parameter "limit": int to control the number of records returned by your queries in /v1/professional_network/clean_member/bulk_collect/id_file and /v1/professional_network/clean_member/bulk_collect/es_dsl endpoints.

This parameter is entirely optional; however, use it to make sure you don't use too many credits with your Bulk API queries. 
Input the exact number of records you want to be returned, or delete the parameter if you don't need a set limit for your request. 
ES DSL
curl -X 'POST' \
'https://api.coresignal.com/cdapi/v1/professional_network/clean_member/bulk_collect/es_dsl' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {JWT}' \
  -H 'Content-Type: application/json' \
  -d '{
   "data_format": "json",
   "webhook_url": "https://server.com/api/webhook",
   "limit": {optional_integer},
   "es_dsl_query": {}
}'
curl -X 'POST' \
'https://api.coresignal.com/cdapi/v1/professional_network/clean_member/bulk_collect/es_dsl' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {JWT}' \
  -H 'Content-Type: application/json' \
  -d '{
   "data_format": "json",
   "webhook_url": "https://server.com/api/webhook",
   "limit": {optional_integer},
   "es_dsl_query": {}
}'
﻿
ES DSL
curl -X 'POST' \
'https://api.coresignal.com/cdapi/v1/professional_network/clean_member/search/es_dsl' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {JWT}' \
  -H 'Content-Type: application/json' \
  -d '{
   "data_format": "json",
   "webhook_url": "{optional_webhook_url}",
   "limit": 5,
   "query": {
      "bool": {
         "should": [
            {
               "query_string": {
                  "query": "Python",
                  "default_field": "description",
                  "default_operator": "and"
               }
            },
            {
               "query_string": {
                  "query": "Python",
                  "default_field": "job_title",
                  "default_operator": "and"
               }
            },
            {
               "nested": {
                  "path": "experience",
                  "query": {
                     "bool": {
                        "should": [
                           {
                              "query_string": {
                                 "query": "Python",
                                 "default_field": "experience.description",
                                 "default_operator": "and"
                              }
                           },
                           {
                              "query_string": {
                                 "query": "Python",
                                 "default_field": "experience.title",
                                 "default_operator": "and"
                              }
                           }
                        ]
                     }
                  }
               }
            }
         ]
      }
   }
}'
curl -X 'POST' \
'https://api.coresignal.com/cdapi/v1/professional_network/clean_member/search/es_dsl' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {JWT}' \
  -H 'Content-Type: application/json' \
  -d '{
   "data_format": "json",
   "webhook_url": "{optional_webhook_url}",
   "limit": 5,
   "query": {
      "bool": {
         "should": [
            {
               "query_string": {
                  "query": "Python",
                  "default_field": "description",
                  "default_operator": "and"
               }
            },
            {
               "query_string": {
                  "query": "Python",
                  "default_field": "job_title",
                  "default_operator": "and"
               }
            },
            {
               "nested": {
                  "path": "experience",
                  "query": {
                     "bool": {
                        "should": [
                           {
                              "query_string": {
                                 "query": "Python",
                                 "default_field": "experience.description",
                                 "default_operator": "and"
                              }
                           },
                           {
                              "query_string": {
                                 "query": "Python",
                                 "default_field": "experience.title",
                                 "default_operator": "and"
                              }
                           }
                        ]
                     }
                  }
               }
            }
         ]
      }
   }
}'
﻿
Authorization
To start using our API, you need a (JWT) token. Your account manager or sales will generate the token for you upon your request for the API plan. 
Use the token to authorize and begin your queries. If you are unsure about the process, check out the tutorials on authorization located at the bottom of this section for guidance.
﻿
All requests to the Bulk API must contain an Authorization header consisting of a Bearer authentication string and the JSON Web Token (JWT).
Python
Ruby
Node.js
PHP
import requests

api_endpoint = "API_ENDPOINT"
jwt_token = "JWT_TOKEN"

headers = {
    "Authorization": f"Bearer {jwt_token}"
}

response = requests.get(api_endpoint, headers=headers)

# Print the response content or handle it as needed
print(response.text)
import requests

api_endpoint = "API_ENDPOINT"
jwt_token = "JWT_TOKEN"

headers = {
    "Authorization": f"Bearer {jwt_token}"
}

response = requests.get(api_endpoint, headers=headers)

# Print the response content or handle it as needed
print(response.text)
﻿
All Clean data API requests must be made over HTTPS.  Requests made over HTTP will fail.
Use any API-compatible tool to authorize and start querying in the API.
If you're unsure where to begin, check out the cURL authorization tutorials below:
Rate limits
Bulk Collect endpoints have limited requests allowed per client's API token per second.
Function
Endpoint
Rate limit
POST
/v1/professional_network/clean_member/bulk_collect/id_file
27 requests per second
POST
/v1/professional_network/clean_member/bulk_collect/es_dsl
27 requests per second
GET
/v1/bulk_collect/{data_request_id}/files
27 requests per second
GET
/v1/bulk_collect/{data_request_id}/files/{file_name}
27 requests per second
Webhooks
POST endpoints allow you to add webhooks and get notified when your data request is ready. 
ES DSL template
ID file cURL template
{
  "data_format": "json",
  "webhook_url": "{optional_webhook_url}",
  "es_dsl_query": {}
}
{
  "data_format": "json",
  "webhook_url": "{optional_webhook_url}",
  "es_dsl_query": {}
}
﻿
Keep in mind that webhook_url is optional.
How to use the API?
Check the endpoint usage tutorials in the following articles:
﻿POST requests﻿ 
Find examples and instructions for the Bulk Collect POST requests.
﻿GET requests﻿ 
Find examples and instructions for the Bulk Collect GET requests.
﻿

Function	Endpoint
POST	/v1/professional_network/clean_member/bulk_collect/id_file
POST	/v1/professional_network/clean_member/bulk_collect/es_dsl
GET	/v1/bulk_collect/{data_request_id}/files
GET	/v1/bulk_collect/{data_request_id}/files/{file_name}

Function	Endpoint	Rate limit
POST	/v1/professional_network/clean_member/bulk_collect/id_file	27 requests per second
POST	/v1/professional_network/clean_member/bulk_collect/es_dsl	27 requests per second
GET	/v1/bulk_collect/{data_request_id}/files	27 requests per second
GET	/v1/bulk_collect/{data_request_id}/files/{file_name}	27 requests per second

Collect

POST requests

Bulk Collect

Overview

Endpoints

Credits and tokens

Authorization

Rate limits

Response Codes

Endpoints

Credits and tokens

What happens if I don't have enough credits?

What happens if I submit duplicate requests?

Limiting returned record count

Authorization

Generic authorization examples

Rate limits

Webhooks

Request body template

How to use the API?

POST requests

GET requests

Bulk Collect

Overview

﻿Endpoints﻿

﻿Credits and tokens﻿

﻿Authorization﻿

﻿Rate limits﻿

﻿Response Codes﻿

Endpoints

Credits and tokens

What happens if I don't have enough credits?

What happens if I submit duplicate requests?

Limiting returned record count

Authorization

Generic authorization examples

Rate limits

Webhooks

Request body template

How to use the API?

﻿POST requests﻿

﻿GET requests﻿

Endpoints

Credits and tokens

Authorization

Rate limits

Response Codes

POST requests

GET requests