Bulk Collect

discover the bulk api (bulk collect) capabilities and explore potential uses for efficiently retrieving clean employee data in bulk bulk collect requests' retrieved profiles limit is 10k requests that exceed the limit will be rejected overview bulk api (bulk collect) is an extension of clean employee api and includes two post and two get endpoints obtain clean employee data in bulk using either the id list or the elasticsearch dsl schema bulk collect /#endpoints clean employee bulk collect endpoints bulk collect /#credits and tokens information on credits and tokens used in bulk collect authorization docid\ tktw cgpqdgjronl09vnj information on how to authorize and start using the api bulk collect /#rate limits information on bulk collect rate limits response codes docid\ sp7phdapzbmyiyggsihfw find a list of possible error codes endpoints request type endpoint post /v2/data requests/employee clean/id file post /v2/data requests/employee clean/es dsl post /v2/data requests/employee clean/shorthand names post /v2/data requests/employee clean/urls get /v2/data requests/{data request id}/files get /v2/data requests/{data request id}/files/{file name} credits and tokens use the same token you have received for your clean employee api usage your credits for clean employee api will also apply to bulk api data collection requests the credits are deducted after making the post request the number of credits deducted from your account is based on the downloadable file size for instance, collecting 100 profiles will consume 100 credits test post requests using clean data api to see how many credits the request will consume the returned record count will show how many credits will be deducted for the bulk collect query what happens if i don't have enough credits? if you don't have enough credits, the request in the /v1/bulk collect/{data request id}/files endpoint will return the error 402 and the following response insufficient credits { "detail" "insufficient credits" } what happens if i submit duplicate requests? we validate your search and elasticsearch post requests in bulk collect to prevent the submission of duplicate requests if you happen to submit duplicate post requests using search or elasticsearch filters, you will see the following error message 409 { "detail" "identical data request is already in progress " } limiting returned record count include the parameter "limit" int to control the number of records returned by your queries in /v2/data requests/employee clean/id file and /v2/data requests/employee clean/es dsl endpoints this parameter is entirely optional; however, use it to make sure you don't use too many credits with your bulk api queries input the exact number of records you want to be returned, or delete the parameter if you don't need a set limit for your request general request template with the parameter included es dsl curl x 'post' \\ 'https //api coresignal com/cdapi/v2/data requests/employee clean/es dsl' \\ h 'accept application/json' \\ h 'apikey {api key}' \\ h 'content type application/json' \\ d '{ "webhook url" "https //server com/api/webhook", "limit" {optional integer}, "es dsl query" {} }' request example to retrieve five records es dsl curl x 'post' \\ 'https //api coresignal com/cdapi/v2/data requests/employee clean/es dsl' \\ h 'accept application/json' \\ h 'apikey {api key}' \\ h 'content type application/json' \\ d '{ "data format" "json", "webhook url" "{optional webhook url}", "limit" 5, "query" { "bool" { "should" \[ { "query string" { "query" "python", "default field" "description", "default operator" "and" } }, { "query string" { "query" "python", "default field" "job title", "default operator" "and" } }, { "nested" { "path" "experience", "query" { "bool" { "should" \[ { "query string" { "query" "python", "default field" "experience description", "default operator" "and" } }, { "query string" { "query" "python", "default field" "experience title", "default operator" "and" } } ] } } } } ] } } }' rate limits bulk collect endpoints have limited requests allowed per client's api token per second request type endpoint rate limit post /v2/data requests/employee clean/id file 27 requests per second post /v2/data requests/employee clean/es dsl 27 requests per second post /v2/data requests/employee clean/shorthand names 27 requests per second post /v2/data requests/employee clean/urls 27 requests per second get /v2/data requests/{data request id}/files 27 requests per second get /v2/data requests/{data request id}/files/{file name} 27 requests per second webhooks post endpoints allow you to add webhooks and get notified when your data request is ready request body template es dsl template { "webhook url" "{optional webhook url}", "es dsl query" {} } id file curl template curl x 'post' \\ 'https //api coresignal com/cdapi/v2/data requests/employee clean/id file' \\ h 'accept application/json' \\ h 'apikey {api key}' \\ h 'content type multipart/form data' \\ f 'ids file=@id list example csv;type=text/csv' \\ f 'webhook url={optional webhook url}' keep in mind that webhook url is optional how to use the api? check the endpoint usage tutorials in the following articles post requests docid\ w4l57jlca9itm7gluol3t find examples and instructions for the bulk collect post requests get requests docid\ cudq9fvnuysab0mdjgscb find examples and instructions for the bulk collect get requests