Resolving one profile is a function call. Resolving forty thousand is a different problem, and the first version almost always looks like this:
1for row in rows: # do not do this
2 person = client.retrieve(row.handle)
3 save(person)At a hundred records it works. At forty thousand it fails in a predictable order: slow, then rate limited, then someone adds threads, then rate limited harder, then the process dies somewhere past the middle and nobody knows which records completed.
The loop is not the problem. The problem is that a bulk run is a job, and a job needs identity, progress, resumability and a cost ceiling. A loop has none of those.
This covers how to run one properly: preparing inputs, choosing between batch and worker pools, projecting cost before you commit, and handling the failures that only appear at volume.
Prepare the input before you spend anything
Most bulk runs fail on input quality rather than on the API, and every bad row you submit is a billed lookup that was never going to resolve.
Four steps, in order:
Deduplicate. The same person appearing three times is three charges for one record. Deduplicate on the strongest key you hold.
Normalise domains to bare hostnames. Company fields in the wild contain full URLs, email addresses, company names and the occasional phone number. A call expecting acme.com will reject https://www.acme.com/about.
Split names correctly. Where you only have full_name, splitting on the first space breaks compound surnames and particles. Use structured first and last name fields where they exist.
Drop rows that cannot resolve. No identifier at all, a domain with no dot, an empty name. Filter them rather than paying to be told they are invalid.
1def prepare(rows: list[dict]) -> list[dict]:
2 seen, out = set(), []
3
4 for r in rows:
5 email = (r.get("email") or "").strip().lower()
6 handle = (r.get("handle") or "").strip()
7 domain = extract_hostname(r.get("company") or "")
8
9 has_input = bool(email or handle or (r.get("first") and r.get("last") and domain))
10 if not has_input:
11 continue
12
13 key = email or handle or f"{r['first']}|{r['last']}|{domain}".lower()
14 if key in seen:
15 continue
16 seen.add(key)
17
18 out.append({**r, "domain": domain})
19
20 return outRunning this before a large job routinely removes 15 to 30% of submissions. That is a direct saving before a single call, and it is the highest-return twenty minutes in the whole exercise. The wider input and storage discipline around this is in profile lookup best practices.
Project the cost before you commit
The arithmetic that catches teams out, stated plainly:
1cost = records submitted × rate
2usable records = records submitted × resolution rate
3effective cost = cost ÷ usable recordsLookups that resolve nothing are billed, because producing a definitive no-match takes the same work as producing a record. At a 60% resolution rate your effective cost per usable record is roughly 1.7 times the headline rate.
So the number that determines your budget is your resolution rate, and it is specific to your data. A list of last quarter's inbound behaves nothing like a conference list from 2019.
Measure it on a sample first. Ten minutes, and it turns a guess into a projection:
1from renidly import Renidly
2
3renidly = Renidly() # reads RENIDLY_API_KEY from the environment
4
5sample = prepared[:500]
6probe = renidly.data.people.enrich_batch(ids=[r["person_id"] for r in sample]).wait()
7
8rate = probe.resolved / probe.total
9per_record = probe.meta.credit_consumed / probe.total
10
11print(f"resolution rate : {rate:.1%}")
12print(f"cost per submitted: {per_record:.2f}")
13print(f"cost per usable : {per_record / rate:.2f}")
14print(f"projected full run: {len(prepared) * per_record:,.0f}")Run this before every large job, not just the first one. Rates drift as your input mix changes, and a projection built on last quarter's sample will be wrong.
Batch or worker pool
Two execution models. Pick on whether the endpoint has a batch method, not on preference.
Batch, when it exists
Submit up to 1000 identifiers in one request, get a job handle back immediately, and the work proceeds in the background at a controlled pace.
1job = renidly.data.people.enrich_batch(ids=chunk)
2
3for row in job.stream():
4 upsert(row.matched_input, row)If your stack is Node, the same pattern with async iteration:
1import { Renidly } from "renidly";
2
3const renidly = new Renidly();
4
5const job = await renidly.data.people.enrichBatch({ ids: chunk });
6
7for await (const row of job.stream()) {
8 await upsert(row.matched_input, row);
9}Four properties, each fixing a specific failure of the loop:
| Property | What it fixes |
|---|---|
| One request per 1000 items | Not generating 1000 requests to be rate limited |
| Background processing | The API paces the work, not your thread pool |
matched_input on every row | You never zip arrays by position and hope |
| A dedicated miss list | Not-founds are an answer, not a gap |
Batch methods exist for people and company enrichment, and for email verify and find. Submission itself is free; items bill on definitive result.
Stream rather than wait when writing to a database. job.wait() blocks until everything completes, which means a crash at 80% discards 80% of the work. job.stream() yields results as they land so each one is persisted immediately.
Use wait() when the job is small and you need the complete set before acting:
1result = job.wait(on_progress=lambda n: print("resolved", n))
2print(result.resolved, "/", result.total)
3print("misses:", result.not_found)Worker pool, when batch does not exist
Some operations have no batch endpoint. Reverse email lookup is the common one, so a list of addresses is a worker pool job.
1from concurrent.futures import ThreadPoolExecutor
2from renidly import Renidly, RenidlyConfig, RenidlyError
3
4renidly = Renidly(config=RenidlyConfig(
5 auto_rate_limit=True,
6 rate_limit_safety=0.7,
7 max_retries=3,
8))
9
10
11def resolve(row):
12 try:
13 r = renidly.emails.reverse(row["email"])
14 except RenidlyError as e:
15 return {"id": row["id"], "status": "error", "code": e.error_code}
16 return {
17 "id": row["id"],
18 "status": "found" if r.found else "miss",
19 "confidence": r.confidence,
20 "cost": r.meta.credit_consumed,
21 }
22
23
24with ThreadPoolExecutor(max_workers=8) as pool:
25 for out in pool.map(resolve, prepared):
26 persist(out)In Node the same bounded concurrency comes from a limiter rather than threads:
1import pLimit from "p-limit";
2import { Renidly, RenidlyError } from "renidly";
3
4const renidly = new Renidly(undefined, {
5 autoRateLimit: true,
6 rateLimitSafety: 0.7,
7 maxRetries: 3,
8});
9
10const limit = pLimit(8);
11
12await Promise.all(prepared.map(row =>
13 limit(async () => {
14 try {
15 const r = await renidly.emails.reverse(row.email);
16 await persist({ id: row.id, status: r.found ? "found" : "miss", cost: r.meta.creditConsumed });
17 } catch (e) {
18 if (e instanceof RenidlyError) await persist({ id: row.id, status: "error", code: e.errorCode });
19 else throw e;
20 }
21 })
22));Choosing
| Use batch when | Use a worker pool when |
|---|---|
| The endpoint has a batch method | It does not |
| Nobody is waiting on the result | You need results within seconds |
| You want resumability without building it | You already run a durable queue |
| The list is large | Items need per-record branching |
Jobs larger than 1000
The cap is 1000 items per job, so a real backfill is a sequence of jobs. Build this once, properly, because you will run backfills more than once.
The principle: track completion per chunk, not per run. A run that dies at chunk 37 should restart at chunk 37, not at chunk 1.
1from renidly import InsufficientCreditsError
2
3CHUNK = 1000
4
5def run_backfill(records: list[dict], run_id: str) -> None:
6 chunks = [records[i:i + CHUNK] for i in range(0, len(records), CHUNK)]
7
8 for idx, chunk in enumerate(chunks):
9 if chunk_done(run_id, idx):
10 continue # already finished, skip
11
12 try:
13 job = renidly.data.people.enrich_batch(ids=[r["person_id"] for r in chunk])
14 except InsufficientCreditsError:
15 alert_ops("backfill paused: balance exhausted")
16 return
17
18 save_job_id(run_id, idx, job.id) # useful for audit and cost questions
19
20 for row in job.stream():
21 upsert(row.matched_input, row)
22
23 mark_chunk_done(run_id, idx)1const CHUNK = 1000;
2
3export async function runBackfill(records: Rec[], runId: string) {
4 for (let idx = 0; idx * CHUNK < records.length; idx++) {
5 if (await chunkDone(runId, idx)) continue;
6
7 const chunk = records.slice(idx * CHUNK, (idx + 1) * CHUNK);
8 const job = await renidly.data.people.enrichBatch({ ids: chunk.map(r => r.personId) });
9
10 await saveJobId(runId, idx, job.id);
11
12 for await (const row of job.stream()) {
13 await upsert(row.matched_input, row);
14 }
15
16 await markChunkDone(runId, idx);
17 }
18}Resubmitting a chunk you already completed is the most expensive mistake available here. It doubles the cost of that chunk for zero additional data, and it is exactly what happens when completion is tracked per run rather than per chunk.
Jobs remain pollable until they expire, so keep your runs tighter than the expiry window. A not-found on a job id means it expired before anyone collected it.
Leave rate limit headroom
Rate limits are per-minute and tier-based. The failure mode at volume is not hitting the limit once, it is hitting it continuously while something else that matters is also trying to use the key.
1renidly = Renidly(config=RenidlyConfig(
2 auto_rate_limit=True,
3 rate_limit_safety=0.7, # bulk work capped at 70% of the ceiling
4 max_retries=3,
5))auto_rate_limit holds you under the ceiling with a sliding sixty-second window rather than letting you discover it through rejections. Regular keys read the limit from your tier; enterprise keys need an explicit rate_limit_per_minute because the ceiling is negotiated.
The safety factor is the parameter that matters at volume. Running a backfill at 100% of your limit on the same key that serves your signup flow is a self-inflicted outage. Set it to 0.7 and the interactive path keeps its headroom.
If the bulk job genuinely needs full throughput, use a separate key for it. Per-request overrides make that straightforward:
1person = renidly.data.people.retrieve(id=pid, options={"api_key": bulk_key})Put a ceiling on the spend
Six lines that prevent an uncomfortable conversation:
1BUDGET, spent = 50_000, 0
2
3for idx, chunk in enumerate(chunks):
4 if spent >= BUDGET:
5 log.warning("budget ceiling reached at chunk %s of %s", idx, len(chunks))
6 break
7 result = renidly.data.people.enrich_batch(ids=chunk).wait()
8 spent += result.meta.credit_consumedEvery response reports the credits it consumed, outside the response data so there is no field collision:
1print(result.meta.credit_consumed, result.meta.remaining_balance)1console.log(result.meta.creditConsumed, result.meta.remainingBalance);Wire a balance alert before you start. A backfill draining the pool your production signup path depends on is an incident, and it happens to most teams once.
Handle the three outcome classes separately
At volume, "it failed" is not useful. Each outcome needs different handling, and lumping them together is how you end up re-paying for work that will never succeed.
| Outcome | Billed | Retry |
|---|---|---|
| Resolved | Yes | No |
| Not found | Yes | No, record the miss |
| Invalid input | No | No, fix the input |
| Opted out | No | Never |
| Rate limited | No | Yes, backoff |
| Transient error | No | Yes, backoff |
| Balance exhausted | No | No, pause the run |
Two of these deserve attention at scale.
Not-found is a real answer and it is billed. It comes back in the not_found list on a batch result rather than as an error. Record it, because a record that failed to resolve this month is a record you should not retry next month without a reason. If you are enriching into a CRM, the write rules that stop a miss from wiping a populated field are in CRM enrichment patterns.
Opt-outs are permanent. Build a durable suppression list on first occurrence and check it before submitting. Otherwise every run you ever execute re-requests the same records forever.
1def submittable(records: list[dict]) -> list[dict]:
2 return [r for r in records
3 if not is_suppressed(r["person_id"])
4 and not missed_recently(r["person_id"], days=90)]That second filter is worth having. A record that nothing could resolve ninety days ago is unlikely to resolve today, and filtering it out of the next run is pure saving. The reasoning behind permanent suppression is in responsible profile lookup.
Make the worker idempotent
Queues retry. Your job will run twice on the same chunk, and without a claim pattern you get duplicate writes and doubled cost tracking.
1def process_chunk(run_id: str, idx: int, chunk: list[dict]) -> None:
2 key = f"backfill:{run_id}:{idx}"
3
4 if not claim(key, ttl_hours=6):
5 return # another worker has it
6
7 try:
8 job = renidly.data.people.enrich_batch(ids=[r["person_id"] for r in chunk])
9 for row in job.stream():
10 upsert(row.matched_input, row)
11 except (ServiceUnavailableError, APIConnectionError):
12 release(key) # transient: allow a retry
13 raise
14
15 mark_chunk_done(run_id, idx)Release the claim on transient failures. Hold it on permanent ones, because they will not succeed on a second attempt.
Decide live versus dataset before you start
Both people and company batch methods accept a flag that forces the freshest per-item resolution rather than the standard dataset read:
1job = renidly.data.people.enrich_batch(ids=chunk, live=True)1const job = await renidly.data.people.enrichBatch({ ids: chunk, live: true });Expect it to cost like a live resolution and take longer per item. The right trade for a small high-value cohort, the wrong one for a first-pass hydration of a hundred thousand cold records.
The pattern that works: split the run. Dataset reads for the bulk, live resolution for the subset that matters. Your target account list gets the fresh treatment; the long tail does not. The comparison of the two modes is in the profile lookup API guide, and deciding which records deserve a refresh at all is covered in CRM data decay.
Monitor the run while it is running
A forty-thousand-record job that silently degrades at record 12,000 is worse than one that fails loudly.
Four things to emit per chunk:
1metrics.gauge("backfill.chunk", idx)
2metrics.gauge("backfill.resolution_rate", result.resolved / result.total)
3metrics.gauge("backfill.credits", result.meta.credit_consumed)
4metrics.gauge("backfill.balance", result.meta.remaining_balance)Watch the resolution rate per chunk rather than only at the end. If it drops sharply partway through, something about that segment of your input is different, usually a geography or a company-size band with thinner coverage. That is worth knowing while you can still stop, rather than after you have paid for all of it.
What breaks only at volume
Five failure modes that never appear in testing.
Memory. Accumulating forty thousand results in a list before writing works until it does not. Stream and persist as you go.
Connection pool exhaustion. A worker pool sized larger than your HTTP client's connection pool produces timeouts that look like API failures and are not.
Database write contention. The lookups are fine; your upserts become the bottleneck. Batch the writes too rather than one transaction per record.
Clock drift on long runs. A job spanning hours against a cursor-paginated endpoint can outlive assumptions about consistency. Prefer completion tracking by chunk over a single long-lived cursor.
The long tail of weird inputs. At forty thousand records you will encounter names, domains and encodings your validation never anticipated. Log the rejects rather than discarding them; the pattern in them usually reveals an upstream data quality problem worth fixing. If the inputs are names and companies rather than identifiers, the resolution paths and their relative reliability are compared in how to find a professional profile by name and company.
Frequently asked questions
How many records can I enrich at once?
Up to 1000 per batch job. Larger runs are a sequence of jobs with completion tracked per chunk.
Should I use batch or a worker pool?
Batch when the endpoint offers it, because it gives you background pacing, a miss list and input-keyed results for free. A worker pool when no batch method exists, such as reverse email lookup.
Do I pay for records that do not resolve?
Yes. Producing a definitive no-match takes the same work as producing a record. Budget on records submitted rather than records returned.
How do I estimate the cost of a large job?
Run a 500-record sample, read the resolution rate and credits consumed from the result, and project. Re-measure before each large run rather than reusing an old figure.
What happens if my process crashes mid-run?
With completion tracked per chunk, the next run skips finished chunks and resumes where it stopped. Without it, you re-pay for everything already done.
How do I avoid hitting rate limits?
Enable client-side rate limiting and set a safety factor below 1.0 so bulk work cannot starve whatever interactive path shares the key.
Should I use live resolution for a bulk run?
Usually not for the whole run. Split it: dataset reads for the bulk, live resolution for the high-value subset.
The one-line version
Prepare the input before you spend anything, because deduplication and normalisation routinely remove a fifth of submissions and every bad row is a billed lookup.
Measure your resolution rate on 500 records and project from that, because misses are billed and the rate is the variable that decides your real cost. Use batch where it exists, stream rather than wait so a crash does not discard completed work, and track completion per chunk so a restart does not re-pay for finished work.
Leave rate limit headroom so the backfill cannot take down whatever else shares the key, put a budget ceiling in the runner, and suppress opt-outs and recent misses before submitting rather than after.
And watch the resolution rate per chunk while the job runs. A segment of your input with thin coverage is worth discovering at chunk 12 rather than at the invoice.
If you want to measure your own resolution rate before committing to a run, there are 100 free credits with no card required, which covers a 500-record sample comfortably. The Quickstart takes about five minutes, and for volume pricing or a procurement conversation, talk to us directly.


