The first version of every enrichment backfill looks the same. Someone pulls the list of records that need hydrating, writes a loop, and runs it.
1for row in rows: # do not do this
2 result = client.enrich(row.handle)
3 save(result)At a hundred records it works. At a hundred thousand it fails in a specific and predictable order: first it is slow, then it hits a rate limit, then someone adds threads, then it hits the rate limit harder, then the process dies at record 60,000 and nobody knows which 60,000 were done.
The loop is not the problem. The problem is that a backfill is a job, and jobs need identity, progress, resumability, and a cost ceiling. A loop has none of those.
This post covers how to run large enrichment workloads properly: when to use the batch endpoints, when a worker pool is actually the better tool, how to make the whole thing resumable, and how to know what it cost while it is running rather than after.
Batch is not "the loop but faster"
The batch endpoints are a different execution model, not a convenience wrapper.
You submit up to 1000 items in one request. You get back a job_id immediately, with an HTTP 202. The work happens in the background at a controlled pace. You poll for progress and collect results as they resolve, keyed back to exactly what you submitted.
Four properties fall out of that, and each one solves a specific failure of the naive loop:
| Property | What it fixes |
|---|---|
| One request per 1000 items | You are not generating 1000 requests to be rate limited |
A job_id you can persist | The job survives your process dying |
| Results keyed to your inputs | You never have to guess which output belongs to which row |
| Balance pre-checked at submit | You find out you cannot afford it before the work starts, not at item 700 |
That last one is worth dwelling on. Submitting a batch pre-checks your balance and charges nothing at submit time. If the pool cannot cover the job, you learn immediately rather than watching a backfill die two thirds of the way through with a partially updated database.
Four batch methods exist, covering the two products where bulk work makes sense:
data.people.enrich_batch(...)/enrichBatch(...)data.companies.enrich_batch(...)/enrichBatch(...)emails.verify_batch([...])/verifyBatch([...])emails.find_batch([...])/findBatch([...])
The simple version, and why it is usually enough
If the job fits in one batch and you can afford to wait for it, the SDK collapses submit-poll-collect into two lines.
1from renidly import Renidly
2
3renidly = Renidly() # reads RENIDLY_API_KEY from the environment
4
5job = renidly.data.people.enrich_batch(
6 handles=["ryanroslansky", "williamhgates"],
7)
8
9result = job.wait(on_progress=lambda n: print("resolved", n))
10
11print(result.status, result.resolved, "/", result.total)
12print("misses:", result.not_found)
13
14for row in result.results:
15 save(row.matched_input, row)job.wait() blocks and polls for you, so you are not writing a sleep loop with backoff by hand. The on_progress callback fires as items resolve, which is what you want wired to a progress bar or a log line rather than silence for four minutes.
Node gets the same shape, with the job handle awaited first and the polling awaited second:
1import { Renidly } from "renidly";
2
3const renidly = new Renidly(); // reads RENIDLY_API_KEY from the environment
4
5const job = await renidly.data.people.enrichBatch({
6 handles: ["ryanroslansky", "williamhgates"],
7});
8
9const result = await job.wait({ onProgress: (n) => console.log("resolved", n) });
10
11console.log(result.status, result.resolved, "/", result.total);
12console.log("misses:", result.notFound);
13
14for (const row of result.results) save(row.matched_input, row);Two fields do the real work here.
matched_input on every resolved row links the result back to the exact string you submitted. This is the field that makes batch safe. You are not zipping two arrays by position and praying the order held. Key your writes on matched_input and reordering, partial resolution, and retries all become harmless.
not_found (notFound in Node) is a separate list of inputs that produced no record. It is not an error and it should not be treated as one. It is a definitive answer, and it is billed, because producing "this does not resolve" takes the same work as producing a record. More on that in the cost section, because it is the single number that determines your budget.
Streaming when you do not want to wait
job.wait() holds until everything is done. For a big job feeding a database, you usually want to persist results as they land instead, so a crash at 80% does not throw away 80% of the work.
1job = renidly.emails.verify_batch([r.email for r in rows])
2
3for row in job.stream():
4 persist(row.matched_input, row)The Node equivalent uses async iteration, and note the extra await because the submit itself returns a promise:
1const job = await renidly.emails.verifyBatch(rows.map((r) => r.email));
2
3for await (const row of job.stream()) {
4 await persist(row.matched_input, row);
5}Stream when the result set is large, when you are writing to a database anyway, or when you want partial results usable before the job finishes. Wait when the job is small and you need the complete set before doing anything with it.
Jobs larger than 1000
The cap is 1000 items per job, so a real backfill is a sequence of jobs. This is where most implementations get sloppy, and it is worth building properly once because you will run backfills more than once.
The rule is: persist the job identity before you submit, not after.
1import time
2from renidly import Renidly, RenidlyConfig, RenidlyError, InsufficientCreditsError
3
4renidly = Renidly(config=RenidlyConfig(max_retries=3, auto_rate_limit=True))
5
6CHUNK = 1000
7
8
9def run_backfill(handles: list[str], run_id: str) -> None:
10 chunks = [handles[i:i + CHUNK] for i in range(0, len(handles), CHUNK)]
11
12 for idx, chunk in enumerate(chunks):
13 if chunk_already_done(run_id, idx):
14 continue
15
16 job_id = get_saved_job_id(run_id, idx)
17
18 if job_id is None:
19 try:
20 job = renidly.data.people.enrich_batch(handles=chunk)
21 except InsufficientCreditsError:
22 alert_ops("backfill paused: balance exhausted")
23 return
24 save_job_id(run_id, idx, job.id) # persist BEFORE waiting
25 job_id = job.id
26 else:
27 job = renidly.data.people.resume_job(job_id)
28
29 result = job.wait()
30
31 for row in result.results:
32 upsert(row.matched_input, row)
33
34 record_misses(run_id, idx, result.not_found)
35 mark_chunk_done(run_id, idx, credits=result.meta.credit_consumed)Saving the job_id before waiting is the whole trick. If your process dies during the wait, the job is still running on the server side and the results are still collectable. Restart, find the saved job_id, and reattach instead of resubmitting work you have already paid for.
Resubmitting a chunk you already submitted is the most expensive mistake in this entire post. It doubles the cost of that chunk for zero additional data.
One caveat on that code: verify the exact reattach method name against your SDK version before shipping it. The submit, wait, and stream helpers are documented; if a resume helper is not exposed in your version, poll the tracking endpoint directly with the saved job_id and the after cursor, which is the same operation one level down.
When a worker pool beats batch
Batch is not always available or always right. Reverse email lookup, for instance, has no batch endpoint, so a list of addresses is a worker pool job.
The pattern is a bounded pool with the SDK rate limiter turned on:
1from concurrent.futures import ThreadPoolExecutor
2from renidly import Renidly, RenidlyConfig, RenidlyError
3
4renidly = Renidly(config=RenidlyConfig(
5 auto_rate_limit=True,
6 rate_limit_safety=0.7, # leave headroom for production traffic
7 max_retries=3,
8))
9
10
11def resolve(row):
12 try:
13 r = renidly.emails.reverse(row.email)
14 except RenidlyError as e:
15 return {"id": row.id, "status": "error", "code": e.error_code}
16 return {
17 "id": row.id,
18 "status": "found" if r.found else "miss",
19 "confidence": r.confidence,
20 "cost": r.meta.credit_consumed,
21 }
22
23
24with ThreadPoolExecutor(max_workers=8) as pool:
25 for out in pool.map(resolve, rows):
26 persist(out)In Node the same bounded concurrency comes from a limiter rather than a thread pool, but the reasoning is identical:
1import pLimit from "p-limit";
2import { Renidly, RenidlyError } from "renidly";
3
4const renidly = new Renidly(undefined, {
5 autoRateLimit: true,
6 rateLimitSafety: 0.7,
7 maxRetries: 3,
8});
9
10const limit = pLimit(8);
11
12await Promise.all(
13 rows.map((row) =>
14 limit(async () => {
15 try {
16 const r = await renidly.emails.reverse(row.email);
17 await persist({ id: row.id, status: r.found ? "found" : "miss", cost: r.meta.creditConsumed });
18 } catch (e) {
19 if (e instanceof RenidlyError) await persist({ id: row.id, status: "error", code: e.errorCode });
20 else throw e;
21 }
22 }),
23 ),
24);rate_limit_safety below 1.0 is the parameter that keeps a backfill from taking down your production path. Set it to 0.7 and the limiter holds you at 70% of your ceiling, leaving the rest for the interactive traffic that a user is actually waiting on. Running a backfill at 100% of your rate limit on the same key that serves your signup flow is a self-inflicted outage.
Enterprise keys need an explicit rate_limit_per_minute (rateLimitPerMinute in Node), since the ceiling is negotiated rather than read from a tier.
Choosing between them
| Use batch when | Use a worker pool when |
|---|---|
| The endpoint has a batch method | It does not, like reverse lookup |
| The list is large and can run in the background | You need results within seconds |
| You want resumability for free | You are already running a durable queue |
| You want one request instead of a thousand | Items need per-record branching logic |
If you are already running a job queue with retries and dead-letter handling, the worker pool integrates into what you have. If you are not, batch gives you most of those properties without building them.
Live enrichment inside a batch
Both people and company batch methods take a live flag. Setting it forces the freshest per-item resolution instead of the standard dataset read.
1job = renidly.data.people.enrich_batch(handles=chunk, live=True)1const job = await renidly.data.people.enrichBatch({ handles: chunk, live: true });Expect it to cost like a live resolution and take a little longer per item. That is the trade, and it is the right trade in exactly two situations: a small, high-value account list where accuracy per record justifies the cost, and a periodic refresh of records you know are stale.
It is the wrong trade for a first-pass hydration of a hundred thousand cold records, where the standard read is what you want and the cost difference across that volume is substantial.
A practical pattern is to split the run: standard read for the bulk, live=True for the subset that matters. Your target account list gets the fresh treatment. The long tail does not.
The number that determines your budget
Here is the arithmetic that surprises teams, and it is worth internalizing before you size anything.
A lookup that runs and returns nothing is billed. Not-founds come back in the not_found list, and they are charged per definitive result, because producing "this does not resolve" takes the same work as producing a record. That is why a miss is a 200 with an empty answer rather than a 404.
So the cost of a backfill is driven by the count of records you submit, and the value is driven by your resolution rate:
1cost = records submitted × cost per lookup
2usable records = records submitted × resolution rate
3effective cost = cost ÷ usable recordsAt a 55% resolution rate, your real cost per usable record is close to double the headline rate. A team that budgets on the headline number and gets 55% back has not been overcharged, they have mis-modelled.
Which means the single most valuable thing you can do before a large backfill is measure your own resolution rate on a sample. Take 500 records that are representative of the full set, run one batch, and read resolved against total. That number is specific to your data. A list of last quarter's inbound behaves very differently from a trade show list from 2019, and no vendor benchmark tells you which one you have.
1sample = handles[:500]
2probe = renidly.data.people.enrich_batch(handles=sample).wait()
3
4rate = probe.resolved / probe.total
5print(f"resolution rate: {rate:.1%}")
6print(f"projected cost for {len(handles)} records:",
7 len(handles) * probe.meta.credit_consumed / probe.total)Run that before every large backfill, not just the first one. Resolution rates drift as your data sources change, and a rate that was 60% last year on inbound leads may be 35% this year on a purchased list.
Make the spend observable while it runs
Every response carries a .meta object sitting outside the response data, so there is no field collision between your data and your billing telemetry.
1print(result.meta.credit_consumed) # what this job cost
2print(result.meta.remaining_balance) # balance after it1console.log(result.meta.creditConsumed, result.meta.remainingBalance);Three things to wire up before you run anything large:
A balance alert. remaining_balance under a floor should page someone. A backfill draining the pool that your production signup path depends on is an incident, and it happens to most teams once.
A per-chunk cost log. Record credits consumed per chunk as you go. When someone asks what the backfill cost, you want a query, not an invoice reconstruction.
A hard ceiling in the runner itself. Decide the maximum the job is allowed to spend and stop when it hits that, regardless of how many records remain.
1BUDGET = 50_000
2spent = 0
3
4for idx, chunk in enumerate(chunks):
5 if spent >= BUDGET:
6 log.warning("budget ceiling reached at chunk %s", idx)
7 break
8 result = renidly.data.people.enrich_batch(handles=chunk).wait()
9 spent += result.meta.credit_consumedA runaway loop is the classic way to turn a routine backfill into an awkward conversation with finance. A ceiling in code costs six lines.
If you also want to price the work before you write any of it, per-route costs are published at GET https://renidly.com/api/panel/credits/routes/costs/, which is public and needs no key. account.route_costs() and account.balance() expose the same thing through the SDK.
Handle the three error classes differently
Not every failure means the same thing, and treating them uniformly wastes money on records that will never resolve.
| Error | Meaning | Retry? |
|---|---|---|
InvalidRequestError | Bad input, read field_errors | No, fix the input |
PermissionDeniedError | Opted out or gated | No, ever, record the exclusion |
InsufficientCreditsError | Balance exhausted | No, pause the whole run |
RateLimitError | Ceiling hit, carries retry_after | Yes, or enable auto_rate_limit |
ServiceUnavailableError | Transient | Yes, with backoff |
APIConnectionError | Network or timeout | Yes, with backoff |
NotFoundError | The batch job expired or does not exist | No, resubmit that chunk |
PermissionDeniedError deserves a permanent suppression list. Individuals can opt out of resolution, and that is a durable state. If you let it fall into your generic retry path, every backfill you ever run will re-request the same addresses, forever, for nothing. Build the exclusion list on the first occurrence and check it before submitting.
NotFoundError on a job handle means the job expired before you collected it. Jobs are pollable until they expire, so if your collector runs on a schedule, make sure the schedule is tighter than the expiry window. Losing a completed job you already paid for is avoidable.
What running this yourself looks like
If you are weighing this against an internal build, the API calls are not the hard part. Here is what you would own instead.
A job orchestration layer. Submit, track, resume, expire, retry, dead-letter. That is a real system, and it is the same system whether you built the data or not.
Rate limit coordination across workers. A sliding window that several processes respect simultaneously requires shared state, usually a Redis-backed token bucket, plus the operational work of keeping that available. Getting it wrong means either leaving throughput on the table or getting throttled in bursts.
Backpressure. When your enrichment layer slows down, your ingest queue grows. Without backpressure, an hour of degraded performance becomes a queue you spend a day draining.
Idempotency everywhere. Queues retry. Without a claim-and-release pattern keyed per record, retries produce duplicate writes and doubled cost.
Freshness scheduling. People change jobs continuously, so a hydrated database is decaying from the moment it is written. A refresh scheduler that prioritizes high-value records, respects a budget, and does not thrash is an ongoing system with an owner, not a project that ships.
Roughly speaking that is one to two engineers permanently, plus infrastructure, plus the on-call rotation. Justifiable if enrichment is your product. Hard to justify if it is one input into a go-to-market motion.
Where Renidly fits
Batch submission with a persistable job handle, server-side pacing, results keyed to your exact inputs, a dedicated miss list, balance pre-checked at submit, and per-call cost telemetry on every response are all in the SDK. You are not building an orchestration layer to get them.
Most enrichment tools give you a rate limit and leave the job semantics to you. Renidly treats bulk work as a first-class execution model, so the code above is the whole implementation rather than the client half of one. The live flag being per-job rather than per-account is part of the same idea: freshness is a decision you make per workload, not an architecture you commit to.
The practical way to test that is a 500-record sample of your own data. You get a resolution rate, a real cost per usable record, and a projection for the full run, in about ten minutes. Those three numbers usually settle the build-versus-buy conversation faster than any evaluation doc, and they are the same three numbers you will want in front of you before any large backfill regardless of what you decide.
If you have not defined which records are worth enriching in the first place, the guide to building precise ICP filters covers narrowing a list before you spend anything hydrating it. And if your input list is email addresses rather than identifiers, reverse email lookup is the entry point.
1pip install renidly1npm install renidlyBoth SDKs cover every endpoint and handle auth, retries, pagination, batch jobs, rate limiting, and typed errors, in the same shape across both languages. The free tier includes 100 credits with no card required, which is enough to run the 500-record probe on your own data before committing to a full run.
The Quickstart takes about five minutes. For volume pricing, an SLA, or a procurement conversation, get in touch.
