The pitch is straightforward and it sounds obviously correct.
You have several data providers. Rather than paying all of them for every record, you try the cheapest one first. If it returns something, you stop. If it does not, you fall through to the next one, and so on until something hits or you run out of providers. You get the coverage of the expensive provider at something close to the price of the cheap one.
Teams build this, run it for a quarter, and then discover their cost per usable record went up.
The logic is not wrong. The ordering is. Cheapest-first optimizes for the price of an attempt, and what you actually care about is the price of a result. Those are the same number only when every provider has the same hit rate, which is never.
This post is about getting the ordering right. It applies to any set of providers, and there is a version of it worth building even if you only use one.
The arithmetic that breaks the naive version
Say provider A costs 1 unit per lookup and resolves 30% of your records. Provider B costs 3 units and resolves 80%.
Cheapest-first says run A, then fall through to B for the 70% that missed.
-
Every record costs 1 for the A attempt.
-
70% of records also cost 3 for the B attempt.
-
Total per 100 records: 100 + 210 = 310 units.
-
Records resolved: 30 from A, plus 80% of the remaining 70, so about 86.
-
Effective cost: roughly 3.6 units per usable record.
Now run B alone. -
Every record costs 3.
-
Total per 100: 300 units.
-
Records resolved: 80.
-
Effective cost: 3.75 units per usable record.
The waterfall wins, but by four percent. That is well inside the margin where the added complexity, the extra failure mode, the second vendor relationship, and the engineering time are not worth it.
Change A's hit rate to 15% and the waterfall becomes more expensive than just calling B. You are paying a full unit on every single record to salvage fifteen of them, and paying it again on the eighty-five you did not.
The rule that falls out of this: a cheap provider only earns its place in the cascade if its hit rate is high enough that the attempts you waste cost less than the calls you avoid. Below that threshold it is a tax on every record you process.
Most teams never run this arithmetic. They order by price because price is the number on the contract, and hit rate is a number they have not measured.
Order by expected value, not by price
The better ordering rule is: first call the provider most likely to resolve this specific record.
Not the cheapest. Not the best overall. The one most likely to work on this one, weighted against what it costs.
For each provider, the number that matters is:
1expected value = P(resolves this record) / cost per attemptSort descending. Call in that order. Stop on a confident hit.
The interesting part is that P(resolves this record) is not a constant. It varies enormously by the shape of the input, and that variance is where the actual savings live.
Segment the input before you route it
This is the step that separates a working waterfall from a cascade of hope, and almost nobody does it.
Providers are not uniformly good. They are good at specific kinds of record. Once you know which kinds, you stop running a cascade and start running a router, and a router is both cheaper and faster because most records only ever hit one provider.
Segment by whatever actually predicts hit rate in your data. Common ones:
| Segment | Why it changes the odds |
|---|---|
| Company size | Coverage of large enterprises is near-universal, small companies is where providers diverge |
| Geography | Every provider has regional strengths and thin spots |
| Seniority | Senior roles are better covered almost everywhere |
| Input type | An email address, a domain plus name, and a profile URL are different problems |
| Record age | A contact captured last month behaves differently from one captured in 2019 |
You do not have to guess at these. Take a few thousand records, run them through each provider, and cross-tabulate hit rate by segment. It costs a bit of credit and a day, and it is the difference between an ordering you believe and an ordering you assumed. The method for running that comparison properly, including building a golden set so you are measuring accuracy rather than fill rate, is in how to test a B2B data vendor.
What usually comes back is that your providers have complementary strengths rather than a strict ranking. That is the finding that makes routing worth building. If one provider dominates in every segment, stop reading, cancel the others, and use the one.
Build a router, not a cascade
Once you have the segment map, the code looks like this. Route first, then fall through only if the routed provider misses.
1from renidly import Renidly, RenidlyError
2
3renidly = Renidly() # reads RENIDLY_API_KEY from the environment
4
5
6def route(record) -> list:
7 """Return providers in expected-value order for this specific record."""
8 if record.email:
9 # Email in hand is the strongest signal available
10 return [try_renidly_reverse, try_secondary]
11 if record.first_name and record.domain:
12 return [try_renidly_find, try_secondary]
13 return [try_secondary, try_renidly_find]
14
15
16def enrich(record):
17 for attempt in route(record):
18 result = attempt(record)
19 if result and result.confidence in ("high", "medium"):
20 return result
21 return None
22
23
24def try_renidly_reverse(record):
25 try:
26 r = renidly.emails.reverse(record.email)
27 except RenidlyError:
28 return None
29 return r if r.found else NoneThe Node version is the same shape, and the thing worth noticing in both is that the fallthrough condition is confidence, not merely "did something come back":
1import { Renidly, RenidlyError } from "renidly";
2
3const renidly = new Renidly();
4
5async function tryRenidlyReverse(record: Record) {
6 try {
7 const r = await renidly.emails.reverse(record.email);
8 return r.found ? r : null;
9 } catch (e) {
10 if (e instanceof RenidlyError) return null;
11 throw e;
12 }
13}
14
15export async function enrich(record: Record) {
16 for (const attempt of route(record)) {
17 const result = await attempt(record);
18 if (result && ["high", "medium"].includes(result.confidence)) return result;
19 }
20 return null;
21}That distinction matters more than the routing does. A cascade that stops on any non-empty response will happily accept a low-confidence guess from the first provider and never try the second one that would have returned a certain answer. You have built a system that optimizes for stopping early rather than for being right.
This is one reason we return a four-tier confidence value rather than a boolean. A waterfall needs a signal it can threshold on, and if a provider only tells you "found" or "not found," your only two options are to accept everything it says or to always fall through, and neither is a waterfall. The same applies to whoever else is in your cascade: if a provider cannot express uncertainty, it can only safely be the last stop.
The double-pay trap
Here is the failure that costs real money and takes months to notice.
Your waterfall runs nightly over a large table. Record 4,412 misses on every provider. Tomorrow night it runs again, misses on every provider again, and you pay the full cascade cost for the second time. And the night after that.
Over a quarter, on a table with a 40% permanent miss rate, you can spend more on records that will never resolve than on the ones that do.
Three things fix it, and you want all three:
Persist the misses, not just the hits. Store the fact that a record failed, which providers were tried, and when. Then check that log before the cascade runs.
Set a per-record cooldown. A record that missed everywhere last week is unlikely to resolve this week. Something like thirty to ninety days before it is eligible again is reasonable for most data, shorter if you are specifically watching for change.
Handle permanent failures separately. Some misses are not "not found yet," they are "will never be found." An opted-out individual, a company that no longer exists, a malformed record. Those need a permanent exclusion flag and should never re-enter the cascade at all. Retrying them forever is pure loss, and it is the single most common source of silent waste in enrichment pipelines.
While you are at it, cache the hits too, keyed on a stable identifier rather than on a name or a handle. Anything you key on a human-readable string will break silently when that string changes, which is the failure mode covered in the primer on identity resolution.
Know when to stop
Every cascade needs a floor, and "try everything" is not a strategy.
The question to ask per tier: given that the previous providers all missed, what is the chance this one hits, and is that worth its cost?
Conditional probability is brutal here. A record that three providers could not resolve is not a random record. It is a hard one, and the fourth provider's headline hit rate does not apply to it. If a provider resolves 60% of records overall, it might resolve 15% of records that already missed three times. At that rate it may not be worth calling at all.
Measure the marginal hit rate per tier, meaning the hit rate conditional on everything above having failed. Not the standalone rate. Teams almost always use the standalone number and consistently overestimate what their last tier is contributing.
A blunt but effective rule: if a tier's marginal contribution is under 10%, cut it. You are paying for the attempt on every record that reaches it, which by definition is your hardest and largest remaining segment.
Measure it properly or do not build it
Four numbers, per provider, per segment. If you are not tracking these, you cannot know whether your ordering is right, and an unmeasured waterfall drifts into being expensive without anyone noticing.
- Attempt count. How many times each provider was called.
- Marginal hit rate. Hits divided by attempts, for that position in the cascade.
- Cost. Actual spend, per provider, per segment.
- Effective cost per usable record. The only number that matters, and the one to compare across orderings.
Getting the third number is harder than it should be with most providers, because spend shows up as a monthly invoice and you are left reverse-engineering which calls caused which charges. Renidly reports the credits consumed on every individual response, so per-segment cost is something you read off a log line rather than reconstruct at month end. Whatever you use, insist on per-call cost visibility before you build a router on top of it, because a router you cannot measure is a router you cannot tune.
Re-run the segment analysis quarterly. Provider coverage shifts, your input mix shifts, and an ordering that was optimal in Q1 is frequently mediocre by Q4.
The argument against building this at all
Now the part where I talk you out of it.
Most teams should not build a waterfall. The complexity is real: multiple vendor relationships, multiple contracts, multiple failure modes, multiple sets of field semantics to reconcile, and a routing layer somebody has to own. The savings are frequently in the range of ten to twenty percent, and I have seen plenty of implementations where the true saving was negative once engineering time was counted honestly.
Build a waterfall when three things are true. Your volume is high enough that ten percent is real money. You have measured genuinely complementary provider strengths rather than assuming them. And you have someone who will own the routing logic as it drifts.
If those are not all true, pick the provider with the best effective cost per usable record on your data, use it alone, and spend the engineering time on something with a larger return. A single well-instrumented provider with good confidence signals will beat a badly ordered three-provider cascade almost every time, and it will beat it on cost, latency, and the number of things that can break at 2 a.m.
One middle path worth knowing about: you can get some of the waterfall benefit inside a single provider by routing between call types rather than between vendors. A cheap dataset read for bulk work and a freshest-available resolution only for the records that matter is the same expected-value logic applied at a smaller scale, with none of the multi-vendor overhead. The mechanics of running that at volume are in batch enrichment at scale.
The one-line version
Order providers by probability of success on each individual record divided by cost, not by price. Segment your input first, because that is where the probability differences live. Cache your misses as aggressively as your hits. Cut any tier whose marginal contribution is under ten percent. And be honest about whether the ten to twenty percent you are chasing is worth the system you are about to own.
The best waterfall is often a router with one branch, and the second best is usually no waterfall at all.
If you want to run the segment analysis described above, there are 100 free credits with no card required, which is enough to cross-tabulate hit rate across a few thousand records and find out whether your assumed ordering survives contact with your actual data. Most do not.
