Landair production systems
Sole AI engineer for a New York brokerage's production AI, measured before it ships
- Situation
- A New York real estate brokerage needs the real owner of each property, often behind LLCs.
- Task
- As its sole AI engineer, I had to make the four signed KPIs measurable and name owners brokers can trust.
- Action
- I turned the signed KPIs into tests, built a Claude resolver that unwinds LLCs, and measured model changes.
- Result
- The resolver finds the real owner on 1,088 of 1,120 labeled properties, and brokers get reports in seconds.
97.1%right owner found, LLCs included (1,088 of 1,120 labeled properties)
Brokers call the human owner, not the LLC on the deed, and any answer the system doubts goes to review first.
How I built it
Turned the four signed KPIs into acceptance tests with the broker founders: scoring rules and labeled validation sets.
Built an agentic Claude resolver: Claude Vision reads scanned deeds, nested LLC ownership chains are unwound to a person, and state, city and identity-record APIs are called cheapest first.
Tested model and scoring changes with a 3-run protocol and confidence bands. Low-confidence results go to human review instead of being returned silently.
Classified all 17,990 comparable sales into a nine-category asset taxonomy.
Built the broker web platform (React, Vercel, Python serverless): property-data lookups run in parallel into one Claude call, and brokers run batches themselves with live progress.
Ran production: 641 pytest files across 3 Python services, CI on every pull request to the two largest, and guarded deploys with rollback and Slack alerts.
Why these choices
More than one run before a verdict
Single runs gave me two false wins that vanished on a re-run. Now every model change is measured more than once on a labeled set. Three runs is how I caught my own model upgrade scoring worse and reverted it.
A plain sort over an LLM comp picker
Against the brokers' own picks, a sort by distance and age beat both the existing engine and an LLM picker. The simple sort shipped.
Fewer paid lookups, owner accuracy held
An offline replay cut identity-data calls from 21.6 to 3.3 per LLC row (35 reviewed rows), and first-named-owner accuracy held. Most outside answers are now stored and read before a new paid call, and a per-run cap and a circuit breaker stop a runaway list.
Fix the candidates, not the ranker
I read the wrong owner answers before tuning anything. On 105 of 107, the right person was in no candidate list (1,344 company-owned calibration lots), so no ranker could fix them. I rebuilt the owner decision as an evidence ledger, each name with its source, role and date, and added sources. A name from one weak source now goes to a person, not the call list.
A held-out set that can say no
A comp order built from evidence gained on the properties it was tuned on. On held-out properties it was no better, so it shipped switched off. Brokers did not get a change that only looked better on the data it was tuned on.
A shadow A/B first
The LLM adjudicator for LLC owners ran beside the live system before it decided anything. It had to gain strict picks and repeat its verdicts across runs.
How it was measured
- Owner resolution: 1,088 of 1,120 labeled Bronx properties right (97.1%), strict top pick 95.5%, committed scorer (Oct 8, 2026). On the 226 entity-owned rows of a September run, 92.5% (85.0% strict). The firm's June estimate for its previous tool was about 50% on owner contacts, a different measure.
- The upgrade the eval killed: a Haiku to Sonnet upgrade of the deed reader scored 53% against 67%, 3 runs each on a 15-row labeled set (Jul 9, 2026), so I reverted it.
- Deed reader on Haiku 5.5: right owner first on 28 of 30 reviewed rows on both passes, against 25 and 26 for Haiku 4.5 (Oct 2026).
- LLM adjudicator for LLC owners, shadow A/B: 2 to 7 more strict picks on 228 LLC rows, and the same verdict 93 to 94% of the time across 4 runs (Sep 30, 2026).
- Comp picker, 37 properties: the distance-plus-age sort matched 44.4% of still-valid broker picks, and an LLM picker 25.0 to 28.5% (Oct 20, 2026). A later rule cut re-picks of comps brokers had removed from 25 to 16 of 104 on held-out properties.
- Asset classes: 118 of 120 random (98.3%) and 149 of 153 stratified (97.4%) broker-labeled sales, when either of a sale's two labels counts. 11 store rows in the key follow my own store rule, and the rules were tuned on these samples (Oct 19, 2026).
- Identity-data lookups, offline replay of 35 reviewed LLC rows: 21.6 to 3.3 per row (692 to 107 calls), first-named owner right on 26, then 27, of 30 (Oct 6, 2026). Rows with an owner phone fell from 29 to 25, all four losses ambiguous name-only matches.
- Broker report, three-way model and effort test: median 41.5 s to 11.1 s on 6 real addresses, 2 passes each, and unsupported claims 17 to 0, LLM-judged (Oct 8, 2026). A small sample.
- Buyer search: 3 to 13 minutes to 25 to 90 seconds per property by batching and parallelizing public-records calls, identical ranked lists on 6 test properties, one per asset type (Oct 13, 2026).
- Name parsing: exact phone matches 16% to 48% and phone coverage 69.4% to 93.9%, same 98-property test (Jun 2026). A rebuild from 4.2M names typed on city forms reads first-name-first names in the right order on 87.7% of 37,397 held-out payroll names, up from 29.7% (last-name-first 97.9% to 96.8%, Oct 2026).
Failures and limits
- One-off evals handed me two false wins before the 3-run rule. Both vanished on a re-run.
- A name-parser inversion was silently reversing more than half of owner names.
- A message path had been failing silently for two months before I root-caused the MMS outage.
- A 1,000-row API read cap had silently piled up 232K duplicate rows.