What I build
when nobody's
asking me to.

Four single-purpose agents (seasonality, logistics, budget, itinerary) with typed Pydantic handoffs, no orchestration framework, and three distinct memory types (episodic, procedural, session).

Problem

Multi-agent systems are usually described in the abstract — no system to point to and say “here’s where that handoff actually breaks.”

Approach

Four single-purpose agents — seasonality, logistics, budget, itinerary — each returning a typed schema, never free text. Hand-rolled orchestration: every handoff is a plain function call I can log and inspect, no framework in between. Three memory types kept deliberately distinct: episodic (trips I’ve actually taken, seeded from this site’s parks page), procedural (standing preferences, refined on correction), and session (current run only).

Outcome

An eval suite across four dimensions — constraint satisfaction, feasibility, consistency, preference alignment — that catches a real failure mode: the model “remembering” something in context without acting on it. Tested and passing offline before a single live API call.

A typed adapter contract for any agent exposing run(input) → output, pluggable scorers — deterministic and LLM-as-judge side by side — and a paired bootstrap regression test on score deltas instead of a raw before/after diff.

Problem

Every agent project I built needed evaluation, and I kept rewriting the same scoring logic from scratch each time — deterministic checks tangled up with LLM-as-judge calls, with no way to tell a real regression from noise.

Approach

A typed adapter contract — a name and a run() — so the framework never learns how an agent works, only that it can call it and get typed output back. Scorers ship as factories, not hardcoded checks: seven deterministic factories (schema validation, numeric bounds, set coverage, sequence integrity, duplicate detection) plus an LLM-as-judge scorer, both behind the same interface, running concurrently under an async runner that holds a semaphore around judge calls so a large suite doesn’t open hundreds of connections at once. Regression detection runs a paired bootstrap on score deltas rather than a raw before/after diff, because eval scores are bounded, skewed, and small-n — exactly where a t-test’s assumptions fail.

Outcome

Its own test suite is real: 80 tests, offline, passing in under a second — including six constructed score-set pairs with known ground truth, such as the same effect size correctly called NO_SIGNIFICANT_CHANGE at n=8 and REGRESSION at n=400. The self-contained toy-agent example runs end-to-end and writes real, diffable run reports. Not yet validated against a live agent — wiring it into the parks planner above, in place of that project’s one-off eval_suite.py, is next.

Multi-touch attribution — Markov removal-effect and a data-driven LightGBM + SHAP model, benchmarked against classic heuristics — validated on 8,000 synthetic journeys with known ground truth before being applied to 267,084 real journeys from the public GA4 e-commerce dataset.

Problem

Allocating marketing budget across channels requires knowing which touchpoints along a customer’s path actually drove the conversion — without that, spend defaults to last-click or intuition and high-value channels go underfunded. But conversions carry no ground truth for what caused them, so the attribution method itself has to be validated before its numbers can guide those decisions.

Approach

Ran seven attribution methods across two datasets: 8,000 simulated journeys with known channel effects, for ground-truth validation, and 267,084 real GA4 journeys (1.63% empirical conversion rate). Each method was scored on true-effect rank recovery (Spearman ρ), calibration, bootstrap stability, and agreement between the Markov and data-driven approaches.

Outcome

Only the data-driven model recovered the true channel ranking on the simulator (ρ = 0.79); every heuristic scored ≈0, and Markov removal-effect scored −0.18 — anti-correlated, confounded by channel frequency, despite being perfectly calibrated and bootstrap-stable (stability ≠ correctness). On real data the two trusted methods agreed only moderately (ρ = 0.49): heuristics and Markov rank channels by touchpoint volume, while the data-driven model piles credit onto the catch-all “other” bucket — a likely data-quality artifact given the dataset’s obfuscation, not a causal signal; those real-data numbers shouldn’t be trusted until “other” is decomposed. 83% of journeys are single-touch, so multi-touch attribution only applies to the remaining ~17% of traffic.