Uber Eats / Engineering case study
Making delivery
predictions observable.
Tracing the signals behind an ETA, preserving what guardrails change, and evaluating where the next model boundary should live.
The work, at a glance
A delivery estimate is the output of a system: model features, courier inputs, fallback paths, and guardrails all shape the number a customer sees. During my internship, I worked on making that system easier to inspect and its architectural tradeoffs easier to measure.
Prediction observability
Added uETA and BYOC metadata to logging paths; extended coverage to background cache refreshes.
Model evaluation
Compared the existing ETD split with a unified checkout–postorder model across stages and fulfillment types.
01 / Context
One delivery. Several predictions.
Uber Eats makes predictions before and after an order exists. Eyeball is the discovery phase, Checkout is the purchase decision, and PostOrder updates the estimate as the order is created, accepted, and adjusted. Each phase has different information available.
Two signals mattered to my observability work. uETA is Uber’s travel-time estimate, used as an input to delivery predictions. ETD is the estimated time of delivery. For BYOC (bring your own courier) orders, the ETD comes from a courier or configured value that can be changed by a minimum-time guardrail. Logging only the final number loses the explanation behind it.
Prediction observability
02 / Observability
Give uETA its own audit trail.
The travel-time signal was being computed, but it was not consistently visible in downstream Heatpipe logs. I added a dedicated TravelTime field at the event-adapter boundary so an engineer could inspect the input that drove a prediction without depending solely on feature-vector JSON.
| Phase | Read from | If absent |
|---|---|---|
| Eyeball | PredictedETAInSecPrediction response | Leave TravelTime nil. |
| Checkout | TIUetaSecondsFeatureVectorV2 | Use DefaultUETATime for a missing or nil feature value. |
Both adapters convert the available value to int64 seconds to satisfy the event schema. At Checkout, the feature-vector JSON also retains ti_ueta_seconds for ML audits. Existing minimum/maximum ETD and calibration logic stays intact. Tests cover extraction, conversion, and fallback behavior; both uETA adapter changes merged.
03 / Data lineage
Keep the value before the guardrail.
For BYOC, the central question was: where did this ETD come from, and what happened to it? The provider can use a live courier feed (VDF), a restaurant configuration, or a constant fallback. A minimum-ETD guardrail then prevents an unrealistically short delivery promise.
I introduced a shared ByocInfo model carrying the source and the ETD captured before that minimum is applied. Putting it in the shared model layer also resolved circular dependency constraints between service domains. The metadata travels alongside the serving value, so the final estimate can change without erasing its origin.
Preserve the input. Guard the promise.
Customer estimate10 min
Logged raw value5 min
Guardrail appliedYes
With a 5-minute input and a 10-minute minimum, the estimate becomes 10 minutes; the logged input stays 5.
Carry the metadata through every branch.
Checkout has both a primary v2 path and a legacy fallback path. I propagated ByocInfo through both, including the prediction metadata returned by the fallback calculator. Otherwise, the audit trail would disappear precisely when the system took its recovery path.
PostOrder adds another concern: multiple component providers contribute to an ETD. The calculator uses explicit source priority—VDF ahead of the constant and restaurant-config alternatives—so attribution does not depend on provider response order. The final workflow adapter converts the duration to seconds for the Kafka payload.
- Scope: Eyeball logs a positive custom ETD only for aggregator fulfillment. Non-BYOC orders retain nil metadata.
- Compatibility: Attach the store-to-metadata map only when it contains entries, preserving legacy nil behavior.
- Validation: Test client-failure fallback traversal, the 5-to-10-minute guardrail case, source attribution, and calculator priority.
- Telemetry: Use M3 metrics to inspect source distributions and whether the guardrail ran.
Checkout and PostOrder merged during the internship. Eyeball was implemented and awaiting review; I did not track a later merge status here.
04 / Background work
The cache has a prediction path, too.
Request-time logging leaves a gap when predictions are computed during a background cache refresh. I worked on an adapter for the cache-miss and stale-cache path, capturing the ML request and response for each restaurant lookup in a batch.
Observe the refresh path
The adapter accounts for absent TravelTime, FeatureVector, and PresentedETDMetadata. The design separates event adaptation from publishing and pairs a primary topic with a shadow topic sharing its schema. I implemented the adapter, but cache-refresh logging remained in progress by the end of the internship—not a completed rollout.
05 / Machine learning
Move the boundary to where information changes.
My second project asked whether Checkout belongs with PostOrder in the model architecture. The existing split places Eyeball and Checkout in a preorder model, with Created, Accepted, and Adjusted handled by a postorder model. The candidate keeps an Eyeball-only model and moves Checkout into a unified model with the later stages.
The hypothesis was that Checkout has richer order context than browsing, making it a useful place to share representations with PostOrder. The candidate still uses two models. The change is which stages each model owns.
Move the boundary, keep the lifecycle
The unified model introduces prediction_mode as a feature to distinguish the four stages. Its feature set includes stage-specific preparation times and elapsed times between order events, alongside shared order, location, travel-time, and fulfillment features.
06 / Evidence
Make the comparison fair before making it favorable.
I used Michelangelo evaluation outputs and Hive queries to compare baseline and candidate predictions on matched orders and stages. Baseline and candidate shared training data, feature pipelines, and hyperparameters. I used chronological train, validation, test, and evaluation windows to avoid leakage.
- Match the cohort. Intersect
workflow_uuidvalues within eachprediction_modeand use the same date range for both models. - Align the target. Preorder actual time runs from order creation to dropoff; PostOrder actual time runs from the prediction timestamp to dropoff. Comparisons must respect that distinction.
- Validate the inputs to the comparison. Check matching row counts and identical average actual delivery times before interpreting metric differences. Matching IDs alone does not prove timestamp alignment.
MAE = avg(|ATD − ETD|)Average absolute prediction error.
bias = avg(ATD − ETD)Positive means underprediction.
I also tracked error percentiles and mean predicted ETD. Tail metrics show what the average conceals; mean ETD reveals whether improved error comes with a longer promise. The multi-head comparison adds the model’s loss metrics for its different prediction heads.
The result depends on the model and fulfillment type.
For single-head Delivery at Checkout, MAE fell 4.2% and p95 error fell 7.5%, while average predicted ETD rose 2.5%. The OTT (over-the-top) cohort showed a larger accuracy gain and a larger increase in the estimate. The multi-head experiment did not reproduce those gains.
Accuracy gains and their tradeoffs
Single-head · Delivery
| Stage | MAE | p95 error | Mean ETD |
|---|---|---|---|
| Eyeball | −4.1% | −8.5% | +0.5% |
| Checkout | −4.2% | −7.5% | +2.5% |
| Created | +0.2% | −4.8% | −0.1% |
| Accepted | +0.1% | +0.2% | −0.1% |
| Adjusted | +0.3% | −2.3% | −0.1% |
Reading the result: Early-stage error improves. Postorder MAE is nearly flat, and Checkout’s more accurate prediction comes with a longer average estimate.
Single-head · Over-the-top (OTT)
| Stage | MAE | p95 error | Mean ETD |
|---|---|---|---|
| Eyeball | −5.9% | −8.1% | +4.4% |
| Checkout | −12.0% | −13.6% | +8.8% |
| Created | −0.5% | +1.6% | +0.2% |
Reading the result: Checkout has the strongest accuracy gain and the largest mean-ETD increase. The OTT comparison covers only these three stages.
Multi-head · Delivery
| Stage | MAE | p95 error | Mean ETD |
|---|---|---|---|
| Eyeball | +0.0% | +0.5% | +0.0% |
| Checkout | +1.8% | +4.1% | −1.1% |
| Created | +0.8% | −4.0% | −0.1% |
| Accepted | +0.5% | +0.1% | −0.2% |
| Adjusted | +0.4% | +0.4% | −0.1% |
Reading the result: Checkout regresses in both MAE and p95. A shorter average estimate does not make the predictions more accurate.
Multi-head · Over-the-top (OTT)
| Stage | MAE | p95 error | Mean ETD |
|---|---|---|---|
| Eyeball | +0.1% | +1.4% | −0.1% |
| Checkout | −0.5% | −2.7% | −0.0% |
| Created | +11.8% | +11.0% | −3.2% |
Reading the result: The Created stage shows a substantial accuracy regression. The multi-head results do not support a general claim that unification improves ETD accuracy.
My takeaway was a conditional result. The single-head architecture was promising for earlier stages, with ETA inflation still to investigate. Postorder gains were limited. The multi-head experiment raised a different question: whether sharing Checkout and PostOrder introduces conflicting signals. The work produced evidence for those next investigations, rather than a blanket recommendation to migrate.
07 / Reflection
Understand the full path of a prediction.
The common thread across both projects was knowing what a number actually represents. In logging, that meant preserving its source, units, and behavior through fallbacks. In evaluation, it meant aligning cohorts and targets, then looking beyond one aggregate metric.
I also learned to formalize requirements earlier, map a feature end to end, and connect implementation decisions to the reason the team needed the work. Shadow-traffic validation, SLATE testing, HDFS failures during training, and multi-head out-of-memory errors were reminders that the environment around a change is part of the engineering problem.
And outside the code: racket sports, hiking, and time with the team made California feel like more than an internship location.
