Thread Futures · model monitoring
Stall-risk model, scored on later launches
Real retail data · UCI Online Retail II Trained on SKUs launched before Oct 2010, then scored quarter by quarter on launches it never saw. Generated by python -m stock.monitor.
Accuracy and calibration by launch quarter
| Launch quarter | SKUs | ROC AUC | PR AUC | Predicted stall rate | Actual stall rate | Alerts |
|---|---|---|---|---|---|---|
| 2010Q4 | 175 | 0.814 | 0.657 | 28.4% | 21.1% | early_units_per_week: PSI 0.316 early_zero_week_share: PSI 2.634 early_customers: PSI 0.279 early_orders: PSI 0.397 early_return_rate: PSI 0.261 launch_list_price: PSI 0.377 launch_month: PSI 8.883 early_trend: KS p=1.5e-05 |
| 2011Q1 | 70 | 0.615 | 0.306 | 20.9% | 7.1% | early_units_per_week: PSI 1.616 early_trend: PSI 0.328 early_zero_week_share: PSI 0.559 early_customers: PSI 1.212 early_orders: PSI 0.881 early_return_rate: PSI 0.413 launch_list_price: PSI 1.513 launch_month: PSI 3.347 AUC 0.62 below 0.7 |
| 2011Q2 | 178 | 0.792 | 0.438 | 13.9% | 1.7% | early_units_per_week: PSI 1.311 early_zero_week_share: PSI 0.899 early_customers: PSI 1.119 early_orders: PSI 1.182 launch_list_price: PSI 0.413 launch_month: PSI 5.24 early_trend: KS p=0.00182 early_return_rate: KS p=1.3e-05 |
Data drift: PSI per feature versus training data
PSI above 0.25 is highlighted (major shift). KS-test p-values are in models/monitoring.json.
| Feature | 2010Q4 | 2011Q1 | 2011Q2 |
|---|---|---|---|
| early_units_per_week | 0.316 | 1.616 | 1.311 |
| early_trend | 0.163 | 0.328 | 0.238 |
| early_zero_week_share | 2.634 | 0.559 | 0.899 |
| early_customers | 0.279 | 1.212 | 1.119 |
| early_orders | 0.397 | 0.881 | 1.182 |
| early_return_rate | 0.261 | 0.413 | 0.07 |
| early_max_markdown | 0.0 | 0.0 | 0.0 |
| launch_list_price | 0.377 | 1.513 | 0.413 |
| launch_month | 8.883 | 3.347 | 5.24 |
| is_seasonal_christmas | 0.0 | 0.0 | 0.0 |
Offcut forecast: error by garment
Simulated cutting runs. Error by garment on the 1,600-run hold-out. P10–P90 range covers 76.8% of outcomes (target 80.0%).
| Garment | Mean abs. error |
|---|---|
| dress | 2.16 pp |
| hoodie | 1.6 pp |
| jeans | 1.23 pp |
| shirt | 1.86 pp |
| tee | 1.22 pp |