Evaluation, Uncertainty, and Human Review
Before a decision function is used in production, define its baseline, test conditions, acceptance criteria, and failure cases. Evaluation should use data that reflects the range of vehicle models, component variants, stations, shifts, and operating conditions expected in scope. Keep development data separate from evaluation data and prevent future outcomes or defect labels from leaking into model inputs.
Useful measures depend on the task and may include detection recall, false-alert rate per operating hour, precision of escalated cases, prediction error, time to detect an exception, review workload, and the proportion of recommendations accepted after review. Report results by relevant defect class, station, equipment type, and operating condition where the data supports it. A simple rules-based baseline helps establish whether a more complex method adds practical value.
Monitor changes in input distributions, product mix, equipment behavior, data completeness, and decision outcomes. Record model or rule version, input references, timestamp, output, uncertainty, reviewer response, and subsequent outcome where permitted. When performance falls outside agreed limits, route the case for review and follow an approved fallback or rollback process.