Bachelor-Thesis (de): Unsicherheitsanalyse für machinelearning-basierte Surrogatmodelle in der szenariobasierten Validierung autonomer Systeme.
Bachelor-Thesis (en): Uncertainty Analysis for ML-based Surrogate Models for the Validation of Autonomous Systems.
Bachelor-Thesis Summary:
The bachelor’s thesis ‘Uncertainties in machine learning-based surrogate models for the scenario-based validation of autonomous systems’ (University of Bremen, Institute of Automation, 2025) examines, within the context of the VaMAI project on the validation of autonomous maritime systems (AUVs), how aleatory and epistemic uncertainties in ML-based surrogate models can be quantified, separated and controlled: Following a systematic literature review on evidential deep learning, Bayesian neural networks (BNNs with HMC/SVI in Pyro), Conformal Prediction and Evidential Neural Networks (ENN based on Aminis’ Deep Evidential Regression, whose UCI benchmarks were successfully reproduced) are compared in sensitivity analyses using metrics such as RMSE, ECE, coverage and a weighted comparison score; In the implementation, Aminis’ ENN is integrated into a criticality space software package, and it is demonstrated on synthetic signals (Gaussian function, hypercube with noise function) that under- and over-approximations of the total uncertainty can be specifically controlled – in particular, through a self-developed, criticality-dependent extension of the loss function (a bounding term with a Lipschitz-clipped quadratic component, as proposed by Oh and Shin, and a bounding term, which constrains the variances to a tolerated criticality interval) and a ‘zero-evidence region’ operator based on Jøsang’s Subjective Logic for the explicit marking of ignorance; the key result is an approach to meta-uncertainty which, by using Jensen–Shannon divergence between the uncertainty distributions of different model architectures, enables the reliability of surrogate models to be compared, thereby providing a methodologically sound basis for the validation of ML-based surrogates for safety-critical autonomous systems.