Best System Identification Using Ordinal Optimization

D. Ahn

Given a number of stochastic systems and a finite sampling budget, we consider an ordinal optimization
problem to find an optimal allocation that maximizes the likelihood of selecting the system with the best
performance. Generalized linear models are used to describe the relationship between system performance and feature vectors, and unknown parameters are estimated using maximum likelihood estimation. We first formulate the problem in a tractable form by characterizing the structural properties of the optimal allocation with the large deviations theory and then obtain a Euclidean approximation for the optimal allocation. This enables us to design a sampling strategy that is near-optimal particularly when the first- and second-best systems are comparable. The proposed sampling strategy turns out to be not only computationally tractable when the model is correctly specified but also applicable to the case of model misspecification.