Yufei Zhang (Imperial College London)

Towards sample-efficient continuous-time reinforcement learning

Abstract: Recently, reinforcement learning (RL) has garnered significant research interest. However, much of this attention and success has been focused on the discrete-time setting. RL for continuous-time systems, despite its natural applicability to a wide range of control tasks in physical systems, remains largely unexplored with limited progress. In particular, characterizing the sample efficiency of continuous-time RL algorithms continues to be a challenging and open problem. In this talk, we introduce a principled framework for systematically analyzing regret bounds for continuous-time RL algorithms in episodic model-based settings. Our framework decomposes the regret analysis into two fundamental components: the finite-sample accuracy of model estimation and the performance gap of a greedy policy derived from the estimated models. The finite-sample accuracy is analyzed using filtering theory and concentration inequalities for correlated observations, while the performance gap is quantified by leveraging the robustness of the control problem. By integrating these insights, we optimize the exploration-exploitation trade-off, achieving the best-known sample complexity in the current literature.