Multi-task reinforcement learning in partially observable stochastic environments

Hui Li, Xuejun Liao, Lawrence Carin

Research output: Contribution to journalArticlepeer-review

43 Scopus citations

Abstract

We consider the problem of multi-task reinforcement learning (MTRL) in multiple partially observable stochastic environments. We introduce the regionalized policy representation (RPR) to characterize the agent's behavior in each environment. The RPR is a parametric model of the conditional distribution over current actions given the history of past actions and observations; the agent's choice of actions is directly based on this conditional distribution, without an intervening model to characterize the environment itself. We propose off-policy batch algorithms to learn the parameters of the RPRs, using episodic data collected when following a behavior policy, and show their linkage to policy iteration. We employ the Dirichlet process as a nonparametric prior over the RPRs across multiple environments. The intrinsic clustering property of the Dirichlet process imposes sharing of episodes among similar environments, which effectively reduces the number of episodes required for learning a good policy in each environment, when data sharing is appropriate. The number of distinct RPRs and the associated clusters (the sharing patterns) are automatically discovered by exploiting the episodic data as well as the nonparametric nature of the Dirichlet process. We demonstrate the effectiveness of the proposed RPR as well as the RPR-based MTRL framework on various problems, including grid-world navigation and multi-aspect target classification. The experimental results show that the RPR is a competitive reinforcement learning algorithm in partially observable domains, and the MTRL consistently achieves better performance than single task reinforcement learning. © 2009 Hui Li, Xuejun Liao and Lawrence Carin.
Original languageEnglish (US)
Pages (from-to)1131-1186
Number of pages56
JournalJournal of Machine Learning Research
Volume10
StatePublished - Jan 1 2009
Externally publishedYes

Fingerprint Dive into the research topics of 'Multi-task reinforcement learning in partially observable stochastic environments'. Together they form a unique fingerprint.

Cite this