In context
Deep reinforcement learning (DRL) was moving from game simulations toward real-world industrial applications, but sample inefficiency and the exploration-exploitation dilemma remained key barriers. This survey, published in the Chinese journal Robot in late 2024, addressed the need for systematic analysis of exploration strategies, which are critical for training robots and autonomous systems in complex, sparse-reward environments.
What was reported
The survey, by Li Shilei and colleagues at Naval University of Engineering, classified exploration policy generation methods for single-agent online DRL based on their relationship to the task policy. It focused on methods in reward space, which add intrinsic rewards to encourage exploration, and in parameter space, which use neuroevolution with fitness functions balancing performance and diversity.
The authors also reviewed hybrid approaches combining action-space and parameter-space exploration, as well as high-level task-target and task-independent exploration methods. They discussed safety constraints for exploration policies, noting that in industrial scenarios like resource scheduling and autonomous driving, exploratory actions must not cause harmful consequences.
The survey highlighted that classical methods, such as adding random noise to task policy outputs, are sample-inefficient and limit DRL's applicability. Newer methods aim to decouple exploration from task policy, improving sample quality and training efficiency.
Why it mattered
This work provided a structured framework for understanding exploration strategies, which is essential for making DRL practical in manufacturing automation, where real-world trials are costly and safety is critical. By categorizing methods and identifying challenges, it offered guidance for developing more efficient and safer learning algorithms for robotic control and autonomous systems.
"How to design an efficient exploration strategy to guide the sampling process... has become a prerequisite for the stable convergence of online DRL algorithms."
Source: 《机器人》期刊 (robot.sia.cn) · Published 2024-11-15 · “在线深度强化学习探索策略生成方法综述”