ACTION-MASKED REINFORCEMENT LEARNING TECHNOLOGY FOR ORDER SCHEDULING

The problem of high-performance and efficient order scheduling is a common combinatorial optimization problem in various industrial contexts. Creation of a model capable of generating schedules balanced in terms of quality and computational time poses a significant challenge due to the large action space. This study proposes a high-performant environment and a reinforcement learning model for allocating orders to resources using a mechanism of invalid action masking. The developed reinforcement learning solution overcomesthe limitations of traditional heuristic and exact methods regarding computation performance and efficiency. The research included the design of a Gymnasium-compatible simulation environment, performance benchmarking, development of optimized environment state updating procedures, feature generation strategies, and evaluation of PPO and MaskablePPO models. The environment implemented incremental updates of the features state and action masks with extensive NumPy vectorization, significantly reducing computational overhead and improving compatibility with deep learning policies. The invalid action masking replaced penalty-based constraints by mandatory restrictions on the agent’s action space to feasible decisions, thus enhancing the policy’s accuracy by focusing on valid and more optimal choices. Datasets containing up to 500 orders were generated, on which PPO and MaskablePPO models from the Stable-Baselines3 library were trained. Each model was trained for 100,000 iterations. Training progress was monitored using TensorBoard. The masked version required 1.49 minutes for training, while the unmasked model completed training in 1.2 minutes. For Masked PPO, the average per-step penalty was 2.41, while for PPO it was 325,000. These results demonstrated that the standard PPO frequently selected invalid actions, collecting heavy penalties, whereas the MaskedPPO accumulated penalties solely related to the schedule length. As a result, on the test dataset, MaskedPPO completed the schedule calculation in 0.18 seconds, producing a schedule with a total duration of 4.590 minutes, compared to 5.4 seconds and 5,127 minutes for standard PPO, which made invalid action attempts in 96 % of the cases. It wasfound that action masking significantly improved the model’s accuracy and convergence despite a slightly longer training time. The results reveal the strong potential of reinforcement learning approaches for order scheduling and combinatorial optimization problems in general. The proposed Masked PPO model resulted in reduced order allocation time compared to the traditional exact CP-SAT method while maintaining higher schedule quality than SPT heuristic approache on problems with 50500 jobs. This work established the foundation for future research involving more complex model architectures based on Set Transformers, Graph Neural Networks, and Pointer Networks. These enable effective generalization and allow the trained policies to be applied to problem instances with higher input dimensions than those seen during training.