Soft actor-critic for data center cooling optimization using EnergyPlus

Details

Supervisors

Macq, Benoît ; Contino, Francesco

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en sciences informatiques, à finalité spécialisée

Abstract

Data centers are characterized by high and continuously growing energy consumption, with a large proportion of this demand arising from cooling systems required to maintain safe thermal operating conditions. Improving cooling efficiency while preserving thermal reliability therefore represents a major challenge for both economic and environmental sustainability. Conventional control strategies are often limited by the complexity of modern HVAC systems and by the difficulty of explicitly modeling all relevant dynamics and uncertainties. This master thesis investigates the application of deep reinforcement learning to the control of data center cooling systems, with the objective of reducing cooling overhead while respecting strict thermal constraints. A simulationbased control framework is developed by coupling a physics-driven EnergyPlus model with a Soft Actor-Critic agent operating in a continuous action space. The controller adjusts chilled-water and supply-air temperature setpoints based on observed thermal and energy-related states. Training and evaluation are conducted in a fully reproducible offline setting, allowing systematic analysis of intermediate policy snapshots. Beyond final performance metrics, this work places particular emphasis on learning dynamics and control behaviour. Offline evaluation of successive policy snapshots reveals how the agent progressively transitions from unsafe, energydriven strategies toward stable and physically consistent operating regimes. The learned policy achieves an average reduction of approximately 2% in Power Usage Effectiveness compared to a baseline controller. Time-series analysis, actuation trajectories, and correlation studies further confirm that the final policy exhibits coherent thermal behaviour and interpretable control patterns. While the observed gains remain moderate, they are consistent with recent state-of-the-art results and highlight the intrinsic difficulty of the problem. The study also discusses limitations related to model simplicity, reliance on simulation, and generalization across operating conditions. Overall, the results support the relevance of entropy-regularized reinforcement learning as a promising approach for data center cooling control, while emphasizing the need for richer models, broader validation, and careful transition toward real-world deployment.