Offline Human-in-the-Loop Reinforcement Learning for Personalized Tuning of a Powered Ankle-Foot Prosthesis

Details

Supervisors

Piette, Eric ; Ronsse, Renaud

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] : ingénieur civil en informatique, à finalité spécialisée

Abstract

enPowered ankle-foot prostheses can inject mechanical work at push-off, yet their controllers still rely on manual parameter tuning performed by an expert rather than by the user. This process is slow, subjective, and hard to scale. This thesis investigates whether reinforcement learning can automate part of this tuning while keeping the user at the center of the objective. It introduces an offline Human-in-the-Loop Reinforcement Learning (HITL-RL) framework that adds a learning layer above the existing mid-level controller of ELSA, the Efficient Lockable Spring Ankle developed at UCLouvain. The developed framework observes compact biomechanical deviations from a normative gait reference, integrates the user's comfort and perceived-assistance scores, and learns how to update three interpretable control parameters between successive walking blocks. Learning is formulated as an Offline Policy Iteration problem with a quadratic cost-to-go, trained without any online exploration on a dataset collected from a real experimental campaign with the prosthesis. Two research questions guide the analysis. The first asks whether such a layer can coexist with the existing controller without weakening its safety guarantees or its clinical interpretability. The second asks whether subjective feedback brings measurable learning value over a purely biomechanical reward in a small-data offline regime. A structured ablation campaign answers both. A single targeted reshaping of the action cost reduces the two main weaknesses of the baseline policy, namely action saturation and sensitivity to gait variability, while preserving its interpretable structure. The hybrid reward reaches the best generalization of the whole campaign, and both reward extremes degrade every scale-free diagnostic, which shows that the human ratings carry information the biomechanical features alone do not capture. The contribution is therefore not a deployed clinical controller but an offline proof of concept and a reusable evaluation pipeline. To the best of our knowledge, it is the first application of Offline Policy Iteration to a powered ankle-foot prosthesis and the first to embed comfort and perceived-assistance scores directly into its reward.