Toward adaptive human-robot collaboration in manipulation tasks via fine-tuned robotic foundation models

Details

Supervisors

Ronsse, Renaud ; Macq, Benoît ; Garcia Ricardez, Gustavo Alfonso ; El Hafi, Lotfi

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] : ingénieur civil électromécanicien, à finalité spécialisée: mécatronique

Abstract

enJapan’s Society 5.0 envisions a future in which humans and robots coexist harmoniously, collaborating seamlessly to build a more advanced, inclusive, and human-centred society. At the core of this vision lies the imperative for robust Human-Robot Collaboration, with a key requirement being the creation of safe and adaptive environments that enable fluid interaction between both agents. This master’s thesis investigates the potential of robotic foundation models specialised in manipulation, which are large data-driven AI systems designed for robotic control, as a pathway toward enabling such collaboration. In particular, it explores whether these models can learn and generalise object manipulation tasks in the presence of humans, a dimension still underexplored in current literature. One of the central challenges in this domain is the lack of datasets that include human interaction, primarily due to the cost and complexity of collecting human-in-the-loop demonstrations. To address this limitation, this work leverages a robotic foundation model in combination with a UR5e robotic arm, fine-tuned using a custom-collected dataset of 2,275 episodes (totalling approximately 7 hours and 37 minutes) across eight distinct manipulation tasks. All demonstrations were collected via teleoperation, enabling precise control over the robot’s actions during data acquisition and diversity in the dataset. For each task, the model was fine-tuned individually, creating task-specific variants aimed at evaluating performance across different task complexities, levels of generalisation, and degrees of human involvement. The spectrum of Human-Robot Interaction explored in this work ranges from scenarios where the human is a passive element in the environment, to cases involving dynamic interactions, and finally to tasks requiring full collaboration, where the robot actively works alongside a human partner. The findings show that while task-specific fine-tuning enhances performance, it also reduces some of the model’s generalisation capabilities. Nonetheless, the results highlight the critical role of data quantity and diversity in unlocking the potential of robotic foundation models. This work provides compelling evidence that meaningful human-robot collaboration is feasible using current models. Moreover, it suggests that future research should explore globally fine-tuned models trained on broader, human-inclusive datasets encompassing varied tasks, objects, and spatial configurations, as a more scalable and generalisable approach than training multiple task-specific models.