Graph-Based retrieval of artworks from narrative inputs for memory-triggering applications

Details

Supervisors

Macq, Benoît ; Lugan, Sébastien

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] : ingénieur civil en mathématiques appliquées, à finalité spécialisée

Abstract

enMany deep-learning approaches to artwork retrieval from narrative inputs offer little interpretability: the reasoning that links a story to a retrieved artwork remains opaque. This master’s thesis proposes a hybrid, graph-based alternative that performs retrieval as an explicit two phase pipeline. The method relies on pre trained neural representations for encoding, but carries out ranking and reasoning inside an explicit heterogeneous graph, using shortest paths (Dijkstra) and cosine similarity. In the first phase, an artwork is retrieved from the semantic content of the story (narrative input), where the emotion inferred from each artwork already contributes to the score; in the second phase, the best artwork from the pool is retrieved, according to a target emotion specified by the user, after being linked to a pre-defined set of nine emotions. In a small exploratory study with ten participants, the method reached an 80% success rate on a controlled image retrieval task and a mean opinion score of 3.1/5 on emotional alignment, while satisfying the five-second inference-time set as a reasonable target. We also evaluated the algorithm with quantitative and qualitative benchmarks against CLIP. These preliminary results suggest that an interpretable, graph-based approach is a promising direction for artwork retrieval from narrative input, offering an explainable alternative to purely deep-learning pipelines.