Low-rank matrix factorization for compressing Transformers

Details

Supervisors

Massart, Estelle

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] : ingénieur civil en mathématiques appliquées, à finalité spécialisée

Abstract

enA compression algorithm framework is developed in this master thesis to compress neural network models. Within this framework, weight matrices that are suitable for compression are identified and then reduced using low-rank approximation methods. Accordingly, a key component of the algorithm is determining the truncation rank to apply to the selected weight matrices. The framework is evaluated on Transformer models, and it is found that the truncation rank configuration chosen by the algorithm yields better compression results than baseline low-rank compression methods. Various scenarios are examined, including the compression of fine-tuned models and pre-trained models, both when samples from the target datasets are available and when they are not.