Fingerprinting PE packer detectors using surrogate-based explainability

Details

Supervisors

Sadre, Ramin ; Bertrand Van Ouytsel, Charles-Henry

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en cybersécurité, à finalité spécialisée: conception et analyse de systèmes

Abstract

Executable packing refers to a series of transformations applied to binary executables, such as compression, encryption, and virtualization. This technique has long been utilized for both legitimate and malicious purposes, including software piracy protection, license management, and malware distribution. Over time, packing methods have become increasingly sophisticated, particularly in bypassing antivirus detection when used in malicious binaries. In malware analysis, packed executables still pose a major challenge for accurate detection today. Over the past two decades, numerous approaches and methods have been proposed in the scientific literature. These rely on heuristics, signatures and machine learning models, performing with various accuracies. Among security products, antivirus use combinations of these techniques, yielding various false positives and negatives rates. The diversity of techniques make these products difficult to rank for performance. In 2023, an experimental toolkit for packing detection was developed by D'Hondt, incorporating re-implementations of various existing techniques. This toolkit serves as a foundation for evaluating and comparing different packing detection methods, enabling researchers to analyze its effectiveness in identifying packed binaries. To enhance its utility, a scoring method is necessary to quantify and compare security products and toolkit's detection capabilities. Such a scoring system would also facilitate the assessment of the toolkit's performance in designing new detection features and improving machine learning models for identifying packed executables. The objective of this thesis is to build a taxonomy of static packing detection techniques and to develop a scoring mechanism so that the toolkit can provide a structured approach to comparing packing detection capabilities of security products. This involves classifying detection techniques and developping automated detection using VirusTotal to get a baseline of security products to compare with. Finally, this entails producing a ranking of in-scope products.