Adversarial Tool for Breaking Static Detection of Executable Packing

Details

Supervisors

Legay, Axel ; Bertrand Van Ouytsel, Charles-Henry ; D'Hondt, Alexandre

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en cybersécurité, à finalité spécialisée: conception et analyse de systèmes

Abstract

Executable packing acts as a double-edged sword in the cyberspace, used to safeguard legitimate software but also for concealing malware. This technique changes how a computer program looks without affecting how it works, making it harder for security tools to detect when something is harmful. The ability to detect whether an executable is packed or not becomes capital in this scenario, as it signals the need for further investigation to understand the nature of the packing and the potential intentions behind it. Before any meaningful analysis of the executable behavior or code can be conducted, it is essential to determine if the executable has been packed and reverse the packing process for thorough investigations. Given that the authors of packed executables actively try to evade detection, analysis and reverse engineering of their code, it is imperative for a packing detection tool to be robust against adversarial attacks. A great deal of research has focused on the static features of executables and their use with machine learning to either detect packing or maliciousness. In particular, some studies defined sets of such features, proving their relevance with many learning algorithms and a more recent study further identified the most significant of them, specifically for static detection of packing. In 2022, an experimental toolkit named Packing Box, presented by D’Hondt in the master thesis "Experimental Toolkit for Studying Executable Packing – Analysis of the State-of-the-Art Packing Detection Techniques", was introduced for static analysis of packed executables. This aims to enable researchers to more effectively analyze and understand the detection of packed executables. Following this, an adversarial study, presented by Jennes in the master thesis "Adversarial Learning on Static Detection Techniques for Executable Packing" , targeted the vulnerabilities of static packing detectors and explored how specific binary alterations could evade static detection mechanisms. This thesis aims to enhance the Packing Box, especially pushing the adversarial study of Jennes one step further by improving the current alterations set then expanding it and by studying their effects and impact on detection mechanisms. Our main contribution is thus to review and enhance existing alterations, to make new ones, to test them and to introduce a new tool based on them, NotPacked++. Furthermore, this attack tool is the weaponization of current adversarial studies and is designed to apply complex combinations of interleaved or interdependent alterations to packed executables that we call super-alterations, making them evade static detection by most common packing detectors. In this way, researchers in packing detection could use the tool to test the vulnerability of their model during the design phase, which is a key component to improve static detectors against malware.