Enhancing malware analysis with concolic execution

Details

Supervisors

Mens, Kim

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en sciences informatiques, à finalité spécialisée

Abstract

enMalware is constantly evolving, making it increasingly difficult to analyze using traditional techniques. To keep up with these changes, new tools and methodologies are required. In this context, the Symbolic Execution for Malware Analysis (SEMA) toolchain was developed. By using symbolic execution, SEMA analyzes binaries to generate System Call Dependency Graphs (SCDGs), which can then be used to classify malware and assist in future detection. Symbolic execution offers many advantages but remains vulnerable to complex obfuscation techniques, including packing. In such cases, concolic execution (a hybrid approach combining concrete and symbolic execution) allows the unpacking phase to be executed concretely before turning into symbolic state. This master’s thesis enhances SEMA by integrating concolic execution using Symbion, a component of the angr framework. Despite its limitations, this study establishes a foundation for future enhancements and research involving concolic execution within SEMA.