Recon 2024

Architecture Analysis of VMProtect 3.8: Demystifying the Complexity
06-30, 11:00–12:00 (US/Eastern), Grand Salon

VMProtect stands as one of the most sophisticated software protection systems employed in obfuscating malware. Increasingly utilized by malware authors, it is crucial for reverse engineers to understand potential attack vectors and key functionalities. This presentation delves into the latest architectural changes of VMProtect 3.8, sharing insights from our extensive research.


The focus will be on the new architecture for the latest VMProtect and techniques for attacking or reversing protected binaries. I will demonstrate how reverse engineering techniques—such as symbolic execution and binary instrumentation—can facilitate the de-virtualization or de-obfuscation of the protected code. Tools like Dynamic Data Resolver (https://blog.talosintelligence.com/dynamic-data-resolver-1-0/), which I wrote earlier, will also assist the reversing process. The research will extend and update former research done on VMProtect like Jonathan Salwan's work (https://github.com/JonathanSalwan/VMProtect-devirtualization) or projects like https://blog.back.engineering/17/05/2021/.

Attendees will gain a comprehensive understanding of VMProtect's inner workings and the ability to develop their tools for analysis, tailored to keep pace with VMProtect's continual evolution.

Importantly, I will discuss whether malware authors deploy VMProtect effectively or make configuration errors. While sometimes daunting, these protections can often be reversed within hours; however, there are instances where the complexity significantly escalates. The talk aims to help attendees identify these variations and will highlight the historical improvements and usage statistics of VMProtect in malware, underscoring the importance of focusing on this technology.

Structure of the Talk: Introduction to VMProtect, Operational Mechanics, Feature Set Overview, Architectural Changes in Version 3.8, Exploring Attack Vectors (Reverse Engineering Techniques,Symbolic Execution,Binary Instrumentation), Analysis of Efficacy (Making a judgment,Exploiting Configuration Errors), Successful Attack Examples, Tool Development Strategies

This talk is designed to empower researchers to better understand and combat the challenges posed by VMProtect, fostering a more profound knowledge base and enhancing custom tool development capabilities.

See also: VMProtect Presentation (9.4 MB)

Holger is a longtime security enthusiast, with more than 25 years of experience in the information security industry. He started his career as a penetration tester and is now working for Cisco Talos as technical leader in the malware and threat hunting sector. He finds new, cutting-edge security threats and analyzes their components. Holger gave talks at international security conferences such as Recon, BlackHat, HackInTheBox, ISC, NorthSec, CiscoLive and others. He is also the author of several offensive and defensive security tools and won the IDA plugin contest with his Dynamic Data Resolver (DDR) IDA plugin in 2020. Recently, he did extensive research on reversing Nim binaries (Recon talk 2023).