Learn, hack!

URL

http://events.ccc.de/congress/2010/Fahrplan/attachments/1764_Deobfuscation%20-%2027C3.pdf

File name

1764_Deobfuscation%20-%2027C3.pdf

File size

1.1 MB

MD5

eb503776e67755836ebf8b9304397485

SHA1

6ec8c24fb785c328d0a68caf1ab29b7e77e3008b

Optimization algorithms present an effective way for removing most obfuscations that are used today. Much of the compiler theory can be applied in removing obfuscations and building fast and reliable deobfuscation systems. By understanding traditional optimization problems and techniques it is possible to develop and customize compiler optimization algorithms for usage in binary deobfuscation/analysis. Analysis of malware binaries is constantly becoming more difficult with introduction of many different types of code obfuscators. One common theme in all obfuscators is transformation of code into a complex representation. This process can be viewed as inverse of compiler optimization techniques and as such can be partially removed using optimization algorithms. Optimization algorithms present an effective way for removing most obfuscations that are used today. Much of the compiler theory can be applied in removing obfuscations and building fast and reliable deobfuscation systems. By understanding traditional optimization problems and techniques it is possible to develop and customize compiler optimization algorithms for usage in binary deobfuscation/analysis. Optimization algorithms are especially successful in following: • Removal of no operation instructions • Simplifying complex instructions • Removal of unconditional jumps • Removal of conditional jumps • Simplifying control-flow graph This presentation shows common obfuscation techniques and a process of adapting optimization algorithms for removing obfuscations. Additionally, a open-source plug-in for the IDA Pro disassembler is presented that demonstrates usability of the proposed optimization process as well as a set of techniques to speed up the process of analyzing obfuscated code.