Topic: Boffins devise newfangled system to block rootkits (HookSafe)

[Samker]

Scientists are set to unveil a lightweight system they say makes an operating system significantly more resistant to rootkits without degrading its performance.

The hypervisor-based system is dubbed HookSafe, and it works by relocating kernel hooks in a guest OS to a dedicated page-aligned memory space that's tightly locked down. The researchers, from Microsoft and the computer science department at North Carolina State University, plan to present their findings Thursday at the 16th ACM Conference on Computer and Communications Security: http://www.sigsac.org/ccs/CCS2009/

The team installed HookSafe on a machine running Ubuntu 8.04, and found the system successfully prevented nine real-world rootkits targeting that platform from installing or hiding themselves. The program was able to achieve that protection with only a 6-percent reduction in performance benchmarks, making HookSafe "the first system that is proposed to enable large-scale hook protection with low performance overhead," the researchers said.

Rootkits that rely on a method known as kernel object hooking involve modifying kernel data hooks. Because they are scattered throughout the operating system memory, and often co-mingled with other kernel data, they are generally hard to protect. Scientists have dubbed the problem the "protection granularity gap" because effective protection requires byte-level granularity while commodity computers allow only for protection at the much broader page level.

The researchers worked around this limitation by relocating almost 5,900 kernel hooks scattered across 41 physical pages to a page-aligned central location. They then used a "thin hook indirection layer to regulate accesses to them with hardware-based page-level protection."

They tested the protected system against nine rootkits written for the Linux 2.6 kernel. Seven of them failed to install at all thanks to the memory protection, while the remaining two failed to hide themselves because of the hook indirection.

The researchers are Zhi Wang, Xuxian Jiang and Peng Ning of North Carolina State University and Weidong Cui of Microsoft Research. A PDF of their paper is available here: http://discovery.csc.ncsu.edu/pubs/ccs09-HookSafe.pdf

(Register)