New Tool Detects Memory Bugs in Closed-Source Programs Without Source Code Access

Researchers have developed MallocSan, a memory safety tool that can detect heap corruption vulnerabilities in compiled C and C++ applications even when the original source code is unavailable. The approach addresses a long-standing challenge in software security: existing memory-checking solutions typically require either access to source code, control over the build process, or specialized hardware.

How MallocSan Works#

The tool operates by intercepting memory allocation calls through the LD_PRELOAD mechanism, injecting an object identifier into the high bits of each protected pointer. When a program attempts to dereference a malformed pointer, the system triggers a fault at the offending instruction. MallocSan then decodes and patches that instruction at runtime, enabling the application to perform per-object bounds checks entirely within user-space on subsequent runs.

For instructions that resist patching, the tool falls back to two alternatives: in-handler emulation or single-stepping. An optional optimization pass can rewrite frequently executed problematic sites offline using profile guidance.

The system also extends its capabilities to vector operations, handling gather and scatter instructions, and allows policies to scope which parts of an application receive coverage.

Performance and Effectiveness#

Testing on seven SPEC CPU 2017 benchmarks showed MallocSan achieving a geometric mean overhead of 4.82x compared to unprotected execution—substantially better than Valgrind Memcheck's 18.55x slowdown. The tool outperformed Memcheck on five of the seven benchmarks tested.

MallocSan maintained significant parallel scaling on multithreaded workloads, including the 644.nab_s benchmark and pigz. In validation using the Juliet test suite, it detected all seeded memory violations without producing false positives on correct code. The tool also caught real-world bugs, including a documented one-byte overread vulnerability in LibTIFF's tiffcrop utility.

Significance for Security#

Memory-corruption errors continue to rank among the most damaging vulnerabilities in C and C++ codebases. By enabling deterministic detection without requiring recompilation or specialized hardware, MallocSan offers a practical path to memory safety for legacy and third-party applications where source-code access is limited or impossible.