New method breaks AI bias in image recognition by shuffling objects and backgrounds

Researchers have developed a new approach to reduce hidden biases in image classification models that cause them to fail when images differ from their training data. The technique, called ForAug, decomposes images into their object and background components and recombines them in varied ways to train more adaptable systems.

The core problem#

Large-scale image datasets like ImageNet contain strong compositional patterns: subjects are typically positioned in the center of frames, appear at consistent scales, and associate with predictable environmental context. While models trained on such data achieve high accuracy on similar images, they struggle dramatically when encountering photos where objects are positioned differently, appear smaller or larger, or sit against unfamiliar backgrounds—a phenomenon called distribution shift.

How ForAug works#

The method separates each training image into foreground and background using segmentation models, then synthesizes a neutral replacement background through inpainting. The isolated foreground object is then positioned on multiple diverse neutral backgrounds at varying positions and scales, followed by standard augmentation techniques. This factorization approach directly manipulates three key sources of bias: object placement, object size, and background context.

Performance gains#

Testing across ten different neural network architectures showed substantial improvements. The technique boosted ImageNet accuracy by as much as 6 percentage points and delivered gains up to 7.3 percentage points on specialized recognition tasks. More impressively, models trained with ForAug showed resilience to distribution shifts—achieving improvements of up to 19 percentage points on standard evaluation benchmarks designed to test robustness.

Diagnostic capabilities#

Beyond training improvements, the method's control knobs allow researchers to quantify specific biases: how much models depend on background details, whether they fixate on object centers, and whether they struggle with size variations. The authors have made code and datasets available for the research community.