New Framework Adapts SAM for Thermal-RGB Scene Understanding in Poor Lighting

A team of computer vision researchers has developed SARTM (Segment Any RGB Thermal Model), a framework that extends the powerful Segment Anything Model (SAM) to handle combined thermal and RGB imagery for improved scene understanding. The work addresses a key limitation of SAM: its training on RGB data alone makes it poorly suited for thermal imaging applications.

Why Thermal-RGB Matters#

Thermal sensors capture heat signatures alongside traditional visible-light data, creating a complementary pair that works especially well when conditions are challenging. Low-light environments, overexposure from bright sunlight, fog, and other adverse weather all become more manageable when thermal data augments standard RGB input. This capability makes thermal-RGB systems valuable for autonomous vehicles, surveillance, search and rescue, and industrial inspection.

The Technical Approach#

Rather than retraining SAM from scratch, the researchers preserved its strong generalization abilities by fine-tuning it with LoRA layers — a parameter-efficient adaptation technique. To bridge the gap between thermal and RGB modalities, they introduced Cross-Modal Knowledge Distillation (CMKD), which uses language information as semantic guidance during training. This module minimizes inconsistencies between the two imaging types and reduces semantic ambiguity.

The team also redesigned SAM's segmentation head and added an auxiliary semantic segmentation branch that pulls together multi-scale features for better fusion across modalities.

Performance Validation#

Testing on three major benchmarks — MFNET, PST900, and FMB — the approach consistently outperformed existing state-of-the-art methods across diverse lighting and weather scenarios. Both quantitative metrics and visual results demonstrated the framework's effectiveness.

The researchers have published their code and pre-trained weights publicly to enable further development in this area.