Researchers Propose Reference-Free Quality Metrics for Converting Flowchart Images into Code

Vision-language models are now commonly deployed to extract flowcharts and other document images into executable code representations. However, in real-world production environments, it's often impossible to validate the quality of this conversion because no reference code exists to compare against. Evaluating output accuracy in these scenarios poses a significant challenge.

The Evaluation Problem#

When systems process arbitrary flowchart images on demand, quality assessment becomes difficult without ground truth. Traditional evaluation metrics rely on comparing generated output to a reference version, but in many production settings this luxury doesn't exist. The inability to measure output fidelity in real time makes it hard to monitor system performance or flag potential failures.

Proposed Framework#

Researchers developed a reference-free evaluation approach that works at inference time using only the input image and generated output. The framework introduces two key metrics:

• RecallOCR: Estimates whether the generated code captures content from the original image by extracting text via OCR and treating it as a proxy reference
• PrecisionVE: Detects fabricated or hallucinated elements by using visual entailment to verify generated elements actually appear in the source image

These combine into F1OCR-VE, a unified quality score computed from their harmonic mean.

Validation Results#

Testing on the FlowVQA dataset demonstrated strong alignment with ground-truth evaluation methods, with correlations of 0.97, 0.91, and 0.94 for recall, precision, and F1 respectively. This high agreement suggests the framework reliably approximates traditional reference-based evaluation without requiring labeled ground truth.