New AI Detection Tool Explains Why It Flags Machine-Generated Text

Detecting whether text was written by humans or machines has become increasingly important as AI language models grow more sophisticated. Researchers have now introduced a new approach that goes beyond simple yes-or-no classifications: NOTAI.AI, a system designed to explain its detection decisions in human-readable terms.

How It Works#

Rather than merely flagging content as AI-generated and providing a confidence score, NOTAI.AI breaks down the reasoning behind each prediction. The system combines three complementary analytical approaches: it measures sentence-level patterns in word probability distributions (called conditional probability curvature), incorporates a neural detector component, and evaluates traditional stylometric markers like vocabulary diversity and sentence complexity. These signals feed into an XGBoost classifier, which then uses TreeSHAP to calculate which features most strongly influenced the final decision.

Explainability as a Core Feature#

What distinguishes this system is its commitment to transparency. Rather than operating as a black box, NOTAI.AI allows users to see exactly which linguistic signals—whether probabilistic patterns, neural network outputs, or stylistic markers—drove the detection. The model can then translate this quantitative evidence into plain-language explanations that ordinary readers can understand.

Testing and Performance#

In experiments using a balanced dataset containing human-authored text alongside both authentic and manipulated AI-generated content, the full model achieved an F1 score of 0.9685 on held-out test data. Notably, the system outperformed simpler variants that relied on just one feature type, demonstrating the value of the multi-signal approach. An automatic evaluation found that human judges rated between 94.5% and 98.6% of the generated natural-language explanations as accurately reflecting the detector's reasoning.

Availability#

The researchers have released the web interface, source code, and a demonstration video publicly, making the tool accessible to researchers and practitioners interested in AI-generated text detection and interpretability.