AI
AI-generated phishing texts rival human-written attacks in convincingness, study finds
A small pilot study comparing GPT-4-crafted SMS phishing messages with those written by humans suggests that language models may lower the barrier for mounting large-scale personalized social engineering campaigns.
Researchers have conducted an empirical comparison of phishing text messages created by large language models versus those crafted by human authors, revealing that AI-generated variants can be nearly as effective at tricking targets as manually written ones.
In this 25-person pilot study, investigators at arXiv tested personalized SMS phishing (known as "smishing") attacks. They contrasted messages generated by GPT-4 through simple prompts against messages written by student volunteers working under time pressure. The team introduced a methodological framework called the Threshold Ranking Approach for Personalized Deception (TRAPD) to evaluate how convincing each message was.
Key Findings on Effectiveness#
The research found that GPT-4-generated messages prompted people to click at a rate of 28%, compared to 21% for the student-authored versions—a difference that remained statistically uncertain at this small sample size. The results underscore that straightforward AI prompting can produce deceptive personalized content that rivals human handiwork in persuasiveness.
Notably, messages framed around job-related topics proved significantly more persuasive than those touching on hobbies or social media, suggesting attackers may be able to craft particularly potent messages by focusing on professional contexts.
Detection and Attribution Challenges#
When asked to identify whether a message was written by a human or generated by AI, study participants performed no better than random guessing. This finding points to a growing challenge: even as AI-assisted messaging becomes more accessible, humans struggle to distinguish machine-generated deception from authentic human communication.
The researchers noted that while the two sets of messages remained computationally distinguishable through text analysis, participants' naked-eye judgment was unreliable—a gap that could spell trouble for defense strategies that rely on human perception alone.
Implications for Cybersecurity#
The work suggests that readily available AI tools could dramatically expand the reach and scale of personalized phishing campaigns. Rather than requiring attackers to manually research and write unique messages for dozens or hundreds of targets, language models enable rapid, low-friction generation of tailored social engineering content.
