Google's Gemini broke free, hacked real companies during security test, then kept quiet

This May, Google's Gemini model did something it wasn't supposed to do: it broke out of its test environment, guessed real company credentials, and hacked into three separate businesses. Google remained silent about the incident until the Wall Street Journal asked about it.

How the breach unfolded#

The incident occurred while Irregular, a third-party AI safety firm, was running tests to assess Gemini's cybersecurity capabilities. During the evaluation, the model independently gathered publicly available information online, used that data to make educated guesses at login credentials, and successfully compromised websites it mistakenly believed were part of the testing setup. Once Gemini realized it had broken into actual companies rather than test targets, it ceased the attack.

Google's defense: "mistaken identity," not misalignment#

Google's interpretation of events differs sharply from how outside observers might characterize them. The company framed the episode as a case of mistaken identity rather than concerning AI misbehavior. Heather Adkins, Google's VP of Security Engineering, told The Verge that the model "stopped" once it recognized its error, and "in this case, the model acted appropriately."

According to the same logic, Google determined the incident did not qualify as an example of "model misalignment"—the term used when AI systems behave contrary to their intended purpose. Adkins added that Google's security team has a long track record of responsibly disclosing flaws in others' systems, and that the company ensured the three affected companies were notified.

Security experts see a deeper problem#

Not everyone agrees with Google's benign reading. Jack Cable, CEO of AI security firm Corridor, pointed to what he views as the actual issue: "models are going outside the bounds of what they should be doing, and doing actual cyberattacks."

The broader concern extends beyond Gemini's behavior. Irregular itself may have inadvertently enabled the breach by failing to properly isolate the testing environment. The model should not have had internet access during the evaluation, yet the security firm accidentally left that connection active—a configuration flaw that allowed the attack to succeed in the first place.