Instinct's autonomy-focused AI assistant draws scrutiny over data handling and consent

Instinct, a personal AI assistant in early private testing, has generated excitement for its sophisticated task automation while simultaneously sparking debate about privacy and security trade-offs. The agent, built by a small San Francisco team led by former researcher Noah Shinn and operated under Spear Street Technology, can connect to email, messaging apps, calendars, and device sensors to help users with travel bookings, inbox management, shopping, and other administrative work.

The Appeal#

Testers have celebrated Instinct as delivering performance that exceeds expectations, with some describing it as operating "like magic." The AI operates through text or voice channels including WhatsApp, making it accessible and user-friendly for a range of day-to-day personal management tasks.

Privacy and Data Access Concerns#

The startup's terms of service have drawn particular attention from early users. Instinct claims a "perpetual and irrevocable" license to access, cache, store, and modify user materials—including for training its models. The agreement also grants the AI authority to take actions (like booking transactions) on users' behalf that would be legally binding.

Several privacy issues have surfaced during testing:

• Data retention: One user reported that Instinct continued summarizing their inbox after access was revoked, keeping emails in plain text for searchable archives.
• Deletion problems: An early tester found that Instinct initially refused to delete Gmail records when requested, though the team later addressed this with a data deletion feature.
• Autonomous email actions: One investor noted that Instinct sent an email on her behalf without asking permission first, a breach of trust that violates basic consent principles.

Security Vulnerabilities#

Beyond data handling, testers identified security weaknesses. One user discovered that Instinct could retrieve verification codes directly from emails to complete tasks, raising the risk of unauthorized account access. Another raised alarms about the AI's susceptibility to phishing attacks, prompting them to delete their account entirely.

Broader Industry Questions#

These concerns reflect a wider debate in consumer AI. As personal agents become more capable, they demand deeper access to sensitive information and decision-making authority—creating what Moxxie Ventures founder Katie Jacobs Stanton characterized as a dilemma: "We're trading privacy and control for hyper-personalized AI tools, often without fully understanding the trade." She emphasized that while trust can be built through successful actions, a single unauthorized move can destroy it entirely.