AI-Powered Teaching Tools Risk Amplifying Gender and Racial Biases in Software Education

Generative AI systems are now widely adopted in educational contexts, but researchers warn that their use in teaching software engineering may perpetuate harmful social biases that shape student outcomes and career paths.

A recent investigation examined how GenAI tools behave when deployed for two common teaching tasks: assembling student teams based on profiles and creating visual content for course materials. The results expose persistent demographic patterns that disadvantage women and people of color.

How Bias Emerges in Team Formation#

When AI systems evaluated student profiles to assign roles within project teams, gender and nationality significantly influenced the outcomes—independent of actual qualifications. Women with identical credentials were more frequently placed in front-end development positions, while men received assignments for higher-status roles more often. This kind of algorithmic gatekeeping can narrow career trajectories early, steering capable students away from sought-after specializations before they even realize it.

Skewed Representation in Generated Images#

The bias manifests differently when these tools generate visual materials for lectures and course slides. When asked to produce images of diverse groups of people, the AI systems generally deliver balanced demographic representation. However, when depicting a single individual—such as a programmer or software engineer—the generated outputs overwhelmingly favor male and light-skinned faces. This subtle but pervasive pattern sends implicit messages about who belongs in technical fields.

Why This Matters in Education#

The consequences of AI-driven bias in classrooms extend beyond immediate discomfort. Early exposure to these messages shapes student self-perception, academic engagement, and long-term career interests. Instructors using these tools may inadvertently signal that certain groups are better suited for particular technical roles, potentially discouraging capable students from pursuing their ambitions.

Path Forward#

The findings underscore the urgency of developing domain-specific auditing and safeguard mechanisms before deploying generative AI in education. Without deliberate intervention, these systems risk automating and institutionalizing the very inequities the tech industry has struggled to address.