SE Researchers Reveal Gap Between ML Best Practices and Actual Implementation

Machine learning is reshaping how software engineers tackle complex problems, yet little is known about how researchers themselves are approaching ML in their work. A new study surveyed software engineering researchers to understand what practices they follow when researching, teaching, and reviewing ML applications in the field.

What the Study Found#

The researchers examined both the practices of SE scholars familiar with ML and the published papers themselves. The analysis revealed a consistent pattern: while certain techniques are widely recognized as best practices, they rarely show up in the actual research literature. For instance, hyperparameter tuning—a recommended step for optimizing machine learning models—appeared in fewer than 20% of reviewed papers.

The study identified common work streams across the field, including data collection, model training, and evaluation. However, significant variation exists in how these activities are executed and documented.

Key Challenges in Practice#

Researchers face persistent obstacles when working with ML in a software engineering context. Major pain points include:

• Data handling issues — preparing, managing, and ensuring quality of datasets
• Model evaluation gaps — difficulty assessing performance, particularly for non-functional properties like reliability and efficiency
• Integration of human judgment — challenges in incorporating domain expertise and human review into the evaluation process

Teaching and Professional Development#

Educational approaches show a mixed picture. Hands-on, practical activities are common in how ML is taught to software engineering students and practitioners. Yet traditional classroom methods remain widespread, suggesting that the field has not fully embraced modern pedagogical approaches for ML instruction.

Bridging the Practice-Research Gap#

The study concludes that despite general agreement on best practices for applying ML to software engineering problems, significant gaps persist between accepted standards and what actually happens in research and education. Closing this divide will require clearer guidelines, more diverse teaching strategies, and greater emphasis on underutilized practices across the community.