DEV Community

Chris Brown
Chris Brown

Posted on • Originally published at code-world-no-blanket.github.io on

ESEM 2025 Overview

#research

Our paper "Exploring the Evidence-Based SE Beliefs of Generative AI Tools" was accepted and recently presented at the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2025) Emerging Results and Vision track. This post summarizes the problem we explored, provides an overview of our emerging results, and and future vision of AI-assisted software engineering. A preprint is available here.

Exploring the Evidence-Based SE Beliefs of Generative AI Tools [ESEM ERV]

Chris Brown and Jason Cusati

  • 🔍 Problem: Generative AI tools are increasingly used to support software development, yet little is known about the "beliefs" of AI tools with regard to evidence-based practices supported by empirical SE research.

  • 🧪 Study: We conducted a conceptual replication study, prompting generative AI tools---ChatGPT, GitHub Copilot, Gemini, Blackbox AI, and Claude---to respond to questions regarding 17 evidence-based claims used in prior work to assess the beliefs of human software developers [Devanbu2016].

  • 📊 Emerging Results: We found generative AI tools typically have positive beliefs aligning with research claims, however can have ambiguous responses. Further, most tools lack credible evidence to support their claims.

  • 💡 Vision: We provide implications for software developers leveraging generative AI tools in practice. We also discuss future research directions to promote:

  1. Evidence-based generative AI for SE;
  2. Evidence-based AI integration within development workflows; and
  3. Generative AI-based techniques for translating research evidence for practitioners.

This was my first time attending ESEM, and I really enjoyed presenting this work and connecting with members of this community. Visiting Honolulu was also an amazing experience 🏖️ We hope this work provides useful insights and motivates future work to enhance evidence-based practices in generative AI-driven software development.

References

[Devanbu2016] P. Devanbu, T. Zimmermann, and C. Bird, “Belief & evidence in empirical software engineering,” in International conference on software engineering, pp. 108–119, 2016.

Top comments (0)