DEV Community: Chris Brown

Project Presentation-Based Assessments in Software Engineering Courses

Chris Brown — Wed, 18 Mar 2026 00:00:00 +0000

Software engineering education is rapidly evolving, and finding ways to effectively evaluate students' understanding and application of practical skills is critical. Traditional exams often fail assessing students' abilities to apply theoretical knowledge to real-world scenarios. In addition, the rise and increased adoption of generative AI tools have made evaluating students' comprehension of development concepts more challenging.

To address this challenge, recent efforts have proposed a variety of approaches including oral exams [Ohmann2026], environment-restricted exams in a containerized environment [Panter2026], proctored online exams [Leong2025], returning to paper-and-pencil exams [Boyer2025]. These practices are useful for preventing AI misuse on assessments, however face challenges such as potential invasions of student privacy (proctoring), require technical implementation and maintenance (containerized exam), and can be difficulty to scale for large classes (oral exams, written exams). Specifically for my undergraduate software engineering course, I needed an exam approach that could:

(a) scale to a large number of students (n = 96); (b) could be incorporated into final project presentations (due to the large number of students/project groups, several teams had to present during the final exam period to complete the class content); (c) prevented AI misuse; and (d) maintained practical relevance to software development and the concepts discussed in class. This innovative approach leverages student presentations of their course projects as a foundation for assessing practical application of class concepts.

To fill this gap, I implemented a project presentation-based assessment format in my undergraduate software engineering class for the final exam last semester. This assessment approach leverages student presentations of their course projects as a foundation for assessing the practical application of class concepts.

Project Presentation-Based Assessment

For project presentation-based assessments, students must critically observe teams' project presentations and respond to scenario-based questions connected with the course content. Questions are designed to assess students' understanding of specific concepts discussed in class, and can incorporate multi question types (e.g., open-ended, fill-in-the-blank, multiple choice, etc.).

In the initial implementation, students had to envision joining two development teams representing final project presentations. The assessment consisted of ten questions, two of which accounted for extra credit, requiring students to respond to scenarios based on live project presentations in the class period. For instance, sample questions included: "Other than “Agile” and the process used by the presenting team (if applicable), what software engineering process would you use to implement this system. Why?" ; "You are in charge of adding a new feature to this product. What approach would you use to deploy the updated application to users? Why?" ; and "As a new developer on the team, which git command should you use to retrieve the latest version of the code in the remote repository for your local development environment?"; etc. A full sample exam is available here.

To apply this approach, we used our final exam period for the class which consisted of half presentations, and half presentations with ongoing exams. Students had the opportunity to answer questions about two out of approxiamtely 10 presentations, each approxiamtely 20 minutes. The exam was hand-written without the use of technology (i.e., laptops). There were also multiple versions of the exam distributed with different questions and in different orders.

Perceptions

Teaching Staff

From a faculty perspective (n = 1, me), I found this exam format provided the following benefits:

helped increase engagement and attention to project presentations (historically, students have been unengaged or left the presentation period after presenting early---leaving the final groups with little to no audience);
provided critical thinking to apply concepts in class to various scenarios (class projects varied widely in topics and domains, including an improved bus transit system, an automated job application assistant, calendars and task management applications, and more.)

While this exam format did scale well, the main challenge was the time for grading. But, the flexible open-ended question format made grading take more time but also it was more interesting to see how students' would apply different concepts from class into responses.

Students

Initial feedback from students was also positive. Many expressed that this format allowed them to provide a more comprehensive demonstration of their skills and found the interactive format to be engaging. Students also shared the real-world contexts made it easier for them to engage thoughtfully with applying the material, and they appreciated the flexibility to draw upon a range of concepts learned throughout the semester. Feedback from studnts was collected within the exam itself, prompting students to understand what questions are discussed in an agile retrospective meeting and (optionally) applying them to this exam format.

1. What Went Well

Some sample quotes from students are below:

"I enjoyed engaging with an actual project an analyzing it"

"Testing on the presentations is creative"

"It was an interesting way to get me to think more in-deth about other groups' projects"

"This final made us use our knowledge of souftware esign in a live practical format, which I think is a good way to gauge student understanding"

"This exam format is incredibly engaging and thought-provoking, and gives us a reason to pay attention to the presentations that many would normally space out during"

"I think the exam structure being a case study of a presentation is good as it makes us really use our knowledge"

...

2. What Didn't Go Well

The main concerns that arose from student feedback include:

Fairness: Students perceived it was unfair that some students had to worry about presenting and taking the exam simultaneously. For instance, the last several groups to go had to both prepare for presenting while also completing the exam for other groups presentations. The order was decided randomly and students were expected to provide slides ahead of time, however the exam disrupted mental preparations, increased stress, etc.

"Having the exam go on while some people are presenting means some students are more stressed than others"

"It's not super fair that some teams had to present during the final and threfore got less time than their peers"

Focus: While some students found value in completing the presentation-based assessments, others found it distracting and hard to focus on both writing responses and paying attention to the presentation at the same time.

"I got distracted by presenters, often losing my train of thought"

"It was a little difficult to pay attention to the presentation and answer questions at the same time".

Space: Students noted there was not enough space to write responses in the exam using the paper format.
Question Types: While there was a mix of question types (multiple choice and free response), some students felt there was too much writing involved with this exam.

"There was a lot of writing and their could have been more MCQs"

3. Improvements

Students provided some really good ideas for improvements to this assessment approach. Beyond practical improvements (i.e., more writing space, more MCQ questions, more clarity on content, and students should have studied more), suggestions for improvement included:

having the instructor give a mock presentation for students to evaluate in this format rather than students, preventing unfairness to students presenting during the exam;
pre-recording presentations to prevent presentations and exams in parallel;
incorporating more creative practical questions (e.g., "what features would you add")

Conclusion

This post introduces project presentation-based assessment, providing an overview of this approach and preliminary feedback from an initial implementation in Fall 2025. I will explore applying some of the suggested improvements for this approach, potentially in the graduate-version of the SE course this semester or future iterations of the undergraduate course. I'm also looking forward to further exploring students' perceptions of this assessment approach and how it influences student learning outcomes and preparation for software engineering careers.

Footnotes and References

[Ohmann2026] Peter Ohmann and Ed Novak."A Multi-Institutional Assessment of Oral Exams in Software Courses". In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pp. 882-888. 2025.

[Panter2026] Kate Panter, Scott Wehrwein, and Hanxiang Du. "Using Containers to Prevent Generative AI Use in Lab Exams". In Proceedings of the 57th ACM Technical Symposium on Computer Science Education V. 2, pp. 1465-1466. 2026.

[Leong2025] Wai Yie Leong. "E-exams and academic integrity: combating cheating with advanced proctoring solutions". In International Conference on Intelligent Technology for Educational Applications, pp. 326-338. Singapore: Springer Nature Singapore, 2025.

[Boyer2025] Lauren Boyer. "Going old school: Some professors return to pen and paper, tech-free classrooms". The College Fix, 2025. https://www.thecollegefix.com/going-old-school-some-professors-return-to-pen-and-paper-tech-free-classrooms

Preparing Students for Modern Software Development Toolchains

Chris Brown — Tue, 09 Dec 2025 16:56:52 +0000

In the software engineering classes I teach (undergraduate and graduate) I'm working to help students gain hands-on exposure to essential tools necessary for real-world software development. While languages, frameworks, and tools constantly change, the toolchain that supports development evolves more gradually—and mastering it can improve students’ readiness to contribute to software projects in internship or industry roles. A list of tool categories I've considered is below, but I'm curious what other essential tools or tool categories should students understand to be industry-ready for modern software development and beyond?

IDEs
Version Control
Static analysis
Debuggers/logging tools
Testing frameworks
CI/CD
AI coding assistants

Are there other things missing, tools you wish new grads knew about before joining your team, or specific tools that are essential within these categories? I'm open to suggestions!

Bridging the Gap Between Research and OSS Communities

Chris Brown — Fri, 17 Oct 2025 00:00:00 +0000

Reflections from the 2025 OSS Community Leadership Summit

I recently had the chance to attend the Community Leadership Summit at All Things Open 2025, one of the largest tech conferences focused on open-source software (OSS). The summit brings together community leaders and individuals interested in growing and empowering a strong open-source community.

The event was organized as an "unconference", where attendees were able to sign up on the spot to lead discussions or give talks on relevant topics. I decided to contribute, and signed up to lead a discussion on "Bridging the Gap Between Research and OSS Communities". This topic was meaningful because our research group does a lot of work focused understanding and improving OSS development, particularly on GitHub.

Bridging the Gap Between Research and OSS Communities

Approximately 15 people joined the discussion, which generally focused on broad ways academic researchers can bridge the gap to OSS communities they study. There was a general consensus on the need to bridge the gap between academic research and open-source in practice---both related to academics doing open-source development and academics doing empirical research on OSS communities. The following themes emerged from our discussion¹:

1. Existing Resources and Organizations: Attendees noted many OSS organizations actively conduct research and maintain their own data initiatives involving open-source ecosystems. Examples discussed include Linux Foundation Research, CHAOSS, SustainOSS, and CURIOSS. I do not have any experience working with these organizations, but hope to pursue potential partnerships and collaborations to better understand how research can inform and be informed by real-world OSS practices.

2. Academic OSPOs: We also discussed Open Source Program Offices (OSPOs) at higher education institutions, which aim to support open-source initiatives and activities at universities. The functions of these offices are heavily university-dependent, but can provide insights on issues related to open source literacy, tech transfer, licensing, OSS curation, and intellectual property. For example, details about the UC Santa Cruz OSPO are available here.

3. Being a Good OSS Citizen: A major thread of conversation focused on how academic researchers can be a "good citizen" in OSS communities. We briefly discussed a well-known example of controversial research-OSS engagement (University of Minnesota vs. the Linux Kernel) and discussed ways for academics and SE researchers to be good citizens. Some suggestions included:

Engineering contributions to OSS repositories by submitting code or pull requests that align with project policies and procedures. It was specifically mentioned that contributing back to OSS seems to be a low priority for researchers;
Disclosing research activities to avoid "doing anthropology without people involved knowing they are subjects of research" (i.e., scraping public email addresses from GitHub users to distribute surveys, which we are guilty of 🫣);
Sharing credit and recognition of original OSS projects by using project DOIs in references for publications, linking to original repositories, etc.;
Practicing "active outreach" by sharing findings and published work with project leadership and communities;
Using and advocating for open licenses to make findings and code open access and available (see next point); and
Supporting open resources by contributing and using open research and educational resources such as the Journal for Open Source Software, "Papers With Code" (which apparently does not exist anymore?), open textbooks, and Teaching Open Source.

3. Tech Transfer: We also discussed specific challenges related to technology transfer in research, in particular how university tech transfer policies can hinder open-source participation. It was brought up that institutions view research as revenue, so even when research efforts rely on OSS and are not making money, researchers may be prevented from contributing back to OSS. Examples include universities holding rights to intellectual property (IP), inappropriate licenses in academic settings that prevent flexible reuse (i.e., Polyform and AFL), and other copyright concerns. There seems to be growing recognition of this problem, and hopefully solutions will arise to support and promote academic tech transfer to OSS communities.

4. Knowledge Transfer: Finally, we discussed knowledge transfer---or ways to convey research findings to OSS practitioners. Currently, most findings in research communities are summarized in research papers and presented in research conferences that practitioners may not read or attend---although one attendee did mention his boss, a non-academic developer without a PhD or research background, reads the conference proceedings from top SE conferences. Another attendee was specifically interested in research exploring anonymous contributions to OSS, which led to a brief discussion about prior work on to this topic [MurphyHill2021] and also how it may not be feasible in OSS contexts. The methods for effective knowledge transfer discussed were:

Attending OSS-focused events to present at tech conferences and open-source summits, like All Things Open, to promote crossover with industry;
Sharing findings through existing resources targeting OSS communities, such as the Linux Foundation; and
Forming small discussion groups of 10-15 people from OSS development to advise research directions, provide insights on what is real vs. perceived in OSS, etc.

To conclude the session, we agreed the most effective approach for bridging the gap between academic researchers and OSS communities is: " just talk to people".

Closing Reflections

This session was a valuable opportunity to connect with OSS stakeholders from various roles who care about academic research and open-source software development. Please reach out if you have other insights or practical ways to connect research with real-world OSS communities. Also, if you are part of an OSS project I’d be interested to hear your thoughts on how researchers can effectively get involved with projects, conduct research to help (and not hurt) development practices, and be good citizens to OSS communities. Thanks to everyone who participated in this discussion, in addition to the organizers of the Community Leadership Summit and All Things Open. More to hopefully come soon...

Footnotes and References

[MurphyHill202] Emerson Murphy-Hill, Jillian Dicker, Margaret Morrow Hodges, Carolyn D. Egelman, Ciera Jaspan, Lan Cheng, Elizabeth Kammer, Ben Holtz, Matthew A. Jorde, Andrea Knight Dolan, Collin Green. "Engineering Impacts of Anonymous Author Code Review: A Field Experiment". Transactions on Software Engineering (TSE), 2021.

These themes were only coded by me based on notes taken during the disucssion and memory. They are listed mostly in order of discussion. ↩

How Progress Ends: A Software Engineering Perspective

Chris Brown — Fri, 17 Oct 2025 00:00:00 +0000

I was recently invited to give a guest lecture for the course ASPT 6004: AI in History and Its Social and Environmental Implications through the Alliance for Social, Political, Ethical, and Cultural Thought (ASPECT) program at Virginia Tech. This interdisciplinary class discusses the challenges that come with artificial intelligence (AI) and machine learning (ML) technologies from a historical and a political perspective. The assigned reading for students for the guest lecture was "How Progress Ends: Technology, Innovation, and the Fate of Nations" by Carl Benedikt Frey (Chapters 1-3). This book examines the history of civilizations to explore how economic and technological progress advances and stagnates. To prepare for the class and align my talk for the guest lecture, I decided to read through the first three chapters of the book---which I found really interesting.

Discussion Overview

The beginning of the discussion focused on my research---including what is software engineering, our work on AI in hiring processes [Vaishampayan2025], AI for SE education [Wang2025], and general perceptions on AI in software development. We also discussed a variety of other topics, including if AI will peak in performance as more content is AI-generated, difficulties with AI policies and usage (i.e., surveiliance), reducing bias in AI, the importance of teaching social science concepts to computing students (see this talk from Greg Wilson), and accountability for AI decisions, why even more advanced models continue to get common problems wrong (e.g., "how many r's are in 'strawberry'" and "who is the pope") and implications for society.

How Progress Ends

In "How Progress Ends", the author provided insights how technologies progress and stagnate---noting many historical examples. We also briefly discussed the students' reading for this week, and how AI could potentially stagnate progress. In addition, I noticed several concepts that could apply to recent literature on AI in software engineering. Two potential issues came to mind.

Social Disruption

The importance of social networks for innovation is no mystery...one of our most important resources is our social network, which acts as our "collective brain." And when networked people are free to explore, they test more technological pathways. [p. 5, Frey2025]

Historical Example 🍻 At the time, many economists perceived Prohibition in the United States would increase productivity and efficiency of workers. However, one negative implication of banning alcohol was "the disappearance of the saloon, [which] disrupted ordinary people’s daily lives and social networks...Skilled workers and craftsmen went there not just to drink but to socialize and exchange ideas. And because these workers were responsible for developing the most inventive contrivances of the era, it should be no surprise that innovation took a hit as saloons across the country were forcibly shut down." [p. 3, Frey2025]. For instance, after Prohibition in the U.S. there was an 18% decline in patents. Frey suggests social networks are critical for innovation, referencing the cofounders of Google and the discovery of mRNA as examples. While Prohibition improved static efficiency, or individuals' operational performance, dynamic efficiency, new ideas and collaborations that lead to economic growth and technological progress, was stagnated.

Relevance to SE 💻 Recently, programmers are integrating AI within software development-related tasks, increasingly replacing human peers in tasks such as tech interview preparation, code reviews, and pair programming. However, this could prevent innovation by limiting the exchange of ideas and transfer of knowledge between individuals. For example, prior work shows humans are the most effective way developers learn about new tools [MurphyHill2011]. However, a recent study also shows that humans pair programming with AI are less likely to engage in knowledge transfer compared to humans, and also lack access to insights derived from tacit knowledge, organization- and team-specific contexts, and domain expertise [Welter2025]. Thus, while generative AI and LLMs can enhance static efficiency, their impacts on dynamic efficiency is still an area of investigation.

Diversity of Thought

The rise of the Chinese state surely encouraged scale, but it came at the price of diversity of thought. [p. 36, Frey2025]

Historical Example 🇨🇳 China was one of the world's early leaders in global innovation and technological progress in the 14th century, yet was rapidly surpassed by Europe in the 15th century. Frey states "The question of why China, despite its early technological superiority, did not produce an industrial revolution, and even ended up being overtaken by the West...[is] one of the greatest paradoxes of Chinese development" [p. 36, Frey2025]. One inhibitor was the increase in lack of diverse thinking, introduced by the Qin and Song dynasties and the Confucian education system. During this time, the selection process for bureaucratic positions was only open to males who could travel to Beijing for the metropolitan exam and pass the examination (95% failed). Meanwhile, individuals were motivated to pursue careers in government. Or, "if Galileo had lived in China, he would have been a bureaucrat and not a scientist" [p. 28, Frey2025]. Simultaneously, "under this system, China did well as long as technological progress was incremental and served the state. Yet this also meant that the Middle Kingdom would have no scientific revolution" [p. 38, Frey2025]. This bureaucratization reinforced a hierarchical and patrimonial structure, discouraging many talented individuals from becoming scientists, inventors, and entrepreneurs.

Relevance to SE 💻 There are a few ways diverse thinking relates to AI in software development today. First, diverse thinking has shown to motivate innovation in technology. For instance, Frey notes recently how "e startups like OpenAI and Anthropic are now challenging Meta and Google" [p. 16, Frey2025]. However, the over-reliance on a few dominant AI models or tools can limit creative approaches to problem-solving and may also inhibit future performance of models, leading to the collapse of generative AI. In addition, bias against individuals from underrepresented backgrounds can inhibit diversity of thought and progress. Diversity in SE teams is critical for success and innovation in software development [Hyrynsalmi2025]. Yet, diverse thinking is under-prioritized in the tech industry. For example, current hiring processes incur bias against individuals from underrepresented backgrounds, prioritizing those with more time and effort to prepare for interviews [Behroozi2019]. AI-based filtering can also exhibit bias against a large population of candidates without standardized credentials (i.e., elite degrees, etc.) and from underrepresented backgrounds [Armstrong2024]. The above example suggests when systems reward only one kind of thinking, it can optimize for incremental improvement rather than innovation and progress.

These thoughts were my own and based on a brief reading of the first three chapters of this book (mostly Chapter 1). If I ever get to finish it, I hope to share more thoughts on progression and stagnation in software development and research, and discuss the role SE research could potentially play in preventing the end of progress.

Footnotes and References

[Vaishampayan2025] Swanand Vaishampayan, Hunter Leary, Yoseph Berhanu Alebachew, Louis Hickman, Brent Stevenor, Fletcher Wimbush, Weston Beck, Chris Brown. "Human and LLM-Based Resume Matching: An Observational Study". Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL 2025).

[Wang2025] Tianjia Wang, Matthew Trimble, Chris Brown. "DevCoach: Supporting Students Learning the Software Development Life Cycle with a Generative AI powered Multi-Agent System". Foundations of Software Engineering, Software Engineering Education Track (FSE-SEET 2025).

[Frey2025] Carl Benedikt Frey." How Progress Ends: Technology, Innovation, and the Fate of Nations". Princeton University Press, 2025.

[MurphyHill2011] Emerson Murphy-Hill and Gail C. Murphy. "Peer interaction effectively, yet infrequently, enables programmers to discover new tools". Proceedings of the ACM 2011 conference on Computer supported cooperative work. 2011.

[Welter2025] Alisa Welter, Niklas Schneider, Tobias Dick, Kallistos Weis, Christof Tinnes, Marvin Wyrich, Sven Apel. An Empirical Study of Knowledge Transfer in AI Pair Programming.

[Hyrynsalmi2025] Sonja M Hyrynsalmi, Sebastian Baltes, Chris Brown, Rafael Prikladnicki, Gema Rodriguez-Perez, Alexander Serebrenik, Jocelyn Simmonds, Bianca Trinkenreich, Yi Wang, and Grischa Liebel. "Making Software Development More Diverse and Inclusive: Key Themes, Challenges, and Future Directions." ACM Transactions on Software Engineering and Methodology 34, no. 5 (2025): 1-23.

[Behroozi2019] Mahnaz Behroozi, Chris Parnin, and Titus Barik. "Hiring is broken: What do developers say about technical interviews?". In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 1-9. IEEE, 2019.

[Armstrong2024] Lena Armstrong, Abbey Liu, Stephen MacNeil, and Danaë Metaxa. "The silicon ceiling: Auditing gpt’s race and gender biases in hiring." In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pp. 1-18. 2024.

Rural Software Developers

Chris Brown — Tue, 14 Oct 2025 16:46:19 +0000

We are researchers at Virginia Tech conducting a study to understand the experiences and perspectives of software professionals living and working in rural environments. The study involves a ~15 minute online survey, open to anyone in the United States who is currently working or has past experience as a software professional. Participants will be entered into a raffle to win one of ten $25 online gift cards.

🔗 Survey link

If you know others who may qualify, we would greatly appreciate it if you share this with your professional network. Thank you for your time!

VL/HCC 2025 Overview

Chris Brown — Mon, 13 Oct 2025 00:00:00 +0000

We had numerous contributions accepted and presented at the AIEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2025) conference. This post summarizes the workshop and main conference contributions from our research group this year. A summary of all the contributions from Virginia Tech are highlighted in this new article from the Center for Human-Computer Interaction at VT (CHCI@VT).

Workshop: Designing for Everyone

Do LLM-Generated Resumes Make Me More Qualified? An Observational Study of LLMs For Resume Generation and Matching Tasks

Swanand Vaishampayan, Chris Brown

🔍 Problem: Large language models (LLMs) are increasingly used to support resume-related hiring tasks for employers and candidates---such as automatically matching resumes to job description requirements and generating or tailoring resume content. However, LLMs can integrate challenges in hiring contexts, such as biased rankings or resume whitening.
🧪 Study: We conducted in observational study to explore the performance of LLMs---GPT-4, Gemini, and Claude---on resume matching and resume generating tasks. For our preliminary study, we leveraged a dataset of real-world resumes from disabled (n = 209) and non-disabled (n = 209) candidates across domains to investigate potential bias.
📊 Findings: We observed moderate alignment across LLMs for resume matching tasks. We found traces of resume whitening (i.e., removing of disability markers) for resume generating tasks. Finally, we found instances of models exhibiting self-preference---or significantly preferring LLM-generated resumes in ranking tasks compared to human-generated resumes.
💡 Implications: We provide design guidelines for enhancing the design of tools for supporting resume matching and resume generation tasks to improve hiring pipelines for candidates and employers.

Understanding User Perceptions of Automated Dark Pattern Detection Online

Ryan Wood, Chris Brown Presented by: Huayu Liang

🔍 Problem: Dark patterns are deceptive design in online user interfaces (UIs) designed to trick users into adopting unintended behaviors. Recently, research is increasingly investigating automated tools to detect dark patterns in UIs---yet we lack knowledge on how automated dark pattern detection techniques impact user experiences online.
🧪 Study: We designed a technology probe, Dark Pattern Detector, a Google Chrome plugin to automatically detect select dark patterns online. We conducted a user study (n = 40) to explore the effectiveness and users' perceptions of our automated dark pattern detection tool. The study consisted of three tasks based on real-world examples of previously defined dark patterns (Disguised Ads, Hidden Costs, and Sneak Into Basket).
📊 Findings: We found our system was mostly effective for notifying users of dark patterns during the study tasks. Most participants also had positive perceptions of Dark Pattern Detector, praising its usefulness and ability to raise awareness about deceptive designs.
💡 Implications: We discuss implications for designing future automated dark pattern detection tools to safeguard users from deceptive designs online.

ParticipantGuide: Promoting Transparency in Human-Centric User Studies

Minhyuk Ko, Shawal Khalid, Chris Brown

🔍 Problem: Human subjects research is critical for advancing human-computer interaction, however recruiting participants for studies is very challenging and burdensome.
💡 Idea: Our extended abstract proposes ParticipantGuide, a method to inform potential subjects of study details to inform decisions about participation in research studies. We will explore this and additional methods to enhance participant recruitment processes in future work.

Graduate Consortium

AI-Guided Exploration of Large-Scale Codebases

Yoseph Berhanu Alebachew

🔍 Problem: Modern software systems are large, complex, and constantly evolving, creating challenges for developers to understand and contribute to codebases.
💡 Idea: Yoseph will devise techniques to support LLM question-answering and interactive visualizations to support program comprehension.

Main Conference

Programmers Without Borders: Bridging Cultures in Computer Science Study Abroad Program

Minhyuk Ko, Mohammed Seyam, Chris Brown

🔍 Problem: Globalization and distributed development is common in modern software engineering practices, yet Computer Science students rarely have the opportunity to participate in study abroad programs offered by universities.
🧪 Study: We interviewed students participating in the first iteration (Spring 2026) of a novel SE-focused study abroad program at Virginia Tech to understand students' experiences and impacts on their computing identity.
📊 Findings: Overall, students found the experience rewarding---consisting of opportunities to experience working with individuals from different cultures and overcoming communication challenges. In addition, we found the program increased students' confidence in participating in a program and getting a full-time computing job after graduation.
💡 Implication: We suggest ways to improve future study abroad programs for software engineering, and discuss takeaways for students and instructors based on our findings.

AutoPrint: Judging the Effectiveness of An Automatic Print Statement Debugging Tool

Minhyuk Ko, Omer Ahmed, Yoseph Berhanu Alebachew, Chris Brown

🔍 Problem: Debugging is one of the most difficult tasks in software development, yet programmers primarily rely on print-statement debugging instead of advanced debugging tools.
🧪 Study: We conducted a judgment study where participants (n = 23) from academic and industry backgrounds were tasked with using AutoPrint, a system we implemented to automatically insert and delete print statements for debugging, in their normal programing activities. After approximately two weeks, we surveyed and interviewed participants to understand their perceptions of automated print statement debugging.
📊 Findings: Participants used AutoPrint for diverse use cases, including working on school projects, personal projects, and web applications. Overall, we found participants viewed the tool fast, simple to use, and more effective than advanced dbugging tools---providing suggestions for UI improvements, AI integration, and automated bug fixing capabilities.
💡 Implication: While we found AutoPrint was viewed as effective, should we encourage tooling to support automated print statement debugging? We discuss future research directions o further evaluate automated print statement debugging and enhance the usability of more advanced debugging systems.

Understanding User and Developer Perceptions of and Responses to Dark Patterns in Online UIs

Huayu Liang, Syeda Afia Hossain, Chris Brown

🔍 Problem: Dark patterns are increasingly integrated into UIs, however we lack knowledge on software developers'---those responsible for implementing software applications---perspectives on dark patterns.
🧪 Study: We conducted two surveys to understand dark pattern perceptions of users (n = 66) and developers (n = 38). We also conducted a repository mining study to investigate themes and sentiments of discussions on GitHub related to dark patterns.
📊 Findings: Users have negative perceptions of dark patterns, but are likely to ignore them due to their ubiquity in online UIs. Developers report rarely implementing dark patterns, but believe they have negative effects on users and support development standards to prevent deceptive designs. On GitHub, discussions about dark patterns primarily focus on their usage, prevention, and examples in software---with comments from developers consisting of significantly more negative sentiments and emotions than non-dark pattern comments.
💡 Implication: We discuss implications for users, developers, and researchers to prevent and mitigate deceptive UI designs in software.

Designing Conversational AI to Support Think-Aloud Practice in Technical Interview Preparation for CS Students

Taufiq Daryanto, Sophia Stil, Xiaohan Ding, Daniel Manesh, Sang Won Lee, Tim Lee, Stephanie Lunn, Sarah Rodriguez, Chris Brown, Eugenia Rho

🔍 Problem: Technical interviews are commonly used to assess software engineers for hiring, yet candidates rarely have opportunities for structured preparation---in particular for think aloud practice to verbalize problem-solving approaches while completing programming tasks.
🧪 Study: We designed a technology probe to simulate technical interviews using conversational AI to support think aloud practice and feedback. We evaluated this preliminary system through a user study (n = 17).
📊 Findings: Participants found the conversational AI tech interview simulation ot be engaging and realistic to an actual interview and appreciated the feedback beyond verbal content analysis (i.e., feedback on pauses, filler words, etc.). Participants also perceived the system helped them learn and improve their performance for future interviews.
💡 Implication: We provide design implications for AI-assisted systems and AI-generated feedback to support technical interview preparation. We also discuss the role of AI and human-AI collaboration for technical interview preparation.

VL/HCC 2025 was a great experience, and it was very meaningful to be back in Raleigh and on campus at NC State---which is where I completed my PhD 🐺

I also served as Workshop Co-Chair, Graduate Consortium Co-Chair, Session Chair, and the program committee for VL/HCC 2025, which were rewarding experiences. We received great feedback on these efforts from the research community, and looking forward to pursuing future work based on these papers. If you would like additional information or are interested in potential collaborations related to any of these projects, please feel free to reach out.

ESEM 2025 Overview

Chris Brown — Mon, 13 Oct 2025 00:00:00 +0000

Our paper "Exploring the Evidence-Based SE Beliefs of Generative AI Tools" was accepted and recently presented at the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2025) Emerging Results and Vision track. This post summarizes the problem we explored, provides an overview of our emerging results, and and future vision of AI-assisted software engineering. A preprint is available here.

Exploring the Evidence-Based SE Beliefs of Generative AI Tools [ESEM ERV]

Chris Brown and Jason Cusati

🔍 Problem: Generative AI tools are increasingly used to support software development, yet little is known about the "beliefs" of AI tools with regard to evidence-based practices supported by empirical SE research.
🧪 Study: We conducted a conceptual replication study, prompting generative AI tools---ChatGPT, GitHub Copilot, Gemini, Blackbox AI, and Claude---to respond to questions regarding 17 evidence-based claims used in prior work to assess the beliefs of human software developers [Devanbu2016].
📊 Emerging Results: We found generative AI tools typically have positive beliefs aligning with research claims, however can have ambiguous responses. Further, most tools lack credible evidence to support their claims.
💡 Vision: We provide implications for software developers leveraging generative AI tools in practice. We also discuss future research directions to promote:

Evidence-based generative AI for SE;
Evidence-based AI integration within development workflows; and
Generative AI-based techniques for translating research evidence for practitioners.

This was my first time attending ESEM, and I really enjoyed presenting this work and connecting with members of this community. Visiting Honolulu was also an amazing experience 🏖️ We hope this work provides useful insights and motivates future work to enhance evidence-based practices in generative AI-driven software development.

References

[Devanbu2016] P. Devanbu, T. Zimmermann, and C. Bird, “Belief & evidence in empirical software engineering,” in International conference on software engineering, pp. 108–119, 2016.

FSE 2025 Preview 2: Workshops

Chris Brown — Wed, 25 Jun 2025 00:00:00 +0000

Foundations of Software Engineering was a great opportunity to connect with and learn from software engineering researchers. In addition to the education and tool demo tracks of the conference, we have several contributions to co-located workshops at FSE 2025 that will be presented later this week!

Towards Evidence-Based Tech Hiring Pipelines [SE 2030]

Chris Brown, Swanand Vaishampayan

🔍 Problem: Software engineers are primarily hiring through a tech hiring pipeline, consisting of processes such as resume matching and technical interviews. However, we lack insights on how to effectively evaluate the capabilities of programmers.
📊 Findings: We offer a roadmap to promote evidence-based hiring in the tech industry grounded in insights from evidence-based decision making---discuss challenging and future research directions to incorporate contextual, experiential, and research-based evidence into tech hiring pipelines.
💡 Implication: We provide implications for improving tech hiring processes, enhancing SE research on hiring, preventing toxicity and promoting diversity and inclusion in hiring processes, and promoting the adoption of evidence-based practices on software engineering in practice.

The First International Workshop on Large Language Model-Oriented Empirical Research (LLanMER) [FSE WS]

Na Meng, Xiaoyin Wang, Chris Brown (Workshop Organizers)

🔍 Problem: There is fast growing interest in the application of large language models (LLMs) in software practice and research. However, lots of questions remain regarding responsible, reliable, and reproducible empirical research with LLMs.
💡 Workshop: Our workshop focuses on methodologies for conducting empirical research with LLMs. We aim to provide a venue to share ideas, discuss obstacles and challenges, brainstorm solutions, and establish collaborations in LLM-oriented empirical research.

Note

If you were attending FSE in Trondheim 🇳🇴, please join us for our workshop on Friday from 2:00-5:30pm in the Sirius room!

From Prompts to Properties: Rethinking LLM Code Generation with Property-Based Testing [LLanMER]

Dibyendu Brinto Bose

🔍 Problem: Large language models are increasingly used for code generation, however evaluating generated code is challenging and unit testing-based techniques are insufficient.
🧪 Study: We propose Property-Based Testing (PBT) as an alternative strategy, and conduct a preliminary evaluation using two code generation models (StarCoder and CodeLlama) on two datasets (MBPP and HumanEval).
📊 Findings: Our results show unit test-based evaluations provide insights on correctness, while PBT is capable of evaluating against additional logical and structural constraints to assess in generated code.
💡 Implication: We suggest that hybrid approaches combining unit and property-based testing can provide a more reliable evaluation for LLM-generated code.

The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants [HumanAISE]

Ebtesam Al Haque, Chris Brown, Thomas D. LaToza, Brittany Johnson

🔍 Problem: Information seeking constitutes a considerable part of software development---however, we lack insights on the impact of AI-assisted tools on software engineers' information needs and information-seeking behaviors.
🧪 Study: We conducted a mixed methods study, surveying and interviewing software practitioners to understand how developers use AI tools for information seeking and the impact of AI tools on productivity and skill development.
📊 Findings: Most participants reported using AI tools to support information seeking. We also observed challenges with AI-assisted information seeking, mixed perceptions on perceived productivity, and positive impacts expanding developer knowledge and skills.
💡 Implication: We discuss the evolution of information seeking behaviors in the age of AI, outlining productivity and learning trade-offs in addition to motivating future directions for AI-assisted developer tools.

Benchmark Infrastructure for LLMs for Code (BI4LLMC)

I was also invited to participate in the International Workshop on Benchmark Infrastructure for LLMs for Code, aiming to devise solutions to address the growing need for effective and rigorous evaluations of LLMs for coding-related tasks.

We hope the discussions at these workshops will spark new ideas, build community, and create new questions for us to explore within the evolving landscape of software engineering and SE research.

FSE 2025 Preview

Chris Brown — Fri, 20 Jun 2025 00:00:00 +0000

The ACM International Conference on the Foundations of Software Engineering (FSE) is one of the leading venues for software engineering-related research---targeting researchers, practitioners, and stakeholders. We're excited to share that several of our recent papers were accepted and will be presented at the main track of the conference this year (a follow-up post will highlight the workshop papers). In this post, we briefly summarize the problems explored, provide an overview of our research findings, and discuss implications from three papers that will appear in the FSE main track:

How do Software Engineering Candidates Prepare for Technical Interviews? [FSE SEET]

Brian Bell, Teresa Thomas, Sang Won Lee, Chris Brown

🔍 Problem: Current tech hiring interview processes are challenging, difficult to prepare for, and rarely faced by candidates in computing curriculum and education.
🧪 Study: We surveyed 131 candidates actively pursuing SE-related roles to understand how they prepare for technical interviews.
📊 Findings: Candidates use varied tools to prepare (mostly YouTube and LeetCode), but rarely train in authentic settings (with coding, communication, and an audience) and lack support in education---leading to increased anxiety and unpreparedness. However, candidates who practice with others (i.e., mock interviews) feel significantly more prepared than those who train alone.
💡 Implication: We provide implications for how candidates, employers, tech interview preparation resources, and CS education can enhance students' preparedness for technical interviews---a process necessary to obtain employment in the tech industry.

DevCoach: Supporting Students Learning the Software Development Life Cycle with a Generative AI powered Multi-Agent System [FSE SEET]

Tianjia Wang, Matthew Trimble, Chris Brown

🔍 Problem: The software development lifecycle (SDLC) is critical for developing and maintaining software systems, yet challenging to effectively incorporate within learning environments. Recent work has explored leveraging generative AI to enhance SE education, but how effectively can it support learning and practical experiences for students learning SDLC concepts?
🧪 Study: We implemented DevCoach, a multi-agent system consisting of AI-powered personas representing roles in SE teams to help students learn and complete tasks related to the SDLC (e.g., requirements, design, implementation, testing, and review). We evaluated our tool with a user study with 20 students across DevCoach and baseline study settings.
📊 Findings: Our results show DevCoach demonstrated the ability to enhance student learning gains, enhanced students' completion of SDLC activities, and improved perceived aspects of learning environments grounded in the Community of Inquiry framework (social, cognitive, and teaching presence) [Garrison1999].
💡 Implications: We discuss the potential of generative AI and multi-agent systems in advancing SE education to offer personalized and collaborative experiences to help learners receive training on other software development concepts.

AutoPyDep: A Recommendation System for Python Dependency Management Utilizing Graph-Based Analytics [FSE Tool Demo]

Dibyendu Brinto Bose, Travis Chan, Matthew Trimble, Chris Brown

🔍 Problem: Managing software dependencies is increasingly complex in modern software development---particularly in popular programming languages such as Python [Neils2024], causing frustration for programmers across varied domains.
🧪 Study: We implemented AutoPyDep, a graph-based recommendation system to support Python dependency management through various features such as centrality analysis of dependencies, visualizations for library interdependencies, and predicting the nature and timing of releases based on historical data. We conducted a preliminary evaluation to assess the prediction capabilities of our tool.
📊 Findings: Using a collected dataset of Python release notes, we found AutoPyDep demonstrated accurate capabilities for release type (New Features (F1 = 0.95), Bug Fixes (F1 = 0.94), Security (F1 = 0.92), and Performance (F1 = 0.89)) and date (mean error ~ 2 months) predictions.
💡 Implication: We briefly discuss implications and future work to enhance AutoPyDep and future dependency management systems.

We look forward to sharing these efforts and engaging with the research community at FSE 2025. We hope these papers---reflecting ongoing threads in our overall research---provide useful insights, and we welcome discussions, critiques, and collaborations that can extend them further. Looking forward to attending the conference and hope to see you in Trondheim, Norway next week! Also stay tuned for an upcoming post on our contributions to several of the FSE workshops.

References

[Garrison1999] Garrison, D. Randy, Terry Anderson, and Walter Archer. "Critical inquiry in a text-based environment: Computer conferencing in higher education." The internet and higher education 2.2-3 (1999): 87-105.

[Neils2024] Niels Cautaerts. "Python dependency management is a dumpster fire". 2024. https://nielscautaerts.xyz/python-dependency-management-is-a-dumpster-fire.html

We Hosted a Workshop for Practitioners---No One Came

Chris Brown — Thu, 05 Jun 2025 00:00:00 +0000

As part of a recent grant, we were required to host a topical workshop to share ideas and promote collaboration among stakeholders. I decided to design our workshop primarily for software practitioners (individuals who apply software engineering concepts in real-world industry settings). As one of our research group goals is to explore how we might bridge the gap between academia and industry---this seemed like the perfect opportunity.

The Workshop

The workshop was organized to be an interactive “working group” on topics related to our grant testing and securing user interfaces in CI/CD pipelines. The event featured research presentations share and collect feedback on preliminary research findings, interactive discussions to uncover challenges in practice, and brainstorming sessions to motivate new solutions and researh directions. We espectially hoped to target practitioners for our workshop to gain industry perspectives and inspire future work with practical implications and relevance.

We started planning the workshop activities in late March (~7 weeks before the event) and finalized the details to share in mid-April (~3.5 weeks before). 20 people registered to attend through our online form (14 in-person and 6 virtual, mostly from VT but some external). We booked a conference room on campus and two graduate students put in a lot of work to prepare presentation slides and activities related to their research from this project.

Yet, when the time for the workshop arrived--- no one showed up. Actually two people came---another graduate student from our research group and a CS faculty member working on related research---and I am extremely grateful for their attendance. However, this was way less than the expected attendance and not a single software practitioner (the intended audience) joined our workshop.

We still had a great time---and received valuable insights and feedback on our work. However, I was very dissapointed we were unable to attract a broader audience. Here is an overview of what we tried to attract practitioners to our workshop, and a brief reflection on what to do differently next time.

Attracting Software Practitioners

	How We Tried to Appeal to Software Practitioners
🧑‍💻	Focused on (what we perceived to be) industry-relevant topics: UI testing 🧪; UI security 🔒; CI/CD pipelines 🔁
🎙️	Invited a world-class speaker with extensive industry experience, who provided a very interesting and highly relevant virtual keynote despite the low turnout
📆	Hosted the event on Saturday to avoid potential conflicts due to work schedules
🗣️	Publicized the workshop to our local developer group Slack channel and LinkedIn
💻	Made the event hybrid to support in-person and virtual attendance
🍕	Provided free food (lunch and refreshments)

Reflection

There are some clear issues that might have prevented attendance---a 4-hour workshop on a Saturday might be too long of a time commitment, hosting it on campus may have deterred non-academic attendees, providing compensation would have encouraged participation, etc. Here are some more takeaways on what we could do better next time.: 📖 Research-to-Practice Relevance: I assumed our research topic (UI testing and security in CI/CD) would be relevant to software developers---and our preliminary findings from this work show it is a challenge for practitioners in open-source development [Gan2025]. However, it may not be a topic developers are actually interested in, or did not resonate with the population we reached out too. Reaching out to the target audience ahead of time to see what issues/topics they care about, then organizing our outreach based on that may be more effective. In addition, better communication regarding the direct relevance and value of the topic/our work could help.; 📢 Broader Promotion: To promote the workshop, I shared messages on the local developer Slack channel (n = 2) and LinkedIn (n = 1), in addition to emails to relevant listservs at Virginia Tech. However, more broad and more frequent publicizing may have helped. I am also personally off most social media now, with the exception of LinkedIn. Yet, it may be beneficial to re-join social media and explore other online platforms to promote our work and engage with development-related communities. Suggestions are appreciated!; 🤝 Deeper Relationships: Finally and most importantly, deeper connections with the target audience could enhance future efforts. This was not my first time messaging this particular group, and I have met and interacted with several members of this community before. However, establishing personal connections and more in-depth relationships can encourage engagement and motivate participation. For instance, attending events and meetup groups hosted by this organization to engage with developers through face-to-face interactions. This will also help gain insights on what our local developer community cares about, motivating future research studies and enhancing the overall relevance of our work.

We will regroup and eventually try again---planning to incorporate some of these items as we potentially explore other ways to showcase our work and help bridge the research-practice gap. Any thoguhts, tips, or suggestions on how to better engage with practitioners as researchers are welcome.

References

[Gan2025]: X. Gan, H. Liang, C. Brown. "Challenges, Strategies, and Impacts: A Qualitative Study on UI Testing in CI/CD Processes from GitHub Developers' Perspectives". International Conference on Software Testing, Verification and Validation (ICST 2025).

Hello World

Chris Brown — Sun, 25 May 2025 00:00:00 +0000

👋 🌍

Welcome to our blog! This is the first post, more will be coming soon 👀...

About Us: We are the Code World, No Blanket lab---a software engineering research group in the Department of Computer Science at Virginia Tech 🦃. Our research uses empirical, interdisciplinary, and automated methods to improve the behavior, productivity, decision-making of software engineers---we study software development and software developers.

More details about us and our research are available here. This group is led by Dr. Chris Brown.

Which Recommendation Would You Prefer?

Chris Brown — Wed, 22 Jan 2020 21:00:30 +0000

Pretend you received these on a pull request you submitted to a GitHub project. Which one would you prefer and why?

Recommendation A

Recommendation B