Workshop on Methodological Issues in Empirical Studies with Human Subjects

The theme of the proposed workshop is “Methodological Issues with Empirical Studies involving Human Subjects”. We aim at gain depth on the adoption of the empirical paradigm applied to SE. Specifically, identifying issues, discussing them, and finding solutions for the issues in currently employed empirical methods. This first workshop will have a focus on empirical studies with human subjects. Such focus could evolve in future editions of this workshop.

The general empirical process consists of six major, consecutive steps (Wohlin et. al): Definition, Design, Implementation, Execution, Analysis, and Reporting. Issues can arise from each step, e.g., while, defining the research goal (during definition), mitigating potential threats to validity (during design), developing the measurement instruments (during implementation), teaching the methods (during execution), preparing the data (during analysis), drawing conclusions (during reporting).

With the steep increase of empirical studies and the advent of secondary studies (aggregation or meta-study), one main concern still affects the quality of the primary studies (e.g., a single experiment). Concerns about different quality issues have been raised for many years; however, due to the complexity of empirical studies in our field – especially when involving human subjects – not all of them seem to be solved. In particular, problems with regard to the appropriate design of experiments and the correct usage of statistical tests are crucial for the trustworthiness of the results. Consequently, we need to better understand the issues and how to mitigate, if not eradicate them.

The workshop’s aim is to increase awareness on the importance of the quality and appropriateness of the methods applied during the different stages of empirical studies to yield reliable results, and to disseminate this knowledge to the broader SE community.

The methodological issues do not only threaten the validity of the studies but make it hard to use them for future work, such as aggregation (developing guidelines for practice).

We believe that: (i) improved understanding of challenges in conducting empirical studies, (ii) considering these challenges within the different phases of experiments, in conjunction with (iii) better reporting, would lead to reusable outcomes. Thus, in the proposed workshop, we are particularly interested in defining recommendations that can be taken up by other researchers to improve the quality of empirical studies in general. The following topics exemplify the envisaged theme:

·        Issues with regard to the motivation and definition of appropriate research questions.

·        Issues with regard to the operationalization, e.g., missing standard metrics.

·        Issues in the selection of appropriate experimental design and its consequences, e.g., on the analysis and conclusions

·        Issues in the systematic identification of threats to validity; and how to address them.

·        Dealing with perceptions and biases.

·        Drawing the appropriate conclusions from the findings

·        Issues with the generalization of findings

·        Understanding failures and successes: lessons learned.

·        Aggregating results from individual studies.

·        Issues with replication or families of studies

·        Quantitative versus qualitative approaches and combined approaches.

Important Dates:

Submission of papers:

February 05, 2018


March 05, 2018


March 19, 2018


May 29, 2018

Preliminary Program:

Workshop Keynote:

The keynote (confirmed) “Studying the Human Side of Software Engineering” will be given by Per Runeson from Lund University, Sweden.

Bio: Dr. Per Runeson is a professor of software engineering at Lund University, Sweden, head of the Department of Computer Science, and the leader of its Software Engineering Research Group (SERG) and the Industrial Excellence Center on Embedded Applications Software Engineering (EASE). His research interests include empirical research on software development and management methods, in particular for software testing and open innovation, and cross disciplinary topics on the digital society in collaboration with social science, humanities, law and business scholars. He has contributed significantly to software engineering research methodology by the books on case studies and experimentation in software engineering. He serves on the editorial boards of Empirical Software Engineering and Software Testing, Verification and Reliability, and is a member of several program committees.

Abstract: Software engineering is a socio-technical research subject, in which we study interplaying systems of humans and technology. This implies that it is hard to scope the studies and to isolate factors that really have effects on the phenomenon under study. What is related to the human, what depends on the technology, and what is the interplay? Still, researchers need to observe humans acting in some kind of context to understand and improve SE. Empirical studies in SE apply both qualitative and quantitative approaches, and each has its pros and cons, but perhaps not the ones we think of at a first thought. This talk focuses on contributions of qualitative and quantitative studies, and how we may synthesize SE knowledge from different types of studies. It further discusses challenges for researchers, primarily with a technical and engineering background, to address the human side of software engineering. Finally, it touches upon the possibly hardest aspect of the human side, namely communicating research findings to stakeholders, to help them make evidence-based decisions in SE.

Invited Speaker:

We have two confirmed invited speakers, Natalia Juristo from UPM and Dieter Rombach from University of Kaiserslautern who will give their perspectives, serving as impulses for discussion.

Session 1


Opening. Introduction of participants


Background: Summary of the current state


Keynote (45’ + 15’ Discussion)

Session 2


Invited talk, presentations, plenary discussions

Session 3


Invited talk, presentations, plenary discussions

Session 4


Discussion and Consolidation of findings


Wrap-up. Ideas for the future

Workshop Organizers:

·        Andreas Jedlitschka (main contact), Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany,
Phone: +49 63168002260

·        Laurie Williams, North Carolina State University, NC, US,

Workshops Advisory Board:

·        Natalia Juristo, UPM, Spain

·        Dieter Rombach, University of Kaiserslautern, Germany

·        Guilherme H. Travassos, COPPE, Brazil

·        Claes Wohlin, BTH, Sweden

Program Committee:

Names with an (*) are confirmed.

Abrahão Silvia, Uni. Politec. de Valencia, Spain

Avgeriou Paris, Univ. of Gronningen, The Netherlands*

Baldassarre Maria Teresa, Univ. of Bari, Italy*

Carver Jeffery, Univ. of Alabahma, US*

Daneva Maya, Univ. of Twente, The Netherlands*

Falessi Davide, California Polytechnic State Univ, US

Franch Xavier, Technical Univ. of Catalunya, Spain*

Genero Marcela, Univ. of Castilla-La Mancha, Spain

Heldal Rogardt, Chalmers Univ. of Tech., Sweden*

Izuireta Clem, Montana State Univ., US*

Jorgensen Magne, Simula Research Labs, Norway*

Kalinowski Marcos, PUC-Rio, Brazil*

Lassenius Caspar, Aalto Univ., Finland*

Männistö Tomi, Univ. of Helsinki, Finland*

Mendez Fernandez Daniel, Tech. Univ. Munich, Germany*

Pfahl Dietmar, Univ. of Tartu, Estonia*

Runeson Per, Lund Univ., Sweden

Russo Barbara, Free Univ. of Bolzano-Bozen, Italy*

Seaman Carolyn, Univ. of Maryland-Baltimore County, US*

Sjøberg Dag, Univ. of Oslo, Norway*

Vegas Sira, UP Madrid, Spain*

Wagner Stefan, Univ. of Stuttgart, Germany*

Last modified: Sunday, 15 October 2017, 8:08 PM