Measuring Software Methodology Usage: Challenges of Conceptualization and Operationalization

Most software engineering research implicitly assumes that development methodologies are useful and that there is a direct relationship between software methodologies and their effects on organizational performance. However, a methodology cannot have an impact if it is not used. The purpose of this paper is, thus, to raise a number of challenges related to the conceptualization and operationalization of methodology usage and to report on a study that compared subjective and objective operationalizations of usage. Results of regression analyses show that these operationalizations do not appear to be strongly related. While self-reported usage is related to self-reported measures of the independent variables of methodology acceptance in the study, the objective and computer-recorded measures show different and distinctly weaker links. There are several explanations to these seemingly contradictory results. Most importantly, the results of this study suggest a need for reconceptualization and better validation of methodology usage constructs in future, empirical software engineering research.