Applying AHP for Collaborative Modeling Evaluation: Experiences from a Modeling Experiment

Abstract

Collaborative modeling is one of the approaches used to enhance productivity in many enterprise modeling and system development projects. Determining the success of such a collaborative effort needs an evaluation of a number of factors which affect the quality of not only the end-products – the models, but also that of other modeling artifacts: the modeling language, the modeling procedure and the support tool. Although a number of quality frameworks have been developed, few of these frameworks have received practical validation and many offer little guidance about how the evaluation is operationalized. The Collaborative Modeling Evaluation (COME) framework presented in this paper offers a holistic approach to the evaluation of the four modeling artifacts. It employs the Analytic Hierarchy Process (AHP), a well-established method from Operations Research, to score the artifacts’ quality dimensions and to aggregate the modelers’ priorities and preferences. Results from a modeling experiment demonstrate both the theoretical and practical significance of the framework.