Course Management Systems (CMS) are an increasingly important part of academic systems in higher education. When choosing a Course Management System for an educational institution, the usability of the system is the key to the effectiveness and efficiency of the online courses that are to be implemented. The goal of this paper is to report the results of a comparative usability study conducted in 2008- 2009 on two different course management systems: BlackBoard and Moodle. 135 students enrolled in the Fall 2008 and Spring 2009 sections of Introduction to Educational Technology participated in the study (72 and 63 respectively). At the beginning of each semester, participants were randomly divided into two groups to experience different CMSs at different times. It can be concluded from this study that in almost every module or function comparison that was made, Moodle was favored by course participants over Blackboard with the exception of the Discussion Board module where scores were not significantly different. At the end of the study, the researchers concluded that use of Moodle in online courses can be a suitable alternative to the current CMS system (BlackBoard). In fact, now that the pilot has showed that Moodle is as effective as BlackBoard, the researchers have already shared their experiences with other faculty members and expanded their investigations by involving numerous other online courses, instructors, and students, because the product showed significant potential for further examination. This study adds to the growing body of studies that are carried out to see if an open source CMS (Moodle) warrants consideration as an alternative to the institution's current course management system. In addition to comparing the students' feedback quantitatively, this study also tried to explain in detail what specific component /function of each CMS students found useful or better than in the other. Rather than focusing only on student satisfaction scores, this study further investigated what aspect of each module for each CMS course participants particularly liked or disliked.
