Fulltext:
Volume & Issue:
Article Type :
Abstract:
Purpose
Qualitative research is a method of inquiry employed in the qualitative evaluation. The quality of qualitative research was shifted from emphasizing on reliability and validity based on neutral facts to trustworthiness of inferences based on the consensus of qualitative researchers and participants. Taiwanese higher education adopts accreditation approach for educational quality assurance. Having experts with deeply understanding of the field being evaluated, peer review is able to make fair judgment and comments in external evaluation to assure the higher education quality. Although the
accreditation approach applies the conceptions of qualitative research methods, such as triangulation for data collection during an evaluation procedure, the quality of qualitative evaluation has been challenged. It has been noticed that reviewers often conducted evaluations in a situated context and inevitably brought their own socio-cultural history and beliefs into the evaluation process. In order to remedy this problem, an objection and appeal system has been developed in Taiwanese higher education evaluation. The institution being evaluated can file an objection if they considered that the
evaluation result is inconsistent with the facts. This study aims to investigate the major issues emerging from the objection reports and the corresponding replies, as well as to examine the gap between the educational quality of peer reviewers and institutional staff based on the objection reports. It provides an exploration of challenges of qualitative evaluation of higher education.
Design/methodology/approach
Applying content analysis methods, this study analyzed the objection reports of 1,156 programs during the 1st cycle of the program accreditation from 2006 to 2010.
Findings
Results showed that the institutions being evaluated tried to alter the conclusion of the on-site-visit reports by questioning the credibility of on-site visit report and providing more evidence (68.4%), weakening the justification of the review (13.4%), and asking reviewers to consider university context (12.0%). However, reviewers often repeated the same conclusion of the evaluation in the replies (48.4%), illustrated their considerations for data incredibility and inconsistency (31.0%), and provided detailed explanation for the evaluation reasoning (12.7%). Cross analysis of the opinions between university staff and reviewers revealed that the reviewers applied similar strategies to reply the objections of institutional staff. Four threats to credibility and consistency exist: 1. The university context was not fully
considered; 2. The on-site visit examined only the isolated phenomena but not dynamic processes; 3. The evaluation judgments could be influenced by personal values; 4. The conclusion was not drawn based on long- term observations and interactions.
Originality/value
To conduct best practice of qualitative evaluation of higher education, institutional staff and external reviewers are required to have appropriate evaluation knowledge and skills. Making reflections from the suggestions of evaluation reports, institution can find out its weakness for institutional development, and make improvements for better educational quality. Through the inquiry of this study, the challenges of qualitative evaluation can be better understood, enabling future improvement of the evaluation process and making better educational policy of higher education.
Author:
Keywords:
APA:
Chen, K. H.-J. (2016). Challenges of qualitative evaluation: Different views of reviewers and university staff in the program accreditation of higher education. Contemporary Educational Research Quarterly, 24(2), 75-109.
Full Text:
