Validity and reliability with educational testing in the pharmacy and medical literature

Purpose:  Validity and reliability are crucial to educational testing. This investigation characterized these psychometric properties in pharmacy education journals, and compared these with medical education journals.

Methods:  Among medical and pharmacy education journals, 2009–2011 tables of content were reviewed and articles selected that used educational testing for students. Two reviewers independently assessed for presence of reliability (internal consistency, inter-rater) and validity (content, construct, criterion) descriptions. As well, we reviewed for authors’ awareness of psychometric properties in the discussion, and pilot testing use. 

Results:  Forty-seven articles (15 pharmacy and 32 medicine) met inclusion criteria. The two reviewers had an agreement of 0.803 (Cohen’s kappa), and reached consensus where disagreement. Our initial data suggested that content validity, criterion validity, construct validity, internal consistency and inter-rater reliability were not differently reported in the medical and pharmacy education literature [p=0.31, p=0.08, p=0.16, p=0.78 respectively; p=0.30 by chi-square]. We confirmed that medical education journals had higher 5-year journal impact factors (JIF) than pharmacy education journals [median 2.3 vs 1.3, p=0.0009 by Mann-Whitney], but on regression, neither JIF nor type (medical, pharmacy) differed in reporting validity and reliability [OR=0.463 for JIF (95%CI 0.10-2.09), OR=1.01 for type (95% CI 0.18-5.82)]. Of note, awareness by authors differed as well. More medical education authors discussed implications with psychometric aspects of educational testing than did pharmacy education authors [50% vs 13%, p=0.02 by chi-square]. We did not find any difference between journal types with use of pilot testing [13% medicine, 13% pharmacy; p=1.00 by chi-square]. 

Conclusion:  Our preliminary results suggest no statistically significant difference between medical and pharmacy literature with reporting validity and reliability in educational testing. Journal quality (by JIF) did not factor into reporting either. However, our review suggested that pharmacy education authors could demonstrate greater awareness of psychometric aspects of educational testing by improving article discussions.