The psychometric analysis of the results of the Practicum Script Concordance Test is carried out in various stages.
In the first stage, the analysis is done based on the results of the reference experts when the test is being done and then the study is complemented with the results of the professionals who take part in the medical training activities.
We use the following methods for the psychometric analysis of the results of both the validating experts’ results and the participants’ results:
This type of analysis enables a first evaluation of the quality of the questions. It is based on the total score on the test, on the results on each of the clinical cases, and on the answers to each of the items (questions) in each clinical case.
At this stage, the aim is to establish the normality of the distributions of the scores for each of these three levels (total test group, clinical case, and items) to verify that they correspond to acceptable profiles for a good assessment test.
The statistical indices calculated include:
Exploratory and confirmatory factor analyses to verify the stability of the factors or dimensions that make up the Practicum Script Concordance Test, taking into account that each training activity (complete test) is composed of clinical cases in different subject areas within the specialty and includes different types of medical decisions (about diagnoses, complementary studies, and treatment).
Comparative analyses among different groups of validating experts, in the cases where a particular test group had been validated by reference panels from different countries, with the aim of stabilizing the reference group or subgroup for the participants in each training activity. The methods used for this type of study are:
Analyses of items, through reliability studies to measure the overall homogeneity of the questions (items) within the test group, also analyzing each question’s capacity for discriminating among participants in function of their responses.
Generalizability studies, first with a crossed-facets design and then with facets with nested questions, which make it possible to calculate the proportion of each element in the test to the overall score observed. These analyses also make it possible to determine the impact of the subgroups analyzed on the overall scores and in particular the variances among different groups of participants, as well as the impact of the order of the cases on the results observed.
N.B. Except when calculating the overall scores, for the study of the coefficients, the grouped results of the items corresponding to each case are used to avoid signal amplification bias in the intercorrelations owing to the self-correlation among items within the same case.
Prof. Dr. Carlos A. Brailovsky Kurlat
International Consultant for the Practicum Script proyect. Professor emeritus at Laval University.