The accurate interpretation of the results obtained from an instrument used to assess change is dependent on whether an invariant relationship exists for each construct being measured. This study examined the dimensionality and long-term stability of a group of items used in three administrations of a questionnaire investigating pre-service teachers' attitudes and beliefs about an initial teacher certification test. Factor analysis was used to analyse these data. Results of the exploratory factor analyses (EFA) reveal that the dimensionalities of these data for each administration were relatively different, with only two factors appearing repeatedly across administrations. In comparison, the results of the confirmatory factor analyses (CFA) suggest that none of the models derived from the EFA analyses provided an adequate fit to the data. The tests for invariance indicate that only five of the twenty-one items under investigation were invariant across administrations. Suggestions for ways to improve the questionnaire are included.