Avaliação de Estudantes

Angoff method to determining passing grades

Fonte: Beta Education Blog

Given the delays over Qualification Exam results, I thought I would provide some information regarding the Angoff Method that IMCA is using to determine the pass rate. (1)

Rationale Exams may have different levels of difficulty, due to the fact that questions are randomly pulled from a test bank. Fixed passing rates, of say 70%, may result in an exam not being "reliable". Reliability deals with the ability of a test to measure a test-taker's ability consistently.(3)(4)  As an example: for a scale to be reliable, it needs to consistently estimate a one-pound object as weighing one pound.  For an exam to be reliable, the scoring method needs to take into account the difficulty of each exams to be able to consistently evaluate a qualified candidate as a qualified candidate.  Major contributions in this field by William H. Angoff (1919-1993) have led to national tests and educational reform to determine cutscores for tests.(2)(3)  The Angoff Method (or its updated version, the modified-Angoff) has been adopted by many standardized tests, including the SATs and CFA.(5)

Method The Angoff method essentially uses field experts to determine the difficulty of each question.  It defines the cutoff score as the lowest score a minimally acceptable candidate is likely to achieve.  For the CIMA(R), a minimally acceptable candidate would be defined as someone who has an adequate knowledge of the learning objectives provided in the Candidate Handbook.  Essentially, it works this way:

• A panel of experts independently rate each question, based on what they believe the % of minimally qualified candidates would answer the question correctly.  As an example, a question that is rated as 50% means that the panel believes that only 50% of the minimally qualified candidates would answer the question correctly.  This would be a much harder question, than one that is rated as 80%.
• The panel then reviews each test question as a group and comes to a consensus for the rating of each question.  This process is performed periodically to include comments made by test-takers.  Questions are then edited, removed, or re-rated.(6)
• At the exam level, the cut score is determined as a function of the ratings for each question in that test.  Often, it is just the average of the ratings.  As an example: an exam that consists of 50 questions with average ratings of 65%, would require at least a score of 65% to be passed.  An easier exam, with a rating of 80% (say), would require a score of at least 80%.

Conclusion There is a substantial empirical evidence showing the benefit of the modified-Angoff method.  However, it has come under some scrutiny with regards to the balance between Type I and Type II errors (false positives and false negatives: passing people who should fail and failing people who should pass).(7)  However, this would be an issue for fixed pass rates as well.  The use of item-judgment methods, specifically the modified-Angoff method, seems to be a significant improvement over fixed pass rates.  In my humble opinion, this creates a sweeter and fairer testing process.

References
1. Source at IMCA.
2. Angoff, William H: Technical Problems of Obtaining Equivalent Scores on Tests, Journal of Educational Measurement, Vol. 1, No. 1 (Jun., 1964)
3. Thorndike, Robert L. and William H. Angoff: Educational Measurement, American Council on Education, 1971
4. http://en.wikipedia.org/wiki/Standard-setting_study
5. http://www.cfainstitute.org/cfaprog/overview/pdf/IntoOur5thDecade.pdf
6. http://www.cwea.org/cert_howcert_testresults_howpassdeter.shtml
7. Tiratira, Niclie: CutOff Scores: The Basic Angoff Method And the Item Response Theory Method, The International Journal of Educational and Psychological Assessment, April 2009, Vol. 1, Issue 1, pp. 39-47