Mostrando postagens com marcador Assessment. Mostrar todas as postagens
Mostrando postagens com marcador Assessment. Mostrar todas as postagens

quinta-feira, 8 de novembro de 2018

Multiple choice questions



Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options

, , , , and 


BMC Medical Education 2018, 18:252

Abstract

Background

Multiple choice questions are commonly used in summative assessment. It is still common practice for tertiary institutions and accrediting bodies to use five-option single best answer multiple choice questions, despite a substantial body of evidence showing that multiple choice questions with only three or four options provide effective and discriminatory assessment.

Methods

In this study we investigated the distribution of distractor efficacy in exams from four large first-year undergraduate courses in chemistry and in anatomy and physiology in a Health Faculty; assessed the impact on overall student score after changing from five-option to four-option single best answer multiple choice questions; and assessed the impact of changing from five options to four options on item difficulty and discrimination.

Results

For the five-option questions analysed, 19% had four effective distractors, which is higher than previous studies, but still a minority of questions. After changing from five to four options, the overall student performance on all multiple choice questions was slightly lower in the second offering of one course, slightly higher in the second offering of another course, and similar in the second offering for two courses. For a subset of questions that were used in both offerings, there were negligible differences in item difficulty and item discrimination between offerings.

Conclusions

These results provide further evidence that five-option questions are not superior to four-option questions, with reduction to four options making little if any difference to overall performance, particularly when MCQ is used in conjunction with other assessment types (including short answer questions, and practical or laboratory assessment). Further areas of study that arise from these findings are: to investigate the reasons for resistance to changing established assessment practice within institutions and by accrediting bodies; and to analyse student perceptions of the impact of a reduced number of options in MCQ-based assessment.

sexta-feira, 4 de maio de 2018

Desenvolvimento Docente







Centro de Desenvolvimento Docente da FMRP-USP tem financiamento aprovado pelo National Board for Medical Examiners


O Centro de Desenvolvimento Docente para o Ensino (CDDE) da FMRP é um órgão criado recentemente na unidade para desenvolver atividades educacionais que visam aperfeiçoar os trabalhos dos professores no ensino e na avaliação dos estudantes, bem como na organização e na gestão dos currículos.

No final de 2017, o CDDE tomou conhecimento que o National Board of Medical Examiners (NBME), que é um organismo responsável por fazer exames dos médicos que pretendem obter treinamento profissional ou trabalhar nos EUA, estava financiando atividades de capacitação de professores da área da saúde na América Latina, para aperfeiçoamento da avaliação dos estudantes. Isso vem sendo feito por meio de um programa permanente denominado Latin American Grants (LAG). O CDDE da FMRP elaborou um projeto, em consórcio com seis outras instituições, que veio a ser um dos três aprovados nesse ano, dentre as 15 propostas apresentadas por universidades de cinco países da América Latina. O projeto da FMRP tem como objetivo principal a elaboração pelas instituições participantes de programas bem definidos de avaliação dos estudantes dos diversos cursos, que contenham aperfeiçoamentos claros, em relação à situação atual. Estes programas devem prever o emprego de métodos mais efetivos de avaliação de conhecimentos e de habilidades clínicas e maior ênfase na avaliação formativa (feedback aos estudantes). 




RESPOSTAS ÀS QUESTÕES

1- Como serão investidos os recursos obtidos no projeto?
Os recursos obtidos (cinquenta mil dólares americanos) serão investidos predominantemente na organização e ministração de cursos e oficinas de capacitação de professores para aperfeiçoar a avaliação dos estudantes. Será feito inicialmente um curso mais prolongado na FMRP (segundo semestre de 2018), que contará com professores estrangeiros convidados. Em seguida, serão feitos, ao longo de 2019, cursos e oficinas mais curtos em cada uma das instituições parceiras. Na fase final do projeto, serão realizadas reuniões e visitas a cada instituição, visando acompanhar e auxiliar na elaboração dos programas de aperfeiçoamento da avaliação dos estudantes de cada curso e de introdução de novos métodos de avaliação de conhecimentos e de habilidades clínicas.

2 - Quais serão as instituições parceiras?
A instituição líder é do projeto é a FMRP, com os seus cursos de Medicina, Fisioterapia e Terapia Ocupacional. Na USP, entra também a FMUSP com o seu curso de Medicina.
Outras instituições e cursos são: UFSCAR (cursos de Medicina, Fisioterapia e Terapia Ocupacional), UNICAMP (curso de Medicina), PUC-SP (cursos de Medicina e Enfermagem), UFMG (cursos de Fisioterapia e Terapia Ocupacional), Universidade de Passo Fundo, RS (cursos de Medicina, Farmácia e Medicina Veterinária) e Escola Bahiana de Medicina e Saúde Pública, BA (curso de Medicina).

3-Quais cursos da FMRP serão beneficiados?
Como mencionado acima, estão incluídos formalmente no projeto os cursos de Medicina, Fisioterapia e Terapia Ocupacional da FMRP. No entanto, pela própria natureza da missão e das atividades do CDDE, pretende-se estender naturalmente as ações de aperfeiçoamento da avaliação do estudante a todos os seus demais cursos (Ciências Biomédicas, Fonoaudiologia, Nutrição e Metabolismo e Informática Biomédica).

4- Como a NBME observará o progresso do projeto?
O NBME tem expertise comprovada em mais de 100 anos de atuação nos EUA na avaliação de estudantes e na elaboração de exames e deve colaborar nas cursos e oficinas a serem realizadas no âmbito do projeto. Além disso, deve ter ações de acompanhamento e, ao final, avaliar se os objetivos estabelecidos no projeto da FMRP, junto com as instituições parceiras, foram atingidos.

Referência: Prof. Dr. Luiz Ernesto de Almeida Troncon Prof. Dr. Valdes Roberto Bollela - Coordenadores do Projeto CDDE-FMRP




quarta-feira, 25 de abril de 2018

Assessment



An Innovative Postexam Review Activity


postexam review assignment


We need to work more with students on seeing exams as something more than just grade generating experiences. Exams can be powerful encounters through which students learn course content and learn about learning. However, given the importance placed on grades, I’m not terribly optimistic about a lot of students discovering on their own what can be learned from an exam experience. We need to frame exams with a stronger focus on learning, and here’s a great example.




It’s a postexam review activity originally used in an introductory biology course where the exams were mostly multiple choice. The activity includes these components:
  • Correcting and Reflecting – With their graded exam returned, students were charged with providing the correct answer for each question missed and identifying the sources they used to correct the answer. Next, they had to explain why their original answer was incorrect. And finally, they had to tell why they missed the question. They did this by selecting from a list of possible reasons—such as misread the question, used incorrect logic, didn’t know enough information—or providing an alternative explanation if their reason wasn’t listed.
  • Examining Study Strategies and Behaviors – This component included a list of 16 study tools (e.g. flash cards and concept maps) and behaviors (e.g. regular class attendance and study times). Here students indicated the tools and actions they used to prepare for the exam and which ones they found most helpful.
  • Using Openended Reflection – Finally, students responded to a three-part question: 1) was the grade an accurate reflection of what they knew for the exam and how well prepared they felt; 2) what could they do to improve their learning prior to the next exam; and 3) what could the instructor do to improve their learning experience.
Students had one week after the exam to complete this assignment. It was worth up to 10 points with the exam worth 100 points. Students did the assignment after each of the three major exams, although the structure changed for Exams 2 and 3 to give students more flexibility in their analysis.
What makes this activity especially interesting is the analysis of the assignment undertaken by faculty and students (two are listed as first authors of this piece). It’s reported in this reference and in a larger follow-up study highlighted in the May issue of the Teaching Professor newsletter. The sample size in this first analysis was small, even so the results raise some interesting questions. Not surprisingly, students were able to correct their answers. But when they tried to explain why their incorrect answers couldn’t be right, only 62% of their responses received a “High Quality” score, using a rubric developed for the analysis. The authors make two points here: students may be able to correct answers and still not understand the material in the exam question; and this discrepancy “underscores a common limitation of multiple-choice questions to accurately assess student understanding.” (p. 87)
Students in this sample were also convinced they were using the best methods to prepare for the exam. They relied heavily on materials provided by the instructor—study worksheets and a prelecture assignment. Any study approach that required motivation and self-direction, such as self-testing, regular study of the content, reading before attending lecture, were used significantly less often and only 11% believed approaches like these held much value, even though the effectiveness of these more self-directed strategies is widely supported by research.
Also, perhaps not surprising, almost half the students didn’t feel their grade on Exam 1 reflected what they knew, and they regularly attributed that discrepancy to careless mistakes. “This suggests that they [students] believe using caution is the only measure they need to take to improve their performance on future exams.” (p. 90)
Yes, this is a labor-intensive assignment, for teachers and students. Something the authors readily agree. But as the follow-up study reveals, it’s an assignment that contributed to improved exam scores and promoted metacognitive development. Students learn the content and they learn more about learning the content. That’s a big payoff, and assignments like this don’t need to occur in every course. We need to think more strategically about where assignments (of various sorts) should be placed within a curricular experience.
References: Andaya, G., Hrabak, V., Reyes, S., Diaz, R. and McDonald, K. (2017). Examining the effectiveness of a postexam review activity to promote self-regulation in introductory biology students. Journal of College Science Teaching, 46 (4), 84-92.
Dang, N., Chiang, J., Brown, H., and McDonald, K. (2018). Curricular activities that promote cognitive skills impact lower-performing students in an introductory biology course. Journal of Microbiology and Biology Education, 19 (1), 1-9.

terça-feira, 31 de outubro de 2017

Digital assessment




Stacey Newbern Dammann, EdD, and Josh DeSantis



For many professors, student assessment is one of the most labor-intensive components of teaching a class. Items must be prepared, rubrics created, and instructions written. The work continues as the tests are scored, papers read, and comments shared. Performing authentic and meaningful student assessment takes time. Consequently, some professors construct relatively few assessments for their courses.
Unfortunately, this practice limits professors’ ability to reliably assess student learning. If a course grade is a mosaic, then each assessment is a tile. A mosaic with just a few tiles only presents a part of the picture. Professors can improve the quality of their assessment mosaic by increasing the number of performances they assess. These smaller and more frequently administered snapshots of student learning are frequently termed formative assessments. The integration of frequent formative assessments improves the validity of course assessment and has been demonstrated to have a variety of benefits, including improving student achievement and helping students develop more agency over their own learning (Nicol & Macfarlane-Dick, 2006). By providing more numerous and authentic measurements of student thinking, professors can improve the fidelity of their assessment mosaic and the reliability of their assessments of student learning.
The proliferation of mobile devices and the adoption of learning applications in higher education simplifies formative assessment. Professors can, for example, quickly create a multi-modal performance that requires students to write, draw, read, and watch video within the same assessment. Other tools allow for automatic grade responses, question-embedded documents, and video-based discussion. In addition to simplifying formative assessment, the use of these tools has been shown to amplify student engagement (Hwang & Chang, 2011).
Emerging tools and approaches open new opportunities for professors to gather more frequent and more authentic formative assessment data. This, in turn, can help students learn the course material and can help professors to tailor their instruction to meet the needs of their students.
Formative Assessment Tools to Consider
There are hundreds of formative assessment tools available. Many, however, perform similar functions. You can narrow your search by considering what approach would work best in your teaching context then identifying a tool you can integrate into your practice. If you are a novice, start small and aim for a win. Choose a course for which you would like to improve the quality of your assessment then select one technology tool to get started with.  
  • Multi-Modal Assessments – Several applications allow professors to create multiple-choice and open-ended items that are distributed digitally and assessed automatically. Student responses can be viewed instantaneously and downloaded to a spreadsheet for later use. Examples of these tools include Socrative (socrative.com) and Poll Everywhere (www.pollev.com). Some tools in this category have unique capabilities. Formative (www.goformative.com) allows professors to upload charts or graphic organizers that students can draw on with a stylus. Formative also allows professors to upload document “worksheets” which can then be augmented with multiple-choice and open-ended questions. Nearpod (www.nearpod.com) allows professors to upload their digital presentations and create digital quizzes to accompany them. Nearpod also allows professors to share three-dimensional field trips and models to help communicate ideas.
  • Video-Based Assessments – Question-embedded videos are an outstanding way to improve student engagement in blended or flipped instructional contexts. Professors may upload their own videos and screen-capture files, or use pre-existing streaming video from YouTube. Once uploaded, videos may be embedded with multiple-choice or short answer items. Using these tools allows professors to identify if the videos they use or create are being viewed by students. EdPuzzle (edpuzzle.com) and Playposit (www.playposit.com) are two leaders in this application category. A second type of video-based assessment allows professors to sustain discussion-board like conversation with brief videos. Flipgrid (www.flipgrid.com), for example, allows professors to posit a video question to which students may respond with their own video responses.
  • Quizzing Assessments – Finally, tools that utilize close-ended questions that provide a quick check of student understanding are also available. Quizizz (quizizz.com) and Kahoot (www.kahoot.com) are relatively quick and convenient to use as a wrap up to instruction or a review of concepts taught. Quizizz works on any device with a browser and features specific apps for iOS and Chrome. Kahoot also has specific apps for iOS, Android, and Windows devices. Kahoot now features a team function and the ability to adjust the time limit for questions. Themes, memes, and avatars are available to choose so professors need only focus on the content of the assessment. Ten to 20 questions are easily administered in both tools; although longer assessments tend to lessen student engagement. Both multiple choice and true/false question format work well with either tool. These provide a fast-paced formative assessment that can address fact-based knowledge as well as higher order thinking.
Considerations
It is important to ensure that your integration of technology is aligned to sound formative assessment design. Formative assessment is most valuable when it addresses student understanding, progress toward competencies or standards, and indicates concepts that need further attention for mastery. Additionally, formative assessment provides the instructor with valuable information on gaps in their students’ learning which can imply instructional changes or additional coverage of key concepts. The use of tech tools can make the creation, administration, and grading of formative assessment more efficient and can enhance reliability of assessments when used consistently in the classroom. Selecting one that effectively addresses your assessment needs and enhances your teaching style is critical. Moreover, it is important that you determine if the tools you select are compliant with your institution’s accessibility and student privacy policies.
Summary
Using tech tools to support assessment can enhance the assessment mosaic of nearly every course. These tech-enhanced formative assessments produce actionable data that can help students learn more efficiently (Yorke, 2003). The design and function of the applications introduced in this article provide a starting point to enhancing your course assessment. The wide array of tech tools available allow professors to select one that match teaching styles and assessment needs. The promise of real-time assessment information and the convenience of collecting data digitally make emerging technologies a great place to advance your teaching practice.
References
Hwang, G. J., & Chang, H. F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers & Education, 56(4), 1023-1031.
Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.
Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher education45(4), 477-501.
Stacey Newbern Dammann, associate professor and chair, Department of Education, York College of Pennsylvania. Josh DeSantis, assistant professor of education, director of Masters of Education Program, York College of Pennsylvania.

terça-feira, 7 de junho de 2016

Assessment



Eliminating Tests Through Continual Assessment

by Leslie Tyler

We’re at an inflection point with our approach to testing and measurement …

“Common Core was such a good idea,” remarked a middle school administrator I recently spoke with. “But then the testing ruined it.” My colleagues and I at Edulastic hear this all the time, as we continue to provide teachers, administrators, and school districts a free, easy-to-use online assessment platform for K-12 teachers that allows them to track student’s progress toward Common Core State Standards as well as giving teachers access to create and share fully customized assignments.

Educators have been working on the transition to the new Common Core State Standards over the past 4-5 years. But last year’s final implementation step – administering the standardized tests meant to ascertain whether students met the more rigorous standards – has caused enough controversy to undo that work, overturning the standards themselves in some states. To date, at least 10 states have abandoned Common Core or have announced intentions to do so. President Obama concurred with the test critics, saying,

“Learning is about so much more than just filling in the right bubble. So we’re going to… make sure that we’re not obsessing about testing.’’

So what went wrong? Besides the wholesale change in the test content and delivery, the primary mistake was placing such a large bet on the outcomes. Results could affect federal funding. Teachers and administrators could be fired and “failing” schools taken over or closed. With these types of penalties, testing changed completely from a vital part of teaching and learning into a ruler to rap knuckles.

The Upside of Tests

Good teachers have been giving tests for centuries to understand what students know and what they still need to learn. Such so-called “formative” tests vary widely in method and definition – from students’ reflecting on their work to a quiz on last night’s reading – but they nevertheless provide essential information to teachers and students about what to cover next.  In fact, recent research shows that formative assessment actually helps students retain what they learn.

While it’s a bit Pollyannaish to propose replacing standardized tests with formative ones, we could eliminate the most negative effects by doing more formative assessments.

Following are some of the biggest testing pain points and ways to alleviate them through low-stakes, continual assessment:

Too much time away from teaching
In a survey Edulastic conducted last summer, we found that educators’ top concern with the new tests was the time required of students: 70% were somewhat or very concerned about it. Unlike formative assessments, which provide immediate data on understanding so that teachers can adjust instruction, educators do not get results from standardized tests until it’s too late to do anything about them. A recent study on testing released by the Council of the Great City Schools found that 39% of school districts had to wait 2-4 months to get test results, often not arriving until after school was out for the year. This is where new products such as Edulastic come in—allowing teachers to save time and instantly see each student’s areas of security and struggle, automatically grading and producing reports.
Increased anxiety for students and teachers
Having just one chance to show what you know, with stiff penalties for failure, increases anxiety for teachers and students. In contrast, formative techniques like pre-tests and post-tests help students focus on and practice the most important concepts. Continual assessment reduces anxiety because it’s designed to reveal what a student has learned and has yet to learn, as opposed to whether the student has succeeded or failed.
Lack of reliable data on mastery or progress
Perhaps the most discouraging thing about our current standardized testing scheme is the scarcity of data it produces on student learning. Continual formative assessment produces thousands of time-series data points, allowing educators to say with confidence that a student has mastered a standard or skill. To get this level of confidence from a single, comprehensive test, students would need to answer dozens of questions for each standard, requiring hours of testing (see pain point #1).

Clearing Roadblocks to Change

Historically, standardized tests aimed to easily compare student performance (and by proxy teacher competency). Unfortunately, they are simply inadequate for this task. But how might we answer vital questions like, “How are our schools doing?” and “What do we need to adjust?”

To answer these questions at all levels – from individual students to whole states – we need more formative assessment practice and better data collection systems. Many teachers and schools already make formative and common assessments part of their curriculum. Grade level teachers review results together to figure out what’s working and what needs to be revised or redone. We need more support for this type of professional development, and programs like Edulastic provide insight into student understanding through webinars, DIY training materials, and in-personal professional development sessions.

Second, we need better, more standardized data collection systems. Providing teachers banks of high-quality assessment items to include in their continual assessment mix will yield comparative data on student performance while promoting learning. Aligning teacher-created formative assessments with standards allows for standardized data collection – instead of standardized tests – to exponentially expand the number of data points available on student proficiency.

We’re at an inflection point with our approach to testing and measurement. Educators have access to tools, providing them better research, technology and data to create a new, more efficient system of comprehensive assessment. If we can’t eliminate standardized tests, we can at least reduce their downside. And spend the time and money saved on assessment practices that promote learning and get us closer to the answer to “How are we doing?”


Authentic Assessment, Deeper Learning: ePortfolios in Higher Education

quinta-feira, 26 de maio de 2016

Entrustable professional activities



Carlos El-Haddad, Arvin Damodaran, H. Patrick McNeil and
Wendy Hu


DOI: 10.1111/imj.12914

Abstract

Consultants regularly need to decide whether a trainee can be entrusted to perform a clinical activity independently. ‘Entrustable Professional Activities’ (EPAs) provide a framework for justifying and better utilising supervisor entrustment decisions for trainee feedback and assessment in the workplace. Since being proposed by Olle ten Cate in 2005, EPAs are emerging as an integral part of many international medical curricula, and are being considered by the Royal Australasian College of Physicians in the current review of physician training. EPAs are defined as tasks or responsibilities that can be entrusted to a trainee once sufficient competence is reached to allow for unsupervised practice. An example might be to entrust a trainee to ‘Initiate and co-ordinate care of the palliative patient’ with only off-site or indirect supervision. Rather than attempting to directly measure each of the many separate competencies required to undertake such a complex task, EPAs direct the trainee and supervisor's attention to the trainee's performance in a limited number of selected, representative, important day-to-day activities. EPA based assessment is gaining momentum, amongst significant concerns regarding feasibility of implementation. While the optimal process for designing and implementing EPAs remains to be determined, it is an assessment strategy where the over-arching goal of optimal patient care remains in clear sight. This review explores the central role of trust in medical training, the case for EPAs, and potential barriers to implementing EPAs based assessment.