Hello,
Included here is a summary of the live comments I made to a group of you at the CAS Annual Meeting in New York City earlier this week regarding the various issues that plagued Part 8 this sitting.
I’ll start by providing some of the process facts and diagnostics for Exam 8.
The initial draft exam (1.0) is created by the writers and the Chair and the Vice Chair for the specific exam.
The exam had 2 subject matter expert consultants that reviewed the material (exam draft 2.0).
The exam had 2 pre-testers (both higher than average performers on the exam when they passed it…note, per feedback from other sessions I have attended, we have changed this process for the upcoming Spring exams to not just be the “uber passers” for pre-testers, but a mix of passing scores from the prior exams) that reviewed the exam (exam draft 3.0).
It is worth noting here that the pre-tester provides important information on the each question and tracks the time that it takes them to complete each question. They also provide diagnostics on clarity and difficulty of questions. Currently, the pre-testers do not take the examination under timed conditions, so they are not granted a reading period and asked to finish the exam in 4 hours. The Exam Committee will revisit how to more effectively utilize the pre-testers and look to include more pre-testers in the process. They have currently been providing the Exam Committee with valuable insights, but we all agree they can provide more insights going forward if we do things differently.
The exam had 3 General Officers that reviewed the material after the pre-tester information is included (exam draft 4.0).
The exam is then sent to a professional proofreader (exam draft 5.0).
Before the exam is printed, there is a Pass Mark Panel meeting where the a-priori pass mark is established. This meeting is purposely held before the printing date to catch any errors that may still exist on the exam after all the reviews discussed above. The attendees at the pass mark panel typically include the chair and vice chair of the exam, the general officers, the pre-testers, and a representative from the Syllabus Committee for that exam.
The process above applies to all upper level exams.
At the Part 8 Pass Mark Panel meeting, it appears that there was a lot more editing done to the exam than what I would have expected. This meeting should not be the time and place to conduct edits, but apparently edits were done at this particular meeting. This proved to be problematic as the time between considering the edits and when the exam needed to be printed was very short. In this short time frame, the review process was not as tight as it should have been and editing issues emerged on the exam as a result of trying to change a finished product (draft 5.0) days before the exam needed to be published. In addition, the pre-testers for part 8 were not able to attend the pass mark panel meeting.
The Exam 8 product that was sent to print also included the following diagnostics:
• The average bloom level for the 2014 exam was estimated to be 3.24 (compared to 3.92 for the 2013 exam)
o This does not mean the exam is necessarily easier, but that there may have been a shift to testing at slightly lower cognitive levels
o The long –term target Bloom Level for this exam is a 3.95.
• The average difficulty of the exam was pegged to be higher for the 2014 exam compared to the 2013 exam.
o A lot of this difficulty comes from asking questions that were not necessarily routine questions, but not all necessarily at highest Bloom levels
o This underscores that difficulty and Bloom’s levels are not measuring the same characteristics
• The exam was estimated to take about 10-15 minutes longer to complete that the 2013 exam (2013 was estimated to take around 180 minutes)
• The a-priori pass mark reflected the difficulty of the 2014 exam with a score in the mid 60’s (the one from 2013 had an a-priori pass mark in the low 70’s)
So, then what happened?!?
1) The editing errors in the instructions page (labeling the test as Exam 5 in some cases).
a. The CAS office takes responsibility for this and will ensure they check the Exam Instructions prior to submitting the exam. This is a classic case of improperly using a template (starting with Part 5) and not checking to make sure all the changes were made in the template.
2) There was a study note accidentally included in the supplemental material
a. The CAS office noticed this when they sent the exam to the printers and asked the printers to exclude the study note from the material. Apparently the printers said they would but they did not. We also did not check what the printers sent back before we sent all the exams to the proctors. The CAS office got their money back and is looking for more professional printing services going forward. There will also be new processes in place to look at the returned product from the printers before anything is sent to the proctors.
3) Editing errors on exams (like a question going from Part a to b and then labeling Part c as Part a again OR completely leaving out a heading for Part d on a question that obviously had four parts to it).
a. The Part Chair owns up to this and these are the kind of things that happen when editing is done too soon before the exam is sent to print. Microsoft Word can do weird unintended things when one makes edits to a document that has so much indentation work in it.
b. The solution to this is to have an independent set of reviewers look at the exam prior to it being printed, even if no edits are made at the Pass Mark panel. These reviewers would be other leaders (actuarial) on the Exam Committee.
4) Question 9 including a label on the column that stated “Loss and ALAE”.
a. This column labeling error was not caught from the original version submitted by the writers. Hence, it went through at least 10 reviewers and no one noticed that the ALAE heading should not have been in there. This is not an excuse, but a fact. Going forward, more attention will be needed in reviewing all titles of columns and in tables to make sure that this kind of mistake does not happen again.
b. This question can absolutely be graded fairly by accounting for any number of ways the candidate did or did not attempt to take ALAE out of the calculations.
5) Question 11 having the wrong formula included in the question.
a. This error was caught in the second draft, but between drafts, somewhere along the way, there was a copying and pasting error in updating the question from one draft to another. Again this is inexcusable, but explains why the error appeared. Again, major attention will need to be given to all equations provided in the exams.
b. This question can absolutely be graded fairly too and the graders are aware of what they need to do to grade all variations of assumptions that candidates may have developed.
Hence, under most typical CAS exam creation conditions, the exam would have been given error free with the exception of #4 above, which was simply not caught through the entire process.
This is still not acceptable, but would have made for a better candidate experience than the exam that was actually provided.
Given the current feedback on the exam, an adjustment for time will prove appropriate. Also, given the errors that were on the exam and the mindset of candidates, an additional adjustment beyond merely time may be made on this exam. This kind of adjustment may be handled as a bulk adjustment to the pass mark, or by adjusting specific questions, or both. The Part 8 team is under clear instructions to consider this kind of adjustment in the recommendation they develop at the grading session.
Some things that are traditionally considered by the Exam Committee when deciding on a time adjustment include:
• Comments and responses on the exam survey, both on the specific exam and in comparison to those for past exams
• The number of questions left blank (while we understand time pressure may result in an answer that is not blank but is not as complete as would be under better time conditions, looking at the number blank as a measure both by itself and in comparison to past exams is generally meaningful)
• The performance on the questions at the end of the exam vs. earlier in the exam (while some candidates do not answer the questions in order, enough do such that exams with significant time issues result in poorer performance on the last few questions than on earlier questions)
• Overall performance on the exam
I am confident that we will grade and adjust the pass mark fairly. There was a suggestion to potentially re-offer Part 8 to candidates that do not pass, but I honestly do not think that will be necessary.
With respect to time being an issue on this exam and any exam for that matter, I have to admit that in all my time on the Exam Committee, this may be the single most difficult parameter to estimate.
We receive time estimates by question from both the writers and pre-testers. Our target for a 4-hour exam is that these time estimates sum to approximately 180 minutes. By targeting 180 minutes rather than 240 minutes, this allows for both underestimation on the part of the writers and/or pre-testers and additional time that might result from working under exam conditions. The part chair and consultants often provide input on time as well; comparing their feel for the overall time needed for the whole exam vs. the prior exam for example, but would not provide specific estimates by question. Obviously, this process for estimating time is not perfect, but I wanted to provide insights on how we try to estimate time.
Allowing more time for exams to be completed (for example, allowing 5 hours to complete the exam) has been discussed over the ages, but unfortunately the proctors and companies that administer the exams state that this would be problematic. The Exam Committee and CAS office will continue to consider this issue in the future.
Another obvious option is to try and ask even fewer questions on the exam (target 2.5 hours of exam material, for example). This is very doable as well, but may come at the cost of not covering enough syllabus material on any one exam, perhaps angering candidates who studied for syllabus material that never made it on the exam itself.
Please be assured that the new Chair of the Exam Committee will be revisiting the “time dilemma” and developing recommendations to get us closer to our ideal of covering adequate syllabus material and allowing candidates to demonstrate their knowledge without the pressure of time.
In conclusion, I still want to apologize for all of this happening on this exam, but I am confident we will be able to grade question 9 and 11 appropriately and that the time adjustment we make in bulk will provide candidates and employers confidence that we are taking into consideration the length issue on the exam. We also will be implementing a stricter and more disciplined set of best standards for both the CAS Office and the Part Chairs in their development of the final examination and the review touch points that occur.
Steven D. Armstrong
Chair of the CAS Examination Committee
Included here is a summary of the live comments I made to a group of you at the CAS Annual Meeting in New York City earlier this week regarding the various issues that plagued Part 8 this sitting.
I’ll start by providing some of the process facts and diagnostics for Exam 8.
The initial draft exam (1.0) is created by the writers and the Chair and the Vice Chair for the specific exam.
The exam had 2 subject matter expert consultants that reviewed the material (exam draft 2.0).
The exam had 2 pre-testers (both higher than average performers on the exam when they passed it…note, per feedback from other sessions I have attended, we have changed this process for the upcoming Spring exams to not just be the “uber passers” for pre-testers, but a mix of passing scores from the prior exams) that reviewed the exam (exam draft 3.0).
It is worth noting here that the pre-tester provides important information on the each question and tracks the time that it takes them to complete each question. They also provide diagnostics on clarity and difficulty of questions. Currently, the pre-testers do not take the examination under timed conditions, so they are not granted a reading period and asked to finish the exam in 4 hours. The Exam Committee will revisit how to more effectively utilize the pre-testers and look to include more pre-testers in the process. They have currently been providing the Exam Committee with valuable insights, but we all agree they can provide more insights going forward if we do things differently.
The exam had 3 General Officers that reviewed the material after the pre-tester information is included (exam draft 4.0).
The exam is then sent to a professional proofreader (exam draft 5.0).
Before the exam is printed, there is a Pass Mark Panel meeting where the a-priori pass mark is established. This meeting is purposely held before the printing date to catch any errors that may still exist on the exam after all the reviews discussed above. The attendees at the pass mark panel typically include the chair and vice chair of the exam, the general officers, the pre-testers, and a representative from the Syllabus Committee for that exam.
The process above applies to all upper level exams.
At the Part 8 Pass Mark Panel meeting, it appears that there was a lot more editing done to the exam than what I would have expected. This meeting should not be the time and place to conduct edits, but apparently edits were done at this particular meeting. This proved to be problematic as the time between considering the edits and when the exam needed to be printed was very short. In this short time frame, the review process was not as tight as it should have been and editing issues emerged on the exam as a result of trying to change a finished product (draft 5.0) days before the exam needed to be published. In addition, the pre-testers for part 8 were not able to attend the pass mark panel meeting.
The Exam 8 product that was sent to print also included the following diagnostics:
• The average bloom level for the 2014 exam was estimated to be 3.24 (compared to 3.92 for the 2013 exam)
o This does not mean the exam is necessarily easier, but that there may have been a shift to testing at slightly lower cognitive levels
o The long –term target Bloom Level for this exam is a 3.95.
• The average difficulty of the exam was pegged to be higher for the 2014 exam compared to the 2013 exam.
o A lot of this difficulty comes from asking questions that were not necessarily routine questions, but not all necessarily at highest Bloom levels
o This underscores that difficulty and Bloom’s levels are not measuring the same characteristics
• The exam was estimated to take about 10-15 minutes longer to complete that the 2013 exam (2013 was estimated to take around 180 minutes)
• The a-priori pass mark reflected the difficulty of the 2014 exam with a score in the mid 60’s (the one from 2013 had an a-priori pass mark in the low 70’s)
So, then what happened?!?
1) The editing errors in the instructions page (labeling the test as Exam 5 in some cases).
a. The CAS office takes responsibility for this and will ensure they check the Exam Instructions prior to submitting the exam. This is a classic case of improperly using a template (starting with Part 5) and not checking to make sure all the changes were made in the template.
2) There was a study note accidentally included in the supplemental material
a. The CAS office noticed this when they sent the exam to the printers and asked the printers to exclude the study note from the material. Apparently the printers said they would but they did not. We also did not check what the printers sent back before we sent all the exams to the proctors. The CAS office got their money back and is looking for more professional printing services going forward. There will also be new processes in place to look at the returned product from the printers before anything is sent to the proctors.
3) Editing errors on exams (like a question going from Part a to b and then labeling Part c as Part a again OR completely leaving out a heading for Part d on a question that obviously had four parts to it).
a. The Part Chair owns up to this and these are the kind of things that happen when editing is done too soon before the exam is sent to print. Microsoft Word can do weird unintended things when one makes edits to a document that has so much indentation work in it.
b. The solution to this is to have an independent set of reviewers look at the exam prior to it being printed, even if no edits are made at the Pass Mark panel. These reviewers would be other leaders (actuarial) on the Exam Committee.
4) Question 9 including a label on the column that stated “Loss and ALAE”.
a. This column labeling error was not caught from the original version submitted by the writers. Hence, it went through at least 10 reviewers and no one noticed that the ALAE heading should not have been in there. This is not an excuse, but a fact. Going forward, more attention will be needed in reviewing all titles of columns and in tables to make sure that this kind of mistake does not happen again.
b. This question can absolutely be graded fairly by accounting for any number of ways the candidate did or did not attempt to take ALAE out of the calculations.
5) Question 11 having the wrong formula included in the question.
a. This error was caught in the second draft, but between drafts, somewhere along the way, there was a copying and pasting error in updating the question from one draft to another. Again this is inexcusable, but explains why the error appeared. Again, major attention will need to be given to all equations provided in the exams.
b. This question can absolutely be graded fairly too and the graders are aware of what they need to do to grade all variations of assumptions that candidates may have developed.
Hence, under most typical CAS exam creation conditions, the exam would have been given error free with the exception of #4 above, which was simply not caught through the entire process.
This is still not acceptable, but would have made for a better candidate experience than the exam that was actually provided.
Given the current feedback on the exam, an adjustment for time will prove appropriate. Also, given the errors that were on the exam and the mindset of candidates, an additional adjustment beyond merely time may be made on this exam. This kind of adjustment may be handled as a bulk adjustment to the pass mark, or by adjusting specific questions, or both. The Part 8 team is under clear instructions to consider this kind of adjustment in the recommendation they develop at the grading session.
Some things that are traditionally considered by the Exam Committee when deciding on a time adjustment include:
• Comments and responses on the exam survey, both on the specific exam and in comparison to those for past exams
• The number of questions left blank (while we understand time pressure may result in an answer that is not blank but is not as complete as would be under better time conditions, looking at the number blank as a measure both by itself and in comparison to past exams is generally meaningful)
• The performance on the questions at the end of the exam vs. earlier in the exam (while some candidates do not answer the questions in order, enough do such that exams with significant time issues result in poorer performance on the last few questions than on earlier questions)
• Overall performance on the exam
I am confident that we will grade and adjust the pass mark fairly. There was a suggestion to potentially re-offer Part 8 to candidates that do not pass, but I honestly do not think that will be necessary.
With respect to time being an issue on this exam and any exam for that matter, I have to admit that in all my time on the Exam Committee, this may be the single most difficult parameter to estimate.
We receive time estimates by question from both the writers and pre-testers. Our target for a 4-hour exam is that these time estimates sum to approximately 180 minutes. By targeting 180 minutes rather than 240 minutes, this allows for both underestimation on the part of the writers and/or pre-testers and additional time that might result from working under exam conditions. The part chair and consultants often provide input on time as well; comparing their feel for the overall time needed for the whole exam vs. the prior exam for example, but would not provide specific estimates by question. Obviously, this process for estimating time is not perfect, but I wanted to provide insights on how we try to estimate time.
Allowing more time for exams to be completed (for example, allowing 5 hours to complete the exam) has been discussed over the ages, but unfortunately the proctors and companies that administer the exams state that this would be problematic. The Exam Committee and CAS office will continue to consider this issue in the future.
Another obvious option is to try and ask even fewer questions on the exam (target 2.5 hours of exam material, for example). This is very doable as well, but may come at the cost of not covering enough syllabus material on any one exam, perhaps angering candidates who studied for syllabus material that never made it on the exam itself.
Please be assured that the new Chair of the Exam Committee will be revisiting the “time dilemma” and developing recommendations to get us closer to our ideal of covering adequate syllabus material and allowing candidates to demonstrate their knowledge without the pressure of time.
In conclusion, I still want to apologize for all of this happening on this exam, but I am confident we will be able to grade question 9 and 11 appropriately and that the time adjustment we make in bulk will provide candidates and employers confidence that we are taking into consideration the length issue on the exam. We also will be implementing a stricter and more disciplined set of best standards for both the CAS Office and the Part Chairs in their development of the final examination and the review touch points that occur.
Steven D. Armstrong
Chair of the CAS Examination Committee
Part 8 Exam - What Happened and What We Are Doing About It
0 commentaires:
Enregistrer un commentaire