Recently the Citrix Exam Team invited six outside experts to the company headquarters in Ft. Lauderdale, FL to assist in working on a new certification exam. It was an honor and privilege to be one of the six. This article will be about my experiences during this week long process.
Note: Do not ask me what exam I assisted with. I will not tell you.
Do not ask me anything about the questions or answers. I will not tell you.
Do not ask my anything about the process we went through for this exam. If it is not in this article, then it was removed in editing to keep me from violating the Non Disclosure Agreement.
The Citrix Exam Team authorized us to write about our experiences as long as detail is kept to a minimum.
The exam team wanted experts from around the world to assist. These are the states and countries represented:
- Great Britain
- New Zealand
- The Netherlands
In addition to the six of us physically there, there were usually one or two others joining via a GoToMeeting session.
The six of us were invited to spend a week assisting the Exam Team in reviewing every question and answer on the exam. While the overall goal of reviewing the questions and answers seemed simple, the process was anything but simple. The days were long and grueling. The daily schedule was:
9AM to 12PM Review session
12 to 1 Lunch
1 to 4 Review session
About 2:30 Snacks (usually ice cream)
4 to 5 Dinner
5 to 8 Review session
As you might be able to guess, we were well fed! I do not know about the others but I sure feel like I put on ten pounds (or 4.55 Kgs) during the week.
It was rare to finish any review session in less than the allotted three hours. The last review session of each day rarely ended before 9:30PM. After the final review session, most of us would head back to the hotel and continue discussions until 11PM or later.
As with any project, the must be a clear blueprint of the goals, blueprints and processes. The Citrix Exam Team has developed a 12 step Test Development Process:
- Conduct Market Analysis and Create Program Definition
- Create Test Definition
- Conduct Job Task Analysis
- Create Blueprint
- Develop Test Items
- Review Test Items
- Conduct Beta Test
- Perform Item Analysis
- Assemble Forms
- Set Cut Score
- Publish Exams
- Maintain Exams
We were involved in Step 6: Review Test Items. We were given wide latitude over the objective being tested, the question being asked and the answers given. If we disagreed with any of these items, we were allowed to:
- remove the exam objective
- reword the question
- delete the question and start over
- reword an answer
- delete the answer and start over
The first concept we had to understand was who was being tested and was expected to pass the exam. There are three possible exam candidates:
- Informed but clearly not qualified
- Minimally qualified
- Clearly qualified
The exam candidate being targeted by almost every hardware and software vendor is the minimally qualified candidate. This candidate has assisted in the assessment and or design of production environments using the products being tested and would need decisions to be reviewed to ensure quality. This was a concept most of us had a hard time coming to terms with. An exam cannot be so difficult that only a true expert can pass but it also cannot be so simple that someone can read a book and pass.
There are entry level exams like the A+ and Network+ for entry level Information Technology (IT) personnel. There are also expert level exams like the CCIE and Microsoft Certified Architect exams for true IT experts in those fields.
For each review session, we were split into two teams of three plus an additional GoToMeeting attendee and a member of the Citrix Exam Team.
Each section of questions and answers had already been written, more or less, by an exam writer. We would look at the exam objective being tested and verify if the question matched the objective. Then we would verify the technical accuracy of the question and if the question matched to the level of candidate being tested. Once the question was done, we would start on each answer verifying the technical accuracy.
There were no trick questions and no trick answers. Every product, technology or utility referenced in either the question or answers had to be real and pertain to what was being tested.
Many of the discussions on the questions and answers were passionate and thorough. Quite a few of the discussions caused entire questions and answers to be deleted and new ones written. So many of the questions and answers were either deleted or rewritten that the Citrix Exam Team is going to list the six of us as exam authors in the Exam Prep Guide in addition to being listed as Subject Matter Experts.
We also had at least one or two exam objectives deleted as being irrelevant to the exam and exam candidate.
By the end of the week we were all exhausted but satisfied that we had produced an exam we would not mind having our names attached too. We completed all the exam questions and answers by the afternoon session on Friday. To celebrate, the Citrix Exam Team took us all out to eat at Arubas on the beach.
I sent this article to several of my new friends that I spent the week with to get some comments from them.
Comment1: All six of us did this work for free. Citrix paid for our travel, hotel and provided a lot of food.
Comment2: It was surprising how well we all six got along. Take six people from different parts of the world and put them together for 12+ hours a day and it was amazing no one wanted to hurt someone. Everyone provided a wide variety of talents and strengths to make the review process, and the week, go as smooth as possible. At the end of the week, we were all satisfied with the certification exam we had help produced and that was going to have our names associated with.
Comment3: The fact that Citrix invited six outsiders to review every question and answer for a certification exam is quite amazing. The exam team is to be commended for allowing us to delete, write and rewrite entire blocks of questions and answers. Their willingness to allow us to do so was a refreshing surprise. If you think writing certification exam questions and or answers is easy, it is not.
Comment4: Did I mention that we were very well fed?
I enjoyed the week I spent with the Citrix Exam Team and my fellow Citrix professionals from around the world. I also have a new appreciation for certification exams. The amount of time and effort the Citrix Exam Team spent just for this one exam was simply amazing. According to the PearsonVue web site, there are 20 Citrix certifications currently being offered. It is hard to imagine the work that is involved for 20 exams. It was also readily evident the entire Citrix Education department takes a lot of pride in the work they do to keep us Citrix professionals educated and certified.