Test Item Selection

Choosing the Appropriate Test Item Type or Validity Continued

In the October 2012 Test Construction and Validity, we took an in-depth look at the quality of an instructor-developed test by examining the validity of test items. This post will focus on applying those concepts through the appropriate selection of test item types.

If this blog was a synchronous type of media (all of us participating at the same time) and I was able to ask a question and receive immediate answers, I would ask you to share the different types of test items that you use. I’m confident that as a group, you would share most of the following:

Each test item type has inherent strengths and weaknesses and choosing the appropriate item type is like choosing a vehicle to carry a load. To choose a suitable vehicle you must know the load specifications and circumstances it will be operated in. The load specifications for your test items types come from your learning objectives.

The objective’s “action verb” provides the class of load and the “conditions” define the operating environment.



Some test conditions require a substantial vehicle

Classification Systems (Taxonomies)

Trucks come in different weight classes. You’re probably familiar with the classic half-ton pickup truck; good for small dry loads and not really good at carrying liquids. So how do you know what class of test item you need? There are multiple systems for classifying learning outcomes (think objectives). You’re probably most familiar with Bloom’s et al, taxonomy of learning domains. Think of the taxonomies as load classifications. For example consider the following learning objective:

First aid students will be able to state the ten steps of cardio-pulmonary resuscitation (CPR) in the order of performance, from memory with complete accuracy. The action verb is to “state”.

Working from a list of action verbs, you would find the verb in the knowledge category within the Cognitive domain. Within the knowledge category are several possible test item types including: true/false, completion, multiple-choice, short answer, and interview.

  To choose the appropriate test item type, you must also consider the learning objective’s conditions. To continue the vehicle analogy, it is like the difference between 2 wheel and 4 wheel drive. If you plan on getting your load (action verb) over muddy or icy conditions you will need a 4-wheel drive vehicle. Since the conditions are specified as “from memory”, I submit the only appropriate choice is an interview. It is the only test item type that can support the action verb (behavior) “state” and the condition “from memory”. Some might question why not the ubiquitous multiple choice format? Once you supply the correct answer to the student it changes the behavior to “recognize” and the condition would be given a list vs. solely from memory.

Below is a table of suggest test item alignments with the various levels within Bloom’s Cognitive domain.

Source: Carnegie Mellon

Other Options

In my blog on Learning Theories and Instructional Design I suggested that each theory has strengths and weaknesses and one of my goals for this Blog is to expand your options, so I am going to include Gagne’s learning outcomes (which is cognitive theory based vs. Bloom’s which is behavioral theory based) as an alternative set of domains. Why? Some day you may wish to include learning objectives (outcomes) that are outside the Behavioral realm; “cognitive strategies” for example.

Robert Gagne’s scheme for learning outcome is:
• Intellectual Skills (How to do something. Generally what is learned is procedural knowledge)
• Cognitive Strategy (Skills that govern one’s own learning, remembering and thinking)
• Verbal Information (Knowledge of facts, events and rules. Something you can state)
• Motor skills (Physical skills: skate, steer an automobile, printing/writing, keyboarding)
• Attitude (amplifies a person’s positive or negative reactions or choice to circumstances)

Note: Intellectual Skills have 4 distinct classes. The same appropriateness of test item type can be applied to Gagne’s learning outcomes (think domains). See a table of appropriate test item types based on Gagne’s learning outcomes.

Learning Aid:

From information to practice. Offer to do a review of one of your co-workers’ tests. Get a copy of the learning objectives and evaluate the test item types use against the action verb AND conditions of the associated learning objective using one of the domains (outcome) schemes. Talk over any discrepancies. Suggest alternatives. CAUTION: this may lead you to the re-evaluate the quality of your objectives.

Best regards,

Cj