Handbook For ICCP Item Writers_____________________________

HANDBOOK FOR ICCP ITEM WRITERS

Provided By The

TEST MANAGEMENT COUNCILS

of the INSTITUTE FOR CERTIFICATION OF COMPUTING PROFESSIONALS
2200 East Devon Avenue, Suite 247 Des Plaines, Illinois (708) 299-4227 (708) 299-4280 FAX Compuserve 74040,3722

Copyright 1995 by the Institute for Certification of Computing Professionals All rights reserved

Handbook For ICCP Item Writers_____________________________

TABLE OF CONTENTS
INTRODUCTION.........................................................................................................................................................3 GENERAL CONSIDERATIONS................................................................................................................................3 WRITING STYLE..........................................................................................................................................................3 AVOIDANCE OF BIAS.................................................................................................................................................3 CONTENT OF ITEMS...................................................................................................................................................3 STRUCTURE OF MULTIPLE-CHOICE TEST ITEMS...............................................................................................4 DEVELOPMENT OF MULTIPLE-CHOICE TEST ITEMS..................................................................................5 THE MULTIPLE-CHOICE ITEM STEM.....................................................................................................................5 THE "EXCEPT" OR "NOT" STEM TYPE...................................................................................................................6 WRITING THE ITEM RESPONSES............................................................................................................................8 THE ITEM KEY: ABSOLUTE CORRECTNESS OR BEST POSSIBLE RESPONSE..............................................9 THE ITEM DISTRACTERS..........................................................................................................................................9 "ALL OF THE ABOVE" AND "NONE OF THE ABOVE" AS RESPONSES..........................................................11 SUMMARY: MULTIPLE-CHOICE TEST ITEMS..............................................................................................12 SOME ADDITIONAL HINTS...................................................................................................................................12 SOME THINGS TO AVOID.....................................................................................................................................13

Handbook For ICCP Item Writers_____________________________

INTRODUCTION
The purpose of this handbook is to assist the item writer in their prepara tion of questions for ICCP examinations. Since these exams consist entirely of multiple-choice types, other forms of item construction have not been included. We have chosen to include both general philosophy and specific mechanics of item construction. This will serve those concerned with question content as well as those who will construct final test items. Although these example items are actual items that have been used in previous ICCP exams, we have chosen them primarily for instructional purposes. They are not indicative of the specific content or level of difficulty which may be encountered in present or future ICCP exams.

GENERAL CONSIDERATIONS
WRITING STYLE The purpose of the words and syntax chosen in writing a test item is to communicate explicit meaning as efficiently as possible. Remember, few written words are read as carefully as those in objective test items. Skill in clear, expository writing are essential to good questions. AVOIDANCE OF BIAS Item designers and writers must keep in mind that ICCP examinations are international and multi-cultural in scope. Therefore, writers must guard against constructing test items that contain elements of national, cultural, or regional bias. For instance, in some countries, the CRT--cathode ray tube--is referred to as a VDT--video display terminal. The five-digit postal Zip Code used in the United States is not a universal convention. CONTENT OF ITEMS The ICCP exams test an applicants grasp of the body of knowledge as defined in the ICCP Official Exam Review Outlines. These outlines define the scope of the exam into which exam items must fit. Writers should refer to the appropriate outline for the content of the exams (and thus the content of items). When submitting new items to ICCP, it is helpful if you classify the item according to the outline section (see the form attached to this guide). Items should not be vendor-specific or product specific. For example, it is proper to write a question about features of spreadsheet programs in general, but it

12/31/94 Version 2.0

Page 3

Handbook For ICCP Item Writers_____________________________

would not be proper to ask about a specific feature of Microsoft Excel or Lotus 1-2-3.1 In addition to classification according to the exam outline, it is also possible to classify the depth of the item, as follows: D1 D2 D3 D4 D5 D6 Recognition Differentiation Description Usage Structure Construction What a concept is called; name or title The external differences between one concept and a similar or neighboring concept The external characteristics of a concept; the definition of the concept How, when and why to use instances of a concept The internal structure of a concept; its components and the relationships of the components How to construct instances of the concept from the components, tailored to a specific purpose or application

This classification is not absolute, but the idea of depth is useful to item writers in suggesting some different approaches to creating new items. Consideration of depth leads to such questions as when should this particular method (concept) be used? or how can this method be combined with other procedures to achieve an objective? Item writers should indicate a target depth level on each question, as well as a classification by competency (subparagraph within the test outline). What is a good question? In general, look for things that a computing professional -- a practitioner like you -- should know to be an effective contributor to his/her organization and to the profession itself. When a new person comes into your organization, what kind of general background things do you expect them to bring with them? What kinds of general things to you provide via your training program? Ask yourself, Is this something that is important for every computing professional to know? In the end, its a judgment call, and your judgment as a professional is important in making this determination. STRUCTURE OF MULTIPLE-CHOICE TEST ITEMS All ICCP exams consist of multiple-choice items, or questions. Each item is composed of a stem and four responses (or answers), one of which is correct. When writing an item, follow these rules of structure: 1. Capitalize the first word of the stem as you would with a sentence; follow accepted rules for capitalization;
1

Microsoft is a registered trademark of Microsoft Corporation. Lotus is a registered trademark of Lotus Development Corporation.

12/31/94 Version 2.0

Page 4

Handbook For ICCP Item Writers_____________________________

2. If the stem is a complete sentence, end the stem with a period or question mark, as appropriate. 3. Identify each response with an upper case letter followed by a right parenthesis; e.g., A), B), C), or D). 4. When using a complete sentence or complete question as the stem, begin all responses with upper-case letters; 5. Should the item stem be incomplete or require internal completion, begin all responses with lower case letters; 6. Do not end a response with a period or other punctuation mark. 7. Identify the correct answer, or key, with an asterisk in front of the response identifier letter; e.g., *B). 8. For emphasis, use upper case letters for an entire word; e.g., NOT is capitalized for emphasis. The examples in this guide follow these conventions.

DEVELOPMENT OF MULTIPLE-CHOICE TEST ITEMS

Multiple-choice test items are currently the most highly regarded and widely used form of objective test item. Almost any understanding or ability that can be tested by means of any other item form can be tested by means of multiplechoice test items. To develop multiple-choice test items, the writer must: 1. Formulate a question or an incomplete sentence that clearly implies a question. This is the STEM of the item. 2. Provide a good answer to the question in a few well-chosen words. This is the KEY of the item. 3. Produce several plausible, but incorrect, answers to the question. These are called the DISTRACTERS. THE MULTIPLE-CHOICE ITEM STEM The function of the stem is to acquaint the examinee with the problem that is being posed. Ideally, it should state or imply a specific question. Al though you can sometimes save words without loss of clarity by using an incomplete statement as the item stem, a direct question is often better. For the sake of variety, however, the item writer should also include some incomplete item stem questions. In designing these questions, take particular care to focus on a specific problem. Generalities will lead to ambiguity and confusing multiple interpretations. Below are three examples of good multiple-choice items.

12/31/94 Version 2.0

Page 5

Handbook For ICCP Item Writers_____________________________

Example 1: The item stem is a complete, direct statement. At what time would a structured walk-through be most effective? *A) Before program coding B) At post-completion audit C) While defining program specifications D) Before generation of hierarchy charts

Example 2: The item stem is an incomplete statement. Logical access to a computer system is BEST controlled by *A) password codes B) encryption of data C) intrusion alarm systems D) complex operating systems

Example 3: The item stem is internally incomplete Records are _____ from an indexed sequential file by placing an identifying code within the record. A) added *B) deleted C) updated D) replaced

THE "EXCEPT" OR "NOT" STEM TYPE Another variation of the multiple-choice question stem is the "Except" or "Not" type. In this construction, the test-taker is asked to select the option that is NOT appropriate. In this form, brevity is critical. Brevity in responses simplifies the task for the test-taker by removing an irrelevant source of difficulty. Brief responses also tend to focus attention on the essential differences among the alternatives offered. In general, this type of item construction should only be used for special situations, such as vocabulary knowledge or, perhaps, knowledge of an exception. Examples 4 and 5 demonstrate acceptable usage of this format.

12/31/94 Version 2.0

Page 6

Handbook For ICCP Item Writers_____________________________

Example 4: Knowledge of an exception: Factors relevant to scheduling jobs in a multiprogramming environment include each of the following EXCEPT A) priorities *B) programmer skill C) memory requirements D) input/output demands

Example 5: Knowledge of vocabulary Which of the following is NOT a current asset? A) Patents *B) Inventory C) Prepaid insurance D) Marketable securities Notice that in both Example 4 and Example 5, the item writer has emphasized the terms of exception, "except" and "not". This is to help the reader notice the word that is critical to understanding the question. Use upper case letters for emphasis where you want to reduce the possibility of confusion.

12/31/94 Version 2.0

Page 7

Handbook For ICCP Item Writers_____________________________

WRITING THE ITEM RESPONSES The item writer should adhere to certain writing practices when designing the responses. These practices are intended to keep the test truly objective. 1. Use parallel structure. If possible, begin each response with the same word, or part of speech. The test-taker will not be confused by multiple grammatical constructions. Neither will he/she be inclined to choose a selection simply because it is different and, therefore, stands out. This is depicted in Example 6. Example 6: Use of parallel structure in responses The MOST common technique for handling overflow records on direct-access storage devices is A) chaining *B) buffering C) overlapping D) randomizing By making all responses the same part of speech the item writer has ensured that the choices are clear and grammatically unbiased. 2. Avoid complex or unduly long responses. These types of response tends to confuse the test-taker. They frequently strain unnecessarily his/her power to concentrate. This particular difficulty is found in Example 7. Example 7: Avoid complex or long responses When we refer to the concept, "ones complement" arithmetic, what precisely do we mean? A) All numbers are taken to less than one B) All arithmetic operations are performed twice for validity checking *C) All numbers are subtracted from one before performing any arithmetic operations D) Negative numbers are formed by inverting the bits in the positive number such that all zero bits are ones and all one bits are zeroes After wading through answers C and D, the test-taker will be strained to remember the actual question.

12/31/94 Version 2.0

Page 8

Handbook For ICCP Item Writers_____________________________

THE ITEM KEY: ABSOLUTE CORRECTNESS OR BEST POSSIBLE RESPONSE Ideally, the intended answer to a multiple-choice question should be a thoroughly correct answer. It should admit no difference of opinion among adequately informed experts. This kind of absolute correctness, however, is difficult to achieve except in formal logical systems. Test designers must base many of their items on propositions that are not absolutely true, but are, nonetheless, strongly probable. Notice in Example questions 1 and 2 above that the item writer has stressed the words "best" and "most" in the item stem. This directs the test-taker not to look for absolute correctness, but rather the best possible option. Another guideline to follow is that the stem of a multiple-choice item should ask a question that has a definite answer. Indeterminate questions may provide interesting topics for discussion, but they do not make good items for testing achievement. Example 8 below demonstrates an indeterminate question. Example 8: Avoid indeterminate questions The practice of structured programming is an outgrowth of A) faster computers B) increased computer storage *C) escalating programmer salaries D) rising costs of program maintenance

In this example, an adequate argument could be mounted for any of the options. Therefore, establishing one correct answer becomes a matter of subjective choice. THE ITEM DISTRACTERS The purpose of a distracter in a multiple-choice test item is to discriminate be tween those who have command of a specific body of knowledge and those who lack it. To do this, a distracter must be a plausible alternative. In other words, it should be appropriate to the question or implied by the stem. One way of obtain ing plausible distracters is to use true statements which do not correctly answer the stem question. Example 9 demonstrates this idea.

12/31/94 Version 2.0

Page 9

Handbook For ICCP Item Writers_____________________________

Example 9: Use of true answers that do not correctly answer the stem In a memory-constrained virtual storage operating system, which one of the following types of interrupt is the MOST common? A) Machine check B) Page interrupt *C) Timer interrupt D) Program interrupt

In this example, all of the choices are truly types of interrupts. However, since the question specifies the most common form, only one response adequately answers the question: the timer interrupt. Multiple-choice items should make frequent use of this device for testing an achievement that is sometimes thought to be testable only by using essay examinations. Another source of plausible distracters are familiar expressions and phrases. Because they have been used in common parlance, they are frequently attractive to students with merely superficial knowledge. See Example 10 below. Example 10: Using familiar expressions as distracters A group of characters that is read or written with each physical read or write operation on a storage medium is called a A) file B) field *C) logical record D) physical record It is highly likely that the test-taker would have heard the terms, "file", "field", and "physical record". But a person with a low-level understanding of the topic may well have not understood them. Therefore, these familiar terms provide excellent distracters at the elementary level of discrimination. On the other hand, obscure distracters are usually undesirable. Example 11 demonstrates this concept.

12/31/94 Version 2.0

Page 10

Handbook For ICCP Item Writers_____________________________

Example 11: Avoid the use of obscure distracters The connections between the subsystems of an application system are known as *A) windows B) asymptotics C) permutations D) catasterizations

If the stem definition and the word "windows" represent an appropriate level of difficulty for this vocabulary test, then the remaining terms used as distracters are obviously too difficult. It is unreasonable to expect the test-taker to know for sure that one of them might not be a better definition than the intended correct answer. Conversely, a distracter which is absurd or highly implausible will contribute little or nothing to the effectiveness of a test item. "ALL OF THE ABOVE" AND "NONE OF THE ABOVE" AS RESPONSES. A common device for adapting multiple-choice items to questions that seem to require several correct answers is to add as a final alternative the re sponse, "all of the above" or "both of the above." A correct answer should not be wrong simply because there are other correct answers. In addition, the experienced test-taker who may have an incomplete knowledge of the subject will have an improved chance of guessing the correct answer. Example 12 demonstrates this type of question. Example 12: Avoid the "all of the above" and "none of the above" During a preliminary study to determine the number of invoices prepared per month, you should A) review a production log B) ask a direct question *C) do both of the above D) do neither of the above

12/31/94 Version 2.0

Page 11

Handbook For ICCP Item Writers_____________________________

If the test-taker knows that option A is correct but is uncertain about B, he/she can eliminate options D and B, greatly improving his/her chances. In a similar vein, the item writer should also avoid the "none of the above" and "neither of the above" options. These item types are more suitable for measuring rote-memory skills such as arithmetic or spelling knowledge.

SUMMARY: MULTIPLE-CHOICE TEST ITEMS

1. Multiple-choice test items should be based on sound, significant ideas that can be expressed as independently meaningful propositions. 2. The stem of a multiple-choice item should state, or clearly imply, a specific direct question. 3. The wording of a multiple-choice item should not follow familiar textbook phraseology so closely that verbal memory without comprehension will provide an adequate basis for response. 4. All of the responses to a multiple-choice test item should be parallel in point of view, grammatical structure, and general appearance. 5. True statements that do not provide good answers to the stem question often make good distracters. 6. Some of the most effective multiple-choice test questions call for a best answer rather than an absolutely correct answer.

SOME ADDITIONAL HINTS

1. The item stem should pose the essence of its question as simply and as accurately as possible. 2. To function properly, a multiple-choice item must be expressed in carefully chosen words and critically edited phrases. 3. If you use acronyms, spell out what they mean to assure clarity; e.g. Automatic Teller Machine (ATM) can be confused with Asynchronous Transfer Mode (ATM). 4. Brevity is desirable in multiple-choice item responses, but it should not be achieved at the expense of importance and significance in the question asked. The stem of a multiple-choice item should be expressed as concisely as possible without sacrificing clarity or omitting essential qualifications.

12/31/94 Version 2.0

Page 12

Handbook For ICCP Item Writers_____________________________

5. The responses "all of the above" and "none of the above" are appro priate only when the answers given to a question are absolutely correct or incorrect. 6. The responses to a multiple-choice item should be expressed simply enough to make clear the essential differences among them. 7. A test-taker who selects the correct response to a multiple-choice item by eliminating the incorrect responses demonstrates useful achievement. 8. The distracters in a multiple-choice item should be definitely less correct than the answer, but plausibly attractive to the uninformed. 9. The intended answer should be clear, concise, correct, and free of clues. 10. Item writers can make some multiple-choice items easier by making the stem more general and the responses more diverse. Conversely, they can make items more difficult by making the stem more specific and the responses more similar.

SOME THINGS TO AVOID

1. Avoid asking the test-taker for a personal opinion; base questions on facts as much as possible. 2. Avoid alphabet soup questions that depend heavily on acronyms. 3. Do not submit trick questions that hinge on some obscure or little-known fact. 4. ICCP exams do not use true-false questions or multiple true-false questions. 5. Dont make the stem or the responses long.

12/31/94 Version 2.0

Page 13

ICCP Item Writer's Guide ICCP EXAMINATION ITEM SUBMITTAL FORM

QUESTION:

ANSWERS: A) B) C) D) OUTLINE SECTION: ___Procedural Programming ___Business Info Systems ___Systems Security ___Programming Languages Book/Article Reference: DEPTH: __Systems Development ___Office Info Systems ___Systems Programming ___Microcomputing & Networks

___Management ___Communications ___Software Engr ___Data Resources Mgt

Comment:

NAME ___________________ ADDRESS ___________________ CITY/STATE ___________________________ZIP ________COUNTRY TELEPHONE ___________________ [CCP ________ ]