Tests

EVALUATION CHECKLIST

Ron Dumont
John Willis

I. CONTEXT OF THE EVALUATION

A. Have all assessment results been placed in context, including the cautious, skeptical interpretation of the following?

1. The student’s actual, real‑life performance in school;

2. The student’s educational history;

3. The student’s personal, familial, and medical history;

4. Reports and referral questions from therapists, parents, and the student (actively solicit reports and questions if necessary);

5. The student’s self-reports;

6. Previous evaluations; and

7. Concurrent findings of other evaluators.

8. Have all data been integrated?

B. Have commonalties been given due weight; and have disparities been acknowledged and discussed?

C. Has knowledge of the need for multiple sources of convergent data been shown?

D. Have the final decisions taken into account the above considerations?

E. Is the choice of tests sufficient to sample behaviors for a specific purpose (e.g., psychoeducational testing)?

F. Is the choice of tests appropriate to the test takers?

G. Is the choice of tests as free from bias as possible, considering the standardization sample and the test-taker population?

H. Do the tests used distinguish between skills rather than lumping them together? For example, a combined reading score based on both oral reading and reading comprehension may mask important differences between the two skills.

II. TEST STANDARDIZATION AND NORMS

A. Do not alter standardized tests especially normed, standardized tests.

1. Were all basal and ceiling rules obeyed, and were they the correct ones, since the rules vary from test to test?

2. Were the test instructions and items read verbatim; with appropriate demonstrations given as instructed?

a. Was there any coaching, helping, or teaching given, except as instructed?

b. Was there any ad libbing during standardized instructions?

c. Did the examiner appear to have practiced the instructions so that they were delivered in a smooth, conversational tone?

e. Was there any unauthorized feedback given?

f. Were all time limits, timed presentations, and timed delays adhered to?

3. Was the test learned and practiced to complete mastery before using it? Was qualified supervision obtained when learning and using a new test?

4. If “testing the limits” has been used, does the report contain clear statements about what was done, why, and how it was done? Has it been made absolutely certain that results of limit‑testing cannot possibly be confused with valid scores?  Was testing the limits done in ways that ensured that it will not influence later items, e.g., by providing extra practice.

5. For students with severe and low‑incidence disabilities, did the examiner adopt appropriate tests rather than adapt inappropriate ones?

6. Were the students tested in their specific native languages (e.g., Puerto Rican vs. Castilian Spanish or American Sign Language vs. Signed English) with tests normed in those languages?

7. Was consideration given to the consequences of taking subtests out of context?

8. Were the scoring keys and test materials kept secure?

9. Were all protocols and record forms original and not photocopies of copyrighted materials?  [This consideration is especially important for response forms used by the examinee because photocopies may alter the difficulty of the task.]

10. Did the evaluator refrain from answering questions from test takers in greater detail than the test manual allows?

B. Was appropriate attention given to test norms?

1. Were all tests used normed on a genuinely representative sample that was:

a. Sufficiently large;

b. Appropriately stratified and randomized for:

i. sexes

ii. geographic regions

iii. racial and ethnic groups

iv. disabilities

v. income and educational levels

vi. other germane variables.

vii. interactions of these variables

c. National, international, or appropriate, clearly specified subgroup;

d. Truthfully and completely presented in test manual;

e. Recent; and

f. The appropriate test level for the student?

C. Is the evaluator aware of possible errors in norms tables?

1. Printed norms.

2. Computerized norms.

D. Does the evaluator take into account the risk of significant changes in scores resulting from movement between norms tables (e.g., the overnight change from Fall norms to Spring norms)?

E. Is the evaluator appropriately skeptical of publishers’ claims?

F. Does the evaluator establish rapport with examinees to obtain accurate scores?

III. RELIABILITY

A. Is the evaluator aware of and has she or he taken into account the Standard Error of Measurement (SEm)?

1. Does the evaluator consistently use 90% (1.65 or 1 2/3 SEm) or 95% (1.96 or 2 SEm) confidence bands?

2. Is the meaning of the confidence band explained clearly in lay terms?

4. Does the evaluator recognize that a test score was obtained once, at a specific time and place?

5. Does the evaluator recognize and explain that the confidence band does not include errors and problems with test administration and conditions?

6. Does the evaluator distinguish clearly between reliability and validity?

7. Does the evaluator distinguish between Standard Error of Measurement (SEm) and Standard Error of Estimate (SEest)?

8. Does the evaluator use the correct confidence band for the appropriate score: raw, ability, W, standard score, percentile rank, etc.?

B. Does the evaluator determine how reliability data were obtained?

1. How large were the samples?

2. Are there reliability data for students similar to the student being tested?

3. Are the time intervals comparable to those we are concerned with?

C. Does the evaluator understand and make appropriate decisions about tests that use rigid “Cut‑off scores” for decision making?

IV. VALIDITY

A. Does the evaluator appreciate the implications of test validity?

1. Validity for what purposes?

2. Validity for what groups?

3. Determine how validity data was obtained

a. How large were the samples?

b. Are there validity data for students similar to the student being tested?

c. What were the criterion measures? Are we using a closed circle of validating tests against other very similar tests?

d. Are the time intervals comparable to those we are concerned with?

B. Does the evaluator understand the concept of construct validity?

C. Does the evaluator understand the relationship between validity and reliability?

F. Does the evaluator understand and take into consideration issues related to “Incremental validity”?

G. Does the evaluator interpret tests only in ways for which validity has been established?

H. Does the evaluator keep a record of all test data for follow-up to establish trends and understand how the test works in the local situation?

I. Does the evaluator demonstrate an appreciation for the limitations of each specific test’s content coverage?

V. SCORING

A. Does the evaluator avoid errors in scoring and recording by:

1.      Using two straight‑edges in the scoring tables;

2.       If necessary, photocopying the norms table and drawing lines and circling numbers;

3.       Checking the accuracy of tables by inspecting adjacent scores;

4.       Reading table titles and headings aloud while scoring;

5.       Rechecking all scores;

6.       Checking them again;

7.       Getting someone else to check them;

8.       Scoring by both age and grade norms, if available, and comparing the results;

9.       Recording the student’s name and the date on all sheets of paper;

10.  Checking the student’s birthdate and age with the student and calculating the age correctly by the rules for the particular test.

11.  Performing thought experiments with tables, e.g., What if the student had made two lucky or unlucky guesses? What if the student were 6 months older or younger? etc.;

12.   Recording all responses verbatim;

13.   Keeping raw data for future use;

14.   Using consistent notations for correct and incorrect answers, no responses, “I don’t know” responses , and examiner’s questions; making sure the examinee cannot determine from the number or direction of pencil strokes which notations  are being made;

15.  Using protractors and templates consistently, carefully, and correctly and, if uncertain, having someone check the scoring;

16.   Following computer‑scoring instructions slavishly, including the sequence in which  the CPU and peripheral equipment are turned on;

17. Checking continued accuracy of computer results by occasionally hand scoring;

18. Being certain to have the latest version of the scoring program, to  know of any new  errors or bugs in it, and  to have the protocols that go with that version;

19. Understanding and clearly explaining differences among standard scores, scaled scores, normal curve equivalents, percentile ranks, and other scores;

20. Using age-equivalent (“mental age”) and grade-equivalent scores sparingly, if at all, explaining them and their myriad limitations clearly,  making sure they have some relationship to reality; and bracketing them with 90% or 95% confidence bands, just as  with standard scores; and

21. Following scoring instructions exactly as described in the test manual, and, when in doubt, obtaining a second opinion or asking the test publisher for clarification.

VI. INTERPRETATION OF EVALUATION RESULTS

A. Does the evaluator distinguish clearly between different tests, clusters, factors, subtests, and scores with similar titles?

1. e.g., “Reading Comprehension” is not the same skill on different reading tests.

2. e.g., “Processing Speed is not the same ability on different intelligence tests.

B. Does the evaluator explain with words and figures all the statistics used in the reports?

C. Does the evaluator explain differences between different statistics for different tests that are included in the reports?

D. Does the evaluator explain the names (e.g., “Below Average”) for the statistics used in the reports?

E. Does the evaluator explain differences between different names for the same scores on various tests that are used in the reports?

F. Does the evaluator distinguish clearly between findings and implications?

G. Do the interpretation and recommendations demonstrate an understanding of the disabilities and the programs, not merely of the tests?

H. Does the identification of a disability demonstrate a reasoned, clinical judgment, rather than simply an exercise in arithmetic?

I. Does the report offer specific, detailed recommendations and give a rationale for each?

J. Does the evaluator eschew boilerplate?

K. Does the evaluator detect and reject unvalidated computer software?

L. Does the evaluator use computer reports to help interpret data and plan reports rather than simply including or retyping the actual printouts in reports?

M. Does the evaluator recognize that students’ skills in related areas may differ dramatically and unexpectedly?

N. Does the evaluator explain the mechanism of the disability?

1. For example, a specific learning disability is a disorder in one or more of the basic psychological processes involved in understanding or in using language, spoken or written that may manifest itself in an imperfect ability to listen, speak, think, read, write, spell, or do mathematical calculations. Does the evaluator make an attempt to describe the student’s specific disorder(s)?

2. Similarly, a serious emotional disturbance must be based on a psychological disorder. Does the evaluator attempt to specify, define, and explain the disorder, not just the behaviors?

O. Does the evaluator report genuinely germane observations from test sessions, while at the same time being clear that behaviors in a test session may be unique to that test session and may never be seen in any other context?

P. Does the evaluator pay attention to the reported observations?  For example, if an evaluator cites the student’s boredom or fatigue, it would not make sense to declare, “Test results are assumed to be valid.” foolish.

Q. Does the evaluator consider practice effects when tests are re-administered? Does the evaluator consider differential practice effects on different subtests?

R. Does the evaluator appraise the entire pattern of the student’s abilities, not merely weaknesses?

S. Does the evaluator revisit the verbatim record of the student’s actual responses before accepting “canned” interpretations from the manual, handbook, or computer printout? For instance, WISC-III Comprehension measures Social Studies achievement as well as “social comprehension,” Picture Completion almost never measures the “ability to distinguish essential from nonessential details,” and young children can earn high scores for “verbal abstract reasoning” with entirely concrete responses to Similarities.

T. Does the evaluator base conclusions and recommendations on multiple sources of convergent data (not just test scores)?

U. Does the evaluator understand and advise about the limitations of norms, especially grade equivalents, for student populations differing markedly from the norm sample?

V. Does the evaluator have a thorough understanding of both standard scores and percentile ranks?

W. Does the evaluator keep up with the field and check interpretations with other professionals?

X. Does the evaluator apply principles of test theory and principles of test interpretation?

Y. Does the evaluator avoid interpretation beyond the limits of the test?

Z. Does the evaluator avoid idiotic interpretations?

---

Stanford-Binet Fifth Edition (SB5)

Broad and Narrow Abilities

Dumont-Willis

These classifications are based on Ron Dumont and John Willis’s understanding of the Gf-Gc classification system.  These may or may not be consistent with the SB5 manuals.

		Nonverbal				Verbal
Broad Gf-Gc		Activity	Narrow Ability	Testlet		Activity	Narrow Ability	Testlet
Fluid Reasoning		Object Series	Induction	Routing		Early Reasoning	Induction	2, 3
		Matrices	Induction	Routing		Verbal Absurdities	Induction, Language Development	4
						Verbal Analogies	Induction, Language Development	5, 6
Crystallized		Procedural Knowledge	Language Development, Listening Ability	2, 3		Vocabulary	Language Development	Routing
		Picture Absurdities	Language Development, General Information	4, 5, 6
Quantitative Reasoning		Quantitative Reasoning	Mathematical Achievement	2, 3, 4		Quantitative Reasoning	Mathematical Knowledge	2, 3
			Mathematical Achievement, Working Memory	5			Mathematical Achievement	4
			Mathematical Achievement	6			Mathematical Achievement, Working Memory	5, 6
Visual Spatial Processing		Form Board	Visualization	1, 2		Position and Direction	Spatial Relations	2, 3, 4
		Form Pattern	Visualization	3, 4, 5, 6			Visualization	5, 6
Short-term Memory		Delayed Response	Working Memory	1		Memory for Sentences	Memory Span, Language Development	2, 3
		Block Span	Memory Span	2, 3, 4, 5, 6		Last Word	Working Memory	4, 5, 6

Induction: Ability to discover the underlying characteristic that governs a problem or set of materials; Language Development: General development or the understanding of words, sentences, and paragraphs (not requiring reading) in spoken native language skills; Listening Ability: Ability to listen and comprehend oral communications; General Information: Range of general knowledge; Mathematical Achievement: Measured mathematics achievement; Mathematical Knowledge: Range of general knowledge about mathematics; Visualization: Ability to manipulate objects or visual patterns mentally and to “see” how they would appear under altered conditions; Spatial Relations: Ability to perceive and manipulate visual patterns rapidly or to maintain orientation with respect to objects in space; Working Memory: Ability to store temporarily and perform a set of cognitive operations on information that requires divided attention and the management of the limited capacity of short-term memory; Memory Span: Ability to attend to and immediately recall temporally ordered elements in the correct order after a single presentation

D. P. Flanagan, S. O. Ortiz, V. Alphonso, & J. T. Mascolo (2002) Achievement Test Desk Reference (ATDR): Comprehensive Assessment and Learning Disability (Boston: Allyn & Bacon)

K. S. McGrew, & D. P. Flanagan (1998), The Intelligence Test Desk Reference (ITDR): Gf-Gc Cross-Battery Assessment (Boston: Allyn & Bacon)

Testing Templates

A handy dandy chart by John Willis showing the range of average scores for a variety of common statistics..
Average Range Scores Parody

For all the templates, click on the tabs at the bottom to access the worksheets

CHC Templates for the ATDR
ATDR-IIv1.3-1

Dumont Willis WAIS Interpretive Worksheet
WAISIVv12 pub-2

Dumont Willis WISC IV Interpretative Worksheet
WISCIV2pub-1

Psycholinguistic Analyis of the Woodcock Johnson Understanding Directions Subtest
UDAnalysisSheet-1

Normal Curve
NormalCurve-1

Dumont Willis SB5 Interpretive Worksheet
STANFORDBINETV9.12.03pub-1

 WPPSI  III Interpretive Worksheet
WPPSIIIIpub

DAS II Interpretive Worksheets
DASIIComputerAssistantv1.2pub-2

WJ III Template
WJIII Template