From Goals to Data and Back Again
of related interest Giggle Time – Establishing the Social Connection A Program to D...
146 downloads
865 Views
2MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
From Goals to Data and Back Again
of related interest Giggle Time – Establishing the Social Connection A Program to Develop the Communication Skills of Children with Autism Susan Aud Sonders ISBN 1 84310 716 3
Playing, Laughing and Learning with Children on the Autism Spectrum A Practical Resource of Play Ideas for Parents and Carers Julia Moor ISBN 1 84310 060 6
Demystifying the Autistic Experience A Humanistic Introduction for Parents, Caregivers and Educators William Stillman ISBN 1 84310 726 0
Addressing the Challenging Behavior of Children with High-Functioning Autism/ Asperger Syndrome in the Classroom A Guide for Teachers and Parents Rebecca A. Moyes ISBN 1 84310 719 8
Freaks, Geeks and Asperger Syndrome A User Guide to Adolescence Luke Jackson ISBN 1 84310 098 3
From Goals to Data and Back Again Adding Backbone to Developmental Intervention for Children with Autism Jill Fain Lehman and Rebecca Klaw
Jessica Kingsley Publishers London and New York
All rights reserved. Except for the quotation of short passages for the purposes of criticism and review, no part of this book or the accompanying CD-ROM except those portions appearing in the appendices may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher. Material that appears in the appendices of this book may be reproduced without modification by educational institutions. Warning: The doing of an unauthorised act in relation to a copyright work may result in both a civil claim for damages and criminal prosecution. The right of Jill Fain Lehman and Rebecca Klaw to be identified as authors of this work has been asserted by them in accordance with the Copyright, Designs and Patents Act 1988. First published in the United Kingdom in 2003 by Jessica Kingsley Publishers Ltd 116 Pentonville Road London N1 9JB, England and th th 29 West 35 Street, 10 fl. New York, NY 10001-2299, USA www.jkp.com Copyright © 2003 Jill Fain Lehman and Rebecca Klaw Library of Congress Cataloging in Publication Data Lehman, Jill Fain. From goals to data and back again : adding backbone to developmental intervention for children with autism/Jill Fain Lehman, Rebecca Klaw. p.cm. Includes bibliographical references and index. ISBN 1-84310-753-8 (alk.paper) 1. Autism in children. 2. Autism in children--Treatment. 3. Children with disabilities--Rehabilitation. I. Klaw, Rebecca, 1952- II. Title. RJ506.A9L446 2003 618.92’898206--dc21 2003041614
British Library Cataloguing in Publication Data A CIP catalogue record for this book is available from the British Library ISBN 1 84310 753 8 Printed and Bound in Great Britain by Athenaeum Press, Gateshead, Tyne and Wear
For Philip, Charles, and Sarah Kaye
— JFL
For Art, Ben, Jeremy, and Bram
— RK
Contents
1.
2.
List of Exercises
11
List of Figures and Tables
11
Acknowledgments
14
Why We Wrote This Book
15
The Big Picture
18
How to use this book
19
Identifying Goals
21
Attention and basic social relatedness
21
Imitation
23
Affect
23
Self-regulation
25
Play
26
Social play
26
Pretend play
27
Drawing
27
Communication
28
Eye gaze
29
Receptive communication (understanding language)
30
Expressive communication: body language and affect
30
Expressive communication: the use of symbols
31
Conversationsl skills/pragmatics
32
Sensory issues
33
Restricted interests and perseverative behaviors
34
Concept development
34
Increasing awareness of others
35
Social skills with peers
36
Respecting social norms
37
School and camp skills
38
Leisure
38
Things to remember
39
3.
Writing Measurable Goals Change in the child and change in the goals
4.
5.
6.
41 42
Phase 1: Emergence
43
Phase 2: Consistency
43
Phase 3: Extension
43
Writing goals that can be measured
44
Ways of recording increasing or decreasing frequency
47
Using quantitative scales
47
Using a qualitative scale
50
Ways of recording increasing or decreasing duration
52
Ways of recording an increasing range of behavior
53
Ways of recording a decreasing prompt level
55
Considering goal-independent factors
57
Things to remember
58
Collecting the Data
61
Data questions
61
Formatting the data sheet
63
Frequency of data collection
65
Special suggestions for teachers
67
Monitoring goal-independent factors
68
Things to remember
71
Putting It All Together — Joey, Tyler, and Mai Lin
73
Joey
73
Tyler
77
Mai Lin
81
Things to remember
84
From Data Collection to Data Analysis
85
Types of data
85
Translating words to numbers to create raw data
89
Transferring the raw data to spreadsheets
92
Keeping in mind what the numbers really mean
95
First impressions: numbers versus pictures
96
Creating simple line graphs
98
7.
Real data is not pretty
101
Things to remember
102
Detecting Change — The Mean Patterns of change
105
Computing the mean
111
Evaluating change using mean values
111
Step 1: Dividing the data for the goal
112
Step 2: Computing the means and charting bar graphs
115
Step 3: Comparing the means
116
Things to remember
8.
9.
105
Measuring Reliability — The Standard Deviation
118
121
Distance as a measure of representation
121
Computing and graphing the standard deviation
124
Evaluating change using mean and standard deviation
125
Things to remember
132
Exceptions That Prove the Rule — Factor Data
133
Identifying an outlier in the data set
133
Predicting sub-patterns of behavior from factors
137
Correlation is not cause
138
Translating factor data to numbers and computing correlation
139
Negative correlation
145
You get what you ask for
146
Data analysis, revisited
151
Things to remember
151
10. Coming Full Circle — Joey, Tyler, and Mai Lin
153
Joey
153
Tyler
164
Mai Lin
171
Things to remember
179
11. Conclusions
181
Appendix A Basic Intervention Goals for Children with Autism Attention and basic social relatedness Imitation Affect Self-regulation Play Increasing the play repetoire Pretend play Drawing Communication Receptive communication (understanding language) Eye gaze Expressive communication (body language and affect) Expressive communication (the use of symbols for communication) Conversational skills/Pragmatics Sensory issues Restricted interests and perseverative behaviors Concept development Increasing social awareness Social skills with peers Social norms School and camp skills Leisure
Appendix B Charts and Handouts The Three Phases of Intervention How Do We Know When Goals Are Complete? Writing Measurable Goals Detecting Change Using The Comparison of Means Sample Data Sheet — 2-year-old, more compromised Sample Data Sheet — 4-year-old, more compromised Sample Data Sheet — 6-year-old, moderately compromised Sample Data Sheet — school-aged child, more able Glossary Directions for Using Excel A very quick reference guide Selecting Deleting Inserting Changing or modifying Copying and Pasting Miscellaneous
Appendix C Exercises
183 183 184 184 185 185 185 185 186 186 186 187 188 188 190 191 191 192 193 193 194 195 195
197 198 199 200 201 202 203 204 205 206 211 211 211 211 211 212 212 213
215
Exercises 3.1
What’s wrong?
216
4.1
Creating data sheets using tables
219
5.1
Writing goals for Joey
221
5.2
Making Joey’s goals measurable
222
5.3
Writing data questions for Joey’s goals
223
5.4
Writing goals for Tyler
224
5.5
Making Tyler’s goals measurable
225
5.6
Writing data questions for Tyler’s goals
226
5.7
Writing measurable goals for Mai Lin
227
5.8
Writing data questions for Mai Lin’s goals
228
6.1
How to produce a line graph
229
6.2
Graphing shortcuts
233
7.1
Computing the mean
237
7.2
Computing multiple means on a single sheet
239
7.3
Computing “before” and “after” means
240
7.4
Producing a bar graph from data for a single goal
241
7.5
Producing a bar graph with multiple goals for one child
242
8.1
Computing the standard deviation
243
8.2
Adding standard deviation information to a bar graph
244
8.3
Experimenting with raw data, mean, and standard deviation in the Statistics Lab
245
9.1
Experimenting with outliers in the Statistics Lab
247
9.2
Computing correlations with the Pearson r
248
9.3
Understanding negative correlation
250
Figures and Tables Figures 4.1
A sample data collection sheet in an easy-to-use format
64
4.2
A sample data collection sheet for multiple children working at the same time on different goals
67
4.3
A sample data collection sheet for multiple children working at the same time on the same goal
67
4.4
A sample data collection sheet showing goal-independent factors marked by an asterisk
69
4.5
The data collection sheet from Figure 4.2 with additional goal-independent factors
70
6.1
Every item on a data collection sheet has two independent properties. We note the first property—whether it’s a goal data or factor data—at the left. The second property—whether the item results in a categorical, ordinal, or interval data set—is labeled on the right. Both properties are used to determine the appropriate statistical tests to apply to the data.
87
6.2
Joey’s data sheet after translation
91
6.3
A portion of Joey’s raw data after it has been entered on a spreadsheet
93
6.4
A portion of Anton’s data from the ABC Workbook
96
6.5
The line graph for Anton’s raw data shows a “rising stairs” pattern
97
6.6
The legend box is necessary when you combine data for two children receiving group intervention on the same dates
99
6.7
The line graph for Anton’s raw data for goal Y
99
6.8
The line graph for Becca’s raw data. Has she made progress on goal X?
100
6.9
The line graph for Celeste’s raw data. Has she made progress?
101
7.1
A single regular pattern of behavior versus two different regular patterns of behavior
106
7.2
One pattern of behavior (pre-transformation) followed by a period of transformation followed by a new pattern of behavior (post-transformation)
107
7.3
The line graph for Anton’s data for goal X
107
7.4
Is this a case of change or not?
109
7.5
With more data before and after, the pattern of change emerges
109
7.6
Our model of cognition assumes that if change occurred within the intervention period then behavior went through three phases: pre-transformation (“BEFORE”), when old patterns dominate; transformation, when learning occurs; and post-transformation (“AFTER”), when behavior is best characterized by the new patterns learned during intervention
114
7.7
Anton’s performance on goal X “before” and “after” intervention
115
7.8
Celeste’s performance on goal X “before” and “after” intervention
117
7.9
Becca’s performance on goal X “before” and “after” intervention
118
8.1
Good representation displayed as a tight clustering of data points around the mean line
122
8.2
The more data points spread out around the mean line the poorer the mean is as a representative value
123
8.3
Standard deviations displayed as error bars around the means. The bar extends both above and below the mean by the amount of the standard deviation. The oval indicates the area of overlap between the bars 124
8.4
The more overlap in the error bars, the less reliable the change indicated by the difference in means
128
8.5
A clear example of change
129
8.6
The large standard deviations argue against the evidence of change provided by the difference in the means
130
8.7
Comparison of means for Celeste
131
9.1
Overlap in the error bars suggests we look further
134
9.2
Post-transformation behavior looks regular except for a single unexpected low value in Session 27
134
9.3
The expected pattern becomes clear after removing the outlier
135
9.4
Translation of the categorical factor specifying location into numbers
140
9.5
Is Becca still learning or is there something interfering systematically with her performance?
141
9.6
Celeste seems to show a systematic regression (followed by recoupment) after every 5 sessions
144
9.7
There is no way to ask the general question, “Does performance vary with location?” Instead, the Pearson r forces us to test a set of more specific questions, each reflecting a different assignment of locations to numbers
149
10.1
Joey’s data collection sheet, as created in Chapter 5
154
10.2
Change in Joey’s ability to show preferences
158
10.3
Change in Joey’s ability to match a photo to an item
159
10.4
Apparent change in Joey’s ability to approximate the word “go”
161
10.5
Lack of change in Joey’s ability to pull an adult to a desired object
162
10.6
Lack of change in Joey’s ability to imitate mouth movements
163
10.7
Tyler’s data collection sheet, as created in Chapter 5
165
10.8
A case of poorer performance after intervention?
167
10.9
Possible change in Tyler’s conversational ability
169
10.10 Mai Lin’s data collection sheet, as created in Chapter 5
172
10.11 Possible change in Mai Lin’s ability to control her voice
174
10.12 Lack of change in appropriate facial orientation
176
10.13 Change in Mai Lin’s ability to form requests as questions
178
C.1
The icon to start the Chart Wizard in Excel 97 is circled in black
229
C.2
The options to select the Chart Type dialog box. Also note the tabs circled at the top of the dialoge box. Each tab gives you access to a different view of some or all of the information to be specified at the step. 230
C.3
When you select the whole chart the data is highlighted on the spreadsheet and a pull-down menu for the chart appears.
C.4
The locations of the function symbol, the rounding up symbol, and the formula bar. 237
235
Tables 10.1
The statistical information for Joey’s goals over a three-month period
156
10.2
The statistical information for Tyler’s goals
166
10.3
The statistical information for Mai Lin’s goals
173
Acknowledgments Writing a book that is intended for both parents and professionals presents unique challenges. Meeting those challenges would not have been possible without the help of Ivie Torres, our parent reader, who reviewed the manuscript with intelligence, honesty, and dedication. Her words of praise, suggestions for improvement, and sense of humor made this project considerably more enjoyable. Our professional reviewers, Stanley Greenspan, Lee Marcus, Diane Twachtman-Cullen, Susan Izeman, and Cory Shulman, were no less helpful. We are particularly grateful to them for providing us with comments in the all too brief time we gave them. Thanks must also go to Mary Hart and Renee Porterfield for lending their expertise in step-by-step reviews of the spreadsheet exercises. Content is one thing, clarity and style another. Philip Lehman was painstaking in his role as editor and heroic in his role as document formatter. Rosann Lehman also read the entire manuscript, searching for inconsistencies, incoherence, and typographical errors. We are grateful for every minute of their time, effort, and patience. Finally, we would like to acknowledge the special roles played by Karen Berkman, Raj Reddy, and Richard Wyckoff in the writing of this book. Each was responsible in his or her own way for professional support or guidance at a critical juncture on the path to its publication.
Chapter 1
Why We Wrote This Book
Intervention for children with autism has come a long way in the last half century. In the early years after Kanner recognized the disorder, the diagnosis went hand in hand with terms like “unreachable” and “uneducable” and with recommendations like “institutionalize.” With the work of Lovaas and others in the 1970s that view began to change. It was finally demonstrated that children with Autistic Spectrum Disorders (referred to as “ASD” throughout this book) could learn, could respond, could even engage effectively (and “affectively”) in the neurotypical world. Since the 1970s many other types of intervention have been used with children with ASD, each based on a particular medical, nutritional, or psychological theory, each with supporters and detractors. Despite the differences among interventionists, the importance of intervention itself is no longer doubted. Parents and professionals know that providing intervention is critical in helping the child with autism cope with the demands of daily living, access education, and achieve long-term independence. If we knew what every child needed, if in fact every child needed exactly the same thing, intervention would be a relatively simple matter. But the last half-century has also shown us that the skills and deficits that can accompany autism are complex. What benefits one child may have little or no impact on the symptoms of another. As a result, at least for now, there is no single intervention or set of interventions that works for every child. All of which leaves us with a fundamental question: how do you choose among interventions for a child? How do you choose among interventions for your child? We believe that whether you are a professional service provider or a family member/service consumer, you answer that question in two steps. The first step is taken when you choose a theoretical approach to intervention. Theory provides the framework or general point of view for structuring the overall program of intervention. As a professional your choice of approach might be based on prior training, an assessment of your own strengths and weaknesses, passionate personal belief, or combinations of these. As a family member your choice might be based on what fits into your family culture, what matches your 15
FROM GOALS TO DATA AND BACK AGAIN
own observations about your child’s learning style, the recommendations of professionals and other parents, and, frankly, how you feel about the people who represent the service options in your community. However you make the decision, the choice of theoretical approach is an important one, but it is not the final one. Once the first step is taken, hundreds upon hundreds of “second steps” must follow. Every day, every therapeutic session, every class period, choices must be made about what to teach and how to teach it now. And here is where the problem occurs, because although theory is a wonderful “top down” organizer, it still leaves a lot to the imagination. Theory lays out a set of possible interventions bounded by principles and guidelines. It may even dictate some specific methods and curricula. But a child—your child—may need help learning almost anything. Theory cannot cover all the possibilities. There is no curriculum for life. If we can’t take the everyday steps of intervention based on theory alone, what other source of information is there to help us? Within the framework of any particular approach, how do you choose which goals to work on and which methods to use during this moment of intervention? We believe the answer to that question is only partly determined by theory. The other crucial factor in making those intervention choices is data. If theory is top down, data is bottom up. Theory says “Choose your goals from column A and your methods from column B,” and “Don’t attempt the third one on the left until you cover the two before it.” Data adds a different voice, saying, “It’s time to move on, that concept is mastered,” and “This child responds best to the second method in the third row.” The two types of information—theoretical and empirical (data-driven)—work together. Time is too short to wander through the vast set of all the things you could possibly do until your data tells you what works. That’s why you need theory. But theory doesn’t cut down the possibilities enough to tell you the best choice for each individual child. And that is why you need data. There is, of course, a third type of information that is critical to making intervention choices: the intangible sources, intuition and feeling. We don’t mean to discount these sources when we focus on the theoretical and empirical. After all, how many of us have believed in a child’s potential despite every objective measure to the contrary? How many of us have watched children do what the theory or data said they couldn’t? While there is no question that intuition and feeling count, sometimes what we feel is uncertainty. For every person who has trusted his or her instinct there is another person who has wondered, “Is this really working?” Subjective information—intuition and feeling—works best when it is tempered by objectivity, and vice versa. We must listen to what theory and data have to tell us but, at the same time, understand the limitations of what they can say.
16
WHY WE WROTE THIS BOOK
Everybody we know who works with children with ASD has a theoretical position. Some use TEACCH, some use a Floor-time model, some use ABA or a discrete trial approach, some use special education curricula, and some mix and match. This book does not assume that you are using any particular model. It does assume, however, that your approach is consistent with a developmental perspective. In other words, we (the authors) have theoretical positions, too. One of us (Rebecca) was trained as a developmentalist. The other (Jill) was trained as a cognitive scientist. Our theoretical biases were evident long before we started to work with children with autism. When these children captured our hearts and minds we saw a chance to apply what we knew in ways that made sense to us, based on our theoretical “upbringings.” In this respect our experiences are not so different from those of our colleagues. What we have found to be different, however, is our attitude toward data. We often come across the belief that developmentally based intervention is incompatible with an empirical mindset. Interestingly, we encounter this belief about as often from those who are empiricists as from those who are developmentalists. But as far as we can see, there is nothing incompatible between the decision to use any particular theory and the decision to use data. When we first came across the notion that there was, we were surprised. When we came across it again and again we were dismayed. That is why we wrote this book. In some sense having a cognitive or developmental predisposition is superfluous to what follows because we can state the ideas behind this book without any mention of cognitive or developmental principles: · Theory alone is not enough. · Data alone is not enough. · If you are a service provider, you need both sources of information to
make good decisions.
· If you are a service consumer, you should expect that both sources will
be used by those who offer intervention for your child.
On the other hand, just saying that data is good doesn’t tell you how to collect good data. And saying that data should inform intervention isn’t the same as telling you how to extract something informative from a pile of numbers. The specifics—the how tos—require an empirical method and the nature of that method is profoundly affected by your approach to intervention. What kind of data you can collect, what statistics you can use, what measurements of progress apply—all of these things depend on the assumptions and expectations that are part of your theory. We are committed to a form of intervention that is based on the use of both theory and data because we know from experience that decisions based on their combination are more powerful than decisions based on either alone. Moreover, 17
FROM GOALS TO DATA AND BACK AGAIN
we want to show you that, contrary to popular belief, such an approach is possible, even straightforward, when the framework for intervention is developmental theory. For the past six years it has worked for us and for our staff. It can work for you. In this book we want to show you the power of empirical developmental intervention.
The Big Picture What we describe in this book is the empirical piece of an empirical developmental approach to intervention. This book is not about child development, or child development in autism, or how to do developmental intervention per se. This book is about collecting and analyzing valid and reliable data in a way that makes sense if you are a developmental interventionist. To that end, the book is organized to follow the flow of intervention from goals to data and back to goals. In Chapter 2 we start by reviewing the sorts of goals that are commonly found in a developmental approach. These are the basis for our examples in all the other chapters. Chapters 3 and 4 focus on how to take a goal and turn it into a statement that describes activity or behavior that is both observable and measurable. The key feature of this process is that the goal always comes first. We don’t ask what we can measure and then choose our goals; we choose our goals and then transform them into data questions in a way that preserves their connection to the underlying theory. By the end of Chapter 4 you will know how to write goals addressing typical developmental issues for children with autism. You will also know how to create an easy-to-use data collection sheet that helps measure a child’s progress on these goals. In Chapter 5 you have the opportunity to practice those skills in the context of case histories for three children: Joey, Tyler, and Mai Lin. Of course, data collection is only half the story. Data analysis is the other half. Chapter 6 begins the process of taking the observations collected during intervention and producing an objective measure of the child’s progress. In this day and age there is no reason why data analysis needs to be a burdensome or mathematically complex chore. Computer spreadsheet programs can be used to organize data efficiently and perform the statistical computations easily. We know the idea of “going on-line” may seem intimidating to some, so Chapter 6 takes you step-by-step through the process of setting your data up on the computer and creating an initial representation of progress in the form of a line graph. Chapters 7, 8, and 9 form the core of the section on data analysis. Each chapter introduces a different statistical tool to the analytical toolbox. Since we have developed a method that relies on spreadsheet functions to do the computational work, however, our focus in these chapters is not on developing the 18
WHY WE WROTE THIS BOOK
mathematical ideas behind the statistics. Instead, the major portion of each chapter discusses what role its statistic plays in interpreting data within a developmental framework. By the end of Chapter 9 you will understand a great deal about how both cognitive and developmental theories characterize a child’s progress toward a goal and about how that characterization can be detected in your data. Chapter 10 takes the analytical methods we’ve developed and applies them to the case histories of Joey, Tyler, and Mai Lin. By revisiting their stories we have the opportunity to close the loop started in Chapter 2, demonstrating not just how analysis is done but also how it feeds back into the next round of intervention decisions and goals. After this exercise, Chapter 11 steps back to review briefly where we are and how we got there. A final point before you get started: the methods we describe are not the kind used in academic research. They are not the stuff of higher mathematics or the ones used by statisticians. The methods are practical and designed for use in the real world with real kids. They are designed to be used by educators, clinicians, and parents even when tired, rushed, and overworked. We believe that using them should be part of your everyday routine. They cannot replace theory and they cannot replace intuition, but they can help to make sense of both.
How to use this book People read books like this one for a lot of different reasons. Although no book can be all things to all readers, we would like to make it as easy as possible for you to use the book in whatever way it serves you best. If you want to learn to use data as part of a developmental approach to intervention, we suggest you read the chapters in order. In addition, we urge you to do each exercise when it is mentioned in the text. The exercises themselves have been separated from the main text as Appendix C so that the casual reader is not distracted. Nevertheless, they offer an opportunity to practice the methods we discuss and are integral to using this book as a self-teaching guide. To use the exercises and other interactive materials on the CD-ROM effectively you should read Chapters 5 through 10 with a computer that has Microsoft® Excel (97 or higher) and Word (97 or higher) programs available. Some of you may be interested in only the data collection method we describe. For you, Chapters 3, 4, and 5 are the most relevant. Those who are interested in only our approach to data analysis will find the relevant material in Chapters 6 through 9. Individuals who do not need to learn to apply the methods in a practical sense should feel free to skip over the exercises and ignore suggestions in the text to open files on the CD-ROM. Teachers or administrators who want to use this book with their students or staff may find the charts and handouts in the three appendices particularly useful, as they summarize various aspects of our approach in a convenient form. 19
FROM GOALS TO DATA AND BACK AGAIN
Although the text of this book is copyrighted, permission is granted for the copying and distribution of material in the appendices for educational purposes. To make distribution easier, the appendices are also available as Microsoft® Word files on the CD-ROM. Finally, all readers should note the existence of a Glossary in Appendix B. In the main body of the book, a new term or idea is identified by boldface italics the first time it is discussed. The Glossary reorganizes all such terms alphabetically so that you can review a definition as needed without having to find the initial occurrence in the text.
20
Chapter 2
Identifying Goals
A great deal has been written both about typical development and about how it is compromised by the symptoms of ASD. We assume that readers of this book are already knowledgeable about these topics (or will learn about them from another source). Our purpose is to teach you how to translate basic developmental growth, as it occurs in children with autism, into measurable goals. Before we can begin to explore the measurement and analysis of progress over time, however, we have to establish a common vocabulary. This chapter reviews areas of developmental focus that are essential to remediation in order to introduce the basic set of topics from which we will create individually designed goals in the examples that follow. Is this basic set of topics or goals adequate to pinpoint areas of need in every child with ASD? Of course not. Each child presents a unique profile of strengths and areas of compromise. Each child requires a team of parents and/or professionals to work and rework the targeted areas of development until the goals match the child’s needs. To be clear, then, the following set of topics is included in this guide in order to ground our discussion in concrete and realistic examples. It is not intended to be an exhaustive list, curriculum, or intervention plan. Not every goal on this list is appropriate for every child and many goals appropriate for your child may not be included. For those who wish to use this framework as a starting point in their discussions with the rest of the team responsible for designing intervention for their children, a listing of basic developmental goals is available as Appendix A.
Attention and basic social relatedness Attending and relating to others is a skill that serves as the precursor to nearly all forms of early learning. This basic skill is often compromised in children with autism. If you don’t attend to the world of people that surround you, you are quite limited in your opportunities for learning. In designing goals that target this area, you need to look for markers of emerging social skills and figure out what typically developing children do to 21
FROM GOALS TO DATA AND BACK AGAIN
indicate that they are willingly attending to social opportunities. These markers, then, become the outcomes that measure progress in growing attention and sociability. For example, a developmentally early goal in this area might be the following: Child will respond to the overtures of familiar/preferred adults with smile, frown, reach, vocalization, or other intentional behavior.
Here you have decided that this child needs to learn to intentionally acknowledge the overtures of familiar people. The child just needs to react when approached by familiar and loving adults. This predictable social acknowledgment can be painfully absent in children with ASD. Another goal at this stage in the development of social awareness might be: Child will stay engaged with familiar adults for increasing lengths of time before attempting to leave or becoming disengaged.
After the child is beginning to socialize with key adults, you look for social interaction to get easier and easier for the child over time. How do you know it is getting easier? By tracking the amount of time that the child can tolerate social give-and-take before attempting to retreat to more solitary activities. Another goal might target more sophisticated acknowledgment of the key players in the child’s life: Child will call family members/teacher and aides by name.
We know from research that individuals with autism have a poorly developed system for recognizing faces. Researchers have suggested that typically developing people have a special center in the brain that distinguishes one face from another in order to help us define, label, and remember others in our world. We are keenly aware of tiny details when it comes to the people in our communities. Individuals with ASD, on the other hand, seem to lack this refined ability to recognize faces. Therefore, learning the names of specific individuals, which is cued by facial recognition, is often very difficult for children with ASD. To address this area of compromise, we write a goal that measures an outcome behavior (calling family members or teachers by name). This is our developmental marker that indicates growing social awareness. Although our goals measure a certain behavior, it is always important to remember what that behavior really means in terms of the child’s social, emotional, and cognitive development. The goals used as examples in this section are merely suggestions. They are intended to give the reader an idea about how to address early skills that fall under the heading of social attention (more advanced skills that mark social awareness are addressed under the heading “Increasing awareness of others,” 22
IDENTIFYING GOALS
below). Remember that even the full list, “Basic Intervention Goals for Children with Autism,” in Appendix A is not intended to be all encompassing. Each child you meet will have his or her own set of skills and deficits in the area of basic social attention.
Imitation Imitating what someone else is doing is the most basic way a child picks up skills. Long before children can manipulate symbols, long before they comprehend a complex communication system, children imitate the actions of others. In children who develop typically (and seemingly effortlessly), imitation is usually motivated by a desire to do just what their siblings, their mothers, or their fathers are doing. It is actually “social imitation” and it leads swiftly to the use of facial expression, gesture, and finally symbols as a means of communication. In many children with ASD, however, imitation is not predominantly social. It is often more mechanical (learning to open a door, fix a toy, or unwrap a candy bar) and thus does not aid in the development of communication and socialization. It is important, therefore, to help children with ASD practice and expand upon this primary technique for acquiring skills. In order to do this, we break down the imitation process into small steps and teach each step, if need be, one at a time: ° Child will imitate with object after demonstration of use of object. ° Child will simultaneously imitate with objects. ° Child will imitate hand movements. ° Child will imitate body movements. ° Child will imitate mouth movements. ° Child will imitate sounds. ° Child will imitate words.
As the child progresses through these steps he or she gains important methods for learning about both the physical and social mechanics of our complex worlds.
Affect What is “affect?” It is the expression of emotion. It is behavior that involves all or some of the following: facial expression; body language; gesture; tone, pace, and volume of communication; and content of communication. Affect has the capacity to influence, impress, touch, and move others. It has social consequence.
23
FROM GOALS TO DATA AND BACK AGAIN
Typically developing children reliably express emotion and appear to understand emotional expression in others. Children with ASD, however, often have great difficulty with both the expression of their own emotions and the understanding of emotional expression in others. It is important, therefore, in the course of designing intervention for children with ASD to find ways to teach the complex components of emotional expression. You might begin with the following goal: Child will look up to caregiver/teacher using a smile as a way of securing ongoing adult attention.
This action, meaning, “I like this…please continue,” is an early social skill upon which survival depends. It is the way that human infants and children engage others in interaction that comforts, excites, interests, and teaches. It is essential for ongoing development. Children with ASD often have difficulty using affect (in this case, a smile) to continue a pleasurable or interesting social experience. They are often ineffectual in their expression of pleasure. Therefore, it may be important to target this skill at the beginning of your intervention. For a child who is developmentally more advanced, you might write the following goal: Child will respond appropriately to common expressions of emotion in family members/familiar individuals in the classroom.
Many children with ASD have difficulty knowing how to respond when someone in their life is angry or sad or excited or scared. They might become distraught when someone is happy or become silly when someone is sad. They might become withdrawn at the expression of any strong emotion. Learning to react appropriately and with some social grace is an important skill not only within one’s family but also at school and in the community. As a final example, consider this goal as representing advanced work in developing appropriate affect: To achieve precision and subtlety in the expression of emotion, Child will use qualifiers to describe gradation of emotional experience (e.g., really disappointed versus a little disappointed).
In learning to understand affect in self and others, it is important not only to learn the basics but also the gradation of each emotion. Emotions occur on a scale for most people. Sometimes they are a little happy, sometimes very happy, and sometimes ecstatically happy. Typically developing children figure out common emotional scales without difficulty. Children with ASD don’t. They tend to see things as black or white, good or bad, always or never. It is important to help children with ASD develop the skills for understanding and regulating
24
IDENTIFYING GOALS
emotional response according to a scale that is similar to the scales of their families and school friends.
Self-regulation Having difficulty remaining alert, focused, and available for learning is a well-studied component of ASD. The ability to control the flood of emotional response when angry or sad or frustrated is also compromised in many of our children. There are entire curricula that help teach children to regulate their state of arousal. Your child may be getting extensive help with this area of difficulty from another professional. However, you might want to include goals from this area in your work with the child as well because difficulty with self-regulation affects your child in the home or in the classroom. Prior to writing this kind of goal, imagine that your team commented on how long it takes the child in question to recover after becoming distressed. The behaviors that express distress in this child linger long after the distressing event is over, interfering with the learning process. So your team decides to target this problem. The team may have collectively figured out four different strategies for calming. They will, perhaps, rock the child in a rocker. They will also repeat calming phrases, such as “It’s okay,” many times. They will give the child a stress ball to squeeze and will help the child focus on a favorite picture. After introducing these strategies to the child, the team can then measure the success of this process by looking for a decrease in the time it takes the child to recover and regroup after an incident that caused distress. Thus, you might write a goal such as: Child will recover from distress within 5 minutes with help from a familiar adult.
Over time, however, you might want to see this child become more independent in his or her self-calming. So a later goal might be: Independently using any or all of four different self-calming strategies, Child will recover from distress within 5 minutes.
Self-regulation is a problem in children with autism. Each child you meet may have a different set of challenges in this area. In Appendix A there is a brief list of suggestions pulled from intervention plans that the authors reviewed. These goals are basic enough to be useful in a number of cases, particularly since they are written in general terms. Of course, you must figure out what each child’s exact areas of difficulty are and create specific goals that reflect the needs of that child.
25
FROM GOALS TO DATA AND BACK AGAIN
Play Children with ASD often have significant difficulty with all or most aspects of play. Play is a pivotal skill in young children. It provides the content of learning and the method by which children practice and solidify information. As a designer of intervention, you need to approach this topic wholeheartedly. In general, you might consider the following areas as targets for intervention: · Learning to enjoy social play. · Developing a repertoire of playful activities. · Beginning to pretend. · Moving from concrete to abstract thinking. · Using drawing as a vehicle for creativity.
A list of common goals that correspond to these general topics is collected in Appendix A. For the purposes of discussion, however, we’ll just look at how to write goals that target social play, pretend play, and drawing.
Social play
Let’s consider a developmentally early goal: Child will joyfully participate in sensory-motor play with a familiar adult.
Sensory-motor play is the earliest form of social play in early childhood. It is not dependent on language or on rules. It rarely depends on sophisticated routine. It is just joyous, sensory-based interaction. Every child needs to begin here—even the child who is good at puzzles or computer games. A more advanced goal might focus on the need to expand the child’s repertoire of playful activities. Children with ASD can be very narrow in their interests. To explore the world in all its complexity, a child needs to get involved with a wide range of playful activities. A goal that reflects this might be: Child will expand his/her play repertoire to include manipulation, sensory-motor activities, art, music, building/construction activities, and early cognitive activities (matching, sorting, and puzzles).
This goal is designed to measure a steadily expanding “vocabulary” of behavior. Progress is indicated by an increasing range of skills. This method of measuring progress is discussed in Chapter 3. The purpose of the goal, however, should be clear. It is important to expand your child’s horizons, giving him or her a myriad of opportunities that promote learning.
26
IDENTIFYING GOALS
Pretend play
Pretend play is a complex skill that is noticeably compromised in most children with ASD. Lack of pretend play is, in fact, a diagnostic marker of autism. In order to teach a child to pretend you need to be aware of the typical sequence with which this skill emerges in young children. Starting at the beginning, your first goal might be something like this: Child will develop interest in the content of pretend play as opposed to the simple mechanics (e.g., interest will move from how the bottle fits into baby’s mouth to helping hungry baby).
Many children with ASD are fascinated by the way things work. They examine toys with the eye of an engineer and the skill of a mechanic. It is a difficult step to move from interest in the movements of a little train’s wheels, for instance, to moving the train around the room making train noises and imagining a real train experience. Children with ASD need help with this transition. When they have achieved this goal with your persistent modeling and encouragement, a new world becomes available to them. Later in the process of promoting pretend play skills with a child, you might write the following: Child will participate, with adults, in increasingly elaborate make-believe, moving from concrete themes (involving everyday, common experience) to abstract themes (involving material never experienced).
This represents a shift into fantasy—the ability to create a mental image of something new and different. This skill is needed for all later learning when the school-age child is presented with information foreign to his or her experiences. In order to learn about pioneer life, for example, a child needs to be able to conjure up a life that has little relationship to the child’s own experience. School-age children learning about pioneers must be able to imagine a world without electricity or cars. Typically developing children can do this, some better than others, because of the imagination that has been developing from the time they were 18 months old. Many children with ASD, however, need to be taught how to pretend. It doesn’t just happen.
Drawing
A final area that might be targeted in your attempt to promote play skills in children with ASD is drawing. Drawing is important because it is another vehicle for creativity and abstract thinking. It is an activity that children often do together and it is a requirement for most children who enter kindergarten. For a variety of reasons, children with ASD are often stunted in their ability to draw and create representations (some children with ASD, however, are very gifted in this area). If your team decides that this skill needs to be strengthened in a partic27
FROM GOALS TO DATA AND BACK AGAIN
ular child, they should be aware of how drawing progresses in the typically developing child. For example, your goals could progress in the following sequence: ° Child will scribble with crayon. ° Child will imitate drawing of vertical line. ° Child will imitate drawing of circle. ° Child will add three parts to incomplete human drawing. ° Child will copy drawing of square. ° Child will draw unmistakable human with body, arms, legs, feet, nose,
eyes, and mouth.
Communication For many children with autistic spectrum disorders, learning to communicate meaningfully and effectively opens the door to the social world. But learning to communicate is often a huge challenge. As educators, parents, and therapists, we are aware of the complexity of teaching someone to communicate. We are also aware that children with ASD don’t always respond to the cues that allow a typically developing child to acquire communication so easily. We can’t just provide an environment rich in language. We can’t simply count on incidental learning of language. There are too many barriers for children with autism. What we can do to teach children with ASD to communicate is break down the process of becoming an effective communicator into small steps and explicitly teach the steps one at a time. This process is not, however, just about the production of language. It is about social connection, all the way from a meaningful look to a whole sentence. In teaching children to communicate, it is helpful to divide the communication goals into the following categories that address the particular problems associated with the diagnosis of ASD: · eye gaze · receptive communication (understanding language) · expressive communication: body language and affect · expressive communication: using symbols · conversational skills (pragmatics).
A lengthy list of common goals that correspond to these sections is provided in Appendix A. For the purpose of discussion, however, let’s look at the categories listed above in greater detail.
28
IDENTIFYING GOALS
Eye gaze
Eye gaze is an early and effective means of communicating. It is the vehicle for two processes known to be problematic in children with autism—joint attention (“Hey, look at that…Did you see that?”) and social referencing (“Is this stranger at the door safe? Is it okay to climb on this shelf ?”). These two processes are thought by some to be the key areas of compromise in children with ASD because it is through joint attention and social referencing that we learn to establish meaning for events. Meaning is constructed, researchers tell us, from three sources of information: 1.
what we see, hear, and feel
2.
what we remember and
3.
what we learn from the feedback our parents, teachers, and therapists give us about a situation.
Consider this: you are a very young child. A stranger comes to your home. Your senses tell you there is someone new in the room. Your memory tells you that this person is unfamiliar, which creates feelings of unease. You then seek more information by looking at Grandma. She is smiling. She looks at you and the stranger and continues to smile. Her feedback interprets the event for you as safe. You adjust your initial reaction and welcome the new person into your environment. How did you construct meaning for this event? You used your senses, your memory, and social referencing. Looking at other people to see if they are experiencing what you are experiencing is a crucial mechanism for attaching common meaning to your world. Attuning your reactions to the reactions of others is a key process in the development of social and emotional skills. Both of the above require consistent eye gaze, which is often difficult for children with ASD. To begin to address this complex area of deficit, you might consider writing the following goal: Child will look back and forth to make sure adult sees what child sees in order to share enjoyment.
or: Child will look towards adult to make sense of an ambiguous situation.
These goals can be shaped into measurable objectives, as discussed in the next chapter. Don’t ignore them because they seem hard to measure. They are, perhaps, the crux of intervention. The lack of joint attention and social referencing may be the point of divergence between typical and atypical development in the young child with autism.
29
FROM GOALS TO DATA AND BACK AGAIN
Receptive communication (understanding language)
It is quite easy to write goals that track language understanding in highly practiced tasks or highly simplified environments. What is less straightforward, however, is writing goals that look for the emergence of language understanding in context. Despite the difficulty of measuring progress in this way, we must keep in mind that the underlying cognitive change we want is not well characterized by behavior in which a child points to a picture in response to a familiar question. The goal of language intervention is increased understanding as part of the child’s natural world. That is where behavior that indicates underlying change will emerge. As an example: Child will look for family members when asked, “Where is Mommy?” or “Where is Daddy?”
This goal not only highlights an early step in the development of receptive language, it also puts that step in the context of the child’s social world. The typically developing child is driven to keep track of important people. This is part of normal social and emotional development. Teaching a child with ASD to understand “Where is Mommy?” or “Where is Daddy?” not only supports the development of communication but heightens the child’s awareness of his/her parents. To further illustrate this point, here are more goals that measure progress in receptive skills as they occur in a situation that is meaningful to both the child and the caregivers. ° Child will respond to his/her name. ° Child will stop action in response to “No!” ° Child will appropriately respond to the command “Stop!” ° Child will move in response to a one-step direction. ° Child will take object or food to someone when requested. ° Child will indicate approval when asked a “Do you want?” question. ° Child will appropriately respond to simple and familiar “Where?” ques-
tions with searching movements.
Expressive communication: body language and affect
It is important to target expressive communication as it occurs before symbolic language develops. Babies, for instance, are great communicators. They use body language and facial expression to communicate complex ideas. There is much to learn about communication before we use words. Consider this goal, for example:
30
IDENTIFYING GOALS
Child will respond to gestures with intentional gestures of his/her own (e.g., reaches out in response to outstretched arms).
or: Child will indicate disapproval using gestures and body language.
Both of these highlight the need to communicate effectively long before language is expected. It is important to work on improving communication while respecting the developmental process. If you emphasize the use of words, for instance, when there is little to no non-verbal expression, you may create communicators that are rote and ineffective in the real world.
Expressive communication: the use of symbols
Typically developing children begin to communicate using symbols with apparent ease. We label things for children and praise them for their fine efforts in repeating those labels, but beyond that, we are merely amused observers of the process. Typically developing children simply begin to talk one day and keep on practicing until they have perfected the process. Children with autism, however, often have great difficulty learning to use symbols for communication. Whether you are using signs, pictures, or words, you are essentially teaching that these almost random items represent real objects or actions in the world. The word “chair” represents all those things you sit upon. The word “swing” represents that wonderful activity you do with Dad. Does it make sense? It doesn’t, really. It is simply a system of symbolic representation upon which society depends. So how do we begin to teach symbolic communication? We teach symbolic communication in the context of the child’s daily life, duplicating, as best we can, the typical developmental process. We teach in small steps, with meaningful strategies. We help the child with autism understand the fundamental concept of symbolic reference through actions that are meaningful to that particular child. The following list represents a small sample of developmentally early goals that target the process of learning to communicate with symbols. Additional goals are included in Appendix A. ° Child will learn fill-in-the-blanks of familiar songs, rhymes, and/or
familiar verbal routines (e.g., “Ready, set, go”).
° Child will use word/sign/picture for “more.” ° Child will indicate that he/she is done with an activity by saying or
signing, “All done.”
° Child will develop consistent vocabulary of symbols used in the absence
of concrete gestures (e.g., child will come into the dining room and say “juice” to mother to request juice without needing to take mother to refrigerator and touch the juice bottle). 31
FROM GOALS TO DATA AND BACK AGAIN
° Child will respond to question, “What’s this?” ° Child will ask question, “What’s this?” ° Child will spontaneously add single words to play, beginning to narrate
play actions.
° Child will ask for help.
Conversational skills/pragmatics
Just because a child with ASD can use many words does not necessarily mean that he or she can use these words meaningfully in conversation. The art of reciprocity is a separate and vital part of mastering language. Choosing the right words and saying them the right way so that you and your conversational partner remain “in sync” is something most of us do relatively easily. Individuals with autism rarely, if ever, find this process effortless. We teach social reciprocity the same way we teach everything else, by breaking it down into manageable pieces of learning, and writing goals that represent small, developmentally sequenced steps. For example, we might begin with something as simple as: Child will use attention-getting words such as “Hey!” when beginning a conversation with a peer in school.
or: Child will use appropriate distance between self and conversational partner.
As the child is becoming a more proficient communicator, we might add the following: Child will use eye contact to signal that he/she is finished talking and that it is now the partner’s turn to talk.
Finally, as the child is becoming quite skilled, you may still have to refine his or her skills with goals such as: Child will initiate conversation that is of interest to social partner.
or: Child will ask questions that are related to the topic, even when the topic is not a preferred subject, to maintain the conversational flow.
32
IDENTIFYING GOALS
Sensory issues In children with autism, there are many differences in the functioning of the central nervous system. For the purpose of this section, the relevant differences relate to how a child reacts to and processes sensory experience as well as how a child plans and organizes responses. Children with ASD typically have difficulty with the following: · Sensory modulation: they might be over- or under-reactive to touch,
sight, or sound.
· Sensory processing: they might have difficulty understanding language
or interpreting visual-spatial, tactile, vestibular, or proprioceptive information.
· Motor planning: they might have difficulty carrying out movement
sequences.
As developmental interventionists, we need to write goals for children who are experiencing difficulties with sensory issues in their home, in their school, and in the community. As we target the impact of sensory issues in the child’s environment, other professionals might be building skills in a more isolated context such as a therapy room. With intervention concerning sensory issues, and perhaps in all types of intervention, a team approach is essential because there are often two avenues for teaching skills. On the one hand, the child needs to learn and practice, one by one, the specific sub-skills that are embedded in each complex task. On the other hand, the child needs to learn when and how to use those sub-skills together and in the actual contexts in which they are required. It’s like teaching someone to play basketball. First you teach the person to dribble, to shoot foul shots, and to do lay-ups. You might need to teach him or her how to defend the basket and how to score. But that is only part of what the person needs to learn. The other piece he or she needs to become proficient in is the experience of putting the bits of learning into action in the appropriate situations. The person needs to practice playing the game. As a teacher, parent, or therapist, you are the one that helps the child “play the game” with sensory issues. You need to identify the areas of compromise and while your team members work to teach isolated skills in their therapy rooms with special equipment or exercises, you help the child put those skills to use together. As a result, your goals might look like the following examples: ° Child will eat a greater variety of foods. ° Child will gain comfort with activities in which his/her feet are off the
ground.
° Child will tolerate proximity of other children. ° Child will become sensitized to, and appropriately label, hot/cold/pain. 33
FROM GOALS TO DATA AND BACK AGAIN
° Child will walk around toys, pets, and people on floor. ° Child will successfully avoid bumping into people. ° Child will develop compensatory strategies for feeling comfort while in
large, open spaces.
° Child will become more comfortable with activities that involve hands
and face.
° Child will become more comfortable with multiple voices singing.
Restricted interests and perseverative behaviors Having restricted interests and/or perseverative behaviors is one of the hallmarks of ASD. Children at both ends of the spectrum can exhibit a need for sameness and a need for ritual, repetition, and control. Many researchers have documented that the incidence of idiosyncratic, solitary behavior decreases as the child becomes more invested in, and comfortable with, the social world. The more the child is engaged, the less the child perseverates on solitary activities. What is our role, then, as interventionists, in this particular phenomenon of autism? We must work to make the child’s world more comfortable. Sometimes we need to alter the environment so that the barriers to overcoming sensory differences are lowered. Sometimes we need to give the child other actions that comfort, yet are interactive. But we always keep the child engaged. Two examples of goals addressing perseverative or idiosyncratic behavior are: Reciting passages from books, videos, TV, and/or radio will decrease. Child will stay focused on shared conversation with caregivers instead of lapsing into private reference.
We write goals like these because we want the idiosyncratic behaviors to decrease. That is the change we want to observe over time. What that change should indicate, however, is not that we have finally eradicated a strange behavior, but that we have increased the child’s comfort and willingness to remain connected to his or her social world.
Concept development There are many concepts that we assume will come into focus if we simply give a child enough opportunities to experience information. But what educators, parents, and therapists have noted is that providing a concept-rich environment isn’t enough for some children with autism. Concept learning typically happens as a by-product of social interaction, something that is difficult for children with ASD. Therefore it takes structured, creative, and strategic teaching to help a child with autism learn common concepts such as time, quantity, or emotions. 34
IDENTIFYING GOALS
As interventionists, we target these apparent gaps in the cognitive profiles of our children. We note that although they can, perhaps, read and do math, they don’t categorize information in common ways, they don’t understand the passage of time, and they remain very puzzled by “Why?” questions. We therefore write goals that highlight the gaps in understanding that we observe in each child. We might write the following: Child will demonstrate understanding of function of familiar objects by selecting the correct item or insisting on the correct item when “mistakenly” given the wrong item.
This goal seeks to teach the function of familiar objects as these functions arise over the course of the child’s day. Perhaps the parent begins to brush the child’s hair with a shoe. What does the child do? Perhaps the therapist puts on slippers when getting ready to go outside, or attempts to open the door with a spoon. What does the child do? According to numerous theorists, the child will more easily retrieve this information in the future when both active problem solving and humor have been involved in the initial learning phase. Consider this goal: To demonstrate an understanding of locative state and prepositions, Child will be able to answer “Where?” questions.
This is a sizable task. It is not the beginning step in teaching the child about where things exist in his or her world; it is a final goal that targets the consistent and meaningful use of these skills. When you ask Johnny where his Woody doll is because together you need him for the house you just built out of Lincoln Logs, his answer will alert you to how well he understands the concept of locative state. That is what you are measuring—language that indicates a comprehension of certain concepts.
Increasing awareness of others This section is a follow-up to the first area for intervention, “Attention and basic social relatedness.” The goals discussed here target more developmentally advanced skills along this dimension. So, while before we were concerned with the child in the small, sheltered world of his or her home or classroom, now our expectations concern a larger world—one that includes various adults, neighbors, and peers. As interventionists, we need to alert the children with whom we work to the various facets of social reciprocity. In order to be truly connected to another human, one needs to be aware of what that other person is doing, is feeling, is seeing, and is hearing. One needs to have clues into what that other person might be thinking and what he or she probably knows. These are essential skills for the child within the family, school, and neighborhood. 35
FROM GOALS TO DATA AND BACK AGAIN
Goals for this topic might include the following: ° Child will note what others are doing and shape his/her behaviors
accordingly.
° Child will predict what others might see or hear in a given situation. ° Child will demonstrate an awareness of the needs of others by spontane-
ously offering help.
° Child will receive a daily compliment for being considerate. ° Child will predict what others might think or feel in a given situation.
Social skills with peers What a formidable task it is to figure out how to break down the process of making and keeping friends into little goals! Making friends is a complex human phenomenon with a rich array of skills required at every step. Some of the children with whom you work may need extensive help in developing social skills with peers and some may need just a few refinements to modesty or bossiness or empathy. You may, for example, need to target the following: Child will successfully initiate conversation/play with peer.
Such a goal is appropriate for children who don’t know how to begin. Even though they may be skilled at asking their parents or siblings to play or talk, these children become uneasy when confronted with kids on a playground or in a classroom. They need, with your help, to learn to invite another child into interaction. They need to make an overture to another that is effective and successful. As you help them learn to do this, you can measure their progress. Later, you may designate the following as the next step: Child will be able to join others already engaged in a play activity or in a conversation (as opposed to having a peer join him or her).
This seems to be a much harder skill for children with ASD. Joining something that is already happening takes a certain melding of behavior. You need to slide in, making few, if any ripples. You need to do what the others are doing so that you are included in their activity. How very difficult for children who feel more comfortable when they have some degree of control. Another aspect of social skills with peers is learning behavior that is conventional in a social setting. You might need, for instance, to teach the child the following: Child will apologize when he/she bumps into someone.
or: 36
IDENTIFYING GOALS
Child will develop tactful responses to describe dislikes and disagreements as they arise during interaction with peers.
This entire topic, “social skills with peers,” is vast and complex. Keep in mind that many of the other sections target related skills; look in the areas of play, expressive communication, conversational skills, respecting social norms, and school and camp skills. Finally (and always), remember that the team that surrounds the child will be the best designer of intervention. Goals may well emerge from the team that are better suited for a particular child than the goals highlighted above or those found in Appendix A.
Respecting social norms Hand in hand with increasing social awareness and developing social skills, most children with autistic spectrum disorders need to be taught standard social expectations. It is easy to assume that the children with whom we work will figure out what they should do and what they shouldn’t. After all, they are surrounded in school by children who know these basic rules, so won’t they learn them, too? The answer is often “no.” Learning to conform to the unwritten rules of social behavior is difficult. It is difficult because children with autistic spectrum disorders are not always aware of common expectations, and if they are aware of these social conventions they may not be motivated to restrain an impulse that runs counter to convention just because their friends might not approve. Think about nose picking, for example. In our experience, a certain child may be aware that a friend frowned upon his or her nose picking last time. But if his or her nose is uncomfortable, that feeling seems to take precedence over any social obligation to follow the rule “Don’t pick your nose in public.” For this rule and others, parents, teachers, and therapists end up teaching the rules of social conformity as the need arises. A formal goal for this particular issue might be: Child will wipe nose on tissue and throw tissue away.
You might find yourself needing to teach lessons in ownership and/or privacy, as demonstrated in the following goals: Child will demonstrate an understanding of ownership by refraining from taking someone else’s food or belongings.
or: Child will demonstrate an understanding of modesty and/or privacy by being fully clothed when leaving the bathroom in public places.
More goals that address common problems in learning about social norms can be found in Appendix A. 37
FROM GOALS TO DATA AND BACK AGAIN
School and camp skills As designers of intervention, we are often in the position of helping children learn a set of skills for successful functioning in a variety of environments. One of the greatest developmental challenges for all children is learning to move from the security and predictability of one’s home and family to the new demands created by multiple environments with multiple caregivers. The skills that are required in school, at camp, or in the community are often unlike skills needed at home or in a quiet, familiar setting. New environments mean new demands. What parents, teachers, and therapists need to do, once again, is figure out together which new demands are posing particular strain on the child in question and design intervention accordingly. You might need to target basic group skills such as: Child will raise hand when wishing to speak and will wait until he/she is called on before speaking out loud.
You might need to target skills specific to the difficulties commonly experienced by children with autistic spectrum disorders, such as: Child will tolerate changes in the schedule when prepared for these changes.
Finally, you might need to design goals that are specific to one child only because no one child with autism is ever identical to another child with autism. Your goal could look like this: Child will tolerate being at both the middle and the end of the line.
This child obviously had difficulty with line position. Is this true for many children? No. Is it true for many children with autism? Probably not. Is it true for at least one child with autism? Yes.
Leisure Leisure is defined by the American Heritage Dictionary as a noun meaning “rest” or “freedom from time-consuming duties, responsibilities, or activities.” If our American concept of leisure were just that, children with autistic spectrum disorders would be skillful masters of leisure time. In America, however, our concept of leisure time for a child is partly defined by doing things with family. Usually, children with autism are not skillful masters of the family trip. The family outing lacks predictability and the presence of novelty can be alarming. Simply leaving the house on the weekend can be irritating for some children with ASD. It is usually the parents of the children with whom we work that define leisure goals. They are the ones that know what needs to be learned so that the 38
IDENTIFYING GOALS
child can experience life as the rest of the family experiences it. They are the ones that point out the skills, big or little, that seem to be missing. We can help families improve leisure skills in their children. It takes the same strategy that everything takes—breaking down expectations into little steps and working on them one by one. Leisure time skills, like everything else, can be written into goals that can then be measured to mark progress over time. Here are a few examples: ° Child will increase repertoire of tolerable family outings. ° Child will keep caregivers informed of where he/she is going during
outings.
° Child will join routine family activities, from start to finish, without
needing to be continually prompted to stay involved.
° Child will join non-routine family activities, from start to finish, without
needing to be coerced.
° Child will be able to tolerate winning and losing.
Things to remember This was a long chapter with a lot of information to absorb. Still, the key ideas are straightforward: T Goals must be written in a way that reflects basic development growth. T Even though we are measuring small behavioral changes, these
behavioral changes represent shifts in the child’s social, emotional, and cognitive development.
T The goals suggested here and in Appendix A are only a beginning in the
process of designing intervention. The actual goals for an individual child are always a team effort that pinpoints specific and often unique areas of need.
39
Chapter 3
Writing Measurable Goals
Recognizing and writing developmentally appropriate goals, like those in the previous chapter, isn’t the same thing as recognizing and writing measurable goals. In our experience even the most well-meaning intervention teams often write goals or objectives that don’t mean what the team intends the goals to mean, and/or can’t be measured in a meaningful way. For example, we recently came across a goal that was written for a young boy who was receiving home-based services. The goal read: George will not be aggressive 80% of the time.
The flip side of this statement is that George can be aggressive 20% of the time and still meet the goal! Twenty percent of a 24-hour day is 4 hours and 48 minutes—meaning 4 hours and 48 minutes of aggression each day was OK. This child, we knew, slept 12 hours a night. He obviously wasn’t aggressive while sleeping. He also went to school for 7 out of the remaining 12 hours of the day and we knew that he wasn’t having any difficulties with aggression in school. So out of the 5 hours of time he was awake and not in school, George would have reached his in-home therapy goal if he remained non-aggressive for a mere 12 minutes. Is this what the team intended? We don’t think so. They may have meant: George will refrain from becoming aggressive with the therapist (kicking, biting, or hitting) four out of every five times he has an emotional outburst in the home.
or they may have meant: During therapy in the home, the frequency of physical aggression towards the therapist will diminish to only one aggressive act (hitting, biting, or kicking) every five sessions.
Or they may have meant something else altogether. If George’s team had written the goal in a clear, specific way, there would have been no confusion concerning 41
FROM GOALS TO DATA AND BACK AGAIN
either the meaning of the goal or how to measure it. That’s what this chapter is about: giving you the skills you need to write goals that reflect the actual behavior you want to measure in a meaningful way.
Change in the child and change in the goals What are we measuring when we write developmental goals for children with autism? Only one thing: change over time. When we work with a child what we want to see is a change in targeted skills and behaviors that may take days, weeks, or months to produce. We are not comparing groups of children to each other or comparing a child to some standard. We are concerned only with a single child’s progress over time. Much of the difficulty in understanding how to approach data collection for developmentally based intervention comes from two different ideas of “progress over time.” On the one hand we need to write goals that reflect changes in small, concrete pieces of behavior. We expect that these kinds of changes can be accomplished in a modest amount of time; this is progress on a small time scale. On the other hand, the knowledge and skills that underlie the behaviors—what we’re most interested in—may take a longer time to change in a general and lasting way. This is progress on a larger time scale. We unify the two time scales by realizing that every large-scale skill or behavioral change must be led through three distinct phases. In the first phase, the child demonstrates certain small-scale behaviors in a single, familiar environment with one teacher or caregiver. Then the child must progress to the second phase in which he or she demonstrates consistent use of the behavior whenever it’s appropriate. Finally, the child must progress again to a phase in which he or she is consistently using the skill in multiple environments with multiple caregivers and natural distractions. This means that each small, concrete goal we write is actually related to other goals in the child’s past and future as part of a deeper, more lasting kind of change. By viewing progress against this backdrop we begin to see how to take large developmental changes and make them measurable. What does the process of balancing small-scale and large-scale progress look like? You and your team begin by figuring out what the child needs to learn next. To help with this decision you might use a curriculum or a set of developmental guidelines. Suppose, for example, that Johnny’s team decides to target the social practice of waving goodbye. Within this large-scale goal there are at least three small-scale goals that follow the child’s progression through the phases of learning. Let’s look at that progression for Johnny, and for Johnny’s goals.
42
WRITING MEASURABLE GOALS
Phase 1: Emergence
In Phase 1 the goal statement describes the emergence of a skill given multiple and supported opportunities in a sheltered, consistent, therapeutic environment. So the first concrete instance of a small-scale goal contributing towards Johnny’s large-scale goal might be written: With multiple prompts and frequent modeling, Johnny will begin to wave goodbye to family members and caregivers leaving his home.
Because learning is most challenging in this phase, you begin at home or in a small classroom with a consistent caregiver. To the extent possible the interactions are based around the child’s interests and natural pursuits. You begin to work with the child in a manner that is tailored to his/her particular learning needs. You offer rigorous support and constant rewards in an effort to help the child move into new skill areas. In short, during the emergence phase, the targeted skills are taught and re-taught, supported and maintained. Only when a child shows progress in this setting under these circumstances does the purpose of intervention refocus.
Phase 2: Consistency
In Phase 2 the goal statements focus on consistency: the frequent and repeated practice of the behavior given specific surroundings. The next concrete instance of a small-scale goal contributing toward Johnny’s large-scale goal might be written: With a decreasing need for prompts, Johnny will wave goodbye to familiar adults and his siblings.
Phase 2 statements reflect your aim to have the child be able to retrieve the skill predictably and reliably. At first you might prompt for the new behavior at every possible opportunity, but once the child easily retrieves that piece of learning when it is needed, you pull back on your prompting so that the child becomes more independent. You may use drills for teaching, you may set up situations for learning within the child’s natural environment, or you may rely on incidental “teachable moments.” However you do it, this is the stage for practice. When a child shows consistency and independence in this setting and under these circumstances, the purpose of intervention is refocused again.
Phase 3: Extension
In the final phase of large-scale progress on this goal, Johnny’s team might write:
43
FROM GOALS TO DATA AND BACK AGAIN
With decreasing need for all prompts, Johnny will wave goodbye in home and school environments when someone who is leaving waves to him or says “goodbye” first.
Followed sometime later by: In all environments, Johnny will consistently wave goodbye when someone who is leaving waves to him or says “goodbye.”
In other words, in Phase 3 the small-scale goals target the extension of the skill to multiple environments and the degree of independence in those environments. You need to make sure the child can do whatever you expect him to do in the grocery store, at the playground, and in school. You also make sure the child can do whatever you expect him to do with other adults, siblings, cousins, neighbors, and peers. Increased support may be required temporarily in each new environment just as familiarity of environment may be required temporarily with each new person. But the temporary reintroduction of support should not be considered a step “backward.” It is part of the natural progression of large-scale change. Overall, the final phase of intervention helps the child take an independent and consistent skill out into the world. Only then is the large-scale goal complete. In designing developmentally based intervention, there is clearly a flow to the learning process that must be reflected in the goals you write. Charts 3.1 and 3.2 in Appendix B summarize the process visually. While understanding the flow is critical, it is not enough. To write small-scale measurable goals (like the examples in this section) we need to know more than just the phase of the large-scale goal they represent. We need to know both if we are looking for emergence, consistency, or extension and under what conditions we should see the emergence, consistency, or extension. We need to know the Who, What, When, and Where of each small-scale step. To get measurable we need to get specific.
Writing goals that can be measured Chapter 2 gave you some ideas about how to write goals that reflect both typical development and the areas of compromise in children with autistic spectrum disorders. The first section of this chapter demonstrated how to set the small concrete goals we work with day-to-day into a larger perspective. But once the team has created a list of objectives that represent the current needs of a certain child—each objective clearly identified as being in a particular phase—how do you shape each goal so that it can be measured reliably? You shape a goal first by qualifying each behavioral outcome and then by quantifying it. To qualify a goal you must be specific about the variables that surround the desired outcome. If you are unclear about the specific conditions under which the
44
WRITING MEASURABLE GOALS
behavior is to occur, the meaning of your data will be unclear as well. In particular, to qualify a goal you must add language that designates: · Where the behavior is to occur. Do you expect the child to exhibit
this behavior in a sheltered and familiar environment? If so, which environment? Do you mean familiar environments that are free of unexpected events? Or do you expect the child to show this behavior in community settings where things can be unpredictable? Qualifying where the behavior should occur means including phrases like “at home,” “in the regular classroom,” or “in multiple community settings.”
· When the behavior is to occur. It might be difficult to measure a new
behavior over a long period of time (like an entire school day) so you should limit the time that you measure a particular goal. Sometimes you might need to measure a targeted behavior during the same time each day to control for changing variables. Other times you might need to measure something in intervals throughout the day to look at the effect of changing variables. In either case, you need to be specific about your time segment. This type of qualification is made by phrases like “during ‘free play’,” “during transitions between classes,” or “in the 2-hour period before bed when Johnny is playing with his siblings.”
· Who observes the desired behavior. Do you expect it to emerge only
with the most trusted adults (“with Mom or Dad” or “with the instructional aide”)? Or do you think the child is ready to show that behavior to many people in his or her community (“with any child on the playground”)? You need to be specific.
· The level of support the child needs to demonstrate the behavior.
Demonstrating a behavior with multiple prompts is a very different skill from demonstrating that same behavior spontaneously. Although the degree of prompting is intimately tied to where the small-scale goal falls in the larger learning progression, specificity is still important. Do you expect the child to need multiple prompts because it is a new and difficult skill (“with multiple prompts”)? Do you expect that the child will need only a single prompt? Or are you intending that the child demonstrate the desired behavior independently, using only natural cues (for example, “when someone waves first”)?
· The type of measurement to be collected. Are you asking the person
collecting the data to estimate? Do you want to know what the best outcome was for the day’s work or do you want to know what the average outcome was? The more specific your goal statement is, the more certain the meaning of the measurements.
45
FROM GOALS TO DATA AND BACK AGAIN
After you’ve qualified it, you need to quantify the goal by choosing a system for measuring progress. Having reviewed many treatment plans for children with ASD, we have identified the following systems of measurement as both practical and meaningful for developmental outcomes. For each goal, choose one of the following types of quantification: · Increasing or decreasing frequency of behavior. You might want to
see a child wave goodbye more and more often when Mom leaves for work. You might want to see fewer and fewer incidents of self-talk in the middle of social play at school. This is the most common system for measurement. It is not, however, the only option you have.
· Increasing or decreasing duration of behavior. You might want to
see a child sustain attention to playful encounters for longer and longer periods of time. You might want to see a gradual reduction in the amount of time it takes a child to respond to his or her name.
· Increasing a range of behavior. This type of measurement is used
when you want to see the gradual acquisition of a set of skills over time. You use this system when you are looking for a child to expand his or her repertoire of play, or when you are measuring, over time, a child’s ability to respond favorably to a variety of people. However, a growing vocabulary—linguistic or behavioral—can’t be measured on a single day. This is a system that measures an accumulation of skills over time.
· Decreasing need for assistance/prompts. This is often the system of
measurement that is used when it is time for the large-scale goal to move from emergence to improvement and finally to consistency. If a child previously needed multiple prompts in order to answer a “Why” question, for example, and now only needs an expectant pause, that is progress! If a child used to need hand-over-hand assistance to wash his or her hands and can now do it with only a set of visual reminders, that is progress! A bonus to using this method of marking progress is that it makes the team aware of how they prompt. Becoming complacent with your level of support is never good for the child.
The process of qualifying and quantifying your goal statement makes it clear to everyone on the team both what you are measuring and how you are measuring it. To complete the process, however, you must choose a specific scale, unit, or quantity that will be used to record the child’s behavior. Each method of quantification allows for different types of observations, as we detail in the following sections.
46
WRITING MEASURABLE GOALS
Ways of recording increasing or decreasing frequency Frequency can be measured in one of three ways: by tally of occurrence (a quantitative scale), by percentage (another quantitative scale), or by a qualitative scale. Each method has appropriate circumstances for its use, as explained below.
Using quantitative scales
A quantitative scale measures progress by recording increasing or decreasing quantity. One type of quantitative scale, the tally of occurrence, is used for measuring progress in the development of a skill when opportunity is constant. Let us show you an example of such a goal. Goal: With increasing frequency, Peter will show pleasure by smiling during structured play with his aide and a small group of peers. Data question: How many times in a ½ hour of structured play did Peter smile to indicate pleasure? 0 1 2 3 4 5 >5
Each example in this section has one part labeled “goal” and one part labeled “data question.” The data question lays out the scale or set of alternatives for the way of measuring progress specified in the goal. We will explain data questions more fully in Chapter 4; for now we are focusing on the bigger picture. In Peter’s case, the team wants to see the emergence of a new behavior. Since a new behavior is most easily learned in a familiar environment the qualifying phrases “with his aide and a small group of peers” and “during structured play” are added to detail the specifics. The general quantifying phrase in the goal statement (“with increasing frequency”) is made specific in the data question (“times in a ½ hour of structured play”). Since there are unlimited opportunities for Peter to smile within the specified environment, a tally of occurrence is an appropriate system of measurement for this goal at this stage. Now let’s think about how this qualified and quantified goal relates to intervention with Peter…maybe Peter only rarely showed pleasure when this goal was written. During the first week of intervention, perhaps Peter smiles only once. The next week, there are three times when he smiles. The following week, Peter not only smiles at his aide, but begins to smile at the other children as well. Do you see what is happening? If you were keeping tallies of occurrence, your data would have registered change over time. Let’s think about another example. Shawnna is a child who has a very difficult time pretending to be anyone other than Shawnna. Her sisters want to play dress-up and Shawnna never joins them. Shawnna is awkward with little figures and doesn’t join her sisters when they play with dolls. A goal is written for Shawnna that is paired appropriately with a quantitative scale:
47
FROM GOALS TO DATA AND BACK AGAIN
Goal: With regular modeling from siblings and adults during therapy sessions in the home, Shawnna will take on the role of another person in play with increasing frequency. Data question: How many times did Shawnna pretend to be someone else while engaged in pretend play with siblings and therapist? 0 1 2 3 4 5 >5
Here again the focus is on the emergence of a new behavior. The goal statement is qualified with phrases indicating where and when (“during therapy sessions in the home”), who (“siblings and adults”), and level of support (“with regular modeling”). Within those constraints, Shawnna has a constant opportunity to initiate the new behavior, so the team chooses a tally of occurrence. The quantifier in the goal statement (“with increasing frequency”) is made specific in the data question. How does this goal relate to intervention and change in Shawnna’s behavior? The team wants to see if they can expand Shawnna’s play skills with modeling so that she can play more easily with her sisters. They set out to work on this every session, modeling and encouraging. The individual collecting data looks for incidents in which Shawnna participates in or initiates some form of role-playing and tallies the occurrences. As Shawnna’s comfort with role-playing increases so will her tally. In contrast to the tally of occurrences, the second type of quantitative scale, percentage, is used for any desired behavioral change in which frequency of opportunity is also measurable. Percentage is calculated by dividing the number of times a behavior does occur by the number of opportunities in which it might have occurred. For example: Goal: With increasing frequency, Peter will respond by saying “Hi” when familiar people say “Hi” to him. Data question: What percentage of the time did Peter respond by saying “Hi” to familiar people who came to his home during his morning therapy session? How many people came to his home? ______ How many times did Peter say “Hi”? ______ Calculate # of greetings ÷ # of opportunities x 100 = ______%
or: Goal: Needing only a single verbal prompt, Shawnna will wait appropriately for her turn on the playground equipment. Data question: What percentage of the time was Shawnna able to wait for her turn at the playground, needing only a single verbal prompt?
48
WRITING MEASURABLE GOALS
# of opportunities: ______ # of times Shawnna waited: ______ Calculate # of waits ÷ # of opportunities x 100 = ______%
Be aware that using percentages as your method of measurement in marking progress can be problematic in many instances. We have observed too often that teams rely on taking a percentage measurement even when it doesn’t make sense. We want you to be familiar with the four serious problems that can be associated with collecting data in this way: 1.
With goals that focus on frequently occurring behaviors, it is unlikely that a therapist or teacher has the luxury of counting all of the opportunities for the behavior in order to calculate the percentage correctly. Each time the observer fails to note that the child missed an opportunity the percentage becomes a little less accurate. Under these circumstances the ratio of successes to opportunities becomes inflated; the child appears to be far more masterful than he or she truly is. Estimating percentages is a common practice that is almost guaranteed to produce inaccurate data. If your team needs to estimate a percentage because counting the child’s successes and dividing them by the number of opportunities for success is too cumbersome, then you should use another method of measuring progress.
2.
Percentages can also be problematic for infrequently occurring behaviors. If there are only one or two opportunities for the target behavior each time you work with the child, the child’s behavior will be easy to record but the percentage data will be too crude to be informative. Why? Consider this: if there is only one opportunity, the child either behaves as desired (receives 100%) or does not behave as desired (receives 0%). If there are two opportunities, the child can score 0%, 50%, or 100%. These figures simply do not give enough gradation to accurately represent the child’s behavior in a meaningful way because the large jump between the possible values does not correspond to large differences in the child’s mastery. As the number of opportunities grows, the use of percentages does become more appropriate…but only until the opportunities become too frequent to count, as explained in 1.
3.
If the number of opportunities fluctuates due to the nature of the developmental skill you are trying to teach, percentages may be misleading. Perhaps Shawnna got a 50% one day and a 40% the next 49
FROM GOALS TO DATA AND BACK AGAIN
day. On paper, this looks like she is regressing slightly. But perhaps she showed a certain new behavior once on the first day (out of two opportunities) and four times on the second day (out of ten opportunities). Even though the percentage dropped, we might want to consider this progress simply because Shawnna managed to show the behavior four times. That information was lost when you calculated the percentages. 4.
It is possible to write goals that try to compensate for problems 1–3 by targeting behaviors that can be measured frequently but not too frequently, and by making sure each session with the child has about the same number of opportunities. Such an approach means letting your data analysis method determine your intervention style. The premise of this book is that the decision of intervention style must come first and is independent of the decision to collect data. Once the commitment to a particular style of intervention has been made then data collection and analysis methods appropriate to that intervention can be learned and practiced.
Using a qualitative scale
A qualitative scale is used for rating behavior that exists on a continuum. Of all the methods we discuss in this chapter, this technique for collecting data on developmental progress can result in the most specific and useful data. Yet we have found that teachers, parents, and therapists rarely use it. Why is this method under-utilized? Perhaps teachers and therapists worry that only a scale that has been established independently can be valid. This is not correct. You know the particular child in question and you understand that development is a process. So put those two areas of knowledge together to devise a series of steps that goes from not being able to do something all the way to being able to do something. By creating a scale of steps that is individualized to a specific child, the data you collect will be more valuable than data collected on a general scale. The four examples that follow are intended to give a feel for how to use qualitative scales to measure progress: Example 1: Goal: With increasing frequency, Peter will match his behavior to the behavior of his peers. Data question: How often did Peter watch what his friends were doing and do it too?
50
WRITING MEASURABLE GOALS
Rarely, even when prompted 1 About half the time, prompted Most of the time, prompted A few times, independently Many times, independently
Example 2: Goal: Shawnna will become more comfortable with messy art activities, that involve using her hands, when they are presented to her in the classroom. Data question: Please rate Shawnna’s participation level with activities involving tactile experiences. Very resistant Somewhat resistant A little resistant Guarded participation Full participation
Example 3: Goal: Isaac will begin to engage in parallel play during “free play” at preschool. Data question: Did Isaac engage in parallel play? Isolative throughout “free play” During gross motor activities only During gross motor and sensory activities During all activities, including playing with trains
Example 4: Goal: With increasing frequency, Beth will learn to use words to describe her distress while completing homework and chores in the evening. Data question: For the majority of the time, how did Beth communicate distress? TANTRUM 1
CRYING
WHINING
VERBALLY
We hear you asking yourself, “Why is it all right to estimate a percentage (‘about half the time’) here when you just finished telling me it was bad practice?” Excellent question! The estimate asked for on this qualitative scale is a choice among a set of broad categories that can be judged accurately. Most adults can accurately distinguish between events that have occurred rarely, about half the time, and most of the time. This is a very different sort of judgment than distinguishing accurately if an event occurred 25% of the time or 30% of the time or 60% of the time. (We’ll explain the difference between qualitative scales and percentages in more technical terms in Chapter 6.)
51
FROM GOALS TO DATA AND BACK AGAIN
As the examples show, the more specific you are about what you expect in the way of progress, the more informative your data results will be. Most systems of measurement look at whether the child did or did not do something with little regard for the process of going from A to Z. If you take the time to figure out the steps that the child will take to accomplish a new skill, you educate everyone who uses the scale. Further, when you look back on the child’s progress and your team notes that the goal has not yet been achieved, a qualitative scale gives you considerable feedback about where the child has stopped progressing. A good team takes this information seriously and plans new strategies accordingly.
Ways of recording increasing or decreasing duration Frequency is not the only aspect of learning that is important. Even a child who shows progress in knowing how and when to use a skill may not participate in the activity or behavior for a long enough period to fully benefit from its rewards. For this child, the team may decide that it is time to focus on duration. When we want to see behavior demonstrated for either longer or shorter periods of time we need quantitative scales that measure duration. Such scales need to be paired with qualifying phrases that clearly ask for average, longest, or shortest instances of the target behavior. The following examples are presented to give you a feel for measuring progress in this way: Example 1: Goal: In the home environment, Peter will stay engaged with familiar adults for increasing lengths of time. Data question: On the average, how long was Peter able to stay engaged with you today? 2–4 min.
5–7 min.
8–10 min.
>10 min.
Example 2: Goal: As Shawnna becomes more social, periods of time in which she chooses to remain self-absorbed and self-contained will diminish. Data question: When given a choice to be alone, what was the longest time before Shawnna sought the company of her mother or therapist? >15 min.
~15 min.2
~10 min.
The symbol “~” means “about” or “approximately.”
2
52
~5 min.
<5min.
WRITING MEASURABLE GOALS
Example 3: Goal: Miguel will learn to tolerate the presence of a dog from a distance without growing anxious or needing to leave the area. Data question: What is the longest period of time that Miguel tolerated the presence of a dog from a distance? 1–5 min.
5–10 min.
10–15 min.
15–20 min.
<20 min.
When considering using increasing or decreasing duration as a system for measurement, you will need to consider increments of time for your scale. Are you looking for an increase or decrease in 2-minute increments? In 5-minute increments? In half-hour increments? This depends completely on the child and what you are measuring. In the case of Peter, 2-minute intervals were picked for his scale because interacting with others is hard for him and it is realistic to expect this to improve in small increments of time. On the other hand, Shawnna is already becoming more social, already demonstrating an interest in reconnecting with familiar people after isolating herself briefly. We expect her interest in isolative play to diminish a little quicker than Peter’s ability to stay engaged increases. Why? Because they are different children in different phases of treatment. Shawnna’s skills are already improving. Peter’s are just emerging. The time intervals in each case reflect the difference in our expectations for change for each child. Let’s think about this with some other examples. If you are working with a child who occasionally presses his or her eye when bored and you want this behavior to diminish as you teach new, more appropriate behaviors to manage boredom, you might measure increasing intervals of time between eye presses. For this child, the appropriate time intervals might be 10 minutes. You know perhaps that he easily goes 10–15 minutes without pressing his eye and as a measure of progress, you want to see the time between eye presses expand to 20 minutes, 30 minutes, and 40 minutes. For another child, however, who is very reactive to noise, you might want the child to tolerate being in the school cafeteria for a minute, then 2, then 3, with a goal of only 5 minutes. For that child, a 1-minute interval might be most appropriate. Just remember that the time intervals you pick will depend on the skill you are targeting and on the child’s individual profile.
Ways of recording an increasing range of behavior A child who participates in an activity for an age-appropriate period of time may, nevertheless, be restricting his or her range of behavior. The child who always wants to play the same game with trains, the child who will play in parallel with peers only at the art table, and the child who describes every event as having occurred yesterday (even when it happened months ago) have at least one thing 53
FROM GOALS TO DATA AND BACK AGAIN
in common. They are all ready to work on large-scale goals to increase the range of their behaviors. Measuring an increasing range of behavior is different from the other systems of measurement we’ve seen. In this type of system, we are interested in steadily increasing a child’s “vocabulary,” as shown in the two examples that follow. Example 1: Goal: During “free play” at school, Peter will expand his repertoire of play to include gross motor, construction/building, puzzles, art, music, and early pretend play. Data question: Please circle the types of activities Peter participated in today during free play: gross motor art
construction/building music
puzzles early pretend play
Peter needs to expand his repertoire of play. For example, he may ask for the computer every day and the teaching staff want to see him, over time, try all different kinds of activities. They don’t expect him to do all six activities every day but their goal is that he eventually participates in all six. So an appropriate way to measure this goal is with an accumulating list. If they noted only how many things Peter did on a given day, they would never be able to figure out if he was doing the same things or different things. By working to complete a list over time, Peter’s team can ensure that he is expanding his interests. In another example, we meet Shawnna again. Shawnna is beginning to understand time. Her vocabulary for time-related words, which indicates her growing knowledge, is expanding. By the end of the school year, Shawnna’s team wants to know that she can use a list of 12 time-related words and phrases. They don’t expect Shawnna to use all 12 words or phrases in a single conversation, but over time they expect her to have used each one of the targeted words or phrases in a way that reflects appropriate understanding of its meaning. They write this goal: Example 2: Goal: To demonstrate a growing understanding of time and sequence, Shawnna will spontaneously use at least 12 time markers in conversation with her speech and language therapist. Data question: Please circle the time markers Shawnna used appropriately and meaningfully today:
54
WRITING MEASURABLE GOALS
now morning today
later afternoon tomorrow
before after night yesterday a long time ago
soon
One of the important things to remember here is that the “vocabulary” to be mastered must be part of a developmentally sound sequence. If you want a child to acquire a set of skills, a set of concepts, a set of words, or a set of activities, choose the set carefully. You don’t want to throw together behaviors that are unrelated to the skill level of the child. Nor does it make sense to expect a child to acquire a range of behaviors that includes skills normally acquired at very different ages.
Ways of recording a decreasing prompt level Monitoring the level of support, as opposed to frequency or duration, tends to be an intermediate step in the process of goal acquisition. It is a useful way to measure progress after a skill has emerged in the child’s repertoire of behaviors but before this skill is independent or reliable. This measurement method is especially important in this population because children with autism who are taught intensively to produce certain behaviors can become dependent on artificial prompts, something often referred to as “prompt dependence.” Therapists, teachers, and parents can also become complacent in their use of prompts, forgetting to push or even nudge the child towards greater independence. It is important to remember, however, that if you use a decreasing need for prompts to measure progress, you must individualize the qualitative scale of prompts according to the needs of the child, the learning task, and the situation. Example 1: Goal: Miguel will use a visual schedule to complete a five-step task (setting the table). Data question: What level of prompt was needed to help Miguel use a visual schedule as a guide in setting the table? needed frequent verbal and gestural prompts needed a few verbal and gestural prompts needed a single verbal prompt needed simple gestural prompt independent
55
FROM GOALS TO DATA AND BACK AGAIN
Example 2: Goal: With a decreasing need for prompts, Peter will wave goodbye to familiar adults in multiple environments. Data question: What level of prompt was needed to help Peter wave goodbye to familiar adults? verbal and physical prompt (hand-over-hand) verbal with modeling verbal only no prompt needed
Example 3: Goal: Isaac will sit quietly “like a pretzel” at circle time with a decreasing need for reminders. Data question: How frequently did Isaac need to be reminded to sit quietly “like a pretzel” at circle time today? aide needed to sit behind, frequent prompting aide sat nearby, few prompts given aide faded, teacher prompted a few times teacher reminded whole group to sit still sat appropriately without reminders
Let’s think about these children for a minute and why decreasing the level of prompting was chosen as a method for measuring progress. Each child—Miguel, Peter, and Isaac—has learned a certain targeted skill in Phase 1 of his treatment for that goal. Miguel had never set the table before but with full support from his mother or therapist he began to complete this task. Now the team wants him to become more independent with this chore (they are moving the goal into Phase 2), so they reduce the level of human-driven prompts by helping him to use a visual schedule. Similarly, Peter has learned to wave goodbye with a lot of support (Phase 1). Eventually, we want him to be able to take the natural cue from a departing family member or friend without our additional support. We want him to respond consistently when someone says, “Bye bye, Peter” and waves. This is a Phase 2 goal. We are reducing the artificial prompts in order for Peter to use an environmental cue to wave goodbye. Finally, in Isaac’s case, we want him to sit correctly at circle-time with only the natural reminder, if needed, from his teacher. We want to decrease the support needed from the aide and increase first his reliance on the teacher for guidance (reliance shared by his classmates) and finally on himself to sit back down independently should he get out of the “pretzel” position. We are moving him from a stage of using the targeted skill with persistent external support such 56
WRITING MEASURABLE GOALS
as the aide (Phase 1) to a stage of natural support and independence (Phase 2). In helping children move from Phase 1 to Phase 2 of goal acquisition in order to master large-scale change, it is very helpful to use decreasing need for prompts as the method for measuring progress. Chart 3.3 in Appendix B presents the material we have covered to this point in the chapter in a concise format. You may find the chart useful as a reminder of the steps in writing measurable goals when designing intervention or explaining the process to others.
Considering goal-independent factors When we measure progress we expect changes in a child’s behavior to be related to the effectiveness of our intervention. But children with autism are often greatly affected by variables that compromise performance even though these variables have nothing to do with the particular skill being taught. A child who is not feeling well, for instance, may communicate less and become harder to engage. Parents have told us repeatedly that when their child is getting ill, he or she can be thrown off course for days before and for days following the illness. We know that changes in routine, changes in environment, changes in teachers, even minor changes in the appearance of teachers can alter a child’s responsiveness as well. The time of the day and the time of the year can be influential. We’ve known many children who have difficulty adjusting to the colder weather, indoor heat, and decreasing light of autumn and winter, or to the early warmth of spring. Understanding and monitoring these goal-independent factors can make an enormous difference in knowing how to interpret the data you collect. Some of the variables that affect performance may be unique to a specific child. You will have to learn these from parents and other team members and through your own history with the child. However, there are common factors to consider monitoring with any child. These include: · overall assessment: availability to engage/learn · who is doing intervention: familiar/unfamiliar · where intervention is done: familiar environment/new environment · day of the week (or day of the “cycle” in some schools) · time of day: morning versus afternoon versus evening · changes in routine · medication: type and time of last dose.
Tracking these kinds of typical goal-independent variables allows you to note, for instance, that the days that a desired behavior is suddenly absent are Tuesday and Thursday—days when the child has a different aide in school. They allow 57
FROM GOALS TO DATA AND BACK AGAIN
you to see, perhaps, that the child is unsuccessful every afternoon when medication is wearing off. Or that the desired behavior is lacking on Mondays when the child returns from a three-day weekend. This information is valuable to the team, allowing you to understand the flow of progress with a keener eye or compensate for differences in your data due to factors like fatigue, illness, and novelty. Just as important, these variables can give you information about what does negatively affect the child and interfere with learning. The team can use that information to devise new treatment goals and/or make adaptations to the interfering variables in order to produce more effective intervention. Exercise 3.1 provides you with an opportunity to practice the skills presented in this chapter. It can be found in Appendix C and in the file “Goal Workbook.doc” on the CD-ROM that accompanies this book.
Things to remember T Developmentally driven objectives can be written into goals that are
measurable.
T Intervention for children with ASD measures only one thing: change over
time.
T There are regular phases to the treatment process. Children begin to
learn a new skill in a single, sheltered setting but before they become fully accomplished they need to demonstrate the skill consistently in multiple environments. The goals we write reflect this process, targeting the emergence, consistency, and extension of the skill over time.
T Every goal needs to be qualified. To qualify the goal you must be
specific about the variables that surround the desired outcome, indicating where, when, and with whom the desired behavior is to be demonstrated as well as the level of support needed for success.
T Every goal needs to be quantified by picking a system of measurement
that fits the particular child and situation. In our experience, one of four systems is adequate to the task: increasing or decreasing frequency of behavior, increasing or decreasing duration of behavior, increasing the range of the behavior, or decreasing the need for assistance or prompts.
T There are three methods for measuring increasing/decreasing frequency:
58
1.
Tally of occurrence (a quantitative scale of measurement) should be used when the behavior could occur any or all of the time.
2.
Percentages (a quantitative scale) should be used only in situations where the number of opportunities can be easily detected and does not fluctuate greatly in different sessions.
WRITING MEASURABLE GOALS
3.
Qualitative scales are used to measure progress that moves through a series of carefully defined steps. A valid qualitative scale can be based on the steps we expect the individual child to pass through on the way from not being able to do something to being able to do it.
T When you want a certain behavior to occur for either longer or shorter
periods of time, measure duration using a quantitative time scale.
T When you want to see a steadily increasing repertoire of skills you need
to measure the child’s range of behavior. The data accumulates new items as they emerge over time.
T Measuring progress by the decreasing need for assistance uses a
qualitative scale of prompts that reflects the needs of the child, the learning task, and the situation. This technique is particularly useful in the stage between the emergence of a desired behavior and the consistent display of the desired behavior.
T In addition to the goal data you collect, it is important to monitor
variables such as time of day, medication, and changes in routine. Such goal-independent factors can help explain uneven and unexpected patterns in a child’s behavior.
59
Chapter 4
Collecting the Data
In the previous chapter, you learned to take a simple statement defining developmental progress and shape it into something measurable by qualifying it and quantifying it. You are now familiar with various methods for quantifying goals and you know the importance of noting goal-independent variables that might affect the data. You also know that there is a flow and pattern to the process of designing intervention with three phases of treatment. In this chapter we discuss something that we have only touched upon in the examples. We are going to show you how to formulate data questions and collect the data. This is not difficult, really, but it does require asking yourself or your team questions about the child’s target behaviors that easily elicit valid responses.
Data questions You have already seen a number of examples of data questions in the sections on quantifying goals. Let’s look at some of these again: Example 1: Goal: Isaac will sit quietly “like a pretzel” at circle time with a decreasing need for reminders. Data question: How frequently did Isaac need to be reminded to sit quietly “like a pretzel” at circle time today? aide needed to sit behind, frequent prompting aide sat nearby, few prompts given aide faded, teacher prompted a few times teacher reminded whole group to sit still sat appropriately without reminders
61
FROM GOALS TO DATA AND BACK AGAIN
Example 2: Goal: As Shawnna becomes more social, periods of time in which she chooses to remain self-absorbed and self-contained will diminish. Data question: When given a choice to be alone, what was the longest amount of time before Shawnna sought the company of her mother or therapist? 15 min.
~15 min.
~10 min.
~5 min.
<5min.
Example 3: Goal: With increasing frequency, Peter will respond by saying “Hi” when familiar people say “Hi” to him. Data question: What percentage of the time did Peter respond by saying “Hi” to familiar people who came to his home during his morning therapy session? How many people came to his home? ______ How many times did Peter say “Hi”? ______ Calculate # of greetings ÷ # of opportunities x 100 = ______%
Example 4: Goal: With increasing frequency, Peter will show pleasure by smiling when with his aide and a small group of peers during structured play. Data question: How many times in a ½ hour of structured play did Peter smile to indicate pleasure? 0 1 2 3 4 5 >5
Example 5: Goal: With a decreasing need for prompts, Peter will wave goodbye to familiar adults in multiple environments. Data question: What level of prompt was needed to help Peter wave goodbye to familiar adults? verbal and physical prompt (hand-over-hand) verbal with modeling verbal only no prompt needed
62
COLLECTING THE DATA
Example 6: Goal: During “free play” at school, Peter will expand his repertoire of play to include gross motor, construction/building, puzzles, art, music, and early pretend play. Data question: Please check the types of activities Peter participated in today during free play: gross motor art
construction/building music
puzzles early pretend play
These are all examples of goals that have been turned into data questions. Each data question reflects both the content defined in the goal and the measurement system selected as most appropriate or useful. Each of these data questions, when answered, gives the team information about progress on that particular goal.
Formatting the data sheet It is important to put data questions into a format that is both easy to read and easy to use. In order to do so we specify the range of answers to the data question, except when requesting a tally or percentage. In this way, the person collecting the data can simply check the applicable answer or fill in an appropriate number. Figure 4.1 shows an example of what a data collection sheet would look like for three of Peter’s goals using our format. Remember that the skills of the individual child and the complexity of a targeted behavior for that child will determine the range of answers. Whether it is intervals of time, levels of prompting, or steps that go from not being able to do something to being able to do that thing, the range of answers that defines a scale are designed for the individual child. For the person checking off answers, the scales act as reminders about what steps this child will take on the road to change. Therefore, the data sheet not only serves the purpose of providing the data collector with an easy system of data collection, it also serves as a reminder about where this child is going and how he or she will most likely get there. Although writing out the goals with the range of possible answers can be time consuming for the team initially, this method serves an important function. It establishes clear definitions of what goals are being measured and how the goals are being measured. In some cases it also indicates the process the child might follow on the way to goal completion. Having this information on hand is critical to consistent data collection either by a single individual across sessions, or by more than one observer working on the same goals with the same child but at different times. Sometimes the person doing the intervention has no opportunity to collect information on a particular targeted skill. If you suspect that there might be 63
FROM GOALS TO DATA AND BACK AGAIN
Child’s name: Peter Smith
Observer’s name:____________
Date:______
Time:______
Please check the types of activities Peter participated in today during free play:
o o o o o o
What percentage of the time did Peter respond by saying “Hi” to familiar people who came to his classroom during his morning preschool session?
How many people came to his classroom?______
gross motor construction/building puzzles art music early pretend play
How many times did Peter say “Hi”?______ Calculate: ______% (# of greetings ÷ # of opportunities x 100)
What level of prompt was needed to help Peter say goodbye to familiar adults?
o Verbal & physical prompt (hand-over-hand) o Verbal with modeling o Verbal only o No prompt needed Figure 4.1 A sample data collection sheet in an easy-to-use format
times when you or a member of your team cannot measure a particular goal, it is important to put that option on your data collection form. You might add a box labeled “Not applicable” or “No opportunity for observation.” This makes it possible to distinguish between times when there is no opportunity and times when the child is unable to perform. One of Shawnna’s goals is an appropriate example: When given a choice to be alone, what was o >15 min. the longest amount of time before Shawnna o ~15 min. sought the company of her mother or o ~10 min. therapist? o ~5 min. o <5 min. o No opportunity for observation
The treatment team wants to measure Shawnna’s growing sociability by monitoring how long she remains self-involved when no one is pushing her to engage. Yet the therapist’s job is to work intensely with Shawnna and she doesn’t often allow Shawnna to disengage for long periods of time. On a day 64
COLLECTING THE DATA
when intervention lasts right until the therapist needs to leave, there would be no opportunity to measure the length of time Shawnna stays disengaged by choice. By adding the box labeled “No opportunity for observation” the data explicitly reflects this decision by the therapist. The importance of this point is discussed further in Chapter 6. In summary, the format for data questions must be chosen to promote and facilitate accurate observation. It is important that your data collection sheets be both easy to read and easy to use. Unless you are taking a tally or collecting numbers to calculate a percentage, your job is to create a set of data questions that can be answered by checking a box or circling the appropriate answer. This requires careful forethought about exactly what should be observed so that the data you collect is an informative measure of the child’s progress to the goal. Exercise 4.1 reviews the features of Microsoft® Word that are needed to create data sheets like the ones seen throughout this book. For additional examples of data sheets designed for individual children in different settings, see Appendix B.
Frequency of data collection Here is a general rule for deciding how often to collect data: more is better. With all children in general, and children with ASD in particular, it is essential that you do not base a conclusion about developmental progress on sporadic data. We all know that almost anything can throw off a child’s performance. Therefore, it is best to track progress steadily and consistently. Some teams with whom we’ve worked see a child daily but collect data only once a week. This system can lead to conclusions about progress that may not be valid. If, for instance, you work with a child daily but only measure progress every Friday, you might be collecting numbers that reflect accumulated fatigue or boredom for the task at hand. If you measure the progress on Wednesday when the normal routine for the child has been interrupted (perhaps Mom goes to work and a babysitter gets the child up and ready for school on those days), your numbers may be lower than usual and not accurately reflect the child’s skill level. The only way to avoid these problems is to take daily data and monitor for goal-independent variables so that you have a way of understanding the ups and downs in the child’s performance. Some teams with whom we’ve worked see a child daily in school but collect data only on Mondays, Wednesdays, and Fridays. Although this system seems to work fairly well (certainly better than a once-a-week system), it leaves the team little knowledge about Tuesdays and Thursdays. They could be missing some valuable piece of information that is only available on one of these other days, as shown in the following example:
65
FROM GOALS TO DATA AND BACK AGAIN
Years ago, we worked with a little boy named Jack whose school-based team tracked his progress on a daily basis by rating him on a scale of 1–5 (poor to excellent) for responsiveness to every subject of the day. What we noted within a few months of school was that his numbers looked significantly lower on Mondays and significantly higher on Thursdays than on the other days of the week. Why? Examining the data, the team discovered two things. On Mondays, following the weekend, Jack was very tired. It turned out that his parents regularly allowed Jack to stay up late on Friday and Saturday nights, sleeping late the following mornings. But on Sunday night, Jack always had difficulty falling asleep at the usual time for a school night. He would repeatedly get out of bed and often didn’t fall asleep until midnight. It was a struggle for his mother to get him up on Monday mornings and he was, the team decided, noticeably tired in school. As soon as the parents realized that they needed to keep Jack on a consistent schedule regardless of whether it was the weekend or not, his performance at school on Mondays improved. On Thursdays, Jack had music therapy as the first school activity in the morning. This session always appeared to calm and focus Jack. The team realized, through their data system, that every Thursday, all day long, Jack’s response to the learning opportunities at school was significantly better than on other days. The team speculated that music therapy exercises readied Jack for the school day and improved overall performance. In response, they were able to create a music period for Jack each morning. The period was designed by the music therapist but carried out by the regular staff. The team’s theory that music experiences readied Jack for school was proven correct, and Jack’s daily overall performance improved with his new schedule.
If Jack’s school team had collected data only on Mondays, Wednesdays and Fridays, they might have discovered and remedied the fatigue issue but they would have missed the relationship between Jack’s performance and his participation in music therapy. Every time you work with a child you have the opportunity to collect data. DO IT! If you have created data collection sheets that are easy to use the task will not be difficult or overwhelming. And if you have created data collection sheets that measure progress accurately the potential rewards for you and the child are enormous.
66
COLLECTING THE DATA
Special suggestions for teachers It is particularly difficult, teachers tell us, to collect data systematically and frequently for multiple children in a classroom. Reorganizing your data collection to be activity-oriented rather than child-oriented can help. For example, if you know that a certain activity is a good setting for data collection, aides or teachers can observe and take data on a variety of goals for a variety of children during that one activity or class period. To do this effectively, you need to have the different children’s goals pulled together on a single sheet,
Opening exercises Date:______
Observer:____________
Student
Data questions
Data 0 1 2 3 4 5 >5
Tanika
How many times did Tanika need to be reminded to remain in her seat?
Joey
Using fidget toys and adaptive o Needed frequent verbal prompts seating, was Joey able to keep o Needed minimal verbal prompts his hands quiet? o Needed visual prompt only o Hands were quiet
Juan
When responding to a question, what percentage of the time did Juan raise his hand instead of calling out the answer?
# of responses: _______________ # of times Juan raised hand: _______________________ Calculate: ______% (# hands up ÷ # responses x 100)
Figure 4.2 A sample data collection sheet for multiple children working at the same time on different goals
Date:______
Teacher:____________
Seatwork goal: Using a visual schedule, the following children will complete the targeted number of tasks independently.
Student
Target # of tasks
Harry
3
Cameron
6
Aishia
4
Tasks complete
Figure 4.3 A sample data collection sheet for multiple children working at the same time on the same goal
67
FROM GOALS TO DATA AND BACK AGAIN
as in Figure 4.2. The observations can then be transferred to each child’s Excel worksheet at a later time. Alternatively, you might design an activity that specifically targets the same goal in a number of children. Suppose, for example, that you want to teach all or most of your students how to complete tasks independently using a visual schedule. It then makes sense to set up time each day for independent work during which children complete tasks using visual support and you collect data using a sheet like the one in Figure 4.3. Once again, after the data has been taken you will enter the observations into each child’s separate Excel worksheet. Creating a sheet to take data for multiple children at one time simplifies the process of collecting the data regularly and requires a minimum of paper shuffling. It requires that you think about and take advantage of certain times of the day when the goals are actively being addressed and/or when the child has multiple opportunities to perform the targeted skills.
Monitoring goal-independent factors Each child has a different set of variables that might affect the steady display of progress. Some children might be thrown off by changes in routine, some by different times of the day, some by illness, and some even by weather or sunlight. When your team identifies a set of circumstances that might affect the child, it is important to begin to collect information about these factors along with the data 1 you collect concerning progress on the goals. How do you do this? You simply add a set of information-gathering questions at the beginning of your data collection sheet. For example, if Peter was a child who had significant ups and downs in his availability for social experience, you might want a global assessment of his availability each day you worked with him. Was it a day in which he was easily engaged? Was it a day in which he avoided interaction from the moment you came until the moment you left? Or was it a more typical day for Peter, in which he was a mixture of avoidant and sociable? Once you are tracking this variable you can consider its role in Peter’s progress. He may have achieved the goal in question but cannot retrieve the targeted skill when he is socially distancing himself. When you notice these circumstances your job changes to address the difficult days, rather than the average days. Figure 4.4 shows a data collection sheet for Peter, with the goal-independent factors marked by an asterisk (*).
We discuss some ways to uncover such factors in Chapter 9. Here we are interested in how you monitor variables that you have already uncovered or that you know to be likely candidates based on parent report, theory, or experience with other children.
1
68
COLLECTING THE DATA
Child’s name: Peter Smith
*Observer’s name:____________
*Date:______
*Time:______ 1 2 3 4 5
*On a scale of 1 to 5 (avoidant to sociable), please rate Peter’s overall mood for the day. Please check the types of activities Peter participated in today during free play.
o o o o o o
What percentage of the time did Peter respond by saying “Hi” to familiar people who came to his classroom during his morning preschool session?
How many people came to his classroom?______
What level of prompt was needed to help Peter say goodbye to familiar adults?
o o o o
gross motor construction/building puzzles art music early pretend play
How many times did Peter say “Hi”?_____ Calculate: ______% (# of greetings ÷ # of opportunities x 100) verbal & physical prompt (hand-over-hand) verbal with modeling verbal only no prompt needed
Figure 4.4 A sample data collection sheet showing goal-independent factors marked by an asterisk (*)
In Figure 4.4 we record an overall assessment of the day (defined here in terms of availability for social experience), who is doing the intervention and collecting the data, the day of the week, and the time of the day. In addition, we might be interested in where the intervention is taking place, whether or not there were any changes in routine that day, and whether the child was on medication. The sheet would then look like Figure 4.5.
69
FROM GOALS TO DATA AND BACK AGAIN
Child’s name: Peter Smith
*Observer’s name:____________
*Date:______
*Time:______ 1 2 3 4 5
* On a scale of 1 to 5 (avoidant to sociable), please rate Peter’s overall mood for the day. *Where did intervention take place?
o home o school o other:
* Were there any changes in routine today?
o yes o no If yes, please explain:______
* Did Peter take Ritalin today?
o yes o no What time did he take it?______
Please check the types of activities Peter participated in today during free play.
o o o o o o
What percentage of the time did Peter respond by saying “Hi” to familiar people who came to his classroom during his morning preschool session?
How many people came to his classroom?______
What level of prompt was needed to help Peter say good-bye to familiar adults?
o o o o
gross motor construction/building puzzles art music early pretend play
How many times did Peter say “Hi”? ____ Calculate______% (# of greetings ÷ # of opportunities x 100) verbal & physical (hand-over-hand) verbal with modeling verbal only no prompt needed
Figure 4.5 The data collection sheet from Figure 4.2 with additional goal-independent factors (*)
70
COLLECTING THE DATA
Figure 4.5 is an example of a data collection sheet that has seven common variables accounted for: · overall assessment of the day (defined in this case in terms of availability
for social experience)
· who is doing the intervention and collecting the data (here it is the same
person but not necessarily)
· day of the week (or day of the “cycle” in some schools) · time of day · location of intervention · possible changes in routine · medication.
Be aware that these are only common variables. Every child is unique, and because of this you might need to keep track of any number of individually designed variables that could be affecting this child’s performance. Goal-independent factors may be very useful or only occasionally useful (we discuss this further in Chapter 9). If you don’t track the variables for each session, however, you will find that it is extremely difficult to recreate them accurately in retrospect. It is far better to have the information available if and when you need it.
Things to remember In order to collect data, you must turn your carefully qualified and quantified goals into data questions. These data questions must then be posed to the parent, therapist, teacher, or other professional on a regular basis. T Data are the answers to the data questions. T Data questions reflect the carefully written goal. T The more specific you are in what you are measuring, the more
informative and useful the data.
T The more frequently you collect data, the more accurate your picture of
progress over time. More is better.
T The team is more likely to collect meaningful data if the data collection
sheet is easy to read and easy to use. Even though the data sheet is time-consuming to design, this “up-front” effort will be rewarded in the long run.
T In addition to monitoring progress on each goal, it is important to
monitor goal-independent variables that may affect treatment outcomes. 71
Chapter 5
Putting It All Together — Joey, Tyler, and Mai Lin
It’s time to put together what we’ve learned so far. Let’s look at three different children, whom we will call Joey, Tyler, and Mai Lin. For each child we’ll tell you a bit about where he or she is developmentally, then work with you to figure out a set of goals. We’ll shape these goals into something meaningful and measurable, design the needed data questions, and finally create a data collection sheet. By the time you’ve worked through these three cases, you’ll be ready to move on to data analysis. Although this chapter gives you a chance to practice goal writing and creating data questions from those goals to measure change over time, it does not discuss the intervention that actually creates the change. That discussion is for a different book! In this book, we want to teach you how to measure and document the results of your intervention. Don’t lose sight, however, of the fact that you can only measure progress if you have progress to measure. Your intervention is the key to the process. Before you meet Joey, Tyler, and Mai Lin, we suggest that you make a copy of all the worksheets for the exercises in this chapter. These can be photocopied from Appendix C (Exercises 5.1 through 5.8) or printed from the file “Goal Workbook.doc” on the CD-ROM that accompanies this book. If you have those sheets readily available, participation in this chapter will be easy.
Joey Joey’s parents describe their 3½-year-old son as a child who seems to have little need to communicate with either of them. He is happy just “doing his own thing.” Joey eats what they give him, he goes to bed without a problem, he tolerates all the daily routines, and when he is left alone, he appears content to spin the wheels on his large collection of cars and trains, or spin his collection of shiny items on the hallway floor. The only thing Joey requests regularly is a drink. He goes into the kitchen and gets 73
FROM GOALS TO DATA AND BACK AGAIN
his cup, which his parents usually leave in a spot that he can reach. If he can’t reach it for some reason, he whines and whimpers. The parents said that Joey would not come and find them for help. Instead, when they hear him whine or whimper, they come in to give him a drink. They wish that Joey would ask them for help more often. “I used to be proud of how independent my second child was,” said Joey’s mother, “but now I see that it isn’t so good.” Joey’s parents wonder if Joey will ever actually speak because he doesn’t even babble the way their older daughter did as a toddler. All he does, according to his mother, is make vowel sounds. His father pointed out that he rarely closes his mouth or lips except when he is eating and even then, it doesn’t look quite right to him. However, recently they have noticed that Joey is beginning to watch their mouths when they are making funny sounds and is trying to purse his lips to blow bubbles. They want to encourage this because they know he needs to use the muscles of his mouth and lips if he is ever to use speech. Joey likes to look at books. Although he mostly flips rapidly through the pages, he does pause and examine any pictures of trains or cars, and recently seems interested in pictures of food. His parents wonder if a picture exchange system would help him communicate while he is learning to talk. Joey loves physical activities, his parents report, and will stay engaged with them, even giving them sustained eye contact, when they push him in the swing. He seems happy to swing for hours, they say. He also loves it when they sit down with him with his collection of shiny objects. He sits and watches for relatively long periods of time if they just keep spinning the objects on the floor. His parents wonder how they could use these preferred activities to encourage communication. Joey’s parents say that more than anything they want to help Joey find a reason to communicate with them and a method that is within his current abilities. They feel strongly that they want to help Joey “make his mark upon the world.” They want to teach him that it is good to express himself and that they will listen.
Given this brief case history, what do you think might be a set of five goals for Joey that reflects what his parents have highlighted as most relevant to them and to him? To narrow the scope of this task, let’s focus all five goals on communication. Take a few minutes and jot down some ideas using the workbook page Exercise 5.1 (from Appendix C). Return here after completing Exercise 5.1. There are many different possibilities for goals for Joey. One group of professionals who looked at this scenario came up with the following list: 74
PUTTING IT ALL TOGETHER – JOEY, TYLER, AND MAI LIN
· To make his mark upon the world and to feel successful in
communicating, Joey will indicate preference when offered the choice of two options.
· To begin to use pictures to make requests, Joey will associate photographs
with preferred items.
· To encourage meaningful vocalization, Joey will attempt to say “go!” after
being cued with “Ready, set…”
· To develop more effective communication through contact gesturing,
Joey will begin to pull his parents to his object of desire.
· To encourage oral imitation as a precursor to oral language, Joey will
imitate mouth movements with increasing frequency.
As a set of goals, these are a good start. They do, however, need to be qualified and quantified further. Using the words in italic as the actual goals, try to shape the goals so that they become measurable (Exercise 5.2). Return here after completing Exercise 5.2. Here is one way to do it: 1. During the same 2-hour period each day, Joey will indicate preference 75% of the time when offered the choice of food or an object to hold or play with. 2. With increasing frequency, Joey will match photographs to preferred items when given the opportunity. 3. After hearing the phrase modeled twice, Joey will attempt to say “go!” after hearing “Ready, set…” during a familiar and preferred activity. 4. With a decreasing need for assistance, Joey will begin to pull his parents and/or familiar adults to his object of desire. 5. Joey will imitate the mouth movements of familiar adults with increasing frequency.
Now we have goals that clearly lend themselves to being measured over time with accuracy and precision. From these goals we compose our data questions. How would you do this? Try Exercise 5.3. Return here after completing Exercise 5.3. 75
FROM GOALS TO DATA AND BACK AGAIN
The team of professionals like you who originally identified Joey’s goals produced the data sheet below. Name: Joey Smith Name of therapist:____________ Date:______ Time:______ Location of intervention: home center On a scale of 1 to 5 (avoidant to sociable), please rate Joey’s overall mood for the day. 1 2 3 4 5
76
1. When offered a choice of food, or object with which to play or hold, was Joey able to indicate preference? (Use same 2-hour period each day when choices are routinely offered)
# of opportunities
# of successes
2. When given an opportunity, was Joey able to match a photograph to a preferred item?
o o o o
3. After hearing the language modeled two times, was Joey able to fill in the word approximation “go” after “Ready, set…” to start a preferred activity?
# of opportunities
4. What level of prompt was needed to get Joey to pull an adult to an object of desire?
o multiple physical and verbal prompts o a verbal prompt (“Show me!”) with outstretched hand o independently (a few times) o independently (consistently)
5. Did Joey attempt to imitate your mouth movements?
o o o o
Calculate:______ % (# of successes ÷ # of opportunities x 100) not today, even with modeling matched after modeling independently matched 1 or 2 times independently matched multiple times # of successes
Calculate:______% (# of successes ÷ # of opportunities x 100)
no attempts noted growing interest in mouth movements a few attempts multiple attempts
PUTTING IT ALL TOGETHER – JOEY, TYLER, AND MAI LIN
What do you think? How do these data questions differ from the ones you produced? Remember that the ones we’ve shown are not the only possible questions. There are many ways to ask valid questions. What’s important is that the questions measure a skill or behavior that you believe to be a good indicator of progress on the underlying goal. Joey is an example of a young child who has very little ability to communicate. He is clearly still in Phase 1 of treatment where his needs are intense and the scope of treatment must be narrowly focused on Joey, his family, and his home. Let’s look at Tyler next. He is a little older and has more communication skills.
Tyler Tyler is a 5-year-old boy who lives in a family where he is nestled between a 6-year-old sister and 4-year-old twin brothers. Tyler’s health is excellent except for allergies that are easily controlled with medication. The family came to us for advice on how to help Tyler, diagnosed recently with Pervasive Developmental Disorder (PDD), communicate more effectively. His parents describe him as being an integral part of their busy family life. “There is never a moment for him to be alone,” his mother said, “and someone is always talking with him.” Yet even within an environment rich with language and intense in social demands, Tyler’s social language is not developing normally or rapidly. One of the things that bothers Tyler’s parents the most is that he continually seems to talk to himself. Tyler recites parts of videos he loves and has memorized, he repeats conversations he has heard days before, and he likes to spell long words over and over again. Right in the middle of a conversation, Tyler will begin to say something totally out of context that has little or no meaning to his conversational partner. His parents think that this tendency towards private reference is a problem because communication should be a dialogue, they say, not a monologue. Another problem Tyler’s parents highlight is that when Tyler does talk to his family members, it isn’t very effective. He often fails to get his siblings’ attention before he speaks to them so his sister and brothers appear to ignore him much of the time. When Tyler comes to his parents to request something, he almost never looks at them, making it hard to know what he is requesting or to whom he is speaking. If they question him to find out the meaning of his communication or to require more information from him, Tyler appears to have difficulty responding. “It’s almost as if he shuts down whenever we start questioning him,” his mother admitted. His father notes that sometimes when he is busy fixing something and not paying attention to whether or not Tyler is talking, that’s when his son seems most willing to engage in verbal exchange. Finally, Tyler seems either overly passive (so different from their other children!) or easily upset. They feel that he needs to learn to effectively say “no” and “I don’t want that” and “not now” just like their other children. 77
FROM GOALS TO DATA AND BACK AGAIN
They feel that this would help Tyler control what happens to him and thus increase his level of participation.
Begin by writing down a few of your ideas for goals concerning Tyler’s communication needs. Return here after completing Exercise 5.4. The goals that were identified by Tyler’s parents with our help were the following: · To help Tyler sustain connection to his social world, incidents of private
reference occurring in the midst of social exchange will decrease in frequency.
· To help him communicate effectively with his siblings, Tyler will learn to
secure his family’s attention before talking.
· To help him effectively make requests, Tyler will look at his communicative
partner as he is talking, or right before he begins to talk, with increasing frequency.
· To help him sustain conversation, Tyler will be engaged in a daily verbal
exchange of increasing length in which the adult partner avoids questioning and quizzing, and relies instead on commenting, narrating, rephrasing, and expressing opinions.
· To help him maintain a more even temperament, Tyler will learn to
effectively and productively indicate disapproval or protest.
Now, just as you did for Joey, try to make these goals more amenable to meaningful data collection. Using the words in italic as the actual goals, shape the goals further so that they become measurable. In this case, write goals assuming that the family will be doing all of the intervention. Return here after completing Exercise 5.5. The original goals just needed a little tweaking. See if your tweaking looks something like ours. It’s okay if it doesn’t—there are a number of ways to qualify and quantify these goals so that progress can be measured.
78
PUTTING IT ALL TOGETHER – JOEY, TYLER, AND MAI LIN
1. During the 1-hour period of time when the family plays together each evening, incidents of private reference occurring in the midst of social exchange will decrease in frequency. 2. During the 1-hour period of time when the family plays together each evening, Tyler will learn to secure his family’s attention before talking at least half of the time. 3. Without needing to be reminded, Tyler will consistently look at his communicative partner as he is requesting food or drink. 4. With an increasing number of verbal exchanges, Tyler will have one conversation a day in which the adults use a facilitative approach (avoiding questioning and quizzing, and relying on commenting, narrating, rephrasing, and expressing opinions). 5. Without being coached by family members, Tyler will learn to effectively and productively indicate disapproval or protest.
What do the corresponding data questions look like? Create your version by doing Exercise 5.6. Return here after completing Exercise 5.6. Here’s our version. We designed this data sheet in conjunction with Tyler’s parents to mark his progress in communication skills within the family. At this time in Tyler’s life, he is both learning new skills (goals 4 and 5), extending skills to new environments (goals 1 and 2) and developing independence (goal 3). His goals span all three phases of treatment. This is not an uncommon picture for children who are progressing in their skill development.
79
FROM GOALS TO DATA AND BACK AGAIN
Name: Tyler Jones
Family member taking data:
Date:______
____________
Time:______ Did Tyler have his allergy medication today?
Yes
No
1 2 3 4 5
On a scale of 1 to 5 (self-absorbed to socially connected), please rate Tyler’s overall mood for the day. 1. Over a 1-hour period before bedtime when the family plays together, how many times did Tyler lapse into private reference?
o o o o
2. Over a 1-hour period before bedtime when the family plays together, how often did Tyler secure his listener’s attention before talking to him or her?
o rarely (needed prompts to do so) o only a few times o about half the time o more than half the time
frequently a few times once or twice not at all
3. On the average, when asking for food or drink, o needed multiple prompts o needed a single prompt did Tyler spontaneously look at you when making a request? o a few times without prompts o consistently without prompts 4. During a single conversation in which a facilitative approach rather than a directive approach was used, how many verbal exchanges were achieved? (tally) 5. Was Tyler able to effectively indicate disapproval/protest?
80
o remained passive or got upset without explanation o indicated disapproval with coaching from family o beginning to protest or say “no!” independently o consistently indicated disapproval when appropriate
PUTTING IT ALL TOGETHER – JOEY, TYLER, AND MAI LIN
Now let’s move on to our final example, Mai Lin.
Mai Lin Mai Lin is an 8-year-old with Asperger Syndrome. She is a gifted musician and an excellent math student. Mai Lin is enrolled in a typical 2nd grade class where she is doing very well. The teachers feel, however, that Mai Lin needs to refine her communication skills so that she can talk more effectively to the people around her. One thing that concerns the teaching staff is that with unfamiliar people, Mai Lin seems to forget the body language and voice tone that normally accompany conversation. She turns away from the listener while talking, talks in an inappropriately quiet or low tone, and her inflection practically disappears. This makes it difficult for Mai Lin to interact with other children at lunch and recess. Mai Lin also has difficulty making herself heard if her initial attempt isn’t successful. If a peer says, “What?” Mai Lin will repeat her words identically without making herself louder or clearer. If the friend says, “What?” again, Mai Lin will continue to repeat her statement or request like “a broken record.” The teaching staff feels that Mai Lin’s “poor mechanics” are impacting her ability to maintain social contact with her friends. In the middle of conversations, Mai Lin tends to suddenly switch topics to music, math, or her current favorite subject. The teachers think she may do this because she is more skilled talking about preferred topics. They would like Mai Lin to become more comfortable sustaining conversation on a range of common topics. Finally, Mai Lin needs to be prompted to make a request in the form of a question. She tends to simply make statements, such as “I can have a drink, please.” Her teachers want Mai Lin to use a question format independently for her requests.
When you write goals this time (Exercise 5.7) try to qualify and quantify them from the outset. For the sake of this exercise, let’s assume that all of Mai Lin’s intervention will occur in the school setting and will focus on communication. Return here after completing Exercise 5.7. Your goals may be different from the ones below but we do need to have a common set of goals for the rest of the exercise. So let’s use the following:
81
FROM GOALS TO DATA AND BACK AGAIN
1. With a decreasing need for reminders, Mai Lin will appropriately face unfamiliar people when she addresses them. 2. Needing no more than a single prompt, Mai Lin will increasingly use appropriate volume and inflection when speaking to unfamiliar people. 3. With consistency, Mai Lin will effectively repair her communicative attempts after a peer says “What?” or indicates in another way that Mai Lin’s words were not understood. 4. Once a day, Mai Lin will sustain general conversation for at least 5 minutes without attempting to suddenly introduce a preferred topic. 5. Needing only an expectant pause as a reminder, Mai Lin will make requests using a question format rather than a statement at least 75% of the time.
Next, design the data questions, making decisions about how to measure Mai Lin’s progress. Return here after completing Exercise 5.8. Once again, your data collection sheet may be different from the one we are going to suggest. Compare and see what you think. Did you remember to add goal-independent factors to your data sheet? Mai Lin is an accomplished young girl. We are mostly “fine-tuning” her skills so that her communication becomes more socially cognizant of and responsive to her partner. Mai Lin is now primarily in Phase 3, learning to extend her skills to multiple people in multiple environments. Note that as she is learning to generalize her skills to the school environment and to multiple peers, Mai Lin might need assistance again (something associated with earlier phases of treatment) until she has learned to generalize her social communication.
82
PUTTING IT ALL TOGETHER – JOEY, TYLER, AND MAI LIN
Name: Mai Lin Wu
Teacher/Aide’s name:
Date:______ Time: ______
____________
Were there any unexpected changes in the school routine today?
Yes
No
If yes, please explain: 1. How much prompting was needed to help Mai Lin appropriately face new or unfamiliar people when speaking to them?
o unable to face unfamiliar person, even with prompt
o o o o
needed multiple prompts needed a single verbal prompt needed a single physical prompt no prompting needed
o no opportunity for observation 2. After a single prompt, if necessary, did Mai Lin use appropriate volume and inflection when speaking to unfamiliar people?
o o o o o
continued in quiet monotone even after prompt appropriate volume/inflection, 1 prompt, 1 or 2 x appropriate volume/inflection, 1 prompt, consistent appropriate volume/inflection, no prompts, 1 or 2 x appropriate volume/inflection, no prompts, consistent
o no opportunity for observation 3. Did Mai Lin effectively repair her communicative attempts after a peer said, “What?” or indicated in another way that Mai Lin was not understood?
o o o o o
skill emerging with assistance from adults skill emerging with multiple cues from peer skill emerging with 1 additional cue from peer repairs attempts herself about half the time consistently repairs communicative attempts
o no opportunity for observation 4. During “talk time” in which Mai Lin has to talk to a teacher for 5 minutes on a variety of common subjects, how many times did Mai Lin attempt to switch the conversation to a preferred topic? 5. What percentage of the time did Mai Lin make requests in the form of a question—as opposed to a statement—needing only a facial cue (expectant pause) as a reminder?
tally of occurrence: # of times: ______
o no opportunity for observation # of requests made:______ # made in correct form or repaired:______ Calculate: ______%
(# of successes ÷ # of opportunities x 100)
o no opportunity for observation
83
FROM GOALS TO DATA AND BACK AGAIN
Things to remember T The process of figuring out appropriate goals, shaping the goals into
measurable objectives, and designing data collection systems gets easier and easier the more you do it.
T We deliberately made these examples complex in order to give you a
chance to see a range of problems and a range of solutions. Many of the goals you target for your own clients or students will be less complex and the process of collecting data will be correspondingly simpler.
T The more information you provide to the person responsible for
collecting the data about what he or she is measuring and how to do it, the better the data will be.
T Even though your data sheets may seem lengthy or wordy, detail and
specificity is what narrows the focus of your intervention and makes it measurable.
84
Chapter 6
From Data Collection to Data Analysis
We collect data to see if there have been measurable and reliable changes in a child’s behavior. We assume that when such changes exist they are a result of intervention and indicate some deeper changes in the child’s knowledge or emotional repertoire. How do we know when the data indicates that there has been change? And how do we know whether that change is reliable? Answering these questions is the topic of this and the following four chapters. It is what data analysis is all about. Before we jump into the particulars, however, we want to remind you of an important difference between this book and other publications in which data analysis is discussed. Some of you may be familiar with terms like t-test, F-test, ANOVA, and p-value from reading journal articles, taking courses in statistics, or being part of an academic research study. You will not find these concepts explained or used in this book because most of the statistical measures you may be familiar with from academic research simply do not apply under the conditions created by real intervention. We don’t want you to change how you do intervention in order to make it appropriate to use these tests. We do want you to use the tests that are appropriate for the kind of intervention you do, even if those tests are less mathematically rigorous. That said, let’s begin.
Types of data Chapters 3 through 5 took you through the steps involved in data collection, the process that turns a desire for behavioral change stated in general terms into a set of questions that can be answered by observing a child’s actions. Data analysis is the process that turns those observations back into meaningful statements about changes in the child’s behavior over a particular period of intervention. An unfilled data collection sheet represents the set of measurements we might make. Each time we work with a child, we choose from among these possibilities, observing the child’s actual performance in a particular situation and recording what we see and hear. So, when analysis begins, the data we have 85
FROM GOALS TO DATA AND BACK AGAIN
collected is spread across many sheets, reflecting behavior across many sessions. There might be many instances of the same sheet or there might be many instances of different sheets, as when different teachers are responsible for tracking different subsets of the child’s goals. Regardless, the same process must be applied to each sheet. We use the sample sheet in Figure 6.1 as the basis for our discussion in this section. If you look at Figure 6.1 you will see the record of Joey’s performance for five goals on January 3, 2001. Each thing that has been circled, checked, crossed out, or written in is called an observation, value, score, or data point (we use these terms interchangeably in what follows to help you get used to hearing all four). Groups of values are called a data set. Taken together, the observations in Figure 6.1 constitute a natural data set—in particular, the set of scores for Joey on January 3, 2001. But there is a second kind of data set that is more central to analysis—the data set that combines the observations for an individual item over time. Each item on a data collection sheet produces such a data set and each set has two properties. These properties determine the kind of analyses we can perform on the data set for that item. You already know the first property —whether the values represent goal data or goal-independent factor data. We’ve identified this property for each item on Joey’s data sheet for you. As the labels at the left of the figure show, Joey’s sheet contains five items that produce goal data (goals 1–5) and five items that produce factor data (the therapist’s name, the date, time, and location of intervention, and a number representing Joey’s overall mood that session). The second property of each item’s data set is its data type. At the beginning of analysis it is fairly easy to determine the type of data produced by each item because we can see all the information we need to decide. On Joey’s sheet we can see, for example, that some of the items produce data points that are words or phrases, some produce numbers from a scale, and some produce percentages. These simple characteristics are important clues to the data’s type. As analysis continues, however, we must translate all the data into numbers. Once we’ve finished the translation process, all the data will look the same and all the telltale clues will have disappeared. So, it’s important to take the time to identify the data’s type now, before we obscure the information by translation. There are three types of data on our collection sheets. Let’s examine them in order, from simplest to most complex. Look at the therapist’s name and the location of intervention on Joey’s sheet. These items produce the simplest sort of data, the kind that permits only the simplest mathematical functions. What sorts of things can we do with the data sets produced from these items? We can count how many observations there are in the data set and, if there are different possible values, how many observations we collected for each value. For example, when we look through the sheets for a particular time period we might find that Joey had eight sessions in 86
FROM DATA COLLECTION TO DATA ANALYSIS
Name: Joey Smith Factor data
Categorical (nominal) data
Name of therapist: Sasha Date: 1/3/01 Time: 3:00-5:00 Location of intervention:
home
o center
On a scale of 1 to 5 (avoidant to sociable), please rate Joey’s overall mood for the day.
Interval data
1 2 3 4 5 Goal data
1. When offered a choice of food, or object with which to play or hold, was Joey able to indicate preference?
# of opportunities
2. When given an opportunity, was Joey able to match a photograph to a preferred item?
o o o o
3. After hearing the language modeled two times, was Joey able to fill in the word approximation “go” after “Ready, set…” to start a preferred activity?
# of opportunities
|||||||||
# of successes |||
Calculate: 33% (# of successes ÷ # of opportunities x 100) not today, even with modeling matched after modeling independently matched 1 or 2 times independently matched multiple times
||||||
# of successes |||
Calculate: 50% (# of successes ÷ # of opportunities x 100)
o multiple physical and verbal prompts 4. What level of prompt was needed to get Joey to pull an adult to an object of o a verbal prompt (“Show me!”) with outstretched hand desire? ý independently (a few times) o independently (consistently) 5. Did Joey attempt to imitate your mouth movements?
o ý o o
Ordinal data
no attempts noted growing interest in mouth movements a few attempts multiple attempts
Figure 6.1 Every item on a data collection sheet has two independent properties. We note the first property—whether it’s goal data or factor data—at the left. The second property—whether the item results in a categorical, ordinal, or interval data set—is labeled on the right. Both properties are used to determine the appropriate statistical tests to apply to the data
87
FROM GOALS TO DATA AND BACK AGAIN
total with six home visits (6/8 or 75%) and two center visits (2/8 or 25%). As you can see in Figure 6.1, data that captures only the categories or names associated with an observation is called categorical or nominal data. Although not usually very interesting by itself, categorical data may, nevertheless, contain important information when correlated with other data (a topic we discuss in Chapter 9). The next type of data can be seen in goals 2, 4, and 5. This type is a little more sophisticated because the data set generated by each item is made up of values from a 4-point scale with a specific and intentional order. We can still compute simple, categorical information about these items—perhaps Joey had 4/8 observations for goal 2 recorded as “Not today, even with modeling” and 4/8 recorded as “matched after modeling.” But we can also compute a new kind of information because, for this type of data, order counts. It meant nothing that “center” was listed after “home” in the location item and switching the order wouldn’t have change the meaning. In contrast, it means something definite that “matched after modeling” comes after “not today, even with modeling” in the list for goal 2. Specifically, it means that during intervention we expect the later choice on the list to be developmentally or cognitively more difficult to achieve than the earlier choice. Similarly, during analysis, we consider scores from later on the list to indicate more growth than scores from earlier on the list. Because the order of the list is meaningful, this type of data is called ordinal. All of the items on your collection sheet that have 4- or 5-point qualitative scales produce ordinal data. Each point on the scale is considered a different category or name, just as each choice of location was a different category or name. Formally, then, ordinal data is a more sophisticated type of measurement than categorical data because the name or category of a data point also indicates how the observation compares to the other possibilities in the ordering. To take another example: in goal 4 it is not just of interest whether Joey needed a verbal prompt, it is of interest particularly because this was a better outcome than if he needed a physical prompt and a less desirable outcome than if he had acted independently. In general, ordinal data can be subjected to all the same sorts of analyses as categorical data, but can, in addition, be used to make relational statements such as “this month’s performance is better than last month’s.” The third type of data that might appear on your collection sheets is also the most sophisticated type of data. Examples of this type for Joey include the date, time, and overall mood factors, and the percentages collected for goals 1 and 3. Each of these items generates a value from a quantitative scale: the date is a point taken from the scale of days in a year; the time is a point (or interval) taken from the scale of hours in a day; overall mood is taken from a numeric 5-point scale; and the percentages are points on a 100-point scale. These scales differ from the ordinal scales in goals 2, 4, and 5 in an important way. Ordinal scales can be used to express relative goodness, but they don’t tell how much of a difference exists 88
FROM DATA COLLECTION TO DATA ANALYSIS
between successive categories. We don’t know in goal 4, for example, that the amount of improvement Joey demonstrates by moving from multiple prompts to a single prompt is equal to the amount of improvement he demonstrates by moving from a single prompt to independence. But we do know, without question, that the amount of time between 3:00 and 5:00 is exactly the same as the amount of time between 4:00 and 6:00. We also know that the amount of increase in success rate between 30% and 40% is exactly the same as the amount of increase in success rate between 40% and 50%. Because the size of the interval between data points is equal (in the arithmetic sense) we call the data sets generated by these items interval data. As with categorical and ordinal data, each point on an interval scale defines a name or category, and as with ordinal data, the categories can be ranked or ordered. What makes interval data different, then, is just that the intervals between the categories are equal and obey the laws of arithmetic. But because of this simple difference interval data allows us to say not only that a child is doing better, but also how much better the child is doing. For example, suppose that during the last period we measured, Joey completed the prompted expression in goal 3 (“Ready, set…”) 40% of the time, while during the current period he completed the expression 80% of the time. We might characterize his performance by saying, “He is now doing twice as well as before,” or “He is now twice as likely to answer appropriately.” However, it is important to understand that such statements are supported by the data because they quantify behavior, not the underlying change. We cannot claim that Joey learned twice as much new information during the current interval, nor can we say that he is “half as autistic” as he was when his performance was only 40% correct.
Translating words to numbers to create raw data When you chose a scale for each goal on your data collection sheet you worded it carefully to reflect both the stage of the goal (for example, “emerging”) and the degrees of progress or change that you wanted to observe during the period of intervention. Using scales with meaningful category names (for example, “not today, even with modeling”) makes the job of the data collector easier and less susceptible to error or individual variation. Unfortunately, meaningful category names have the opposite effects on computers. The tools that are available to make data analysis easier require that we turn the words into numbers. Translating the data on your collection sheets into numbers will always be the first step in your data analysis. For the purposes of translation there are five different cases to consider. The examples provided for each case refer to the translated values that are circled in Figure 6.2.
89
FROM GOALS TO DATA AND BACK AGAIN
1.
Interval values require no translation. The calculated frequency in goals 1 and 3 and the number circled for the overall mood are the numeric values for those observations. The date and time can also be left as they are at this point in the analysis. (We’ll look at different ways to treat date and time in Chapter 9).
2.
Items in an accumulating list also require no translation.1 For this type of goal the data sheet specifies the actual items you want the child to add to his or her repertoire of words, skills, or emotions. In each session one or more of the items will be circled or checked. Every checked item is part of the observation to be recorded in the spreadsheet. Eventually, you want to see each of the targeted items listed at least once in the spreadsheet. Once each item has been listed, the child has achieved his or her goal of increasing a range of behaviors. This method of measurement does not depend on further analysis to verify change over time—change is evident from the data itself. Since no additional analysis is required, we do not discuss this type of goal again in the remaining chapters.
3.
Factors that have categorical values (the name of the therapist and the location of the intervention) need to be translated, but we delay the translation to numbers until we are ready to use the factor in computing a correlation statistic (see Chapter 9).
4.
Observations based on a qualitative scale (goals 4 and 5) must be changed to numeric form. Remember that each qualitative scale was created after identifying step-by-step changes in behavior. The team considered each step to be categorically different, a separate marker of progress. The translation process assigns sequential numbers to each step: 1 to the first value, 2 to the second value and so on. In this way the numbers reflect growth in functioning with 1 representing the lowest level expected of the child and 4 or 5 (depending on how many steps you identified) representing what we expect to be the best outcome for this period of intervention. Notice that the process of translating the scale to numbers creates an important characteristic in
Although there is no example of this kind on Joey’s sheet, you can see one in Chapter 5 (where the goal was to extend Peter’s play repertoire to include gross motor play, construction/building activities, puzzles, art, music, and early pretend play).
1
90
FROM DATA COLLECTION TO DATA ANALYSIS
Name: Joey Smith Name of therapist: Sasha Date: 1/3/01 Time: 3:00–5:00 Location of intervention:
home
center
On a scale of 1 to 5 (avoidant to sociable), please rate Joey’s overall mood for the day. 1 2 3 4 5 1. When offered a choice of food, or object with which to play or hold, was Joey able to indicate preference?
# of opportunities
2. When given an opportunity, was Joey able to match a photograph to a preferred item?
o o o o
3. After hearing the language modeled two times, was Joey able to fill in the word approximation “go” after “Ready, set…” to start a preferred activity?
# of opportunities
4. What level of prompt was needed to get Joey to pull an adult to an object of desire?
o multiple physical and verbal prompts o a verbal prompt (“Show me!”) with outstretched hand ý independently (a few times) 3 o independently (consistently)
5. Did Joey attempt to imitate your mouth movements?
o ý o o
|||||||||
# of successes |||
Calculate: 33% (# of successes ÷ # of opportunities x 100) not today, even with modeling N/A matched after modeling independently matched 1 or 2 times independently matched multiple times
||||||
# of successes |||
Calculate: 50% (# of successes ÷ # of opportunities x 100)
no attempts noted growing interest in mouth movements a few attempts 2 multiple attempts
Figure 6.2 Joey’s data sheet after translation
91
FROM GOALS TO DATA AND BACK AGAIN
your data: numbers whose difference is 1 now represent different categories of behavior. 5.
Goals without associated observations (goal 2) must be explicitly recorded as “N/A,” “no observation,” or a similar phrase.
When you’ve assigned the appropriate type of value to each factor and observation, you’ve translated the child’s behavior into the raw data we work with in a spreadsheet. You may find that circling or highlighting the value for each observation on the data sheet will make it easier to enter the data accurately on the computer. As you scan down the circled values, notice that all the values for goals are numbers and that you can no longer tell the ordinal data from the interval data. Don’t let appearances fool you: we have changed the way the data looks, but not what the data means. The interpretation of the data must always be based on its underlying meaning.
Transferring the raw data to spreadsheets The second step in beginning a data analysis is to transfer the data on your collection sheets to a spreadsheet. Using a spreadsheet lets us further organize the data, apply statistical formulas to it, and automatically generate graphs that help make sense of it. If you haven’t used a spreadsheet program before, this process may seem difficult and time-consuming at first, but it will become easier and faster with practice. More importantly, taking the time to learn how to use a spreadsheet will make your analysis more reliable and more informative. There are many ways we might format each child’s data in the individual sheets, rows, and columns of an Excel workbook. The format we’ve chosen to use is shown in the portion of Joey’s spreadsheet in Figure 6.3. Whether you use this format or not, it is important that you use your chosen format consistently. Let’s look at the spreadsheet in Figure 6.3 more closely. You may find it helpful at this point to open the file “Joey Jan-Mar2001.xls” in Excel on your computer. There are 7 features of our organization that we want you to notice:
92
1.
The name of the file includes both the child’s name and the time period covered by the data. This makes it easy to find this particular period of Joey’s intervention without opening and closing a large number of files. It also helps you avoid overwriting this file by mistake with a new file for Joey on different dates (or with a new file for a different child on the same dates).
2.
The name of the sheet (circled at the bottom) tells us it contains the raw data for this period of intervention. Everything you do in data analysis begins with the raw data. For this reason, raw data should be placed on its own sheet in the workbook and all subsequent analysis performed on
FROM DATA COLLECTION TO DATA ANALYSIS
a copy. (The handout in Appendix B titled “Directions for Using Excel” covers the mechanics of copying data and other basic operations in Excel.) By observing this precaution you will not have to start over should you mistakenly corrupt the data by transforming it in an inappropriate way. By working with a copy of the data, you always have an on-line version to go back to if necessary. 3.
The top row of the sheet is used for labels that explain the information in each column. The remaining rows record the data for each session, one session per row, arranged in chronological order. The cells in the first column of each row label the session dates explicitly. This may not seem important now, but it will prove invaluable if you have to return to the collection sheets to check for data entry errors or comments that might have been written by the observer. In addition, your data may have different uses over time. For example, at some point in the future you may find yourself wondering how a child behaved during the same week or month in a previous year. As another example, someone may want to combine your data for a child with other professionals’ data sets to examine behavior across therapies on a day by day basis.
file name
worksheet name
Figure 6.3 A portion of Joey’s raw data after it has been entered on a spreadsheet
93
FROM GOALS TO DATA AND BACK AGAIN
4.
While the left-most column is used for the date, the remaining columns contain the data in the order dictated by the labels in the first row. Choose a meaningful label for each factor and goal on the collection sheet; do not use the number of the goal to label the column. Again, this may seem like unnecessary work at the moment, but over time you are likely to forget which column represents which goal, making it easy to go to the wrong data in response to a question. Moreover, if you are not careful when you perform tests using the data in a column, you may pick up the goal number as if it were a piece of data, changing the outcome of the test. Finally, consider that the same goal may occur in a different place on subsequent data sheets for this child or on the various data sheets used by different professionals working with this child. If you use a brief phrase (like “Match photo”) you will be able to track progress within a child’s development more easily. In short, the small effort taken to choose a meaningful label now will minimize confusion and make for more accurate statistics in the future.
5.
Along with meaningful labels for the goals, we include a reminder of the scale that was used. In the case of percentages we include the approximate number of opportunities per session that the child had for that goal during this period of intervention. We’ll need this information to interpret the results of the analysis. By including it now we save ourselves the trouble of having to go back to the data collection sheets to find it.
6.
Notice the blank entry (cell E4) under the “Match photo” goal for January 4. On that day Joey did not have the opportunity to match pictures to preferred items. Having no opportunity is different from having an opportunity but not being able to match (the lowest value on the qualitative scale for this goal).2 The observer who took the data left all the items unchecked to indicate that Joey had no opportunity to work on this goal. We translated the absence of a check mark into the raw data value “N/A” then, in turn, into a blank cell in the spreadsheet. For the statistical routines to work correctly it is critical that you leave blank any cell for which you did not collect data. Do not enter 0, “N/A,” “No opportunity,” or any other value in these cells.
In this way “Match photo” is different from “Imitate mouth” where, as long as the therapist talks and plays face-to-face with Joey, we can say that Joey had the opportunity to imitate.
2
94
FROM DATA COLLECTION TO DATA ANALYSIS
7.
Finally, note that the labels for the factor data are italicized while the labels for the goal data are in boldface. Use a visual convention to separate your factor data from your goal data. In the following chapters we will apply certain statistics to each column of goal data and other statistics to factor-goal pairs. Sometimes factor data will be represented by words on the raw data sheet (e.g., location in column B) but sometimes it will be represented as numbers (e.g., overall mood in column C). Having a simple visual cue that distinguishes between the two kinds of numbers (rather than relying solely on memory) will help to make sure the correct statistics are applied to the correct columns.
The accuracy of your raw data is critical to the accuracy of your statistics and resulting interpretation. Faulty data usually leads to faulty conclusions. Adopting a consistent approach to data organization across children is one way to improve accuracy. There are two additional good habits that are worth cultivating. First, you’ll make fewer transcription mistakes due to fatigue (and boredom) if you enter your data as it is collected rather than in one long session just before you’re ready to do an analysis. Second, don’t assume you did it right; always check the values in your spreadsheet against the values on the collection sheet after you’ve finished entering them. The best way to catch any transcription mistakes you’ve made (and we all make them) is to have a colleague or friend read the values from the collection sheet while you check the numbers on the spreadsheet. If this isn’t practical, waiting an hour or more before you check the collection sheets against the spreadsheets can help keep you from repeating an error.
Keeping in mind what the numbers really mean There is one final feature to notice about the data as it now appears on your spreadsheet: most of the context that defines the meaning of the numbers has been removed from view. As we focus increasingly on what we can do with the data when it is represented as numbers, don’t lose sight of the fact that the meaning of those numbers depends on the way the goals and scales have been worded. A value of 4 in one column does not necessarily mean the same thing as a value of 4 in another column. Even more importantly, keep in mind that the meaning of the numbers depends upon the accuracy and consistency of the observer and, most importantly, upon how well the behaviors being observed and the scales we have specified truly reflect the underlying cognitive and emotional changes we desire. This reminder is critical at this point in the book because we are about to switch our thinking from particular children with histories and meaningful goals to data sets with no real information behind them. In developing the tools 95
FROM GOALS TO DATA AND BACK AGAIN
for data analysis our examples will be based primarily on the information in the ABC Workbook on the CD-ROM that accompanies this book. The ABC Workbook contains data sets for three fictitious children: Anton, Becca, and Celeste. We’ve given names to the data sets because it makes the discussion easier to read, but we’ve chosen the names to remind you of the variables A, B, and C because these are our “variable” children. Their data has no real goals behind it and may, in fact, be used to represent different goals at different times for narrative purposes. To help you keep this in mind we have also labeled the goals in these data sets with variables (X, Y, and Z) rather than the meaningful abbreviations you would use for real data. If this sounds confusing, don’t worry, it will begin to make sense shortly. And rest assured that once we’ve developed them, we will bring our analytical tools back to the cases of Joey and the other children (see Chapter 10). In the meantime, we suggest you keep the ABC Workbook available on-line as you read the remainder of this and the next three chapters. The exercises designed to teach you how to use the analytical tools in Excel require the ABC Workbook as well.
First impressions: numbers versus pictures If you open the ABC Workbook to the sheet labeled “Anton Goal X” you will find 30 observations for the period between January 3rd and March 10th for the fictitious child named Anton (see Figure 6.4 if you are uncertain which sheet we mean).
96
FROM DATA COLLECTION TO DATA ANALYSIS
As you glance down the column for goal X you may notice that the first group of 10 values contains only 1s and 2s, the second group of ten values contains 2s and 3s, and the final group of ten values is made up of 3s and 4s. So, if we assume that lower numbers correspond to weaker performance and higher numbers correspond to stronger (more desired) performance, it seems clear that Anton’s behavior has become increasingly appropriate over this time period. We made the trend in this example fairly obvious and specifically pointed out the meaningful pattern in order to help you see it. In general, however, the ability to look at even a small column of numbers and judge whether or not change has taken place is a skill that takes a lot of practice. To develop this skill, and to work with larger or more complex data sets, you need a few simple but powerful tools. The first one we’ll add to your analytical toolbox is graphing. Graphing your data is often the second step of data analysis. The power of graphing can be seen by comparing Figure 6.5 to column B of Anton’s spreadsheet. For most people the visual representation of the data shown in Figure 6.5 is easier to understand and characterize than the column of numbers on the raw data sheet. You may also find that even when there is a clear trend in the data, as in Anton’s case, a picture is more compelling than a list of numbers when showing your data to others. Indeed, one of the reasons for entering your data in a spreadsheet is that the spreadsheet program has built-in graphing functions to help everyone on the team see the trends. This graph, for example, looks like a set of stairs rising, a good metaphor for (and fairly reliable predictor of ) positive change.
4-point scale (1,2,3,4)
Anton, Goal X
00 6/ 3/
0 28
/0
0 2/
21
/0
0 2/
/0 14
2/
0
00 7/ 2/
31
/0
0 1/
24
/0
0 1/
17
/0
0 1/
/0 10
1/
1/
3/
00
5 4 3 2 1 0
Figure 6.5 The line graph for Anton’s raw data shows a “rising stairs” pattern
97
FROM GOALS TO DATA AND BACK AGAIN
Creating simple line graphs Just as there are different types of data, there are different types of graphs that can be used to present each data type. Figure 6.5 is an example of the conven3 tional choice for showing a trend over time: the line graph. Exercise 6.1 (in Appendix C) takes you step-by-step through the creation of Anton’s line graph for goal X using Excel. Exercise 6.2 takes the basic method taught in Exercise 6.1 and introduces a number of helpful shortcuts and variations. There is a bit of an art to producing a good graphical representation of your data. On the one hand you must make sure that enough information is included in the various titles, labels, and legends to clearly explain what the picture represents. On the other hand, too much information in too small a space leads to visual clutter that may obscure the point you want to make. How you balance these two requirements—visual clarity and information accuracy—is partly a question of personal style. You don’t need to become an expert on graphical representation to create graphs that are clear, useable, and readable by others. There are, however, a few features you should be aware of: · Who and What: There should never be any question about which
child’s data it is or which goal is being graphed. We tend to put this information in the title of the graph because that is generally the most prominent text. In addition, when the child’s name and goal are in the title it isn’t necessary to include a legend, which helps cut down the visual clutter. The exception is for a graph containing data for more than one goal. In these cases the legend is necessary and the title may be used for other purposes, as shown in Figure 6.6.
· When: Because we are always interested in measuring change within
some time interval, the time interval must be clearly specified as well. In Figure 6.6 the dates of intervention have been associated with the increments shown by the “tick marks” (little lines) on the horizontal axis. Since the dates are explicit and recognizable, we don’t need an additional label for the axis. Again, our choice preserves information with an eye toward reducing visual clutter. Sometimes the tick marks aren’t informative enough, however, and additional information should be provided. As an example, see Figure 6.7 where we use the axis label to indicate that the tick marks represent a sequence of sessions.
· How: How was behavior measured for this goal? The vertical axis
reflects the values of the data points. Graphing programs usually
Technically, a line graph is not an appropriate way to represent ordinal data because the continuous line implies that the values between the points are well-defined. People are rarely misled by this aspect of the representation, however, so we will sacrifice a bit of technical correctness for the sake of convention.
3
98
FROM DATA COLLECTION TO DATA ANALYSIS
Comparison of Progress on Goal X
Anton
00
0
6/
3/
/0
0 2/
28
/0
0 2/
21
/0
00
14
2/
7/
0
2/
/0
0 1/
31
/0
0
24
/0
1/
17
1/
10
/0
0
Celeste
1/
1/
3/
00
4-point scale (1,2,3,4)
4.5 4 3.5 3 2.5 2 1.5 1 0.5 0
Figure 6.6 The legend (box at right) is necessary when you combine data for two children receiving group intervention on the same dates
5 4 3 2 1
28
25
22
19
16
13
10
7
4
0
1
4-point scale (1,2,3,4)
Anton, Goal Y
Sessions
Figure 6.7 The line graph for Anton’s raw data for goal Y
determine the increments on the vertical axis automatically, based on the values of the data set being graphed. Division of the vertical axis is one of the few features of a graph that can be complicated to change. Unfortunately, the values chosen by Excel are often inappropriate for the kind of data we use. Someone looking at just the vertical axis in Figure 6.6, for example, might believe that the child’s behavior was rated on a continuous scale of values from 0 to 4.5. In fact, the only values for goal X that the observer could have chosen were the numbers 1, 2, 3, or 4. So we use the title of the axis to remind the reader of the actual scale and explicitly counter the misinformation in the graph. 99
FROM GOALS TO DATA AND BACK AGAIN
· How many: Figure 6.6 (and Exercise 6.2) demonstrates that more than
one data set can be placed on a single graph. Should you put all the data for a child in one graph? How many goals can you combine? The decision depends on both clarity and the purpose of the graph. Combining data sets in a single graph will usually make each trend a little bit harder to spot—the more you add the harder it gets. Sometimes the difficulty in reading the graph must be tolerated because you want a direct comparison among data sets. Keep in mind, however, that to combine data sets on a single graph the horizontal and vertical axes must mean the same thing. You cannot meaningfully combine goals with different size scales or goals in which the data was collected on different dates.
There is one additional aspect of graphing you need to understand: the visual effect of length versus height in the axes. It is possible to mislead someone about the meaning of your data by changing the ratio of these dimensions. For any single graph you don’t need to worry about this problem because Excel usually makes reasonable choices. If you are going to compare graphs side by side, however, it is important that they have the same relative dimensions and increment markings on their axes. Within the guidelines we’ve given you there is still room for variation (and you’ll see some of that variability in the graphs throughout this book). Developing your own style is fine, just remember that the general aims when producing graphs are similar to the general aims for your data collection sheets—be clear and require as little effort as possible on the part of the observer for a correct interpretation. Adopting consistent conventions helps both these aims. Becca, Goal X
4-point scale (1,2,3,4)
5 4 3 2 1
00 6/ 3/
0 28
/0
0 2/
21
/0
0 2/
/0 14
2/
0
00 7/ 2/
31
/0
0 1/
24
/0
0 1/
17
/0
0 1/
/0 10
1/
1/
3/
00
0
Figure 6.8 The line graph for Becca’s raw data. Has she made progress on goal X?
100
FROM DATA COLLECTION TO DATA ANALYSIS
Real data is not pretty Most of the time, the information in a data set is not as clear cut as in Anton’s case. Compare, for example, Anton’s data (Figure 6.5) with Becca’s data for the same goal (Figure 6.8). What should we conclude? Has Becca’s behavior changed? Can we be certain that she has learned the underlying information or skill we wanted her to learn? Perhaps her poor performance in the session on January 31st can be safely ignored; but maybe the additional low score toward the end of February should make us cautious when judging her progress. The case for Celeste (Figure 6.9) is even more confusing. She seems to have been making steady progress then appears to have regressed (around February 7th) then looks as if she may have gotten back on track. Should we conclude that she has reached her goal? Should we conclude that she has not? Celeste, Goal X
4-point scale (1,2,3,4)
5 4 3 2 1
00 6/ 3/
0 28
/0
0 2/
21
/0
0 2/
/0
2/
14
00 7/ 2/
0 31
/0
0 1/
24
/0
0 1/
17
/0
0 1/
/0 10
1/
1/
3/
00
0
Figure 6.9 The line graph for Celeste’s raw data. Has she made progress?
If your impulse is to conclude that Celeste has not reached her goal, then suppose for a moment that the scale being used to measure Celeste’s progress was translated as: 5 or more prompts →1 3–4 prompts →2 1–2 prompts →3 0 prompts →4 Looking at the scale, do we really want to conclude that Celeste’s data does not represent significant progress? Perhaps it is unreasonable, given her stage of development, to expect Celeste to do better than the pattern shown (one occurrence of unprompted behavior in the first ten sessions (10%), but three occurrences of unprompted behavior in the last ten sessions (30%)). Perhaps we wouldn’t expect better than 30% for any child of this age or level of develop101
FROM GOALS TO DATA AND BACK AGAIN
ment. Remember: the numbers themselves are meaningful only with respect to the ideas behind them. What constitutes progress depends critically on what is reasonable to expect for this particular child within the regularities of typical development. Becca and Celeste’s cases more accurately reflect the kinds of graphs you are likely to see when working with real data. Cognitive change is a gradual thing and rarely characterized by constant forward progress. Both adults and children grow tentatively, requiring small steps that make sense within the larger context of our lives to convince us of the usefulness of enduring the discomfort of change. As a result, real data, like real behavior, is often too “messy” to understand using only a tool as simple as the graphing function discussed in this chapter. A commitment to an empirical approach to intervention requires that we outfit ourselves with stronger tools: the mean and standard deviation. We turn to these in the next chapter.
Things to remember How our data is organized: T Spreadsheet file: the file name includes both the child’s name and the
time interval during which the data was collected. The file contains all the data collected for that child during that time interval.
T Sheet name within the file: indicates what kind of data (e.g., raw data). T Raw data sheet: the first row labels factors and goals, the remaining
rows contain the session date in the first column followed by columns containing the value assigned to each factor and goal on that date.
Important principles for labeling and entering your data: T Reduce data entry mistakes by typing in your data each time it is
collected (rather than in a marathon session) and by having your entries verified by a colleague or by yourself at a later time.
T Once your raw data set has been entered and checked, don’t touch it.
Always work with a copy.
T Label the session dates clearly and explicitly in the first column. T Choose a meaningful label for each factor and goal. T Leave empty any cell for which data was not collected during a session.
Empty cells correspond to the N/A or “No opportunity for observation” values on your data collection sheet.
T Use a visual convention (e.g., italics) to separate your factor data from
your goal data.
102
FROM DATA COLLECTION TO DATA ANALYSIS
T Consistency, consistency, consistency! Whether it is in organizing
your data, labeling your spreadsheets, or producing your graphs, the more consistent you are, the fewer mistakes you will make and the clearer your understanding and presentation of ideas will become over time.
T The meaning of the numbers depends on more than just the performance
of the child. Data must always be understood in terms of the way the goals have been phrased and the scales worded, the accuracy of the observer, and how well the behaviors being observed actually reflect the underlying changes we desire.
T Cognitive change is gradual and rarely characterized by constant
forward progress. For this reason, although line graphs can be helpful in understanding your data, they usually do not provide enough information to draw conclusions.
103
Chapter 7
Detecting Change — The Mean
In Chapter 6 we saw that data visualization tools such as graphing have strengths and weaknesses. When a trend in the data is clear-cut, a picture of that trend can be a compelling description. On the other hand, when there is no trend, or when the trend is unclear, a picture may require a thousand words of explanation and still produce no firm conclusion. In this chapter we introduce the use of simple statistical methods by examining the idea of change as it applies to developmental intervention. Our discussion of change leads us to a number of other naturally related concepts: regularity, expectation, predictability, and noise. Some of these concepts may be unfamiliar to you at first while others may seem obvious. Since these concepts are deeply intertwined, however, whichever concepts you’re most comfortable with will eventually lead you to the others.
Patterns of change When we work with a child to change a behavior, skill, or emotional response, we are taking part in what is, ideally, a three-step process. In the first step, the team identifies in the child a consistent way of acting that is ineffective, inappropriate, or maladaptive. The team believes that this way of acting represents the child’s responses to some stimuli or situation at this moment in time. In the second step, the team chooses intervention methods that modify the underlying beliefs, knowledge, or emotions that cause the child’s responses, and these methods are used with the child over a period of time. In the third step—and here’s the ideal part—the team identifies in the child a consistent way of acting that is more effective, appropriate, or adaptive than before. Is there a simpler way to say this? Sure. For the purposes of intervention we define change as the transformation of one regular pattern of behavior into another regular pattern of behavior. Consider the left-hand graph in Figure 7.1. Common intuition tells us that no change has occurred in this child’s behavior. Indeed, looking at a graph like this the team might conclude, “If something doesn’t change soon we’ll have to 105
FROM GOALS TO DATA AND BACK AGAIN
try a new approach.” Referring to our definition of change, we would say, more specifically, that no change has occurred because there is only one regular pattern of behavior, not two. Example of Abrupt Change
4 3 2 1 0 1
2
3
4
5
6
Sessions
7
8
9
10
4-point scale (1,2,3,4)
4-point scale (1,2,3,4)
Example of No Change 4 3 2 1 0 1
2
3
4
5
6
7
8
9
10
Sessions
Figure 7.1 A single regular pattern of behavior versus two different regular patterns of behavior
Now consider the right-hand graph in Figure 7.1. The child whose behavior is reflected by this data does show change, both intuitively and by our definition. First we see one regular pattern (the values of 2 in the first five sessions) then we see a second, different regular pattern (the values of 4 in the last 5 sessions). When we observe a regular pattern we form expectations about future behavior. To put it another way: regularity has predictive value. At the end of Session 5 in the right-hand graph, for example, we might predict that “if nothing changes” this child will continue to receive a score of 2 for this goal. When our expectations are violated and our prediction proven wrong (in Session 6), we assume it is because something has changed (indeed, unless we’ve identified other obvious factors, we assume that the cause of that change is related to the intervention). As time goes on and the new regularity is detected, we establish a new set of expectations and make new predictions. At the end of Session 10, for example, we might say, “I think he’s got it!” The kind of abrupt transformation of behavior shown in the right-hand graph of Figure 7.1—a large and consistent step forward occurring virtually overnight—is rare, but not impossible. Sometimes it does feel as if a child suddenly “gets it.” When we are trying to modify deep and largely unconscious behaviors, however, more gradual change is typical. Figure 7.2 shows an example of a more gradual transformation phase, similar to the “rising stairs” pattern of change we saw for Anton in Chapter 6. Our definition of change alone isn’t quite enough to help us interpret this graph. Are we looking at one occurrence of change or two? The definition is inadequate to settle this question because it is expressed in terms of behavior (as it should be!), not in terms of points on a graph. The definition doesn’t tell us, for example, how many data points are required to establish a regular pattern, or how many data points there are in the period of transformation. 106
DETECTING CHANGE – THE MEAN
4-point scale (1,2,3,4)
Example of Gradual Change
5 4 3
PRE POST
2 1 0 1
2
3
4
5
6
7
8
9
10
11
12
13
Sessions Figure 7.2 One pattern of behavior (pre-transformation) followed by a period of transformation followed by a new pattern of behavior (post-transformation)
A person who doesn’t know the meaning behind the data might see three regular patterns and two distinct episodes of change. With the developmentalist’s eye, however, we are likely to see only one episode of change; we explain the regularities in this way because they meet our expectation that cognitive change typically requires a gradual transformation of behavior. The important point in this example is that we can use our knowledge of development to make a prediction about the kind of pattern of change we are likely to see, then match the actual data to that prediction. In general: our interpretation of the data depends on our expectations and our expectations depend on our knowledge. When should a pattern be considered regular? This is not a trivial question. Figure 7.3 repeats Anton’s data from Chapter 6 to help clarify.
4-point scale (1,2,3,4)
Anton, Goal X
00 6/ 3/
0 28
/0
0 2/
21
/0
0 2/
/0 14
00
2/
7/ 2/
0 31
/0
0 1/
24
/0
0 1/
17
/0
0 1/
/0 10
1/
1/
3/
00
5 4 3 2 1 0
Figure 7.3 The line graph for Anton’s data for goal X
107
FROM GOALS TO DATA AND BACK AGAIN
When we said that the pattern of change in Figure 7.2 was similar to Anton’s we were making a judgment based on our visual perception of the overall shapes of the two graphs (“a set of rising stairs”). Visual perception is a powerful tool for judging regularity and that is one of the reasons we graph data. In Anton’s case, however, we need more than just visual perception to judge whether the graph meets our definition of change. Generalizing from the pattern in Figure 7.2 we might be comfortable picking out the middle portion of the data (the nine data points between 1/24 and 2/14) as the transformation stage. But how should we interpret Anton’s pre-transformation and post-transformation periods (the sessions prior to 1/24 and after 2/14, respectively)? Neither the pre-transformation behavior nor the post-transformation behavior is as obviously regular as it was in Figure 7.2. As in the previous example, it is knowledge that is not directly apparent from the data that allows us to say that there are two different regular patterns of behavior in Anton’s graph. We know, for example, that no child is entirely consistent in his or her behavior. This knowledge, alone, might be enough to make us confident in our interpretation. If we look back at the meaning of the scale used during observation, however, we might also find that the wavering in both the pre- and post-transformation scores is easy to dismiss as an artifact of that scale. For example, suppose the translation of the scale for this goal was: 3–5 prompts →1 1–2 prompts →2 no prompts (at least once) →3 no prompts (consistently) →4 Then if Anton’s pre-intervention pattern of behavior regularly required either two or three prompts, his scores would alternate between 1 and 2. In other words, the regularity in behavior would cross our category boundaries and create the sort of pattern shown in the pre-transformation portion of his graph. Similarly, if at the end of the observation period Anton still needed an occasional prompt, his scores would alternate between 3 and 4 and we would expect the post-transformation pattern in Figure 7.3, given our scale. Knowledge—of children in general and Anton’s scale in particular—makes the difference in our interpretation. Now consider Daniel’s graph in Figure 7.4. How many regular patterns do you see? Two people looking at this graph might conclude different things about whether and when change occurred, even if they had the same knowledge about development, rate of cognitive change, the particular child, and the particular child’s goals. Adding more knowledge—even knowledge about Daniel—won’t necessarily alter their interpretations. This kind of data is typical of what you see when measuring real behavior in real situations. In the previous chapter we called it “messy.” The technical term is “noisy.” Noise is the set of values that creates 108
DETECTING CHANGE – THE MEAN
an apparent lack of regularity when such regularity actually exists. You can think of noise as regularity’s “evil twin.” Noise interferes with our ability to perceive regularities. It undermines the certainty we feel about our interpretation. At its worst, a noisy graph of successful learning can be visually indistinguishable from a graph that shows true lack of change.
4-point scale (1,2,3,4)
Daniel, Noisy Change
5 4 3 2 1 0 1
2
3
4
5
6
7
8
9
10
11
12
13 14
15
Sessions Figure 7.4 Is this a case of change or not?
We used knowledge about how we measured Anton’s goals to compensate for the noise in Figure 7.2. Another way to reduce noise can be to extend the period over which we make observations. This may help because our sense of regularity depends on detecting a consistent pattern over time. The graph in Figure 7.5 shows the 15 sessions in Figure 7.4 as part of a longer stretch of time. If we had been able to measure Daniel’s behavior for this goal before Session 1 we would
5 4 3 2 1
22
19
16
13
10
7
4
0
1
4-point scale (1,2,3,4)
Daniel, View Extended Over Time
Sessions Figure 7.5 With more data before and after, the pattern of change emerges
109
FROM GOALS TO DATA AND BACK AGAIN
have seen the first regular pattern of behavior clearly. And if we could continue to measure Daniel’s behavior for this goal after Session 15 we would see a second regular pattern emerge. Two regular patterns separated by a period of growth—now that looks familiar! What just happened? The data that looked too noisy to interpret with any certainty in Figure 7.4 didn’t change, yet most of the noise seems to have disappeared. Why? Remember that our definition of behavioral change involves three parts: two different patterns of regularity separated by a period of transformation. When we tried to divide Daniel’s first graph into those three parts, it was difficult to see where one part ended and the next began. Every possible division left all three parts looking noisy. When we broadened our view in time the noise appeared as one segment. Moreover, it shifted into the segment in which it is most easily explained—the segment that reflects the period of learning transformation. Our knowledge, in combination with enough data, allowed us to explain 1 the noise. Unfortunately, for most kinds of meaningful change we can’t know how much data is going to be enough to picture the whole process for a particular child. We simply don’t know how long it will take a given child to learn a new skill, behavior, or emotional response in an adaptive and lasting way. In addition, we don’t always know exactly where the child is in the learning process when we begin to collect data. We often target a behavior when we think the time is ripe, which is just another way of saying that the data we collect at the beginning of intervention may already reflect some of the noise typical of the transformation stage. It seems as if everywhere we turn there is potential noise threatening the accurate interpretation of our data. How do we decide when a portion of our graph is just a little noisy and when that portion is really showing that lasting change requires additional intervention? The unfortunate truth is that we can’t eliminate noise completely, nor can we always explain it away. We’ve already discussed some ways we can minimize it: design goals to be meaningful reflections of the behavior we’re trying to teach, design scales to look for the degree of change that could occur within the time interval being measured, practice good data collection habits, and use our knowledge judiciously in interpreting the graphs. But, in the end, we have to accept that real behaviors, like real children, are inherently noisy beasts. If we can accept that our data has noise then we can acknowledge that not every data point carries useful information. But how do we decide which of the data points reflect true behavior and trends, and which ones are noisy? If we’re going to ignore some of the information in our data we need to be certain that If you think we’re going to point out that this example shows one more reason why more data is better, you’re right.
1
110
DETECTING CHANGE – THE MEAN
we do it fairly, not just in whatever way makes us (and the child) look good. In addition, if we’re going to use our interpretation to convince others, we need to be certain that they would make the same choices. In short, our method of dealing with noise must be consistent across children and repeatable by others.
Computing the mean The mean is a statistic that helps eliminate some of the effects of noise in a way that is both fair and uniform (it won’t do it by itself, so we’ll have to introduce its partner, the standard deviation, in the next chapter). It does its job by giving us a single representative value for a set of data. In essence, the mean clears away much of the visual clutter, thereby removing a good deal of the subjective component that comes with using line graphs for evaluation. Technically, the mean of a set of numbers is the set’s arithmetic average (the sum of the items in the set, divided by the number of items in the set). In Excel, the mean is computed using the AVERAGE function. Exercises 7.1 and 7.2 in Appendix C provide practice using this function. When you’ve completed Exercise 7.2 you’ll see that the mean is the same for each of Anton’s, Becca’s, and Celeste’s data sets. Recall from the previous chapter, however, that the line graphs for these data sets led us to qualitatively different evaluations of each child’s progress on goal X. Of course, when we looked at the line graphs we were looking for evidence of change. What part does the mean play in our search?
Evaluating change using mean values In well-designed psychology experiments, the typical method for detecting change requires three steps: first, the abilities of a group of participants are measured; second, the participants perform whatever activity the researcher has hypothesized will promote learning; and third, the participants’ abilities are measured again. The researcher then compares the participants’ test scores before the activity to their scores after the activity. A significant difference in those scores is taken as evidence that the experimental activity had an effect. What counts as a significant difference depends on the size of the group and the statistic the researcher chooses for the analysis. The choice of statistic depends, in turn, on the experimental design, access to a control group that did not undergo the experimental activity, any underlying assumptions about the participants being tested, and so on. Of course, this method is nearly useless as soon as you leave the psychology lab. Most of the statistics appropriate for studying large groups do not apply to studying a single child in intervention. Moreover, there is little or no “before” intervention measurement in our work. We can’t gather extensive baseline data for each goal before we begin to try to alter the child’s behavior, but instead must 111
FROM GOALS TO DATA AND BACK AGAIN
move almost immediately from identifying a goal to both collecting data on performance and providing behavior-changing experiences. Nor is there a clear period “after” intervention. If we have chosen natural activities for the context of our intervention, learning opportunities may occur when we are not observing and over which we have no control. Having reoriented the child so that the natural social world continues to intervene and reinforce a new behavior, the “experiment” goes on long after the goal has been marked “achieved” in our reports. So, we must adapt standard experimental design to our circumstances. Like the researcher we are concerned with demonstrating a difference between “before” and “after” behaviors. But we must replace formal group statistics with a test for change that makes sense for a single child in the real world. Not surprisingly, the test we use is based on our definition of change as a three-part process and our expectations about how that process unfolds. At the heart of the test is a comparison of means. There are three steps to performing such a comparison for each goal: 1.
Divide the data for that goal as closely as possible into thirds. We consider the first third of the data to correspond to “before” intervention and the last third of the data to correspond to “after” intervention. Copy the “before” and “after” data to a new spreadsheet. Ignore the middle third of the data.
2.
Compute the mean of the “before” data and the mean of the “after” data separately.
3.
Compare the two means in terms of the scale for the goal. If the two means reliably reflect different categories of behavior then conclude that change has occurred.
In the remaining sections of this chapter we look at each of these steps more closely.
Step 1: Dividing the data for the goal
Open the sheet titled “Anton Goal X Before-After” in the ABC Workbook to see a division of data as specified in Step 1. Because Anton’s data contains 29 observations, perfect thirds are not possible. To ignore the least amount of data, we copied the first ten and last ten observations from Anton’s raw data and left the nine middle cells behind. We also put blank rows between the sets to make sure it was easy to tell where the “before” data stopped and the “after” data started. The sheets titled “Becca Goal X Before-After” and “Celeste Goal X Before-After” were created in the same way (the first few steps of Exercise 7.3 in Appendix C take you through the mechanics).
112
DETECTING CHANGE – THE MEAN
At first glance dividing the data into thirds may seem like an arbitrary way to proceed, but some well-defined division is necessary if our method is going to be consistent and repeatable. The critical aspect of this step is not the fraction we’ve chosen (more about that below) but that the division is made independent of the characteristics of any particular data set. What we’re not allowed to do is look at the data and say, “Hmmm, we’ll just call this section over here ‘pre-transformation’ and that section over there ‘post-transformation’ and see what happens.” The rules of the game say that if we have successfully produced change through intervention then we must be able to fit the data to the hypothesis, not the other way around. In this case, our hypothesis is that during this time interval there has been a change in the child’s underlying knowledge, beliefs, or emotions such that he or she is able to behave in a way that is consistently 2 different from his or her behavior prior to intervention. But why thirds? The choice comes from the theory of how cognitive change is reflected in behavior. If change is occurring within the time frame of our analysis then each of change’s three phases—pre-transformation regularity, transformation, and post-transformation regularity—should be uniquely reflected in its own subset of the data. “Before” corresponds to that period of time during which the child’s behavior is best explained by his or her old ways of thinking and feeling. We expect pre-intervention patterns of behavior to dominate the first subset of the data even as we begin working on a goal because the kinds of lasting change we are interested in tend to occur slowly. Assuming our intervention is working, the further along we go in time, the less dominant the pre-intervention patterns should be. So, we expect the middle subset of our data to correspond to the transformation stage of change. The period of transformation is regular only in its irregularity. During this time we cannot predict the child’s actions from session to session. He or she will do better on some days than on others, try new responses in some situations but revert to old behaviors when too challenged or uncertain. It is the knowledge that this part of our data should be noisy that allows us to ignore it. Finally, our definition of change requires the evidence of a new pattern of regular behavior in the child. So we expect the remaining subset of our data to correspond to “after”—that period of time during which the child’s behavior is best explained by the new ways of thinking and feeling that have resulted from intervention. Figure 7.6 overlays these ideas on the line graph for Anton’s data.
2
In standard psychology experiments statistics are used to reject the null hypothesis. What this means is that you hypothesize that there has been no change as a result of the experiment. Then you hope that the statistic shows that the probability of the before and after scores being from the same set of participants is so low that you can safely reject the hypothesis of no experimental effect. Yes, our way of phrasing it is far less confusing. As long you remember to fit the data to the hypothesis and not the other way around, you’ll do fine with the simpler characterization.
113
FROM GOALS TO DATA AND BACK AGAIN
Does each phase of transformation really correspond to exactly one third of the data all the time? No, of course not. But thirds are a good approximate solution to the practical constraints. As a general rule, we want at least ten data points in each of the “before” and “after” data sets to compute reasonable statistics (don’t even bother to analyze data with fewer than five values in each set). If we have one observation every day for each goal this method requires a month of behavior (30 days) as the minimum time span for analysis. This works out nicely in cognitive terms—another general principal is that lasting changes in many kinds of behavior take at least a month of practice. Of course, if we collect data less frequently than once a day we have to adjust the time span for the analysis accordingly. For example, if you work with a child five days a week you need at least six weeks to collect ten “before” and ten “after” data points. If you work with a child only three times a week, you need nearly three months of data for an adequate pool of observations. Since quarterly analysis is required by many educational systems and service providers, dividing the data into thirds makes it possible to meet their constraints on a three-times-a-week-or-more 3 schedule and still have at least ten observations for each mean.
5 4 3 2 1 0
“BEFORE”
LEARNING
3/6/00
2/28/00
2/21/00
2/14/00
2/7/00
1/31/00
1/24/00
1/17/00
1/10/00
“AFTER”
1/3/00
4-point scale (1,2,3,4)
Anton, Goal X
Figure 7.6 Our model of cognition assumes that if change occurred within the intervention period then behavior went through three phases: pre-transformation (“BEFORE”), when old patterns dominate; transformation, when learning occurs; and post-transformation (“AFTER”), when behavior is best characterized by the new patterns learned during intervention
If you work with a child only once a week you need about seven months of data to perform a comparison of means as we describe it. This is impractical for professionals who must report on a quarterly basis. You can increase the amount of data you collect without increasing the number of intervention sessions by combining your data with data taken by another observer (a parent, for example) on days you do not work with the child. If you increase your data set in this way you must also check the correlation between observer and performance, as discussed in Chapter 9. You still may not manage to collect ten observations from each of the first and last third of the time interval; the fewer observations you have, the less certain you should be of your conclusions.
3
114
DETECTING CHANGE – THE MEAN
Theoretical and practical justifications aside, it is still the case that sometimes there will be change and that change will be obscured by the decision to divide the data into thirds. That is the price of consistency and repeatability. As we work through the examples in Chapters 8, 9, and 10 we will point out patterns that may arise in the data due to this decision. For now, let’s continue with our explanation of the comparison of means.
Step 2: Computing the means and charting bar graphs
Having separated out “before” and “after” from the raw data, the next step is to compute the mean for each subset. When we perform this step for Anton’s data, we find that the “before” mean equals 1.4 and the “after” mean equals 3.6. Exercise 7.3 (Appendix C) gives you a chance to practice this step, building on the skills learned in Exercises 7.1 and 7.2. As we noted in Chapter 6, a picture may give a clearer and more compelling explanation of the pattern in the data than the numbers alone. To create a graph that contrasts the “before” and “after” means for a goal we use the column bar graph, as shown in Figure 7.7. Exercise 7.4 (Appendix C) takes you step-by-step through the process of creating bar graphs like this one.
4-point scale (1,2,3,4)
Comparison of Means, Anton, Goal X 4.0 3.0 1/3-1/26
2.0
2/18-3/10
1.0 0.0 Before (1.4) versus After (3.6) Figure 7.7 Anton’s performance on goal X “before” and “after” intervention
Sometimes we want to start with a broad view of the child’s growth over the analysis period before focusing in goal by goal. Most graphing programs make it easy to show all of a child’s goals on a single graph (see Exercise 7.5, Appendix C). Unfortunately, these programs even allow us to combine goals that shouldn’t be combined. It is inappropriate to combine goals that have different size scales because the resulting graph can create perceptual biases. In other words, you can combine goals with 4-point scales in one graph but you must produce a separate graph for goals with 5-point scales and another for the goals with frequency values (100-point scales). 115
FROM GOALS TO DATA AND BACK AGAIN
Step 3: Comparing the means
All right, we’ve divided the data, computed the means and produced column bar graphs. It’s time to figure out what all this prep work actually tells us. Progress: yes or no? Since the differences in Anton, Becca, and Celeste’s line graphs for goal X were what originally sent us down this path, let’s see if using the mean has helped clarify the situation. We start with the easiest case: Anton. It seemed clear from his line graph that he had made real progress on this goal after about two and a half months of intervention. At the very least we expect that a comparison of means for Anton’s data will show the same result. In Step 2 we computed the “before” and “after” means for Anton’s data on goal X and found that they were 1.4 and 3.6, respectively. Figure 7.7 showed the same information in graphic form. Step 3 says that the means must represent different categories of behavior if we’re going to conclude that there has been change. How do we know whether the means reflect different categories of behavior? The label on the vertical axis of the graph and the label at the top of Anton’s spreadsheet remind us that this data was collected using a qualitative 4-point scale. Recall that for this type of data, each step on the scale reflects a different category of behavior. In other words, we chose each step of the scale to reflect what we felt would be a qualitatively different level of performance by Anton. When we translated the steps on the scale into numbers, each successive level of performance was assigned a number that was one higher than the level before. A score of 1 in Anton’s data means that Anton’s behavior that day fell in the lowest category on the scale. A score of 2 means Anton’s behavior was described by the next higher category, a 3 means the next higher, and a score of 4 means that Anton’s behavior fell in the highest category on the scale. Because of this correspondence, we define a qualitative scale’s category width to be 1.00 (Chapter 9 contains an example of how to calculate category width for percentage data and Chapter 10 has an example of category width for tallies). Since the difference between Anton’s means is about 2, Anton’s performance progressed across two category boundaries. Impressive progress in such a short time. Because Anton is one of our “variable” children, we don’t have access to the information behind the data to interpret the results of the analysis further. To understand the nature of this degree of change in a real child we would now go back to the wording of the scale that underlies this graph. What exactly was goal X and what are the skills associated with achieving it? What does it mean to have started at an average score of 1.4 in January and what does it mean to have ended up at an average score of 3.6 in March? Is this a shift in an emerging goal or does this show solidification of a skill across multiple environments? Without going back we cannot go forward—we can’t use our interpretation to think about what to do next. Although we have gone as far as we can in Anton’s case,
116
DETECTING CHANGE – THE MEAN
remember that what we do with statistical information is part of the larger on-going process of intervention. We will have a chance to explore the whole process in Chapter 10 when we apply the analytical tools we are developing here to the cases of Joey, Tyler, and Mai Lin. Let’s skip Becca for a moment, and jump ahead now to Celeste’s data. Recall that Celeste’s performance produced the most difficult line graph to interpret in Chapter 6. The comparison of means in Figure 7.8 makes the situation clear: the change we expected has not occurred. We know this because we must apply the same rules to Celeste’s data that we applied to Anton’s data. Given a qualitative scale with behaviorally distinct steps, different categories of behavior are defined by a numerical difference of at least 1.00. The difference in Celeste’s means is in a promising direction—perhaps Celeste is just entering the transformation stage—but the size of the difference (0.60) falls short of our boundary condition (a difference of at least one category width). Because it is much less than a category width (1.00), the difference is too small to justify the idea that there has been a “reliable” category change.
4-point scale (1,2,3,4)
Comparison of Means, Celeste, Goal X 4.0 3.0 1/3-1/26
2.0
2/18-3/10
1.0 0.0 Before (2.4) versus After (3.0)
Figure 7.8 Celeste’s performance on goal X “before” and “after” intervention
The question of just what we mean by “reliable” is at the heart of the analysis of Becca’s performance on goal X. The graph comparing Becca’s “before” and “after” means (2.1 and 3.1) is shown in Figure 7.9. Without the word “reliable” in Step 3 of our method, we could conclude that Becca’s data matches the pattern of change we hypothesized and move confidently on to interpreting the change in terms of the original scale. But our confidence has to be tempered by the actual value of the difference of the means (1.00), which is just enough to meet our boundary condition. Remember that the mean is a representative value indicating, on the average, what the child’s behavior was over a predicted stage of the intervention. Just as in the case of dividing the data into thirds, we need to 117
FROM GOALS TO DATA AND BACK AGAIN
have a consistent, well-defined boundary condition for category change; but when our interpretation relies completely on that arbitrary boundary, we need more evidence.
4-point scale (1,2,3,4)
Comparison of Means, Becca, Goal X 4.0 3.0 1/3-1/26
2.0
2/18-3/10
1.0 0.0 Before (2.1) versus After (3.1)
Figure 7.9 Becca’s performance on goal X “before” and “after” intervention
Without question there are other kinds of evidence we should always consider: parent report (“She’s really much more social than before the holidays. She joins the rest of the family after dinner almost every night now.”), teacher report (“Yes, I’ve started to see her interacting with the other kids at lunchtime, too.”), and so on. These typical sources are informative and important, but subjective. Since one of the reasons we use data is specifically to provide a different (objective) source of information, it would be a shame to give up on this new source after so much work. Luckily, our statistical toolbox has just the right-sized hammer for the problem. If the mean establishes a representative value for performance, its partner, the standard deviation, describes just how representative that value is. Standard deviation is the tool we use to measure the reliability of change. It is the topic of the next chapter.
Things to remember T Change is the transformation of one regular pattern of behavior into a
different regular pattern of behavior.
T Regularity has predictive value. This means that when we notice a
regular pattern we expect that pattern to continue and rely on that pattern to predict future behavior.
T Data analysis depends on our expectations; our expectations depend on
our knowledge.
118
DETECTING CHANGE – THE MEAN
T Noise is the phenomenon that hides regularity. Real behaviors, like real
children, are inherently noisy.
T To compute change for a goal:
1.
Divide the data for that goal into thirds. The first third of the data corresponds to “before” intervention; the last third of the data corresponds to “after” intervention. Ignore the middle third of the data, as it corresponds to the period of transition.
2.
Compute the mean of the “before” data and the mean of the “after” data independently.
3.
Compare the two means in terms of the scale for the goal. If the two means reliably reflect different categories then change has occurred.
T Whether the data matches our expected pattern of change is something
we can determine from a comparison of means (and standard deviations). But to understand the pattern of change in behavioral terms we must go back to the underlying meaning of the data that is found in the wording of the scale and the original discussion that led to the goal being measured.
119
Chapter 8
Measuring Reliability — The Standard Deviation
There are two reasons we chose a comparison of means as the statistical measure for detecting change. First, we recognize that the circumstances under which parents, teachers, and therapists collect data don’t satisfy the conditions for using more powerful statistical tests. Second, because the mean acts as a representative value for a set of observations, using it seems to be at least a partial solution to the problem of dealing with noise in a uniform and repeatable way. Our choice introduces a new problem, however. While it’s true that every set of numbers has a mean, there is nothing about the way we compute the mean that guarantees that it’s a good representative value, nothing that guarantees that it will allow us to predict behavior accurately in the future. How certain can we be that the information given by a comparison of means reflects a reliable change in behavior? How well each mean represents its values determines how certain we can be of the story our data tells. The standard deviation measures how representative the mean is, and therefore measures the certainty of our story.
Distance as a measure of representation When we vote for political candidates, we expect that, if elected, they will represent us by making the same sort of decisions we would make in their place. Realistically, we don’t expect every decision to be made exactly as we would have made it. Instead, our sense of how good or bad our representatives are depends upon how often they make decisions the way we would or how different their decisions are from what ours would have been. Typically, the language we choose to reflect our judgment of how well a candidate represents us is based on a distance metaphor. So, for example, we might talk about “how far apart” our views are or “how close we are” to the candidate on a particular issue.
121
FROM GOALS TO DATA AND BACK AGAIN
When judging whether the mean is a good representative for a set of observations, the intuition is similar. Of course in intervention, our candidate (the mean) has to be faithful to its constituency (the data) on only one issue: representing the child’s behavior for a particular goal. And although the concept of distance is used metaphorically in the political arena, in statistics distance is a real and measurable thing. To further our intuition about the relationship between distance and representation, let’s look at two different data sets. The first, Figure 8.1, shows Anton’s scores for goal X during the post-transformation phase of intervention and a line at the mean value (3.6) for this data set. Observations versus Mean
4-point scale (1,2,3,4)
Anton, Goal X
5 4 3 2 1 0 1
2
3
4
5
6
7
8
9
10
Session in "After" phase
Figure 8.1 Good representation displayed as a tight clustering of data points around the mean line
There are two things we want you to notice about this figure. First, each observation is close to the mean. In fact, the largest difference between the mean and any observation is 0.6, which is less than the size of one category width on the rating scale. The second thing to notice is that the mean line is actually closer to the values of 4 than to the values of 3. To push our political metaphor, we might say that when called upon to serve, the mean would act a little bit more like a 4 than like a 3. Since there are more 4s than 3s in the data set this makes sense: a value that is closer to 4 is more representative of the set than one that is closer to 3 or exactly half way between 3 and 4. The small distances between the data points and the mean make the mean value for Anton’s data a good representative of Anton’s performance in the past. Because of this, we feel confident using it to predict Anton’s performance in the future. By “electing” the value of 3.6 to represent his data we are saying that 122
MEASURING RELIABILITY – THE STANDARD DEVIATION
from this point forward we expect to see “votes” of 3 and 4, reflecting continuing regularity, in Anton’s behavior. Now let’s look at Figure 8.2, which contrasts Celeste’s post-transformation scores for goal X with the mean (3.0) for those values. Although the mean for Celeste’s data was computed with exactly the same formula as the mean for Anton’s data, the observations don’t seem to cluster around the line the way they did in Figure 8.1. Although a few values lie on the mean line, the smallest distance between any of the remaining observations and the mean is 1.00, which is both a full category width and almost twice as big as the largest distance in Anton’s data. Suppose you were the value of 1 observed in the fourth “after” session. Would you have elected 3 as your representative?
Observations versus Mean
4-point scale (1,2,3,4)
Celeste, Goal X
5 4 3 2 1 0 1
2
3
4
5
6
7
8
9
10
Session in "After" phase
Figure 8.2 The more the data points spread out around the mean line the poorer the mean is as a representative value
The large and varied distances between Celeste’s data points and her mean indicate that the mean is a poor representative for its “constituency.” Its value is a compromise among data points that have little in common. Because of this, we cannot be confident that the mean will be a good representative of Celeste’s behavior in the future, either. We should not use it to predict values of 3, or any other regularity, in her behavior going forward.
123
FROM GOALS TO DATA AND BACK AGAIN
Computing and graphing the standard deviation The standard deviation is a number that reflects our notion of representation as a function of distance from, or clustering around, the mean. The formula for this statistic is rather ugly and cumbersome if you are not a computer. Fortunately, we can let Excel handle the actual computation by applying the STDEV function in our spreadsheets. STDEV is taught in Exercise 8.1 (Appendix C). Since our method for judging change involves two means, we also need to compute two standard deviations, one for the “before” data and one for the “after” data set. Over time you will come to recognize the presence or absence of change directly from these four numbers. As you build that skill, however, you may find it helpful to look at a visual representation of the critical information; and, of course, it may be important to have a visual for explaining your analysis to others on the team. In Chapter 7 you learned how to produce bar graphs in Excel in order to contrast the “before” and “after” means. To show the standard deviation around each mean we augment the bar graph with error bars. An example is given in Figure 8.3. Exercise 8.2 takes you step-by-step through the process. Notice that the error bar for each mean extends both above and below that mean by the amount of its standard deviation. Also note the area of overlap between the bars indicated by the oval. This overlap will help us determine the reliability of change.
Comparison of Means mean + standard deviation
4-point scale (1,2,3,4)
6.0
mean – standard deviation
mean
4.0 2.0 0.0 Before
versus
After
mean = 2.6
mean = 3.9
sd = .85
sd = .92
Figure 8.3 Standard deviations displayed as error bars around the means. The bar extends both above and below the mean by the amount of the standard deviation. The oval indicates the area of overlap between the bars 124
MEASURING RELIABILITY – THE STANDARD DEVIATION
Evaluating change using mean and standard deviation The method of data analysis we introduced in Chapter 7 was problematic because it did not give a concrete definition of reliability. Now that you know how to compute the standard deviation for a set of observations, we can reformulate the method to remove that problem.
Computing the comparison of means statistic
To compare the “before” and “after” means for each goal: 1.
Divide the data for the goal into thirds. The first third of the data corresponds to “before” intervention and the last third of the data corresponds to “after” intervention. Copy the “before” and “after” data to a new spreadsheet. Ignore the middle third of the data.
2.
Compute the mean and standard deviation of the “before” data and the mean and standard deviation of the “after” data separately.
3.
Compare the two means in terms of the scale for the goal. If the difference between the means is not a full category width apart, assume that there has been no reliable change.
4.
If the two means are at least a full category width apart, graph the means and standard deviations and compare the reliability of the change by looking at the amount of overlap in the error bars.1 (a)
If there is no overlap conclude that the difference in means signals reliable change and a new regular pattern of behavior. If the “after” standard deviation is more than about half a category width consider continuing intervention to achieve consistency in the new skill or behavior. Otherwise, the goal is achieved in its current form.
(b)
If there is overlap, it is crucial to consider outliers and correlations (Chapter 9), as well as all the other sources of information available before coming to a conclusion. If no additional information is available, however, use the following guideline: • If the overlap is less than 1/3 the difference in the means, assume that the overlap reflects normal variability in performance and conclude that reliable change has occurred.
1
For those who prefer numbers to pictures, use the following formula: Overlap = Meanbefore + Standard Deviationbefore – (Meanafter – Standard Deviationafter).
125
FROM GOALS TO DATA AND BACK AGAIN
As in (a), if the “after” standard deviation is large, consider continuing intervention to achieve consistency.
• If the overlap is greater than or equal to 1/3 the difference between the means, assume that the change in behavior is not yet reliable. This procedure is also available as Chart 8.1 in Appendix B. Step 4b of this procedure is complex because the issues surrounding reliability can be complex and a judgment about the child’s performance requires both thought and discussion. Your decision—whether to ignore the overlap, to continue with the goal in modified form, or continue with the goal as is—must be sensitive to both the degree of overlap and why that overlap arises. In general, the more overlap there is, the less likely it is that reliable change has occurred. But in a specific instance there may be reasons why the overlap can exist even when reliable change has occurred. The two most common reasons (outliers and factors that systematically interfere with performance) are discussed in detail in the next chapter. However, even if those reasons do not explain the overlap, the team may identify other reasons for the child’s variability in performance through discussion and such discussion should be considered seriously. Remember: data analysis provides only one source of information. When all else fails, the team can choose to rely on the boundary value of 1/3 for their decision making. There is nothing magical about using this fraction—we need to choose some number to keep the analysis consistent and repeatable. Some teams may feel that this fraction suggests reliable change too easily, other teams may feel that it is too rigid a criterion. What is important is that you choose a value and use it consistently. What is more important still is that you understand what the boundary condition means. We add error bars to the graph to indicate how good a representative value the mean is and how much regularity there was in the data set it represents. If the means are at least a category width apart and they are good representatives of the data, then we can believe in the behavioral change the data signals. But, if the means are not good representatives of the data, we need to know just how poor they are. The poorer the representation, the larger the standard deviation will be. As the standard deviation around each mean grows, the amount of overlap between the error bars increases. If the standard deviations/error bars around each mean are large enough relative to the distance separating the means then we can no longer rely on the change in mean to indicate a change in behavior. Let’s see if this really makes sense. Figure 8.4 contrasts three graphs. In each graph the “before” mean and the “after” mean are the same, but as we proceed top to bottom through the figure the size of the standard deviations around the means grows. In the top graph, the top of the “before” mean error bar rises just a little bit above the bottom of the “after” mean error bar. In the middle graph, the overlap is larger—about a third of the total difference between the means. In the 126
MEASURING RELIABILITY – THE STANDARD DEVIATION
bottom graph the overlap is almost as big as the entire difference between the means: the top of the “before” error bar comes up almost as far as the “after” mean and the bottom of the “after” error bar comes down almost to the “before” mean. Glance back at Figures 8.1 and 8.2 to remind yourself visually that a bigger standard deviation means less clustering of observations around the mean. Now think about the fact that the observations represented by Figure 8.4 cannot be smaller than 1 or larger than 5 (given the 5-point scale). So, as the standard deviation grows around a “before” mean that is close to the bottom of the scale, the values that are very different from the mean are probably large values. Similarly, as the standard deviation grows around an “after” mean that is close to the top of our scale, the values that are very different from that mean are probably small values. To put it another way, as the standard deviations get bigger, the data in the pre-transformation phase and the data in the 2 post-transformation phase have more values in common. The overlap in area of the graph corresponds (very roughly) to an overlap in the data sets. The more the data sets share values, the less it seems that there are the two different regular patterns of behavior required by our definition of change. The graphs in Figure 8.4 were chosen to make a point; they are not the only patterns that you might see in your data. In the remainder of this chapter and again in Chapters 9 and 10 we examine and discuss a number of additional scenarios. But we don’t have the room (and you probably don’t have the patience) to examine all the possible cases. So instead of presenting hundreds of examples, we’ve set up a small laboratory in Exercise 8.3 to help you develop a feel for the relationships among raw data, mean, and standard deviation. We encourage you to spend some time experimenting in the lab before you continue. OK, we’re finally ready to tackle the problems that pushed us to examine the standard deviation in the first place: Anton, Becca, and Celeste’s data sets. As usual, Anton is our poster boy for change. His “before” and “after” means are two full categories apart and, when we add the error bars around the means, there is no overlap (Figure 8.5). His “after” standard deviation is much less than a category width. All in all, Anton’s data gives every indication that his behavior has changed reliably and settled in to a new, more desirable pattern. If, for example, goal X is measuring emergence of a skill with support, it looks like it’s time to move this goal into Phase 2, reducing support and solidifying the skill 2
If the two means are not close to the top and bottom of the scale then they are less likely to have at least a category’s width between them. If they are close to the middle of the scale and still a category apart, the argument is more complicated but the conclusion is the same: the more overlap, the less you should trust the reliability of the change. For those of you who are worried about the effect of outliers on the comparison of means, we discuss outliers in Chapter 9.
127
FROM GOALS TO DATA AND BACK AGAIN
A small overlap in the standard deviations 5 4 3 2 1 0
The same means, but some overlap 5 4 3 2 1 0
The same means with a large overlap 5 4 3 2 1 0
Figure 8.4 The more overlap in the error bars, the less reliable the change indicated by the difference in means
128
MEASURING RELIABILITY – THE STANDARD DEVIATION
with more natural rewards. If the goal is already in Phase 2 and the scale measures progress in a familiar surrounding, it looks like it’s time to consider Phase 3 for this goal, generalizing it to new environments. However we decide to build on Anton’s progress, we can at least feel comfortable predicting that “if nothing changes” his behavior will continue (under the current goal qualifiers) in the new, more adaptive way. Becca’s data was a problem in Chapter 6 and remained a problem in Chapter 7. Recall that the means for her “before” and “after” data sets were exactly one category width apart. Since that’s a boundary value in our statistical method, an interpretation of progress could have gone either way depending largely upon the evidence contributed by other sources. Adding standard deviation information to her graph, however, argues against an interpretation of reliable change (Figure 8.6), as the overlap in the error bars covers nearly all of the difference between the means. Comparison of Means, Anton, Goal X
4-point scale (1,2,3,4)
5 4 3 2 1 0 Before (mean=1.4, sd=.52
After mean=3.6, sd=.52)
Figure 8.5 A clear example of change
If we look at Becca’s data (“Becca Goal X Before–After” in the ABC Workbook), it’s clear that the overlap in the standard deviations around Becca’s “before” and “after” means comes from two sources. First, there are a few values of 3 in the pre-transformation phase that would look more at home in the “after” data set. Second, there is a value of 1 and a value of 2 in the post-transformation phase that are more indicative of pre-transformation behavior.
129
FROM GOALS TO DATA AND BACK AGAIN
4-point scale (1,2,3,4)
Comparison of Means, Becca, Goal X
6.0 4.0 2.0 0.0 Before
After
(mean=2.1, sd=.74 mean=3.1, sd=.99)
Figure 8.6 The large standard deviations argue against the evidence of change provided by the difference in the means
On the surface these results look like Becca might be on the way to a new regular pattern of behavior but needs more practice. Another way of saying this is that Becca’s post-transformation phase may have been longer than we expected. If we keep providing intervention for this goal we will have the opportunity to extend our view over time, gathering more data as we did for Danny in Chapter 7. And as we saw in that example, extending our view can push a portion of the data more solidly into the transformation stage where its lack of regularity fits our expectations. So, if Becca is on the way to lasting change and we continue intervention, then we would expect to see a smaller standard deviation around the “after” mean of the extended data set as her new behavior becomes reliably consistent. If you find yourself questioning the story we just told about Becca’s graph, arguing that there are other explanations for these results…good for you! No interpretation can be considered complete until we’ve gone back and looked at the original ideas and arguments that caused us to choose the goal and scale that underlie the data. Of course, since Anton, Becca, and Celeste are cases we made up for illustrative purposes we don’t have that information and can’t perform that critical step of the analysis. Don’t worry, though; we’ll have the chance to complete three interpretations in Chapter 10 when we analyze Joey, Tyler, and Mai Lin’s data. In the meantime, let’s turn to Celeste. Celeste’s data failed the test for change at step 3 of the comparison of means, so adding error bars to the graph we created in Chapter 7 isn’t strictly necessary. Still, it is useful to note that the tale told by the standard deviations is consistent with our previous interpretation. Figure 8.7 shows that what difference there is in the means is overwhelmed by variability in behavior both “before” and “after” intervention. We design intervention based on the expecta130
MEASURING RELIABILITY – THE STANDARD DEVIATION
tion that the three phases of change can occur in terms of a given measurement scale and within a given interval of time. In Celeste’s case this expectation has not been met.
4-point scale (1,2,3,4)
Comparison of Means, Celeste, Goal X
6.0 4.0 2.0 0.0 Before
After
(mean=2.4, sd=.84 mean=3.0, sd=1.05)
Figure 8.7 Comparison of means for Celeste
Perhaps Celeste is not ready for goal X as it is currently qualified. She may not be ready to act adaptively without a great deal of support. She may not be ready to use the new skill outside of familiar settings or with unfamiliar people. On the other hand, perhaps the type of quantification chosen for goal X is the problem. If the scale contains categories that are too far beyond Celeste’s reach at this point in time, she may be responding to intervention but over too small a portion of the scale to be noticeable. The point we want to make here is that although analyzing the data does not give us the information we expected, it gives us useful information nonetheless. In Celeste’s case it signals to the team that it is time to re-evaluate the assumptions that underlie the choice of this goal. So, we’re done, right? With graphs in hand we’re ready to add our data analysis to the other sources of information presented at the next team meeting. Actually, we’re close, but no, we’re not done quite yet. While we have wrung pretty much all the information we can out of the goal data alone, there’s still the factor data to consider. What it has to add to the story is the topic of Chapter 9.
131
FROM GOALS TO DATA AND BACK AGAIN
Things to remember T The standard deviation is a statistic that measures representation as a
function of distance from, or clustering around, the mean. The smaller the standard deviation, the better the mean is as a representative of prior behavior and the better it predicts future behavior.
T If the standard deviations around the “before” and “after” means are large
enough relative to the difference separating the means then we can no longer rely on the difference in mean to indicate a reliable change in behavior.
T Not even the most compelling graph of the “before” and “after” means
and standard deviations completes an analysis. Every interpretation requires that we go back and look at the original ideas and arguments that caused us to choose the goal and scale that underlie the data before making the next set of intervention decisions.
132
Chapter 9
Exceptions That Prove the Rule — Factor Data
Like all statistical methods, the comparison of means is based on the idea of finding similarities and differences between the data we expected and the data we collected. When we analyze data we do it against the backdrop of those expectations. So far our method allows us to recognize the pattern of change that is typical of developmental learning when it appears in the data over a given time period. But what happens when our expectations aren’t met? The tools in our computational toolbox—graphing, mean, and standard deviation—are good ones to have when things go right, but they aren’t particularly useful when things go wrong. When we don’t see the behavioral growth we’re expecting, we’d like to be able to tell a better story than just “too much noise” or “no change.” We’d like to know if there are reasons behind the apparent lack of progress. We’d like to know if there are regularities hidden within the noise. Having had the forethought to collect factor data, now is the time to look to that source of information to help clarify the picture.
Identifying an outlier in the data set Figure 9.1 shows the means for Anton’s data for goal Y. The standard deviation of the “after” data is large and overlaps a bit of the space marked off by the error bar of the “before” data. Applying the comparison of means method, we know that we must consider other relevant sources of information before attributing developmental change to these results. It’s possible that the large standard deviation in the “after” data does reflect ongoing difficulty in the consistent use of the new skill. When we look at Figure 9.2, however, we see that another possibility exists. The values for goal Y in most of Anton’s last ten sessions are exactly what we would expect if Anton’s progress was solid. The exception, a value of 1, appears in Session 27. This unusually low value has two effects on the comparison of means in the previous figure: it pulls 133
FROM GOALS TO DATA AND BACK AGAIN
the “after” mean down and it contributes most of the standard deviation around that mean. A single value that stands out against an otherwise regular pattern of behavior is called an outlier.
Comparison of Means, Anton, Goal Y
4-point scale (1,2,3,4)
5.0 4.0 3.0 2.0 1.0 0.0 Before
After
(mean=1.9, sd=.57 mean=3.4,sd=.97)
Figure 9.1 Overlap in the error bars suggests we look further
4-point scale (1,2,3,4)
Anton, Goal Y
6 4 2 0 1
2
3
4
5
6
7
8
9 10
21 22 23 24 25 26 27 28 29 30 Sessions
Figure 9.2 Post-transformation behavior (sessions 21 through 30) looks regular except for a single unexpected low value in Session 27
If we recompute the means without the observation in Session 27 (see Figure 9.3), Anton’s data set for goal Y looks every bit as clear and exciting as his data for goal X did in the previous chapter. As tempting as it is to simply ignore the outlier and move onto Anton’s next goal, however, good practice demands that we look for some explanation for his performance in Session 27 before dismissing it as uncharacteristic. Broadly speaking, there are three possible explanations for Anton’s difficulties: 134
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
4-point scale (1,2,3,4)
Comparison of Means, Anton, Goal Y, Revised (session 27 omitted)
5.0 4.0 3.0 2.0 1.0 0.0 Before
After
(mean=1.9, sd=.57 mean=3.7, sd=.50)
Figure 9.3 The expected pattern becomes clear after removing the outlier
1.
His score is related to a goal-independent factor we are already tracking.
2.
His score is related to a goal-independent factor we are not tracking.
or 3.
His score is related to a goal-dependent factor of which we are unaware.
Recall that we track particular goal-independent factors like time of day, observer, location of intervention, and medication regimen because we already have good theoretical or empirical reasons for believing that those factors can affect performance in children with autism. In other words, our willingness to spend the time and energy tracking those factors is due to a tentative expectation (or hypothesis) that they could explain performance in this particular child as well. To test the hypothesis that Anton’s poor performance might be explained by a factor we are already tracking, we look for a factor that, like goal Y, has a value for Session 27 that is different from its typical values. We might find, for example, that the day of Session 27 was the only time in the “after” phase that Anton missed his medication or that he received intervention in the afternoon. If we did find that Session 27 was the only day that intervention was switched to the afternoon, our hypothesis would be supported. We might conclude that Anton is a “morning person,” at least with respect to the skills involved in goal Y. Then, having some justification, we may use Anton’s recomputed means (Figure 9.3) as the more realistic representation of his progress and plan the next phase of goal Y accordingly. In addition, the team now has some 135
FROM GOALS TO DATA AND BACK AGAIN
evidence for the usefulness of this goal-independent factor in predicting Anton’s behavior and might choose to spend some time and energy to understand it more fully in order to provide adaptations or intervention designed to extend his competency in the afternoons. Suppose, however, that there were days in the “after” phase when Anton had intervention in the afternoon and his performance on goal Y did not suffer. Such observations would violate our hypothesis that time of day might account for his performance in Session 27 and force us to explore alternatives. If we find that none of the factors we are tracking seems to offer much of an explanation for Anton’s bad day then we must consider whether there is some other goal-independent or goal-related factor that we should be measuring. In other words, an outlier can be seen as an opportunity to form a new hypothesis about factors that affect progress. In trying to form a new hypothesis it may be helpful to look at Anton’s performance on other goals during the same session. For example, if his performance seems to have suffered across the board on that day, then the data suggest that the relevant factor is goal independent. But if, for example, the scores for social goals were uniformly lower in Session 27, but those for communication goals were not, this suggests that Anton’s sensitivity might be to some aspect of the social activity that isn’t required by the communication skills. Another possibility is that goal Y contains the only clear outlier, suggesting that the difficulty is related to something specific to that goal. Additional information might be found in Anton’s data collection sheets in the form of a notation or comment that identifies an aspect of the skill or situation that was unusually problematic. Or it may be that team members who were present on that day can recall the events and identify a possibility. If a hypothesis can be formed, the team must consider next whether to track the new factor to test the hypothesis, or target the new factor intentionally in a revised version of the current goal. If no information is available, however, then the team must decide between two alternatives: to take the data as is, continue intervention for a while and then re-evaluate, or to discount the outlier and simply move the goal to its next phase. Remember that an outlier is just a single value that stands out against an otherwise regular pattern. Nothing in this definition requires the outlier to be a particularly low value for a goal. In fact, an unusually high value can also make progress look worse than it is if it occurs in the “before” portion of the data where it would increase both the mean and standard deviation in the “before” phase. The effect on the comparison of means, then, would be the same as that produced by an unusually low value in the “after” phase: there will be less of a difference between “before” and “after” due to a single value. An unusually high score may also exaggerate performance, making the child’s gains look better than they probably are. This happens when the high 136
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
value occurs in the “after” data, pulling that mean up. You don’t want to let exaggerated progress mislead you into refocusing intervention too soon. Of course, paying attention to an outlier that is uncharacteristically high may be even more important than paying attention to an outlier that is uncharacteristically low because explaining an unusually high value may lead you to identify a factor that improves performance. Such a factor can give you insight into the child’s strengths that can, in turn, lead to more effective adaptations and intervention. What if there is more than one outlier in the graph? With large “before” and “after” data sets you might see more than one point that stands out against an otherwise clear pattern of behavior. If you find you need to explain each outlier with a different factor, be wary. Every new explanation you must bring to the analysis should make you that much less certain that you’re looking at a regular pattern in the child’s behavior. If, on the other hand, the multiple outliers seem to be explained by the same factor—that is, the values of the factor are the same on the outlier days and different from the values of the factor on non-outlier days—then you’re not really looking at multiple outliers, you’re looking at a consistent sub-pattern within the data.
Predicting sub-patterns of behavior from factors By definition an outlier is identifiable because it isn’t part of a pattern. Sometimes, however, a factor has influenced behavior in so many sessions that it obscures the pattern of change we are looking for by weaving a pattern of its own into the data. The statistic for detecting when this happens is correlation. The basis of this statistical method is very similar to the reasoning we went through in the previous section: we say that two variables are correlated when variation in the value of 1 variable seems to predict variation in the value of the other variable. Although we will usually be looking for a correlation between factor data and goal data, in general it is possible for two factors or even two goals to be correlated. Correlation is a useful tool because of its predictive power. We take advantage of the predictive aspect of correlation every day when, for example, we choose a route to take to work or a movie to see at the theater. How do these decisions rely on correlation? Suppose there are a few different ways to drive from home to work all covering more or less the same distance according to the odometer. Nevertheless, over time we notice that which route we take makes a big difference in how long it takes to get there. What we’ve discovered is a correlation between route and travel time. The correlation isn’t perfect—we can’t predict with certainty that on a given day the “best” route will be the fastest. There may be a detour or an accident or especially heavy traffic. But we assume, all other things being equal, that variation in route predicts variation in arrival time in that one route gets us to work earlier than the others.
137
FROM GOALS TO DATA AND BACK AGAIN
Similarly, our experience may lead us to notice that there is a high correlation between an actor and our enjoyment or dislike of a movie. Again, the correlation isn’t likely to be perfect—not even the casting of a favorite actor can save some movies and sometimes actors we dislike occasionally appear in great films. All other things being equal, however, we assume that the particular value of one variable (actor) predicts a better or worse outcome in another variable (our movie-going experience) when we make our choice. The phrase “all other things being equal” is an important aspect of the examples given above. We don’t, in the driving example, use correlation to choose between highway roads and surface roads. Highways and surface streets are in different categories with respect to our expectations and predictions. We know that highways are generally faster than surface streets for medium or long distances and can rely on that knowledge independent of our experience with particular routes of either type. In the same way, we don’t typically decide among movies on the basis of actor when one movie is a drama, one is a comedy, and the third is a horror film. Again, the differences among the categories seem to be more important in making the decision than a variable that might correlate within each category. This same reasoning applies to our developmental data. In general we look for a correlation between variables only within each phase of change because the phases themselves are assumed to define substantially different categories (“before” intervention, during learning, and “after” intervention).
Correlation is not cause It is extremely important to understand that the ability to predict behavior is not the same thing as understanding the cause of the behavior. It would never occur to you to think, for example, that the road itself is what causes you to be on time. You know from your commonsense understanding of the world that the real cause of when you arrive at work is a complex combination of factors like weather, traffic patterns, stop light timing, bus stops, etc. These factors are also the reason why there is a correlation between route and arrival time. Unfortunately, when we don’t have a good understanding of the causes behind a correlation, we often make the mistake of believing that the factor we use to predict behavior is also the cause of that behavior. Here’s an example we see far too often: a child (we’ll call her Kyra) who has a social goal (for example, to share toys with peers) is supposed to take medication every morning. When we look at Kyra’s data we find a high correlation between medication and sharing. In other words, when Kyra takes her medication she does well on the social goal and when she misses her medication she does poorly on the social goal. Given this result we would be correct if we predicted that on days when Kyra does not take her medication she will have difficulty sharing toys with her school peers. We would not be correct, however, if, solely on the basis of this data 138
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
and correlation, we concluded that the lack of medication caused her poor performance (or that the medication caused her skilled performance on days when it was taken). We would even be incorrect if we concluded that the lack (or presence) of medication contributed to her poor (or good) performance. How can that be? Well, suppose, for example, that on days when Kyra wakes up feeling agitated and experiencing a lot of tactile sensitivity her tantrums are so severe at medication time that her parents decide not to fight that particular battle. When Kyra gets to school, the same tactile defensiveness makes it impossible for her to be near the other children, listen to their requests, and share her toys. In other words, the same difficulty that caused Kyra to miss her medication (her sensory system) also causes her poor social performance. So, even though missing her medication did not cause or contribute to her poor social performance, it will be correlated with it. Although the driving and medication examples look different on the surface, underneath they make the same point: there is always the possibility that the correlation between 2 variables reflects nothing more than a shared common cause. Keep these examples in mind the next time you find yourself tempted to claim a causal relationship between a factor and goal performance on the basis of the factor’s predictive value alone.
Translating factor data to numbers and computing correlation The statistical function we use for detecting whether there is a correlation between a factor and observed behavior is the Pearson r. There are other statistics that can be used depending upon whether the data you’ve collected is categorical, ordinal, or interval-based, but the PEARSON r is available in most spreadsheet programs and can be used for all these types (with some caution, as explained below). Exercise 9.2 takes you step-by-step through the use of this statistical function in Excel. Recall that when we converted our data from collection sheets to spreadsheets we delayed the translation of non-numeric factors into numbers. Before we can use the Pearson r we must complete that step. There are two cases: 1.
When the type of the factor is categorical (nominal) we typically assign 1 to the first category/name we come across in the data, two to the second category/name, and so on, as shown in Figure 9.4. (This straightforward approach works fine when there are only two categories but can lead to an unintended hypothesis when there are more than two categories, as discussed in the section “You get what you ask for,” below).
2.
When the type of a factor is ordinal or interval we assign numbers in a sequence that preserves the order information, just as we did when we 139
FROM GOALS TO DATA AND BACK AGAIN
converted our original qualitative scales to raw data (see Chapter 6). Even so, some factors can be translated in more than one way. Consider session date as an example. The date of each session can be translated into many different ordinal time scales. For example, we could translate session dates into session numbers with the earliest date being given the number 1, the next date being given the number 2, and so on. Or each session date might be translated into a number between 1 and 7 to represent its day of the week. We might even decide to use session date to distinguish weekdays from weekends by assigning 1 to Monday through Friday sessions and 2 to sessions that fall on Saturday or Sunday. Which of these choices we make will affect the degree of correlation, a fact we look at in more detail shortly.
Figure 9.4 Translation of the categorical factor specifying location into numbers
The result of computing the Pearson r is always a number between –1.00 and +1.00. Ignore the sign for the moment (we’ll discuss it in the next section). Just looking at the number, we use the following guideline to characterize the degree of correlation between 2 variables: 140
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
· If the Pearson r is less than 0.55, there is no correlation. · If the Pearson r is between 0.55 and 0.74, there is a moderate
correlation.
5-point scale
· If the Pearson r is between 0.75 and 1.00, there is a strong correlation.
Comparison of Means, Becca, Goal Y
5-point scale
6.00 4.00 2.00 0.00 Before
After
(mean=2.0, sd=.55 mean=3.4, sd=1.06)
Figure 9.5 Is Becca still learning or is there something interfering systematically with her performance?
As with the boundary conditions in the comparison of means, these numbers should never be taken as absolutes. Other sources of information may suggest interpreting a value near a boundary as belonging to the weaker or stronger cor1 relation category. 1
These values were chosen assuming that the “before” or “after” data set has about 10 values in it. If you have fewer than 10 values, r must be higher to reach the same conclusion; if you have a lot more values, r can be a bit lower.
141
FROM GOALS TO DATA AND BACK AGAIN
All right, let’s look at some examples. If you open the ABC Workbook to the sheet titled “Becca Goal Y,” you will see the full data set for the graphs in Figure 9.5 as well as the location data specifying where Becca received intervention. This factor is categorical and the values for the factor are shown in their original and translated forms. Becca’s “before” data for goal Y looks pretty typical but her “after” data and the comparison of means seem to indicate that the goal requires continued intervention. We would like to understand if the irregular behavior in the period that we expected to be “after” learning is really due to inadequate practice or if it might have some other explanation. Since the line graph for the data in the “after” phase cannot be characterized as a pattern disturbed by an outlier, we must consider whether there could be some more systematic interference with Becca’s performance. Specifically, what happens if we consider the location factor? Might Becca’s performance on this goal be better on days when she receives intervention at home rather than at playgroup, or vice versa? Calculating the correlation between location and goal Y we find little evidence for a predictive relationship between location and behavior in the “before” data (r = 0.56), but a strong correlation in the “after” data (r = 0.82). In other words, it would seem that we can predict something about how Becca will do on goal Y at this point in her learning simply by knowing where intervention will occur. What is the specific prediction? Looking at Becca’s spreadsheet we see that most of the time her low scores on goal Y are paired with location values of 1. Since the categorical value of “home” was assigned to 1, the data is telling us that she does more poorly at home for this goal than she does when she is at playgroup. This is a fairly unusual outcome—we expect a child to perform well at home before she can move a skill into an outside environment. Becca’s example gives us the opportunity to point out, more concretely, why we calculate the correlation separately for the “before” and “after” data. Although the predictive value of location is strong for goal Y during the “after” phase of intervention, the same factor held little predictive value in the “before” phase. How can that be? A possible explanation is that the new goal was difficult enough that Becca’s struggle to perform during the initial phase of intervention didn’t depend on where the intervention was given. After Becca began to catch on, however, there was something about the environment that made a difference in her ability to perform the skill. Either location 1 (home) contains some element that is making it more difficult for Becca, or alternatively, there is something about location 2 (playgroup) that makes the skill more accessible or easier to perform. Note that if we had computed a single statistic over the combined “before” and “after” data the result would have been r = 0.64, a moderate value that only masks the real pattern in the data. What do the results of the correlation analysis mean for the team? If there were enough data points they could calculate the “after” means for each location 142
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
separately and draw their conclusions based on the revised analysis. Even though there aren’t enough data points to do this (we’d like at least 10 points in each location) the trend still seems clear. So, the team might decide to break goal Y into 2 goals during the next period of intervention. The two new goals would be qualified with respect to location and worked on and measured independently. At home, the focus would be to try to identify factors that interfere with Becca’s performance in order to adapt the environment (or teach additional skills) to overcome the interference. At playgroup, where the skill looks solid, the focus would depend on the next stage for goal Y: possibly performance with less support, for longer duration, or some similar extension. The Pearson r can uncover fairly complex relationships. Consider Celeste’s data for goal Z. Figure 9.6 shows the graphs for the data found on the sheet titled “Celeste Goal Z” in the ABC Workbook. This is the first analysis of percentages we’ve shown, so it’s useful to note that the mean and standard deviation are computed in exactly the same way for interval data as for the ordinal data we’ve worked with so far. The comparison of means is also computed in the same way, except that we haven’t defined yet what a “category width” is for this type of data. We are in a position to do that now. Remember that the comparison of means uses the idea of category width to define how the behavioral change we are looking for would be reflected in the data. In ordinal data like our 4- and 5-point qualitative scales, the minimum category width required for reliable change is 1.00 because that is the value between steps that define categorically different skill levels. We need to be true to this principle when defining category width for interval data as well. In the case of percentages, then, the definition depends on the nature of the behavior over which the percentages are calculated. If there were about ten opportunities to observe performance on the goal every session then 10% would be the minimum category width because each success can affect the percentage score by not less than 10%. If there were only five opportunities in a session then 20% would be the minimum category width because the child’s scores could only have been 0% (0/5), 20% (1/5), 40% (2/5), 60% (3/5), 80% (4/5), or 100% (5/5). As noted on her data sheet, Celeste had about ten opportunities per session to practice her skills for goal Z. This gives a minimum category width of 10% so the comparison of means (24.67% versus 37.33%) passes the initial analytical test for change: a difference of at least one category. The standard deviations are less than a category width and the error bars barely overlap so we are fairly confident that Celeste has made reliable progress on this goal. When we look at the line graphs for her data, however, we see an interesting pattern. It looks as if Celeste is having a recurring problem: over each set of five sessions, she does progressively better on goal Z, only to suffer a small setback during the first session in each new set of five. How can we see if this hypothesis is right? 143
% successful
FROM GOALS TO DATA AND BACK AGAIN
% successful
50.00 40.00 30.00 20.00 10.00 0.00 Before
After
(mean=24.67, sd=5.81 mean=37.33, sd=7.04)
Figure 9.6 Celeste seems to show a systematic regression (followed by recoupment) after every 5 sessions
To test the hypothesis we want to compute the correlation between the position of the session in a five-day sequence and performance on goal Z. We begin by assigning each session a number between 1 and 5, as shown in column C of Celeste’s spreadsheet. As with Becca’s data, we then compute the Pearson r separately for the “before” and “after” data sets because the effect of session position could well be different at different points in the development of the new skill. The correlation between session number and goal Z is 0.76 in the “before” data set, and 0.80 in the “after” data set—pretty good indications that this sub-pattern in Celeste’s data (increasing performance over each set of five sessions with regression between sets) is predicted by this factor. We now have statistical evidence for her cyclic behavior to go along with our more intuitive analysis. Given this information, the team is likely to consider it worth the effort to try to uncover what might be producing the correlation.
144
EXCEPTIONS THAT PROVE THE RULE – FACTOR DATA
Negative correlation Recall that when we claim two variables are correlated we mean that variation in the value of one variable predicts variation in the value of the other variable. So far we’ve seen only cases in which an increase in one variable predicts an increase in the other. In Becca’s “after” data the higher value for location correlated with the higher values for goal Y, while in Celeste’s case rising scores accompanied rising session numbers. When increase predicts increase (or decrease predicts decrease) we call it a positive correlation. We made an arbitrary choice when we translated the location factor in Becca’s data into numbers. We could have chosen the opposite assignment (1 for playground and 2 for home) without changing the meaning of the factor. Similarly, when we created a factor reflecting position in a five-day sequence for Celeste’s data, we simply chose numbers that corresponded to the meaning of the words we used to describe the pattern we saw (1–5 for the first through the fifth position in the sequence). But as long as we were consistent and chose an ordinal numbering scheme we could have done that assignment differently as well. Although it would have seemed odd, we could have numbered the positions in the sequence 5, 4, 3, 2, and 1 without changing the meaning of the data or the hypothesis we wanted to test. Reversing the way each factor was assigned numbers would have left the hypothesis intact, but what about the degree of correlation between each factor and its goal? Wouldn’t that have changed? No, it wouldn’t. The only effect of reversing the assignments would have been to reverse the relationship between the variables. Where increase had predicted increase, increase would now predict decrease. When an increase in one variable predicts a decrease in the other variable (or decrease predicts increase) it is called negative correlation and it is signaled by a negative sign in the Pearson r statistic. The degree of the relationship—how well one variable predicts the other—remains the same, only the direction—increasing predicts decreasing—changes. You can explore this phenomenon further in Exercise 9.3 (Appendix C). As we mentioned in the introduc