The Social Validity Manual A Guide to Subjective Evaluation of Behavior Interventions in Applied Behavior Analysis
This page intentionally left blank
The Social Validity Manual A Guide to Subjective Evaluation of Behavior Interventions in Applied Behavior Analysis
Stacy L. Carter Texas Tech University Lubbock, TX
AMSTERDAM • BOSTON • HEIDELBERG • LONDON • NEW YORK • OXFORD • PARIS • SAN DIEGO • SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Academic Press is an imprint of Elsevier
Academic Press is an imprint of Elsevier 32 Jamestown Road, London NW1 7BY, UK 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA 525 B Street, Suite 1900, San Diego, CA 92101-4495, USA Copyright © 2010 Elsevier Inc. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (⫹44) (0) 1865 843830; fax (⫹44) (0) 1865 853333; email:
[email protected]. Alternatively, visit the Science and Technology Books website at www.elsevierdirect.com/rights for further information Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress ISBN: 978-0-12-374897-3 For information on all Academic Press publications visit our website at elsevierdirect.com Typeset by Macmillan Publishing Solutions (www.macmillansolutions.com) Printed and bound in the United States of America 10 11 12 13 14 10 9 8 7 6 5 4 3 2 1
Dedication
This book is dedicated to my parents, Tracy and Judy; my wife, Narissra; and my son, Ezra. Thank you for everything.
This page intentionally left blank
Contents
List of Figures
xiii
List of Tables
xv
Preface
1.
Background of Social Validity Defining Social Validity What is Social Validity? Development of a Concept Dimensions of Applied Behavior Analysis Importance of Social Validity Is Social Validity Important? Purpose of Social Validity Positive Behavior Supports Origin of PBS Movement: Association for Behavior Analysis Origin of PBS Movement: Normalization/Inclusion Origin of PBS Movement: Person-Centered Values PBS Criterion I: Comprehensive Lifestyle Change and Quality of Life PBS Criterion II: Life Span Perspective PBS Criterion III: Ecological Validity PBS Criterion IV: Stakeholder Participation PBS Criterion V: Social Validity PBS Criterion VI: Systems Change and Multicomponent Intervention PBS Criterion VII: Emphasis on Prevention PBS Criterion VIII: Flexibility with Respect to Scientific Practices PBS Criterion IX: Multiple Theoretical Perspectives Effects of the PBS Movement
2.
xvii
1 1 3 4 6 6 8 11 12 12 13 14 15 16 16 17 17 18 18 18 18
Conceptualizations of Social Validity Dimensions of Social Validity Habilitative Validity Working Model of Treatment Acceptability Decision-Making Model of Treatment Acceptability Expansive View of Treatment Acceptability
21 22 24 25 27 vii
viii
Contents
Epidemiological Conceptualization of Social Validity Assessment Model of Social Validity Maintenance Model of Social Validity Distributive Model of Treatment Acceptability
3.
Instruments for Evaluating Social Validity Interviews Direct Observation Performance Criteria Formal Treatment Acceptability Measurement Instruments Treatment Evaluation Inventory Treatment Evaluation Inventory-Short Form Treatment Acceptability Rating Form Treatment Acceptability Rating Form Revised Intervention Rating Profile Intervention Rating Profile-15 Children’s Intervention Rating Profile Behavior Intervention Rating Scale Intervention-Process Rating Scale (IPRS) Abbreviated Acceptability Rating Profile Comparison of Treatment Acceptability Measures Problems Related to Social Validity Measurement Methods for Increasing the Use of Treatment Acceptability Instruments Increase Availability of Instruments and Other Measures Increase Consultant Awareness of Instruments Increase Competency in the Administration and Scoring of Instruments Increase Knowledge of Potential Uses of Acceptability Information Re-emphasize the Importance of Social Validity
4.
28 31 31 32
40 47 47 48 49 49 51 52 52 52 56 57 59 59 60 61 61 64 65 65 66 66
Research on Social Validity Early Protocols for Measuring Social Validity Methodological Variations Literature Reviews of Treatment Acceptability Research Findings TEI and TEI-SF IRP and IRP-15 Teacher Acceptability Measure (TAM) CIRP BIRS AARP TARF-R Summary
69 74 75 81 81 91 97 97 99 100 101 102
ix
Contents
5.
Increasing the Social Significance of Treatment Goals Consumer Significance Personal Preferences/Values Choice Habilitative Potential Awareness of Coercion Immediate versus Long-term Benefits Consultant Training Personal/Professional Agenda Goals Clarity and Specific Outcome Criteria of Goals Complexity and Comprehensiveness of Goals Potential for Actualizing the Goal Instrument/Method Society Normalization Use of Available Resources and Funding Strategies to Boost the Social Validity of Treatment Goals Step 1—Incorporate Semi-structured Interviews Step 2—Gather Information from Multiple Sources Step 3—Consider Different Aspects of the Individual’s Life Step 4—Rank Treatment Goals Hierarchically Step 5—Operationalize Expected Outcomes in Both Clinical and Social (Lay) Terminology
6.
111 111 117 122 124 126 126 126 127 128 128 128 129 131 134 134 135 136 136 137 138 139 140
Enhancing the Appropriateness of Treatment Procedures Preliminary Factors to Consider When Planning for Social Validity Determine How the Information Will Be Used Consider the Reliability and Validity of Measures Consumer Acceptability Characteristics of Consumers Knowledge of Treatment Consultant Acceptability Characteristics of Consultant Affiliations of Consultant Familiarity with Recent Advances in the Field Interaction Style of Consultant Assessment Methods Design of Treatment Treatment Intrusiveness Implementation Difficulty Other Variables Societal Appropriateness
144 144 144 146 147 150 150 150 151 152 154 155 156 156 163 163 165
x
Contents
Strategies to Improve the Social Validity of Procedures and Programs Step 1—Explain to Consumers Why Information Is Being Collected and How It Will Be Used Step 2—Look for Problems First Step 3—Determine How Social Validity Should Be Measured Step 4—Determine When Social Validity Should Be Measured (Pre- or Posttreatment) Step 5—Determine Who Should Rate the Acceptability of the Procedures Step 6—Determine the Conditions Under Which Ratings Should Be Obtained Step 7—Determine What Type of Training Should Be Provided to Raters Step 8—Allow Time for Behaviors to Become Established
7.
169 169 169 170 171 171 172 173 173
Improving the Importance of Treatment Effects Disability Policy Concepts Core Concept 1: Antidiscrimination Core Concept 2: Individualized and Appropriate Services Core Concept 3: Classification Core Concept 4: Capacity-Based Services Core Concept 5: Empowerment/Participatory Decision Making Core Concept 6: Service Coordination and Collaboration Core Concept 7: Protection from Harm Core Concept 8: Liberty Core Concept 9: Autonomy Core Concept 10: Privacy and Confidentiality Core Concept 11: Integration Core Concept 12: Productivity and Contribution Core Concept 13: Family Integrity and Unity Core Concept 14: Family Centeredness: Services to the Whole Family Core Concept 15: Cultural Responsiveness Core Concept 16: Accountability Core Concept 17: Professional and System Capacity-Building Core Concept 18: Prevention and Amelioration Conducting Social Validity Assessments Consumer Importance Consultant Importance Societal Importance Strategies for Attaining Social Importance for the Effects of Treatment Ensure that Consumers Have a Clear Understanding of Treatment Effects Provide Normative Comparisons Collect Data on Associated Variables
180 181 181 181 182 182 182 182 183 183 183 183 183 184 184 184 184 185 185 186 187 196 200 204 205 205 206
xi
Contents
Identify Experts to Evaluate the Effects Conduct Cost Analyses
8.
Social Validity and Ethics Social Validity and Ethical Values Aversive Procedures and Social Validity Social Validity Within Ethical Guidelines How Social Validity Can Support Ethical Guidelines Competence Professional and Scientific Responsibility Respect for People’s Rights and Dignity Concern for the Welfare of Others Contribution to Community and Society
9.
211 211 218 219 220 221 222 223 223
Organization of Social Validity Data Micro Level Organization of Data Medial Level Organization of Data Macro Level Organization of Data Validity of Data Examination of Validity Using Messick’s Principles
10.
207 208
230 234 235 238 239 243
The Future of Social Validity
References Index
263 283
This page intentionally left blank
List of Figures
Figure 2.1
Witt and Elliott’s “Working Model” of social validity.
Figure 2.2
Reimers et al. decision-making model of treatment acceptability.
Figure 3.1
Semi-structured interview for social validation.
Figure 3.2
Student-Assisted Functional Assessment Interview.
Figure 3.3
Treatment Evaluation Inventory.
Figure 3.4
Treatment Acceptability Rating Form-Revised.
Figure 3.5
Intervention Rating Profile.
Figure 3.6
Intervention Rating Profile-15 (IRP-15).
Figure 3.7
Children’s Intervention Rating Profile (CIRP).
Figure 3.8
Behavior Intervention Rating Scale (BIRS).
Figure 3.9
Abbreviated Acceptability Rating Profile (AARP).
Figure 5.1
Distributive model of treatment goals.
Figure 6.1
Distributive model of treatment acceptability.
Figure 7.1
Distributive model of treatment effects.
Figure 7.2
Checklist for examining recognition of core concepts of disability policy.
Figure 7.3
Sample graph of goal attainment scaling.
Figure 9.1
Social validity measurement inventory.
Figure 9.2
Table for organizing social validity data.
Figure 10.1
Distributive model of social validity.
xiii
This page intentionally left blank
List of Tables
Table 1.1
Wolf’s Three-Component Definition of Social Validity.
Table 1.2
Dimensions of ABA.
Table 1.3
PBS Criteria.
Table 1.4
PBS Critical Features.
Table 2.1
Key Variable Equivalencies in Epidemiological Model by Winett et al. (1991).
Table 3.1
Some General Methods for Assessing Social Validity.
Table 3.2
Characteristics of Treatment Acceptability Instruments.
Table 3.3
Ways to Increase the Use of Treatment Acceptability Instruments.
Table 4.1 Categories of Variables Derived from Kazdin’s 1980 Studies Which Influenced Future Research. Table 4.2
Summary of Treatment Acceptability Research Published Between 1990 and 2005.
Table 4.3
Variables with Contradictory Findings Regarding Influences on Acceptability.
Table 5.1
Checklist for Exploring the Social Significance of Treatment Goals.
Table 5.2
Treatment Goal Prioritization.
Table 5.3 Factors in Determining the Actualization Potential of a Treatment Goal. Table 6.1
Treatment Component Value Matrix.
Table 6.2
Treatment Intrusiveness Measure. xv
xvi
List of Tables
Table 7.1
Validation Levels of Social Validity Data.
Table 7.2
Core Concepts of Disability Policy.
Table 7.3
General Example of Goal Attainment Scale.
Table 7.4
Quality-of-Life Domains and Indicators Identified by Schalock and Verdugo (2002).
Table 7.5
Questions for Expert Evaluators to Answer.
Table 8.1
Right to Effective Behavioral Treatment.
Table 8.2
American Psychological Association Ethics Code 2002.
Table 9.1
Social Validity Measurement Matrix.
Table 9.2
Messick’s Six Principles of a Unified Theory of Construct Validity.
Table 9.3 Questions to Consider for Examining the Validity of Social Validity Data. Table 10.1
Reasons for Limited Impact of ABA.
Preface
At the heart of applied behavior analysis is social validity. Wolf (1978) referred to social validity as how applied behavior analysis was finding its heart, and in many ways social validity does appear to function like the heart of a much larger body. The field of applied behavior analysis has been mostly controlled by a “brain” that organizes information and responds in a logical, objective manner. Social validity appears to function in a more illogical, subjective manner that leads the field of applied behavior analysis down paths where it would typically not venture. The examination of objective, observable data has been a hallmark of applied behavior analysis, while social validity has made the field incorporate subjective, self-report data. Applied behavior analysis has been dominated by single-subject research designs, while social validity has led the field more into group research designs. Just like in a romance novel, the heart of applied behavior analysis has taken over at times and caused it to go places, seek after things, and act in ways that sometimes don’t seem quite logical, but rather act out of a passion centered only in the heart. The passion from this heart has taken the field of applied behavior analysis into realms where new discoveries have been made and new mysteries have developed. Many of the findings from social validity research are contradictory and fleeting at times, which makes research in this area one of the most interesting and one of the most difficult to interpret. This book is designed to provide an introduction into the obscure side—the dark, passionate side, if you will—of applied behavior analysis. A history of the development of social validity within the field of applied behavior analysis is provided along with the impact that it has had within the field. In L. Frank Baum’s The Wizard of Oz, the Tin Woodman and the Scarecrow debated the value of both a heart and a brain. The Scarecrow said, “I shall ask for brains instead of a heart; for a fool would not know what to do with a heart if he had one.” The Tin Woodman responded by xvii
xviii
Preface
saying, “I shall take the heart, for brains do not make one happy, and happiness is the best thing in the world.” Each made excellent points, and the field of applied behavior analysis has the potential to make use of both the objective, logical skills associated with the brain and the impassioned, subjective input connected to the heart. By using the brain and heart in combination, the field of applied behavior analysis can potentially make greater strides in using science to improve the lives of everyone in society and also make them happier. As stated by Wolf: Our use of subjective measures does not relate to internal causal variables. Instead, it is an attempt to assess the dimensions of complex reinforcers in socially acceptable and practical ways. It is an evolutionary event that is occurring as a function of the contingencies of the applied research environment, contingencies that our founders would probably say they appreciate, if we had the nerve to ask them for such subjective feedback on our behavior (1978, p. 213).
Metaphorically speaking, the approach to social validity taken in this book can be compared to putting together a puzzle with a huge number of pieces. Without having a method for putting the pieces together, initial efforts toward completing the puzzle may consist of random trialand-error responses with periodic success. This book attempts to offer some strategies for putting the pieces of the puzzle together that will hopefully reduce the randomized responses and result in faster completion of the puzzle. One strategy for completing a puzzle is to begin by sorting and organizing the border pieces, since they are typically easier to recognize because of the recognizable straight edge. Completing the border of the puzzle offers some perspective to the magnitude of completing the rest of the puzzle. The border pieces of the puzzle can be compared to the conceptualization of social validity that is used throughout this book. While several different conceptualizations are discussed, this book uses a conceptualization that is very liberal toward its inclusion of variables considered to represent and have influence on social validity. This wide-ranging view of social validity results in a large puzzle border within which a large variety of pieces can and need to be placed in order to complete the puzzle. Another strategy in completing a puzzle is to sort the pieces according to color similarities and attempting to connect pieces of like color. This book attempts to do this with social validity by organizing the chapters according to treatment goals, treatment procedures, and treatment effects. This type
Preface
of organization is intended to increase possible recognition among similar aspects of treatment and allow for additional connections to occur. In completing a puzzle, there are often sections of the puzzle which are not directly associated with the predominant picture contained within the puzzle. These pieces arrange to form a background against which the primary picture can be discriminated. The background within a puzzle is highly important although not directly a part of the primary picture within the puzzle. The background adds to the central puzzle picture by making the picture more recognizable and supports the positioning and the structural stability of the central picture. Upon putting together all the available pieces to the puzzle, there may in many cases be pieces that are missing. If enough pieces can be put together, the picture represented in the puzzle can still be determined even with some pieces missing. This book focuses on a number of variables and concepts that may not be considered direct measures of social validity, but which offer up a background upon which social validity can be more readily discriminated. By including a large number of associated variables, the evidence for social validity will be provided with a highly stable structure even if some pieces are missing and the importance of social validity will be seen as the predominant big picture that all behavior treatments are attempting to illustrate. Stacy L. Carter, PhD April 1, 2009 Lubbock, TX
xix
This page intentionally left blank
Chapter 1
Background of Social Validity Defining Social Validity What is Social Validity? Development of a Concept Dimensions of Applied Behavior Analysis Importance of Social Validity Is Social Validity Important? Purpose of Social Validity Positive Behavior Supports Origin of PBS Movement: Association for Behavior Analysis Origin of PBS Movement: Normalization/Inclusion Origin of PBS Movement: Person-Centered Values PBS Criterion I: Comprehensive Lifestyle Change and Quality of Life
PBS Criterion II: Life Span Perspective PBS Criterion III: Ecological Validity PBS Criterion IV: Stakeholder Participation PBS Criterion V: Social Validity PBS Criterion VI: Systems Change and Multicomponent Intervention PBS Criterion VII: Emphasis on Prevention PBS Criterion VIII: Flexibility with Respect to Scientific Practices PBS Criterion IX: Multiple Theoretical Perspectives Effects of the PBS Movement
DEFINING SOCIAL VALIDITY What is Social Validity? The construct of social validity has been conceptualized in many different ways. Schwartz and Baer (1991) point out that numerous meanings have been ascribed to the construct of social validity, which have thus catapulted the construct into a mass of confusion. Even the terminology associated with social validity has been used in various ways that make determining exactly what social validity consists of extremely confusing at times. While the literature within the field of behavior analysis coined the Copyright © 2010 Elsevier Inc. All rights reserved
1
2
Chapter | 1 Background of Social Validity
term “social validity,” the term has not been used consistently even within the field where it was first used. Other fields of social science, as well as business and marketing, have used various terminology to describe similar constructs, such as social importance, social relevance, social significance, consumer satisfaction, clinical importance, clinical significance, educational relevancy, applied relevance, applied importance, ecological validity, cultural validity, and cultural significance. The sheer number of terms, which have in some instances been used interchangeably with social validity, may have resulted in some confusion about exactly what is involved in defining, measuring, and evaluating social validity. While social validity can be associated with several different disciplines and defined in many different ways, the type of social validity referred to within this book will be the social validity that was developed from the field of applied behavior analysis. There have been several proposed variations on how social validity should be defined, how it should be assessed, and how it should be used within the field of applied behavior analysis. Several of these variations will be described within this book, and each will be discussed in relationship to the original definition of social validity proposed by Wolf (1978). For the purposes of this book, the term social validity will refer to the evaluation of the degree of acceptance for the immediate variables associated with a procedure or program designed to change behavior. This definition is derived from the definition provided by Wolf (1978). He defines social validity as (1) the social significance of the goals of treatment; (2) the social appropriateness of the treatment procedures; and (3) the social importance of the effects of treatments (Table 1.1). The most frequent method for determining the degree of acceptance for a procedure or program has been to ask those receiving, implementing, or consenting to a treatment about their opinions of the treatment. These opinions are then used to make decisions about current or future uses of the treatment.
TABLE 1.1 Wolf’s Three-Component Definition of Social Validity 1. Social significance of the goals of treatment 2. Social appropriateness of the treatment procedures 3. Social importance of the effects of treatment
Chapter | 1 Background of Social Validity
Development of a Concept Wolf (1978) describes, with an apology, his personal experiences with social validity that were associated with the development of a definition for the term. He describes how a last-minute deadline resulted in the following quote from Don Baer being included in the introductory issue of the Journal of Applied Behavior Analysis (JABA) to describe the purpose of the journal: “It is for the publication of applications of the analysis of behavior to problems of social importance.” Wolf states that he continued to think about the meaning of the term social importance and how it might be viewed by others in the field. He explains that the term social importance relied on subjective information such as opinions or judgments made by other people. Wolf feels that including this type of subjective component in the seminal issue of JABA conflicted with the objective nature of behavior analysis, which was considered to be in line with other natural sciences that avoided subjective evaluations of variables. Wolf goes on to note that the field of behavior analysis had separated itself from other social sciences (such as sociology, anthropology, psychiatry, etc.) that included subjective observations, and the board of editors for JABA wanted the journal to exemplify only strict objective measurements. The tradition of direct observation of behavior was considered necessary by Skinner (1969), as he states: Much can be done in the study of behavior with methods of observation no more sophisticated than those available, say, to Faraday, with his magnets, wires, and cells. Eventually the investigator may move to peripheral areas where indirect methods become necessary, but until then he must forego the prestige which attaches to traditional statistical methods (p. 111).
While Wolf continued to struggle with a definition of social importance that would fit the objective nature of the journal, he found that the journal readers, article authors, and reviewers of manuscripts did not complain about the lack of a definition of social importance and they appeared to be able to readily recognize its presence or absence. This appears to be similar to the statement made by Supreme Court Justice Potter Stewart, who avoided providing a succinct definition for pornography but stated “I know it when I see it….” The point is that Wolf finds that social importance is recognized, accepted, and valued even though it may not lend itself to an objective method of measurement.
3
4
Chapter | 1 Background of Social Validity
Dimensions of Applied Behavior Analysis In developing an understanding of social validity, it is important to review the development of the field of applied behavior analysis. The JABA has become the primary outlet for disseminating research in the area of applied behavior analysis. In the introductory issue of this journal, Baer, Wolf, and Risley (1968) outline the dimensions of applied behavior analysis, and these dimensions are still frequently referred to in the literature. They describe seven criteria needed for a study to be considered an example of applied behavior analysis (Table 1.2). These criteria are essential for adhering to the scientific principles established within the field of applied behavior analysis.
Criteria I: Applied The applied criterion refers to how target behaviors are selected. With applied research, target behaviors must be selected because of their importance to society. With nonapplied research, target behaviors may be selected for reasons that provide convenience to the researcher. Applied research focuses on target behaviors that are highly relevant to society and that typically reveal immediately important relationships between the behavior and certain stimuli.
Criteria II: Behavioral The behavioral criterion focuses on the pragmatic aspect of research. Applied research should focus on the physical events performed by individuals rather than on other factors only related to physical performance
TABLE 1.2
Dimensions of ABA
Baer, Wolf, and Risley’s (1968) Seven Dimensions of Applied Behavior Analysis 1. 2. 3. 4. 5. 6. 7.
Applied Behavioral Analytic Technological Conceptually systematic Effective Generality
Chapter | 1 Background of Social Validity
of these events, such as verbal descriptions of performance, unless independently supported by other measures.
Criteria III: Analytic The analytic criterion refers to the reliability of the research. In order for a study to meet the criterion of being analytic, it must reasonably demonstrate some level of control over a behavior. Nonapplied research may require extensive repetition of control over behavior, but applied research has to rely on achieving as much control over behavior as can be demonstrated to make the control believable to a select audience. In other words, applied research cannot always achieve the level of demonstrated control over behavior that can be achieved in nonapplied research, due to allowances provided within social settings.
Criteria IV: Technological The technological criterion concerns the description of the techniques used to promote behavior change. Applied research requires that all components of a treatment be completely and accurately identified and reported. This criterion does not distinguish nonapplied research, but requires that enough information be provided on the techniques used in an experiment so that others might replicate these same techniques in exactly the same manner.
Criteria V: Conceptually Systematic The conceptually systematic criterion relates to how techniques relate to basic principles that are considered part of the field of behavior analysis. Applied research must convey how specific procedures have been derived from basic principles so that similar procedures might be derived from those same basic principles. This criterion was deemed necessary to promote the field of applied behavior analysis beyond what might be considered a series of unrelated techniques or procedures.
Criteria VI: Effective The effective criterion relates to the relevance of outcomes to practical situations. Nonapplied research may seek to produce outcomes that have theoretical importance, while applied research must produce results that have some practical value. Applied research requires outcomes that are large enough to be considered important to those who deal closely with a behavior.
5
6
Chapter | 1 Background of Social Validity
Criteria VII: Generality The generality criterion involves the enduring qualities of a treatment over time, over settings, or across related behaviors. Since applied research must deal with socially important behaviors, the changes that take place in these behaviors may be important in ways different from the way in which the changes were initially introduced. Demonstrating that treatments are effective in more than one way increases the value of the procedure and the importance to society. Examining these criteria in light of social validity, it seems that the applied criteria and the effective criteria play the most important roles in understanding social validity, because both of these criteria focus on aspects of social determination or judgment. In addition, the generality criterion may be highly important toward gaining a more comprehensive understanding of the influences on social validity. Although some of the dimensions of applied behavior analysis may play more important roles in the assessment of social validity, as stated by Schwartz and Baer (1991), state-of-the-art social validity assessment should address all the dimensions of applied behavior analysis.
IMPORTANCE OF SOCIAL VALIDITY Is Social Validity Important? Social validity has been viewed as essential to the survival of the field of applied behavior analysis and as a diversionary trap that would lead to the demise of the field (Baer & Schwartz, 1991). They state, “We know little about the accurate and valid assessment of what gets called social validity—and we know a great deal less about the survival of fields” (p. 231). While either of these views may potentially be correct, what appears most important is to gain an understanding of the construct and what role it might play in benefiting society. If social validity has some value to society, it should be embraced and further utilized in the field of behavior analysis rather than avoided and misunderstood. Baer and Schwartz state that the primary problems facing the field of applied behavior analysis with regard to social validity were how to assess it and how to apply it to improve behavior interventions. In some cases it may be argued that social validity is not important. An expert may develop a program that is considered highly effective at
Chapter | 1 Background of Social Validity
achieving a desired outcome, which may on the surface appear to be a very useful program regardless of whether nonexperts agree. An extreme hypothetical example of this might be as follows: An expert is assigned a case in which a client is displaying aggressive acts toward others within a residential facility. The client frequently strikes, pushes, and bites others throughout the day during typical daily activities. The expert designs a program intended to reduce the frequency of aggressive acts toward others. The expert trains the staff at the facility to place the client in a time-out room following each aggressive act. The program is determined to show an immediate reduction in the frequency of aggressive acts when compared to data obtained prior to the treatment. On the surface, this treatment may appear to have produced the desired outcome (reduction of aggression), but it actually neglects to mention a number of critical elements that would need to be considered before the treatment could truly be considered a success. First, the treatment does not clarify the amount of time the client would be required to remain in the time-out room. This problem has been highly relevant in intermediate care facilities, due to the potential for abusing this type of procedure. The staff at the facility may choose to leave the client in the time-out room for extended periods of time because the aggressive behavior that the patient displays is difficult to manage and can be avoided while he or she remains in the time-out room. In addition, the procedure may not be truly effective because the time spent in isolation reduces the opportunities for the client to engage in aggressive behavior; this lack of opportunity would certainly reduce the frequency of aggressive acts. There are numerous other concerns regarding this program, such as the potential lack of socialization that may occur, lack of training in appropriate skills, and so on. The point of this example is not to identify all the potentially inappropriate elements, but rather to demonstrate that just because a program is considered effective does not mean that it will be considered appropriate by those closely involved in implementation of the program, or that it should even be implemented in the first place. For instance, other treatments may be available that can potentially produce similar outcomes and avoid some of the deficiencies apparent in the program described. If these other treatments are available, they might certainly be more appropriate than the treatment described, although they may not produce more effective results.
7
8
Chapter | 1 Background of Social Validity
Kazdin (1980) points to three reasons why determining the acceptability of treatments was important to overall social validity. The first reason was that several techniques for treating problem behaviors may be available that have comparable efficacy, but may not be equally acceptable to the person receiving the treatment. The second reason involves ethical and legal issues, which require that procedures avoid infringing on the rights of the individuals receiving the treatments. The final reason addressed by Kazdin (1980) involves the value of identifying specific variables that could influence the acceptability of treatments. If influential variables could be identified, they could be used to encourage individuals to seek out treatment, initiate treatment, and adhere to treatment regimens. With the increased dissemination of treatments involving less intrusive procedures and changes in educational law, a reexamination of the concept of social validity appears warranted. Michaels, Brown, and Mirabella (2005) report that experts were making fewer recommendations for the use of consequence-based treatments than they had in the past. Educational legislation such as the No Child Left Behind Act (No Child Left Behind Act of 2001, 20 U.S.C. § 6301 et seq.) and the Individuals with Disabilities Education Act Amendments (IDEA, 1997, 20 U.S.C. § 1401) has mandated the incorporation of parents, teachers, principals, and others in the development and implementation of treatments for children receiving special education services. More recent changes to IDEA have also promoted the use of empirically based treatments in the least restrictive environment. In light of these changes, the concept of social validity may benefit from further evaluation in terms of relevance to decisions made regarding treatments for behavior change programs.
Purpose of Social Validity One purpose of social validity may be, as indicated earlier, to ensure the survival of the field of applied behavior analysis. While this may seem like an arrogant statement and has definitely been challenged by many— to the point of declaring that social validity is basically nonessential— social validity still appears to have a purpose and a place within applied behavior analysis. Schwartz and Baer (1991) state that the purpose of social validity assessment is to provide information that would enable a treatment program to survive. In order for a treatment program to survive, it must not only be effective, it must also address problems considered
Chapter | 1 Background of Social Validity
relevant by consumers, it must do so in a manner that consumers can tolerate or even possibly enjoy, and it must produce outcomes that are considered valued. While these may be only a few of the requirements for the survival of a treatment program, the measurement of social validity may provide a method for identifying elements of a treatment program that could lead consumers to discontinue its use, avoid it, or even seek to ban its further implementation. One purpose of social validity may be to obtain feedback from consumers regarding how well specific elements of a treatment program are liked or disliked, in order to either make changes to the treatment program or to provide a rationale to consumers regarding why a treatment contains undesirable elements. Social validity assessment can provide information on how to quickly respond to consumers in a way that will provide them with a reason to continue the treatment or a new understanding of the need for the treatment. Schwartz and Baer (1991) point out that several criticisms of social validity are based on the fear that subjective measurement would replace objective measurement in applied behavior analysis and that this would be detrimental to the field. Schwartz and Baer state that social validity was never intended to replace objective measures related to a complete applied behavior analysis program, but was rather intended to be a supplement to a well-developed, objectively measured program. They affirmed that social validity was meant to be used as a measure secondary to any other objective measures that had previously been shown necessary for the development of a program that could be considered an example of applied behavior analysis. They go on to describe two ways that social validity was important to the field of applied behavior analysis. The first reason social validity was important involved the advancement and survival of the field of applied behavior analysis. They describe how it was important to understand what programs are liked and disliked by the public so that behavior analysts could continue to be a resource that consumers would value. The second reason Schwartz and Baer find social validity important involved understanding why some programs were liked and others disliked. They indicate that understanding why this occurred would eventually allow program developers to predict whether the programs they developed would be accepted or rejected. They also point out that in order to do this, there was a need to publish instances of both high
9
10
Chapter | 1 Background of Social Validity
social acceptance and low social acceptance, so all potential predictors could be investigated. They go on to describe how almost all of the research in applied behavior analysis demonstrated high social acceptance, which indicates that all procedures based on the principles of applied behavior analysis are socially valid. They also state that if this were an actual representation, then social validity was not being legitimately assessed, because applied behavior analysis could not be that good. If the literature on social validity of applied behavior analysis procedures were accurately representative, then the procedures developed on these principles and the entire field of applied behavior analysis should be very apt to survive and flourish. The appearance of near-total social acceptance of applied behavior analysis procedures in the literature, as indicated by Schwartz and Baer, could create a false sense of security regarding the viability of these procedures. This false sense of security could potentially result in limited development, expansion, or modification to these procedures in order to maintain high levels of social acceptance or to respond to changes in social validity. Kunkel (1987) notes that the field of applied behavior analysis appeared to be in a rut and indicated that the field needed to broaden its parameters in order to have a more substantial impact. Kunkel reflects on the rut he observed in the field: “What are the underlying reasons for this? And how can we get out of this rut? The answers reflect in large measure the field’s self-imposed definitions of acceptable procedures and consequent narrow substantive focus” (p. 330). Baer (1987) also recommends that the field should go beyond the parameters it had limited itself to in order to expand and suggests that expansion would potentially always be needed within the field. Hopkins (1987) indicates awareness of a lack of enthusiasm in the field of applied behavior analysis, but suggests that the field was gaining an understanding of the variables that were impeding the progress of the field and this could potentially lead toward renewed interest in the field. The field of applied behavior analysis seems to be in a state of avoiding or limiting an understanding or incorporation of the construct of social validity. Reviews of the literature have repeatedly reported decreasing trends in research on social validity measures. Finney (1991) states that social
Chapter | 1 Background of Social Validity
validity is not a static phenomenon and that it needs to be constantly reevaluated in order to obtain accurate measures. The limitations, such as potentially undue potency regarding the social validity of applied behavior analysis programs and the declining trends in social validity research in applied behavior analysis, appear to have left the door open for another movement that attempts to address some of these limitations. Namely, the positive behavior supports (PBS) movement appears to exploit this limitation in applied behavior analysis by focusing on the development of behavior programs that have a heavy emphasis on programs highly conducive to their consumers’ environments.
POSITIVE BEHAVIOR SUPPORTS The PBS approach is an expansion of the ecological–behavioral perspective toward developing treatments for individuals with disabilities. The ecological–behavioral perspective is concerned with the development of treatments that recognize the functionality of behaviors targeted for treatment and the interaction of these behaviors with the environment. These perspectives avoid addressing behaviors in isolation by including multiple areas of potential concern into developing treatment programs. Schalock (2004) describes a paradigm shift in regard to the provision of services to individuals with disabilities. This paradigm shift involves increased emphasis on the development of functional behaviors, increased emphasis on behaviors improving the personal well-being of the individual receiving services, increased individualization of services and supports, and expanded focus on demonstration of increased personal competence of the individual receiving services. The PBS movement is a culmination of a number of theoretical perspectives that attempts to provide a wellrounded approach to treatment development and implementation. The PBS movement is described by Carr et al. (2002) as emerging from three major sources. These sources are 1. applied behavior analysis, 2. the normalization/inclusion movement, and 3. person-centered values approaches. These major sources are described in the following sections.
11
12
Chapter | 1 Background of Social Validity
Origin of PBS Movement: Association for Behavior Analysis Carr et al. describe how applied behavior analysis has provided a conceptual framework for PBS and procedures for assessing and treating behaviors. They note how PBS incorporated concepts from applied behavior analysis, such as the three-term contingency (antecedent, behavior, and consequence), establishing operations, setting events, stimulus control, generalization, and maintenance, as a foundation for the development of PBS. In addition, several techniques from applied behavior analysis have been incorporated, such as functional assessment and several other treatment procedures such as fading, shaping, reinforcement contingencies, and so on. Although recognition was given to the strong emphasis on applied behavior analysis in the development of PBS, Carr et al. explain that PBS evolved beyond its roots in applied behavior analysis to assume its own identity. This independent identity is grounded in a response to the unique conditions apparent in applied settings that necessitated modifications to assessment strategies, treatment strategies, and methods for measuring outcomes. PBS strategies were considered a needed progression in applied science that relied on alternative methods considered more compatible with the varied contexts of natural settings. In general, the methodologies promoted within applied behavior analysis were too rigid, too narrowly focused, or too radical to be implemented with integrity in certain environments. In addition, the outcomes produced by applied behavior analysis techniques were considered at times to be limited to very specific behaviors that might not always be supported by natural environmental contingencies, but rather relied on artificial contingencies derived from the techniques. PBS developed as an alternative means of alleviating these concerns through incorporation of techniques that presumably provided a better fit within capricious natural environments.
Origin of PBS Movement: Normalization/Inclusion The influence of the normalization/inclusion movement on PBS has been to promote an awareness of the need to incorporate individuals with disabilities into settings that are the same as those of individuals without disabilities. The approach within PBS is to recognize settings that have been artificially developed and seek to reorganize or eliminate these environments and create environments that are supportive for everyone
Chapter | 1 Background of Social Validity
involved. Examples of this described by Carr et al. include the development of social groups for individuals with disabilities that encouraged involvement with individuals without disabilities and selection of living arrangements that match those of individuals without disabilities rather than segregated group home situations. While traditional applied behavior analysis treatment programs place considerable focus on addressing specific clinical problems, the PBS approach requires that a treatment program provide a means for the individual to improve access to community activities and relationships with others (Dunlap & Fox, 1996).
Origin of PBS Movement: Person-Centered Values Person-centered values are considered a foundation in the development of PBS due to the emphasis placed on humanistic values, which are said to override the importance of empirical data. Carr et al. describe evaluation procedures of PBS as incorporating a technological criterion as well as a values criterion. “Technological criterion” refers to the effectiveness of a particular strategy or technique for changing behavior. This criterion appears to be secondary to the values criterion, which focuses on the capacity for techniques and strategies to enhance the dignity of individuals and their opportunities for choice within their environment. Horner et al. (1990) outline three criteria that treatments must meet before being considered as treatment options within the PBS model (Table 1.3). The three criteria that must first be considered are (1) the intrusiveness of a treatment upon the individual targeted for treatment, (2) the social acceptability of the treatment, and (3) the degree to which a competent professional supervises and monitors the treatment. These criteria, outlined by Horner et al., appear to place more emphasis on values than on the effectiveness of particular treatments. TABLE 1.3
PBS Criteria
Three Criteria for Consideration as a PBS Treatment (Horner et al., 1990) 1. The intrusiveness of a treatment upon the individual targeted for treatment 2. The social acceptability of the treatment 3. The degree to which a competent professional supervises and monitors the treatment
13
14
Chapter | 1 Background of Social Validity
Carr et al. describe three processes used to integrate the values criterion into the PBS model of service delivery. These processes include person-centered planning, self-determination, and a wraparound approach. Person-centered planning focuses on first determining the needs of an individual when developing treatment programs and formulating goals for the individual. This is considered to be in contrast with a programcentered approach, which first determines the services readily available and selects from these services what would be beneficial for the individual. The person-centered planning approach is considered to be open to more options, while the program-centered approach appears to limit individual options. The construct of self-determination involves incorporating methods to allow individuals with disabilities opportunities to be involved in decisions concerning them and to provide input and allow for choice regarding the development of treatments and selection of goals designed for them. The process of incorporating wraparound services, as described by Carr et al., involves incorporating an entire system of individuals who would be involved in providing supports. These individuals extend beyond those who traditionally work directly with the individual with disabilities and include a team of experts, family members, advocates, and others dedicated to empowering the individual with disabilities. Each of these major areas has influenced the development of PBS into an applied science that is unique and independent of previous approaches within the social sciences. When the critical features of PBS are examined individually, they may not differentiate from techniques frequently used within other approaches, but it is the combination of all the critical features that identifies PBS as a unique approach. Carr et al. outline nine critical features of PBS that, in combination, make it unique and distinguish it from other approaches (Table 1.4). Each of these critical features will be described in order to provide a background from which to understand how this overall approach may have been more responsive in developing socially valid programs than applied behavior analysis approaches.
PBS Criterion I: Comprehensive Lifestyle Change and Quality of Life The first critical feature of PBS noted by Carr et al. involves a comprehensive lifestyle change and quality of life. This is perhaps the most critical feature of PBS and the feature that is considered to make this approach
Chapter | 1 Background of Social Validity
TABLE 1.4 PBS Critical Features Nine Critical Features of PBS (Carr et al., 2002) 1. 2. 3. 4. 5. 6. 7. 8. 9.
Comprehensive lifestyle change and quality of life Life span perspective Ecological validity Stakeholder participation Social validity Systems change and multicomponent intervention Emphasis on prevention Flexibility with respect to scientific practices Multiple theoretical perspectives
stand apart from other approaches in particular applied behavior analysis. Whereas applied behavior analysis approaches are sometimes considered to be highly focused on changing the behavior of a specific individual or group of individuals who are targeted for treatment, PBS attempts to change the complete lifestyle of the individual targeted for treatment, as well as the lifestyle of those associated with the treatment. One of the central differences between PBS and applied behavior analysis appears to be in the order of importance and the order of causation. Applied behavior analysis places importance on changing a problem behavior first in order to support relative changes in lifestyle and enhance quality of life. PBS places primary importance on conducting changes in lifestyle and quality of life, which in turn are supported by reductions in problem behaviors. Carr et al. describe how PBS is measured using more molar levels of analysis and data collection, such as social interactions, daily activities, and schedules, which are different from the more micro-level of measurement and analysis characteristic of applied behavior analysis, such as number of aggressive acts or frequency of self-injury. PBS is focused on a grander scale that examined how individuals participate in life activities, how they enjoy their activities, and how the life of those around them is improved.
PBS Criterion II: Life Span Perspective The second critical feature of PBS involves a life span perspective. This feature of PBS is differentiated from applied behavior analysis in that
15
16
Chapter | 1 Background of Social Validity
it redefines the maintenance of behaviors and it proposes examining changes in behavior and lifestyle for lengthy periods of time that include decades of change rather than months of success. Within applied behavior analysis, maintenance is measured by behavior change that persists when a treatment procedure is removed or discontinued. Within PBS, the measurement of maintenance is replaced with the examination of how treatment might be further developed or modified to ensure continued success, but treatment is never completely removed. The treatments developed within PBS are considered to be pliable in that they might never be removed but rather modified to meet the changing lifestyles of those involved with the treatment.
PBS Criterion III: Ecological Validity The third critical feature of PBS is ecological validity. This feature of PBS differentiates it from much of the research conducted within applied behavior analysis, due to its focus on using individuals who are already present in an environment to implement procedures rather than introducing researchers or others not already involved in a setting to implement procedures. In addition, the settings focused on in PBS are those an individual targeted for treatment would typically attend. Carr et al. explain that most research conducted within applied behavior analysis was done with highly trained researchers and in highly controlled settings. PBS, in contrast, focuses on conducting all elements of treatment using individuals who would naturally be involved with a treatment and conducting these treatments in natural environments where control was limited.
PBS Criterion IV: Stakeholder Participation Stakeholder participation was the fourth critical element described by Carr et al. This feature involved including a high level of input from consumers in the development and evaluation of treatments. Applied behavior analysis procedures were usually considered to be developed and evaluated by experts, but PBS differs from that approach by encouraging a method of exchanging information that is reciprocal among experts, consumers, and others. This involves having families and others provide information that can be actively used for (1) assessment purposes, (2) selection of
Chapter | 1 Background of Social Validity
comprehensive treatments, (3) practicality of treatments, and (4) selection of desirable outcomes. This feature appears to rely on a high level of collaboration among everyone involved in designing a PBS program.
PBS Criterion V: Social Validity The fifth critical feature of PBS is social validity. Carr et al. state that the social validity aspects of PBS have been amplified from the typical applied behavior analysis. This amplification includes consideration of a number of additional treatment elements to be evaluated as part of social validity. Social validity within PBS considers whether a treatment is practical for implementation, whether implementers of the treatment consider it valuable, whether the treatment can be appropriately implemented in various environments, whether the treatment produces subjectively acceptable results, and whether the treatment has produced an outcome that is considered beneficial. Social validity is described by Carr et al. as being more highly emphasized within PBS than it had previously been within applied behavior analysis.
PBS Criterion VI: Systems Change and Multicomponent Intervention The sixth critical feature of PBS was systems change and multicomponent interventions. Carr et al. explain that applied behavior analysis had focused a large amount of research on how single treatments could be used to produce changes in behaviors, but that these single treatments were ineffective in actual practice. Single interventions could only be effective under certain circumstances; multiple strategies were needed to address the numerous functions that may need to be addressed to change the behavior of an individual. PBS focuses on programs that can make a broader change, such as those implemented at the systems level. In addition, PBS treatments are not limited to demonstrations of the effectiveness of a single treatment, but incorporate treatments that may be effective when presented in a complete program of multiple treatments. This has been referred to as an “ecology of support” (Lucyshyn, Olson, & Horner, 1995), where multiple environments are continually rearranged to promote long-term maintenance of socially appropriate behaviors.
17
18
Chapter | 1 Background of Social Validity
PBS Criterion VII: Emphasis on Prevention An emphasis on prevention is the seventh critical feature of PBS. This feature is considered to be consistent with the legislation regarding early intervention and prevention of problem behaviors. This feature de-emphasizes the typical reactive approach to problems in times of crisis and places most emphasis on maintaining the nonoccurrence of problem behavior.
PBS Criterion VIII: Flexibility with Respect to Scientific Practices Flexibility with respect to scientific practices is the eighth critical feature of PBS described by Carr et al. This feature of PBS involves expanding assessment procedures, treatment techniques, and data to include those that may not have the experimental rigor required of other disciplines. The realities of operating within natural environments are considered to preclude the need for designing treatments that can satisfy the rigorous criteria required of experimental laboratory work or even achieve the level generally described within the literature on applied behavior analysis.
PBS Criterion IX: Multiple Theoretical Perspectives The final critical feature described by Carr et al. focuses on the incorporation of multiple theoretical perspectives. This feature is centered on the idea that the most important outcome produced by a program is not at the individual level but rather at the systems level. These multiple perspectives focus on developing changes in environmental contexts to support an entire social system as well as an individual within the system. These changes are not individual techniques but rather comprise reallocation of resources that are continuously adapted to meet the needs of an individual. In addition, the multiple theoretical perspective of PBS places value on recognizing cultural variables that may influence the development of programs in settings which include individuals from numerous cultural backgrounds.
Effects of the PBS Movement These critical features encompass the essential focus of PBS and in combination differentiate it from other applied sciences. A possible strength
Chapter | 1 Background of Social Validity
of the PBS approach is that it has been very efficient in making use of indigenous sources of reinforcement. PBS approaches have incorporated existing sources of reinforcement and useful aspects of highly preferred strategies that are frequently used by individuals implementing interventions. These approaches may appear much more familiar to staff, since the strategies may include several aspects of techniques that they have used for many years or with which they are very familiar. This may play a very important part in attaining a level of acceptable social validity, in that procedures that are familiar to those rating the acceptability of a treatment may find those familiar components to be more acceptable than those components that are less familiar. Michaels et al. (2005) describe a possible paradigm shift based on the PBS movement. They surveyed experts in PBS and found a decrease in the acceptability of consequence-based decelerative techniques and a decrease in the use of such procedures, although many had used these types of procedures in the past. The experts indicated that ethical reasons, along with the development of more effective treatment alternatives, were most influential in altering their acceptance of specific treatments. Their findings revealed that treatment acceptability can change over time, especially in the face of large-scale movements such as PBS, changing ethical guidelines, and the development of new treatment methodologies. Kennedy (2002a) may have provided some insight into why PBS has become a prevalent method for addressing problem behaviors. In describing his rationale for using behavior maintenance as the primary indicator for determining social validity, he points out that skills that are maintained in social contexts after treatment goals have been achieved are considered more socially valued than skills that have not been maintained. Since PBS focuses more on rearranging existing environments than on introducing new elements to the environment, these methods may have been considered more valuable because of their long-term maintenance in environments that are resistant to change. According to the Association for Positive Behavior Supports (APBS), the critical features of PBS include a life span perspective, a comprehensive lifestyle change, ecological validity, stakeholder participation, and social validity. The life span perspective necessitates consideration of the individual’s personal preferences, current repertoire of skills,
19
20
Chapter | 1 Background of Social Validity
age-related variables, family variables, and community variables. In addition, the life span perspective promotes choice making within behavior interventions to effectively involve the individual at all levels of the intervention. Comprehensive lifestyle change involves consideration of future environments in which the individual will participate and how an intervention may generalize into these novel settings over time, as well as support the development of new skills for future learning opportunities. This criterion involves more than reduction of problem behavior in a particular setting or increasing a particular behavior or skills set. The focus is more on modifying an entire way of life in a manner that will have lasting effects. Ecological validity comprises the concept that an intervention employs aspects of the environment familiar to the individual targeted for intervention. The use of artificial reinforcement is minimized and more natural sources of reinforcement are recruited for the intervention. The people and aspects of the environment that an individual targeted for intervention may typically come into contact with are used rather than staff or items that are unfamiliar to the person. This limits the use of interventions that may be effective within a highly controlled setting but have minimal portability to actual environments where an individual spends time (Reuf, Turnbull, Turnbull, & Poston, 1999). The social validity criterion involves using empirically validated procedures that are considered nonaversive and based on reinforcement procedures. In summarizing the development of the PBS movement, Dunlap (2006) recognizes the need for the approach to remain grounded in Association for Behavior Analysis (ABA) techniques while allowing the incorporation of novel approaches as well: Although the methods and techniques of ABA are essential, it is also clear that the dynamic problem-solving approach demonstrated so well by these ABA pioneers is the legacy that PBS had adopted. If PBS is to truly expand its ability to foster new solutions that will affect human well-being on a meaningful scale, it should retain this special brand of openness and empirical problem solving as a central, defining characteristic well beyond these current nascent years of development (p. 60).
Chapter 2
Conceptualizations of Social Validity Dimensions of Social Validity Habilitative Validity Working Model of Treatment Acceptability Decision-Making Model of Treatment Acceptability Expansive View of Treatment Acceptability
Epidemiological Conceptualization of Social Validity Assessment Model of Social Validity Maintenance Model of Social Validity Distributive Model of Treatment Acceptability
DIMENSIONS OF SOCIAL VALIDITY Baer et al. (1968), in describing the dimensions of applied behavior analysis, distinguish applied research from nonapplied research by how target behaviors for study were chosen. They state that with nonapplied research, target behaviors could be selected because of their importance to theory, convenience, performance ease, and/or recording simplicity. In contrast, applied research chooses target behaviors because of their relevance to, importance to, and interest to society. This need to determine the degree to which society appreciates the methods and outcomes produced by applied research influenced Wolf (1978) to define the need to measure the impact on society, and has influenced several others to develop models and conceptualizations of social validity and its components. Models and conceptualizations of social validity and its components have focused on multiple combinations of factors (Lennox & Miltenberger, 1990), epidemiological research and social marketing frameworks (Winett et al., 1991), behavior maintenance (Kennedy, 2002a), measures of habilitation (Hawkins, 1991), and distributive influences (Carter, 2008a, 2008b). In addition, models have been devised to Copyright © 2010 Elsevier Inc. All rights reserved
21
22
Chapter | 2 Conceptualizations of Social Validity
demonstrate how components of social validity, such as treatment acceptability, may influence other aspects of treatment, such as treatment integrity (Witt & Elliott) and treatment adherence (Reimers, Wacker, Cooper, and De Raad, 1992). The original conceptualization of social validity presented by Wolf (1978) asked behavior analysts to extend their views of how to measure the overall importance of their treatments. Most previous indicators of treatment importance were related to systematic demonstrations of the effectiveness of a treatment to change a specified target behavior. Wolf proposed incorporating a subjective method of determining the importance of treatment programs, which involved asking consumers of treatments how satisfied they were with the treatment. While this use of verbal report was a direct deviation from the traditional dimensions of behavior analysis established by Baer et al. (1968), it did follow a more basic tenet of the scientific approach of incorporating the principle of parsimony. The value of focusing on socially important behaviors has always been a primary focus of applied behavior analysis but has appeared to play a secondary role to demonstrations of effectiveness. Finney (1991) points out that Wolf offered a simple, pragmatic solution to including information about the importance of treatments from consumers; previously, this importance may have been determined by journal editors and reviewers who had no direct connection to the actual treatment. This proposed shift by Wolf to include more consumer input may also have been a hallmark in the field of behavior analysis because of the potential contribution it may have provided to the survival of the field (Schwartz & Baer, 1991)—even though the proposal by Wolf stepped beyond the boundaries of what was previously considered acceptable practice in the field of applied behavior analysis. While adherence to strict protocols and highly objective data collection have separated the field of applied behavior analysis from other less rigorous disciplines, it may have been necessary for the field to expand beyond its original confines. As noted by B.F. Skinner, the field must be prepared to change.
HABILITATIVE VALIDITY Hawkins (1991) argues that the term social validity was misleading and inappropriate for describing the types of activities associated with assessment in this area. He indicates that the term social validity implied assessment of outcomes which were important to society. He states that
Chapter | 2 Conceptualizations of Social Validity
numerous treatments target goals, behaviors, and outcomes that are primarily personal rather than influential upon society. He recommends that the term social validity be replaced by the more correct term “consumer satisfaction,” which he argues would more accurately define the process of assessing the opinions of consumers of behavior treatments. Additionally, Hawkins argues that the term “habilitative validity” is more in line with an applied behavior analytic perspective. Habilitative validity referred to the extent to which the goals, procedures, and outcomes of treatment resulted in increasing the benefits and decreasing the costs to the individual and others. Hawkins states that research needs to focus on associations and the predictive value of measures of consumer satisfaction on consumer behavior and other events. He argues that it would be beneficial to determine how well consumer satisfaction ratings predicted future consumer behavior such as attendance at treatment sessions, adherence to treatments, and recommending the treatment to others. The predictive potential of consumer satisfaction and other events, such as treatment effectiveness and the overall increase in benefits and reduction in cost to the individual, should also be examined. Hawkins states that measures of consumer satisfaction could be improved by asking the right type of questions. He states that questions of consumers should be directed toward scenarios that consumer satisfaction ratings might predict. Questions such as this might include asking a consumer to imagine that the treatment goal has been achieved, the treatment procedures have been implemented, or the treatment outcome has been achieved, and then asking the consumer to consider how much more beneficial life might be for the individual who received the treatment. Hawkins concludes that measures of consumer satisfaction could potentially be useful for increasing the habilitative validity of treatment goals, procedures, and outcomes, especially when combined with more objective data. The objective methods for validating habilitative behaviors described by Hawkins included using normative averages of behavior for comparison, examining average correlations between performance and resulting effects, testing various behaviors and levels of performance in relation to their associated outcomes, and testing outcomes associated with alternative treatments. Hawkins argues for a definitive shift in the way that social validity is described and in the way that social validity information is utilized. Most importantly, Hawkins,
23
24
Chapter | 2 Conceptualizations of Social Validity
although highly critical of social validity, does indicate that evaluating outcomes as perceived by consumers has the potential to allow the field of applied behavior analysis to stand out from other similar disciplines. Several conceptualizations have been developed regarding the social appropriateness of the procedures used within a treatment. Referred to as treatment acceptability, these conceptualization have attempted to explain the relationships among specific elements of treatment acceptability (Witt & Elliott, 1985), and determine the influence that treatment acceptability has upon other treatment variables (Reimers, Wacker, & Koepple, 1987). These conceptualizations have also attempted to organize numerous factors attributing to treatment acceptability (Lennox & Miltenberger, 1990), and to explain how treatment acceptability measurement might be distributed across different societal dimensions (Carter, 2008b).
WORKING MODEL OF TREATMENT ACCEPTABILITY Witt and Elliott (1985) describe a “working” model of treatment acceptability which focused on the most significant variables influencing treatment acceptability. Their model includes the four following variables: (1) acceptability of treatment, (2) use of treatment, (3) integrity of treatment, and (4) effectiveness of treatment. Each of these variables was described as occurring in a sequential order and having a reciprocal influence upon each other (see Figure 2.1). Acceptability of a treatment is the first-order variable which influences the second-order variable of use of treatment. According to this model, a treatment must first be accepted before it is used; then, once a treatment is accepted, its likelihood of being used increases. In turn, a treatment that is used more frequently is more likely to be accepted. The third-order variable is integrity of the treatment, which can be judged only once a treatment is being used. Consequently, the integrity of a treatment may speak to the likelihood that a treatment will be used in the future. The fourth-order variable is the effectiveness of treatment, which may be in direct relation to the degree of treatment integrity. In turn, the effectiveness of a treatment may increase or decrease the integrity with which a treatment is implemented. Finally, the effectiveness of the treatment may be highly influential upon the acceptability of the treatment. The acceptability of a treatment may also influence the effectiveness of a treatment, through the increased use of the treatment, and enhance integrity of the treatment procedures.
25
Chapter | 2 Conceptualizations of Social Validity
Acceptability of treatment
Use of treatment
Effectiveness of treatment
Integrity of treatment
FIGURE 2.1 Witt and Elliott’s “Working Model” of social validity. From Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. Kratochwill (Ed.), Advances in school psychology (Vol. 4, pp. 251–288). Hillsdale, NJ: Erlbaum. Reprinted with permission.
The relationships among these variables have been supported and refuted in various studies. While this model provides a very parsimonious approach to understanding the relationships among some elements of treatment acceptability, it may not completely reflect the comprehensive concept of treatment acceptability. Subsequent conceptualizations of treatment acceptability have extended this model and attempted to validate specific relationships among additional variables.
DECISION-MAKING MODEL OF TREATMENT ACCEPTABILITY Reimers et al. (1987) developed a model of treatment acceptability which was more complex than previous models and included an emphasis on the level of understanding of a treatment. Their model consisted of a flowchart from which decisions could be made for responding to numerous potential outcomes related to the implementation of a treatment (Figure 2.2). The importance of understanding the treatment was directly related to the acceptability of the treatment, and the initial decision to be made regarding a recommended treatment was to determine if the treatment was clearly understood. Within this model, a good understanding of a treatment is a prerequisite to determining the degree to which a treatment was considered acceptable. In other words, a treatment could not be considered acceptable or unacceptable by someone who did not first have a good understanding of the treatment. When a treatment is not clearly understood, the result is low compliance and
26
Chapter | 2 Conceptualizations of Social Validity
Recommended treatment
Good understanding
Poor understanding Low compliance
Low acceptability
High acceptability
Low effectiveness
Low compliance
High compliance
Provide education
Low effectiveness Modify treatment or propose new treatment
Low effectiveness Low maintenance Re-assess Modify treatment or propose new treatment
High effectiveness Much disruption to family or classroom
Little disruption to family or classroom
Variable maintenance
High maintenance
FIGURE 2.2 Reimers et al. decision-making model of treatment acceptability. Reimers, T. M., Wacker, D. P., & Koepple, G. (1987). Acceptability of behavioral treatments: A review of the literature. School Psychology Review, 16, 212–227. Copyright 1987 by the National Association of School Psychologists, Bethesda, MD. Reprinted by permission of the publisher. www.nasponline.org
low effectiveness, so education to ensure that the treatment is clearly understood is required. Once a treatment is clearly understood, it could be considered to have high or low acceptability, with either of these potentially resulting in low compliance, low effectiveness, and the need to modify the treatment. In addition, treatments with either low or high acceptability might result in high compliance with the treatment. High compliance with a treatment could then result in either high or low levels of effectiveness. Treatments with low effectiveness may result in low maintenance of treatment effects and the need to re-assess and modify the treatment. Highly effective treatments may result in a considerable amount of disruption to the environment, which may make maintenance of the treatment highly inconsistent and thus require modification. Conversely, highly effective treatment may also produce minimal disruption to the environment and result in high levels of maintenance. This model predicts situations such as a treatment resulting in minimal degrees of
Chapter | 2 Conceptualizations of Social Validity
behavior change, low maintenance of behavior change, or unexpected disruption to the environment which requires that the treatment be modified or replaced by a different treatment. Once a treatment is modified or a different treatment introduced, the decision-making flowchart would be followed again from the beginning.
EXPANSIVE VIEW OF TREATMENT ACCEPTABILITY Lennox and Miltenberger (1990) conceptualized a component of social validity which combined numerous factors considered to encompass the construct of treatment acceptability. They described 12 factors which were grouped into the four categories of efficacy considerations, secondary effects, legal/social implications, and practical considerations. Efficacy considerations included motivational variables and treatment effectiveness as factors. Motivational variables include those variables which are functionally related to the target behavior. In other words, those motivating variables are those that aid in generating or maintaining target behaviors. The examination of motivational/maintaining variables as related to the concept of treatment acceptability relies upon the potential of these variables to increase the effectiveness of treatments by including functionally equivalent replacement behaviors. Lennox and Miltenberger described several methods for identifying motivational/maintaining variables, such as interviews, direct observations, and experimental analyses. Treatment efficacy information was comprised of empirical evidence that supported the treatment as a legitimate behavior change agent. The category of secondary effects included the factors of side effects and abuse potential. Side effects referred to treatment effects on behaviors or states which were not the primary target of the intervention. Lennox and Miltenberger pointed out that side effects could be adaptive or maladaptive; either could impact how acceptable a treatment would be rated. Abuse potential referred to misuse of procedures by those responsible for implementing a procedure. In general, the more susceptible to possible abuse a treatment is, the less acceptable it would appear to be as a treatment. The category of legal/social implications included treatment restrictiveness/intrusiveness, treatment precedence, social acceptability, and regulatory factors. Restrictiveness/intrusiveness concerned how aversive treatments were within a hierarchy of least to most potentially distressing
27
28
Chapter | 2 Conceptualizations of Social Validity
treatments. Treatment precedence referred to how frequently a procedure had been used to address a specific target behavior, with higher prevalence rates with appropriate outcomes being considered more acceptable. The social acceptability factor considered the results of previous research regarding the acceptability of treatments and how this information might influence future acceptability of treatments. Regulatory factors were concerned with how laws, legislation, guidelines, rules, and the like impacted the acceptability of treatments. The category of practical considerations included staff competence, staff cooperation, treatment efficiency, and cost effectiveness. The factor of staff competence dealt with how well staff were able to carry out treatment procedures, while the staff cooperation factor concerned how willing staff were to implement a treatment. Treatment acceptability was considered to be increased when staff were competent and cooperative toward treatments and decreased when these factors were not achieved. Treatment efficiency as a factor involved how quickly a treatment produced results. Treatments that were able to produce quick results could increase the acceptability of the treatments, although other factors, such as the use of punishment procedures which produce quick results, might override this influence. Cost effectiveness as a factor concerned the feasibility of a particular treatment in comparison to the feasibility of other treatments which might influence the acceptability of the treatment. The work by Lennox and Miltenberger appears important because it provided a more comprehensive conceptualization of treatment acceptability than had been previously described.
EPIDEMIOLOGICAL CONCEPTUALIZATION OF SOCIAL VALIDITY Winett, Moore, and Anderson (1991) extend the concept of social validity by proposing a broader definition of social validity. In their model, measuring against the elements of this broader definition, one would interactively test and shape a treatment in order to establish the social validity of the treatment. The first step in Winett, Moore, and Anderson’s model addresses the first aspect of social validity, which involves ensuring that appropriate behaviors are targeted for treatment and that appropriate goals are selected for the treatment. Their model involves determining important problems that need to be addressed by
Chapter | 2 Conceptualizations of Social Validity
determining the verifiable importance of a target behavior and goals through procedures derived from the field of epidemiology. These procedures included several criteria for possible use in prioritizing behaviors to target for treatment and determining the appropriateness of the goals of a treatment. Within this model, target behaviors would be prioritized based on the associated risks with undesirable outcomes relative to the individual and within the general population. Treatments and treatment goals chosen based on these criteria would be considered socially valid because of their importance determined from studies which may be expensive and take years to conduct. Although these treatments and goals might be considered to have social validity because of their importance revealed by these studies, Winett, Moore, and Anderson pointed out that such treatments may still lack social acceptability because objective data are not the only basis for making acceptability determinations. The second step in the Winett et al. model is focused on arranging treatments which are considered to be socially acceptable. Their model incorporates a social marketing framework, centered around four key variables and one additional variable to design treatments, which would potentially enhance the acceptability of these treatments. Table 2.1 provides an outline of these social marketing variables and their corresponding equivalent variables used in the development of behavior treatment. These variables would be examined when developing a treatment, through information gathering on each of these variables, pilot-testing treatments, refining treatments to fit the target population, and retesting treatments until they appear to be well designed. Following this social marketing process should increase the social validity of the treatments. The third step in the Winett et al. model focuses on the third aspect of social validity, involving the social importance of treatment effects. Their model discusses determining whether a treatment has significant outcomes under ideal and practical-world situations. They also discuss considering the scope of outcomes in relation to an individual and to an entire population. Their model includes criteria for treatment effectiveness which would require a treatment to achieve some level of expected outcome in order to be considered socially valid, rather than simply being highly accepted by consumers. The model proposed by Winett et al. (1991) appears to be most relevant in developing a comprehensive database on social validity among specific
29
30
Chapter | 2 Conceptualizations of Social Validity
TABLE 2.1 Key Variable Equivalencies in Epidemiological Model by Winett et al. (1991) Key Variables Marketing Variables
⫽
Social Validity Variables
Product (program, intervention, or system)
⫽
Treatment components
Price (social and psychological costs)
⫽
Side effects
Place (where product is available)
⫽
Intrusiveness (removal from less restrictive setting; time and effort)
Promotion (dissemination)
⫽
Consultation practices (knowledge provided to consumers; language used to describe treatment)
Positioning (particular niche of product as distinct from similar products)
⫽
Reinforcement-based, punishmentbased, PBS, behavior analytic, counseling, cognitive, etc.
populations which could be examined for extended periods of time and used to derive future estimates of social validity. It also would provide a well-researched basis from which to begin selecting and designing treatments which should be considered socially valid among those familiar with the research and whose determinations of social validity were primarily grounded in objective data. This level of analysis would be difficult for practitioners to achieve alone, but it might be reasonable for practitioners to contribute to some larger research database examining social validity. Finney (1991) commented on how the Winett et al. model considered highincidence disorders to be more important than low-incidence disorders, because high-incidence disorders were likely to have more of an impact on a larger segment of society. Finney pointed out that low-incidence disorders might also have a large dysfunctional impact on society which might require expenditures of large amounts of money, time, and effort if not addressed through prevention or early treatment. In addition, considering the need for a potentially lengthy process of testing and refinement required under the Winett et al. model, a quote from Baer et al. (1968)
Chapter | 2 Conceptualizations of Social Validity
should be considered: “[S]ociety rarely will allow its important behaviors, in their correspondingly important settings, to be manipulated repeatedly for the merely logical comfort of a scientifically skeptical audience.”
ASSESSMENT MODEL OF SOCIAL VALIDITY Gresham and Lopez (1996) conceptualized social validity as a product of the social validity assessments. They described how social validity could be enhanced by improving upon the procedures used to assess social validity. Social validity could be influenced if social validity assessment procedures were used more frequently during the treatment process and provided more useful data which could serve in making decisions. They proposed the need for recognition and incorporation of several practices for conducting assessments of the social validity of school-based practices. Their recommendations included the incorporation of multiple sources of information which go beyond the consumers’ opinions of treatments and rely more heavily on data obtained from normative comparisons, from sources such as developmental norms, structured interviews, and archival data. Each of their recommendations described methods for measuring social validity in relevant manners; the results of the measurements could then be used to make modifications during the treatment process in order to increase the overall social validity of a treatment. While they stated that consumers’ opinions were important toward marketing a treatment or a program, they espoused that the opinions of consumers were not always in the best interest of the individual targeted for treatment. By focusing on assessment methods other than consumer opinion, they felt that more socially valid treatments could be provided to their clients.
MAINTENANCE MODEL OF SOCIAL VALIDITY Kennedy (2002a & 2002b) proposed a conceptualization of social validity that centered on the examination of behavior maintenance as the primary indicator of social validity. This conceptualization was similar to that of Gresham and Lopez in that it placed more emphasis on normative comparisons rather than on consumers’ opinions. Kennedy argued that using maintenance of behavior as a primary indicator of social validity could (1) provide a means of analyzing and understanding how social contexts are affected by behavior change, (2) allow for empirical investigation of diverse sources of social control, and (3) allow for
31
32
Chapter | 2 Conceptualizations of Social Validity
parsimonious incorporation of subjective evaluations and normative comparisons. Kennedy argued that skills developed during an intervention could be considered to have social value if they are maintained because the person’s social environment would provide opportunities for continued reinforcement of the skills. In addition, if skills developed during an intervention were not maintained, because of lack of continued opportunities to obtain reinforcement for displaying the skills in the person’s environment, the intervention would be considered not socially valid. Kennedy explained that the sources of social control which influenced the construct of social validity were numerous and that applied behavior analysis did not have a technology for assessing these numerous sources of control. By using behavior maintenance as the primary indicator of social validity, techniques for promoting behavior maintenance could be manipulated and evaluated to search for causes related to maintenance or the lack of maintenance. This could improve upon the methods used to study social validity by allowing opportunities for experimental manipulations related to behavior maintenance, which could then be understood as social validity manipulations. Kennedy opined that both subjective evaluations and normative comparisons were easily integrated into the evaluation and “repair” of behavior maintenance. Both of these techniques could be used to determine reasons for lack of behavior maintenance, and the results combined with maintenance data to determine influences on social validity. Kennedy also provided specific guidelines for incorporating subjective evaluations and normative comparisons in the assessment of behavior maintenance, which described the time, content, type of interventions derived, and experimental designs for assessment. While the construct of social validity proposed by Kennedy appears to have potential for improving research on social validity, empirical examples using these procedures have not appeared in the literature. These techniques may require an extended amount of time due to the lack of research focusing primarily on behavior maintenance.
DISTRIBUTIVE MODEL OF TREATMENT ACCEPTABILITY Carter (2008b) addressed a component of social validity with a proposed model of treatment acceptability. This model offers a combination of
Chapter | 2 Conceptualizations of Social Validity
Strupp and Hadley’s (1977) tripartite model of therapeutic outcome and Wolf’s (1978) conceptualization of social validity. This combination was first recommended by Bernstein (1989), who provided a matrix which delineated nine possible pairings for measuring social validity. Carter’s distributive model separated overall treatment acceptability into three distinct categories considered to be influential. The three categories included societal influences, consultant influences, and influences associated with consumers of treatments. Societal influence was defined as opinions, evaluations, and actions of large segments of society. These large segments of society included groups such as lawmakers, legislators, professional organizations, parent organizations, university programs, and the like. These groups may develop laws regarding treatments, may formulate opinions regarding treatments, may promote research in treatment development, or may advance movements regarding treatments. An example of societal influence may be educational laws such as IDEA (2004), which mandates that functional behavioral assessments be used for treatment development under certain circumstances. This mandate appears to make treatments developed without these procedures less acceptable, at least in certain circumstances. Other examples of societal influences on treatment acceptability may include movements such as PBS and research in functional assessment technology. PBS has advanced the use of reinforcement-based procedures rather than consequence-based procedures. This movement has been associated with a shift in the acceptability of treatments among PBS experts, as indicated by the research of Michaels et al. (2005). Another example of societal influence may involve dissemination of research developments such as functional assessment methodology. This methodology has been shown to influence the acceptability of treatments derived from functional assessments and those derived without functional assessments (Mccausland, Grey, Wester, & McClean, 2004). The second category of influence described by Carter was consultant acceptability. This category was concerned with the individual who develops, recommends, and evaluates treatments. Factors such as a consultant’s professional training, experience/history with certain treatments, and individual competencies may influence the acceptability of a treatment or may prevent a potentially acceptable and effective treatment from being recommended or implemented. Consultants are limited
33
34
Chapter | 2 Conceptualizations of Social Validity
by the professional training which they have received, and most consultants are limited by ethical guidelines which prevent them from implementing treatments with which they have little or no competencies. While consultants can seek out assistance from other professionals in order to implement certain treatments, competent professionals may not always be available and thus certain treatments may have to be avoided. These limitations may preclude the need for development and recommendation of other treatments which may then be considered acceptable or nonacceptable by consumers who have no opportunities for comparison with other unavailable treatments. Consultants who have been trained in a particular orientation may be inclined to select and promote treatments derived from that particular orientation. Consultants may also belong to specific professional organizations which promote the use of specific treatments, and this may influence the consultants’ personal acceptance of these or other treatments, which in turn may be passed along to consumers. Some other ways in which consultants might influence treatment acceptability would be in their selection of specific treatment components and the manner in which they choose to evaluate the acceptability of the treatments they design. Consultants may choose to include reinforcement techniques or punishment techniques in developing a treatment, or may choose techniques which can be implemented quickly or which require extended amounts of time to implement. The consultant may alter the acceptability of a treatment by choosing to include these types of techniques, which have been demonstrated to influence the acceptability of treatments (Kazdin, 1980; Witt, Elliott, & Martens, 1984). The manner in which a consultant evaluates the acceptability of a treatment may also influence acceptability. Consultants may choose to evaluate acceptability with instruments with widely varying degrees of reliability and validity. The third category of influence which Carter described was consumer acceptability. This category was comprised of factors associated with the individuals receiving treatment or those closely connected to the treatment who are frequently asked to rate the acceptability of treatments (such as parents, guardians, etc.). The factors in this category were mostly related to specific characteristics of the consumers of treatments, which for those receiving treatment included the severity of the problem they were displaying. Treatment acceptability has been shown
Chapter | 2 Conceptualizations of Social Validity
to vary according to the severity of the behavior which is being treated (Kazdin, 1980; Reimers, Wacker, Cooper, & DeRaad, 1992; Tingstrom, 1990). The characteristics of those who rate the acceptability of treatments has also been shown to influence their acceptability ratings of the treatments. These variables are numerous but include characteristics such as socioeconomic class (Heffer & Kelley, 1987), gender (Kazdin, 1980), and marital distress (Miller & Kelley, 1992). Carter stated that each of these categories could interact, and when combined formed an overall treatment acceptability which was frequently discussed and measured. This model attempted to address the reciprocal nature of treatment acceptability. Skinner (1972), in discussing issues of control, described how the experimenter develops conspicuous controls over the pigeon to evaluate the effects of the controls, but that the pigeon also exerts a level of control by influencing the apparatus and procedures developed by the experimenter. Skinner pointed out that “some such reciprocal control is characteristic of all science” (p. 161). This model encourages the examination of such reciprocal control among society, consultants, and consumers with regard to treatment acceptability. The benefits of this type of distributive model were described as providing a means of evaluating certain factors in a more discrete manner than had been done previously and potentially providing more information on how factors in each category of the model influenced overall treatment acceptability. The distributive model was also described as providing a framework for determining the causes of contradictory findings in treatment acceptability research and for examining interactions among the different categories. Additionally, it was argued that the model provided a beneficial means for evaluating potential trends within treatment acceptability, which might be occurring at different rates for each of the different subdivisions of the model. While this model may provide some benefits conceptually, no empirical evidence was provided to support the proposed benefits of the model. In order to offer a truly adequate representation of social validity, numerous variables should be considered. Strupp and Hadley (1977) pointed out a number of issues related to the incorporation of a tripartite model for analyzing mental health and psychotherapy procedures. They indicated that judgments may vary depending on who conducts
35
36
Chapter | 2 Conceptualizations of Social Validity
a judgment evaluation, that evaluators have uniquely vested interests that are not necessarily invalid, that the limitations of judgment evaluations need to be emphasized, and that the only accurate judgment is one that is comprehensive and examines all facets of their tripartite model. Their recommendations are highly relevant to the examination of social validity in a manner that is highly comprehensive and recognizes and emphasizes the limitations inherent in the current state of the science for conducting such measurements. Framing social validity research on a conceptualization that is highly comprehensive appears advantageous for recognizing correlations among variables that may have previously gone unrecognized. Based on this rationale, the conceptualization used throughout this book is founded on the distributive model of treatment acceptability proposed by Carter (2008b). The model is one that is fluid, in that it can easily be expanded when new relevant variables are identified. Although this model is highly limited and in need of supportive data, it does offer a highly comprehensive framework from which to explore the various dimensions of social validity.
Chapter 3
Instruments for Evaluating Social Validity Interviews Direct Observation Performance Criteria Formal Treatment Acceptability Measurement Instruments Treatment Evaluation Inventory Treatment Evaluation Inventory-Short Form Treatment Acceptability Rating Form Treatment Acceptability Rating Form Revised Intervention Rating Profile Intervention Rating Profile-15 Children’s Intervention Rating Profile Behavior Intervention Rating Scale Intervention-Process Rating Scale (IPRS) Abbreviated Acceptability Rating Profile
Comparison of Treatment Acceptability Measures Problems Related to Social Validity Measurement Methods for Increasing the Use of Treatment Acceptability Instruments Increase Availability of Instruments and Other Measures Increase Consultant Awareness of Instruments Increase Competency in the Administration and Scoring of Instruments Increase Knowledge of Potential Uses of Acceptability Information Re-emphasize the Importance of Social Validity
The term treatment acceptability, a component of social validity, was defined by Kazdin (1980) as judgments of treatments by actual or potential consumers of the treatments, such as nonprofessionals, clients, laypersons, and others. Wolf (1978) stated that social validity refers to decisions based on the social importance of interventions on three related levels. First, the objectives of the treatment must be socially significant, in that Copyright © 2010 Elsevier Inc. All rights reserved
37
38
Chapter | 3 Instruments for Evaluating Social Validity
the desired outcome is both important and pertinent. Second, the treatment procedures must be deemed as socially appropriate, in that they are generally considered acceptable by society. Third, the effects of the treatment should have clinical significance. Kazdin (1980) noted that social validity research has mainly focused on the appropriateness of treatment procedures, an area of research also known as treatment acceptability. Research in the area of treatment acceptability is relatively new to the area of education and psychology. Recognition of the need for research in the area of social validity became more evident with an elaboration of the topic in an article by Baer et al. (1968). Prior to this publication, the primary emphasis with regard to educational and behavioral treatments had been treatment efficacy (Witt, 1986). Wolf (1978) described social validity to include: (1) significance of treatments (important vs. trivial or insignificant); (2) acceptability of treatments (judgments regarding fairness/appropriateness of treatment); and (3) importance of treatment effects (statistical significance vs. practical significance). The examination of social validity of treatments has only recently become an area of interest to the fields of education and psychology. The need for research regarding societal validation of treatments has been proposed by Baer et al. (1968) and further defined by Wolf (1978). Initial research in the area of treatment acceptability was conducted by Kazdin (1980) and has grown with the development of several instruments for measuring treatment acceptability. Although the number of instruments available for measuring treatment acceptability has increased, the use of these instruments appears to have decreased. Several studies have demonstrated a decreasing trend in the use of social validity measurement (Carr, Austin, Britton, Kellum, & Bailey, 1999; Dunlap, Clarke, & Streiner, 1999; Kennedy, 1992; Poling & Ehrhardt, 2000). Research on social validity has typically taken a back seat to other types of research which focus on demonstrations of techniques to increase or decrease specific behaviors (Kennedy, 2002b). This is similar to research on areas such as treatment integrity, generalization, maintenance, and so on. Decreasing trends in measurement of social validity have been noted in light of increasing trends in the experimental analysis of problem behaviors. As stated by Kennedy, “the current state-of-the-science in behavior analysis focuses on experimental analyses of events occurring
Chapter | 3 Instruments for Evaluating Social Validity
within a few seconds of problem behavior in artificial environments over very brief periods of time (e.g., 5–15 min)” (2002b, p. 143). Kennedy (2002b) went on to explain that one of the most important areas for behavior analytic researchers to consider was the social context in which behavior change occurred. Considering the limited emphasis on social validity, much of the research that has been conducted on social validity has been secondary to the primary topic of many research articles. These types of research articles typically have a primary focus of demonstrating techniques for behavior change and then reporting that the treatment was considered socially valid based on some form of informal rating scale. While this method may be very useful, it may be a misrepresentation of the social validity of certain treatments, since it is very likely that only when a treatment receives favorable social validity ratings would this information be included in the article. In other words, a treatment that was effective at changing behavior but was not considered socially valid would most likely not be published in a journal, or the social validity information would not be included. What is possibly more disturbing is that only those informal instruments which provide favorable social validity ratings may be considered valid, while instruments or measures that do not support a treatment deemed useful by a researcher may be considered invalid. Although the dimensions of applied behavior analysis require consideration of social validity, it does not appear that most journals are requiring this type of information as a condition for publication. Some methods for alleviating this trend may be to make social validity assessments more efficient, more available, more comprehensive, and more of a requirement for dissemination of research findings. Social validity assessment can be conducted in several ways and at several different points in time. Table 3.1 outlines some of the different methods for assessing social validity. While there are several methods for measuring social validity, the most frequently used method is the questionnaire or rating scale. Many times, these are informal instruments developed for the purpose of evaluating a specific treatment program and do not provide any validation of the instrument. These types of instruments are prevalent within the literature and tracking their use is difficult because in many cases they may be analogous to a “one-hit wonder” that is never seen or heard from again in the literature.
39
40
Chapter | 3 Instruments for Evaluating Social Validity
TABLE 3.1
Some General Methods for Assessing Social Validity
General Methods for Assessing Social Validity ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■
Consumer comments Informal discussions Interviews Inventory sheets Questionnaires Surveys Rating scales Mixed-item forms Formal instruments Comparison (performance criteria, template, normative, peer, self, etc.) Observations Treatment efficacy Treatment generalization Treatment maintenance
Other more formal instruments for assessing treatment acceptability that have undergone some level of validation are also used, and these can be somewhat easier to track in the literature. These instruments are not always the primary focus of a research study, and it is often necessary to read an article very closely to determine when and where these instruments were used. Some of the more common strategies for gathering information on social validity will be described and some of the formal instruments used to evaluate treatment acceptability will be reviewed in detail.
INTERVIEWS Interviews are one method of assessing social validity that can provide a large amount of information in a short period of time. While information from interviews may be plentiful, it may also not be highly relevant toward determining the social validity of a treatment. An unstructured interview may lead to discussions that are not directly related to the assessment of social validity, unless the interviewer is highly skilled at redirecting the person being interviewed. Another interview option may be a semi-structured interview which delineates questions relevant to
Chapter | 3 Instruments for Evaluating Social Validity
Semi-Structured Interview for Soical Validation Consultee’s Name Date Consultant’s Name School A. Social Significance of Goals 1. 2. 3. 4. 5. 6. 7. 8. 9.
What behaviors lead yout to request consulation? Which behaviors are the most problematic for in your classroom? Describe how these behaviors cause classroom problems. If these problematic behaviors were decreased or eliminated, how would this affect (the client)? Other students in your classroom? Your teaching in your classroom? Do you see these behaviors as skill deficits? Performance deficits? What do you base this on? Defince each behavior as specifically as possible. How do these behaviors affect other students in your classroom? Students in other classroom? How do theses behaviors affect other school personnel (e.g., principal, other teacher, staff, etc.)? Which behavior (s) do you think would be the most beneficial for to change how? Why? Which behavior (s) would have the greatest long-term benefits for ? Why? B. Social Acceptability of Procedures
10. How do you feel about the procedures we discussed to change ’s behavior? 11. Which aspects of the intervention do you like the most? Why? Which do you like the least? Why? 12. Which aspects of this intervention would be the most difficult to implement? Why? Which aspects would be the least difficult to implement? Why? 13. Here are some ways in which we could change the intervention. Do these changes make the interevention more acceptable and easier to implement? Why? What would you recommend for the further changes? 14. What, if any, potential negative effects might this intervention have on ? On other student in your classroom? 15. Do you think this intervention is likely to be effective in solving ’s problem? Why? Why not? What are some ways we could determine whether or not the intervention had solved ’s problem? C. Social Importance of Effects 16. Describe how well you think the intervention worked. 17. What behavior changes did you observe? Did these changes make a difference in ’s behavior in your classroom? In other school settings (e.g., other classrooms, cafeteria, playground, etc.)? 18. Is ’s behavior now similar to that of the average student in your classroom? If not, do you think that continued use of the intervention would accomplish this goal? Why or why not? How long do you think this might take if we continued this intervention? 19. Are you satisfied with the outcomes of this internvention? How satisfied are you? Why? 20. Do you think this intervention would work with similar problems in the future? Why or why not? 21. Would you recommend this intervention to other teacher? Why or why not? What aspects of this interventio would you change before recommending this intervention to other teachers?
FIGURE 3.1
Semi-structured interview for social validation.
Gresham, F. M., & Lopez, M. F. (1996). Social validation: A unifying construct for school-based consultation research and practice. School Psychology Quarterly, 11, 204–227. Reprinted with permission.
specific areas of social validity. Gresham and Lopez (1996) provided an example of a semi-structured social validity interview which allocated questions to three areas of social validity (Figure 3.1). The interview consists of 21 base questions with additional follow-up questions. The first nine questions focus on the significance of treatment goals, the next six questions relate to acceptance of treatment, and the last six questions are concerned with the importance of treatment effects. Their interview may be a useful approach to gathering information on social validity,
41
42
Chapter | 3 Instruments for Evaluating Social Validity
but extended applications of the usefulness of the interview have not been disseminated. Lane (1997) offered another example of a semi-structured interview to assess social validity that focused on perspectives of students targeted for treatment of academic difficulties, titled the Children’s Social Validity Interview (CSVI). The CSVI consists of Yes/No questions, Likert-type rating scale responses, and open-ended items. The CSVI is interesting in that it asks children to evaluate how often they use the intervention in various settings. This touches on another method for assessing social validity, which focuses on how a treatment is used by consumers. Recurring use of a treatment is an indicator of social validity (Gresham & Lopez, 1996; Kennedy, 2002a). The recurrent use of treatment may be assessed by asking consumers or via direct observation of the use of a treatment. Treatments that are used more often may be considered to have higher social validity than treatments that are available but used less frequently. This aspect of measuring social validity may be a useful indicator, but should be evaluated within the larger scheme of social validity. Some treatments that are frequently used may also have limited acceptability, such as punishment procedures. Punishment procedures may be frequently used but may be considered highly unacceptable by others. Some information collected within functional behavior assessment questionnaires and interviews can provide evidence of social validity, and it might be beneficial for future research to consider correlating these types of assessments with measures of social validity. An example of a functional assessment interview that provides a significant amount of information that could be indicative of social validity is the StudentAssisted Functional Assessment Interview (Kern, Dunlap, Clarke, & Childs, 1994). This interview consists of four sections of items that deal with identifying and evaluating aspects of a classroom that a student likes or dislikes (see Figure 3.2). The first section includes a three-point Likert-type scale that concerns some broad features of a classroom environment, as well as some more specific evaluations of such things as availability of rewards. The second section consists of some fill-in-theblank questions and some open-ended items that focus on classroom problems and personal interests of the student. The third section consists of a five-point Likert-type scale that asks the student to rate how well they like a number of academic subjects. The fourth section directly relates to section three by asking follow-up questions concerning
43
Chapter | 3 Instruments for Evaluating Social Validity
STUDENT-ASSISTED FUNCTIONAL ASSESSMENT INTERVIEW
STUDENT DATE INTERVIEWER
SECTION I 1.
In general, is your work too hard for you?
Always
Sometimes
Never
2. In general, is your work too easy for you?
Always
Sometimes
Never
3.
When you ask for help appropriately, do you get it?
Always
Sometimes
Never
4.
Do you think work periods for each subject are too long?
Always
Sometimes
Never
5.
Do you think work periods for each subject are too short?
Always
Sometimes
Never
6.
When you do seatwork, do you do better when someone works with you?
Always
Sometimes
Never
7.
Do you thinkk people notice when you do a good job?
Always
Sometimes
Never
8.
Do you think you get the points or rewards you deserve when you do good work?
Always
Sometimes
Never
9.
Do you think you would do better in school if you received more rewards?
Always
Sometimes
Never
10. In general, do you find your work interesting?
Always
Sometimes
Never
11. Are there things in the classroom that distract you?
Always
Sometimes
Never
12. Is your work challenging enough for you?
Always
Sometimes
Never
FIGURE 3.2
Student-Assisted Functional Assessment Interview.
From Kern, L., Dunlap, G., Clarke, S., & Childs, K. E. (1995). Student-assisted functional assessment interview. Diagnostique, 19, 29–39. Reprinted with permission.
specifically what the student likes or dislikes about a certain academic subject. This functional assessment interview seeks to obtain a great deal of information from students about their personal preferences in general and as they relate to specific academic subjects. The information that is
44
Chapter | 3 Instruments for Evaluating Social Validity
SECTION II 1. When do you think you have the fewest problems with school?
in (Target behavior)
Why do you not have problem during this/these time(s)?
2. When do you think you have the most problem with school?
in (Target behavior)
Why do you have problem during this/these time(s)?
3. What changes could be made so you would have fewer problem with ? (Target behavior)
4. What kind of rewards would you like to earn for good behavior or good school work
5. Waht are your favorite activities at school?
6. What are your hobbies or interests?
7. If you had the chance, what activities would you like to do that you don’t have the opportunity to do now?
Probes
FIGURE 3.2 (Continued)
45
Chapter | 3 Instruments for Evaluating Social Validity
SECTION III Rate how much you like the following subjects: not at all Reading
1
2
3
4
very much 5
Math
1
2
3
4
5
Spelling
1
2
3
4
5
Handwriting
1
2
3
4
5
Science Social Studies
1 1
2 2
3 3
4 4
5 5
English
1
2
3
4
5
Music
1
2
3
4
5
P.E. Computer
1 1
2 2
3 3
4 4
5 5
Art
1
2
3
4
5
SECTION IV What do you like about Reading?
What don’t you like about Reading?
What do you like about Math?
What don’t you like about Math?
What do you like about Spelling?
What don’t you like about Spelling?
What do you like about Handwriting?
What don’t you like about Handwriting?
What do you like about Science?
FIGURE 3.2 (Continued)
fair
46
Chapter | 3 Instruments for Evaluating Social Validity
What don’t you like about Science?
What do you like about Social Studies?
What don’t you like about Social Studies?
What do you like about English?
What don’t you like about English?
What do you like about Music?
What don’t you like about Music?
What do you like about P.E.?
What don’t you like about P.E.?
What do you like about Computers?
What don’t you like about Computers?
What do you like about Art?
What don’t you like about Art?
Probes
FIGURE 3.2 (Continued)
Chapter | 3 Instruments for Evaluating Social Validity
obtained could be useful in identifying certain aspects of treatment that may prove effective, but possibly even more importantly the interview may allow for some of the likes and dislikes of the student to be incorporated into a treatment. By including definite preferences of the student into the treatment, it may be possible to increase the social validity of the treatment components as evaluated by the student. Obtaining similar information from treatment mediators may also prove beneficial for developing treatment procedures with high acceptability.
DIRECT OBSERVATION Direct observation is another method for assessing social validity which can provide high-quality information but may require a substantial amount of time and effort. Observing the behaviors of individuals receiving treatment allows for numerous types of comparisons to be conducted. Comparison may be made to others functioning in the same or similar environments. Direct observations can be made of the individual targeted for treatment and similar observations made of others functioning in the same environment. These observations can then be used to compare how closely the individual targeted for treatment resembles others considered to be functioning well in the same type of environment.
PERFORMANCE CRITERIA In addition, performance criteria can be established prior to implementation of a treatment and resulting behavior can be compared to the pre-established criteria. Fawcett (1991) recommended using proficiency criteria at three different levels for comparison of posttreatment behavior performance. The three levels consisted of ideal performance levels, normative performance levels, and deficient performance levels. Ideal performance levels would be representative of consistent behavior that would be highly efficient and further improvement would not be necessary. Normative performance levels would consist of behavior that was efficient and comparable to that of others functioning in the same environment. Deficient performance levels would lack efficiency and would require further improvement before the individual could function effectively within the environment. This type of comparison to specific criteria provides a highly structured method for determining the social importance of treatment effects.
47
48
Chapter | 3 Instruments for Evaluating Social Validity
FORMAL TREATMENT ACCEPTABILITY MEASUREMENT INSTRUMENTS The model for measurement of treatment acceptability was established by Kazdin (1980), with the development of the Treatment Evaluation Inventory (TEI). Kazdin developed the TEI as a Likert-type rating scale which questioned raters about several factors related to a specific treatment. While the TEI has served as the standard format for obtaining formal measures of treatment acceptability, several other informal measures have also been described in the literature. Most of the informal measures of treatment acceptability have involved asking various questions about how a consumer or someone close to the consumer liked the overall treatment or certain aspects of the treatment. Some of these measures may have included open-ended questions; others asked the raters to rank the usefulness, effectiveness, ease of implementation of a treatment, or other factors using self-designed scales or sets of questions that appeared important to the specific treatment. It is difficult to determine how reliable or valid any of these informal measures may be, because in many instances the questions are not readily available for replication in other studies and no validity data are provided. While these informal measures of treatment acceptability appear to have value for the specific purpose for which they were designed, they may not be highly beneficial in developing a comprehensive understanding of treatment acceptability that can be generalized. It appears that the most pertinent information on treatment acceptability that has accrued from the literature has stemmed from the use of more formal measures of treatment acceptability that could be readily obtained and scrutinized as to their reliability and validity. These more formal measures of treatment acceptability will now be described. As stated earlier, the most frequently used formal measure of treatment acceptability appears to be the TEI. This formal measure, along with the Intervention Rating Profile (IRP; Witt & Elliot, 1985), have been the most frequently used and modified measures of treatment acceptability within the behavior intervention literature. These measures have been modified and revised in several ways by researchers in order to improve reliability and validity, in order to reduce the administration time, and to make the instruments more useful for specific target populations. The following provides a review of some of the original formalized instruments
Chapter | 3 Instruments for Evaluating Social Validity
developed to measure treatment acceptability and the modifications, revisions, and extensions of these instruments.
Treatment Evaluation Inventory The TEI was originally designed to assess treatments recommended for children with behavior disorders (Kazdin, 1980). The TEI was developed in an effort to design a measure to evaluate the acceptability of a treatment independently of the efficacy of the treatment and which could be considered to have high validity. Kazdin originally procured 45 items that appeared to be relevant to the evaluation of a treatment by a client. He then narrowed the items to 16 which were most relevant to the treatment of children and to punishment procedures, which was the topic of his original research. These 16 items were validated against items taken from the Evaluative dimension of the Semantic Differential (Osgood, Suci, & Tannenbaum, 1957) which were considered to be related to the acceptability of treatments. A principal component analysis of the TEI rotated to varimax criterion resulted in item loadings from 0.61 to 0.95 on a unitary factor accounting for 51.4% of the variance. All but one item did not load highly on other factors, and this item was removed, resulting in a 15-item measure (see Figure 3.3). The TEI consists of items which ask direct questions regarding specific treatment procedures. Ratings are made on a seven-point Likert-type scale, with descriptive anchor points that vary depending on the question. Total scores are obtained by summing all items, with higher summed total scores indicating greater levels of treatment acceptability. The TEI is considered to be the most frequently used measure of treatment acceptability, but it has been used very little in actual clinical intervention studies to measure acceptability of behavioral interventions with children and families. Several modifications to the TEI have been developed and will be described next.
Treatment Evaluation Inventory-Short Form The TEI was abridged and published as the Treatment Evaluation Inventory-Short Form (TEI-SF; Kelley, Heffer, Gresham, & Elliot, 1989). This revised form was used to measure acceptability of treatments for children with behavior disorders. The original TEI was reduced by six items, resulting in a nine-item, shorter version of the original instrument. The rationale for shortening the original instrument
49
50
Chapter | 3 Instruments for Evaluating Social Validity
The Treatment Evaluation Inventory Please complete the items listed below. The items should be completed by placing a checkmark on the line under the question that best indicates how you feel about the treatment. Please read the items verycarefully because a checkmark accidentally placed on one space rather than another may not represent the meaning you intend. 1. How acceptable do you find this treatment to be for the child’s problem behavior? _______ _______ _______ _______ _______ _______ _______ Not at all Moderately Very acceptable acceptable acceptable 2. How willing would you be to carry out this procedure yourself if you had to change the child’s problems? _______ Not at all willing
_______
_______
_______ Moderately willing
_______
_______
_______ Very willing
3. How suitable is this procedure for children who might have other behavioral problems than those described for this child? _______ _______ _______ _______ _______ _______ _______ Not at all Moderately Very suitable suitable suitable 4. If children had to be assigned to treatment without their consent, how bad would it be to give them this treatment? _______ Very bad
_______
_______
_______ Moderately
_______
_______
_______ Not bad at all
_______
_______
_______ Not cruel at all
5. How cruel or unfair do you find this treatment? _______ Very cruel
_______
_______
_______ Moderately cruel
6. Would it be acceptable to apply this procedure to institutionalized children, the mentally retarded, or other individuals who are not given an opportunity to choose treatment for themselves? _______ _______ _______ _______ _______ _______ _______ Not at all Moderately Very acceptable to acceptable acceptable to apply this apply this procedure procedure 7. How consistent is this treatment with common sense or everyday notions about what treatment should be? _______ _______ _______ _______ _______ _______ _______ Very different Moderately Very or inconsistent consistent consistent with everyday notions
FIGURE 3.3 Treatment Evaluation Inventory. From Kazdin, A. E. (1980). Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis, 13, 259–273. Reprinted with permission.
was considered to be a need to expedite the time for completing acceptability ratings within the time constraints of typical clinical practice. The items are rated using a five-point Likert-type scale with fixed, anchored points. The descriptions on the Likert-type scale range from 1 (strongly disagree) to 5 (strongly agree). Total scores are obtained
51
Chapter | 3 Instruments for Evaluating Social Validity
8. To what extent does this procedure treat the child humanely? _______ _______ _______ _______ _______ Does not treat Treats them humanely at moderately all humanely
_______
_______ Treats them very humanely
9. To what extent do you think there might be risks in undergoing this kind of treatment? _______ _______ _______ _______ _______ _______ _______ Lots of risks Some risks are No risks are are likely likely likely 10. How much do you like the procedures used in this treatment? _______ _______ _______ _______ _______ Do not like Moderately them at all like them
_______
_______ Like them very much
_______
_______ Very effective
11. How effective is this treatment likely to be? _______ Not at all effective
_______
_______
_______ Moderately effective
_______
12. How likely is this treatment to make permanent improvements in the child? _______ _______ _______ _______ _______ _______ _______ Unlikely Moderately Very likely 13. To what extent are undesirable side effects likely to result from this treatment? _______ _______ _______ _______ _______ _______ _______ Many Some No undesirable undesirable undesirable side effects side effects side effects likely likely likely 14. How much discomfort is the child likely to experience during the course of treatment? _______ Very much discomfort
_______
_______
_______ Moderate discomfort
_______
15. Overall, what is your general reaction to this form of treatment? _______ _______ _______ _______ _______ Very negative Ambivalent
_______
_______ No discomfort at all
_______
_______ Very positive
FIGURE 3.3 (Continued)
by summing all items, with higher summed scores indicating greater levels of acceptability. The internal consistency of this instrument was reported to be 0.85 (Kelley et al.).
Treatment Acceptability Rating Form The Treatment Acceptability Rating Form (TARF; Reimers & Wacker, 1988) was devised from Kazdin’s (1980) original measure of treatment acceptability to measure parent’s acceptability of treatments devised within a clinical setting. It consists of 15 items on a seven-point Likert-type scale. The method for obtaining total scores is not described (Reimers, Wacker, & Cooper, 1991). The internal consistency of this instrument was reported to range from 0.80 to 0.91. The TARF was
52
Chapter | 3 Instruments for Evaluating Social Validity
created to improve upon the TEI by including factors of effectiveness and cost of treatment. The TARF was also developed to provide an acceptability instrument more suitable for clinical populations.
Treatment Acceptability Rating Form Revised The Treatment Acceptability Rating Form Revised (TARF-R; Reimers et al., 1991) is a revised and expanded form of the TARF used to measure acceptability of treatments devised within clinical settings (see Figure 3.4). The TARF-R consists of 20 questions with 17 questions pertaining to treatment acceptability, in which two questions address problem severity, and one question that addresses understanding of the intervention. The questions are rated on a six-point Likert-type scale with varying anchor point descriptors for each item. Total scores are obtained by summing all items, with higher summed scores representing greater levels of acceptability. The internal consistency of this instrument was reported to be 0.92.
Intervention Rating Profile The Intervention Rating Profile (IRP) was developed to extend research in treatment acceptability to educational treatments and to make practitioners more aware of interventions considered acceptable by teachers. The IRP consists of 20 items which are rated on a six-point Likert-type scale (see Figure 3.5). The Likert-type rating scale ranges from 1 (strongly disagree) to 6 (strongly agree). Total scores are obtained by summing all items, with higher summed scores indicating greater levels of acceptability. A total score of 70 would represent a moderate level of acceptability. Principal component analysis with varimax rotation revealed a primary factor accounting for 41% of the variance and four secondary factors (Tarnowski & Simonian, 1992). The internal consistency of this instrument was reported to be 0.89.
Intervention Rating Profile-15 Modifications were made to the IRP to shorten the instrument and to increase item loading on a single factor. This modified version of the IRP was published as the Intervention Rating Profile-15 (see Figure 3.6) and used to measure educational interventions (IRP-15; Martens, Witt,
53
Chapter | 3 Instruments for Evaluating Social Validity
TREATMENT ACCEPTABILITY RATING FORM—REVISED (TARF-R) Please complete the items listed below. The items should be completed by placing a check mark on the line under the question that best indicates how you feel about the psychologist’s treatment recommendations. 1. How clear is your understanding of this treatment? —— Not at all clear
——
——
—— Neutral
——
—— Very clear
2. How acceptable do you find the treatment to be regarding your concerns about your child? —— Not at all acceptable
——
——
—— Neutral
——
—— Very acceptable
——
—— Very willing
3. How willing are you to carry out this treatment? —— Not at all willing
——
——
—— Neutral
4. Given your child’s behavioral problems, how reasonable do you find the treatment to be? —— —— —— —— —— —— Not at all Neutral Very reasonable reasonable 5. How costly will it be to carry out this treatment? —— —— —— —— Not at all Neutral costly
——
—— Very costly
6. To what extent do you think there ight be disadvantages in following this treatment? —— Not at all likely
——
——
—— Neutral
——
—— Many are likely
7. How likely is this treatment to make permanent improvements in your child’s behavior? —— —— —— —— —— —— Unlikely Neutral Very likely 8. How much time will be needed each day for you to carry out this treatment? —— —— Little time will be needed
——
—— Neutral
——
—— Much time will be needed
——
—— Very confident
9. How confident are you that the treatment will be effective? —— Not at all confident
——
——
—— Neutral
10. Compared to other children with behaviorial difficulties, how serious are your child’s problem? —— —— —— —— —— —— Not at all Neutral Very serious serious
FIGURE 3.4
Treatment Acceptability Rating Form-Revised.
From Reimers, T. M., & Wacker, D. P. (1988). Parents’ ratings of behavioral treatment recommendations made in an outpatient clinic: A preliminary analysis of the influence of treatment effectiveness. Behavior Disorders, 14, 7–15. Reprinted with permission.
Elliot, & Darveaux, 1985). The modifications to improve the internal consistency of the instrument included generating eight new items and removing other items, resulting in a total of 15 items. The items are rated using a six-point Likert-type rating scale with ranges from 1
54
Chapter | 3 Instruments for Evaluating Social Validity
11. How disruptive will it be to the family (in general) to carry out this treatment? —— Not at all disruptive
——
——
—— Neutral
——
—— Very disruptive
——
—— Very affordable
——
—— Very affordable
12. How effective is this treatment likely to be for your child? —— Not at all effective
——
——
—— Neutral
13. How affordable is this treatment for your family? —— Not at all affordable
——
——
—— Neutral
14. How much do you like the procedures used in the proposed treatment? —— Do not like them at all
——
——
—— Neutral
——
—— Like them very much
15. How willing will other family members be to help carry out this treatment? —— Not at all willing
——
——
—— Neutral
——
—— Very willing
16. To what extent are undesirable side-effects likely to result from this treatment? —— —— No side-effects are likely
——
—— Neutral
——
—— Many side-effects are likely
17. How much discomfort is your child likely to experience during the course of this treatment? —— —— No discomfort at all
——
—— Neutral
——
—— Very much discomfort
——
—— Very severe
18. How severe are your child’s behavioral difficulties? —— Not at all severe
——
——
—— Neutral
19. How willing would you be to change your family routine to carry out this treatment? —— Not at all willing
——
——
—— Neutral
——
—— Very willing
20. How well will carrying out this treatment fit into the family routine? —— Not at all well
——
——
—— Neutral
——
—— Very well
21. To what degree are your child’s behavioral problems of concern to you? —— No concern at all
——
——
—— Neutral
——
—— Great concern
FIGURE 3.4 (Continued)
(strongly disagree) to 6 (strongly agree). Total scores are obtained by summing all items, with higher summed scores indicating greater levels of acceptability. A moderate level of acceptability would require a total summed score of 52.5. The internal consistency of this instrument was reported to be 0.98.
55
Chapter | 3 Instruments for Evaluating Social Validity
Strongly Disagree
Disagree
Slightly Disagree
Slightly Agree
Agree
Strongly Agree
The purpose of this questionnaire is to obtain information that will aid in selection of classroom interventions. These interventions will be used by teachers of children with behavior problems. Please circle the number which best describes your agreement or disagreement with each statement.
1. Teachers are likely to use this intervention because it requires few technical skills.
1
2
3
4
5
6
2. Teachers are likely to use this intervention because it requires little training to implement effectively.
1
2
3
4
5
6
3. Most teachers would find the intervention suitable for the behavior problem described.
1
2
3
4
5
6
4. Most teachers would find this intervention appropriate for behavior problems in addition to the one described.
1
2
3
4
5
6
5. The child’sbehaviorproblemis severe enough to warrant use of this intervention.
1
2
3
4
5
6
6. This intervention would be appropriate for use before making a referral.
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
9. This intervention is practical in the amount of time required for contact with school staff.
1
2
3
4
5
6
10. This intervention is practical in the amount of time required for recordkeeping.
1
2
3
4
5
6
11. This intervention is practical in the amount of out-of-schooltimerequired for implementation.
1
2
3
4
5
6
12. This intervention would not be disruptive to other students.
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
15. This would be an acceptable intervention for the child’s problem behavior.
1
2
3
4
5
6
16. This intervention would not result in negative side effects for the child.
1
2
3
4
5
6
17. This interventionwouldnotresult in risk to the child.
1
2
3
4
5
6
18. This intervention would not be considered a “last resort.”
1
2
3
4
5
6
19. Overall,this intervention would be beneficial for the child.
1
2
3
4
5
6
20. I would be willing to use this intervention in the classroom setting.
1
2
3
4
5
6
7. This intervention would not be difficult to implement in a classroom with 30 other students. 8. This intervention is practical in the amount of timer equired for parent contact.
13. It would not be difficult to use this intervention and still meet the needs of other students. 14. This intervention should prove effective inchanging the child’s problem behavior.
FIGURE 3.5
Intervention Rating Profile.
From Witt, J. C., & Martens, B. K. (1983). Assessing the acceptability of behavioral interventions used in classrooms. Psychology in the Schools, 20, 510–517. Reprinted with permission.
Chapter | 3 Instruments for Evaluating Social Validity
Slightly Agree
Agree
Strongly Agree
7. I would be willing to use this intervention in the classroom setting. 8. This intervention would not result in negative side-effects for the child. 9. This intervention would be appropriate for a variety of children. 10. This intervention is consistent with those I have used in classroom settings. 11. This intervention was a fair way to handle the child’s problem behavior. 12. This intervention is reasonable for the behavior problem described. 13. I liked the procedures used in this intervention. 14. This intervention was a good way to handle this child’s behavior problem. 15. Overall, this intervention would be beneficial for the child.
Slightly Disagree
1. This would be an acceptable intervention for the child’s problem behavior. 2. Most teachers would find this intervention appropriate for behavior problems in addition to the one described. 3. This intervention should prove effective in changing the child’s problem behavior. 4. I would suggest the use of this intervention to other teachers. 5. The child’s behavior problem is severe enough to warrant use of this intervention. 6. Most teachers would find this intervention suitable for the behavior problem described.
Disagree
The purpose of this questionnaire is to obtain information that will aid in the selection of classroom interventions. These interventions will be used by teachers of children with behavior problems. Please circle the number which best describes your agreement or disagreement with each statement. Strongly Disagree
56
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
1
2
3
4
5
6
FIGURE 3.6 Intervention Rating Profile-15 (IRP-15). From Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. R. Kratochwill (Ed.), Advances in school psychology (4th ed.) (pp. 251–288). Mahwah, NJ: Erlbaum. Reprinted with permission.
Children’s Intervention Rating Profile The Children’s Intervention Rating Profile (CIRP; Witt & Elliot, 1985) was a modification of the IRP designed to assess children’s views of the treatment acceptability of educational interventions. The instrument consists of seven questions related to the perceived fairness and
57
Chapter | 3 Instruments for Evaluating Social Validity
I agree
I do not agree
1. The method used to deal with the behavior problem was fair.
+ ---- + ---- + ---- + ---- + ---- +
2. This child’s teacher was too harsh on him.
+ ---- + ---- + ---- + ---- + ---- +
3. The method used to deal with the behavior may cause problems with this child’s friend.
+ ---- + ---- + ---- + ---- + ---- +
4. There are better ways to handle this child’s problem than the one described here.
+ ---- + ---- + ---- + ---- + ---- +
5. The method used by this teacher would be a good one to use with other children.
+ ---- + ---- + ---- + ---- + ---- +
6. I like the method used for this child’s behavior problem.
+ ---- + ---- + ---- + ---- + ---- +
7. I think that the method used for this problem would help this child do better in school.
+ ---- + ---- + ---- + ---- + ---- +
FIGURE 3.7
Children’s Intervention Rating Profile (CIRP).
From Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. R. Kratochwill (Ed.), Advances in school psychology (4th ed.) (pp. 251–288). Mahwah, NJ: Erlbaum. Reprinted with permission.
expected effectiveness of a treatment (see Figure 3.7). The questions on this instrument were written at a fifth-grade reading level. The items are rated using a six-point Likert-type rating scale. Total scores are obtained by summing all items, with higher summed scores representing greater levels of acceptability. The internal consistency of this instrument ranged from 0.75 to 0.89.
Behavior Intervention Rating Scale The Behavior Intervention Rating Scale (BIRS; Von Brock & Elliott, 1987) was developed to create a new instrument for measuring treatment acceptability in school settings. The BIRS is a modification of the IRP-15 which added nine items, for a total of 24 questions included on the instrument (see Figure 3.8). The items are rated using a six-point Likert-type rating scale which ranges from 1 (strongly disagree) to 6 (strongly agree). Total scores are obtained by summing all items, with higher summed scores indicating greater levels of acceptability. The internal consistency of this instrument was reported to be 0.97.
Behavior Intervention RatingScale—TeacherVersion To be completed by the Teacher: Please evaluate the intervention by circling the number which best describes your agreement or disagreement with each statement. Please circle only one number for each item. Use 1 indicating you strongly disagree with the statement, 3 indicating a neutral response, and 6 indicating you strongly agree with the statement. Strongly Disagree
Strongly Agree
Neutral
1. This would be an acceptable intervention for the child's problem behavior.
1
2
3
4
5
2. Most teachers would find this intervention appropriate for challenging behaviors.
1
2
3
4
5
3. The intervention should prove effective in changing the child's problem behavior.
1
2
3
4
5
6
4. I would suggest the use of this intervention to other teachers.
1
2
3
4
5
6
5. The child’s behavior problem is severe enough to warrant the use of this intervention.
1
2
3
4
5
6
6. Most teachers would find this intervention suitable for the behavior problem described.
1
2
3
4
5
6
7. I would be willing to use this [intervention] in the classroom setting.
1
2
3
4
5
6
8. The intervention would not result in negative side effects for the child.
1
2
3
4
5
6
9. The intervention would bean appropriate intervention for a variety of children.
1
2
3
4
5
6
10. The intervention is consistent with those I have used in classroom settings.
1
2
3
4
5
6
11. The intervention was afair way to handle the child's problem behavior.
1
2
3
4
5
6
12. The intervention is reasonable for the behavior problem described.
1
2
3
4
5
6
13. I like the procedures used in the intervention.
1
2
3
4
5
6
14. This intervention was a good way to handle the child's challenging behaviors.
1
2
3
4
5
6
15. Overall, the intervention would be beneficial for the child.
1
2
3
4
5
6
16. The intervention would quickly improve the child's behavior.
1
2
3
4
5
6
17. The intervention would produce a lasting improvement in the child's behavior.
1
2
3
4
5
6
18. The intervention would improve the child's behavior to the point that it would not noticeably deviate from other classmates' behavior.
1
2
3
4
5
6
19. Soon afterusing the intervention, the teacher would notice a positive change in the problem behavior.
1
2
3
4
5
6
20. The child's behavior will remain at an improved level even after the intervention is discontinued.
1
2
3
4
5
6
21. Using the intervention should not only improve the child's behavior in the classroom, but also in other settings (e.g., other classrooms, home).
1
2
3
4
5
6
22. When comparing this child with a well-behaved peer before and after use of the intervention, the child's and the peer's behavior would bemore alike after using the intervention.
1
2
3
4
5
6
23. The intervention should produce enough improvement in the child's behavior so the behavior is no longer a problem in the classroom.
1
2
3
4
5
6
24. Other behaviors related to the problem behavior also are likely to be improved by the intervention.
1
2
3
4
5
6
6 6
FIGURE 3.8 Behavior Intervention Rating Scale (BIRS). From Von Brock, M., & Elliott, S. (1987). Influence of treatment effectiveness information on the acceptability of classroom interventions. Journal of School Psychology, 25, 131–144. Reprinted with permission.
59
Chapter | 3 Instruments for Evaluating Social Validity
Intervention-Process Rating Scale (IPRS) This instrument was based upon both the TEI and the IRP (Kutsick, Gutkin, & Witt, 1991). It was designed to measure treatment acceptability, but appeared to be more directed toward measuring the acceptability of processes related to service delivery, such as consultation approaches. The instrument includes 11 items which are rated using a six-point Likert-type scale. The scale reportedly has a Cronbach alpha coefficient of 0.95. Examples of some the items on the scale include “Overall, the method by which this intervention was planned was a good one” and “Most teachers would implement this strategy exactly as planned.”
Abbreviated Acceptability Rating Profile An additional abbreviated form of the IRP-15 was developed to measure educational interventions and published as the Abbreviated Acceptability Rating Profile (AARP; Tarnowski & Simonian, 1992). Tarnowski and Simonian developed the AARP through modification of the IRP-15 by eliminating seven items and maintaining eight items that were reworded to improve readability (see Figure 3.9). The items are rated using a six-point Likert-type rating scale which ranges from 1 Strongly Slightly Slightly Disagree Disagree Disagree Agree 1. This is an acceptable treatment for the child’s behavior. 2. The treatment should be effective in changing the child’s behavior.
Agree
Strongly Agree
1
2
3
4
5
6
1
2
3
4
5
6
3. The child’s behavior is severe enough to justify the use of this treatment.
1
2
3
4
5
6
4. I would be willing to use this treatment with my child.
1
2
3
4
5
6
5. This treatment would not have bad side effects for the child.
1
2
3
4
5
6
6. I liked this treatment.
1
2
3
4
5
6
7. The treatment was a good way to handle the child’s problem.
1
2
3
4
5
6
8. Overall, the treatment would help the child.
1
2
3
4
5
6
FIGURE 3.9
Abbreviated Acceptability Rating Profile (AARP).
From Tarnowski, K. J., & Simonian, S. J. (1992). Assessing treatment acceptance: The abbreviated acceptability rating profile. Journal of Behavior Therapy & Experimental Psychiatry, 23, 101–106. Reprinted with permission.
60
Chapter | 3 Instruments for Evaluating Social Validity
(strongly disagree) to 6 (strongly agree). Total scores are obtained by summing all items, with higher summed scores indicating greater levels of acceptability. The internal consistency of this instrument was reported to be 0.98. Tarnowski and Simonian (1992) found the AARP to be a reliable and valid instrument for measuring treatment acceptability. In comparison to the IRP-15, Tarnowski and Simonian indicated that the AARP was more simplistic, easier to read, and required less administration time. The psychometric properties of the AARP were initially evaluated using a sample of 60 parents seen for routine pediatric outpatient visits at a large urban hospital. The initial sample, when subjected to a principal component analysis, resulted in all items loading on a unitary factor (Acceptability) that accounted for 84.9% of the variance, with item loadings ranging from 0.89 to 0.96. The initial sample was crossvalidated in a second independent sample of 80 mothers by subjecting the data to identical analysis; this test resulted in item loadings ranging from 0.89 to 0.98 on a single factor that accounted for 90.3% of the variance. The AARP resulted in reduced completion time to approximately one-half the ten minutes required to complete the IRP-15. Readability analyses conducted using the Harris-Jacobson Wide Range Readability Formula (Harris & Jacobson, 1982) resulted in indices of 5.0 for the AARP, compared to 7.9 for the IRP-15. Based on their findings, Kelley et al. (1989) considered the AARP to be an improvement over the IRP15 as well as other measures of acceptability.
COMPARISON OF TREATMENT ACCEPTABILITY MEASURES Finn and Sladeczek (2001) critiqued nine treatment acceptability measures which were the: TEI, TEI-SF, TARF, TARF-R, IRP, IRP for teachers, AARP, CIRP, and BIRS. These nine measures were selected after an extensive database search of studies investigating behavioral intervention acceptability. The evaluation covered eight areas of treatment acceptability as follows: (1) definition of treatment acceptability; (2) content and purpose; (3) test reliability; (4) test validity; (5) statistical analysis; (6) sample characteristics; (7) scoring procedures; and (8) uses of the measure in research and practice. The evaluation indicated no single measure of treatment acceptability to be more comprehensive than
Chapter | 3 Instruments for Evaluating Social Validity
any other. Table 3.2 provides a summary of some of the characteristics of treatment acceptability instruments.
PROBLEMS RELATED TO SOCIAL VALIDITY MEASUREMENT As noted by Baer, Wolf, and Risley (1987), social validity findings may be subject to frequent false positive findings, because of the rudimentary ways in which social validity is measured. In other words, treatments may be frequently found to have social validity because the instruments that are being used are not efficient at accurately assessing the construct, especially with varying population in various contexts. It also appears that many of the informal measures of treatment acceptability have been administered posttreatment and may therefore be influenced by attempts to avoid disagreeing with the researcher, lack of familiarity with other treatment options, and fear of the removal of a successful treatment. Hawkins (1979) provided some criteria for determining the validity of an instrument for measuring consumers’ satisfaction with services: If their verbal reports on a consumer satisfaction questionnaire correlate with such measures as their verbal reports to others on the same topic, their referral of the learner for similar help again, their hiring the learner, or their admitting the learner to more advanced programs, the questionnaire is shown to be serving its purpose. Such validity evidence on consumer satisfaction measures does not seem to be available yet (p. 510).
METHODS FOR INCREASING THE USE OF TREATMENT ACCEPTABILITY INSTRUMENTS With the improved instruments for assessing treatment acceptability and the advances in treatment acceptability research, professionals have a well-developed basis from which to make treatment decisions and conduct further research. These advances should make it easier for researchers and practitioners to incorporate more formalized instruments into daily practice. It might also be presumed that the evaluation of specific treatments, specific populations of clients, and raters from specific areas of the country with varying backgrounds should become more prevalent. Even with the advances made toward more reliable, more valid, more expedient, and more population-specific treatment
61
62
TABLE 3.2
Characteristics of Treatment Acceptability Instruments
Instrument Name
Type of Instrument
Validity
Reliability
Details
Treatment Acceptability Inventory (TEI; Kazdin, 1980)
Consists of 15 items; uses a seven-point Likert-type scale; descriptive anchor points vary depending on question
Item loadings from 0.61 Internal consistency ranged from 0.35 to to 0.95 on a unitary 0.96 factor accounting for 51.4% of variance
Treatment Acceptability Rating Form (TARF; Reimers & Wacker, 1988)
Consists of 15 items; uses a seven-point Likert-type scale; varying descriptive anchor points
Internal consistency ranged from 0.80 to 0.91
Developed from the TEI for use with parents; incorporated factors of effectiveness and cost of treatment
Treatment Evaluation Inventory-Short Form (TEISF; Kelley et al., 1989)
Consists of nine items rated using a five-point Likert-type scale with fixed anchor points
Internal consistency of 0.85
Revised version of the TEI; requires less time to complete than the TEI
Treatment Acceptability Rating Form-Revised (TARFR; Reimers, Wacker, & Cooper, 1991)
Consists of 20 items rated using a seven-point Likert-type scale; varying descriptive anchor points for each item
Internal consistency of 0.92
Revised version of the TARF; includes items addressing problem severity and understanding of treatment
Considered the first and one of the most frequently used instruments Chapter | 3 Instruments for Evaluating Social Validity
Consists of 20 items evaluated by a six-point Likert-type scale
One primary factor (41% of the variance) and four secondary factors
Internal consistency of 0.89
Designed to evaluate educational treatments
Intervention Rating Profile15 (IRP-15; Martens et al., 1985)
Consists of 15 items; uses a six-point Likerttype scale
Principal component analysis resulted in a unitary factor
Internal consistency of 0.98
Modified the IRP to increase internal consistency
Children’s Intervention Rating Profile (CIRP; Witt & Elliot, 1985)
Consists of seven questions evaluated using a seven-point Likert-type scale
Principal component analysis resulted in a unitary factor
Internal consistency ranged from 0.75 to 0.89
Modified the IRP for children; items written at fifth-grade reading level
Behavior Intervention Rating Scale (Von Brock & Elliot, 1987)
Consists of 24 items; uses a six-point Likerttype scale
Internal consistency of 0.97
Modified version of the IRP-15
Abbreviated Acceptability Rating Profile (AARP; Tarnowski & Simonian, 1992)
Consists of eight items rated using a six-point Likert-type scale
Internal consistency of 0.98
Modified the IRP-15 to reduce completion time and improve readability
Unitary factor accounted for 84.9% of the variance with item loadings ranging from 0.89 to 0.96
Chapter | 3 Instruments for Evaluating Social Validity
Intervention Rating Profile (IRP; Witt & Elliot, 1985)
From Carter, S. L. (2007). Review of recent treatment acceptability research. Education and Training in Developmental Disabilities, 42, 301–316. Reprinted with permission.
63
64
Chapter | 3 Instruments for Evaluating Social Validity
TABLE 3.3 Ways to Increase the Use of Treatment Acceptability Instruments Methods to Consider for Increasing the Use of Treatment Acceptability Instruments 1. 2. 3. 4. 5.
Increase availability of instruments and other measures Increase consultant awareness of instruments Increase competency in the administration and scoring of instruments Increase knowledge of potential uses of acceptability information Re-emphasize the importance of social validity
acceptability instruments, there does not appear to be an increase in the use of these instruments. One potential reason for the minimal usage of treatment acceptability instruments is their inability to detect small changes among consumers. Elliot, Busse, and Gresham (1993) pointed out that rating scales are not sensitive enough to detect small changes in behavior. While it should be noted that treatment acceptability instruments are limited in their capacity to detect small changes in consumer acceptance, they are one of the most easily administered methods for gathering information quickly. The advantage of using treatment acceptability instruments is that they may be supportive of and supported by other information. In combination with other information, treatment acceptability instrument data may provide substantial evidence of social validity. As discussed previously, there appears to be a decreasing trend in overall social validity research. Table 3.3 provides some recommendations for increasing the use of social validity assessment measures and each of these recommendations are discussed below.
Increase Availability of Instruments and Other Measures One strategy for overcoming the decreasing trend in social validity research could be to make well-developed instruments for assessing treatment acceptability more readily available. In many research articles the actual instrument is not printed, or only a select number of items from the instrument are printed. In many cases it is difficult to obtain the actual instrument, as it may require writing to the author or obtaining the instrument from another source such as a book. These treatment
Chapter | 3 Instruments for Evaluating Social Validity
acceptability instruments might be made more accessible by making them available for downloading or by offering ready-to-use hard copies of the instruments for a fee.
Increase Consultant Awareness of Instruments Making the instruments more available does not necessarily guarantee that they will be used, which is why it may be dependent upon training programs in psychology, education, and applied behavior analysis to emphasize awareness of these instruments, the importance of these instruments, and competency training in the administration and scoring of these instruments. Practitioners who have instruments readily available for use, are aware of the different types of instruments, and have an understanding of the importance of these instruments may be more likely to use the instruments.
Increase Competency in the Administration and Scoring of Instruments The focus of most training programs in applied behavior analysis is on direct observation of behavior and use of single-subject research designs. This is the foundation upon which behavior analysis was developed. In contrast, the development of instruments for measuring social validity, such as the TEI or IRP, involved indirect, subjective measurement of behavior and group research design methods. This contrast may contribute to a lack of familiarity with using some of the instruments for collecting information on social validity. This lack of familiarity could be remedied by ensuring that training programs in applied behavior analysis and continuing education courses offer preparation on at least how to administer and accurately score various social validity instruments. Consideration might also be given to more advanced training on how to use group research designs and advanced statistical procedures, in order to develop and validate new instruments for collecting social validity data. Advanced training might also involve evaluating how measurement of other conceptual variables might be correlated with social validity and could potentially increase the evidence to support social validity. In addition, university training programs and continuing education courses should focus on how to use the information obtained from the instruments for both specific and more generalized purposes.
65
66
Chapter | 3 Instruments for Evaluating Social Validity
Increase Knowledge of Potential Uses of Acceptability Information Considering the limited reported use of social validity measures, practitioners who do collect data on social validity may receive little reinforcement of this practice. Consultants may need to be provided with new skills for using social validity information, so that they do gain some reinforcement for doing so. Consultants who regularly collect social validity data may use the information to improve the services that they offer and thus become more valuable to consumers. This can be advantageous to the consultant in that consumers who find services highly valuable frequently spread this news to other consumers who may seek out the services of the consultant, thus resulting in greater earning potential and job security. Consultants may also find that treatment acceptability instruments may be useful for avoiding disputes regarding the acceptability of specific treatments, or may help to settle disputes regarding the acceptability of treatments. Miltenberger (1990) recommended that professionals could collect representative acceptability ratings from the area or school in which they work for a number of different treatments and problem behaviors. The professional could then use this information when making treatment decisions. The only way to identify the numerous potential uses for social validity data is to actually collect the data and examine the potential ways that the data could be used. With limited availability of social validity data, there is also limited opportunity to use the data in decision making, planning for services, developing treatments, and the like. Making social validity data more available for consumers, consultants, and other segments of society may reduce the number of missed opportunities where this type of data could have been helpful, and may increase the value of such data.
Re-emphasize the Importance of Social Validity Based on the decreasing trends in the dissemination of social validity data, it could be said that social validity is not considered as important as it may have once been. Most behavior analysts, if asked, would likely agree that social validity is a highly important part of practice within applied behavior analysis. Increased promotion of the need to include
Chapter | 3 Instruments for Evaluating Social Validity
data on social validity could lead to a renewed interest and potentially the development of improved methods for measuring social validity. The positive behavior support movement has done an excellent job at promoting the need to include social validity data into the dissemination of research. The research on positive behavior support appears to show something of a renewed interest in social validity, and some of the techniques being used offer a fresh perspective toward measuring social validity. Many different combinations of data have been used to offer evidence of social validity. This type of renewed interest and effort should be expanded within the practice of applied behavior analysis.
67
This page intentionally left blank
Chapter 4
Research on Social Validity Early Protocols for Measuring Social Validity Methodological Variations Literature Reviews of Treatment Acceptability Research Findings TEI and TEI-SF IRP and IRP-15
Teacher Acceptability Measure (TAM) CIRP BIRS AARP TARF-R Summary
The three dimensions of social validity described by Wolf (1978), which include the significance of the goals of treatment, the appropriateness of the treatment procedures, and the importance of the effects of treatment, have been researched using varying methods and to varying degrees. The most thoroughly researched dimension of social validity appears to have been the social appropriateness of the treatment procedures. This dimension has been termed treatment acceptability by Kazdin (1980a). The significance of the goals of treatment appears to have received the least attention in the research literature, while the social importance of the effects of treatment has received a good deal of attention depending upon the manner in which this dimension is defined. The primary focus of this chapter will be to outline the research which has been conducted regarding the social appropriateness of treatment procedures, otherwise known as treatment acceptability. Research regarding the other two dimensions of social validity (the significance of the goals of treatment and the social importance of the effects of treatment) will also be described in less detail.
EARLY PROTOCOLS FOR MEASURING SOCIAL VALIDITY Treatment acceptability, which focuses on the social appropriateness of treatment procedures, has been one of the most frequently investigated Copyright © 2010 Elsevier Inc. All rights reserved
69
70
Chapter | 4 Research on Social Validity
components of social validity. Kazdin (1980a) developed the initial foundations for this type of research in an attempt to gain insight into some of the variables which influenced how acceptable a treatment would be rated. His research model has served as the primary method for conducting research in this area and proven beneficial in developing a broad understanding of a number of variables which would have otherwise taken a much longer period of time to understand. Prior to this study, treatment acceptability had been indirectly evaluated through anecdotal interviews and questionnaires. But none of these reports provided any measure of validity. The initial research conducted by Kazdin (1980a), which was the first to validate a measure of treatment acceptability, will be described in detail and the methodology—which has proven to be highly influential on continuing treatment acceptability research—will be outlined. Kazdin (1980a), in developing and designing the TEI, presented case descriptions and treatments for a child whose behavior warranted treatment to 88 undergraduates recruited from introductory psychology courses. The participants listened to a case description and four different treatments presented from cassette tapes which were played by the researcher. The participants heard one of two different case descriptions. The two case descriptions varied on the type of problems described; the nature of the problem; the identity of the person seeking treatment; the setting in which the treatment would take place; and the characteristics of the child, such as the presence of a diagnostic label, the level of intelligence, age, and gender of the child. The treatments were described to the participants as if they had been conducted with the child and were based on treatments described within the literature. The treatments included reinforcement, time-out from reinforcement, drug treatment, and electric shock. Although each of the treatment descriptions were based on descriptions published within the literature on behavior interventions, the treatments had not necessarily been described for treating the problem described within the case description, but rather for other behavior problems. After listening to the case description and one of the four treatments, the participants completed the TEI and items from the Semantic Differential (Osgood, Suci, & Tannebaum, 1957). The participants then listened to another treatment, for which they also completed ratings, and repeated this process until they had heard and rated all four treatments.
Chapter | 4 Research on Social Validity
Kazdin (1980a) then evaluated potential differences in the ratings of the treatments and found that the TEI ratings and the Semantic Differential ratings differentiated among the four treatments. The TEI determined statistically significant differences among all the treatments, with reinforcement as the most acceptable treatment, followed by time-out, drug therapy, and electric shock. Additional analyses determined that the acceptability differences in the treatments were not attributable to the differences in the case descriptions. In addition, it was determined that the gender of the participant completing the ratings was not a significant factor in the treatment acceptability differences. In order to extend his findings, Kazdin (1980a) conducted a second study which evaluated the potential impact of the severity of the problems presented in the case descriptions. This study replicated the previously described study with 94 participants who were provided with one of four case descriptions and the same treatments described in the previous study. The case descriptions differed as to the gender of the child described and for the severity of the problem behavior, which was described as either moderate or severe. The moderate problem behavior was described in the same manner as the case descriptions in Kazdin’s (1980a) previously described study. The severe problem behaviors included descriptions such as potentially resulting in permanent damage from self-injury, damaging property, being extremely disruptive, being associated with severe levels of mental retardation, and occurring in an institution. The results revealed that reinforcement was significantly more acceptable than the other treatments, and electric shock was significantly less acceptable than the other treatments, while time-out and drug therapy did not differ from each other. The severity of the problems described within the case descriptions significantly influenced the acceptability of the treatments, with higher acceptability ratings being associated with more severe problem behaviors. In addition, Kazdin (1980a) found that males rated reinforcement as less acceptable than females and rated electric shock as more acceptable than females. Kazdin (1980a) pointed out that this gender difference in participants’ ratings accounted for only a small percentage of the variance in acceptability ratings. The methodology described by Kazdin (1980a) has served as a foundation for numerous studies on treatment acceptability, which have replicated and extended the primary features of this methodology. The
71
72
Chapter | 4 Research on Social Validity
methodological features of Kazdin’s (1980a) research which have been highly influential include the following: 1. 2. 3. 4.
Use of college students as raters Presentation of case descriptions Presentation of treatment descriptions Completion of a treatment acceptability rating instrument
College students have been the most frequently used participants for rating the acceptability of treatments. Several other groups of participants have been used to rate the acceptability of treatments, such as teachers, parents, professionals, and so on, but college students have remained a convenient sample from which to quickly obtain large numbers of acceptability ratings. The presentation of case descriptions which detail the individual targeted for treatment, the severity of the problem behaviors, the setting in which the treatment will be conducted, and other variables was originally presented via audio cassette tape to the raters, in order to maintain consistency of the information being presented. Case descriptions have been presented in various formats, such as written descriptions, verbal presentations, and video presentations. The treatments in Kazdin’s (1980a) studies contained information regarding the type of treatment and the specific components of treatment presented via audio cassette. Other studies have included various information related to treatments, including highly detailed descriptions of the treatments, information regarding assessment data used to develop the treatments, information on the effectiveness of treatments, training to increase knowledge of treatments, and opportunities to experience implementing treatments. Treatment acceptability instruments have been used frequently, and several variations of these instruments have been developed for specific purposes and for specific populations, as described in the previous chapter. The two experiments conducted by Kazdin (1980a), which demonstrated the use of the TEI as a valid instrument for evaluating treatment acceptability, have highly influenced the inclusion of variables examined in a line of research studies conducted in this area. These experiments identified several potential variables and combinations of variables which could potentially influence treatment acceptability. See Table 4.1 for some categories of variables which have become prevalent in the
73
Chapter | 4 Research on Social Validity
TABLE 4.1 Categories of Variables Derived from Kazdin’s 1980 Studies Which Influenced Future Research Manipulations of Treatment Variables
Case Description Variables
Rater Variables
Different types of treatment
Age
Gender
Different uses of treatments
Gender
Reinforcement vs. punishmentbased
Severity of behavior
Behavior vs. medical
Type/nature of behavior
Different uses of treatments
Intelligence Diagnosis Relevant caretakers
research on treatment acceptability. The primary variable considered was the different types of treatments described, of which Kazdin (1980a) incorporated only four treatments. This variable has developed into a line of research that has since examined numerous different treatments and countless variations and descriptions of these treatments. Another category of variable investigated by Kazdin (1980a) was characteristics associated with the individual receiving the treatment. These included the age, gender, type of problem displayed, nature of the problem, severity of the problem, the setting in which the problem behavior was displayed, the diagnosis of the individual, their level of intelligence, the caretakers connected to the individual, and so on. Although all of these factors were not directly examined by Kazdin (1980a) in isolation, they have since become a focus of several studies examining treatment acceptability. Another category of variables which has been further studied since Kazdin’s (1980a) research has been factors associated with the person rating the treatment, such as gender. While Kazdin (1980a) only examined the gender of the participants in his study, several other factors related to the participants rating the acceptability of treatments have since been examined, including their relationship to the individual receiving treatment, their knowledge of the treatments, their socioeconomic status, and so on.
74
Chapter | 4 Research on Social Validity
METHODOLOGICAL VARIATIONS Although treatment acceptability research methodology has primarily consisted of analogue research, several variations of this methodology have been incorporated to extend the ecological validity of the findings. Variations have included presentation of case descriptions of problem behavior and treatments by use of audio tape (Kazdin, 1980a, 1980b), written summaries (Cavell, Frentz, & Kelley, 1986), video presentations (Martens et al., 1985), and combinations of presentation formats, including actual implementation of treatments (Reimers et al., 1992). Participants may rate the treatment acceptability of one treatment applied to several different case descriptions, or rate treatment acceptability of several treatments applied to the same or several different case descriptions. While analogue research methodology has provided insight into the acceptability of several treatments when hypothetically applied to numerous different cases covering a wide range of variables, the findings may be limited in ecological validity when compared to a clinical research methodology (Miltenberger, 1990). Clinical research methodology in treatment acceptability has typically involved having clients and/or consultees rate treatment acceptability prior to, during, and/or after implementation of a treatment. This type of clinical research for assessing treatment acceptability has not been highly prevalent within the literature. Armstrong, Ehrhardt, Cool, and Poling (1997) reviewed a collection of five years of research articles in The Journal of Developmental and Physical Disabilities and reported that 13% of the empirical articles provided information on social validity. When acceptability data are presented along with an empirical study, there is usually little opportunity for replication, because of the specificity of the treatments involved, the characteristics of the clients, and/or the individuals providing the ratings. While clinical research methodology to investigate treatment acceptability appears to provide more ecologically valid findings and naturalistic evaluation of treatment acceptability in relationship to other variables, the analogue method of investigating treatment acceptability provides a much larger and more representative data base more quickly, especially with treatments for low incidence populations.
Chapter | 4 Research on Social Validity
LITERATURE REVIEWS OF TREATMENT ACCEPTABILITY Reimers et al. (1987) conducted a review of the literature on behavioral interventions and found five primary factors that were considered to affect treatment acceptability. These factors were problem severity, treatment approach, time needed to implement treatment, side effects of treatment, and cost. Treatments for more severe problem behaviors were generally considered more acceptable than treatments for less severe problem behaviors, with some evidence for increased acceptability when more restrictive treatments were matched with more severe problems and less restrictive treatments were matched to less severe problems. Reinforcement-based procedures were generally rated more acceptable than punishment-based procedures. Treatments which required less implementation time were generally rated as more acceptable in comparison to treatments requiring more implementation time. One study reported that treatments with adverse side effects were rated less acceptable than those reporting no side effects. Cost was a factor suggested by Reimers et al. to influence treatment acceptability, although no studies were reported to have examined this factor as a variable. Elliott (1988) reviewed 20 empirical studies on treatment acceptability of behavioral interventions for school children as rated by teachers, children, and psychologists. Elliott concluded that treatment acceptability was a complex construct influenced by several variables. Additionally, Elliott indicated that educational consumers rated positive treatments as more acceptable than reductive treatments, and that there was a moderate-to-strong relationship between pretreatment acceptability ratings and perceived treatment effectiveness. Miltenberger (1990) conducted a review of the literature on treatment acceptability research conducted during the 1980s and suggested that the most acceptable treatments would be those which were least restrictive, required little time, had the fewest side effects, were least disruptive to other students, were consistent with the rater’s training or orientation, were presented with the most appropriate rationales, were considered to be necessary for behavioral improvement, and were considered to be most effective. Rasnake (1993) conducted a selected review of treatment acceptability research and concluded that numerous variables had been found to be influential regarding ratings of treatment acceptability. She
75
76
Chapter | 4 Research on Social Validity
suggested that the research findings on treatment acceptability should be used cautiously and that, because of limitations within this body of research, the overall acceptability of treatments were debatable. Carter (2007) reviewed 15 years of research on treatment acceptability published between 1990 and 2005. Table 4.2 provides a summary of some of the studies reviewed. Overall conclusions from the review indicated that acceptability of treatments was usually obtained in isolation rather than in combination or package treatments. In addition, several variables were found to inconsistently influence treatment acceptability, and several idiosyncratic, highly volatile variables were being evaluated in treatment acceptability research. Research on treatment acceptability conducted prior to the 1990s primarily focused on treatments to address problem behaviors. Hawkins (1991) criticized the limited focus of treatment acceptability research and suggested expanding the measurement of acceptability to include other aspects of treatment. A recent trend in acceptability research has been to examine the acceptability of practices other than treatment procedures. While treatment acceptability research has focused primarily on the specific practices and procedures related to treatments, this new focus of acceptability research has begun to focus on the procedures and practices related to other areas of service delivery. This type of research is an attempt to extend the concept of treatment acceptability to include practices such as assessment (Shapiro & Eckert, 1994), training programs (Cross Calvert & McMahon, 1987), and consultation (Colton & Sheridan, 1998; Freer & Watson, 1999; Graham, 1998; Kutsick, Gutkin, & Witt, 1991; Martens, Kelly, & Diskin, 1996; Schill, Kratochwill, & Elliott, 1998; Sheridan & Steck, 1995). Kutsick et al. (1991) developed the IPRS, which was reported to measure treatment acceptability. While this study reported findings of treatment acceptability, it primarily focused on three consultation processes. The consultation processes included a collaborative approach, a psychologistdirected approach, and a teacher-directed approach. They provided 240 teachers with written case descriptions which varied on the consultation approach, the severity of the child’s problem described, and the type of treatment. They found that the collaborative consultation approach was significantly more acceptable than the other consultation approaches.
Summary of Treatment Acceptability Research Published Between 1990 and 2005
Studies
Treatment Variable Manipulation
Tingstrom (1990)
Client Variable Manipulation Rater Variable Manipulation Mild vs. severe problem behavior revealed significantly higher acceptability of a timeout procedure for more severe problem behavior
Waas & Anderson (1991)
Teacher vs. school psychologist ratings of timeout revealed no significant differences on acceptability Second-graders, fifth-graders, and college student revealed similarities and differences
Miller & Kelley (1992)
Behavior interventions and medication revealed similarities and differences
Reimers et al. (1992)
Mildly intrusive treatments rated more acceptable for mild problem behaviors and highly intrusive treatments rated more acceptable for severe problem behaviors
Chapter | 4 Research on Social Validity
TABLE 4.2
Mothers vs. fathers, maritally distressed couples vs. nonmaritally distressed couples, and parents of children with and without behavior problems revealed similarities and differences Analog case descriptions and clinical cases revealed similar ratings
77
(Continued)
78
TABLE 4.2
(Continued)
Studies
Treatment Variable Manipulation Paradoxical treatment found less acceptable than nonparadoxical treatment
Rasnake, Martin, Tarnowski, and Mulick (1993)
Differences and similarities revealed among six treatments for self-injurious behavior
Spreat & Walsh (1994)
Highly intrusive treatments less acceptable than less intrusive treatments
Burgio et al. (1995)
Treatment setting influenced acceptability
Fairbanks & Stinnett (1997)
Age of institutional staff influenced acceptability; knowledge of behavioral principles and years of work experience did not Sex, age, level of mental retardation, restrictiveness of residence, behavior descriptors, severity of selfinjury, and frequency of behavior revealed no influence on acceptability
Raters’ personal estimates of treatment success and membership in the Psychology Division of AAMR were found to influence acceptability
Diagnostic label of learning disabled, behavior disordered, and attention deficit disorder did not influence acceptability
Teachers rated negative intervention more acceptable than did school psychologists and school social workers
Chapter | 4 Research on Social Validity
Betts & Remer (1993)
Client Variable Manipulation Rater Variable Manipulation
Reinforcement technique rated more acceptable and spanking rated less acceptable than other techniques
Miller, Manne, and Palevsky (1998)
Description of treatment as medically related did not influence acceptability
Stinnett, Crawford, Gillespie, Cruce, and Langford (2001) Elliot & Fuqua (2002)
Differences found among four treatments for hair pulling
Carter (2005)
Differences found based on professional recommending a treatment
Description of child as healthy vs. having cancer did not influence acceptability
Pediatric nurses, parents of healthy children, and parents of children with cancer revealed some similarities and differences in acceptability
Label of ADHD vs. no label did not influence acceptability
Graduating from an urban vs. a rural high school influenced acceptability
Chapter | 4 Research on Social Validity
Jones, Eyberg, Adams, and Boggs (1998)
Age and severity of problem did not influence acceptability
From Carter, S. L. (2007). Review of recent treatment acceptability research. Education and Training in Developmental Disabilities, 42, 301–316. Reprinted with permission.
79
80
Chapter | 4 Research on Social Validity
Their results were similar to previous research in that they determined reinforcement-based treatments were more acceptable than punishmentbased treatments. Their findings differed from previous findings in that the severity of the problem behavior did not influence treatment acceptability. This study was unique because it appeared to focus on the acceptability of consultation approaches, which represented an extension beyond variables directly related to treatments. This type of research appears to support Hawkins’s (1991) recommendation to extend acceptability research to include other aspects of service delivery. Aaroe and Nelson (1998) conducted a review of the literature on acceptability focusing on aspects of the curriculum that included preferred placements for services and types of instructional activities preferred. They reviewed 10 years of research within 11 journals published by the Council for Exceptional Children and found 11 articles which met their criteria for review. They reported that rating scales were used in two of the studies, while the majority of the studies reviewed used interviews. Their findings concluded that students with disabilities enjoyed receiving instruction in resource classrooms. Additional conclusions were that students with disabilities preferred activities which promoted feelings of competence and which produced high success rates. Eckert and Hintze (2000) conducted a review of acceptability research that focused on school-based service delivery practices rather than treatments. Their review included an examination of two studies focusing on the acceptability of mathematics interventions and two studies focusing on the acceptability of interventions for children diagnosed with attention deficit hyperactivity disorder (ADHD). In addition, they examined five studies which focused on the acceptability of school-based consultation approaches and four studies which focused on the acceptability of schoolbased assessment procedures. They concluded from their review that several variables may influence the acceptability of school-based treatments focusing on academic skills and ADHD. They also concluded that influences on the acceptability of school-based consultation may stem from the school situation, the consultant, the consultee, and the client. Additionally, they suggested that their review provided some support for recently developed assessment procedures and indicated that future research should consider methodological issues when evaluating the acceptability of school-based practices.
Chapter | 4 Research on Social Validity
RESEARCH FINDINGS Examination of the research on treatment acceptability reveals three types of variable manipulation within the published literature. These specific variables can be broadly categorized as: (1) manipulations involving treatment variables (e.g., type of treatment, effectiveness of treatment, and mediator of treatment), (2) manipulations involving client/case variables (e.g., severity of client’s problem, age/gender of client, diagnostic label of client), and (3) manipulations involving the consumer variable or the individual from which treatment acceptability ratings are obtained (e.g., professional affiliation of the raters, raters’ knowledge of treatment, and geographic location of raters’ school). A review of the research on treatment acceptability will be provided as conducted by the various treatment acceptability rating scales. Treatment acceptability research has involved a myriad of psychotherapeutic treatment techniques, such as marital therapy (Bornstein et al., 1983). This review will only include treatments that are considered to have a basis in the principles of applied behavior analysis as defined by Baer et al. (1968). Emphasis will be placed on acceptability of actual behavioral treatments rather than acceptability of other practices related to treatments, such as consultation techniques. The review is organized according to the specific treatment acceptability instrument used to collect data. In addition, the factors found to influence treatment acceptability will be examined within three lines of research which stem from Kazdin’s (1980a) studies, namely: variations involving the treatment, variations involving the individual targeted for treatment (the client), and variations involving the individual completing the acceptability evaluations (the rater). While this is not intended to be an exhaustive review, it will hopefully provide a sufficient number of studies to represent some of the more frequently used treatment acceptability instruments and the numerous factors which have been found to influence treatment acceptability.
TEI and TEI-SF The studies conducted by Kazdin (1980a), which were previously described, were part of the development and validation of the TEI. These studies provided information on several different variables, including
81
82
Chapter | 4 Research on Social Validity
those associated with the treatment, those associated with the consumer targeted for treatment, and the raters who completed the acceptability ratings. The treatment variables examined in Kazdin’s (1980a) initial studies included a comparison of highly intrusive versus less intrusive treatments and treatments based on reinforcement techniques versus treatments based on punishment techniques. The treatments described within Kazdin’s (1980a) studies represented an increasing level of intrusiveness among the treatments, with reinforcement being the least intrusive, followed by time-out from reinforcement, drug treatment, and electric shock (the most intrusive treatment). The highly intrusive treatments were found to be less acceptable than the less intrusive treatments. The treatment based on reinforcement was found to be the most acceptable treatment over all the other treatments, which were based on punishment techniques. A client variable was also examined by Kazdin (1980a), which involved the severity of the problem behavior displayed by the child described in the case description. The more severe problem behavior resulted in increased acceptability ratings. In addition, the rater variable of gender was found to produce different acceptability rating, with males rating electric shock as more acceptable than females. This study, which has proven to be highly influential in subsequent research, demonstrated how treatment variables, client variables, and rater variables could be manipulated and examined as to their relationship to treatment acceptability (treatment ⫽ level of intrusiveness; reinforcement vs. punishment) (client ⫽ severity of problem behavior) (rater ⫽ gender).
Treatment Variables Other variables associated with treatment have been investigated using the TEI. Kazdin (1980b) evaluated the influence that the context in which a treatment was presented had upon treatment acceptability. This study consisted of two experiments which followed a similar methodology to his previous study on acceptability (Kazdin, 1980a). The first experiment involved 144 undergraduate students from psychology courses. The participants rated the acceptability of a positive reinforcement procedure and three time-out procedures (isolation, withdrawal of attention, and contingent observation). The treatments were presented via audiotape to address the behavior problems of a child described in one of two case descriptions which varied as to the gender of the children, the level
Chapter | 4 Research on Social Validity
of intelligence, and the behaviors described. The findings revealed statistically significant differences among all the treatments, with positive reinforcement as the most acceptable, followed by contingent observation, withdrawal of attention, and isolation. In the second experiment, the TEI was used to determine that the isolation time-out procedure was more or less acceptable based upon the context in which it was presented to 104 undergraduate college students recruited from psychology courses. When the isolation time-out procedure was presented as part of a behavioral contract or as a supplement to a withdrawal of attention procedure, it was rated as more acceptable than when it was presented as the sole treatment. This study demonstrated that associating a highly intrusive treatment with a less intrusive treatment could increase the acceptability of the less acceptable treatment. Jones et al. (1998) continued the examination of intrusiveness of treatments. They assessed the treatment acceptability of six child management techniques (positive reinforcement, response cost, differential attention, time-out, overcorrection, and spanking) using the TEI-SF with 20 mothers of children referred for treatment of disruptive behavior. Mothers rated the positive reinforcement technique as more acceptable than other techniques, while spanking was rated as less acceptable than the other techniques. Cavell, Frentz, and Kelley (1986b) conducted a study, similar to Kazdin’s (1980b) study, which examined the impact of the context in which a treatment was described with the acceptability of the treatment. Middle and high school teachers read case scenarios involving the withdrawal of an effective treatment. The cases differed on the rationale which was provided for withdrawing the treatment. The results determined that the rationale provided to the teachers for withdrawing the treatment influenced the acceptability of the treatment. This study demonstrated that rationales used to describe the withdrawal of a treatment could influence the acceptability of the treatment. In addition, this study demonstrated that the variable of context was influential on acceptability of treatments with a population other than college students. Blampied and Kahan (1992) examined the context in which five punishment procedures were administered. They randomly recruited 201 community members from New Zealand to rate the acceptability of these
83
84
Chapter | 4 Research on Social Validity
procedures when administered at school or at home. They found that the procedures were more acceptable when administered at home rather than at school. The language used to describe a treatment is another treatment variable that has been examined using the TEI. Singh and Katz (1985) presented case descriptions to 96 undergraduate psychology students and asked them to rate the acceptability of four different treatments (differential reinforcement of incompatible behavior, positive practice overcorrection, time-out, and humanistic parenting) using the TEI. The results indicated the following ranking for acceptability, with the greatest acceptability first: differential reinforcement of incompatible behavior, humanistic parenting, and positive practice. Time-out received ratings as least acceptable. Following this study, the same participants were provided formal educational training on three behavioral treatments (differential reinforcement of incompatible behavior, positive practice overcorrection, and time-out) and then the participants reevaluated all four treatments. The training provided information such as specific details of each treatment, empirical data outlining the effectiveness of each treatment, and potential side effects of the treatments. At posttraining, the participants’ treatment rating revealed the rankings to be differential reinforcement of incompatible behavior, positive practice overcorrection, and time-out. The humanistic parenting treatment was rated lower before education than after education. The apparent appropriateness of treatments is another treatment variable that has been investigated using the TEI. Cavell, Frentz, and Kelley (1986a) used the TEI with 120 middle and high school teachers to assess the acceptability of four paradoxical treatments with different rationales and an ineffective contingency contract for a hypothetical case of school disruption and truancy. They found that continuing the ineffective contingency contract was rated as more acceptable than all paradoxical treatments, regardless of the rationale provided. Betts and Remer (1993) conducted research similar to that of Cavell et al. (1986a) on the apparent appropriateness of treatments. They evaluated the acceptability of paradoxical versus nonparadoxical interventions using the TEI-SF. Paradoxical interventions were defined as techniques designed to eliminate undesirable behavior by encouraging the
Chapter | 4 Research on Social Validity
undesirable behavior. Participants in this study were 97 undergraduates taking part in a semester-long simulation of a family arguing with a rebellious adolescent daughter. The participants were grouped into “families” and participated in four role-play exercises and a family therapy session prior to evaluating either a paradoxical or a nonparadoxical intervention using the TEI-SF. The findings indicated that paradoxical interventions were less acceptable than nonparadoxical interventions, although paradoxical interventions were considered to be an acceptable intervention overall.
Previous Treatment Information Kazdin (1981) used the TEI to investigate the acceptability of four treatments for a child who displayed either aggression or hyperactivity. During two experiments, undergraduate college students rated the acceptability of reinforcement of incompatible behavior, positive practice, time-out from reinforcement, and medication. In the first experiment, the treatments were described as having either strong or weak therapeutic effects. In the second experiment, the treatments were described as having no side effects or as having undesired side effects. The acceptability ratings, from most to least acceptable, for both experiments were as follows: reinforcement of incompatible behavior, positive practice, time-out from reinforcement, and medication. The results of experiment one demonstrated that the description of strong or weak therapeutic effects did not influence the acceptability ratings. The results of experiment two revealed that the presence of undesired side effects significantly decreased the acceptability of all treatments. Singh, Watson, and Winton (1987) partially replicated the study by Kazdin (1981) with 96 mothers of children diagnosed with mental retardation. They found that the mothers rated the acceptability of the treatments in a hierarchy matching the findings of Kazdin (1981). Hobbs, Walle, and Caldwell (1984) used the TEI to examine the acceptability of three treatments for child noncompliance. This study measured 20 mothers’ acceptability ratings after they implemented each of the procedures with their own child in a brief parent training session. Although differences in the effectiveness of the techniques were noted, no significant differences in the acceptability of the treatments resulted. In a similar posttreatment acceptability study, Hobbs, Walle, and
85
86
Chapter | 4 Research on Social Validity
Hammersly (1990) evaluated the influence of three variables on treatment acceptability. Following a training session using three treatments (social reinforcement, time-out, and a combination of social reinforcement and time-out) with their child, 20 mothers rated the acceptability of these treatments using the TEI. In addition, information from rating scales and direct observations was obtained on the severity of their child’s problem, adverse side effects displayed by their child, and effectiveness of the treatments with their child. A multiple regression revealed that the treatment effectiveness and adverse side effects influenced treatment acceptability ratings. Kalfus and Burk (1989) examined the influence of previous treatment information on treatment acceptability. They administered the TEI to 105 graduate-level education students and 53 undergraduate-level psychology students to evaluate the acceptability of five treatment procedures for a case vignette of a child who engaged in pica. Five treatment procedures were rated in the following order of most acceptable to least acceptable: positive reinforcement, overcorrection, contingent removal of a pica item, differential attention, and time-out. Treatment history information was manipulated by providing either a long case description or a shorter case description. The absence of treatment history information increased acceptability ratings by undergraduate psychology students and decreased acceptability ratings by graduate education students. The findings from this research supported previous research by identifying positive reinforcement as the most acceptable treatment, time-out as the least acceptable treatment, and other treatments receiving ratings falling between positive reinforcement and time-out. This study also evaluated the influence of different treatment mediators. The treatment mediator identity was manipulated by presenting the treatment mediator as either a psychologist or as a combination of parents and teachers. The identity of the treatment mediator (parents and teachers, or psychologist) did not influence acceptability ratings. Tingstrom, McPhail, and Bolton (1989) continued the examination of previous treatment information using the TEI by manipulating information specifically stating that a treatment was effective, ineffective, or providing no information on effectiveness. This study also manipulated the age of a child targeted for treatment. Participants were 302 undergraduate students enrolled in psychology courses who provided acceptability ratings of four
Chapter | 4 Research on Social Validity
school-based treatments. The results determined that when a treatment was reported as effective, it was rated as more acceptable than when it was rated as ineffective. When no information was provided on the effectiveness of the treatment, only small increases in acceptability were noted over treatments described as ineffective. The age differences in child targeted for treatment did not influence acceptability ratings.
Client Variables Another area which has been examined in relation to treatment acceptability using the TEI are variables associated with the individual targeted for treatment, which will be referred to as client variables. Frentz and Kelley (1986) examined the acceptability of five reductive treatments for a range of problem behavior. Mothers with children ranging in age from 2 to 12 years were presented case scenarios and then completed the TEI for the different treatments. The results revealed that the mothers rated the reductive procedures as more acceptable when the procedures were described as being applied to more severe behavior problems. When these same procedures were applied to less severe behavior problems, the acceptability ratings decreased. This study provided evidence indicating a relationship between the severity of the problem behavior and the acceptability of treatments. Similarly, Miltenberger et al. (1989) evaluated community-based and institution-based staff members’ TEI ratings for four treatments to address mild and severe disruptive behaviors. They found that the severity of problem was influential on some of the acceptability ratings provided by community-based and institutionbased staff. Lindeman, Miltenberger, and Lennox (1992) also found evidence, using the TEI, that the severity of problem behavior influenced acceptability ratings of superintendents of public residential facilities for individuals with mental retardation.
Rater Variables Differences in the characteristics of those rating the acceptability of treatments have been evaluated using the TEI. Kazdin, French, and Sherick (1981) examined acceptability differences among parents, clinical staff members, and child psychiatric patients for four alternative treatments. Results from the TEI determined that the treatments associated with positive reinforcement were more acceptable than those associated with punishment. In addition, the parents and staff members rated
87
88
Chapter | 4 Research on Social Validity
the treatments as more acceptable overall than the children did. This study provided further support for the existence of differential acceptability for reinforcement-based versus punishment-based procedures within populations closely involved with the treatments described. The study also demonstrated that acceptability could be dependent upon the population from which the ratings were obtained. Epstein, Matson, Repp, and Helsel (1986) used the TEI to examine differences in acceptability ratings between regular education teachers and special education teachers. They also evaluated the influence of labeling a child as having a learning disability versus a label of mental retardation. They found no differences between the ratings by the teachers or based on the label designated to the child targeted for treatment. They did determine that teachers were able to distinguish between treatments based on the acceptability of the treatments. The TEI has been used to examine rater variables among professionals from different divisions of an organization. Spreat and Walsh (1994) assessed factors associated with decisions regarding acceptability of behavior modification programs by members from differing divisions of the American Association of Mental Retardation (AAMR). Surveys were mailed to 400 members of Region IX of the AAMR. Of these surveys, 198 were returned, representing an approximately 50% return rate. The survey consisted of a case vignette with manipulations occurring among nine variables with randomly assigned values. The variables manipulated for the client described in the case vignette included sex, age, level of mental retardation, restrictiveness of residence, behavior descriptors, severity of self-injury, and frequency of behavior. A modified TEI was used to rate the acceptability of each case vignette. None of the client variables were found to be statistically significant in influencing treatment acceptability. Treatment intrusiveness variables were also manipulated within the case vignette and included descriptions of previously used procedures, likely side effects, and different treatments with varying levels of restrictiveness. The intrusiveness factors found to influence acceptability were restrictiveness of the proposed treatments and whether other procedures had been previously tried. The findings indicated that the strongest indicator of treatment acceptability was the respondents’ personal estimates of probable treatment success. Members of the Psychology Division of the AAMR rated treatments as slightly more acceptable than
Chapter | 4 Research on Social Validity
members of other divisions of AAMR (treatment ⫽ restrictiveness; previous treatments; no effect ⫽ side effects) (client ⫽ no effects ⫽ sex, age, level of MR, restrictiveness of residence, behavior descriptors, severity of selfinjury, and frequency of behavior) (rater ⫽ differing divisions of AAMR, personal estimates of treatment success). Rater variables have been examined on some personal levels using the TEI, such as with parents of children who had or did not have a specific diagnosis. Gage and Wilson (2000) randomly assigned 30 parents of children with ADHD (as determined by the Child Behavior Check List and a diagnosis by a professional) and 30 parents of children without an ADHD diagnosis to three different conditions. The three experimental conditions consisted of (1) medication, (2) behavioral treatment, and (3) a combination of both. Following a case vignette, the TEI was used to measure acceptability of treatments. The data were analyzed using three ANOVAs for each treatment scenario across TEI ratings. Results indicated that parents of children with ADHD rated medications and combination of both treatments statistically significantly higher than parents of children without ADHD. Parents of children without ADHD rated the behavioral treatment statistically significantly higher than parents of children with ADHD. Parents of children with ADHD rated the combination of treatments statistically significantly higher than the other treatments. No other significant effects were found within the parents of children with ADHD group. Among parents of children without ADHD, the results revealed statistically significant differences among all treatments, with the behavioral treatment receiving the highest acceptability treatment, followed by the combination treatment, and then the medication treatment. Heffer and Kelley (1987) found that specific affiliations of raters influenced ratings of treatment acceptability. They used the TEI to assess mothers’ ratings of five child management techniques. They found that raters from different socioeconomic classes and from different races rated acceptability of treatments differently. Tingstrom and Silver (1990) replicated and reevaluated data collected by Tingstrom et al. (1989). The focus of this study was to evaluate the potential influence of race as a variable influencing the acceptability of treatments. The protocol from the Tingstrom et al. study was used to collect data from 256 black undergraduate students. These data were then combined with results from 253 white undergraduate students from the Tingstrom et al. study.
89
90
Chapter | 4 Research on Social Validity
The results concluded that race was not an influential variable in ratings of acceptability for the school-based treatments. Miller and Kelley (1992) evaluated three rater variables (gender, marital adjustment, and child behavior) on treatment acceptability. The TEI was administered to 69 married couples with young children to assess the acceptability of six interventions (positive reinforcement, response cost, medication, room time-out, chair time-out, and spanking) for a hypothetical child displaying noncompliance and aggressiveness. The couples also completed a marital adjustment scale and a child behavior problem inventory. Findings revealed that gender influenced treatment acceptability ratings for all interventions except chair time-out, with mothers providing higher ratings than fathers for all interventions except spanking and medication. When compared to non-distressed couples, those couples in marital distress provided significantly higher acceptability ratings for room time-out and significantly lower ratings for positive reinforcement. Couples of children displaying problem behaviors rated medication as more acceptable and spanking as less acceptable than couples of children without behavior problems. Miller et al. (1998) used the TEI-SF to examine the acceptability of five behavioral treatments (positive reinforcement, chair time-out, response cost, overcorrection, and reprimands) for general or medically related noncompliance of a hypothetical child described as either healthy or with cancer. Ratings were obtained from parents of children receiving medical treatment for cancer, pediatric nurses, and parents of medically healthy children. Results revealed no significant differences in acceptability ratings based on the type of noncompliance (general vs. medically related) or based on the child being described as healthy or with cancer. Parents of children receiving medical treatment for cancer rated the response cost and the time-out procedure significantly lower than the other participants. Parents of healthy children rated the positive reinforcement procedure significantly less acceptable than did the other groups. In summary, the research using the TEI and the TEI-SF has determined that some treatments are consistently more acceptable than others. The specific variables associated with treatments that influence treatment acceptability include reinforcement-based or punishment-based procedures (Kazdin, 1980a, 1980b; Kazdin et al., 1981), the context in which
Chapter | 4 Research on Social Validity
the treatment is presented (Cavell et al., 1986b), the language used to describe the treatment (Singh & Katz, 1985), the apparent appropriateness of the treatment (Betts & Remer, 1993; Cavell et al., 1986a), previous treatment information (Kalfus & Burk, 1989; Tingstrom et al., 1989), side effects (Kazdin, 1981), and restrictiveness of treatments (Spreat & Walsh, 1994). The client variables that have been found to influence treatment acceptability include severity of behavior problem for which treatment is targeted (Frentz & Kelley, 1986; Miltenberger et al., 1989; Lindeman et al., 1992). The rater variables which have been found to influence treatment acceptability include adult ratings compared to child ratings (Kazdin et al., 1981), membership in differing divisions of a professional association (Spreat & Walsh, 1994), race (Heffer & Kelley, 1987), income level (Heffer & Kelley), gender (Miller & Kelley, 1992), marital adjustment (Miller & Kelley), parenting a child with cancer (Miller et al., 1998), and parenting a child with ADHD (Gage & Wilson, 2000).
IRP and IRP-15 As described in Chapter 3, the IRP was specifically designed to evaluate the acceptability of treatments typically used in classroom situations to address student problem behaviors. The IRP and subsequent modifications have been used frequently, and results of research with these instruments will be evaluated in the same manner as the TEI. Manipulations of treatment variables, client/student variables, and rater variables often overlap, but each of the research studies will be arranged according to what can be determined to be the primary focus of the study.
Treatment Variables In the initial development of the IRP, Witt and Martens (1983) demonstrated differences among reinforcement-based treatments typically used in schools and punishment-based treatments typically used in schools. They administered the IRP to preservice and student teachers who rated the acceptability of six different behavior interventions. The results were similar to previous findings using the TEI, with reinforcementbased treatments rated as more acceptable than punishment-based treatments. In addition, the study determined that reinforcement-based treatments were more acceptable when they required minimal amounts of implementation time and when they were described as targeting mild
91
92
Chapter | 4 Research on Social Validity
behavior problems. This study is notable because it was one of the first to design an acceptability instrument, other than the TEI, which could be used for specific types of treatments. Witt, Elliott, et al. (1984) examined further the findings from this study. They used the IRP to examine the relationship between the amount of time required by a teacher to implement a treatment and treatment acceptability. They also examined the relationship of the severity of the problem behavior exhibited by the student described in the case vignette to treatment acceptability. This would be considered a student variable rather than a treatment variable. Their participants were preservice and student teachers who were presented a case vignette describing either a reinforcementbased treatment or a punishment-based treatment to address either a severe or a mild problem behavior. They were also provided with information on the amount of time required by the teacher to implement the treatment. Their findings were similar to those of Witt and Martens (1983) in that the acceptability of the reinforcement-based treatments was increased when the treatment was described as requiring less teacher time. Their findings were different from those of Witt and Martens (1983) with respect to the severity of the problem behavior, which did not significantly influence acceptability ratings. Witt, Martens, and Elliott (1984) conducted an additional replication of this line of research with teachers in grades K-12. Their findings provided further evidence that treatments which required less time to implement were rated as more acceptable by teachers. They also determined that the severity of the problem behavior had no significant influence on acceptability ratings. Elliott, Witt, Galvin, and Peterson (1984) examined the acceptability of school-based treatments among regular and special education teachers. They found that severity of problem behaviors influenced the acceptability of treatments. In addition, they found that the treatments which were less time intensive were rated as more acceptable; more timeintensive procedures and reinforcement procedures were rated more acceptable than punitive procedures. Similarly, Witt, Moe, Gutkin, and Andrews (1984) evaluated the acceptability of school-based treatments among regular and special education teachers. They presented teachers with treatments using pragmatic, humanistic, and behavioral terminology. They found that the teachers provided higher acceptability ratings to treatments described using pragmatic terminology. Their
Chapter | 4 Research on Social Validity
findings revealed that the severity of the problem behavior influenced treatment acceptability, with higher acceptability ratings being assigned to treatments addressing more severe behaviors. They also found that less experienced teachers provided higher ratings of treatments than more experienced teachers. Broughton and Hester (1993) extended the research indicating that treatment acceptability among teachers is influenced by the amount of time required to implement a treatment. They examined the influence of administrative and community support on the acceptability ratings of two classroom interventions as rated by teachers. They found that the presence of administrative and community support increased treatment acceptability.
Consumer Variable Manipulation Martens et al. (1985) shortened the IRP to form the IRP-15, which they administered to regular and special education teachers. They manipulated the amount of time required to implement treatments and the severity of the problem behavior described within the case description. Their findings supported previous findings which found that the amount of time required to implement a treatment influenced the acceptability of the treatment, as well as an interaction between the severity of the problem behavior described in the case description. These results supported findings by Elliott et al. (1984) and Witt et al. (1984). Tarnowski, Rasnake, Mulick, and Kelly (1989) examined the influence of three levels of self-injurious behavior severity on the acceptability of six behavior treatments. Intermediate care facility staff completed the IRP-15, and the results indicated a positive correlation between problem severity and the punishment-based treatments. Tarnowski, Mulick, and Rasnake (1990) replicated the Tarnowski et al. (1989) study with residential treatment staff from a facility adhering to a behavioral treatment philosophy. Findings from the initial study were supported, with higher acceptability ratings being provided for all treatments by the staff from the facility following the behavioral treatment philosophy. This study demonstrated an influence on acceptability by the severity of the problem behavior for punishment-based treatments and demonstrated differences based upon the philosophical policies of the raters’ workplace. Expanding upon the research involving the severity of the problem behavior, Harris, Preller, and Graham (1990) used the IRP-15 to examine
93
94
Chapter | 4 Research on Social Validity
the acceptability of two cognitive-behavioral treatments and two behavioral treatments. They provided 203 teachers with descriptions of a selfmonitoring procedure, a self-instructional strategy, a token economy, and a social reinforcement technique to address mild or severe student problem behaviors. They found that the teachers rated all the treatments acceptable, but also noted significant differences in the ratings based on whether the treatments were applied to a mild or severe problem behavior.
Rater Variable Manipulation Witt and Robbins (1985) conducted two experiments using the IRP with preschool, Head Start, and regular education teachers. The first experiment evaluated the acceptability of six school-based interventions. Findings revealed differential reinforcement of other behavior (DRO) as the most acceptable, followed by differential reinforcement of low rates of behavior (DRL), time-out, staying after school, and corporal punishment as the least acceptable. Experiment two evaluated the influence of the person implementing a treatment that involved staying after school. The findings determined that when a teacher was the treatment mediator, acceptability ratings were higher than when a principal was described as the treatment mediator. Additional analyses revealed that less experienced teachers rated treatments more acceptable than more experienced teachers, which supported findings by Witt, Moe, Gutkin, and Andrews (1984). An additional finding supported previous findings using the IRP, which found that acceptability ratings were influenced by the severity of the problem behavior being addressed (Witt & Martens, 1983; Witt, Moe, Gutkin, & Andrews, 1984). Tingstrom (1990) further evaluated the influence of the individual described as implementing a procedure. He used the IRP-15 to assess 103 teacher ratings of a time-out procedure when implemented by a teacher or a school psychologist for either mild or severe problem behavior. Findings revealed no significant differences based on the individual described as implementing the treatment. Significantly higher ratings were provided for time-out when applied to severe problem behavior versus mild problem behavior.
Raters’ Knowledge of Treatment Rasnake et al. (1993) evaluated the association of knowledge of behavior principles to treatment acceptability ratings. Participants included 57
Chapter | 4 Research on Social Validity
directive care staff members employed at an intermediate care facility. A case description was presented to the participants with manipulations of severity levels of self-injurious behavior. Treatment vignettes using six different behavioral interventions (differential reinforcement of other behavior, differential reinforcement of incompatible behavior, stimulus control, overcorrection, physical restraint, and contingent shock) were rated by participants. The IRP was used to rate treatment acceptability. Participants also completed a 25-item multiple forced-choice instrument as a means of measuring prior knowledge of behavioral principles. The data were analyzed using Pearson correlation coefficients. The researchers found that an increased knowledge of behavioral principles did not result in increased treatment acceptability ratings. In addition, the stimulus control intervention was the only intervention to receive an overall mean acceptability rating at the “acceptable” level. Fairbanks and Stinnett (1997) evaluated treatment acceptability of different behavior interventions associated with different diagnostic labels as rated by members of different professional groups. A vignette was presented to 31 teachers, 33 school psychologists, and 33 school social workers from three school districts in southwestern Illinois. A copy of the IRP-15 was completed by the participants to rate the acceptability of the proposed treatment described within the vignette. Variables manipulated included the diagnostic label of the child described within the case vignette. Diagnostic labels manipulated were learning disabled (LD), behavior disordered (BD), and Attention Deficit Disorder (ADD). The case vignette described a third-grade boy who displayed behaviors such as excessive talking, out-of-seat, and overly active. A three-way ANOVA was used to evaluate the data. Findings indicated no significant differences based on the diagnostic label of the child described in the case vignette. Among the members of the different professional groups, teachers rated the negative intervention (time-out from reinforcement with praise) more acceptable than did school psychologists or school social workers. Also, school psychologists rated the negative intervention more acceptable than did school social workers.
Diagnostic Label of the Client Stinnett et al. (2001) manipulated the variable of attaching a label to the client described in the case description and measured the impact this
95
96
Chapter | 4 Research on Social Validity
manipulation had on treatment acceptability. Their study also evaluated the acceptability by teachers-in-training of two treatments, Ritalin versus special education placement, for students with a label of ADHD versus those without a diagnosis. Participants were recruited from undergraduate teacher education courses at a medium-sized university in the Southwest and consisted of 27 males and 117 females. These teachersin-training were asked information regarding the high school from which they graduated and grouped by graduation from a rural versus an urban high school. The participants were predominantly Caucasian (87.5%), with 50% having graduated from a rural high school and 49% having graduated from an urban high school. Participants were presented with a case vignette with manipulations involving label (ADHD vs. no label) and treatment (Ritalin vs. special education placement). Participants rated acceptability of interventions using the IRP-15. These data were analyzed with other data collected using three 3-way ANOVAs. The impact of labeling on the child in the case vignette with ADHD versus no label was found to have no statistical significance, and no statistically significant difference was found between the two treatments (Ritalin vs. special education placement). The location of raters’ high schools was found to influence the acceptability of treatments, with participants who graduated from rural high schools rating treatments as more acceptable than participants who graduated from urban high schools. In summary, the IRP and subsequent modifications have identified several variables which appear to influence the acceptability of treatments. The treatment variables that have been found to be influential include reinforcement versus punishment-based procedures (e.g., Witt, Elliott, et al., 1984; Witt & Martens, 1983), amount of time required to implement a treatment (Elliott et al., 1984; Martens et al., 1985; Witt, Elliott, et al., 1984; Witt & Martens, 1983; Witt, Martens, & Elliott, 1984), terminology used to describe the treatment (Witt, Moe, et al., 1984), the treatment mediator (Witt & Robbins, 1985), and availability of administrative and community support (Broughton & Hester, 1993). The client variables found to influence acceptability include the severity of the behavior problem (Elliott et al., 1984; Harris et al., 1990; Martens et al., 1985; Tarnowski et al., 1989, 1990; Tingstrom, 1990; Witt & Martens, 1983; Witt & Robbins, 1985; Witt, Moe, et al., 1984). The rater variables which have been demonstrated to influence acceptability include teacher
Chapter | 4 Research on Social Validity
experience (Witt & Robbins, 1985; Witt et al., 1984), professional group membership (Fairbanks & Stinnett, 1997), and location of raters’ high school (Stinnett et al., 2001).
Teacher Acceptability Measure (TAM) Power, Hess, and Bennett (1995) used the TAM (a shortened version of the IRP-15) to examine 147 teachers’ acceptability ratings of a daily report card procedure, a response cost procedure, and stimulant medication to address the problem behaviors of a child diagnosed with ADHD. They found that the teachers rated the daily report card procedure significantly more acceptable than the other procedures. They also found that the stimulant medication was rated as less acceptable when described in isolation than when described in combination with a behavior treatment. This finding supported previous findings indicating that the context in which a treatment was presented influenced the acceptability of the treatment (Cavell et al., 1986b). In addition, their findings demonstrated that a teacher’s knowledge of ADHD and years of teaching experience were not influential regarding acceptability ratings.
CIRP Elliott et al. (1986) used the CIRP to evaluate the treatment acceptability of several frequently used school-based treatments for problem behavior. Sixth-grade students were presented with case descriptions of 12 treatments for which they provided acceptability ratings. The results determined the most acceptable treatments to be private student–teacher interactions, group reinforcement, and negative sanctions. The least acceptable treatments were found to be public reprimands and punitive group contingencies. Shapiro and Goldberg (1986) examined the acceptability of a dependent group contingency, an independent group contingency, and an interdependent group contingency on spelling performance. They administered the CIRP to sixth-grade students following exposure to all three group contingencies. The resulted revealed that the students rated the independent group contingency as the most acceptable treatment. In a partial replication, Goldberg and Shapiro (1995) used the CIRP to evaluate the acceptability of an interdependent group contingency and a dependent group contingency on spelling performance among 85
97
98
Chapter | 4 Research on Social Validity
sixth-grade students. The students rated the acceptability of the group contingencies both before and after exposure to the contingencies. The results demonstrated that the students rated the interdependent contingency more acceptable prior to exposure to both contingencies. Following exposure to both group contingencies, the ratings showed no differences between the two treatments. Comparisons of acceptability among different types of raters have been examined for various forms of group contingencies. Elliott, Turco, and Gresham (1987) used both the IRP and the CIRP to evaluate the acceptability of three group contingencies among children, teachers, and psychologists. The teachers and psychologists rated the independent group contingency and the interdependent group contingency as acceptable and rated the dependent group contingency as unacceptable. They found that the children rated all the group contingencies acceptable. Waas and Anderson (1991) used the CIRP to reveal differences in treatment acceptability ratings of second-graders, fifth-graders, and college students. The participants rated the acceptability of a behavior contingency intervention, group counseling, and special education class placement. The findings revealed that college students rated the behavior contingency intervention and special education class placement significantly lower than did the second- and fifth-graders. No significant differences in acceptability ratings were found for the group counseling treatment. Arra and Bahr (2005) used the IRP and the CIRP to examine the acceptability of three remedial mathematics interventions. They administered a modified IRP to student teachers before and after implementation of a cognitive, behavioral, or traditional mathematics intervention. They also obtained acceptability ratings, using a modified CIRP, from 55 students who taught using the mathematics intervention for 6 weeks. Preimplementation ratings were obtained with both the IRP and the CIRP using treatment scenarios describing the use of the three interventions. Postimplementation ratings were obtained based on experiences with the intervention to which the raters were assigned during the 6-week intervention. The results revealed no statistically significant differences of acceptability among the three interventions for the student teachers or the students on both pre- and postimplementation. In addition, no statistically significant differences were revealed between pre- and postexposure to the intervention for either the
Chapter | 4 Research on Social Validity
student teachers or students. The study did report that all three of the interventions appeared to have overall high acceptability.
BIRS Von Brock and Elliott (1987) used the BIRS to evaluate the influence of effectiveness information and problem severity on treatment acceptability. Case descriptions were developed which described the severity of the problem behavior displayed by a child as either mild or severe. Treatment descriptions of a token economy, a response cost procedure, and a time-out procedure were developed, along with information describing varying degrees of effectiveness for each of the treatments. The case descriptions and treatments with varying effectiveness information were used by 216 teachers to rate the acceptability of the treatments. The results concluded that both the severe problem behavior and increased effectiveness information influenced the acceptability of the treatments. The time-out procedure was found to be significantly less acceptable than the other two treatments. Manipulation of the information about level of treatment effectiveness as an influence on treatment acceptability was also examined by Clark and Elliott (1988). They distributed a case vignette, a treatment acceptability measure, and a questionnaire concerning general knowledge of techniques to 133 elementary school teachers from Nebraska and Louisiana. The case vignettes manipulated two potential treatments: modeling–coaching (a form of) versus overcorrection method (a form of), and two levels of outcome effectiveness (weak vs. strong therapeutic effects). Treatment acceptability was rated using the BIRS. Findings indicated a statistically significant preference for the modeling–coaching treatment when compared to the overcorrection treatment, and statistically significant positive influence on acceptability by strong therapeutic outcome effects. They also determined a positive correlation between knowledge of techniques and acceptability ratings. Miller, DuPaul, and Lutz (2002) evaluated the acceptability of three psychosocial treatments for childhood depression using the BIRS. A total of 228 members of the National Association of School Psychologists rated cognitive restructuring and self-control therapy as more acceptable than social skills training.
99
100
Chapter | 4 Research on Social Validity
Olive and Liu (2005) compared posttreatment acceptability ratings on the BIRS between teachers and parents. Following successful implementation of a treatment for challenging behavior, parents rated the treatments more acceptable than teachers. In addition, the acceptability of the treatments increased in relation to the overall behavior change resulting from the treatment. Pisecco, Huzinec, and Curtis (2001) evaluated the influence of characteristics of the child described in a case vignette on teacher acceptability ratings of behavioral and medication treatments. They presented 159 elementary school teachers with case vignettes which varied the specific subtype of ADHD and the gender of the child. Then the teachers completed the BIRS for a description of a daily report card procedure, a response cost technique, a classroom lottery, and medication. Their results concluded that the daily report card was the most acceptable treatment, and they also found that medication was more acceptable as a treatment for boys than for girls. In a similar study, Curtis, Pisecco, Hamilton, and Moore (2006) used the BIRS to examine cross-cultural differences in acceptability ratings of some classroom treatments for students diagnosed with ADHD. Teachers from the United States and New Zealand rated the acceptability of treatments described within vignettes, which included a daily report card procedure, a response cost technique, a classroom lottery, and medication. The results revealed cultural differences in acceptability ratings, with teachers from the United States providing higher acceptability ratings than teachers from New Zealand. In addition, an interaction was found to occur with the gender and nationality of the student described in the vignettes and acceptability ratings, which was similar to findings by Pisecco et al. Some other instances of the BIRS being used to rate acceptability of various treatments are cueing procedures for children with ADHD (Posavac, Sheridan, & Posavac, 1999) and mnemonic instruction for students with learning disabilities (Scruggs & Mastropieri, 1989).
AARP Age/Gender of Client The impact of client age on treatment acceptability was evaluated by Elliott and Fuqua (2002) using the AARP. They evaluated the acceptability of four interventions (habit reversal, hypnosis, medication, and punishment)
Chapter | 4 Research on Social Validity
101
for treating trichotillomania. The study presented case vignettes to 239 college students in which the age of the client (8 years, 16 years, 26 years) and the severity of hair pulling (mild vs. severe) were manipulated. The results showed significant differences among the four treatments, with hypnosis and habit reversal being rated as the most acceptable interventions. The age of client and severity of trichotillomania did not significantly alter the ratings. Miltenberger and Lumley (1997) used the AARP to investigate the influence of treatments addressing the function of problem behavior on acceptability ratings. They randomly assigned 132 direct care staff to one of two conditions which described aggressive behavior maintained by attention or escape. They found that a time-out procedure was rated more acceptable than a guided compliance procedure, but the function of the problem behavior was not influential on treatment acceptability ratings.
TARF-R Reimers et al. (1992) conducted both analog and clinical examinations of 40 parent ratings of treatment acceptability using the TARF-R. Parents rated the acceptability of one of three treatments (positive reinforcement, time-out, or medication) for a case description of a fictional child displaying either mild or severe problem behaviors. In addition, parents were given written descriptions and provided demonstrations of positive treatment packages (e.g., verbal praise, differential reinforcement, token systems, etc.) which were recommended for implementation with their child. Parents rated the acceptability of the recommended treatment package prior to implementing the package and at 1-, 3-, and 6-month follow-ups. Findings of the analog evaluations revealed higher acceptability ratings for positive reinforcement and time-out when recommended for the mild problem behaviors and higher ratings for medication when recommended for the severe problem behaviors. The examination of influence of child problem severity on acceptability for the clinical ratings was conducted by dividing the parents into two groups based on ratings obtained from a problem behavior checklist. Findings from the clinical group indicated that parents of children with less severe problems rated the positive treatment packages more acceptable than did parents of children with more severe problem behavior; this held true on all ratings obtained except for the 3-month follow-up, which revealed no differences in ratings. Findings demonstrated similar ratings of treatment acceptability by parents across analog
102
Chapter | 4 Research on Social Validity
and clinical contexts. Additional analyses revealed that treatment acceptability ratings may influence treatment compliance and treatment efficacy.
SUMMARY The research on treatment acceptability has produced a large amount of information on variables which may influence how a treatment may be accepted. The importance of identifying the variables associated with high or low treatment acceptability lies in the fact that these variables may play an important role in the implementation of treatments, as well as the maintenance of behavior change. Several researchers have conceptualized or empirically demonstrated the importance of treatment acceptability. Wolf described how treatments that are not socially valid may cause clients to run away. Kazdin (1980b) discussed how low acceptance of treatment may deter clients from seeking out treatment, initiating treatment, or maintaining treatment protocols. Low treatment acceptability may reduce the likelihood that the treatment will be implemented with a sufficient level of integrity (Kazdin, 1980b). Conversely, high treatment acceptability may increase the likelihood that the treatment will be resistant to premature cessation (Kazdin, 1981; Kelley et al., 1989; Witt & Elliot, 1985; Witt et al., 1984). Overall, research on treatment acceptability has been conducted using several instruments, including the TEI or the IRP and variations of these instruments. The results of these studies have provided information on numerous variables that influence acceptability ratings. The variables which have been found influence treatment acceptability, through use of both the TEI and the IRP or variations of these instruments, include reinforcement versus punishment-based procedures (Kazdin, 1980a, 1980b; Kazdin et al., 1981; Witt & Martens, 1983; Witt, Elliott, et al., 1984), context in which the treatment is presented (Broughton & Hester, 1993; Cavell et al., 1986b; Power et al., 1995), the language used to describe the treatment (Singh & Katz, 1985; Witt, Moe, et al., 1984), previous treatment information (Clark & Elliott, 1988; Kalfus & Burk, 1989; Tingstrom et al., 1989; Von Brock & Elliott, 1987), the severity of the behavior problem (Elliott et al., 1984; Frentz & Kelley, 1986; Harris et al., 1990; Martens et al., 1985; Tingstrom , 1990; Von Brock & Elliott, 1987; Witt & Martens, 1983; Witt, Moe, et al., 1984; Witt & Robbins,
Chapter | 4 Research on Social Validity
103
1985), group membership (Fairbanks & Stinnett, 1997; Olive & Liu, 2005; Spreat & Walsh, 1994), and adult rating compared to child ratings (Elliott et al., 1987; Kazdin, 1981). Variables found to influence acceptability, measured by the TEI and subsequent variations of this instrument exclusively, include the apparent appropriateness of the treatment (Betts & Remer, 1993; Cavell et al., 1986a), side effects (Kazdin, 1981), restrictiveness of treatments (Spreat & Walsh, 1994), and parenting a child with ADHD (Gage & Wilson, 2000). The IRP and variations of this instrument have exclusively determined the following variables to be influential on acceptability: amount of time required to implement a treatment (Elliott et al., 1984; Martens et al., 1985; Witt, Elliott, et al., 1984; Witt & Martens, 1983; Witt, Martens, et al., 1984), the treatment mediator (Witt & Robbins, 1985), the gender of the client (Curtis et al., 2006; Pisecco et al., 2001), nationality of the client (Curtis et al., 2006), teacher experience (Witt, Moe, et al., 1984; Witt & Robbins, 1985), teacher nationality (Curtis et al., 2006), and location of raters’ high school (Stinnett et al., 2001). Several studies on the acceptability of treatments have produced contradictory results (see Table 4.3). These contradictory results appear to reveal the perpetual variability of the concept of treatment acceptability. There may be several possible reasons for these contradictory findings. Wolf (1978) pointed out the apparent discrepancies frequently reported between client-reported data and observer-obtained data. He stated three potential reasons for these discrepancies, which included the influence of situational contingencies, the lack of appropriate recording measures, and the inaccuracy of self-reporting capabilities. Situational contingencies, such as attempts to appear agreeable, fear of being ostracized, or coercion, might influence acceptability ratings. The use of inappropriate measurements could influence acceptability ratings when measures are defective or do not incorporate the behaviors viewed as important to raters. The inaccuracy of self-reporting capabilities was described by Wolf as the inability of humans to adequately judge changes in environmental situations. Some consistent findings have been revealed by the research on treatment acceptability, but much of the research is reflective of numerous inconsistencies and variations that make some of the findings difficult to interpret. These inconsistent findings may also be reflective of the unique variables associated with the individuals involved in these
104
Chapter | 4 Research on Social Validity
TABLE 4.3 Variables with Contradictory Findings Regarding Influences on Acceptability Severity of Client Problem Behavior as Influential Variable Significant Influence
No Significant Influence
Kazdin (1980a)
Witt et al., (1984)
Kazdin (1980b)
Witt et al., (1984)
Witt & Martens (1983)
Kutsick et al., (1991)
Elliott, Witt, Galvin, & Peterson (1984)
Spreat & Walsh (1994)
Witt, Moe, Gutkin, & Andrews (1984)
Elliott & Fuqua (2002)
Martens et al., (1985) Witt & Robbins (1985) Frentz & Kelly (1986) Miltenberger et al. (1989) Tarnowski et al. (1989) Preller & Graham (1990) Tingstrom (1990) Lindeman et al. (1992)
Side Effects of Treatment as Influential Variable Significant Influence
No Significant Influence
Kazdin (1981)
Spreat & Walsh (1994)
Treatment Mediator as Influential Variable Significant Influence
No Significant Influence
Witt & Robbins (1985)
Kalfus & Burk (1989) Tingstrom (1990)
Gender of the Client as Influential Variable Significant Influence
No Significant Influence
Pisecco, Huzinec, & Curtis (2001)
Kazdin (1980a)
Curtis, Pisecco, Hamilton, & Moore (2006)
Kazdin (1980b) Spreat & Walsh (1994)
Age of Client as Influential Variable Significant Influence
No Significant Influence Tingstrom, McPhail, & Bolton (1989)
105
Chapter | 4 Research on Social Validity
TABLE 4.3
(Continued) Spreat & Walsh (1994) Elliott & Fuqua (2002)
Experience Level of Teachers as Influential Variable Significant Influence
No Significant Influence
Witt & Robbins (1985)
Power, Hess, & Bennett (1995)
Witt, Moe, Gutkin, & Andrews (1984)
Professional Affiliation of Rater as Influential Variable Significant Influence
No Significant Influence
Spreat & Walsh (1994)
Epstein et al., (1986)
Fairbanks & Stinnett (1997)
Racial Affiliation of Raters as Influential Variable Significant Influence
No Significant Influence
Heffer & Kelley (1987)
Tingstrom & Silver (1990)
Rater’s Knowledge of Treatments as Influential Variable Significant Influence
No Significant Influence
Singh & Katz (1985)
Rasnake et al. (1993)
Clark & Elliott (1988)
Power, Hess, & Bennett (1995)
studies and the statistical procedures used to evaluate these idiosyncratic variables. Skinner (1969) explained the difficulty with using statistical procedures to examine individual variables: The complex system we call an organism has an elaborate and largely unknown history which endows it with a certain individuality. No two organisms embark upon an experiment in precisely the same condition nor are they affected in the same way by the contingencies in an experimental space. (It is characteristic of most contingencies that they are not precisely controlled, and in any case they are effective only in combination with the behavior which the organism brings to the experiment.) Statistical techniques cannot eliminate this kind of individuality; they can only obscure and falsify it (1969, pp. 111–112).
Much of the research on treatment acceptability has relied upon statistical procedures which may be less sensitive to the changes in
106
Chapter | 4 Research on Social Validity
individual variables. Skinner continued his explanation of examination of individuality: An analysis which recognizes the individuality of the organism is particularly valuable when contact is made with other disciplines such as neurology, psychopharmacology, and psychotherapy, where idiosyncratic sets of variables must also be considered. The rigor of the analysis is not necessarily threatened. Operant methods make their own use of Grand Numbers; instead of studying a thousand rats for one hour each, or a hundred rats for ten hours each, the investigator is likely to study one rat for a thousand hours. The procedure is not only appropriate to an enterprise which recognizes individuality, it is at least equally efficient in its use of equipment and of the investigator’s time and energy. The ultimate test of uniformity or reproducibility is not to be found in the methods used but in the degree of control achieved[—]a test which the experimental analysis of behavior usually passes easily. When effects on behavior can be immediately observed, it is most efficient to explore relevant variables by manipulating them in an improvised and rapidly changing design. Similar practices have been responsible for the greater part of modern science (1969, p. 112).
The difficulty with social validity research is that the effects on behavior may not be immediately observable. The changes in behavior are in many cases only observable after a lengthy experience with a particular treatment. Regardless, much in the way of social validity research could still be accomplished by examining some of the more immediate preferences for treatment, as has been recently demonstrated by Hanley, Piazza, Fisher, and Maglieri (2005). Including measurements similar to these may have incremental value in an overall examination of social validity. The different treatments examined, the varying contexts in which treatments were evaluated, and subtle differences among the raters in each of these studies could all be part of the reasons for contradictions in research findings. The use of several different instruments for measuring treatment acceptability, along with various statistical methods used to determine significance of findings, may also be a potential reason for these contradictions. The potential instability of influences on treatment acceptability or of the concept of treatment acceptability may also be related to contradictory research findings. There appear to be numerous reasons for the apparent contradictions that exist in the research on treatment acceptability, and these contradictions, along with consistent relevant influences, should continue to be examined to gain a better understanding of the concept of treatment acceptability.
Chapter 5
Increasing the Social Significance of Treatment Goals Consumer Significance Personal Preferences/Values Choice Habilitative Potential Awareness of Coercion Immediate versus Long-term Benefits Consultant Training Personal/Professional Agenda Goals Clarity and Specific Outcome Criteria of Goals Complexity and Comprehensiveness of Goals Potential for Actualizing the Goal Instrument/Method
Society Normalization Use of Available Resources and Funding Strategies to Boost the Social Validity of Treatment Goals Step 1—Incorporate Semistructured Interviews Step 2—Gather Information from Multiple Sources Step 3—Consider Different Aspects of the Individual’s Life Step 4—Rank Treatment Goals Hierarchically Step 5—Operationalize Expected Outcomes in Both Clinical and Social (Lay) Terminology
A socially valid treatment requires consideration of the goals, procedures, and outcomes associated with treatment. While each of these aspects of treatment plays an important part in attaining social validity, one of the most important aspects may lie in the quality, accuracy, and specificity of treatment goals. Without correctly identifying and aiming toward an important treatment goal, the chances of achieving a socially valid treatment may be greatly diminished. The purpose of this chapter is to provide an overview of some areas to consider when developing and Copyright © 2010 Elsevier Inc. All rights reserved
107
108
Chapter | 5 Increasing the Social Significance of Treatment Goals
evaluating the social importance of treatment goals. Strategies for identifying relevant treatment goals will be described, along with recommendations for increasing the potential social importance of these goals. Each of these areas will be described within a framework similar to the distributive model of treatment acceptability described by Carter (2008b). Wolf (1978) noted that in the development of socially significant treatment goals, recognition must be given to all relevant individuals or groups who may have a vested interest in the treatment. While each of the interested individuals or groups may express priorities that vary substantially, it appears necessary to gain an understanding of how these different factions may influence the development of treatment goals. Targeting socially significant treatment goals may require consideration of several different factors at various levels of influence. Carter (2008b) presented a model for examining treatment acceptability which delineated three broad areas of influence consisting of societal influence, consultant influence, and consumer influence. This same model may be useful in examining the various sources of influence on the social significance of treatment goals. Figure 5.1 displays a distributive model of the social significance of treatment goals which extends Carter’s model in an effort to develop an overall model of social validity. This distributive model of the social significance of treatment goals will be used to present the information in this chapter. The establishment of socially important treatment goals may require a period of pretreatment assessment. This type of assessment may take many forms and could be a rather lengthy process, depending on the complexity of the problems. In any case, a good rule to remember when considering various treatment goals is the principle of parsimony. While complex goals may be impressive, typically the most important goals are those that are simple and concise. The hallmark of applied behavior analysis has been the use of direct observations of clearly definable and observable behaviors. The use of observation procedures may be a very important part of identifying socially important treatment goals. Observational assessment can take several forms, such as narratives, antecedent-behavior-consequent (ABC) recording, frequency counts, interval recording procedures, and others. Each of these direct observational procedures may have value for
Societal significance
Significance of goals
Consultant significance
Consumer significance
FIGURE 5.1
Consultant Training Personal agenda Professional agenda Habilitative potential Similarity to goals in comparable cases • Normalization • • • • •
• • • • • • •
Society Maintain/Increase conformity Normalization Funding Resources
Goal Clarity Outcome criteria Complexity Potential for actualizing the goal • Comprehensiveness • • • •
Consumers Personal preferences Input on selection Choice Habilitative potential Awareness of coercion Normalization Immediate vs. long-term benefits
Instrument/Method • Validity & reliability • Ease of implementation • Specificity to group • Clarity of items
Chapter | 5 Increasing the Social Significance of Treatment Goals
• • • •
Distributive model of treatment goals . 109
110
Chapter | 5 Increasing the Social Significance of Treatment Goals
determining socially important treatment goals, but consideration should be given to the time and effort required to conduct these types of observations for the purpose of assessing important treatment goals. Some of these observation procedures may be most beneficial for closely monitoring behavior change rather than for determining socially important treatment goals. Measures other than direct observation, such as ratings, interviews, questionnaires, and the like, can also be valuable sources of information (Schwartz & Olswang, 1996). When conducting any type of assessment, the purpose of the assessment should always be considered in order to determine what is and is not potentially valuable. Selecting important treatment goals may frequently involve examining data from narratives or ABC recordings, but in many situations professional judgments become necessary for selecting treatment goals in a timely manner. Macmann and Barnett (1999) offer some strategies for reducing errors when using professional judgments. While they explain that most studies involving professional judgments have focused on diagnostic decision making, the recommendations may still be useful when applied to treatment goal selection. Their recommendations include examining a number of conceivable hypotheses, offering a wide range of potential interventions to consumers, using decision aids rather than relying on memory, and accepting uncertainties by describing to consumers variability in potential outcomes. Using their recommendations could improve the use of professional judgment when selecting treatment goals that have potential social significance. Additionally, Schalock and Luckasson (2005) offered a highly comprehensive set of guidelines for making clinical judgments in the field of mental retardation. They indicated that clinical judgment was rooted in a high level of expertise on a topic, extensive experience, and availability of considerable data on a subject. They pointed out that clinicians in the field of mental retardation were required to make a number of clinical judgments and that the future of the field suggested an increase in the need to make clinical judgments. They differentiated clinical judgment from ethical judgments and professional judgments, which focused more on evaluating values and following professional guidelines, respectively. According to Schalock and Luckasson, clinical judgments were characterized as “being systematic (i.e., organized, sequential, and logical), formal (i.e., explicit and reasoned), and transparent (i.e., apparent and communicated clearly)” (p. 1).
Chapter | 5 Increasing the Social Significance of Treatment Goals
111
They provided six clinical judgment strategies, as follows: (1) conducting a thorough social history; (2) aligning data and its collection to the critical question(s) at hand; (3) applying broad-based assessment strategies; (4) implementing intervention best practices; (5) planning, implementing, and evaluating individualized supports; and (6) reflecting cultural competence and linguistic diversity. A thorough social history allows for increased understanding of an individual’s past, present, and potential future experiences. Aligning data to critical questions at hand involves eliminating unnecessary information and focusing on the essential questions that need to be answered. Applying broadbased assessment strategies includes incorporating a number of nontraditional assessment techniques to address the individual differences of clients, such as verbal capacities, visual impairments, and so forth. Implementing intervention best practices relates to knowledge of the current research base and alignment with the new perspectives and conceptualizations in the field. Planning, implementing, and evaluating individualized supports concerns the areas where supports can be implemented, such as independent living, socialization, medical, and so on. In addition, the functions of specific supports are considered, such as making friends, maintaining a home, accessing health care, and so forth, as well as the personal outcomes related to these functions, such as number of friends obtained and improvement in physical health. Reflecting cultural competency and linguistic diversity involves recognizing the degree to which an individual’s abilities or disabilities can be clearly differentiated from other aspects of the individual (such as cultural practices, language barriers, etc.). Each of these strategies offers a framework for conducting clinical judgments, and consideration of these strategies could be beneficial when deciding on treatment goals that will likely have a high degree of social significance.
CONSUMER SIGNIFICANCE Personal Preferences/Values Developing socially significant treatment goals based on information at the level of the consumer may rely upon obtaining numerous types of input from the consumer. At this level the individuality of the person targeted for treatment becomes most important. Gathering information
112
Chapter | 5 Increasing the Social Significance of Treatment Goals
directly from the individual targeted for treatment, or from those directly connected to the consumer, may involve asking questions to gain an understanding of what is important to the individual. Hawkins (1979) described asking consumers for feedback about potential or existing treatment goals as a measure of interobserver agreement that can be useful for designing treatments are appropriate to the situation and the primary concerns of the consumer. He also pointed out that this was a form of social validation for treatment goals. Values include the expectations that people have regarding the behavior of themselves or others. Carr (1996) explains that while science provides the strategies for changing behavior, values provide the foundation for determining what is worth changing. The goals of treatment which are selected may reflect the values of the consumer, the consultant, or segments of society. Consumers may have values which are ingrained in them by culture, religion, ethnicity, and so on. These consumer values may influence the willingness or reluctance to agree to a particular treatment goal. Some religious and cultural groups actively promote a philosophy of “spare the rod, spoil the child,” which emphasizes physical punishment practices for children as a method of child rearing, or ideologies such as “children should be seen and not heard.” Consumers who adhere to such values may be more likely to accept treatment goals which focus on restricting a child’s behavior to a very limited range of behaviors reflective of obedience. These types of values may enhance a willingness of consumers to accept treatment goals which minimize creativity on the part of the child and minimize the child’s opportunities to experience failure. In contrast, other consumers may adhere to values which promote freedom of expression and experiential learning. These consumers may be more willing to accept treatment goals which focus on increasing a large number of behaviors which allow for more creativity and for more opportunities to experience failure. Rutzen, in a study on the social importance of treating malocclusion, summed up the need to consider the degree to which a physical deformity deviates from a typical population in his statement: “If a defect is not readily discernible by others, it is less likely to affect social interaction and thereby role performance and the attainment of socially desirable goals” (1973, p. 239). This statement appears applicable toward considering the social importance of the goals of a behavior intervention.
Chapter | 5 Increasing the Social Significance of Treatment Goals
113
Selection of specific behaviors to target for change should include consideration of the degree to which the targeted behaviors are deviant from typical populations. Developing a behavior intervention which focuses on a behavior which is not considered socially important by others may lead to socially insignificant changes or possibly behavior changes which could be considered socially deviant from normative groups. A behavior intervention which targets in-seat behavior for a child in a classroom may be highly desirable, but consideration should be given to what is considered a reasonable amount of inappropriate in-seat behavior by first observing the in-seat behavior of other students in the classroom. Similarly, a behavior intervention which aims to improve the on-task academic behavior of a child should consider the quality of performance and the resulting decrease in other nonacademic activities. A very high level of on-task academic behavior may be beneficial for academic endeavors, but might be considered socially deviant when examined within the context of certain normative groups. The values of consultants may come from sources similar to those of consumers, but might also encompass the values outlined by professional organizations, employers, or training programs. In many cases, consultant values may differ from personal to professional values and may not always concur with the values of the consumers they serve. Consultants may have to balance their values when establishing treatment goals aligned with the values of their consumers. For example, a consultant may want to establish a treatment goal for a child in which the child does not engage in any fighting with peers. Parent consumers might not value such a goal if they value aggressive behavior in order to accomplish tasks and feel that their child may need to fight for what he or she wants. In cases such as this, it is up to the consultant to find a middle ground that ensures that the child will be able to defend himself or herself and avoid violating the rights of others. For additional examples, see Bailey (1987), who provided several examples of values conflicts occurring with early childhood services. Selection of treatment goals that have social significance may rely upon the personal perspective of the various individuals evaluating the treatment goals. Lucyshyn and Albin (1993) recommend incorporating goals that are directed toward specific individuals closely connected to the individual targeted for treatment. These same individuals would eventually
114
Chapter | 5 Increasing the Social Significance of Treatment Goals
be the ones who evaluate the social validity of the program. Their suggestions involve inclusion of both child-focused goals and family-focused goals when designing behavior support for families. Goals for family members may address issues such as respite care and recruiting or training care providers. These family goals can also be highly specific, such as indicating that a family member (such as a father, sister, aunt) will learn new skills such as sign language. Their approach appears highly comprehensive in an effort to address all areas where behavior support may be relevant to promote behavior change through lifestyle modification. Consumers, consultants, and broad segments of society may differ in the social perspectives related to treatment goals. The opinions of individuals from each of these categories may not be in agreement, but they may provide some insight into how certain treatment goals might be appreciated by these various groups. In addition, obtaining information from these various groups might be beneficial for modifying treatment goals in a manner that could be somewhat appealing for individuals from more than one social category. Dardig and Heward (1981) offered three goals for improving the effectiveness and efficiency of communications among team members involved in developing Individualized Education Programs (IEPs). They suggested that each team member’s opinion be offered equal weight and consideration, that a process for resolving disagreements be available, and that the process be documented in an efficient manner. These suggestions are an effort to organize team meetings in a way that makes them more cooperative and less competitive. A consultant may derive information from consumers through interviews, rating scales, and other means and then use this information to determine treatment goals. This method may be sufficient in many cases, and may accurately reflect the wants and needs of the consumer. In other cases the consultant may need to seek out additional input from consumers prior to selecting treatment goals. This may involve reviewing with the consumer all of the information collected and seeking clarification on issues that are unclear before choosing specific treatment goals. This type of follow-up prior to treatment-goal development may in some cases prove beneficial in more accurately defining specific treatment goals that are highly relevant for the consumer. Identifying consumer preferences regarding treatment goals may be accomplished by conducting interviews with the consumers or with
Chapter | 5 Increasing the Social Significance of Treatment Goals
115
those highly familiar with the consumer, but a behavior consultant may need to filter some of the information that is provided during interviews. The information acquired from interviews needs to be validated by comparing the information with interviews from others highly connected with the individual targeted for treatment. The PBS approach offers a general protocol for the development and delivery of services that promotes the involvement of others beginning with the initial referral. Hieneman et al. (1999) described the PBS protocol as including four steps as follows: (1) organization of a team, (2) functional assessment of problem behavior, (3) development of a behavior support plan, and (4) ongoing assessment of the intervention. This protocol begins with formalizing a team of individuals who can offer multiple perspectives and from which some validation of the significance treatment goals might be offered through the group selection of the goals. This type of team development approach might be considered time intensive, but previous examinations of the involvement of parents in service delivery versus a clinician-only delivery model has found that parent involvement is a time-efficient and cost-effective model for providing intervention services (Schreibman, Koegel, Mills, & Burke, 1984). Lucyshyn and Albin (1993) pointed out that when developing comprehensive family behavior support plans, the input from family members should always be valued; expressions of emotions may be cathartic for the family members and should not be considered immaterial. They note that if the views of families are not recognized and validated, future problems may evolve in the consultation relationships. Person-centered planning is a strategy frequently used for program planning and development of treatment goals that involves a team of individuals (Kincaid, 1996; Smull & Harrison, 1992; Vandercook, York, & Forest, 1989). These team members attempt to identify opportunities for the individual targeted for treatment to work toward achieving his or her wants and needs. Kincaid described five essential goals of personcentered planning that included: (1) participating in community life, (2) gaining and maintaining satisfying relationships, (3) expressing preferences and making choices, (4) fulfilling respected roles in society, and (5) continued development of personal competencies. In addition to identifying opportunities for personal development, the process identifies barriers and promotes short- and long-term treatment goal development.
116
Chapter | 5 Increasing the Social Significance of Treatment Goals
The goodness-of-fit approach for behavior service development and delivery has frequently been used for parents of young children (Bailey et al., 1990; Thomas & Chess, 1977). This model recognized the uniqueness of individual families and their children when selecting specific treatments that will match their lifestyles and personal preferences. Bailey (1987) recommended using collaborative goal setting to avoid conflicts between parents and professionals. He described five requirements necessary to implement collaborative goal setting with families, as follows: (1) take a systems perspectives point of view, (2) systematically assess family needs, (3) use effective listening and interviewing techniques, (4) negotiate values to reach a joint solution, and (5) help families match needs with available resources. Albin, Lucyshyn, Horner, and Flannery (1996) described a similar contextual-fit model for attaining goodness of fit in the development of behavior support plans. Good contextual fit was described as compatibility between specific features of a behavior plan and three classes of variables that included consumer variables, plan implementer variables, and environmental variables. Albin et al. described how the absence of good contextual fit may only be discovered after a behavior plan is unsuccessfully implemented, but noted that this could potentially be avoided by gathering additional information during the development of a behavior plan. They offered a Family Assessment Interview Protocol that was designed to provide an understanding of the ecology of a family prior to developing a behavior plan. The protocol consists of several questions within three categories, as follows: (1) family characteristics, (2) family social construction of child and activity settings, and (3) family vision of successful activity settings. In addition to the family interview, Albin et al. recommended using a Goodness-of-Fit Survey after a behavior plan has been developed but prior to implementation of the plan and then periodically during plan implementation. The Goodnessof-Fit Survey consists of 20 questions rated using a five-point Likert-type scale. The questions focus around how well the behavior plan recognizes the needs of the family, matches their values, and fits into their routines. Each of the instruments recommended by Albin et al. provides a structured method for assessing the personal preferences and values of consumers, rather than relying on informal comments or conversations with consumers. Another example of a worksheet for evaluating the social significance of target behaviors is provided by Cooper, Heron, and Heward (1987).
Chapter | 5 Increasing the Social Significance of Treatment Goals
117
Their worksheet consists of 10 questions that are answered either “Yes” or “No” and a place for comments. The questions focus on aspects of potential target behaviors such as whether the behavior will provide more opportunities for reinforcement, whether the target behavior will promote more advanced skills, whether the target behavior will really help the client, and so on. The use of each of these instruments fits with the recommendations by Macmann and Barnett (1999) for reducing error in professional judgments and provides a framework for ensuring that socially important treatment goals are identified. Table 5.1 provides an example of a checklist that examines each of the areas suggested for consideration when planning for socially significant treatment goals. Section one of the checklist focuses on consumer preferences and values.
Choice The opportunity to choose is another means of allowing consumer input and potentially enhancing the significance of treatment goals. Some consumers may verbally express a choice of treatment goals when provided with potential options. In these cases it may be beneficial to validate these choices by repeating the opportunity to choose on more than one occasion and comparing these choices. Consistency in choice of treatment goals would appear to promote more significant treatment goals. The need to prioritize treatment goals is an important aspect of choice that involves gathering and reviewing information. Consumers may not rely on reviews of information when making choices about treatment goals, but if they are provided with a structured method for prioritizing treatment goals they may find the information useful. The use of a memory aid may be valuable for this purpose and for making more informed choices regarding treatment goals, and may reduce errors in professional judgment, as suggested by Macmann and Barnett (1999). Schalock (2001) offered five recommendations to guide the selection of outcome measures utilized by organizations. The recommendations were as follows: (1) select outcomes that are within the organization’s mission statements, (2) select a minimum number of outcomes that are relevant and obtainable, (3) select outcomes that are within the organization’s evaluation capabilities, (4) select outcomes that reflect age- and functional-level differences, and (5) select outcomes that the organization
118
Chapter | 5 Increasing the Social Significance of Treatment Goals
TABLE 5.1 Checklist for Exploring the Social Significance of Treatment Goals Section One: Consumer Preferences/Values
Yes
No
Comments
Yes
No
Comments
Does the treatment goal match the personal preferences of the consumer? Does the treatment goal match the values of the consumer? Is the consumer comfortable with the potential outcomes related to the treatment goals? Are there additional treatment goals that need to be addressed? Is this treatment goal the most important need of the consumer? Is this treatment goal reasonably attainable? Do the treatment goals involve preferred activities of the consumer? Do the treatment goals involve preferred settings/ environments of the consumer? Do the treatment goals involve other individuals valued by the consumer? Are the treatment goals supportive of other goals of the consumer? Are the treatment goals supportive of the longterm goals of the consumer?
Section Two: Normalization Do the treatment goals increase opportunities for normalization? Are treatment goals focused on age-appropriate activities/behaviors? Do treatment goals focus on multiple life areas, such as home, school, work, family, friends, etc.? Do treatment goals promote skills frequently displayed by peers in typical environments? Do treatment goals support existing skills needed for normalization? Do treatment goals address requirements/ expectations for participation in multiple settings?
119
Chapter | 5 Increasing the Social Significance of Treatment Goals
TABLE 5.1 (Continued) Section Three: Choice
Yes
No
Comments
Yes
No
Comments
Yes
No
Comments
Do the treatment goals promote opportunities for choice? Will the treatment goals potentially generate high rates of reinforcement? Will the treatment goals potentially generate high-quality reinforcement? Will the treatment goals allow for immediate reinforcement? Do treatment goals reduce the response effort associated with acquiring reinforcement?
Section Four: Habilitative Potential Do the treatment goals provide opportunities for developing new skills? Do the treatment goals promote new social skills? Do the treatment goals promote new work/ educational skills? Do the treatment goals promote new leisure skills? Do the treatment goals promote skills necessary to access new environments? Do the treatment goals focus on promoting shortterm competencies? Do the treatment goals focus on promoting longterm competencies through generalization and maintenance?
Section Five: Awareness of Coercion Do the treatment goals clearly denote that the goals can be changed? Do the treatment goals promote the skills necessary to refuse treatment? Do the treatment goals incorporate consumerdeveloped strategies/techniques? Do the treatment goals allow consumers to negotiate specific aspects of treatment? Do the treatment goals allow consumers to seek a compromise regarding treatment?
120
Chapter | 5 Increasing the Social Significance of Treatment Goals
is committed to follow over time. These recommendations could be useful if adapted to improve measurement of the social significance of treatment goals. When adapted for selecting socially significant treatment goals, the recommendations might include the following: 1. Select treatment goals that are consistent with the consumer’s values, preferences, lifestyle, and so on. 2. Select a minimum number of treatment goals that have high relevance and achievability. 3. Select treatment goals that can be evaluated by the consultant. 4. Select treatment goals that recognize the individual characteristics and abilities of the consumer. 5. Select treatment goals that are important enough to follow over an extended period of time. Dardig and Heward (1981) described a six-step process for prioritizing annual Individualized Education Program (IEP) goals using a team-based approach. They offered nine criteria for prioritizing these annual goals, which are responded to using a five-point Likert-type scale. The six-step, team-based process is as follows: (1) team member introductions, (2) listing and discussion of possible goals, (3) determination of criteria for prioritizing goals, (4) individualized team member ratings, (5) synthesis of individual responses, (6) prioritized list of annual goals. Cooper et al. (1987) provided a matrix for prioritizing target behaviors which allows each target behavior to be rated according to prioritization criteria which can be differentially weighted to meet the needs of different situations. The ratings can then be tallied and target behaviors ranked from highest to least priority. This type of matrix may also be beneficial for prioritizing treatment goals and promoting discussions of choice related to treatment goal selection. With this considered, a matrix for prioritizing treatment goals was developed that can be used to rank potential treatment goals (see Table 5.2). This Treatment Goal Prioritization sheet can provide a total score across eight questions focused on prioritizing treatment goals rather than only target behaviors. The total scores for each potential treatment goal can then be compared and ranked according to the highest total score, with higher total scores indicating higher priority consideration. Matching law has explained how choice or response allocation occurs relevant to the accumulation of the most available reinforcement
Treatment Goal Prioritization
Rate your agreement/disagreement with each of the questions below for the following treatment goal. Treatment Goal: ___________________________________________________________________________________________ Strongly Disagree
Slightly Disagree Disagree
Slightly Strongly Agree Agree Agree
1. This is the best treatment goal that could be chosen.
1
2
3
4
5
6
2. This treatment goal focuses on the most important issues.
1
2
3
4
5
6
3. This treatment goal increases opportunities to engage in activities that may currently be limited.
1
2
3
4
5
6
4. This is a reasonable treatment goal to accomplish.
1
2
3
4
5
6
5. This treatment goal would not have bad side effects.
1
2
3
4
5
6
6. This treatment goal will increase opportunities for reinforcement.
1
2
3
4
5
6
7. This treatment goal will promote other needed skills.
1
2
3
4
5
6
8. This treatment goal is needed more than most other goals.
1
2
3
4
5
6
Chapter | 5 Increasing the Social Significance of Treatment Goals
TABLE 5.2
121
122
Chapter | 5 Increasing the Social Significance of Treatment Goals
(Herrnstein, 1961, 1970). Mace and Roberts (1993) described some factors that could influence choice, such as rate, quality, and immediacy of reinforcement. In addition, they explained that response effort and combinations of factors may influence choice. Considering these factors as they apply to consumers’ choice of treatment goals may be a valuable means for increasing the significance of treatment goals. While each of these factors (rate, quality, immediacy, response effort) may only be accurately measured through direct observation, it may be possible in some cases to predict the potential influence of each of these factors. For example, a treatment goal that focuses on increasing a leisure skill activity such as snow skiing might provide high-quality reinforcement for the consumer, but, depending on the availability of conditions such as snow, the rate of reinforcement may be rather lean unless other alternative outlets can be offered, such as magazines on skiing, video game skiing, and the like. In addition, it might be difficult to provide immediate reinforcement for a treatment goal related to increasing snow skiing, and the response effort associated with such a goal could be great. Treatment goals that exploit the factors considered to influence choice would likely increase the significance of the goals and possibly the overall effectiveness of the procedures. Several articles that focus on treatment goals of increasing communication through the use of functional communication training (FCT) have recommended manipulating the factors considered to influence choice. These articles generally recommend selecting an FCT response that can be immediately recognized and reinforced at high rates with high-quality reinforcement. They also suggest developing an FCT response that is easy to display, so as to reduce the response effort associated with the response. These same principles may be applicable to incorporating consumer choice into treatment goal selection. Section three of the Checklist for Exploring Social Significance of Treatment Goals (Table 5.1) focuses on consumer choice.
Habilitative Potential The need to select treatment goals which promote the incorporation of new useful skills appears highly important, especially when these new skills may result in access to new sources of reinforcement for the consumer. Hawkins (1991) referred to behaviors that maximize occasions
Chapter | 5 Increasing the Social Significance of Treatment Goals
123
for reinforcement and minimize opportunities for punishment as having habilitative potential. Schalock and Kiernan (1990) provided five criteria for determining what to measure when conducting an outcome analysis of habilitation services. Their recommendations were as follows: (1) attributed to service, (2) sensitive to change and intervention, (3) obtainable, (4) objective, and (5) prioritized. These recommendations appear to be important because they can be sensitive to the personal preferences of the consumers and the evaluation may be easily conveyed to consumers. Modifying Schalock and Kiernan’s criteria to organize and evaluate social validity data, the recommendation may be as follows: 1. 2. 3. 4. 5.
Goal attainment can be attributed to treatment. Goals are specific and sensitive to behavior changes. Goals are achievable. Goals are quantifiable. Goals are prioritized.
In order for consumers to appreciate the significance of treatments, it is important for these consumers to recognize, when appropriate, that the treatments were responsible for achievement of new habilitative skills. This recognition should influence the posttreatment social validity when the information is clearly conveyed to the consumers. Goals that are specific and sensitive to behavior change may be more clearly conveyed to consumers during selection of goals and measurement of the effects of treatment. Selecting goals that can be quantifiably measured may increase the specificity of the goals and improve estimates of achievability of goals. All of these factors, along with consumer preferences, should be considered when prioritizing goals, and when weighed appropriately should promote the social significance of the goals. The habilitative potential of treatment goals should also involve examination of possible long-term and short-term habilitative effects. Horner, O’Neill, & Flannery (1993) recommended some areas where measurement of long- and short-term habilitative effects could occur in an effort to increase the validity of behavior treatments. These included the following: (1) measures of physical integration, (2) measures of social integration, (3) measures of variety of activities, (4) measures of the availability of preferences or choice, and (5) measures of the social roles
124
Chapter | 5 Increasing the Social Significance of Treatment Goals
held by a consumer. Each of these recommendations attempts to quantify lifestyle changes and could be highly related to changes in quality of life. Horner et al. recommended including these types of measures in an effort to demonstrate potential indirect influences, even if only simultaneous rather than causal relationships could be demonstrated. This recommendation would expand the number of environmental variables included in a behavior program and make these programs more comprehensive by manipulating numerous variables across an increased number of settings, activities, life areas, and so forth. Hawkins (1991) has stressed the importance of social comparison of habilitative skills, whereby the performance of skills targeted for treatment are viewed in relation to these same skills performed by others considered to be highly competent in performing these skills. Horner et al. (1993) have recommended that these comparisons should focus on skill sets or activities in which the individual targeted for treatment can engage in the activity in a manner that is functionally equivalent to how the activity typically functions for others in the same environment. The function of the activity is considered to be more important than the manner in which the activity is performed. This focus on function promotes generalization and maintenance of skill sets by ensuring that the skill sets are useful, expected in specific environments, and come in contact with existing contingencies of reinforcement, and there are frequent opportunities to display the skill sets (Carr, 1980; Fox, 1989; Horner, Williams, & Knobbe, 1985; Stokes & Osnes, 1988).
Awareness of Coercion In many cases consumers may experience some form of coercion to work on a particular goal. This type of influence on the social significance of treatment goals may be quite obvious, or might occur in less conspicuous manners. Skinner (1953) described counter-control as the degree to which an individual may resist or avoid a treatment. Consumers who do not have the capability of choosing to participate, choosing not to participate, or choosing to partially participate in a treatment have a decreased degree of counter-control. Consumers who are fully informed of their right not to participate in a treatment are considered to have a higher degree of counter-control. While this is reasonable, studies by Milgram (1974) have shown that other factors associated with
Chapter | 5 Increasing the Social Significance of Treatment Goals
125
the consultant can also be highly influential toward reducing the consumer’s likelihood of choosing to refuse treatment. In the studies by Milgram (1974), the subjects of the treatment were presented with a scenario where they were to administer a shock of increasing intensity to a person who did not answer questions correctly. The person answering the questions was a confederate in another room who could not be seen but only heard by the subject. During the experiment, the experimenter told the participants that the experiment required them to continue whenever the subject protested against continuing. In order to discontinue the experiment, the subjects would have to be very assertive in their requests not to participate. This type of covert coercion has been reportedly used by telephone and Internet service providers who make their services readily available and state that services can be discontinued at any time upon request. Many consumers of these services have reported that when they attempt to discontinue the services, the sales representative made the process very difficult by ignoring their requests, offering other options, repeatedly transferring their call, applying cancellation fees, and so on. The consumers of these services have in many cases had to be very assertive and determined to eventually have their services cancelled. An example of overt coercion that is frequently experienced by parents of preschoolers surrounds the issue of potty training. While most all children eventually become competently potty trained, parents are often pushed into training their child because of requirements placed upon them by day care centers, child care providers, or others who insist that the child be potty trained prior to receiving these types of services, and in many cases may be prohibited from continuing in a day care setting if the child is found to not be competently potty trained. This type of overt coercion may cause parents to push a child into potty training activities that the child and parent may find extremely frustrating. Similarly, parents of children who displayed hyperactivity have reported being told directly that their child could not continue attending a classroom unless the child was provided with medication to address the hyperactivity. In other cases, parents have reported that school personnel have implied that their child could not continue to attend the classroom unless provided with medication. Carey and Bourbon (2004, 2005) have described several strategies for reducing the amount of counter-control associated with treatments in
126
Chapter | 5 Increasing the Social Significance of Treatment Goals
classroom settings. Their strategies included asking students how they would like something to be, determining student preferences regarding treatments, allowing students to choose treatments, incorporating opportunities for students to negotiate treatments, and allowing for compromise with treatments.
Immediate versus Long-term Benefits The potential benefits of treatment goals should be evaluated as to their long- and short-term gains for the individual. Consumers may value more immediate results and therefore be more concerned with goals that are readily attainable in the short term. Treatment goals that only offer benefits after a long period of treatment, but have a lasting effect, may be more acceptable in situations where an undesirable behavior has been present for a long period of time. In such cases the behavior would have a long history of reinforcement and may be highly resistant to treatment. Consumers may be willing to accept a treatment goal that lacks immediate effects if they feel the eventual benefits will be worth the time, effort, and wait involved. The ideal treatment goal would provide both short- and long-term benefits. These types of goals resolve immediate concerns and offer lasting support over an extended period of time.
CONSULTANT Training Consultants need to have a variety of competencies, including training paraprofessionals (Barnard, Christophersen, & Wolf, 1974; Neef, 1995; Phillips & Mudford, 2008; Wood, Luiselli, & Harchik, 2007), training parents (Anan, Warner, McGillivary, Chong, & Hines, 2008; Gross, Miltenberger, Knudson, Bosch, & Brower Breitwieser, 2007; Mueller et al., 2003), training teachers (Moore et al., 2002), training graduate students (Iwata et al., 2000), working with principals (Gillat & SulzerAzaroff, 1994), and using technology in training (Macurik, O’Kane, Malanga, & Reid, 2008; Wallace, Doney, Mintz-Resudek, & Tarbox, 2004). Not all consultants have these experiences, as not all training programs provide these types of opportunities. When consultants are faced with a situation that falls outside their level of expertise, they are
Chapter | 5 Increasing the Social Significance of Treatment Goals
127
ethically obligated to seek assistance from someone who can supervise their efforts, seek out additional training, or refer the case to someone with the appropriate background to provide the services. A consultant who has limited training will have a limited array of skills to offer consumers. While all consultants have limitations, it appears beneficial for a consultant to continually acquire new skills through a variety of training experiences. Consultants who persistently attain new skills will have more to offer consumers and can potentially influence the social validity of a treatment program by providing consumers with more options. Having more options will allow the consultant and consumer to work more closely toward developing a treatment that can fit well into the environment and can be easily implemented. In addition, consultants who are familiar with various strategies for training consumers, such as utilization of technology or working with principals, may increase social validity by promoting more highly trained treatment mediators who can implement treatments more efficiently.
Personal/Professional Agenda Consultants need to have an awareness of both their personal and their professional agendas when selecting treatment goals for consumers. In some cases, certain treatment goals may be more interesting for the consultant than other treatment goals. Consultants may be inclined to promote these treatment goals among consumers because of their personal interests, even though these goals may not be the highest-priority goals for the consumer. Consultants need to understand that working on particular treatment goals may be highly interesting and may allow for some unique experiences but may not be in the best interest of certain consumers. The consultant can deal with these personal agendas by carefully selecting cases that offer opportunities to work on these specific types of treatment goals. For example, a consultant who is highly interested in and specializes in self-injurious behavior might seek out these types of cases from among numerous referrals, and refer other cases to other professionals. In situations where a consultant does not have the option to pass on certain referred cases, the consultant may want to be certain to use the Treatment Goal Prioritization sheet (Table 5.2) as a self-check against personal bias in treatment goal selection.
128
Chapter | 5 Increasing the Social Significance of Treatment Goals
GOALS Clarity and Specific Outcome Criteria of Goals Treatment goals should be precisely worded so that they can be clearly understood by everyone who is closely involved with the consumer, and the goals should be easily communicated to others in the community. In addition, treatment goals should define specific outcome criteria which will allow successful movements toward reaching the goal to be readily recognized. Much of what has been written on writing behavior objectives can be easily applied toward the development of clear treatment goals. Treatment goals should be developed in a manner that offers a clear definition of the behaviors that are involved in achieving the goal. Goals that are too vague may promote confusion among those involved. Vague treatment goals result in some people believing that the goals have been achieved while others may not consider the goals to have been achieved.
Complexity and Comprehensiveness of Goals When selecting treatment goals, consideration should be given to the amount of time and effort that will be required to achieve the goals. A consultant needs to achieve a balance between treatment goals that are comprehensive enough to achieve important treatment effects, but must also consider the difficulty related to implementing a comprehensive plan. Lucyshyn and Albin (1993) recommended considering the support strategies that are available or may be provided to ensure the implementation of comprehensive support plans. Their seven implementation support strategies were: 1. 2. 3. 4. 5. 6. 7.
written descriptions of intervention procedures, meetings with family members to develop home-based interventions, user-friendly implementation checklists, behavioral rehearsal of specific interventions, work group meetings to develop materials, direct coaching of parents, and follow-up telephone consultation.
Determining how much support can be provided during implementation and the competencies of those individuals responsible for implementing
Chapter | 5 Increasing the Social Significance of Treatment Goals
129
the procedures may be an important factor in determining if a treatment goal is too complex. In some cases where complex or comprehensive treatment goals are necessary, it may be advisable to break the treatment goals down into smaller steps that can be more readily accomplished.
Potential for Actualizing the Goal While it may be difficult or impossible to know the likelihood of a treatment goal being accomplished, it is still somewhat necessary to speculate and focus on goals that are achievable. Treatment goals that can be realized may promote acceptance of additional goals, while treatment goals that are extremely difficult to achieve may lead to consumer reluctance. Wolf (1978) noted that if consumers are not accepting of the services that they receive, they may express this by not accessing these types of services in the future. Considering the actualization potential of a treatment goal might include examination of a number of different factors (see Table 5.3). One factor that might need consideration is the history of reinforcement associated with the behaviors targeted for change. This has to do with the length of time that a behavior has been occurring and the type of
TABLE 5.3 Factors in Determining the Actualization Potential of a Treatment Goal Factors to Consider when Evaluating the Actualization Potential of a Treatment Goal History of reinforcement Success of previous treatments Level of treatment mediator training Amount of time available from staff/consultant Level of functioning of the consumer Number of supportive/adaptive skills in the repertoire of the consumer Number of steps necessary to reach goal Amount of supervisory-level support
130
Chapter | 5 Increasing the Social Significance of Treatment Goals
reinforcement schedule that has been in place to maintain the behavior. A behavior that has occurred for a short period of time and is maintained by a near-continuous schedule of reinforcement might respond quickly to an appropriate behavior intervention. A behavior that has occurred for several years and has been maintained by a variable reinforcement schedule may be less likely to respond quickly to a behavior intervention. Considering the success of previous treatments may provide some indication of how future treatments may progress. If previous treatments have produced only gradual changes in behavior, it may indicate that the behavior has some resistance to change and future treatments may require more time to achieve the degree of behavior change desired. When considering previous treatments, it is always important to note the degree of treatment integrity that was present. There are potentially many cases where it has been noted either verbally or within paperwork that a treatment was being provided, but the integrity of the treatment may have been poor. Another factor to consider when speculating on the actualization potential of treatment goals may be the level of training of the treatment mediators. Training of treatment mediators can in some cases be accomplished rather quickly if they are familiar with similar procedures. If the treatment mediators need an extensive amount of training to implement a treatment, this may slow down the implementation of a treatment program and may result in treatment integrity issues that have to be repeatedly addressed through retraining activities. Associated with this is the amount of time that the treatment mediators have to devote toward training on a treatment program, as well as the time available for the consultant. Scheduling times to conduct training sessions between a consultant and a treatment mediator could result in delays in program implementation and the actualization of a treatment goal. Another factor that might be relevant to the actualization of treatment goals may be the level of functioning of the consumer targeted for behavior change. A consumer who has severe deficits may require more time to achieve a goal, and the goals that are set for these individuals need to be carefully selected so as not to set goals that are too easily attainable or too difficult to attain. Considering the number of adaptive skills that a consumer has which can support the behaviors encompassing the treatment goal may be a way to judge the potential for actualizing a goal. The number of steps involved in achieving a goal should be carefully considered and
Chapter | 5 Increasing the Social Significance of Treatment Goals
131
matched to the number of steps that a consumer can potentially follow, and all the steps related to achieving a goal should be identified to avoid any hidden steps being revealed once the goals have been established. Finally, recognition should be given to the amount of supervisory support that is available for implementation of a given program. In many cases a consultant is not the direct supervisor over treatment mediators and thus must rely on other supervisors. In cases where supervisory support is not provided, the treatment mediators may put forth minimal effort in the implementation of a treatment program and the result can be a low degree of treatment integrity and difficulty achieving treatment goals. In other cases, supervisory support may be readily available and cases of incompetence or insubordination can be reported and readily addressed by a supervisor.
Instrument/Method Validity and Reliability Consultants must determine the need for instruments that have proven validity and reliability. There are several instruments that have proven validity and reliability, such as the TEI, IRP, AARP, and others. While each of these instruments can be effective toward attaining valuable measures of acceptability, it also appears that the instruments do not always meet the needs of consultants or consumers. In numerous instances in the literature, these instruments have been modified to meet the specific needs of consumers and consultants. In numerous other instances, these instruments have been avoided in lieu of selfdeveloped instruments to evaluate social validity. There may be several possible reasons for this trend, such as demonstrated validity and reliability are not always important, the standardized instruments do not meet the needs of the consumer or consultant, the instruments are not easily accessible, consultants are not knowledgeable of various instruments that may be useful, or combinations of these and/or other factors. Basically, consultants should make themselves aware of what instruments are available and choose the best methods for their situation.
Ease of Implementation Instruments or methods for collecting social validity data need to offer a quick and efficient means for gathering information. In many cases
132
Chapter | 5 Increasing the Social Significance of Treatment Goals
social validity data is never collected, and if the available procedures are not easy to implement, then they will most likely never be used. Considering the time and effort that is required to conduct a functional assessment of behavior, develop a treatment, train treatment mediators, collect data on behavior changes, conduct integrity checks, and so on, it seems highly important that social validity measures not be overly time consuming. Schwartz and Baer (1991) recommended that social validity questionnaires be highly comprehensive and allow consumers numerous options for responding. When a highly comprehensive questionnaire is used, time allowances should be recognized, especially when these questionnaires are used in conjunction with other similar instruments such as functional assessment interviews. Each of these types of instruments takes time to complete, and in many cases there may be a considerable amount of redundancy across different forms, scales, interviews, and other materials. It might be beneficial for consultants to work toward synthesizing some of the different measures into a comprehensive tool that addresses numerous variables, such as functional assessment background data and social validity data, and also reduces redundancy. At present, such an instrument does not appear to be available. Another option would be to rely upon other related factors that can offer insight into consumer preferences and be indicative of social validity, such as quality-of-life measures, measures of habilitation, vocational aptitudes measures, and so forth,, from which information may be available. Information from other successful programs may be indicative of treatment goals that are considered socially valid by specific consumers. The easier it is to obtain information about potentially socially significant treatment goals, the more readily the measures will potentially be used by consultants.
Specificity to Target Individual/Group Consultants should consider the characteristics of the individuals who will be evaluating the social validity of treatment goals and choose measures that best match these characteristics. Several instruments have been developed for specific populations, such as institutional staff, teachers, parents, children, and so on. Instruments developed for use
Chapter | 5 Increasing the Social Significance of Treatment Goals
133
with teachers concentrate on variables frequently of concern to teachers, such as difficulty of implementation. Instruments designed for children present the information at a level of understanding appropriate for their age group. Instruments frequently used for parents have items that easily readable and are often presented by a consultant who reads the items for the parents. Each of these instruments is targeted for a specific population of consumers and is usually easy to administer. Consultants should become familiar with the instruments that are available and select those that are most appropriate for their purposes. When instruments are not available for a specific population, consultants should consider modifying some of the existing instruments or utilizing other types of measurement that might be more appropriate.
Clarity of Items Treatment acceptability instruments need to clearly describe the questions being posed to consumers. In order to more accurately obtain information from consumers on rating scales, the items need to be specific about exactly what is being rated. Differentiating ratings about the goals, procedures, and effects of treatment may be one method for offering clarity to the items on a rating scale. Broader questions on a rating scale should specify all the areas that need to be considered in making a rating. Including specific information about what exactly is being rated may provide an opportunity for a consultant to make minor adjustments to treatment programs to improve social validity without having to rework an entire program to achieve the same results. Rating scales may be administered at various times during the course of treatment development and implementation, and these ratings at different time periods may provide useful comparisons. In addition to this strategy, it may also be beneficial to include some comparison items within a rating scale that is administered after a treatment has been implemented and effects have been achieved. Items such as this might specify that raters evaluate an aspect of a treatment program based on their perceptions prior to implementation, during implementation, and after implementation. These types of ratings obtained after treatment effects have been demonstrated could be in addition to ratings obtained before treatment and during implementation in order to add to the reliability of those ratings.
134
Chapter | 5 Increasing the Social Significance of Treatment Goals
SOCIETY Normalization The survival of the culture of a society depends upon some level of structure which acts to maintain or increase conformity among its members. Skinner (1953) stated: What a man eats and drinks and how he does so, what sorts of sexual behavior he engages in, how he builds a house or draws a picture or rows a boat, what subjects he talks about or remains silent about, what music he makes, what kinds of personal relationships he enters into and what kinds he avoids—all depend in part upon the practices of the group of which he is a member (p. 415).
At the level of society, treatment goals that aim toward progressing the standards of society should be considered to have a greater degree of social validity. Skinner (1953) explained how the culture of a society is shaped by operant conditioning: “Behavior comes to conform to the standards of a given community when certain responses are reinforced and others are allowed to go unreinforced or are punished” (p. 415). Skinner also explained how these cultural manners and customs were continually changing through a process similar to natural selection. These types of changes have been demonstrated in the usage of seatbelts in vehicles, acceptable locations for smoking, maintaining extra food and water in the event of an emergency, and many other examples. Each of these has been promoted and incorporated as a standard practice within society and violations of some of these practices can now result in undesirable comments from others or even fines. Recognition of this need to conform to societal standards may be beneficial toward developing treatment goals that have social significance at the societal level. Normalization is an ideology of promoting integration of individuals with disabilities into environments that are accessed by the general population. Brown et al. (2004/1977) advocated for the integration of children with severe disabilities into regular education settings and stated that these services should be “delivered in as normal a fashion as possible” (p. 4). Wolfensberger (1972, 1983) stated that culturally normal behaviors of individuals with disabilities should be supported with procedures, settings, and goals that are typical for those within the general population. While normalization may be a societal movement, it may also have a high degree of importance among consumers of treatments. Consumers
Chapter | 5 Increasing the Social Significance of Treatment Goals
135
of behavior services, in many cases, are highly motivated to achieve a degree of participation and acceptance similar to their peers, but their current behaviors may prevent this from occurring. This makes the issue of normalization relevant toward the development of socially significant treatment goals, because consumers may simply have their “eye on the prize” in a way that may compete with or outweigh the manner in which the “prize” is obtained. Treatment goals should reflect the specific types of activities, settings, and so on in which consumers would like to increase their access or participation. Consumers of applied behavior analysis services may consider treatment goals that focus on increasing opportunities to participate in specific, highly important activities, and consideration should be given toward goals that promote normalization of their activities while recognizing the need for these activities to be age appropriate. Age-appropriate treatment goals may allow for increased normalization by focusing on skills that are typically displayed by peer groups of the consumer. Turnbull and Turnbull (1996) described a process called Group Action Planning and outlined some of the typical life areas where normalization can be supported for individuals with problem behaviors. They included families, friendships, community, schools, living situations, and employment. For each of these areas, they described action components such as inviting needed supports, creating connections with others, envisioning great expectations for the future, engaging in creative problem solving, and celebrating successes. Each of these components encourages active involvement on the part of the individual targeted for behavior services. This type of Group Action Planning could be a useful resource when determining treatment goals. Identifying lifestyle areas where normalization could be increased and utilizing existing skills of the consumer to support these goals may lead to more significant treatment goals. Section two of the Checklist for Exploring the Social Significance of Treatment Goals (Table 5.1) focuses on areas of normalization. These questions can provide some structure for evaluating the potential significance of treatment goals.
Use of Available Resources and Funding Society is frequently concerned with the availability of resources and funding, as well as how these resources are utilized. A treatment program that makes use of existing resources and funds would most likely be more appropriate than a program that requires development of new
136
Chapter | 5 Increasing the Social Significance of Treatment Goals
resources or acquisition of funds. In order for a treatment program to have a significant effect, it may be necessary to substantially reorganize the existing environment, which may necessitate acquiring new resources or funding. In such cases it may be beneficial to delineate how some existing resources will still be used and how new resources or funding may produce multiple outcomes within the environment.
STRATEGIES TO BOOST THE SOCIAL VALIDITY OF TREATMENT GOALS There are several strategies which could potentially improve the selection of socially significant treatment goals and enhance overall social validity. 1. 2. 3. 4. 5.
Incorporate semi-structured interviews Gather information from multiple sources Consider different aspects of the individual’s life Rank treatment goals hierarchically Operationalize expected outcomes in both clinical and social (lay) terminology
Step 1—Incorporate Semi-structured Interviews The use of a semi-structured interview which focuses on identifying relevant treatment goals can be a beneficial source from which to begin developing treatment goals. This type of interview may provide some consistency toward gathering information which can direct the selection of the most significant goals of treatment. This type of interview may be implemented individually or with a group. Individual administrations may provide a means of comparing different responses and looking for consensus opinions among individuals with differing opinions. Group administration may also allow for identification of a consensus of opinions, but may also limit information from individuals who are uncomfortable discussing their opinions within groups. Factors such as time limitations and the complexity of problem behaviors may influence the selection of individual or group interviews. Gresham and Lopez (1996) provide an example of a semi-structured interview which can be used for all levels of social validation for school-based treatments. Their interview is divided into a section focusing on goal
Chapter | 5 Increasing the Social Significance of Treatment Goals
137
selection, a section on treatment procedures, and a section on effects of treatments. The section on goal selection contains items such as “What behaviors led you to request consultation,” “Describe how these behaviors cause classroom problems,” and “Define each behavior as specifically as possible.” The items in this section of their interview appear to primarily prompt for information about the problem behavior to be addressed, rather than focusing on the goals of the treatment. While it is necessary to place importance on properly defining the problem behaviors, it is also important to identify what are considered to be significant goals related to the behaviors. An example of this might be when a treatment is designed to reduce the occurrence of off-task behaviors, while the expected goals of those working with the client may be to improve the quality of work produced. This may be a problem of considering only the immediate goals of treatment without incorporating the extended goals of treatment. The immediate goal of the treatment for off-task behavior described is to reduce the amount of time that the client is engaged in off-task behavior. The extended goal of the treatment for off-task behavior may be to increase the quality of work produced. A treatment may be developed which increases on-task behavior, but with relatively no improvement in quality of work, which might be considered a significant treatment goal by a consultant, but may be considered irrelevant by others working with the client who expect the increase in on-task behavior to result in improved work quality. Without proper consideration of the immediate and extended goals of treatment, the goals may lack significance among those closely connected to the client. The identification of significant treatment goals may require investigating a wider range of potential goals than has been previously practiced within the field of applied behavior analysis. The area of positive behavior supports may provide some insight into how applied behavior analysis may improve on development of socially significant treatment goals. The focus of positive behavior supports, which stresses examination of all areas of life impacted by changes in behavior, when developing a treatment may be an important part of selecting significant treatment goals.
Step 2—Gather Information from Multiple Sources This strategy is important to ensure that a representative sample of people who are directly associated with an individual targeted for treatment
138
Chapter | 5 Increasing the Social Significance of Treatment Goals
provide input into what treatment goals they consider most important. To increase the potential for addressing socially valid treatment goals, those individuals who will eventually rate the social acceptability of the treatment should be allowed to address their concerns regarding the goals of treatment. If the goals considered important by those most closely connected to the treatment are incorporated into the treatment, and they in turn are the same individuals who rate the acceptability of the treatment goals, it should be expected that they would provide higher acceptability ratings for the treatment goals. This could be difficult in some cases, especially when there are several individuals who consider very different goals to have importance, but the more input from these individuals that can be obtained, the better chance a treatment should have to address a number of these goals or at least a portion of the goals.
Step 3—Consider Different Aspects of the Individual’s Life The various areas of a person’s life should be considered when designing a socially valid treatment. Although initially a person’s primary problem may be considered to be having difficulty making new friends, it is important to find out what other areas of his or her life may be in need of intervention. Persons who have difficulty making new friends might benefit from a treatment which focuses on improving their hygiene so that others are not distracted or offended by lack of cleanliness. The goals that might appear most obvious to the consultant developing a treatment might overlook some of the goals considered most relevant by the client and those most closely associated with the client. When planning the goals of treatment, a consultant should consider gathering information from the same or similar group of individuals who will be rating the social validity of the treatment goals. Developing goals with input from one group and assessing social validity from another group may result in more information from more people but may reveal highly varied opinions about the significance of treatment goals. When consultants are only able to gather information from a select group to develop treatment goals, they may compare the goals to goals developed for other clients in similar situations to determine the potential for these goals to be considered socially significant.
Chapter | 5 Increasing the Social Significance of Treatment Goals
139
Step 4—Rank Treatment Goals Hierarchically Horner et al. (2005) reported that the social validity of single-subject research is enhanced when dependent variables with high social importance are targeted for treatment. With this considered, each of the goals of treatment should be prioritized and ranked from most to least important. The level of importance for each treatment goal should be based upon input from a majority of the individuals who will be closely impacted by the treatment or program goals. When ranking treatment goals, it is important to incorporate all the treatment goals that are considered highly important, but it is also important to minimize the number of goals associated with any particular treatment. A good rule of thumb may to limit the number of goals to three in order to avoid overly complex goals and to ensure that the most important goals are addressed. In addition, it may not be feasible to accomplish several goals with a single treatment, and subsequent goals may need to be addressed using other treatments. Proposing to address too many goals with a single treatment may lead to reductions in social validity if the goals cannot be effectively achieved. Typically, treatment goals may be singular, but to increase the potential social validity it may be necessary to expand upon a singular goal to include some observable changes associated with alternative settings, with different people, or to supplement other behaviors. For example, a treatment designed to reduce inappropriate social interactions might have a goal statement such as “John will reduce his inappropriate belching to zero occurrences during social interactions with others.” This singular treatment goal might be expanded to include “John will stand at an appropriate distance from others (approximately 3 feet) when engaging in a conversation” and “John will increase his interactions with others.” The singular goal of reducing belching during social interactions could be accomplished by having John avoid all social interactions, which may make the reduction of belching irrelevant; or, if belching is reduced but John stands to close to someone during a conversation, the reduction of belching may seem less important. In cases where a number of goals are highly relevant, it may be necessary to incorporate several treatments or a treatment “package” program to adequately address all goals.
140
Chapter | 5 Increasing the Social Significance of Treatment Goals
Step 5—Operationalize Expected Outcomes in Both Clinical and Social (Lay) Terminology Operationally defining the expected outcomes in both clinical and social terminology is not a typical component discussed in most texts outlining the procedures for developing behavior treatments. This may involve determining if extinguishing only one of multiple target behaviors will reach the expected level of outcome. For example, if a child displays aggression toward others, property destruction, and selfstimulatory behaviors, the parents and others working with the child may not consider a treatment which results in a reduction of aggression and self-stimulation successful unless it also produces reductions in property destruction. In contrast, a treatment which extinguishes aggression and property destruction might be considered a success by parents and others even if the self-stimulation is not effective or even if it increases. This treatment would have limited clinical success for the three targeted behaviors, but may be considered socially acceptable based upon the combination of target behaviors which were reduced. Additionally, clinical and social outcome expectations should encompass differences across environments and individuals. The clinical and social outcomes for a classroom environment may be very different from the expectations on a playground. One may require a student to remain seated, refrain from talking, and look at a teacher, while the other setting may involve running, talking, and interacting with peers. In order to potentially increase the social validity of the goals, it may be important to define these goals not only in clinical terminology but also in lay terminology. An explanation of the goals of treatment should be informative to those closely connected to the treatment by explaining the goals in a manner that understandable and meaningful to them. While a highly trained consultant may understand a goal that results in “a 45% reduction in the rate of problem behavior,” other untrained individuals may require alternative explanations. The wording of these goals should incorporate easily recognizable points of reference such as “the treatment will reduce his aggression so that he can independently play with others on the playground.” By providing clear and easily understandable goals prior to implementing a treatment, the accomplishment of these goals should be more easily recognized and the expectations of the consultant and others will be in agreement.
Chapter 6
Enhancing the Appropriateness of Treatment Procedures Preliminary Factors to Consider When Planning for Social Validity Determine How the Information Will Be Used Consider the Reliability and Validity of Measures Consumer Acceptability Characteristics of Consumers Knowledge of Treatment Consultant Acceptability Characteristics of Consultant Affiliations of Consultant Familiarity with Recent Advances in the Field Interaction Style of Consultant Assessment Methods Design of Treatment Treatment Intrusiveness Implementation Difficulty Other Variables Societal Appropriateness Strategies to Improve the Social Validity of Procedures and Programs Step 1—Explain to Consumers Why Information Is Being
Copyright © 2010 Elsevier Inc. All rights reserved
Collected and How It Will Be Used Step 2—Look for Problems First Step 3—Determine How Social Validity Should Be Measured Step 4—Determine When Social Validity Should Be Measured (Pre- or Posttreatment) Step 5—Determine Who Should Rate the Acceptability of the Procedures Step 6—Determine the Conditions Under Which Ratings Should Be Obtained Step 7—Determine What Type of Training Should Be Provided to Raters Step 8—Allow Time for Behaviors to Become Established
141
142
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
Wolf (1978) first stressed the importance of the concept of treatment appropriateness as a component of social validity, which he described as a necessary element for maintaining and expanding the use of behavioral procedures by society. Kazdin (1980) referred to this as “treatment acceptability” and defined this as judgments of treatments by actual or potential consumers of the treatments, such as nonprofessionals, clients, laypersons, and others. The conceptualization of treatment acceptability described by Lennox and Miltenberger (1990) incorporates factors influencing treatment acceptability that may only be available to the practitioner designing or recommending the treatment. This comprehensive view of treatment acceptability extends the more traditional view of treatment acceptability, as defined by Kazdin, by incorporating factors such as the ethical guidelines of a profession, the influence of meta-analyses and literature reviews, the expertise of the practitioner, practitioner history with a treatment, and practitioner bias. Other models of treatment acceptability (see Witt & Elliott, 1985; Reimers et al., 1987) appear to be predictive and focus more on clear understanding of treatments and how treatment acceptability may lead to increases or decreases in factors such as use of treatment, integrity of treatment, and effectiveness of treatment. The information on treatment acceptability and appropriateness is presented following Carter’s (2008b) model of treatment acceptability (see Figure 6.1). This model distributes treatment acceptability along the areas of society, consultant, and consumer, each of which are considered to be major influences on overall treatment acceptability. When developing a treatment or program which will potentially have high social validity, planning should begin as soon as a referral is accepted. Consideration should be given as to how the social validity information will be used, as well as who will be using the information. Social validity information may only be needed to ensure the satisfaction of a treatment for those closely connected to the treatment, to evaluate the acceptability of an innovative procedure, or in order to develop a database to share with others. In addition, the individuals who will be using the information may influence the extent to which social validity data is collected. If the information is only to be used by the consultant who is developing the treatment, data collection may be less extensive than if the data is to be used by a human subject board or by a school system.
Treatment acceptability
Consultant acceptability
• • • • • • •
Consumer acceptability
FIGURE 6.1
Society Laws Legislation Ethical boards Professional associations Review boards Movements/trends
Consultant Training History with treatment Ethical guidelines Familiarity with recent research Knowledge of client situation Interaction style Assessments conducted
Rater Gender Knowledge of treatment Socioeconomic class Location of high school Parenting a child w/medical disorder • Marital distress • • • • •
• • • • •
Treatment Design elements Reinforcement Punishment Implementation time Intrusiveness
• • • •
Acceptability Instrument/Method Validity & reliability Ease of implementation Specificity to group Clarity of items
Client • Severity of problem
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
• • • • • •
Societal acceptability
Distributive model of treatment acceptability.
143
144
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
Depending on how the information will be used and who will use the information, the data may need to include rigorous measures of validity and reliability. The extent to which social validity data may be considered convincing may depend upon adherence to a certain conceptualization of social validity. The reliability of the information may be considerably more difficult to ensure, due to the apparent perpetually changing aspects of this construct. The aim for reliability may be to provide a representative sample which could be considered reliable within a certain period of time. Each of these preliminary factors will be discussed, followed by more specific suggestions for enhancing the potential social validity of a treatment or program.
PRELIMINARY FACTORS TO CONSIDER WHEN PLANNING FOR SOCIAL VALIDITY Determine How the Information Will Be Used Developers of treatments and programs to address problem behaviors have an ethical responsibility to consider the social implications of these treatments and programs. These developers or consultants may simply need to collect some data to aid in designing a treatment which will potentially be acceptable to a small group of individuals who are closely connected to the treatment. Gathering this type of data may be sufficient for meeting the ethical requirement of considering the best interest of the client and may involve asking a few informal questions of a small number of individuals. More extensive information may need to be gathered on social validity if the information is to be used to satisfy the requirements of a human subject board, to establish norms within a school system, or to develop a regional database from which inferences will be drawn. Each of these instances may require incorporating more formal measures of social validity from a wider range of individuals over a longer period of time.
Consider the Reliability and Validity of Measures In order to ensure that social validity measurements are valid, the use of informal measures without previously established validity norms should be minimized. While informal measures may be supplemental to more standardized measures of social validity, they may not provide evidence of validity when used in isolation. One potential reason for the continued use
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
145
of informal measures of social validity may be the relevance of the information that these informal measures can provide. Informal measures may be quickly developed and designed to measure specific aspects of a treatment or targeted towards specific population of raters. The flexibility with more informal measures of social validity may continue to be popular because of the type of information they can provide, which might be considered highly relevant and useful to the developer of the treatment. More standardized measures of social validity may only probe a limited number of areas considered relevant to the treatment developer, or may only produce general information which does not allow the treatment developer to use the information to improve the validity of future treatments. Schwartz and Baer (1991) appeared to recognize the limitations of more formal social validity assessments toward providing accurate information which could prove useful in the remediation of treatments considered unacceptable and in the construction of more acceptable treatments. They suggested that social validity assessments be improved to ensure that the information being collected is comprehensive and highly viable. They recommended five means for improving social validity questionnaires that included the following: 1. Questionnaires should encompass numerous response options for raters rather than confining the rater to a limited number of responses, 2. Require the rater to provide responses that span the entire range of the scale in order to ensure that the rater was actively considering all possible options, 3. Questionnaires should specify the exact portion of the treatment being rated instead of an overall general rating of the entire program, 4. Social validity assessments should cover all the relevant areas associated with treatment, rather than focusing on a single aspect of treatment such as decreases in target behaviors, and 5. Social validity assessments should pool very specific information from raters rather than more general statements in order to make the information more usable. By incorporating the recommendations by Schwartz and Baer, it may be possible to make formalized measures of social validity more appealing and useful to practitioners. If formal measures can be improved in a manner that allows them to be easily used in a variety of settings with a variety of
146
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
consumers while producing highly relevant information, the use of informal measures may become less advantageous. Providing evidence of reliability of social validity may be required in some instances where the information may be used toward program planning and for measuring the relative changes in acceptability that may be related to different periods of data collection. Increasing the validity and reliability of social validity information may rely in part upon adhering to a solid conceptualization of social validity. Once a clear conceptualization of social validity is discerned, the most appropriate types of measurement instruments can be identified, as well as the most appropriate point in time to utilize measurements. One method which provides evidence of reliability may involve measuring acceptability in the same manner at different points in time.
CONSUMER ACCEPTABILITY The actual or potential consumers of treatments have been defined by Kazdin (1980) as nonprofessionals, clients, laypersons, and others. A number of variables related to the consumer have been found to influence treatment acceptability. Within the proposed distributive model, each of these variables is described as consumer acceptability, which helps to comprise overall treatment acceptability. Consumer acceptability is described within this model as two sets of variables related to the actual clients who potentially receive treatments and those who may assist in the selection of treatments, such as parents, guardians, mediators of treatments, and so on, who rate the acceptability of treatments. One client variable that has been found to influence acceptability is the severity of the client’s problem behavior (Kazdin, 1980; Tingstrom, 1990). Reimers et al. (1992) found increases in acceptability ratings when less intrusive treatments, such as positive reinforcement, were recommended for less severe behaviors and when more intrusive treatments, such as medication, were recommended for more severe behavior problems. This indicated that selecting a treatment based on the severity of a client’s problem behaviors may influence the acceptability of the treatment. One of the first goals in assessing consumer social validity should be to gather information regarding problems perceived by consumers in association with treatment programs. Consumers should first be encouraged
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
147
to identify and voice their concerns about treatment programs, rather than having them focus first on the positive aspects of a treatment program. Schwartz and Baer (1991) recommended that when social validity information is collected from consumers, they should be informed of why this information is being collected and how the information will be used. The program designer should then make attempts to incorporate reasonable concerns into a more socially acceptable treatment. Failure to provide consumers with a rationale for the collection of social validity data may make consumers consider their input as undervalued and may cause resistance toward future attempts to solicit social validity information.
Characteristics of Consumers Several variables related to the characteristics of those who rate the acceptability of treatments have been found to influence treatment acceptability. Parental stress derived from parenting a child with a medical disorder or marital difficulties have been shown to decrease the acceptability of treatments (Gage & Wilson, 2000; Miller & Kelley, 1992; Miller et al., 1998). In addition, simply involving parents in some training procedures has been shown to increase parental stress (Benson & Turnbull, 1986; Gallagher, Beckman, & Cross, 1983) and potentially decrease treatment acceptability. Some other consumer variables that have been found to influence treatment acceptability include severity of problem behavior (Reimers et al., 1992), the gender of raters (Kazdin, 1980; Miller & Kelley, 1992), the raters’ knowledge of the treatment (Singh & Katz, 1985), raters’ socioeconomic class (Heffer & Kelley, 1987), and geographic location of raters’ high school (Stinnett et al., 2001). Many of the variables, such as gender and marital distress, appear to be completely independent from influence by a consultant or from societal factors related to treatments. Lucyshyn and Albin (1993) recommended that interventions proposed for family-based behavior support plans should be consistent with objective data collected and should incorporate specific features unique to the family. They proposed four requirements for treatments summarized as follows: (1) treatments should be logically related to the hypothesis derived from the functional assessment; (2) the treatments should be informed by the strengths of the family; (3) the treatment should have potential for reducing the stress of the entire family; and (4)
148
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
the treatment should be consistent with the environment and routines in which the family functions. Their requirements for treatment rely upon gathering information from families through functional assessments and extensive interviews that cover a wide range of topics, such as family strengths/weaknesses, available resources, sources of stress, daily routines, and the like. They also recommend presenting potential treatments in the form of a table that specifies how each component of the treatment is important. They provided an example of a table with column headings such as hypothesis, ecological/lifestyle, teaching new skills, emergency procedure, and so forth. Under each of the column headings, the specific component of a family behavior support plan could be written in an effort to explain the relevance of each of the components of treatment. This approach appears to be a potentially useful method for presenting information to consumers so that they may gain a better understanding of how the specific components of behavior plans were designed and selected for inclusion in the program. Table 6.1 provides a similar method for explaining the rationale for selecting specific components of a treatment package in a treatment component value matrix. This matrix can be modified to reflect the areas of need considered most relevant for different consumers or programs. The column headings reflect the various components of a treatment package. The specific components can be written in underneath the treatment component number. The treatment components can be arranged in order of those considered to potentially be most important, under a lower component number, to those considered less important, listed under a higher component number. The areas where the treatment components may have influence are listed vertically in the first column. These areas are based upon lifestyle categories considered important for individuals with problem behaviors (see Mirenda, 1993; Turnbull & Ruef, 1997; Turnbull & Turnbull, 2002, 1999; Turnbull, Ruef, & Reeves, 1994). An “X” can be placed in the corresponding row and column to reflect that a specific treatment component is designed to address that area of need. In place of an “X,” a statement can be written as to how the treatment component is designed to influence that area of need. Once completed, this matrix can be a visual for consumers when explaining the rationale for including each component. Including this type of rationalization for treatment components may allow for
Treatment Component Value Matrix Treatment Component #1
Based on functional assessment Teaches replacement behavior Improves family relationships Improves friendships Improves employment opportunities Improves educational opportunities Improves community participation
Treatment Component #2
Treatment Component #3
Treatment Treatment Treatment Component #4 Component #5 Component #6
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
TABLE 6.1
Improves living situation 149
150
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
increased understanding among consumers of the need for each component and promote increase acceptance of the treatment procedures. Lucyshyn and Albin also recommend that the following six factors be considered when selecting strategies for inclusion in a family-focused behavior support plan. 1. 2. 3. 4. 5.
The amount of time available to commit to implementation; Implementer preferences in treatment strategies; Skills possessed by family members; Skills possessed by classroom teachers; Additional implementation support resources available to the family; and 6. The quality of the relationship between the parents and classroom teacher.
Knowledge of Treatment Variables such as knowledge of treatment may be dependent upon information provided by a consultant, or possibly by societal influences, such as professional training or affiliations with associations or movements such as PBS. In addition, parents, friends, or relatives of individuals with medical or psychiatric diagnoses may frequently become involved in associations, movements, or support groups to obtain information. This may make variables discussed within societal acceptability especially influential with these potential raters of treatments. This influence on knowledge demonstrates how consumers, consultants, and society interact and subsequently influence treatment acceptability.
CONSULTANT ACCEPTABILITY Characteristics of Consultant The designers of treatments are typically consultants who have the training and experience necessary to develop, implement, and monitor treatments. Consultants can be highly influential in the overall acceptability of treatments; that influence is referred to as consultant acceptability within this model. Consultants may vary greatly with regard to their training, history with treatments, and competencies. Each of these factors can be highly prominent in the types of treatments that they propose
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
151
and develop. Consultants with training in PBS may propose and develop more treatments that fit with this orientation than consultants without this training, just as those with training in cognitive behavioral therapies may propose and develop more treatments from this orientation. Ethical guidelines typically require that professionals practice only within those areas in which they are competent, and with the wealth of information available and the numerous areas of specialization, most consultants can only be competent in a limited number of areas. These limitations influence consultant acceptability by increasing or decreasing their experience and knowledge of certain treatments and may influence their judgments of the acceptability of certain treatments. Singh and Katz (1985) found that formal educational training could change acceptability ratings by college students and this implies that the formal training that consultants receive may be influential in how acceptable they find treatments. It seems reasonable that consultants would only recommend treatments that they consider acceptable themselves and this could be considered an ethical obligation within their profession. This demonstrates how societal acceptability may be influencing consultant acceptability. Bernstein explained the influence of society on consultant activity as follows: “Discussion by members of society of the social validity of our professional activities is inevitable, and inevitably results in public policies which affect our professional behavior” (1989, p. 97).
Affiliations of Consultant A more obvious demonstration of how societal acceptability may combine with and influence consultant acceptability can be seen when examining the professional affiliations of consultants. Spreat and Walsh (1994) found treatment acceptability differences among members of different divisions of the American Association on Mental Retardation. They determined that members of the Psychology Division rated treatments slightly more acceptable than members of other divisions. Similarly, Fairbanks and Stinnett (1997) found differences in treatment acceptability among teachers, school psychologists, and social workers. Social workers rated punishment-based treatments as less acceptable than the other two professional groups. While membership in a professional organization has been shown to influence treatment acceptability, it does not clarify whether the differences among these groups were
152
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
prevalent prior to membership within the group or whether membership in the group shaped their judgments of treatment acceptability through training, experiences, ethical guidelines, and so forth.
Familiarity with Recent Advances in the Field The treatments developed or recommended by consultants may depend upon several factors other than their past training, history with specific treatments, or professional association memberships. Consultants who continually hone their skills in efforts to maintain familiarity with the most recent research developments may be more likely to use treatments that have only recently been developed or shown to be highly effective. This can be seen with the advances made in functional analysis approaches, which were described previously. A consultant who was familiar with these approaches or who was trained in these approaches would be more likely to incorporate these techniques than a consultant who was not familiar with these approaches. In addition, a consultant who is not familiar with these techniques may ethically be discouraged from using these techniques without proper training. Consultants who use functional analysis approaches may be more likely to recommend certain treatment procedures suggested by the findings of the assessment over other arbitrary techniques. This is an example of consultant acceptability where one treatment is considered more acceptable than another treatment because of the familiarity and use of certain assessment techniques. A consultant may also influence the acceptability of a potential treatment by simply not presenting it as an option to a consumer. Hastings and Noone (2005) outlined how in some cases, ethical guidelines require an assessment to determine the function of behavior prior to implementing a treatment. Functional assessments may incorporate descriptive, experimental, or informant-based procedures in order to develop treatments that address the maintaining variables of problem behavior (Hall, 2005). When treatment procedures using reinforcement and punishment-based methods are developed on the basis of a functional assessment, the treatment may introduce a functionally equivalent replacement behavior that can be taught to the individual. This may make a treatment appear less intrusive, because it introduces a habilitative factor for the individual that in most cases is a required component of any complete educational program. Carr and Durand (1985) demonstrated
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
153
how children could be taught, through FCT, to solicit reinforcement that was equivalent to the reinforcement they were receiving for engaging in problem behaviors. Their research provides an example of how functional assessments may be used to develop a habilitative component that could make an intrusive treatment more acceptable. Consumers of treatment may be unaware of these ethical guidelines and may not incorporate the potential benefits of a functional assessment toward developing an intervention when they are rating the acceptability of an intervention. An exception to this could be if the risks and benefits of a functional assessment are described to the consumers and they clearly understand the relevance of such an assessment. Weigle and Scott (2000) found that teachers differentially rated the acceptability of a brief interruption procedure and differential reinforcement when provided with information on the function of problem behavior. Umbreit (1995) found increased acceptability of treatments based on a functional assessment over those not based on a functional assessment. Most likely, a functional assessment may only directly influence the acceptability of the treatment for the practitioner developing the treatment (rather than the consumer), by meeting ethical requirements, increasing the potential effectiveness of a treatment, or generating the least restrictive treatment. Unless consumers are provided with training on the importance of certain practices such as functional assessment, the influence of these practices on consumers’ ratings of acceptability would be minimal. In most cases the consumer would not have the expertise to compare the treatment based on a functional assessment to a treatment which was not based on a functional assessment, unless they were provided with information in this area. However, Lennox and Miltenberger’s conceptualization of treatment acceptability, which includes factors influencing the practitioner’s choice of treatment, would consider a functional assessment a potential component of treatment acceptability by realizing the role that a functional assessment could play in developing a more acceptable treatment. The value of a successful treatment which reduces problems resulting in lost time, extensive effort, and increased expenses may be considered invaluable and receive high acceptability ratings by those closely associated with the problem, regardless of the difficulty associated with assessing the treatment. The conflict that exists is associated with the potential for other treatments which may involve less extensive assessment procedures and produce the same or similar results. A functional assessment can potentially
154
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
include an extended analog evaluation that repeatedly exposes an individual to reinforcement for inappropriate behavior (Hastings & Noone, 2005). The influence of these assessment factors (extensive vs. less extensive) on the acceptability of treatments is currently unknown. In addition, the impact of a functional assessment on the acceptability of treatments is currently inconclusive (Hastings & Noone, 2005).
Interaction Style of Consultant Consultants may influence the acceptability of a treatment through their interactions with a consumer. A consultant, while gathering knowledge about a consumer and the consumer’s situation, may become more or less inclined to recommend certain treatments. The consultant may feel that a particular consumer may respond better to a certain treatment for a variety of reasons. Factors such as the level of functioning of a consumer, the availability of resources to the consumer, or the severity of the consumer’s problems may all influence the treatment recommendations made by a consultant. Consultants may also influence acceptability by the amount of information that they provide to consumers and the presentation method that they use with consumers. Singh and Katz (1985) found that acceptability ratings changed after raters were provided with training on specific details of treatments, empirical data on the effectiveness of treatments, and potential side effects of treatments. This type of information would typically be provided to the consumer by a consultant who developed the treatment. Similarly, Kazdin and Cole (1981) and Witt et al. (1984) found that the terminology or jargon used to described a treatment, such as humanistic or behavioral jargon, influenced the acceptability of treatments. A consultant who uses a specific type of terminology to describe a treatment to a consumer may influence the acceptability of a treatment as rated by a consumer. While consultants are obligated to provide enough material to consumers to gain informed consent, the amount of information they provide and the manner in which they present the information might vary depending on the complexity of the treatment, the restrictiveness of the treatment, the characteristics of the consumer, or numerous other variables. Each of the interactions described between consultants and consumers may be examples of how consultant acceptability influences overall treatment acceptability. Research on the acceptability of various consultation models may offer some insight into how consumer–consultant interactions influence social
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
155
validity. Distinct models of consultation delivery have been developed, such as direct behavioral consultation (DBC; Watson & Robinson, 1996) and conjoint behavioral consultation (CBC; Sheridan, Kratochwill, & Bergan, 1996). Each of these consultation models promotes different levels and types of interactions between consultants and consumers. Research examining satisfaction and acceptability of different consultation practices has found the CBC model to be preferable over other approaches (Cowan & Sheridan, 2003; Freer & Watson, 1999; Sheridan & Steck, 1995). The CBC model has been described as incorporating a structured group effort toward addressing behavioral needs of a specific individual where all parties are held responsible for the success of the programs developed (Sheridan & Colton, 1994; Sheridan & Kratochwill, 1992). In contrast, other consultation approaches may only involve targeted individuals such as parents or teachers. While additional research is needed to clarify the specific types of interactions considered valuable by consumers, consultants, and segments of society, it appears that obtaining input from a group of interested individuals may prove beneficial toward enhancing social validity.
Assessment Methods Another area in which consultants may influence overall acceptability is through the method used to assess treatment acceptability. Consultants are frequently the individuals who select the manner in which treatment acceptability will be measured. Consultants may choose to measure acceptability informally, by asking a few questions, or they may use a more formal instrument that has established validity and reliability. A consultant who chooses to use an informal procedure may ask questions that are not valid or reliable toward assessing treatment acceptability and thereby obtain measures that are not accurate. Conversely, some more formal treatment acceptability instruments do not provide specific cut-off points for distinguishing between acceptable and unacceptable treatments. Some of these formal instruments rely upon general rules of thumb, such as using the midpoint of a scale for cut-off points for distinguishing acceptable and unacceptable treatments. Whether informal procedures or formal instruments are used, the consultant may still influence the acceptability of a treatment through the selection of individuals who are allowed to complete the acceptability ratings. The consultant may choose to obtain acceptability ratings from one
156
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
person, or may obtain ratings from numerous individuals involved with the treatment. In addition, consultants may choose to selectively obtain acceptability ratings, or they may obtain acceptability ratings on most or all treatments they develop. As recommended by Lennox and Miltenberger (1990), consultants may obtain several acceptability ratings on highly similar treatments which they may use to compile local, regional, or national measures of acceptability. These more global measures could be used to influence their use of specific treatments or could be used to influence the committees and associations weighing societal acceptability factors.
Design of Treatment The design elements of treatments have been shown to consistently influence the overall acceptability of the treatments. Consultants construct specific treatment packages and are frequently responsible for training the individuals who implement the treatments on the components of these packages. This makes the actual treatment elements highly dependent upon the consultant who develops them and are therefore included as part of consultant acceptability. While the actual components of treatment may independently influence acceptability, they cannot first be considered unless they are included and described by the consultant. Several treatment variables have been found to influence treatment acceptability and are incorporated into this distributive model as part of consultant acceptability, since they are considered dependent upon the consultant to include them in the development of a treatment. Treatment variables that have been found to influence treatment acceptability include treatment intrusiveness related to use of reinforcement and punishment techniques (Kazdin, 1980), cost-effectiveness regarding the amount of time required to implement treatment (Witt et al., 1984), and the apparent appropriateness of the treatment (Cavell et al., 1986). Each of these treatment variables would depend upon how the consultant chose to design the treatment and the choice to include specific treatment variables, such as reinforcement, punishment, reductions in implementation time, or components to increase the apparent appropriateness of treatments.
Treatment Intrusiveness Lennox and Miltenberger (1990) described treatment restrictiveness/ intrusiveness as a well-known concept in the field of mental retardation which attempts to measure the amount of physical or psychological
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
157
stress placed upon the individual receiving the treatment. The typical progression of treatment restrictiveness/intrusiveness involves beginning with the least restrictive/intrusive treatment that is considered to potentially be effective, and if the treatment is deemed ineffective, attempting a more intrusive treatment which is considered to be potentially effective. Behavioral procedures that solely involve the use of reinforcement have frequently been demonstrated to be more acceptable than treatments that incorporate the use of punishment procedures (Miltenberger, 1990; Reimers et al., 1987). Reinforcement-based procedures that focus on increasing appropriate behavior appear to be less intrusive than punishment-based procedures that focus on reducing an inappropriate or unwanted behavior. Reinforcement-based techniques typically allow an individual to gain greater access to things he or she considers valuable (such as food, tokens, praise, free time, etc.) without directly attempting to suppress other behaviors. Punishment-based procedures can be considered to be highly intrusive by directly suppressing a behavior that is a part of the individual’s repertoire through the introduction of unpleasant/ painful stimuli or removing something that the individual values. Punishment-based procedures may be particularly intrusive when they focus on suppressing a specific behavior and do not provide training to introduce a more appropriate behavior to replace the behavior that was suppressed. Thomas and McGuire (1988) have indicated that various punishment procedures have been periodically renamed in attempts to reduce the stigmatizing effect of terms frequently associated with punishment. They also point out that renaming procedures such as physical restraint with terms such as “hugging” or “interrupting” do not change the degree of intrusiveness of a procedure, although the terms may be more appealing. Treatments that incorporate substances such as lemon juice (e.g., Matson, Manikam, & Ladatto, 1990) are designed to create a level of physical discomfort in order to discourage problem behavior. Electric shock is another technique incorporating physical discomfort that has been used as an aversive treatment for a variety of potentially life-threatening problem behaviors, such as self-injury (Duker & Seys, 2000). In addition, treatments that incorporate the use of response effort may be considered to influence the intrusiveness of a treatment because they may increase the amount of physical discomfort experienced by an individual. These procedures manipulate the effort required by an individual to complete a task or to engage in a behavior (Piazza, Roane, Keeney, Boney, & Abt,
158
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
2002). In some cases, increasing response effort may increase the degree of physical discomfort associated with a specific behavior and result in a decrease in the occurrence of the problem behavior. In other situations, response effort may be used to shape an individual’s behavior in order to make the behavior functional or act more efficiently on the environment. Treatments that have an impact on a student’s access to the same privileges received by other students could be considered to influence the intrusiveness of the treatment. Treatments such as time-out that often remove a student from a typical school setting/classroom or deny the student access to materials or opportunities for socialization may be considered highly intrusive. A less intrusive treatment would not impact the opportunity for a student to remain in a classroom setting or participate in an activity with others or with generally available materials. A treatment that requires a student to be placed in a setting that is more restrictive than where the child was previously placed would be highly intrusive. Treatments that involve removing a child from a regular education classroom and placing him or her in a special education classroom, an alternative school, or other more restrictive environment are examples of how placement may influence the intrusiveness of a treatment. Treatments that rely upon the influence of peers as a means of providing encouragement/reinforcement or enforcing rules in order to make the treatment effective also increase the effectiveness of a treatment (Smith & Fowler, 1984). Interdependent and dependent group-oriented contingencies are some examples of treatments that exploit the use of peers (Litow & Pumroy, 1975). Gresham and Gresham (1982) found that interdependent and dependent group contingencies were more effective in reducing disruptive behaviors than an independent group contingency, and suggested that the group cooperation element of these treatments may have been responsible for their greater effectiveness. They noted that within these two treatments, the children praised each other for appropriate behaviors and reprimanded each other for engaging in inappropriate behaviors. While these procedures may be effective, they may also be considered highly intrusive due to the influence upon a student’s peers. A possible unwritten component of this type of treatment, noted by Witt and Elliott (1982), involved group contingencies causing members of the group to seek revenge against an individual in the group who failed to comply with the required classroom expectations.
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
159
Other procedures, such as school–home notes, could employ unwritten components. Abramowitz, Eckstrand, O’Leary, and Dulcan (1992) referred to school–home notes as reinforcement methods that consist of both school and parental input to improve children’s classroom behavior. School–home notes often require parents to implement a portion of the procedure in addition to the procedures implemented by a teacher. While these treatments are typically designed to provide reinforcement, they could incorporate unwritten or unspecified components such as punishment implemented by parents or siblings at home, which are not specifically stated within the school–home notes. Parents may enforce restrictions upon a child based upon the child’s actions at school. These restrictions could inadvertently impact the siblings of the student by preventing them from playing with their sibling or preventing them from attending an event or participating in an activity. This impact could potentially motivate the sibling to seek revenge upon the student who is being punished by the school–home note. The use or recruitment of others to enhance or implement a treatment, or making the goals of the treatment public (Hayes et al., 1985), appear to influence the intrusiveness of a treatment. The more people who are involved in a treatment, the higher the possibility that someone could abuse the treatment for personal gain or pleasure, unless appropriate checks and balances are ensured. While a treatment may be abused by anyone, when professionals such as teachers or personnel who are paid to implement a treatment are the sole or primary individuals who are implementing a treatment, the potential intrusiveness of a treatment appears to decrease. This appears to be particularly important in comparison to treatments that incorporate minimally trained or untrained individuals such as significant others. In addition, treatments that clearly define the specific procedures to be implemented appear to be potentially less intrusive than treatments that have less clearly defined procedures or increase the opportunity for inclusion of unwritten components of treatment. Carter, Mayton, and Wheeler (2009) developed the Treatment Intrusiveness Measure (TIM) to evaluate the intrusiveness of behavior treatments frequently used with individuals with developmental disabilities (see Table 6.2). The instrument consists of 33 yes/no questions which focus on the presence or absence of various components of a treatment. “Yes” responses are scored with as one point and “No” responses are scored as zero points. The cumulative total points for questions provide a Base Level
160
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
TABLE 6.2 Treatment Intrusiveness Measure Answer questions 1–33 to determine Base Level of Intrusiveness Score (BLIS) Please answer “yes” or “no” to the following questions by placing a check mark in the appropriate box to the right of the question.
YES
NO
1. Does the treatment have the potential to impact the health or safety of the individual?
2. Does the treatment restrict the movement of an individual?
3. Does the treatment include providing the individual with nonnutritious foods?
4. Does the treatment involve providing the individual with controlled or restricted substances?
5. Does the treatment provide unlimited access to food or drinks?
6. Does the treatment prevent behaviors that are sometimes beneficial to the person or considered a natural biological response?
7. Does the treatment involve locking windows or doors?
8. Does the treatment involve requiring the individual to repeatedly perform a task?
9. Does the treatment require the individual to engage in a physically challenging activity?
10. Does the treatment involve physical prompting?
11. Does the treatment have the potential to impact the health or safety of the individual?
12. Does the treatment procedure attempt to minimize or eliminate a behavior?
13. Does the treatment incorporate pain or unpleasant stimuli?
14. Is the individual required to participate in the treatment?
15. Is the individual prevented from discontinuing the treatment?
16. Is the individual prevented from avoiding parts of the treatment?
17. Does the treatment cause physical discomfort?
18. Does the treatment involve an aversive stimulus?
19. Does the treatment involve increasing the effort required to engage in a problem behavior?
161
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
TABLE 6.2
(Continued)
Please answer “yes” or “no” to the following questions by placing a check mark in the appropriate box to the right of the question.
YES
NO
20. Does the treatment involve a change in placement for the individual?
21. Does the treatment involve removing an individual from his or her typical setting for any period of time?
22. Does the treatment involve restricting or limiting the individual’s privileges?
23. Does the treatment involve restricting the individual’s access to materials?
24. Does the treatment involve restricting the individual’s access to activities?
25. Does the treatment involve peers?
26. Does the treatment involve parents/guardians/care providers?
27. Does the treatment involve siblings?
28. Does the treatment involve possible unwritten components?
29. Does the treatment involve implementation of delayed consequences in multiple settings?
30. Does the treatment involve any form of public posting or announcement?
31. Does the treatment involve multiple individuals responsible for implementing the treatment?
32. Is the treatment implemented by nonprofessionals or volunteers?
33. Are the treatment components only vaguely defined?
Total number of “Yes” responses indicates BLIS
Optional Questions to Determine Modified Level of Intrusiveness (MLIS) Please answer “yes” or “no” to the following questions by placing a check mark in the appropriate box to the right of the question.
YES
NO
Does the treatment increase access to reinforcement for the individual?
(Continued)
162
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
TABLE 6.2 (Continued) Please answer “yes” or “no” to the following questions by placing a check mark in the appropriate box to the right of the question.
YES
NO
Is the treatment based on a prior functional assessment?
Does the treatment include a habilitative component?
Is the individual allowed to provide input into the development of the treatment?
Are the individual’s preferences incorporated into the treatment?
Is the individual allowed to choose the treatment?
Is the individual allowed to compromise with the treatment?
Are the components of the treatment clearly defined?
Are well-defined data collection techniques provided and included as part of the treatment?
Are treatment integrity checks clearly described and included as part of the treatment?
Enter BLIS Enter total “Yes” responses and subtract from BLIS BLIS ⫺ Total “Yes” from above questions ⫽ Modified Level of Intrusiveness Score (MLIS)
_____
BLIS ⫽ 0–5 indicates low level of intrusiveness. BLIS ⫽ 6–10 indicates mild level of intrusiveness. BLIS ⫽ 11–15 indicates moderate level of intrusiveness. BLIS ⫽ 16–20 indicates high level of intrusiveness. BLIS ⫽ ⬎21 indicates very high level of intrusiveness. From Carter, S. L., Mayton, M. R., & Wheeler, J. J. (2009). The development of an instrument to evaluate treatment intrusiveness for individuals with severe and challenging behavior. Research in Developmental Disabilities, 30, 58–69. Reprinted with permission.
Intrusiveness Score (BLIS) that represents a degree of treatment intrusiveness present for the treatment, with higher scores related to greater degrees of intrusiveness. An additional set of 10 questions is also provided, which addresses the factors considered to be highly relevant to the social validity of a treatment. The incorporation of these questions allows for an additional Modified Level of Intrusiveness Score (MLIS) which can be used to examine treatment intrusiveness as related to social validity.
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
163
Implementation Difficulty Lennox and Miltenberger addressed cost-effectiveness as a factor in treatment acceptability under the category of practical considerations. They discussed that from an administrative point of view, the acceptability of treatments may be reduced when they require more time, effort, expense, and so on. Witt, Elliott, and Martens (1984) found that increased time requirements for implementing an intervention negatively influenced teacher ratings of treatment acceptability. In some cases, the amount of time required to implement a plan may be unavoidably extensive, and may only be offset by offering support to treatment mediators such as frequent breaks, extra staff support, specialized training, reduction of other responsibilities, and the like. In other situations, it may be important to consider the “goodness of fit” (as discussed in Chapter 5) in an effort to determine treatment elements that can be easily incorporated into existing routines. While this approach may be beneficial toward reducing implementation effort and time, it must also be effective toward changing the targeted behaviors. Although decisions regarding treatment should not be solely based on factors associated with cost, the availability of treatments with potentially comparable effectiveness but differing costs should not be overlooked. Evaluating cost-effectiveness using a cost-benefit analysis is discussed further in Chapter 7.
Other Variables Other treatment variables found to influence acceptability that may be dependent on the consultant could include treatment efficacy (Von Brock & Elliott, 1987), the apparent appropriateness of the treatment (Cavell et al., 1986), and the mediator of the treatment (Kalfus & Burk, 1989). Lennox and Miltenberger referred to this time required to produce results as treatment efficiency. They discussed treatment efficiency as a practical consideration of treatment acceptability that was not well established for many procedures and was difficult to determine from research studies, because of the different parameters used within these studies. Michaels et al. (2005) noted that a research base exists which supports the effectiveness of decelerative consequence-based procedures, and that some of these procedures yielded more immediate results than proactive antecedent approaches. Cavell et al. (1986b) found a decreased acceptance of treatment protocols that were presented with different rationales for following the specified
164
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
protocol. The type of rationale that a consultant offers consumers may be connected to the training and information that consumers are provided by a consultant, but the specific of the training and type of information provided may also play a role toward influencing acceptability of treatments. The situational context in which treatment rationales are provided should also be considered. Cavell et al. (1986a) found that continuing an ineffective treatment was considered more acceptable than introducing a paradoxical treatment, regardless of the rationale that was provided. In this case, it appears that the situational context of having another treatment already in place negated the rationale for a treatment that did not appear obviously appropriate. Consultant should recognize the resistance of consumers to treatments which do not readily appear to be appropriate regardless of the rationale that might be provided. This type of situation may be frequently encountered when discussing the need for a functional analysis which programmatically introduces potential reinforcement for inappropriate behavior over a period of time. The typical protocol for conducting a functional analysis does not have an apparent connection with reducing inappropriate behavior; instead, it usually requires a rationale regarding this as an effective assessment procedure which will inform the development of an effective treatment. While the advantages of conducting a functional analysis should not be overlooked, in some situations it may be advantageous to consider a structural analysis (Conroy & Stichter, 2003; Stichter, Sasso, & Jolivette, 2004; Wheeler, Carter, Mayton, & Thomas, 2002; Wacker, Berg, Asmus, Harding, & Cooper, 1997). Wacker et al. (1997) describe a structural analysis as consisting of manipulating antecedent variables potentially related to the occurrence of a target behavior and examining the effects of these manipulations. While a structural analysis does not reveal the variables maintaining a specific behavior, the protocol may provide a more apparent alternative toward reducing an inappropriate behavior. This method offers the advantage of immediately conducting an examination of potential treatments rather than waiting to develop treatments after examining the results, as in a functional analysis. This immediate introduction of treatment for inappropriate behavior would be more apparent and require a less complex rationale than a functional analysis, which could be helpful in certain situations. Each of these variables may be dependent on the consultant for selecting treatments considered highly effective and for training individuals
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
165
to implement treatments. While each of these factors may be dependent upon the consultant, they may also independently influence acceptability when consumers have prior experience or knowledge of treatments. This type of influence on acceptability would not depend on the consultant to provide information on the treatment, but would rather be a direct influence on the consumers because of their history with the treatment.
SOCIETAL APPROPRIATENESS Schwartz and Baer (1991) proposed several recommendations for improving social validity assessment, such as expanding the definition of consumers to represent more individuals from the community and beyond who may influence the use of treatments. By examining how overall treatment acceptability may be distributed across several influential factors, such as large segments of society which may influence the development and use of treatments, the concept of societal acceptability was developed. Societal acceptability incorporates the influences that arise from a broader segment of society, rather than just those who are typically involved in the development and implementation of specific treatments used for individuals. These influences are generally developed through the evaluation of opinions, arguments, and actions of large groups of people. Laws and legislation such as the Individuals with Disabilities Education Act (2004) have promoted the use of empirically based treatments in the least restrictive environment. This in turn appears to designate treatments without an empirical basis as less acceptable, as well as treatments that are not implemented in the least restrictive environment. Other large segments of society which influence the acceptability of treatments are professional associations, parent organizations, treatment review committees, and university-affiliated training/treatment programs. Each of these may design new treatments, develop ethical obligations, and promote movements that encourage or discourage the use of specific treatments. An example of how research in the assessment and design of treatments has resulted in changes in the acceptability of treatments can be seen by examining the influence of functional assessment technology on the use of particular treatments. Research by Carr and Durand (1985) and Iwata, Dorsey, Slifer, Bauman, and Richman (1982, 1994) has had a profound influence on educational legislation, assessment of behavior, and use of
166
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
treatments that incorporate specific reinforcement techniques rather than punishment. Hanley, Iwata, and McCord (2003) described how, prior to the development of functional analysis methodologies, treatments typically involved arbitrary reinforcement or punishment to overpower unknown sources of reinforcement. With the introduction of functional analysis approaches, the source of reinforcement for behaviors could be determined and treatments could be designed to exploit specific reinforcement contingencies. Mccausland et al. (2004) found that treatments based on functional assessment information for aggressive behavior were more acceptable than treatments which were not derived from functional assessments. These procedures have influenced legislation such as IDEA (2004), which now mandates functional behavior assessments under certain circumstances, and have potentially influenced the acceptability of treatments by making those treatments developed without the assistance of a functional assessment potentially less acceptable due to these legislative mandates. One movement that has been highly influential with regard to educational legislation has been the ideology of PBS. This movement has promoted the incorporation of lifestyle considerations and person-centered values in the development of treatments, as well as less use of consequence-based decelerative techniques (Anderson & Freeman, 2000). This movement has been highly influential within society and has been shown to have a direct influence on the acceptability of treatments by individuals responsible for the development and implementation of treatments. Michaels et al. (2005) described a possible paradigm shift based on the PBS movement. They surveyed experts in PBS and found a decrease in the acceptability of consequence-based decelerative techniques and a decrease in the use of such procedures, although many had used these types of procedures in the past. The experts indicated that ethical reasons were most influential in altering their acceptability of specific treatments, along with the development of more effective treatment alternatives. Their findings revealed that treatment acceptability can change over time, especially in the face of large-scale movements such as PBS, changing ethical guidelines, and the development of new treatment methodologies. All of these influences combine to form the construct of societal acceptability, which is considered a part of overall treatment acceptability. Societal acceptability may influence overall treatment acceptability by limiting the possible types or numbers of interventions that may be considered, by
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
167
pushing for the use of specific types of interventions, and by restricting the manner in which treatments may be implemented. In addition, societal acceptability may influence both consultants and consumers of treatments. The laws and regulations that are developed by larger segments of society have direct influence on consultants, who must abide by ethical and legal guidelines when developing treatments. These laws and regulations may have both a direct and an indirect influence on the consumers of treatments. In many cases the associations, boards, and committees responsible for developing ethical and legal guidelines are composed of or informed by consultants and consumers of treatments. While societal acceptability may directly influence consultants, it may also influence consumers of treatments indirectly, through consultants who modify the treatments they develop to meet ethical and legal guidelines. Societal acceptability may also directly influence consumers of treatments, since they may be members of associations, committees, and so on. Consumers also may be directly exposed to the research and rhetoric associated with movements such as PBS, Gentle Teaching (McGee, Menolascino, Hobbs, & Menousek, 1987), Toughlove (York, York, & Wachtel, 1982), and the like. Legislation such as the No Child Left Behind Act (No Child Left Behind Act of 2001, 20 U.S.C. § 6301 et seq.) and the Individuals with Disabilities Education Act Amendments (IDEA, 1997, 20 U.S.C. § 1401) have mandated that interventions be empirically validated to demonstrate the effectiveness of the interventions. These mandates appear to increase the relevance of treatment acceptability research, due to the disparity between empirically validated treatments and their acceptability (Lerman & Vorndran, 2002). As Linscheid (1993) noted, many treatments with demonstrated effectiveness may not have high treatment acceptability in comparison to treatments without empirical validation. The awareness of possible unwanted side effects associated with treatments is another concern relevant to all levels (consumer, consultant, societal) of social validity. Spreat and Walsh (1994) surveyed members of AAMR and found that likely side effects associated with treatment for aggression were not significantly related to treatment acceptability, but were significantly related to treatment for self-injury. Michaels et al. (2005) found that PBS experts associated secondary effects with ineffectiveness of decelerative consequence-based procedures. The PBS experts stated that secondary effects, such as dangerous physical
168
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
interactions, social isolation, long-term change difficulties, and so on, made the procedures ineffective. It is important to note that side effects associated with a decelerative consequence-based procedure which produces immediate results need to be evaluated, along with any detrimental effects on the person and the environment which may continue for a longer period of time, when implementing a proactive antecedent-based approach that produces less immediate results. Abuse potential as a factor of treatment acceptability was described by Lennox and Miltenberger (1990) as the susceptibility of treatment to misuse by those implementing the procedures. While several treatments have the potential to result in physical abuse, through forceful interactions such as physical restraint, some types of overcorrection, and the use of aversive stimuli, there are other types of more subtle abuse which can result with the use of less intrusive procedures. These may occur with lengthy exclusionary time-out procedures or failing to honor the terms of a behavioral contract. Although these types of abuse may not result in long-term problems for the individual subjected to these treatments, the potential for abuse with these procedures may be amplified for someone who has been exposed to the abuse of these procedures in the past. Progar et al. (2001) found differentially high rates of aggression in a participant exposed to the same treatment implemented by novel staff and staff with whom the participant had a previous history. Their findings indicated that the participant’s history with staff that had previously been associated with frequent demands and possibly aversive situations influenced the participant’s current behavior. Individuals who have a history of being abused with a particular treatment may have a very different perspective on the acceptability of these procedures than someone who has not been exposed to abuse by these procedures. The influence that societal acceptability has upon both consultants and consumers is reciprocal, in that societal acceptability is both directly and indirectly influenced by both consultants and consumers. Consultants and consumers may directly influence societal acceptability by becoming members of the boards and committees that develop laws and ethical guidelines, or through litigation such as class action lawsuits. They may also indirectly influence societal acceptability by developing new treatments or technologies related to treatments, such as functional analysis approaches. Consultants and consumers of treatments may indirectly
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
169
influence societal acceptability through movements or campaigns for certain rights associated with treatments.
STRATEGIES TO IMPROVE THE SOCIAL VALIDITY OF PROCEDURES AND PROGRAMS Some general priorities which should be considered when attempting to develop socially valid procedures should include the following: 1. Explain to consumers why information is being collected and how it will be used 2. Look for problems first 3. Determine how social validity should be measured 4. Determine when social validity should be measured (pre- or posttreatment) 5. Determine who should rate the acceptability of the procedures 6. Determine the conditions under which ratings should be obtained 7. Determine what type of training should be provided to raters 8. Allow time for behaviors to become established
Step 1—Explain to Consumers Why Information Is Being Collected and How It Will Be Used As explained by Schwartz and Baer (1991), consumers need to be informed of why they are being asked to provide information, and they also need to be aware of how the information will be used. Consumers who are granted this type of disclosure may be more reluctant to provide specific information. In addition, consumers who are provided an understanding that the information that they provide will be used to improve the quality of services provided may have a vested interest in the information that they provide. This act of participating in the development of treatments may result in increased appreciation of the treatments that are developed, as opposed to treatments developed without consumers being aware of how their input was incorporated.
Step 2—Look for Problems First By looking for problems first, the aim is to correct problems early on in the process of developing and implementing treatments. It may easier to focus
170
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
on the strengths of a treatment program and disregard potential problems that may be incurred with the treatment. This can lead to problems being glossed over and treatments developed and implemented that are highly problematic. Identifying these problems early on may help to avoid having to modify a treatment after it has been deemed too problematic.
Step 3—Determine How Social Validity Should Be Measured When determining how to measure the procedures of a treatment program, consideration should be given to the level of understanding of the relevant consumers. Different types of measurement will be required to gather information from elementary students with disabilities than from adults with professional training in the development of behavior treatments. One group might require straightforward questions to be read to them and another group might be capable of responding to written instructions requiring responses to comprehensive questions. The method of measuring social validity might encompass allowing clients to experience different treatments, allowing them to choose their preferred treatment, and observing their responses to these treatments, as has been demonstrated by Hanley, Piazza, Fisher, Contrucci, and Maglieri (1997) and Hanley et al. (2005). Hanley et al. (2005) provided a unique example of how the preferences of two children with severe behavior disorders could be incorporated into the selection of treatments with and without a punishment component. They used a concurrentchains arrangement to evaluate the children’s preferences between a FCT procedure with and without a punishment component. They found that both the children preferred the FCT procedure with the punishment component over the FCT without punishment. The FCT procedure with punishment was also found to be the most effective of the two treatment options for both the children. Hanley et al. (2005) described how treatment acceptability instruments such as rating scales and questionnaires were not appropriate for individuals who are limited in their capacity to express their preferences. While others may advocate for a consumer with limited communication, they may not always make decisions in the best interest of the consumer, and may not have enough familiarity with the consumer to make difficult decisions for them. The procedures described by Hanley et al. (1997, 2005) offer a viable means for identifying how a consumer with limited communication skills could
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
171
demonstrate preferences through behaviors within carefully arranged scenarios. If the relevant consumers encompass a large number of individuals, such as community members, it may be necessary to use a method of measurement which can be completed quickly and without requiring a large amount of prompting from the data collector. It would simply not be time-efficient to read through a lengthy questionnaire to a large group of people if the questionnaire had to be individually administered. This might only be accomplished if the questionnaire could be administered to an entire group at the same time.
Step 4—Determine When Social Validity Should Be Measured (Pre- or Posttreatment) When to evaluate treatment procedures may depend upon the availability of relevant consumers, the capabilities of relevant consumers, and time allowances for implementation of treatment. The availability of relevant consumers will be a factor in obtaining both pre- and posttreatment ratings of treatment acceptability. Typically, prior to implementing a treatment, a consultant is ethically obligated to fully disclose the components of treatments in a manner that can be understood by all relevant consumers. By doing this, the consultant can increase the potential social validity of the treatment components when rated by these relevant consumers after treatment implementation. In other words, preimplementation disclosure of treatment procedures should increase postimplementation acceptability. This may be the case only when preimplementation disclosure of procedures informs necessary changes to the procedures in response to various concerns or objections by relevant consumers. When changes to procedures are not made to address concerns or objections by a single consumer, it might be predictive of a future lack of social validity by this consumer, unless other factors such as treatment effectiveness are able to overcome prior objections.
Step 5—Determine Who Should Rate the Acceptability of the Procedures When determining who will provide ratings of procedures, it may be important to know how the information may be used. In addition, it is
172
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
necessary to have a clear understanding of the rules and regulations regarding the use of specific procedures. Some procedures may not be permissible under certain circumstances or according to operation guidelines of a facility or school. These rules and regulations can be used as an initial form of social validity from a more general group of raters who have previously determined some procedures appropriate and others inappropriate. Other raters who provide information on the social validity of procedures should be knowledgeable of the existing rules and regulations in order to incorporate this information into their evaluations. Raters who are not familiar with some of the necessary background information to provide an informed rating may need to be given access to the information or provided training on how to include various information into their ratings. An example of how this might take place could involve training raters on a hierarchy of intrusiveness of treatment procedures that defines certain treatment procedures (such as reinforcement) as less intrusive than punishment-based procedures. These hierarchies may be obtained from some facility operational manuals, state associations, or from textbooks (e.g., Alberto & Troutman, 1999).
Step 6—Determine the Conditions Under Which Ratings Should Be Obtained Wolf (1978) recommended that a set of conditions be established under which raters of social validity could be expected to provide the best evaluations. He suggested that these conditions would most likely consist of educating the raters about treatment options, limiting the use of coercion associated with obtaining acceptability ratings, and ensuring anonymity of the raters. Raters who have no familiarity with a specific treatment recommended by a consultant, or with alternatives to a recommended treatment, may not be capable of sufficiently evaluating the treatment. A consultant could potentially make a treatment seem more or less appealing by withholding certain information about the treatment or providing a limited number of alternatives to the treatment. If a consultant described a treatment but withheld information about potential side effects that may be likely to occur, the rater may be more accepting of the treatment than if the side effects were fully described. A consultant might also make a treatment seem more appealing by describing the treatment in relation to other more severe forms of treatment, such as comparing a time-out procedure with electric shock treatment.
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
173
While the optimal conditions for obtaining evaluations may vary depending upon the raters, these as well as other variables should be considered when determining who should provide ratings of acceptability and under what conditions these ratings should be obtained.
Step 7—Determine What Type of Training Should Be Provided to Raters Wolf (1978) suggested that raters may need to be trained how to observe their behavior and their conditions in order to make accurate judgments. Without this type of training, he noted that raters may not be aware of when a situation changes to their benefit or to their detriment. This type of training may provide raters with the tools they need to more accurately report on their personal situations or the situations of others. This type of training could be elaborate or could consist of a minimal amount of information to make the raters aware of certain distinguishing features associated with a procedure. As a rule of thumb, it might be that the determination for how elaborate the training provided should be is how well it matches the severity of the situation. For example, social validity raters may need more extensive training when rating a treatment which involved highly intrusive components, such as electric shock, or which addressed life-threatening behaviors, such as self-injury. Less extensive training might suffice for social validity raters of treatments involving only reinforcement-based procedures and addressing behaviors which are considered only slightly socially undesirable.
Step 8—Allow Time for Behaviors to Become Established In some cases, it may be necessary to implement a behavior intervention that is less acceptable for a short period of time to allow specific behaviors to become established. Skinner (1969) described how the social acceptability of a program developed by Allyon and Haughton (1962) increased after clients spent a period of time under less acceptable conditions in order to establish a behavioral repertoire. Allyon and Haughton changed the methods used by institutional staff when attempting to get 30 schizophrenic women to enter the dining room during dinner time. The staff were experiencing tremendous difficulty ensuring that all the clients were entering the dining area on time and were exerting great effort using various strategies. Allyon and Haughton changed the staff behavior by instructing
174
Chapter | 6 Enhancing the Appropriateness of Treatment Procedures
them to not exert any effort toward getting the clients into the dining hall. Instead, a bell was rung which indicated the availability of the dining hall to the clients, and any client who went into the dining hall within 30 min of hearing the bell ring would be allowed to eat her dinner. Clients who did not enter the dining room within 30 min missed the opportunity to eat that meal. The results of the study were that all the clients eventually were able to enter the dining hall within 30 min and even shorter periods of time. Skinner explained how this program was initially dependent upon food deprivation until establishment of the behavior of entering the dining hall within a specific period of time after the bell was rung. After the appropriate behavior was established, the food deprivation was no longer necessary. He stated that this type of removal of food deprivation from the program would make the program more socially acceptable. This example is an example of how the acceptability of a program may be enhanced by the presence of a prior program with less acceptable components. This represents a possibility of order effects when evaluating the acceptability of a behavior treatment or program. This progression of acceptability could possibly be influential regardless of the order of the treatments. A prior treatment program that is highly acceptable could possibly influence acceptability ratings of a subsequent treatment. The possibility of order effects influencing acceptability ratings could potentially occur with the acceptability of the goals of treatment, as well as with treatment outcomes.
Chapter 7
Improving the Importance of Treatment Effects Disability Policy Concepts Core Concept 1: Antidiscrimination Core Concept 2: Individualized and Appropriate Services Core Concept 3: Classification Core Concept 4: CapacityBased Services Core Concept 5: Empowerment/Participatory Decision Making Core Concept 6: Service Coordination and Collaboration Core Concept 7: Protection from Harm Core Concept 8: Liberty Core Concept 9: Autonomy Core Concept 10: Privacy and Confidentiality Core Concept 11: Integration Core Concept 12: Productivity and Contribution Core Concept 13: Family Integrity and Unity Core Concept 14: Family Centeredness: Services to the Whole Family
Copyright © 2010 Elsevier Inc. All rights reserved
Core Concept 15: Cultural Responsiveness Core Concept 16: Accountability Core Concept 17: Professional and System CapacityBuilding Core Concept 18: Prevention and Amelioration Conducting Social Validity Assessments Consumer Importance Consultant Importance Societal Importance Strategies for Attaining Social Importance for the Effects of Treatment Ensure that Consumers Have a Clear Understanding of Treatment Effects Provide Normative Comparisons Collect Data on Associated Variables Identify Experts to Evaluate the Effects Conduct Cost Analyses
175
176
Chapter | 7 Improving the Importance of Treatment Effects
The third level of social validity defined by Wolf (1978) involves the social importance of the effects of treatment. This area of social validity has been somewhat addressed in most research in the ABA literature by the emphasis on clinical treatment effectiveness. Justifiably, in many cases, this has been a predominant factor in the judgment of whether a treatment has any value. Witt and Elliott (1985) define treatment as a strategy or approach that is directed toward improving some aspect of functioning. While a treatment that does not produce important effects is truly not a treatment at all. Recognizing treatments which produce clear clinical effects can advance research in social validity by eliminating consideration of ineffective procedures and concentrating on treatments with a clear evidence base. When evaluating the importance of treatment effects with respect to social validity, clinical effects need to be considered along with social effects. Treatments may be considered to have clinical effectiveness when they are determined to have a causal relationship with changes in behavior that are supported by data. This type of clinical effectiveness may not be sufficient to ensure that the behavior changes are valued by consumers. A treatment that increases on-task behavior at a clinically significant level may not be appreciated by consumers unless the on-task behavior is accurate and efficient to the overall relevance of the task. A child whose behavior is frequently off task may be taught to demonstrate high rates of on-task behavior, but unless the on-task behavior produces some quality product, consumers may consider the treatment to be only minimally important. Determining the importance of the effects of a treatment involves the social influences associated with the outcome of a treatment. These influences have been referred to as social importance or social impact and encompass a multitude of changes in social experiences resulting from changes in individual behavior. These measures of social impact may be considered secondary to the primary purpose of a behavior change program, but may be a highly critical component of achieving a socially valid intervention. The measurement of social impact can be obtained in several ways, such as surveys, rating scales, observations, and/or peer comparison. Some of the obvious types of social impact may be recognized within child potty training effects, where social stigmatization is associated with public enuresis or encopresis, and behavior interventions which result in successful potty training can result in changes in the social responses from others, such as
Chapter | 7 Improving the Importance of Treatment Effects
Societal importance
Importance of effects
Consultant importance
Consumer importance
FIGURE 7.1
177
Society • Expert evaluation • Cost analysis • Conformity to local/ national norms
Consultant • Evaluation of data • Significant changes in behavior • Normative comparison
Consumers • Understanding of treatment effects • Quality of life • Empowerment/selfdetermination
Distributive model of treatment effects.
being allowed to attend certain daycare or preschool settings. Figure 7.1 illustrates a distributive model of treatment effects. The examination of data is essential for recognizing the importance of treatment effects. Kazdin (1994) described three types of data collection important for evaluating social validity: comparative data, subject evaluation data, and social impact measure data. Comparative data consists of obtaining measures of behavior from a pool of individuals considered to be functioning well or within a range considered normal. This normative range of behaviors would then be compared to behaviors selected for treatment in order to clarify specific deviances that exist. By making these types of comparisons, the effects of treatments can be validated against behaviors normal or typical for a specific situation. Subject evaluation consists of having consumers collect data on how important some of their individual behaviors are to them and then using this information to develop treatment goals. These procedures might be modified for consumers who are incapable of collecting personal data by allowing others closely connected to them to collect the data for them. This type of data may help to prioritize the most significant behaviors to select for treatment. After a treatment has been implemented, these same behaviors can be reevaluated by the consumer to determine if the same
178
Chapter | 7 Improving the Importance of Treatment Effects
behaviors are still considered high priority, or if they are considered less detrimental to the consumer because of the treatment. Social impact data can consist of measures of behavior contained in official reports, such as school office referrals or agency injury reports, police records, and the like, or unofficial reporting, such as parental complaints or teacher narrative accounts. Each of these sources of data can be used to enhance the development of treatment goals by providing a written record of occurrences of certain events. These same sources can then be used to compare pre- and posttreatment records for number of occurrences and for other relevant factors noted in the recordings, such as intensity, situation, other individuals present, and so on. Fawcett (1991) described how the goals, procedures, and effects of treatments could be individually validated on at least three different levels. The levels described by Fawcett appear to involve validation at macro, medial, and micro levels. Macro levels involve validation procedures which focus on the global or overall impact associated with the goals, procedures, or effects of treatment. Micro levels of validation focus on highly specific influences of treatment goals, procedures, and effects. The medial level of validation appears to provide a middle ground between the macro and micro levels where validation can take place (see Table 7.1). The goals of treatment could be validated on a broad social context, by specific behavioral categories, or based upon the discrete responses associated with accomplishing the goal. Validating goals at the broad social level might involve compiling lists of the overall associated objectives related to the goal. For example, validating a broad social goal for aggressive behavior might include improving
TABLE 7.1 Validation Levels of Social Validity Data Treatment Goals
Treatment Procedures
Treatment Effects
Macro
Broad social goals
Comprehensive programs
Distal
Medial
Behavioral categories
Treatment packages
Intermediate
Micro
Discrete responses
Specific techniques
Proximal
Chapter | 7 Improving the Importance of Treatment Effects
179
relationships with other, developing new relationships, and avoiding negative consequences. Validating goals at the medial level of behavioral categories might involve making comparisons to behavioral norms associated with the goal. For example, a goal targeting reductions in aggressive behavior might include increasing problem-solving skills, improving communication, or increasing access to preferred reinforcement. Validating goals at the micro level might involve determining specific behaviors associated with acquisition of the goal. For example, reducing aggression might involve implementing a differential schedule of reinforcement, implementing a functional communication training program, or using a specific punishment procedure. Treatment procedures at the macro level would include comprehensive treatment programs designed to address a number of problem behaviors and associated problems. These types of programs are typically developed for use with large numbers of individuals over an extended period of time. Medial-level treatment procedures would include treatment packages containing more than one specific behavioral technique to address a common problem. This may be the most frequently used form of treatment procedures. However, it may be difficult to determine the degree to which each of the techniques contributes to the behavior change. The micro level of treatment procedures involves specific behavioral techniques. This level of treatment procedures allows the contribution of the specific technique to the behavior change to be evaluated in isolation from other techniques. The macro level of treatment effects involves distal effects which include changes that are generally related to the behavior change. An example of this might be a treatment outcome which focuses on improving personal hygiene and produces an increase in acceptance by groups of individuals. The medial level of treatment effects involve intermediate effects, or those effects which are more closely related to the behavior change. Using the example of improving personal hygiene, an intermediate effect might involve increased interactions from existing friends. The micro level of treatment effects involves proximal outcomes which are the most closely related to the behavior change. The example of improving personal hygiene might, at the proximal level, include increased frequency of teeth brushing or hair washing. Fawcett stated that validation of the social importance of goals, procedures, and effects required precise descriptions of what is to be observed.
180
Chapter | 7 Improving the Importance of Treatment Effects
DISABILITY POLICY CONCEPTS Another means for examining the effects of treatment may involve examining how closely the effects of treatment line up with the concepts contained within disability policy. Turnbull, Beegle, and Stowe (2001) used a combination of literature review, action research, and qualitative data to conceive 18 core concepts of disability policy that affected families who had children with disabilities. The 18 core concepts listed in Table 7.2 offer some areas where treatment effects could potentially be measured. It could be possible to measure the impact that a treatment
TABLE 7.2 Core Concepts of Disability Policy 18 Core Concepts of Disability Policy Affecting Families of Children with Disabilities (Turnbull et al., 2001) 1. Antidiscrimination 2. Individualized and appropriate services 3. Classification 4. Capacity-based services 5. Empowerment/participatory decision making 6. Service coordination and collaboration 7. Protection from harm 8. Liberty 9. Autonomy 10. Privacy and confidentiality 11. Integration 12. Productivity and contribution 13. Family integrity and unity 14. Family centeredness 15. Cultural responsiveness 16. Accountability 17. Professional and system capacity-building 18. Prevention and amelioration
Chapter | 7 Improving the Importance of Treatment Effects
181
has toward instilling or enhancing the component variables that comprise any of the core concepts of disability policy. In other words, the core concepts of disability policy could serve as a frame of reference for deciding on areas to assess change pre- and posttreatment. These changes can provide evidence of social validity at several levels, since the actual changes in behavior can be important to the consumer receiving the treatment, adherence to disability policy may be important to consultants, and compliance with disability policy is representative of effects considered important by society. Each of the 18 core concepts developed by Turnbull et al. will be described.
Core Concept 1: Antidiscrimination The core concept of antidiscrimination refers to avoiding bias against an individual because of his or her disability. This concept may contribute to social validity if instances of equality can be measured and determined to be associated with treatment effects. Turnbull et al. (2001) pointed out that equality should be viewed as to the opportunities that are made available. In other words, equality of treatments may involve extensive accommodations for individuals with disabilities that nondisabled individuals do not receive, but these accommodations are necessary to allow for comparable opportunities for individual both with and without disabilities.
Core Concept 2: Individualized and Appropriate Services The concept of individualized and appropriate services relates to the services that are designed specifically to address the needs of a person and are not simply chosen from a checklist without consideration of the individual’s specific needs. In addition, these services are meaningful to the individual and represent the most appropriate services based on criteria such as effectiveness, intrusiveness, comprehensiveness, and so on. This concept covers all dimensions of social validity (goals, procedures, and effects of treatment) and provides several opportunities for measurement to take place which can provide support for the social validity of a treatment program.
Core Concept 3: Classification The concept of classification is connected to the qualifications necessary to receive particular services. This concept may be difficult to relate to social validity, but may be seen if the goals of treatment are in some way
182
Chapter | 7 Improving the Importance of Treatment Effects
connected to current or potential classification. In some cases, the treatment effects may potentially prevent a more severe classification, by introducing new skills or eliminating behaviors that might have resulted in a classification that implied more severe impairment. The potential for obtaining social validity data related to the concept of classification may be limited, but under certain circumstance this may be possible.
Core Concept 4: Capacity-Based Services The concept of capacity-based services refers to the provision of services that are unique to the individual strengths and weaknesses of an individual, as well as the provision of resources, priorities, and concerns related to addressing these unique characteristics. Social validity can be enhanced by examining the degree to which the unique characteristics of an individual are being addressed through the treatment programs that are developed for the individual.
Core Concept 5: Empowerment/Participatory Decision Making The concept of empowerment/participatory decision making involves allowing responsibility for decision making to rest with the individual who has a disability to the greatest extent possible. Social validity can associated with the extent that a treatment program supports, maintains, or increases an individual’s capacity to participate in decision making.
Core Concept 6: Service Coordination and Collaboration The concept of service coordination and collaboration relates to the provision of services that are “seamless” and do not result in significant difficulties concerning systematic lapses in service or disagreement among service providers. Measuring the occurrence of these types of difficulties and/or the development of processes to prevent these difficulties (such as partnerships, collaborative agreements, concise protocols, and the like) could be used as evidence of social validity.
Core Concept 7: Protection from Harm The concept of protection from harm is concerned with ensuring that individuals are safeguarded against potentially detrimental situations or
Chapter | 7 Improving the Importance of Treatment Effects
183
consequences. Social validity could be demonstrated by documenting the removal of procedures with detrimental side effects, eliminating selfinflicted harmful behaviors, documenting increases in self-protection strategies, and the like.
Core Concept 8: Liberty The core concept of liberty reflects individuals’ freedom from actions that would prevent their physical movements. Social validity could be reflected in the determination of treatment effects which result in increased physical freedom, such as removal of restraints or seclusionary procedures.
Core Concept 9: Autonomy The concept of autonomy refers to individuals’ right to act on their own behalf. Social validity could be achieved through this concept by measurements of self-determination or self-direction.
Core Concept 10: Privacy and Confidentiality The concept of privacy and confidentiality refers to the prevention of personal information from becoming public. Social validity could be equated to increases or decreases in the public availability of information that are associated with a treatment program.
Core Concept 11: Integration The concept of integration has to do with having permission to participate in activities that are typical for someone who does not have a disability. Social validity could be reflected through measures of varying degrees of integration associated with treatment effects.
Core Concept 12: Productivity and Contribution The concept of productivity and contribution relates to how individuals with a disability are able to offer something of value to themselves, to others, or to their community. Social validity of treatment effects could be measured by determining the extent to which individuals with disabilities consider themselves to be offering something of value to
184
Chapter | 7 Improving the Importance of Treatment Effects
themselves or others. This could also be achieved by determining the extent to which others in the community value the contributions of the individual with the disability.
Core Concept 13: Family Integrity and Unity The core concept of family integrity and unity refers to how policies and practices benefit the union of families. Social validity could be related to how well the treatment effects support the cohesion of family units. This type of measurement could comprise something as simple as counting the number of family contacts pre- and posttreatment.
Core Concept 14: Family Centeredness: Services to the Whole Family The concept of family centeredness is concerned with offering services that can meet the needs of the entire family unit. Social validity could be accounted for by the comprehensiveness of the treatment programs to include provisions for all the members of a family unit when programs are developed for a person with a disability.
Core Concept 15: Cultural Responsiveness The core concept of cultural responsiveness deals with how well programs address the diversity within individuals with disabilities and their families. A measure of social validity could be comprised of detailing the extent to which a treatment program recognizes diversity and responds appropriately to diversity.
Core Concept 16: Accountability The concept of accountability involves measurement of expected outcomes. Turnbull et al. (2001) discussed a number of issues related to accountability, such as procedural safeguards, legislative oversight, budgetary oversight, and so on. Social validity could be related to the extent to which a treatment program promotes or supports measures of accountability. An example of this might be a contingency contract which introduces an impartial third party to rule over any disagreements that might arise between the parties involved in the contract.
Chapter | 7 Improving the Importance of Treatment Effects
185
Core Concept 17: Professional and System Capacity-Building The concept of professional and system capacity-building is concerned with improving the professionals working with individuals with disabilities and the systems that support individuals with disabilities. Social validity of treatment effects could be identified by the influence that they have on building or improving professionals or service delivery systems. An example of this might be when a treatment is developed that requires a qualified person to implement the procedures, or requires new systems to be put in place to monitor the treatment (such as review processes).
Core Concept 18: Prevention and Amelioration The concept of prevention and amelioration focuses on efforts to avoid the occurrence of certain disabilities or reduce the occurrence of certain disabilities. Social validity could be demonstrated through treatment programs that are directed toward preventing behaviors that would further a disabling condition. An example of this would be treatment programs that offer early intervention strategies to prevent innocuous behaviors from developing into severe problem behaviors. Turnbull et al. (2001) describe how these core concepts of disability have multiple relationships with each other and may result in some overlap in actual practice. It should be understood that these core concepts are concerned with disability policy, and basing social validity on these concepts should be considered an opportunity to develop measures that are supportive of social validity. Measuring and relating each of these core concepts and using them as evidence of social validity should be considered only in light of other, more traditional information. These core concepts may be considered to have some general importance to society, so documenting that treatment goals, procedures, and effects are supportive of these core concepts does indicate a level of societal social validity. Figure 7.2 presents a checklist for noting if goals, procedures, and effects of treatment address the core concepts outlined by Turnbull et al. (2001). Using this checklist may allow a consultant to evaluate the extent to which a treatment program attends to the core concepts of disability policy. This information could be used to broaden or refocus the core concepts of disability policy that are included within a treatment program. Identifying the actual measures used to address each of the
186
Chapter | 7 Improving the Importance of Treatment Effects
Directions: Place check in each box for which a treatment goal, procedure, or effect addresses a core concept. Core Concepts of Disability Policy
Goals of Treatment
Treatment Procedures
Treatment Effects
Total Concepts Addressed
1. Antidiscrimination 2. Individualized anda appropriate services 3. Classification 4. Capacity-based services 5. Empowerment/participatory decisionmaking 6. Service coordination and collaboration 7. Protection from harm 8. Liberty 9. Autonomy 10. Privacy and confidentiality 11. Integration 12. Productivity and contribution 13. Family integrity and unity 14. Family centeredness 15. Cultural responsiveness 16. Accountability 17. Professional and system capacity building 18. Prevention and amelioration Column Totals
FIGURE 7.2 Checklist for examining recognition of core concepts of disability policy.
core concepts of disability would potentially be beneficial and could be accomplished by noting the strategies used within each of the individual categories. While identifying how treatments address the core concepts of disability policy does provide some evidence of social validity, additional evaluations should also be conducted on a more individualized basis and be considerate of situational variations.
CONDUCTING SOCIAL VALIDITY ASSESSMENTS Gresham and Lopez (1996) outlined nine practical recommendations for conducting social validity assessments. Their target audience was school psychologists, who could use the recommendations to socially validate interventions developed during behavior consultation activities
Chapter | 7 Improving the Importance of Treatment Effects
187
in school settings. While their focus was limited, the implications for their recommendations are relevant to conducting a comprehensive evaluation of social validity for most any treatment. The nine recommendations they made were as follows: (1) use developmental norms, (2) use a functional/habilitative approach, (3) use semi-structured interviews, (4) use relevant judges, (5) use archival data, (6) use behavioral markers, (7) use concurrent choices, (8) use integrity assessments, and (9) use combined social validation procedures. Strategies for incorporating each of the recommendations by Gresham and Lopez, along with additional strategies, will be described in an effort to develop a comprehensive resource for evaluating the importance of treatment effects when conducting social validity assessments.
Consumer Importance When considering the importance of treatment effects from the perspective of consumers, their personal characteristics should be considered. Semi-structured interviews have been described as one means of gathering important information from consumers and can be highly relevant when evaluating social validity (Gresham & Lopez, 1996). While interviews may offer a great deal of information, they are highly dependent upon the individuals who are available and willing to complete interviews. Using the same interviews over a lengthy period of time could also lead to refinement of the interview process, but it may also have a tendency to promote generalizations across consumers that does not address their specific circumstances or characteristics. This may lead to a very limited reflection of variables considered important. To always use the same measures of treatment effectiveness would not promote consumer importance of treatment effects. Kiresuk and Sherman described how using a limited range of evaluation instruments was insufficient. In our outpatient therapy unit, for instance, there are patients of both sexes, of varying marital status, ages ranging from 16 to 80, education level ranging from less than eighth grade to Ph.D. candidates. The use of the same psychometric or rating device for all patients inevitably led to evaluating some patients on variables that were irrelevant to their particular dilemma or circumstances (1968, p. 444).
Consumer Understanding of Treatment Effects Kiresuk and Sherman (1968) suggested that concrete treatment goals should be developed between the consumer and the consultant when
188
Chapter | 7 Improving the Importance of Treatment Effects
they establish their initial contract. They stated that these treatment goals should be clearly related to tangible activities where progress can be noticeably measured. They described a method for evaluating the effects of treatment elements, referred to as Goal Attainment Scaling (GAS). Their measurement method included three steps as follows: (1) goal selection and scaling, (2) random assignment of the consumer to a treatment, and (3) follow-up on progress of goals and scale values. Their procedure involved goals being selected by a team of professionals who reviewed the information available from the consumer. Bailey et al. (1990) and Maher (1983) have successfully demonstrated alternative methods of selecting goals that involve input from the actual consumer, although they emphasize that sometimes consultants need to promote goals that are not the most preferred by consumers. After goals are selected, they developed a series of potential treatment effects arranged hierarchically from least favorable effects to highly preferred effects. They suggest that at least two of the potential treatment effects be defined well enough for someone who was unfamiliar with the consumer to accurately determine the presence or absence of certain treatment effects. They also associated a score for each of the potential treatment effects. The most likely treatment effect was assigned a score of zero, the least favorable treatment effect was assigned a score of ⫺2, and the most highly preferred treatment was scored ⫹2. Treatment effects considered between most likely and least or most favorable were scored ⫺1 or ⫹1 respectively (see Table 7.3 for general example of a GAS). In addition, they suggested assigning a weight to each of the goals according to the priority of importance of the goal. After the goals have been weighted and potential treatment effects arranged hierarchically, a treatment is implemented for a specified period of time and then a follow-up evaluation is conducted where the treatment effects are rated according to the scale that was developed. Kiresuk and Sherman point out that this offers a chance to make modifications to the treatment program or discontinue treatment when appropriate. Additionally, they offer formulas for converting the scale scores into standardized scores so that the scores can be easily compared. The scores on the scale can be evaluated through visual analysis, using standardized T-scores, or using weighted percentage involvement scores. Each of these scoring procedures may have value, but it is important to understand that different types of scores may be viewed differently by
General Example of Goal Attainment Scale
Scale Levels
Goal 1
Goal 2
Goal 3
Goal 4
Goal 5
Most-preferred treatment effect (2)
81–100% reduction of problem behavior
81–100% fluency of replacement behavior
70–100% increase in socialization activities
70–100% increase in community activities
70–100% increase in work activities
Preferred treatment effect (1)
61–80% reduction of problem behavior
61–80% fluency of replacement behavior
50–69% increase in socialization activities
50–69% increase in community activities
50–69% increase in work activities
Expected treatment effect (0)
50–60% reduction of problem behavior
50–60% fluency of replacement behavior
20–49% increase in socialization activities
20–49% increase in community activities
20–49% increase in work activities
Nonpreferred Less than 50% treatment effect (⫺1) reduction of problem behavior
40–50% fluency of replacement behavior
Less than 20% increase in socialization activities
Less than 20% increase in community activities
Less than 20% increase in work activities
Least-preferred Increase in problem treatment effect (⫺2) behavior
Less than 40% fluency Increase in antisocial behavior of replacement behavior
Increase in community activity refusal
Increase in work refusal
Chapter | 7 Improving the Importance of Treatment Effects
TABLE 7.3
189
190
Chapter | 7 Improving the Importance of Treatment Effects
Most preferred treatment effect (2) Preferred treatment effect (1) Expected treatment effect (0) Nonpreferred treatment effect (–1) Least preferred treatment effect (–2) Goal 1
Goal 2
Goal 3
Goal 4
Goal 5
FIGURE 7.3 Sample graph of goal attainment scaling.
different individuals. Consumers may identify more with simple visual analysis (see Figure 7.3 for a sample graph of GAS data), consultants may consider weighted percentage scores valuable for evaluating treatment effects across different consumers and treatments, and society may require T-scores as evidence for the effectiveness of treatments. Maher (1983) suggested that for programs such as public schools, where goals for reading and mathematics have been predetermined, GAS scores could be evaluated using multiple regression analyses. Pretest scores could be evaluated as predictor variables of posttest scores. Lucyshyn and Albin (1993) suggested that the success of certain modifications within a family behavior support plan could be evaluated by using other aspects of the family for comparison. They indicated that accommodations could be appraised (1) by comparing how congruent the procedures were with the personal characteristics of the child; (2) by comparing the consistency of the procedures with the goals, values, and beliefs of the family; and (3) by the sustainability of the procedures over time. This type of evaluation primarily focuses on how well the procedures align with other aspects of the family that are not considered problematic. Lucyshyn and Albin recommend conducting this type of evaluation by discussing and observing typical routines of the family and noting how well certain procedures support the other activities, characteristics, and preferences of the family. They reported using four questions to guide the evaluation of treatment effects during ongoing implementation and for final evaluations, as follows: 1. Has the child’s behavior, skills, and/or lifestyle improved in a meaningful way?
Chapter | 7 Improving the Importance of Treatment Effects
191
2. Have family members and school personnel implemented the interventions with sufficient fidelity? 3. Do family members and school personnel find the goals, interventions, and outcomes of the support effort acceptable? 4. Has the support effort strengthened the family as a whole? Each of these questions emphasizes factors that may be highly relevant to consumers and would potentially promote the importance of treatment effects for consumers. A treatment effect such as strengthening the family as a whole may be highly relevant to consumers, but the importance may not be readily apparent to consultants or society in general. These factors would most likely need to be correlated with other factors, such as divorce rates, level of family dysfunction, child abuse rates, and so forth, in order to make such treatment effects appear more clearly important to consultants and consumers.
Quality of Life Another concern for consumers when considering the importance of treatment effects is measurement of associated factors that may be correlated with certain effects of treatment. One such factor is quality of life. The importance of some treatment effects could be directly related to measures of consumer satisfaction with their lives. While quality of life is still considered an emerging field, there is some consensus on at least eight core domains that encompass quality of life (Schalock & Luckasson, 2005; Schalock & Verdugo, 2002). Table 7.4 presents these eight domains as described by Schalock and Verdugo, along with three of the most common indicators of these domains. Schalock and Luckasson described how to use these domains by developing a set of interview questions that were specific to each of these domains. These questions could be answered by the individual for whom quality of life was being assessed or by others highly familiar with the individual. An example of some questions developed by Schalock and Luckasson, related to the domain of social inclusion, were as follows: “What are [the child’s] preferred indoor and outdoor activities with other people?” and “How does he act around the various people around him?” (2001, p. 48). Schalock and Luckasson provided an example of how questions for each of these domains could be used to compare quality of life before and after some significant incident in the life of an individual, such as being severely burned. They presented family and residential
192
Chapter | 7 Improving the Importance of Treatment Effects
TABLE 7.4 Quality-of-Life Domains and Indicators Identified by Schalock and Verdugo (2002) Eight Quality-of-Life Domains
Common Indicators of the Domain
1. Emotional well-being
Contentment Self-concept Freedom from stress
2. Interpersonal relations
Interactions with others Relationships Emotional supports Physical supports Financial supports Feedback from others
3. Material well-being
Financial status Employment Housing
4. Personal development
Education Personal competence Personal performance
5. Physical well-being
Health Activities of daily living Leisure
6. Self-determination
Autonomy/personal control Personal goals Personal values Choices
7. Social inclusion
Community integration Community participation Community roles Social supports
8. Rights
Human (respect, dignity, equality) Legal (citizenship, access, due process)
staff the same set of questions, only using past tense to discuss preburn incident behavior and using present tense to discuss and provide examples of postburn incident behaviors. They also conducted direct observations of the individual who experienced the burns. The data from the interviews and the observations were summarized within a table representing pre- and postincident behavior organized by domain. Based on the summary of the information in the table, they were able
Chapter | 7 Improving the Importance of Treatment Effects
193
to determine negative impacts on five of eight quality-of-life domains. Questions highly specific to the individual which could be objectively measured might also be developed when conducting an assessment of these domains of quality of life. Green and colleagues have published a number of studies that demonstrate how specific indices of happiness can be observed (e.g., Green, Gardner, & Reid, 1997; Green, & Reid, 1996). These studies offer a highly objective means for observing behaviors associated with concepts such as quality of life. Another method for measuring quality of life is the Quality of Life Questionnaire (Keith, Schalock, & Hoffman, 1986; Schalock, Keith, Hoffman, & Karan, 1989). This questionnaire consists of 28 questions that are rated on a three-point scale. The sum of all the items on the questionnaire is referred to as the Quality of Life Index. These index scores can range from 28 to 84, with lower scores representing lower quality of life and higher scores representing higher quality of life. The items on the questionnaire can also be examined according to three factors which produce a resulting Quality of Life Factor Score. The three factors are comprised of questions concerning environmental control, community involvement, and social relations. Schalock et al. (1989) demonstrated that the Quality of Life Index could be used as an outcome measure for rehabilitation programs focusing on individuals with mild to moderate disabilities in communitybased living and work environments. Quality of life has been shown to have an inverse relationship with extant consumer needs and to have a positive relationship with intelligence (Keith et al., 1986). Examination of quality of life when evaluating effects of treatments may reveal some relationships and may enhance the importance of treatment effects for consultants. Weisgerber (1991) discussed the importance of recognizing variations in quality of life that are in direct relation to life stages. Weisgerber described four life stages that included: 1. 2. 3. 4.
Birth, infancy, early childhood School years Productive years Senior years
Within each of these life stages, there are some overriding factors that are important in establishing a high quality of life, but these factors also change with regard to their topography and others are transient.
194
Chapter | 7 Improving the Importance of Treatment Effects
For example, factors such as health, safety, and social relationships are important across all life stages, but the type of social relationships needed by a very young child are quite different from those needed by an adult. Very young children need social relationships that provide assistance with personal hygiene, nutrition, health care, and so on. An adult may need social relationships that offer friendship, occupational assistance, sexual fulfillment, and the like. Transient factors may include factors such as work and rearing children. Work-related factors may not be important for young children or retired adults, but may have a high relevance for middle-aged adults. Weisgerber points out that for individuals with disabilities, the transition through these stages may present added difficulties for achieving a high quality of life because of restrictions on movements, social interactions, and potentially high levels of dependency on others. Weisgerber (1991) outlined four factors that influence the successful attainment of a high quality of life. The factors were as follows: 1. 2. 3. 4.
Impact of significant others Individual determination to get things done Understanding individual strengths and weaknesses Social competence
Each of these factors may influence attainment of high quality of life, but these may also be areas where behavioral treatments may be targeted to increase the importance of treatment effects. The impact of significant others can directly influence the quality of life attained by individuals with and without disabilities. When developing behavior treatments, consultants should consider how significant others may be supported, in addition to the individual targeted for behavior change. Families can promote an adaptive, supportive environment for their children, or they may pose physical abuse, psychological abuse, or other maladaptive environments for their children (McCubbin, Sussman, & Patterson, 1983). A behavioral treatment that only focuses on the immediate factors related to changing the behavior of a child may miss opportunities to achieve a larger environmental change which could enhance the child’s quality of life and increase the importance of the treatment effects. Self-determination as a means of improving quality of life may be directly related to the ability to accomplish specific goals. Wehmeyer defined self-determination as “acting as the primary causal agent in
Chapter | 7 Improving the Importance of Treatment Effects
195
one’s life and making choices and decisions regarding one’s quality of life free from undue external influence or interference” (1996, p. 24). Wehmeyer (1999) described his model for self-determination as including 11 component elements that included (1) choice-making skills; (2) decision-making skills; (3) problem-solving skills; (4) goal setting and attainment skills; (5) self-observation, evaluation, and reinforcement skills; (6) self-instructional skills; (7) self-advocacy and leadership skills; (8) internal locus of control; (9) positive attributions of efficacy and outcome expectancy; (10) self-awareness; and (11) self-knowledge. Self-report scales have been developed to measure self-determination and instructional approaches have been developed to address needs related to self-determination. Since the conceptualization of selfdetermination is highly related to quality-of-life issues, it may also be a potentially valuable associated measure to provide support for the social importance of treatment effects and overall social validity. Understanding individual strengths and weaknesses may influence quality of life by allowing an individual to place more emphasis on their strengths in order to achieve success. Individuals who are unaware of their weaknesses may continually place themselves in situations where they fail, whereas individuals who are more aware may choose to place themselves in situations with a higher likelihood of success. For example, a person who has poor gross motor skills may experience considerable frustration when attempting to engage in some sporting activities, but may achieve success in sport-related activities such as being a spectator of sports events or playing video sports games. While experiencing failure and finding alternative methods for overcoming obstacles is important, it should be available as a choice in which an individual is taught the skills necessary to participate in multiple options. Behavior treatment programs may address this by focusing on the strengths an individual has in his or her repertoire and expanding upon these strengths to allow for increased interactions and choice. This focus on strengths is frequently employed within functional communication training programs, where the communication response that is taught is one which incorporates existing skills such as the ability to say “No” or to hold up a hand as a gesture of refusal. Social competence is an area where behavior treatments can specifically address the skills necessary to increase appropriate social interactions. Several examples exist of treatments designed to directly improve skills
196
Chapter | 7 Improving the Importance of Treatment Effects
required to demonstrate increased social competence (see Nikopoulos & Keenan, 2004; Petursdottir, McComas, McMaster, & Horner, 2007; Reeve, Reeve, Townsend, & Poulson, 2007). Considering that in many cases inappropriate behavior may be maintained by social variables, increasing social competence may be readily incorporated into behavior treatments as a means of promoting appropriate behavior and subsequently promoting increased quality of life. The PBS approach to behavior consultation has promoted addressing these larger issues by incorporating large-scale environmental changes to support behavior change programs and promote increased quality of life. Clarke, Worcester, Dunlap, Murray, and Bradley-Klug (2002) provided a case example that incorporated numerous different variables in an effort to evaluate the impact of the positive behavior supports developed to address the self-injurious behavior displayed by a 12-year-old female with a diagnosis of autism and several medical conditions. A number of multicomponent treatment packages were developed to address various academic tasks and transition activities. They used both direct and indirect measures to evaluate several effects of the positive behavior supports. They determined the effects of the treatment packages to include the following: (1) reductions in problem behavior, (2) increases in appropriate engagement, (3) reductions in time necessary to complete tasks, (4) increased happiness behaviors, (5) increased ratings of positive affect, (6) increased positive interactions with adults, (7) improvement in quality-of-life ratings, (8) satisfaction with the procedures among team members, and (9) some evidence of potential maintenance of procedures. This case example offered information on a number of different variables that were considered to be effects of treatment. Considering factors related to quality of life as well as numerous other variables associated with treatment effects may be important when focusing on improving the importance of treatment effects and ultimately overall social validity.
Consultant Importance Evaluating Data The primary concern of consultants regarding the importance of treatment effects has historically been objective measures of treatment impact. Consultants are ethically obligated to recommend treatments that have a
Chapter | 7 Improving the Importance of Treatment Effects
197
demonstrated record of producing the intended results. Achievement of these or similar results in many cases validates the services rendered by the consultant from the perspective of the consultant. Whaley and Malott (1971) described consultant obligations for novel techniques as follows: “When using a new procedure for the first time, the therapist almost has a moral obligation to collect data and perform the treatment so that it can be published as a scientific experiment which demonstrates the effectiveness of the therapy. In doing so, he not only helps the individual under study, but may also indirectly help hundreds or even thousands of other individuals.” (pg. 44). Consultants should put effort toward gathering information from multiple sources that can be used to validate the importance of treatment effects and to ensure that they are meeting their ethical obligations with the treatments they recommend. The consultant is responsible for guaranteeing that treatment effects are important for themselves, for supporting or refuting the evaluations of consumers, and for meeting the necessary requirements to address societal evaluations of treatment importance. The incorporation of direct observation of treatment effects may be an essential element toward determining the relative importance of these effects and validating ratings and comments offered by consumers. Sheridan (1995) has discussed how effects may be rated differently depending on the actual experiences and culture of the rater. A parent rating the importance of treatment effects for a child may provide highly different ratings of the effects than the actual child targeted for treatment. This requires the use of multiple raters to gain an overall average rating and possibly direct observations to clarify inconsistencies. These direct observations may need to be conducted on several occasions in several different environments, especially when ratings differentiate substantially from observation data. Consultants are also responsible for determining how the individual components of treatments support the production of important effects. Dietz (1982) described how the use of treatment packages may be deemed effective by the client or the public if a socially important aspect of behavior is increased or decreased. Dietz explained that it is necessary to demonstrate how the components of a treatment package were influential toward the treatment effects. This provides an example of how both societal and individual importance may be satisfied but consultant importance may be lacking. This would appear to heavily depend
198
Chapter | 7 Improving the Importance of Treatment Effects
upon the background training of the consultant and expectations of the treatment, which should be established when selecting the goals of treatment. Consultants trained in applied behavior analysis may be inclined toward extended examination of the importance of individual treatment components. Consultants trained in positive behavior supports may be more accepting of treatment packages without examination of individual components. The philosophy behind PBS is to incorporate treatments in broad packages that involve lifestyle changes, with less emphasis on evaluating each individual component of a treatment package.
Significant Change in Behavior Consultants need to recognize the differing criteria that may be used by consumers, consultants, and society when evaluating the importance of treatment effects. Individuals within each of these areas may use different information to evaluate the importance of treatment effects, or they may differentially weigh the value of certain information. Some of the basic elements of visual examination of single-subject research can potentially be influential on social validity. Treatment data that clearly demonstrate the immediacy of treatment effects and that have minimal or no overlap between baseline and treatment phases could influence perceived importance of the treatment by the consultant and potentially for a consumer who is provided with an explanation of the data. Some consumers may place a great deal of value in the verbal reports of others impacted by changes in behavior. Complete extinction might be very necessary for a behavior such as aggression, while a behavior such as shouting might only require a noticeable reduction in occurrence. If a family or staff member is being aggressively attacked, they may not consider a treatment to have important effects unless the behavior is completely extinguished. In contrast, treatment for a behavior such as shouting might be considered by family members to have important effects if there is a significant reduction in occurrence and changes that are recognized by comments from others in the environment. Word of mouth can be very influential, and this type of feedback may be highly valued, especially when previous feedback has been overwhelmingly negative and after a treatment is introduced the verbal feedback becomes predominantly positive. Consultants may value information that is highly objective, such as direct observation, which may not always be in agreement with verbal
Chapter | 7 Improving the Importance of Treatment Effects
199
reports. Society may value evidence regarding beneficial changes within the overall system, such as reduced staffing requirements and the like, when considering the importance of treatment effects.
Maintenance of Treatment Effects Maintaining treatment effects over a period of time is an important dimension of any behavior treatment. Consultants need to be knowledgeable of the extent to which treatment procedures and effects are upheld or are likely to be upheld after the consultant discontinues or minimizes his or her involvement with the treatment program. Kennedy (2002b, 2005) has described how the continued use of treatment procedures over an extended period of time after consultants are no longer involved provides evidence of social validity. Kennedy referred to this continued use of treatments as sustainability and explained that if a treatment is sustained over time, it must be representative of some degree of social validity. Kennedy explained that while sustainability of a treatment over time may have an apparent relationship to social validity, it may also be influenced by other factors, such as organizational procedures, policies, laws, and so forth. Kennedy (2005) describes these factors as limitations of using sustainability as an indicator of social validity. Another way to consider these factors may be from a more inclusive conceptualization of social validity that considers procedures, policies, laws, and the like developed by segments of society as indicators of societal social validity. These other factors that may be influencing the sustainability of a treatment may make sustainability an unreliable indicator of social validity among consumers directly connected to the treatment. Alternatively, the development of policies, procedures, laws, and so on is an indicator of social importance among segments of society. Kennedy (2005) applied the concept of sustainability as evidence of social validity to the prevalence of published research on social validity. He provided evidence that social validity data are infrequently reported in the professional literature (see Carr et al., 1999; Kennedy, 1992) and implied that measurement of social validity would not be considered valid based on the sustainability criteria. While this may be correct at the level of the consultant or practitioner, at other, broader levels, segments of society representing the professional literature have not dropped the concept and still include the concept within the stated purposes of the journals. This would indicate that
200
Chapter | 7 Improving the Importance of Treatment Effects
although the measurement procedures of social validity are not considered useable or appropriate at the level of the practitioner, broader segments of society still find the concept of sustainability socially valid.
Normative Comparison Gresham and Lopez (1996) recommended using developmental norms as a means of evaluating social validity. The use of developmental norms can be a useful method for determining specific treatment goals and for evaluating the effectiveness of treatments by comparing current development with what is expected at various developmental levels. Several developmental norms have been developed that can provide a reference for designing goals and evaluating treatments. Similarly, norms have been developed for areas other than development which focus more on functioning levels. One such measure is the Assessment of Social Competence (ASC; Meyer et al., 1985). The ASC provides a range of social skills for individuals with disabilities that are divided into a hierarchy of functioning. The social skills are categorized according to the functioning that the skill serves in various environments. Definitions of behaviors used to accomplish each social function are provided and ranked across four levels of increasing sophistication and degree of social acceptance. Using this type of hierarchically ordered norms can be a useful method for determining the importance of treatment effects. Pretreatment social competencies can be determined and compared to posttreatment competencies in order to determine progress along the hierarchy.
Societal Importance Community Expert Evaluation Supporting the local and national norms of society is important for acquiring a high level of societal validity. One method of measuring social validity for a treatment program is to gather feedback from community experts knowledgeable in areas relevant to the treatment outcomes. Whang, Fawcett, and Matthews (1984) used local businesspeople as experts for evaluating the social validity of a treatment program designed to increase on-the-job social skills. Similarly, Seekins, Fawcett, and Matthews (1987) evaluated the social validity of a treatment program designed to increase consumer advocacy by obtaining ratings on the
Chapter | 7 Improving the Importance of Treatment Effects
201
program outcomes from a local politician and a local public administrator. Each of these programs was noted by Schwartz and Baer (1991) as an example of how social validity could be obtained from a broader range of consumers which extended beyond those most closely connected to the treatment. While this type of information could be informative and beneficial, Schwartz and Baer pointed out that this type of social validity information would need to be weighed against information obtained from those sources most directly involved with the treatment program. In addition, this type of information from experts could be highly susceptible to false positive ratings of social validity, especially if these experts are being paid for their participation. An expert who is paid for an evaluation may be inclined to provide the type of responses that are expected, in an effort to secure opportunities to receive payment for future evaluations. It is also unknown if a researcher would repeatedly use the same expert if the social validity evaluations received from the expert were consistently poor. This could be remedied by using experts who anonymously reviewed programs and who were selected at random from a general pool. This type of remedy could also potentially further remove the expert pool from the local community, which could also influence social validity ratings due to an unfamiliarity with the resources, opportunities, and general practices of the community where the treatment was implemented. It may be important to have experts who are familiar with the local community and experts who can provide a more general perspective. Some potential questions for experts evaluators are provided in Table 7.5. These questions would need to be modified to obtain the specific type of information desired, but consideration should be given to questions that elicit positive responses rather than more objective responses.
Cost Analysis Evaluating the ratio of costs to aspects of treatment is an important strategy that could prove highly beneficial toward influencing societal social validity. While these types of procedures could be highly relevant to decision making, the methods have been underutilized in fields such as education (Rice, 1997). Levin and McEwan (2001) delineated a number of different terms associated with cost analysis which included costeffectiveness, cost-benefit, cost-utility, and cost-feasibility. They noted that although the terms were related, each included characteristics that
202
Chapter | 7 Improving the Importance of Treatment Effects
TABLE 7.5 Questions for Expert Evaluators to Answer Please provide an example or explanation for each of the questions. Does the individual have the skills necessary to succeed in social situations? Does the individual have the skills necessary to succeed in vocational settings? Does the individual have the skills necessary to succeed in independent/ supported living situations? Does the individual have the skills necessary to succeed in educational settings? Does the individual have the skills necessary to develop/maintain friendships? Does the individual have the skills necessary to achieve/maintain adequate health? Does the individual have the skills necessary to avoid legal infractions? How does the individual compare to others functioning in vocational, social, educational, and living situations? Are the individual’s behaviors permissible in most settings? Are the individual’s behaviors productive? Does the individual’s behavior permit increased opportunities for the person? Does the individual’s behavior conform to the general standards upheld by society? What additional skills does the individual need to succeed in various settings?
made it unique. Cost-effectiveness evaluated the combination of the cost of a treatment and the effects of the treatment on some other outcome. Cost-benefit analysis referred to evaluating the monetary expenses and gains achieved from a treatment. Cost-utility referred to an evaluation of the expenses incurred with a treatment and the perceived value or satisfaction of a treatment. Cost-feasibility involved an evaluation of the cost of a treatment to determine if the treatment could actually be implemented within a particular framework of resources. Each of these types of cost analyses could prove beneficial in providing evidence for societal social validity. Treatments that are found to have the most to offer for the least cost or effort may be deemed more valuable by a society with limited resources.
Chapter | 7 Improving the Importance of Treatment Effects
203
Jacobson, Mulick, and Green (1998) conducted a cost-benefit estimate of providing early intensive behavioral interventions for young children with autism. Their findings indicated that considerable monetary savings could be attained by families and society by providing these intervention services even when varying degrees of skill attainment were estimated for the children. These types of analyses could prove valuable if they were to become more prevalent and the information were to be disseminated in a manner that could potentially influence public policy. Changes in public policy can be an indicator of societal social validity. In some cases, cost may refer to issues other than simply monetary expenses. In these cases, cost may refer to factors such as time and effort. While these cost factors may be most highly relevant to treatment mediators, they may also be important to larger segments of society when making decisions about the allocation of resources. Treatments that involve a great deal of time and effort may require more rotation of personnel to implement the procedures. Some of these issues can be incorporated into the cost analyses previously described. In other situations, it may be beneficial to subject treatments to parametric analyses in order to determine what amount of time or effort is necessary to obtain the expected treatment outcomes. Cooper et al. (2007) describe parametric analyses as involving examination of varying degrees or intensities of the independent variable on treatment outcomes. Parametric studies have been conducted examining such factors as duration of reinforcement (Twohig & Woods, 2001), magnitude of reinforcement (Carr, Bailey, Ecott, Lucker, & Weil, 1998), and frequency of reinforcement (Murray & Kollins, 2000). These types of analyses could be influential in promoting social validity by determining how resources may be distributed more efficiently to produce the most effective outcomes.
Conformity to Local/National Norms Behavior treatments are predominantly used to improve the quality of people’s lives, and this can be more readily achieved by promoting at least a minimal level of conformity with societal standards. Individuals who engage in more drastic forms of nonconformity with societal rules (such as robbery, assault, and so on) typically have encounters with the legal system and may be subject to penalties such as imprisonment. It
204
Chapter | 7 Improving the Importance of Treatment Effects
is not the role of a behavior analyst to attempt to control the lives of consumers or influence them to engage in total conformity with rules or laws. But, behavior analysts do engage in arranging contingencies whereby consumers can have greater access to reinforcement, and in many cases this occurs by increasing conformity to certain rules, guidelines, standards, laws, and the like. In many cases, the rules must also be altered to allow individuals to be successful. Society must uphold certain agreed-upon standards, and the potential societal social validity would be increased if behavior treatments result in fewer breaches of these standards. As stated by Kazdin and Matson: Although conformity is not a goal of training in any general sense, it is definitely an implicit goal for several areas of rehabilitation. Community adjustment will be maximized to the extent that persons who initially are regarded as deviant, can perform within the range of behavior evident in everyday life. It is one matter for a person to be unique in areas that may be accepted in everyday life but quite another for a person to have several signs of being different that perpetuate public rejection and stigmatization (1981, p. 48).
STRATEGIES FOR ATTAINING SOCIAL IMPORTANCE FOR THE EFFECTS OF TREATMENT Incorporating a wide range of variables into analyses of treatment effects may be an important strategy for increasing the social importance of treatment effects. Kennedy (2002b) recommended that nine criteria related to social validity be incorporated into analyses of problem behavior in order to increase the understanding and potential importance of behavior change procedures within social contexts. The criteria are presented as questions to be answered. Answering “yes” to any or all of the nine questions would potentially increase the social validity of the treatment procedures and effects. The nine criteria are as follows: 1. Is behavior change demonstrated in typical settings? 2. Does the intervention promote movement into the least restrictive environment? 3. Is the intervention conducted by families and/or school and community personnel? 4. Is the person’s entire day/week impacted by the intervention? 5. Is the intervention maintained over time? 6. Is the intervention enabling and skill building? 7. Is the recipient of the intervention happier?
Chapter | 7 Improving the Importance of Treatment Effects
205
8. Do secondary consumers value the intervention? 9. Will taxpayers view the intervention as worthy of public funding? This conceptualization is an expanded view of social validity that incorporates numerous variables that can provide evidence of the importance of treatment effects. Each of these nine criteria appears highly relevant to efforts to improve the social importance of treatment effects. Some other strategies that may be beneficial toward enhancing the social importance of treatment effects are as follows: 1. 2. 3. 4. 5.
Ensure that consumers have a clear understanding of treatment effects Provide normative comparisons Collect data on associated variables Identify experts to evaluate the effects Conduct cost analyses
Each of these strategies is described.
Ensure that Consumers Have a Clear Understanding of Treatment Effects In many cases consumers may not realize the importance of certain treatment effects, especially when the effects are not apparent. The relevance of treatment effects that result in prerequisite skills to some more advanced skill may not be readily recognized, and consumers may need to be provided instruction on the importance of acquiring prerequisite skills. If consumers are not aware that certain treatment effects can have important implications, they may have a tendency to disregard these treatment effects. An example of this might be training an individual with severe physical and mental disabilities to turn the body toward the location of a sugary milk mixture; this may initially seem to lack any degree of importance. If the implications of such a treatment are explained as having relevance toward future self-help skills, the same treatment effects may be considered to have more importance.
Provide Normative Comparisons Conducting normative comparisons of consumer target behaviors before and after treatment may be an efficient means for increasing the social importance of treatment effects. Normative comparisons might consist of
206
Chapter | 7 Improving the Importance of Treatment Effects
data collected on similar behaviors of peers, co-workers, roommates, and so on, or data collection may be conducted from more comprehensive published normative tables of behaviors. Either of these methods may influence social validity for consumers and consultants. The use of more standardized norms, such as those included in standardized assessment instruments, may be necessary to influence social validity within larger segments of society.
Collect Data on Associated Variables Rutzen (1973) conducted a study examining the social importance of an orthodontic treatment for malocclusion (crooked teeth). His study compared several potential variables among groups of treated and untreated people diagnosed with malocclusion. The variables investigated included education, occupation, income, courtship status, measures of social position, measures of self-esteem, and personality traits. Rutzen found few differences among the treated and untreated groups and stated that the differences were barely statistically significant. The absence of large differences was attributed in part to the treatment not impacting the socialization of the treated person beyond that of an untreated person who may compensate during social interactions. While this study focuses on a treatment for physical appearance, it does provide an early, well-developed example for examining the social validity of a treatment that included collecting data on a number of variables associated with the treatment. In addition, Rutzen recommended collecting informant and observational data on social discrimination from a representative sample of individuals with diagnosed malocclusion, to determine the seriousness of the problem and to use as a comparison for those treated for malocclusion. The data that are available for demonstrating the importance of treatment effects may be enhanced by consideration of a number of other related factors. While important treatment effects may be represented by data demonstrating significant clinical changes in behavior, obvious practical changes in behavior, and/or sustainability of the effects, examining data on other associated factors may also be valuable. Other factors may include concepts such as quality of life, side effects, selfdetermination, happiness, and the like. While some of these concepts may rely on subjective data, these types of data may still be valuable
Chapter | 7 Improving the Importance of Treatment Effects
207
for providing additional evidence for potential indirect effects of treatment. If these types of factors are seen as valuable by consumers, they may be influential toward enhancing the social validity of the treatment effects. In addition, some of these associated variables may be examined through objective observational data such as happiness indices (see Green & Reid, 1999). Graphing these associated variables in conjunction with data on treatment effects may be an efficient method for conveying the data to others in a manner that can be easily understood.
Identify Experts to Evaluate the Effects Appropriately identifying the individuals who rate the effects of treatments is highly important toward attaining social validity. These experts need to have specialized knowledge regarding the treatment effects, a vested interest in the outcomes of treatment, or both. In some cases, it may be most important to obtain expert raters who will experience associated benefits from the treatment effects displayed by the individual targeted for treatment. These experts may include family members, friends, direct-care staff, teachers, peers, classmates, and the like. In other cases, it may be important to obtain individuals who have specialized knowledge which allows them to objective rate the performance quality of a certain set of behaviors. This type of expert can be especially helpful, particularly when attempting to produce effects that may be habilitative across a wide range of environments. For example, a child may have the quality of his or her singing rated very highly by family members, but a trained professional voice coach may be highly critical of the same singing. While each of these experts’ ratings could be considered valid evaluations, it may be dependent upon how the overall goal of treatment is defined as to how appropriate their ratings are for the situation. A treatment which results in a nonverbal child singing a song may be rated with a high level of social validity by the child’s parents, because they have defined the overall goal of treatment as helping the child become verbal. The professionally trained voice coach may rate the same singing as having low social validity because the overall goal has been defined as helping the child become a professional singer.
208
Chapter | 7 Improving the Importance of Treatment Effects
Fawcett (1991) recommended a number of experts who should be considered as raters of social validity as follows: 1. 2. 3. 4. 5.
Similar clients or consumers People most affected by the problem behavior Paid and unpaid helpers Those allocating or providing funding for programs Those with media influence (e.g., journalists, elected officials, consumer advocates) 6. Researchers Each of the experts recommended by Fawcett may be viable to include in an evaluation of the social validity of treatment effects. However, consideration should be given in advance as to how the ratings will be used and how limitations by particular raters might be addressed. It may be difficult to obtain ratings that are consistent across all of these various types of raters, so it may be valuable to determine in advance which of these rater’s input is most important to the type of treatment effects that are being evaluated.
Conduct Cost Analyses Determining the viability of treatments in relation to the expense, time, and/or effort required to implement the treatments could have a potentially strong influence on the social validity of the treatment. If the same treatment effects can be achieved from a treatment that utilizes fewer resources, the social validity of this more efficient treatment can potentially be enhanced. This type of information must be considered as a single influential factor among several others that might be weighted more heavily. For example, evidence for a highly cost-efficient treatment may increase the social validity of the treatment, but if the treatment incorporates severe punishment procedures or is connected to unwanted side effects, the cost-analysis information may become somewhat irrelevant.
Chapter 8
Social Validity and Ethics Social Validity and Ethical Values Aversive Procedures and Social Validity Social Validity Within Ethical Guidelines How Social Validity Can Support Ethical Guidelines Competence
Professional and Scientific Responsibility Respect for People’s Rights and Dignity Concern for the Welfare of Others Contribution to Community and Society
Ethics are a critical component of most any professional field. The presence or lack of ethical behavior within a profession can have considerable impact on the individuals receiving professional services. Skinner (1953, 1971, 1974) discussed naturalistic ethics as involving individuals’ history of reinforcement. According to Skinner, actions considered ethical were those that sought to maximize reinforcement for themselves and for the norms of society. Hayes and Tarbox (2007) describe ethical conduct as being comprised of two phases. The first phase consists of making a comparison between different courses of action that may be available. This comparison is made with recognition of some standard to what is considered right and wrong. The second phase of ethical conduct involves taking a course of action based on the comparisons made in the initial phase. The description of ethical conduct provided by Hayes and Tarbox is based on a behavioral perspective that is different from most conventional interpretations of ethical conduct, in that it does not presume the need for conscience choice, personal virtue, or the power of will to be utilized by the individual engaging in ethical conduct. Hayes and Tarbox explain that this difference may lead to misinterpretation of situations where values are highly involved, since conventional thinking Copyright © 2010 Elsevier Inc. All rights reserved
209
210
Chapter | 8 Social Validity and Ethics
deems values to fall outside the realm of scientific consideration. Hayes and Tarbox state: “The notion that societal values are reflected in the actions of societal members under particular sets of conditions, along with the consequences of those actions, presents a sizeable opportunity for the field of behavior analysis” (2007, p. 693). From a behavioral perspective, the contingencies in place are responsible for the occurrence of ethical conduct and are reflective of societal values. Societal values can be reflected in the laws, policies, rules, guidelines, and so on that provide the framework for society. Turnbull, Wilcox, Stowe, Raper, and Hedges (2000) provided a rationale for the inclusion of PBS in the 1997 reauthorization of the Individuals with Disabilities Education Act (IDEA, 1997). They indicated that the Fifth and Fourteenth Amendments to the Constitution of the United States provide numerous justifications of the inclusion of PBS with IDEA 1997. As stated by Turnbull et al.: These constitutional grounds are (a) substantive due process, (b) procedural due process, (c) least restrictive alternatives and the liberty/autonomy interest (including requirements of justified governmental purpose, means-ends correlation, efficacy, and therapeutic purpose), (d) equal protection, (e) judicial deference to professional judgment (the doctrine of presumptive validity, and (f) the requirement of nexus between person, intervention, and place (2000, p. 225).
They provide specific examples of how each of these grounds is relevant to PBS as a component of IDEA 1997. In addition, Turnbull, Wilcox, Stowe, and Turnbull (2001) provided guidelines for agencies using PBS in compliance with IDEA 1997 provisions. These guidelines focus on including PBS in IEP development and in disciplinary situations. Some of the guidelines they provide include such strategies as ensuring proper understanding of PBS procedures, being well trained in implementation and decision-making processes, ensuring that the PBS will be beneficial for the student, and maintaining detailed and accurate documentation of all aspects of the process of selecting and developing PBS techniques. Meller, Martens, and Hurwitz (1990) found that treatments with low acceptability were correlated with increased propensity to blame the psychologist who developed the procedures and to engage in legal recourse. They compared treatment acceptability ratings with ratings on a scale designed to measures variables including liability for injury. Their findings suggest that treatment acceptability data may have some
Chapter | 8 Social Validity and Ethics
211
value toward predicting the potential likelihood of litigation related to various treatments. While this is an area that needs additional research, it could provide valuable information toward establishing appropriate safeguards when implementing various treatments.
SOCIAL VALIDITY AND ETHICAL VALUES Ethical standards are derivations of the values within society and fluctuate with the knowledge base and conditions present in a society. Wolf (1978), in conceptualizing social validity, simultaneously promoted the associated ethical considerations necessary in applied behavior analysis. Examination of the validity of goals, procedures, and effects of treatments incorporates the values present within a society upon which ethical standards are based. Basically, an examination of social validity has considerable overlap with an examination of ethical values. Obtaining social validity data from parents who frequently use physical punishment with a child may also be reflective of the ethical values of the parents, which could be based on limited information such as memories of the techniques that were used by their parents on them. Social validity data from consultants may be reflective of the ethical values within their training programs, professional organizations, employment, and the like. Social validity data obtained from larger segments of society may reveal ethical values that are dependent upon popularity of movements, the prevalence of rhetoric, economic conditions, and other factors. Increasing measurement of social validity and utilizing the information may be another method for incorporating ethical values into the practice of applied behavior analysis. Although more direct examinations of ethics are necessary, social validity does assist in incorporating the values of society and offers a method of checks and balances for behavior treatments (Adkins, 1997).
AVERSIVE PROCEDURES AND SOCIAL VALIDITY Having checks and balances for behavior treatments is warranted; without these measures, the potential for misuse of procedures increases. While these insurances are necessary, they must also be considered in light of the situations in which a treatment is being used. One of the best examples of this was offered by Linscheid (1993). Linscheid described using electrical stimulation in a series of cases and some of the associated difficulties in acquiring permissions, regardless of the evidence on
212
Chapter | 8 Social Validity and Ethics
success of the treatment and the potential risks associated with the clients’ behaviors. Linscheid described that the treatment was developed to treat pediatric rumination, in response to a request for assistance from a pediatrician who “practically demanded that we do something to produce a significant and rapid change in the rate of rumination” (1993, p. 347). The treatment was for a 9-month-old infant who had been hospitalized for 6 weeks for rumination for which no medical reasons could be determined. The infant was losing weight and at risk of dying if the rumination continued. A procedure involving increased attention for the absence of rumination and withdrawal of attention when rumination occurred was found to be ineffective. The treatment consisted of connecting a device to the infant’s leg which delivered a half-second of electrical stimulation upon a treatment mediator pressing an activation button. The electrical stimulation was provided at the first sign of rumination and terminated as soon as the rumination stopped. The treatment resulted in immediate decreased rates of rumination and by the third day of treatment no rumination occurred. The treatment remained in place for 10 days, after which no further treatment was deemed necessary, as the infant had gained a substantial amount of weight. Another case described by Linscheid involved a 14-year-old girl who engaged in rumination and self-induced rectal prolapsing. The rumination was successfully treating using electrical stimulation and permission was requested to use electrical stimulation for the rectal prolapsing. Linscheid stated that permission was obtained after several months, but they were unable to procure a behavioral psychologist who was willing to implement the treatment and the girl ultimately died while receiving an operation to address the rectal prolapsing. Linscheid stated: “Our anger arose from the fact that we were sure there was an effective behavioral, nonsurgical treatment for both her rumination and her rectal prolapsing, and it was only bureaucratic regulations produced by misconceptions and fears that prevented her from receiving the treatment” (1993, p. 351). In another case described by Linscheid, contingent electrical stimulation was found to be successful toward eliminating the severe self-injurious behavior of a 12-year-old female, but after moving back to her home in an institution the self-injurious behavior returned. At this point permission to continue the electrical stimulation as treatment was denied by a state official. Linscheid reported that the girl continued to engage in the
Chapter | 8 Social Validity and Ethics
213
self-injurious behavior for more than 12 years, despite numerous other procedures that she was provided. The examples provided by Linscheid (1993) illustrate a clear clash between societal values and the values held by consumers and consultants. In these cases, it appears that the overall social validity of the procedures was mixed, with the consumers and the consultant deeming the procedures as socially valid while larger segments of society found the procedures to not have social validity. It is interesting to note that Linscheid initially believed that the electrical stimulation would be a highly traumatic experience for the 9-month-old infant described earlier. After implementing the treatment, he found that the infant appeared to not experience trauma and appeared happy and interactive, which was in contrast to his predictions. He stated, “If anything, I was ‘shocked’ by how simple, nonintrusive, and effective the program proved to be” (1993, p. 348). The perceptions of electric shock were changed for Linscheid following actual implementation of the treatment. Larger segments of society do not have the opportunity to experience such implementation, and this leads to consideration of how the availability of experience and information may be influential in shaping the values of individuals within society. Linscheid commented that most people were concerned about the use of electric shock because of fears associated with misuse of the procedures and from misunderstandings of the procedures. Some attempts to influence the social validity of procedures have focused on training to increase knowledge of procedures. These studies have primarily focused on the use of medications for treating disorders of attention. The results of these studies have shown mixed results on how improving knowledge may influence treatment acceptability. Power, Hess, and Bennett (1995) found that combined knowledge of ADHD and knowledge of treatments for ADHD were only minimally correlated with treatment acceptability. Vereb and DiPerna (2004) found that knowledge of treatments for ADHD was uncorrelated with treatment acceptability, and knowledge of medication was negatively correlated with treatment acceptability among teachers. While these studies indicate variable influence of knowledge of treatments on treatment acceptability, they did not include decelerative procedures such as electric shock. Such procedures may be qualitatively different from other procedures because of the controversy associated with their use and the potential side effects. As noted
214
Chapter | 8 Social Validity and Ethics
by Irvin and Lundervold, “decelerative procedures can cause physical and psychological discomfort or distress, for individuals with whom they are applied and for those who apply them” (1988, p. 332). As described by Linscheid (1993), the ethical debate over the use of aversive procedures in behavior treatments has been filled with misconceptions, lack of information, lack of regard for effectiveness data, emotional responses, and censorship. Linscheid was involved in the development of one of the most frequently referenced treatments in the debates regarding the use of aversive treatments, named the Self-Injurious Behavior Inhibiting System (SIBIS; Cunningham & Linscheid, 1976; Linscheid, 1993; Linscheid, Iwata, Ricketts, Williams, & Griffin, 1990). In summarizing the debate over the use of electric shock, Linscheid stated the following: There has been an assumption that anyone willing to conduct this kind of research must be either ignorant of positive programming procedures, backward in his or her approach to treatment, or unconcerned about the rights and welfare of disadvantaged individuals with disabilities. There has also been an assumption that a belief in only positive procedures is equated with caring and that advocacy for the occasional, restricted use of aversive conditioning procedures when warranted somehow indicates a lack of compassion. I think it is unfortunate that this controversy has been relegated to this level (1993, p. 363).
The debate over the use of aversive procedures appears to be highly connected to measurement and understandings of social validity, as the stances promoted among professional organizations are influential on consultant activities and their use and/or acceptance of certain procedures. These professional organizations in some cases may reflect the values of larger segments of society and may possibly be influential on shaping the values of larger segments of society. Based on the debates, ABA developed a task force to examine client’s rights with regard to behavior treatment (Van Houten et al., 1988). The document that resulted offered six statements concerning the rights of clients, as an ethical guide for practitioners engaged in behavior treatments, and was later adopted with a majority vote by the ABA as a position statement (see Table 8.1). These client rights provide some ethical guidelines for behavior analysts when developing treatments, but, as indicated by Bernstein (1989), they do not directly address the debate over the use of aversive procedures. While the debate is not directly addressed by the statements, a client’s
Chapter | 8 Social Validity and Ethics
TABLE 8.1
215
Right to Effective Behavioral Treatment
Six Statements Regarding the Right to Effective Behavioral Treatment (from Van Houten et al., 1988) Statement #1: An individual has a right to a therapeutic environment. Statement #2: An individual has a right to services whose overriding goal is personal welfare. Statement #3: An individual has a right to treatment by a competent behavior analyst. Statement #4: An individual has a right to programs that teach functional skills. Statement #5: An individual has a right to behavioral assessment and ongoing evaluation. Statement #6: An individual has a right to the most effective treatment procedures available.
right to the most effective treatment procedures available does appear to endorse the use of aversive procedures when deemed appropriate. Considering the need for behavior analysts to become involved in public policy, Bernstein recommended an ongoing dialogue as she stated: “Behavior analysts must continue to discuss the social validity of our work with individual consumers of our services, policy makers, and policy implementers. It is a dialogue we need in order to be able to provide interventions that are both effective and acceptable” (1989, p. 97). While the ABA International (ABAI) has developed a position statement on the right to effective treatment, other associations, such as The Association for the Severely Handicapped (TASH) and AAMR, have taken stances against the use of aversive procedures. The APBS has promoted a highly influential movement, especially within school settings, that focuses on nonaversive treatments. The PBS movement uses procedures developed in applied behavior analytic research, although procedures that incorporate aversive procedures are not included in the array of options available to PBS researchers. Considering the widespread popularity of this movement, especially within educational settings, it could be implied that aversive procedures are not socially valid. This would be based on societal acceptance of the PBS approach, which does not include aversive
216
Chapter | 8 Social Validity and Ethics
procedures. It could prove beneficial to closely examine movements such as School-Wide PBS to determine how these approaches have been capable of making a substantial impact on educational settings, whereas more traditional ABA approaches have had a limited impact (Heward, 2005). In contrast, behavior analysts are obligated to provide consumers with the most effective treatment available. The PBS approach emphasizes predominantly antecedent-based treatments and those incorporating reinforcement procedures, while excluding punishment procedures (Carr et al., 2002). Research by Brown, Michaels, Oliva, and Woolf (2008) indicates that experts in ABA and PBS are currently less accepting of punishment procedures than they were during previous decades. The primary factors related to these experts’ decreased use of punishment procedures were ethical considerations, perceived ineffectiveness of punishment procedures, and research-based alternatives to punishment. Leslie (1997) has pointed out that promoting positive reinforcement while minimizing punishment procedures is a potentially effective public relations policy, but this type of promotion also promotes some ethical judgments. These three ethical components described by Leslie were that positive reinforcement is more morally acceptable than punishment procedures, that positive reinforcement does not generate unwanted side effects, and that positive reinforcement is equally or more effective than punishment. Leslie went on to clarify that these views that are being promoted are not necessarily supported by the research literature. Leslie pointed out that in some cases, functional analysis data may suggest the potential value in implementing a punishment procedure, such as with severe self-injurious behavior. In contrast, social validity data consistently demonstrate a lack of acceptance of punishment procedures. Essentially, social validity data can serve as a counterbalance for evaluating the incorporation of punishment procedures. Although punishment techniques are consistently considered less acceptable than reinforcement procedures, and there does appear to be an indication among large segments of society that treatments including aversive procedures are not considered acceptable, there may still be instances where punishment procedures are necessary. Hanley et al. (2005) indicated that punishment procedures were a necessary component of treatments for two children with severe behavior disorders; if
Chapter | 8 Social Validity and Ethics
217
treatment options had been limited to reinforcement procedures, as promoted within the PBS approach, the children would not have received effective treatments. Their study has additional relevance in that the two children were given the option of receiving treatment without a punishment component and the children repeatedly chose the treatment that included a punishment component. The researchers implied that this preference for punishment was indicative of acceptance of the treatment procedures. This study is interesting in that it provides the perspective of consumers who were actually receiving a treatment, rather than another rater who had never experienced the treatment. In addition, the treatment was a punishment procedure, which has consistently been considered less acceptable than reinforcement procedures in numerous studies. The elements of this study reflect comments made by Linscheid (1993), who pointed out that negative perceptions of punishment procedures were frequently voiced by those who had never had firsthand experience with these procedures and that having direct experience with such treatments could change these perceptions. Wolery and Gast (1990) described how social validity was relevant to the debate on the use of aversive procedures on at least three different levels. The first level involved utilizing social validity as a criterion that must be established prior to using a particular treatment. This criterion appears reasonable on the surface, as treatments that are considered unacceptable should not be used. But if this is looked at more closely, based on the sample of stakeholders from which social validity data are collected, there may be considerable variation in the degree of acceptance of a treatment. The use of social validity as a criterion for selecting a treatment is reasonable, but it should be recognized that specific stakeholders may be more or less accepting of a treatment, and unless data are collected from a representative sample, the data are subject to bias. The second level on which social validity was relevant to the debate on the use of aversive procedures involved using social validity as a criterion in determining the value of a manuscript submitted for publication. As indicated by Wolery and Gast, inclusion of social validity data enhanced the overall importance of a research study and increased the likelihood of having a journal article be accepted. The third level described by Wolery and Gast was that social validity data had been used to justify both sides of the debate on the use of aversive procedures. They acknowledged that
218
Chapter | 8 Social Validity and Ethics
while there was general agreement on the social significance of the goals of treatment which focused on decreasing maladaptive behavior such as aggression and self-injury, disagreement existed as to the appropriateness of the procedures and the importance of treatment effects. Turnbull (1981) and Turnbull et al. (1986) discussed the potential use of an interest analysis to examine the interests of stakeholders in a given situation. An interest analysis was described as a potential debating tool that could be used to weigh and balance the interest of stakeholders. Turnbull et al. included this technique in developing a model for evaluating the moral aspects of aversive treatments. The technique is similar to the model of social validity discussed throughout this book, in that it considers the interest of several stakeholders, including the consumers such as the persons receiving the treatment and their families, the consultant or therapist, and society, as well as the system that is in place.
SOCIAL VALIDITY WITHIN ETHICAL GUIDELINES In order to examine how social validity and ethics can be highly correlated, it seems necessary to examine the different types of ethical standards that influence the practice of applied behavior analysis. Several organizations provide ethical guidelines or standards regarding professional practice using behavior treatments. The American Psychological Association (2002) provides ethical standards for professional practice. These standards are based on a set of five guidelines that are considered principles for psychologists to aspire towards (see Table 8.2). The first principle, of beneficence and nonmaleficence, encourages psychologists to offer some type of benefit to the consumers and to avoid doing anything that might be harmful to their clients. The second principle, of fidelity and responsibility, deals with psychologists’ relationships with others and ensuring that their services are clearly understood and provided in a professional manner. The third principle, of integrity, involves accuracy, honesty, and truthfulness in practice. The fourth principle, of justice, relates to fairness and avoidance of biases in practice, as well as recognition of the limitations of practice for a psychologist. The fifth principle, of respect for people’s rights and dignity, refers to maintaining an appreciation for the rights and welfare of clients and the community in general.
Chapter | 8 Social Validity and Ethics
219
TABLE 8.2 American Psychological Association Ethics Code 2002 General Principles of the American Psychological Association Ethics Code 2002 Principle A: Beneficence and Nonmaleficence Principle B: Fidelity and Responsibility Principle C: Integrity Principle D: Justice Principle E: Respect for People’s Rights and Dignity
The Council for Exceptional Children (CEC) offers eight principles which encompass their ethical code for educators who work with people with exceptionalities. Similarly, the Behavior Analyst Certification Board® (BACB) has developed arguably one of the most comprehensive set of rules of conduct for behavior analysts who acquire this type of certification. These guidelines are extensive and are posted on the BACB website. While each of these sets of guidelines is helpful toward gaining an understanding of the differences and commonalities among the ethical positions of various organizations, they are also beneficial toward understanding social validity. The values emphasized by these organizations represent areas of practice that have some degree of social importance. This is evidenced simply through the time and effort expended to develop the ethical guidelines to support activities in these areas. In addition, most all of these guidelines refer to respecting individuals and society, which can be accomplished by incorporating information on social validity. In other words, to behave ethically within professional services involving behavior treatments, there must be some recognition of the preferences, wants, and needs of consumers and society. Broadly defined, social validity offers a means for meeting this type of ethical requirement by offering methods for integrating societal concerns.
HOW SOCIAL VALIDITY CAN SUPPORT ETHICAL GUIDELINES Schalock and Luckasson (2005) stated two reasons that professional organizations have formalized ethical principles: first, to describe their
220
Chapter | 8 Social Validity and Ethics
system of moral behavior, and second, to describe the rules of conduct with respect to particular actions carried out by members of the organization. Schalock and Luckasson summarized the ethical principles that were contained in most organizational documents into five basic principles, which included: (1) competence, (2) professional and scientific responsibility, (3) respect for people’s rights and dignity, (4) concern for the welfare of others, and (5) contribution to community and society. These principles, as related to social validity, can provide a framework for conducting social validity research. Each of these will be described.
Competence First, behavior analysts should become competent in the various conceptualizations and methods used to gather information on social validity. Based on the lack of social validity that is reported within the literature on applied behavior analysis, it could be deemed that there is a competency deficit among practitioners with regard to social validity assessment. In a cursory review of textbooks, the topic of social validity is typically given a limited amount of attention, which would indicate a lack of importance of the topic. In addition, courses in applied behavior analysis may focus heavily on accurate measurement in the assessment of behaviors and the design of treatments, but may provide relatively minimal attention to measurement of social validity. In order to enhance the ethical principle of competence, textbooks and training programs need to consider the relative importance of social validity and how this concept can offer a highly beneficial means for incorporating the most significant, appropriate, and important variables into a behavior treatment program. One reason for this lack of attention may be the subjective nature of this type of data collection and the group design of most research studies. This is a definite break from the objective, single-subject design tradition of applied behavior analysis. Including additional skills sets, such as the use of subjective measures and group research designs, into ABA training programs would seem to only increase the competence of behavior analysts. In addition, these skills may come to be seen as complementary to more traditional ABA methods, and new mixed methods of research could evolve. While adherence to the scientific rigor of ABA methodology needs to continue, several other nontraditional techniques
Chapter | 8 Social Validity and Ethics
221
may prove to have incremental validity, among which the examination of social validity from a broad perspective may be one. Subsequently, expanding the tools and comprehensiveness of ABA may result in an increase in the attractiveness of the procedures and potentially greater acceptance of the procedures within educational and other settings. Viewing nontraditional procedures as supplemental and complementary to traditional measures, without allowing these to overtake or dilute the value of these traditional measures, may increase the survival value of the profession of ABA. By some comparisons, it may appear that the PBS movement, which incorporates an array of non-ABA approaches, has gained widespread acceptance and popularity. There has been constant concern over the public image of ABA, and it may be beneficial to examine some other movements that appear to be gaining momentum in the public eye and incorporating some of these elements into promoting ABA.
Professional and Scientific Responsibility The applied dimension of ABA, as outlined by Baer et al. (1968), requires that target behaviors in applied research be based on their importance to society. Social validity directly addresses this dimension of ABA, offering a method to increase professional responsibility to the field. Improving the examination of social validity would appear to increase the ethical responsibility of behavioral practitioners by demonstrating an expansive effort toward determining the importance of their efforts to consumers and larger segments of society. Without improving and promoting the methods for evaluating social validity, practitioners may resort to less rigorous methods for these evaluations. Based on the prevalence of informal methods of social validity measurements, it appears that it is readily acceptable to develop a social validity assessment composed of a few questions for which reliability and validity are unknown. This type of informal assessment may be necessary in some cases, but appears to have become an acceptable standard within the field. Developing social validity measurements at the drop of a hat, without any need to validate the procedures, seems to lack some scientific responsibility. As stated by Skinner: Scientists have also discovered the value of remaining without an answer until a satisfactory one can be found. This is a difficult lesson. It takes considerable
222
Chapter | 8 Social Validity and Ethics
training to avoid premature conclusions, to refrain from making statements on insufficient evidence, and to avoid explanations which are pure invention. Yet the history of science has demonstrated again and again the advantage of these practices (1953, p. 13).
While waiting for appropriate and satisfactory methods to develop is part of scientific responsibility, it does not seem plausible to do so when some satisfactory methods do exist and without attempting to modify these existing methods. Relying upon scales and measures that have proven reliability and validity contributes to scientific responsibility. In addition, actively developing and validating new measurement scales and measurement paradigms when typical measurements are not applicable may be an area that needs further consideration and would adhere to a principle of professional and scientific responsibility.
Respect for People’s Rights and Dignity By incorporating input from consumers on goals, procedures, and effects of treatments, social validity offers a viable means for respecting the dignity of consumers and recognizes their right to have this type of input. Allowing consumers to have ample opportunity to contribute toward the development and evaluation of behavior treatments affords them a level of dignity. It provides some degree of respect for their opinions and recognizes that it is their life and that they and their ideas are important. Allowing for this type of input can make consumers perceive that they are a part of the process and that they are not being subjected to something that they are unable to influence. While ineffective treatments may be considered socially valid by some consumers, it may still be valuable and ethical to gather this information anyway. A close examination of the aspects that make an ineffective treatment socially valid for some consumers may have value toward improving the social validity of more effective treatments. In addition, it may increase the likelihood of allowing a consumer to recognize the benefits of a more effective treatment if the consumer is allowed to have input at all levels of treatment development. Kazdin (1980b) suggested prioritization of the influence that specific variables have on treatment acceptability, which would provide one means of isolating specific variables to manipulate in efforts to increase social validity of treatment with proven effectiveness. Rasnake et al. (1993) recommended incorporating a large
Chapter | 8 Social Validity and Ethics
223
number of variables into treatment acceptability research in order to address most all of the different influences on treatment acceptability research. Each of these approaches offers possibilities toward respecting the rights and dignity of consumers in a manner that acknowledges the ethical principles of most professional organizations.
Concern for the Welfare of Others Social validity is concerned with measuring the impact of treatment goals, procedures, and effects on not only the direct recipients of treatment but also on others that may indirectly influenced by the treatment. Social validity assessment frequently incorporates treatment mediators, family members, friends, peer groups, etc. This inclusion in social validity assessment offers a method for measuring and ensuring that the welfare of others is considered in treatment programs. Developing a more standardized process for conducting social validity assessments could increase the focus on others welfare, by expecting that a wide array of individuals are included in the process. Conducting social validity assessments in a manner that includes numerous others may also reveal variables that have previously gone unrecognized.
Contribution to Community and Society By examining various stakeholders, social validity provides an excellent method for evaluating the contributions made to society through treatments. Baer and Schwartz (1991) recommended incorporating the views of professional organizations, activist groups, typical taxpayers, and others into social validity evaluations. Recognizing the voices of large segments of society is a clear strength of social validity research and is complementary to promoting ethical behavior. Including techniques such as cost-benefit analyses, expert evaluations, and needs assessments into comprehensive social validity evaluations can provide clear indicators of the degree to which a treatment program is contributing to society. The measures and processes included in a comprehensive assessment of social validity are highly commensurate with adherence to most ethical principles. It appears that advanced efforts in conducting social validity assessments could be beneficial toward promoting ethical conduct as outlined by several professional organizations. If not toward promoting ethical
224
Chapter | 8 Social Validity and Ethics
conduct, it may be beneficial toward identifying areas where ethical principles are being highly recognized and other areas where additional efforts are needed. In essence, social validity data may be one type of tool for evaluating practices in line with established ethical principles or standards. Schalock and Luckasson (2005) outlined relationships between clinical judgment strategies and professional ethical principles and standards. They noted that each of the clinical judgment strategies was embedded in the ethical principles and standards, but also noted that clinical judgment was not the same as professional ethics. The same holds true for social validity assessment. While many of the activities involved in conducting social validity research are embedded in professional ethical principles, the activities are not the same. Ethical principles are aspirational, in that they represent a standard of practice that is always slightly out of reach. However close one comes to achieving the level representative of an ethical principle, it always can be accomplished at a slightly more representative level. While social validity exists on a similar continuum, it focuses on measuring the degree to which validity is actually considered to be present with respect to other measures of social validity. Whereas, in many cases, ethical principles are compared to a standard that could be considered a metaphysical representation, social validity is compared to other similar measures of social validity to evaluate what has actually been achieved.
Chapter 9
Organization of Social Validity Data Micro Level Organization of Data Medial Level Organization of Data Macro Level Organization of Data
Validity of Data Examination of Validity Using Messick’s Principles
Organizing social validity data in a manner that allows for useful evaluation is an important process to consider. If social validity data are not systematically organized and evaluated, any trends that may be occurring may not be recognized. In addition, variables related to social validity may appear to be ubiquitous and the research may provide little toward informing actual practice. Schalock and Kiernan (1990) described how habilitation services could be evaluated using three types of analyses, which they described as outcome analysis, process analysis, and impact analysis. They described outcome analysis as involving objective data sets that focus on individual experiences, such as changes in living skills, community involvement, employment, and so forth. Process analysis involved examination of categorical groups of individuals, services, geographic areas, and the like. Impact analysis offered a comparison of groups receiving habilitation services and those who did not receive services or who received different services. While their focus was habilitation services, these analyses can also provide some useful insight toward organizing and evaluating social validity data, with some modifications. On a larger scale, Schalock (2001) described the concept of outcome measurement as vital in determining the quality and enhancing the Copyright © 2010 Elsevier Inc. All rights reserved
225
226
Chapter | 9 Organization of Social Validity Data
accountability of education, health care, and social services. Schalock described a paradigm shift that was occurring in the field of disabilities that involved increased requirement for accountability of services to ensure that available resources were being utilized effectively. Evaluation of social validity is a measure of accountability that was presented as a challenge by Wolf (1978) and as an ethical obligation of those responsible for implementing applied behavior analytic services. Recognizing, organizing, and measuring outcomes related to treatments may be one method to improve the understanding of social validity. Schalock (2001) differentiated between personal and organizational outcome measures. Personal outcome measures are those that are based on changes or ratings obtained directly from the individual or consumer. Organizational outcomes are those measures associated with evaluating a program in order to improve upon the services provided. Each of these outcomes can be divided into performance outcomes and value outcomes. Performance outcomes focus on factors such as efficiency, effectiveness, competency, levels of functioning, and the like. Value outcomes focus on satisfaction ratings, preferences, rated importance, and the like. Schalock’s Program Evaluation Model, which was based on an integration of the literature on outcomes evaluation, recognized four “cells” in which outcomes could be assigned. These cells encompassed an organizational-performance cell, an organization-value cell, an individual-performance cell, and an individual-value cell. An evaluation of the emphasis on each of the outcomes recognized within each of the four cells was considered to reveal the “personality” of an organization. A predominant focus on the variables associated with the organizationperformance cell indicated a primary emphasis on systematic issues within the organization, such as coordinating service delivery, finances, staffing issues, and so on. An organization with this type of emphasis was considered to have a personality of “stability.” A principal focus on the organizational-value cell stressed supporting the community and consumer, satisfying consumers, promoting competence among staff, and ensuring access for consumers. A major emphasis on outcomes associated with this cell was recognized as having a personality of “outreach.” The person-performance cell focused on outcomes related to an individual’s health, education, finances, level of functioning, and living situation. A primary focus on this cell revealed an organization’s
227
Chapter | 9 Organization of Social Validity Data
personality as being one of “rehabilitation.” The person-value cell highlighted individual variables such as social inclusion, relationships, dignity, self-determination, and personal development. With measurement revealing this cell as the primary focus, an organization was referred to as having a personality of “advocacy.” Considering the usefulness of Schalock’s Program Evaluation Model, a review of measures typically used in social validity research was conducted and arranged according to how the measurement type was most frequently used with respect to consumer social validity, consultant social validity, and societal social validity. Direct and associated measures of social validity related to goals, treatments, and effects of treatments are represented in the Social Validity Measurement Matrix displayed in Table 9.1. While this matrix does not identify the “personality” of a program, it does offer a quick means for evaluating the type of social validity measures being used (direct or associated), as well as the predominant focus of the measurements (goals, procedures, or effects).
TABLE 9.1
Social Validity Measurement Matrix
Direct measurements
Associated measurements
Goals of Treatment
Treatment Components
Effects of Treatment
Interviews
Formal rating scales
Expert evaluation
Questionnaires
Questionnaires
Model comparison
Ranking/prioritization worksheets
Exposure to treatment/trials
Normative comparison
Values/Preferences Informal discussions
Treatment intrusiveness
Clinical effects
Person-centered planning
Functional assessment
Cost-benefit analysis
Goodness of fit
Treatment integrity
Status checks (health, education, financial, etc.)
228
Chapter | 9 Organization of Social Validity Data
Schalock (2001) offered a three-step process for evaluating the major outcomes upon which an organization is focused and initiating redirection of the organization to outcomes that are more desirable. Step one involves determining a baseline of outcomes currently being used by the organization. The measurement of the outcomes in place is accomplished by completing an inventory of current outcome measures, identifying the factors that best describe the outcomes used by the organization. A tally of the factors identified from each of the four cells in the Program Evaluation Model (organization-performance, organization-value, individual-performance, and individual-value) allows for assignment of a “personality” (stability, outreach, rehabilitation, and advocacy) to the organization. Step two of the process involves identifying the outcomes desired by the organization using the same outcome inventory, but selecting the factors that the organization wishes to measure. A tally of selected factors related to each of the categories in the Program Evaluation Model reveals the desired “personality” of the organization. The third step in the process involves determining what needs to occur for the organization to align its services with the desired outcomes. Schalock provides three recommendations for achieving this that include: (1) fostering a culture of change, (2) developing and evaluating strategic plans and performance goals, and (3) implementing an outcomes-oriented monitoring system. The Program Evaluation Model and process delineated by Schalock may prove useful toward organizing, measuring, and evaluating efforts to achieve increased social validity. In many cases, efforts at determining social validity are unevenly distributed and may not be an accurate representation of social validity as initially conceptualized by Wolf (1978). The conceptualization of social validity described by Wolf consists of recognition of three potentially different aspects of treatment. The social significance of the goals of treatment, the social appropriateness of the treatment procedures, and the social importance of the effects of treatments may each require different types of outcome measurement in order to accurately reflect the influence of each area. In many cases, certain areas may be overemphasized or underemphasized. A measurement that primarily focuses on the immediacy of behavior change, the rate of behavior change, and the degree of behavior change from baseline may be overemphasizing the social importance of treatment effects by
229
Chapter | 9 Organization of Social Validity Data
stressing the clinical importance of the treatment rather than the practical or social importance of treatment effects. In other instances, a measure that focuses primarily on the low coercion of a treatment, the lack of side effects, or the low abuse potential of a treatment may be overemphasizing the social appropriateness of the treatment. By adapting the strategies used in conjunction with Schalock’s Program Evaluation Model, it may be possible for agencies or consultants to determine the primary focus on which they base their evaluations of social validity. Procedures frequently reported as part of social validity measurement were reviewed and organized into categories that attempt to reflect the primary emphasis of Wolf’s (1978) three aspects of social validity. In order to follow the model utilized by Schalock for evaluating program outcomes, an inventory of factors related to measurement of aspects of social validity was developed into the Social Validity Measurement Inventory (see Figure 9.1). After the inventory is completed, the number of items circled within each column can be tallied and the sum used to represent the emphasis placed on that type of social Baseline Directions: For each column, circle the descriptions that best depicts the factor or type of measurement currently being used most frequently. Desired Measurement Directions: For each column, circle the description that best depicts the factor or type of measurement that you would prefer to have or consider most valuable. Goals measurement Examination of past goals Consumer values Consumer preferences Consumer future plans Person-centered planning Goodness of fit Consumer expectations Prioritization of goals
Treatment measurement Consumer knowledge of treatment Consumer acceptability Willingness to implement Evidence base/reasonableness of treatment Functional assessment Treatment integrity Complexity/Time necessary to implement Treatment intrusiveness
Goal attainment scaling Potential long-term benefits of goals Potential short-term benefits of goals Habilitative potential
Presence of reinforcement/punishment Replacement behavior
Consumer confidence in goal achievability Comprehensiveness of goals
Side effects
Potential impact of goals on others (family, friends, etc.) Potential influence on quality of life Predicted cost-benefit analysis
Resources available to support implementation Exposure to treatment/treatment analysis
Consumer-perceived change in severity of behavior problem Consumer perception of replacement behavior Consumer competency with replacement behavior Status level (health, education, social, financial, etc.) Impact on others (family, friends, etc.) Quality of life
Consultant experience with treatment
Cost-benefit analysis
Baseline # Desired #
Baseline # Desired #
Baseline # Desired #
FIGURE 9.1
Presence of coercion Consumer confidence in treatment
Abuse potential
Social validity measurement inventory.
Effects measurement Overall change in behavior Immediacy of behavior change Degree of behavior change Consumer satisfaction with effects Expert recognition of effects Normative comparison Clinical outcomes Influence on habilitation (self-help, recreation, etc.) Baseline comparison Peer comparisons
230
Chapter | 9 Organization of Social Validity Data
validity measurement. The inventory can be completed by an individual or by multiple respondents, in which case the average number of items circled per category can be used to represent the overall measurement emphasis. The inventory can be used as a baseline measure of the type of social validity measurements currently in place and as a method for establishing targets for the types of measurements that the individual or group desires. More or less emphasis on specific types of social validity measurements may depend on the purposes and/or mission of the individual or group. As noted earlier, Schalock (2001) provided three recommendations for organizations that were attempting to align their services with the outcomes they desired. These recommendations included: (1) fostering a culture of change, (2) developing and evaluating strategic plans and performance goals, and (3) implementing an outcomes-oriented monitoring system. For individuals or organizations that are attempting to develop, modify, or evaluate their measurement of social validity, it may be important to incorporate a number of processes to support these efforts. Some of these processes, along with variations of these analyses, will be described within each of the three levels of data organization.
MICRO LEVEL ORGANIZATION OF DATA The organization and evaluation of data at the micro level are primarily involved with capturing social validity as noted by the consumers of treatment. At this level, the focus is on data related to a specific implementation of a treatment or treatment program. The consumers of the treatment are local and highly involved in the receipt, development, or implementation of treatment. Micro level data are the most prevalent social validity data and possibly the most difficult to summarize. The difficulty with summarizing this type of data is that they are directly related to the individual characteristics of the consumers, which varies the type of data collected and the instruments used to collect the data. Numerous studies report social validity data using author-developed instruments or variations of validated instruments. The rationale for author-developed instruments appears to be related to the wide range of behaviors, goals, and characteristics of consumers, which may not be readily addressed by more standardized instruments. While this is
Chapter | 9 Organization of Social Validity Data
231
understandable, it does make comparisons difficult, because it is difficult to compare the social validity of similar cases when different instruments have been used to measure social validity. In addition, these instruments are not standardized and may contain items that may or may not be related, depending on the conceptualization of social validity on which the instrument was developed. Regardless of the wide range of variation that exists, it appears that this type of data may be most useful for gaining an understanding of the social validity of a specific application of a treatment for a specific consumer and using the information to develop future treatments. Components of a treatment considered to have high social validity may be incorporated into future treatments in an effort to ensure that these future treatments also have high social validity. In addition, it may be valuable to consider the characteristics of consumers and any others who may be evaluating social validity. Figure 9.2 offers an example how this type of information might be organized. This figure arranges treatment according to the level of intrusiveness, as described by Alberto and Troutman (1999), or as a medical treatment or cognitive-based treatment. Some broad characteristics of raters are provided, although these could be modified to include more specific characteristics if this was considered viable. The treatment components can be listed at the top of the figure and provided a code which can then be used within the grid. By completing this type of table for several treatment programs developed by a consultant, it may be possible to examine the data for potential trends or patterns relevant to a consultant’s caseload. This information may only be useful to consultants who are attempting to gain a better understanding of the treatment acceptability for the programs they are developing and implementing over a period of time. They could then possibly compare these data with similar data collected by other consultants, in an effort to identify areas or methods that consistently result in high acceptability for certain broad categories of clients, behaviors, or raters. An additional use of these types of data may be to evaluate the activities of consultants who may design a social validity instrument, to gain a better understanding of the process they are using to develop and implement treatments. Consultant may use this information to gain an
232
List Treatment Components with Level of Intrusiveness 1)______________________________ Level I treatment
Level II treatment
2)______________________________ Level III treatment
Level IV treatment
Medical treatment
3)______________________________
Cognitive-based treatment
High acceptability
Moderate acceptability
Low acceptability
Client Child Adult Mild behaviors Moderate behaviors Severe behaviors Social behaviors Academic behaviors
Child Adult Client Parent or relative Teacher Hired staff Consultant Other Instrument Informal TEI or variation IRP or variation Other instrument
FIGURE 9.2
Table for organizing social validity data.
Chapter | 9 Organization of Social Validity Data
Raters
Chapter | 9 Organization of Social Validity Data
233
understanding of what works for them and what does not work within their practice. The consultants can then possibly modify their consultation approach and potentially impact the overall social validity of a treatment. One strategy that a consultant could employ would involve examining the similarities and differences of conducting social validity measurements at different points in time during the consultation process. By conducting these types of examinations, the consultant may be able to target a specific point or activity used during the consultation process when measures of social validity change. For example, a consultant may measure social validity prior to implementation of a treatment and determine that there is a high degree of acceptability for the treatment. The consultant may then again measure the acceptability of the treatment immediately following implementation of the treatment and determine that acceptability has changed dramatically. In this scenario, the consultant may be inclined to examine factors related to actual implementation of the treatment, such as implementation difficulty or possibly the occurrence of extinction bursts. The consultant could then evaluate his or her own practices related to training treatment mediators to a higher degree of fluency with treatment procedures or incorporating more strategies to respond to extinction bursts. Another strategy that consultants might use to evaluate their own practices might be to compare the social validity of similar treatment programs offered to consumers with similar treatment goals while using different consultation approaches. Differing formal consultation approaches could be compared, or the same approach might be used but with more or less emphasis on certain activities, such as interviews or follow-along activities. Using instruments that have been validated has the advantage of allowing for some broader comparisons. This is still somewhat problematic, since the characteristics of consumers vary widely and these have been shown to influence social validity. Although consumer characteristics vary widely, it should be possible to collect enough data over a period of time to make some comparisons among consumers with similar characteristics or among similar treatments. It might also be possible, when using the same social validity instruments, to investigate possible changing trends among consumers with similar characteristics. These trends would need to be validated against other evidence, possibly from trends determined among professionals or among trends evident within society.
234
Chapter | 9 Organization of Social Validity Data
Rasnake (1993) stated that a better way to provide social validation of behavior treatments was needed. She recommended that multiple variables and multiple dimensions which may influence acceptability be included in evaluation of treatments. She indicated that these evaluations should concern themselves with identifying social validity related to specific variable combinations, such as treatment techniques, individual clients, and varying situations. The implementation of this recommendation could prove extremely difficult or impossible following the previous model of large-group analogue research which has dominated this area of research. Rasnake suggested that social validity may be only realistically examined on a case-by-case basis and that more global statements about acceptability would not be valid. This type of case-bycase evaluation of treatment acceptability has been conducted in several empirical studies which include information on the acceptability of their treatments. Although several of these case-by-case evaluations exist, they vary widely in the type of information obtained and the formality of the instruments used. At present, no attempt has been made to summarize the findings from these types of acceptability evaluations.
MEDIAL LEVEL ORGANIZATION OF DATA One recommendation which could prove beneficial toward gaining a better understanding of treatment acceptability might come from an attempt to organize and analyze the information obtained from several case-by-case evaluations. Schalock and Kiernan (1990) described process analysis as including a description of the individuals to whom services are provided, the actual services provided, the associated costs, and the aggregated outcomes. This type of analysis is similar to a medial level organization of social validity data, wherein the primary concern is to determine the social validity of treatments across a number of unrelated consumers. Schalock and Kiernan recommended that a process analysis of habilitation services include characteristics of individuals served, types of services provided, costs of services provided, and aggregated outcome data. Each of these areas can be useful in medial level organization of social validity data. The characteristics of consumers receiving treatment is important to consider. Schalock and Kiernan suggest three criteria to follow when selecting personal characteristics: selecting characteristics that are measurable, selecting characteristics that will potentially have
Chapter | 9 Organization of Social Validity Data
235
strong effects on outcomes, and selecting characteristics that have variation. The types of treatment provided can also be organized according to the specific type of treatment, according to specific combinations of treatments, or according to level of intrusiveness. The costs associated with treatment may involve an examination of the monetary expenses incurred or an evaluation of the amount of time and effort required to implement a treatment. Several studies have demonstrated that the time required to implement a treatment is a highly relevant factor influencing social validity. Other outcomes that may be relevant include employment status, social activities, and quality of life, among others. Treatment precedence refers to previous effectiveness of a treatment with a specific class of behavior. Lennox and Miltenberger described both a local and national precedence of treatments. Local precedence of treatments could be determined from use of the treatment within an immediate social or professional community, while national precedence could be determined from reviews of literature. Lennox and Miltenberger cautioned that precedence should be considered as a factor in treatment acceptability only when it is accompanied by evidence for the effectiveness of the treatment, since ineffective treatments may be used frequently. Treatment precedence as described by Lennox and Miltenberger appears different from treatment history associated with a specific individual. Treatment history refers to an individual’s past experience with a specific treatment. The difficulty in considering treatment precedence as a factor in treatment acceptability is in how the specific classes of behavior are derived. Behaviors may be classified by the topography of the behavior or by the function of the behavior. The precedence of effective treatments could vary depending on whether the treatment is applied to a topographically defined class of behavior, such as aggression, or to a functionally defined class of behavior, such as escape maintained behavior.
MACRO LEVEL ORGANIZATION OF DATA Lennox and Miltenberger recommended that treatment acceptability research be compiled into a database to represent an area such as a school district and then used by school personnel in determining what types of treatment were more or less acceptable in their district. This would require a significant period of time and some level of standardization
236
Chapter | 9 Organization of Social Validity Data
within the evaluations, but could potentially provide some insight into the general acceptability of treatments in certain areas, for certain consumers, and under certain circumstances. While this type of information might initially only be relevant for practitioners working in certain regions of the country, with certain clients, or under certain circumstances, it might eventually prove useful toward developing a database which could reflect how treatments might be accepted. While this type of database would most likely need to be continually updated, and would need to include several subcategories, it might provide a means for determining trends, movements, or biases associated with certain treatments, regions of the country, or other factors. This type of analysis would not be unlike the current methods used to examine the political inclinations of states within the United States. One depository of information on the social validity of treatments can be found within the professional literature. Wolery and Gast (1990) pointed out a trend in the literature that occurred in the 1970s, following publication of research on the effectiveness of overcorrection procedures. They indicated that there was an increase in publications on the same topic, coming from researchers at different locations across the country. Based on the increased prevalence of overcorrection procedures during this period of time, they suggest that the procedure was socially valid during that time period. They also used this example as an indicator that social validity was variable over time, by implying that the procedures were currently not seen as acceptable. They also provided a number of other examples of how social acceptability of procedures could be tracked through time that included institutional placement of individuals with mental retardation, use of behavior techniques without a prior functional assessment, and segregated schools. While their technique is valuable, it should be noted that tracking social validity trends by examining prevalence rates within professional literature may lead to some false positive conclusions. Wolery and Gast suggested that dissemination potential of professional research may be judged on the availability of social validity data, but within the professional literature these data are rarely reported. When these data are reported, they will most likely reflect adequate social validity, because instances of inadequate social validity would most likely not be reported. In addition, several other factors are involved in determining
Chapter | 9 Organization of Social Validity Data
237
the appropriateness of a manuscript for publication in a professional journal. These types of reviews and bibliometric analyses may provide some evidence for trends in social validity, but may not offer any reasons for why these trends may be occurring. As stated by Critchfield (2002, p. 426), “bibliometric methods describe, but do not explain.” In essence, determining trends based on prevalence rates of procedures within professional publications should be clarified with additional supporting information, such as conference presentations on the procedures made during the same time period, statements or data from behavior consultants practicing during the same time period, establishment of specific movements, popularity of movements, and the like. Only through examination of several variables will this type of trend analysis reveal any rationale for how these trends developed or could potentially be shaped. Some examples of how survey data can be used to gather macro level treatment acceptability data and track trends that may be occurring with the use of specific treatment procedures have been demonstrated within recent literature (e.g., Brown et al., 2008; Michaels et al., 2005). These studies gathered and compared data on PBS and ABA experts’ willingness to use various treatment procedures in their current practices and in their professional practices during previous decades. Information on rationales for using and not using decelerative consequence-based treatments were obtained along with specific circumstances where these procedures might be considered valuable. Statistical comparisons were performed with quantitative data collected, and sorting and classification of qualitative data were also conducted. These studies reveal highly interesting potential trends among these experts that imply a paradigm shift toward decreased acceptance of aversive procedures. In addition, several rationales were identified to account for this paradigm shift, such as ethical reasons, availability of alternative techniques, personal experiences with people with disabilities, and the ineffectiveness of punishment procedures to produce long-term behavior change. These rationales could be valuable toward developing future research protocols that could evaluate the influence of these rationales on social validity by manipulating variables such as specific ethical training, training on procedures that could be used as alternatives to punishment, or provision of increased personal experiences with individuals with disabilities. Organizing social validity data
238
Chapter | 9 Organization of Social Validity Data
in a manner similar to that exemplified by Michaels et al. and Brown et al. could provide a useful manner for examining the data and determining trends and areas in need of additional research. Turnbull et al. (1986), in discussing procedures for establishing standards of morality, described three methods that might be used by a society to establish these types of standards. While the focus was on morality, these methods should be examined as to their relevance to organizing social validity data. The first method involved determining what the standards would be if developed by a “reasonable person.” This would involve defining what a “reasonable person” encompasses and then determining how they came to value certain standards. For social validity, this method seems plausible, but also seems limited in scope, and could lead to examination of numerous idiosyncratic variables with little or no relevance to an overall examination of social validity. The second method described by Turnbull et al. involved developing a societal standard from which to make comparisons. The societal standard would be developed by conducting a cross-sectional survey of society. This method could be valuable for establishing some standards against which social validity could be compared, and this type of investigation has been conducted on a small scale, primarily with college students and teachers in the professional literature. The obvious limitations of such an endeavor would involve the limited number of goals, procedures, and effects that could be evaluated and the potential lack of representation of individuals with low-incidence disabilities, although they may be exposed frequently to behavior treatments. The third method described by Turnbull et al. involved having professional organizations collaborate on establishing moral standards. This approach would be valuable for social validity, but would most likely be met with a great deal of conflict. The methods described by Turnbull et al. offer some techniques that may eventually prove to be valuable toward improving how social validity data are organized and examined.
VALIDITY OF DATA Another procedure that may be useful toward organizing and examining data on social validity may be a consideration of the overall meaningfulness of the information being collected and used to represent social
Chapter | 9 Organization of Social Validity Data
239
validity. Validity has been described as an evaluative judgment of the relationship between empirical evidence and interpretations and actions derived from the empirical evidence (Cronbach, 1971; Messick, 1989, 1995). The principles of validity apply to all forms of assessment, including quantitative and qualitative methods, by combining scientific inquiry and rational arguments to substantiate or contradict the interpretations and consequences resulting from the data collected (Messick, 1995). Social validity is a construct that is comprised of numerous different factors, which Wolf (1978) arranged into three categories encompassing treatment goals, treatment procedures, and treatment effects. While several instruments have been validated for use in evaluating treatment acceptability, this comprises only one of the three components of social validity described by Wolf. This type of validity, which focuses on representativeness of test scores and error of measurement inclusive in test scores, is similar to traditional psychometric evaluation. While this type of validity may cover certain instruments used to evaluate treatment acceptability, it does not appear to encompass the larger construct of social validity espoused by Wolf. It should be recognized that multiple types of measurement are necessary for conducting a comprehensive appraisal of social validity, many of which are difficult to submit to traditional psychometric evaluation. Wolery and Gast (1990) questioned whether social validity could be of any use if the construct existed on a continuum without any clearly delineated criteria that could be agreed upon. What Wolery and Gast appear to be concerned with is the validity of social validity. While this seems ironic, it does seem possible to address this issue by examining social validity data against standards imposed on more traditional types of validity frequently used in psychological assessment.
Examination of Validity Messick (1995) proposed a unified concept of construct validity that encompasses all of the traditional psychometric concepts of validity. This unified theory of validity includes six principles that recognize all of the critical issues related to examining validity as a unified concept (see Table 9.2). While this theory of construct validity was developed within a traditional psychological assessment paradigm, it appears to have relevance for evaluation of social validity data. Similarly, this
240
Chapter | 9 Organization of Social Validity Data
TABLE 9.2 Messick’s Six Principles of a Unified Theory of Construct Validity Principle 1: Content Relevance and Representativeness Principle 2: Substantive Theories, Process Models, and Process Engagement Principle 3: Scoring Models as Reflective of Task and Domain Structure Principle 4: Generalizability and the Boundaries of Score Meaning Principle 5: Convergent and Discriminant Correlations with External Variables Principle 6: Consequences as Validity Evidence
unified theory has been used to develop standards for evaluating the validity of functional behavior assessments (Shriver, Anderson, & Proctor, 2001). Messick explained the relevance of using the six principles of construct validity: The six aspects of construct validity afford a means of checking that the theoretical rationale or persuasive argument linking the evidence to the inferences drawn touches the important bases; if the bases are not covered, an argument that such omissions are defensible must be provided (1995, p. 747).
In measuring social validity, it may be crucial to admit that some previous efforts toward measuring this construct have been highly limited, although the implications of these limited efforts have been afforded considerable recognition. By adapting Messick’s six aspects of construct validity to evaluate social validity data, it may be possible to identify if the important bases are touched upon and remediate or provide a defense for omissions.
Content Relevance and Representativeness Messick’s (1995) principle of content relevance and representativeness places importance on the boundaries of inclusiveness that characterize a construct. Messick (1995) explains that both content relevance and representativeness are traditionally measured via expert judgment. Experts judge the adequacy of the range of items included within an assessment to ensure that they are comprehensive with respect to the construct
Chapter | 9 Organization of Social Validity Data
241
domain being measured, cover the most important and meaningful parts of the construct, and are technically sound. When considering how this applies to evaluation of social validity data, at least two factors appear that should be addressed related to the principle of content relevance and representativeness. The first factor involves the representation of the most important components of social validity. Following Wolf’s (1978) conceptualization of social validity, the most important components of social validity are the social significance of treatment goals, the social appropriateness of treatment procedures, and the social importance of treatment effects. Each of these components should be included in a comprehensive measurement of social validity. A second factor to consider under the principle of content relevance and representativeness is the importance of the information that is gathered when conducting a social validity assessment. The variables that are most important in a social validity assessment may vary depending upon the purpose for which the assessment is being conducted. Validating the construct of social validity may be done for different purposes, such as determining consumer social validity, consultant social validity, or societal social validity. For example, social validity assessment conducted specifically for the purpose of understanding consumer social validity may focus heavily on the values, preferences, and priorities of the consumers, while an assessment of societal social validity may require extensive cost-benefit analyses and conformity with societal standards. Messick (1995) explained that construct validation may need to include evidence on the relevance of the data to the applied purpose and the utility of the data in applied settings. Recognition of these different purposes should be part of the overall evaluation of social validity data and may offer a defensible argument for the omission of some forms of measurement.
Substantive Theories, Process Models, and Process Engagement Messick (1995) described the substantive theories, process models, and process engagement principle of a unified theory of construct validity as focusing on tasks that accurately reflect the content being measured and employing consistent empirical evidence. This principle indicates that the focus of some data collection procedures may be relevant and others may not be relevant. A valid assessment should provide evidence
242
Chapter | 9 Organization of Social Validity Data
that the most important components of a construct are being gauged by the items on a measurement scale. This principle relates to examination of social validity data by taking into account instances of professional judgment and requiring some standard formatting that consistently produces similar results. Some examples of this may involve the use of forms for prioritizing treatment goals, using formal questionnaires, and employing standardized treatment acceptability measures.
Scoring Models as Reflective of Task and Domain Structure The principle of scoring models focuses on the internal structure of an assessment and the criteria and rubrics that are used to score the assessment. In traditional psychological assessment, each of these should be consistent with what is known or theorized about the underlying processes that combine to produce effects. Procedures such as factor analysis can determine the consistency of items on a test in relation to a specific construct. Some measures of treatment acceptability have reported on the results of factor analyses, such as the TEI, the IRP, and the AARP. Other measures of social validity may be reliant upon theory, such as the threefactor conceptualization of social validity proposed by Wolf (1978).
Generalizability and the Boundaries of Score Meaning The construct validity principle regarding generalizability and boundaries of score meaning, described by Messick (1995), centers on the reliability of data and ability to make inferences about the data across tasks, time, observers, and raters. For social validity data, reliability data have been reported for several treatment acceptability instruments, but many other instruments do not address this issue. The boundaries of how social validity data can be used to make inferences may depend upon how well the measurement addresses the specific area of concern. For example, measures reflecting a high degree of consumer-related social validity may not imply similarly high social validity among other members of society. At present, it appears that some inferences can be made, such as that treatment procedures which are predominantly composed of reinforcement techniques are generally more acceptable than those comprised of punishment techniques. While this factor may be generalized, other types of generalization do not provide consistent evidence, and additional research is needed to clarify these inconsistencies.
Chapter | 9 Organization of Social Validity Data
243
Convergent and Discriminant Correlations with External Variables The principle of convergent and discriminant correlations with external variables is concerned with comparing assessment data with other measures or behaviors and clarifying any relationships that exist. Convergent relationships indicate that there is a correspondence between the data sets, and assist in substantiating the measurement meaning. Discriminant relationships indicate dissimilarity among the data sets and assist in substantiating the distinctness of a measure. Treatment acceptability instruments have been correlated with other measures, such as the IRP and the CIRP. One problem that seems to exist is the frequent use of informal measures that demonstrate wide variability in the types of items that they include. In addition, more research is needed to determine the relationships among social validity data and other types of data such as quality of life, self-determination, and so on.
Consequences as Validity Evidence The principle of consequences as validity evidence refers to the longterm and short-term decisions that are derived from assessment. Messick (1995) indicated that it was important to accumulate evidence of beneficial outcomes from assessments and ensure that detrimental outcomes were minimized. The intent of social validity assessment is to promote acceptance of behavior programs within society. While this would appear to be beneficial, it can also lead to some decisions that may have detrimental effects. Potential detrimental effects of social validity may occur when a consumer considers an ineffective treatment to be highly acceptable and refuses to accept any other, more effective treatment, or when segments of society condemn a specific treatment even though it may be highly beneficial in certain cases.
Using Messick’s Principles Using the principles defined by Messick (1995) may allow for examination of the rationale underlying obtaining and using social validity data. Table 9.3 provides some questions related to each of Messick’s principles which can be used for examining social validity data. It is not expected that social validity data will always or ever meet all the criteria
244
Chapter | 9 Organization of Social Validity Data
TABLE 9.3 Questions to Consider for Examining the Validity of Social Validity Data Content Relevance and Representativeness 1. Do the data collected represent the goals, procedures, and effects of the treatment adequately? 2. Is the most important information included for the target audience? Substantive Theories, Process Models, and Process Engagement 3. Are devices used that promote consistency in processes, such as memory aids, templates, standard forms? 4. Are interviews conducted in a consistent manner? Do these interviews produce consistent information across those interviewed? 5. Are standardized treatment acceptability instruments used? 6. Does the assessment process involve collection of unnecessary/ unrelated information? Scoring Models as Reflective of Task and Domain Structure 7. Are instruments used that have demonstrated evidence as to their item integrity to a specific construct? 8. Are assessment measures based on a clear conceptualization of social validity? Generalizability and the Boundaries of Score Meaning 9. Does the measurement instrument report adequate reliability? 10. Is evidence available to support inferences made regarding the data? Convergent and Discriminant Correlations with External Variables 11. Does the assessment measure being used correlate well with other similar measures? 12. Are the assessment data distinct from other types of unrelated data? Consequences as Validity Evidence 13. Do the data promote beneficial outcomes? 14. Are there detrimental outcomes derived from the data?
described by Messick, but these questions do provide a method for evaluating the adequacy of the information obtained. When rationales and inferences drawn from social validity data do not adhere to Messick’s principles, there may be reasonable ways to account
Chapter | 9 Organization of Social Validity Data
245
for these instances. While this type of examination may appear extensive, it does appear to address at least some of the questions posed by Wolery and Gast regarding the value of social validity data, by offering a means to evaluate the data. It may also be possible to categorize the overall adequacy of social validity data using Messick’s principles, but it still seems necessary to determine what is or is not enough. These are questions that will need to be determined by dissemination of various amounts of social validity information for different types of cases and evaluating the critiques of these varying degrees of information. Increasing the scientific rigor of social validity data and organizing the data using methods that are easily conveyable should be a major focus of research on social validity. Increasing the validity of social validity assessments can allow for a clearer picture of what is and is not important within society to develop. In addition, improving the measurement of social validity may produce a greater recognition of the need to examine the concept and potentially improve the services offered within the field of applied behavior analysis. Incorporating several different methods for organizing social validity data may be valuable toward recognizing trends or paradigm shifts that have occurred. Further advancement of the identification trends in social validity data could also potentially reveal some strategies for predicting the emergence of new trends or paradigm shifts. At present, the organization of social validity data is very rudimentary, which can be somewhat expected considering that the measurement of social validity is still considered rudimentary. Organizing and using data on social validity to make decisions and determine trends may in turn promote the development of improved measurement instruments and vice versa. These two activities should work together to enhance the field and allow for improved decision making among organizations and individual consultants.
This page intentionally left blank
Chapter 10
The Future of Social Validity Research on social validity has progressed considerably since Wolf (1978) first introduced the term, and a large amount of information has been gathered and discussed. Although a number of variables which are influential on social validity have been discovered, the relative importance of these variables and other contingencies linked to these variables remain unclear. The inconsistent research findings that are apparent in the literature on social validity provide some insight into the difficulty involved in making the findings useful to practitioners. While some variables, such as the use of reinforcement-based procedures as opposed to punishment-based procedures, appear to generally produce consistently higher acceptability ratings, the majority of these variables are highly influenced by other situational or personological variables. In order to provide practitioners information which can be useful, the interaction of a number of variables on social validity will need to be investigated. A practitioner attempting to develop socially valid treatments needs to have a means for determining which variables may be most influential, what relative situational and personological variables need to be considered, and in what combination these variables may influence social validity. To promote the development and use of socially valid treatments and programs, the focus of future research may need to be expanded and assimilated into a more highly organized series of research studies. Although a number of studies have been conducted on a number of variables related to social validity, there has been a lack of consistent methodology, measurement, and examination of trends. Each of these areas appears important to gaining an understanding of the concept of social validity. Past research has incorporated a number of different methodologies, such as inconsistent description of cases and treatments, inconsistent presentation of information (such as audiotape presentation, written summaries, and video presentations), and inconsistent Copyright © 2010 Elsevier Inc. All rights reserved
247
248
Chapter | 10 The Future of Social Validity
recruitment of raters. Inconsistencies in measurement have been seen in the use of a number of different formal and informal instruments which are present in the literature. These measurements have been obtained from raters at various points such as before, after, and during treatment implementation. Comparisons of ratings of acceptability pre- and posttreatment implementation are rare and make most examinations of these findings nearly impossible. The examination of trends over time with social validity is very rare and has not been done in relation to changing social, legal, or educational policies. As mentioned in Chapter 1, the generality criterion described by Baer et al. (1968) for determining if a study analyzes applied behavior may play an important part in understanding the variables influencing social validity. A treatment may be considered to promote generality if the change in target behavior continues after the treatment has been withdrawn, or when the changes in behavior extend to other nontarget behaviors or nontreatment settings. With regard to social validity, many of the influences which appear to influence the acceptability of treatments appear to originate from areas that are not direct components of the treatment. For example, the incorporation of reinforcement or punishment-based procedures could be considered a direct component of treatment which could be readily identified in most cases. In contrast, the severity of the problem behavior displayed by the client may not be considered a direct component of treatment; nor would the consultant’s membership in a professional organization, or numerous other personological variables of those rating the acceptability of the treatment. Although these variables may not be direct components of a treatment, they have been shown to influence the acceptability of treatments. These types of variables could be viewed in a similar manner to the generality of behavior change. Some treatments may be producing changes in areas that are not targeted by the treatment, such as changes in philosophical points of view, changes in treatments implemented with other individuals, or policy changes. Baer et al. recommended planning for generality of behavior change, and this recommendation seems to also be an appropriate recommendation for promoting the social validity of treatments. Once a clear understanding of the indirect variables that influence social validity is attained, planning for the influence of these variables can be incorporated into developing socially valid treatments and programs.
Chapter | 10 The Future of Social Validity
249
Kazdin (1980b) recommended that more research was needed to identify the scope of variables that influence treatment acceptability and to determine how much influence each of these variables has with regard to treatments. He suggested that future research should allow raters to provide input on which variables were most influential in their ratings of acceptability of treatments. This type of information could allow for these variables to be prioritized along a continuum of most-to-least influential upon the acceptability of a treatment. This type of continuum of influence might only be developed with measures that are valid and reliable, and would need to be adjusted according to the type of treatment being evaluated, the description of the client for whom treatment is targeted, and the type of rater who is proving the acceptability ratings. Developing an overall continuum of variables influencing treatment acceptability might prove difficult, but this type of continuum might be possible if incorporated into the distributive model of treatment acceptability (Carter, 2008b) described in Chapter 2. This would allow for specific continuums to be developed which could potentially fit the societal, consultant, and consumer forms of treatment acceptability. The degree to which specific variables influenced treatment acceptability might be determined if applied to each of these categories separately. This type of continuum has already been initiated for the category of consultant acceptability by the research of Michaels et al. (2005), which determined that PBS experts identified ethical reasons as most influential in altering their acceptability of specific treatments, along with the development of more effective treatment alternatives. While this is only the beginning of the development of a continuum of variables influencing consultant acceptability, it does demonstrate how such a continuum might be developed. Schwartz and Baer (1991) proposed extending the assessment of social validity to a larger sampling of consumers. They provided a larger definition of consumers, which included direct consumers, indirect consumers, members of the immediate community, and members of the extended community. Direct consumers were comprised of those who are the primary recipients of treatment. Indirect consumers are those who are not directly treated, but are those who arrange for or purchase a treatment for someone else. Parents who arrange treatment services for their child are an example of this type of indirect consumer. These
250
Chapter | 10 The Future of Social Validity
indirect consumers may also consist of individuals who are strongly affected by a treatment because the treatment is directed toward someone with whom they are closely affiliated. Members of the immediate community are those who may not be closely affiliated with the individual receiving treatment, but may be impacted by the treatment due to frequent interactions with the individual receiving treatment. These immediate community members may include neighbors, classmates, or co-workers. Members of the extended community include members of the same community who do not know the individual receiving treatment, but may observe the individual receiving treatment or participating in community activities. These individuals may not interact with the individual receiving treatment, but may praise or denounce the treatments for a variety of reasons. Schwartz and Baer thus recognized a larger community from which social validity measures should be gathered. Their description of who should be targeted for procuring social validity measurements may be highly relevant toward the future of social validity research. Previous research has been somewhat limited as to who represents relevant groups from which to obtain social validity measures. Their recommendations for an expansion of these relevant groups seems promising and could lead to a better understanding of social validity from a more comprehensive perspective. Bernstein conveyed the importance of exploring social validity issues at several levels: Behavior analysts need to continue to engage in dialogue, both among ourselves and with the larger society, over social validity issues . . . . It is a dialogue we need in order to be able to provide interventions that are both effective and socially acceptable (1989, p. 97).
Continuing to encourage dialogue concerning social validity seems highly relevant with the recent development of movements, such as PBS, which promote the avoidance of aversive, consequence-based treatments. Michaels et al. (2005) reported that PBS experts’ perceptions of aversive treatment procedures had changed over time toward less acceptance of these types of procedures. Additionally, Brown et al. (2008) surveyed ABA experts regarding the treatment acceptability of decelerative consequence-based behavior treatments and found a similar decreasing trend of acceptance of these procedures. Based on these
Chapter | 10 The Future of Social Validity
251
studies, which were limited in their sample size (73 PBS experts and 57 ABA experts), it appears that a similar paradigm shift toward less acceptance of punitive procedures may be occurring for both PBS and ABA experts. This type of research appears to provide an excellent basis for encouraging discussions about social validity within the fields of ABA and PBS, as well as with society at large. Schwartz and Baer (1991) considered one of the primary issues for future research in social validity to be expanding the accuracy of social validity assessments. In addition to recommendations to improve the construction of social validity assessments, they recommended four functional methods for enhancing the measurement of social validity. First, they recommended establishing procedures for comparing the goals and outcomes of treatment against some standard level of performance. Similar to Van Houten (1979), they suggested the possibility of using peer behavior as a means of comparison, but warned that finding appropriate peer models could be difficult in some cases. They pointed out that for residents of state institutions, using peer models from the same institution might result in lowered outcome standards, whereas using models from the community might result in higher standards of competence. They suggested that a decision should be made as to whether current or future environmental standards were more important for the individual targeted for treatment. From a PBS model of behavior intervention, the answer to this question appears to be that both environments are important and standards should be developed for functioning within both these environments. The PBS approach appears to incorporate environmental modifications across several settings and therefore seems to adapt a number of different standards of performance across these different settings. Some research on the social validity of school-wide positive behavior supports has emerged that incorporates a number of variables, such as school safety, procedural acceptance, implementation difficulty, and the like (Metzler, Biglan, Rusby, & Sprague, 2001; Nelson, 1996; Nelson, Martella, & Marchand-Martella, 2002). Kern and Manz (2004) noted that examination of social validity for school-wide positive behavior supports had been limited to Tier One intervention procedures and ratings obtained from school staff. In addition, the measures used in these studies appear to have limited validity and reliability, focus
252
Chapter | 10 The Future of Social Validity
heavily on effectiveness of procedures, and may have a tendency to promote favorable responses from raters. Examination of the social validity of school-wide positive behavior supports is an area that is developing and appears to need additional research to gain a better understanding of the variables related to these procedures. Gresham and Lopez (1996) expanded on how standards could be determined by suggesting the use of developmental norms to determine competent levels of behavior performance. They suggested that these normative data could be used to establish goals for treatment outcomes. In addition, they recommended using information from published literature describing functional relationships of a wide range of behaviors to establish goals of treatment. The information from these published studies could be used as standards to which to compare treatment outcomes. This type of approach appears viable except in cases of lowincidence-type behavior for which representative published research might not exist. Schwartz and Baer (1991) stated that competent behavior could be judged by comparing the behavior of an individual targeted for treatment to others functioning in the community. They also suggested that those completing the judgments of competent behavior should demonstrate that they are qualified to make these types of judgments. They suggested that qualified judges should be determined by referring to the original consumer set to which the problem behavior was most relevant. If the problem behavior was relevant to only a single consumer, such as the parent of a child, then the parent would be considered qualified to judge the outcomes of a treatment. In cases where the problem behavior was relevant to a larger group of community members, then qualified judges would include representative members of the community. Similarly, Gresham and Lopez (1996) recommended that relevant judges be determined based on the specific type of outcomes targeted, such as including teachers to judge academic outcomes and law enforcement officers to judge school violence. The second functional recommendation by Schwartz and Baer (1991) for enhancing social validity assessment involved ensuring that measurements of social validity are obtained as unobtrusively as possible. Typical observation methods may allow consumers to become aware
Chapter | 10 The Future of Social Validity
253
that they are being observed and may ethically require consent prior to conducting such observations. Similarly, the use of questionnaires and interviews can be obtrusive and increase the likelihood that consumers may behave in an atypical manner that is not representative of their usual behavior, or they may provide responses in order to satisfy a personal agenda, such as attempting to appear agreeable or embellishing descriptions of problems. Schwartz and Baer suggested using archival data as a means of obtaining data unobtrusively. Gresham and Lopez (1996) made similar recommendations regarding use of archival data, such as school records, which could be used as evidence to support the social validity of treatments. Schwartz and Baer’s third recommendation involved measuring behavioral correlates of satisfaction. These behavioral correlates are often fundamentally obvious, but may be late to emerge and thereby not allow ample time for the treatment developer to respond in order to remediate problems with social validity. Given this difficulty, these types of behavioral correlates may best be used in combination with other measures of social validity and primarily as supportive in nature. Some examples of these types of behavioral correlates might be attendance at treatment sessions, enthusiasm toward treatment, and discussions of treatment with others. Those who regularly attend treatments and arrive promptly might thereby provide supportive evidence of high social validity with the treatment. Those who do not attend regularly or who arrive late to treatment programs might provide supportive evidence of low social validity. Similarly, consumers who recommend a treatment to others, and show high enthusiasm for a treatment program, could be used as supportive evidence of high social validity. Those who discourage others from initiating the treatment or are unenthusiastic about the treatment might support low social validity. Gresham and Lopez (1996) described the ultimate behavioral correlate or marker for the consultant as “repeat business” in which consultees call back for assistance with future problems. An additional behavioral correlate may involve the number of consumers who decide to seek legal recourse. Meller et al. (1990) noted that treatment acceptability could be a useful tool for risk management programs. They indicated that consumers’ satisfaction with treatments may in many cases be more influential in decisions to seek legal action than
254
Chapter | 10 The Future of Social Validity
the actual outcomes of the treatment. This indicates that consumers may place more value on choosing to undergo a specific treatment than they do on evaluating the eventual outcomes of the treatment. While the value of social validity research to risk management programs needs to be researched further, it is an interesting area that could potentially expand social validity measurement. Francisco and Butterfoss (2007) recommended incorporating measurement of social validity into public health practices. They suggested that social validity could be beneficial to public health practices for several reasons, which included recognition of community involvement in the development of community health programs, increased legitimacy of public health practices, increased consideration of procedure with public health, increased involvement of broader audiences in development and implementation of programs, improved services to the community, and improvements in summarizing public health data. It is interesting that researchers in the field of public health are promoting the use of social validity to potentially improve a number of areas within their field when, in contrast, the field of applied behavior analysis continues to publish very few examples of social validity research. In addition, the measures used to evaluate social validity are considered rudimentary, and there are very few indications of interest in improving the measurement of social validity (Baer et al., 1987). That other fields such as public health are recognizing the value of social validity should come as a wake-up call to the field of applied behavior analysis to vehemently pursue expanding and improving on the measurement and evaluation of social validity. One purpose of this book is to offer some ideas that may serve to spark some initiative toward expanding research on social validity. Although many of the ideas presented may also appear rudimentary, they may serve as a framework for developing new methods for conceptualizing, measuring, and evaluating social validity. The fourth recommendation by Schwartz and Baer (1991) to enhance measurement of social validity involves offering choice. They stated that the ultimate measure of social validity is the range of alternative treatment options that a consumer will reject when choosing a treatment program. A treatment can be described as preferred if it is chosen over a number of viable alternative treatments. Based upon this type of measure, it could be implied that ABA procedures have limited social
Chapter | 10 The Future of Social Validity
255
validity within education settings, because ABA procedures are frequently rejected for alternative approaches in education. Heward (2005) compiled a list of 14 reasons why the impact of ABA on educational practices has been limited (see Table 10.1). Each of these reasons identifies an opportunity for behavior analysts to work toward improving the social validity of ABA procedures within educational settings. Based upon Heward’s reasons, the social validity of ABA procedures could be improved in educational settings by finding ways to make the procedures, processes, and assumptions more compatible with education. As stated by Heward, “The primary goal of behavior TABLE 10.1 Reasons for Limited Impact of ABA Heward’s (2005) List of Reasons Why ABA Has a Limited Impact on Education 1. ABA’s basic assumptions about the purpose and process of education are incompatible with the views of many educators. 2. ABA’s data do not interest educators. 3. ABA’s data do not matter because educational decisions are seldom informed by data on student learning. 4. ABA’s empirical pragmatism is antithetical to education’s retreat from objective science. 5. ABA seems too simplistic. 6. Other approaches promise more. 7. ABA’s use of reinforcement goes against current beliefs in education. 8. ABA is an easy mark for criticism. 9. Some teachers view ABA as a threat to their creativity and independence. 10. ABA places the responsibility for student learning on teachers and schools. 11. Implementing behavioral approaches yields too little reinforcement for teachers. 12. Behavioral educators have insufficient understanding and control of the contingencies that govern the adoption and maintenance of effective practices. 13. ABA has yet to prove its value to the students about whom society cares most. 14. Improving education is not an urgent mission for society.
256
Chapter | 10 The Future of Social Validity
analysts working in education should not be getting education to do more and better ABA: our goal should be helping education do better” (p. 340). Improving education may include making ABA procedures more acceptable within education so that other, less effective procedures can be rejected. Such endeavors might involve researching ways to make ABA procedures more interesting, less threatening, more reinforcing to educators, and so on. In order for consumers to reject alternative treatments, they must be able to make valid choices. Schwartz and Baer (1991) stated that in order for consumers to make a valid choice, they must first have extensive experience with all the treatment options in a nearly concurrent arrangement. This implies that consumers should have ample opportunity to experience all aspects of a given treatment to an extent deemed appropriate for them. This type of experience should occur for all treatment options during relatively similar points in time. This could prove difficult for some treatments, as it may require a lengthy period of time for the individual to be exposed to all the aspects of a treatment. It could also be difficult to ensure that different treatments are discernable to the consumer. Different treatments may be in place at the same time in different settings, but the consumer has to be capable of coming in contact with the treatment components in order to make a choice between the treatments. Two different treatments that are primarily reactionary in nature may appear very similar if there is no occurrence of the target behavior, but might appear very different if the target behavior occurs under one treatment condition but not under the other treatment condition. Hanley et al. (2005), in a study described earlier, has provided an elegant example of how choice could be used to provide support to the social validity of a treatment program. Winett et al. (1991) proposed a new model for determining social validity which involved a number of interactive, a priori steps to determine which behaviors were most important, and provided a method for promoting the acceptance of treatments designed to address these important behaviors. In addition, they described methods to promote the acceptance of behaviors selected for treatment as the most important behaviors to target for treatment. Essentially, as described by Baer and Schwartz (1991), the Winett et al. model is quite profound and challenges the
Chapter | 10 The Future of Social Validity
257
current state of applied behavior analysis. Their model creates a situation where behavior analysts are imposing their subjective decisions upon larger segments of society and using marketing techniques to persuade these larger segments of society to agree to or accept these subjective decisions regarding treatments and the importance of implementing these treatments in specific ways. Baer and Schwartz went on to suggest cautious investigation of the methods by Winett et al. They also recommended that this investigation of new strategies should not override or eliminate the old strategies, which may still prove beneficial to a more comprehensive understanding of social validity. Rasnake et al. (1993) outlined a number of limitations within the research on treatment acceptability. She pointed out that a large number of studies used college students as participants, which limits the generalizability of the results to actual consumers and those with extensive experience with various treatments. The commonly used analogue design of treatment acceptability research limits the amount of information that can be included about a treatment and within the case description. If this type of information could be included, the acceptability ratings might vary, as has been demonstrated in the studies evaluating the influence of context on acceptability. Rasnake described how these limitations prevent the findings from studies on treatment acceptability from being useful toward making treatment decisions. She suggested that treatment acceptability may only be relevant on a case-by-case basis, because of the tremendous amount of information that needs to be included when measuring treatment acceptability. She recommended that future research on treatment acceptability include information considered relevant as per the NIH Consensus Conference Statement (NIH, 1991). This statement outlines the importance of biological, environmental, and personological variables in making treatment decisions. Johnston (1993) urged that the future development of a behavioral technology be dependent upon formalized planning and management that exceeded the informal traditions associated with bridging experimental research to more applied practices. Johnston defined behavioral technology as “behavior change procedures the nature of whose influence has been established by experimental analysis in the terms of the
258
Chapter | 10 The Future of Social Validity
natural science of behavior and for which applied empirical evaluation has established reliable and general effects” (p. 324). The future development of techniques for establishing social validity for behavior treatments may be dependent upon more formalized planning and management of specific strategies for conducting research on social validity. Kazdin (1980) offered an excellent strategy for conducting research on treatment acceptability that formalized the methodology and influenced most all subsequent research on the acceptability of treatments. This formalization of procedures may be what is required to further expand research in this area and advance the methods used to be more representative of a behavioral technology. Expanding the distributive model of treatment acceptability described by Carter (2008b) may offer some direction toward formalized planning of social validity research. The expanded distributive model of social validity is presented in Figure 10.1. Expanding social validity measurement to incorporate numerous associated variables could eventually result in new protocols and procedures for evaluating social validity, and most any advances in social validity appear to be highly needed for the field of applied behavior analysis. Baer et al. (1987) pointed out that even using invalid techniques to evaluate social validity could possibly be better than not evaluating social validity. They indicated that simply allowing consumers opportunities to express their concerns with treatments might reveal otherwise unrecognized problems with treatment programs. This point of view is quite forgiving and recognizes the limitations involved with measuring social validity. In addition, this viewpoint allows for some trial-and-error efforts toward developing new technologies for measuring social validity. As noted by Baer, Wolf, and Risley, “the discipline is now moving into large-scale dissemination, valid social-validity assessments will soon become crucial to survival; yet this aspect of our measurement technique has seen very little inquiry and development” (1987, p. 323). Considering the importance of developing new technology for measuring social validity, it is hoped that the recommendations provided within this book can promote some discussion of, and potentially a framework for, developing social validity assessments that are more comprehensive, more accurate, and more relevant to the interests of consumers, consultants, and society. In Johnston’s (1993) model for developing and evaluating behavioral technology, he offered several questions to be answered when developing
• • • • • • •
Consumers Personal preferences Input on selection Choice Habilitative potential Awareness of coercion Normalization
Immediate vs. long-term benefits
Society • Expert evaluation • Cost analysis • Conformity to local/national norms
Goal Clarity Specific outcome criteria Complexity Potential for actualizing the goal Comprehensiveness
Consultant Training Personal agenda Professional agenda Habilitative potential Similarity to goals in comparable cases • Normalization • • • • •
Society • Maintain/Increase conformity • Normalization • Funding • Resources
Consultant Significance
Significance of goals
Consumer significance
Societal significance
Societal importance
• • • • • •
Society Laws Legislation Ethical boards Professional associations Review boards Movements/trends
Societal acceptability
Social Validity
Importance of effects
• • • • • • •
Treatment acceptability Consultant acceptability
Consultant importance Consumer
Consumer acceptability
Consumers • Understanding of treatment effects • Quality of life • Empowerment/selfdetermination
reliability
Rater Gender Knowledge of treatment Socioeconomic class Location of high school Parenting a child w/medical disorder • Marital distress • • • • •
• • • • •
Treatment Design elements Reinforcement Punishment Implementation time Intrusiveness
• Ease of implementation • Specificity to group • Clarity of items
259
FIGURE 10.1 Distributive model of social validity.
Client • Severity of problem
Acceptability instrument/Method
• Validity &
importance Consultant • Evaluation of data • Significant change in behavior • Normative comparison
Consultant Training History with treatment Ethical guidelines Familiarity with recent research Knowledge of client situation Presentation of treatment Assessments conducted
Chapter | 10 The Future of Social Validity
• • • • •
Instrument/Method • Validity & reliability • Ease of implementation • Specificity to group • Clarity of items
260
Chapter | 10 The Future of Social Validity
new procedures. Questions to consider when developing new procedures included determining the nature of the problem, determining the goals for the procedure, determining the behaviors of interest, determining the controlling variables, and determining the relevant principles and procedures. Determining the nature of the problem involved considering the perspectives of parties closely interested in the procedure, as well as the various environments that are or may be related to the effectiveness of the procedure. Determining the goals for the procedure included examining the potential short- and long-term effects of the procedures, as well as recognizing the concerns of all parties involved with the procedure. Determining the behaviors of interest involved gaining an understanding of interrelated social contingencies that may exist or develop with the use of a procedure. This required examination of behaviors of several different individuals who may be connected to the procedure. Determining the controlling variables involved examining the past, present, and future influences on behavior that may influence the procedure. Determining the relevant principles and procedures consisted of specifying the elements of the procedure that are necessary to make the procedure comprehensive. The questions to consider when analyzing and refining new procedures consist of determining the procedure’s effects, determining the components and their effects, determining how components produce effects, and determining how to improve the procedure. Determining the procedure’s effects requires a thorough examination of all planned and unplanned effects related to the procedure. This also needs to be extended to the component parts of the procedure, which need to be singled out and examined with regards to their effects. In addition, the components of the procedure need to be examined as to how individual components produce specific effects. Finally, improvements to the procedure need to be evaluated with consideration of effects over extended periods of time and under various circumstances. The questions described by Johnston (1993) to consider when evaluating and refining procedures include determining the effects under applied conditions, determining necessary refinements and their effects, determining if the conditions of application maintain the procedure, and determining if the procedure meets its original goals. The first
Chapter | 10 The Future of Social Validity
261
question, regarding effects under applied conditions, involves examining the procedure in environments where there is less control over specific variables. The second question relates to defining refinements to the procedures and measuring subsequent changes that may be related to these changes in the procedure. The third question requires monitoring the procedure over an extended period of time to determine the extent to which the procedure continues to function as intended. The final question, regarding evaluating and refining procedures, relates to assessing the degree that the original goals of the procedure meet the initial expectations of the procedure, using methods appropriate for measuring the goals (which may involve questioning others or direct observations). The questions developed by Johnston (1993) refer specifically to behavioral change procedures, but the intent of the model is to provide a thorough strategy to follow when developing technology in an organized manner. Johnston points out that answering the questions within the model may require the efforts of several different researchers over a lengthy period of time, and the answers may only be revealed in retrospect. Additionally, Johnston suggested that researchers should be aware of the current status and need for development of technology within the field of behavior analysis. One area that appears to be in need of continued development is social validity research. As noted by Wolery and Gast (1990), a considerable amount of work needs to be done toward defining social validity and determining how it should be measured. Wolery and Gast stated that in order for social validity to be truly useful, it should encompass all three of the dimensions described by Wolf (1978), and the amount or degree of validity for each of these dimensions should be reported. The distributive model of social validity presented within this book is an attempt to further the development of social validity research and offer some potential measures that can provide support to measurements of social validity. While this expanded model of social validity requires further refinement from several different researchers, it could potentially provide a format for organizing and managing social validity research. What is
262
Chapter | 10 The Future of Social Validity
still needed is additional empirical support for each of the components of this distributive model. The model is not entirely based on empirical data, but rather on verbal “analysis” and extrapolation. A warning appears necessary to avoid potential misuse of the proposed model: there is a need for clearly demonstrated reliability and generality of the effects of the procedures promoted within the distributive model of social validity presented. This warning was best expressed by Johnston in describing the requirements of a behavioral technology: The details of a procedure’s influence on behavior cannot be established by careful extrapolation, reasoned speculation, or personal conviction. This kind of interpretation is tempting because of the convenience and apparent fit of our guesses; however, if everyone engaging in such speculations leaves the subsequent experimental responsibilities to someone else, the latter may accumulate like garbage waiting to be taken out—and eventually may be worth as much (1993, p. 325).
This caution regarding this type of interpretation should be taken seriously and understood regarding the future use of this distributive model of social validity. While this is a critical concern, it seems necessary to make some attempt to offer a method for planning and managing social validity that can be further validated. Horner provided an insightful comment on the future of behavior analysis: If we as behavior analysts are to offer a technology of real behavior change, we will need (a) a research agenda that is guided not just by gaps in the literature but by a coherent conceptual model; (b) a research agenda that is relevant for behavior change at multiple levels of our society; and (c) research that is responsive to the needs of families, friends, and self-determined individuals as well as teachers, caregivers, and clinicians. This is an important time in the history of behavior analysis. We have the research and understanding to contribute to society in ways that are at the heart of societal interests (2002, p. 467).
The development of social validity research has progressed in the 30 years since Wolf (1978) defined the concept, but there also appears to be a large number of informal, commonsense approaches that may have resulted in considerable confusion and inconsistencies regarding the research in this area. It is hoped that the models and techniques presented within this text may offer a means toward resolving some of these inconsistencies and informalities and lead toward a better understanding of social validity.
References
Aaroe, L., & Nelson, J. (1998). Views about key curricular matters from the perspectives of students with disabilities. Current Issues in Education, 1, 1–8. Abramowitz, A. J., Eckstrand, D. O., O’Leary, S. G., & Dulcan, M. K. (1992). ADHD children’s responses to stimulant medication and two intensities of a behavioral intervention. Behavior Modification, 16, 193–203. Adkins, V. K. (1997). Social validity and naturalistic ethics: Wolf and Quine. Behavior and Social Issues, 7, 153–157. Alberto, P. A., & Troutman, A. C. (1999). Applied behavior analysis for teachers (5th ed.). Upper Saddle River, NJ: Prentice Hall. Albin, R. W., Lucyshyn, J. M., Horner, R. H., & Flannery, K. B. (1996). Contextual fit for behavioral support plans: A model for “Goodness of Fit”. In L. K. Koegel, R. L. Koegel, & G. Dunlap (Eds.), Positive behavioral support: Including people with difficult behavior in the community (pp. 81–98). Baltimore: Brookes. Allyon, T., & Haughton, E. (1962). Control of the behavior of schizophrenic patients by food. Journal of the Experimental Analysis of Behavior, 5, 343–352. American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. APA: Author, Available at http://www.apa.org/ethics. Anan, R. M., Warner, L. J., McGillivary, J. E., Chong, I. M., & Hines, S. J. (2008). Group intensive family training (GIFT) for preschoolers with autism spectrum disorders. Behavioral Interventions, 23, 165–180. Anderson, C. M., & Freeman, K. A. (2000). Positive behavior support: Expanding the application of applied behavior analysis. The Behavior Analyst, 23, 85–94. Arra, C. T., & Bahr, M. W. (2005). Teachers’ and students’ preferences for mathematics interventions: Implications for teacher acceptability in consultation. Journal of Educational and Psychological Consultation, 16, 157–174. Armstrong, K. J., Ehrhardt, K. E., Cool, R. T., & Poling, A. (1997). Social validity and treatment integrity data: Reporting in articles published in the Journal of Developmental and Physical Disabilities, 1991–1995. Journal of Developmental and Physical Disabilities, 9, 359–367. Association for Positive Behavior Supports (2007). PBS standards of practice: Individual level. Available for download from http://apbs.org/whatsnew. html#standards_of_practice. 263
264
References
Baer, D. M. (1987). Weak contingencies, strong contingencies, and many behaviors to change. Journal of Applied Behavior Analysis, 20, 335–337. Baer, D. M., & Schwartz, I. S. (1991). If reliance on epidemiology were to become epidemic, we would need to assess its social validity. Journal of Applied Behavior Analysis, 24, 231–234. Baer, D., Wolf, M., & Risley, T. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97. Baer, D. M., Wolf, M. M., & Risley, T. R. (1987). Some still current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 20, 313–327. Bailey, D. B. (1987). Collaborative goal-setting with families: Resolving differences in values and priorities for services. Topics in Early Childhood Special Education, 7, 59–71. Bailey, D. B., Simeonsson, R. J., Winton, P. J., Huntington, G. S., Comfort, M., Isbell, P., O’Donnell, K. J., & Helm, J. M. (1990). Family-focused intervention: A functional model for planning, implementing, and evaluating individualized family services in early intervention. Journal of the Division for Early Childhood, 10, 157–171. Barnard, J. D., Christophersen, E. R., & Wolf, M. M. (1974). Supervising paraprofessional tutors in a remedial reading program. Journal of Applied Behavior Analysis, 7, 481. Benson, H. A., & Turnbull, A. P. (1986). Approaching families from an individualized perspective. In R. H. Horner, L. H. Meyer, & H. D. Fredericks (Eds.), Education of learners with severe handicaps: Exemplary service strategies (pp. 127–157). Baltimore: Brookes. Bernstein, G. S. (1989). In response: Social validity and the report of the ABA task force on the right to effective treatment. The Behavior Analyst, 12, 97. Betts, G. R., & Remer, R. (1993). The impact of paradoxical interventions on perceptions of the therapist and ratings of treatment acceptability. Professional Psychology: Research and Practice, 24, 164–170. Blampied, N. M., & Kahan, E. (1992). Acceptability of alternative punishments. Behavior Modification, 16, 400–413. Bornstein, P. H., Fox, S. G., Sturm, C. A., Ballweg, B. J., Kirby, K. L., Wilson, G. L., Weisser, C. E., Andre, J. C., & McLellarn, R. W. (1983). Treatment acceptability of alternative marital therapies: A comparative analysis. Journal of Marital and Family Therapy, 9, 205–208. Broughton, S. F., & Hester, J. R. (1993). Effects of administrative and community support on teacher acceptance of classroom interventions. Journal of Educational and Psychological Consultation, 4, 169–177. Brown, F., Michaels, C. A., Oliva, C. M., & Woolf, S. B. (2008). Personal paradigm shifts among ABA and PBS experts. Journal of Positive Behavior Interventions, 10, 212–227. Brown, L., Wilcox, B., Sontag, E., Vincent, B., Dodd, N., & Gruenewald, L. (2004). Toward the realization of the least restrictive educational environments
References
265
for severely handicapped students. Research & Practice for Persons with Severe Disabilities, 29, 2–8. (Reprinted from The American Association for the Education of the Severely/Profoundly Handicapped Review, 2(4), December, 1977.) Burgio, L. D., Cotter, E. M., Stevens, A. B., Hardin, J. M., Sinnott, J., & Hohman, M. J. (1995). Elders’ acceptability ratings of behavioral treatments and pharmacotherapy for the management of geriatric behavioral disturbances. The Gerontologist, 35, 630–636. Carey, T. A., & Bourbon, W. T. (2004). Countercontrol: A new look at some old problems. Intervention in School and Clinic, 40, 3–9. Carey, T. A., & Bourbon, W. T. (2005). Countercontrol: What do the children say? School Psychology International, 26, 595–615. Carr, E. G. (1980). Generalization of treatment effects following educational intervention with autistic children and youth. In B. Wilcox & A. Thompson (Eds.), Critical issues in educating autistic children and youth (pp. 118–134). Washington, DC: Department of Education, Office of Special Education. Carr, E. G. (1996). The transfiguration of behavior analysis: Strategies for survival. Journal of Behavioral Education, 6, 263–270. Carr, E. G., Dunlap, G., Horner, R. H., Koegel, R. L., Turnbull, A. P., Sailor, W., Anderson, J. L., Albin, R. W., Koegel, L. K., & Fox, L. (2002). Positive behavior support: Evolution of an applied science. Journal of Positive Behavior Interventions, 4, 4–16. Carr, E. G., & Durand, V. M. (1985). Reducing behavior problems through functional communication training. Journal of Applied Behavior Analysis, 18, 111–126. Carr, J. E., Austin, J. L., Britton, L. N., Kellum, K. K., & Bailey, J. S. (1999). Assessment of social validity trends in applied behavior analysis. Behavioral Interventions, 14, 223–231. Carr, J. E., Bailey, J. S., Ecott, C. L., Lucker, K. D., & Weil, T. M. (1998). On the effects of noncontingent delivery of differing magnitudes of reinforcement. Journal of Applied Behavior Analysis, 31, 313–321. Carter, S. L. (2005). College students’ acceptance of potential treatments for ADHD. Psychological Reports, 97, 258–264. Carter, S. L. (2007). Review of recent treatment acceptability research. Education and Training in Developmental Disabilities, 42, 301–316. Carter, S. L. (2008a). Further conceptualization of treatment acceptability. Education and Training in Developmental Disabilities, 43, 135–143. Carter, S. L. (2008b). A distributive model of treatment acceptability. Education and Training in Developmental Disabilities, 43, 411–420. Carter, S. L., Mayton, M. R., & Wheeler, J. J. (2009). The development of an instrument to evaluate treatment intrusiveness for individuals with severe and challenging behavior. Research in Developmental Disabilities, 30, 58–69.
266
References
Cavell, T., Frentz, C., & Kelley, M. (1986a). Acceptability of paradoxical interventions: Some nonparadoxical findings. Professional Psychology: Research and Practice, 17, 519–523. Cavell, T. A., Frentz, C. E., & Kelley, M. L. (1986b). Consumer acceptability of the single case withdrawal design: Penalty for early withdrawal? Behavior Therapy, 17, 82–87. Clark, L., & Elliott, S. N. (1988). The influence of treatment strength information on knowledgeable teachers’ pretreatment evaluations of social skills training methods. Professional School Psychology, 3, 241–251. Clarke, S., Worcester, J., Dunlap, G., Murray, M., & Bradley-Klug, K. (2002). Using multiple measures to evaluate positive behavior support: A case example. Journal of Positive Behavior Interventions, 4, 131–145. Colton, D. L., & Sheridan, S. M. (1998). Conjoint behavioral consultation and social skills training: Enhancing the play behaviors of boys with attention deficit hyperactivity disorder. Journal of Educational and Psychological Consultation, 9, 3–28. Conroy, M. A., & Stichter, J. P. (2003). The application of antecedents in the functional assessment process. Journal of Special Education, 37, 15–25. Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Cowan, R. J., & Sheridan, S. M. (2003). Investigating the acceptability of behavioral interventions in applied conjoint behavioral consultation: Moving from analog conditions to naturalistic settings. School Psychology Quarterly, 18, 1–21. Critchfield, T. S. (2002). Evaluating the function of applied behavior analysis: A bibliometric analysis. Journal of Applied Behavior Analysis, 35, 423–426. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp.443–507). Washington, DC: American Council on Education. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. Cross Calvert, S., & McMahon, R. (1987). The treatment acceptability of a behavioral parent training program and its components. Behavior Therapy, 2, 165–179. Cunningham, C. E., & Linscheid, T. R. (1976). Elimination of chronic infant ruminating by electric shock. Behavior Therapy, 7, 231–234. Curtis, D. F., Pisecco, S., Hamilton, R. J., & Moore, D. W. (2006). Teacher perceptions of classroom interventions for children with ADHD: A crosscultural comparison of teachers in the United States and New Zealand. School Psychology Quarterly, 21, 171–196. Dardig, J. C., & Heward, W. L. (1981). A systematic procedure for prioritizing IEP goals. The Directive Teacher, 3, 6–8. Dietz, S. M. (1982). Defining applied behavior analysis: An historical analogy. The Behavior Analyst, 5, 53–64.
References
267
Duker, P. C., & Seys, D. M. (2000). A quasi-experimental study on the effect of electrical aversion treatment on imposed mechanical restraint for severe selfinjurious behavior. Research in Developmental Disabilities, 21, 235–242. Dunlap, G. (2006). The applied behavior analytic heritage of PBS: A dynamic model of action-oriented research. Journal of Positive Behavior Interventions, 8, 58–60. Dunlap, G., & Fox, L. (1996). Early intervention for serious problem behaviors. In L. K. Koegel, R. L. Koegel, & G. Dunlap (Eds.), Positive behavioral support: Including people with difficult behavior in the community (pp. 31–50). Baltimore: Brookes. Dunlap, G., Clarke, S., & Streiner, M. (1999). Intervention research in behavioral and developmental disabilities: 1980 to 1997. Journal of Positive Behavior Interventions, 1, 170–180. Eckert, T. L., & Hintze, J. M. (2000). Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly, 15, 123–148. Elliott, S. N. (1988). Acceptability of behavioral treatments: Review of variables that influence treatment selection. Professional Psychology: Research and Practice, 19, 68–80. Elliott, A. J., & Fuqua, R. W. (2002). Acceptability of treatments for trichotillomania: Effects of age and severity. Behavior Modification, 26, 378–399. Elliott, S. N., Busse, R. T., & Gresham, F. M. (1993). Behavior rating scales: Issues of use and development. School Psychology Review, 22, 313–321. Elliott, S. N., Turco, T. L., & Gresham, F. M. (1987). Consumers’ and clients’ pretreatment acceptability ratings of classroom group contingencies. Journal of School Psychology, 25, 145–153. Elliott, S. N., Witt, J. C., Galvin, G. A., & Peterson, R. (1984). Acceptability of positive and reductive behavioral interventions: Factors that influence teachers’ decisions. Journal of School Psychology, 22, 353–360. Epstein, M. H., Matson, J. L., Repp, A., & Helsel, W. J. (1986). Acceptability of treatment alternatives as a function of teacher status and student level. School Psychology Review, 15, 84–90. Fairbanks, L. D., & Stinnett, T. A. (1997). Effects of professional group membership, intervention type and diagnostic label on treatment acceptability. Psychology in the Schools, 34, 329–335. Fawcett, S. B. (1991). Social validity: A note on methodology. Journal of Applied Behavior Analysis, 24, 235–239. Finn, C. A., & Sladeczek, I. S. (2001). Assessing the social validity of behavioral interventions: A review of treatment acceptability measures. School Psychology Quarterly, 16, 176–206. Finney, J. W. (1991). On further development of the concept of social validity. Journal of Applied Behavior Analysis, 24, 245–249. Fox, L. (1989). Stimulus generalization of skills and persons with profound handicaps. Education and Training in Mental Retardation, 24, 219–229.
268
References
Fox, P., & Emerson, E. (2001). Socially valid outcomes of intervention for people with MR and challenging behavior. Journal of Positive Behavior Interventions, 3, 183–189. Francisco, V. T., & Butterfoss, F. D. (2007). Social validation of goals, procedures, and effects in public health. Health Promotion Practice, 8, 128–133. Freer, P., & Watson, T. S. (1999). A comparison of parent and teacher acceptability ratings of behavioral and conjoint behavioral consultation. School Psychology Review, 28, 672–684. Frentz, C., & Kelley, M. L. (1986). Parents’ acceptance of reductive treatment methods: The influence of problem severity and perception of child behavior. Behavior Therapy, 17, 75–81. Gage, J. D., & Wilson, L. J. (2000). Acceptability of attention-deficit/hyperactivity disorder interventions: A comparison of parents. Journal of Attention Disorders, 4, 174–182. Gallagher, J. J., Beckman, P. J., & Cross, A. H. (1983). Families of handicapped children: Sources of stress and its amelioration. Exceptional Children, 50, 10–19. Gillat, A., & Sulzer-Azaroff, B. (1994). Promoting principals’ managerial involvement in instructional improvement. Journal of Applied Behavior Analysis, 27, 115–129. Goldberg, R., & Shapiro, E. S. (1995). In-vivo rating of treatment acceptability by children: Effects of probability instruction on student’s spelling performance under group contingency conditions. Journal of Behavioral Education, 5(4), 415–432. Graham, D. S. (1998). Consultant effectiveness and treatment acceptability: An examination of consultee requests and consultant responses. School Psychology Quarterly, 13, 155–168. Green, C. W., Gardner, S. M., & Reid, D. H. (1997). Increasing indices of happiness among people with profound multiple disabilities: A program replication and component analysis. Journal of Applied Behavior Analysis, 30, 217–228. Green, C. W., & Reid, D. H. (1996). Defining, validating, and increasing indices of happiness among people with profound multiple disabilities. Journal of Applied Behavior Analysis, 29, 67–78. Green, C. W., & Reid, D. H. (1999). Reducing indices of unhappiness among individuals with profound multiple disabilities during therapeutic exercise routines. Journal of Applied Behavior Analysis, 32, 137–147. Gresham, F. M., & Gresham, G. N. (1982). Interdependent, dependent, and independent group contingencies for controlling disruptive behavior. The Journal of Special Education, 16, 101–110. Gresham, F. M., & Lopez, M. F. (1996). Social validation: A unifying construct for school-based consultation research and practice. School Psychology Quarterly, 11, 204–227. Gross, A., Miltenberger, R., Knudson, P., Bosch, A., & Brower Breitwieser, C. (2007). Preliminary evaluation of a parent training program to prevent gun play. Journal of Applied Behavior Analysis, 40, 691–695.
References
269
Hall, S. S. (2005). Comparing descriptive, experimental and informant-based assessments of problem behaviors. Research in Developmental Disabilities, 26, 514–526. Hanley, G. P., Iwata, B. A., & McCord, B. E. (2003). Functional analysis of problem behavior: A review. Journal of Applied Behavior Analysis, 36, 147–185. Hanley, G. P., Piazza, C. C., Fisher, W. W., Contrucci, S. A., & Maglieri, K. A. (1997). Evaluation of client preference for function-based treatment packages. Journal of Applied Behavior Analysis, 30, 459–473. Hanley, G. P., Piazza, C. C., Fisher, W. W., & Maglieri, K. A. (2005). On the effectiveness of and preference for punishment and extinction components of function-based interventions. Journal of Applied Behavior Analysis, 38, 51–65. Harris, A. J., & Jacobson, M. D. (1982). Basic reading vocabularies. New York: Macmillan. Harris, K. R., Preller, D. M., & Graham, S. (1990). Acceptability of cognitivebehavioral and behavioral interventions among teachers. Cognitive Therapy and research, 14, 573–587. Hastings, R. P., & Noone, S. J. (2005). Self-injurious behavior and functional analysis: Ethics and evidence. Education and Training in Developmental Disabilities, 40, 335–342. Hawkins, R. P. (1979). The functions of assessment: Implications for selection and development of devices for assessing repertoires in clinical, educational, and other settings. Journal of Applied Behavior Analysis, 12, 501–516. Hawkins, R. P. (1991). Is social validity what we are interested in? Argument for a functional approach. Journal of Applied Behavior Analysis, 24, 205–213. Hawkins, R. P. (1991). Is social validity what we are interested in? Argument for a functional approach. Journal of Applied Behavior Analysis, 24, 205–213. Hayes, L. J., & Tarbox, J. (2007). Ethics and values in behavioral perspective. In J. W. Jacobson, J. A. Mulick, & J. Rojahn (Eds.), Handbook of intellectual and developmental disabilities (pp. 691–717). New York: Springer. Hayes, S. C., Rosenfarb, I., Wulfert, E., Munt, E. D., Korn, Z., & Zettle, R. D. (1985). Self-reinforcement effects: An artifact of social standard setting?. Journal of Applied Behavior Analysis, 18, 201–214. Heffer, R., & Kelley, M. (1987). Mothers’ acceptance of behavioral interventions for children: The influence of parent race and income. Behavior Therapy, 18, 153–164. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of reinforcement. Journal of the Experimental Analysis of Behavior, 4, 267–272. Herrnstein, R. J. (1970). On the law of effect. Journal of the Experimental Analysis of Behavior, 13, 243–266. Heward, W. L. (2005). Reasons applied behavior analysis is good for education and why those reasons have been insufficient. In W. L. Heward,
270
References
T. E. Heron, N. A. Neef, S. M. Peterson, D. M. Sainato, G. Cartledge, R. Gardner, III, L. D. Peterson, S. B. Hersh, & J. C. Dardig (Eds.), Focus on behavior analysis in education: Achievements, challenges, and opportunities (pp. 316–348). Upper Saddle River, NJ: Merrill/Prentice Hall. Hieneman, M., Presley, J., Gayler, W., Nolan, M., DeTuro, L., & Dunlap, G. (1999). Facilitator’s guide: Positive behavioral support. Tallahassee, FL: State of Florida, Department of State. Hobbs, S. A., Walle, D. L., & Caldwell, H. S. (1984). Maternal evaluation of social reinforcement and time-out: Effects of brief parent training. Journal of Consulting and Clinical Psychology, 52, 135–136. Hobbs, S. A., Walle, D. L., & Hammersly, G. A. (1990). The relationship between child behavior and acceptability of contingency management procedures. Child and Family Behavior Therapy, 12, 95–102. Hopkins, B. L. (1987). Comments on the future of applied behavior analysis. Journal of Applied Behavior Analysis, 20, 339–346. Horner, R. H. (2002). On the status of knowledge for using punishment: A commentary. Journal of Applied Behavior Analysis, 35, 465–467. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practices in special education. Exceptional Children, 71, 165–179. Horner, R. H., O’Neill, R. E., & Flannery, K. B. (1993). Effective behavior support plans. In M. E. Snell (Ed.), Instruction of students with severe disabilities (pp. 184–214). New York: Macmillan. Horner, R. H., Williams, J. A., & Knobbe, C. (1985). The effect of “opportunity to perform” on the maintenance skills learned by high school students with severe handicaps. Journal of the Association for Persons with Severe Handicaps, 10, 172–175. Horner, R. H., Dunlap, G., Koegel, R. L., Carr, E. G., Sailor, W., Anderson, J., Albin, R. W., & O’Neill, R. E. (1990). Toward a technology of “nonaversive” behavioral support. Journal of the Association for Persons with Severe Handicaps, 15, 125–132. Individuals with Disabilities Education Act Amendments of 1997, 20 U.S.C. § 1401 (26). Individuals with Disabilities Education Act of 2004, P.L., 108–446. Irvin, L. K., & Lundervold, D. A. (1988). Social validation of decelerative (punishment) procedures by special educators of severely handicapped students. Research in Developmental Disabilities, 9, 331–350. Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1994). Toward a functional analysis of self-injury (Reprinted from Analysis and Intervention in Developmental Disabilities, 2, pp. 3–20, 1982). Journal of Applied Behavior Analysis, 27, 197–209. Iwata, B. A., Wallace, M. D., Kahng, S. W., Lindberg, J. S., Roscoe, E. M., Conners, J., et al. (2000). Skill acquisition in the implementation of functional analysis methodology. Journal of Applied Behavior Analysis, 33, 181–194.
References
271
Jacobson, J. W., Mulick, J. A., & Green, G. (1998). Cost-benefit estimates for early intensive behavioral intervention for young children with autism— General model and single state case. Behavioral Interventions, 13, 201–226. Johnston, J. M. (1993). A model for developing and evaluating behavioral technology. In R. Van Houten & S. Axelrod (Eds.), Behavior analysis and treatment (pp. 323–343). New York: Plenum. Jones, M. L., Eyberg, S. M., Adams, C. D., & Boggs, S. R. (1998). Treatment acceptability of behavioral interventions for children: An assessment by mothers of children with disruptive behavior disorders. Child & Family Behavior Therapy, 20(1), 15–26. Kalfus, G. R., & Burk, B. K. (1989). The influence of case history and treatment mediator on ratings of acceptability of child treatment. Child & Family Behavior Therapy, 11(2), 45–55. Kazdin, A. E. (1980a). Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis, 132, 259–273. Kazdin, A. E. (1980b). Acceptability of time-out from reinforcement procedures for disruptive child behavior. Behavior Therapy, 11, 329–344. Kazdin, A. E. (1981). Acceptability of child treatment techniques: The influence of treatment efficacy and adverse side effects. Behavior Therapy, 12, 493–506. Kazdin, A. E. (1994). Behavior modification in applied settings. Pacific Grove, CA: Brookes/Cole. Kazdin, A. E., & Cole, P. M. (1981). Attitudes and labeling biases toward behavior modification: The effects of labels, content, and jargon. Behavior Therapy, 12, 56–68. Kazdin, A. E., French, N. H., & Sherick, R. B. (1981). Acceptability of alternative treatments for children: Evaluations by inpatient children, parents, and staff. Journal of Consulting and Clinical Psychology, 49, 900–907. Kazdin, A. E., & Matson, J. L. (1981). Social validation in mental retardation. Applied Research in Mental Retardation, 2, 39–53. Kelley, M., Heffer, R., Gresham, F., & Elliott, S. (1989). Development of a modified treatment evaluation inventory. Journal of Psychopathology and Behavioral Assessment, 11, 235–247. Keith, K. D., Schalock, R. L., & Hoffman, K. (1986). Quality of life: Measurement and programmatic implications. Lincoln, NE: Region V Mental Retardation Services. Kennedy, C. H. (1992). Trends in the measurement of social validity. Behavior Analyst, 15, 147–156. Kennedy, C. H. (2002a). The maintenance of behavior change as an indicator of social validity. Behavior Modification, 26, 594–606. Kennedy, C. H. (2002b). Toward a socially valid understanding of problem behavior. Education and Treatment of Children, 25, 142–153. Kennedy, C. H. (2005). Single-case designs for educational research. Boston: Allyn and Bacon.
272
References
Kern, L., Dunlap, G., Clarke, S., & Childs, K. (1994). Student-assisted functional assessment interview. Diagnostique, 19, 29–39. Kern, L., Dunlap, G., Clarke, S., & Childs, K. E. (1995). Student-assisted functional assessment interview. Diagnostique, 19, 29–39. Kern, L., & Manz, P. (2004). A look at current validity issues of school-wide behavior support. Behavioral Disorders, 30, 47–59. Kincaid, D. (1996). Person-centered planning. In L. K. Koegel, R. L. Koegel, & G. Dunlap (Eds.), Positive behavioral support: Including people with difficult behavior in the community (pp. 439–465). Baltimore: Brookes. Kiresuk, T. J., & Sherman, R. E. (1968). Goal attainment scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4, 443–453. Kunkel, J. H. (1987). The future of JABA: A comment. Journal of Applied Behavior Analysis, 20, 329–333. Kutsick, K. A., Gutkin, T. B., & Witt, J. C. (1991). The impact of treatment development process, intervention type, and problem severity on treatment acceptability as judged by classroom teachers. Psychology in the Schools, 28, 325–331. Lane, K. L. (1997). Students at-risk for antisocial behavior: The utility of academic and social skills interventions. Doctoral Dissertation. University of California, Riverside. Lennox, D. B., & Miltenberger, R. G. (1990). On the conceptualization of treatment acceptability. Education and Training in Mental Retardation, 25, 211–224. Lerman, D. C., & Vorndran, C. M. (2002). On the status of knowledge for using punishment: Implications for treating behavior disorders. Journal of Applied Behavior Analysis, 35, 431–464. Leslie, J. (1997). Ethical implications of behavior modification: Historical and current issues. Psychological Record, 47, 637–648. Levin, H. M., & McEwan, P. J. (2001). Cost-effectiveness analysis: Methods and applications (2nd ed.). Thousand Oaks, CA: Sage. Lindeman, D. P., Miltenberger, R. G., & Lennox, D. B. (1992). Acceptability of behavioral interventions: Perceptions of superintendents of public residential facilities. Behavioral Residential Treatment, 7, 35–44. Linscheid, T. R. (1993). The development and evaluation of the self-injurious behavior inhibiting system: A personal perspective. In R. Van Houten & S. Axelrod (Eds.), Behavior analysis and treatment (pp. 345–365). New York: Plenum. Linscheid, T. R., Iwata, B. A., Ricketts, R. W., Williams, D. E., & Griffin, J. C. (1990). Clinical evaluation of the self-injurious behavior inhibiting system (SIBIS). Journal of Applied Behavior Analysis, 23, 53–78. Litow, L., & Pumroy, D. K. (1975). A brief review of classroom group-oriented contingencies. Journal of Applied Behavior Analysis, 8, 341–347.
References
273
Lucyshyn, J. M., & Albin, R. W. (1993). Comprehensive support to families of children with disabilities and behavior problems: Keeping it “friendly”. In G. H. S. Singer & L. E. Powers (Eds.), Families, disability, and empowerment: Active coping skills and strategies for family interventions (pp. 365– 407). Baltimore: Brookes. Lucyshyn, J. M., Olson, D., & Horner, R. H. (1995). Building an ecology of support: A case study of one young woman with severe problem behaviors living in the community. Journal of the Association for Persons with Severe Handicaps, 20, 16–30. Macmann, G. M., & Barnett, D. W. (1999). Diagnostic decision making in school psychology: Understanding and coping with uncertainty. In C. R. Reynolds & T. B. Gutkin (Eds.), The handbook of school psychology (3rd ed., pp.519–548). New York: Wiley. Macurik, K. M., O’Kane, N. P., Malanga, P., & Reid, D. H. (2008). Video training of support staff in intervention plans for challenging behavior: Comparison with live training. Behavioral Interventions, 23, 143–163. Maher, C. A. (1983). Goal attainment scaling: A method for evaluating special education services. Exceptional Children, 49, 529–536. Martens, B. K., Kelly, S. Q., & Diskin, M. T. (1996). The effects of two sequential-request strategies on teachers’ acceptability and use of a classroom intervention. Journal of Educational and Psychological Consultation, 7, 211–221. Martens, B. K., Witt, J. C., Elliott, S. N., & Darveaux, D. X. (1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research and Practice, 16, 191–198. Matson, J. L., Manikam, R., & Ladatto, J. (1990). A long-term follow-up of a recreate the scene, DRO, overcorrection and lemon juice therapy program for severe aggressive biting. Scandinavian Journal of Behavior Therapy, 19, 33–38. Mccausland, D., Grey, I. M., Wester, G., & McClean, B. (2004). Effects of functional versus non-functional explanations for challenging behaviours on treatment acceptability. Journal of Intellectual Disabilities, 8, 351–369. McCubbin, H., Sussman, M., & Patterson, J. (Eds.), (1983). Social stress and the family. New York: Hawthorn. McGee, J. J., Menolascino, F. J., Hobbs, D. C., & Menousek, P. E. (1987). Gentle teaching: A non-aversive approach to helping persons with mental retardation. New York: Human Sciences. Meller, P. J., Martens, B. K., & Hurwitz, S. (1990). Variables influencing perceptions of liability: A case for treatment acceptability assessment. School Psychology Quarterly, 5, 237–255. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp.13–103). New York: Macmillan. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
274
References
Metzler, C. W., Biglan, A., Rusby, J. C., & Sprague, J. R. (2001). Evaluation of a comprehensive behavior management program to improve school-wide positive behavior support. Education and Treatment of Children, 24, 448–479. Meyer, L. H., Reichle, J., McQuarter, R. J., Cole, D., Vandercook, T., Evans, I. M., Neel, R., & Kishi, G. (1985). The assessment of social competence (ASC): A scale of social competence functions. Minneapolis: University of Minnesota Consortium Institute. Michaels, C. A., Brown, F., & Mirabella, N. (2005). Personal paradigm shifts in PBS experts: Perceptions of treatment acceptability in decelerative consequence-based behavioral procedures. Journal of Positive Behavior Interventions, 7, 93–108. Milgram, S. (1974). Obedience to authority: An experimental view. New York: Harper & Row. Miller, D. L., & Kelley, M. L. (1992). Treatment acceptability: The effects of parent gender, marital adjustment, and child behavior. Child & Family Behavior Therapy, 14(1), 11–23. Miller, D. N., DuPaul, G. J., & Lutz, J. G. (2002). School-based psychosocial interventions for childhood depression: Acceptability of treatments among school psychologists. School Psychology Quarterly, 17, 78–99. Miller, D. L., Manne, S., & Palevsky, S. (1998). Brief report: Acceptance of behavioral interventions for children with cancer: Perceptions of parents, nurses, and community controls. Journal of Pediatric Psychology, 23, 267–271. Miltenberger, R. G. (1990). Assessment of treatment acceptability: A review of the literature. Topics in Early Childhood Special Education, 10(3), 24–38. Miltenberger, R., Lennox, D., & Erfanian, N. (1989). Acceptability of alternative treatments for persons with mental retardation: Ratings from institutional and community-based staff. American Journal on Mental Retardation, 93, 388–395. Miltenberger, R. G., & Lumley, V. A. (1997). Evaluating the influence of problem function on treatment acceptability. Behavioral Interventions, 12, 105–111. Mirenda, P. (1993). AAC: Bonding the uncertain mosaic. AAC Augmentative and Alternative Communication, 9, 3–9. Moore, J. W., Edwards, R. P., Sterling-Turner, H. E., Riley, J., DuBard, M., & McGeorge, A. (2002). Teacher acquisition of functional analysis methodology. Journal of Applied Behavior Analysis, 35, 73–77. Mueller, M. M., Piazza, C. C., Moore, J. W., Kelley, M. E., Bethke, S. A., Pruett, A. E., Oberdorff, A. J., & Layer, S. A. (2003). Training parents to implement pediatric feeding protocols. Journal of Applied Behavior Analysis, 36, 545–562. Murray, L. K., & Kollins, S. H. (2000). Effects of methylphenidate on sensitivity to reinforcement in children diagnosed with attention deficit hyperactivity disorder: An application of the matching law. Journal of Applied Behavior Analysis, 33, 573–591.
References
275
National Institutes of Health. (1991). Treatment of destructive behaviors of persons with developmental disabilities (NIH Publication No. 91-2410). Washington, DC: U.S. Government Printing Office. Neef, N. A. (1995). Research on training trainers in program implementation: An introduction and future directions. Journal of Applied Behavior Analysis, 28, 297–299. Nelson, J. R. (1996). Designing schools to meet the needs of students who exhibit disruptive behavior. Journal of Emotional and Behavioral Disorders, 4, 147–161. Nelson, J. R., Martella, R. M., & Marchand-Martella, N. (2002). Maximizing student learning: The effects of a comprehensive school-based program for preventing problem behaviors. Journal of Emotional and Behavioral Disorders, 10, 136–148. Nikopoulos, C. K., & Keenan, M. (2004). Effects of video modeling on social initiations by children with autism. Journal of Applied Behavior Analysis, 37, 93–96. No Child Left Behind Act of 2001, 20 U.S.C. 70 § 6301 et seq. Olive, M. L., & Liu, Y. (2005). Social validity of parent and teacher implemented assessment-based interventions for challenging behaviour. Educational Psychology, 25, 305–312. Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). Measurement of meaning. Urbana, IL: University of Illinois Press. Petursdottir, A., McComas, J., McMaster, K., & Horner, K. (2007). The effects of scripted peer tutoring and programming common stimuli on social interactions of a student with autism spectrum disorder. Journal of Applied Behavior Analysis, 40, 353–357. Phillips, K. J., & Mudford, O. C. (2008). Functional analysis skills training for residential caregivers. Behavioral Interventions, 23, 1–12. Piazza, C. C., Roane, H. S., Keeney, K. M., Boney, B. R., & Abt, K. A. (2002). Varying response effort in the treatment of pica maintained by automatic reinforcement. Journal of Applied Behavior Analysis, 35, 233–247. Pisecco, S., Huzinec, C., & Curtis, D. (2001). The effect of child characteristics on teachers’ acceptability of classroom-based behavioral strategies and psychostimulant medication for the treatment of ADHD. Journal of Clinical Child Psychology, 30, 413–421. Poling, A., & Ehrhardt, K. (2000). Applied behavior analysis, social validation, and the psychopharmacology of mental retardation. Mental retardation and Developmental Disabilities Research Reviews, 5, 342–347. Posavac, H. D., Sheridan, S. M., & Posavac, S. S. (1999). A cueing procedure to control impulsivity in children with attention deficit disorder. Behavior Modification, 23, 234–253. Power, T. J., Hess, L. E., & Bennett, D. S. (1995). The acceptability of interventions for Attention-Deficit Hyperactivity Disorder among elementary and middle school teachers. Developmental and Behavioral Pediatrics, 16, 238–243.
276
References
Progar, P. R., North, S. T., Bruce, S. S., DiNovi, B. J., Nau, P. A., Eberman, E. M., et al. (2001). Putative behavioral history effects and aggression maintained by escape from therapists. Journal of Applied Behavior Analysis, 34, 69–72. Rasnake, K. L., Martin, J., Tarnowski, K. J., & Mulick, J. A. (1993). Acceptability of behavioral treatments: Influence of knowledge of behavioral principles. Mental Retardation, 31, 247–251. Rasnake, L. K. (1993). Treatment acceptability research: Relevance to treatment selection decisions. Child and Adolescent Mental Health Care, 3, 31–47. Reeve, S. A., Reeve, K. F., Townsend, D. B., & Poulson, C. L. (2007). Establishing a generalized repertoire of helping behavior in children with autism. Journal of Applied Behavior Analysis, 40, 123–136. Reimers, T. M., & Wacker, D. P. (1988). Parents’ rating of the acceptability of behavioral treatment recommendation made in an outpatient clinic: A preliminary analysis of the influence of treatment effectiveness. Behavioral Disorders, 14, 7–15. Reimers, T. M., Wacker, D. P., & Cooper, L. J. (1991). Evaluation of the acceptability of treatments for their children’s behavioral difficulties: Ratings by parents receiving services in an outpatient clinic. Child & Family Behavior Therapy, 13(2), 53–71. Reimers, T. M., Wacker, D. P., Cooper, L. J., & De Raad, A. O. (1992). Acceptability of behavioral treatments for children. School Psychology Review, 21, 628–643. Reimers, T. M., Wacker, D., & Koepple, G. (1987). Acceptability of behavioral interventions: A review of the literature. School Psychology Review, 16, 212–227. Reuf, M. B., Turnbull, A. P., Turnbull, H. R., & Poston, D. (1999). Perspectives of five stakeholder groups: Challenging behavior of individuals with mental retardation and autism. Journal of Positive Behavior Interventions, 1, 43–58. Rice, J. K. (1997). Cost analysis in education: Paradox and possibility. Educational Evaluation and Policy Analysis, 19, 309–317. Rutzen, S. R. (1973). The social importance of orthodontic rehabilitation: Report of a five-year follow-up study. Journal of Health and Social Behavior, 14, 233–240. Schalock, R. L. (2001). Outcome-based evaluation (2nd ed.). New York: Kluwer Academic/Plenum Publishers. Schalock, R. L. (2004). The emerging disability paradigm and its implications for the policy and practice. Journal of Disability Policy Studies, 14, 204–215. Schalock, R. L., & Kiernan, W. E. (1990). Habilitation planning for adults with disabilities. New York: Springer Verlag. Schalock, R. L., Keith, K. D., Hoffman, K., & Karan, O. C. (1989). Quality of life: Its measurement and use. Mental Retardation, 27, 25–31.
References
277
Schalock, R. L., & Luckasson, R. (2005). Clinical judgment. Washington, DC: American Association on Mental Retardation. Schalock, R. L., & Verdugo, M. A. (2002). Handbook on quality of life for human service practitioners. Washington, DC: American Association on Mental Retardation. Schill, M. T., Kratochwill, T. R., & Elliott, S. N. (1998). Functional assessment in behavioral consultation: A treatment utility study. School Psychology Quarterly, 13, 116–140. Schreibman, L., Koegel, R. L., Mills, J. I., & Burke, J. C. (1984). Training parent-child interactions. In E. Schopler & G. B. Mesibov (Eds.), The effects of autism in the family (pp. 187–205). New York: Plenum. Schwartz, I. S. (1991). The study of consumer behavior and social validity: An essential partnership for applied behavior analysis. Journal of Applied Behavior Analysis, 24, 241–244. Schwartz, I. S., & Baer, D. M. (1991). Social validity assessments: Is current practice state of the art?. Journal of Applied Behavior Analysis, 24, 189–204. Schwartz, I. S., & Olswang, L. B. (1996). Evaluating child behavior change in natural settings: Exploring alternative strategies for data collection. Topics in Early Childhood Special Education, 16, 82–101. Scruggs, T. E., & Mastropieri, M. A. (1989). Mnemonic instruction of LD students: A field-based evaluation. Learning Disability Quarterly, 12, 119–125. Seekins, T., Fawcett, S. B., & Matthews, R. M. (1987). Effects of self-help guides on three consumer advocacy skills: Using personal experiences to influence public policy. Rehabilitation Psychology, 32, 29–38. Shapiro, E. S., & Eckert, T. L. (1994). Acceptability of curriculum-based assessment among school psychologists. Journal of School Psychology, 32, 167–183. Shapiro, E. S., & Goldberg, R. (1986). A comparison of group contingencies for increasing spelling performance among sixth grade students. School Psychology Review, 15, 546–557. Sheridan, S. M. (1995). Building social skills in the classroom. In S. Goldstein, L. Braswell, M. Goldstein, S. Sheridan, & S. Zentall (Eds.), Understanding and managing children’s classroom behavior (pp. 375– 396). New York: Wiley. Sheridan, S. M., & Colton, D. L. (1994). Conjoint behavioral consultation: A review and case study. Journal of Educational and Psychological Consultation, 5, 211–228. Sheridan, S. M., & Kratochwill, T. R. (1992). Behavioral parent-teacher consultation: Conceptual and research considerations. Journal of School Psychology, 30, 117–139. Sheridan, S. M., Kratochwill, T. R., & Bergan, J. R. (1996). Conjoint behavioral consultation: A procedural manual. New York: Plenum.
278
References
Sheridan, S. M., & Steck, M. C. (1995). Acceptability of conjoint behavioral consultation: A national survey of school psychologists. School Psychology Review, 24, 633–647. Shriver, M. D., Anderson, C. A., & Proctor, B. E. (2001). Evaluating the validity of functional behavioral assessment. School Psychology Review, 30, 180–192. Singh, N., & Katz, R. (1985). On the modification of acceptability ratings for alternative child treatments. Behavior Modifications, 9, 375–386. Singh, N. N., Watson, J. E., & Winton, A. S. W. (1987). Parents’ acceptability ratings of alternative treatments for use with mentally retarded children. Behavior Modification, 11, 17–26. Skinner, B. F. (1953). Science and human behavior. New York: Free Press. Skinner, B. F. (1969). Contingencies of reinforcement: A theoretical analysis. New York: Meredith. Skinner, B. F. (1971). Beyond freedom and dignity. New York: Knopf. Skinner, B. F. (1972). Cumulative record (3rd ed.). New York: AppletonCentury-Crofts. Skinner, B. F. (1974). About behaviorism. New York: Knopf. Smith, L. K. C., & Fowler, S. A. (1984). Positive peer pressure: The effects of peer monitoring on children’s disruptive behavior. Journal of Applied Behavior Analysis, 17, 213–227. Smull, M. W., & Harrison, S. B. (1992). Supporting people with severe retardation in the community. Alexandria, VA: National Association of State Mental retardation Program Directors. Spreat, S., & Walsh, D. E. (1994). Impact of treatment efficiency and professional affiliation on ratings of treatment acceptability. Mental Retardation, 32, 227–233. Stichter, J. P., Sasso, G. M., & Jolivette, K. (2004). Structural analysis and intervention in a school setting: Effects on problem behaviour for a student with an emotional/behavioural disorder. Journal of Positive Behavior Interventions, 6, 166–177. Stinnett, T. A., Crawford, S. A., Gillespie, M. D., Cruce, M. K., & Langford, C. A. (2001). Factors affecting treatment acceptability for psychostimulant medication versus psychoeducational intervention. Psychology in the Schools, 38, 585–591. Stokes, T. F., & Osnes, P. G. (1988). The developing applied technology of generalization and maintenance. In R. H. Horner, G. Dunlap, & R. L. Koegel (Eds.), Generalization and maintenance: Lifestyle changes in applied settings (pp. 5–19). Baltimore: Brookes. Strupp, H. H., & Hadley, S. W. (1977). A tripartite model of mental health and therapeutic outcome. American Psychologist, 32, 187–196. Tarnowski, K., Mulick, J., & Rasnake, L. K. (1990). Acceptability of behavioral interventions for self-injurious behavior: Replication and interinstitutional comparison. American Journal on Mental Retardation, 95, 182–187.
References
279
Tarnowski, K., Rasnake, L. K., Mulick, J., & Kelly, P. (1989). Acceptability of behavioral treatments for self-injurious behavior. American Journal on Mental Retardation, 93, 575–580. Tarnowski, K. J., & Simonian, S. J. (1992). Assessing treatment acceptance: The abbreviated acceptability rating profile. Journal of Behavior Therapy and Experimental Psychiatry, 23, 101–106. Thomas, A., & Chess, S. (1977). Temperament and development. New York: Bruner/Mazel. Thomas, D. R., & McGuire, P. S. (1988). Balancing right to treatment with intrusiveness: The psychotherapist judgment rule. Behavioral Residential Treatment, 3, 211–222. Tingstrom, D. H. (1990). Acceptability of time-out: The influence of problem behavior severity, interventionist, and reported effectiveness. Journal of School Psychology, 28, 165–169. Tingstrom, D. H., & Silver, N. C. (1990). Acceptability of school-based interventions: A replication with a black sample. Journal of Psychology, 124, 587–589. Tingstrom, D. H., McPhail, R. L., & Bolton, A. B. (1989). Acceptability of alternative school-based interventions: The influence of reported effectiveness and age of target child. Journal of Psychology, 123, 133–140. Turnbull, H. R. (1981). Legal precedent and the individual case. Exceptional Education Quarterly, 2, 81–90. Turnbull, A. P., & Ruef, M. (1997). Family perspectives on inclusive lifestyle issues for individuals with problem behavior. Exceptional Children, 63, 211–227. Turnbull, A. P., Ruef, M., & Reeves, C. (1994). Family perspectives on lifestyle issues for individuals with problem behavior (Monograph No. P-11). Lawrence, KS: University of Kansas, Beach Center on Families and Disability. Turnbull, A. P., & Turnbull, H. R. (1996). Group action planning as a strategy for providing comprehensive family support. In L. K. Koegel, R. L. Koegel, & G. Dunlap (Eds.), Positive behavioral support: Including people with difficult behavior in the community (pp. 99–114). Baltimore: Paul H. Brookes. Turnbull, A. P., & Turnbull, H. R. (1999). Comprehensive lifestyle support for adults with challenging behavior: From rhetoric to reality. Education and Training in Mental Retardation and Developmental Disabilities, 34(4), 373–394. Turnbull, A. P., & Turnbull, H. R. (2002). Comprehensive lifestyle support for adults with challenging behavior: From rhetoric to reality. In J. Lucyshyn, G. Dunlap, & R. Albin (Eds.), Families and positive behavior support: Addressing problem behavior in family contexts (pp. 57–72). Baltimore: Brookes. Turnbull, H. R., Beegle, G., & Stowe, M. (2001). The core concepts of disability policy affecting families who have children with disabilities. Journal of Disability Policy Studies, 12, 133–143.
280
References
Turnbull, H. R., Guess, D., Backus, L. H., Barber, P. A., Fiedler, C. R., Helmstetter, E., & Summers, J. A. (1986). A model for analyzing the moral aspects of special education and behavioral interventions: The moral aspects of aversive procedures. In P. R. Dokecki & R. M. Zaner (Eds.), Ethics of dealing with persons with severe handicaps: Toward a research agenda (pp. 167–210). Baltimore: Brookes. Turnbull, H. R., Wilcox, B. L., Stowe, M., Raper, C., & Hedges, L. P. (2000). Public policy foundations for positive behavioral interventions, strategies, and supports. Journal of Positive Behavior Interventions, 2, 218–230. Turnbull, H. R., Wilcox, B. L., Stowe, M., & Turnbull, A. P. (2001). IDEA requirements for use of PBS: Guidelines for responsible agencies. Journal of Positive Behavior Interventions, 3, 11–18. Twohig, M. P., & Woods, D. W. (2001). Evaluating the duration of the competing response in habit reversal: A parametric analysis. Journal of Applied Behavior Analysis, 34, 517–520. Umbreit, J. (1995). Functional analysis of disruptive behavior in an inclusive classroom. Journal of Early Intervention, 20, 18–29. Vandercook, T., York, J., & Forest, M. (1989). The McGill Action Planning System (MAPS): A Strategy for Building the Vision. Journal of the Association for Persons with Severe Handicaps, 14, 205–215. Van Houten, R., Axelrod, S., Bailey, J. S., Favell, J. E., Foxx, R. M., Iwata, B. A., & Lovaas, O. I. (1988). The right to effective behavioral treatment. Journal of Applied Behavior Analysis (JABA), 21, 381–384. Van Houten, R. (1979). Social validation: The evolution of standards of competency for targets. Journal of Applied Behavior Analysis, 12, 581–591. Vereb, R. L., & DiPerna, J. C. (2004). Research brief: Teachers’ knowledge of ADHD, treatments for ADHD, and treatment acceptability: An initial investigation. School Psychology Review, 33, 421–428. Von Brock, M., & Elliott, S. (1987). Influence of treatment effectiveness information on the acceptability of classroom interventions. Journal of School Psychology, 25, 131–144. Waas, G. A., & Anderson, G. P. (1991). Outcome expectancy and treatment acceptability: Perceptions of school-based interventions. Professional Psychology: Research and Practice, 22, 149–154. Wacker, D. P., Berg, W. K., Asmus, J., Harding, J., & Cooper, L. J. (1997). Experimental analysis of antecedent influences on challenging behaviours. In J. Luiselli & M. Cameron (Eds.), Antecedent control: Innovative approaches to behavioral support (pp. 67–86). Baltimore: Brookes. Wallace, M. D., Doney, J. K., Mintz-Resudek, C. M., & Tarbox, R. S. (2004). Training educators to implement functional analysis. Journal of Applied Behavior Analysis, 37, 89–92. Watson, T. S., & Robinson, S. L. (1996). Direct behavioral consultation: An alternative to traditional behavioral consultation. School Psychology Quarterly, 11, 267–278.
References
281
Weigle, K. L., & Scott, J. R. (2000). Effects of functional analysis information on ratings of intervention effectiveness and acceptability. Journal of the Association for Persons with Severe Handicaps, 25, 217–228. Weisgerber, R. A. (1991). Quality of life for persons with disabilities: Skill development and transitions across life stages. Gaithersburg, MD: Aspen. Wehmeyer, M. L. (1996). Self-determination as an educational outcome: Why is it important to children, youth and adolescents with disabilities?. In D. J. Sands & M. L. Wehmeyer (Eds.), Self-determination across the life span: Independence and choice for people with disabilities (pp. 15–34) Baltimore: Brookes. Wehmeyer, M. L. (1999). A functional model of self-determination: Describing development and implementing instruction. Focus on Autism and Other Developmental Disabilities, 14, 53–61. Whaley, D. L., & Malott, R. W. (1971). Elementary principles of behavior. Englewood Cliffs, NJ: Prentice-Hall. Whang, P. L., Fawcett, S. B., & Matthews, R. M. (1984). Teaching job-related social skills to learning disabled adolescents. Analysis and Intervention in Developmental Disabilities, 4, 29–38. Wheeler, J. J., Carter, S. L., Mayton, M. R., & Thomas, R. A. (2002). Structural analysis of instructional variables and their effects on task engagement and self-aggression. Education and Training in Mental Retardation and Developmental Disabilities, 37, 391–398. Winett, R. A., Moore, J. F., & Anderson, E. S. (1991). Extending the concept of social validity: Behavior analysis for disease prevention and health promotion. Journal of Applied Behavior Analysis, 24, 215–230. Witt, J. C. (1986). Teachers’ resistance to the use of school-based interventions. Journal of School Psychology, 24, 37–44. Witt, J. C., & Elliott, S. N. (1982). The response cost lottery: A time efficient and effective classroom intervention. Journal of School Psychology, 20, 155–161. Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. Kratochwill (Ed.)Advances in school psychology: Vol. 4 (pp. 251–288). Hillsdale, NJ: Erlbaum. Witt, J. C., & Martens, B. K. (1983). Assessing the acceptability of behavioral interventions used in classrooms. Psychology in the Schools, 20, 510–517. Witt, J. C., & Robbins, J. R. (1985). Acceptability of reductive interventions for the control of inappropriate child behavior. Journal of Abnormal Child Psychology, 13, 59–67. Witt, J. C., Elliott, S. N., & Martens, B. K. (1984). Acceptability of behavioral interventions used in the classrooms: The influence of amount of teacher time, severity of behavior problem, and type of intervention. Behavioral Disorders, 9, 95–104. Witt, J. C., Martens, B. K., & Elliott, S. N. (1984). Factors affecting teachers’ judgements of the acceptability of behavioral interventions: Time
282
References
involvement, behavior problem severity, and type of intervention. Behavior Therapy, 15, 204–209. Witt, J. C., Moe, G., Gutkin, T. B., & Andrews, L. (1984). The effect of saying the same thing in different ways: The problem of language and jargon in school-based consultation. Journal of School Psychology, 22, 361–367. Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214. Wolfensberger, W. (1972). The principle of normalization in human services. Toronto: National Institute on Mental Retardation. Wolfensberger, W. (1983). Social role valorization: A proposed new term for the principle of normalization. Mental Retardation, 21, 234–239. Wolery, M., & Gast, D. (1990). Re-framing the debate: Finding middle ground and defining the role of social validity. In A. C. Repp & N. Singh (Eds.), Perspectives on the use of nonaversive and aversive interventions for persons with developmental disabilities. Sycamore, IL: Sycamore Publishing Co. Wood, A., Luiselli, J., & Harchik, A. (2007). Training instructional skills with paraprofessional service providers at a community-based habilitation setting. Behavior Modification, 31(6), 847–855. York, P., York, D., & Wachtel, T. (1982). Toughlove. Garden City, NY: Doubleday.
Index
A
C
ABA International (ABAI), 215 Abbreviated Acceptability Rating Profile (AARP), 59–60, 100–101 American Association on Mental Retardation (AAMR), 88, 151, 215 American Psychological Association Ethics Code 2002, 219 Applied behavior analysis (ABA), 12, 108, 135, 214, 221, 254–255 dimensions, 4 analytic criterion, 5 applied criterion, 4 behavioral criterion, 4–5 conceptually systematic criterion, 5 effective criterion, 5 generality criterion, 6 technological criterion, 5 limited impact on education, 255 versus PBS, 15 Assessment model, of social validity, 31 Association for Positive Behavior Supports (APBS), 19, 215 Aversive procedures, and social validity, 211–218
Children’s Intervention Rating Profile (CIRP), 56–57, 97–99 Children’s Social Validity Interview (CSVI), 42 Comparative data, 177 Conceptualizations, of social validity, 21, 28–31, 228, 241 assessment model, 31 dimensions, 21–22 habilitative validity, 22–24 maintenance model, 31–32 treatment acceptability: decision-making model, 25–27 distributive model, 32–36 expansive view, 27–28 working model, 24–25 Construct validity, Messick’s principles of, 240 consequences, as validity evidence, 243 content relevance and representativeness, 240–241 convergent and discriminant correlations, with external variables, 243 generalizability and boundaries, of score meaning, 242 scoring models, as reflective of task and domain structure, 242 substantive theories, process models, and process engagement, 241–242 Consultant acceptability, 33 assessment methods, 155–156 consultant: affiliations, 151–152 characteristics, 150–151 interaction style, 154–155 implementation difficulty, 163 recent advances in, 152–154 treatment design, 156 treatment intrusiveness, 156–162 treatment variables, 163–165 Consultant importance, of social validity assessments: behavioral changes, 198–199 data evaluation, 196–198 normative comparison, 200 treatment effects, maintenance of, 199–200
B Background, of social validity, 1 ABA, dimensions of, 4–6 concept development, 3 importance, 6–8 meaning, 1–2 PBS movement, 11 effects, 18–20 origin, 12–14 PBS criterion, 14–18 purpose, 8–11 Base Level of Intrusiveness Score (BLIS), 159, 162 Behavior Analyst Certification Board (BACB), 219 Behavior Intervention Rating Scale (BIRS), 57–58, 99–100 Behavioral technology, definition of, 257, 258, 260, 262
283
284
Consumer acceptability, 34, 146 consumers, characteristics of, 147–150 treatment, knowledge of, 150 Consumer importance, of social validity assessments, 186 quality of life, 191–196 treatment effects, consumer understanding of, 187–191 Consumer significance, of treatment goals, 111 choice, 117, 120–122 coercion, awareness of, 124–126 habilitative potential, 122–124 immediate versus long-term benefits, 126 personal preferences/values, 111–117, 118–119 Consumers, characteristics of, 147–150 Council for Exceptional Children (CEC), 218–219 Code of Ethics, 219
D Data collection: comparative data, 177 social impact measure data, 177 subject evaluation data, 177 Definition, of social validity, 1–2, 37–38 Dimensions, of social validity, 21–22 applied research versus nonapplied research, 21 Direct consumers, 249 Direct observation, for social validity assessment, 47 Distributive model: of social validity, 259 of treatment acceptability, 32–36, 143 of treatment effects, 190 of treatment goals, 109
E “Ecology of support”, 17 Ethics and social validity, 209 aversive procedures and social validity, 211–218 ethical values, and social validity, 211, 218–220 social validity support, to ethical guidelines, 220 community and society, contribution to, 224 competence, 220–221 people’s rights and dignity, respect for, 222–223
Index
professional and scientific responsibility, 221–222 welfare of others, concern for, 223 Extended community, members of, 250
F Functional communication training (FCT) response, 122, 170 Future, of social validity, 247–262
G Goal attainment scaling (GAS), 187, 188, 189 sample graph, 190
H Habilitation services: impact analysis, 225 outcome analysis, 225 process analysis, 225 Habilitative validity, 22–24
I Immediate community, members of, 250 Importance, of social validity, 6–8, 66–67 Indirect consumers, 249–250 Individualized Education Program (IEP), 114, 120 Individuals with Disabilities Education Act (2004), 8, 165, 167 Instruments, for social validity evaluation, 37 availability, of instruments, 64–65 competency in administration and scoring, of instruments, 65 consultant awareness, of instruments, 65 direct observation, 47 formal measure, of treatment acceptability, 48 Abbreviated Acceptability Rating Profile, 59–60 Behavior Intervention Rating Scale, 57–58 Children’s Intervention Rating Profile, 56–57 Intervention Rating Profile, 52, 55 Intervention Rating Profile-15, 52–54, 56 Intervention-Process Rating Scale, 59 Treatment Acceptability Rating Form, 51–52 Treatment Acceptability Rating Form Revised, 52, 53, 54
285
Index
Treatment Evaluation Inventory, 49, 50, 51 Treatment Evaluation Inventory-Short Form, 49–51 importance, of social validity, 66–67 interviews, 40–47 performance criteria, 47 social validity data, potential uses of, 66 social validity measurement, problems related to, 61 treatment acceptability instruments, 61, 64 treatment acceptability measures, comparison of, 60–61, 62–63 Intervention Rating Profile (IRP), 48, 52, 55, 91 diagnostic label of client, 95–97 rater variable manipulation, 94 raters’ knowledge of treatment, 94–95 treatment variables, 91–93 Intervention Rating Profile-15 (IRP-15), 52–54, 56, 60, 91 consumer variable manipulation, 93–94 diagnostic label of client, 95–97 raters’ knowledge of treatment, 94–95 Intervention-Process Rating Scale (IPRS), 59, 76 Interviews, 40–47
J Journal of Applied Behavior Analysis (JABA), 3, 4 Journal of Developmental and Physical Disabilities, The, 74
L Local precedence versus national precedence, of treatments, 235
M Macro level organization, of social validity data, 235–238 Maintenance model, of social validity, 31–32 Medial level organization, of social validity data, 234–235 Messick’s principles: of construct validity, 240 consequences, as validity evidence, 243 content relevance and representativeness, 240–241 convergent and discriminant correlations, with external variables, 243
generalizability and boundaries, of score meaning, 242 scoring models, as reflective of task and domain structure, 242 substantive theories, process models, and process engagement, 241–242 data validation, 243–245 Micro level organization, of social validity data, 230–234 Modified Level of Intrusiveness Score (MLIS), 162
N National precedence versus local precedence, of treatments, 235 No Child Left Behind Act, 8, 167 Normalization/inclusion movement, of PBS movement, 12–13
O Organizational outcomes versus personal outcome measures, 226–227
P PBS criterion, 13 comprehensive lifestyle change and quality of life, 14–15 critical features, 15 ecological validity, 16 emphasis on prevention, 18 flexibility with respect to scientific practices, 18 life span perspective, 15–16 multiple theoretical perspectives, 18 social validity, 17 stakeholder participation, 16–17 systems change and multicomponent interventions, 17 Performance criteria, for social validity: deficient performance levels, 47 ideal performance levels, 47 normative performance levels, 47 Person-centered planning approach, 14, 115 Person-centered values, of PBS movement, 13–14, 15 Personal outcome measures versus organizational outcomes, 226–227 Planning, for social validity: informal measures, reliability and validity of, 144–146 information usage determination, 144
286
Positive behavior supports (PBS) movement, 11, 215, 251 effects, 18–20 origin: applied behavior analysis, 12 normalization/inclusion movement, 12–13 person-centered values, 13–14 PBS criterion: comprehensive lifestyle change and quality of life, 14–15 critical features, 15 ecological validity, 16 emphasis on prevention, 18 flexibility with respect to scientific practices, 18 life span perspective, 15–16 multiple theoretical perspectives, 18 social validity, 17 stakeholder participation, 16–17 systems change and multicomponent interventions, 17 technological criterion, 13 versus ABA, 15 Procedures and programs, social validity of, 169–174 Program Evaluation Model, 226, 227, 228–229 Program-centered approach, 14 Purpose, of social validity, 8–11
R Research, on social validity, 69 methodological variations, 74 protocols, for social validity measurement, 69–73 research findings, 81 AARP, 100–101 BIRS, 99–100 CIRP, 97–99 IRP and IRP-15, 91–97 TARF-R, 101–102 teacher acceptability measure (TAM), 97 TEI and TEI-SF, 81–91 treatment acceptability, literature reviews of, 75–80
S Semi-structured interview, for social validation, 40–41, 42, 136–137, 186–187 Social impact measure data, 177
Index
Social importance: meaning, 3 of treatment effects, 204–208 Social validity assessments, 39, 40, 185 consultant importance: behavioral changes, 198–199 data evaluation, 196–198 normative comparison, 200 treatment effects, maintenance of, 199–200 consumer importance, 186 quality of life, 191–196 treatment effects, consumer understanding of, 187–191 societal importance: community expert evaluation, 200–201 cost analysis, 201–203 local/national norms, conformity to, 203–204 Social validity data, 225 data validation, 238 examination, 239–243 using Messick’s principles, 243–245 macro level organization, 235–238 medial level organization, 234–235 micro level organization, 230–234 potential uses, 66 validation levels, 178 Social validity measurement: problems related to, 61 protocols for, 69–73 Social validity measurement inventory, 229 Social Validity Measurement Matrix, 227 Societal appropriateness, of treatment procedures, 165–169 Societal importance, of social validity assessments: community expert evaluation, 200–201 cost analysis, 201–203 local/national norms, conformity to, 203–204 Stewart, Potter, 3 Student-Assisted Functional Assessment Interview, 42, 43–46 Subject evaluation data, 177
T Teacher acceptability measure (TAM), 97 The Association for the Severely Handicapped (TASH), 215 Three-component definition, of social validity, 2
Index
Treatment acceptability, 37, 38, 69–70 analogue research methodology, 74 clinical research methodology in, 74 contradictory findings, 104 decision-making model, 25–27 distributive model, 32–36, 143 expansive view, 27–28 efficacy considerations, 27 legal/social implications, 27–28 practical considerations, 28 secondary effects, 27 formal measure, 48 Abbreviated Acceptability Rating Profile, 59–60 Behavior Intervention Rating Scale, 57–58 Children’s Intervention Rating Profile, 56–57 Intervention Rating Profile, 48, 52, 55 Intervention Rating Profile-15, 52–54, 56 Intervention-Process Rating Scale, 59 Treatment Acceptability Rating Form, 51–52 Treatment Acceptability Rating Form Revised, 52, 53, 54 Treatment Evaluation Inventory, 48, 49, 50, 51 Treatment Evaluation Inventory-Short Form, 49–51 instruments: characteristics, 62–63 usage, 61, 64 literature reviews, 75–80 measures, 60–61, 62–63 research methodology, 74 working model, 24–25 Treatment Acceptability Rating Form (TARF), 51–52 Treatment Acceptability Rating Form Revised (TARF-R), 52, 53, 54, 101–102 Treatment appropriateness, of social validity, 142 Treatment component value matrix, 148, 149 Treatment effects, importance of, 175, 204–208 disability policy, 179 accountability, 184 antidiscrimination, 180–181 autonomy, 182 capacity-based services, 181 classification, 181 core concepts, 180 cultural responsiveness, 184
287
empowerment/participatory decision making, 182 family centeredness, 183 family integrity and unity, 183 individualized and appropriate services, 181 integration, 183 liberty, 182 prevention and amelioration, 184–185 privacy and confidentiality, 183 productivity and contribution, 183 professional and system capacitybuilding, 184 protection from harm, 182 service coordination and collaboration, 182 distributive model, 190 at macro level, 178, 179–180 at medial-level, 178, 180 at micro level, 178, 180 social validity assessments, conducting, 185 consultant importance, 196–200 consumer importance, 186–196 societal importance, 200–204 Treatment Evaluation Inventory (TEI), 48, 49, 50, 51, 81 client variables, 87 rater variables, 87–91 treatment information, 85–87 treatment variables, 82–85 Treatment Evaluation Inventory-Short Form (TEI-SF), 49–51, 81–91 Treatment Goal Prioritization sheet, 120, 121 Treatment goals, social significance of, 107 actualization potential, 129–131 boosting strategies, 136–140 checklist for, 118–119 clarity and specific outcome criteria, 128 complexity and comprehensiveness, 128–129 consultant: personal/professional agenda, 127 training, 126–127 consumer significance, 111 choice, 117, 120–122 coercion, awareness of, 124–126 habilitative potential, 122–124 immediate versus long-term benefits, 126 personal preferences/values, 111–117, 118–119 distributive model, 109 instrument/method, 131–133
288
Treatment goals, social significance of (continued ) at macro level, 178 at medial-level, 178 at micro level, 178 society: normalization, 134–135 resources and funding, availability of, 135–136 Treatment history, 86, 235 Treatment Intrusiveness Measure (TIM), 159 Treatment precedence, 28, 235 Treatment procedures, appropriateness of, 141 consultant acceptability: affiliations, of consultant, 151–152 assessment methods, 155–156 characteristics, of consultant, 150–151 implementation difficulty, 163 interaction style, of consultant, 154–155 recent advances, 152–154 treatment design, 156 treatment intrusiveness, 156–162
Index
treatment variables, 163–165 consumer acceptability, 146 consumers, characteristics of, 147–150 knowledge of treatment, 150 at macro level, 178, 179–180 at medial-level, 178, 180 at micro level, 178, 180 planning, for social validity: informal measures, reliability and validity of, 144–146 information usage determination, 144 social validity of procedures and programs, strategies for improving, 169–174 societal appropriateness, 165–169
U Unstructured interview, 40
V Validation, of social validity data, 238, 239–245
