34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page i
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH SECOND EDITION
Edited by
Robert L. Kane, MD Minnesota Chair in Long-Term Care and Aging University of Minnesota School of Public Health
34411_CH00_FM_i_xii.qxd
9/18/07
World Headquarters Jones and Bartlett Publishers 40 Tall Pine Drive Sudbury, MA 01776 978-443-5000
[email protected] www.jbpub.com
4:45 PM
Page ii
Jones and Bartlett Publishers Canada 6339 Ormindale Way Mississauga, Ontario L5V 1J2
Jones and Bartlett Publishers International Barb House, Barb Mews London W6 7PA UK
Jones and Bartlett’s books and products are available through most bookstores and online booksellers. To contact Jones and Bartlett Publishers directly, call 800-832-0034, fax 978-443-8000, or visit our website www.jbpub.com. Substantial discounts on bulk quantities of Jones and Bartlett’s publications are available to corporations, professional associations, and other qualified organizations. For details and specific discount information, contact the special sales department at Jones and Bartlett via the above contact information or send an email to
[email protected]. Copyright © 2006 by Jones and Bartlett Publishers, Inc. All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner. ISBN-13: 978-0-7637-3441-1 ISBN-10: 0-7637-3441-1 Library of Congress Cataloging-in-Publication Data Understanding health care outcomes research / Robert L. Kane.— 2nd ed. p. ; cm. Includes bibliographical references and index. ISBN 0-7637-3441-1 1. Outcome assessment (Medical care) [DNLM: 1. Outcome Assessment (Health Care) 2. Research Design. W 84.1 U55 2006] I. Kane, Robert L., 1940– R853.O87U53 2006 362.1—dc22 6048 2005018079 Production Credits Acquisitions Editor: Michael Brown Editorial Assistant: Kylah McNeill Production Director: Amy Rose Production Editor: Renée Sekerak Production Assistant: Rachel Rossi Associate Marketing Manager: Marissa Hederson Manufacturing Buyer: Therese Connell Composition: Auburn Associates, Inc. Cover Design: Kristin E. Ohlin Printing and Binding: Malloy, Inc. Cover Printing: Malloy, Inc. Printed in the United States of America 11 10 09 08 07 10 9 8 7 6 5 4 3 2 7196
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page iii
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Author Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix PART I—BASIC CONCEPTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1—Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Robert L. Kane Why Look at Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 An Outcomes Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Types of Study Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Measuring Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Conceptual Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Summary: Organization of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Chapter 2—Designing an Outcomes Research Study . . . . . . . . . . . . . . . . 23 David M. Radosevich Introduction: Types of Study Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Evaluating the Threats to Outcomes Research . . . . . . . . . . . . . . . . . . . . . . 25 Statistical Conclusion Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Other Threats to Internal Validity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 External Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Quasi-Experimental Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Summary: General Guidelines for Designing a Health Outcomes Research Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Chapter 3—Isolating the Effects of Treatment . . . . . . . . . . . . . . . . . . . . . . 59 Paul L. Hebert Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Conceptual Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 iii
34411_CH00_FM_i_xii
iv
9/9/05
9:17 AM
Page iv
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
Statistical Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Example: The Effectiveness of Influenza Vaccination . . . . . . . . . . . . . . . . 76 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Chapter 4—Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Jennifer R. Frytak, Robert L. Kane Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 The Nature of Measurement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Measurement Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Reliability and Validity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Interpreting Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Advantages of Multiple- Versus Single-Item Measures . . . . . . . . . . . . . . 110 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 PART II—SPECIFIC MEASURES: OUTCOMES . . . . . . . . . . . . . . . . . . . 121 Chapter 5—Generic Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Matthew L. Maciejewski Introduction: Why Use Generic Measures?. . . . . . . . . . . . . . . . . . . . . . . . 123 What Health Domains Are Typically Measured in Generic Measures? . . 125 What Are the Criteria for Generic Measures? . . . . . . . . . . . . . . . . . . . . . . 131 Practical Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Preference Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Choosing a Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Health Utility Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Chapter 6—Condition-Specific Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Adam Atherly Introduction: Condition-Specific Versus Generic Measures . . . . . . . . . . 165 Why Not Generic Health Status Measures? . . . . . . . . . . . . . . . . . . . . . . . 167 Condition-Specific Health Status Measures . . . . . . . . . . . . . . . . . . . . . . . 168 Other Alternatives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 The Choice of a Condition-Specific Measure . . . . . . . . . . . . . . . . . . . . . . 171 The Role of Condition-Specific Versus Generic Measures . . . . . . . . . . . 177 Choosing a Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Chapter 7—Satisfaction with Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Maureen A. Smith, Chris Schüssler-Fiorenza, Todd Rockwood Introduction: The Importance of Patient Satisfaction . . . . . . . . . . . . . . . . 185 Theoretical Models of Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 Interpreting Satisfaction Ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page v
Table of Conents
v
Methods of Measuring Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Existing Satisfaction Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 Literature Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Summary/Future Directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 PART III—SPECIFIC MEASURES: RISK ADJUSTERS. . . . . . . . . . . . . 217 Chapter 8—Severity and Comorbidity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Maureen A. Smith, Nicole M. Nitz, Sara K. Stuart Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Relationship Between Severity of Illness and Comorbidity . . . . . . . . . . . 220 Severity of Illness and the Domains of Health . . . . . . . . . . . . . . . . . . . . . 222 Components of Severity of Illness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Reasons to Include Severity and Comorbidity Measures . . . . . . . . . . . . . 229 How to Choose a Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Issues Specific to Severity Measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 Issues Specific to Comorbidity Measures . . . . . . . . . . . . . . . . . . . . . . . . . 248 Specific Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Chapter 9—Demographic, Psychological, and Social . . . . . . . . . . . . . . . 265 Todd Rockwood, Melissa Constantine Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Three Types of Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 Demographic Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 Psychological . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Social . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 PART IV—ANALYSIS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Chapter 10—Capturing the Effects of Treatment . . . . . . . . . . . . . . . . . . 307 Jeremy Holtzman Introduction: The Importance of Understanding Treatment . . . . . . . . . . . 307 What Is Treatment?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Components of Treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 Understanding the Components of Treatment . . . . . . . . . . . . . . . . . . . . . . 313 Need for Variation in Treatment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 The Treatment Variation Defines the Treatment . . . . . . . . . . . . . . . . . . . . 319 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Chapter 11—Cost-Effectiveness Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 335 John A. Nyman Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Types of Cost Effectiveness Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
34411_CH00_FM_i_xii
vi
9/9/05
9:17 AM
Page vi
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Health-Related Quality of Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 A Hypothetical Illustration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 PART V—RESEARCH ISSUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Chapter 12—Implementing Outcomes Research in Clinical Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 David M. Radosevich Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Organizational Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Implementation Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 Regulatory Demands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 Appendix 12A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 Chapter 13—Practical Advice About Outcomes Research . . . . . . . . . . . 371 Robert L. Kane Introduction: Organizing One’s Thinking . . . . . . . . . . . . . . . . . . . . . . . . . 371 The Search for Simple Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Adjusting for Case Mix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Getting Follow-up Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 Using Extant Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 Basic Analysis Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 Disease Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Quality Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Operational Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page vii
Preface
This book originally began as an exercise to bring together diverse material on health outcomes research. Since the original publication, we have held five summer institutes to teach the material contained in the book. These courses combined didactic material with an opportunity to develop a model study. The experience gained from these courses provided useful insights into material that was not available in the first edition. This edition expands the scope of the presentation as well as updating information and providing new insights into familiar topics. We owe a great debt to the students of these workshops. They taught us a great deal about how to better present this information and what would be of interest to both clinicians and researchers. We hope that this volume reflects those lessons. What was once a fairly obscure pursuit in 1997, when the first edition appeared, has now become a major enterprise. Everyone seems to be jumping on the quality bandwagon. However, the approaches to conducting health outcomes research vary a great deal. We strongly believe there continues to be a real need for a text that provides both an overview and insights into some of the more subtle aspects of this pursuit. We view our task as providing neophytes with enough information to get them launched while offering even experienced researchers some new ideas about how to improve their designs and approaches to this complex task. The authors of the various chapters have been actively involved in conducting outcomes research studies; several of them have also taught in our continuing series. Robert L. Kane, MD vii
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page viii
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page ix
Author Biographies
Adam Atherly, PhD is an Assistant Professor in the Department of Health Policy and Management in the Rollins School of Public Health at Emory University. Dr. Atherly received his PhD in Health Services Research, Policy, and Administration from the University of Minnesota. His main area of research is in the economics of aging and consumer decisions regarding health plan choice and health. Dr. Atherly has been involved in health outcomes research, including scale development, evaluation of efforts to improve quality of care, and patient safety and cost effectiveness analysis. He also works with the National Center for Environmental Health in the Centers for Disease Control on the economics of asthma and respiratory illness. Chris Schüssler-Fiorenza, MD is a post-doctoral Research Fellow and General Surgery Resident in the Division of General Surgery at the University of Wisconsin Medical School, where she is conducting research on outcomes for colorectal cancer patients. Melissa Constantine is a doctoral student in the Division of Health Services Research and Policy at the University of Minnesota. Her dissertation is focused on evaluating the impact of social interaction and the social construction of reality on the quality of informed consent processes in clinical medicine. Her research interests are focused on the intersection of bioethics and human behavior in the social environment. Jennifer R. Frytak, PhD specializes in analysis of health care outcomes and costs with an emphasis on prospective research. As a Senior Researcher at i3 Magnifi Health Economics and Outcomes Research, she is responsible for designing research protocols, conducting analyses, and ix
34411_CH00_FM_i_xii
x
9/9/05
9:17 AM
Page x
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
disseminating results to clients. Previously, she worked as a researcher at the United Health Group, Center for Health Care Policy and Evaluation, where she studied specialty care use among older adults in managed care, the business case for quality, the effect of utilization review on cost and quality, and underlying explanations for the hospital volume-treatment outcome relationship. Dr. Frytak earned a PhD in Health Services Research and Policy at the University of Minnesota. Paul L. Hebert, PhD is an Assistant Professor of Health Policy in the Department of Health Policy at Mount Sinai School of Medicine in New York, NY. Dr. Hebert is a health economist with an expertise in outcomes research and cost effectiveness. His experience in outcomes research includes assessments of the effectiveness of influenza and pneumococcal vaccination for Medicare beneficiaries; the effects of disease management on outcomes for persons with congestive heart failure; the effects of home blood pressure monitors on outcomes for persons with hypertension; the effectiveness of case management for frail elderly individuals in assisted living centers; and the effects of HMO enrollment on Medicare beneficiaries with diabetes. Jeremy Holtzman, MD, MS is an Assistant Professor in the Division of Health Services Research and the Department of Medicine at the University of Minnesota. He is also a member of the Clinical Outcomes Research Center at the University of Minnesota and a practicing Internist at Hennepin County Medical Center. He has conducted a number of outcomes research studies including studies of the outcomes of vascular disease and hip arthroplasty. Robert L. Kane, MD is the former Dean of the University of Minnesota School of Public Health, where he currently holds an endowed chair position in Long-Term Care and Aging. He directs the University of Minnesota’s Center on Aging, the Minnesota Area Geriatric Education Center, the Clinical Outcomes Research Center, and an AHRQ-funded Evidencebased Practice Center. He has conducted numerous research studies on both the clinical care and the organization of care for older persons, especially those needing long-term care. He is a graduate of Harvard Medical School. Matthew L. Maciejewski, PhD is an Associate Professor in the Department of Health Services at the University of Washington and a Core Investigator in the Health Services Research and Development Center of Excellence at the VA Puget Sound Health Care System. His research interests include the economic and quality of life effects of diabetes and obe-
34411_CH00_FM_i_xii
9/9/05
9:17 AM
Page xi
Author Biographies
xi
sity, Medicare managed care, VA utilization and costs, and research methods. Dr. Maciejewski has conducted evaluations of selection bias into Medicare HMOs by Medicare beneficiaries with and without diabetes, community-based primary care for veterans; the effect of VA medication co-payment increases on medication use; health care utilization and costs of veterans with diabetes and/or hypertension. He is a co-investigator on an NIH-funded randomized trial of weight loss in people with diabetes. Dr. Maciejewski obtained his PhD from the University of Minnesota in 1998. Nicole M. Nitz, MS is a researcher at Ingenix Pharmaceutical Services, specializing in health outcomes and health economics, which she has applied in research on a variety of disease states including depression, epilepsy, schizophrenia, and asthma. Prior to joining Ingenix, Ms. Nitz was a Health Outcomes Researcher at Eli Lilly and Company. She was responsible for the design of economic and outcomes studies integrated in clinical trials for new drug applications, a large observational study of patient outcomes, and incorporating health outcomes and health economics strategies into product marketing plans. She is a candidate for a PhD in Health Services Research at the University of Minnesota and earned her MS in Health Services Research, Policy, and Administration at the University of Minnesota. John A. Nyman, PhD is a health economist and Professor in the Division of Health Services Research and Policy, School of Public Health at the University of Minnesota. His research interests include the standardization of cost-utility analysis, the theory of demand for health insurance, physician behavior, long-term care markets and policy, and gambling as a public health issue. Dr. Nyman has conducted or is conducting costeffectiveness studies on a wide variety of interventions including: the drug Gemfibrozil‚ a falls prevention program for the elderly, mindfulness meditation training for organ transplant patients, and treatment protocols for bulimia, diabetes, and anorexia. He teaches courses in cost-effectiveness analysis in health care, the economics of the health care sector, and health insurance. Dr. Nyman holds a PhD in economics from the University of Wisconsin. David M. Radosevich, PhD, RN is an epidemiologist and Assistant Professor at the University of Minnesota. He is the Director of Transplant Information Services (TIS), a corporation that coordinates, collects, and disseminates health outcomes information for the solid organ transplant program. In addition, he is the Deputy Director of the Clinical Outcomes Research Center (CORC) in the School of Public Health. Dr. Radosevich is
34411_CH00_FM_i_xii
xii
9/9/05
9:17 AM
Page xii
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
actively involved in studies evaluating the effectiveness of health programs for chronic disease, disease management, and organ donation in Minnesota. He frequently consults with clinical researchers, medical group practices, and health plans in outcomes measurement, study design and analytic methods. He currently teaches graduate courses in the epidemiology of aging and conducting health outcomes research. Todd Rockwood, PhD is an Assistant Professor in the Division of Health Services Research and Policy in the University of Minnesota School of Health. He is a sociologist with special training in survey methods. As a core member of the CORC staff, he collaborates on outcomes studies with a variety of health professionals. He has been active in developing condition-specific quality of life measures. Maureen A. Smith, MD, MPH, PhD is an Assistant Professor in the Department of Population Health Sciences at the University of Wisconsin– Madison Medical School. Her research has examined access to health care, quality of care, and health outcomes for aging and chronically ill persons. Dr. Smith has conducted extensive studies on health outcomes and quality of life for older stroke patients. In related work, she examines the disparities in access to health care and health outcomes in older adults, including those with lung, breast, and colorectal cancer. Sara K. Stuart received her Bachelor of Science degree from the University of Wisconsin–Oshkosh and is currently attending the University of Wisconsin–Madison Medical School. During the summer of 2004 she held a Shapiro Research Fellowship with the Department of Population Health Sciences, where she examined the role of patient age in the outcomes of cancer care.
34411_CH01_001_022
9/9/05
9:21 AM
Page 1
Part I Basic Concepts
34411_CH01_001_022
9/9/05
9:21 AM
Page 2
34411_CH01_001_022
9/9/05
9:21 AM
Page 3
1 Introduction Robert L. Kane
WHY LOOK AT OUTCOMES Quality of care has assumed increased importance. Several factors are responsible, including the fact that much has been made of the dangers of medical errors (Kohn, Corrigan, & Donaldson, 2000). Demands for accountability have increased under pressures to see society become more prudent purchasers of care. The growing cost of health care raises renewed questions about its value. As a by-product, assessing outcomes of care has taken on new importance. The focus of much clinical research has broadened to address larger questions about the ultimate impact of care. The “outcomes” examined in outcomes research are more likely to approximate what one ultimately wants health care to achieve—improvements in functional status and quality of life. Outcomes research differs from other medical research in another important way: It is more inclusive of what is considered an intervention. Whereas most medical research may examine the effects of a particular drug or surgical intervention, outcomes research may examine the effects of such elements as counseling or even reorganizing the way care is delivered. Hence, outcomes research may ask not only are individuals with CHD better off with angioplasty or medical therapy (a valid outcomes research question) but are individuals with CHD who get their health care in HMOs better off than others. Like Moliere’s bourgeois gentilhomme who suddenly discovered he had been speaking prose all his life, health care providers seem to have awakened to the need to examine the results of their labors. The observations 3
34411_CH01_001_022
4
9/9/05
CHAPTER 1
9:21 AM
Page 4
INTRODUCTION
about the large variation in the rates of various medical activities and utilization of care stirred interest in whether these differences had any effect on outcomes (Chassin et al., 1987; Leape et al., 1990; Wennberg & Gittelsohn, 1982; Wennberg, Freeman, Shelton, & Bubolz, 1989). The rise of managed care, with its industrial accountability and productivity models, stimulated a revised way of thinking about care. As the variation data generated a press for greater consistency, which was translated into a demand for clinical guidelines, it became quickly evident that medicine does not have a vast store of empirically verified information about the relationship between what is done and the results. Evidence-based medicine is de rigueur. Much attention has been devoted of late to methods for assessing the quality of the medical literature and summarizing the findings from it (Sackett, 1997). The Cochrane Collaborating Centers have developed a library of reports that assess the literature on various topics and make recommendations for practice (see www. cochrane.org/index0.htm). The Agency for Healthcare Research and Quality has chartered a set of evidence-based practice centers to conduct systematic literature reviews and report on the findings with a direct goal of providing the bases for practice recommendations (see www.ahrq.gov/ clinic/epcix.htm). Coincident with all this attention to outcomes has been a growth in outcomes research programs. Most academic medical centers now have some entity charged with leading research on the outcomes of care. Many managed care programs have such a unit, either directed at research per se or linked more closely to clinical activity under the general heading of quality improvement. Indeed the prototype for such an organization is the Institute for Health Care Delivery Research in Intermountain Health Care, which has pioneered applied outcomes research (www.ihc.com/xp/ihc/physician/ research/institute/). Outcomes analysis can be undertaken for several reasons: 1. To make market decisions. In an ideal world, consumers looking for help might want to know how well a given clinician has performed in treating their specific problem. Likewise, those acting on behalf of consumers (e.g., benefits managers) might want such information to help in their decisions about with whom to contract. 2. For accountability. Several agencies have a stake in the quality of medical care. Formal regulatory activity is vested in the government and in professional societies. Payers may also be concerned that the
34411_CH01_001_022
9/9/05
9:21 AM
Page 5
Why Look at Outcomes
5
care they are buying is of adequate quality. In effect, the same information on the outcomes achieved can be analyzed at the level of a clinician or a clinic or a hospital (if the sample size is large enough). In conducting such analyses, however, appropriate adjustments for casemix and other relevant risk factors are needed in both cases. 3. To improve the knowledge base of medicine. The substrate for evidence-based medicine (EBM) is good evidence on which to base it. Solid outcomes information is the crucial building block for the EBM edifice. The enthusiasm for establishing guidelines for care has been somewhat dampened by the growing realization that the empirical database for most of these recommendations is quite weak and they are forced to rely on clinical consensus judgments. Although some would hold that the only real science comes from randomized controlled trials (RCTs), much can be learned by carefully applying epidemiological analyses to large databases of wellcollected experiential information. Outcomes research should be seen as complementing, not competing with RCTs. It attempts to address a particular type of medical knowledge, that is, a better understanding of how treatment in the real world affects a wide range of outcomes. Outcomes can be expressed in different ways. Perhaps the simplest and most direct measure is survival, although some ethicists might seek to complicate even this determination. Clinicians are most familiar with clinical measures ranging from death to values of specific parameters like blood pressure. Outcomes can also be derived from symptoms or even the results of physical examinations. They can be the results of simple tests, like blood levels, or more complex physiological measures. Another set of outcomes relies on information collected from patients. This data usually reflects how they have experienced the illness and the effects it has had on their lives. These outcomes include measures of functioning as well as measures of affect. Satisfaction with care and with life in general can be considered part of this set of outcomes. In general, clinicians place greater faith in the data they get from laboratory tests and their own observations than what patients report, but this prejudice may not be appropriate. One cannot actually measure “health” in a laboratory. One may not be able to relate the results of the colorimetric reaction to outcomes that a researcher might be interested in. For example, knowing the oxygen saturation in the great toe of a person with impaired circulation may be wonderful, but it is more salient to know
34411_CH01_001_022
6
9/9/05
CHAPTER 1
9:21 AM
Page 6
INTRODUCTION
that the patient still can’t walk. Patient-derived information can be as valid as—or even more valid than—that obtained from a machine. For example, the results of a scale based on patient perceptions of events may be as valid as the inference placed on the results of a colorometric reaction that is interpreted as reflecting the level of enzymatic activity. Why Outcomes May Be Hard to Sell Looking directly at the outcomes of care (as opposed to concentrating on the process of care) makes a lot of sense. In the best traditions of the famous bank robber, Willie Sutton, that is where the treasure can be found. However, using outcomes may be less satisfying than one may wish. Clinicians have difficulties with outcomes on several grounds: 1. The outcomes of care may be due to many things, only some of which are under the clinician’s control. Outcomes are rarely the product of a single individual’s efforts. Instead, they result from the collaboration and coordination of many people working within a system. System failure is at least as deadly as individual error (Berwick, 1989). It is much more satisfying to be able to say one did all the right things, even if something bad happened. Some estimates suggest that medical care has only a limited effect on the overall health of a population; numbers in the range of 10 to 25 percent are bandied about. It seems reasonable to assume that the size of the effect of treatment on specific sick people is larger, but other factors will influence the results. It is not necessary that treatment explain all (or even most) of the variance on outcomes to make it worthwhile to examine its effectiveness. One can change the risk of a successful outcome by several orders of magnitude by interventions that fail to explain even a modest amount of the variance in outcomes. 2. Although theory suggests that outcomes and process measures are closely linked, the correlation between process and outcomes is often weak. Hence, a poor outcome does not necessarily indicate what needs to be done differently. At best, outcomes can only suggest to an investigator where to look for more information about the process of care. In clinical practice, they are often best considered as screeners. Rather than examining the processes of care for all the care provided,
34411_CH01_001_022
9/9/05
9:21 AM
Page 7
Why Look at Outcomes
3.
4.
5.
6.
7
a pattern of poor outcomes can suggest which types of care (or which providers) need closer scrutiny. Outcomes information usually requires extra effort (and expense) to collect. Medical record keeping is notoriously inconsistent (Weed, 1968a, 1968b). Much information is recorded as judgments and definitions vary widely. Omissions are frequent. What does “within normal limits” mean? Medical practice does not routinely gather systematic information about the outcomes of care. At best, clinicians are generally aware of only those patients who return for further care. Rarely do they systematically follow the course of those who do not return, although these may be the outcomes of greatest interest. Even less often do they systematically collect data on other variables that might influence the outcomes. Outcomes are essentially probability statements. Because outcomes can be influenced by many different factors, one should not try to judge the success of any single case; instead, outcomes are addressed in the aggregate. The rate of success is compared. Thus, outcomes reflect the experience of a clinician, not the results of any single effort. Because outcomes rely on group data, there must be enough cases to analyze. For many clinicians, the volume of cases around a specific condition is too small to permit rapid aggregation for analysis. One must either collect cases over several years or use a group of physicians as the unit of analysis. Both strategies have disadvantages. Outcome results take a long time to assemble. First, one has to accumulate cases. For each case, one has to wait for the outcomes to become evident. As a result, by the time an outcomes report is available, the care reported on may have occurred some time ago. The results may no longer seem fresh.
Given all these problems, it is little wonder that people would rather talk about outcomes than deal with them. It is much more comfortable to test the extent to which care complies with extant orthodoxy, but one quickly runs into a paradox. Despite all the attention to EBM, thoughts about what constitutes appropriate care are still more often based on beliefs than hard evidence. Before one endorses an orthodoxy, a person would like to have better proof that a given approach really leads to better outcomes. Consensus should not be confused with wisdom. Imagine what would have happened if there had been a consensus conference in the mid-19th century
34411_CH01_001_022
8
9/9/05
CHAPTER 1
9:21 AM
Page 8
INTRODUCTION
on cupping and leaching. Developing such linkages means having a data system that can provide the needed grist for the analytic mill.
Collecting Outcomes Information Two strategies are available to collect outcomes information: 1. Routine medical practice can incorporate system data collection and feedback to track outcomes of care. The rise of managed care, with its improved information systems and its concerns about efficiency, may prove a catalyst for this effort. 2. Special practices can be designated to operate data collection activities under some scientific aegis that would systematically collect data on outcomes and relate them to the process of care (much the way academic centers conduct clinical trials to test new therapies). Practitioners would then rely on the validated processes for assessing their quality of care. Having recognized the discrepancy between what one knows and what one believes, medicine was at an impasse. One camp, anxious for fast results, pushed for creating practice guidelines based on the best available information and filling in the rest with expert opinion. They argued that, at worst, such a strategy would produce the equivalent of a higher quality textbook. The other camp maintained that enforcing arbitrary rules not based on empirical evidence was equivalent to codifying beliefs. They urged greater restraint until a better science base was developed. The early experience with guideline writing confirmed the weak science base that underlay much of clinical practice (Field & Lohr, 1992). The big question then was how to remedy the situation—with systematic outcomes research the obvious answer. The choice of the best research strategy remained the question. The classical view of quality of medical care used a framework that divided such work into structure, process, and outcome (Donabedian, 1966). Structure referred to such aspects as the training of the care providers or the equipment of the facility in which the care was provided. Process addressed what was done: Was the correct (appropriate) action taken? Was it done skillfully? Outcomes referred to the results of these actions. There was an assumption that these three aspects are directly
34411_CH01_001_022
9/9/05
9:21 AM
Page 9
An Outcomes Approach
9
related, but that belief has often proved hard to demonstrate empirically. One explanation was that the “lore” or medicine was just that: a set of beliefs and traditions that were poorly grounded in empirical evidence. Another interpretation was that the effects of care were simply too subtle to be easily revealed by most studies, especially nonexperimental ones. The weak relationships often found between process and structure on the one hand and outcomes on the other cut both ways. Investigators seeking to demonstrate the validity of their outcomes findings may turn to structural and process correlations. Turning the system on its head, one might test the validity of guidelines by assessing whether those adhering to the guidelines achieved better results than those who do not. If outcome measures work, one would expect to find better outcomes among those providers judged by some external standard to give better care. What does it mean when care provided in teaching hospitals is no better than that offered in community hospitals? On the one hand, the measures may be insensitive; alternatively, there may be less difference than one might suspect. If the results are the inverse of what was expected, there will obviously be greater cause for concern, but failure to find a difference where orthodox teaching said one should be found may raise at least as many questions about the orthodoxy as challenges to the validity of the observation.
AN OUTCOMES APPROACH An outcomes approach requires more than simply collecting data on the outcomes of care. Rather, it should be considered in terms of an outcomes information system. Careful and complete data collection for purposes of both outcomes ascertainment and risk adjustment has to be combined with proper analyses. The basic model for analyzing the outcomes of care is the same whether one uses a RCT or an epidemiological approach. The model is summarized as follows: Outcomes = f(baseline, patient clinical characteristics, patient demographic/ psychosocial characteristics, treatment, setting)
This pseudoequation indicates that clinical outcomes are the result of several factors, which can be classified as risk factors (baseline status, clin-
34411_CH01_001_022
10
9/9/05
CHAPTER 1
9:21 AM
Page 10
INTRODUCTION
ical status, and demographic/psychosocial characteristics) and treatment characteristics (treatment and setting).1 The goal of the analysis is to isolate the relationship between the outcomes of interest and the treatment provided by controlling for the effects of other relevant material. The latter is often referred to as risk adjustment.
Risk Adjustment The patient’s baseline status is very important. With a few exceptions (such as plastic surgery and elective orthopedics), patients never get better than they were before the episode that started the need for treatment in the first place. Thus, really two types of baseline status information need to be collected: 1. status at the outset of treatment (which can be used to show how much change has occurred since treatment began) 2. usual status before the onset of the problem that requires treatment (which defines the upper bound of just how much improvement is possible or likely) Information on baseline status basically corresponds to what will be later collected to assess outcomes. Patient clinical characteristics cover a lot of territory. One of the reasons clinicians make diagnoses is to group patients into classes that share a need for a given type of therapy and/or suggest an expected course. Knowing a patient’s diagnosis would thus play a central role in building an outcomes data system. Many patients have more than one diagnosis. It is necessary for purposes of analysis to identify one diagnosis as the primary diagnosis and to treat the others as modifiers.2 These are often referred to as comorbidities. Diagnoses can be further refined in terms of their implications for outcomes by addressing characteristics that suggest varying prognoses. These are termed severity measures. In addition to severity, one may be concerned about other modifiers of diagnoses such as duration of the problem and history of previous episodes. In general, it is usually safer to be as inclusive as possible. Because clinicians are especially distrustful of nonrandomized controlled trials, they need a great deal of reassurance that all possible differences between groups have been considered. By including
34411_CH01_001_022
9/9/05
9:21 AM
Page 11
An Outcomes Approach
11
elements that seem unnecessary, the investigator may eventually gain greater acceptance for the results. Nothing is more frustrating than presenting an analysis, especially one that challenges conventional wisdom, only to have the clinical audience say: “Yes, but did you consider . . . ?” A policy of inclusion is not an automatic talisman against rejection, but it can help avoid it. At some point, of course, the cost of collecting seemingly irrelevant data can be overwhelming. A reasonable compromise must be struck. If the clinician audience is involved in planning the study, at least those elements that seem most important can be covered. Other clinical information may address different risk factors (e.g., exposure to toxins, diet, habits). The other set of patient information concerns demographic and psychosocial factors. Some obvious items, such as age and gender, seem to need no justification, but even they should be thoughtfully addressed. A specific conceptual model that indicates the expected influence of each variable is a critical fist step in planning an outcomes study. Others, such as education and social support, may exert their effects more subtly. The relevance of specific elements may vary with the condition being examined. Other psychosocial variables, like the patient’s cognitive or emotional state, may influence the effects of treatment on other outcomes.
Treatment Setting refers to both the physical location where the care is provided as well as the organization of that site. It can also address other attributes such as the philosophy of care provided. For example, one may want to compare the same basic care provided in an inpatient and outpatient context. Alternatively, one may want to address the level of risk aversion or the extent of staffing for apparently similar models of care. One site may have a philosophy of encouraging patients to do as much as possible for themselves; another may be inclined to provide a lot of services to assist patients in performing basic activities, either because they are concerned about safety or they feel that doing things for patients may be faster in the long run. At its most basic level, treatment can refer simply to gross types; for example, does medical management work better than surgical? It can even be simply a proxy for care given in hospital versus another or by one physician versus others. Measuring the effects of treatment first requires a clear, useful taxonomy for treatments. Surprisingly little work has gone into cre-
34411_CH01_001_022
12
9/9/05
CHAPTER 1
9:21 AM
Page 12
INTRODUCTION
ating such schema. Just as one needs to think not only about formal treatments like prescribed drugs but also about over-the-counter medications, the definition of a therapy may not be limited to what is done in a clinical setting. Informal care may play a substantial role. In some cases, the treatment may extend over several sites. For example, much of the care formerly rendered in hospitals is now provided in nursing homes and even at home. A simple model to classify treatment can be derived from drug therapy, where one talks about such constructs as type, dosage, duration, and timing. A similar approach can be applied to other treatments like surgery. The next level of analysis might ask whether the same treatment in different hands produces different results. At this point, the issue becomes individual skill. Treatment relates directly to what has been termed process of care under the taxomony created by Donabedian (1966), which can be said to be composed of two basic aspects: (1) doing the right/appropriate thing and (2) doing it well. The goal of outcomes research is to establish what treatment is appropriate for a given situation by isolating the effects of treatment from the effects of other factors that influence outcomes. It is harder to use outcomes to address skill compared to appropriateness, but in the end, that is the only real way. Although some may try to “tease out” the skill component by using some sort of direct analysis, such a strategy will not readily distinguish between skill and appropriateness. A more precise approach is first to ascertain what type of care produces the best (or at least acceptable levels of ) results for a given problem (or group of patients). Then, one can apply the same deductive analytic approach to examining those cases where the appropriate care was given to look for differences across providers. Where such differences are found, they can be said to reflect differences in skill.
TYPES OF STUDY DESIGNS There is substantial confusion about the relationship of study design to outcomes research. Many people seem to equate outcomes research with epidemiological study designs and thus wonder what role randomized controlled trials (RCTs) can play in outcomes research. Indeed, RCTs that examine the outcomes of interest to outcomes research are as much outcomes research as studies that use other designs. The emphasis on other study designs in outcomes research reflects the more expansive interventions considered reasonable for study (many of which do not lend themselves to
34411_CH01_001_022
9/9/05
9:21 AM
Page 13
Types of Study Designs
13
RCTs) and the issue with generalizability of RCTs when one is interested in the effect of health care on the broader population of interest. Clinical research worships at the shrine of the RCT. The basic difference between an RCT and well-conducted prospective observational study is the allocation of patients. In an RCT, the allocation is not under the control of either the medical practitioner or the patient. In observational studies, there is always a possibility of selection bias. Patients either may elect certain practitioners or certain types of care or the practitioners may assign care on the basis of differences in clinical status. Indeed, no one in their right mind would assume that care is assigned randomly. The science of medicine depends on matching treatment to need. The real question from the perspective of scientific study is whether some unmeasured factor might be responsible for the choice of treatment. Random assignment obviates that risk. It does not necessarily mean that the experimental and control groups are equivalent. (It is still possible to get differences by chance.) However, it does means that any differences are not systematic; in other words, they do not reflect bias. Those using observational methods are under great pressure to prove the comparability of the treated and untreated groups. Even when all measured variables are examined, there always remains the possibility of some systematic difference of an unmeasured variable. The ability to assign subjects randomly to either experimental or control status confers an aura of science that is unsurpassed.3 Indeed, serious questions of bias arise whenever the decision to treat or not (or how to treat) is determined by some external force. Those reviewing the results of nonrandomized studies need to be reassured that potential risk factors have been identified and addressed. Nonetheless, there remains a concern that the experimental and control groups are not completely comparable; hence, that unknown factors may account for differences found. A number of statistical procedures have been developed to address this issue, but the level of comfort with the results of these efforts varies with the discipline. Clinicians, who are usually not statistically sophisticated, need a lot of reassurance that the experimental and control groups are comparable. In recent years, biostatisticians have promoted propensity scores as a way of providing clinicians with more comfort about well-conducted observational studies (D’Agostino, 1998). In essence, propensity scores identify the variables that might be associated with using or not using a given service. Clinically homogeneous risk subgroups are created on the basis of these measured variables and the results compared across each of these subgroups. Some researchers, especially economists, still worry about unmeasured
34411_CH01_001_022
14
9/9/05
CHAPTER 1
9:21 AM
Page 14
INTRODUCTION
variables and have developed procedures that attempt to adjust for these. One of the most common is the use of instrumental variables (IVs) (Angrist, Imbens, & Rubin, 1996; Lee, 1994). These are variables that are statistically associated with the likelihood of treatment but not with the outcomes. By using these IVs, the researchers can presumably adjust for unmeasured effects. The problem lays in finding IVs that fit the bill. In most cases, it is hard to identify a variable that is associated with getting care but not with its outcomes. The most common IVs are measures of access to care. RCTs may encourage false confidence; they are not a guarantee of good science. Problems with attrition, for example, may create new sources of bias. Standards for the conduct and reporting of RCTs, like CONSORT (Begg et al., 1996), promote better research quality. RCTs have real limitations. In general, randomized trials use great care in design to specify inclusion criteria. Because RCTs are complicated and difficult to mount, they are usually restricted to very tightly targeted groups of patients. Often the investigators are not actively concerned about how the subjects are obtained and rely on random allocation to distribute any differences equally across the two groups. As a result, randomized trials often trade internal validity (tightness of comparisons) for external validity (generalizability). Thus, randomization does not provide the protective shield that some think. Even if the groups are more comparable (and such a distribution is not assured by random assignment), the pertinent analyses may still require looking at the data within subclasses. It does not seem feasible to rely exclusively on RCTs for all, or even most, of the needed empirical data linking outcomes to the process of care. There are those who maintain that nothing but randomized controlled trials can provide real evidence of efficacy. Epidemiological models applied to observational data can never be absolutely sure that differences found were not due to unobserved variations in the two groups. Random allocation is a powerful tool, but both because of other limitations (especially in regard to examining the effectiveness of a treatment; i.e., how it actually works in practice) and simply for reasons of logistics, epidemiological (observational) studies will inevitably play a major role. It is crucial that these latter studies be carefully designed to minimize their limitations (Campbell & Stanley, 1963; Cook & Campbell, 1979). (Chapter 2 provides a more detailed discussion about the alternative approaches.) In effect, both approaches require some level of extrapolation and inference. The RCT requires a heavy set of inferences to extrapolate the results based on extensive participant selection and fixed interventions to clinical
34411_CH01_001_022
9/9/05
9:21 AM
Page 15
Measuring Outcomes
15
practice. The epidemiological approach requires a substantial amount of inference in the analysis itself, but the translation to practice is thus much easier because many of the relevant variables have already been addressed. Because the epidemiological approach is essentially a naturalistic technique that relies on data collected as part of extant practice, questions will arise about the comparability of those who receive different forms of care. The assignment to treatment groups is not based on chance. Factors, both overt and more subtle, determine who gets what care. The burden of proof lies with the investigator. In truth no amount of evidence can absolutely guarantee comparability, but a lot of informational benefit can accrue from using carefully analyzed information derived from real practice. A much more important problem in using clinical information is its quality. Clinical investigators quickly appreciate that clinical data is not recorded systematically or thoroughly. Patient information is entered when patients visit the system. No systematic follow-up is obtained. Much of the information recorded summarizes clinicians’ summary impressions rather than capturing the presence of specific signs and symptoms. Two clinicians may opt to record quite disparate information, even when they use the same headings. Investigators seeking to mount outcomes studies will need to plan these studies to include prospective data collection and incorporate deliberate steps that attend to the quality of information at each stage. Most good observational studies require a prospective design with standardized, systematic data collection on all aspects (i.e., case mix, treatment, and outcomes).
MEASURING OUTCOMES Outcomes come in a variety of sizes and shapes. The selection of an outcome measure should be based on a clear sense of what one wants to measure and why. Outcome measures can be both generic and specific to a given problem. The generic measures are useful for looking at policy issues or reflecting the bottom line effects of care on health status or even aspects of quality of life. They provide a sort of lingua franca that can be used to compare the treatments for various conditions in analyses such as cost effectiveness. Because much medical care can affect specific signs and symptoms but may not have a profound impact on the greater spheres of life, most clinicians are accustomed to looking at the more limited effects of care. These are more closely linked to specific interventions and hence are usually more satisfying to see. Condition-specific outcomes, as the name implies,
34411_CH01_001_022
16
9/9/05
CHAPTER 1
9:21 AM
Page 16
INTRODUCTION
will vary with the condition being treated, although some measures may prove useful for more than one condition. Generic measures address larger constructs and hence their causal links to specific treatment events may be more difficult to trace. The generic measures can include both measures of function in various sectors (e.g., selfcare, social activity, emotional state) as well as satisfaction with the care provided, the way it is provided, and perhaps even the setting in which it is provided. It is not always easy to separate opinions about the quality of care from feelings about the results of treatment. Although someone may feel satisfied that a clinician did his best even if the results are disappointing, it is likely that patients will be more satisfied when the results are favorable. Both generic and condition-specific outcomes measures (as well as the other components of the outcomes equation) often need to be aggregated to create some sort of summary measure. The aggregation process is complex. There is a strong temptation to simply add raw scores to generate a total score, but such a step is foolhardy. In the simplest case, it implies an equal weighting among the components, an assumption that is not automatically true. Even worse, the components may take on different weights because of the way the answers are constructed. For example, a response with five categories may receive a score of 1–5, whereas a dichotomous answer would be 0,1. There is no a priori reason to suspect that a 5 on the first scale is any more important than a 1 on the second. Even when the responses are in some apparent order, a response of 5 is not necessarily five times more than one of 1. Deciding how to weight the components of a summary scale properly can be a serious undertaking. Ordinarily, one needs some construct to use as the basis for “norming” the values placed on each component. Techniques that vary in sophistication and ease of implementation (usually inversely) can be applied to obtaining the value weights of different constituencies. In the outcomes trade, these values are usually referred to as utility weights. Sometimes they are directly related to overt concepts; sometimes they are inferred from observed behaviors. The science of measurement has come a long way. Before an outcomes measure can be said to have attained its pedigree, it must pass a series of tests. Basically, the criteria for a useful measure are that it is reliable (i.e., it will yield the same results consistently); it is valid (i.e., it measures what it says it does); and it is responsive (i.e., it can detect meaningful increments of change) (Guyatt, Deyo, Charlson et al., 1989). Some measures have been extensively studied; others are more novel. Few if any can be used on all occasions. The astute outcomes researcher must weigh the measure’s reputation against its actual content and the
34411_CH01_001_022
9/9/05
9:21 AM
Page 17
Conceptual Modeling
17
application intended. For example, some measures work well with some populations but not with others. They may cover only a limited portion of the full performance spectrum or be better at distinguishing among some aspects of function than others.
CONCEPTUAL MODELING There are five key steps in outcomes research. Although they are performed sequentially, they are not as independent as they might seem. Greater clarification of later steps may entail revising earlier ones. In the end, any presentation must be internally coherent. The individual steps must be shown and they must relate to one another. The five steps are as follows: 1. 2. 3. 4. 5.
Define a researchable question. Develop a conceptual model. Identify the critical dependent and independent variables. Identify appropriate measures for each. Develop an analysis plan.
In most cases, the research question precedes the underlying model, but not necessarily. Asking a researchable question is much harder than simply posing a question. A researchable question must be answerable by direct means. It is not a philosophic proposition. One test of the completeness and directness of the question will come from the conceptual model. Frequently the question will be modified after the model is refined. A critical step in developing an outcomes study is the creation of a conceptual model. This need will be stressed frequently in this book because it is so central to successful outcomes work. In essence, the conceptual model indicates what is believed to cause the outcome. It identifies what are the critical pathways and what other factors are likely to affect these. It should identify which variables, chosen to represent the various components of the basic outcomes equation described earlier, are pertinent to the study at hand. The variables themselves and their relationship both to the outcomes of interest and to each other should be specified. The process of creating a conceptual model is itself iterative. One starts from a set of premises based on theory and/or clinical insights. As one fleshes out the model and becomes ever more specific about just what is involved, one can begin to consider how to operationalize this model. This operationalization may necessitate revisiting the model.
34411_CH01_001_022
18
9/9/05
CHAPTER 1
9:21 AM
Page 18
INTRODUCTION
A conceptual model is not necessarily the same as a theoretical model. No disciplinary theory needs to drive the model. Instead, it should explicate clearly what process the investigator believes is occurring—or at least what elements need to be controlled in the analysis. Such a model can be based on clinical experience as well as a review of prior work. Working the model through provides a way to think about what factors are most important. Figure 1–1 offers a simple illustration of a conceptual model for looking at the outcomes of congestive heart failure. The items in the boxes are operationalized aspects of the basic elements that are addressed in the outcomes equation described earlier. The arrows indicate an expected effect. In this model, the effects of treatment are expected to interact with the clinical factors to produce outcomes. Once these elements have been identified, they can be operationalized. Each one can be captured in one or more measures. The delineation of the model and the specification of variables represent two of the major components of a research design. A familiar quote in outcomes research is that what cannot be measured does not exist. In one sense, the concept is attractive. One needs to be able to reduce complex attributes to measurable representations in order to study them and to compare their presence across programs. However, one must
Clinical factors • cardiac output • severity • duration • etiology • comorbidity • prior status
Patient factors • age • gender • occupation
Treatment • specific medications • diet • exercise • case management
Outcomes • cardiac output • symptoms • function • complications • quality of life • employment/work loss
Figure 1–1 Conceptual Model of Treatment and Outcomes in Congestive Heart Failure
34411_CH01_001_022
9/9/05
9:21 AM
Page 19
Conceptual Modeling
19
approach measurement with respect. Measurement involves distortion; it is by nature a process of abstraction and something is inevitably lost in the process. Likewise, the commitment to measurement should not be construed as endorsing the idea that everything that can be measured is useful. Perhaps one of the most memorable misuses of measurement was the theory behind the conduct of the Vietnam War. Body counts and arbitrary definitions of successful missions do not necessarily lead to a successful conclusion. Quantitative analysis works best when it serves conceptual thinking, not as a substitute for it. Clinical intuition and insight are valuable gifts, which should not be discarded or devalued in the face of quantitative science. In his autobiography, Colin Powell describes an intelligence unit in Vietnam that received endless amounts of data on the enemy’s shelling patterns. All this information was entered into a computer regression model that eventually produced the result that shelling was heavier on moonless nights, an observation that any combat veteran could have provided (Powell, 1995). Outcomes research shares some of these problems. On the one hand, if its findings do not agree with clinical wisdom, they are distrusted. On the other hand, if they support such beliefs, they are extraneous. Life is generally too complicated to attempt outcomes analysis without some sort of framework. Some analysts may believe that the data will speak for itself, but most appreciate the value of a frame of reference. Even more important, with so much information waiting to be collected, one needs some basis for even deciding where to look for the most powerful answers. Using outcomes wisely requires having a good feel for what question is being asked and what factors are likely to influence the answer. Outcomes research is largely still a clinical undertaking, although it has become sophisticated. At its heart is a clinical model of causation. Before an outcomes study can be planned, the investigator needs to develop a clear model of the factors that are believed to be most salient and their relationship to the outcomes of interest. Some factors will play a direct role; others may influence events more indirectly. Each needs to be captured and its role defined. This model forms the basis of the analysis plan. The third key ingredient is the analysis plan.4 The conceptual model provides a general framework for the analysis, but the specifics depend on several factors, primarily the nature of the variables. Most analyses, especially those that rely on an epidemiological approach, have to be multivariate. One or another variation of regression models is likely to be employed. Although multivariate modeling can take into account the effects of inter-
34411_CH01_001_022
20
9/9/05
CHAPTER 1
9:21 AM
Page 20
INTRODUCTION
vening variables, nonrandom assignment invariably raises questions about the comparability of treatment and control groups. Even groups that seem extremely comparable on the basis of variables examined may vary widely along some other parameter. Some researchers have proposed statistical models to deal with this so-called selection bias. Special models are developed to identify and deal with the correlated error associated with such a bias. These corrections use factors that are common to both the equation that describes the factors associated with care use and that looks directly at outcomes; at least one variable should apply to the former, but not the latter. Interpreting the results of regression equations can be complicated. Fundamentally, the major question is whether the independent variable of greatest interest (usually treatment) is significantly related to the dependent variable (i.e., the outcome) after the effects of other factors has been considered. This relationship can be examined in two ways: (1) the extent to which a change in the risk factor affects the dependent variable (e.g., the odds ratio) and (2) the capacity of the full equation to explain the variance in the model. It is quite feasible for a variable to be significantly related to the dependent variable in an equation that explains very little of the overall variance. Conversely, explaining the variance does not examine the relationship between the independent variables and the dependent variable. In epidemiological terms, the size and strength of a coefficient from the regression equation reflect the power of the relationship, whereas the amount of variance explained describes the power of the overall model. It is possible to have a significant relationship among variables and still not explain much of the total variance in the distribution of the dependent variable. Because outcomes may be influenced by many things—not all of them measurable— many outcome equations do not explain large amounts of the variance, although the adjusted relationship between variables of interest may be very significant. Being able to establish a clear relationship between a treatment and its purported effects is important even when that relationship does not account for all, or even most, of the effect. A clear understanding of how a treatment influences outcomes for defined subgroups of patients lays the foundation for meaningful guidelines about what constitutes appropriate care.
SUMMARY: ORGANIZATION OF THE BOOK The next three chapters in this introductory section address overarching design issues; two address study design issues and one is on measurement principles. This section of the book (Chapters 2–4) is organized to discuss
34411_CH01_001_022
9/9/05
9:21 AM
Page 21
References
21
the implications of the basic outcomes model. Each component is discussed at some length to identify the issues that must be considered and to suggest some measures that may prove useful (along with caveats about using them). Chapters 5–7 address outcomes measures including generic measures, condition-specific measures, and satisfaction. Chapters 8 and 9 cover the major components of risk adjustment, including severity of illness, comorbidity, and demographic and psychosocial characteristics. Chapter 10 addresses cost effectiveness, a growing area of related interest in outcomes research. Chapter 11 discusses treatment and proposes a taxonomy for this central component. The final two chapters (12 and 13) address some overarching issues in conducting outcomes research. Chapter 12 provides some practical issues in implementing research studies in a clinical setting. The last chapter then offers some final thoughts for those who are anxious to launch into outcomes studies. Although these observations are intended primarily for neophytes, the authors hope that even more experienced outcomes researchers may gain some useful insights from them.
REFERENCES Angrist, J.D., Imbens, G.W., & Rubin, D.B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–472. Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., et al. (1996). Improving the quality of reporting of randomized controlled trials: The CONSORT statement. Journal of the American Medical Association, 276, 637–649. Berwick, D.M. (1989). Continuous improvement as an ideal in health care. New England Journal of Medicine, 320(1), 53–56. Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally. Chassin, M.R., Kosecoff, J., Park, R.E., Winslow, C.M., Kahn, K.L., Merrick, N.J., et al. (1987). Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. Journal of the American Medical Association, 258, 2533–2537. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. D’Agostino, R.H., Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17, 2265–2281. Donabedian, A. (1966). Evaluating the quality of medical care. Milbank Memorial Fund Quarterly, XLIV(3), 166–206. Field, M., & Lohr, K. (Eds.). (1992). Guidelines for clinical practice: From development to use. Washington, DC: Institute of Medicine, National Academy Press.
34411_CH01_001_022
22
9/9/05
CHAPTER 1
9:21 AM
Page 22
INTRODUCTION
Guyatt, G.H., Deyo, R.A., Charlson, M., Levine, M.N., & Mitchell, A. (1989). Responsiveness and validity in health status measurement: A clarification. Journal of Clinical Epidemiology, 42(5), 403–408. Kohn, L.T., Corrigan, J.M., & Donaldson, M.S. (Eds.). (2000). To err is human: Building a safer health system. Washington, DC: National Academy Press. Leape, L.L., Park, R.E., Solomon, D.H., Chassin, M.R., Kosecoff, J., & Brook, R.H. (1990). Does inappropriate use explain small-area variations in the use of health care services. Journal of the American Medical Association, 263, 669–672. Lee, L.F. (1994). Semiparametric instrumental variable estimation of simultaneous equation sample selection models. Journal of Econometrics, 63(2), 341–388. Powell, C.L. (1995). My American journey/Colin L. Powell with Joseph E. Persico (1st ed.). New York: Random House. Sackett, D.L. (1997). Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone. Weed, L.L. (1968a). Medical records that guide and teach. New England Journal of Medicine, 278(11), 593–600. Weed, L.L. (1968b). Medical records that guide and teach. New England Journal of Medicine, 278(12), 652–657 (concl.). Wennberg, J., & Gittelsohn, A. (1982). Variations in medical care among small areas. Scientific American, 246, 120–135. Wennberg, J.E., Freeman, J.L., Shelton, R.M., & Bubolz, T.A. (1989). Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. New England Journal of Medicine, 321, 1168–1173.
NOTES 1. Terminology varies a great deal with respect to the use of the term “risk factors.” Some people use it interchangeably with disease severity. Others use it more generically to refer to the whole set of factors that can influence the outcomes of care (even including treatment). In this book, we have tried to use it consistently to refer to those factors besides treatment that can affect outcomes. 2. It would be possible to deal with clusters of diagnoses, but the numbers of combinations could quickly become unmanageable. 3. Random assignment does not confer an absolute protection against bias. It simply reduces the likelihood that such bias has occurred. It is still important to examine the characteristics of the experimental and control groups to look for such bias and to consider the value of subgroup analysis where the effects of treatment may be greater with one portion of the sample than another. 4. This book does not attempt to discuss the intricacies of the analytic methods for nonexperimental studies. Investigators should consult with a methodologist and/or statistician before any outcomes analysis is undertaken.
34411_CH02_023_058
9/9/05
9:35 AM
Page 23
2 Designing an Outcomes Research Study David M. Radosevich
INTRODUCTION: TYPES OF STUDY DESIGNS A health outcomes research study design is a plan for executing the study. At a minimum, the design depicts the groups studied; for example, treatment and control group, instances of the treatment and the timing, and frequency of health outcomes measures. The design provides a high-level overview of the health outcomes study and insights into the plan for analysis. Finally, the design should specify whether the individuals studied are randomly assigned to either receive the treatment of interest or no treatment, also referred to as a control group. Control over treatment assignment through randomization is the basis for distinguishing two types of outcomes studies: experiments and quasiexperiments. Random assignment of subjects is central to controlling for extraneous differences between groups, but it does not guarantee comparability; it simply asserts that any differences are due to chance. Without randomization of study participants, the outcomes researcher runs the risk of individuals with particular characteristics having a higher probability of being included in the study or one of the study groups. These differences can arise from patient self-selection or from clinician decisions about who should get treatment. Selection bias or self-selection has the potential to confound the treatment–outcome relationship, thereby biasing results. Some of these differences can be measured and controlled for in the analysis, but others may remain unmeasured and uncorrected. Overall, selection bias may be the greatest threat to the validity of health outcomes research studies. 23
34411_CH02_023_058
24
9/9/05
CHAPTER 2
9:35 AM
Page 24
DESIGNING AN OUTCOMES RESEARCH STUDY
Although the randomized controlled trial (RCT) is considered the “gold standard” for clinical research, most outcomes studies are conducted as quasiexperiments, which lack control over the assignment of participants to receipt of treatment. As a consequence, the outcomes researcher is faced with controlling for self-selection and underlying differences between treatment and no treatment groups by other means such as the timing of the outcome measurement (relationship to randomization? can be done in observational studies) or statistical adjustment. Many treatments cannot be practically investigated using an experimental design. In outcomes studies conducted in health plans, fairness is a frequently voiced concern regarding allocating individuals on the basis of randomization (Disease Management Association of America Outcomes Consolidation Steering Committee, 2004). Consequently, the quasi-experimental design, also called the observational study (Kelsey, Whittlemore, Evans, & Thompson, 1996), serves as the backbone of health outcomes research.
Self-Criticism in the Design Process There is no perfect health outcomes research study. Every investigator must weigh trade-offs between internally valid designs, like the RCTs, and quasi-experiments where inferences could be erroneous because of an inability to randomly assign treatments to study participants. Designing an outcomes research study requires a process of self-criticism and self-evaluation. This is accomplished by raising questions concerning the validity of the study design or the accuracy of inferences drawn. In this iterative process of self-criticism, the outcomes researcher comes to recognize the imperfection of the study design and its strengths and limitations and identifies strategies for strengthening the overall design. In truth, validity encompasses all the grey areas of research and is always context specific. It is mistaken to interpret the validity of study designs as simply good or bad. The goal of a health outcomes research study is to achieve the best approximation of the truth of the treatment–outcomes relationship. Does a given treatment cause a particular outcome? Understanding and evaluating the threats to the validity of human inferences about the results of an outcomes study are critical to success. This involves addressing four study design questions. The remainder of this chapter discusses the implications of these questions and the common threats to the validity of health outcomes studies.
34411_CH02_023_058
9/9/05
9:35 AM
Page 25
Evaluating the Threats to Outcomes Research
25
EVALUATING THE THREATS TO OUTCOMES RESEARCH Validity concerns the truth or falsity of propositions about cause (Cook & Campbell, 1979). Although a discussion of the multiple threats to study designs is outside the scope of this chapter, a selected few, which are frequently encountered in outcomes research study designs, need to be considered in planning and implementation. They are listed in Table 2–1. For a complete discussion of validity and study designs, the reader is referred to the texts by Campbell and Stanley (1963); Cook and Campbell (1979); and Shadish, Cook, and Campbell (2002). For a more humorous treatment of validity, also referred to as bias, see the papers by David Sackett (1979) and Alvin Feinstein (Feinstein, Sosin, & Wells, 1985).
Table 2–1 Adaptation of Cook and Campbell’s Scheme (1979) for Classifying Threats to the Validity of Health Outcomes Research Internal Validity • Statistical Conclusion ° ° ° ° °
Low statistical power Fishing and error rate problems Violated assumptions and inappropriate statistical tests Reliability of measures Inconsistent implementation of the intervention
• Internal Validity ° ° ° ° °
Selection Regression to the mean Attrition Missing data History
External Threats • Construct Validity ° Inadequate conceptual design ° Mono-operation and monomethod biases ° Treatment diffusion • External Validity ° Person ° Setting ° Time
34411_CH02_023_058
26
9/9/05
CHAPTER 2
9:35 AM
Page 26
DESIGNING AN OUTCOMES RESEARCH STUDY
Internal Versus External Validity Under Cook and Campbell’s scheme, threats to validity can be classified as either internal or external. This distinction neatly divides threats into those that concern the validity of conclusions drawn about the relationship between the treatment and the outcome and whether the results are externally applicable to other persons, places, and time. Internal validity is the minimum necessary to interpret an outcomes study. All outcomes studies need to be internally valid; that is, the study design avoids errors that could compromise conclusions. For example, the researcher wants to avoid drawing spurious conclusions regarding results because the subjects in the groups being compared are not comparable. Issues around external validity concern the generalizability or representativeness of study results. Can the results of an outcomes study be applied across different populations of persons, in different settings, and in other periods of time? Generalizability questions usually can be traced to the methods of recruitment of study subjects. RCTs have been criticized for their lack of generalizability, because study conclusions are limited to the population being studied. Recruitment may employ strict inclusion and exclusion criteria for enrollment; therefore, individuals recruited bear little resemblance to individuals seeking health care in the “real world.” Many RCTs rely on volunteers, who themselves are highly self-selected. In contrast with RCTs, quasi-experiments have the potential for being more representative. Even RCTs can encounter selection bias when the rate of follow-up is poor or even worse when it is different in treatment and experimental groups. The standard way to handle such loss is through a process known as intention to treat (ITT). Basically, the last observation is carried forward as the final observation for that subject. Thus, someone who leaves treatment early is retained at the state when they were last observed. This approach is generally conservative for treatments designed to improve the situation, but it can have the opposite effect if the treatment is simply designed to slow the rate of deterioration. Thus, it must be employed thoughtfully. A second aspect of external validity concerns the validity of inferences drawn about higher order constructs or traits that cannot be directly observed. Can one generalize from the operational definitions used in the study to abstract constructs? From this perspective, external validity concerns the measurements concepts, the interrelationship of the concepts with one another, and integrity of the treatment investigated. This form of
34411_CH02_023_058
9/9/05
9:35 AM
Page 27
Evaluating the Threats to Outcomes Research
27
validity is referred to as construct validity. There is a theoretical basis for construct validity in two approaches to constructing outcomes measures: latent-trait theory and factor analysis. According to latent-trait theory, the individual’s responses to an item on an outcomes measure depend on the level of the attribute present (Streiner & Norman, 1995). Factor analysis, on the other hand, attempts to represent a set of variables as a smaller number of constructs (Kim & Mueller, 1978). Both latent-trait analysis and factor analysis are useful techniques for confirming construct validity.
Four Study Design Questions The process for evaluating study designs was best articulated in educational psychology by Campbell and Stanley (1963). Their disciples built on this early work, expanding it to include applications in health services research, epidemiology, and clinical research. More recent work (Cook & Campbell, 1979; Shadish et al., 2002) stressed the importance of four critical questions in the design of scientific experiments. These questions, which reflect four major threats to the validity of outcomes study designs, have been restated to make them relevant to outcomes research. 1. 2. 3. 4.
Is there a relationship between the treatment and outcome? Is the observed relationship between treatment and outcome causal? What concepts explain the treatment outcome relationship? How representative is the treatment and outcome relationship across persons, settings, and times?
Each question relates to a form of validity: statistical conclusion; internal, construct, and external validity, respectively. The process of designing a health outcomes research study involves its critique and redesign. Epidemiologists describe the threats to validity as biases or systematic errors in the design or implementation of a study (Szklo & Nieto, 2000). In addition to confounding, biases are categorized according to type: selection or information. Selection bias was mentioned earlier. Information biases are unique to epidemiological studies and concern misclassification of the treatment (or exposure), the outcome, and imprecise definition of study variables. Two types of misclassification errors are recognized in clinical studies. First, one could falsely conclude that the individual received the treatment of interest or has the outcome of interest.
34411_CH02_023_058
9/9/05
CHAPTER 2
28
9:35 AM
Page 28
DESIGNING AN OUTCOMES RESEARCH STUDY
These are referred to as false positives. The second type of misclassification error is one in which the individual did receive the treatment or the outcome of interest, but it was not noted. These latter are referred to as false negatives. Together, these errors define the accuracy of treatment and outcomes measures, thereby defining the sensitivity and specificity of measures (Szklo & Nieto, 2000). In applying the internal and external validity scheme discussed earlier, information biases bridge both forms of validity. Epidemiologists have recognized that misclassification significantly biases the results of observational studies. For a full appreciation of how these biases impact outcomes studies, several sources offer a fundamental review of epidemiological methods (Kelsey et al., 1996; Szklo & Nieto, 2000). What follows is a brief review of threats to validity that are important in health outcomes research.
STATISTICAL CONCLUSION VALIDITY Question 1: Is there a relationship between the treatment and the outcome? This statistical question concerns whether the treatment and the outcome covary and the strength of association between them. Five threats to statistical validity are commonly observed in outcomes research study designs. These include low statistical power, fishing and error rate problems, violated assumptions of statistical tests, reliability of the outcome measures, and inconsistent implementation of the intervention. Low Statistical Power All too frequently, the first question asked by researchers is: How many subjects do I need for my study? This question is always premature before planning the study and preparing an analysis strategy. Planning for statistical power begins by addressing the following questions: • • • • •
What is the research question? Who is the target population? How will study subjects be recruited? How large an effect is expected? How much variation is anticipated?
34411_CH02_023_058
9/9/05
9:35 AM
Page 29
Statistical Conclusion Validity
29
• What analysis is needed to answer the question? • It is feasible to study the question? All health outcomes studies need to be designed to detect differences between persons receiving the treatment and those not receiving the treatment. The goal is to detect a true effect. Formally stated, the primary concern of statistical power is the likelihood of detecting the truth about the treatment-outcome relationship. The determinants of sample size can be best understood in the context of hypothesis testing. For example, in a study to investigate the difference between the risk of occurrence of adverse outcomes between a “new” medical treatment e and usual care signified by c, one sets up a hypothetical test scenario as follows. Null Hypothesis:
Pe Pc
Alternative Hypothesis:
Pe Pc
where Pe represents the probability of the event among experimental subjects and Pc the probability of the event among controls. Statistics test the likelihood that an observed difference occurred by chance. If a study is designed to test for differences in the adverse event rates between the “new” treatment and usual care, the determinant of the sample size is statistical significance level, also called the type I error rate (); it reflects the likelihood that one sees a difference that could simply have occurred by chance. This is equivalent to the risk of drawing a false conclusion that there is a difference between Pe and Pc. By contrast, a type II error claims no difference when in fact one exists. Statistical power, or one minus the type II error (), is the probability of discovering a true difference between Pe and Pc. Next, the size of the difference considered important is addressed. The latter is defined in terms of the effect size, a standardized difference, which reflects how large a difference one wants to be able to demonstrate statistically. Finally, one considers the number of subjects or the number of groups necessary. Examining the interrelationship of the type I error rate, the type II error rate, and the magnitude of the effect being sought is referred to as statistical power analysis. Many factors under the direct control of the outcomes researcher directly affect the statistical power of outcomes studies. Figure 2–1 shows the impact of various threats to validity on sample size. In the center of the figure, there is general function for estimating sample size (Friedman,
34411_CH02_023_058
30
9/9/05
CHAPTER 2
9:35 AM
Page 30
DESIGNING AN OUTCOMES RESEARCH STUDY
Competing Risk
Lost to Follow-up
Health Outcome
Lack of Standardization
2 x Variability x [Constant (␣, )]2 Delta2 Compensatory Rivalry and Equalization
Lag Non-compliance
Diffusion
Figure 2–1 Threats to Validity and How They Impact the Statistical Power of a Health Outcomes Study
Furberg, & DeMets, 1996). Delays in implementing a treatment intervention (lag), individuals not taking prescribed study medications (noncompliance), and treatments spilling over to individuals assigned to the control group (diffusion, compensatory rivalry, and equalization) all increase the number of individuals needed to detect differences between treatment and control groups. In the numerator, poor standardization of the study, individuals being lost to follow-up (attrition), persons dying from causes other than target condition (competing risk), and selecting a health outcome measure with poor responsiveness characteristics inflate sample size. Low statistical power is a recurring threat to the validity of health outcomes research study designs. The best advice is to seek the counsel of an expert, preferably someone who will conduct the analysis. Planning and implementing a health outcome study are collaborative endeavor. Because statistical power is critical to designing and planning an outcomes study, statistical power should always be specified in advance through an a priori power analysis. Using the results from published studies and knowledge regarding the outcomes measure, it is possible to make an “educated guess” regarding the likely size of the effect of the intervention. This is quantified in terms of an effect size or detectable difference. It
34411_CH02_023_058
9/9/05
9:35 AM
Page 31
Statistical Conclusion Validity
31
may be asked another way: How big a difference is needed to convince the target audience that the treatment effect is meaningful? Although this enhances the efficiency of study designs and eliminates frivolous outcomes studies, most statistical power analysis is done at the end of the study. This post hoc power analysis is only justified under circumstances in which the investigator lacks advanced knowledge regarding the size of treatment differences. Post hoc power analysis should always be done where no statistically significant differences were found in the analysis to be sure a real difference has not been overlooked. The sample size needed to support a claim of no difference is usually much larger than that needed to show a difference. The following are some guidelines for statistical power analysis. • Consult with an expert; remember that statistical power analysis and estimating sample size are collaborative endeavors. • Specify sample size in advance. • Set standards for considering statistical power. 1. 2. 3. 4.
Type I error less than 5% Type II error less than 20% Lowest common denominator for comparison Plan for available data; response rates, eligibility, missing data
• Be guided by research objectives; consider monetary and indirect costs. • Parameters on which sample size is based should be evaluated as part of interim monitoring. • Be conservative in estimates of statistical power. Design a study for the smallest subgroup analysis planned and the available data. Guided by the study objectives, always consider the monetary and nonmonetary costs of a study and be conservative in estimates. It is generally a good idea to continuously evaluate the study assumptions as the research unfolds. Most analysts perform statistical power analysis using any one or a combination of three resources. Formulas for the direct calculation of statistical power can be found in a number of different sources (Friedman et al., 1996; Murray, 1998; Schlesselman, 1982). In general, most formulas incorporate
34411_CH02_023_058
32
9/9/05
CHAPTER 2
9:35 AM
Page 32
DESIGNING AN OUTCOMES RESEARCH STUDY
measures of the type I error, type II error, and the effect size. The arithmetic expression in the center of Figure 2–1 provides a conceptual depiction of the interrelationship of these elements. Tables for frequently used statistical tests can be found in resources such as Cohen (1988). Finally, shareware over the Internet and commercially available software, such as nQuery Advisor (Elashoff et al., 2000), are available. Statistical power analysis calculations are more involved when it comes to multivariate analysis. Because these frequently involve complex computations and multiple design parameters, these are best left to skilled biostatisticians, epidemiologists, and health services researchers.
Fishing and Error Rate Problems Most intuitively recognize that if a researcher conducted a hundred tests of statistical significance, 5 percent would be statistically significant (at the 5 percent level) by chance. Yet frequently outcomes studies are designed to make multiple comparisons and ignore chance in interpreting the statistical tests. These are collectively referred to as “fishing” and “error rate” problems. The inflation of type I errors is particularly troublesome in outcomes studies, especially in studies using multiple outcomes and multidimensional scales. This threat to validity arises when investigators fail to specify their end points in advance of conducting their study or the primary outcomes are ill defined. In the absence of specifying primary end points, the investigator incorporates multiple outcomes measures in the study. When analyzing the results, each of the outcomes is treated as having primary importance to answering the study question, thereby converting the study to one that is more exploratory than hypothesis driven. A second threat involves the use of multidimensional measures; for example, a new treatment is hypothesized to improve the quality of life of participants. The investigator chooses a multidimensional scale to measure quality of life, such as the Medical Outcomes Study 36-Item Short Form Health Survey (SF-36) (Ware, 1991; Ware & Sherbourne, 1992; Ware, Snow, Kosinski, & Gandek, 1993) or the Sickness Impact Profile (Bergner, 1989; Bergner, Bobbitt, Carter, & Gilson, 1981). Without specifying a specific subscale from these measures, the investigator increases the likelihood of type I errors by treating all the subscales as equally important to confirming the hypothesis.
34411_CH02_023_058
9/9/05
9:35 AM
Page 33
Statistical Conclusion Validity
33
Various methods have been devised to adjust for an inflated type I error rate (Rosner, 1995; Shadish et al., 2002): 1. 2. 3. 4.
The LSD approach Bonferroni correction Tukey’s method Scheffé’s method
A thorough discussion of these techniques can be found in most biostatistics (Rosner, 1995) and epidemiologic (Last, 2001) texts. In general, the approaches adjust the type I error rate downward, making it more difficult to reject the null hypothesis (no difference) and thereby reducing spurious associations. Strategies for minimizing error rate problems in health outcomes research studies include the following: • • • •
Recognize the problem of making multiple comparisons. Establish a priori the primary outcomes for the study. Incorporate greater specificity in outcomes measures. Make adjustments for multiple comparisons by selecting one of the accepted statistical techniques, such as Tukey or Scheffé.
Violated Assumptions of Statistical Tests This threat to statistical conclusion validity involves selecting an inappropriate statistical test to answer the study question and violating the assumptions for the statistical tests being used. Although a discussion of the full range of statistical tests applicable to health outcomes research is beyond the scope of this chapter, some general guidelines should be kept in mind. The nature of the variables affects what statistical tests should be used. Tests differ for categorical or continuous variables and for nominal, ordinal, interval, or ratio scales. A variety of techniques could be used for analyzing outcomes. For categorical outcomes, such as death, morbidity, and hospitalization, the rigorous assumptions of normally distributed errors can be relaxed. In some instances, categorical data may be desirable because it allows the researcher to contrast elements that are critical to understanding the effects
34411_CH02_023_058
34
9/9/05
CHAPTER 2
9:35 AM
Page 34
DESIGNING AN OUTCOMES RESEARCH STUDY
of a treatment. For example, whereas a linear model that uses mean age may not get at the effects of age 85+, a model that compares those 85+ to those younger might address the issue more directly. In the biomedical literature, logistic regression is widely used to analyze categorical outcomes, such as death or morbidity (Allison, 2001; Hosmer & Lemeshow, 2002; Kleinbaum & Klein, 2002; Le, 1998). Based on a logit function, the technique can be simultaneously used to adjust for covariates and expanded to ordered categorical end points called ordered logit. Multivariate logistic regression is frequently used for analyzing categorical outcomes data in biomedical studies. Logistic regression yields the odds ratio as the measure of association between the treatment of interest and the outcome (Szklo & Nieto, 2000). A variant, multinomial logit can be used when there is more than one outcome of interest. Analysis of categorical outcomes can be further strengthened if the investigator knows about the timing of occurrence of the outcome. In this case, time-to-event analysis, also called survival analysis, has been widely used (Allison, 1995; Hosmer & Lemeshow, 1999; Kleinbaum, 1996; Le, 1997). Survival analysis is a powerful technique to analyze time-to-event data. In general, survival analysis improves statistical power for analyzing categorical outcomes by using the time to occurrence of an event to weight the importance of that event. However, this may be easier said than done. The timing of some outcome events may be hard to determine precisely or impossible to obtain. An outcomes investigator might procure death records to determine the date of death for a subject or administrative data to ascertain the date of hospitalization, but the onset of an acute myocardial infarction (MI) can be clouded by an uncertain history of previous MIs, misdiagnosis, and a failure to seek medical care needed to record the event. Finally, one rarely knows the specific date of onset of some conditions such as disability or chronic diseases such as diabetes mellitus. A second analytic issue is best illustrated by the use of general linear regression in the analysis of continuous outcomes such as health status scale scores, blood pressures, and laboratory values. Replicate outcomes, such as baseline and follow-up health status measures, have the potential to be correlated. The correlated nature of these repeated measures makes it unjustified to use traditional fixed-effects models. Using general linear regression fails to account for the correlated nature of the outcomes measure, thereby artificially increasing the type I error rate for the statistical test (Murray, 1998). In recent years, mixed-model methods have become widely used to handle correlated data (Liang & Zeger, 1993; Littell, Milliken, Stroup, &
34411_CH02_023_058
9/9/05
9:35 AM
Page 35
Statistical Conclusion Validity
35
Wolfinger, 1996; Murray, 1998). This analytic method has two applications in outcomes studies. Mixed-model methods are used in the analysis of data from studies with repeated outcomes measures. This is equivalent to repeated measures analysis and is the basic design for the pretest/posttest study. Outcome study participants are repeatedly measured (e.g., serial blood pressure readings), completing a health status survey at regular intervals. This approach is important in outcomes studies because replicate measures are highly correlated. General linear regression fails to take this into account. Mixed-model methods are appropriate when the units of assignment or sampling include factors other than the individual, a frequently encountered problem in health services, and public health research. Public health services are frequently delivered at the level of the community. Individuals within communities are more similar than individuals outside the community. The community as a source of variability, called a random effect, is nicely handled using mixed-model methods. By extension, mixed effects can be hospitals, schools, clinics, health plan, or any other type of grouping. These mixed-model methods can be applied to categorical or continuous outcomes. Some sources of random error beyond the individual participant occur within study designs that draw participants from other units of interest, such as hospitals, clinics, health plans, and communities. These are often referred to as hierarchical or nested outcomes designs in which the treatment or intervention may be influenced by the level of the group—hospital, clinics, and so on. These designs lend themselves to mixed-model methods because of the correlated nature of their data within the level of the group (Murray, 1998). These are called mixed models because there are two or more random effects: random effects at the level of the participant, and random effects at the level of the group. Table 2–2 summarizes the appropriate types of analytic approaches for different combinations of distributions, random effects, and data types. Table 2–3 gives recommended guidelines for analyzing outcomes study data.
Reliability of Outcomes Measures The failure to reliably measure either the treatment or the outcome could result in a misclassification of the treatment status, the outcome, or both. The reliability of a measure imposes an upper bound on that measure’s ability to discriminate. Unreliable measures can attenuate the relationship between the treatment status and the outcome, mask a relationship, or create a spurious relationship. This underscores the importance of the routine
34411_CH02_023_058
36
9/9/05
CHAPTER 2
9:35 AM
Page 36
DESIGNING AN OUTCOMES RESEARCH STUDY
Table 2–2 Classification Scheme for Statistical Approaches Useful in Analyzing Health Outcomes Data Distribution Design Characteristics
Normal Distribution
Nonnormal Distribution
One Random Effect
General Linear Model— Ordinary Least Squares Linear Regression
Generalized Linear Model—Logistic Regression
Two or More Random Effects/Replicate Outcomes Measures
General Linear Mixed Model
Generalized Linear Mixed Model—Nonlinear Mixed Models
Time-to-Event
Survival Analysis—Kaplan-Meier Life Table Methods and Cox Proportional Hazards Regression
measurement of reliability of study measures and implementing corrective steps to increase reliability. The following actions can be taken to improve reliability (Shadish et al., 2002): increase the number of measurements (i.e., increasing the number of raters or items) and choose better quality measures (i.e., better raters and better training of raters; quality items). Inconsistent Implementation of the Intervention One of the more serious threats to the validity of outcomes studies in field settings is the inconsistent implementation of the intervention. Treatment implementation is notoriously unreliable in observational studies. In natural settings, the investigator rarely has control over the treatment implementation. In community settings, treatments frequently lack standardization and are often idiosyncratic to the settings in which they occur. Epidemiologists have long recognized that treatments implemented inconsistently can lead to spurious results (Szklo & Nieto, 2000). If the implementation of the intervention lacks standardization, results are more likely to suggest that there is no treatment effect. Although this is classified as a statistical threat, remedies focus on tight quality control of the treatment implementation and careful monitoring of the implementation. Systematic training of study subjects and staff involved in treatment implementation is critical. This involves the use of implementation manuals, the development and implementation of programs for training staff and subjects, and continuous reinforcement of expectations in order to improve adherence.
34411_CH02_023_058
9/9/05
9:35 AM
Page 37
Other Threats to Internal Validity
37
Table 2–3 Recommended Guidelines for Analyzing Health Outcomes Study Data 1. What is the nature of the study outcome? a. Categorical outcomes can be either nominal, dichotomous, or ordered categorical. b. Continuous variables are in a raw form or require a transformation, e.g., cost data is highly skewed and should log transformed. c. Time-to-event analysis. 2. Is the data highly correlated? Aside from random errors within the subject, are there different sources of random error? a. Correlated outcomes data and data with multiple sources of error are best handled using some form of mixed-model method. Linear mixed- and nonlinear mixed-model methods are useful along with generalized estimating equation approaches. b. Noncorrelated data might use a general linear model or logistic regression. 3. Adopt generally acceptable standards for statistical power. Type I error less than 5% Type II error less than 20% Lowest common denominator for comparison Plan for available data: response rates, eligibility, missing data
In general, it is easier to measure outcomes than it is to measure treatments. Outcomes studies should measure treatment. This is accomplished by monitoring whether a standard treatment was delivered, received by the subject, and adhered to. The processes of delivery, receipt, and adherence should be incorporated into all outcomes studies. If the researcher has measured the treatment, it is possible to compare the outcomes for those receiving varying levels of the treatment. However, it is possible that subjects self-select levels of treatment. Hence, this is generally viewed as weak evidence for an outcomes effect. It is better to use this data to supplement the preferred analytic method, intent-to-treat (Shadish et al., 2002).
OTHER THREATS TO INTERNAL VALIDITY Question 2: Does the observed relationship likely reflect causation from the treatment to the outcome or might this same relationship reflect the effects of other factors? This distinction concerns the validity of inferences drawn about the observed relationship. This concern falls into
34411_CH02_023_058
38
9/9/05
CHAPTER 2
9:35 AM
Page 38
DESIGNING AN OUTCOMES RESEARCH STUDY
what has previously been described as threats to internal validity. Shadish, Cook, and Campbell (2002) describe nine threats to internal validity that might bias inferences drawn about the treatment outcome relationship. Five of these threats hold particular relevance to health outcomes research study designs: selection, regression to the mean, attrition, missing data, and history.
Selection Selection is the most serious internal validity threat in health outcomes research studies. As mentioned earlier, selection occurs because at the beginning of the study, on average, individuals in the treatment group differ from those in the nontreatment group in both known and unmeasurable ways. This difference frequently occurs because the treatment cannot be randomly assigned to individuals. Selection is a major problem in casecontrol studies (Schlesselman, 1982) in which the investigator has difficulty finding a comparable control group for cases that are of study interest. If the study involves hospitalized patients, exposure and risk increase the likelihood of hospitalization, leading to a higher rate of exposure among hospitalized cases than hospitalized controls. The observed distortion in the relationship is referred to as Berkson’s Bias (Berkson, 1946; Last, 2001). The treatment–outcome relationship could be confounded by differences between the treatment and control groups. For example in epidemiological studies of obesity and all-cause mortality, the relationship is confounded by cigarette smoking. Smokers often have a leaner body mass but are at increased risk of sudden death, cardiovascular disease, and cancer from cigarette smoking. One approach to deal with the problems of selection is the use of propensity scores (Rosenbaum, 2002). Logistic regression is used to predict membership in either the treatment or control group. Propensity scores derived from the logistic regression are used to match subjects, thereby minimizing group differences across study variables. However, propensity scores cannot account for unmeasured variables that may be the source of the selection bias. Selection can interact with other threats to internal validity, such as history, attrition, and regression. The following are examples of these interactions: • Selection-history interaction: An outside event affects one of the groups more than another.
34411_CH02_023_058
9/9/05
9:35 AM
Page 39
Other Threats to Internal Validity
39
• Selection-attrition interaction: Participants in one group are more likely than others to withdraw or drop out from the program. • Selection-regression interaction: This is a problem of differential regression. In other words, one of the groups by being sicker or healthier is more likely to be average at a later date.
Regression to the Mean Some outcomes studies are designed by selecting individuals on the basis of being very sick or healthy. For example, in orthopedic surgery studies, one selects study subjects on the basis of those having the poorest functioning and in need of a joint replacement. Using the same functional status measure before surgery and after surgery, individuals typically look “average” after surgery and hence appear to have improved. This tendency to obtain scores approaching the average with remeasurement is called regression to the mean. Because all outcomes measures carry some level of uncertainty or error in their administration, outcomes measures are never perfectly reliable. The lack of reliability in outcomes measures exaggerates regression to the mean; that is, an unreliable measure is more prone to regress to the mean with replicate administration. In order to minimize the risk of regression, do not select comparison groups on the basis of extreme scores and use measures with demonstrated reliability. Poor reliability in an outcomes measure can be obviated by avoiding single-item indexes and employing multi-item scales. Attrition The bane of most outcomes studies is attrition, also referred to as experimental mortality. Study participants fail to complete the outcomes measure that is administered or they drop out of the study. The more frequently an outcomes measure is planned for collection, the greater the possibility that there will be attrition. This is a special type of selection that occurs because subjects drop out after the study begins or certain data is missing. Using the earlier orthopedic surgery example, following surgery, individuals fail to return for follow-up and hence do not complete the planned-for outcomes measures. Randomization of subjects fails to control for the effects of attrition. Individuals with poorer results from the treatment or
34411_CH02_023_058
40
9/9/05
CHAPTER 2
9:35 AM
Page 40
DESIGNING AN OUTCOMES RESEARCH STUDY
those with less education might be less likely to return for follow-up or complete study measures. This selective attrition biases results across groups making results applicable to those that are better educated and most benefited by the treatment. A related facet of attrition that is a form of selection is survival selection. This occurs when there is a correlation between patient survival and the treatment studied. For example, in observational studies involving patients with AIDS, those surviving longer are more likely to receive the treatment (Glesby & Hoover, 1996). When treated and untreated patients are compared, the treated group appears to have a longer survival. Survival bias can also distort results. If only survivors are compared, the group with the better survival rate may appear worse because the most vulnerable died. One way to counter this effect is to include those who died in the assessment of outcomes; for example, death may be treated as the worst functional state.
Missing Data In outcomes studies, data will always be missing. The best way to minimize threats posed by missing data is good quality control. This includes careful study management, well-defined project protocols, and clear and wellthought-out operations. Continuous monitoring for quality control minimizes missing data. In addition, it is always best to use available data rather than discarding study variables or cases. Missing data is positive information. Murphy’s Law for outcomes research could read: “If there are any ways in which data can be missing, they will be” (Cohen & Cohen, 1983). Observations needed for conducting outcomes research could be missing for a number of reasons. Attrition, which was discussed earlier, is one reason for missing data. In health outcomes questionnaires, individuals may skip questions either accidentally or deliberately. In other cases, information requested might be difficult or impossible for participants to provide (e.g., questions are too personal or difficult), data systems crash and cannot be recovered, or measuring instruments, such as automatic blood pressure machines, fail. Missing data threatens the integrity of outcomes research and greatly complicates statistical analysis. It threatens the validity of statistical conclusions drawn, particularly if the method for handling missing data is unacceptable and introduces systematic bias. Missing data effectively reduces data for analysis by attenuating statistical power; thereby, reducing the likelihood of detecting differences.
34411_CH02_023_058
9/9/05
9:35 AM
Page 41
Other Threats to Internal Validity
41
The best solution for missing outcomes is improved quality control in the data-collection process. In order to effectively reduce problems posed by missing data, it is critical to distinguish between the types of missing data (Cohen & Cohen, 1983). Is the data missing for the outcome or the treatment? If the outcome is missing, the investigator is faced with dropping the subject from the study. This can lead to a comparison of unbalanced groups, less-representative samples, and a loss of statistical power. Is data randomly or selectively missing? Health survey researchers expect a certain amount of random nonresponses in every study. If the pattern of nonresponse is equally distributed across all subjects, it should not introduce a systematic bias. Selectively missing data poses a more serious problem. Selectively missing data is frequently encountered in studies of special populations, such as the elderly or persons with mental health problems. In studies of the elderly, those with cognitive deficits are less likely to provide risk factor data than those with full cognitive function (Radosevich, 1993), but individuals who are unable to provide requested information about their baseline status are at higher risk for poor health outcomes. Are many versus few items missing? As a general rule, no more than 1 to 2 percent of values should be missing for outcomes study variables. If the pattern of missing values shows that certain data is missing more frequently, then questionnaires and data collection forms should be revised. Dropping variables with high rates of missing values may be safer than dropping subjects (Cohen & Cohen, 1983). Some investigators elect to drop variables from their analysis if extensive data is missing. If the data being dropped makes no material contribution to the outcomes study, dropping it is of little consequence. In that case, the investigator might reconsider why the variable was included in the study. Resources were wasted and information is being lost. On occasion, the investigator chooses to drop participants from the study. In many advanced statistical packages used for analyzing health outcomes data, this procedure is referred to as listwise deletion. If the data is an outcome, as noted earlier, dropping participants might be perfectly justified. However, beyond 1 or 2 percent of participants, this could introduce significant attrition bias into studies. The outcomes study overall loses statistical power and becomes less representative of the target population. This selective loss of subjects is an unacceptable strategy for handling missing data. Pairwise deletion of participants is generally found in studies using correlation methods or bivariate techniques. Associations are examined only
34411_CH02_023_058
42
9/9/05
CHAPTER 2
9:35 AM
Page 42
DESIGNING AN OUTCOMES RESEARCH STUDY
for the paired observations in which the study factor of interest and outcome are both present. If data is randomly missing, this approach might work, but the investigator is unclear as to the study population. A number of acceptable methods for handling missing data in outcomes studies have been suggested. First, use dummy variable codes for missing values. This means that in the place of a missing value for a variable, one employs a dummy code that flags the variable as missing. In the analysis, this strategy has the effect of creating an additional variable for the missing factor and quantifying potential bias introduced by the absence of a value for that variable. A second group of techniques involves the interpolation of outcomes values: (1) carrying the last observed outcome forward, (2) interpolation between known outcomes, and (3) assuming the worst outcome. Different assumptions underlie each of these approaches. For example, in a study of the long-term follow-up of mechanical heart valve recipients, individuals lost to follow-up are assumed to have died from a heart valve–related condition. This involves assuming the worst-case scenario. As an alternative, one might assume the individual was still alive because that was the person’s status at the time of his or her last contact. Finally, mean substitution is an extension of linear regression techniques and is frequently used where the outcome variable is derived from a multiitem scale. The basis for this approach is that the best predictor of a missing value is the other values for the same individual. For multi-item scales such as the 10-item Physical Functioning Scale (PF-10) score (McHorney, Kosinski, & Ware, 1994; Ware, 1991; Ware & Sherbourne, 1992; Ware et al., 1993), a mean scale score is computed on the basis of available items. If the individual completes 7 of the 10 items comprising the PF-10, the scale score is based on the available seven items. This approach underscores an additional advantage of using multi-item scales.
History History concerns events that occur between the treatment and the outcome that are outside the control of the researcher. For example, in an observational study of the effectiveness of a primary care–based treatment program for diabetes mellitus, the introduction of a new drug to treat diabetes is likely to affect the outcome. In the real world, it is impossible to isolate routine care from external changes that occur in the health care environment.
34411_CH02_023_058
9/9/05
9:35 AM
Page 43
Construct Validity
43
Because it is impossible to control for outside events, outcomes researchers have employed several strategies to control for history effects. First, investigators have attempted to isolate the study participants from the outside. This can more easily be accomplished in the laboratory than in the field. In laboratory experiments, study participants receive the experimental intervention in a setting isolated from the field. In field settings, assignment groups could be separated from one another; for example, hospitals and clinics located in different communities. A second strategy to reduce history effects is to use nonreactive outcomes measures. Examples of these measures include laboratory tests and physiological measures that are less susceptible to outside effects. A third strategy is to use a control group drawn from a comparable group of participants. If the intervention and the control groups are comparable and outcomes measurements occur at the same time, history effects would be uniform across the study groups. They might minimize the effect of treatment but will not inflate it.
CONSTRUCT VALIDITY Question 3: What constructs are involved in the particular causeand-effect relationship? Constructs are abstractions inferred from specific observations. Outcomes research is generally concerned with drawing conclusions about attributes that cannot be directly observed. For example, one cannot directly observe physical functioning in a subject but can observe the manifestations of physical functioning (e.g., walking, climbing stairs, standing from a seated position). Physical functioning is a construct. Construct validity involves understanding of how the concepts used in the model relate to one another and how they are measured. There are a number of threats to construct validity (Shadish et al., 2002).
Inadequate Conceptual Design Inadequate preoperational explication of constructs simply means that the measures and treatments have not been adequately defined and explained prior to implementing the study. Many investigators fail to adequately define and analyze the concepts they are studying. Before commencing an outcomes study, an operating model needs to be spelled out. At the very least, the following conceptual planning needs to occur (see Chapter 1):
34411_CH02_023_058
44
9/9/05
CHAPTER 2
9:35 AM
Page 44
DESIGNING AN OUTCOMES RESEARCH STUDY
• Develop and define concepts in the operating model. • Create an operational model of the study that shows the relationship between concepts. • Operationally define and critique the concepts. Good conceptual planning is at least as important as choosing the right measures.
Mono-operation and Mono-method Bias Mono-operation concerns using only a single measure of each concept; for example, using a single outcomes measure or treatment measure. Single operations of study constructs pose the risk of not measuring the concept in the correct way or measuring irrelevant facets of the concept. One way of reducing this threat is by using a number of instances of the concept. For example, to reduce this bias in the measurement of treatment, the design could incorporate various doses of an experimental drug. This strategy would enable the investigator to demonstrate a dose-response relationship. Alternatively, the investigator might increase the number and type of interventions. For example, in a diabetes mellitus disease management program, different forms of patient coaching might be employed: nurse telephone calls, patient learning materials, and physician coaching. Mono-method bias is a related threat to validity, wherein a single method is used to collect data. For example, a study of the effectiveness of a particular diabetes intervention might use only self-reported survey data to answer the question. This design is susceptible to a mono-method bias threat. It would be better to include other measures of effectiveness such as laboratory values or medical records review. The distinction between mono-operation and mono-method bias is often not clear. For this reason, they might be lumped as the monobias threats. They include what measures are used to assess the concepts and what data collection methods are employed. Treatment Diffusion Treatment diffusion is a recurring problem in observational studies. Participants in the control group sometimes receive some of the treatment.
34411_CH02_023_058
9/9/05
9:35 AM
Page 45
External Validity
45
This is sometimes called a “spillover” or “diffusion effect.” For example, in a study of the effects of a hospital nursing care model, hospital units not receiving the intervention could be exposed to the intervention through staff contact between units implementing the care model and those not implementing the care model. Unknown to the investigator, nursing units assigned to the control condition could implement facets of the nursing care model. The diffusion of the treatment would be likely to attenuate differences in the outcomes between the treatment and control conditions. At a minimum, one needs to look for the possibility of a diffusion effect. Other designs can mitigate this effect—for example, by allocating treatment to different physicians or clinics—but this design imposes other problems.
EXTERNAL VALIDITY Question 4: How representative is the relationship across persons, settings, and times? Randomized controlled trials are the backbone of biomedical research and the “gold standard” for determining the efficacy of medical therapies. Through randomization of participants to treatment conditions, the RCT gains in terms of internal validity. However, a major limitation of the RCT is the lack of generalizability. Because RCTs use strict criteria for the inclusion and exclusion of study subjects, results are not as representative of the persons and settings of greatest interest in health care and outcomes research. Moreover, RCTs are costly to implement. Although observational studies suffer from many of the threats to internal validity discussed earlier, they more successfully represent the populations that are receiving the care. The representativeness, also called generalizability, applies to three facets of study designs: the individuals participating in the study, where the treatment occurs, and the timing or time interval for the study. No single design can adequately address the threats to validity. There are trade-offs. The most discussed and major tradeoff is between internal and external validity. Some argue that timely, representative, and less-rigorous observational studies are to be preferred over internally valid study designs. There are no hard and fast rules. Table 2–4 summarizes the threats to validity discussed. These are a few of many possible threats but are the ones that hold greatest relevance to health outcomes research studies. For those listed, the table briefly defines the threat, provides an underlying cause, and gives some possible solutions.
34411_CH02_023_058
46
9/9/05
CHAPTER 2
9:35 AM
Page 46
DESIGNING AN OUTCOMES RESEARCH STUDY
Table 2–4 Adaptation of Cook and Campbell’s Scheme (1979) for Classifying Threats to the Validity of Health Outcomes Research Validity Threat
Definition
Underlying Cause
Possible Solution
Statistical Conclusion Validity Low statistical power
Study design does not permit detecting a true effect
Inadequate sample size and responsiveness of outcomes measure
Increase sample size; choose an outcomes measure with optimal responsiveness characteristics
Fishing and error rate problems
Multiple comparisons increase the likelihood of making a type 1 error
Too many hypotheses; lack of a primary hypothesis
Identify primary and secondary hypotheses; post hoc adjustments for making multiple comparisons
Violated assumptions of statistical tests and inappropriate statistical tests
Inappropriately applied statistical test or the assumption of the statistical test is violated
Careless analysis; plan for analysis not well thought out; failure to consult with an analytic expert
Consult with an analytic expert; use a statistical method that takes into account the correlated nature of outcomes data
Reliability of measures
Unreliable outcome measures
Selecting unstable measures; lack of standardization of measurement
Monitor the quality of measurement; select measures based on sound psychometric properties
Treatment implementation
Inconsistent implementation of the treatment
Lack of standardization of treatment implementation; lack of clarity regarding treatment implementation
Monitor the quality of treatment implementation; take corrective measures to assure standardization; closely monitor the treatment implementation; incorporate treatment measures into study design
34411_CH02_023_058
9/9/05
9:35 AM
Page 47
External Validity
47
Table 2–4 Adaptation of Cook and Campbell’s Scheme (1979) for Classifying Threats to the Validity of Health Outcomes Research Validity Threat
Definition
Underlying Cause
Possible Solution
Internal Validity Selection
Differential selection of subjects to the treatment and control groups
Failure to randomize treatment to subject groups
Risk adjustment; propensity analysis
Regression to the mean
Selection of sicker or healthier subjects for the study more likely to result in outcomes at follow-up that look average
Recruitment criteria for the study focuses on sicker or healthier subjects; unreliable outcome measures
Use of a control group with similar characteristics to the treatment group; improve the reliability of the outcome measure
Attrition and missing data
Subjects drop out or leave the study before its completion
Inadequate followup leaves subjects lost to follow-up; death from an unrelated cause
Quality control of the data collection process
History
Events that occur during the study that affect treatment implementation and outcomes
Changes in routine treatment (e.g., introduction of a new medication, changes in reimbursement, patient management) that could have an effect on the outcome
Monitor and document external factors that could affect treatment implementation and outcomes
Construct Validity Inadequate explication of constructs
Study concepts are poorly defined and their interrelationship not well spelled out
Failure to develop an operating model for the study; muddled thinking about the
Adequate planning of the outcomes study with focus on measures and their interrelationship
continues
34411_CH02_023_058
48
9/9/05
CHAPTER 2
9:35 AM
Page 48
DESIGNING AN OUTCOMES RESEARCH STUDY
Table 2–4 (continued) Adaptation of Cook and Campbell’s Scheme (1979) for Classifying Threats to the Validity of Health Outcomes Research Mono-method and monooperation biases
Using single methods to collect data; using a single measure of the treatment and outcome
Treatment diffusion
In natural settings, the treatment spills over to groups not intended to receive the intervention
question Cost prohibitive to use multiple measures of treatment and outcomes; using single methods of data collection
Inadequate segregation of treatment and control group subjects; rivalry between groups not given the treatment and those receiving the treatment
Employ multiple methods to collect the factors of study interest, e.g., written surveys, personal interviews, and physiological testing; employ multiple approaches in measuring the treatment and outcomes, e.g., self-report, interview, observation Whenever possible, plan to give the control subjects the treatment after the study has concluded; blind subjects to the treatment; give control subjects a “sham” therapy
External Validity Representativeness to person, setting, and time
Results of the study limited by person, setting, and time
Inclusion and exclusion criteria limit the findings
Replicate studies across different populations, in diverse settings, and at other points of time
34411_CH02_023_058
9/9/05
9:35 AM
Page 49
Quasi-Experimental Designs
49
QUASI-EXPERIMENTAL DESIGNS Types of Quasi-experimental Designs Most outcome studies will be observational and hence will rely on quasiexperimental designs. A bare-bones experimental design has an intervention and an outcome. In the absence of a control group, these are sometimes referred to as preexperiments. The preexperimental design could be expanded by adding control groups and pretest measures. All outcomes study designs can be described using a standard nomenclature (see Table 2–5). The preexperimental design consisting of an intervention and outcome could be depicted as follows: X
O
In this post-test-only design, an outcome (O) is observed only after a treatment (X). This type of design is frequently used in medical practice and is referred to as a case study; patients receive a treatment and the researchers observe an outcome. A number of problems are associated with this design, selection, history, and attrition, to name a few. From a statistical perspective, this design is not interpretable. One cannot observe covariation because the design fails to incorporate either a pretest or a control
Table 2–5 Standard Nomenclature for Describing Quasi-experiments O — outcomes measures or an observation X — treatment X — removed treatment R — random assignment of subjects/groups to separate treatments NR — no random assignment of subjects/groups to separate treatments Subjects/groups separated by dashes – – – – are not equated by random assignment Subject/groups divided by a vertical dashed line are not necessarily equivalent to one another
34411_CH02_023_058
50
9/9/05
CHAPTER 2
9:35 AM
Page 50
DESIGNING AN OUTCOMES RESEARCH STUDY
group. The simple randomized controlled trial (RCT) adds both a pretest and a control group. The RCT could be depicted as follows: R
O1
R
O1
X
O2 O2
In this design, participants are randomly assigned (R) to treatment and control conditions. A preintervention observation (O1) is made before the treatment (X) is delivered, followed by a postintervention observation (O2). If the outcome is a measure of physical functioning, it makes intuitive sense to have a measure of functioning before an intervention (e.g., joint replacement surgery or an exercise program). Statistically, this enables the researcher to observe covariation, a necessary prerequisite for statistical conclusion validity. However, in practice, many investigators omit the O1 measures and rely on randomization to produce equivalent groups. For irreversible end points, a post-test-only design with randomization would be essentially equivalent to a randomized control trial. For example, in study where survival was the primary outcome, it makes little sense to think about a preintervention measure. The status of the participant is alive at the time of his or her recruitment. Preintervention observations might include measures of comorbidity and severity for risk adjustment; but randomization, if performed correctly, assures that treatment and control groups are comparable at the time the intervention condition is delivered. Nonetheless, it may prove valuable to collect baseline characteristics to use in the analyses. If the investigator lacks control over allocating participants to the treatment or control conditions, then the study is described as quasi-experimental. Using the standard nomenclature, a quasi-experiment investigating the effectiveness of a disease management program might look like the following: O1 O1
X
O2 O2
In this design, the dashed line is used to signify that the groups are not randomized. From the outset, selection is a serious internal threat to validity. In order to draw valid inferences about differences in the outcomes between the two groups, investigators would need to be able to statistically adjust for differences between treatment and control groups. Any conclusions will be confounded by morbidity differences between the groups. Although a discussion of all possible outcomes study designs is beyond the scope of this chapter, a few design characteristics are worth noting,
34411_CH02_023_058
9/9/05
9:35 AM
Page 51
Quasi-Experimental Designs
51
using the scheme of Shadish and his colleagues (2002): designs without control groups, designs without pretest measures, and combination designs.
Designs Without Control Groups The post-test-only design described earlier can be improved by adding a pretest measure. This type of approach has been used in evaluating the effectiveness of programs such as disease management and health education (Linden, Adams, & Roberts, 2003). O1
X
O2
The investigator makes a pretreatment observation and looks for a change in that measure with follow-up. Although this provides some evidence for change that could be a result of the intervention, it fails to rule out other things that might have happened to the participants (history), such as other treatments, practice changes, or statistical regression to the mean. One improvement to this design is to add additional pretest measures. O1
O2
X
O3
Adding multiple pretest measures reduces the threat of statistical regression, but one cannot rule out the possibility that other external factors might have led to the changes that occurred. This type of design lends itself to situations in which repeated pretest measures are available to the investigator. For example, in a study intended to reduce the use of medical services, prior use of services might serve as pretest measures. The lack of a trend in the use of health care services before the intervention strengthens the argument for the effect of the intervention minimizing threats of regression or age. Shadish and his colleagues (2002) discuss other types of designs without control groups such as the removed-treatment design, repeated-treatment design, and designs that use nonequivalent observations. The interested reader is referred to this source for a more comprehensive discussion.
Designs Without Pretest Measures The pretest is an observation taken before the intervention condition in order to ascertain the preliminary status of the participant. In many outcomes research studies, it is impossible to obtain a pretest measure; for
34411_CH02_023_058
52
9/9/05
CHAPTER 2
9:35 AM
Page 52
DESIGNING AN OUTCOMES RESEARCH STUDY
example, participants are emergent cases. In this type of study, a nonequivalent control group might be used. One of the more frequently used designs without a pretest is the post-test-only design with nonequivalent groups. NR
X
O1
NR
O2
Here the dashed horizontal line indicates that the group receiving the intervention is different from the control group. Campbell and Stanley (1963) called this the static group comparison. Participants receiving the treatment are compared to those who did not, thereby establishing the effect of the intervention. Certainly, the biggest problem with this type of design is selection; participants in one group could systematically differ from those in the other group leading the observations made. One approach to dealing with this threat is to add an independent pretest sample. NR
O1
NR
O1
X
O2 O2
Here the vertical dashed line signifies that the participants at time 1 and time 2 may be different. These observations are independent of one another. This design is used frequently in epidemiology and public health where it is impossible to collect data on the same group of participants pretest and posttest. The level of intervention is at a group or system level and participant level of control is less critical to the study question. For example, what are the effects of community level intervention to increase smoking cessation?
Combination Designs Some quasi-experimental designs use both pretests and control groups. The simplest design is the nonequivalent treatment and control group design with a dependent pretest and posttest. NR
O1
NR
O1
X
O2 O2
34411_CH02_023_058
9/9/05
9:35 AM
Page 53
Summary: General Guidelines
53
The pretest and posttest make it simpler to evaluate the study for attrition (i.e., dropout of subjects) and regression to the mean. However, because participants are not randomized to treatment conditions, differential selection remains a problem; for example, participants receiving the treatment condition are sicker and heavier users of health care services than those receiving the control condition. One way to improve this design is to add pretest measures or switch interventions. NR
O1
O2
NR
O1
O2
X
O3 O3
This type of design might be beneficial where there are ethical concerns about withholding a therapy that could be beneficial to the participant or demoralizing to participants in the control condition. NR
O1
NR
O1
X
O2 O2
O3 X
O3
This brief list of typical designs that have been used in health outcomes research is not exhaustive but merely represents some of the more commonly found designs and some thoughts about how these might be improved.
SUMMARY: GENERAL GUIDELINES FOR DESIGNING A HEALTH OUTCOMES RESEARCH STUDY Evaluate the Threats to Validity This chapter has identified the threats to validity that are frequently encountered in outcomes research studies. For more exhaustive and comprehensive discussion, the reader is encouraged to explore some of the references cited. The most complete treatment of threats to validity can be found in the works of Cook and Campbell (1979) and Shadish and colleagues (2002). These authors have built on the earlier work of Campbell and Stanley (1963), establishing the nomenclature for classifying and describing study designs and characterizing biases found in observational studies. The reading is a bit turgid, but worth the effort to gain an appreciation of the multiple layers of quasi-experiments.
34411_CH02_023_058
54
9/9/05
CHAPTER 2
9:35 AM
Page 54
DESIGNING AN OUTCOMES RESEARCH STUDY
The most important message to remember is the need to identify all potential threats to the validity as one is planning a study. Because quasiexperiments are especially susceptible to internal validity threats—including selection, mortality, and statistical regression—much of the effort is focused in this area. The outcomes researcher needs to engage in a continuous process of self-criticism, preferably involving input from peers with expertise in the area of study. Present a proposal formally to colleagues for their review. Although this can be a particularly humbling experience, even for those viewed as experts in their field, the finished product will be much improved. Construct and statistical conclusion validity are frequently ignored from the outset of design. Investigators will embark on a study before sufficient work has been done developing and refining an operating or conceptual model for their work. As discussed in Chapter 1, this oversight frequently leads to poor operationalization of study variables, ignoring and omitting key factors from their study, and a muddled analysis plan. The conceptual work and statistical plan need to be undertaken before the beginning of the study. Dealing with these threats is no less important than the work of coming up with a sound internally valid study design. If a researcher cannot visualize what the final product will look like, it is advisable not to start.
Draw a Study Logistical Plan When protocols are developed from randomized controlled trials, the study investigator frequently develops a flow diagram called a schedule of events, which demarcates the timing of measurements for the clinical trial. This schedule of data collection provides direction for the study manager about what data needs to be collected, when the data is collected, and from whom the data is collected. The schedule includes all the variables collected as part of the study, the study subjects’ personal characteristics, their risk factor profile, data necessary for risk adjustment (e.g., comorbidity, disease severity), and outcomes measures such as laboratory values, health outcomes questionnaires, and adverse events. Importantly, the schedule of events includes when the study data is collected and from which study subjects this data is collected. It is likewise helpful to diagram the overall design of the study. For outcomes research studies, the graphical design demarcates study groups, time dimensions, outcomes variables, multidimensionality, and possible contrasts for analysis.
34411_CH02_023_058
9/9/05
9:35 AM
Page 55
References
55
Graphics can also be an important adjunct in presenting the results of the study. The graphical image provides greater depth and dimensionality that are impossible to communicate verbally. For an excellent discussion of the graphical display of quantitative information, the reader is encouraged to review works by Tufte (1990, 1997, 2001).
Use Design and Statistical Controls Statistical control, or risk-adjustment control, can never overcome the effects of a poorly designed study. In general, the best strategy is to use a combination of sound study design and statistical controls in implementing the health outcomes research study. Shadish and colleagues (2002) refer to this as the “primacy of control by design.” Design involves adding control groups, making additional observations before the treatment intervention, and timing data collection. Analyzing outcomes data requires statistical techniques that are often beyond the skills of most investigators. The use of correlated methods, described as mixed-model methods, and time-to-event analysis, called survival analysis, requires advanced statistical course work. Because of the complexity of analysis, sound study design must involve input from a skilled data analyst at an early stage of the planning process. This assures that the study question has been clarified, the analysis plan fits the study design, the right variables are being collected, and the study can produce the desired results.
REFERENCES Allison, P.D. (1995). Survival analysis using the SAS® System: A practical guide. Cary, NC: SAS Institute Inc. Allison, P.D. (2001). Logistic regression using the SAS® System: Theory and application. Cary, NC: SAS Institute Inc. Bergner, M. (1989). Quality of life, health status, and clinical research. Medical Care, 27(3, Supplement), S148–S156. Bergner, M.B., Bobbitt, R.A., Carter, W.B., & Gilson, B.S. (1981). The sickness impact profile: Development and final revision of a health status measure. Medical Care, 19, 787–805. Berkson, J. (1946). Limitations of the fourfold table analysis to hospital data. Biometrics Bulletin, 2, 47–53.
34411_CH02_023_058
56
9/9/05
CHAPTER 2
9:35 AM
Page 56
DESIGNING AN OUTCOMES RESEARCH STUDY
Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally and Company. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Company. Disease Management Association of America. Outcomes Consolidation Steering Committee. (2004). Disease management: Program evaluation guide (1st ed.). Washington, DC: Author. Elashoff, J.D., Oliver, M.R., Yeghiazarian, K., Zheng, M., Jamshidian, M., & Koyfman, I. (2000). nQuery Advisor (Version 4.0). Los Angeles, CA. Feinstein, A.R., Sosin, D.M., & Wells, C.K. (1985). The Will Rogers phenomenon. New England Journal of Medicine, 312(25), 1604–1608. Friedman, L.M., Furberg, C.D., & DeMets, D.L. (1996). Fundamentals of clinical trials. St. Louis, MO: Mosby. Glesby, M.J., & Hoover, D.R. (1996). Survivor treatment selection bias in observational studies. Annals of Internal Medicine, 124(11), 999–1005. Hosmer, D.W., & Lemeshow, S. (1999). Applied survival analysis: Regression modeling of time to event data. New York: Wiley-Interscience. Hosmer, D.W., & Lemeshow, S. (2002). Applied logistic regression (2nd ed.). New York: John Wiley and Sons, Inc. Kelsey, J.L., Whittlemore, A.S., Evans, A.S., & Thompson, W.D. (1996). Methods in observational epidemiology (Vol. 26). New York: Oxford University Press. Kim, J.-O., & Mueller, C.W. (1978). Introduction to factor analysis: What it is and how to do it (Vol. 13). Beverly Hills, CA: Sage Publications. Kleinbaum, D.G. (1996). Survival analysis: A self-learning text. New York: SpringerVerlag. Kleinbaum, D.G., & Klein, M. (2002). Logistic regression: A self-learning text. New York: Springer-Verlag. Last, J.M. (Ed.). (2001). A dictionary of epidemiology (4th ed.). New York: Oxford University Press. Le, C.T. (1997). Applied survival analysis. New York: Wiley-Interscience. Le, C.T. (1998). Applied categorical data analysis. New York: John Wiley and Sons, Inc. Liang, K.-Y., & Zeger, S.L. (1993). Regression analysis for correlated data. Annual Review of Public Health, 14, 43–68. Linden, A., Adams, J.L., & Roberts, N. (2003). An assessment of the total population approach for evaluating disease management program effectiveness. Disease Management, 6(2), 93–102. Littell, R.C., Milliken, G.A., Stroup, W.W., & Wolfinger, R.D. (1996). System for mixed models. Cary, NC: SAS Institute Inc.
34411_CH02_023_058
9/9/05
9:35 AM
Page 57
References
57
McHorney, C.A., Kosinski, M., & Ware, J.E. (1994). Comparisons of the costs and quality of norms for the SF-36 survey collected by mail versus telephone interview: Results from a national survey. Medical Care, 32(6), 551–567. Murray, D.M. (1998). Design and analysis of group-randomized trials (Vol. 27). New York: Oxford University Press. Radosevich, D.M. (1993). Factors associated with disability in the elderly. University of Minnesota. Rosenbaum, P.R. (2002). Observational studies (2nd ed.). New York: Springer-Verlag. Rosner, B. (1995). Fundamentals of biostatistics (4th ed.). Belmont, CA: Duxbury Press. Sackett, D.L. (1979). Bias in analytic research. Journal of Chronic Diseases, 32, 51–63. Schlesselman, J.J. (1982). Case-control studies: Design, conduct, analysis. New York: Oxford University Press. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company. Streiner, D.L., & Norman, G.R. (1995). Health measurement scales: A practical guide to their development and use (2nd ed.). Oxford: Oxford University Press. Szklo, M., & Nieto, E.J. (2000). Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers. Tufte, E.R. (1990). Envisioning information. Cheshire, CT: Graphics Press. Tufte, E.R. (1997). Visual explanations: Images and quantities, evidence and narrative. Cheshire, CT: Graphics Press. Tufte, E.R. (2001). The visual display of quantitative information. Cheshire, CT: Graphics Press. Ware, J.E. (1991). Conceptualizing and measuring generic health outcomes. Cancer, 67(3), 774–779. Ware, J.E., & Sherbourne, C.D. (1992). The MOS 36-Item Short-Form Health Survey (SF-36). Medical Care, 30(6), 473–483. Ware, J.E., Snow, K., Kosinski, M., & Gandek, B. (1993). SF-36 Health Survey: Manual and interpretation guide. Boston: The Health Institute, New England Medical Center.
34411_CH02_023_058
9/9/05
9:35 AM
Page 58
34411_CH03_059_082
9/9/05
9:38 AM
Page 59
3 Isolating the Effects of Treatment Paul L. Hebert
INTRODUCTION Health outcomes research often starts with deceptively simple questions, like which treatment is best or which is the best hospital or best health plan? The difficulties in answering these questions arise from the complexity of health itself. Health is not produced by medical therapy alone. The medical treatment a patient receives can be an important determinant of a patient’s health, but so is the patient’s age and gender, the severity of his or her condition, and other comorbid conditions and socioeconomic status. The best physicians or the best treatments may improve a patient’s health, but discovering what works best is complicated by the fact that the sickest patients may seek care from the best doctors or be treated with the most aggressive therapy. The goal of outcomes research is to disentangle from these complex relationships the contribution that a particular treatment makes to a patient’s health. The first section of this chapter examines how an outcomes research starts with a conceptual model of what the treatment in question is, the factors that play a role in how the treatment is determined, and how the treatment interacts with other determinants of health to produce some observable health outcome. The second section briefly discusses statistical analyses for isolating the effect of the treatment. Studies designs appropriate for outcomes research are treated in greater detail in Chapter 2. The third section provides an example.
59
34411_CH03_059_082
60
9/9/05
CHAPTER 3
9:38 AM
Page 60
ISOLATING THE EFFECTS OF TREATMENT
CONCEPTUAL MODEL As noted in Chapter 1, outcomes research begins with a conceptual model. The conceptual model maps out relationships between key constructs in the health, medical treatment, and health outcomes of study subjects. It is not necessarily a theoretical model. Relationships could be based on theories of behavior from psychology or economics, but clinical experiences and prior empirical analyses should also be reflected. Key questions to be addressed in a conceptual model should be readily inferred from that model.
What Is the Outcome? The conceptual model should clearly indicate what health outcome is the focus of the analysis. Many different types of outcomes exist. A researcher could be interested in the effect of a treatment on a physiological measure such as blood pressure, on the probability of an event such as death or hospitalizations, or on more subjective outcomes such as patient satisfaction. Later chapters discuss methods for measuring many of these outcomes. In all cases, the outcomes chosen should be sensitive to changes in the treatment in the population under investigation.
What Are the Treatments Under Comparison? Treatments can differ in terms of what was done and how it was done; comparisons between treatments can take many forms. A research study can compare treatments of different kinds or of different intensities or dosages or can compare the same treatment across different settings or providers. The conceptual model should identify what the treatment is and what is the basis of comparison. Oftentimes, identifying the treatment and the basis of comparison is straightforward. In an assessment of the effectiveness of influenza vaccination, the influenza vaccination is the treatment and the basis of comparison is between those who got a flu shot and those who did not.1 Sometimes defining the treatment is more complex. For example, what is the treatment in a study of the effectiveness of palliative care? One could define “palliative care” strictly as a referral to a palliative care consult team and compare
34411_CH03_059_082
9/9/05
9:38 AM
Page 61
Conceptual Model
61
patient and family member satisfaction across people who did and did not receive a palliative care consult. Alternatively, one could define the treatment of palliative care as consisting of a panel of treatments: assessment and treatment of pain and nausea, defining goals of care, and discharge planning. Any patient who receives this panel of treatments could be said to have received palliative care, regardless of whether the patient received it from a palliative care consult team. Some patients may receive only some components. The alternative definitions of the treatment might lead to profound differences in the estimation of the effectiveness of palliative care.
How Is the Treatment Assignment Determined? In a randomized trial, treatment assignment is determined by a figurative flip of a coin, but this is not the case in nonrandomized study designs. Constructing a conceptual model that carefully describes how a patient gets into one treatment group or the other will help an outcomes researcher identify the possible biases that are inherent in the observational data. For example, the sickest patients are often sent to academic health centers for care, and Medicare HMOs tend to attract healthier Medicare beneficiaries. Careful measurement of initial health status, including the severity of illness, important comorbid conditions, and demographics, is required to make an unbiased assessment of the outcomes of care at academic health centers or in Medicare HMOs. Allocation of treatment is not only a function of underlying health status. Economics, personal preferences, institutional biases, resource availability, and many other factors can play a role in the treatment a person receives. By listing these factors and showing how they relate to the receipt of a treatment and to outcome under investigation, a researcher can see what statistical steps are needed to prevent these factors from confounding an assessment of treatment effectiveness. As discussed later, confounding occurs when some factor is independently correlated with both the treatment a patient receives and the health outcome under investigation.
What Are the Other Important Explanatory Variables? Identifying risk factors is critical for risk adjustment, but even in randomized trials where randomization is used to achieve a balance of risk
34411_CH03_059_082
62
9/9/05
CHAPTER 3
9:38 AM
Page 62
ISOLATING THE EFFECTS OF TREATMENT
factors between treatment groups, identifying all factors that contribute importantly to the outcome can still be statistically useful. As will be described later, one goal of a statistical analysis is inference: determining the extent to which the observed differences between two treatment groups might be due to chance alone. This involves assessing how much of the variation in the outcome can be attributed to the treatment. A researcher can improve his or her chances of appropriately declaring that differences in outcomes between treatment groups is statistically significant by controlling for other factors, or explanatory variables, that contribute to the variation in the outcome. For example, advanced age is clearly a risk factor for mortality. Even if two treatment groups do not differ in terms of mean patient age, a statistical analysis of the effects of the treatment on mortality could be improved by incorporating age as an explanatory variable. Because age explains a great deal of the variation in mortality, including age as an explanatory variable in a multivariate model can decrease the amount of unexplained variation in mortality. This can result in an appropriately smaller confidence interval for the estimated treatment effect and a greater likelihood of appropriately concluding that the effect of the treatment was statistically significant.
What Is the Unit of Analysis? In outcomes research the unit of analysis is often, but not always, the patient. In many health outcome studies, a patient is observed to receive one of several treatments and his or her outcomes are recorded; thus, the unit of analysis is the patient. However, the unit of analysis can be more aggregate as is sometimes the case in epidemiological studies. In epidemiology, “ecological” analyses aggregate health and other statistics by geographic area and compare outcomes across areas that differ with respect to their use of a particular treatment. For example, a research study could compare county-level hospitalization rates across counties with varying levels of managed care market penetration. The unit of analysis in this case is the county, not the person. The unit of analysis can be less aggregate than the patient. In an analysis of in-hospital prescribing errors, for example, the unit of analysis could be the administration of a particular class of drug to a particular group of patients. A patient may receive hundreds of administrations of drugs, only
34411_CH03_059_082
9/9/05
9:38 AM
Page 63
Statistical Model
63
some of which are subject to rules that define appropriate dosing errors. Outcomes research could address whether a particular treatment—such as a computerized order entry—reduces the likelihood of an inappropriate administration. In this case, the unit of analysis would be the administration of a particular drug.
STATISTICAL MODEL The conceptual model informs the analytical model. In an analytical model, the outcomes researcher defines measures for each of the key constructs in the conceptual model and determines the appropriate statistical procedures to conduct. Many of the remaining chapters in this book are devoted to appropriate ways to measure these constructs. These measures must be incorporated into an analytical framework. There are two goals of a statistical analysis: estimation and inference. Estimation is concerned with getting the direction and size of the treatment effect correct. Getting the treatment effect right means that if treatment A really is better than treatment B, then the statistical analysis should show better outcomes among patients who receive treatment A. Inference is concerned with getting the p-value right. It is the basis for deciding that the estimate of the difference in outcomes between two treatment groups is sufficiently large that it could not be attributed to chance alone.
Estimation Statisticians talk about the difference between the true treatment effect and the estimate of the treatment effect. Although it is never possible to know for certain what the true treatment effect is, certain ways of estimating the treatment effect are better than others. Well-conducted randomized placebo-controlled double-blinded trials are the gold standard for estimating treatment effects. However, even among well-conducted trials, some variation in estimated treatment effects will exist. Some trials will yield estimates that are higher and some that are lower, but across many repeated trials, one expects that these types of trials should yield estimates of the effect of treatment that cluster around the unobservable, true effective. At the other end of the spectrum are ways of estimating the effect of treatment that will repeatedly overestimate or underestimate the true treatment
34411_CH03_059_082
64
9/9/05
CHAPTER 3
9:38 AM
Page 64
ISOLATING THE EFFECTS OF TREATMENT
effect. An estimate that systematically and persistently over- or underestimates the true effect is said to be biased. In statistics, a biased estimate of the effect of a treatment is an estimate that differs from the true effect of the treatment in some systemic, nonrandom way. Bias can be introduced to an analysis in a number of ways, commonly through confounding. Confounding occurs when an unmeasured factor is correlated with both the treatment assignment and the outcome. If sicker people get flu shots and the researcher does not fully measure and account for this factor, then unmeasured differences in health between flu shot users and nonusers can confound an estimate of the effect of influenza vaccination, resulting in a biased estimate of the effectiveness of flu shots. This causes an outcomes researcher to falsely attribute a difference in outcomes to a treatment, when it should in fact be attributed to the confounding factor. This confounding factor could be a health-related factor or behavior that increases the risk or severity of a poor health outcome, in which case one refers to it as a risk factor. For example, smoking is (negatively) correlated with receiving a flu shot and positively correlated with being hospitalized for respiratory diseases such as influenza. Smoking is a risk factor for influenza that could confound an analysis on the effectiveness of influenza vaccination if appropriate statistical adjustments are not taken. The process of removing the effects of risk factors on the estimated treatment effect is often referred to as risk adjustment. Contrast this example with that of HMO membership, which is also correlated with receiving a flu shot and with hospitalization for influenza or pneumonia: People who belong to an HMO are more likely to receive a flu shot and because HMOs try to discourage hospitalizations, people in HMOs are less likely to be hospitalized for pneumonia. HMO membership could confound an assessment of the effectiveness of vaccination, but one would not call it a risk factor. This demonstrates the importance in a conceptual model of describing all of the factors that go into determining treatment assignment, not just health-related risk factors. Figures 3–1 through 3–3 show several ways that an unmeasured factor can enter into an outcomes study. An unmeasured factor is represented by a broken-line circle. A broken-line arrow signifies an undetected correlation between the factor and another element in the model. Figure 3–1 depicts a situation in which an unmeasured risk factor is correlated with both patient characteristics and outcomes, but not with the choice of treatment. For example, in a randomized trial of the effectiveness of influenza vaccination, both treatment groups (vacinees and nonvacinees) may con-
34411_CH03_059_082
9/9/05
9:38 AM
Page 65
Statistical Model
65
Unmeasured Risk Factor
Outcome Health Status
Initial Health Status
Treatment: What was done and how well?
Figure 3–1 Unmeasured Factors Uncorrelated with Treatment
tain smokers, but in this circumstance, smokers are not disproportionately represented in either treatment group. In this situation, the investigators may be able to discern the effect of the treatment in question without concern that smoking is confounding the estimate even if they did not measure smoking status. Figure 3–2 depicts a situation in which the unmeasured factor is correlated with one of the treatments and the outcome, but not with the underlying health characteristics of the patients in the treatment or control group. For example, an outcomes researcher may wish to explore differences in health outcomes at two surgical facilities. A patient enters a hospital for a given surgery and then is discharged to a rehabilitation facility. The quality of the surgery, the quality of the discharge planning, and the quality of the rehabilitation facility are all intrinsically linked. The investigator risks attribution bias by falsely attributing a difference in outcomes to one aspect of
34411_CH03_059_082
66
9/9/05
CHAPTER 3
9:38 AM
Page 66
ISOLATING THE EFFECTS OF TREATMENT
Outcome Health Status
Initial Health Status
Treatment: What was done and how well?
Correlate of Treatment
Figure 3–2 Unmeasured Factor Associated with Both Treatment and Outcome (Attribution Bias)
treatment (i.e., the surgery) when in truth the difference in outcome may have been to poor discharge planning or poor rehabilitation. An unobserved factor—the quality of rehabilitation—is correlated with the site of treatment and the outcome. This leads to a biased estimate of the relative quality of the two surgical facilities. Figure 3–3 shows another damaging way in which an unmeasured factor can enter an outcomes study. In this scenario, some significant, unmeasured risk factor is correlated with the underlying health characteristics of the patient and the treatment in question. This clearly leads to biased estimates of the treatment effect. For example, if only the sickest patients receive a given treatment, then that treatment may appear to be less effective, or even harmful, relative to other treatments. This type of bias is called selection bias. Selection bias can work in both directions. Some clinicians may opt to treat those patients with the best prognoses. Observational studies are inevitably prone to selection bias, but the extent may be more apparent in some circumstances. To avoid selection bias, control and treatment groups should be constructed in such a way that the only difference between the two groups is the treatment itself. In this way, any observed changes in health status at the conclusion of the trial can be logically attributed to the intervention. The following describes several techniques for controlling selection bias in outcomes studies. More detailed information on study designs and ways to avoid biases are discussed in Chapter 2.
34411_CH03_059_082
9/9/05
9:38 AM
Page 67
Statistical Model
67
Outcome Health Status
Initial Health Status
Treatment: What was done and how well? Unmeasured Risk Factor
Figure 3–3 Unmeasured Risk Factor Associated with Initial Health Status and Treatment (Selection Bias)
Controlling for Selection Bias Through Random Assignment When the investigator has control over the assignment of patients to treatment groups, the appropriate procedure is to assign individuals to the groups using some random process. The goal of randomization is to distribute patient characteristics equally across intervention and control groups. Randomized assignment does not guarantee that the treatment and control groups are equivalent; it only provides some statistical assurance that the probability of having nonequivalent control and intervention groups is small. Controlling for Selection Bias Through Natural Experiments Natural experiments are situations in which an intervention and a control group are created through some natural, systemic processes, but that processes is believed to be unrelated to the underlying characteristics of the patients. Because the process is unrelated to patient characteristics, the two groups created by the process are believed to be similar except for the intervention in question; thus, any difference in outcomes between the two groups is assumed to be attributable to the intervention. For example, a study of the outcomes of patients in coronary care unit versus noncoronary care unit wards was conducted using data from a natural experiment. The
34411_CH03_059_082
68
9/9/05
CHAPTER 3
9:38 AM
Page 68
ISOLATING THE EFFECTS OF TREATMENT
investigators found patients who had been assigned to a coronary care unit, but who were admitted to a general medical service because there was insufficient room in coronary care unit. The process by which the patients got into the control group (medical service) or treatment group (coronary care unit) was based on the availability of a coronary care unit bed, which was believed to be unrelated to the health characteristics of the patient. Controlling for Selection Bias Using Multivariate Statistics Often in outcomes research, the comparison groups of interest were created neither by direct experimental control nor by a natural process that would suggest equivalent groups. In order to make meaningful comparisons of the outcomes of the two units, the investigator must measure and statistically account for differences in initial patient risk factors. Having measured the relevant variables, investigators typically use multivariate regression analysis to disentangle the effect of the treatment on health outcomes from the contribution of these other factors that may affect the outcome. A number of good texts discuss the use of multivariate regression analysis in this context (Hirsch & Riegelman, 1996; Kennedy, 1993). Generally, investigators estimate an equation of the form: Outcomei 0 1PatientDemogi 2Comorbidi 3Severityi 4Treatment eI
where treatment is equal to zero if the patient is in the control group and 1 if the patient is in the treatment group. If the error from this regression (ei) is uncorrelated with the treatment—that is, there is no unmeasured factor that is correlated with treatment and the outcome of treatment—then the coefficient on the treatment variable represents an unbiased estimate of the effect of the treatment on the outcome. Controlling for Selection Bias Using Matching and/or Propensity Scores Rubin (1983) has noted that simply measuring confounding factors and including them as explanatory variables in a multivariate statistical analysis does not always eliminate bias in the estimated treatment effect. Researchers can still get biased responses if the separation of independent variables is excessive. Separation refers to a situation in which extremely different distributions of independent variables between treatment and control groups exist. For an example of perfect separation consider the case
34411_CH03_059_082
9/9/05
9:38 AM
Page 69
Statistical Model
69
where only very old persons get flu shots. A multivariate model could not tell whether any observed differences in outcomes were due to age or vaccination. Rubin has shown that far less extreme forms of separation—both in terms of means and in terms of variances of independent variables—can have deleterious consequences for estimates of the effect of the treatment (Rubin, 2001). One way to address the problem of separation is to match patients in the treatment group to patients in the control group that have similar characteristics. When there are only a few important characteristics and the researcher has a large number of observations, this can be done by stratifying. For example, if age and smoking were the only important characteristics in determining who gets a flu shot, then a researcher could match each old smoker who got a flu shot to one or more old smokers who did not. Similar stratifications could be made for young smokers and for nonsmokers, both old and young. This would create databases of treatment and control subjects that were balanced on important covariates of age and smoking status. When a large number of factors influence treatment choice, propensity score matching is a useful technique to achieve such a balance. With propensity score matching, researchers first estimate the probability of receiving the treatment as a function of baseline patient characteristics. The propensity score is the predicted probability of receiving the treatment from this regression. Patients who received treatment are then matched to patients who did not receive the treatment, but have similar linear propensity scores and other important prognostic variables. Through this matching, a comparison group of patients who did not receive the treatment is created that have observable baseline patient characteristics that are similar to patients who did receive the treatment. Propensity score matching is similar to a matching in a matched case control study (discussed in Chapter 2), except that in a case control study, patients who had a particular outcome are matched to patients who did not have that outcome. For example, in a case control trial of influenza vaccine effectiveness, patients who were hospitalized for pneumonia could be matched to similar patients who were not hospitalized, and then influenza vaccination status could be compared between these two groups. If influenza vaccination were protective, one would expect to see fewer vaccinees in the hospitalized group (the cases) than the nonhospitalized group (the controls). In contrast, a vaccine study based on propensity score matching would match vaccinees to nonvacinees and look for differences
34411_CH03_059_082
70
9/9/05
CHAPTER 3
9:38 AM
Page 70
ISOLATING THE EFFECTS OF TREATMENT
in hospitalizations between the two matched groups. A protective effect of vaccination would be evidenced by fewer hospitalizations in the vaccinated group. Propensity matching, or any other matching algorithm, can only hope to reduce the bias that is inherent in observable patient characteristics. If, after matching observable patient characteristics, patients still differ on important factors that cannot be observed by the researcher, then the estimated treatment effect can still be biased. In addition, matching can limit the generalizeability of the findings—or the extent to which the findings in a particular study apply to the general population of patients with the studied condition and treatments. Often, for some people in the treatment group, no comparable match can be found in the control group, and some people in the control group will not match to patients in the treatment group. Nothing can be said about the effect of the treatment in patients who cannot be matched. Creating Equivalent Groups Through Instrumental Variables Instrumental variable models are a sophisticated form of multiple regression analysis that is appropriate when potential confounding variables are either unknown or difficult to measure. Because this type of analysis has increasing importance to outcomes research, a heuristic explanation of the techniques is discussed. For a more detailed discussion, see Newhouse and McClellan (1998). The most important element of these models is the concept of an instrumental variable: some measurable event or characteristic that gets individuals into a treatment group, but has nothing to do with the outcome in question (except perhaps through its effect on the treatment that the patient receives). In theory, such a variable is comparable to the random number generator that assigns individuals to treatment and control groups in a randomized controlled trial. The results of the random number generator got individuals into one group or another, but had nothing else whatsoever to do with the outcome of the experiment. If an investigator can identify a naturally occurring variable or variables that fit these characteristics, then an instrumental variable model may produce unbiased estimates of a treatment effect. For example, a study of the effect of more aggressive treatment of coronary artery disease following a post myocardial infarction used the distance from the patient’s home to the nearest hospital as the instrumental variable (McClellan, McNeil, & Newhouse, 1994). It was argued that when
34411_CH03_059_082
9/9/05
9:38 AM
Page 71
Statistical Model
71
a patient suffers a myocardial infarction, he or she is taken immediately to the nearest facility; characteristics of the nearest facility had nothing to do with the patient’s underlying health conditions. However, if the nearest facility engaged in aggressive treatment of coronary artery disease— specifically, providing a high number of coronary artery bypass graft (CABG) surgeries to patients presenting with myocardial infarction—then that patient was likely to receive a CABG. Researchers could construct two datasets of persons hospitalized for myocardial infarction—one consisting of patients who lived relatively near an aggressive facility and one consisting of patients who lived relatively distant. Presumably, the only difference between patients in these two datasets is that in one, more patients received a CABG. Differences in outcomes between these two groups could then logically be attributed to greater use of a CABG. A crucial problem with instrumental variables is finding a suitable instrument. Most candidate instrumental variables turn out to be related to both access to care and outcomes. Unfortunately, they are hard to identify for many studies.
Inference The second goal of a statistical analysis is inference. Every estimate of the treatment effect has some uncertainty associated with it. As described, most health issues are dominated by random, unobservable, or unknown factors. There is a chance that differences in outcomes between two treatment groups—even in a well-designed randomized trial—are just a result of these random chances and have little to do with the effect of a treatment. Even a large randomized trial of two placebos would not likely yield exactly the same number of outcome events in both placebo groups. The purpose of inference is to estimate the degree to which the observed difference is attributable to chance. The purpose of this brief section is not to give an exhaustive discussion of statistical inference, which is an extensive literature, but to give common situations where inferences can be compromised. Key concepts in inference include the confidence intervals and p-values. Researchers worship small p-values and confidence intervals that do not include zero, but what exactly is a p-value? Simply put, a p-value is the probability that the results of an experiment could be as discrepant or more discrepant with the null hypothesis than they were observed to be. In outcomes research the null hypothesis is usually that there are no differences in
34411_CH03_059_082
72
9/9/05
CHAPTER 3
9:38 AM
Page 72
ISOLATING THE EFFECTS OF TREATMENT
outcomes between treatment groups. The experiment is conducted, outcomes are compared between the treatment groups, and a p-value is estimated as a measure of how discrepant the outcomes were from the null hypothesis. For example, if one were testing whether a coin is fair and flipped the coin 100 times only to find that it turned up heads 99 times, the p-value for the null hypothesis that the coin is fair is the probability of getting an outcome as bad as or worse than 99 heads in 100 flips. In this case, a worse outcome would be 100 consecutive heads or 100 consecutive tails. These are extremely unlikely events with a fair coin, so the p-value is very small. The researcher would have a good degree of confidence that the coin is not fair. Some researchers do not like to report p-values, but prefer confidence intervals instead. A confidence interval for a treatment effect is a range of effect within which a researcher has a specified level of statistical confidence that the treatment effect lays. (This is related to the p-value in that the p-value measures the probability of the treatment effect being outside the confidence interval.) For example, if a researcher reports a reduction in the relative risk of hospitalization for flu shot users of 0.75 0.25, she is saying that there is a 95 percent chance that the effect of vaccination lay somewhere between 1.0 (no reduction in risk) and 0.50 (a halving of the risk of hospitalization). A confidence interval that does not include 0 (or 1.0 in the case of a relative risk) indicates that the treatment had an effect that was larger than could be explained by chance alone. The benefit of confidence intervals over p-values is that the confidence interval gives information on the range of the effect in addition to the probability that the treatment effect is different from zero. A very small p-value can be associated with a very small treatment effect, if the sample size is extremely large. The p-value or confidence interval only conveys confidence or uncertainty about the statistical properties of the estimated treatment effect. The data in a statistical analysis has certain patterns of variability in it. That variability is reflected in the confidence interval, with a greater variability associated with greater uncertainty and larger confidence intervals. There can be many other sources of uncertainty in an outcomes research. For example, a researcher could be uncertain of the true influenza vaccination status of an individual if vaccination status was derived from a self-report, or a researcher could have concerns that not all hospitalizations were observed for patients in the study. These sources of uncertainty are not captured in a statistical confidence interval. Consequently, researchers should be careful not to put undue importance on a small p-value.
34411_CH03_059_082
9/9/05
9:38 AM
Page 73
Statistical Model
73
How Can a Researcher Get the Wrong p-Value? Just as the estimated treatment effect can be biased, the p-value can also be biased. Improper statistical techniques can lead to confidence intervals that are either too wide or too narrow. Confidence intervals that are too narrow lead to p-values that are too small and the improper conclusion that there is statistical evidence of a treatment effect. The most common way to get a p-value that is improperly small is to assume that the data in the analysis is independent when it is in fact not independent, but correlated. Two observations are said to be independent if the fact that one observation is above the mean has no bearing on whether the other observation is above or below the mean. For example, for two people drawn at random from Medicare beneficiaries nationwide, the probability that one person’s blood pressure is above average should have no bearing on whether the second person’s blood pressure is above or below average. The statement cannot be made with the same confidence, however, when the two blood pressure readings are taken from people within the same family or from the same person at different times. Because people in the same family may share dietary habits and genetic characteristics, the fact that one person’s blood pressure is elevated suggests that a family member may also have elevated blood pressure. If correlated data are treated as if they were independent, the resulting p-values and confidence intervals are usually too small. To see this, consider an outcomes study of the effect of HMO participation on beneficiary blood pressure. The goal is to measure and compare blood pressure for a sample of individuals in HMO and fee-for-service plans. Taking 20 blood pressure readings on five people in each group would yield 100 observations from each group, but because readings from the same people are correlated, one would not expect this to provide as much information on the mean blood pressure in the two groups as would taking one blood pressure reading from each of 100 beneficiaries in each group. Consequently, one would expect the uncertainty about an HMO effect on blood pressure to be greater with only five subjects per group, and the p-value on the estimate of an HMO effect of blood pressure should be correspondingly larger. Thus, intuition suggests that treating correlated data as if it were uncorrelated yields p-values that are too small. The following are three common ways for data in an outcomes study to be correlated and a brief reference to statistical solutions to the problems.
34411_CH03_059_082
74
9/9/05
CHAPTER 3
9:38 AM
Page 74
ISOLATING THE EFFECTS OF TREATMENT
1. Correlation due to nonrandom survey sampling techniques. Data from large surveys are often correlated because of the survey sampling technique used to collect the data. Survey sampling refers to the methods used to identify a sample of people from whom a researcher gathers information, perhaps through the use of a survey. In simple random sampling, researchers identify a universe or list of potential subjects—for example, all Medicare beneficiaries or all persons with telephone service in a state—and then sample persons at random from within this list. Outcomes research that utilizes Medicare administrative data often uses simple random sampling. Administrators at the Centers for Medicare and Medicaid Services maintain databases of Medicare claims for a simple random sample of beneficiaries and make these databases available to qualified researchers. This technique can be an expensive way to conduct some surveys. If the survey is being conducted in person, the researcher might want to limit the geographic areas where she or he takes surveys in order to limit the required travel time. A researcher could randomly select geographic areas by first choosing ZIP codes in which to conduct the survey, and then randomly selecting persons within those ZIP codes to survey. Data from people selected from different ZIP codes may be independent, but people within the same ZIP code may have many things in common, including similar levels of socioeconomic status and environmental exposures, resulting in data that are not independent within ZIP code. The Medicare Current Beneficiary Survey (Adler, 1994) is an example of a survey conducted in this way. More complex sampling techniques are also frequently used. The National Health Interview Survey contacts 40,000 households and then collects information on individuals within each household. Not only is data from members of the same household likely correlated, but households within the same geographic area are also likely to be correlated. The solution to these problems is to analyze data using software that accounts for the complex structure of the survey. Software such as SUDAAN is specifically designed for this purpose; most statistical software packages have modules within them to analyze data from surveys with complex sampling structures. These software packages take into account the various levels on which data is correlated and give p-values that are consistent with that correlation structure. 2. “Clustering” observations by site in multicenter trials. Similar to survey sampling, clustering occurs in multisite trials because observations
34411_CH03_059_082
9/9/05
9:38 AM
Page 75
Statistical Model
75
from the same site—for example, patients from the same hospital, clinic, or firm within a clinic—share common, unobserved characteristics. Quality of care may vary by site so that patients from a high-quality hospital tend to have better-than-average outcomes, resulting in data from that site being “clustered” near the top of the outcomes scale. As with all correlated data, this could lead to p-values that are too small. Several statistical solutions to clustered data exist (Localio, Berlin, Ten Have, & Kimmel, 2001). These methods are usually organized into two types: conditional and unconditional. Generally speaking, conditional methods measure the treatment effect within each center and then combine results across centers. These techniques can only be used if there is a mix of treatments and outcomes in each site; that is, there cannot be a site in which, by chance, all study subjects were randomized to the treatment group. Models that use conditional methods are also known as marginal models. These include fixed-effects models (also known as dummy variable models), random-intercept models (also known as random-effects models in the econometrics literature) (Wooldridge, 2002), or fixed- and random-effects models (also known as mixed models or random coefficient models). Clearly, a major problem with this literature is that researchers from different disciplines use different terms to refer to the same concepts and, in the case of the term “fixed effects,” the same term to refer to different concepts. Unconditional methods estimate the treatment effect over the entire population of patients (hence, they are also sometimes called population-averaged models), and then correct the standard errors and confidence intervals to account for the clustering within site. Unconditional models include survey sampling techniques described briefly earlier; generalized estimating equations (GEE) (Diggle, Liang, & Zeger, 1994), which treat the correlation between observations as another parameter to estimate; and bootstrap resampling techniques that resample at the site level instead of the patient level. 3. Repeated measures on the same individual. Many outcomes assessments use databases that consist of multiple observations on a panel of study subjects. For example, an outcomes study of nurse management for patients with congestive heart failure may follow two groups of patients— one group who receives help in managing their disease from a nurse and a typical care group—over a period of time. The researchers could be interested in changes in functional status over time, and thus could take measures of an appropriate health status measure, such as the Short Form-36
34411_CH03_059_082
76
9/9/05
CHAPTER 3
9:38 AM
Page 76
ISOLATING THE EFFECTS OF TREATMENT
(see Chapter 7) at baseline and every 3 months for 18 months. Each person in the study would contribute eight observations to the analysis: one at baseline and seven follow-up observations. The researchers could then assess whether the patients in the nurse group maintained functioning longer than those in the usual care group. Because patients who are high functioning at baseline are more likely to be high function at follow-up, the SF-36 scores for the same person are likely correlated, and statistical tools that ignore this correlation will likely generate p-values that are artificially small. This type of analysis is called repeated measures analysis, longitudinal data analysis, panel data analysis, or growth curve modeling. Generally speaking, techniques appropriate for clustered data are also appropriate for repeated measures analysis, although generalized estimating equations and mixed models are the most popular.
Group Randomized Trials Sometimes the most appropriate unit of analysis is not the individual, but a group of individuals. For example, in the Minnesota Heart Health Program (Luepker et al., 1994), three pairs of communities were selected to participate in a study to investigate the effects of a community-wide health education program on morbidity due to coronary heart disease. Although the intervention (the health education program) was given at the community level, data were collected at the person level. Appropriate statistical techniques need to be used to adjust for the correlation between members in the same community and between observations taken within the same community over time. Murray (1998) describes appropriate statistical tests for group-randomized trials, focusing on mixed (fixed- and random-effects) models.
EXAMPLE: THE EFFECTIVENESS OF INFLUENZA VACCINATION The following is a brief example of an outcomes analysis of the effect of influenza vaccination on the probability of hospitalization for pneumonia or influenza.
34411_CH03_059_082
9/9/05
9:38 AM
Page 77
Example: The Effectiveness of Influenza Vaccination
77
Conceptual Model Figure 3–4 shows a conceptual model for an assessment of the effectiveness of influenza vaccination in elderly Medicare beneficiaries. The unit of analysis is the individual beneficiary. The treatment in question is the receipt of a flu shot and the basis of comparison is between patients who did and did not receive the treatment. The influenza vaccine changes from one flu season to the next in anticipation of the dominant influenza strain for the coming flu season. Sometimes the match between vaccine and virus is good and sometimes it is not. For this reason, the analysis is conducted over several flu seasons to get an average effect of influenza vaccinations. The proposed outcome is hospitalization for pneumonia and influenza. Pneumonia is a serious complication of influenza in the elderly, so it should be sensitive to differences in influenza vaccination status. Figure 3–4 shows important risk factors include patient demographics, patient smoking status, and comorbid conditions, such as diabetes, heart failure, and asthma. These are important confounding variables because they are correlated with both pneumonia hospitalizations and with the
Instrumental variables Chronic ulcers Chronic dermatitis
Unobserved risk factors Smoking status
Treatment Influenza Vaccination
Potential confounder HMO status
Patient risk factors Age, Race, Education Clinical risk factors Comorbid conditions - Diabetes - Asthma - Heart failure
Outcome Pneumonia hospitalization
Other explanatory variables Medigap insurance Virus strain
Figure 3–4 Conceptual Model for an Assessment of the Effect of Influenza Vaccination on Hospitalizations for Pneumonia
34411_CH03_059_082
78
9/9/05
CHAPTER 3
9:38 AM
Page 78
ISOLATING THE EFFECTS OF TREATMENT
receipt of a flu shot. Previous research has shown that people who have asthma are both more likely to get a flu shot and are more susceptible to contracting pneumonia. This is consistent with theoretical models of preventive behavior, such as the Health Beliefs Model. According to the Health Beliefs Model (Becker, 1974), a person is more likely to take a preventive measure such as vaccination if he or she feels more susceptible to flu or its complications, such as pneumonia. Age and smoking status are also important risk factors and potential confounders. Smoking is especially troubling because one assumes that is unobserved in the database for this analysis. Figure 3–4 also shows that Medicare HMO enrollment is correlated with both the receipt of an influenza vaccination and hospitalization for pneumonia. This association would be especially important if the analysis was being conducted using Medicare administrative data because Medicare HMOs do not always submit claims to Medicare for the hospitalizations of their beneficiaries. Therefore, not all hospitalizations for HMO beneficiaries are accounted for in the Medicare administrative data. This omission poses a risk of attribution bias: attributing a protective effect to influenza vaccination when it should be attributed to the fact that hospitalizations for HMO beneficiaries are often not observed. For this reason, it may be a good idea to restrict the analysis to Medicare fee-for-service beneficiaries only. Supplemental or “Medigap” insurance refers to a supplemental policy that most Medicare beneficiaries purchase that helps covers the out-ofpocket costs associated with a Medicare hospitalization. Medicare charges beneficiaries a deductible of several hundred dollars when they are hospitalized. Medigap policies cover this deductible, so hospitalization is cheaper for beneficiaries with Medigap than without. Studies have shown that this reduced price induces more hospitalizations for persons with Medigap. This might be especially true for persons with conditions such as pneumonia, for which outpatient treatment is often a viable option. Medigap coverage should not have an effect on the receipt of a flu shot because flu shots are free to Medicare beneficiaries. Nevertheless, it could explain some of the variation in pneumonia hospitalizations so an outcomes researcher should incorporate it into the statistical model. The same is true for dominant strain of the virus. Certain strains of influenza (influenza A[H3N2] in particular) are more virulent than others. This substantially affects hospitalizations in a given year, but has little effect on vaccination because the vast majority of vaccinations is given before the dominant strain of the virus is known for certain.
34411_CH03_059_082
9/9/05
9:38 AM
Page 79
Example: The Effectiveness of Influenza Vaccination
79
Figure 3–4 also suggests some variables that may serve as instrumental variables. Studies have shown that persons with certain conditions, such as ulcers and chronic dermatological conditions, are more likely to get a flu shot not because they feel they are more susceptible to flu, but because these conditions put them in constant contact with their physicians. A person who goes to his or her physician during weeks when flu shots are being administered is more likely to be reminded to get a flu shot. These variables are correlated with getting a flu shot but not correlated with hospitalization for pneumonia, except potentially through the flu shot itself, and so fit the definition of an instrumental variable.
Statistical Model An appropriate analytical model for this conceptual model is described by the equation: Hospitalizationi,t f(Flu Shoti,t, Demographicsi, Comorbiditiesi,t, Medigapi,t, VirusStraint)
The subscripts i and t refer to patient i in flu season t; that is, there are potentially several observations per beneficiary. The probability of hospitalization is observed for each person as either a zero if the person was not hospitalized in year t or a 1 if the person was. Because a hospitalization is a discrete event, this equation is best estimated by a logit or a probit model. The odds ratio on the variable flu shot gives the effect of the treatment. In this case, the treatment effect is expressed in terms of a reduction in the odds of hospitalization that is attributable to vaccination. Before estimating the equation, researchers should ensure that the separation in independent variables is not too great. For example, if vaccinees tended to be very old and nonvaccinees very young, then a researcher would have a difficult time estimating the effectiveness of vaccination that is independent of age. An appropriate strategy would be to stratify the analysis by age, and test for a vaccine effect in separate samples of older beneficiaries and younger beneficiaries. If vacinees and nonvacinees tend to be different across many characteristics, then propensity score matching would be warranted. Because there are multiple observations per person, the observations are not all independent. Observations on the same person are probably correlated because each person has an underlying propensity to contract
34411_CH03_059_082
9/9/05
CHAPTER 3
80
9:38 AM
Page 80
ISOLATING THE EFFECTS OF TREATMENT
influenza that does not change from one year to the next and is not captured by the other variables in the model. As a result, a simple logistic model that assumes all observations are independent might underestimate the p-value on the flu shot odds ratio. Estimating the equation using generalized estimating equations is one solution to this problem. An instrumental variable approach to estimating treatment effect would entail estimating two equations: a treatment equation and an outcome equation. Treatment equation: FluShoti,t f(Instrumentsi,t, Demographicsi, Comorbiditiesi,t, Medigapi,t, VirusStraint)
Outcome equation: Hospitalizationi,t f(Prob(FluShoti,t), Demographicsi, Comorbiditiesi,t, Medigapi,t, VirusStraint)
In the treatment equation, the probability of getting a flu shot is estimated as a function of all the explanatory variables in the first equation plus the instruments. As just described, instruments for this analysis are indicators for dermatological conditions and ulcers, which are correlated with vaccine use, but not with the underlying probability of contracting pneumonia. The predicted probability of receiving a flu shot is then calculated from the results of the treatment equation and is used in place of actual flu shot use in the outcome equation; that is, in the outcome equation, hospitalization is estimated as a function of each of the right hand variables in the second equation, but the variable for flu shot use is replaced with the predicted probability from the treatment equation.
SUMMARY This chapter provides a framework for assessing the effect of treatments in health care outcomes research. As discussed, the first step is constructing a conceptual model that details the relationships between major concepts in the outcomes study. The next step is to represent that conceptual model in the form of an analytic or statistical model. In order to report an unbiased treatment effect, researchers must strive to create treatment and control groups that differ only by the treatment in question. Two ways of doing this were discussed: randomized experiments and nonrandomized, statistical analyses.
34411_CH03_059_082
9/9/05
9:38 AM
Page 81
Notes
81
This chapter also includes a discussion of the two goals of a statistical analysis: estimation and inference. The concern of estimation is to derive an unbiased estimate of the treatment effect. The concern of inference is to assess the degree of statistical uncertainty that one has in that estimated treatment effect. Potential pitfalls to achieving these goals were also discussed. The next chapters will elaborate on some of these issues. REFERENCES Adler, G. (1994). A profile of the Medicare Current Beneficiary Survey. Health Care Financing Review, 15, 153–163. Becker, M.H. Ed. (1974). The health belief model and personal health behavior. Health Education Monograph, 2(4), 324–473. Chalmers, T.G., Celano, P., Sacks, H.S., & Smith, H.J. (1983). Bias in treatment assignment in controlled clinical trials. New England Journal of Medicine, 309, 977–986. Diggle, P.J., Liang, K.Y., & Zeger, S.L. (1994). Analysis of longitudinal data. Oxford: Oxford University Press. Hirsch, R.P., & Riegelman, R.K. (1996). Statistical operations—Analysis of health research data. Cambridge, MA: Blackwell Science. Kennedy, P. (1993). Guide to econometrics. Cambridge, MA: MIT Press. Localio, A.R., Berlin, J.A., Ten Have, T.R., & Kimmel, S.E. (2001). Adjustments for center in multicenter studies: An overview. Annals of Internal Medicine, 135(2), 112–123. Luepker, R.V., Murray, D.M., Jacobs, D.R., Jr., Mittelmark, M.B., Bracht, N., Carlaw, R., Crow, R., Elmer, P., Finnegan, J., Folsom, A.R., et al. (1994). Community education for cardiovascular disease prevention: Risk factor changes in the Minnesota Heart Health Program. American Journal of Public Health, 84(9), 1383–1393. McClellan, M., McNeil, B.J., & Newhouse, J.P. (1994). Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables. Journal of the American Medical Association, 272(11), 859–866. Murray, D.M. (1998). Design and analysis of group-randomized trials. New York: Oxford University Press. Newhouse, J.P., & McClellan, M. (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health, 19, 17–34. Rubin, D.B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2, 169–188. Wooldridge, J.M. (2002). Econometric analysis of cross sectional and panel data. Cambridge, MA: MIT Press.
NOTES 1. Even here, issues of dosage and timing may be relevant.
34411_CH03_059_082
9/9/05
9:38 AM
Page 82
34411_CH04_083_120
9/9/05
9:42 AM
Page 83
4 Measurement Jennifer R. Frytak Robert L. Kane
INTRODUCTION Measurement is fundamental to conducting health outcomes research. The ability to analyze clinical, social, and economic observations requires some method of quantifying them. Despite measurement’s long and frequent use, poor measurement has been an ongoing problem in conducting outcomes research. Measurement involves abstracting reality. The goal is to distort as little as possible while gaining information in a form that will be analytically useful. Measurement of variables of interest is linked to all steps of the scientific process: conceptualization of the study, analysis of the data, and interpretation of the results. One’s measures are only as good as one’s conceptualization of the concept; one’s data is only as good as one’s measures; and one’s results are only as good as one’s data. Designing an outcomes study involves two critical steps: (1) conceptualization and (2) operationalization of the variables of interest. The conceptual model can be based on prior empirical work or driven by theory. The conceptual model should display the relationships between the outcomes of interest and the factors expected to influence these outcomes. The model will drive the analysis plan. The model becomes a road map for both the data collection and the analysis phases of the study. It is a useful tool to organize concepts, hypotheses, timing of interventions, and so forth, as well as to clarify the purpose of the study. 83
34411_CH04_083_120
84
9/9/05
CHAPTER 4
9:42 AM
Page 84
MEASUREMENT
In the second step, measurement is used to quantify the conceptual model. The operationalization process involves specifying the assumptions that translate theoretical/conceptual variables into empirical ones (that is, measurement). When specifying these assumptions, it is useful to have a conceptual model of the variable one is trying to measure, especially if the variable is complex. For example, different conceptual models of healthrelated quality of life result in different dimensions of health being measured. The investigator must describe how each variable in the study will be collected (e.g., survey, claims data, record review), the level of aggregation for each variable, and whether existing measures will be used or new ones will need to be developed. Achieving a good balance between conceptualization and measurement is a challenge in any study. The two steps are necessarily linked, but there is a risk of focusing too heavily on one step and neglecting the other. On the one hand, an investigator may develop a complex theory on the role of psychosocial factors in achieving desirable outcomes for patients with congestive heart failure, but then rely on easily available proxy variables for psychosocial factors to test the hypotheses. In this scenario, if the proxy variables do not measure well what they are supposed to measure, the investigator will not be able to provide adequate evidence to test the theory. On the other hand, investigators often assume that one can measure anything that one can name without much thought to an underlying conceptual model for developing the measure. Several different investigators might develop substantively different measures of disability, but call the topic by the same name. It is easy to imagine the potential problems of comparability and generalizability that could arise. The level of sophistication in measuring concepts grows as one understands them better. In practice, one may be reduced to using crude measures to first address a particular problem such as analyzing an unanticipated insight from one’s data set. Overall, a good rule of thumb is that a researcher should not attempt to measure something that he or she cannot conceptualize nor should one use a measure that cannot be analyzed correctly.
THE NATURE OF MEASUREMENT Measurement is an easy concept to visualize, but a difficult one to define simply. Commonly, it is defined as the assignment of numbers to charac-
34411_CH04_083_120
9/9/05
9:42 AM
Page 85
The Nature of Measurement
85
teristics of objects, events, or people according to rules (Stevens, 1981). A researcher attempts to map numbers onto a set of objects so that the numbers represent the underlying relationships between the objects (Pedhazur & Schmelkin, 1991). For example, a ruler is used to measure the lengths of two pieces of string. The ruler assigns a number to each length, say 4 inches and 8 inches. Using this method, it is possible to state numerically that one string is half the length of the other. This provides a numeric description of the attribute. Imagine how difficult it would be in clinical or any other research if one had to rely strictly on verbal descriptions of characteristics of objects as a means for comparisons among the objects. When choosing existing measures or developing new ones, it is important to keep in mind that all measurement is imperfect for two reasons: Anything we measure is an abstraction of reality. Anything we measure is measured with error.1 One can never measure anything directly. Instead, particular features or attributes of an object, person, or phenomenon are measured (Nunnally & Bernstein, 1994). For example, a reseacher does not measure an elderly woman, per se, but may measure her functional disability. Recognizing that only selected attributes are measured emphasizes that any measurement is an abstraction from the whole person in this case and from “reality.” Realistically, one often ends up sacrificing richness of a construct for a quantifiable and generalizable measure. Other than her ability to function, little information about this elderly woman is captured. In fact, even the information on functioning may be incomplete because possible barriers and positive influences in her environment may dramatically affect what she can do and these are unknown to us. Also, because one is constrained to using manmade instruments and protocols to measure attributes of an object or person, all measurements are subject to error even in the most controlled situations. Inherently, measurement is an indirect process because researchers are confined to their senses (or at most extensions of them) (Blalock & Hubert, 1982). Historically, clinicians have been most confident measuring observable physical attributes of patients. However, in the current environment of outcomes research, many of the attributes of interest are not directly observable (e.g., quality of life and patient satisfaction). Phenomena such as quality of life, functional disability, depression, and even diseases like coronary artery disease, diabetes, or Alzheimer’s disease can be thought of as latent constructs. Latent constructs are abstract ideas formulated by the
34411_CH04_083_120
86
9/9/05
CHAPTER 4
9:42 AM
Page 86
MEASUREMENT
clinician or investigator to explain certain phenomena that are observed in a clinical setting or elsewhere. Measurement of latent constructs is even more indirect than measuring observable attributes, because the clinician will never be able to confirm absolutely the former’s existence. For example, a clinician has confidence in a given diagnosis only to the degree that the patient’s symptoms are consistent with some past experience with the disease. Similarly, the clinician’s confidence in measuring body temperature with a thermometer is based on a past experience of accurate readings with the given thermometer and his competence in using the thermometer. Physical attributes are often measured as indicators of a latent construct such as disease. For example, blood sugar levels are associated with a complex series of metabolic processes that constitute diabetes mellitus. (Even blood sugar measurements are not direct; they rely on indirect chemical reactions that produce color changes.) Everyday clinical practice relies on many latent constructs and involves the measurement of both observable and unobservable phenomena. Measuring the magnitude of latent constructs often involves constructing a scale. One assumes that the strength of the latent construct in each individual causes each item in a scale to take on a given value (DeVellis, 1991). An individual who has trouble functioning independently should score higher on a functional disability scale than a person who has no trouble functioning. Measuring observable physical attributes relies on a properly calibrated medical instrument or machine such as a thermometer, a blood pressure cuff, and so forth. It is common when measuring physical properties—both physical attributes solely or physical attributes as indicators of a latent construct—to gain a false sense of security that one is accurately measuring what one intends to measure. Clinicians and investigators are more wary of a measurement based on scales. For example, they may have less confidence in a level of depression obtained by the Geriatric Depression Scale than the clinical conclusions of a trained psychiatrist. However, if one considers all the opportunities for conceptualization and measurement error when measuring either observable or unobservable phenomena, accepting any type of measurement involves taking a leap of faith. Consider the following example. Measuring blood pressure generally involves many approximations. The force used to constrict an arm is considered to reflect the pressure in a blood vessel. Careful study suggests that this relationship can be influenced by a number of factors. For example, the circumference of the arm affects the force needed to constrict the artery. Different cuff sizes influence this rela-
34411_CH04_083_120
9/9/05
9:42 AM
Page 87
Measurement Issues
87
tionship. The circumstances around making the blood pressure determinations can dramatically influence results. Some people may experience higher blood pressures because of the anxiety induced by a visit to the doctor. Dramatic improvements can occur simply by having someone else take the measurement in a more informal setting. In truth, blood pressure determinations are samples of a parameter that varies widely. Careful studies of blood pressure measured periodically over a 24-hour period show wide fluctuations for the same person. Clearly, the single determination of blood pressure at a given point in time is but a pale approximation of the complex physical phenomenon it represents.
MEASUREMENT ISSUES Measurement is a complex task and several issues need to be considered when either choosing a measurement scale to include in a study or developing a new scale. Consider the construct of functional disability and handicap as an illustrative example to demonstrate the variety of measurement approaches and issues investigators have tackled to capture the idea of functional disability. Scales of this type are generally described as activities of daily living (ADL) scales and instrumental activities of daily living (IADL) scales. ADLs address basic tasks that are needed for independent living; IADLs deal with somewhat more complex tasks needed to maintain an independent lifestyle in the community. No single approach to the measurement of any construct is universally accepted. This is evident from the sheer number of ADL and IADL scales available for use (see Chapter 5). McDowell and Newell reviewed over 50 ADL scales and chose to include six in their book (McDowell & Newell, 1996). These scales differ in a number of dimensions: content, metric, method of scoring, length, method of administration, and rigor of determining reliability and validity. Given these circumstances, these scales would not likely yield similar scores of disability or be easily comparable.
Conceptualization of the Measure First, determine clearly what construct to measure and how the information will be used. Theory and a conceptual model can serve as guides to decide how broad or narrow the measure needs to be in several respects.
34411_CH04_083_120
88
9/9/05
CHAPTER 4
9:42 AM
Page 88
MEASUREMENT
1. Is the intended measure generic or disease-specific? (For a complete discussion of this issue, see Chapters 5 and 6 on generic and diseasespecific outcome measures, respectively.) For instance, an investigator may be interested in disability in a nursing home population. Initially, it seems that a generic measure may serve the study well. However, if the study question specifically deals with disability of arthritis patients in nursing homes, a disease-specific measure may also be considered. 2. What is the scope of the measure in terms of content? The content of different disability indices varies widely. The following topics have appeared in disability scales: self-care, mobility, travel, body movement, home management, medical condition, senses, mental capacity, work, resources, social interaction, hobbies, communication, and behavior problems (McDowell & Newell, 1996). One could choose several of these items or all of them. 3. What range should the measure cover? The wider the range is, the less measurement precision. Think of a microscope with several lenses. One is set for low power and provides an overview of the field but little detail. The high-power lens enables one to study the structure of a cell but only over a small area. How much detail is required may depend, in part, on who is asking the question. A geriatrician might do well to ask about a series of ADLs, but an occupational or physical therapist would break down each of these ADLs into multiple subcomponents and check each one. 4. What level of precision is needed from the measure for the study? This question often involves both responsiveness to change and concerns about floor and ceiling effects. For example, using the functional disability questions from the Sickness Impact Profile (SIP) to capture changes in functioning in a well elderly population may be a mistake because the SIP has been shown to differentiate well between those with poor health but not well between states in basically healthy older adults (Andresen et al., 1995). This represents a ceiling effect, an inability to demonstrate improvement. Similarly, the ADL scale would be a bad choice to measure functioning in a well population, because it generally measures a range of very low functioning. A floor effect is an inability to demonstrate deterioration (i.e., measure a trait below a given range). Well-articulated goals for the chosen measure will help ensure the quality of the data obtained. For example, in terms of disability measures, it
34411_CH04_083_120
9/9/05
9:42 AM
Page 89
Measurement Issues
89
should be clear whether one is interested in what an individual does do versus what an individual can do. The nature of the data collection will also influence the findings. Actual structured observations are distinct from reports, either self-reports or proxy reports, and these in turn differ from testing performance under structured circumstances. Likewise, specific scales may be conceptually inappropriate for certain populations. A scale used for assessing depression in younger adults may not be appropriate for assessing depression in the elderly and vice versa. Questions about physical symptoms related to depression in adults may reflect normative health changes in the elderly. Scales developed to measure disability in a younger developmentally disabled population may include items addressing workforce issues that are not applicable to the retired elderly. Too often scales take on an artificial reality after they have been in use for a while and people assume that they are applicable to any study. Finally, it is important to be aware of the analysis requirements of the study before data collection of the measures is undertaken. The nature of the data will direct the mode of analysis. For example, categorical data will be analyzed through some variation of nonparametric test, whereas continuous data will be analyzed with parametric tests. Some data can be used in both forms. The choice of analysis depends on what is believed to be happening. Parametric tests usually assume some form of constant relationship, whereas nonparametric analyses enable the researcher to look at specific points of change. For example, one could treat age or weight as a continuous variable and examine the relationship with the overall value or one could examine the effects among specific subgroups. The data source for each variable and any problems with the quality or availability of the data should be clear in advance. It is useful to make dummy tables that lay out how each variable will be used in the analysis phase. The format of the tables will vary depending on whether the variables are continuous or categorical. Striking a balance between measures that are clinically attractive and psychometrically sound can be difficult. Many clinically developed measures use an approach that makes good clinical intuitive sense but may violate the basic principles of psychometrics. For example, consider a widely used measure in assessing the effects of spine surgery, the Oswestry Personal Care measure (Fairbank, Couper, Davies, & O’Brien, 1980). Table 4–1 illustrates one item from that scale; other scale items have a similar design. This item addresses self-care and appears to be in some order of severity. Each level of personal care is assigned a point value that is summed, implying that the scale is continuous. However, closer inspection
34411_CH04_083_120
90
9/9/05
CHAPTER 4
9:42 AM
Page 90
MEASUREMENT
Table 4–1 Sample Item from the Oswestry Personal Care Measure 0 I can look after myself normally without causing extra pain. 1 I can look after myself normally but it causes extra pain. 2 It is painful to look after myself and I am slow and careful. 3 I need some help but manage most of my personal care. 4 I need help every day in most aspects of self care. 5 I do not get dressed, wash with difficulty, and stay in bed.
reveals several problems. The first several levels seem to follow a progression from no problem to pain to slowness and pain. The next level is needing some help and then needing more help. The last level involves inability to perform specific tasks, including staying in bed rather than getting up. Thus, within a single item of a measure there are several major shifts in focus and metric by which to assess the level of disability.
The Information Source When choosing an already developed scale, certain issues must be considered. In some cases, there may be a choice between using a report and making a direct measure of performance. The former usually addresses what is typically done, whereas the latter tests performance under controlled conditions (Hoeymans et al., 1996). Measures taken off the shelf have been developed for a specific information source with particular respondents in mind. The respondent may be the patient, the clinician, or some other proxy. Usually the decision is based on who will be the most accurate source of information for the characteristic of interest. The effects of relying on proxies vary with the type of information sought. Proxies cannot be expected to have insight into all areas, such as pain or quality of life. Most items from a medical history can be reliably obtained from proxies (assuming the proxy has had sufficient opportunity to observe the patient), but some cannot (Weiss et al., 1996). With respect to physical functioning, clinicians tend to offer less optimistic reports of functioning than their patients (Janse et al., 2004). The nature of the relationship between the proxy and the patient may affect the accuracy of the
34411_CH04_083_120
9/9/05
9:42 AM
Page 91
Measurement Issues
91
report even beyond the extent of familiarity. Family members may offer a different perspective from staff (Kane et al., in press). Historically, medicine has overlooked patient input and relied heavily on physician judgment. Ironically, in many cases physicians get their information from the patients. Passing the information through an extra filter may make it more or less accurate. Research has shown that proxies of elderly individuals tend to report lower functioning than the elderly themselves (Magaziner, Simonsick, Kashner, & Hebel, 1988; Rubenstein, Schairer, Wieland, & Kane, 1984). Each type of respondent has its own perspective; it is important to be aware of the characteristics of the information source that may bias the results or inhibit getting the necessary information. Using a standardized form loses the richness of information obtained. A general depression scale does not capture all the unique details of a given patient’s situation. This loss is offset by the ability to quantify and compare information across patients. A certain level of cautious skepticism about established measures is useful. Many established scales have been developed on the basis of expert opinion. Although they are clinically intuitive and hence attractive to clinicians, they may not have been subjected to psychometric analysis. It is possible that they are building on past misconceptions rather than measuring what was intended.
Mode of Data Collection The mode of administration can affect the results of surveys. The two primary methods of survey administration are interviewer-administered questionnaires (IAQ) and self-administered questionnaires (SAQ). Interviews are usually administered face to face or by telephone. Historically, selfadministered questionnaires were conducted either by mail or using a dropoff technique (giving someone a questionnaire in the clinic to fill out). Alternative methods for SAQ used more and more, include Web based, e-mail, kiosk (personal computer set up in clinic), and the like. Most research on the measurement error associated with mode of administration has focused on differences between mail, telephone, and face-toface surveys (Biemer, Groves, Lyberg, Mathiowetz, & Sudman, 1991; Dillman & Tarnai, 1988; Groves, 1989), but research is starting to evaluate alternative forms of SAQ (Dillman, 1999). Research on mode of survey administration has demonstrated effects on the measurement of attitudes and perceptions (Lynn, 1998) as well as behavior (Rockwood, Sangster, &
34411_CH04_083_120
92
9/9/05
CHAPTER 4
9:42 AM
Page 92
MEASUREMENT
Dillman, 1997). Health-related issues are not immune to this effect, including symptom report (Epstein, Barker, & Kroutil, 2001), impact of treatment (Fowler et al., 1998), and satisfaction (Fowler, Gallagher, & Nedrend, 1999). Effects on the basic psychometric properties of established scales have also been demonstrated (Rockwood, Kane, & Lowry, 1999). Social desirability in response is probably the most researched and best understood effect associated with mode of administration. Social desirability occurs when respondents alter their responses to conform to social norms and expectations (Groves, 1989). These effects can be both positive and negative; that is, individuals will underreport “negative issues,” such as illicit drug use (Aquilino, 1994; Aquilino & Sciuto, 1990) and tend to overreport “positive issues,” such as health status (Fowler et al., 1998). Research to date tends to converge on a couple of key findings. First, SAQ methods of administration, either mail or more complex methods such as Audio Computer Assisted Self Interviewing (ACASI), tend to have less error than any other methods of administration (Schwarz, Strack, Hippler, & Bishop, 1991; Tourangeau & Smith, 1996). Second, a face-to-face interview can produce good measurement (Aquilino, 1994; Aquilino & Sciuto, 1990), but it is not as consistent as self-administered methods (Tourangeau & Smith, 1996). Finally, the telephone mode is most susceptible to social desirability effects (Aquilino, 1994; Dillman & Tarnai, 1988; Groves, et al., 1988) and should be avoided when asking about sensitive issues (Fowler et al., 1998). A range of context effects occurs in survey methods. Context effects refer to how the response to a question may be affected by where or how it is placed in the questionnaire. Some effects associated with measurement in survey methods occur regardless of mode. Measurement error can result from the order of questions, response categories, and wording (Schwarz & Sudman, 1992). For example, in asking about happiness in life and marital happiness, the order in which one asks the items can significantly alter the response (Schuman & Presser, 1996; Schwarz & Hippler, 1995). Likewise, one can alter the frequency with which a behavior is reported to occur by altering the response categories (Schwarz, Bless, Bohner, Harlacher, & Kellenbenz, 1991; Schwarz, Knauper, Hippler, Noelle-Neumann, & Clark, 1991). These effects have been demonstrated within mode, but their manifestation also varies across modes (Dillman, Sangster, Tarnai, & Rockwood, 1996; Schwarz, Strack, Hoppler, & Bishop, 1991). For example, the effect of response alternatives on the reporting of behavior is not as strong in the mail mode as it is in the telephone mode (Rockwood et al., 1997). This interaction further complicates issues associated with mode of administration. These issues become
34411_CH04_083_120
9/9/05
9:42 AM
Page 93
Measurement Issues
93
especially important when mixed (i.e., more than one type of administration) modes of survey administration are considered for use in a study. Two basic forces have fostered mixed modes of survey administration: cost and response rates. Reducing the costs associated with data collection is becoming more necessary. A cheaper method of collecting data, such as the mail mode, is often used first with a more expensive method, such as telephone, used for the nonrespondents (Dillman & Tarnai, 1988). Another reason is reduction of nonresponse error or alternatively increasing the response rate (Groves & Couper, 1998; Groves, Dillman, Eltinge, & Little, 2002). Because some individuals are more likely to respond to one mode versus another, utilizing multimodes can significantly increase the response rate. Either of these instances requires an implicit trade-off between measurement error and nonresponse error (Groves, 1989; Groves et al., 2002). Historically, researchers have tended to opt for reducing nonresponse error; however, recent research suggests that measurement error in general and specifically error that accompanies the use of mixed-mode surveys to reduce nonresponse error could actually be introducing more error (Martin, 2004; Groves et al., 2002). A final note about measurement concerns surveys conducted at multiple points in time. For example, outcomes research participants are often recruited in clinic and are asked to complete a baseline survey at that point in time; then a subsequent follow-up survey is conducted using the telephone mode. Such designs, although making it generally easier to recruit subjects and conduct the study, should be carefully considered prior to use. An example from the authors’ own experience illustrates the problem. We were approached to help an investigator who had conducted an outcomes study in which patients were recruited and initially surveyed over the telephone and then were later surveyed by mail. In addition to the survey, the investigator had collected clinical data (functional, diagnostic, as well as assessment). The clinical data demonstrated a significant improvement in function, disease state as well as clinical assessment, but the survey data (SF-36) demonstrated a significant reduction in health-related quality of life (HRQoL). The investigator was seeking help in explaining the finding in the data. Based upon additional data collection, the decrease in HRQoL was shown to be attributable to using the telephone mode first and the mail mode second. Indeed, the HRQoL had actually improved. Fortunately, this investigator had collected clinical as well as self-reported data. Had the researcher relied solely on self-report data, the investigator would have drawn the wrong conclusion from the research.
34411_CH04_083_120
94
9/9/05
CHAPTER 4
9:42 AM
Page 94
MEASUREMENT
SCALING Once researchers understand what variables they are interested in studying, they must decide how the variable will be quantified. The term scaling refers to how one assigns numbers to the characteristics of the objects being measured. Each variable in the study must be scaled (see Table 4–2). Some variables are scales that consist of a single item, whereas other often more complex variables are scales that consist of an aggregation of multiple items. For a single-item scale or for each single item in a multiple-item scale, the response set must be scaled. The response set is the choice set that the respondent has for answering a given question. The response set determines the type of scale. With multiple-item scales, both the choice of response set and aggregation of the individual items determine the type of scale. Typically, one thinks of variables as either categorical or continuous. This is a helpful dichotomy to apply to scales, because the type of scale
Table 4–2 Types of Measurement Scales Scale
Aggregation
Example of Response Set
Nominal
Categorical
Gender: female 1, male 0
Ordinal
Categorical
Can you dress and undress? 1. yes, without difficulty 2. yes, with minor difficulty 3. yes, with major difficulty 4. no, not able to
Interval
Continuous
How much difficulty do you have dressing and undressing? (put an x at the point that best represents your level of difficulty) No Difficulty
Ratio
Continuous
Unable to Dress Alone
5 feet 60 inches ——— ————— 4 feet 48 inches
34411_CH04_083_120
9/9/05
9:42 AM
Page 95
Scaling
95
will determine the eventual statistical procedures that can be used with the scale. The following are four types of measurement scales (Stevens, 1981): 1. A nominal measure has a categorical response set with no particular ordering. Numbers are used as labels in order to classify objects into distinct categories; all objects can fit into only one category and every object must fit into a category. Classification is prevalent in outcomes research. Examples of nominal measures include gender, race, hospital type, and diagnosis-related groups (DRGs). A simple yes/no question would be a nominal level measure. 2. An ordinal measure has an ordered categorical response set. Numbers are used to represent a rank ordering between a set of objects given a particular attribute. Distances between the objects are not equal and cannot be meaningfully interpreted. An example of an ordinal measure would be an ADL question on dressing. Can you dress and undress? 1. Yes, without difficulty 2. Yes, with minor difficulty 3. Yes, with major difficulty 4. No, not able to The distance between minor difficulty and no difficulty is not necessarily the same as the distance between minor and major difficulty. 3. An interval measure has a continuous response set. The distance between each category is assumed to be the same (i.e., the numbers can be meaningfully interpreted). The Fahrenheit temperature scale is an interval scale. A more clinically oriented example of an interval level scale would be a visual analog scale. To obtain an interval level response, one could have respondents rate their level of difficulty dressing and undressing on a scale from 0 to 10. No difficulty would correspond to 0 and the inability to dress oneself alone would score a 10. Respondents would be told that distances between successive categories are equal. 4. A ratio measure is an interval measure with a true zero. The absolute temperature scale is a ratio scale. A clinical example of a ratio measure would be height. There is a true zero and when the unit of measurement changes, the ratio still stays the same. A person who is 5 feet tall is 1.25 times taller than one who is 4 feet tall. The ratio of
34411_CH04_083_120
96
9/9/05
CHAPTER 4
9:42 AM
Page 96
MEASUREMENT
these two heights stays the same when converted to inches. Magnitude estimation is a process believed to yield ratio level scores. Suppose bathing is assigned as the benchmark ADL with a score of 500. Raters would then assign a weight of 250 to an ADL that contributed half as much to overall dependency. The choice of measure implies the range of acceptable statistics to use with the measure. Parametric statistics (regression, averages, etc.) are inappropriate for nominal and ordinal (categorical) measures but appropriate for interval and ratio level continuous measures. Nominal and ordinal measures should be analyzed using nonparametric statistics. Table 4–1 shows the relationships among these types of measures. A good rule of thumb is to collect continuous data or at least not dichotomous data as often as possible. One can convert from continuous to categorical forms, but not easily the reverse. Such conversions to categorical data, although relying on less powerful statistics, may allow one to look for specific points of impact. For example, overall age may not show an effect, but the effect may lie in one particular subgroup (e.g., 85). For some analyses, a mean change is less meaningful than the proportion that moves from one category to the other, but it is easier to categorize or establish a cutoff point post hoc. Dichotomous measures require larger sample sizes to demonstrate an effect and are less reliable than continuous measures (Streiner & Norman, 1995). To measure the magnitude of continuous underlying attributes, several techniques are used: rating scales, comparative methods, magnitude estimation, and econometric methods. Using rating scales, the magnitude of an attribute is estimated by asking the respondent about the characteristic directly by using such tools as a visual analog scale or a Likert scale. Likert-type scales involve summing individual responses to a set of questions on an agree/disagree continuum (e.g., strongly agree, agree, disagree, strongly disagree) to calculate a score of disability. The response set is symmetric. Scale values are not assigned to individual items; only the total score is scaleable. Although individual items in the Likert scale are ordinal, total scores are treated as interval by most investigators without introducing substantial bias (Nunnally & Bernstein, 1994), as long as the scale is measuring only one underlying characteristic. Disputes over constructing the Likert scale arise around issues such as the appropriate number of
34411_CH04_083_120
9/9/05
9:42 AM
Page 97
Scaling
97
categories and whether there should an odd or even number of categories. The number of categories generally ranges from five to nine. Likert-based scaling methods are widely used, because they are inexpensive to design, easy to understand, and easy to administer. Comparative methods involve an initial scaling of items on an interval level scale (usually calibrating to a normal distribution) by a group of judges prior to obtaining actual responses. Items are scaled by either asking the judges to rank a large number of statements from most favorable to least favorable or to compare each item to each other item to distinguish which of the pair has more of the attribute in question. After the scale has been calibrated, a respondent chooses which statements apply; scores to individual items are aggregated by summing or averaging. These methods guarantee interval-level data, but scale construction is expensive, difficult, and does not guarantee unidimensionality of the construct or unbiased rankings by the judges (McIver & Carmines, 1981). Most of the early functional disability scales relied on another comparative method, the Guttman scaling approach. This type of scale is hierarchical and deterministic in nature, and unidimensionality of the construct of interest is required (Nunnally & Bernstein, 1994). Given the hierarchical nature of the scale, developmentally oriented constructs, such as ADLs, work best with this method of scaling. Katz and colleagues (1963) developed a scale of ADLs based on the assumption that one loses and gains functions in a certain order. A person who is dependent at functional level three, is necessarily dependent at levels one and two. Deterministic models of scaling assume no measurement error. Each score on the scale indicates the exact pattern of responses to the dependency questions. In practice, coefficients of stability and reproducibility are used to determine the degree of deviation from perfect ordering in the respondents of the sample. Finding constructs suitable for use with the Guttman method is difficult and the data obtained is ordinal. Even the ADL construct has proved problematic. Research conducted with a sample of Medicaid-eligible disabled elderly and a national sample of noninstitutionalized elderly shows that many hierarchical patterns of dependency other than the Katz method are possible (Lazaridis, Rudberg, Furner, & Cassel, 1994; Travis & McAuley, 1990). The common practice of using simple counts of ADLs assumes a hierarchy, equal weighting of each ADL in the count, and interval level properties. These assumptions are somewhat tenuous. In an effort to move beyond ordinal properties and equal weighting assumptions of standard measures, various scaling techniques have been used to weight health or dependency states. For example, magnitude estimation techniques were used to obtain a weighted ratio level scale of func-
34411_CH04_083_120
98
9/9/05
CHAPTER 4
9:42 AM
Page 98
MEASUREMENT
tional dependency (Finch et al., 1995). In magnitude estimation, a reference item from a given construct is chosen and given a scale value. All remaining items are rated numerically with respect to how similar or dissimilar each is from the reference item. Using bathing as the standard (500), an expert panel rated 13 function domains based on their judgment of how the need for human assistance to perform the function contributes to overall dependency. The panel then assigned weights from 0 to 100 to the level of assistance (e.g., a little assistance, a lot of assistance, complete assistance) needed to perform the functional activity. Because the scale exhibits ratio-level properties, a composite score is calculated. Finch et al. found that their scale was more sensitive to the nature and extent of functional losses than simple counts of ADLs and IADLs, because it did not assume equal weight and did not arbitrarily dichotomize level of dependency for the ADLs and IADLs. Cost-effectiveness analysis in health care has sparked the development of various utility measures of health care quality. (See Chapter 11 for a discussion of cost effectiveness.) A numerical value is assigned to health states with 1 for a healthy state and 0 for dead. Aggregate weights for each state can be determined by a variety of scaling methods, including rating scales and magnitude estimation, but economists favor the standard gamble or time trade-off methods that are based on economic theory (Kaplan, 1995). The standard gamble asks the respondent to choose between an outcome that is certain but less than ideal or gamble on an outcome that is uncertain but leads to either perfect health or death given set probabilities. The time trade-off method discards the difficult concept of probabilities and offers respondents a choice between perfect health for a set amount of time or less than perfect health for a variable amount of time. Economists argue that aggregation is more plausible using these methods in comparison to rating scales with ordinal-level properties. However, these methods assume rational decision making on the part of the respondents. Research suggests that this is not necessarily the case (Tversky & Kahneman, 1981). Moreover, these methods are difficult for respondents to understand and time-consuming to administer (Streiner & Norman, 1995).
Weighting/Aggregation For multi-item scales, once it is understood how the responses in the questions are scaled, one needs to think about how the items in the scale
34411_CH04_083_120
9/9/05
9:42 AM
Page 99
Scaling
99
are combined. A common way to combine the items is to add them. However, simply adding the items may obscure important assumptions; one is assuming the weights for each item as well as the metric. Adding up the scale items assumes that each item contributes equally to the total score. However, the choice of a response set may inadvertently weight the items. Assume that one response set is 1 2 3 4 5. Choosing a response set of 1 3 5 7 9 for a different item assumes that the highest response of the latter set is 9 times greater than the lowest response, whereas in the first set, it was only 5 times greater. Also, one should only aggregate items that measure a single underlying construct. Apples and oranges should not be added together. It is not appropriate to combine several different constructs into a single score. Inadvertent weighting is possible if different aspects of a single construct are measured with different numbers of items. For instance, if an overall measure of functional disability had three questions dealing with self-care and five questions dealing with mobility, mobility would be weighted more heavily in the overall score. In combining subscales, this unintentional weighting should be corrected by dividing the subscale score by the number of items in the subscale (i.e., using the averages of the subscales). Unintentional weighting in scales can also occur if items in a scale are very highly correlated. Thus, a high score on question 2 would lead to high scores on items highly correlated with question 2, giving certain aspects of the underlying construct extra weight. Careful consideration of the cut point in a response set is also important because it will dramatically affect the overall level of the construct measured in a given population. For example, suppose a researcher wants to do a simple count of disabilities and the response categories consist of the following: without help from someone, with a little help from someone, and with a lot of help from someone. Defining disability as doing something with a lot of help results in not capturing the part of the population who have moderate levels of disability. To weight a scale, one can derive the weights empirically or theoretically. Empirical weights are often the coefficients from a multiple regression of total score on the individual items in the scale—or the item-total correlations (scale item-total score) can be used as the weights. Generally, the regression weights are unstable across samples due to a large sampling error, and much evidence suggests that weighting makes little difference especially when scales contain at least 20 items (Nunnally & Bernstein, 1994). However, there is some evidence to the contrary if the number of items is small and the items in the scale are relatively heterogeneous
34411_CH04_083_120
100
9/9/05
CHAPTER 4
9:42 AM
Page 100
MEASUREMENT
(Streiner & Norman, 1995). Functional status measures may benefit from weighting given the heterogeneity of the items covered in these scales. Weighted or unweighted, once the total score has been calculated, it is important to be able to compare scores across similar scales. The best method is to transform the total score of each scale into a normalized standardized score (Streiner & Norman, 1995).
RELIABILITY AND VALIDITY An important part of selecting a measure for use in a study is establishing the usefulness of the measure. Traditionally, this process involves assessing the reliability and validity of a measure. Assessing reliability involves showing that measurement is performed in a reproducible manner. Assessing validity means measuring what one intended to measure. A scale that always overweighs patients by exactly 10 pounds is an example of a reliable scale but not a valid one. Conversely, if the scale randomly overweighed and underweighed patients, the scale would be unreliable and invalid. Reliability is a necessary, but not sufficient, condition for validity. As noted earlier, measurement is always subject to error. Measurement error lies at the heart of reliability and validity. One way to think about error in a measure is as random and nonrandom measurement error. Reliability is largely a function of random error that is completely unsystematic in nature. Random error could be caused by such things as keypunch and coding errors, ambiguous instructions, interviewer mistakes, or inconsistent responses. As random error in the measure increases, the reliability of the measure decreases. Validity is largely a function of nonrandom measurement error or systematic error. Nonrandom error generally occurs when a measure is measuring more than one construct or when the method of collecting the data systematically biases the results (Carmines & Zeller, 1979; Maruyama, in press). A common misperception is that once the reliability and validity of a measure have been established, they are no longer an issue when the measure is used in future studies. This is false because measurement error is inherent to a particular sample; thus, reliability and validity are artifacts of the given sample as well. Reliability is a function of the instrument and the user. Comparable to using a scalpel, a scale may perform well in one set of hands and not in another. Clinicians should at least ponder whether their studies vary from previous studies along various dimensions (e.g., the pop-
34411_CH04_083_120
9/9/05
9:42 AM
Page 101
Reliability and Validity
101
ulation of interest, the study setting, the method of data collection, and interviewer background) that might threaten the reliability and validity of the measures. Reliability Reliability is a measure of reproducibility and is solely an empirical issue. Reliability limits the ability of a measure to detect a difference. The amount of variance a measure can explain will never exceed its reliability. Common types of reliability are internal, test-retest, and interrater. One may be interested in reproducibility of individual test scores across similar items in a test, of individual test scores over time, and of individual test scores by different observers. Specifically, the investigator must have repeated measures across some dimension (e.g., multiple items measuring the same construct, multiple test scores over time for the same subjects, and multiple raters scoring the same patients). Fundamentally, reliability can be expressed as follows (Carmines & Zeller, 1979; Crocker & Algina, 1986; Streiner & Norman, 1995): Reliability
true variance true variance error variance
As the equation implies, the greater the amount of variance attributable to error, the lower the reliability. This reliability coefficient, a form of an intraclass correlation coefficient (ICC), is interpreted as the percentage of variance in a measure that results from true patient (or respondent) variability. The ICC provides information on the “extent to which repetition of the test yields the same values under the same conditions in the same individuals” (Guyatt, Walter, & Norman, 1987). For all reliability coefficients, a coefficient of 1 would indicate perfect reliability (i.e., no measurement error) and a coefficient of 0 represents no reliability. Because the ICC is based on analysis of variance (ANOVA) techniques, it is appropriately used only with continuous data. A common form of internal reliability used with scales is Chronbach’s alpha coefficient, which is a derivation of the ICC. (The Kuder-Richardson formula 20 is appropriate for dichotomous data [Kuder & Richardson, 1937]). This type of reliability is referred to as internal consistency (i.e., the items in the scale are homogeneous). The basic question posed is how closely each item in a scale is related to the overall scale. All variability
34411_CH04_083_120
102
9/9/05
CHAPTER 4
9:42 AM
Page 102
MEASUREMENT
other than error variance and item variance is treated as fixed because multiple observations occur due to the items in the scale rather than from multiple observers. The formula for Chronbach’s (1951) alpha coefficient is:
n n1
1
2i T2
n number of items in the scale i standard deviation of each scale item T standard deviation of the total scale score High values of alpha occur when the items comprising the total scale score are highly correlated. If the correlation between all possible pairs of items in the scale were zero, the scale would have zero reliability. However, one does not want perfect correlations, which would suggest that items are redundant. High levels of the alpha coefficient should be taken with a grain of salt, because they are sensitive to the number of items in the scale. By increasing the number of items in a scale, internal consistency seems to be increasing, but this may not actually be the case. It is possible that some of these scale items may be redundant or poor indicators of the attribute of interest. An adequate alpha level for scales used to compare groups is .70 to .80 (Nunnally & Bernstein, 1994). Examining internal consistency using the item-total correlation is also a popular method. Items in a scale that do not correlate highly with the total score minus that item are not desirable. Another form of reliability is test-retest reliability. In other words, the same test is given to the same individuals after a period of time elapses. The correlation between the scores from each test is considered the reliability of the measure. Reliability may be overestimated as a result of memory if the time interval between the tests is too short, or reliability may be underestimated if the time interval is too long and real changes within respondents occur. It is not possible to separate this form of reliability from the stability of the measure (Pedhazur & Schmelkin, 1991). Thus, a catch22 exists when dealing with stability of measurement and outcomes research because one is inherently measuring change. To quote Kane and Kane, “We cannot be certain about the degree that functional status fluctuates without reliable measurements, and we cannot readily test the measures for reliability without assuming some stability over time in the characteristic measured” (Kane et al., 1981).
34411_CH04_083_120
9/9/05
9:42 AM
Page 103
Reliability and Validity
103
Kappa () is a commonly used statistic to assess inter-rater agreement in situations with dichotomous outcomes (i.e., nominal scales) such as presence or absence of a condition when more than one observer/rater is used. The kappa moves beyond a measure of simple agreement by taking into account the proportion of responses that is expected by chance:
P1oPPe e
Po is the observed proportion of agreement (the addition of the diagonal elements of a 2 2 table divided by the total number of responses). Pe is the expected agreement between the two raters for the two outcomes given the marginal distributions of a 2 2 table. A hypothetical example is shown in Table 4–3. The expected agreement for presence of the condition is calculated as (80 100)/200 40 and the expected agreement for absence of the condition is calculated as (120 100)/200 60. The total expected agreement (Pe ) is calculated as (40 60)/200 .5. Kappa .1 because agreement is little better than what was expected by chance. The value is very sensitive to the marginals; if the marginals are not well balanced, kappa will be low even when agreement is high (Feinstein & Cicchetti, 1990). This measure is appropriate only for nominal data. However, attempts have been made to develop a weighted kappa for use with ordinal data. The weighted kappa focuses on disagreement, typically using quadratic weights calculated as the square of the amount of discrepancy from exact agreement (Streiner & Norman, 1995). Using Pearson’s correlation, percent agreement, and chi-square is not recommended for assessing inter-rater reliability (Bartko, 1991; Streiner & Norman, 1995). The classical test theory notion of reliability as a ratio of true variance to total variance is limiting, because it does not break down multiple
Table 4–3 Example of Interrater Reliability Rater 1 Presence of Rater 2
Absence of
Total
Presence of Absence of
45 35
55 65
100 100
Total
80
120
200
34411_CH04_083_120
104
9/9/05
CHAPTER 4
9:42 AM
Page 104
MEASUREMENT
sources of error. Essentially, each type of reliability coefficient just discussed yields a different estimate of reliability. For some studies with multiple potential sources of error, total variability may be seen more practically as the sum of all variability in the measure that includes both patient/respondent variability (between subject) and measurement error (within subject) and other forms of variability such as observer, site, training, and so forth, if the investigator deems it appropriate. Generalizability theory, an extension of analysis of variance (ANOVA) techniques, examines all measured sources of variance simultaneously and can address all types of reliability in a single study (Crocker & Algina, 1986). This technique has not been well utilized in the health sciences literature. This is the most appropriate approach to inter-rater reliability. When assessing reliability, it is important to keep in mind that it is a measure of reproducibility, not a measure of accuracy. It does not specifically deal with the issues of sensitivity and specificity. Sensitivity is the ability to detect those who have a disease, and specificity is the ability to detect those who do not. These determinations incorporate an established cut point for the characteristic of interest and the prevalence of the characteristic in the population. Generally, one must be traded one to get the other.
Validity Reliability is a prerequisite for validity, but establishing the latter poses great challenges. Validity lies at the heart of the measurement process, because it addresses whether the scale is measuring what it was intended to measure. However, validity is difficult to establish, because it is subject to many confounding factors. For example, abstract ideas turned into scales are subject to the definitional whims of the individual clinician or investigator (i.e., naming a factor score from a factor analysis). Also, the interview situation may affect whether one measures what is intended. An interview of a nursing home resident on the topic of quality of life in the nursing home conducted in the presence of the nursing home administrator may produce invalid results. Results could also be affected if the interview was conducted by phone or with a proxy rather than face to face with the resident. Given all the possibilities of bias, validity becomes a matter of degree; a perfectly valid indicator is not achievable. In practice, validity is a process of hypothesis testing that moves beyond operational definitions to demonstrate relationships between the mea-
34411_CH04_083_120
9/9/05
9:42 AM
Page 105
Reliability and Validity
105
sure of interest and other measures or observable physical properties. According to Streiner and Norman (1995), “Validating a scale is really a process whereby we determine the degree of confidence we can place on inferences we make about people based on their scores from that scale” (p. 146). Although all types of validity boil down to the same thing, a whole body of literature exists on the different types and complexities of validity. Validity is a unitary concept and types of validity are merely a useful tool for discussion. It is not simply a case of demonstrating one aspect in order to establish validity (Guion, 1980). Generally, the three main types of validity discussed are criterion-related validity, content validity, and construct validity. The investigator is responsible for determining the approach to assessing validity that best suits the construct of interest. Criterion-related validity is assessed by correlating the measure of interest with a gold standard or an already well established measure of the characteristic (the criterion). This correlation can be assessed concurrently to establish concurrent validity or the correlation can be assessed in the future to establish predictive validity. An example of predictive validity would be the development and use of a new section on the medical school entrance exam to predict an individual’s likelihood of pursuing primary care. One would not be able to know how well the section did at predicting the percentage of new primary care doctors until the class of medical school students graduates. Using the results of this exam section for admission or other type of decision prior to graduation will bias the correlation. Casemix adjusters are often validated on the basis of their ability to predict an outcome of concern such as death. In many of these cases, the data on both predictors and outcomes may already have been collected. The main problem with this type of validity is that the more abstract the concept is, the less likely it will be to find a criterion for assessing a measure of it. In clinical practice, screening tests often are validated using a more comprehensive diagnosis as a criterion score. Content validity assesses the degree to which the items in the measure cover the domain of interest. For example, how well does a specific ADL scale represent the entire range of disability? The more representative the sample of disability, the more valid one’s inferences will be. If the scale does not take into account the difficulty of dressing and bathing, one may obtain an inaccurate picture of the extent of an individual’s ability to function independently. The main problem with content validity is that in the health sciences, it is generally impossible to sample the entire domain due to the “fuzziness” of many of the concepts. For many measures, content
34411_CH04_083_120
106
9/9/05
CHAPTER 4
9:42 AM
Page 106
MEASUREMENT
validity reduces to a form of face validity, that is, the judgment by the medical or research community that the measure really measures the construct and perhaps even the judgment of the respondents that the questions in the measure make sense. Construct validity refers to the validity of measures of unobservable constructs. Because the measures cannot be observed and have no agreedupon criterion or content, construct validity is an exercise in hypothesis testing, in which the construct of interest is one of the variables. One is interested in whether the scores on the measure of the construct reflect the hypotheses about patient behavior. Some common methods of establishing construct validity follow. First, one can examine group differences on the measure. If one expects two groups to differ in a predicted manner on a measure, one can directly test this difference. Often investigators administer the scale to a group of individuals known to have the characteristic of interest and a group known not to have the characteristic. The group known to have the characteristic is expected to score higher on the measure than the other group. However, this method neglects the fact that in practice the scale will need to discriminate among individuals in the middle range of the trait (Streiner & Norman, 1995). Second, correlational studies help determine convergent and discriminant validity. If two scales are both supposed to measure functional disability (preferably using two different methods), they should be highly correlated; this is convergent validity. Two scales measuring different constructs, such as functional disability and mental health, should not be highly correlated; this is discriminant validity. Third, Campbell and Fiske (1959) propose a model for interpreting traits across methods. The multitrait multimethod matrix presents all of the intercorrelations resulting when each of several traits is measured by each of several methods. This method is based on the concept that reliability is the agreement between two efforts to measure the same trait through maximally similar methods. Validity is represented in the agreement between two attempts to measure the same trait using maximally different methods. See Campbell and Fiske (1959) for a complete description. Fourth, confirmatory factor analysis is used to determine whether or not the data is consistent with an underlying theoretical model (i.e., whether there is in fact a unique construct). This process of validation deals with the internal structure of the construct. Pedhazur and Schmelkin (1991) describe factor analysis as “a family of analytic techniques designed to identify fac-
34411_CH04_083_120
9/9/05
9:42 AM
Page 107
Reliability and Validity
107
tors, or dimensions, that underlie the relations among a set of observed variables. . . . The observed variables are the indicators (measures, items) presumed to reflect the construct (i.e., the factor).” (p 66) Construct validity is an ongoing process. (Ideally, construct validation requires a pattern of consistent findings involving different investigators using different theoretical structures across a number of different studies [Carmines & Zeller, 1979]). Unfortunately, the house of cards investigators build trying to validate a measure can be easily toppled by a few inconsistent findings.
Sensitivity to Change A goal of the medical community is to produce positive change in patient health status through appropriate treatment of disease. A goal of outcomes research is to assess the effectiveness of medical treatment. Necessarily, many outcome measures are expected to be able to measure change over time (Guyatt et al., 1987; Guyatt, Deyo, Charlson, Levine, & Mitchell, 1989; Kirshner & Guyatt, 1985). Besides reliability and validity, researchers argue that measures used to assess treatment effects must be responsive to changes in patient health status over time. This property is often called responsiveness or sensitivity to change in the literature. One can have reliable instruments that are not responsive and responsive instruments that are not reliable. For example, a repeated measure may give the same results every time, but it is unresponsive if it does not detect improvements in functioning that are known to have occurred. Good reliability only demonstrates that the measure adequately discriminates between individuals at a point in time. If the goal of measurement is to detect change due to treatment through group differences, a measure that reflects the responsiveness of the scale to change is needed. Death is usually quite reliable, but it may be insensitive in detecting change in arthritis. Assessing the effects of medical interventions often involves the use of change scores (i.e., mean change in the outcome variable over time). However, using the reliability of the change score as a measure of responsiveness is inappropriate. A uniform response to treatment would result in a change score reliability coefficient of zero, because the variance of the change score is zero if all patients improve an equal amount (Streiner &
34411_CH04_083_120
108
9/9/05
CHAPTER 4
9:42 AM
Page 108
MEASUREMENT
Norman, 1995). Guyatt, Walter and Norman (1987) suggest creating a ratio of the clinically important difference (if available) to the variability of scores within stable patients. If the clinically important difference is unavailable, the mean change score is used. Others correlate change scores on the scale to change scores of physiologic measures (Meenan et al., 1984) or use receiver-operating characteristic to determine the ability of the scale to detect clinically important change compared to some external criterion (Deyo & Centor, 1986). Using change scores for responsiveness seems to be an intuitive choice, because clinicians are concerned about health status before and after a medical intervention. Moreover, investigators argue that besides addressing responsiveness, the use of change scores is necessary to correct for baseline differences between the experimental and control group, particularly in nonrandomized trials. However, several researchers caution against the use of change scores. Norman (1989) argues that none of these methods establishes the statistical connection between reliability and responsiveness (expressed as a ratio of variances) that has been shown to exist (see Reliability section). Change scores should not be used unless the variance between subjects exceeds the error variance within subjects, or more practically, unless the reliability of the measure exceeds .5 (Norman, 1989). Responsiveness can be viewed as follows (Norman, 1989; Streiner & Norman, 1995): Responsiveness
variance due to change variance due to change error variance
A unitless measure of responsiveness results that expresses the proportion of variance in the change score due to true change resulting from the treatment. Nunnally and Bernstein (1994, p. 112) further caution that “the major problem in working directly with the change score is that they are ridden with a regression effect” (i.e., regression to the mean). Patients who were above the mean prior to treatment tend to have negative change scores (do worse) after treatment and patients who scored below the mean prior to treatment tend to have positive change scores (do better) due to random variation. To address this phenomenon, residualized change scores (i.e., difference between actual posttest score and posttest score predicted by a regression equation) are suggested for assessing individual change and analysis of covariance techniques for overall treatment effects (Chronbach & Furby, 1970; Streiner & Norman, 1995). Unfortunately, neither of these methods handles error effectively or performs well in quasi-experimental designs (Nunnally, 1975).
34411_CH04_083_120
9/9/05
9:42 AM
Page 109
Interpreting Change
109
INTERPRETING CHANGE Evidence of a clinical outcome needs to be interpreted in a clinical context. As noted in the first chapter, a conceptual model can prove useful in encouraging such thinking, but extra care must be taken in interpreting results. A good example of the measurement hierarchy that underlies such outcomes can be seen in the case of rheumatoid arthritis (RA) (Spiegel et al., 1986). RA patients can have acute flare-ups that are often reflected in clinical measures like joint counts (i.e., the number of swollen and inflamed joints) or laboratory tests like sedimentation rates. More functional measures include performance such as walk time and grip strength. Although these performance measures will be greatly affected by the acute status of the patient’s joints, they can also reflect other aspects of the disease. For example, joint deformities will limit the ability to perform either test, even if there is no acute exacerbation. A higher level of function is reflected in activities of daily living (ADLs). The ability to perform ADLs is the result of the patient’s acute status and the presence of joint deformities, but it is also influenced by other factors, such as mental attitude. A depressed patient may be less inclined to perform ADLs. Another way of thinking about the issues around interpreting measures is to consider the context. For example, a study of cataract surgery found that the procedure was associated with visual improvement; tests of visual acuity showed consistent improvement. However, not all patients who underwent the procedure showed corresponding functional improvement. On further examination, it was discovered that patients with substantial cognitive impairment failed to improve their functional ability (Elam et al., 1988). A reliable, valid, and responsive health status measure still has one final hurdle before it deserves widespread use in the clinical community. Changes in health status measures must be understood by clinicians and others (i.e., what constitutes a meaningful or clinically significant change on the measure). Treatments may be statistically significant without being clinically significant. This phenomenon occurs more frequently in large samples, because statistical significance can be achieved with extremely small differences between the treatment and control group. For example, a group of stroke patients without the use of an arm enroll in a trial to examine various forms of physical therapy. One of those therapies electrically stimulates and strengthens an individual’s muscles in the arm. Over time, a statistically significant improvement in the mean strength of that arm muscle can be detected between the treatment (n 5000) and control groups
34411_CH04_083_120
110
9/9/05
CHAPTER 4
9:42 AM
Page 110
MEASUREMENT
(n 5000). However, most of the patients do not have any substantial or clinically meaningful associated improvement in the ability to use the arm in daily living. Attempts at interpretation have either been distribution based or anchor based (Deyo & Patrick, 1995). Distribution-based interpretation typically involves the effect size. An effect size is the mean change in a variable divided by the standard deviation of that variable (Cohen, 1977). Kazis and colleagues (1989) believe that translation of changes in health status into a standard unit of measurement and using general effect–size thresholds provide a clearer understanding of clinical significance. General effect–size thresholds are 0.2—small effect, 0.5—moderate effect; and 0.8—large effect (Cohen, 1977). However, these classifications do not apply universally; because in some interventions, a 0.2 may be clinically significant and a 0.8 may not be clinically significant in others. To compare health status scores across instruments, some argue that the denominator of the effect size should be the standard deviation of the change in scores (Katz et al., 1992). Anchor-based interpretations tie the health status measure to external measures or events. Changes in score on quality-of-life instruments have been calibrated to patient global ratings of change with the effect that a 0.5 change on a 7-point Likert scale represented clinical significance (Jaeschke et al., 1989). Changes in mental health scores for a population of schizophrenics are related to the probability of subsequent major life events such as psychiatric hospitalization, arrest, victimization, and suicide attempt (Harman et al., in press). From a policy perspective, differences in mental health scores resulted in differential use of outpatient mental health services; individuals in the lowest third of scores (sickest) had three times the average expenditures as those in the highest third (Ware et al., 1984). Other examples of anchoring health status include tying to prognosis or predicting future events, using quality-adjusted life years, receiving a particular diagnosis, tying to traditional clinical classification systems (e.g., the New York Heart Association Functional Classes), and for (dichotomous outcomes) the number needed to treat to prevent one bad outcome (Deyo & Patrick, 1995).
ADVANTAGES OF MULTIPLE- VERSUS SINGLE-ITEM MEASURES Three good reasons to choose multiple-item scales over single questions are improved reliability, scope, and precision (Haley, McHorney, & Ware,
34411_CH04_083_120
9/9/05
9:42 AM
Page 111
Summary: Advantages of Multiple- Versus Single-Item Measures
111
1994; Nunnally & Bernstein, 1994; Spector, 1992). Single-item scales are not as reliable as multiple-item scales, because multiple-item scales average out measurement error in a construct when they are summed to obtain a total score. Generally, individual items have considerable measurement error, but it is not typically assessed. Second, in many clinical instances, a single item is inadequate to capture the complexity of a construct such as quality of life or depression. Finally, multi-item scales make it possible to discriminate more finely between the degrees of an attribute. Categorical response sets enable more discrimination than a simple yes/no question. For example, one can collect much more detailed information about dressing by asking multiple-category questions. One should ask whether an individual can (a) dress independently, (b) dress with limited help from assistive devices, (c) dress independently, taking excessive time, (d) dress with help from another person, or (e) is unable to dress him- or herself at all. Answers to these specific questions provide important information about the level of disability. Aggregated multiple-category questions measuring a single construct provide more precise answers.
Shortcomings of Traditional Multiple-Item Scales Multi-item scales are preferable to single variables for complex constructs, but several shortcomings of traditional test theory and scale construction make these scales less than perfect (Crocker & Algina, 1986; Hambleton & Swaminathan, 1985; Streiner & Norman, 1995). First, on such scales it is not possible to determine how people with different levels of a characteristic perform on a given item of the scale. Each scale item is assumed to tap the underlying characteristic identically. Second, scales are not generally test free or sample free. Individuals that complete different items on a scale cannot be compared (test free) nor can scales generally be used in different groups without reestablishing the psychometric properties (e.g., reliability and validity) of the scale (scale free). In theory, a scale constructed according to item response theory (IRT) will produce a scale that is sample and test free. The scale will be reproducible across diverse groups and over repeated tests. Psychometric properties will not need to be reestablished. Moreover, individuals completing different items on the scale can be easily scored and compared. For example, scores of impaired individuals completing the easiest 10 items on the scale could be compared to the scores of unimpaired individuals completing the most diffi-
34411_CH04_083_120
112
9/9/05
CHAPTER 4
9:42 AM
Page 112
MEASUREMENT
cult 7 items. This method of scaling is an extension of Guttman scaling. Thus, a hierarchical structure is assumed. Also, assumed is unidimensionality and local independence. In other words, the items in the scale can only be measuring a single underlying trait (unidimensionality) and the probability of answering item j positively is not related to the probability of answering item k positively (local independence). Local independence would be unmet if item k depends upon item j or earlier items. Health-related scale applications of IRT are limited but gaining popularity (Avlund, Kreiner, & SchultzLarson, 1993; Fisher & William, 1993; Haley et al., 1994). Rather than the deterministic method of Guttman scaling, which assumes no error, IRT is based on the probability of responding positively to a scale item based on the amount of the underlying trait that an individual possesses. The more traits an individual has, the more likely that individual will respond positively to the scale item. Specifically, it is the “probability that a randomly chosen member of a homogeneous group will respond correctly to an item” (Crocker & Algina, 1986, p. 341). Item characteristic curves (ICC) are plots of the relationship between the person’s performance on any item and the underlying trait (Streiner & Norman, 1995) (see Figure 4–1). The curves are S shaped and based on the logistic curve. Three logistic IRT models are in use and vary in their levels of complexity. The models vary on three parameters: (1) discrimination or steepness of slope (a), (2) difficulty or location along the trait continuum (b), and (3) guessing or where the bottom of the ICC flattens out (Streiner & Norman, 1995). In the one parameter model, the ICCs represent the proportion of individuals responding positively as a function of the latent trait and the Choice A (with certainty)
Wheelchair bound
p Choice B (with uncertainty) 1-p
Perfect health
Death
Figure 4–1 Standard Gamble Method. Source: Based on G. Torrance and D. Feeny, Utilities and Quality-Adjusted Life Years, International Journal of Technology Assessment in Health Care, Vol. 5, No. 4, pp. 559–575, © 1989.
34411_CH04_083_120
9/9/05
9:42 AM
Page 113
Summary: Advantages of Multiple- Versus Single-Item Measures
113
difficulty of the item. The one parameter model is equivalent to the Rasch model. Item response theory in its original form may be used to model the dichotomous responses of subjects to a number of questions or test items. Dichotomous responses (i.e., True-False, Yes-No, Correct-Wrong, SuccessFailure) can represent either original scales or the result of the binary transformation of the Likert scale. For example, a five-level Likert satisfaction scale with a neutral point (from Extremely Dissatisfied to Extremely Satisfied) can be dichotomized into Dissatisfied or Satisfied by applying an arbitrary cutoff value. An item response model with one parameter for item difficulty is known as a Rasch model. Although Rasch’s derivation used a different approach, in effect it can be expressed as a fixed-effect logit model in which logit value of the probability of answering a test item correctly is a difference between person ability and item difficulty. In the two-parameter model, the ICCs represent the proportion of individuals responding positively to an item as a function of the latent trait, the difficulty of the item, and the discrimination of the item. In the threeparameter model, the ICCs represent the proportion of individuals responding positively as a function of the latent trait, the difficulty of the item, the discrimination of the item, and guessing on the item. The following is the formula for the probability (Pi) that a randomly chosen member of a homogeneous group will respond correctly to an item. Pi ( )
ek 1 ek
where one-parameter model: k a( bi) two-parameter model: k ai( bi) three-parameter model: k ai( bi) Pi ( ) ci
(1 ci )e k 1 ek
where k amount of the latent trait a discrimination—the ability of an item to discriminate between different levels of the trait (the slope of the ICC) b difficulty—the greater the difficulty, the more latent trait the respondent must have to score positively on the scale (location of the ICC on the trait [x] axis)
34411_CH04_083_120
114
9/9/05
CHAPTER 4
9:42 AM
Page 114
MEASUREMENT
c pseudo-guessing parameter—accounts for guessing in people with low levels of the trait (ICC does not approach zero) Choosing the appropriate model depends on whether guessing could be a problem and whether items in the scale are believed to differentially tap levels of the underlying trait. The less complex the model, the more accurate and easier it is to estimate. The investigator must consider the effects on the results when all the assumptions of IRT are not met (Crocker & Algina, 1986). Moreover, classical reliability measures can be translated for use with IRT applications, but validity is still a concern. Latent traits are not the same thing as constructs; IRT does not provide a guarantee that items are measuring what they were intended to measure (Crocker & Algina, 1986). A latent trait may be responsible for the performance of a set of items, but these items may not be tapping the construct that was intended. Goodness-of-fit measures to assess how well the model fits the data are still under development, especially for the two- and three-parameter models. Ideally, IRT enables the creation of test-free and sample-free measurement. A scale representing a hierarchy of a trait enables people to be compared even if they took different items on the scale or are from different subpopulations. Potential economies of scale in scale administration are possible. However, large sample sizes are generally needed to estimate these models (minimum of 200 for the one-parameter model and 1000 for the three-parameter model) as well as specialized software (Crocker & Algina, 1986). Also, a good understanding of the trait is essential to construct items for a unidimensional scale. Work is now under way to employ IRT to shorten questionnaires. This procedure involves a computer program that chooses the next question to be asked based on prior responses. A few questions can thus locate a respondent on the scale, but each set of questions actually asked may be unique to that respondent. Latent Variable Structural Equation Modeling Poor measurement has serious implications for analyzing the data. Errors in calculating the variance of a single variable measured with error result in an overestimate of the variance. In multiple regression analysis, one assumes that each variable is measured without error. For reasons discussed earlier, this is an unrealistic assumption. When variables are mea-
34411_CH04_083_120
9/9/05
9:42 AM
Page 115
Summary: Advantages of Multiple- Versus Single-Item Measures
115
sured with error, estimation problems result (Kmenta, 1986). Random measurement error in the dependent (outcome) variable (left-hand side of the equation) yields unbiased regression coefficients, but the R2 (i.e., the total variance explained by the model) is decreased. Random measurement error in independent variables (right-hand side of the equation) will produce biased and inconsistent regression coefficients, so these coefficients should not be used for confidence intervals and t-tests. Measurement error in only one independent variable in the regression model will affect the coefficients of variables free of error, and it will not be possible to determine the direction of the bias in the coefficients (Bollen, 1989). Alternative methods of coefficient estimation are needed in the presence of measurement error (Bollen, 1989; Kmenta, 1986). The idea of a measurement model has been discussed throughout this chapter. Through this process, unobservable or observed constructs are operationalized by connecting them to one or more observed measures. Unobserved latent variables are typically associated with a sizable amount of measurement error. Developing a multiple-item scale of an unobservable construct averages out the measurement error, but the psychometric properties of the scale (e.g., reliability and validity) still remain a function of the sample used in the development of the scale. Measurement error in a single-item measure is rarely addressed; it is not possible to calculate a Chronbach’s alpha for this type of measure. When using unobserved constructs in a model, latent variable structural equation modeling offers a potential advantage over traditional analysis techniques such as multiple regression, because it involves both a measurement model (confirmatory factor analysis) and structural equation modeling (less restrictive regression modeling). Latent variable structural equation modeling enables the investigator to estimate accurately both direct and indirect causal relationships among the variables in the face of measurement error and causality that does not flow in a single direction; traditional regression approaches do not enable this (Joreskog & Sorbom, 1989). Multiple indicators are used to measure the latent variables in the model. For example, family socioeconomic status may be measured by four variables: parental educational attainment, types of jobs held by the parents, size of home, and family income. Using multiple indicators enables the reliabilities and validities of the variable to be estimated directly. The coefficients are unbiased because all random and nonrandom error are extracted by the measurement model. The latent variable
34411_CH04_083_120
116
9/9/05
CHAPTER 4
9:42 AM
Page 116
MEASUREMENT
structural equation model uses only the true variance associated with each variable to estimate the coefficients. SUMMARY Measurement can involve extremely complex elements and is a critical component of the overall process of assessing outcomes. Because measurement issues appear to address attributes that are familiar, they seem readily accessible. It is easy to underestimate the importance as well as the complexities of measurement for a study. In essence, measuring involves creating abstract representations for clinical realities. These conceptions are often fragile entities and must be handled with care. As statistical capacities expand, the numbers of assumptions also increase. Investigators must at least be sensitive to the potential pitfalls and willing to accept the idea that they exist even when they are not recognized by untrained eyes. In many cases, technical assistance is needed from experts who are familiar with measurement and its subtle complexities. REFERENCES Andresen, E.M., Patrick, D.L., et al. (1995). Comparing the performance of health status measures for older adults. Journal of the American Geriatrics Society 43(9), 1030–1034. Aquilino, W. (1994). Interview mode effects in surveys of drug and alcohol use: A field experiment. Public Opinion Quarterly, 58, 210–240. Aquilino, W., & Sciuto, L.L. (1990). Effects of interview mode on self-reported drug use. Public Opinion Quarterly, 54, 362–395. Avlund, K., Kreiner, S., & Schultz-Larson, K. (1993). Construct validation and the Rasch model: Functional ability of healthy elderly people. Scandinavian Journal of Social Medicine, 21(4), 233–245. Bartko, J. (1991). Measurement and reliability: Statistical thinking considerations. Schizophrenia Bulletin, 17(3), 483–489. Biemer, P.P., Groves, R.M., Lyberg, L.E., Mathiowetz, N.A., & Sudman, S. (1991). Measurement errors in surveys. New York: Wiley. Blalock, J., & Hubert, M. (1982). Conceptualization and measurement in the social sciences. Beverly Hills, CA: Sage Publications. Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley. Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin, 56, 81–105. Carmines, E.G., & Zeller, R.A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage Publications.
34411_CH04_083_120
9/9/05
9:42 AM
Page 117
References
117
Chronbach, L., & Furby, L. (1970). How we should measure “change”—Or should we? Psychological Bulletin, 74, 68–80. Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York: Academic Press. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston. DeVellis, R.F. (1991). Scale development: Theories and applications. Newbury Park: Sage Publications. Deyo, R.A., & Centor, R.M. (1986). Assessing the responsiveness of functional scales to clinical change: An analogy to diagnostic test performance. Journal of Chronic Disease 39(11), 897–906. Deyo, R.A., & Patrick, D.L. (1995). The significance of treatment effects: The clinical perspective. Medical Care 33(4), AS286–AS291. Dillman, D.A. (1999). Mail and electronic surveys: The tailored design method (2nd ed.). New York: Wiley. Dillman, D.A., Sangster, R.L., Tarnai, J., & Rockwood, T.H. (1996). Understanding differences in people’s answers to telephone and mail surveys. In M.C. Braverman & J.K. Slater (Eds.), Advances in survey research. San Francisco, CA: Jossey-Bass Publishers. Dillman, D.A., & Tarnai, J. (1988). Administrative issues in mixed mode surveys. In R.M. Groves, P.P. Beimer, L.E. Lyberg, J.T. Massey, W.L.I. Nichools, & J. Waksberg (Eds.), Telephone survey methodology. New York: Wiley, pp. xx, 581. Duncan, O.D. (1984). Notes on social measurement: Historical and critical. New York: Russell Sage. Elam, J.T., Graney, M.J., et al. (1988). Functional outcome one year following cataract surgery in elderly persons. Journal of Gerontology, 43, 122–126. Epstein, J.F., Barker, P.R., & Kroutil, L.A. (2001). Mode effects in self-reported mental health data. Public Opinion Quarterly, 65(4), 529–549. Fairbank, J.C., Couper, J., Davies, J.B., & O’Brien, J.P. (1980). The Oswestry Low Back Pain Disability Questionnaire. Physiotherapy, 66(8), 271–273. Feinstein, A., & Cicchetti, D. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. Finch, M., Kane, R.L., et al. (1995). Developing a new metric for ADLs. Journal of the American Geriatrics Society, 43(8), 877–884. Fisher, J., & William, P. (1993). Measurement related problems in functional assessment. American Journal of Occupational Therapy, 47(4), 331–338. Fowler, F.J., Jr., Roman, A.M., et al. (1998). Mode effects in a survey of Medicare prostate surgery patients. Public Opinion Quarterly, 62(1), 29–46. Fowler, F.J.J., Gallagher, P.M., & Nedrend, S. (1999). Comparing telephone and mail responses to the CAHPS survey instrument. Medical Care, 37(3), MS41–MS49. Groves, R.M. (1989). Survey errors and survey costs. New York: Wiley. Groves, R.M., Beimer, P.P., Lyberg, L.E., Massey, J.T., Nicholls, II, W.L., & Waksberg, J. (1988). Telephone survey methodology. New York: Wiley.
34411_CH04_083_120
118
9/9/05
CHAPTER 4
9:42 AM
Page 118
MEASUREMENT
Groves, R.M., & Couper, M. (1998). Nonresponse in household interview surveys. New York: Wiley. Groves, R.M., Dillman, D.A., Eltinge, J.L., & Little, R.J.A. (2002). Survey nonresponse. New York: Wiley. Guion, R.M. (1980). On trinitarian doctrines of validity. Professional Psychology, 11, 385–398. Guyatt, G., Walter, S., & Norman, G. (1987). Measuring change over time: Assessing the usefulness of evaluative instruments. Journal of Chronic Diseases, 40(2), 171–178. Guyatt, G.H., Deyo, R.A., Charlson, M., Levine, M.N., & Mitchell, A. (1989). Responsiveness and validity in health status measurement: A clarification. Journal of Clinical Epidemiology, 42(5), 403–408. Haley, S.M., McHorney, C.A., & Ware, J.E.J. (1994). Evaluation of the MOS SF-36 physical functioning scale (PF-10): I. Unidimensionality and reproducibility of the Rasch item scale. Journal of Clinical Epidemiology, 47(6), 671–684. Hambleton, R., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA, Sage Publications. Harman, J.S., Manning, W.G., et al. (in press). Interpreting clinical and policy significance in mental health services research. Hoeymans, N., Feskens, E.J.M., et al. (1996). Measuring functional status: Cross-sectional and longitudinal associations between performance and self-report (Zutphen Elderly Study 1990–1993). Journal of Clinical Epidemiology, 49(10), 1103–1110. Jaeschke, R., Singer, J., Guyatt, G.H. (1989). Measurements of health status: Ascertaining the minimal clinically important difference. Controlled Clinical Trials, 10 (4); 407–415. Janse, A.J., Gemke, R.J., B.J., Uiterwaal, C.S.P.M., van der Tweel, I., Kimpen, J.L.L., & Sinnema, G. (2004). Quality of life: Patients and doctors don’t always agree: A metaanalysis. Journal of Clinical Epidemiology, 57, 653–661. Joreskog, K.G., & Sorbom, D. (1989). LISREL 7: A guide to the program and application. Chicago: SPSS, Inc. Kane, R.L., Kane, R.A., Bershadsky, B., Degenholtz, H.B., Kling, K., Totten, A.M., et al. (in press). Proxy sources for information on nursing home residents’ quality of life. Journal of Gerontology: Social Sciences. Kaplan, R.M. (1995). Utility assessment for estimating quality-adjusted life years. In Sloan, F.A. (Ed.), Valuing Health Care. New York, NY, Cambridge University Press: 31–60. Katz, J.N., Larson M.G., et al. (1992). Comparative measurement sensitivity of short and longer health status instruments. Medical Care, 30(10), 917–925. Kazis, L.E., Anderson, J.J., Meenan, R.F. (1989). Effect sizes for interpreting changes in health status. Medical Care, 27(3), S178–S189. Kirshner, B., & Guyatt, G. (1985). A methodological framework for assessing health indices. Journal of Chronic Disease, 27(3), S178–S189. Kmenta, J. (1986). Elements of econometrics (2nd ed.). New York: Macmillan Publishing Company. Kuder, G., & Richardson, M. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160.
34411_CH04_083_120
9/9/05
9:42 AM
Page 119
References
119
Lazaridis, E.N., Rudberg, M.A., Furner, S.E., & Cassel, C.K. (1994). Do activities of daily living have a hierarchical structure? An analysis using the longitudinal study of aging. Journal of Gerontology: Medical Sciences, 49(2), M47–M51. Lynn, P. (1998). Data collection mode effects on responses to attitudinal questions. Journal of Official Statistics, 14(1), 1–14. Magaziner, J., Simonsick, E., Kashner, E., & Hebel, J. (1988). Patient-proxy response comparability on measures of patient health and functional status. Journal of Clinical Epidemiology, 41(11), 1065–1074. Martin, E. (2004). Presidential address: Unfinished business. Public Opinion Quarterly, 68(3), 439–450. Maruyama, G. (in press). Fundamentals of structural equation modeling. McDowell, I., & Newell, C. (1996). Measuring health: A guide to rating scales and questionnaires. New York: Oxford University Press. McIver, J.P., & Carmines, E.G. (1981). Unidimensional scaling. Newbury Park: Sage Publications. Meenan, R., Anderson, J., et al. (1984). Outcome assessment in clinical trials: Evidence for the sensitivity of a health status measure. Arthritis and Rheumatism, 27(12), 1344–1352. Norman, G.R. (1989). Issues in the use of change scores in randomized trials. Journal of Clinical Epidemiology, 42(11), 1097–1105. Nunnally, J.C. (1975). The study of change in evaluation research: Principles concerning measurement, experimental design, and analysis. E.L. Struening & M. Guttentag (Eds.), Handbook of evaluation research (Vol. 1, pp. 101–138). Beverly Hills, CA: Sage Publications. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Pedhazur, E.J. & Schmelkin, L.P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum Associates. Rockwood, T.H., Kane, R.L., & Lowry, A. (1999). Mode of administration considerations in the development of condition specific quality of life scales. Paper presented at the 7th Health Surveys Conference, National Center for Health Statistics, Williamsburg, VA. Rockwood, T.H., Sangster, R.L., & Dillman, D.A. (1997). The effect of response categories on questionnaire answers: Context and mode effects. Sociological Methods and Research, 26(1), 118–140. Rubenstein, L., Schairer, C., Wieland, G., & Kane, R. (1984). Systematic biases in functional status of assessment of elderly adults: Effects of different data sources. Journals of Gerontology, 39(6), 686–691. Schuman, H., & Presser, S. (1996). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Thousand Oaks, CA: Sage Publications. Schwarz, N., Bless, H., Bohner, G., Harlacher, U., & Kellenbenz, M. (1991). Response scales as frames of reference: The impact of frequency range on diagnostic judgments. Applied Cognitive Psychology, 5, 37–49. Schwarz, N., & Hippler, H.-J. (1995). Subsequent questions may influence answers to preceding questions in mail surveys. Public Opinion Quarterly, 59(1), 93–97.
34411_CH04_083_120
120
9/9/05
CHAPTER 4
9:42 AM
Page 120
MEASUREMENT
Schwarz, N., Knauper, B., Hippler, H.-J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55(4), 570–582. Schwarz, N., Strack, F., Hippler, H., & Bishop, G. (1991). The impact of administration of mode on response effects in survey research. Applied Cognitive Psychology, 5, 193–212. Schwarz, N., & Sudman, S. (Eds.). (1992). Context effects in social and psychological research. New York: Springer-Verlag. Spector, P.E. (1992). Summated rating scale construction: An introduction. Newbury Park, CA: Sage Publications. Spiegel, J.S., Ware, J.E., et al. (1986). What are we measuring? An examination of selfreported functional status measures. Arthritis and Rheumatism, 31(6), 721–728. Stevens, S.S. (1981). SCALING: A sourcebook for behavioral scientists (pp. 22–41). G. Maranell (Ed.). Chicago: Aldine Publishing Company. Streiner, D., & Norman, G. (1995). Health measurement scales: A practical guide to their development and use (2nd ed.). New York: Oxford University Press. Tourangeau, R., & Smith, T.W. (1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275–304. Travis, S.S., & McAuley, W.J. (1990). Simple counts of the number of basic ADL dependencies for long-term care research and practice. Health Services Research, 25(2), 349–360. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458. Ware, J.E., Jr., Manning, W.G., et al. (1984). Health status and the use of outpatient mental health services. American Psychologist, 39(10), 1090–1100. Weiss, A., Fletcher, A.E., et al. (1996). Use of surrogate respondents in studies of stroke and dementia. Journal of Clinical Epidemiology, 49(10), 1187–1194. Young, Y., German, P., et al. (1996). The predictors of surgical procedure and effects on functional recovery in elderly with subcapital fractures. Journals of Gerontology. Series A, Biological Sciences and Medical Sciences, 51(4), M158–M64.
NOTES 1. This concept originally came from physics and is associated with the Uncertainty Principle proposed by Werner Karl Heisenberg, who won the Nobel Prize in Physics in 1932.
34411_CH05_121_164
9/9/05
9:59 AM
Page 121
Part II Specific Measures: Outcomes
34411_CH05_121_164
9/9/05
9:59 AM
Page 122
34411_CH05_121_164
9/9/05
9:59 AM
Page 123
5 Generic Measures Matthew L. Maciejewski
Two classes of measures can be used to assess outcomes: conditionspecific measures and generic measures.1 Condition-specific measures (discussed in Chapter 6) focus on symptoms and signs that reflect the status of a given medical condition. They may also assess the direct sequelae of a disease on a person’s life. As a result, they are likely to be sensitive to subtle changes in health. Generic measures, on the other hand, are comprehensive measures that assess a single aspect or multiple aspects of health-related functioning in daily life. These measures can be applied in different types of diseases, treatments, and patients (Patrick & Erickson, 1993). For example, the Sickness Impact Profile (SIP) has been used to evaluate physical and social health in rheumatoid arthritis, low back pain, and myocardial infarction patients (Deyo & Diehl, 1983; Deyo, Leininger, & Overman, 1982; Ott et al., 1983). The 36item Short-Form Health Survey (SF-36) was used to compare the several chronic conditions studied in the Medical Outcomes Study (MOS) (Kravitz et al., 1992; Safran, Tarlov, & Rogers, 1994). The SF-36 has become the most widely used generic measure today and has been translated into many different languages.
INTRODUCTION: WHY USE GENERIC MEASURES? Generic measures capture elements that transcend single diseases. They can thus be used to compare the effects of treatments across diseases. Compared to condition-specific measures, they trade sensitivity for breadth. 123
34411_CH05_121_164
124
9/9/05
CHAPTER 5
9:59 AM
Page 124
GENERIC MEASURES
Generic measures are designed to capture the physical, psychological, and social aspects of health that have been outlined in the World Health Organization’s definition of health (World Health Organization, 1948). Health has been evaluated in terms of both quantity of health and quality of health. Many of the traditional outcome measures are based on counts of the frequency of occurrence of specific events that rely on definitional separations between function and dysfunction. These traditional measures tend to have face validity and to be measured across populations. Many of this category of health measures—such as mortality, morbidity, and average life expectancy—can be easily aggregated to provide summary figures that are easy to understand. Vital statistics rely on assessments of traditional health measures. Generic measures differ from these traditional health measures because they reflect the importance or value attributed to overall health and functioning. Generic measures can be used for several clinical and research purposes. In clinical practice, providers who are interested in assessing patient health broadly will find generic measures useful as bottom-line indicators of the effects of treatments on health status and quality of life. Generic measures can be used to measure health along the entire range of health from well-being to disability. These measures can augment other types of clinical data and provider perceptions of patient health that focus on symptoms and signs of disease. Providers can obtain a “natural history” of a patient’s perceived health status and quality of life by using generic measures, such as Cooperative Information Project (COOP) charts or the SF-36, to track changes in their patients. Patients’ perception of physical, emotional, and social health cannot be obtained any other way. Clinicians and patient proxies may be able to make statements about a patient’s experience, but the proxy responses can over- or underestimate patient health in ways that vary by domain and type of proxy (Boyer, Novella, Morrone, Jully, & Blanchard, 2004; Sneeuw et al., 1999; Sneeuw, Sprangers, & Aaronson, 2002; Von Essen, 2004). Such potential bias does not invalidate generic measures; it simply requires their careful use. Generic measures also have several useful applications in research. In clinical trials, the inclusion of generic (and condition-specific) outcomes measures can complement the analysis of morbidity and mortality by providing information from the patient’s perspective of the impact of treatment on relevant aspects of a patient’s experience. In addition, generic measures that assign relative values to different health states, such as the
34411_CH05_121_164
9/9/05
9:59 AM
Page 125
What Health Domains Are Typically Measured in Generic Measures?
125
Health Utilities Index Mark 3, the EQ-5D, or the Quality of Well-Being Scale, enable the construction of quality-adjusted life years that are the typical denominator in a cost-effectiveness ratio. These types of generic measures, called health utility or health preference measures, assign values to health states that reflect patient preferences for being in a given health state (e.g., death) (Patrick & Erickson, 1993). For example, the EQ-5D is able to identify 243 possible health states with five questions (EuroQol Group, 1990), whereas the Health Utilities Index Mark 3 can identify 972,000 possible health statuses with eight questions (Feeny et al., 2002). Generic measures can serve two purposes: 1. Assessing treatment effects in terms most relevant to patients 2. Constructing a measure of effectiveness for cost-effectiveness analyses There are two other potential uses for generic measures: risk adjustment and profiling. The predictive power of generic measures in cost estimation has been compared to diagnosis- or pharmacy-based measures of patient risk (Pope, Adamache, Walsh, & Khandker, 1998; Maciejewski et al., 2005). Parkerson and colleagues (2005) found that the Duke Health Profile was more predictive of primary care charges than diagnoses or provider severity measures. In this type of risk adjustment, generic measures potentially explain variation in costs. Finally, generic measures may also be useful in profiling health care organizations, such as hospitals or nursing homes. Kane and associates (2004) profiled 40 nursing homes using a generic quality-of-life measure and found that it was possible to differentiate nursing homes by using a quality-of-life measure that was completed by the nursing homes’ residents. Profiling is typically done using health outcomes or costs, but this novel approach to profiling via quality-of-life measures is likely to increase in the future.
WHAT HEALTH DOMAINS ARE TYPICALLY MEASURED IN GENERIC MEASURES? Several different domains of patient experience can be assessed with generic measures. Traditional measures of health, such as morbidity and mortality, are crude endpoints that assess the quantity of health in a given population or across populations. They are easy to measure and compare
34411_CH05_121_164
126
9/9/05
CHAPTER 5
9:59 AM
Page 126
GENERIC MEASURES
across populations and over time, but yield no information about the social or emotional functioning of a population. Generic quality-of-life measures, on the other hand, are typically designed to assess a range of domains.
Quality-of-Life Measures A host of generic measures is designed to assess domains within patient experience that are based on the concept of health-related quality of life (HRQoL). HRQoL refers to the experience and importance of different domains of health that are affected by disease and treatment (Patrick & Erickson, 1993; Ware, 1995).2 Five concepts define the scope of HRQoL (Patrick & Deyo, 1989): 1. 2. 3. 4. 5.
Impairments Functional states Health perceptions Social opportunities Duration of life.
These five concepts represent the different ways that dysfunction and treatment affect well-being and quality of life. Domains or health constructs have been operationalized to translate these five concepts of quality of life into a measurable form. Domains of functioning vary in importance depending on the condition and population in question. The question asked limits what is answered. Although there is no consensus about the inclusive set of domains, all investigators can agree that quality of life means different things to different people. This point has two implications. First, researchers may think about the domains of health in different ways, which may lead to generic measures assessing the same domains of health in different ways. Second, patients or survey respondents may value or weight various constructs differently. For example, a study comparing responses to the EQ-5D by patients in a clinical trial and the general public found that the clinical trial participants scored 223 of 243 states more highly (Polsky, Willke, Scott, Schulman, & Glick, 2001). Researchers must decide whether to use measures that are weighted individually by each respondent or to assign weights to each question in a construct explicitly, according to some social weighting scheme. If a generic measure has a weighting scheme that is derived from the general public, but the patient
34411_CH05_121_164
9/9/05
9:59 AM
Page 127
What Health Domains Are Typically Measured in Generic Measures?
127
sample of interest is very different from the general public, the choice of weighting scheme may be important. In this context, values play subtle but important roles at several levels in assessing health. First, the choice of measure may be impacted by the researcher’s perception of domains that will matter to the patient population. Second, the values implicit in the weighting scheme for the chosen measure will also impact the observed responses. The five concepts of HRQoL provide a way of thinking about quality of life as it relates to well-being and health status. However, they are extremely general concepts that do not directly relate to observable aspects of patient health. The eight domains of health listed in Table 5–1 define these general concepts in terms of measurable, less-abstract constructs: (1) physical functioning, (2) social functioning, (3) emotional functioning, (4) sexual functioning, (5) cognitive functioning, (6) pain, (7) vitality, and (8) overall well-being. For example, functional states can be assessed in physical, emotional, social, and cognitive domains. The five concepts of HRQoL overlap with the domains of health, but there is not a one-to-one relationship. Health measured in a single domain may relate to several aspects of HRQoL.
Table 5–1 Eight Domains of Health Domain
Examples of possible survey items
Physical functioning
Range of motion of limbs, feeding and bathing, walking outdoors, shopping, and cleaning
Social functioning
Visits with friends and family, restrictions on working, ability to baby-sit grandchildren
Emotional functioning
Feeling depressed or psychologically distressed, feeling happy
Sexual functioning
Problems with libido, impotence, or sexual satisfaction
Cognitive functioning
Problems remembering important dates or events, awareness of current time and place
Pain/discomfort
Feeling bodily pain when getting out of bed, feeling aches when lifting even light objects
Vitality
Lacking energy, feeling tired, needing to nap frequently
Overall well-being
Feeling satisfied with overall health, feeling content in general
34411_CH05_121_164
128
9/9/05
CHAPTER 5
9:59 AM
Page 128
GENERIC MEASURES
Physical Functioning Physical functioning refers to the range of an individual’s mobility and independence in three types of physical ability: 1. fitness or physiologic health 2. basic activity restrictions (activities of daily living [ADLs]) 3. more advanced independent living restrictions (instrumental ADLs [IADLs]) Fitness is commonly assessed by moderate or vigorous physical exertions such as walking several miles or playing tennis. The ability to perform basic self-care ADLs, such as toileting, dressing, and feeding, are fundamental to an assessment of physical health status. ADLs are assumed to measure a hierarchy of physical functioning based on Guttman scaling (Katz, Ford, Moskowitz, Jackson, & Jaffee, 1963; Lazaridis, Rudberg, Furner, & Cassel, 1994). Greater self-sufficiency based on IADLs, such as shopping, doing chores, and cleaning, are also indicators of physical functioning. Together, these three sets of physical tasks represent the range of functionality (Kane & Kane, 1981) that forms a crude hierarchy (Finch, Kane, & Philp, 1995). The specific choice of measure depends on the population addressed. Different sets tap different parts of the functional spectrum better.
Social Functioning Social functioning captures the range of an individual’s social interaction and interdependence in four ways: 1. 2. 3. 4.
social role limitations involvement in the community closeness of interpersonal relationships coping
Social role limitations refer to an individual’s ability to perform a social responsibility, such as mother or friend, to that individual’s expectations. Involvement in the community refers to the network of family and friends that an individual comes into contact with on a regular basis. This involvement is a measure of the degree of integration with his or her environment,
34411_CH05_121_164
9/9/05
9:59 AM
Page 129
What Health Domains Are Typically Measured in Generic Measures?
129
whereas social role limitations focus more on the person’s performance in that network. The third aspect of social functioning—closeness of relationships—refers to the quality of the social network, as opposed to the quantity of contacts in the community. This concept reflects social support and the comfort derived from meaningful relationships. It is based on interdependence relative to other people, as is community involvement. Coping refers to the individual’s ability to tolerate and maintain social relations while burdened with illness.
Emotional Functioning Emotional functioning refers to an individual’s range of affective wellbeing in terms of positive and negative emotions and the stability of emotions. For example, the SF-36 has a question about the amount of time the patient has been very nervous, happy, and downhearted (Ware & Sherbourne, 1992). The stability of emotions refers to the emotional fluctuations an individual undergoes over the course of an illness or treatment.
Sexual Functioning Sexual functioning refers to an individual’s ability to engage in his or her usual sexual relations. None of the most widely used generic measures, such as the SF-36, Sickness Impact Profile, Nottingham Health Profile, or Quality of Well-Being Scale, include questions about sexual functioning, but a widely used health utility measure (Health Utility Index Mark 2) does have one question on fertility. Several condition-specific measures have been developed to assess sexual functioning because sexual functioning has been found to be an important element of decreased quality of life in people with prostate cancer, strokes, brain tumors, and hysterectomies.
Cognitive Functioning Cognitive functioning refers to an individual’s range of intellectual ability in three ways: (1) memory, (2) reasoning abilities, and (3) orientation. The ability to remember significant dates and events in the past or future is a common measure of cognitive functioning, but thinking is a much more
34411_CH05_121_164
130
9/9/05
CHAPTER 5
9:59 AM
Page 130
GENERIC MEASURES
complex process. Several different generic measures test memory, including the Mental Status Questionnaire (Kahn, Goldfarb, Pollack, & Peck, 1960). Reasoning ability refers to an individual’s ability to do simple reasoning or computational tasks. Although these are best tested directly, it is possible to ask respondents to note problems they are experiencing in these areas. The SIP has a question asking the patient if he or she has trouble making plans, making decisions, or learning new things (Bergner, Bobbit, Carter, & Gilson, 1981). Orientation captures an individual’s awareness of current surroundings and is a common aspect of cognitive functioning queried in older adult patients.
Pain The domain of pain assesses the degree of debilitating physical discomfort distinct from physical functioning. Pain commonly refers to bodily pain and itching experienced by an individual. Some people use other words to describe uncomfortable sensations, like aches and cramps. Pain is assessed according to its intensity, duration, and frequency. Depending on the population and condition of interest, intensity and duration may be more important than frequency. For example, patients with chronic arthritis may constantly experience joint pain, with some episodes being more intense and longer than others.
Vitality The degree of vitality that an individual feels is captured by two constructs: (1) energy and (2) sleep and rest. This domain includes positive and negative ranges of vitality. For example, two questions on the SF-36 ask if patients feel full of pep and if patients feel tired. Sleep is also included in this domain because an individual’s vitality will be influenced by his or her amount of rest. Two questions on the SIP on sleep and rest ask patients if they sit during much of the day and if they sleep or nap during the day.
Overall Well-Being Overall life satisfaction, or overall well-being, is a global evaluation of an individual’s sense of contentment. This domain provides a comprehen-
34411_CH05_121_164
9/9/05
9:59 AM
Page 131
What Are the Criteria for Generic Measures?
131
sive assessment of the patient’s sense of his or her health status and happiness. In essence, overall well-being implicitly incorporates the physical, psychological, and social dimensions and their interactions. This domain is commonly assessed with broad questions asking patients about their overall health and well-being. The best-known question used to assess overall health is, “In general, would you say your health is excellent, very good, good, fair, or poor?” Overall well-being is a useful domain to include in most studies because this deceptively simple question has been found to be a good predictor of mortality (Idler & Kasl, 1991; Mossey & Shapiro, 1982). Combined with the other domains of health, assessment of overall well-being can provide a complete conception of HRQoL.
WHAT ARE THE CRITERIA FOR GENERIC MEASURES? Many criteria have to be considered when choosing a generic measure for a study. These criteria include whether the measure contains domains that are relevant to the study objectives and types of patients being studied, the measure’s validity and reliability, the measure’s sensitivity and responsiveness, and practical concerns about the measure. Table 5–2 lists the conceptual issues, practical concerns, and psychometric properties that one must consider when choosing a generic measure for a particular study. Most of the criteria apply to both unidimensional and multidimensional measures, but issues such as weighting apply specifically to the latter. The relevance of each criterion to selecting the most appropriate generic measures follows.
Domains of Health A conceptual model should dictate the quantitative and qualitative aspects of health to be measured, specifically the domains of health that are relevant to the study objectives and population being examined. Most constructs of HRQoL involve multiple domains, but one domain or aspect may be more important. In this case, unidimensional measures may be more important. For example, ADLs may be the appropriate measure if physical functioning is paramount in a population of older adult patients. The domains salient to a particular condition or patient population should influence the choice of measure, not the other way around. Multidimensional measures have the advantage of capturing health status across domains that may not have been expected by the investigator, but
34411_CH05_121_164
132
9/9/05
CHAPTER 5
9:59 AM
Page 132
GENERIC MEASURES
Table 5–2 Criteria for Choosing a Generic Measure Criterion
Impact on measure
Domains of health
Choice of domains affects the treatment effects observed.
Range of health
Range of measure affects the coverage of the spectrum of performance and change in health status.
Clinical relevance
The degree to which the measure taps into health domains relevant to the population.
Level of emphasis
The emphasis determines the relative weight of each domain in the measure.
Sensitivity
The ability of the measure to detect subtle but meaningful variations in health status without significant floor or ceiling effects that might limit its usefulness in the population of interest.
Responsiveness
The ability of the measure to detect important changes over time.
Reliability
Yields consistent, interpretable results.
Validity
Provides information about the dysfunction of interest and measures the domains that it was designed to measure.
Practical concerns
The burden of administration influences the response rates and rate of item completion from patients.
care must be taken to avoid the “kitchen sink” approach. Without a conceptual model, observing and reporting unexpected findings may exacerbate the perception of post hoc rationalization. Unidimensional measures are likely to be more sensitive to minor changes in that one domain, whereas an aggregated measure may dilute this effect. These trade-offs must be considered when determining which domains of health should be measured in a given investigation.
Range of Health The range of health refers to the possible stages of dysfunction that patients may experience. Patient health lies on a continuum that spans from
34411_CH05_121_164
9/9/05
9:59 AM
Page 133
What Are the Criteria for Generic Measures?
133
well-being to impairment to death (Field & Lohr, 1992; Nagi, 1991; World Health Organization [WHO], 1980). The following illustrates six points along the continuum of dysfunction: Well-Being➝Disease➝Impairment➝Functional Limitation➝Disability➝Death
Well-being, a concept first made prominent by the WHO, refers to a state of complete social, mental, and physical health. Disease refers to an interruption of bodily processes due to infection, trauma, or metabolic imbalance. Impairment is a greater degree of illness that results in “a loss or abnormality of an anatomical, physiological, mental, or emotional nature” (Nagi, 1991). Functional limitations are characterized by manifestations at the level of the entire organism, not just dysfunction in an organ or one system. Disability, the stage prior to death, refers to an “inability or limitation in performing socially defined roles and tasks” (Nagi, 1991). Different measures assess health at different points along the continuum, which can be seen in how questions are worded. Positively worded questions may tap into well-being, whereas negatively worded questions may tap into impairment, functional limitations, or disability. Neutrally worded questions with a range of possible responses may enable patients to respond to the entire range from well-being to disability. For example, an SIP question on social functioning asks for concurrence with the following statement, “I isolate myself as much as I can from the rest of my family,” whereas a more neutrally worded question from the SF-36 states, “During the past 4 weeks, to what extent has your physical health or emotional problems interfered with your normal social activities with family, friends, neighbors, or groups?” Another issue related to the range of health that a measure assesses is how significant are the floor and ceiling effects. A floor effect is the inability to measure health below a certain range. For example, the SF-36 is most sensitive to the health status effects of disease and impairment when healthy populations become ill. It does not do as well in distinguishing the very ill from the ill. Conversely, a ceiling effect is the inability to capture health above a certain range. ADLs may trace the loss of function, but saying that someone has no dependencies does not describe well just how functional he or she is. A conceptual model of the health status and disease will help to identify the suitable stages of dysfunction to be measured.
34411_CH05_121_164
134
9/9/05
CHAPTER 5
9:59 AM
Page 134
GENERIC MEASURES
Clinical Relevance Clinical relevance refers to the relationship between the health status domains in a generic measure and the expected path of patient dysfunction or recovery in the population of interest. Knowing the expected clinical course and potential adverse effects of both the disease and its treatments will help guide the choice of relevant measures. Mortality may not be clinically relevant for most clinical investigations in which death is not a likely occurrence (e.g., arthritis). Clinical relevance also refers to the usefulness of information derived from a measure. Results that can be translated into practice are more relevant than measures that are harder to interpret. Generic measures run the risk of being hard to put into a clinical context: What does a score of “x” on a scale of well-being mean? To interpret a certain score, it is important to determine a priori the norms against which scores are to be assessed. For example, the SF-36 and Quality of Well-Being Scale have published community norms against which scores from other populations can be compared.
Level of Emphasis or Dimensionality The level of emphasis refers to the relative focus placed on different constructs or domains in an instrument. Essentially, this translates into the number of questions used to capture a certain construct. Most generic measures avoid levels of emphasis across several domains to address all domains equally. In some cases, the investigator may want to emphasize one or two domains over others because they are especially relevant to the problem being studied. For example, in a study of lower back pain, one may want to address physiologic functioning and pain in more detail. However, other domains, such as social functioning, may also be relevant in a study of lower back pain. The relative focus that domains receive will be an important consideration in the choice of existing measures.
Sensitivity Basic measurement constructs are discussed at greater length in Chapter 4. Here they are briefly addressed in the context of their application to generic measures. The sensitivity of a measure refers to its ability to detect
34411_CH05_121_164
9/9/05
9:59 AM
Page 135
What Are the Criteria for Generic Measures?
135
small changes in health across different patient populations. These changes can result from disease progression or the effects of treatment. In general, generic measures are less sensitive than disease-specific measures or unidimensional measures. For example, the SF-36 asks general questions about pain but not about the intensity of pain. A more focused measure of pain would explore the location, intensity, nature, and duration of the pain.
Responsiveness The responsiveness of a measure refers to its ability to detect changes in health over time. Responsive measures will show a significant improvement for patients whose health is improved by treatment. Clinicians may want to know how big a change in a generic measure score is clinically meaningful, so that they have a quality-of-life basis for modifying treatment. There is a corresponding question related to responsiveness that parallels the one asked for clinical relevance: What does a change of “x” on a scale of well-being mean? Responsiveness can be interpreted into clinically meaningful terms via clinical anchors, statistical anchors, or effect sizes. Clinical anchors can be community norms or score differences between patients with and without a condition of interest. Statistical anchors tend to take the form of a minimally important difference, which is the difference in a generic measure that corresponds to a meaningful change in a global scale that asks the patient whether he or she improved over time (Norman, Sridhar, Guyatt, & Walter, 2001). Effect sizes are constructed by comparing treatment and control group outcomes, which are then standardized by their (pooled) standard deviations to evaluate how big an impact the treatment had on quality of life (Kazis, Anderson, & Meenan, 1989).
Reliability Reliability is a critical psychometric property that a good generic measure should demonstrate; it is virtually a precondition to its use. In psychometric terms, reliability has several meanings. For scales, internal reliability refers to the degree of concordance between any item and the overall scale score. Most reports of internal consistency for generic measures use Chronbach’s alpha.3 The statistic produces a range from 0 to 1;
34411_CH05_121_164
136
9/9/05
CHAPTER 5
9:59 AM
Page 136
GENERIC MEASURES
the latter represents perfect internal consistency where all questions measure the same construct.4 A generic measure is said to be reliable if—on repeated administrations— consistent results are obtained. Reliability is commonly assessed by administering the same instrument to the same population at two points in time (test-retest) or by having two separate decision makers evaluate the same set of cases (interrater). Although this topic is discussed in greater detail in Chapter 4, it is important to note here that reliability in one setting may not be equivalent to reliability in another. Especially in the context of interrater or test-retest reliability, the performance of one set of judges or respondents may not predict well how a different group will respond, particularly if the groups differ on relevant characteristics.
Validity Validity indicates whether an instrument is measuring the constructs it was designed to measure. Testing validity is usually much more challenging than establishing reliability. Validity can be assessed several ways, including content validity and construct validity. Measures that have content validity actually measure the domain or domains that they are supposed to measure. Construct validity represents how well the questions within a measure relate to one another and to other measures (Nunnally & Bernstein, 1994). PRACTICAL CONSIDERATIONS In almost all cases, investigators are advised to use an extant generic health status measure (or portions of it) instead of trying to create one de novo. (The development work involved is extensive and would dwarf most applied studies.) Several practical considerations must be addressed when choosing a generic health status measure. These practical issues should be considered once the conceptual model and psychometric issues are dealt with, not the other way around. These considerations include the following: • The length of time required to administer and complete the questionnaire • The appropriate format for the survey (telephone, face-to-face, or selfadministered) • The use of proxy respondents
34411_CH05_121_164
9/9/05
9:59 AM
Page 137
Practical Considerations
• • • •
137
The cost of administration (data collection and data entry) The complexity of the measurement and scoring methods The acceptability of the survey to patients/respondents and clinicians The expected format for presenting the results
Policymakers and clinicians generally find single, comprehensive values of health or quality of life more palatable than a set of scale values associated with different domains. (The simplicity of traditional generic measures, such as infant mortality or life expectancy, may partly explain their persistence in public policy analysis.) However, scores may not be meaningful per se. To get someone’s attention, the score has to be understood in a context that is significant to the audience. Moreover, the same score may mean different things to different people. For those working actively with a scale, score values may become rectified as facts, but, for most people, they are not. Treating score values as indisputable, objective indicators of underlying health should be avoided. Investigators must balance the burden of survey administration on interviewers and respondents with the breadth of health domains covered. Respondent attention will vary with the underlying nature of the respondents and the situation. Long questionnaires cannot be completed in a busy clinic while people are waiting to see the physician. There is a trade-off between the length of a questionnaire and the response rate for the overall measure and for particular items in the measure that has to be balanced. Older adult or chronically ill populations may have problems of comprehension and/or fatigue. The time frame of a question can be important. There is often a trade-off between the time span needed to generate enough events and the accuracy of recall. Different types of information can be reliably remembered for various times. Questionnaires must be designed to minimize the difficulty in answering them. Pilot testing to ensure that the questions are interpreted as they were intended can prevent disastrous mistakes. In many cases, the cost of survey administration will influence the choice of generic measure. As a general rule, complicated, high-quality data on patient quality of life will be expensive to collect. It usually requires interviewers to administer the questions and interpret them or probe for further responses. New forms of interactive computerized interviews may reduce the need for interviewers, but several logistical issues must be resolved before this approach is widely available.
34411_CH05_121_164
138
9/9/05
CHAPTER 5
9:59 AM
Page 138
GENERIC MEASURES
The acceptability of the survey to patients and clinicians is important to ensure reasonable response rates. The COOP charts developed by Nelson and colleagues (1990) at Dartmouth are unique among multidimensional generic measures in accompanying survey questions by pictures associated with Likert-type responses. For example, a question about physical functioning has answers ranging from very little work to very heavy work with a picture of someone doing dishes as little work and someone running as very heavy work (Nelson, Landgraf, Hays, Wasson, & Kirk, 1990). This intuitive format was highly acceptable to patients and clinicians. The results were also easily interpreted because the COOP charts have only one question for each health construct. Investigators have to balance presentation of results and the psychometric properties of their measure. Surveys that use single measures may have deficiencies in validity and reliability. By contrast, the results from complex, multidimensional measures may provide results that require careful interpretation. Each of these criteria should be considered for all measures, but all do not need to be satisfied before a generic measure can be considered useful. Often, it is a trade-off between the ideal characteristics for a measure and the practicalities of application.
PREFERENCE WEIGHTING Having collected results of multidimensional scales, the investigator must struggle with how to combine them into a single, meaningful aggregated index. Both clinicians and policymakers are usually looking for some sort of bottom-line summary measure that captures the overall impact of the results. Some means of weighting the relative importance of the individual components is needed. Simply adding components without making any attempt to weight them does not escape the problem. Such addition uses an unstated equal weighting, which may be at least as biased as any attempt to apply more deliberate weights. Preference, or utility, weighting refers to placing value judgments on health states achieved or avoided by treatment (Sackett & Torrance, 1978). These weights reflect the relative importance of various states of health when compared with an anchor, such as perfect health or death. Preference-weighted summary measures are critical when performing cost-effectiveness and quality-adjusted life-year calculations. There are three major approaches to assigning preference weights:
34411_CH05_121_164
9/9/05
9:59 AM
Page 139
Preference Weighting
139
1. Magnitude estimation is a process by which respondents compare a number of different health states and rank them, in terms of how much more or less preferable the states are to a baseline health state. 2. Utility weights are typically derived from one of two different approaches to having respondents choose between a defined health state and a probabilistic, desirable, or undesirable health state, which enables a determination of patient preferences. In the standard gamble method, current health with unknown probability P is compared to an alternative health state with unknown probability (1 P) and the respondent is asked to pick the probability P that would make the respondent indifferent between the two options. In the time trade-off method, two health states are again compared, but length of time in each health state is varied instead of the probability of being in each state. 3. Statistical weights are generated via regression analysis, in which each survey question is included as an independent variable to predict an outcome of interest. For example, all the questions in the Beck Depression Inventory could be used as independent variables in a regression to predict whether a group of patients have a clinical diagnosis of depression. The regression coefficients are the statistical weights that are used to generate an overall score. Both unidimensional and multidimensional health status measures must explicitly include preference weights in one way or another. However, weighting in multidimensional measures is given emphasis here because these measures commonly have a large number of questions across several domains that are more complicated to aggregate. Weighting is more straightforward in unidimensional measures, because several questions may be added together to obtain a single value for only one domain. For both types of generic measures, assessing preferences for different health states will improve the clinical relevance of index values. Four steps are involved in conducting preference weighting: 1. 2. 3. 4.
Choice of judges Description of health states Selection of measurement method Assignment of weights
There are four ways to measure preference weights and each has its merits. Weights can be arbitrarily assigned to questions, inferred from
34411_CH05_121_164
140
9/9/05
CHAPTER 5
9:59 AM
Page 140
GENERIC MEASURES
observations of social behavior, obtained from the literature on preference weighting, or derived through original data collection efforts (Patrick & Erickson, 1993). The fifth way to assign weights is really a variant of the first. Not overtly assigning weights to items (i.e., weighting each item equally) does not avoid the problem; it is as arbitrary as assigning specific weights. Different people, or judges, may assign different values to each of the various states to be summed. The composition of the judging panel is as much a philosophical issue as a scientific issue. There are no fixed rules about whose opinions should be used. Potential raters include the patients themselves, care professionals, family members, regulators, policymakers, and the general public. In some cases, it is possible to allow each patient to provide his or her own value weights. When comparable measures are needed, a more consistent scheme of weighting is preferred. One can use a single source or the weights can be calculated by combining the scores of different judges. The choice of a reference panel is important because different groups may generate quite different weights. For example, consumers place more weight on IADLs, whereas professionals weight ADLs more heavily (Chen & Kane, 2001; Kane, Rockwood, Finch, & Philp, 1997). Arbitrary assignment of weights is the most common approach. A variant of the default approach in which all questions implicitly have equal weights makes an even graver error by using the direct weight of questions where the range of possible responses varies. Not only do the weights not reflect any theoretical value structure, they are based on a spurious factor. Collecting information on preference weights from population surveys is the most resource intensive but may also be the most useful. Information derived from patients with experience with the illness under investigation may be different from that based on a sample of the general population, most of whom have had no direct exposure to the problem at hand. A number of different methods have been developed for direct derivation of preference weights, including the standard gamble, time tradeoff, psychophysical, and multidimensional methods (Froberg & Kane, 1989a, 1989b, 1989c, 1989d; Patrick & Erickson, 1993). Each of these methods requires careful consideration. The accurate assessment of preference weights is important for cost-effectiveness studies and aggregation of generic measures into index scores. Investigators must determine the best way for their research problems; what they cannot afford is to ignore the issue.
34411_CH05_121_164
9/9/05
9:59 AM
Page 141
Choosing a Measure
141
CHOOSING A MEASURE Health status measures have regularly addressed the four Ds—death, disease, disability, and discomfort—identified by White and colleagues (1961) more than four decades ago. Improved psychometric work has simply enabled more sophisticated measurement of these elements. Clinical investigators now have a variety of alternative generic measures that can be used as substitutes or complements to these traditional indicators of health.
Traditional Measures The most commonly used measures—mortality, morbidity, and utilization—are frequently used because they are the most accessible from medical records, health departments, and hospital charts. Mortality is a useful endpoint when there is a reasonable expectation that the problem being studied has a chance of leading to premature death. Mortality is most meaningful if expressed as the proportion of deaths from a particular cause over a defined time interval. Mortality suffers from both floor and ceiling effects.5 On the one hand, its absence says little about any other point on the continuum of dysfunction; on the other hand, it is hard to get worse. Morbidity can be assessed in several ways. It may reflect the incidence or prevalence of a disease, or it may be assessed as days of work missed or bed disability days. Evaluations relying solely on morbidity measures may exclude important extremes in outcomes, such as excellent health or death. These types of floor and ceiling effects are a concern in any generic status measure. Morbidity usually focuses only on physical health, but it also can capture the consequences of mental health and work-related limitations. If a broader range of dysfunction and other domains of health are relevant, then morbidity is not as useful as other, more comprehensive measures (Kaplan et al., 1989). Utilization of health services measures has been used as a proxy of health status. Utilization is difficult to interpret as a measure of health because of differences in access to services and other factors related to the population’s utilization. Cultural and economic factors in the patient population of interest may distort the relationship between health and utilization data (Johnson, Goldman, Orav, & Garcia, 1995; Meredith & Siu, 1995).
34411_CH05_121_164
142
9/9/05
CHAPTER 5
9:59 AM
Page 142
GENERIC MEASURES
Unidimensional Health Status Measures Generic health status measures that assess only one domain of health have been developed for use in quality-of-life assessments. Unidimensional measures can be either a single indicator based on one question or a single index based on a summation of several questions tapping the same domain. For example, the Mental Health Inventory is a five-question survey of mental health that is part of the SF-36 but can be used independently as a unidimensional measure. See Table 5–3 for selected unidimensional measures. ADLs and IADLs are two measures that capture a single domain of health—physical functioning. The PULSES Profile (Moskowitz & McCann, 1957) and the Functional Activities Questionnaire (Pfeffer et al., 1982) are two additional measures of physical function for use in elderly populations . The Change in Function Index is a general physical functioning measure that can be used in adult populations (MacKenzie, Charlson, DiGioia, & Kelley, 1986). Discussions of ADL and IADL measures of physical functioning, as well as selected measures of emotional, cognitive, and social functioning, follow. The health domain of emotional functioning also can be measured with an array of unidimensional measures. The Zung Self-Rating Anxiety and Self-Rating Depression Scales have all been developed to assess psychological and emotional functioning (Kane & Kane, 1981). Cognitive functioning can be assessed using the Mental Status Questionnaire (Kahn et al., 1960). Unidimensional measures of pain, vitality, and overall well-being are not discussed here; clinicians interested in measures of these domains should refer to the review by McDowell and Newell (1996).
Physical Functioning Measures One of the earliest measures of physical functioning—ADLs—was developed by Katz and colleagues in 1963 to assess basic concepts of selfcare. A fairly standard set of activities is now used to assess patients’ degree of independence or capacity for self-care, including dressing, bathing, using the toilet, transferring in and out of a bed/chair, and feeding. Two ADL scales—the Katz Index and the Barthel Index—are widely considered the best-known and validated measures of physical functioning. A third, the Functional Independence Measure (FIM), is widely used in the
34411_CH05_121_164
9/9/05
9:59 AM
Page 143
Choosing a Measure
143
Table 5–3 Selected Unidimensional Measures Measure
Domain of health
Number of questions
Focus of questions
Physical functioning
6 dichotomous questions to create a scale of dependency
Bathing, dressing, toileting, transfer, continence, feeding
Physical functioning
10 questions on performance based on 100point scale
Feeding, grooming, toileting, walking, climbing stairs, continence
Profile of Mood States (McNair & Lorr, 1964)
Emotional functioning
65 questions
Affective mood states of anxiety, depression, anger, vigor, fatigue, and bewilderment
Beck Depression Inventory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961)
Emotional functioning
21 questions
Body image, sadness, feeling of failure, dissatisfaction, social withdrawal, low energy
Mental Status Questionnaire (Kahn et al., 1960)
Cognitive functioning
10 questions that sum to 10
Orientation based on current date, location, birthday
RAND Social Health Battery (Donald & Ware, 1984)
Social functioning
11 questions
Social resources and contacts in family and community
MOS Social Support Survey (Sherbourne & Stewart, 1991)
Social functioning
20 questions
Emotional support, informational support, tangible support, positive social interaction, affection
Katz Index of ADLs (Katz et al., 1963) Barthel Index of ADLs (Mahoney & Barthel, 1965)
34411_CH05_121_164
144
9/9/05
CHAPTER 5
9:59 AM
Page 144
GENERIC MEASURES
rehabilitation arena (Granger, Hamilton, Linacre, Heinemann, & Wright, 1993; Keith, Granger, Hamilton, & Sherwin, 1987). In its original form, trained raters administer the six questions in the Katz Index using detailed statements about each activity. Based on the description of bathing (e.g., sponge bath, tub bath, or shower), a patient is rated either independent (without assistance or assistance with only one part of the body) or dependent (assistance with more than one part of the body). An overall score ranges from fully dependent to fully independent in all activities, based on a ranking of activities in order of dependence. The ranking of activities from smallest loss of independence to greatest loss is bathing, dressing, toileting, transferring, continence, and feeding (Katz et al., 1963). Patients are considered less than fully independent if they cannot do any one of the activities, less independent still if they cannot bathe but do one additional activity, and further dependent as they do fewer activities. This schema has been found to be a reliable and valid measure of physical functioning. It is now commonly used as a self-administered questionnaire. The Barthel Index uses a 100-point scale (Mahoney & Barthel, 1965), a weighted measure of the self-care activities of feeding, grooming, bathing, toileting, walking, climbing stairs, and continence (control of bladder and bowels). The Barthel Index is designed to be administered by medical staff. This hierarchical index is novel because each activity has a value associated with dependence or independence that sums to 100 for independence in all activities. In addition, continence is broken out into two dimensions, and walking and climbing stairs are added as basic elements of self-care. This measure also has been found to be a valid and reliable measure of physical functioning, primarily in older or more chronically ill patients. Like the Katz Index, the Barthel Index is a good predictor of mortality and hospitalization. Investigators can assess a higher level of physical functioning by measures of the IADLs. These activities commonly include cooking, cleaning, doing laundry, shopping, using transportation, keeping track of money, taking medications, and using the telephone. Each of these activities requires a greater level of skill and mobility than the basic self-care concepts captured in ADLs. Although numerous other measures have been developed, only one IADL measure—the Comprehensive Older Person’s Evaluation (COPE) scale—is discussed here. The items in the COPE scale include using the telephone, handling money, securing personal items, tidying up, and preparing meals (Pearlman, 1987). The measure is designed to be administered by medical staff who ask 100 questions of the patient that can be summed to 100 in an
34411_CH05_121_164
9/9/05
9:59 AM
Page 145
Choosing a Measure
145
overall IADL score. When a measure of ADLs and a measure of IADLs are used, a broad range of physical functioning can be assessed. They can be combined into a single continuous scale (Finch et al., 1995). Important issues in using ADL measures are the consistency in measures across settings (e.g., hospital, home) and formats (e.g., self-rating, direct observation). The information obtained from the same survey questions has been shown to vary by setting and format (Dorevitch et al., 1992; Myers, 1992; Myers, Holliday, Harvey, & Hutchinson, 1993). Self-rated functioning was found to be closer to observations made directly by clinicians than was functioning rated by formal and informal caregivers (Dorevitch et al., 1992). Clinicians underestimate the difficulty of several IADL tasks when compared with the self-rated evaluations by patients (Myers et al., 1993). Clinicians should consider these and other sources of response bias when administering any measures of quality of life and functioning.
Emotional Functioning Measures Emotional functioning can also be assessed independent of other measures, as in the case of ADLs for physical functioning. Unidimensional measures of emotional functioning tend to focus on a range of either positive or negative emotions (e.g., depression) to the exclusion of the other end of the continuum. The Profile of Mood States (POMS) survey is a 65-question survey that assesses affective mood states in a number of areas. There is also a POMS Brief that is 30 questions long. The specific affective mood states assessed in POMS are Tension-Anxiety, Depression-Dejection, AngerHostility, Vigor-Activity, Fatigue-Inertia, and Confusion-Bewilderment (McNair & Lorr, 1964). The Beck Depression Inventory is a 21-question survey that has been widely used to assess depression in adolescents and adults (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961). BDI scores of 10–18 indicate mild to moderate depression, 19–29 indicate moderate to severe depression, and scores above 30 indicate severe depression.
Cognitive Functioning Measures The Mental Status Questionnaire (MSQ) is a unidimensional measure of cognitive functioning based on orientation. The MSQ has been administered in the National Health and Nutrition Examination Survey to older
34411_CH05_121_164
146
9/9/05
CHAPTER 5
9:59 AM
Page 146
GENERIC MEASURES
adult respondents. This interviewer- or self-administered survey contains 10 questions about the date, location, the person’s birthday, and the current and previous U.S. presidents (Kahn et al., 1960; Patrick & Erickson, 1993). The answers to these 10 questions are scored by giving 1 point for each correct answer. The overall, unweighted score is then divided into three categories: no cognitive dysfunction (0–2 errors), moderate dysfunction (3–8 errors), and severe dysfunction (9–10 errors). The MSQ has been found to be highly reliable in terms of both internal consistency and test-retest reliability. However, the MSQ does not break down cognitive functioning into other spheres, such as remote memory or reasoning abilities. The Philadelphia Geriatric Center (PGC) Extended MSQ, developed by Lawton (1968), is a fuller version of the MSQ that includes several questions about more distant events to test remote memory (Kane & Kane, 1981). Questions about remote memory include the names of the patient’s mother and father. This version of the MSQ rounds out aspects of cognitive functioning but does not capture all aspects of cognitive functioning that may be pertinent to different clinical populations. There is some evidence that the levels of performance need to be adjusted for education and race. Because it was designed to be used primarily with older adults, it may not be as useful in younger groups. A widely used, more comprehensive cognitive measure is the Multidimensional Mental Status Examination (MMSE) developed by Folstein and colleagues (1985). As its name implies, it taps a variety of dimensions of cognitive functioning, including spatial recognition. It can be administered by a modestly trained individual and has been used effectively in a variety of cultures, but, again, some adjustments for education should be made in interpreting the results. A newer version is the Modified Mini-Mental State (3MS) Examination (Teng & Chu, 1987).
Social Functioning Measures Any unidimensional measure of social functioning, such as the RAND Social Health Battery (Donald & Ware, 1984), should cover aspects of social role limitations, involvement in the community, closeness of interpersonal relationships, and coping. The RAND Social Health Battery is an 11-item, self-administered scale that was used in conjunction with the Health Insurance Experiment. The battery queries social functioning related to family and community life (Donald & Ware, 1984). For exam-
34411_CH05_121_164
9/9/05
9:59 AM
Page 147
Choosing a Measure
147
ple, one question asks the patient how many close friends he or she has. The RAND Battery demonstrated reasonable reliability in both internal consistency (0.68) and test-retest (0.68) terms but lower validity. Although the RAND measure does not have strong psychometric properties, it is a short, easily administered social functioning measure. An alternative unidimensional scale is the Social Support Survey constructed for use in the MOS (Sherbourne & Stewart, 1991). This 20-item self-administered survey covers five areas of social functioning: 1. 2. 3. 4. 5.
Emotional support Informational support Tangible support Positive social interaction Affection
This measure has demonstrated high reliability (0.97) and validity. This unidimensional survey is also short and easy to administer. Most unidimensional measures of social and cognitive functioning are based on work in psychiatry instead of the older, sociology-based measures just presented. Those interested in these domains of health should refer to empirical literature that explores these aspects of health more fully.
Multidimensional Health Status and Health Utility Measures A wide range of multidimensional measures that captures two or more domains of health is available to investigators. The following major multidimensional measures are reviewed here: • • • • • •
The 36-Item Short-Form Health Survey (SF-36) The Sickness Impact Profile The Nottingham Health Profile The Duke University of North Carolina Health Profile The Quality of Well-Being Scale The Dartmouth COOP Charts
Three health utility measures also included in this review are the Health Utilities Index Mark 2, Health Utilities Index Mark 3, and the EuroQol
34411_CH05_121_164
148
9/9/05
CHAPTER 5
9:59 AM
Page 148
GENERIC MEASURES
EQ-5d. Table 5–4 lists these multidimensional and health utility measures and the domains they cover.
The SF-36 The SF-36 (Ware & Sherbourne, 1992) was developed to capture multiple domains of health across the entire continuum of health status (McHorney, Ware, & Raczek, 1993). It is, however, most useful in generally healthy populations. As noted previously in this chapter, eight domains of health are measured by the SF-36. The physical functioning scale is a 10-item set of questions that capture the presence and extent of physical limitations. Questions query patients about lifting and carrying groceries, bending and kneeling, walking moderate distances, and other ADL-type functions. Role functioning is measured using a 4-item scale for limitations due to physical problems and a 3-item scale for limitations due to emotional problems. Bodily pain is measured by a 2-item scale that asks about the frequency of bodily pain/discomfort and the extent of interference with normal activities due to pain. Mental health is captured using the 5-item Mental Health Inventory developed for the Medical Outcomes Study (Ware & Sherbourne, 1992). Four mental health conditions are tapped: (1) anxiety, (2) depression, (3) loss of behavioral/emotional control, and (4) psychological well-being. Vitality, in terms of energy and fatigue, is measure using a 4-item scale. Social functioning is measured using two questions that ask about physical and emotional health-related effects on social activities. Last, general health perceptions are assessed using a 5-item scale that queries patients about self-perceived health. These eight scales can be scored independently into easily interpreted patient evaluations. The SF-36 scales are not meant to be aggregated into a global or overall assessment of patient health, although they commonly are. Aggregation across scales blurs the distinctions within each domain of health that the measure was designed to tap. The purported strength of the SF-36 is the number of domains that it measures and the relatively broad range of health that it covers. However, the measure works best to distinguish states of illness from wellness. It is not as effective in distinguishing changes in dysfunction among those already disabled. The reliability, content validity, and construct validity of the SF-36 have been evaluated in numerous studies, and it has been found to be highly
√
√
√ √ √ √
√
√
√
√
√
√
DUKE
QWB
COOP
HUI-2
HUI-3
EQ-5D
√
√
√
√
SF-36 36-ltem Short-Form Health Survey (Ware & Sherbourne, 1992) SIP Sickness Impact Profile (Bergner et al., 1981) NHP Nottingham Health Profile (Hunt, McKenna, & McEwen, 1980) DUKE Duke University of North Carolina Health Profile (Parkerson, Broadhead, & Tse, 1990; Parkerson et al., 1989) QWB Quality of Well-Being Scale (Kaplan et al., 1989) COOP Dartmouth COOP Charts (Nelson & Berwick, 1989) HUI-2 Health Utility Index Mark 2 (Feeny et al., 1995) HUI-3 Health Utility Index Mark 3 (Feeny et al., 2002) EQ-5D EuroQol EQ-5D (EuroQol Group, 1990)
√
√
√
√
√
√
√
√
Vitality
√
√
√
√
Overall well-being
9:59 AM
√
√
√
√
√
Pain
9/9/05
Key:
√
√
√
NHP
√
√
√
SIP
√
√
√
Social functioning
SF-36
Cognitive functioning
Emotional functioning
Physical functioning
Table 5–4 Domains Covered by Selected Multidimensional and Health Utility Measures
34411_CH05_121_164 Page 149
Choosing a Measure 149
34411_CH05_121_164
150
9/9/05
CHAPTER 5
9:59 AM
Page 150
GENERIC MEASURES
reliable and valid for diverse patient groups and individuals (McHorney, Ware, Lee, & Sherbourne, 1994; McHorney et al., 1993). Validity was assessed according to both psychometric and clinical criteria. To keep the health survey brief enough to be useful in clinical settings, certain domains of health were omitted. The SF-36 does not capture cognitive functioning. If this domain of health is relevant to the patient population and/or medical condition of interest, the SF-36 can be used in conjunction with other measures or can be replaced with one of several measures that do capture this domain. A utility-based version of the SF-36, the SF-6D, has been constructed to enable researchers to calculate quality-adjusted life years from the SF-36 (Brazier, Roberts, & Deverill, 2002).
Sickness Impact Profile The SIP was developed at the University of Washington to provide a measure of health status sensitive enough to detect changes in health status over time or between groups (Bergner et al., 1981). This enables comparisons across types and severities of medical conditions and across demographic and cultural subgroups. The SIP consists of 6 domains, which are further divided into 12 subdomains of health. The 6 domains are physical functioning, emotional functioning, social functioning, cognitive functioning, pain, and overall well-being. These domains are divided into sleep and rest, eating, work, home management, recreation and pastimes, ambulation, mobility, body care and movement, social interaction, alertness behavior, emotional behavior, and communication. The SIP contains 136 statements that are used to produce a percentage score for each of the 12 subdomains. The statements are divided unevenly among the 12 subdomains but can be summed to obtain an overall assessment of health status. In addition, a physical index score can be obtained by combining body care and movement, ambulation, and mobility scale scores. A psychosocial index score can be obtained by combining the emotional behavior, social interaction, and communication scale scores. Reliability and validity of the SIP have been demonstrated in a number of field trials and in subsequent comparative analyses of generic measures. In a series of three field trials early in its development, the SIP obtained high reliability with Cronbach’s alphas of 0.94 and above (Bergner, 1993). Validity in these three field trials was also found to capture accurately patient dysfunction and to correlate higher with clinical
34411_CH05_121_164
9/9/05
9:59 AM
Page 151
Choosing a Measure
151
ratings of dysfunction than clinical ratings of sickness. In a study of patients with head injury, the SIP was found to correlate highly with neurologic and neuropsychologic severity indices (Temkin, Dikmen, Machamer, & McLean, 1989). One of the strengths of the SIP is that it covers a large number of domains and subdomains of health-related quality of life. In addition, the SIP is sensitive to a wide variety of conditions and patient populations. The two greatest limitations to the SIP are its length and the omission of the vitality domain. At 136 statements, the SIP is one of the longer generic surveys. The breadth of domains covered is commendable, but the length of the instrument makes it impractical for use in most clinical investigations or practice settings. One way to overcome the problem of length is to use subsets of the SIP. The SIP omits the health domain of vitality from the survey, a critical domain for many patient populations and conditions. Overall, the SIP is a comprehensive alternative to the SF36 if more detailed scale scores for a large number of subdomains are desired. Not only can the SIP provide scale scores for each of the 12 subdomains, but index scores for physical and psychosocial functioning can be obtained along with a composite or overall health status score. The various levels of aggregation are unique to the SIP. Care must be taken to combine scales according to the methods detailed by Bergner and colleagues (Bergner, 1993). In response to the perceived need for a shorter version of the SIP, de Bruin, Buys, and colleagues (1994) developed the SIP68, a 68-item version of the SIP. The SIP68 covers six subdomains to provide reliable and valid results in a less burdensome format. The six subdomains of the SIP68 are (1) somatic autonomy, (2) mobility control, (3) psychological autonomy and communication, (4) social behavior, (5) emotional stability, and (6) mobility range (Post, de Bruin, DeWitte, & Schrijvers, 1996). Somatic autonomy refers to basic ADLs. Mobility control refers to hand and arm control and basic control over one’s body (de Bruin, Buys, et al., 1994). Psychological autonomy and communication measures cognitive functioning and one’s ability to communicate. Social behavior refers to social functioning, and emotional stability refers to emotional functioning. Last, mobility range refers to IADL-type activities of shopping and personal business affairs. These six subdomains predicted scores from the original 136-item SIP nearly perfectly (R2 0.96). Although further testing and validation of the SIP68 remain to be done, early results indicate that it is a valid and reliable generic measure without the burden of the longer version.
34411_CH05_121_164
152
9/9/05
CHAPTER 5
9:59 AM
Page 152
GENERIC MEASURES
Nottingham Health Profile The Nottingham Health Profile (NHP), developed in England, covers all domains of health except cognitive functioning and overall well-being (Hunt, McKenna, & McEwen, 1980). The NHP is a two-part instrument with a total of 45 questions. The first part consists of a 38-item survey that asks patients about physical functioning, vitality, pain, and emotional and social functioning. Vitality is subdivided into questions about sleep and energy (McEwen, 1993). The second part of the NHP includes 7 questions about the problems caused by the patient’s present state of health in seven areas: job or work, home management, social life, home life, sex life, interests/hobbies, and holidays. The NHP was tested for face, content, and criterion validity in its assessment of physical, social, and emotional domains of health. Validation studies have been done on older adult patients, general practice clinic patients, patients with peripheral vascular disease, and patients with nonacute conditions. In addition, reliability has been demonstrated in studies of patients with osteoarthritis and patients with peripheral vascular disease (McEwen, 1993). The NHP has been primarily used in clinical studies and population surveys in Europe. The strengths of the NHP are ease of administration, ease of interpretation, usefulness as a measure of general health status in a variety of conditions and populations, and high reliability and validity. These strengths are similar to those of the SF-36 and SIP, but the NHP provides an alternative approach to health because it focuses on the departure from health. Because the NHP was developed in England, it has a slightly different cultural foundation. Nonetheless, it has been successfully incorporated into studies in the United States. The limitations of the NHP include the possibility of false positives due to the severity of dysfunction used in survey questions and its focus on the negative aspects of health as opposed to concepts of well-being and positive health. The first limitation is a potential problem in many patient populations in which sensitivity to minor health changes is important. Despite these limitations, the NHP has been shown to be valid and reliable across several domains of health. The range of domains covered and its distinct cultural heritage provide sufficient reasons to consider its use in qualityof-life investigations.
34411_CH05_121_164
9/9/05
9:59 AM
Page 153
Choosing a Measure
153
Duke Health Profile The 63-item Duke University Health Profile (DUHP) measures symptom status, physical functioning, social functioning, and emotional functioning. On further application in clinical trials, the survey was found to have several conceptual problems that led to difficulties in scoring and interpretation of scales (Parkerson, Broadhead, & Tse, 1990; Parkerson et al., 1989). Self-esteem was the only indicator of emotional health, and social role performance was the only measure of social functioning. These unidimensional constructs do not assess the presence and extent of dysfunction in these two domains of health. To correct for these conceptual problems and to provide a shorter, more practical survey instrument, Parkerson and colleagues created the 17-item Duke Health Profile (1993). This generic measure was constructed to maintain the convergent and discriminant validity of the larger DUHP as well as high internal consistency and test-retest reliability. Clinical validity was also assessed to determine how well the Duke survey could distinguish between patients with different types of physical and mental health problems. The physical health of patients was assessed by questions using ADLtype measures of sleeping, walking up a flight of stairs, and running. Mental health was assessed by questions such as self-perceived depression and nervousness. A patient’s social health was captured by questions about family relationships, involvement in family and social groups, and other factors. These questions demonstrated reasonably good reliability, with Cronbach’s alpha values between 0.55 to 0.78, and reasonable test-retest correlations (Parkerson et al., 1990). Psychometric and clinical validity were not as strongly validated. Much more extensive validation for this measure is necessary, a task that Parkerson and colleagues have begun to pursue (Parkerson, Broadhead, & Tse, 1991; Parkerson et al., 1993). One of the great strengths of the Duke health survey is its brevity. However, this brevity also compromises the psychometric properties of reliability and validity. This measure—and its longer predecessor—are comprehensive measures that may prove useful in many investigations after more testing in different patient populations and diagnostic groups. More work is necessary to ensure that the benefits of using this measure are not eclipsed by the costs of unreliable and uninterpretable results.
34411_CH05_121_164
154
9/9/05
CHAPTER 5
9:59 AM
Page 154
GENERIC MEASURES
Quality of Well-Being Scale The Quality of Well-Being (QWB) scale is a 38-item generic measure that can be used to assess three domains of functioning: physical functioning, social functioning, and vitality (Kaplan et al., 1989). The QWB was the primary quality-of-life instrument used to rank health states in Oregon’s health reform effort. Four scales capture the entire continuum of health from death to optimal functioning, which is uncommon among most of the widely used measures. This is achieved by combining mortality and morbidity into a single, comprehensive score (Kaplan, Anderson, & Ganiats, 1993). Overall health from death to optimal functioning is evaluated by aggregating the three scale scores in a linear function to obtain values ranging from death at 0.0 to optimal health at 1.0. The QWB can be used to obtain point-in-time assessments of health status as well as projections of health using “well-years.” An overall wellness score can be obtained from the QWB by aggregating subscales and converting them to well-years. Well-years integrate mortality and morbidity into health status in terms of well-years of life (Kaplan et al., 1993). Utility weights are used to combine responses from the various questions into a single score. The QWB does not weight all questions equally but assigns negative weights according to the degree of dysfunction associated with a given question. The scale starts at 1.0 for optimal health, and responses to the survey shift the score away from optimal health as dysfunction is indicated. The utility weights were validated on a general population sample in San Diego, California. The generalizability of the weights has been demonstrated in a study of rheumatoid arthritics (Balaban, Sagi, Goldfarb, & Nettler, 1986). The QWB has been used in a wide range of clinical studies, but validation studies comparing this measure to others have yet to be done. This generic instrument has been used in studies of rheumatoid arthritis, coronary artery bypass grafts, antihypertensive medications, acquired immunodeficiency syndrome (AIDS), and cystic fibrosis (Bombardier et al., 1986; Kaplan et al., 1989; Orenstein, Pattishall, Ross, & Kaplan, 1990; Weinstein & Stason, 1985). Validity and reliability for this measure have not been thoroughly explored, a point that Kaplan and colleagues acknowledge (Kaplan et al., 1989). This sort of validation is necessary before the QWB can be used with confidence with conditions and populations different from those already examined.
34411_CH05_121_164
9/9/05
9:59 AM
Page 155
Choosing a Measure
155
Dartmouth COOP Charts The COOP charts developed as part of the Cooperative Information Project at Dartmouth University were devised as generic measures that suited the clinical setting by satisfying five practical criteria (Nelson & Berwick, 1989): 1. 2. 3. 4. 5.
Produce reliable and valid data on a range of domains Fit into routine office practice Apply to a range of illnesses Yield easily interpretable results Provide the clinician with useful information of patient functioning
The COOP charts employ a unique format of survey presentation. Each question about a given domain of health, for example, physical functioning, is accompanied by pictures associated with responses. These charts are used to assess the domains of physical, emotional, and social functioning, as well as overall well-being. The nine charts focus on physical, emotional, role, and social functioning; overall health; and changes in health, pain, overall quality of life, and social support (Nelson et al., 1990). Each of the charts provides five possible answers and is not meant to be aggregated. Convergent and discriminant validity have been assessed by comparing responses from COOP charts and the health survey used in the RAND Health Insurance Experiment. The COOP charts were validated with patients in four outpatient settings: 1. 2. 3. 4.
Patients in primary care clinics Elderly patients in Veterans Affairs (VA) clinics Hypertensive and diabetic patients in university hospital clinics Patients with chronic diseases in several sites in the Medical Outcomes Study
High convergent validity was demonstrated using the multitraitmultimethod technique (Nelson et al., 1990). Reliability was assessed by surveying patients two times with a 1-hour interval and with a 2-week interval. High reliability was demonstrated in the 1-hour interval, but much lower reliability was found in the 2-week interval. Nelson and colleagues argue that health status in patients visiting a physician’s office should change markedly over a 2-week period, so low 2-week reliability should
34411_CH05_121_164
156
9/9/05
CHAPTER 5
9:59 AM
Page 156
GENERIC MEASURES
not be a concern. The COOP charts are a novel approach to evaluating health with generic measures. Future research should be conducted using COOP charts in populations with a range of illnesses, both acute and chronic, not captured in earlier work. The patient and clinician acceptance of the nine COOP charts was quite high and their practicality did not entirely sacrifice validity and reliability. This measure, however, omits several potentially important domains of health (e.g., cognitive functioning, pain, and vitality).
HEALTH UTILITY MEASURES Several widely used generic health utility measures are used to assess patient preferences for the purpose of conducting cost-effectiveness or cost-utility analyses. These measures include the Health Utilities Index Mark 2, Health Utilities Index Mark 3, and the EuroQoL EQ-5D. The Health Utilities Index Mark 2 (HUI-2) is a seven-item survey that was developed by researchers at McMaster University (Feeny, Furlong, Boyle, & Torrance, 1995; Torrance et al., 1996). The seven questions ask about sensation, mobility, emotion, cognition, self-care, pain, and fertility. The fertility question has three possible responses, but all other questions have either four or five responses. The HUI-2 has 24,000 empirically derived health states. Utilities were derived using a standard gamble. The index was originally derived to evaluate the long-term sequelae of childhood cancer but has been applied to many other populations with different conditions. The Health Utilities Index Mark 3 is an eight-item survey that improved upon the seven-item Health Utilities Index Mark 2 (Feeny et al., 2002). The eight questions ask about vision, hearing, speech, ambulation, dexterity, emotion, cognition, and pain. This measure has 972,000 possible health states because the number of possible responses to each question has been increased to a minimum of five and a maximum of six. The EuroQol EQ-5D is a five-item survey that was developed with a European team of researchers to generate a simple health state valuation measure (EuroQol Group, 1990). The five questions ask about mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Each question has three responses. The EQ-5D also has a visual analog scale for patients to self-rate their health status. The EQ-5D can generate 243 health states, which were derived from a time trade-off approach to utility calculation. In addition, the visual analog scale has been used to
34411_CH05_121_164
9/9/05
9:59 AM
Page 157
Summary
157
assign utilities to health states (www.euroqol.com). In 1998 health state valuations based on adults in the United States were constructed, which simplifies its use for American populations (Johnson, Coons, Ergo, & Szava-Kovats, 1998).
RESOURCES A large number of online and offline resources are available to assist with identification of generic measures that can be useful in clinical or research operations. The MAPI Institute in France (www.mapi-research.fr) and its related Quality of Life Instrument Database (www.qolid.com) are good sources for a broad array of generic and condition-specific measures. More information on the Health Utilities Indices and the EuroQol EQ-5D can be found at the following Web sites: www.healthutilities.com and www.euroqol.org, which also have extensive reference lists.
SUMMARY The evaluation of health status and health-related quality of life often deserves greater consideration than may seem necessary at first. Careful attention to conceptualization of health domains and an exact definition of quality can facilitate the valid and reliable assessment of patient health. Five main points can serve as guidelines: 1. It is best to determine which of the eight domains are salient to your problem, and then choose the generic measure that captures those domains. Each measure will have one or more questions that focus on a specific aspect of functioning, and different measures incorporate different combinations of domains. No single measure will work best in all possible patient populations and medical conditions. Choosing the measure appropriate for one’s study is a critical first step in effective outcomes research. 2. Generic measures are the best way to capture multidimensional aspects of health. These types of measures are designed to assess patient health across several domains (physical/social/emotional/ cognitive functioning, pain, vitality, and overall well-being). Measures may obtain a single index number from values in separate
34411_CH05_121_164
158
9/9/05
CHAPTER 5
9:59 AM
Page 158
GENERIC MEASURES
domains, such as the SIP, or may obtain individual values for each domain. In either case, generic measures are the ones to use if overall patient health is the desired outcome. 3. Generic measures should be collected at baseline (as well as followup) to indicate where a patient’s course began. The measurement of health status to indicate improving or worsening health is only meaningful if a before/after comparison can be made. Otherwise, only point-in-time evaluations can be made. 4. The more easily understood the measure, the more useful it is. Generic measures generally have life anchors to relate a numeric value to a state or condition of health. In other words, results based on scale scores need to be placed in the context of daily life to be easily interpreted. The values obtained from a generic measure must have a clinical context to be useful sources of information. 5. Health utility measures can serve dual purposes. They can provide patient-based assessments of treatment and also serve as measures of effectiveness for cost-effectiveness or cost-benefit analysis. These measures are useful if economic analysis is accompanying an outcomes analysis.
REFERENCES Balaban, D.J., Sagi, P.C., Goldfarb, N.I., & Nettler, S. (1986). Weights for scoring the quality of well-being instrument among rheumatoid arthritics. A comparison to general population weights. Medical Care, 24(11), 973–980. Beck, A.T., Ward, C.H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571. Bergner, M. (1993). Development, testing, and use of the Sickness Impact Profile. In Walker, B.R., & Rosser, S. (Eds.), Quality of life assessment: Key issues in the 1990s. Boston: Kluwer Publishing Company. Bergner, M., Bobbit, R.A., Carter, W.B., & Gilson, B.S. (1981). The Sickness Impact Profile: Development and final revision of a health status measure. Medical Care, 19(8), 787–805. Bombardier, C., Ware, J., Russell, I.J., Larson, M., Chalmers, A., & Read, J.L. (1986). Auranofin therapy and quality of life for patients with rheumatoid arthritis: Results of a multicenter trial. American Journal of Medicine, 81, 565–578. Boyer, F., Novella, J.L., Morrone, I., Jully, D., & Blanchard, F. (2004). Agreement between dementia patient report and proxy reports using the Nottingham Health Profile. International Journal of Geriatric Psychiatry, 19(11), 1026–1034.
34411_CH05_121_164
9/9/05
9:59 AM
Page 159
References
159
Brazier, J., Roberts, J., & Deverill, M. (2002). The estimation of a preference-based measure of health from the SF-36. Journal of Health Economics, 21, 271–292. Chen, Q., & Kane, R.L. (2001). Effects of using consumer and expert ratings of an activities of daily living scale on predicting functional outcomes of postacute care. Journal of Clinical Epidemiology, 54(4), 334–342. de Bruin, A.F., Buys, M., deWitte, L.P., & Diederiks, J.P.M. (1994). The Sickness Impact Profile: SIP68, a short generic version, first evaluation of the reliability and reproducibility. Journal of Clinical Epidemiology, 47(8), 863–871. de Bruin, A.F., Diederiks, J.P.M., DeWitte, L.P., Stevens, F.C., & Philipsen, H. (1994). The development of a short generic version of the Sickness Impact Profile. Journal of Clinical Epidemiology, 47(4), 407–418. Deyo, R.A., & Diehl, A.K. (1983). Measuring physical and psychosocial function in patients with low back pain. Spine, 8, 635–642. Deyo, R.A., Inui, T.S., Leininger, J.D., & Overman, S.S. (1982). Physical and psychosocial function in rheumatoid arthritis: Clinical use of a self-administered health status instrument. Annals of Internal Medicine, 142(5), 879–882. Donald, C.A., & Ware, J.E., Jr. (1984). The measurement of social support. In Greenley, J.F. (Ed.), Research in Community and Mental Health. Westport, CT: JAI Press. Dorevitch, M.I., Cossar, R.M., Bailey, F.J., Bisset, T., Lewis, S.J., Wise, L.A., et al. (1992). The accuracy of self and informant ratings of physical capacity in the elderly. Journal of Clinical Epidemiology, 45(7), 791–798. EuroQol Group. (1990). EuroQoL—A new facility for the measurement of health-related quality of life. Health Policy, 16, 199–208. Feeny, D., Furlong, W., Boyle, M., & Torrance, G.W. (1995). Multi-attribute health status classification systems: Health utilities index. PharmacoEconomics, 7, 490–502. Feeny, D., Furlong, W., Torrance, G.W., Goldsmith, C.H., Zhu, Z., DePauw, S., et al. (2002). Multiattribute and single-attribute utility functions for the Health Utilities Index Mark 3 system. Medical Care, 40, 113–128. Field, M., & Lohr, K. (Eds.). (1992). Guidelines for clinical practice: From development to use. Washington, DC: Institute of Medicine, National Academy Press. Finch, M., Kane, R.L., & Philp, I. (1995). Developing a new metric for ADLs. Journal of the American Geriatrics Society, 43(8), 877–884. Folstein, M., Anthony, J.C., Pachad, I., Duffy, B., & Gruenberg, E.M. (1985). The meaning of cognitive impairment in the elderly. Journal of the American Geriatrics Society, 33, 228–233. Froberg, D.G., & Kane, R.L. (1989a). Methodology for measuring health-state preferences— I: Measurement strategies. Journal of Clinical Epidemiology, 42(4), 345–354. Froberg, D.G., & Kane, R.L. (1989b). Methodology for measuring health-state preferences— II: Scaling methods. Journal of Clinical Epidemiology, 42(5), 459–471. Froberg, D.G., & Kane, R.L. (1989c). Methodology for measuring health-state preferences— III: Population and context effects. Journal of Clinical Epidemiology, 42(6), 585–592. Froberg, D.G., & Kane, R.L. (1989d). Methodology for measuring health-state preferences— IV: Progress and a research agenda. Journal of Clinical Epidemiology, 42(7), 675–685.
34411_CH05_121_164
160
9/9/05
CHAPTER 5
9:59 AM
Page 160
GENERIC MEASURES
Granger, C.V., Hamilton, B.B., Linacre, J.M., Heinemann, A.W., & Wright, B.D. (1993). Performance profiles of the functional independence measure. American Journal of Physical Medicine and Rehabilitation, 72, 84–89. Hunt, S.M., McKenna, S.P., & McEwen, J.A. (1980). A quantitative approach to perceived health status: A validation study. Journal of Epidemiology and Community Health, 34, 281–285. Idler, E.L., & Kasl, S. (1991). Health perceptions and survival: Do global evaluations of health status really predict mortality? Journal of Gerontology, 46(2), S55–S65. Johnson, J.A., Coons, S.J., Ergo, A., & Szava-Kovats, G. (1998). Valuation of EuroQoL (EQ-5D) health states in an adult U.S. sample. PharmacoEconomics, 13, 421–433. Johnson, P.A., Goldman, L., Orav, E.J., & Garcia, T.H. (1995). Comparison of the medical outcomes study short-form 36-item health survey in black patients and white patients with acute chest pain. Medical Care, 33(2), 145–160. Kahn, R.L., Goldfarb, A.I., Pollack, M., & Peck, A. (1960). Brief objective measures for the determination of mental status in the aged. American Journal of Psychiatry, 117, 326–328. Kane, R.A., & Kane, R.L. (1981). Assessing the elderly: A practical guide to measurement. New York: Springer Publishing Company. Kane, R.L., Bershadsky, B., Kane, R.A., Degenholtz, H.B., Liu, J., Giles, K., et al. (2004). Using resident reports of quality of life to distinguish among nursing homes. Gerontologist, 44(5), 624–632. Kane, R.L., Rockwood, T., Finch, M., & Philp, I. (1997). Consumer and professional ratings of the importance of functional status components. Health Care Financing Review, 19(2), 11–22. Kaplan, R.M., Anderson, J.P., & Ganiats, T.G. (1993). The Quality of Well-Being Scale: Rationale for a single quality of life index. In B.R. Walker & S. Rosser (Eds.), Quality of life assessment: Key issues in the 1990s. Boston: Kluwer Publishing Company. Kaplan, R.M., Anderson, J.P., Wu, A.W., Mathews, W.C., Kozin, F., & Orenstein, D. (1989). The Quality of Well-Being Scale: Applications in AIDS, cystic fibrosis, and arthritis. Medical Care, 27(3), S27–S43. Katz, S., Ford, A.B., Moskowitz, R.W., Jackson, B.A., & Jaffee, M.W. (1963). Studies of illness in the aged. The index of ADL: A standardized measure of biological and psychosocial function. Journal of American Medical Association, 185(12), 914–919. Kazis, L.E., Anderson, J.J., & Meenan, R.F. (1989). Effect sizes for interpreting changes in health status. Medical Care, 27(3 Suppl), S178–S189. Keith, R., Granger, C., Hamilton, B., & Sherwin, F. (1987). The functional independence measure: A new tool for rehabilitation. In M.G. Eisenberg & R.C. Grzesiak (Eds.), Advances in clinical rehabilitation (Vol. 1., pp. 6–18). New York: Springer-Verlag. Kravitz, R.L., Greenfield, S., Rogers, W., Manning, W.G., Zubkoff, M., Nelson, E.C., et al. (1992). Differences in the mix of patients among medical specialties and systems of care: Results from the medical outcomes study. Journal of the American Medical Association, 267(12), 1617–1623. Lawton, M.P. (1968). The PGC Mental Status Questionnaire: Mimeograph from the Philadelphia Geriatric Center.
34411_CH05_121_164
9/9/05
9:59 AM
Page 161
References
161
Lazaridis, E.N., Rudberg, M.A., Furner, S.E., & Cassel, C.K. (1994). Do activities of daily living have a hierarchical structure? An analysis using the longitudinal study of aging. Journal of Gerontology: Medical Sciences, 49(2), M47–M51. Maciejewski, M.L., Liu, C.-F., Derleth, A., McDonnell, M., Anderson, S.M., & Fihn, S.D. (2005). The performance of administrative and self-reported measures for risk adjustment of VA primary care patient costs. Health Services Research, 40, 887–904. MacKenzie, C.R., Charlson, M.E., DiGioia, D., & Kelley, K. (1986). A patient-specific measure of change in maximal function. Archives of Internal Medicine, 146(7), 1325–1329. Mahoney, F.I., & Barthel, D.W. (1965). Functional evaluation: The Barthel Index. Maryland State Medical Journal, 14, 61–65. McDowell, I., & Newell, C. (1996). Measuring health: A guide to rating scales and questionnaires (2nd ed.). New York: Oxford University Press. McEwen, J. (1993). The Nottingham Health Profile. In Walker, S.R. & Rosser, R. (Eds.), Quality of life assessment: Key issues in the 1990s. Boston: Kluwer Publishing Company. McHorney, C.A., Ware, J.E., Jr., Lee, J.F.R., & Sherbourne, C.D. (1994). The MOS 36-Item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Medical Care, 32(1), 40–66. McHorney, C.A, Ware, J.E., Jr., & Raczek, A.E. (1993). The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), 247–263. McNair, D.M., & Lorr, M. (1964). An analysis of mood in neurotics. Journal of Abnormal and Social Psychology, 69, 620–627. Meredith, L.S., & Siu, A.L. (1995). Variation and quality of self-report health data: Asian and Pacific Islanders compared with other ethnic groups. Medical Care, 33(11), 1120–1131. Moskowitz, E., & McCann, C.B. (1957). Classification of disability in the chronically ill and aging. Journal of Chronic Disease, 5, 342–346. Mossey, J.M., & Shapiro, E. (1982). Self-rated health: A prediction of mortality among the elderly. American Journal of Public Health, 72, 800–808. Myers, A.M. (1992). The clinical Swiss Army knife: Empirical evidence on the validity of IADL functional status measures. Medical Care, 30(5), MS96–MS111. Myers, A.M., Holliday, P.J., Harvey, K.A., & Hutchinson, K.S. (1993). Functional performance measures: Are they superior to self-assessments? Journal of Gerontology: Medical Sciences, 48(5), M196–M206. Nagi, S.Z. (1991). Some conceptual issues in disability and rehabilitation, Appendix A. In Pope, A.M. & Tarlov, A.R. (Eds.), Disability in America: Toward a national agenda for prevention. Washington, DC: National Academy Press. Nelson, E.C., & Berwick, D.M. (1989). The measurement of health status in clinical practice. Medical Care, 27(3), S77–S90. Nelson, E.C., Landgraf, J.M., Hays, R.D., Wasson, J.H., & Kirk, J.W. (1990). The functional status of patients: How can it be measured in physicians’ offices? Medical Care, 28(12), 1111–1126.
34411_CH05_121_164
162
9/9/05
CHAPTER 5
9:59 AM
Page 162
GENERIC MEASURES
Norman, G.R., Sridhar, F.G., Guyatt, G.H., & Walter, S.D. (2001). Relation of distributionand anchor-based approaches in interpretation of changes in health-related quality of life. Medical Care, 39, 1039–1047. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Orenstein, D.M., Pattishall, E.N., Ross, E.A., & Kaplan, R.M. (1990). Quality of well-being before and after antibiotic treatment of pulmonary exacerbation in cystic fibrosis. Chest, 98, 1081–1084. Ott, C.R., Sivarajan, E.S., Newton, K.M., Almes, M.J., Bruce, R.A., Bergman, M., & Gilson, B.S. (1983). A controlled randomized study of early cardiac rehabilitation: The sickness impact profile as an assessment tool. Heart and Lung, 12(2), 162–170. Parkerson, G.R.J., Broadhead, W.E., & Tse, C.K.J. (1990). The Duke Health Profile: A 17item measure of health and dysfunction. Medical Care, 28, 1056–1072. Parkerson, G.R.J., Broadhead, W.E., & Tse, C.K.J. (1991). Development of the 17-item Duke Health Profile. Family Practice, 8, 396–401. Parkerson, G.R.J., Connis, R.T., Broadhead, W.E., Patrick, D.L., Taylor, T.R., & Tse, C.K. (1993). Disease specific versus generic measurement of health-related quality of life in insulin dependent diabetic patients. Medical Care, 31(7), 629–639. Parkerson, G.R.J., Hammond, W.E., Michener, J.L., Yarnall, K.S.H., & Johnson, J.L. (2005). Risk classification of adult primary care patients by self-reported quality of life. Medical Care, 43, 189–193. Parkerson, G.R.J., Michener, J.L., Wu, L.R., Finch, J.N., Muhlbaier, L.H., Magruder-Habib, K., et al. (1989). Association among family support, family stress, and personal functional health status. Journal of Clinical Epidemiology, 42, 217–229. Patrick, D.L., & Deyo, R.A. (1989). Generic and disease-specific measures in assessing health status and quality of life. Medical Care, 27(3), S217–S232. Patrick, D.L., & Erickson, P. (1993). Health status and health policy: Quality of life in health care evaluation and resource allocation. New York: Oxford University Press. Pearlman, R.A. (1987). Development of a functional assessment questionnaire for geriatric patients: The Comprehensive Older Person’s Evaluation (COPE). Journal of Chronic Disease, 40, 85S–98S. Pfeffer, R.I., Kurosaki, T.T., Harrah, C.H., et al. (1982). Measurement of functional activities in older adults in the community. Journal of Gerontology, 37, 323. Polsky, D., Willke, R.J., Scott, K., Schulman, K.A., & Glick, H.A. (2001). A comparison of scoring weights for the EuroQol derived from patients and the general public. Health Economics, 10, 27–37. Pope, G.C., Adamache, K.W., Walsh, E.G., & Khandker, R.K. (1998). Evaluating alternative risk adjusters for Medicare. Health Care Financing Review, 20, 109–129. Post, M.W.M., de Bruin, A.F., DeWitte, L.P., & Schrijvers, A. (1996). The SIP68: A measure of health-related functional status in rehabilitation medicine. Archives of Physical Medicine and Rehabilitation, 77(5), 440–445. Sackett, D.L., & Torrance, G.W. (1978). The utility of different health states as perceived by the general public. Journal of Chronic Diseases, 31, 697–704.
34411_CH05_121_164
9/9/05
9:59 AM
Page 163
References
163
Safran, D.G., Tarlov, A.R., & Rogers, W.M. (1994). Primary care performance in fee-forservice and prepaid health care systems: Results from the MDS. Journal of the American Medical Association, 271(20), 1579–1586. Sherbourne, C.D., & Stewart, A.L. (1991). The MOS Social Support Survey. Social Science and Medicine, 32, 705–714. Sneeuw, K.C., Aaronson, N.K., Sprangers, M.A., Detmar, S.B., Wever, L.D., & Schornagel, J.H. (1999). Evaluating the quality of life of cancer patients: Assessments by patients, significant others, physicians and nurses. British Journal of Cancer, 81(1), 87–94. Sneeuw, K.C., Sprangers, M.A., & Aaronson, N.K. (2002). The role of health care providers and significant others in evaluating the quality of life of patients with chronic disease. Journal of Clinical Epidemiology, 55(11), 1130–1143. Temkin, N.R., Dikmen, S., Machamer, J., & McLean, A. (1989). General versus diseasespecific measures: Further work on the Sickness Impact Profile for head injury. Medical Care, 27(3), S44–S53. Teng, E.L., & Chu, H.C. (1987). The Modified Mini-Mental State (3MS) Examination. Journal of Clinical Psychiatry, 48, 314–318. Torrance, G.W., Feeny, D.H., Furlong, W.J., Barr, R.D., Zhang, Y., & Wang, Q. (1996). Multiattribute utility function for a comprehensive health status classification system: Health Utilities Index Mark 2. Medical Care, 34, 702–722. Von Essen, L. (2004). Proxy ratings of patient quality of life—Factors related to patientproxy agreement. Acta Oncol, 43(3), 229–234. Ware, J.E. Jr., (1995). The status of health assessment, 1994. Annual Review of Public Health, 16, 327–354. Ware, J.E., Jr., & Sherbourne, C.D. (1992). The MOS 36-Item Short-Form Health Survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30(6), 473–483. Weinstein, M.C., & Stason, W.B. (1985). Cost-effectiveness of interventions to prevent or treat coronary heart disease. American Review of Public Health, 6, 41–63. White, K.L., Williams, T.I., & Greenberg, B.G. (1961). The ecology of medical care. New England Journal of Medicine, 265, 885–892. World Health Organization. (1948). Constitution of the World Health Organization. Geneva, Switzerland: WHO Basic Documents. World Health Organization. (1980). International classification of impairments, disabilities, and handicaps: A manual of classification relating to the consequences of disease. Geneva, Switzerland: World Health Organization.
NOTES 1. Generic measures may also be used as independent variables to control for health status differences between patients or groups. Usually, generic measures used as independent variables are collected at baseline to adjust for patient differences without confounding outcomes through the contemporaneous measurement of the outcome of interest and health status. Contemporaneous measurement of outcomes and health
34411_CH05_121_164
164
9/9/05
CHAPTER 5
9:59 AM
Page 164
GENERIC MEASURES
status can lead to health status affecting how a person reports outcomes. For example, being depressed may influence how a person reports his or her overall health status. 2. The term HRQOL is used in an effort to restrict discussions of quality of life to the domains where health care can be expected to play some role. 3. Cronbach’s alpha is an estimate of the expected correlation of one test with an alternative form containing the same number of items (Nunally & Bernstein, 1994). 4. One would not actually want a state of perfect internal consistency because one variable would summarize all relevant information and each additional variable would add nothing new to the overall score. 5. A floor effect is a value that observations cannot fall below. A ceiling effect is a value that observations cannot exceed (Nunally & Bernstein, 1994).
34411_CH06_165_184
9/9/05
10:01 AM
Page 165
6 Condition-Specific Measures Adam Atherly
The previous chapter discussed the use of generic health status measures in the measurement of outcomes. Health services researchers have successfully used generic health status measures to measure outcomes in a wide variety of settings. However, generic health status measures are only one alternative available to the outcomes researcher; condition-specific health status measures offer both advantages and limitations compared to their generic cousins. In using either, collecting baseline data will be important in interpreting the results. INTRODUCTION: CONDITION-SPECIFIC VERSUS GENERIC MEASURES Condition-specific outcome measures are outcome measures designed to measure changes in the most salient aspects of a specific condition. They reflect the aspects of functioning that are closely tied to the condition. Condition-specific measures are available for many different diseases and afflictions. There are essentially two types of condition-specific measures: 1. clinical, which use primarily signs, symptoms, and tests 2. experiential, which capture the impact of the disease or problem on the patient, often in ways very akin to those used in generic measures Condition-specific measures are designed to be extremely sensitive to the detection of small treatment effects. Generic measures (discussed in the 165
34411_CH06_165_184
166
9/9/05
CHAPTER 6
10:01 AM
Page 166
CONDITION-SPECIFIC MEASURES
previous chapter) may fail to detect small changes for a number of reasons. First, condition-specific measures are designed to tap the domains of greatest interest for a particular condition. Generic health status measures, in contrast, cast a broad net across different facets of health. For example, the SF-36 (described in detail in the previous chapter) measures across a broad spectrum; it attempts to tap eight conceptually separate aspects of functioning and mental well-being (Ware & Sherbourne, 1992). The numbers of variables in each domain are presented in Table 6–1, which illustrates both the strengths and weaknesses of a broad health status measure. The SF-36 may be able to tap several different dimensions of an intervention that is expected to affect health in a variety of ways. For example, certain drugs may improve physical functioning while causing fatigue. A generic health status measure could incorporate the sum of both of these effects. However, this breadth and flexibility is also a weakness of the generic health status measures. “Condition-specific health status measures are measures designed to assess specific diagnostic groups or patient populations, often with the goal of measuring responsiveness or ‘clinically important’ changes” (Patrick & Deyo, 1989, p. S217).
Table 6–1 Numbers of Variables Included in the Concepts Examined in the SF-36 Concept Physical functioning
No. of questions 10
Role limitations due to physical problems
4
Social functioning
2
Bodily pain
2
General mental health
5
Role limitations due to emotional problems
3
Vitality
4
General health perceptions
5
34411_CH06_165_184
9/9/05
10:01 AM
Page 167
Why Not Generic Health Status Measures?
167
WHY NOT GENERIC HEALTH STATUS MEASURES? According to Patrick and Deyo (1989): “Generic health status measures purport to be broadly applicable across types and severities of disease, across different medical treatments or health interventions and across demographic and cultural subgroups” (p. S217). The generic health status measures may not isolate the variable(s) of greatest interest. For example, the SF-36 may not be a good choice to evaluate the effect of an intervention designed to help control pain. Because only 2 of the 36 questions deal with pain (see Table 6–1), much of the information provided by the SF-36 may simply be irrelevant (technically, this is low content validity). By not isolating the dimensions of greatest interest, a true treatment effect can be masked. In contrast, a conditionspecific measure can be specifically designed to assess pain and can focus directly on the precise area of interest. In effect, one is trading depth for breadth. This involves both the range of the domains addressed and the response range within each domain. Whereas generic measures are typically designed to cover a wide range, condition-specific measures may hone in on what is especially salient for that condition. This focus is important not just to enable measures to be concise, but also to ensure that the health status measure is sensitive to clinically important differences or changes in health status. Determining whether or not a given treatment had an effect requires that the estimated effect is both statistically significant and clinically significant. Statistical significance, which refers to the likelihood that a result occurred by chance, is determined by such factors as the sample size and variance. The clinical significance of the treatment effect must be determined by the investigator. A generic health status measure may miss clinically significant treatment effect for several reasons. First, there may be a floor or ceiling effect. A scale is considered to have a ceiling effect when individuals who are rated as perfectly healthy by the scale can be found to have health problems on other scales (Bombardier et al., 1995); the converse is a floor effect. For example, the SF-36 is designed to be used to distinguish the health status of reasonably healthy populations from those who are ill, but it is not as useful in discriminating between the ill and the very ill. If the SF-36 was used to assess the effect of an intervention on a population of frail elderly, the scale might miss a true treatment effect because the entire sample was bunched at the lower end of the scale both before and after the
34411_CH06_165_184
168
9/9/05
CHAPTER 6
10:01 AM
Page 168
CONDITION-SPECIFIC MEASURES
intervention. The condition-specific measure can be aimed at the proper segment of the distribution (i.e., avoid ceiling and floor effects). The second reason a generic health status measure may miss a clinically significant treatment effect is because it lacks sensitivity (Kessler & Mroczek, 1995). For some illnesses, a successful treatment may not result in an increase in the scores of an overall health status measure. For example, generic health status measures were insensitive to positive health changes resulting from successful treatment of benign prostatic hyperplasia, changes successfully captured by condition-specific measures (Barry et al., 1995). More important, generic health status measures may simply fail to tap the necessary dimensions of health. For example, a successful treatment for hypertension would not necessarily affect the scores of a self-administered generic health status measure such as the SF-36, because the effects of blood pressure control are not perceptible to the patients, although successful treatment of the hypertension can have a profound influence on the longterm health of the individual.
CONDITION-SPECIFIC HEALTH STATUS MEASURES A condition-specific measure can successfully tap the domains of greatest interest, but these advantages come at a price. Condition-specific measures have several drawbacks. In order to measure a given condition more precisely, condition-specific measures cast a far narrower net than do generic measures. As a result, some unanticipated (or anticipated in a separate domain) effects from an intervention could be missed (Bombardier et al., 1995). A second drawback associated with condition-specific health status measures is the difficulty of comparing the results on a study to those of other studies when condition-specific measures are used. Improvements in diabetes care cannot be readily compared to a decrease in arthritis symptoms. In many settings, the investigator is not merely interested in finding a treatment effect, but also in estimating the importance of the treatment effect. Most readers prefer some clinical translation of a numerical scale. What does a 3-point increase in the Arthritis Impact Measurement Scales (AIMS) mean clinically (Kazis, Anderson, & Meenan, 1989)? How does a 3-point AIMS increase compare to an improvement on
34411_CH06_165_184
9/9/05
10:01 AM
Page 169
Condition-Specific Health Status Measures
169
another scale? One possibility is to compare the findings to previous studies. However, for most conditions, a plethora of scales is available. If the investigator is concerned not merely with finding the presence of a treatment effect, but also in comparing the treatment with other similar studies, a condition-specific measure may complicate that comparison, although there are statistical techniques such as “effect sizes” available to aid in that task (Guyatt, Feeny, & Patrick, 1993). However, some generic health status measures, such as the SF-36 and EuroQol 5D, have become sufficiently widely used that a comparison across different treatments, conditions, or populations may be more easily made in some situations. Condition-specific outcome measures have another strength; they are often intuitively appealing, especially to clinicians. If one wishes to investigate the impact of a treatment for arthritis, a commonsense approach is to use a scale that specifically taps the dimensions of health affected by arthritis. The ultimate goal of outcomes research is to provide insights that lead to greater efficiency and a higher quality of care. To actually have an impact on the delivery of care, the researcher must be able to persuade the medical care community that the findings truly reflect reality, rather than simply being a theoretical abstraction. An excellent example of the strengths and limitations of the two types of measures was provided in a study of patients who underwent cataract surgery (Damiano et al., 1995). This study used both a generic health status measure (the Sickness Impact Profile, SIP) and a vision-specific measure (the VF-14) to evaluate the impact of the surgery. A postoperative improvement in visual acuity was found to be unrelated to the SIP score; conversely, the VF-14 was found to be highly sensitive to changes in visual acuity. The authors conclude that the SIP is simply not sensitive enough to measure changes such as those caused by cataract surgery. However, the SIP did provide interesting insights. Several behaviors measured by the SIP that were not expected to be related to vision, such as “I act irritable and impatient,” were found to be highly correlated with presurgery better-eye visual acuity. This suggests that vision may impact health in ways not detected by a vision-specific measure. Almost all measures of complications are condition specific. By definition, a complication of treatment is an untoward outcome associated with the treatment of a condition. Therefore, a complication is necessarily condition specific. (The definition of complications is discussed in greater detail in Chapter 8 on comorbidities.)
34411_CH06_165_184
170
9/9/05
CHAPTER 6
10:01 AM
Page 170
CONDITION-SPECIFIC MEASURES
OTHER ALTERNATIVES Although a discussion of the use of condition-specific health status measures versus the use of generic health status measures is useful, the best alternative often is to try to combine the two approaches (as in the Damiano and colleages [1995] study mentioned previously). The generality of the generic measures is both their strength and weakness; the generic measures exhibit breadth, but not depth. The generic measures are able to find treatment affects across many domains, but may fail to focus intently enough on the domains of most interest. Conversely, the condition-specific measures exhibit great depth, but little breadth. If resources permit, the combined strategy is probably the best alternative and is a relatively common approach (Patrick & Deyo, 1989; Fitzpatrick et al., 1998; Guyatt, Feeny, & Patrick, 1991). This was the course followed by Bombardier and colleagues (1995) in a study of pain and physical function after knee surgery. Among patients who reported knee pain, a generic measure (the SF-36) was unable to distinguish patients in need of knee surgery; the knee pain–specific measure (the Western Ontario and McMaster Universities Osteoarthritis Index [WOMAC] [Bellamy et al., 1983]) was able to do so. After surgery, patients often recovered enough that the WOMAC was unable to distinguish among patients, although some were extremely disabled, whereas the SF-36 was able to do so. Some people worry that responses to the survey may be altered by the order in which the questionnaires are administered (i.e., whether the generic or condition specific is given first). Previous research has found that the order in which questions are asked can alter responses to the questions, based on factors such as interviewee fatigue, relevance of the question or topic, and learning effects (Sudman & Bradburn, 1974). However, the little work that has been done examining the effect of survey ordering (generic versus condition specific) suggests that this is not a major issue and that responses do not vary based on order (McColl et al., 2003). If it is necessary to use a single instrument, two approaches are available. First, a generic instrument can be modified for a specific condition. For example, an (unsuccessful) attempt was made to modify the SIP to make it more sensitive to head injury (Temkin et al., 1989). Items deemed nonapplicable were removed and items believed particularly applicable to head injury were added. The scale was then reweighted. In this case, the modified SIP was no more effective than the unmodified SIP, so it was discarded. One drawback of this approach is that the strength of generic
34411_CH06_165_184
9/9/05
10:01 AM
Page 171
The Choice of a Condition-Specific Measure
171
measures is the ability to compare to other studies that have used the same measure. Once a scale has been modified and reweighted, it is no more comparable to the original than a completely unrelated condition-specific measure. Second, a generic instrument can have a condition-specific supplement attached to it. The goal of this approach is to have the condition-specific supplement not overlap measurement of the domains covered by the generic measurement, but instead to expand more deeply into domains of particular interest (Patrick & Deyo, 1989). This approach retains the comparability of the generic measurement (because the measure is retained unchanged), but also taps the domains of greatest interest in the supplement. A final alternative is to use a battery of condition-specific measures. Because one of the major shortcomings of condition-specific measures is their narrowness and specificity (Kessler & Mroczek, 1995), the use of several different condition-specific tests may be an option. For example, Kjerulff and Langenberg (1995), in a study examining fatigue among patients having hysterectomy, used four different fatigue measures (Symptom Fatigue Scale, Profile of Mood States, and two scales from the Medical Outcome Study Short-Form General Health Survey). Although each of the four scales ostensibly measured the latent construct “fatigue,” each provided separate insights into how fatigue was affected by hysterectomy. Fatigue was broken down into three separate components: frequency of fatigue, the extent to which fatigue is problematic for the respondent, and the extent to which fatigue causes limitation of activity. The correlation of the components was high, but each predicted different events. For example, the extent to which fatigue is problematic was the best predictor of the number of physician contacts. Each scale contained unique information, and the authors of this study suggest that measurement of all three components is necessary for a complete analysis of fatigue. Table 6–2 summarizes the options for using condition-specific measures, either alone or in combination.
THE CHOICE OF A CONDITION-SPECIFIC MEASURE The Conceptual Model Simply choosing the “best” measure available for a given condition from a statistical perspective is inadequate. The choice of a particular conditionspecific measure should be guided more strongly by the investigator’s
34411_CH06_165_184
172
9/9/05
CHAPTER 6
10:01 AM
Page 172
CONDITION-SPECIFIC MEASURES
Table 6–2 Ways to Use Condition-Specific Measures Option
Discussion
Use a condition-specific and a generic measure.
Administering two different measures is potentially costly; otherwise, this is best option.
Modify generic measure.
A modified measure is not comparable with an unmodified measure; lose main advantage of generic measure.
Attach condition-specific supplement to generic . measure
Solid option; retains comparability of generic measure; necessary to find or develop supplement.
Use battery of condition-specific measures.
Pick condition-specific measures that tap all domains of interest. May be easier, cheaper, and more thorough to use generic and condition-specific measures; battery doesn’t retain the easy comparability of generic measures.
conceptualization of what the condition-specific measure ought to measure than by narrow statistical guidelines. The first step in picking a measure is to understand the natural history of the disease and to construct a theory regarding precisely how the intervention will impact the condition. With that model in place, available conditionspecific measures can be evaluated to find one that taps the exact domain where the intervention is expected to have an impact. In many cases, a disease can affect the life of an individual in many distinct ways. More importantly, a single disease can impact multiple domains of a single ill person’s life. The selection of appropriate domains to study is the key to the selection of an appropriate condition-specific health status measure. The selection of the domains is a difficult task (Kessler & Mroczek, 1995): The first issue that has to be confronted in selecting outcome measures concerns the appropriate domains. This is an easy issue to address in the abstract, because the researcher usually wants to measure all domains that might be importantly affected by the medical intervention under investigation. It is much more difficult to determine what these domains are in practice, however, because the intervention effects can be complex. (p. AS109)
34411_CH06_165_184
9/9/05
10:01 AM
Page 173
The Choice of a Condition-Specific Measure
173
One way to conceptualize this issue is to consider the types of information that can be drawn from study participants regarding the nature of their ailments. A health-status measure can be thought of as evaluating symptoms, signs, tests, or function. (See Table 6–3.) A symptom would be something reported by the patient but not confirmed by other “scientific” means. Symptoms are typically the easiest and lowest cost type of event to measure; one simply needs to ask the patient (Sherbourne & Meredith, 1992). Some domains, like pain, are very difficult (although possible) to measure in any other way. As the opinions of patients as to how they feel, symptoms are inherently subjective. There is a prejudice against using subjective patient opinions; patient opinions are not considered to be as scientific as opinions rendered by trained medical professionals. The major difficulty with symptoms is establishing validity. For example, many different health questionnaires ask patients to rate their own level of pain. What precisely does this measure? When asking a person how severe his or her pain is, is it compared to the worst pain imaginable? The worst pain the individual has ever felt? The worst pain felt recently? The level of pain the individual typically feels? The level of pain the individual fears he or she might feel? Self-reported health measures are strongly influenced by such factors as ethnicity (Meredith & Siu, 1995) and social class (Koos, 1954). One example of this is the classic study of the relationship between culture and pain, which showed that individual responses to pain depend on social, family, and culturally patterned
Table 6–3 Measurement by Condition-Specific Measures Definition
Example
Symptoms
Reported, but not confirmed by other means
Pain
Signs
Result reported by medical profession
Heart murmur
Test
Objective, reproducible finding by medical professional
Blood pressure
Function test
Measurement of item related to condition, but not condition itself
Test of patient’s ability to walk up stairs
34411_CH06_165_184
174
9/9/05
CHAPTER 6
10:01 AM
Page 174
CONDITION-SPECIFIC MEASURES
responses (Zborowski, 1952). Methods to standardize pain reporting to permit comparisons across patients have been developed (Kane, Bershadsky, Rockwood, Saleh, & Islam, 2005). However, it is not clear that symptoms reported by patients are, in fact, less reliable than other types of measures. Symptoms have inherent face validity. Moreover, patient feelings can provide unique insights. For example, studies of self-reported health have found that it is one of the best available predictors of mortality (Idler & Kasl, 1991). The problem may be less one of lower actual validity than one of lower perceived validity. A sign is a result reported by a medical professional after a direct examination of the patient. Signs are opinions or reports expressed by medical professionals. For example, a physician listening to a patient’s heart may report hearing a heart murmur. Signs are typically considered more valid than symptoms, although that may be as a result of professional prejudice rather than empirical truth. The validity and reliability of a professional opinion are dependent on such factors as the training of the professional, the focus and quality of the instrument, and the level of ambiguity of the topic (Feinstein, 1977). Although clinicians place much faith in clinical observation, the accuracy of such observations is well worth testing. A test is an objective finding by a medical professional, such as a laboratory test. A test is typically considered to be superior to symptoms and signs because of better validity. When a population is tested for a disease, for example, the exact same procedure and exact same criteria can be used for every single member of the population. Every member of the population can give a blood sample of the same size. Every blood sample can be treated in the same way. The exact same antibody threshold can be used every time. With tests, extremely high validity can be established. Then again, trust in tests may be due to professional prejudice rather than true relationships. With many tests, interpretation of the results is necessary. For example, in an echocardiogram, after the test is complete, a medical professional needs to interpret the ultrasonic record. The ultrasonic record is shown as shadows on a monitor. The rater must make a judgment about the presence of potential anomalies. The quality, reliability, and validity of the test therefore depend entirely on the quality, reliability, and validity of the interpretation. Another example of this is found in radiology. The rate of correct interpretations of mammograms of patients with cancer was between 74 and 96 percent (Elmore et al., 1994). The correct interpretation rate for patients without cancer ranged from 89 to 35 percent. Tests can be just as fallible as symptoms or signs.
34411_CH06_165_184
9/9/05
10:01 AM
Page 175
The Choice of a Condition-Specific Measure
175
Although one may feel confident in using a direct physiological measure as an outcome, even the simplest measure can produce unforeseen problems. For example, blood pressure seems like a straightforward parameter; but even as simple a measure as this can be presented in many ways. The way the variable is defined can affect the result and even dramatically alter the interpretation. An analysis by Berlowitz and his colleagues (1997) showed that depending on how blood pressure was used (e.g., last diastolic blood pressure [DBP] determination >90, mean DBP over 1 year) the relative performance of clinical sites changed. Finally, a function test does not attempt to measure aspects of the condition directly, as do symptoms, signs, and tests, but rather measures the impact of the condition on day-to-day life. Many generic health status measures operate on the functional level, but most of them use reported function rather than direct testing of performance. A function test may be directed at a single joint (like range of motion) or at the lower or upper extremities (e.g., walking or grip strength). Care needs to be exercised with measures that utilize function tests. A test that measures, for example, a patient’s ability to walk a specified distance measures just that: the ability of a patient to walk a specified distance. This point may seem quite obvious, but is often overlooked. Investigators often use function tests to measure some underlying disease state. For example, if arthritis limits the ability to walk, does a test of a person’s ability to walk measure arthritis? To answer “yes” involves a leap of faith (and is also a logical fallacy). (See Chapter 4 on measurement for more discussion of this topic.) This leap can be made, but should be made only in the presence of a strong conceptual model. Consider the following as an example of the type of mistake that this approach could allow. An investigator attempts to measure the relationship between an intervention and arthritis, using walking distance as a function test. The intervention is exercise. The experimental group increases their fitness level as a result of the exercise and therefore walk farther, although the arthritis is unchanged. Increased scores on the function test, driven by changes in fitness levels, are then falsely interpreted as improvements in arthritis.
Hierarchy of Measurement Each type of test has different shortcomings and will tap a different domain of the impact of the condition on the patient. Many domains can
34411_CH06_165_184
176
9/9/05
CHAPTER 6
10:01 AM
Page 176
CONDITION-SPECIFIC MEASURES
be measured by each of the different methods. For example, a measure of health could be a function test (like performance ADL) or a symptom reported by a patient (self-reported health). Rheumatoid arthritis can provide an example of the hierarchy of measure. Spiegel and colleagues (1986) examined some of the available conditionspecific tests for rheumatoid arthritis to explore which domains each measure covered. First, symptoms are reported by the patient: morning stiffness and pain. Next, signs can be discovered upon physical examination: tenderness, swelling of joints, and joint deformity. Tests are available, such as X-rays and laboratory tests for the presence of an inflammatory disorder. A series of performance tests is available, notably a grip test and a walk time test. Finally, generic health status measures such as activities of daily living can measure functional status. Which is the correct test to use? Rheumatoid arthritis is a chronic, symmetrical arthritis affecting synovial lining of joints. Initially, the patient typically experiences swelling and tenderness of affected joints, followed by pain and stiffness. Eventually, the range of motion of the joints may become limited, joints may become deformed, and cysts may form. Other problems associated with rheumatoid arthritis include malaise and anemia. Although other complications are much less common, rheumatoid arthritis can affect almost every organ system, including the heart and lungs. This is an example of a disease primarily associated with the musculoskeletal system having complications in many different systems (which argues for the inclusion of a generic health status measure along with a condition-specific measure). There is no cure for rheumatoid arthritis. Interventions include exercises and splints for sleeping to increase range of motion (also surgery in extreme cases). Symptomatic pharmacologic therapy can reduce inflammation and pain. Rheumatoid arthritis is rarely fatal, but pain, suffering, and impairment can be extreme. Up to 15 percent of patients become fully incapacitated (Fishman, 1985). With this background, the question of which test to use can be re-asked. Which measure is appropriate depends entirely on the research question. Measures that address treatment for an acute exacerbation may be very different from those that address performance. Acute measures concentrate on evidence or effects of inflammatory response, such as joint counts and sedimentation rates. Using performance measures like walk time and grip strength to detect acute events must consider that they are strongly related to joint deformity. Regardless of a person’s acute status, extensive joint deformity may dramatically impair performance. In contrast, clinical esti-
34411_CH06_165_184
9/9/05
10:01 AM
Page 177
The Role of Condition-Specific Versus Generic Measures
177
mates of disease activity and pain by physicians and patients appear to be strongly related to disease activity, but not as strongly to functional status. Global functioning may be affected by a person’s mental state. Depressed or discouraged patients may find themselves less able to cope with social roles at the same level of disease activity as those unimpaired by such feelings. The choice of an appropriate outcome measure for rheumatoid arthritis depends entirely on the expected impact of the intervention and knowledge of the relationship of the measure to other clinically significant factors. When selecting a particular condition-specific measure, the investigator should have built a conceptual model to facilitate the identification of important domains. In this rheumatoid arthritis example, it is necessary to understand the natural history of the disease and the expected impact of the intervention prior to the selection of the condition-specific measure. The bottom line is that in order to select an appropriate condition-specific measure, investigators must know what they wish to measure.
THE ROLE OF CONDITION-SPECIFIC VERSUS GENERIC MEASURES Generic and condition-specific measures can complement each other. The domains measured by condition-specific measures may resemble those addressed by generic measures, but they are treated differently. When an investigator selects a condition-specific measure, the reason is typically to measure domains more deeply than they are measured by generic measures. The cataract eye surgery study mentioned earlier provides an example of this goal (Damiano et al., 1995). If it is decided to use both condition-specific and generic measures, the investigators should be clear as to why both are being included in the study. Often, generic measures are used as a safety net; they are designed to capture unexpected results of the intervention. This strategy is associated with statistical trolling, casting a wide net without an underlying model in the hopes that something will prove interesting in retrospect. This strategy is often used early on in the research of a new treatment. As noted in the first chapter, hypothesisdriven investigations will lead to stronger conclusions and avoid misleading observations based on chance findings. Typical statistical measures, such as t-tests, are designed to test hypotheses. If a 95 percent significance level is used to test hypotheses, then type I
34411_CH06_165_184
178
9/9/05
CHAPTER 6
10:01 AM
Page 178
CONDITION-SPECIFIC MEASURES
errors (rejecting the null hypothesis when the null hypothesis is true) will occur 5 percent of the time. Therefore, in 20 tests, one would be expected to be significant simply due to chance. Simply throwing a battery of tests at a problem, without any underlying conceptual model, can lead to false conclusions. Instead, generic measures should be incorporated to test specific hypotheses. An intervention’s primary impact may be in one domain, but the intervention may also be expected to have secondary impacts in several different domains. For example, knee surgery might affect not just mobility, but could potentially also affect other domains such as mental health (if increased mobility reduced depression caused by isolation). Bear in mind that more generic effects are usually the result of more complex interactions. Another reason for incorporating generic measures into a study is that overall health, as measured by generic health measures, may differentially affect the intervention’s impact on the main (condition-specific) outcome measure. Consider, for example, back surgery as an intervention for patients with back pain. The purpose of the intervention is to decrease pain. A condition-specific outcome measure, such as the Roland Low Back Pain Rating Scale (Roland & Morris, 1983), could be used to measure the levels of pain both before and after the intervention. The success of the intervention may depend on overall health status. People suffering from a multitude of ailments may not gain much relief from the surgery, even if the surgery worked perfectly. Such a result does not mean the intervention was a failure. Rather, it means the success of the surgery depends on overall health status. A condition-specific measure can convey the result that the surgery worked, which is necessary information for the narrow evaluation of the surgery. A generic measure could reveal that for some patients, the surgery had no positive impact on the overall well-being of the patient. The latter information is necessary for evaluating, for example, the cost effectiveness of a treatment, or it could be used in targeting a treatment toward populations likely to gain benefit from the treatment. The notion that generic and condition-specific measures should be used in tandem has been criticized by Dowie (2002) as being inconsistent with the goal of making decisions on the allocation of scarce health care resources in a transparent and explicitly coherent, preference, and evidencebased process. Dowie argues that in situations when the generic and condition-specific measures disagree, there must be a transparent prespecified decision rule for selecting which measure to accept as definitive. However, if the decision maker has already decided that one or the
34411_CH06_165_184
9/9/05
10:01 AM
Page 179
The Role of Condition-Specific Versus Generic Measures
179
other measure (generic or condition specific) will be disregarded if it disagrees with the preferred measure, what is the purpose in using both? Further, the argument that a condition-specific measure can find small but important differences is also criticized; if a change is too small to impact overall health, what is the purpose in inflating the change to make it seem larger? This is likened to selecting a larger map to make two cities seem further apart; the distance between the cities is the same regardless of the scale of the map. Similarly, if a change in health status is too small to impact overall health, what is the purpose of administrating a measure sensitive enough to detect irrelevant differences? However, this argument presupposes that there is agreement on the definition of a meaningful difference. Generic measures typically find changes in health-related quality of life. In contrast, condition-specific measures are often more akin to clinical measures and can more easily find clinically relevant differences—but what is the purpose in making a clinically relevant change in a person’s health that has no effect on their health-related quality of life? Clinicians and decision analysts often disagree about this question. Decision analysts argue that treatment should be focused on the person, not the condition, which leads to the conclusion that a generic measure—examining the impact on the entire person—is preferable to a condition-specific measure, which focuses only on the condition. Most measures have a set of patient preferences implicitly built in that determines the relative value of different health states. Generic measures often reflect community preferences, whereas condition-specific measures may reflect the preferences of the particular population with the condition (Feeny, 2002). Many condition-specific measures cover multiple domains and may measure health-related quality of life, as do the generic measures. Key difference is the preference weights built into the measures; conditionspecific measures more accurately reflect the preferences of the target population. The argument that differences important to the individual’s healthrelated quality of life will be found by a generic measure also assumes that the generic measure is the final arbitrator of meaningful differences. However, this is a difficult argument given the dramatic differences in widely accepted generic measures in the number of questions, number of possible levels of responses, and included domains, as detailed in the previous chapter. For example, on a 0–1 scale, the difference between perfect health and any level of illness in the EQ-5D is 0.12 points (from a score of 1 to 0.88) (Brazier & Fitzpatrick, 2002). Therefore, a decline in health from
34411_CH06_165_184
180
9/9/05
CHAPTER 6
10:01 AM
Page 180
CONDITION-SPECIFIC MEASURES
full health to 0.95 health units (for example) is undetectable by the EQ-5D, although a different measure, such as the SF-36, might be able to demonstrate such a change. If different generic measures disagree on the effect of a treatment or condition on an individual’s health-related quality of life, how can a particular generic measure be considered definitive?
CHOOSING A MEASURE Selecting the best condition-specific health status measure for a study can be difficult. Investigators can either create a new measure or use an already developed measure. Despite the appeal of a customized measure, the work involved is substantial. Moreover, the acceptance of the results of the study may hinge on the acceptance of the measure. Pioneering investigators must first provide strong and convincing evidence that their measure is reliable, internally consistent, and valid before even beginning to discuss the substantive results of the study. Further, the results of the study with the new measure will be hard to compare with those from any previous study. These drawbacks, combined with the time and cost associated with the development of a new measure, argue strongly against developing a new measure unless no available measures are acceptable. The better option is to choose a condition-specific measure developed and validated by other investigators. For many conditions, there is a multitude of condition-specific measures (for example, for the measurement of arthritis, at least five standard condition-specific outcome measures are available) (Patrick & Deyo, 1989). The book Measuring Health by McDowell and Newell (1996) presents nearly 100 measures for common conditions, such as pain, mental status, depression, and physical disability. The Ovid database interface provides access to the Health and Psychosocial Instruments database that contains more than 15,000 references to articles and measures. The selection of a particular condition-specific measure from the measures available should be guided by statistical, theoretical, and practical criteria. Statistically, the investigator should seek measures that are reliable, valid, unbiased, and precise in the range where effects are expected, and easy to implement (Kessler & Mroczek, 1995). An extended discussion of these issues is available in Chapter 4. Theoretically, the measure should cover the domains of greatest interest. The determination of the domains must be driven by a theoretical model of the disease or condition and how it will interact with the treatment.
34411_CH06_165_184
9/9/05
10:01 AM
Page 181
Summary
181
Practical considerations should be taken into account. The mode of administration of the measure needs to be consistent with the overall design of the study. For example, it would be improper to have study participants self-administer a measure designed for use in a phone survey. Measures that have been used previously in similar studies should be given preference over infrequently used or new measures, provided they seem to capture the salient information on the effects being targeted. Widely used measures will facilitate comparisons of the results of the study with previous studies. The investigator should also consider the distribution of expected scores in the study population. Measures that minimize the probability of floor or ceiling effects should be selected. For example, if the study population consists of frail elderly, avoid as much as possible measures designed for healthy populations. Some measures require the use of complicated scoring algorithms. Some are also bulky or complicated. All other things equal, simpler and shorter are better. The time frame of measures also varies. For example, the SF-36 asks about the previous 4 weeks. Other measures may ask about the previous 6 months. Some may ask about right now. A measure should be selected with a time frame appropriate for the intervention and condition. It is also worth considering how the results of the study will be analyzed. The method of analysis should be established prior to the beginning of the study. This enables the investigator to select appropriate statistical tests and thereby to conduct a power analysis and determine the necessary sample size.
SUMMARY The proper selection of the outcomes measure is a key to a successful outcomes study. The best approach to picking a measure is to first acquire an understanding of the natural history of the condition, then develop a theory of how the intervention will interact with the condition. This will allow for the discovery of the domains in which the intervention is expected to impact. Condition-specific measures are focused, precise measures that are able to delve deeply into the domains of greatest interest. Condition-specific measures should be teamed with appropriate generic measures so that the intervention can be evaluated, not just with regard to its narrow impact on the condition, but also for its impact on overall health across a multitude of domains.
34411_CH06_165_184
182
9/9/05
CHAPTER 6
10:01 AM
Page 182
CONDITION-SPECIFIC MEASURES
Condition-specific measures can play a central role in assessing the outcomes of care. Rather than engaging in a debate about their relative merits compared to generic measures, they are best seen as complements to generic measures. Some condition-specific measures come directly from clinical practice; others must be created. Care must be exercised not to place undue confidence in the reliability of measures derived directly from practice.
REFERENCES Barry, M., Fowler, F., O’Leary, M., et al. (1995). Measuring disease-specific health status in men with benign prostatic hyperplasia. Medical Care, 33(S4), AS145–AS155. Bellamy, N., Buchanan, W.W., Goldsmith, C.H., et al. (1983).Validation study of WOMAC: A health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. Journal of Rheumatology, 15(12), 1833–1840. Berlowitz, D.R., Ash, A.S., Hickey, E.C., et al. (1997). Outcomes of hypertension care: Simple measures are not that simple. Medical Care, 35(7), 742–746. Bombardier, C., Melfi, C., Paul, J., et al. (1995). Comparison of a generic and a disease specific measure of pain and physical function after knee replacement surgery. Medical Care, 33(S4), AS131–AS144. Brazier, J., & Fitzpatrick, R. (2002). Measures of health-related quality of life in an imperfect world: A comment on Dowie. Health Economics, 11, 17–19. Damiano, A., Steinberg, E., Cassard, S., et al. (1995). Comparison of generic versus disease specific measures of functional impairment in patients with cataract. Medical Care, 33(S4), AS120–AS130. Dowie, J. (2002). Decision validity should determine whether a generic or conditionspecific HRQOL measure is used in health care decisions. Health Economics, 11, 1–8. Elmore J., Wells, C., Lee, C., et al. (1994). Variability in radiologists’ interpretations of mammograms. New England Journal of Medicine, 331(22), 1493–1499. Feeny, D. (2002). Commentary on Jack Dowie: Decision validity should determine whether a generic or condition-specific HRQOL measure is used in health care decisions. Health Economics, 11, 13–16. Feinstein, A.R. (1977). Clinical biostatistics. St. Louis, MO: C.V. Mosby. Fishman, R.A. (1985). Normal-pressure hydrocephalus and arthritis. New England Journal of Medicine, 312(19), 1255–1256. Fitzpatrick, R., Davey, C., Buxton, M., et al. (1998). Evaluating patient-based outcome measures for use in clinical trials. Health Technology Assessment, 2, i–iv, 1–74. Guyatt, G.H., Feeny, D.H., & Patrick, D.L. (1991). Issues in quality-of-life measurement in clinical trials. Control Clinical Trials, 12, 81S–91S. Guyatt, G.H., Feeny, D.H., & Patrick, D.L. (1993). Measuring health-related quality of life. Annals of Internal Medicine, 118, 622–629.
34411_CH06_165_184
9/9/05
10:01 AM
Page 183
References
183
Idler, E.L., & Kasl, S. (1991). Health perceptions and survival: Do global evaluations of health status really predict mortality? Journal of Gerontology, 46(2), S55–S65. Kane R.L., Bershadsky, B., Rockwood, T., Saleh, K., & Islam, N.C. Visual Analog Scale pain reporting was standardized. Journal of Clinical Epidemiology, 58(6), 618–623. Kazis, L.E., Anderson, J.J., & Meenan, R.F. (1989). Effect sizes for interpreting changes in health status. Medical Care, 27(3 Suppl), S178–S189. Kessler, R., & Mroczek, D. (1995). Measuring the effects of medical interventions. Medical Care, 33(S4), AS109–AS119. Kjerulff, K., & Langenberg, P. (1995). A comparison of alternative ways of measuring fatigue among patients having hysterectomy. Medical Care, 33(S4), AS156–AS163. Koos, E.L. (1954). The health of Regionville. New York: Columbia University Press. McColl, E., Eccles, M., Rousseau, N., et al. (2003). From the generic to the condition specific? Instrument order effects in quality of life assessment. Medical Care, 41(7), 777–790. McDowell, I., & Newell, C. (1996). Measuring health. A guide to rating scales and questionnaires (2nd ed.). New York: Oxford University Press. Meredith, L.S., & Siu, A.L. (1995). Variation and quality of self-report health data: Asian and Pacific Islanders compared with other ethnic groups. Medical Care, 33(11), 1120–1131. Patrick, D.L., & Deyo, R.A. (1989). Generic and disease-specific measures in assessing health status and quality of life. Medical Care, 27(3), S217–S232. Roland, M., & Morris, R. (1983). A study of the natural history of back pain, Part I: Development of a reliable and sensitive measure of disability in low-back pain. Spine, 8(2), 141–150. Sherbourne, C., & Meredith, L. (1992). Quality of self-report data: A comparison of older and younger chronically ill patients. Journal of Gerontology, 47(4), S204–S211. Spiegel, J.S., Ware, J.E., Ward, N.B., et al. (1986). What are we measuring? An examination of self-reported functional status measures. Arthritis and Rheumatism, 31(6), 721–728. Sudman, S., & Bradburn, N. (1974). Response effects in surveys. Chicago: Aldine Publishing Co. Temkin, N.R., Dikmen, S., Machamer, J., et al. (1989). General versus disease-specific measures: Further work on the Sickness Impact Profile for head injury. Medical Care, 27(3), S44–S53. Ware, J.E.J., & Sherbourne, C.D. (1992). The MOS 36-Item Short-Form Health Survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30(6), 473–483. Zborowski, M. (1952). Cultural components in response to pain. Journal of Social Issues, 8, 16–30.
34411_CH06_165_184
9/9/05
10:01 AM
Page 184
34411_CH07_185_216
9/9/05
11:48 AM
Page 185
7 Satisfaction with Care Maureen A. Smith Chris Schüssler-Fiorenza Todd Rockwood
INTRODUCTION: THE IMPORTANCE OF PATIENT SATISFACTION Patient satisfaction is often defined as “a healthcare recipient’s reaction to salient aspects of his or her service experience” (Pascoe, 1983). Research on patient satisfaction has grown in importance over the last 30 years. Initially, interest in patient satisfaction arose from sociological research in the 1950s indicating that increased patient satisfaction could improve clinical outcomes by improving compliance with appointment keeping, medication use, and following treatment recommendations (Williams, 1994). Further research showed that satisfied patients were also less likely to sue for malpractice (Hickson et al., 1994). Satisfaction was also viewed as a direct goal of health care (Cleary & McNeil, 1988). A patient-centered approach to medicine has recently emerged and recast the patient in a more active role, which has made the patient’s perspective more important (Mead & Bower, 2000). Currently, increasing patient participation in the health care process is thought to have intrinsic benefits and to promote better health outcomes (Greenfield, Kaplan, & Ware, 1985). The rise in consumerism and the attempt to apply consumer models to health care has required that patients (now customers) have adequate information to make informed choices about their health care (Hudak, McKeever, & Wright, 2003). Health care organizations have begun to conduct patient satisfaction surveys and use the results as a marketing tool. At the same time, there is a growing emphasis on evidence-based medicine and on evaluating the quality of medical care. Another goal of patient 185
34411_CH07_185_216
186
9/9/05
CHAPTER 7
11:48 AM
Page 186
SATISFACTION WITH CARE
satisfaction research is to incorporate patients’ views on what values should be associated with different outcomes when defining quality care (Cleary & McNeil, 1988). Patient satisfaction surveys are currently implemented as a measure of health care quality, linked with national accreditation measures, and may be tied to financial reimbursements to providers. As Donabedian (1966) advocated, “Achieving and producing health and satisfaction, as defined for its individual members by a particular society or subculture, is the ultimate validator of the quality of care.”
THEORETICAL MODELS OF SATISFACTION There is no universally accepted theoretical model of patient satisfaction. A commonly cited model, however, is the expectancy disconfirmation model. This model is adapted from the customer service literature, in which the customer compares his or her expectations with the service performance. When service performance exceeds expectations, it results in customer satisfaction, and if it fails to meet expectations, the result is customer dissatisfaction. A modification of this model is called the cognitive-affect model of satisfaction, in which perception of service performance includes a cognitive evaluation, affective response, and a direct effect on satisfaction (Oliver, 1993). The affective response can be tempered or increased by attributions of causality and perceptions of equity (Figure 7–1). Whether cognitive evaluation or affective response predominates may also be a function of the service provided. In Oliver’s study of satisfaction with respect to cars and with regard to an educational course, disconfirmation was the best predictor of satisfaction with cars, whereas affect was a better predictor of satisfaction with the course (Oliver, 1993). Because health care services are more similar to educational services than using a car, these findings suggest that including affect/attribution in the conceptual model of patient satisfaction may improve the understanding of patient satisfaction (Thompson & Sunol, 1995). An addition to Oliver’s model is the concept of feedback loops, in which a patient’s satisfaction can affect his or her subsequent behavior which then provides feedback to the service provider (Crow et al., 2002; Strasser, Aharony, & Greenberger, 1993). Patient characteristics, values, beliefs, and experiences affect both their expectations and their attributive style. Providers also can shape patient expectations through information about treatment and outcomes.
Measurement Technique May affect evaluation that is recorded
Behavorial Reaction
Theoretical Models of Satisfaction
Figure 7–1 Model of Patient Satisfaction. Adapted from Oliver, 1993; Crow et al., 2002.
Equity/Inequity
Satisfaction
11:48 AM
Attribution
Positive/Negative Affect
Disconfirmation
9/9/05
Individual Characteristics: - Values, Beliefs, Experiences - Personality - Health Status - Sociodemographics
Attribute Performance
Expectations
34411_CH07_185_216 Page 187
187
34411_CH07_185_216
188
9/9/05
CHAPTER 7
11:48 AM
Page 188
SATISFACTION WITH CARE
INTERPRETING SATISFACTION RATINGS
Expectations and Psychosocial Determinants Although the expectations-disconfirmation model is frequently studied and frequently critiqued, one review found that only 20 percent of patient satisfaction research articles included expectations in their study (Crow et al., 2002). Although, some studies, particularly in the United States, have found positive association between expectations and satisfaction, the amount of variance of satisfaction explained by expectations remains low, at no more than 20 percent and often much less than that in some studies (Thompson & Sunol, 1995). There have been several theorized mechanisms for this. One explanation incorporates the concept of an assimilation effect, which states that when performance varies only slightly from expectations, people’s perceptions tend to place it at their expectations, and it is only at the extremes that satisfaction or dissatisfaction is recorded (Thompson & Sunol, 1995). In addition, there is evidence that satisfaction is expressed not just with unexpected good performance, but that patients may also mark satisfied if nothing unexpected happens, meaning dissatisfaction may only be expressed at markedly unexpected negative performances (Williams et al., 1995). People may also tolerate a range of performances from a defined adequate level to their ideal performance called a zone of tolerance, and they will be satisfied with any performance that falls inside. The less choice a consumer has, the wider the person’s zone of tolerance (Edwards, Staniszweska, & Crichton, 2004). Another critique of the use of expectations is the argument that many patients come in with unformed expectations. This appears to occur more frequently in the United Kingdom than in the United States and is cited as a reason why expectations have not been found to correlate well with satisfaction in the United Kingdom, in contrast to the United States where a positive relationship has been found (Crow et al., 2002). However, even patients who do not have specific expectations with respect to medical testing, treatment, and outcome may still have expectations for the interpersonal interaction and information obtained during the visit. In addition, it is unclear how the health care experience and its results provide feedback to the patient and can inform his or her expectations and attributions in the future. Expectations in the context of medical care may be more complex, because an important component of many medical care encounters is to
34411_CH07_185_216
9/9/05
11:48 AM
Page 189
Interpreting Satisfaction Ratings
189
provide patients with diagnostic information and information about what to expect about treatments, complications, duration, and outcome. Thus, patients who come in with unformed expectations may then acquire expectations about their treatment and outcome, and patients with formed expectations about treatment that differ from the physician’s preferred course of care may change their expectations in the face of a persuasive explanation from the physician. How these new expectations then correlate with satisfaction with treatment and outcomes has not been well studied. However, patients who obtain information about what to expect have increased satisfaction (Crow et al., 2002; Krupat, Fancey, & Cleary, 2000), and complaints of lack of information can result in the desire to switch physicians (Keating et al., 2002). Another major problem with expectations and satisfaction ratings is that they are conditioned by social norms and people may become habituated to lowered expectations over time. Thus, satisfaction surveys may show that people with poor care are satisfied, but this is because they have lowered expectations, not because they are receiving high-quality care. If satisfaction surveys are used uncritically, high satisfaction rates can be cited to maintain the status quo even if the quality of care is inadequate instead of serving to advance the patient/consumer perspective (Crow et al., 2002; Weithman, 1995). Thus, in order to fully understand consumer preferences in health care, one may need to ask patients about their ideal expectations, especially in cases in which patients’ desires differ from what is normatively available. Attribution is also an important component of the conceptual model. Williams and colleages (1998) studied how positive and negative effects are transformed into satisfaction ratings. They proposed a model in which the patient—when assigning satisfaction ratings—also takes into account the duty of the provider and whether the provider is culpable for the event. Hence, a client with a negative experience who perceives the service provider as not culpable for the experience or assumes that the service is outside the duty of the service provider will likely still mark satisfied. Equity judgments can also modify satisfaction ratings of negative affective experiences if they are perceived as equitable and may be particularly salient in countries with a national health service (Thompson & Sunol, 1995). The attribution part of the model may explain why trust in provider and evidence of caring are highly associated with satisfaction (Joffe, Manocchia, Weeks, & Cleary, 2003). If the patient trusts or believes that the provider has done his or her best and that negative performance is out of the provider’s control, the patient will still report high satisfaction.
34411_CH07_185_216
190
9/9/05
CHAPTER 7
11:48 AM
Page 190
SATISFACTION WITH CARE
Casting a patient into an active consumer role assumes that the patient accepts that role, when in fact the patient may still believe in a passive role and may not be able to or may choose not to question medical authority. The consumer model assumes the existence of a patient opinion, that the patient considers his or her opinion legitimate, and that the patient is willing to express his or her opinion (Williams, 1994). It is theorized that older patients still retain a more passive role with respect to medical authority, which might explain the higher satisfaction rates found in some studies, but this has never been directly tested (Crow et al., 2002; Sitzia & Wood, 1997). In addition, psychosocial pressures work to transform patients’ negative affective response into positive ratings. Even a patient who accepts a more active role is still in a dependent position in the health care system. Patients have a need for a positive working relationship and there are psychological incentives for patients to maintain a positive outlook about their care in part to maintain a positive outlook for their outcome (Edwards et al., 2004). This is in line with cognitive consistency theory in which patients need to justify their time, effort, and discomfort to themselves and thus report they are satisfied (Sitzia & Wood, 1997).
Dimensions of Satisfaction Patient satisfaction appears to be simultaneously a unidimensional and a multidimensional construct. Patients make unique summary judgments about their overall experiences with care, suggesting that a global measure of satisfaction may be appropriate (Aharony & Strasser, 1993). However, summary judgments of satisfaction show high rates of satisfaction and low response variability. When satisfaction is broken down into different components, there is greater evidence of dissatisfaction with certain components (Sitzia & Wood, 1997) and there is substantial evidence that satisfaction is multidimensional (Abramowitz, Cote, & Berry, 1987; Meterko, Nelson, & Rubin, 1990). Separate scales have been developed to measure more specific aspects of medical care such as access to services, technical quality, and a provider’s interpersonal skills. A review by Ware and colleagues (1983) of the acute care setting listed interpersonal manner, technical quality of care, accessibility/convenience, finances, efficacy/outcomes of care, physical environment, and availability as the eight dimensions of satisfaction. The lack of a widely accepted conceptual framework for dimensions of satisfaction not only leads to considerable variation of the dimensions
34411_CH07_185_216
9/9/05
11:48 AM
Page 191
Interpreting Satisfaction Ratings
191
between studies but also to the neglect of certain dimensions and outcomes from most studies (Hall & Dornan, 1988; Wensing, Grol, & Smits, 1994). Also little information is available on the relative importance of the various dimensions to patients (Sitzia & Wood, 1998). However, there appear to be some commonalities among important dimensions across health care settings such as interpersonal relations, communication, being treated with respect, and trust. These dimensions may be related to the overall conceptual model of satisfaction in terms of how patients attribute their experiences. However, there are clearly differences in dimensions across settings as well and the current conceptual models do not account for these differences. An overarching conceptual model should incorporate the range of possible dimensions across all health care settings. We propose a conceptual model of satisfaction dimensions that builds on the Donabedian framework for quality assessment (structure, process, and outcome) (Donabedian, 1988). This model incorporates components of health services access and cost into the structural aspect of satisfaction, and incorporate care and health services quality into the process aspect of satisfaction with care. In addition, the World Health Organization (WHO) definition of health incorporates the concept of physical, mental, and social well-being (WHO, 1948). These dimensions can also guide approaches to thinking about health care and subdividing Donabedian’s categories. In different health care settings, these components of health are prioritized differently and this approach allows different settings of care to be more precisely distinguished. For example, social environment is critical for nursing home satisfaction, whereas medical aspects may be more important for ambulatory care and inpatient care, and the physical environment may be relevant for nursing home and hospital care. In this model, structural aspects of satisfaction include access, cost, and physical environment factors related more to the physical and social (access/cost) side of health care. Process aspects of satisfaction include technical quality that is mainly focused on physical and sometimes mental health, interpersonal relations/ communications and continuity/coordination of care reflective of the social side of medicine and important for psychosocial well-being and satisfaction. Outcomes aspects include satisfaction with health (physical, mental, and social) as an outcome of care (Figure 7–2). This model would enable one to identify dimensions that have been less frequently studied such as outcome and particularly its psychological and social components. In order to simplify the model for each setting of care, one would need more information on how patients prioritize dimensions as
Cost
Physical Environment
Figure 7–2 Dimensions of Satisfaction
Access
Psychosocial Needs Interpersonal
Technical Quality
Coordination/ Continuity
Physical
Mental
Outcome
Social
CHAPTER 7
11:48 AM
Process
9/9/05
Structure
192
Quality
34411_CH07_185_216 Page 192
SATISFACTION WITH CARE
34411_CH07_185_216
9/9/05
11:48 AM
Page 193
Interpreting Satisfaction Ratings
193
empirical studies suggest that priorities are different across different settings. For health plans, the structural aspects of quality (access, cost, and physical environment such as technology) are valued highly (Crow et al., 2002), whereas for inpatient care, the physical living environment and interpersonal interactions particularly with nursing care are more important (Crow et al., 2002; Rubin, 1990). The patient’s condition may also affect their satisfaction. For example, patients who present with an acute physical complaint in an ambulatory setting may prioritize outcome, whereas patients who present with a chronic disease might prioritize the process dimensions of satisfaction.
Outcomes and Satisfaction Although patient satisfaction is often considered as an outcome of health care, it is also affected by outcomes of care. One model of satisfaction argues that the major goal of patients who present with a health problem is not satisfaction but the resolution of the health problem (Sitzia & Wood, 1997). Despite this important insight, only a small minority of studies investigate the effect of outcomes on patient satisfaction (Hall & Dornan, 1988; Wensing et al., 1994). There is evidence, however, that outcomes affect satisfaction, although it is also possible for patients to be satisfied with their health care even with poor outcomes. Understanding these apparently contradictory results requires investigation of the feedback loops that influence satisfaction attitude formation. One study found that immediately after a visit to an acute care clinic, satisfaction was most related to lack of unmet expectations and receiving an explanation for a symptom’s cause and an estimate of likely system duration, whereas at 2 weeks and 3 months, satisfaction was related to improvement of the underlying symptom (Jackson, Chamberlin, & Kroenke, 2001). In this study satisfaction correlated with absolute symptom impairment as had been found previously (Jackson, Chamberlain, & Kroenke, 2001; Kane, Maciejewski, & Finch, 1997). One interpretation of these findings is that satisfaction at a later point in time “is really measuring satisfaction with the patient’s health outcome rather than the individual physician” (Jackson et al., 2001). An alternative interpretation might be that for the types of health conditions studied, the patients (who were not asked about expectations of outcomes) expected complete resolution of their symptoms and thus any residual symptoms were a source of dissatisfac-
34411_CH07_185_216
194
9/9/05
CHAPTER 7
11:48 AM
Page 194
SATISFACTION WITH CARE
tion. Another study that looked at sources of patients’ unmet expectations found that half the patients interviewed focused on length of symptom duration rather than on severity and that these patients interpreted refractory symptoms as a sign that the physician was off track (Kravitz et al., 1996). In addition to physical outcomes, the way illness affects functional and emotional well-being is very important for patients but can be overlooked by physicians (Kravitz et al., 1996). For certain disease states, the patients’ relationship to the involved body part, the meanings attached to treatment outcome, as well as how significantly the impairment affects occupational and social function may also greatly affect satisfaction (Hudak, McKeever, & Wright, 2004).
METHODS OF MEASURING SATISFACTION Measurement Methods All of the standard social research methods are capable of measuring satisfaction: archival, ethnographic, focus groups, and survey research. The first two of these methods have received little attention in the study of satisfaction, but can provide useful information. Data from existing sources, such as patient complaint records, can provide a means of identifying areas in which patients have expressed dissatisfaction with the performance of the organization or health care professionals. Open-ended interviews with patients can provide a detailed understanding of the dynamics of the health care provision process from a patient’s point of view and may be useful in exploring or generating hypotheses, particularly as an initial step in areas that have not been well researched. Focus groups have several advantages, but share the generalizability problem also found with archival and ethnographic methods. Focus groups can provide qualitative as well as quantitative data; they permit detailed exploration of specific events and present an opportunity for spontaneous information to emerge. The most common method to assess satisfaction is by survey. A closed response format survey relies on the standardization of measurement whereby all respondents are presented with the same questions and are constrained to respond in a uniform manner. They enable a sample of the population to be studied and the findings to be generalized to an entire population.
34411_CH07_185_216
9/9/05
11:48 AM
Page 195
Methods of Measuring Satisfaction
195
Psychometric Testing Reliability and Validity The reliability and validity of measures of satisfaction are crucial, although in the past they were often not tested or reported. One review article found that only 6 percent of instruments met minimal reliability and validity testing criteria (Sitzia & Wood, 1998). The main types of reliability at issue are inter-item reliability or internal consistency reliability of multi-item scales (for example, Cronbach’s alpha), test-retest reliability, and interobserver reliability for those measures administered by telephone or in-person interview. Test-retest reliability is reported even less frequently than internal consistency reliability and it may be difficult to discern how it has been affected by changing perceptions over time or poor recall. Surveys also need to be validated against internal and external criteria. Although measures are often tested for internal discriminant validity, which is the ability of patients to discriminate among different features of care, external discriminant validity or how patients’ discriminations compare to independent ratings is rarely tested (Aharony & Strasser, 1993). Approaches to external validation include comparison of patient ratings of care to those of other sources, comparison of the ratings to other variables theoretically related to patient satisfaction, and experimental manipulation of features of care (Rubin, 1990). The external validation of patient satisfaction and ratings of technical quality are particularly important. Otherwise, satisfaction measures will lack face validity, which is vital if these measures are to be used and accepted in the evaluations of health plans, hospitals, and clinicians (Ben-Sira, 1976; Davies & Ware, 1988; DiMatteo & Hays, 1980). Finally, satisfaction measures need to be validated in more than one population or setting of care. Many existing measures of satisfaction have not been validated in this way and ongoing research in this area is important. The response format for individual items has an impact on reliability and validity. The five-point Likert “agree-disagree” response scale requires more items per dimension to achieve validity and may be susceptible to acquiescent response bias. Ware and Hayes found that the “excellent–poor” format produced responses that were less skewed, had greater variability, and performed better on validity tests compared to a six-point “very satisfied–very dissatisfied” scale (Ware & Hays, 1988), although this finding was not repli-
34411_CH07_185_216
196
9/9/05
CHAPTER 7
11:48 AM
Page 196
SATISFACTION WITH CARE
cated in a more recent study (Hendriks, Vrielink, Smets, van Es, & De Haes, 2001). Hendriks and colleagues (2001) did find that word responses next to the items resulted in fewer missing items. Different subpopulations may respond better to certain response formats; for example, a study found that older adults preferred a 10-item visual analogue form and this version had greater response variability (Castle & Engberg, 2004). The CAHPS program chose a 0 to 10 numeric scale after performing cognitive testing and psychometric field data analysis, because of its ease of use in telephone administration and greater distribution of responses compared to a 5-point scale (Harris-Kojetin, Fowler, Brown, Schnaier, & Sweeny, 1999). Bias A number of biases may affect survey measures such as nonresponse bias, acquiescent response bias, and sociopsychological artifacts. Satisfaction surveys are plagued by low response rates, often less than 50 percent (Aharony & Strasser, 1993; Barkley & Furse, 1996; Sitzia & Wood, 1998). This is in contrast to the epidemiological literature where a response rate that is less than 80 percent is considered substandard. Nonresponse bias can greatly affect the validity of the results. Studies have shown that the demographics, utilization patterns, and health status differed between respondents and nonrespondents (Fowler et al., 2002; Lasek, Barkley, Harper, & Rosenthal, 1997; Mazor, Clauser, Field, Yood, & Gurwitz, 2002; Zaslavsky, Zaborski, & Cleary, 2002). Less-satisfied patients are less likely to respond, thus caution must be taken when interpreting the results of studies with satisfaction rates below 80 percent (Eisen & Grob, 1979; Ley, Bradshaw, Kincey, & Atherton, 1976; Mazor et al., 2002). Acquiescent response bias is the tendency to agree with statements regardless of actual content. It can be reduced by varying positive and negative statements in a survey. Sociopsychological artifacts refer to how responses may be affected by fear of retribution or social desirability. Social desirability is the tendency of respondents to offer answers that are consistent with values the respondent believes are held by the interviewer (Groves, 1989; Groves et al., 1988; Locander, Sudman, & Bradburn, 1976). This social influence, which pressures the respondent to provide an answer that is in line with normative expectations or self-enhancing presentation, is influenced both by the mode of survey administration as well as the substantive content of questions. It can be reduced by guaranteeing confidentiality (Singer, Von Thurn, & Miller, 1995).
34411_CH07_185_216
9/9/05
11:48 AM
Page 197
Methods of Measuring Satisfaction
197
Utilization Patterns, Survey Timing, and Reference Group Survey results may also be influenced by utilization patterns with high users of care and non- or low users of care rating satisfaction differently. Thus, surveys that enable differentiation between the satisfaction survey results of high users versus non- or low users of care enable greater focus and, in addition, analysis of sicker subpopulations whose contact time and satisfaction with health care may differ. The timing of survey administration can also influence results, because a patient’s perceived satisfaction with care may change over time (Aharony & Strasser, 1993). One cause of this change may be symptom improvement over time (Jackson et al., 2001). The reference standpoint of a satisfaction instrument is also important. A general reference asks patients to rate health care given to people in general; whereas a personal reference asks them how satisfied they are with their own health care. The personal referent has been found to be more specific and useful as an evaluation measure. However, it must also be noted that people tend to rate satisfaction with their own care higher than they do when using a general referent (Pascoe, Attkisson, & Roberts, 1983; Ware, Snyder, Wright, & Davies, 1983). It is unclear whether the skewed response distributions of the personal referent measure may be in part influenced by psychosocial artifacts such as the tendency to give socially desirable responses, reluctance to criticize provider, or because of fear of retribution (Hays & Ware, 1986). Other possible explanations include rating the general referent more negatively because of biases against doctors and health care in general or conversely a positive bias toward rating one’s own care higher because the tendency of people to think they are better off than others (Hall & Dornan, 1988).
Reporting Versus Rating The Consumer Assessment of Health Plan Study chose to have patients report their experiences with different attributes of health plans thought to be important to patient satisfaction rather than rating their satisfaction with these attributes. Thus, for example, instead of asking patients to rate their satisfaction with waiting time before seeing a doctor, the survey asks how long the patient had to wait. This avoids the problem that different patients may have different expectations of care and also avoids some of the subjectivity
34411_CH07_185_216
198
9/9/05
CHAPTER 7
11:48 AM
Page 198
SATISFACTION WITH CARE
associated with rating. However, this study maintained the global satisfaction rating items. The advantage of this approach is that it provides more objective information about patients’ experiences of the details of care and facilitates quality improvement projects. In addition, future customers of health plans can make their own value judgments about different aspects of care based on these reports. The disadvantage of this approach is that with only a summative rating of patient satisfaction, there is less information on the relative importance of various attributes of health plans to consumers.
EXISTING SATISFACTION MEASURES Health Plan The major dimensions of patient satisfaction with health plans are access, cost, quality of physicians, and plan administration (Table 7–1). In evaluating satisfaction of health plans, it may be necessary to also survey disenrollees to get a more accurate assessment of satisfaction because the most dissatisfied members may have left the plan, causing an increase in the satisfaction ratings due to the remaining members. One study found that adding disenrollee results resulted in a statistically significant decrease in satisfaction scores (Bender, Lance, & Guess, 2003). Another issue that needs further study in evaluating the quality of and patient satisfaction with health plans is the relationship between provider satisfaction and patient satisfaction. Patient satisfaction has been found to be associated with provider satisfaction (Haas et al., 2000), although the mechanisms have not been fully elucidated. One study suggested that sources of provider dissatisfaction are associated with restriction of choice of hospitals, strong influences of the managed care plan on practice, and types of financial incentives (Landon et al., 2002). These are issues that patients care about but may not experience directly because it is not always possible to know the choices that are not offered. If the goal of patient satisfaction research is to assess health plan quality and provide a basis for informed choice for consumers, understanding the relationship between patient and provider satisfaction or assessing provider satisfaction directly (particularly with issues of access of which the patient may not be fully aware, such as problematic financial assessments, restrictions on treatments, or referrals offered) may be a useful adjunct in assessment of quality.
Hargraves et al., 2003
Davies & Ware, 1991
Consumer Satisfaction Survey (CSS)
Source(s)
Consumer Assessment of Health Plans (CAHPS 2.0)
Name of instrument
Subscales = 0.80–0.97
Plan Rel/ICC 0.94/ = 0.58 0.88/ = 0.86 0.90/ = 0.75 0.95/ = 0.62 0.94/ = 0.51 0.88/na 0.82/na 0.93/na 0.96/na
Reliability
Content claimed predictive
Construct
Validity evaluated
Self, telephone, face to face
Mail, telephone administration Supplemental items Confirmatory Factor Analysis
Notes
11:48 AM
Access Finances Technical Quality Communication Choice and Continuity Interpersonal Care Services Covered Information Paperwork Costs of Care General Satisfaction Overall Care Time Spent Outcomes Overall Quality Overall Plan Plan Satisfaction
Doctors Who Communicate Courteous/Helpful Office Staff Getting Needed Care Customer Service Overall Ratings: Personal Doctor or Nurse Quality of Health Care Specialist Health Plan
Getting Care Quickly
Dimension
9/9/05
47 items
46 items/ 20 min
Items/Time
Table 7–1 Measures of Health Plan Satisfaction
34411_CH07_185_216 Page 199
Existing Satisfaction Measures 199
34411_CH07_185_216
200
9/9/05
CHAPTER 7
11:48 AM
Page 200
SATISFACTION WITH CARE
Consumer Assessment of Health Plans In 1995, the Agency for Health Care Policy and Research (AHCPR) sponsored a major collaborative effort headed by the Research Triangle Institute, RAND, and Harvard Medical School to develop a set of standardized, psychometrically tested instruments to evaluate consumers’ experiences of health plans. Since that time, the CAHPS 2.0 has become the most widely used survey instrument to evaluate patients’ experiences of health plans (Hargraves, Hays, & Cleary, 2003). It has undergone extensive psychometric testing and two revisions since it was first introduced. CAHPS is designed to provide information to consumers rather than purchasers of health plans (AHCPR, 1996). Items were developed through extensive cognitive testing and focus groups, which resulted in a higher concentration of items concerning access, interpersonal skills, and communication and fewer items on provider technical skills. Initial CAHPS research focused on health plan instruments, but the CAHPS project has expanded to include instruments to assess group provider care. Further projects to develop instruments evaluating hospital care, individual providers, ambulatory care experience, nursing home care experiences and research into cultural comparability, and the use of CAHPS results for quality improvement are ongoing (Agency for Healthcare Research and Quality, 2003b). Consumer Satisfaction Survey The Consumer Satisfaction Survey (CSS) was originally developed in 1988 by the Group Health Association of America (GHAA) (Davies & Ware, 1991) to enable employers and business coalitions to compare satisfaction across health plans and other employee health benefit options. It was developed with the advice and input of health plans and employers and also used a number of items from the Patient Satisfaction QuestionnaireIII. It utilizes the “excellent” to “poor” response scale, which some research suggests has superior psychometric properties (Ware & Hayes, 1988). A second version of the CSS was released in 1991 and was expanded to include more health plan-specific features such as range of services covered, availability of information, paperwork, and costs. The CSS has been extensively fielded by both mail and telephone procedures; national norms for several items are now available. CSS contributed substantially to the Annual Member Health Care Survey (AMHCS) developed by the National Committee on Quality
34411_CH07_185_216
9/9/05
11:48 AM
Page 201
Existing Satisfaction Measures
201
Assurance (NCQA) (Kippen, Strasser, & Joshi, 1997; NCQA, 1995). After CAHPS 1.0 was developed, the NCQA working together with AHCPR and the CAHPS consortium merged the CAHPS survey and AMHCS, which resulted in a new core set of items and developed a Health Plan Employer Data and Information Set (HEDIS) supplement to the revised CAHPS core. This new instrument was called the CAHPS 2.0H and has been used for NCQA accreditation and with HEDIS since 1999 (Agency for Healthcare Research and Quality, 2003b).
Hospital Both the literature review by Rubin (1990) and the review by Crow and colleagues (2002) found that nursing, physicians, interpersonal relations, and communication were important to patient satisfaction in the hospital setting (Table 7–2). Physical environment or ward atmosphere is also important. Several studies found that patient and staff ratings of quality of care were particularly associated with assessments of overall quality of nursing, overall quality of medical care, and ward atmosphere (Rubin, 1990). Hospital CAHPS Hospital CAHPS (HCAHPS) is currently in development. It underwent pilot testing of a draft instrument in 2003 and has a planned implementation date of summer of 2005. HCAHPS covers eight domains that were chosen through an extensive literature review, use of consumer focus groups, domains that were previously covered by CAHPS, and domains that performed well in terms of reliability and validity on pilot tests. Domains that the HCAHPS developers chose not to include are billing, emergency room/department, emotional support, family, food, privacy, technical skills, and convenience. HCAHPS may also be combined with surveys that hospitals currently use for internal purposes, which has the advantage of enabling comparison between hospital results for the core items but enables individual hospitals to customize the survey to evaluate issues particular to that hospital. Internal consistency reliability, hospital level reliability, and construct validity were tested during the pilot study. The study also looked at case mix differences and potential response bias (Agency for Healthcare Research and Quality, 2003a).
106 items
Admissions Nursing and Daily Care Medical Care Hospital Environment Information Discharge
Concern for Patient Doctor Communication Medication Nursing Services Discharge Information Pain Control Physical Environment Global Ratings (hospital care, doctor care, nursing care)
Dimension Construct
Validity evaluated Pilot studies early 2004, mail, telephone implementation summer 2005
Notes
Mail or telephone = .87–.95 for Content administration claimed; subscales Construct; Convergent; Discriminant; Predictive
= .51–.89 composites
Reliability
CHAPTER 7
11:48 AM
Meterko et al., 1990
32 items
Items/Time
9/9/05
Patient Judgments on Hospital Quality (PJHQ)
HCAHPS (draft) CAHPS II AHRQ
Source(s)
202
Name of instrument
Table 7–2 Measures of Hospital Satisfaction
34411_CH07_185_216 Page 202
SATISFACTION WITH CARE
34411_CH07_185_216
9/9/05
11:48 AM
Page 203
Existing Satisfaction Measures
203
Patient Judgments on Hospital Quality The Patient Judgments on Hospital Quality (PJHQ) does not focus exclusively on patient satisfaction but also looks at how patients judge key attributes of hospital quality (Nelson, Hays, Larson, & Batalden, 1989). The survey items were constructed from literature reviews, preexisting questionnaires, focus groups with patients, and interviews with hospital administrators, physicians, and nurses. The survey development process has been extensively described in an eight-article supplement to the journal Medical Care (Meterko et al., 1990). In addition to the standard form, a short form containing 69 items has also been developed (Hays, Larson, Nelson, & Batalden, 1991). The PJHQ system has been used extensively in nonpoor populations with adult patients as well as with the parents of pediatric patients (McGee, Goldfield, Riley, & Morton, 1997). There is less information on its use with inpatients who have psychiatric or organic brain syndrome diagnoses or in poor populations.
Ambulatory Care Ambulatory care encompasses a variety of care, including preventive care, acute symptom care, and chronic care (Table 7–3). Issues of interpersonal relationships with providers, access and waiting times, and continuity of care are important. Although the importance of outcomes of care may vary depending on type of problem, outcomes clearly influence the satisfaction of patients presenting with acute physical problems (Jackson et al., 2001; Marple, Kroenke, Lucey, Wilder, & Lucas, 1997). When asking about a primary provider, it is important to note that women may have additional difficulties in completing survey instruments that assume only one primary care doctor, because women often see a gynecologist regularly either as their only doctor or in conjunction with their internist. Thus, they may be confused on items asking them to rate their regular doctor. The Women’s Primary Care Satisfaction Survey was designed to overcome this problem and to be more focused on attributes of primary care that affect women (Scholle, Weisman, Anderson, & Camacho, 2004). Ambulatory CAHPS Ambulatory CAHPS (ACAHPS) is a set of surveys currently under development for the evaluation of patient experiences and satisfaction with
CAHPS, 2004b Access Doctor Communication Office Staff Courtesy, Helpfulness, Respect Shared Decision Making Coordination/Integration Health Promotion Education Customer Service
subscales = .82–.94
Construct
Construct, discrim.
Validity evaluated
In development
Notes
CHAPTER 7
11:48 AM
A-CAHPS
General Satisfaction Technical Care Interpersonal Care Waiting Time
= .82–.89
Reliability
9/9/05
7 items
Visit-Specific Ware & Satisfaction Hays, Questionnaire 1988 (VSQ)
Dimension Interpersonal Manner Communication Technical Competence Time Spent with Dr. Financial Aspects Access to Care General Satisfaction
Items/Time
Marshall, et 50 items/ Patient al.,1993 9–12 min Satisfaction QuestionnaireForm III (PSQ III)
Source(s)
204
Name of instrument
Table 7–3 Measures of Ambulatory Satisfaction
34411_CH07_185_216 Page 204
SATISFACTION WITH CARE
34411_CH07_185_216
9/9/05
11:48 AM
Page 205
Existing Satisfaction Measures
205
ambulatory care. The goal of CAHPS is to develop instruments that can assess patient satisfaction at a number of levels in the health care system, including sites of care and clinicians. The CAHPS developers identified the following domains as important functional areas in ambulatory care: access, doctor communication, coordination of care, shared decision making, office staff courtesy, helpfulness and respect, cultural sensitivity, customer service, health promotion, and education. For each level of the health care system, the plan is to have a questionnaire that will consist of core items that will enable comparisons of result across survey users. It also will have supplemental items to enable greater focus on a specific function of the health care system or specific attributes of respondents. In addition, the survey will collect information on patients’ global evaluations and the characteristics of survey respondents. Although a number of planned items for the surveys have been developed and tested, the CAHPS development team is in the process of soliciting input from stakeholders with respect to further items for each function and for how the functions fit into the various levels of the health care system (CAHPS, 2004b). Given that the health plan and hospital versions of the CAHPS survey have set a standard for evaluation of patient satisfaction in these areas, it is likely that ACAHPS will accomplish the same thing in the area of ambulatory care. Visit-Specific Satisfaction Instrument The Visit-Specific Satisfaction Instrument (VSQ) was designed to be a focused, short instrument to measure satisfaction with specific features of care at the time of a specific medical visit (Ware & Hays, 1988). It was developed using prior visit-specific satisfaction surveys as a guide. Several response formats were tested with the VSQ and the “excellent–poor” format was ultimately recommended (Ware & Hays, 1988). Both a 51-item version and a 7-item version were psychometrically tested. A 9-item version was developed for use in the Medical Outcomes Study and remains available (Davies & Ware, 1991). Patient Satisfaction Questionnaire III The original Patient Satisfaction Questionnaire (PSQ) was developed by Ware in the 1970s and it originally contained 80 items that were selected from a large pool of items obtained through extensive literature reviews, focus groups, analyses of reasons for disenrollment, and analyses of
34411_CH07_185_216
206
9/9/05
CHAPTER 7
11:48 AM
Page 206
SATISFACTION WITH CARE
open-ended questions. It is one of the most commonly used measures of patient satisfaction (Ware & Karmos, 1976; Ware, Snyder, & Wright, 1976; Ware et al., 1983; Ware, Wright, & Snyder, 1976; Ware & Young, 1976). The current version is a 50-item instrument that uses a 5-point Likert response scale (strongly agree to strongly disagree) and that was developed for use in the Medical Outcomes Study (Marshall, Hays, Sherbourne, & Wells, 1993). A short form, the PSQ-18, also exists and takes 3 to 4 minutes to complete; its subscales substantially correlate with those of the PSQ-III (RAND Health, 1994).
Long-Term Care Patient satisfaction is a major goal of long-term nursing care and the essential domains focus on elements required for well-being and quality of life (Table 7–4). Perhaps since long-term care has not been traditionally under medical auspices, nursing homes do not fit as well under a medical model; instead, psychosocial and environmental aspects of care predominate. A literature review of 16 nursing homes instruments found through content analysis that the essential domains for nursing home satisfaction surveys are activities, care and services, caregivers, environment, meals, well-being (Robinson, Lucas, Castle, Lowe, & Crystal, 2004). They noted that only 25 percent of instruments had solicited the opinions of nursing home residents in their development. The authors also interviewed a sample of nursing home residents in three New Jersey nursing homes. They identified essential content areas and found that 25 percent of these areas were covered by the New Jersey residents but not by the survey instruments. Items not in published surveys but identified as important by these nursing home residents were availability of outdoor activities, consistency of staff, good personal/grooming care, being treated as an adult, close proximity to family and friends, and opportunities to discuss concerns and problems (Robinson et al., 2004). Nursing home and long-term care surveys have important differences from surveys of other types of care (e.g., hospital, ambulatory). The instruments tend to rely more on interviews that can be subject to more psychosocial artifacts. It is helpful to have an instrument that is suitable for those with cognitive impairments. Often, family members or caretakers may assist in filling out or answering the questionnaires. Family members and caretakers are also used at times as proxies, although caution needs to be taken that surveys do not mix the results of the two because one study found that regular visitors tended to
Geron, 1987
McCusker, 1984
Home Care and Terminal Care Satisfaction Scale (HCTC)
58 items (home care); 34 items (terminal care)
61 items
87–113 items
11 items
General Satisfaction; Availability of Care; Continuity of Care; Physician Availability; Physician Competence; Personal Qualities of MD; Communication with MD; Involvement of Pt/Fam in treatment decisions; Freedom from Pain; Pain Control
Homemaker Home Health Aide Case Management Meal Grocery
Domains vary depending on population studied
Physician Services; Nursing Services; Environment; Global Satisfaction
Dimension
Patient and caretaker satisfaction In-person interview with patient (home care)/surviving relative (TC) Self-administered; Caretaker
Convergent for several subscales Discriminant for several subscales
Term. Care subscales: = .59–.90
In-person interview
In-person or telephone
In-person interview
In development
Notes
Face validity claimed
Construct by factor analysis and Pearson correlations
None
Not reported
Validity evaluated
Home care subscales: = .10–.75 Caretaker subscales: = .50–.85
Subscales: = .26–.83 = .54–.88 = .58–.88 = .49–.79 = .46–.87
None
= .69–.74 for subscales
Reliability
11:48 AM
Home Care Satisfaction Measure (HCSM)
Case, 1996
Zinn et al., 1993
Nursing Home Resident Satisfaction Scale (NHRSS)
Items/Time
9/9/05
American Health Care Association Satisfaction Assessment Questionnaires (SAQs)
AHRQ
Source(s)
NH-CAHPS
Name of instrument
Table 7–4 Measures of Long-Term Care Satisfaction
34411_CH07_185_216 Page 207
Existing Satisfaction Measures 207
34411_CH07_185_216
208
9/9/05
CHAPTER 7
11:48 AM
Page 208
SATISFACTION WITH CARE
rate nursing home care more highly than residents (Gasquet, Dehe, Gaudebout, & Falissard, 2003). Finally, the concerns and satisfaction of family members/caretakers themselves may be important in evaluating long-term and terminal care (Atherly, Kane, & Smith, 2004). CAHPS Nursing Home Survey The Centers for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality (AHRQ) are working together to develop an instrument that can provide information on experiences of both short-term and long-term nursing home residents. They are currently conducting a national field test of a draft instrument and alternative sampling strategies. The CAHPS team (NH-CAHPS) is also exploring the possibility of surveying family members of nursing home residents about their experiences with nursing homes (CAHPS, 2004a). Nursing Home Resident Satisfaction Scale The Nursing Home Resident Satisfaction Scale (NHRSS) was designed to be used as an indicator of nursing home quality and to provide feedback to nursing home administrators (Zinn, Lavizzo-Mourey, & Taylor, 1993). The items were selected based on a literature review and then tested and refined using a pilot study. The advantages of the NHRSS are that the interview is short, easy to administer, and the results are not affected by mild levels of cognitive impairment. However, the instrument validity is not known. In addition, the small pilot sample limits the amount of information on the generalizability of the instrument to other populations and sensitivity of the instrument to discriminate between facilities. American Health Care Association Satisfaction Assessment Questionnaires The American Health Care Association (AHCA) developed a set of satisfaction assessment questionnaires (SAQs) to measure the satisfaction with care of nursing residents and their families. There are separate SAQs for cognitively intact residents, rehabilitation residents, medically complex residents, family members of the cognitively intact and cognitively impaired residents, and residents of assisted living facilities. Guided by the six core principles identified as appropriate for the long-term industry
34411_CH07_185_216
9/9/05
11:48 AM
Page 209
Existing Satisfaction Measures
209
(Case, 1996), the AHCA used a development process that included inperson and phone interviews conducted by the University of Wisconsin and nationwide polling of customers by the Gallup Association (AHCA, 1999). The questionnaires cover many domains. For example, the SAQ for cognitively intact residents examines overall satisfaction, family and community involvement, independence and respect, programs, facility setting, meals and dining, health care, doctor’s care, staff, safety and security, roommates and other residents, and moving in/out. The surveys use a five-point “excellent to poor” rating scale. The SAQs are widely used; however, a major disadvantage is that no psychometric testing of the instruments has been undertaken and thus, in the absence of reliability and validity data, it is difficult to know how to interpret the results. Home Care Satisfaction Measure The Home Care Satisfaction Measure (HCSM) is used to assess the satisfaction of older adults with five common home care services: homemaker, home health aide, home-delivered meals, grocery, and case management services (Geron, 1997). The survey was developed using focus groups that included ethnic minorities, preexisting satisfaction measures from other types of health care services, and it underwent further testing and refinement. The incorporation of the perspectives of the service recipients, including those of ethnic minorities, is an advantage of the HCSM. By looking at satisfaction with specific services, this instrument can be used as a guide for quality improvement projects, which is another advantage. However, because the survey excluded older adults whose cognitive impairments precluded a structured interview and because all the sampled participants received case management services, its generalizability may be limited particularly in the subset of the older adult population with greater memory and cognitive impairments. In addition some of the reliabilities of the subscales were low, requiring cautious interpretation of its results. Home Care and Terminal Care Satisfaction Scale The Home Care and Terminal Care (HCTC) satisfaction scale was developed to measure the satisfaction of chronically and terminally ill patients and their families with home care as well as their preferences for location of care (McCusker, 1984). Three instruments were developed: one to measure the
34411_CH07_185_216
210
9/9/05
CHAPTER 7
11:48 AM
Page 210
SATISFACTION WITH CARE
satisfaction of the patients themselves, one to measure the evaluations of the patients’ caretakers, and a third postbereavement version designed to be administered to caretakers following the patient’s death. The instruments adapted items from previous acute care satisfaction measures and added items suggested by project investigators and staff. The measures were tested and revised. However, some subscales did not perform well and their use was not recommended. In addition, the timing of the administration of interviews may matter particularly with the postbereavement version.
LITERATURE REVIEWS In addition to the reviews already discussed that examined patient satisfaction in a particular health care setting, a major literature review of the measurement of satisfaction in health care was performed by Crow and associates (2002) as part of the National Health Service’s (NHS) Technology Assessment Program. The goal of the review was to examine and summarize results of methodological studies, studies on the determinants of satisfaction in health care in different settings as well as to determine where the gaps are in existing knowledge in order to point to future research. For methodological issues, they look at studies examining modes of surveying and response rates as well as survey design issues. For the determinants of satisfaction, they review studies looking at patient-related determinants such as expectations, health status, socioeconomic and demographic characteristics, and health service–related determinants. They examine health service–related determinants by setting and including general/primary care, in-hospital satisfaction, and hospital outpatient care as well as studies examining the patient-practitioner relationship (Crow et al., 2002).
SUMMARY/FUTURE DIRECTIONS The development of the CAHPS generated interest in and substantial effort toward rigorous assessment of patients’ experiences with health care and their overall satisfaction. This work is important for looking at health care performance and quality, but a number of areas in patient satisfaction research still need exploration. Given that patients today are inundated with health information from the media, Internet, drug company advertising,
34411_CH07_185_216
9/9/05
11:48 AM
Page 211
References
211
and other sources, more work needs to be done on how patients form expectations and how these are changed or formed through the health care encounter. Patients’ preferences of care and their rank ordering of the attributes of health care that affect their satisfaction still remain to be explored. Current patient satisfaction research focuses almost exclusively on the processes of care to the exclusion of outcomes. Research into outcomes is particularly important because providers often only assess physical outcomes, but it may be that functional, psychological, and social outcomes play a larger role in patient satisfaction. One criticism of using outcomes as a measure of quality is that one can have a high-quality process and a bad outcome. It seems that assessment of patient satisfaction can bridge that gap to some extent, because patients can still be satisfied with overall health care even in the face of poor outcomes, if they perceive that the process was good and the outcome was beyond the provider’s control. Given the increasing use of patient satisfaction as a proxy for quality, it is particularly important to understand the relationship between patient perceptions of quality processes and quality outcomes.
REFERENCES Abramowitz, S., Cote, A.A., & Berry, E. (1987). Analyzing patient satisfaction: A multianalytic approach. Qrb. Quality Review Bulletin, 13(4), 122–130. Agency for Health care Policy and Research (AHCPR). (1996). Technical overview of Consumer Assessment of Health Plans (CAHPS) (No. AHCPR Pub. No. 97-R013). Rockville, MD: Public Health Service, Agency for Health Care Policy and Research. Agency for Healthcare Research and Quality. (2003a, February). Update on Hospital CAHPS (HCAHPS). Retrieved January 24, 2005, from www.ahrq.gov/qual/cahps/ hcahpsupdate.htm. Agency for Healthcare Research and Quality. (2003b, March). CAHPS® and the National CAHPS® Benchmarking Database: Fact sheet. Retrieved January 24, 2005, from www. ahrq.gov/qual/cahpfact.htm. Aharony, L., & Strasser, S. (1993). Patient satisfaction: What we know about and what we still need to explore. Medical Care Review, 50(1), 49–79. American Health Care Association (AHCA). (1999). SAQ of the cognitively intact resident. Atherly, A., Kane, R.L., & Smith, M.A. (2004). Older adults’ satisfaction with integrated capitated health and long-term care. Gerontologist, 44(3), 348–357. Barkley, W.M., & Furse, D.H. (1996). Changing priorities for improvement: The impact of low response rates in patient satisfaction. Joint Commission Journal on Quality Improvement, 22(6), 427–433.
34411_CH07_185_216
212
9/9/05
CHAPTER 7
11:48 AM
Page 212
SATISFACTION WITH CARE
Bender, R.H., Lance, T.X., & Guess, L.L. (2003). Including disenrollees in CAHPS managed care health plan assessment reporting. Health Care Financial Review, 25(1), 67–79. Ben-Sira, Z. (1976). The function of the professional’s affective behavior in client satisfaction: A revised approach to social interaction theory. Journal of Health and Social Behavior, 17(1), 3–11. CAHPS. (2004a, March). The CAHPS Nursing Home Survey (NH-CAHPS). Retrieved January 10, 2005, from www.cahps-sun.org/Products/NHCAHPS/NHCAHPSIntro.asp. CAHPS. (2004b). An introduction to Ambulatory CAHPS. Washington, DC: U.S. Department of Health and Human Services, Agency for Health Care Policy and Research. Case, T. (1996). A quality assessment and improvement system for long-term care. Quality Management in Health Care, 4(3), 15–21. Castle, N.G., & Engberg, J. (2004). Response formats and satisfaction surveys for elders. Gerontologist, 44(3), 358–367. Cleary, P.D., & McNeil, B.J. (1988). Patient satisfaction as an indicator of quality care. Inquiry, 25(1), 25–36. Crow, R., Gage, H., Hampson, S., Hart, J., Kimber, A., Storey, L., & Thomas, H. (2002). The measurement of satisfaction with healthcare: Implications for practice from a systematic review of the literature. Health Technology Assessment, 6(32), 1–244. Davies, A.R., Jr., & Ware, J.E., Jr. (1988). Involving consumers in quality of care assessment. Health Affairs, 7(1), 33–48. Davies, A.R., Jr., & Ware, J.E., Jr. (1991). GHAA’s consumer satisfaction survey and user’s manual (2nd ed.). Washington, DC: Group Health Association of America. DiMatteo, M.R., & Hays, R. (1980). The significance of patients’ perceptions of physician conduct: A study of patient satisfaction in a family practice center. Journal of Community Health, 6(1), 18–34. Donabedian, A. (1966). Evaluating the quality of medical care. Millbank Memorial Fund Quarterly, 40(3), 166–206. Donabedian, A. (1988). The quality of care. How can it be assessed? Journal of the American Medical Association, 260(12), 1743–1748. Edwards, C., Staniszweska, S., & Crichton, N. (2004). Investigation of the ways in which patients’ reports of their satisfaction with healthcare are constructed. Sociology of Health and Illness, 26(2), 159–183. Eisen, S.V., & Grob, M.C. (1979). Assessing consumer satisfaction from letters to the hospital. Hospital & Community Psychiatry, 30(5), 344–347. Fowler, F.J., Jr., Gallagher, P.M., Stringfellow, V.L., Zaslavsky, A.M., Thompson, J.W., & Cleary, P.D. (2002). Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Medical Care, 40(3), 190–200. Gasquet, I., Dehe, S., Gaudebout, P., & Falissard, B. (2003). Regular visitors are not good substitutes for assessment of elderly patient satisfaction with nursing home care and services. Journals of Gerontology. Series A, Biological Sciences and Medical Sciences, 58(11), 1036–1041. Geron, S.M. (1997). The Home Care Satisfaction Measures (HCSM): Study design and initial results of item analyses. Boston: Boston University School of Social Work.
34411_CH07_185_216
9/9/05
11:48 AM
Page 213
References
213
Greenfield, S., Kaplan, S., & Ware, J.E., Jr. (1985). Expanding patient involvement in care. Effects on patient outcomes. Annals of Internal Medicine, 102(4), 520–528. Groves, R.M. (1989). Survey errors and survey costs. New York: Wiley. Groves, R.M., Biemer, P.P., Lyberg, L.E., Massey, J.T., Nicholls, W.L., & Waksberg, J. (Eds.). (1988). Telephone survey methodology. New York: Wiley. Haas, J.S., Cook, E.F., Puopolo, A.L., Burstin, H.R., Cleary, P.D., & Brennan, T.A. (2000). Is the professional satisfaction of general internists associated with patient satisfaction? Journal of General Internal Medicine, 15(2), 122–128. Hall, J.A., & Dornan, M.C. (1988). Meta-analysis of satisfaction with medical care: Description of research domain and analysis of overall satisfaction levels. Social Science and Medicine, 27(6), 637–644. Hargraves, J.L., Hays, R.D., & Cleary, P.D. (2003). Psychometric properties of the Consumer Assessment of Health Plans Study (CAHPS) 2.0 adult core survey. Health Services Research, 38(6 Pt 1), 1509–1527. Harris-Kojetin, L.D., Fowler, F.J., Jr., Brown, J.A., Schnaier, J.A., & Sweeny, S.F. (1999). The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Consumer Assessment of Health Plans Study. Medical Care, 37(3 Suppl), MS10–MS21. Hays, R.D., Larson, C., Nelson, E.C., & Batalden, P.B. (1991). Hospital quality trends. A short-form patient-based measure. Medical Care, 29(7), 661–668. Hays, R.D., & Ware, J.E., Jr. (1986). My medical care is better than yours. Social desirability and patient satisfaction ratings. Medical Care, 24(6), 519–524. Hendriks, A.A., Vrielink, M.R., Smets, E.M., van Es, S.Q., & De Haes, J.C. (2001). Improving the assessment of (in)patients’ satisfaction with hospital care. Medical Care, 39(3), 270–283. Hickson, G.B., Clayton, E.W., Entman, S.S., Miller, C.S., Githens, P.B., Whetten-Goldstein, K., & Sloan, F.A. (1994). Obstetricians’ prior malpractice experience and patients’ satisfaction with care. Journal of the American Medical Association, 272(20), 1583–1587. Hudak, P.L., McKeever, P., & Wright, J.G. (2003). The metaphor of patients as customers: Implications for measuring satisfaction. Journal of Clinical Epidemiology, 56(2), 103–108. Hudak, P.L., McKeever, P.D., & Wright, J.G. (2004). Understanding the meaning of satisfaction with treatment outcome. Medical Care, 42(8), 718–725. Jackson, J.L., Chamberlin, J., & Kroenke, K. (2001). Predictors of patient satisfaction. Social Science & Medicine, 52(4), 609–620. Joffe, S., Manocchia, M., Weeks, J.C., & Cleary, P.D. (2003). What do patients value in their hospital care? An empirical perspective on autonomy centred bioethics. Journal of Medical Ethics, 29(2), 103–108. Kane, R.L., Maciejewski, M., & Finch, M. (1997). The relationship of patient satisfaction with care and clinical outcomes. Medical Care, 35(7), 714–730. Keating, N.L., Green, D.C., Kao, A.C., Gazmararian, J.A., Wu, V.Y., & Cleary, P.D. (2002). How are patients’ specific ambulatory care experiences related to trust, satisfaction, and considering changing physicians? Journal of General Internal Medicine, 17(1), 29–39.
34411_CH07_185_216
214
9/9/05
CHAPTER 7
11:48 AM
Page 214
SATISFACTION WITH CARE
Kippen, L.S., Strasser, S., & Joshi, M. (1997). Improving the quality of the NCQA (National Committee for Quality Assurance) Annual Member Health Care Survey Version 1.0. American Journal of Managed Care, 3(5), 719–730. Kravitz, R.L., Callahan, E.J., Paterniti, D., Antonius, D., Dunham, M., & Lewis, C.E. (1996). Prevalence and sources of patients’ unmet expectations for care. Annals of Internal Medicine, 125(9), 730–737. Krupat, E., Fancey, M., & Cleary, P.D. (2000). Information and its impact on satisfaction among surgical patients. Social Science & Medicine, 51(12), 1817–1825. Landon, B.E., Aseltine, R., Jr., Shaul, J.A., Miller, Y., Auerbach, B.A., & Cleary, P.D. (2002). Evolving dissatisfaction among primary care physicians. American Journal of Managed Care, 8(10), 890–901. Lasek, R.J., Barkley, W., Harper, D.L., & Rosenthal, G.E. (1997). An evaluation of the impact of nonresponse bias on patient satisfaction surveys. Medical Care, 35(6), 646–652. Ley, P., Bradshaw, P.W., Kincey, J.A., & Atherton, S.T. (1976). Increasing patients’ satisfaction with communications. British Journal of Social & Clinical Psychology, 15(4), 403–413. Locander, W., Sudman, S., & Bradburn, N. (1976). An investigation of interview method, threat and response distortion. Journal of the American Statistical Association, 71(354), 269–275. Marple, R.L., Kroenke, K., Lucey, C.R., Wilder, J., & Lucas, C.A. (1997). Concerns and expectations in patients presenting with physical complaints. Frequency, physician perceptions and actions, and 2-week outcome. Archives of Internal Medicine, 157(13), 1482–1488. Marshall, G.N., Hays, R.D., Sherbourne, C.D., & Wells, K.B. (1993). The structure of patient satisfaction with outpatient medical care. Psychological Assessment, 5(4), 477–483. Mazor, K.M., Clauser, B.E., Field, T., Yood, R.A., & Gurwitz, J.H. (2002). A demonstration of the impact of response bias on the results of patient satisfaction surveys. Health Services Research, 37(5), 1403–1417. McCusker, J. (1984). Development of scales to measure satisfaction and preferences regarding long-term and terminal care. Medical Care, 22(5), 476–493. McGee, J., Goldfield, N., Riley, K., & Morton, J. (1997). Collecting information from health care consumers: A resource manual of tested questionnaires and practical advice. Gaithersburg, MD: Aspen Publishers, Inc. Mead, N., & Bower, P. (2000). Patient-centredness: A conceptual framework and review of the empirical literature. Social Science & Medicine, 51(7), 1087–1110. Meterko, M., Nelson, E.C., & Rubin, H.R. (1990). Patient judgments of hospital quality: Report of a pilot study. Medical Care, 28(9 Suppl), S1–S56. National Committee for Quality Assurance (NCQA). (1995). Annual Member Health Care Survey Manual, Version 1.0. Washington, DC: Author. Nelson, E.C., Hays, R.D., Larson, C., & Batalden, P.B. (1989). The patient judgment system: Reliability and validity. Qrb. Quality Review Bulletin, 15(6), 185–191.
34411_CH07_185_216
9/9/05
11:48 AM
Page 215
References
215
Oliver, R.L. (1993). Cognitive, affective, and attribute bases of the satisfaction response. Journal of Consumer Research, 20(3), 418–430. Pascoe, G.C. (1983). Patient satisfaction in primary health care: A literature review and analysis. Evaluation & Program Planning, 6(3–4), 185–210. Pascoe, G.C., Attkisson, C.C., & Roberts, R.E. (1983). Comparison of indirect and direct approaches to measuring patient satisfaction. Evaluation & Program Planning, 6(3–4), 359–371. RAND Health. (1994). Patient Satisfaction Questionnaire (PSQ-18)-short form. Retrieved January 10, 2005, from www.rand.org/health/surveys/PSQ18.html. Robinson, J.P., Lucas, J.A., Castle, N.G., Lowe, T.J., & Crystal, S. (2004). Consumer satisfaction in nursing homes: Current practices and residence priorities. Research on Aging, 26(4), 454–481. Rubin, H.R. (1990). Can patients evaluate the quality of hospital care? Medical Care Review, 47(3), 267–326. Scholle, S.H., Weisman, C.S., Anderson, R.T., & Camacho, F. (2004). The development and validation of the primary care satisfaction survey for women. Women’s Health Issues, 14(2), 35–50. Singer, E., Von Thurn, D.R., & Miller, E.R. (1995). Confidentiality assurances and response: A qualitative review of the experimental literature. Public Opinion Quarterly, 59(1), 66–77. Sitzia, J., & Wood, N. (1997). Patient satisfaction: A review of issues and concepts. Social Sciences & Medicine, 45(12), 1829–1843. Sitzia, J., & Wood, N. (1998). Response rate in patient satisfaction research: An analysis of 210 published studies. International Journal for Quality in Health Care, 10(4), 311–317. Strasser, S., Aharony, L., & Greenberger, D. (1993). The patient satisfaction process: Moving toward a comprehensive model. Medical Care Review, 50(2), 219–248. Thompson, A.G., & Sunol, R. (1995). Expectations as determinants of patient satisfaction: Concepts, theory and evidence. International Journal for Quality in Health Care, 7(2), 127–141. Ware, J.E., Jr., & Hays, R.D. (1988). Methods for measuring patient satisfaction with specific medical encounters. Medical Care, 26(4), 393–402. Ware, J.E., Jr., & Karmos, A.H. (1976). Development and validation of scales to measure patient satisfaction with health care services: Volume II. Perceived health and patient role propensity. Final report. June 30, 1972– March 31, 1976 (Vol. 274). Carbondale: Southern Illinois University. Ware, J.E., Jr., Snyder, M.K., & Wright, W.R. (1976). Development and validation of scales to measure patient satisfaction with health care services: Volume I. Part B. Results of scales constructed from the Patient Satisfaction Questionnaire and other health care perceptions. Final report. June 30, 1972–March 31, 1976 (Vol. 447). Carbondale: Southern Illinois University. Ware, J.E., Jr., Snyder, M.K., Wright, W.R., & Davies, A.R. (1983). Defining and measuring patient satisfaction with medical care. Evaluation & Program Planning, 6(3–4), 247–263.
34411_CH07_185_216
216
9/9/05
CHAPTER 7
11:48 AM
Page 216
SATISFACTION WITH CARE
Ware, J.E., Jr., Wright, W.R., & Snyder, M.K. (1976). Development and validation of scales to measure patient satisfaction with health care services: Volume IV. Key health concepts: Methodological appendix. Final report. June 30, 1972–March 31, 1976 (Vol. 245). Carbondale: Southern Illinois University. Ware, J.E., Jr., & Young, J. (1976). Development and validation of scales to measure patient satisfaction with health care services: Volume III. Conceptualization and measurement of health as a value. Final report. June 30, 1972–March 31, 1976, (Vol. 203). Carbondale: Southern Illinois University. Weithman, P.J. (1995). Contractualist liberalism and deliberative democracy. Philosophy and Public Affairs, 24(4), 314–343. Wensing, M., Grol, R., & Smits, A. (1994). Quality judgments by patients on general practice care: A literature analysis. Social Sciences & Medicine, 38(1), 45–53. Williams, B. (1994). Patient satisfaction: A valid concept? Social Sciences & Medicine, 38(4), 509–516. Williams, B., Coyle, J., & Healy, D. (1998). The meaning of patient satisfaction: An explanation of high reported levels. Social Sciences & Medicine, 47(9), 1351–1359. Williams, S., Pauly, M.V., Rosenbaum, P.R., Ross, R., Schwartz, J.S., Shpilsky, A., & Silber, J. H. (1995). Ranking hospitals by the quality of care for medical conditions: The role of complications. Transactions of the American Clinical & Climatological Association, 107, 263–273. World Health Organization (WHO). (1948). Preamble to the Constitution of the World Health Organization. Paper presented at the International Health Conference, New York. Zaslavsky, A.M., Zaborski, L.B., & Cleary, P.D. (2002). Factors affecting response rates to the Consumer Assessment of Health Plans Study survey. Medical Care, 40(6), 485–499. Zinn, J.S., Lavizzo-Mourey, R., & Taylor, L. (1993). Measuring satisfaction with care in the nursing home setting: The nursing home resident satisfaction scale. Journal of Applied Gerontology, 12(4), 452–465.
34411_CH08_217_264
9/9/05
12:02 PM
Page 217
Part III Specific Measures: Risk Adjusters
34411_CH08_217_264
9/9/05
12:02 PM
Page 218
34411_CH08_217_264
9/9/05
12:02 PM
Page 219
8 Severity and Comorbidity Maureen A. Smith Nicole M. Nitz Sara K. Stuart
INTRODUCTION The importance of adjusting for severity and comorbidity in health outcomes research can be illustrated by two cancer patients. Patient 1 has been diagnosed with stage I cancer and has no concurrent health problems, whereas Patient 2 has been diagnosed with stage IV cancer (the same site as Patient 1) and has type I diabetes and congestive heart failure. It is unlikely that these two patients will receive the same treatment or have equivalent outcomes, even though these two patients have an identical diagnosis. Suppose that a new treatment is starting to be used for this type of cancer, and an outcomes researcher undertakes a study to determine the effectiveness of this treatment in saving lives. Some of the patients in the study are relatively healthy like Patient 1 and others are severely ill, similar to Patient 2. If healthy patients are more likely to receive the treatment, a simple comparison of outcomes between patients who did and did not get the treatment is likely to show that it saves many lives. Why? The healthier patients are much less likely to die than severely ill patients for reasons completely unrelated to receiving the treatment. The treatment appears to save lives because patients who receive treatment are more likely to be healthy at the beginning of the study. Without adjusting for severity (i.e., stage) of disease or comorbidities, conclusions drawn from this study will be wrong. As shown in this example, researchers often seek to compare outcomes achieved by two or more groups of patients. Occasionally, patients are randomly allocated to each of the groups (although this does not guarantee that the groups are comparable). More often, groups of interest are formed natu219
34411_CH08_217_264
220
9/9/05
CHAPTER 8
12:02 PM
Page 220
SEVERITY AND COMORBIDITY
rally through an unknown selection process. The problems generated by this selection process were brought to national attention when the Health Care Financing Administration announced that it would publish hospital mortality rates (Green, Wintfeld, Sharkey, & Passman, 1990). These published mortality rates generated extensive controversy, particularly because the data did not control for the initial severity of illness of each hospital’s patients (Iezzoni, Ash, Coffman, & Moskowitz, 1992) or comorbidity (Greenfield, Aronow, Elashoff, & Watanabe, 1988). If the rates were not adjusted for the initial severity of illness or patient comorbidity, higher death rates in some hospitals might not necessarily represent poor quality of care. It would be unfair to identify and publicly penalize those hospitals whose results seemed poor because they were admitting the sickest patients. This chapter focuses on defining severity and comorbidity, explaining the importance of measuring both, providing an overview of how to choose a method, and giving brief summaries of some specific measures used in health care outcomes research.
RELATIONSHIP BETWEEN SEVERITY OF ILLNESS AND COMORBIDITY Although often considered separately from severity measures in the literature, comorbidity is a component of the patient’s overall severity of illness. Defining the concepts of severity of disease, comorbidity, and severity of illness by referring to the literature is confusing, because many measures use the terms interchangeably. These definitions can be clarified by outlining the relationship of these constructs to each other.
Severity of Disease Severity of disease usually refers to the severity and importance of a specific diagnosis (often the principal diagnosis), irrespective of a patient’s other health conditions. As in the previous patient example, a person with a principal diagnosis of cancer may be staged from I to IV, with I being less severe (representing local disease) and IV being the most severe (representing widely metastatic spread).
Severity of disease = f Importance of the principal diagnosis Severity of the principal diagnosis
34411_CH08_217_264
9/9/05
12:02 PM
Page 221
Introduction: Relationship Between Severity of Illness and Comorbidity
221
Comorbidity The term “comorbid conditions” typically refers to one or more additional diagnoses for a patient, not the principal diagnosis that brought the patient into the health care system. However, comorbidity is essentially an artificial concept. In some instances, coding for payment or certification requires that clinicians deem one problem as the principal cause, but in truth, patients do not necessarily have a principal diagnosis. They may have multiple diagnoses. Deeming one as principal and the others as secondary may be an artifact imposed by the clinician (or the investigator). Measures of comorbidity implicitly or explicitly weight the severity of each individual diagnosis and the importance of each diagnosis to the overall level of comorbidity for the patient. For example, the Charlson Comorbidity Index sums scores for approximately 20 diseases, ranging from stroke to liver disease (Charlson, Pompei, Ales, & MacKenzie, 1987). Diseases not included in calculating the score are implicitly weighted zero. Each disease receives a score from 1 to 6, representing the importance of the disease to the overall level of comorbidity and relative risk of death.
Comorbidity = f Importance of each secondary diagnosis Severity of each secondary diagnosis
Severity of Illness Information about the severity of all of a patient’s diagnoses can be combined to get a score for a particular patient’s overall level of illness. Consequently, overall severity of illness is a function of both the severity of disease and comorbidity. Similarly to comorbidity indices, these measures weight both the severity of each diagnosis and the importance of each diagnosis to the overall level of illness for the patient. Severity-of-illness measures assess the contribution to risk from the overall illness level, often as a sum of specific diseases. When the outcomes study focuses on a particular disease of primary interest, it may be more useful to measure comorbidity rather than overall severity of illness, although both are important. However, severity of illness can also be defined generically without reference to a specific diagnosis. This approach identifies variables representing basic disorders of organ functioning and scores these variables identically across all patients. The types of severity measures used in an
34411_CH08_217_264
222
9/9/05
CHAPTER 8
12:02 PM
Page 222
SEVERITY AND COMORBIDITY
individual study will depend on the underlying conceptual model and the research questions being asked. Overall severity of illness = f
Importance of the principal diagnosis Severity of the principal diagnosis Importance of each secondary diagnosis Severity of each secondary diagnosis
SEVERITY OF ILLNESS AND THE DOMAINS OF HEALTH During data analysis, outcomes measures are adjusted statistically for the potentially confounding effects of each group’s initial severity of illness, which, as just stated, incorporates not only the severity of the primary disease, but also the severity and number of comorbid conditions. The final outcome measures are often based on the concept of “health,” which may include several domains (e.g., physical, social, emotional, and cognitive functioning). Traditional measures of risk adjustment are usually based on the medical concept of physiological “illness.” Most current conceptualizations of severity of illness include only one or two domains of health (Aronow, 1988; Pompei, Charlson, Ales, MacKenzie, & Norton, 1991). This approach may have substantial limitations, particularly if omitted domains such as cognitive, emotional, or social functioning influence the final outcome measure. Many traditional severity measures focus on physiological functioning alone. For example, Pompei and colleagues (1991) suggest that physicians use several clinical concepts to relate a patient’s initial level of illness at admission to the outcomes of hospitalization. These include the patient’s overall condition, functional status, physiological severity of illness, burden of comorbid disease, and instability (i.e. ability to withstand an acute illness). The majority of these clinically defined concepts (e.g., physiological severity, comorbidity, and instability) are subcategories of the larger domain of physical functioning (see Figure 8–1). The concept of functional status tends to be more inclusive; although, it often focuses on physical functioning, other domains such as cognition may be implicitly measured as well. By subcategorization, this conceptualization conveys more precise definitions for measures of the initial severity of illness. Nonetheless, the limitations of this approach are most apparent in its omissions. Major domains
34411_CH08_217_264
9/9/05
12:02 PM
Page 223
Severity of Illness and the Domains of Health
Physical Functioning
Instability Functional Disability
223
Emotional Functioning
Cognitive Functioning
Social Functioning
Physiologic Severity Comorbidity Symptoms
Vitality
Figure 8–1 Domains of Health
of health are excluded (e.g., emotional, cognitive, and social functioning). Many of these excluded domains have not traditionally been considered part of primary medical practice, and patients with these problems have been referred to other types of health professionals. The conceptualization of initial severity of illness as domains related only to physical health may bias the conclusions of an outcomes research study. For example, clinical depression has been shown to worsen the outcomes of a number of conditions, such as stroke. For these conditions, ignoring the domain of emotional functioning may seriously hamper the ability to test the relationship between the proposed intervention and the eventual outcome. To address these issues, Iezzoni has proposed a broader conceptualization of risk that combines the domains of health with the concepts of illness severity (Iezzoni, 1994). Several additional dimensions of risk are identified, including the patient’s age, sex, attitudes, and preferences for outcomes and cultural, ethnic, and socioeconomic attributes or behaviors. Which domains are most important for risk adjustment? The answer varies depending on the outcome and disease of interest. Iezzoni (1994) suggests the “medical meaningfulness” test for assessing the validity of a risk-adjustment approach. This test asks: Is the source of the risk (i.e.,
34411_CH08_217_264
224
9/9/05
CHAPTER 8
12:02 PM
Page 224
SEVERITY AND COMORBIDITY
domain) linked to the outcome of interest? An evaluation of this potential linkage determines whether a particular risk-adjustment strategy achieves validity and clinical credibility.
COMPONENTS OF SEVERITY OF ILLNESS It is useful to clarify the components of illness severity that are most important for outcomes research. The following conceptualization utilizes the domains of health and the clinical concepts of severity (e.g., physiologic severity and comorbidity), along with information about the patient’s principal diagnosis. General physiologic severity can be measured by variables that reflect basic physiologic functioning, such as heart rate, blood pressure, white blood cell count, and level of consciousness. These variables are scored identically for all patients, irrespective of the principal diagnosis. Abnormal values of these variables reflect disorders of homeostasis and basic organ function. Traditionally, most severity measures have focused on quantifying this component. These traditional measures are useful in classifying severity for acutely ill hospitalized patients; they may be less useful in classifying severity in other settings such as ambulatory care. The physiologic severity of the principal diagnosis can be crucial to the eventual outcome. Most patients can be classified as more or less severe, based on information specific to the principal diagnosis. For example, two patients with a diagnosis of breast cancer may vary dramatically in severity depending on whether the cancer is local or widely metastatic. For breast cancer patients, the most important classification of severity may involve the degree of metastasis. In contrast, patients with stroke may be considered more or less severe depending on the level of paralysis or loss of consciousness. In some cases, the components of severity will overlap. For example, loss of consciousness is a general measure of physiologic function, but it is also an extremely important determinant of severity for a patient with a principal diagnosis of stroke. For other cases, it may be difficult to identify a principal diagnosis, as in the case of older patients who have multiple coexisting conditions such as diabetes, ischemic heart disease, and kidney disease. The presence of multiple diagnoses for a single patient necessitates additional identification and classification of these comorbid conditions. The number and severity of comorbid conditions is a major component of the overall severity of illness. For example, in an otherwise healthy
34411_CH08_217_264
9/9/05
12:02 PM
Page 225
Components of Severity of Illness
225
patient with pneumonia, the prognosis may be excellent. However, a primary diagnosis of pneumonia in a patient with AIDS or metastatic cancer may indicate an immediately life-threatening condition with an extremely poor prognosis. In addition, it may not be sufficient to identify the presence of a comorbid condition; severity of the condition and its relationship to the primary diagnosis should be considered as well. A myocardial infarction in a patient with diabetes controlled completely by diet may have a relatively good prognosis; the opposite may be true in a diabetic patient with substantial end-organ damage, such as end-stage renal disease. In this case, the severity of the comorbid condition (diabetes) may dramatically influence a patient’s life expectancy. The relationship of the comorbid condition to the primary diagnosis may also provide valuable information. Severe pneumonia in a patient with a primary diagnosis of low back pain has different implications than a severe pneumonia in a patient with a primary diagnosis of AIDS. The outcome variable of interest may capture one or many domains of health. Incorporating a baseline measure of the outcome variable into a study improves a risk-adjustment strategy for several reasons. First, it assures that all domains considered important for that particular outcome are included in the measures of initial severity. Second, because the baseline measure is identical to the final outcome measure, measurement error for that variable will be consistent across the initial and final time periods. Unfortunately, although baseline measures are useful indicators of initial severity, it is not always possible or meaningful to collect them. For example, if the outcome of interest is death, the baseline measure is meaningless (because all patients are initially alive). However, if the outcome of interest is functional status (e.g., activities of daily living, or ADLs), a baseline measure can provide valuable information regarding the patient’s progress over time. Baseline measures of other relevant domains of health can be equally important. Suppose the researcher wishes to study the relationship between occupational therapy (the intervention) and functional status at discharge from the hospital (the outcome). The relationship between the intervention and outcome may be substantially affected by the patient’s initial cognitive status. For example, a patient with Alzheimer’s disease may have a completely different response to occupational therapy when compared to someone who is cognitively intact. A baseline measure of cognitive functioning enables the researcher to incorporate this concern into the analysis. It is important to note that, in practice, these five components of severity of illness are not mutually exclusive (Table 8–1). For example, some severity of illness measures focus on general physiological severity, but also
34411_CH08_217_264
226
9/9/05
CHAPTER 8
12:02 PM
Page 226
SEVERITY AND COMORBIDITY
Table 8–1 Components of Severity of Illness • General physiological severity, irrespective of the principal diagnosis • Physiological severity of the principal diagnosis • Number and severity of comorbid conditions • Baseline measures of the outcome domain • Baseline measures of other relevant domains
include information about the patient’s comorbid conditions. These five components are useful for purposes of discussion, but the real issue is whether the combination of severity measures chosen is appropriate and comprehensive enough for the task at hand. This chapter will focus on traditional measures of severity of illness, which usually incorporate elements of the first three components (severity of comorbid conditions, general physiological severity, and physiological severity of the principal diagnosis and the number).
Comorbidity Versus Complications Comorbid conditions are diseases other than the principal diagnosis that influence the outcome of treatment. As mentioned previously, comorbidity is a component of a person’s overall severity of illness and must be differentiated from complications. Investigators should be wary in adjusting for comorbidity, lest they inadvertently adjust for complications of the disease or treatment, and thus adjust away the outcome of interest. It is important to differentiate between the constructs of comorbidity and complications. “Comorbidities, or coexisting diagnoses, are diseases unrelated in etiology or causality to the principal diagnosis,” whereas complications are the “sequelae of the principal diagnosis” or its treatment (Iezzoni, 1994, p. 52). By expanding these definitions of comorbidity and complications, one can look at coexisting conditions in three ways: 1. Comorbidities 2. Complications of disease 3. Complications of treatment
34411_CH08_217_264
9/9/05
12:02 PM
Page 227
Components of Severity of Illness
227
This expanded definition explicitly considers both the causality of the secondary condition and the time sequence of events. We will use “occurrence” of a disease to mean the time period of diagnosis, because knowing the true moment of the inception of the illness is often impossible, especially for chronic diseases. Comorbidities are conditions that are unrelated in etiology or causality to the principal diagnosis (e.g., cancer diagnosed after stroke). In terms of chronology, comorbid conditions may either precede, be concurrent with, or occur after the onset of the principal disease. Having a particular comorbid disease may increase (or decrease) the likelihood of a positive or negative outcome for the primary illness. In terms of measuring comorbid disease, one should control for comorbid diseases that occurred before the onset of the primary disease and before the treatment. Because the concept of a comorbid disease is rather arbitrarily defined by clinical logic, the decision to include in the measure conditions that were identified after the onset of the primary disease will depend on the likelihood that the comorbid condition actually existed prior to primary disease or whether there is any chance that the comorbid condition could be a consequence of either the primary disease or its treatment. Complications of the disease are conditions that are directly causally/ etiologically related to the principal disease itself, including its treatment. Specifically, one can view complications of the disease as diseasespecific outcomes. For example, in the case of diabetes and myocardial infarction, having the principal disease increases the likelihood that the secondary condition will develop. This implies that the complication must be concurrent with or occur after the principal disease. Complications may also be related to comorbid diseases, especially when these diseases have a synergistic effect in increasing the vulnerability of the patient to infection. Disease complications may occur either before or after treatment is initiated, and when they occur will determine how the investigator should treat these complications. When an outcomes study is focused on hospitalizations, investigators reviewing hospital discharge records may identify complications of the disease that occur after admission. However, investigators should view complications of the disease that occur after the start of treatment as outcomes, rather than controlling for these factors. Doing otherwise can yield perverse results. Preadmission complications of the primary disease may be incorporated into the measure of the severity of disease at admission. Including disease
34411_CH08_217_264
228
9/9/05
CHAPTER 8
12:02 PM
Page 228
SEVERITY AND COMORBIDITY
complications that occur prior to the intervention in a measure of disease severity and level of comorbidity should not bias the estimates of the effectiveness of the treatment. To the extent that treatment should address these disease complications, however, one would not want to adjust for their presence. Complications of the treatment for the principal disease occur when a treatment for the principal disease causes the patient to suffer the secondary condition. One should be able to detect this type of complication by looking at differences in incidence rates for the condition across different types of treatment or no treatment. However, one does not want to control for treatment complications, because doing so will bias the estimate of treatment effect. A condition can be both a complication of disease and a complication of treatment, in which case the disease increases the likelihood of occurrence; a particular treatment increases the likelihood even further than if no treatment had been used. This type of complication must occur concurrent with or after the treatment, and therefore after the diagnosis of the principal disease. In practice, it may be impossible to differentiate between disease complications and treatment complications when they occur after the start of the treatment. Any complication that occurs after the beginning of the treatment the researcher should regard as an outcome of the treatment. One would not want to control for these posttreatment complications in the analysis, except with the complete understanding that doing so may bias the estimates of the effects of the treatment by controlling for the negative consequences of treatment. Although distinctions between “complications” and “comorbidities” can be made by means of definitions, the etiological relationship between diseases and particular episodes of illness may not always appear as clear-cut. Therefore, each study requires a clear conceptualization identifying which conditions will be considered complications (versus comorbidities) to avoid inadvertently controlling for complications of the treatment. Controlling for comorbidities enables the investigator to attribute justifiably a better outcome to the treatment rather than to a healthier group of treatment subjects. On the other hand, controlling for complications of the treatment will make it appear that the treatment had a far better effect on outcomes than it actually did. Including treatment complications in the comorbidity measure will increase the variance explained in the model, but by explaining away the outcome (Shapiro, Park, Keesey, & Brook, 1994).
34411_CH08_217_264
9/9/05
12:02 PM
Page 229
Reasons to Include Severity and Comorbidity Measures
229
REASONS TO INCLUDE SEVERITY AND COMORBIDITY MEASURES The primary rationale for assessing severity and comorbidity is to eliminate potential sources of observed variation in outcomes between groups to help to isolate the effects of treatment. Differences between groups in the levels of severity and comorbid disease may generate differences in the measured outcomes for these groups and confound the effect of the intervention. Comorbidity measures also help to establish the patient’s usual health status before treatment. Because medical treatments rarely make patients healthier than they were before the onset of the episode of illness, the definition of pretreatment state provides an estimate for the maximum possible improvement in health available from the treatment. Finally, an investigator may want to include measures of severity and comorbidity to enhance the credibility of the study. Every physician seems to think she or he treats the sickest patients. If others believe severity or comorbidity to be an important predictor of the outcome, even in the absence of compelling evidence or theory, the omission of a measure of both could call the results into question. There are several major reasons to incorporate severity and comorbidity measures explicitly into the conceptual and analytical models driving outcomes research. These include adjusting for selection bias, improving the ability of the model to predict outcomes, and forming a basis for subgroup analyses. The following sections examine the reasons to include these measures explicitly in outcomes models.
Identification of Selection Bias In most outcomes research, groups are not randomly allocated to the intervention of interest. This creates the potential for substantial selection bias. If sicker patients are less likely to receive the intervention, comparison of the intervention and nonintervention groups is likely to show a favorable outcome for patients who received the intervention. This favorable outcome is not due to the effect of the intervention, but is the result of a healthier set of patients in the intervention group. Two criteria must be met in order for selection bias to occur (see Figure 8–2). First, severity and/or comorbidity must be related to the outcome of interest. Second, the severity and/or comorbidity must influence which patients receive the intervention.
34411_CH08_217_264
230
9/9/05
CHAPTER 8
12:02 PM
Page 230
SEVERITY AND COMORBIDITY
Severity/Comorbidity
Intervention
Outcome
Figure 8–2 Identification of Selection Bias (Severity/Comorbidity influences the receipt of intervention and the outcome.)
One practical way to identify components of severity and types of comorbidities important in creating potential selection bias is to look at the indications for the intervention. For example, suppose the outcomes research question is whether coronary angioplasty is effective in reducing mortality in a group of patients with angina. The comparison group is composed of patients with angina who do not receive angioplasty. If either the severity of coronary occlusion or the presence of specific comorbidities is an indicator (or contraindication) for angioplasty and is related to eventual mortality, then it is likely that substantial selection bias will occur if the severity of coronary occlusion and adjustment for comorbidity are not included in the analysis. The direction of the resulting bias cannot always be predicted in advance. For example, if sicker patients are more likely to receive the intervention, the intervention group may do more poorly than the comparison group, but this is due to the initial poor health of patients receiving the intervention, not the intervention itself. Because the direction of the bias cannot always be predicted, it is important to include severity and comorbidity measures in conceptual models of outcomes research and to use these models in developing the statistical methods for final analysis. Procedures that are appropriate for a person with few or no other illnesses may be inappropriate for a patient with other concurrent diseases. Clinicians and patients select treatments that are appropriate for the “package” of the primary illness and other existing conditions. Selection bias occurs when the group of patients receiving treatment A are fundamentally different from the group of patients receiving treatment B because one or more of the factors influencing the outcome also deter-
34411_CH08_217_264
9/9/05
12:02 PM
Page 231
Reasons to Include Severity and Comorbidity Measures
231
mined which treatment the patient received (Iezzoni, Shwartz, Ash, Mackiernan, & Hotchkin, 1994). The decision to include a measure of severity and/or comorbidity and which one to use will depend on which treatment the analysis will focus on and the clinical logic of how treatments are selected.
Independent Predictors of Outcome Even when there is no evidence of biased selection, severity and comorbidity are often believed to be important factors predicting the outcome. In this case, severity and comorbidity are strongly related to outcome (see Figure 8–3), but are not related to whether or not a patient ends up in the intervention group. Although there is no risk of biased conclusions, including severity and comorbidity in the statistical models may substantially improve their fit and consequently improve the precision of the estimates for the impact of the intervention. Here, the benefit of measuring severity is indirect, but again argues for including these measures in conceptual and statistical models of the outcome. Including measures of severity and comorbidity may also reveal the independent effects of other factors; for example, increasing age is highly associated with a greater number and severity of illnesses. Greenfield and colleagues (1987) found that when they controlled for the effect of comorbidity, there was still a difference by age in patterns of care for breast cancer, suggesting a possible age bias in treatment given to breast cancer patients.
Severity/Comorbidity
Intervention Figure 8–3 Independent Predictors of Outcome
Outcome
34411_CH08_217_264
232
9/9/05
CHAPTER 8
12:02 PM
Page 232
SEVERITY AND COMORBIDITY
Subgroup Analysis The relationship between the intervention and outcome may differ depending on severity of the primary disease and the presence as well as severity of any comorbid conditions. Researchers talk about this phenomenon as a statistical interaction of severity or comorbidity with treatment. Subgroups of patients with more or less severe disease or with varying types, numbers, and severity of comorbidities may respond differently to an intervention. In this case, severity and comorbidity may or may not be directly related to the intervention or the outcome. However, the impact of the intervention is substantially different, depending on the initial levels of severity and the presence of comorbidities. This difference in effect can be illustrated by the clinical logic used in selecting a treatment. Clinicians may recommend that patients with certain comorbid diseases not use a particular treatment because the treatment may not work as well or may be detrimental to people with those comorbid conditions. However, standard clinical treatment procedures do not always account for all possible conditions and the effect may differ by the severity or variety of comorbid conditions that were not considered to be clinically relevant to the choice of treatment or outcome. The impact of severity and comorbidity on the relationship between the intervention and outcome is shown in Figure 8–4. When this situation occurs, subgroups should be analyzed separately or appropriate statistical interaction terms should be included in the model.
Severity/Comorbidity
Intervention Figure 8–4 Subgroup Analysis
Outcome
34411_CH08_217_264
9/9/05
12:02 PM
Page 233
How to Choose a Measure
233
HOW TO CHOOSE A MEASURE A variety of tools is available for evaluating measures of severity and comorbidity. The list of criteria shown in Table 8–2 should help the investigators choose the most appropriate measures.
Included Domains The investigator must make sure that all relevant domains are included. (See Chapter 1 for a discussion of conceptual models.) It is unlikely that all important domains will be incorporated into a single measure. Consequently, several measures are often used to cover all domains considered important for risk adjustment. For example, an investigator might consider initial levels of physical functioning, emotional functioning, and cognition relevant to the outcome of interest. Separate measures of each domain could be chosen or measures that combine one or more domains could be used. The relevant domains can be chosen by one of several methods, including: • Iezzoni’s (1994) test of medical meaningfulness: Is the domain linked to the outcome of interest? • Test of influence: Is the domain included in the conceptual model of the outcome?
Table 8–2 Criteria for Choosing a Measure • Included domains of health • Reliability • Validity 1) Prognostic endpoint 2) Population of interest 3) Setting 4) Timing of measurement 5) Range of scores 6) Data sources
34411_CH08_217_264
234
9/9/05
CHAPTER 8
12:02 PM
Page 234
SEVERITY AND COMORBIDITY
The test of influence is more inclusive and incorporates the concept of subgroup analysis developed earlier. In this case, severity or comorbidity may not be directly related to the outcome, but if it substantially influences the impact of the intervention on outcome, then severity of illness should be measured and included in the analysis on that basis.
Reliability Measures of severity and comorbidity must be reliable, consistently produce the same measurements under the same conditions, particularly given the controversy over the effectiveness of statistical risk adjustment strategies. Because unreliable measures cannot be valid, the entire validity of an outcomes study could be questioned if unreliable measures are used and concerns about possible selection bias are raised.
Validity An important, but often overlooked, question to ask before using any severity and/or comorbidity measure is: How was the measure validated? Most measures are validated by showing that they predict an outcome measure of interest, such as mortality. Therefore, some measures explicitly incorporate prognosis into their construction. However, it is important to note that this validity is conditional. A particular measure is valid only in situations with similar characteristics. For example, a measure validated by predicting in-hospital mortality for adult patients admitted to an intensive care unit can be considered reasonably valid for use in a similar in-patient population, but should definitely not be assumed valid for use in a study examining the functional status of children with rheumatoid arthritis in an outpatient setting. A good example of a measure for which this issue has been addressed is the Charlson Index; several variations of the Charlson Index have been adapted for different settings. The more closely an investigator’s study resembles the circumstances where a measure was validated, the more likely it is that the measure will be successful in capturing the important components of severity and/or comorbidity. The validity of a measure is particularly important if it does not “work.” That is, if a severity or comorbidity measure utilized in an outcomes analysis cannot be shown to be statistically related to the outcome of interest or
34411_CH08_217_264
9/9/05
12:02 PM
Page 235
How to Choose a Measure
235
to identify important subgroups for analysis, then the fundamental validity of a measure should be questioned. It is indeed possible that the measure has never been validated for that purpose. To avoid this unfortunate situation, a researcher should closely examine the circumstances under which a measure has been validated and consider the consequences of not meeting one of the five main validation criteria (see Table 8–3). Prognostic Endpoint A basic reason to measure severity of disease/illness and comorbid diseases is the assumption that they affect the study outcome, independently or by confounding the effect of the treatment or primary disease. Most measures have been validated for only one measure of outcome. For example, many studies validated endpoints while assessing mortality: measure validation outcomes include outpatient services utilization, disability, and appropriateness of care (Kahn et al., 1988; Charlson et al., 1987; Charlson, Szatrowski, Peterson, & Gold, 1994; Parkerson, Broadhead, & Tse, 1993; Greenfield, Blanco, Elashoff, & Ganz, 1987; Greenfield et al., 1988; Verbrugge, Lepkowski, & Imanaka, 1989). Wood and colleagues (1981) use three categories to classify the endpoint used to validate measures (i.e., the prognostic endpoint): • Medical meaningfulness (e.g., death, nursing home placement, functional status) • Economic meaningfulness (e.g., resource consumption) • Administrative meaningfulness (e.g., length of stay) A measure should be selected that has been validated against the outcome of interest in the current study. There is some overlap; some measures have been validated for more than one type of prognostic endpoint, and this makes them particularly useful in studies examining more than one type of outcome. For example, an investigator might choose a measure that was created to predict mortality. However, if the investigator is interested in an outcome other than mortality, a measure shown to predict mortality might not be the most appropriate one to use. This concern is especially relevant for ambulatory care, where death is an unlikely endpoint and disability is more likely; a measure created to predict mortality may not predict disability
Organ system Patients involvement, risk of complications
DS (Clinical)
Multiple settings
Hospital
Treatment difficulty LOS
CSI
Medical record review
Medical record review
Discharge abstract
Medical record review
Score: 1–4 for each of 820 disease groups and overall Stages: 1.0–3.0 for each of 420 diagnoses
Any time period
Score: 467 categories
Score: 0–71
Range of scores
Any time period
Entire hospitalization
Worst value during 24 hours
Timing of measurement
Severity
Severity
Severity
Severity
Typically used in measuring severity or comorbidity?
12:02 PM
Patients
Hospital
Total charges, Patients LOS (length of stay)
DRGs
Hospital intensive care unit
In-hospital mortality
Data requirements
CHAPTER 8
Adult patients
Setting
9/9/05
APACHE II
Measure
Patient population
236
Relevant outcome
Validity
Table 8–3 Measurement Properties for Severity and Comorbidity Measures
34411_CH08_217_264 Page 236
SEVERITY AND COMORBIDITY
Patients
Patients
Adult patients Outpatient primary care setting
1-year mortality risk
In-hospital mortality
Follow-up visit, greater than 6 outpatient visits, hospital admission, and
Charlson (Original)
ICED
Duke Severity of Illness
Hospital
Entire hospitalization Any time period
Survey patient Score: 0–100 at time of visit or review charts for visit anytime
Medical record review Medical record review Patient survey, or medical review, using checklist
Score: 0–3
Score: 0–6
How to Choose a Measure
continues
Severity
Comorbidity
Comorbidity
Severity
Severity
12:02 PM
Hospital
Score: 0–4
Worst value over first 48 hours
Medical record review
Relative weights with the average set equal to 100 for each scale
Entire hospitalization
Discharge abstract
9/9/05
Hospital
Patients
In-hospital mortality and morbidity
Medisgroups (Original)
Hospital
Patients
In-hospital mortality, LOS total charges
DS (Scale)
34411_CH08_217_264 Page 237
237
Use of outpatient services
ACGs
Outpatient
Adult patients Outpatient
Hospitalization Patients using and 1-year prescribed mortality drugs
CDS
Male patients VA hospital
ICD-9 diagnoses codes from insurance claims
HMO pharmacy database
Medical record review
Use charges over 1-year period
Over a 1-year time period
Any time period
Timing of measurement
51 categories
Score: 0–35
Score: 0–3
Range of scores
Comorbidity
Comorbidity
Comorbidity
Typically used in measuring severity or comorbidity?
12:02 PM
Mortality from comorbid disease
Data requirements
CHAPTER 8
outpatient medical charges within 18 months
Setting
9/9/05
Kaplan & Feinstein
Measure
Patient population
Validity
238
Relevant outcome
Table 8–3 continued
34411_CH08_217_264 Page 238
SEVERITY AND COMORBIDITY
Medical exam of patient and health history (some studies use medical record review)
Any time period
13 organ Comorbidity systems scored (0 no disease to 4 lifethreatening) and added together Score: 0–52
Comorbidity List of 30 comorbidities and is determined if patient has each or not; placed in analysis as separate binary variables
12:02 PM
VA hospital
Mortality
CIRS
Entire hospitalization
9/9/05
Elderly patients (55)
ICD-9 diagnoses codes from in-patient database
LOS, hospital Adult patients Hospital charges, and in-hospital death
Elixhauser
34411_CH08_217_264 Page 239
How to Choose a Measure 239
34411_CH08_217_264
240
9/9/05
CHAPTER 8
12:02 PM
Page 240
SEVERITY AND COMORBIDITY
very well. Nonetheless, the endpoint used to establish the measure need not necessarily be identical to the study outcome to explain some variance in the study outcome. A measure’s inability to predict the relevant outcome in a given study may be interpreted in several ways. First, the measure may not capture all important components of severity of illness/disease and comorbidity important to the outcome. Second, severity of illness/disease and the specific comorbid conditions that were measured simply may not predict the outcome at all. Third, severity of illness/disease and comorbidity may not predict the outcome in the study population due to unique characteristics of the study sample. The situation is clear if the measure chosen was validated against the same prognostic endpoint and in a similar population to those in the present study. Some external factor not accounted for in the model may affect the outcome itself. For example, if the study uses claims data from an insurance plan and hypothesizes that patients with greater comorbidity and more severe disease/illness will use more mental health services, varying coverage limits on the number of mental health visits may make it difficult to assess the true sizes of the effects of severity and comorbidity. In this case, the unmeasured confounding variables differed between the study population and the population used to validate the measure, and this is consistent with the third explanation just listed. In relation to comorbidity, measures of prognostic endpoints as well as the disease of primary interest will influence which comorbid conditions are weighted more heavily. In some measures, physical domains are weighted more heavily than cognitive or emotional domains. For example, if mortality is used as the endpoint, many cognitive and psychiatric conditions will be weighted lightly compared to physical conditions.
Population of Interest Numerous populations of interest exist in outcomes research. These include children, young adults, the elderly, psychiatric patients, patients with end stage renal disease; the list is essentially endless. It may be difficult to find a measure validated on a population identical to the proposed population. Some compromises are better than others. If an investigator plans to study a population of hospitalized adult patients with chronic renal disease, it is reasonable to use a measure validated on all hospitalized adult patients, although it would be best to combine this measure with other markers known to be important in chronic renal disease (e.g., creatinine levels). It is
34411_CH08_217_264
9/9/05
12:02 PM
Page 241
How to Choose a Measure
241
unlikely that a measure developed for psychiatric patients would be useful. For comorbidity, the population used to validate the measure influences which conditions are weighted most heavily. There are two ways to calculate the weights: 1. for a population as a whole (for example, for the in-patient population in a hospital) 2. for the specific population with the disease of interest (for example, diabetics) The weights calculated using a general population might be less applicable to a specific population. In particular, secondary conditions may have different implications for health relative to one disease compared to another. For example, physical diseases may be worse when cognitive impairments are present.
Setting The setting is closely tied to the population. The largest distinction is typically between in patient and outpatient care. The population in one setting may be fundamentally different from that in another, and setting can simply be viewed as another descriptor of the population. The number of settings in which patients receive care is rapidly expanding. For example, the frail elderly are cared for at home, in nursing homes, in adult day care centers, in assisted living centers, and so on. As a result, it is unlikely that a measure for the frail elderly will have been validated in every setting. As before, it is important to make conscious compromises to the validity of a measure by considering this issue explicitly before a method is chosen. In particular, if a measure has been found to be valid in several settings similar to the one of interest, it makes a good choice for use in a setting where no validated measures are available.
Timing of Measurement The timing of data collection for severity and comorbidity measures can have profound consequences for the results. The most important issue is to relate the timing of data collection for severity and comorbidity to the
34411_CH08_217_264
242
9/9/05
CHAPTER 8
12:02 PM
Page 242
SEVERITY AND COMORBIDITY
intervention of interest. Ideally, information on severity and comorbidity should be based on observations made prior to the application of the intervention. Often, some overlap is unavoidable, but the consequences of confounding the severity of the initial illness and comorbidities with the effects of the intervention should be considered explicitly. The timing of data collection of severity and comorbidity are discussed separately in this section.
Severity Ideally, information about the severity of illness/disease should be collected at two time points (see Figure 8–5): • Prior to symptom onset • At presentation (i.e., after symptom onset, but prior to intervention) Measurement of the patient’s burden of disease prior to symptom onset may be as important as measurement of severity after the onset of symptoms. For example, one patient may be debilitated and living in a nursing home, whereas another patient is completely independent and lives alone in his or her own home. Both patients have identical strokes and become paralyzed on the right side of their body. The first patient has a substantially higher probability of a poor outcome. Information on these patients’ level of functioning prior to the onset of their acute stroke is essential to predicting their long-term outcome. Many severity measures incorporate information about the patient after the intervention of interest has already been implemented. Depending on the study, this may completely invalidate the use of that measure. In all
Baseline measurement prior to symptom onset
Measurement of severity at presentation
Symptom Onset
Intervention
Figure 8–5 Timeline for Measurement of Severity of Illness
Outcome
34411_CH08_217_264
9/9/05
12:02 PM
Page 243
How to Choose a Measure
243
cases, it makes use substantially more problematic. In particular, if an investigator believes that any components of the severity measure might be affected by the intervention, a different measure should be used or the time points for data collection should be restricted to the time prior to the intervention. If the original measure were naively used to adjust for severity of illness, it could easily adjust away the effects of the intervention! The timing of measurement is particularly easy to illustrate for baseline measures of the outcome variable. These measures differ from the final measures of outcome only in the timing of data collection. For example, suppose the outcome of interest is physical functioning at discharge for patients who present to the hospital with acute stroke. Information can be collected on the patients’ level of functioning at admission to the hospital and prior to the onset of symptoms. These values can be used to adjust for the potentially confounding effects of differences in both the severity of the initial stroke and differences in the baseline levels of functioning prior to the stroke.
Comorbidity Measuring the comorbid conditions prior to the treatment will ensure that treatment complications or posttreatment disease complications are not inadvertently included in the comorbidity measures. Inclusion of complications could make the treatment appear to be more beneficial than it really is; therefore, complications of the intervention should never be adjusted for in the analysis. As discussed previously, it is difficult in practice to separate complications of treatment from complications of the disease that occur after the treatment. For example, a diabetic enters the hospital for an MI and develops a kidney infection. It is unclear whether this is due to the treatment for the MI or to the care received in the hospital after being admitted for the MI. There are several plausible mechanisms by which hospital care could influence the development of a kidney infection. Therefore, the safest strategy is to exclude from the comorbidity measure any complication that occurs after the beginning of the treatment. The duration and concurrence of the comorbid conditions should also be considered. Should the comorbidity measure only include conditions that are active? Some conditions may have long-term consequences for the patient’s reaction to medical care. For example, previous hip fracture may make it more difficult for an older patient’s subsequent hip fracture to heal. What kind of time frame will be used to determine whether or not the
34411_CH08_217_264
244
9/9/05
CHAPTER 8
12:02 PM
Page 244
SEVERITY AND COMORBIDITY
disease is still “active” if the data is not collected directly from the patient? The time frame used depends on clinical rationale and the availability of data. For example, a hip fracture that occurred 30 years ago may have less clinical relevance than one that occurred 1 year ago.
Range of Scores The range of scores is most important for categorical measures of severity and comorbidity because each additional category slightly lowers the statistical power of a study. For example, DRGs use almost 500 separate categories to classify patients. This number of categories is clearly manageable from a statistical standpoint only if the sample size is in the tens of thousands. Continuous (interval) measures do not have this problem, although they may have other types of problems such as the presence of influential outliers.
Data Sources Dramatic differences can be apparent in information quality, quantity, validity, and reliability for the different types of data used in outcomes research. There are currently three main types of data: medical records, administrative databases, and patient surveys. Medical records and administrative databases would be classified as secondary data sources, whereas patient surveys are a primary data source. Each type has different benefits and costs. Certain outcomes (e.g., pain) are extraordinarily difficult to collect in a reliable and valid manner from medical records or administrative databases. The solution (albeit a more expensive one) is to ask the patient directly. However, other types of data may only be available from medical records, such as the administration and timing of various therapies or medications, whereas data on hospital costs may be available only from administrative records. Medical records and administrative database data have the advantage of being relatively easy to obtain and on a large number of subjects. However, external factors influence what type of data was originally collected, and these may affect the usefulness of this data for the study. Most computerized or other secondary databases were not collected with outcomes research as a central or even peripheral concern; therefore, the data will contain little detail.
34411_CH08_217_264
9/9/05
12:02 PM
Page 245
How to Choose a Measure
245
The choice of a measure will determine the data source and vice versa, but information from different sources can be combined to measure severity and comorbidity. Measuring severity and comorbidity by more than one method improves the probability of identifying a successful measure, particularly if all five criteria for validity are not met for each individual measure. Most severity and comorbidity measures have been developed for use with medical records or administrative databases. Not much effort has gone into measuring severity of illness or comorbidity using information obtained directly from patients, whereas many outcome measures have been derived for use in direct patient surveys. It is important to remember that baseline measures of the outcomes of interest can almost always be used to measure the initial severity of illness. The usefulness of this approach has probably contributed to the paucity of severity measures developed for use in patient surveys. The best source of data depends on the study and the particular biases of the data sources. Prospective data collection is almost always preferable, because it permits collecting data that may otherwise not be available, as well as in the level of detail sought. However, this type of primary data collection is not always possible due to time or budgetary constraints. A brief description of secondary data sources follows to make investigators aware of where gaps in knowledge occur in the databases, why they occur, and what biases may result from these gaps.
Medical Records Diagnoses Although one might think that the medical records prepared by clinicians would be the most accurate method of determining other illness concurrent with the disease of primary interest and the severity of illness/ disease, this data is not infallible. Clinicians vary in both the thoroughness of their recording and their diagnostic acumen. For example, the diagnosis of dementia may be inconsistently made and rarely will clinicians provide enough information to stage it. Errors of both omission and commission may occur. Some active diagnoses may not be recorded and some inactive problems may be retained. Insurance limitations on which treatments are paid for may influence what the clinician records as the “official” record. A provider may choose a related problem or symptom that is secondary to the primary diagnosis. If a patient with depression had no coverage for mental health and is being prescribed antidepressants, a provider might record
34411_CH08_217_264
246
9/9/05
CHAPTER 8
12:02 PM
Page 246
SEVERITY AND COMORBIDITY
“sleep disturbance” as the diagnosis. The patient is probably having problems with sleep (because sleep disturbances are a symptom of the depression), and so the visit will be covered (antidepressants can also be prescribed for sleep). Medical records may also be difficult to abstract. For example, handwritten notes can cause significant abstraction error.
Administrative Computerized Diagnosis Records Several issues associated with using administrative diagnosis records may affect the choice of a measure. Standard terms for diseases and procedures are found in two common reference books, The International Classification of Disease, 9th revision (U.S. Health Care Financing Administration, 1991), referred to as ICD-9, and The Physician’s Current Procedural Terminology, 4th edition (American Medical Association, 1994), referred to as CPT-4. First, the mapping of diseases to diagnostic ICD-9-clinical modification (ICD-9-CM) and Current Procedural Terminology (CPT-4) codes used in most databases is not necessarily very precise. How a measure “maps” the clinical concept of the disease to the diagnostic and procedure codes may vary from measure to measure. Subtle distinctions may be lost or distorted. For example, should a measure then include ICD-9 codes for “minor depression” and “major depression” in the same way when obviously the two diseases are of different severities? A comparison of ICD-9-CM codes and clinical concepts of comorbidity found that some conditions cannot be precisely mapped to the ICD-9-CM or procedural codes (Romano, Roos, Luft, Jollis, & Doliszny, 1994). An ICD-10, which emphasizes function and more primary care conditions, has been developed but is not widely used. Chronic or asymptomatic conditions may be less prevalent in computerized administrative databases. For example, for prostatectomy and cholecystectomy patients, hospital discharge data had less information on certain chronic conditions than did anesthesiologists’ notes. As a result, these patients had lower scores on the ASA (American Society of Anesthesiologists) Physical Status measure (Roos, Sharp, & Cohen, 1991). However, the exclusion of some types of information does not mean that the measure will have less predictive power for the outcome. In this example, data from the hospital discharge notes and from the anesthesiology notes had similar predictive powers for 1-year mortality and 90day readmission.
34411_CH08_217_264
9/9/05
12:02 PM
Page 247
How to Choose a Measure
247
The number of secondary diagnoses allowed in the administrative database may be limited, either by budgetary constraints or by limitations on the number of diagnoses for which the clinic is paid. Again, this missing information may not have a large impact on the results of a study; the size of this effect depends on the measure used, the outcome of interest, and the study question. For example, if the study hypothesizes that those with chronic diseases such as rheumatoid arthritis will have a higher mortality rate for hip fractures, the investigator will want to be sure that the computerized records are comprehensive enough to include chronic disease diagnoses when acute events are recorded.
Drug Prescription Data Many computerized insurance databases of prescriptions contain records of prescriptions filled by patients, rather than prescriptions written by clinicians. Generally, this distinction may not be very important, but if one wants to use prescriptions as a proxy for illnesses, the prescribing behavior of clinicians is a more accurate reflection than only the subset of the prescriptions actually filled. Prescription databases may also have a coverage bias; insurance claims or health plan records may not reflect all prescriptions filled, particularly if the drug either is not covered by the insurance carrier or the cost of the drug is less than the copayment the patient pays for prescription drugs. For example, some chronic diseases such as arthritis may be treated with nonprescription (over-the-counter) drugs, which would not be recorded in the database of filled prescriptions. Other drugs such as penicillin (and even digoxin) are inexpensive enough that they would not be systematically found in the database. In addition, the types or brands of drugs included in the database may be limited to those on the formulary, particularly in the case of managed care pharmacies. Using prescription databases is even more problematic if there was a change in the formulary during the period of the study so that a drug that was once covered is no longer covered or vice versa. Likewise, it pays to be alert to changes in FDA regulations that permit drugs once sold only by prescription to be sold over the counter. For example, the change to over-the-counter status for medications used to treat yeast infections may have spurred a sudden drop in the number of prescriptions for these drugs, but this decline does not indicate a real decrease in the incidence of yeast infections.
34411_CH08_217_264
248
9/9/05
CHAPTER 8
12:02 PM
Page 248
SEVERITY AND COMORBIDITY
ISSUES SPECIFIC TO SEVERITY MEASURES There has been much debate over whether severity is a generic or diagnosis-specific construct. There are three general types of measures (Iezzoni, 1994). First, “generic” measures rate the severity of a patient’s condition using variables that are measured identically for every patient. The other two types of measures utilize diagnosis-related information. The second measure can be called “diagnosis specific.” This measure is developed only for patients with a certain diagnosis (e.g., patients with pneumonia). The third measure can be called “diagnosis considered.” It considers diagnoses by calculating severity ratings for each of a patient’s diagnoses and combining them into an overall score. The choice of generic, diagnosis-specific, or diagnosis-considered measures of severity depends mainly on the hypotheses and populations of interest. Generic measures can usually be collected on all types of populations. Measures developed separately for each diagnosis (i.e., “diagnosis specific”) are useful when only one, or at most several, condition is being studied. When data on a large number of patients with many disparate conditions is being collected, severity rating systems that “consider diagnoses” may be most appropriate. It is important to note that more than one type of measure can be used, and different measures may add different types of information to the analysis.
ISSUES SPECIFIC TO COMORBIDITY MEASURES In an ideal research world, investigators would be able to create a customized comorbidity measure for each primary diagnosis. However, given time and budget restrictions, this is not usually possible, so the investigator must resort to using a general measure created for use in another study. One must consider the practical losses incurred in using a general measure, especially the ability of the entire model to explain variance in the outcome. Additionally, using a general measure rather than disease-specific measure will influence the size of the estimated effect of “comorbidity” on the outcome; the impact of “comorbidity” on the outcome, often measured by relative risks or regression coefficients, will vary depending on the measure used. A general measure may still be significantly related to the outcome, but the ability of the model to explain variance in the outcome may be lower than for a model using a disease-specific comorbidity measure.
34411_CH08_217_264
9/9/05
12:02 PM
Page 249
Issues Specific to Comorbidity Measures
249
An alternative to using the general measure alone is for the investigator to use the general measure in combination with individual variables for other relevant diseases not addressed by the general measure. However, the more individual variables included, the more degrees of freedom are used (which are limited by the size of the sample).
Disease of Primary Interest Several critical issues should be considered in either choosing an already developed measure of comorbidity or creating a new measure. The study’s primary disease or set of diseases will influence the choice of a measure for comorbid disease, because comorbid diseases are defined only in relation to the principal disease, and the types of data collected will differ depending on the primary disease. For example, in a study that compares outcomes among health care plans for a broad range of illnesses (from rheumatoid arthritis to stroke), the comorbidity measure appropriate to the outcomes for rheumatoid arthritis will likely be quite different from the comorbidity measure used in comparing outcomes for stroke. The diversity of diseases this type of study examines suggests using several comorbidity measures depending on which outcomes the investigator is analyzing. For example, investigators could employ a measure previously used to predict outpatient utilization in comparing outcomes for rheumatoid arthritis, but they should use a measure that predicts mortality when comparing stroke outcomes.
Conditions or Diseases Included A crucial issue in constructing a comorbidity measure is what diseases or conditions to include. Some measures limit the conditions to only those considered important (i.e., diseases not on the list have an implicit weight of 0). Other measures are comprehensive, assigning virtually all diagnostic codes some weight to incorporate them in the comorbidity scale whether or not they are relevant to the study outcome. A structured list can make it easier to abstract from the medical records. One way to contain the length of such a list is to include only those with the largest effect on the outcome. In addition, such an explicit list also can account for differences in terminology used in medical records.
34411_CH08_217_264
250
9/9/05
CHAPTER 8
12:02 PM
Page 250
SEVERITY AND COMORBIDITY
Measures of comorbidity may also include other risk factors that are not “diseases,” such as functional status, age, disease prognosis, or expected response to treatment (Charlson et al., 1994; Charlson et al., 1987; Greenfield et al., 1988; Greenfield et al., 1987; Kahn et al., 1988; Parkerson et al., 1993). When measures include these other factors, the investigator will not be able to differentiate between the effect due to the comorbidity and the effect due to these factors. One should be careful not to include in the analysis these other factors if the comorbidity measure already includes them. For example, in using age-adjusted Charlson comorbidity scores, one should not also include age as a separate variable in the regression. As an alternative to using a preexisting comorbidity measure, each relevant comorbid disease can be used in the analysis separately. This approach enables more precise estimates of the specific effects of comorbid conditions, but it can generate a large number of independent variables. For example, if the investigator thinks that osteoarthritis has a particular effect on outcomes of hip fractures, she or he may put into the regression a variable representing whether or not the patient has osteoarthritis. The coefficient on this variable would tell the investigator the particular association between rheumatoid arthritis and outcomes of hip fracture. However, this method can become very complex if many diseases are expected to influence the outcomes; the investigator may run into problems with the size of the study sample relative to the number of variables in the analyses.
Unweighted Versus Weighted The easiest method of measuring comorbid conditions would be to simply count the number of illnesses the patient has aside from the primary diagnosis. However, this simple summary approach has several weaknesses. First, by not explicitly assigning any weights, the actual weight each comorbid condition carries is the same; that is, this “unweighted” measure assumes that rheumatoid arthritis and stroke exert an equally important effect on the outcome, implicitly assigning each a weight of 1.0. This assumption may not be true. Second, this type of measure does not adjust for the severity of the comorbid conditions themselves. When one explicitly assigns weights, a greater importance can be given to cases in which the comorbid condition is more severe. For example, if the comorbid condition is cancer, assign a larger weight to patients with a stage IV diagnosis than to patients with a stage I diagnosis.
34411_CH08_217_264
9/9/05
12:02 PM
Page 251
Specific Measures
251
The weights assigned to particular comorbid conditions will vary for different primary diagnoses; that is, a comorbid condition may have a greater influence on the outcome when one considers one primary diagnosis rather than when one considers another. In addition, weights may be assigned not only to specific conditions, but also to the severity of the comorbid condition. In considering the weights assigned to each comorbid condition, the investigator should also examine the method used to combine the weighted comorbid disease into a score and examine whether this method is relevant to the disease of primary interest. For example, the Charlson Comorbidity Index uses an additive scale, assuming that the effect of combined diseases is equal to the added sum of their individual scores, weighted for severity (Charlson et al., 1987). Counting the number of conditions may not be the appropriate method of aggregating the scale. Some diseases may have synergistic effects (i.e., combined with another disease, their effect is multiplicative rather than additive) (Satariano, 1992).
SPECIFIC MEASURES In determining whether to use a comorbidity measure versus a severityof-illness measure, the researcher should consider both the study design and the construction of the measure. If the study focuses on one or a few particular diagnoses, a measure of comorbidity will be useful. If the study concerns comparisons of groups but does not focus on specific diseases, the researcher may find a severity of illness measure more appropriate. Finally, an important issue to remember is that measures used to assess comorbidity were often created to assess severity of illness; researchers have adapted these measures as comorbidity measures simply by excluding the disease of primary interest when calculating scores for the measure. Labeling a measure as a “comorbidity measure” is arbitrary and is based on how the researcher intends to use it and not necessarily how it was created. When used as comorbidity measures, the research may simply exclude the disease of primary interest from the data when calculating the measure. In using measures in this manner, the investigator should carefully review the outcome against which the measure was validated and the population in which it was validated. The number of measures in the literature has expanded dramatically over the last two decades. Many of these measures have been through several revisions. It would be impossible to provide a detailed analysis of all of these severity and comorbidity measurement systems within the scope of
34411_CH08_217_264
252
9/9/05
CHAPTER 8
12:02 PM
Page 252
SEVERITY AND COMORBIDITY
this chapter. The tools developed in the preceding section provide a useful starting point for a brief analysis of several representative measures. From among the range of available comorbidity measures several existing measures have been selected for illustrative purposes, including the Acute Physiology, Age, and Chronic Health Evaluation (APACHE), DiagnosisRelated Groups (DRGs), Computerized Severity Index (CSI), Disease Staging (DS), Medisgroups, Charlson Comorbidity Index, Index of Coexistent Disease (ICED), Duke Severity of Illness, Kaplan and Feinstein, Chronic Disease Score (CDS), Ambulatory Care Groups (ACGs), Elixhauser et al., and Cumulative Illness Rating Score (CIRS). Each of these measures is included because it has demonstrated adequate reliability and validity over time. Table 8–3 summarizes the circumstances under which these methods were originally validated.
Acute Physiology, Age, and Chronic Health Evaluation Intensive care units (ICUs) admit patients with a wide variety of diagnoses, and these patients account for a substantial portion of hospital expenditures (Knaus, Draper, & Wagner, 1983). In most cases, patients are admitted to ICUs because of broad-based organ dysfunction (e.g., cardiovascular or respiratory collapse), not because of their primary diagnosis. The original APACHE scoring system was developed to predict the risk of death in these patients and consisted of two parts: an acute physiology score (APS) and a chronic health evaluation (CHE) score. Thirtythree variables comprised the APS and represented measures of generic physiological functioning during the first 32 hours (e.g., heart rate, respiratory rate, urine output, hematocrit). Each variable was weighted from 1 to 4 based on a literature review and the consensus from a panel of physicians; all variables were totaled to get a final APS value. The CHE was a four-category scale (from A to D) representing physiological reserve. Consequently, each patient was assigned a category (such as 20-A) that combined information regarding both of these measures. Due to criticism of the large number of variables and time allowed for data collection, the APACHE scoring system was subsequently revised (Wong & Knaus, 1991). The APACHE II was developed in 1985 and was explicitly based on statistical modeling of in-hospital mortality (Knaus, Draper, Wagner, &
34411_CH08_217_264
9/9/05
12:02 PM
Page 253
Specific Measures
253
Zimmerman, 1985) (see Table 8–3). The timing of data collection was reduced to the first 24 hours. The number of APS variables was reduced from 33 to 12, and the threshold and weights of variables were modified. Surgical procedures and age were incorporated into the score, and chronic health points were assigned only if the patient had a history of severe organ system insufficiency or was immunocompromised. The subsequent APACHE II summed the values from three parts: APS points, age points, and chronic health points. A second revision (Knaus et al., 1991) was developed (APACHE III) to improve the predictive value of the scoring system and to incorporate information about ICU organization and management that may contribute to hospital care.
Diagnosis-Related Groups In the late 1970s, there was increasing interest in containing the growth in health care costs. In order to compare costs across hospitals, measures of hospital case mix were needed. DRGs were developed to define hospital case mix by grouping patients with similar clinical attributes and output utilization patterns (Fetter, Shin, Freeman, Averill, & Thompson, 1980). Length of stay was considered to be the primary utilization pattern of interest. DRGs classified each patient into a major group based on the patient’s primary diagnosis. Other variables such as surgical procedures and the patient’s age were used to refine the classification so that patients with similar lengths of stay were grouped together. The result was 383 diagnosisrelated groups that were medically meaningful and also grouped patients with respect to a major indicator of output utilization (length of stay). In 1983, Medicare began paying hospitals based on an expanded set of 467 DRG categories (Vladeck, 1984) (see Table 8–3). As a result of their widespread use as a payment mechanism and criticism regarding the relative paucity of information about severity of illness, DRGs have recently been refined through the addition of information regarding secondary diagnoses (Freeman et al., 1995). Nevertheless, because DRGs use information from the entire hospitalization, they may represent the effects of therapeutic interventions in addition to the underlying severity of the initial illness. Their use as a risk-adjustment tool remains limited, although their value for reimbursement purposes makes them an important focal point for the development of more refined measures of severity of illness.
34411_CH08_217_264
254
9/9/05
CHAPTER 8
12:02 PM
Page 254
SEVERITY AND COMORBIDITY
Computerized Severity Index After criticism of the DRGs’ ability to classify severity of illness, the CSI was developed as an alternative (see Table 8–3) (Horn et al., 1991). The CSI is based on the ICD-9 coding system (U.S. Health Care Financing Administration, 1991), which indicates the existence of disease by assigning a code to a specific diagnosis. The CSI extends the ICD-9 coding system by incorporating diagnosis-specific information about disease severity. The definition of severity used for developing the CSI was treatment difficulty (proxied by length of stay) due to the extent and interactions of a patient’s diagnoses (Iezzoni et al., 1992). The CSI uses explicit criteria to assign a severity level from 1 to 4 for each diagnosis. This system also combines information about diagnoses using diseasespecific weighting rules and assigns an overall patient severity level from 1 to 4.
Disease Staging The original clinical version of DS is based on a purely conceptual model of disease severity (Gonnella, Hornbrook, & Louis, 1984). Its unique conceptual basis makes it worth a brief examination, although the reliability and validity of the original version have not been reported in detail (see Table 8–3). In DS, each diagnosis represents a disease (not a symptom or physical abnormality) that is conceptualized based on the organ system involved (e.g., cardiovascular) and the etiology (e.g., degenerative). A level of severity (from 1.0 to 3.0, with 4.0 representing death) is assigned to each disease based on the degree of complications and generalized systemic involvement. A panel of medical consultants specified disease-specific staging criteria for approximately 400 diseases. A computerized version of DS was developed empirically using information from the entire hospitalization (see Table 8–3) (Markson, Nash, Louis, & Gonnella, 1991). This version incorporates information regarding the stage of each of a patient’s diseases as well as other patient characteristics. Three different scales are produced for each patient, based on separate predictions of in-hospital mortality, length of stay, and in-hospital charges. As mentioned earlier, the use of information from the entire hospitalization may confound the effects of therapeutic interventions with the underlying severity of the initial illness and limits the broad applicability of this measure for risk adjustment.
34411_CH08_217_264
9/9/05
12:02 PM
Page 255
Specific Measures
255
Medisgroups Medisgroups classifies patients into one of five severity groupings at admission based on relative degree of organ failure (see Table 8–3) (Brewster et al., 1985). Patients are initially categorized by the reason for admission, although this categorization is not used in the final severity grouping. The assignment of a patient to a severity group is based on Key Clinical Findings (KCFs) that are objective indicators of abnormal physiology (e.g., electrocardiogram, chest X-ray, respiratory rate). The level of each KCF receives a score from 1 to 4 and the most severe KCF determines the patient’s severity grouping (even if it is unrelated to the reason for admission).
Charlson Comorbidity Index The Charlson Comorbidity Index was originally created for use with medical chart abstracts to predict mortality for hospitalized patients (see Table 8–3). It was then tested for its ability to predict 1-year mortality rates from comorbid disease in a population of women treated for breast cancer. Data about the comorbid conditions was collected from reviews of hospital records. Comorbid conditions are classified based on a clinical judgment of prognostic relevance; resolved conditions are excluded. The measure incorporates information about the severity of several common conditions. Weights for each disease were calculated from the adjusted relative risk of mortality associated with each disease; that is, diseases were weighted based on the ratio of the incidence rate of mortality for patients with the disease compared to the incidence rate of mortality for those without the disease. If patients with the disease were just as likely to die as were patients without the disease, the relative risk of mortality for the disease would be 1.0. Weights for the relative risks were basically assigned according to a 3-point ordinal scale, with 1 point assigned to the conditions with the lowest relative risks and 3 points to those with the highest, but an exceptional rating of 6 was also used. Conditions with relative risks below 1.2 were dropped from the measure. Diseases with relative risks between 1.2 and 1.5 were assigned 1 point; diseases with relative risks of 1.5 to 2.5 were assigned 2 points; and diseases with relative risks of 2.5 to 6.0 were assigned 3 points; in addition, a small number of conditions with relative risks greater than 6 were assigned 6 points. In all, 19 conditions are included in the weighted measure. The scores for each disease are then
34411_CH08_217_264
256
9/9/05
CHAPTER 8
12:02 PM
Page 256
SEVERITY AND COMORBIDITY
summed to create an index. The comorbidity index significantly predicts 1-year survival. Moreover, the model with the comorbidity index explains more variance in the outcome than the model with the disease entered individually (Charlson et al., 1987). The comorbidity index has been further refined by combining information about the subject’s age with the comorbid diseases, in which each decade of age over 40 years adds another point to the total comorbidity index score. The Charlson Index worked as well as the Kaplan and Feinstein (Kaplan & Feinstein, 1974) method of classifying comorbid diseases in differentiating between patients with low and high levels of comorbidity and in explaining variance in mortality (Charlson et al., 1994; Charlson et al., 1987). Additional studies have modified this instrument for use with (computerized) claims data by “mapping” the clinical disease definitions to specific ICD-9-CM codes and for use with specific diseases of interest (Deyo, Cherkin, & Ciol, 1992; Roos et al., 1991; Romano, Roos, & Jollis, 1993a, 1993b). The Charlson Index has also been used in studies in which the researchers created data- and population-specific weights (Cleves, Sanchez, & Draheim, 1997; Ghali, Hall, Rosen, Ash, & Moskowitz, 1996; Klabunde, Potosky, Legler, & Warren, 2000).
Index of Coexistent Disease ICED was created to predict risk of mortality for hospitalized patients (see Table 8–3). The index consists of three subscales: (1) initial severity of the comorbid conditions at admission; (2) instability of comorbid conditions (complications) around time of admission; and (3) functional status. The authors derived weights for this scale based on the Disease Staging system, a severity-of-illness measure discussed previously. In the initial severity and instability subscales, weights for each comorbid condition are ordinal. This information is derived from medical records, where any condition that is mentioned at least twice by the physician is scored. The functional status subscale assigns points for functional status in 11 body systems based on information from nurses’ notes. These three subscales are combined into a single index that assigns subjects to one of four ordinal categories, ranging from a nondisease state to a condition of life-threatening illness. The number of categories has varied in different conceptions of the measure from 7 to 3. ICED was validated for an elderly population with primary diagnoses of breast, prostate, and colorectal cancers (Greenfield et al., 1988; Greenfield et al., 1987).
34411_CH08_217_264
9/9/05
12:02 PM
Page 257
Specific Measures
257
Duke Severity of Illness Checklist This checklist was created to measure overall severity of illness, but has been used as a measure of comorbidity by omitting the disease of primary interest in scoring the measure. For each disease, this instrument measures symptom level, complications, prognosis without treatment, and expected response to treatment. A severity score is then calculated for each disease by summing across these four domains. An overall score is created by summing across the severity scores for individual illnesses with the highest weights assigned to the most severe diagnoses. The score is determined by use of a checklist completed by either the medical provider at the time of the patient encounter or by an abstractor looking over medical charts (Parkerson et al., 1993; Parkerson et al., 1989). This measure was validated for primary care adult outpatients to predict outcomes such as medical charges and hospital admission (see Table 8–3) (Parkerson, Broadhead, & Tse, 1995). Some of the information required (prognosis without treatment and expected response to treatment) may be more difficult to derive from medical charts.
Kaplan and Feinstein The Kaplan and Feinstein (1974) measure is a significant predictor of mortality from comorbid disease. It was originally validated when used as a comorbidity measure for mature-onset diabetes in an all-male VA hospital population (see Table 8–3). Diseases were categorized as diagnostically, prognostically, or pathogenically comorbid. Data was abstracted from medical charts (Kaplan & Feinsterin, 1974). Although weights for this measure are based on clinical judgment, it was equally able to differentiate between low- and high-risk groups for mortality from comorbid disease and explains as much variance as the empirically created Charlson Comorbidity Index (Charlson et al., 1987).
Chronic Disease Score This measure was created in part to take advantage of the availability of large administrative databases available from health maintenance organization (HMO) pharmacies (see Table 8–3) (Von Korff, Wagner, & Saunders,
34411_CH08_217_264
258
9/9/05
CHAPTER 8
12:02 PM
Page 258
SEVERITY AND COMORBIDITY
1992). The instrument was created as an indicator of chronic disease morbidity and health status in outpatients using drugs. This measure has been used as a measure of comorbidity by excluding the disease of interest from the measure. Because it does not include psychotropic drugs, it may also be used as originally constructed as a comorbidity measure when the disease of interest is psychological. The Chronic Disease Score maps the diseases most likely to be associated with certain classes of drugs using expert judgment. The measure includes 17 disease categories associated with drug classes. The measure then assigns each of these classes of drugs a weight based on clinical judgments of the severity of these particular conditions relative to the other conditions, as well as the combination of classes of drugs being used. The weighted scores for each class of drugs are then summed to obtain a total CDS. This measure requires that the patient population have the same pharmacy data set. Unlike measures that are based on disease diagnoses, this measure is very susceptible to the economic and clinical environment from which the data is derived. Because this measure infers diagnoses from drug prescriptions, it is susceptible to technological changes in the types of drugs and medical therapies prescribed; this is especially a concern with regulatory changes that change the status of prescription drugs to over the counter. Inferring diagnoses is also difficult if a disease may be treated with either drugs or other treatments. For example, the measure will not count as high cholesterol a patient who has been prescribed a low-cholesterol diet rather than drugs. In addition, it does not address some diseases that are often managed on an outpatient basis, such as AIDS. This measure has been refined and validated with a sample of 250,000 managed care enrollees (Clark, Von Korff, Saunders, Baluch, & Simon, 1995).
Ambulatory Care Groups Measures such as the Ambulatory Care Groups (ACGs) are actually case-mix measures that have been used as “comorbidity” measures by excluding the disease of primary interest when the investigators calculate the scores. Case-mix measures attempt to group patients by the risk (of mortality, of utilization) they share due to their illnesses, but the groups are not necessarily ranked. Severity-of-illness measures tend to rank patients based on their level of illness. ACGs use ICD-9 diagnoses codes from administrative insurance claims data to create a case-mix measure to predict utilization of outpatient ser-
34411_CH08_217_264
9/9/05
12:02 PM
Page 259
Specific Measures
259
vices. Each ICD-9-CM code is assigned to one of 34 ambulatory diagnostic groups (ADGs), which are based on clinical judgments of the stability, chronicity, and expected services use associated with the diagnosis. The ADGs are then further grouped, after intermediate steps, into 51 ACGs, mutually exclusive categories based on number of outpatient visits and total outpatient charges for the period of a year. This measure was validated in a population of adult ambulatory patients (Weiner, Starfield, Steinwachs, & Mumford, 1991; Berlowitz, Rosen, & Moskowitz, 1995).
Elixhauser et al. This measure was created to be used with ICD-9 diagnoses codes from administrative in-patient databases (see Table 8–3). The list of comorbidities in this index is larger than that of most comorbidity measures; 30 diagnoses were determined to be significant in predicting outcomes, which included hospital charges, length of stay, and in-hospital mortality. Any secondary diagnoses in the same DRG as the diagnosis of primary interest are not counted as comorbid diseases nor are health problems that are typically acute, such as pneumonia or respiratory failure. The comorbidities are not weighted into one score, but are included in statistical models as binary (dichotomous) indicator variables. This measure was originally validated on all California hospital in-patients for one year, although the study also analyzed the results on 10 diagnosis-specific subgroups of patients (Elixhauser, Steiner, Harris, & Coffey, 1998). Two other studies also validated this method, as well as compared its predictive ability to another comorbidity measure (Southern, Quan, & Ghali, 2004; Stukenborg, Wagner, & Connors, 2001).
Cumulative Illness Rating Scale This measurement was designed to organize comorbidities by organ system as well as rate the severity of each (see Table 8–3). Comorbidities are classified by 13 organ systems and each condition is rated from 0 (no condition present) to 4 (extremely severe). The scores for all 13 categories are then summed to give an overall score, which can range from 0–52 (Linn, Linn, & Gurel, 1968). The overall score has also been computed in other ways, such as summing the number of positive systems, obtaining a mean score, or counting the number of diseases rated 3 or 4 (Extermann, 2000). This measure has
34411_CH08_217_264
260
9/9/05
CHAPTER 8
12:02 PM
Page 260
SEVERITY AND COMORBIDITY
also been adapted for use specifically in the geriatric population, hence the name of the CIRS-Geriatric (CIRS-G). The CIRS-G contains diseases that are more specific to elderly patients (Miller et al., 1992). SUMMARY This chapter has examined the conceptual and methodological issues involved in measuring severity of illness/disease and comorbidity in outcomes research. Many measures have been developed, but all have limitations. Most focus on physiological functioning to the exclusion of other domains of health. Others have not been adequately tested for reliability and validity across a wide range of settings. Few measures are available for nonhospital settings such as nursing homes or ambulatory care clinics. Consequently, compromises must often be made when conducting an outcomes research study. When choosing a set of measures, each investigator should strive to make explicit and conscious compromises to the data collection and analytic strategy based on a well-conceived conceptual model. This conceptual model should explicitly link the intervention of interest to the outcome, incorporate information regarding all important and related domains of initial health, and set a specific timeline for data collection that will not confound the relationship of severity of illness/disease or comorbidity with the complications of the intervention. This approach enables a closer approximation in answer to the fundamental question of outcomes research: Is the patient truly better off as a result of the intervention? REFERENCES American Medical Association. (1994). Physician’s current procedural terminology (4th ed.). Chicago: Author. Aronow, D.B. (1988). Severity-of-illness measurement: Applications in quality assurance and utilization review. Medical Care Review, 45(2), 339–366. Berlowitz, D.R., Rosen, A.K., & Moskowitz, M.A. (1995). Ambulatory care casemix measures. Journal of General Internal Medicine, 10(3), 162–170. Brewster, A.C., Karlin, B.G., Hyde, L.A., Jacobs, C.M., Bradbury, R.C., & Chae, Y.M. (1985). MEDISGRPS: A clinically based approach to classifying hospital patients at admission. Inquiry, 22(4), 377–387. Charlson, M.E., Pompei, P., Ales, K.L., & MacKenzie, C.R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40(5), 373–383.
34411_CH08_217_264
9/9/05
12:02 PM
Page 261
References
261
Charlson, M., Szatrowski, T.P., Peterson, J., & Gold, J. (1994). Validation of a combined comorbidity index. Journal of Clinical Epidemiology, 47(11), 1245–1251. Clark, D.O., Von Korff, M., Saunders, K., Baluch, W.M., & Simon, G.E. (1995). A chronic disease score with empirically derived weights. Medical Care, 33(8), 783–795. Cleves, M.A., Sanchez, N., & Draheim, M. (1997). Evaluation of two competing methods for calculating Charlson’s Comorbidity Index when analyzing short-term mortality using administrative data. Journal of Clinical Epidemiology, 50(8), 903–908. Deyo, R.A., Cherkin, D.C., & Ciol, M.A. (1992). Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. Journal of Clinical Epidemiology, 45(6), 613–619. Elixhauser, A., Steiner, C., Harris, D.R., & Coffey, R.M. (1998). Comorbidity measures for use with administrative data. Medical Care, 36(1), 8–27. Extermann, M. (2000). Measuring comorbidity in older cancer patients. European Journal of Cancer, 36(4), 453–471. Fetter, R.B., Shin, Y., Freeman, J.L., Averill, R.F., & Thompson, J.D. (1980). Case mix definition by diagnosis-related groups. Medical Care, 18(2 Suppl), 1–53. Freeman, J.L., Fetter, R.B., Park, H., Schneider, K.C., Lichtenstein, J.L., Hughes, J.S., Bauman, W.A., Duncan, C.C., Freeman, D.H., Jr., & Palmer, G.R. (1995). Diagnosisrelated group refinement with diagnosis- and procedure-specific comorbidities and complications. Medical Care, 33(8), 806–827. Ghali, W.A., Hall, R.E., Rosen, A.K., Ash, A.S., & Moskowitz, M.A. (1996). Searching for an improved clinical comorbidity index for use with ICD-9-CM administrative data. Journal of Clinical Epidemiology, 49(3), 273–278. Gonnella, J.S., Hornbrook, M.C., & Louis, D.Z. (1984). Staging of disease. A case-mix measurement. Journal of the American Medical Association, 251(5), 637–644. Green, J., Wintfeld, N., Sharkey, P., & Passman, L.J. (1990). The importance of severity of illness in assessing hospital mortality. Journal of the American Medical Association, 263(2), 241–246. Greenfield, S., Aronow, H.U., Elashoff, R.M., & Watanabe, D. (1988). Flaws in mortality data. The hazards of ignoring comorbid disease. Journal of the American Medical Association, 260(15), 2253–2255. Greenfield, S., Blanco, D.M., Elashoff, R.M., & Ganz, P.A. (1987). Patterns of care related to age of breast cancer patients. Journal of the American Medical Association, 257(20), 2766–2770. Horn, S.D., Sharkey, P.D., Buckle, J.M., Backofen, J.E., Averill, R.F., & Horn, R.A. (1991). The relationship between severity of illness and hospital length of stay and mortality. Medical Care, 29(4), 305–317. Iezzoni, L.I. (1994). Chapter 2. Dimensions of risk. In L.I. Iezzoni (Ed.), Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press. Iezzoni, L.I., Ash, A.S., Coffman, G.A., & Moskowitz, M.A. (1992). Predicting in-hospital mortality. A comparison of severity measurement approaches. Medical Care, 30(4), 347–359. Iezzoni, L.I., Shwartz, M., Ash, A.S., Mackiernan, Y., & Hotchkin, E.K. (1994). Risk adjustment methods can affect perceptions of outcomes. American Journal of Medical Quality, 9(2), 43–48.
34411_CH08_217_264
262
9/9/05
CHAPTER 8
12:02 PM
Page 262
SEVERITY AND COMORBIDITY
Kahn, K.L., Park, R.E., Brook, R.H., Chassin, M.R., Kosecoff, J., Fink, A., Keesey, J.W., & Solomon, D.H. (1988). The effect of comorbidity on appropriateness ratings for two gastrointestinal procedures. Journal of Clinical Epidemiology, 41(2), 115–122. Kaplan, M.H., & Feinstein, A.R. (1974). The importance of classifying initial co-morbidity in evaluating the outcome of diabetes mellitus. Journal of Chronic Diseases, 27(7–8), 387–404. Klabunde, C.N., Potosky, A.L., Legler, J.M., & Warren, J.L. (2000). Development of a comorbidity index using physician claims data. Journal of Clinical Epidemiology, 53(12), 1258–1267. Knaus, W.A., Draper, E.A., & Wagner, D.P. (1983). Toward quality review in intensive care: The APACHE system. Qrb. Quality Review Bulletin, 9(7), 196–204. Knaus, W.A., Draper, E.A., Wagner, D.P., & Zimmerman, J.E. (1985). APACHE II: A severity of disease classification system. Critical Care Medicine, 13(10), 818–829. Knaus, W.A., Wagner, D.P., Draper, E.A., Zimmerman, J.E., Bergner, M., Bastos, P.G., Sirio, C.A., Murphy, D.J., Lotring, T., Damiano, A., et al. (1991). The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest, 100(6), 1619–1636. Linn, B.S., Linn, M.W., & Gurel, L. (1968). Cumulative Illness Rating Scale. Journal of the American Geriatrics Society, 16(5), 622–626. Markson, L.E., Nash, D.B., Louis, D.Z., & Gonnella, J.S. (1991). Clinical outcomes management and disease staging. Evaluation & the Health Professions, 14(2), 201–227. Miller, M.D., Paradis, C.F., Houck, P.R., Mazumdar, S., Stack, J.A., Rifai, A.H., Mulsant, B., & Reynolds, C.F., III. (1992). Rating chronic medical illness burden in geropsychiatric practice and research: Application of the Cumulative Illness Rating Scale. Psychiatry Research, 41(3), 237–248. Parkerson, G.R., Jr., Broadhead, W.E., & Tse, C.K. (1993). The Duke Severity of Illness Checklist (DUSOI) for measurement of severity and comorbidity. Journal of Clinical Epidemiology, 46(4), 379–393. Parkerson, G.R., Jr., Broadhead, W.E., & Tse, C.K. (1995). Health status and severity of illness as predictors of outcomes in primary care. Medical Care, 33(1), 53–66. Parkerson, G.R., Jr., Michener, J.L., Wu, L.R., Finch, J.N., Muhlbaier, L.H., MagruderHabib, K., Kertesz, J.W., Clapp-Channing, N., Morrow, D.S., Chen, A.L., et al. (1989). Associations among family support, family stress, and personal functional health status. Journal of Clinical Epidemiology, 42(3), 217–229. Pompei, P., Charlson, M.E., Ales, K., MacKenzie, C.R., & Norton, M. (1991). Relating patient characteristics at the time of admission to outcomes of hospitalization. Journal of Clinical Epidemiology, 44(10), 1063–1069. Romano, P.S., Roos, L.L., & Jollis, J.G. (1993a). Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: Differing perspectives. Journal of Clinical Epidemiology, 46(10), 1075–1079. Romano, P.S., Roos, L.L., & Jollis, J.G. (1993b). Further evidence concerning the use of a clinical comorbidity index with ICD-9-CM administrative data. Journal of Clinical Epidemiology, 46(10), 1085–1090.
34411_CH08_217_264
9/9/05
12:02 PM
Page 263
References
263
Romano, P.S., Roos, L.L., Luft, H.S., Jollis, J.G., & Doliszny, K. (1994). A comparison of administrative versus clinical data: Coronary artery bypass surgery as an example. Ischemic Heart Disease Patient Outcomes Research Team. Journal of Clinical Epidemiology, 47(3), 249–260. Roos, L.L., Sharp, S.M., & Cohen, M.M. (1991). Comparing clinical information with claims data: Some similarities and differences. Journal of Clinical Epidemiology, 44(9), 881–888. Satariano, W.A. (1992). Comorbidity and functional status in older women with breast cancer: Implications for screening, treatment, and prognosis. Journal of Gerontology, 47 (Health Module), 24–31. Shapiro, M.F., Park, R.E., Keesey, J., & Brook, R.H. (1994). The effect of alternative casemix adjustments on mortality differences between municipal and voluntary hospitals in New York City. Health Services Research, 29(1), 95–112. Southern, D.A., Quan, H., & Ghali, W.A. (2004). Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Medical Care, 42(4), 355–360. Stukenborg, G.J., Wagner, D.P., & Connors, A.F., Jr. (2001). Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Medical Care, 39(7), 727–739. U.S. Health Care Financing Administration. (1991). The international classification of diseases, 9th revision, clinical modification: ICD-9-CM (4th ed.). Washington, DC: U.S. Dept. of Health and Human Services, Public Health Service, Health Care Financing Administration. Verbrugge, L.M., Lepkowski, J.M., & Imanaka, Y. (1989). Comorbidity and its impact on disability. Milbank Quarterly, 67(3–4), 450–484. Vladeck, B.C. (1984). Medicare hospital payment by diagnosis-related groups. Annals of Internal Medicine, 100(4), 576–591. Von Korff, M., Wagner, E.H., & Saunders, K. (1992). A chronic disease score from automated pharmacy data. Journal of Clinical Epidemiology, 45(2), 197–203. Weiner, J.P., Starfield, B.H., Steinwachs, D.M., & Mumford, L.M. (1991). Development and application of a population-oriented measure of ambulatory care case-mix. Medical Care, 29(5), 452–472. Wong, D.T., & Knaus, W.A. (1991). Predicting outcome in critical care: The current status of the APACHE prognostic scoring system. Canadian Journal of Anaesthesia, 38(3), 374–383. Wood, W.R., Ament, R.P., & Kobrinski, E.J. (1981). A foundation for hospital case mix measurement. Inquiry, 18(3), 247–254.
34411_CH08_217_264
9/9/05
12:02 PM
Page 264
34411_CH09_265_304
9/9/05
10:49 AM
Page 265
9 Demographic, Psychological, and Social Todd Rockwood Melissa Constantine
INTRODUCTION Demographic, psychological, and social phenomena cover a wide range from directly observable characteristics or events, such as sex and social interaction, to abstract constructs such as depression and anxiety. A brief example illustrates some of the complexity involved in the conceptualization, measurement, and analytic usage of demographic, psychological, and social variables. When one is asked about “sex,” a range of things could be considered relevant. It becomes important to think not just about what one means when one says “sex” but what its purpose is in research (dependent, independent, or control variable) and how one intends to measure it. Almost every study measures “sex” as male or female, but there are problems with such measurement. If the focus is on gender identity or role, it is not safe to assume that this is what has been measured if all one has determined is male or female. Alternatively, a significant difference among the sexes might reflect differences in how males versus females were treated, and thus the inference should be about treatment, not about sex. In an outcomes research study on fertility, reproductive physiology would be important. Alternatively, in a study focusing on depression, gender identity might be equally or more important. Phenomena in this class are not straightforward (Summers, 1970; Blalock, 1974; Bohrnstedt & Borgatta, 1981; Campbell & Russo, 2001). These things can be objective and directly observable, subjective and not observable, or both. Thus, the use of these variables must be carefully thought out at all levels (conceptually, operationally, and analytically) (Campbell & Overman, 1988; Campbell & Russo, 2001). Simply measuring them because others do, or doing so because they might be relevant, can lead to problems. 265
34411_CH09_265_304
266
9/9/05
CHAPTER 9
10:49 AM
Page 266
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
THREE TYPES OF USAGE Psychosocial variables can be outcomes (dependent variables), things that affect outcomes (independent variables), or things that one controls for in the analysis (control variables). Figure 9–1 illustrates simple examples of each type. In the top part of the figure, depression is illustrated as the primary outcome of the research. The intent of the research is to focus on the impact that counseling and returning to work has on depression that emerges as a response to some type of traumatic loss. In the middle, depression is an independent variable; that is, returning to work after a traumatic loss could be influenced both by an individual receiving counseling as well as the severity of their depression. In the final instance, depression is used as a control variable; the purpose of the research is to evaluate how much influence counseling has on returning to work after a traumatic loss, controlling for the severity of depression. In developing outcomes research, it is important to identify at the beginning of the research what role, if any, psychosocial variables play in the research. Their thoughtless inclusion can lead to spurious findings. If a psychosocial variable is not the outcome (dependent variable), then prior to including it in the study one needs to consider two things: 1. Why the variable has, or should have, a relationship to the outcome 2. Whether it is needed as a control variable DEMOGRAPHIC VARIABLES Demography is the study of populations (Shryock et al., 1976; Weeks, 1981). Most demographic research is based on two variables: age and sex. Although these two variables explain the dynamics of populations for demographic research, they are only two of many variables that could be potential demographic factors in outcomes research. The following discussion is not meant to be an exhaustive list of demographic variables, but instead highlights some of the major ones that are used in outcomes research. Age In outcomes research and in fact almost all research, age is measured chronologically; the amount of time a person has been alive. How age is
34411_CH09_265_304
9/9/05
10:49 AM
Page 267
Demographic Variables
267
Dependent Return to Work
Traumatic Loss
Depression
Counseling
Independent Counseling
Traumatic Loss
Return to Work
Depression
Control Depression
Traumatic Loss
Counseling
Return to Work
Figure 9–1 Psychosocial Variables as Dependent, Independent, and Control Variables
measured should depend primarily upon the nature of the research. Although age is generally measured in years, it is not necessarily the only measure of age. For example, in a study of neonatal intensive care, age could be measured in hours or days. There are two primary methods of measuring chronological age: (1) asking people how old they are, and
34411_CH09_265_304
268
9/9/05
CHAPTER 9
10:49 AM
Page 268
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
(2) asking for their date of birth. In the case of the latter, it is important to also record the date when the question was asked; all too frequently an individual is asked what date they were born on, but the date the question was asked was not recorded and as a result it is not possible to accurately determine age. Measuring age chronologically assumes that the effect of age is more or less constant across all individuals; that is, a 50-year-old is a 50-year-old regardless of what has occurred during that person’s life. Although for some research such a conceptual definition of age is appropriate, it is not necessarily appropriate for all research. All of us at one time or another have said or observed about someone that the person is a “young 50” or an “old 50.” This label reflects a different conceptual definition of age. Age marks not simply the passage of time; it represents the cumulative effects of the passage of time and the events that have occurred during that time. In the past decade, more refined measures have been developed to assess what is being commonly referred to as “real” age, which are intended to reflect the force of mortality. Gerontologists view age in terms of the force of mortality or the likelihood of dying. This concept is more akin to the idea of real age. More attention is being focused on this concept of real age. For example, the chronological ages of the authors of this chapter are 44 and 45, but using a commercial Web site (www.realage.com), the “real” ages are 41 and 40. This conceptualization of age, which has long been used by gerontologists, is slowly making inroads into academic and scholarly research. However, current measures of real age have been primarily used for commercial purposes and have not yet demonstrated good validity, and thus their use in outcomes research should be judicious.
Sex Demographically speaking, sex is about reproductive physiology; is one male or female? Historically, much has been attributed to sex from characterizations such as “females are hysterical,” “males repress emotion,” to health behaviors and gender identity/role. Although at various points in time the correlation between sex and gender might have been high, it is no longer safe to assume that female means feminine or male mean masculine, particularly in a multidimensional conceptualization of gender identity. When asking if one is female or male, it should be assumed that all one is determining is reproductive physiology.
34411_CH09_265_304
9/9/05
10:49 AM
Page 269
Demographic Variables
269
Residence In outcomes research, two issues around residence can be of concern. Location can be used for many different purposes: as an indicator of access to health services in general or particular types (e.g., a tertiary care facility), but also as a factor directly related to other issues associated with health (exposure). At the grossest level, it can be identified as rural/urban. Data with geographic distinctions often comes from secondary sources that are linked to primary data collection. The Area Resource File (www. arfsys.com) contains a wealth of information regarding health care issues at the county level from data available from the Census Bureau as well as health-related studies such as the Behavioral Risk Factor Surveillance Survey (www.cdc.gov/brfss). The key to using this data is ensuring that data collected from study participants allows one to link the primary data to these secondary resources, such as county of residence, ZIP code, actual street address (if permitted by the institutional review board, IRB), or nearest intersection when an address cannot be obtained. The second, often ignored aspect of residence is the characteristics of the dwelling in which one lives or the larger built environment that surrounds each resident (Rubin, 1998; Diez Roux, 2003; Jackson, 2003; Northridge et al., 2003). Although not relevant in all research, the impact of the built environment in terms of health, safety, and recovery can be significant for certain research questions (Rubin, 1998; Lucy, 2003). For example, in an outcomes study of recovery following hip surgery for individuals who return to home, it is critical to identify if sleeping, dining, and bathing facilities are all on the same level. As more attention is devoted to this area with a focus on public health and community (Perdue et al., 2003), the importance of the built environment is reflected in diverse research, from how hospitals are built to floor plans for homes (Douglas & Douglas, 2004).
Race In health research, race and ethnicity are often mistakenly treated as synonymous. They are not just different; the historically assumed correlation between them is decreasing. Although there are numerous definitions of race, they more or less generally converge around a common theme: a population that lives within a specified geographic area and shares the same gene pool. Given the mobility in contemporary society, the historical notion and
34411_CH09_265_304
270
9/9/05
CHAPTER 9
10:49 AM
Page 270
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
importance of race are rapidly diminishing. Although in a few instances— such as diabetes and obesity in the Pima community—race might still be relevant (Lillioja & Bogardus, 1988; Ravussin, 1993; Pratley, 1998; Bennett, 1999; Imperatore et al., 2001), such relevance is not a forgone conclusion for all populations (Clancy et al., 2002). There is the danger of the ecologic fallacy, whereby characteristics or outcomes that are associated with a population are attributed to individuals. In research in communities such as the Pima community, the focus is often on genetic factors, an attribute or characteristic of an individual, not a population. Although the genetic predisposition for a condition could be distributed throughout the population, it is critical to distinguish between genotypic attributes of an individual and the phenotypic attributes expressed in a population. The larger issue deals with the propensity to infer ethnicity from race (Snipp, 2003; Takeuchi & Gage, 2003). The historical correlation between race and ethnicity is declining, especially in industrialized nations. Thus, the temptation to infer ethnicity from race needs to be discouraged. African American is not a race; it is not necessarily even an ethnicity; it could even represent multiple subcultures within a single city. Even issues that might appear simple, such as the use of tobacco in Southeast Asians, demonstrate significant ethnic and cultural differences between Hmong, Vietnamese, Laotians, and Cambodians.
Marital Status Marital status is an unfortunate use of terminology. Strictly speaking, marital status is just an indication of whether or not the individual is in a legally binding relationship that is recognized by the state, which at this point requires that it involve two individuals and have three levels: never, current, and past (divorced). In research, the concept is used somewhat differently; the focus is on relationship(s) and role(s). For example, the desired measure is not necessarily if the person is married, but if they are living in a marriage-like relationship versus living alone. The variable is primarily intended to assess the presence or absence of characteristics associated with relationship(s), intimacy, support (i.e., emotional, physical, fiduciary, etc.), and issues related to the stability and permanency in roles and behavior. In self-report (survey research), the variable is generally operationalized as: Are you currently: (1) married or living in a marriagelike relationship, (2) widowed, (3) divorced, or (4) single. Although options
34411_CH09_265_304
9/9/05
10:49 AM
Page 271
Psychological
271
2 and 3 are a subset of 4, those states can significantly impact the individual and thus they are kept as separate categories. PSYCHOLOGICAL Of the three areas discussed in this chapter, psychological factors are the most complicated. Not only is the domain broad, it is still largely unexplored. The intent of the following material is not to present a summary of the state of knowledge at this point in time nor to cover the full range of psychological/psychiatric issues, but rather to briefly discuss the issues and measures in three general areas: mind-body connection, affect function, and cognitive function. Mind-Body Connection Although historically it is a central part of medical practice, the attention to the mind-body linkage in research is relatively new. The mind-body concept has at its roots traditional Western medical research as well as complementary and alternative medicine. Research into the mind-body connection comes in many forms: from linkages between the neurology and psychology research community in the form of psychoneuroimmunology to popular culture diagnosis, treatments, and cures (e.g., iradology, determining health status by studying the iris, which seems vaguely reminiscent of phrenology, the study of the skull). Measures regarding the mind-body connection are found in all types of research, but the application of the measures should be done thoughtfully, not just because it is in vogue. Measures in this area can be extremely useful in outcomes research in particular areas, such as behavioral medicine (irritable bowel syndrome, fibromyalgia, etc.). Within traditional medicine, the oft-derided placebo effect is not necessarily a transitory phenomenon, as in the case of the arthroscopic versus placebo study for the treatment of osteoarthritis (Moseley et al., 2002). The study demonstrated it can have just as positive outcomes as surgical intervention. The placebo effect is something solidly located in the mind-body connection. The goal of the following material is to discuss some of the major concepts as well as measures in this area. A summary of the measures that will be discussed for each area is found in Table 9–1. Admittedly, the grouping and placement of the concepts in one area versus another is somewhat arbitrary.
34411_CH09_265_304
272
9/9/05
CHAPTER 9
Table 9–1
10:49 AM
Page 272
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Mind-Body Measures
Domain Well-being
Instrument
Description
General Well-Being Schedule Developed for the National (Fazio, 1977) Center for Health Statistics; is intended to measure well-being in the general population Index of Well-Being (Campbell et al., 1976)
Composed of eight semantic differential items and one life satisfaction item
Spheres of Control Battery (Paulhus, 1983)
Three dimensional measures focusing on: 1. personal efficacy (control over one’s own life) 2. interpersonal control (expectancies of outcomes in social situations) 3. sociopolitical control (perception of ability to influence society)
Internality, Powerful Others, and Chance Scales (Levenson, 1981)
Three scales focusing on different aspects of life: 1. internality (perception of control over one’s own life) 2. powerful others (belief that others control events in one’s life) 3. chance (belief that chance determines events and outcomes in life)
Locus of Control General
Health
Multidimensional Health Three-dimensional Locus of Control Scale instrument for health-specific (Wallston & Wallston, 1981) locus of control based upon Levenson’s conceptual framework: 1. internal (individual is in control of own health) 2. chance (health/illness are due to chance)
34411_CH09_265_304
9/9/05
10:49 AM
Page 273
Psychological
Domain
Instrument
273
Description 3. powerful others (health/ illness are in control of external agents)
Mental Health Locus of Control Scale (Hill & Bale, 1980)
Bipolar instrument focused on beliefs about control relative to therapeutic changes. Pole 1: internal (beliefs that patient is responsible for changes) Pole 2: therapist (therapist is responsible for changes)
Pain General
Visual Analog Scales (VAS) and Numeric Rating Scales (NRS) (McDowell & Newell, 1996) Standardization of SelfReported Pain (Kane et al., 2002)
VAS includes a range of methods from 10cm horizontal or vertical lines NRS usually ranges from 0 to 10. Poles in both versions are usually labeled: no pain (0) and pain as bad as it could be, worst pain imaginable, unbearable.
Mcgill Pain Questionnaire (Turk et al., 1985; Turk & Melzack, 1992)
Weighted scale designed to profile three aspects of pain: 1. sensory-discriminative 2. motivational-affective 3. cognitive-evaluative
Medical Outcomes Study Pain Measures (Stewart & Ware, 1992)
Basic multi-item measure intended to measure the impact of pain on daily living; not intended to provide a detailed measure of pain
Pain and Distress Scale (Zung, 1983)
Measure intended to assess mood and behavior changes associated with acute pain
continues
34411_CH09_265_304
274
9/9/05
CHAPTER 9
Table 9–1 Domain
10:49 AM
Page 274
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
continued Instrument
Description
Specific Cancer
Brief Pain Inventory (Cleeland, 1991)
Measures two aspects of pain: 1. sensory 2. functional limitations due to pain
Back
Oswestry Low Back Pain Disability Questionnaire (Fairbank et al., 1980)
Instrument focuses on the impact of pain and contains a rating of pain intensity and assessment of the impact of pain in nine areas: personal care, lifting, walking, sitting, standing, sleeping, sex life, social life, and traveling
Back Pain Classification Scale (Leavitt & Garron, 1979a, 1979b)
Screening tool developed for clinical use to discriminate between back pain due to psychological versus physiological factors
Fatigue, Energy, Consciousness, Energized and Sleepiness (Shapiro et al., 2002)
Adjective checklist developed to assess function in five areas: fatigue, energy, consciousness, energized, and sleepiness
Perceived Stress Questionnaire (Cohen et al., 1983)
Short 10-item survey focused on perception of stress and ability to deal with stress
Perceived Stress Questionnaire (Levenstein et al., 1993)
Longer instrument assessing perceived stress, contains seven subscales: harassment, overload, irritability, lack of joy, fatigue, worries, and tension
Social Readjustement Rating Scale (Holmes & Rahe, 1967)
Event checklist that contains a weighted value of the amount of stress for each event
Fatigue/ Insomnia
Stress Perceived
Experiences
34411_CH09_265_304
9/9/05
10:49 AM
Page 275
Psychological
Domain
Other
Instrument
275
Description
Life Stressors and Social Resources Inventory (LISRES) (Impara & Plake, 1998)
The LISRES measures two aspects: life stressors (9 subscales) and social resources (7 subscales). Assessment occurs for 8 domains: physical health status, housing and neighborhood, finances, work, relationship with spouse or partner, relationships with children, relationships with extended family, and relationships with friends and social groups
Life Stressor ChecklistRevised (Wolfe & Kimerling, 1997)
Screen of life events that meet the definition of a trauma according to DSM-IV ; assesses occurrence, symptomatology, and functioning for 30 events
Illness Behavior Questionnaire (Pilowsky & Spence, 1983)
Assessment of adjustment/ responses to illness that are not functional, including hypochondriacal, denial, and changes in affect Instruments have been developed to assess a wide range of health-related issues based on the model of readiness by Prochaska and colleagues
Readiness to Change –Alcohol (Rollnick et al., 1992) –Drugs (Hodgins, 2001) –Smoking (McConnaughy et al., 1983)
Well-being Well-being has a wide range of meanings—from economic well-being to emotional well-being. Although some of these measures could be important for specific research, usually for outcomes research, general well-being measures are the focus. The use of well-being measures needs to be balanced against health-related quality of life (HRQoL) measures. HRQoL is a subset of well-being research. HRQoL is discussed in Chapter 5 and is focused on the relationship between health and quality of life (QoL), which
34411_CH09_265_304
276
9/9/05
CHAPTER 9
10:49 AM
Page 276
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
represents just a part of the general conception of well-being. Measures in this domain, although health influenced, do not specifically seek to measure a direct relationship between health status and well-being. The measures are general in nature and thus should only be used when general well-being is relevant; otherwise, measures of HRQoL should be used. Two commonly used measures of well-being are the General Well-Being Schedule (Fazio, 1977) and the Index of Well-Being (Campbell et al., 1976). Locus of Control Locus of control in health research has grown out of psychological work on mastery and efficacy (Rotter, 1966). Mainstream mastery and efficacy research tends to focus on control over life. A large number of scales have been developed that represent different conceptualizations of the concept of control. Although most share internal and external aspects (“my life is under my control,” “my life is controlled by the environment around me”), others include chance (see Internality, Powerful Others, and Chance [Levenson, 1981], Table 9–1) and environmental influence (the ability of individuals to change their social environment, see Spheres of Control Battery [Paulhus, 1983]) as part of the conceptualization. Instruments in this area, which emerged 30 years ago, have been used extensively in health research. These instruments and others have been useful in research as dependent variables (programs to empower patients), independent variables (to explain the success of intervention), and as controls (to evaluate the outcomes of an intervention controlling for the subjects’ beliefs about who or what is responsible for their health). The two most commonly utilized instruments are the Multidimensional Health Locus of Control Scale (MHLCS) (Wallston & Wallston, 1981) and the Mental Health Locus of Control (MHLC) (Hill & Bale, 1980) (see Table 9–1). The MHLCS is drawn primarily from conceptualizations of locus of control (Levenson, 1981). Pain Pain is probably one of the most frequently used measures, but least understood phenomenon in health research. It is common to everyone; very few individuals have not experienced pain, but the pain experience is not necessarily shared. Pain is not a simple construct, but a combination of physiological, psychological, and social forces.
34411_CH09_265_304
9/9/05
10:49 AM
Page 277
Psychological
277
The most commonly used measures of pain are simple visual analog scales (VASs), usually 10cm long or numeric rating scales (NRSs, e.g., 0–10) with the ends anchored with no pain and worst pain imaginable (McDowell & Newell, 1996). Although these scales are useful in assessing pain for treatment and even demonstrate utility in research, they are not without their drawbacks. Given the personal differences in the pain experience, the ability to compare pain scores among individuals or across populations is limited. Methods have been proposed by which responses to VASs or NRSs could be standardized to facilitate such comparisons, using relatively few items (Kane et al., 2002). Although these simple measures of pain are often used, if the research is focused specifically on pain or on pain within a specific population (e.g., cancer), then other measurement tools are used. A large number of diverse instruments have been developed to assess pain. The Brief Pain Inventory (Daut et al., 1983) is an intensive measure of pain that includes the location(s) of pain, its severity, and its quality. The McGill Pain Questionnaire focuses on the sensory nature of the pain (Turk et al., 1985; Turk & Melzack, 1992). Several instruments focus on pain and function, such as the MOS Pain Measure (Stewart & Ware, 1992) or instruments such as the Oswestry Low Back Pain Disability Questionnaire (Fairbank et al., 1980). These different instruments reflect different aspects of pain and the conceptualization of pain. For example, in orthopedics research, instruments that focus on the relationship between pain and function would represent a conceptual and operational definition that is most consistent with the research. Alternatively, research in fibromyalgia would probably be better served by combining several measures such as the Brief Pain Inventory, which focuses on severity and quality of the pain, and the McGill Pain Questionnaire, which would provide information regarding the sensory aspect of the pain experience. Stress The concept of stress is often unrecognized in outcomes research, perhaps because of the diversity in conceptualization and content. Stress can range from emotional, physiological, social, and even economic. Although there is diversity in the conceptualization of stress, most conceptualizations are based in either perception or experience. Perhaps no other concept illustrates the need for consistency between conceptual and operational measures. Some researchers had conceptually defined stress as a perceptual
34411_CH09_265_304
278
9/9/05
CHAPTER 9
10:49 AM
Page 278
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
phenomenon—the individuals’ perceptions of their ability to deal with stressful events—but these researchers had chosen what is probably one of the most commonly used stressful experiences questionnaires, the Social Readjustment Rating Scale (Holmes & Rahe, 1967). In this instance, there is a fatal mismatch between the conceptual and operational definitions of stress. The Perceived Stress Questionnaire (Cohen et al., 1983) instrument would be more consistent with their conceptual definition of stress. If stress is a focus of the research, it is often important to include measures of both aspects, perception of stress, and stressful experiences, as well as measures of resources that can be used to deal with stress (such as the Life Stressors and Social Resources Inventory [Impara & Plake, 1998]). Other Some other areas could be important in the study of mind-body. Individuals will often deny illness or inflate or fabricate illness (hypochondria) and such mind-body interactions can affect outcomes research. Instruments such as the Illness Behavior Questionnaire (Pilowsky & Spence, 1983) focus on nonfunctional adjustment or responses to illness such as those just identified. A final area that has received considerable research attention over the past decade is readiness to change. Primarily based on the work of Prochaska, “stages of change” research has been used not just in interventional research, such as alcohol or tobacco addiction research, but in general public health research as well (Prochaska et al., 1992; Prochaska & Norcross, 2001). A number of instruments have been developed to assess an individual’s readiness to change for a number of different conditions, from addiction (alcohol, tobacco, etc.) to health behaviors (exercise) (McConnaughy et al., 1983; Rollnick et al., 1992; Hodgins, 2001).
Affect This area of psychological factors has received the most attention in outcomes research. Research has focused on these issues as outcomes (dependent variables), factors that predict outcomes (independent variable), and items that need to be controlled for. The aforementioned construct validity becomes very important here. For example, although physiological manifestations are often associated with depression—such
34411_CH09_265_304
9/9/05
10:49 AM
Page 279
Psychological
279
as sleep, diet, alertness, and awareness—they may be in contradiction with one another; sleeping a lot as well as sleep disruptions are both considered signs of depression. Part of the reason for this apparent paradox is that researchers are dealing with a psychosocial construct that has diverse manifestations in all areas of life; physiological, social, and psychological. As with stress, there is a range of conceptual definitions and corresponding operational measures for most of these phenomena; thus, it is important to make sure that they are conceptually and operationally consistent. Most of the concepts discussed later are characterized as “negative emotions” and as a result are closely related to each other, creating an overlap between them in terms of conceptual definitions as well as empirical measures. These concepts are also closely related to some of the social constructs that will be discussed in the final section of the chapter. Therefore, one has to carefully ask not just why they should be included in the study, but if they are included, careful consideration must be made regarding the construct conceptually (what is it?), measurement (how to measure it?), and analytically. The first and most important measurement issue associated with this class of phenomena is that the conceptual and operational definitions of the variables be the same. (Measurement issues are discussed in greater detail in Chapter 4.) For example, a researcher may wish to include “depression” in a research project. One could conceptualize depression in terms of a diagnostic state, but at the measurement level, use screen tools such as the CESD or geriatric depression scale, thereby creating a mismatch between the conceptual and operational definitions of depression. If the conceptual usage of the term is to indicate a clinical diagnosis of depression, then screening tools for depression are not the appropriate measurement strategy. A range of meanings is associated with validity and reliability relative to the measurement of psychosocial factors, depending upon what is being measured. In the measurement of nonobservable, subjective phenomena (such as mood or affect) the central concern is construct validity (Nunnally & Bernstein, 1994; Campbell & Russo, 2001). Subjective phenomena are defined as constructs that have no directly observable manifestation and thus, as a result, they may or may not exist. Establishing construct validity is a cognitive, labor-, and time-intensive process directed at establishing that the construct (e.g., depression) exists and that it can be measured in a meaningful way. In outcomes research, one often takes for granted that the construct exists and focuses on other types of validity, such as predictive
34411_CH09_265_304
280
9/9/05
CHAPTER 9
10:49 AM
Page 280
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
or discriminant validity (Nunnally & Bernstein, 1994). Although this is an acceptable approach, it is important to note that these other types of validity are predicated upon construct validity being established (Streiner & Norman, 1995). The researcher should be familiar with the work that has established the validity of the construct. The other types of validity—predictive, discriminant, or content— although secondary to construct validity, are still important. In outcomes research, it is important that the measures used be capable of detecting meaningful change (Lieberson, 1985). The difficulty emerges in determining what is meaningful change: clinical, lifestyle, functional, and so forth. Returning to the earlier depression example, if the focus is on the use of serum serotonin reuptake inhibitors (SSRIs) in a population that manifests signs of depression but doesn’t necessarily have a formal diagnosis of depression, then the appropriate measurement approach to depression will depend on whether one wants to focus on measures of depressive affect, depressive symptoms, or depressive experiences that are most consistent with the conceptual definition of depression being used. It is not possible to cover the full range of issues that could be relevant to clinical outcomes research from this domain. The following material will focus on the primary concepts that have been either used in outcomes research or are potentially important for outcomes research in particular areas. Table 9–2 presents a brief list of some of the measurement instruments discussed. In outcomes research, depression and anxiety tend to be the most common and frequently used. Outcomes research has shown that anxiety can influence recovery from surgery and other medical procedures (independent variable). Alternatively, much of outcomes research is conducted on different ways of treating depression (dependent variable). In evaluating specific interventions, such as surgical versus medical treatment of a condition, it may be important to control for factors such as depression and anxiety. Depression In the postindustrial age, depression and sadness have become dominant states; as a result, depression is the most prevalent mental health problem in the United States (Shaver & Brennan, 1991). Clinically, depression is usually defined as a complex of symptoms generally characterized by dysphoria and loss of interest and pleasure in “things.” When looked at psychosocially, it also integrates into the conceptual definition issues associated with loneliness and social isolation.
34411_CH09_265_304
9/9/05
10:49 AM
Page 281
Psychological
281
Table 9–2 Psychological Measures Domain
Instrument
Description
Depression Screening
Other
Self-Rating Depression Scale Assesses affective, somatic, (Zung, 1965) psychomotor, and psychological aspects of depression in general population Center for Epidemiologic Studies Depression Scale (CESD) (Radloff, 1977)
Instrument emphasizes measurement of affective components of depression: mood, guilt, worthlessness, helplessness, as well as psychomotor, appetite, and sleep
Geriatric Depression Scale (Yesavage et al., 1982)
Instrument specifically for administration in geriatric populations in the clinical setting
Carroll Rating Scale (Carroll et al., 1981)
Based upon the work of Hamilton, this instrument was designed to assess severity of depression for 17 symptoms associated with depression
Depressive Experiences Questionnaire (Blatt et al., 1976)
Measures two aspects of depression: anaclitic (emotional dependence on others) and introjective (stringent standards for self, guilt, self-esteem, etc.)
Self-Rating Anxiety Scale (Zung, 1971)
A 20-item instrument that focuses on psychological as well as physiological factors associated with stress
Hamilton Anxiety Scale (HAMA) (Hamilton 1959; Bruss et al., 1994)
Developed to quantify the severity of anxiety symptomatology, often used in psychotropic drug evaluation; generally used for clinical administration
Anxiety
continues
34411_CH09_265_304
282
9/9/05
CHAPTER 9
10:49 AM
Page 282
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Table 9–2 continued Domain
Instrument
Description
State Trait Anxiety Measure (Spielberger, 1983)
The self-report inventory consists of 20 items to assess state anxiety, and another 20 items to assess trait anxiety; these two parts differ in the item wording and in the response format (intensity vs. frequency)
Endler Multidimensional Anxiety Scales (EMAS) (Endler et al., 1991)
The instrument has three modules: the EMAS-S, which assesses state anxiety, the EMAS-T, which assess trait anxiety, and the EMAS-P, which assesses perception of threat (what is causing the anxiety)
Affect Balance Scale (Bradburn, 1969)
Short instrument assessing reaction to events as well as state over the past 2 weeks; includes measures of positive and negative affect
Memorial University of Newfoundland Scale of Happiness (MUNSH). (Kozma & Stones, 1980)
Developed to assess general perspective of quality of life in the elderly, includes four subscales: positive affect, negative affect, general positive experience, and general negative experience
PGC Moral Scale (Lawton, 1975)
Developed to assess well-being in the elderly; instrument has 22 items and contains measures in three areas: dissatisfaction-loneliness, agitation, and attitudes toward one’s own aging
Other Measures
34411_CH09_265_304
9/9/05
10:49 AM
Page 283
Psychological
Domain
Instrument Mini-Mental State Exam (Folstein et al., 1975)
Short Portable Mental Status Questionnaire (Pfeiffer, 1975)
283
Description Instrument contains 11 items and assesses a range of issues, including orientation to time and place, recall, addition, ability to follow instruction, as well as motor skills Instrument adapted from the Mental Status Questionnaire for administration in the community setting as brief screen for cognitive function; mental status questionnaire has been demonstrated to be effective in institutional settings, but not in community settings; Short Portable Mental Status Questionnaire adapted from the Mental Status Questionnaire for administration in the community
Although there is some commonality around the meaning of depression, its manifestations and range are large, from suicidal ideation to sadness or feeling “blue.” Given such a large range, this concept presents many challenges when it comes to ensuring a match between the conceptual and operational levels and it can be difficult to identify appropriate measures. As noted, if one is dealing with diagnosing depression, then the DSM-IV (Diagnostic and Statistical Manual of Mental Disorders–Fourth Edition) (American Psychological Association, 1994) is the appropriate tool, but often in outcomes research, alternative measures such as the Self-Rating Depression Scale (Zung, 1965) or the Center for Epidemiologic Studies Depression Scale (CESD) (Radloff, 1977) are used. This schism can create a mismatch between measurement and either conceptual thought (diagnosis of depression is the concept) or analytical use (results are treated as being diagnostic). Depression is similar to stress in that it can be characterized as an internal state, through which the external world is viewed and interacted with,
34411_CH09_265_304
284
9/9/05
CHAPTER 9
10:49 AM
Page 284
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
or it can be characterized as a result of interaction with the world. Multiple tools have emerged to capture these differing orientations to depression. Depression defined as an internal state tends to dominate most medical and outcomes research. Instruments such as the Zung and the CESD have emerged as general population screeners for depressive affect and have been demonstrated to work effectively. A large number of specialized instruments, such as the Geriatric Depression Scale (Yesavage et al., 1982), have evolved to target the depressive affect in particular populations; a number of tools have emerged to assess depression in particular populations, for example, cancer, such as the Functional Assessment of Chronic Illness Inventory (Cella et al., 1993). Alternatively, some instruments have focused on environmental factors that contribute to depression. Generally, these instruments have been either directly adapted from a clinical tool, such as the adaptation of the Hamilton Rating Scale (Hamilton, 1960) for nonclinical administration (Carroll Rating Scale (Carroll et al., 1981)), or based more on psychosocial theory, such as the Depressive Experiences Questionnaire (Blatt et al., 1976). Regardless of which approach to measurement is used, it is important that an instrument is chosen carefully and that the content of the instrument reflects the conceptual domain as well as the research. For example, if the conceptual definition of depression for a research project is characterized by mood as well as withdrawal, then an instrument that emphasizes loneliness as well as mood should be used. Anxiety The distinction between depression and anxiety is becoming somewhat blurred (Hamilton, 1983). Part of this blurriness results from the transition from viewing anxiety as primarily an aspect of social interaction to seeing anxiety as an affective state or trait of an individual. Even more than depression, anxiety has a range of conceptual and operational definitions in outcomes research, from situational and well defined—“white-coat anxiety”—to a vague state experienced in anticipation of some ill-defined pending doom, to one that is based in social phenomena characterized by perceived danger and powerlessness. As a result, a wide range of descriptors is commonly utilized in measuring aspects of anxiety: “How often do you feel apprehensive?” “How often do you find that you can’t stop worrying?” Thus, despite being a widely used term, anxiety is rarely clearly
34411_CH09_265_304
9/9/05
10:49 AM
Page 285
Psychological
285
defined. However, such a clear definition is necessary prior to selecting how to measure the construct. The vagueness of the construct leads to diverse approaches to measuring it. Most of the early measures focused on adjective checklists, such as the Anxiety Adjective Checklist (Zuckerman & Lubin, 1965). Later measures started to include a more diverse set of measures, including physiological conditions and social isolation (Zung’s Self-Rating Anxiety Scale [Zung, 1971], Hamilton Anxiety Scale [Hamilton, 1959; Bruss et al., 1994]). Later work has attempted to refine the conceptualization of anxiety as well as differentiate state from trait anxiety. This work is reflected in the instruments such as the State-Trait Anxiety Inventory (STAI) (Spielberger, 1983), which was initially conceptualized as a research instrument to study anxiety in adults. It is a self-report assessment device that includes separate measures of state and trait anxiety. State anxiety is characterized as a transitory response in which the individual experiences anxiety, apprehension, or tension. It can manifest both emotionally as well as physiologically, but is regarded as being short-term. Alternatively, trait anxiety is identified as a persistent and stable anxiety response to the environmental factors. The Endler Multidimensional Anxiety Scales (EMAS) (Endler et al., 1991) assess state and trait anxiety as well, but also include a measure regarding the perception of what is causing anxiety. Given the overlap that often occurs when dealing with psychological phenomena such as depression and anxiety, an alternative is to utilize more general instruments to assess psychological states. A number of instruments, such as the Affect Balance Scale (Bradburn, 1969) and the Multiple Affect Adjective Checklist (Zuckerman & Lubin, 1965), provide multidimensional assessments. Although they lack some of the specificity of depression or anxiety scales, they represent an integrative approach to the assessment of affect. Such an approach has advantages in that it recognizes the similarity between constructs and attempts to look at affect generally, without attempting to make direct attributions. Although measures associated with positive emotions are rarely used in health research, there is no reason that they cannot be adopted for use. They can extend the range of affect measurement beyond simply the absence of negative feelings. Instruments such as the Rosenburg Self-Esteem Scale (Rosenburg, 1965), the Memorial University of Newfoundland Scale of Happiness (Kozma & Stones, 1980), or the Affectometer 2 (Kammann & Flett, 1983) can be used to assess happiness and related positive emotions.
34411_CH09_265_304
286
9/9/05
CHAPTER 9
10:49 AM
Page 286
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
When affect is considered in research, anxiety and depression tend to dominate conceptual definitions and measurement tools. It is important to note that anxiety and depression are multifaceted and their meanings are fluid. Alternative constructs, such as morale, can be useful in outcomes research. In aging research, an instrument such as the PGC Morale Scale (Lawton, 1975) utilizes measures of depression, anxiety, usefulness, and loneliness, as well as other issues to develop three scores: agitation, attitude toward one’s own aging, and lonely dissatisfaction.
Cognitive Function Measures of cognitive function are not intended to assess intelligence; rather they are focused on assessing basic issues associated with cognition. These measures serve a variety of purposes, ranging from an outcome to a screening tool used as part of the inclusion criteria for a study. For example, a recent study used a battery of cognitive function measures to assess the impact of dialysis on the cognitive abilities of patients undergoing treatment. This research demonstrated a significant change in cognitive ability not just during the dialysis process, but also in the time between treatments. At the simplest level, basic measures of orientation are often used—does a person know who he or she is, does the person know what time it is, and does the person know where he or she is. Although such measures are relatively blunt, if a person is not oriented to time and place, then cognitive function is impaired. Alternatively, if the person is oriented to person, time, and place, it is not a given that cognitive function is not impaired. A number of tests have evolved as screens for cognitive function. The minimental state exam (MMSE) (Folstein et al., 1975) assesses recall, addition, and the ability to follow instruction, as well as motor skills. Other instruments such as the Mental Status Questionnaire (Kahn et al., 1984) have proven effective in institutional settings, but not in community settings, and the Short Portable Mental Status Questionnaire (Pfeiffer, 1975) was adapted from the Mental Status Questionnaire for administration in the community.
SOCIAL Individuals are not blank slates, immune to the effects of history and social/environmental context. How individuals see themselves, as well as
34411_CH09_265_304
9/9/05
10:49 AM
Page 287
Social
287
their perception and integration of the social environment, can greatly affect behavior. Any research that even peripherally involves or requires a human component should include consideration of social context. The inclusion of variables associated with social context requires the researcher to have an idea about how the world works, or more specifically, an idea about how the small piece of the world under examination works (Lieberson, 1985). The selection of variables for inclusion should be guided by this conception. There are numerous social/psychosocial perspectives that can provide a framework for understanding individual beliefs, perceptions, and behavior. Consideration of the social context is most commonly done at the macro level, such as socioeconomic status (SES) or measures of organizational structure, but appear less frequently when the unit of analysis is the individual, dyad, or small group. By understanding the dynamic of social interaction at the interpersonal level, other factors such as SES may become more salient and germane or their irrelevance demonstrated. A useful perspective to study social phenomena is a basic understanding of roles. Roles are particularly important when interaction is between nonequivalent actors (e.g., patient–physician); the character of the physician– patient relationship calls into question patient role adoption. From the patients’ perspective, they may adhere to the traditional sick role as described by Parsons (1975). This role feeds into what has historically been the dominant model of provider–patient interaction, the paternal model. For example, in Western society, women have been socially and medically cultured to the belief that good and responsible parenting begins with prenatal care (Lippman, 1991; Madlon-Kay et al., 1992; Anderson, 1999). However, with the increased emphasis in the last decade on consumerism in health care and corresponding expectations that patients become more actively responsible for their own health care, the patient role identity may be one that obligates patients to be more actively involved in decision making with their care than the Parsonian model allows—and from the provider perspective, how providers view their role as medical doctors, perhaps as the holders of technical knowledge or as the conveyors of information, must be considered. Just as the patient expectation of the doctor role can affect patient behavior, the doctoral expectation of patient role, which is determined by medical doctors’ role identity, will affect their interaction with patients. Therefore, the delivery of care can vary at the individual level, regardless of the elaborate controls used to present a common clinical encounter among all patients.
34411_CH09_265_304
288
9/9/05
CHAPTER 9
10:49 AM
Page 288
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
The health care environment can be intimidating. Patients often defer to their physician for determination of appropriate medical care. For example, observation of two different obstetricians, within the same practice, with similarly situated patients resulted in very different outcomes. When discussing the option of a particular prenatal screening, one doctor told his patients that 90 percent of patients have the screening test. As a result, almost all of that doctor’s patients agreed to have the screening test. Alternatively, another doctor told his patients that only 50 percent had the screening test, and as a result only about 50 percent of that doctor’s patients agreed to have the screening test. This differing rate reflected differences in each of the doctors’ beliefs regarding their role as holder of technical knowledge and their obligation to direct clinical decision making. A paternalistic model of patient–provider interaction has dominated the practice of medicine until fairly recently. The past 30 years have brought a shift away from the paternalistic model to a consumer-oriented model of patient self-determination. One expects to see a range of behavior and role enactments, falling on the continuum between the paternalistic model and the self-determined model. Role theory entails a consideration of norms. A norm is a belief about the acceptability of behavior. It is an evaluative criterion that specifies a rule for behavior. Norms are based in cognitive beliefs of approval or disapproval. They tend to persist over time, although they can change (Michener & DeLamater, 1994; Borgatta & Montgomery, 2000). They influence behavior from the individual to societal levels. The expectations that serve as the evaluative standard for norms are largely socially determined. It is a comparison level learned from others who the actor takes as referents (Michener & DeLamater, 1994). Through socialization, the individual becomes aware of and internalizes norms, which then become internal drivers of behavior. Internalized norms reflect the society and subgroups to which the individual is exposed or a part of. Internal norms have considerable influence on individual behavior; they govern and constrain choices the individual makes. As such, factors such as ethnicity and culture may become relevant. Primary socialization occurs throughout an individual’s formative years. Therefore, it is reasonable to expect cultural norms that are closely linked to ethnicity have a strong effect on the development of individual norms and consequently role identity across various social environments. Both of these contribute to the considerations based in the social construction of reality. The social construction of reality somewhat springs from the interactionist perspective. At the risk of oversimplification, the
34411_CH09_265_304
9/9/05
10:49 AM
Page 289
Social
289
social construction of reality holds that the social world is perceived by the individual as fact and reality and includes widely accepted “institutionalized” behavior, codes of conduct, and appropriate belief systems and behaviors (Berger & Luckmann, 1990). Through the socialization process, a reality of everyday life (REL) is constructed. Included in the REL are appropriate standards of behavior for oneself as well as others, particularly with respect to those occupying acknowledged roles within one’s REL. This model, although complex, is becoming increasingly important with the relatively new emphasis that is being placed on health disparities research. Research in this area emphasizes the alternative realities of everyday life that exist between different cultural and ethnic groups. With this limited understanding of the social construction of reality, it is still evident that social context, both proximal and distal, can be a good predictor of individual attitude and behavior. What is also evident is that the social context includes not just consideration of the environment in which the intervention may occur, but appraisal of the REL and the intersubjective understanding between the various actors. Too often it is assumed that the RELs of the patient and the provider are the same.
Social Economic Status SES has been demonstrated to be related to a spectrum of health outcomes and diseases, from outcomes associated with cardiovascular disease (Kaplan & Keil, 1993) to dementia (Cobb et al., 1995). Measures of SES range from measures that utilize relatively few variables, such as residence, income, occupation, and education, to recently developed measures that focus on social capital (Oakes & Rossi, 2003; see Table 9–3). Historically, measures of SES, such as the indexes developed by Hollingshead (Hollingshead, 1957; Hollingshead & Redlich, 1958), Duncan (1961), or Nam-Powers (Nam & Terrie, 1986), have dominated the measurement of SES. Both Duncan’s and Nam-Powers’s indexes use three variables: income, education, and occupation. The instruments vary in terms of the weight assigned to each of the three different variables. Both draw on the U.S. Census Bureau’s coding system for occupation and as a result, if the methodology is used, it is essential to use the most recent coding for occupation. In comparison, the Hollingshead index uses two variables income and a relatively simple coding of occupation (seven levels). What is often ignored in the usage of these indexes is the original intent behind the development of the measures; they were developed as measures of social stratification and
34411_CH09_265_304
290
9/9/05
CHAPTER 9
10:49 AM
Page 290
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Table 9-3 Social Measures Domain
Instrument
Description
SES Indexes
Hollingshead Index of Social Position (Hollingshead, 1957; Hollingshead & Redlich, 1958)
SES measure based originally on three factors: residence (location), occupation, and education; given the complexity of coding residence, the twofactor model in which residence is omitted is traditionally used
Duncan Socioeconomic Index (Duncan, 1961)
SES measure based on three factors: occupation, income, and education; note that the coding of occupation is complex and revised frequently (see http://webapp.icpsr. umich.edu/GSS/rnd1998/ appendix/apdx_g.htm http://www.ipums.umn.edu/ usa/pwork/seia.html)
Nam-Powers Socioeconomic SES measure based on three Score (Nam & Terrie, 1986) factors: education, occupation, and income (see www. ssc.wisc.edu/cde/cdewp/ 96-10.pdf for methodology) Social Capital
Capital SES (CAPSES) (Oakes & Rossi, 2003)
A relatively new measure; a composite of material capital (income, owned materials, investments, expected wealth such as inheritance), human capital (fixed endowments— athleticism, beauty, innate cognitive skill, education), and social capital (have relationships with others, have support)
34411_CH09_265_304
9/9/05
10:49 AM
Page 291
Social
Social Support
Social Functioning/ Adjustment
291
The Social Support Questionnaire (Sarason et al., 1983)
Instrument asks respondents to indicated if they (1) have someone they can turn to for support for 27 items and (2) how satisfied they are with that support
Medical Outcomes Study Social Support Survey (Sherbourne & Stewart, 1991)
Instrument is focused on the measurement of functional social support and resources; has four subscales: tangible support, affectionate support, positive social interaction, and emotional or informational support
Duke-UNC Functional Social Support Survey (Broadhead et al., 1988)
Intended as a measure of satisfaction with functional and emotional social support; assesses satisfaction in eight areas
Duke Social Support and Stress Scale (Parkerson et al., 1989)
Instrument provides a rating for support received from and the amount of stress caused by family and nonfamily sources
Social Functioning Schedule (Remington & Tyrer, 1979)
Instrument assesses function in 12 areas: employment, household chores, contribution to household, money, self-care, material relationships, care of children, patient–child relationships, patient–parent relationships, household relationships, hobbies, and spare time activities; is conducted as a semistructured interview and is a large instrument
Social Adjustment Schedule (Weissman & Bothwell, 1976)
Intended to assess the quality of social relationships and role performance in a number of areas: work (outside and inside the home), spare time, family, and financial; for each of the areas, five aspects are assessed continues
34411_CH09_265_304
292
9/9/05
CHAPTER 9
10:49 AM
Page 292
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Table 9–3 continued Domain
Instrument
Description
Social Maladjustment Scale (Clare & Cairns, 1978)
Scale assesses three issues: material condition, social management, and satisfaction; for six different areas: housing, occupation, economic situation, leisure/social activities, family and domestic relationships, and marriage
Social Dysfunction Scale (Linn et al., 1969)
Instrument designed to assess function in three areas: selfesteem, interpersonal, and performance; primarily focuses on negative aspects
Complex Organizations Commitment
Organizational Commitment Questionnaire (Mowday et al., 1979)
A 15-item instrument to assess global organizational commitment; has also been modified to assess professional commitment or commitment to job as well (Gunz & Gunz, 1994; Millward & Hopkins, 1998)
Affective, Normative, and Continuance Commitment (Meyer & Allen, 1997)
Instrument is designed to assess three aspects of commitment to an organization: effect (sense of belonging, attachment), normative (loyalty, job tenure), and continuance; continuance scale has two subscales to assess: (1) level of sacrifice the individual is willing to make to remain in the organization and (2) alternative work possibilities
Work Control (Dwyer & Ganster, 1991)
Designed to assess a worker’s perception of control over work environment, including control over performance,
Work Control
34411_CH09_265_304
9/9/05
10:49 AM
Page 293
Social
Domain
Instrument
293
Description timing, and procedures associated with work tasks
Job Routinization and Formalization (Bacharach et al., 1990)
Instrument is designed to assess aspects of conflict between employee preference and organizational needs; instrument has measures in four areas: routinization, pervasiveness of rules, record keeping, and formalization
Role Overload (Dwyer et al., Intended to measure inconsisand Bacharach et al., 1990) tencies between activities and tasks demanded and the time and resources available for completing the task Job Overload (Caplan et al., 1980)
11-item instrument designed to measure quantitative aspects of pace and amount of work
Job Interdepdendence (Pearce & Gregersen, 1991)
Two dimensions assessed in this measure: interdependence and independence; interdependence measures focus on reciprocal relationships and independence measures address autonomy
Inventory of Stressful Events (Motowidlo et al., 1986)
Instrument developed to assess stressful events for nurses; measures composed of 45 items that focus on the stress associated with events associated with the provision of nursing care
Frustration with Work (Peters et al., 1980)
Short three-item instrument assessing frustration with job/work
Job Stress Scale (Parker & Decotiis, 1983)
Instrument assesses job stress for two dimensions: time and anxiety; measures of time stress include measures related to work as well as impact of work on nonwork life; anxiety measures focus solely on anxiety due to job
34411_CH09_265_304
294
9/9/05
CHAPTER 9
10:49 AM
Page 294
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
social class. The measures are influenced by the work of Weber and colleagues (1978) and Marx and colleagues (1906) and reflect a notion of class that dominated social thought from the late 1800s to the mid-1900s. Recent studies in public health have demonstrated the importance of moving away from such classical notions by noting the importance of economic variables in the study of health and SES (Daly et al., 2002). Alternatively, new concepts such as social capital have emerged, which may prove to be more useful in the study of outcomes than the traditional concept of SES.
Social Capital Measures of social capital originally focused on nation-state or community measures (what is the social capital of a community). Research has demonstrated a relationship to health and health outcomes when looking at the nation-state and community levels (Evans et al., 1994; Wilkinson, 1996). However, the measurement of social capital at the community level has proved to be a difficult task and the measure of capital seems to be specific to each study and generalizing the findings is difficult. Each researcher has developed individual methods for measuring community social capital, and although there is some overlap, it is largely a measurement area in which there is little standardization. Attempts have been made to develop individual level measures of social capital, such as the capital SES (CAPSES) (Oakes & Rossi, 2003). The CAPSES attempts to measure capital in three areas: material capital (income, wealth, etc.), human (the individual, abilities, etc.), and social (relationships and support). The content of the CAPSES clearly demarcates it as a measure of social capital as opposed to SES (social class). As such, it has several advantages for outcomes research. It is focused on the situation of the individual as opposed to traditional measures of SES in which social class is used to inform inferences about an individual. Additionally, it incorporates measures of social relationships and support, which are often important measures in outcomes research to assess informal care.
Social Support As with many of the psychological measures, there is a diverse number of conceptual definitions of social support, social function, and related
34411_CH09_265_304
9/9/05
10:49 AM
Page 295
Social
295
measures. The assessment of social support in outcomes research can range from very simple items, such as “Is there anyone who can help you change your bandages at home?” to more complex measures that attempt to identify who provides what type of support and how satisfied the individual is with that support. Three different measures of social support are presented in Table 9–3. The social support questionnaire asks the respondent if he or she has someone to rely on for support; it uses 27 different items and also asks the person to rate his or her satisfaction with the support provided by that person (Sarason et al., 1983). The Medical Outcomes Study Social Support Survey measures functional social support and resources (Sarason et al., 1983). The measurement of social support and resources is broken into four subscales assessing tangible, affectionate, positive social interaction, and emotional or informational support. The Duke-UNC Functional Social Support Survey is primarily intended as a measure of satisfaction with support in eight social and emotional areas (Broadhead et al., 1988). Finally, the Duke Social Support and Stress Scale combines an assessment of both the amount of support received from as well as stress caused by family and nonfamily sources.
Social Function/Adjustment Measures of social support focus on the dynamic relationship between the individual and the environment whereas measures of social function/ adjustment focus on assessing the individual’s status and function. As such, they are distinct from measures of social support and should not be confused with social support. Instruments assessing social function tend to be long due to the complexity of the phenomena. The Social Functioning Schedule (Remington & Tyrer, 1979) assesses function in 12 areas, including work, home/family, and hobbies. Within each area, a number of aspects are asked, including behavioral (frequency of event) as well as psychological (stress associated with activity). It is a good instrument to assess social function but is a long, semistructured interview (45 minutes). Measures of social adjustment tend to either take a positive (adjustment) or negative (maladjustment) approach. As with the measures of affect, in theory they represent the “poles” on the continuum, but in reality, measurement is not as advanced as the theory and there is significant overlap between “adjustment” and “maladjustment” scales. Most of the instruments to assess social
34411_CH09_265_304
296
9/9/05
CHAPTER 9
10:49 AM
Page 296
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
adjustment originate in work associated with drug/criminal rehabilitation or neurosis. In Table 9–3, three instruments are briefly reviewed. The Social Adjustment Schedule (Weissman & Bothwell, 1976) focuses on a range of social relationships, including community, work, and home for each area in which the five aspects are assessed. The Social Maladjustment Scale (Clare & Cairns, 1978) focuses on six areas ranging from occupation to housing and marriage. For each area, the material conditions, the individual’s ability to manage it, and overall satisfaction are assessed.
Complex Organization The final areas of social factors traditionally assessed in outcomes research are aspects of formal organization. These measures are not employed as often as they should be, especially as controls associated with multicenter studies. A number of issues are associated with outcomes research and complex organizations. For example, in a multicenter study or in studies that occur in large institutions (hospitals, large clinics), assessing differences in individual or group commitment to organization, career, or profession could be important, particularly if the intervention is organizationally based (e.g., guidelines or pathways). Many instruments exist to assess organizational commitment. The Organizational Commitment Questionnaire was originally developed to assess organizational commitment (Mowday et al., 1979). It has been adapted to assess professional and job commitment (Gunz & Gunz, 1994; Millward & Hopkins, 1998). Other instruments focus on commitment to supervisors (Becker et al., 1996) or assess organizational commitment across multiple concepts, such as the Affective, Normative, and Continuance Commitment instrument (Allen & Meyer, 1990). For studies in which the activity associated with care is changed through changing work roles, instruments that assess job characteristics would be relevant, for example, control over work (Work Control; Dwyer & Ganster, 1991); job routinization and formalization; role overload (Bacharach et al., 1990); overload (Job Overload; Caplan et al., 1980); interdependence (Job Interdependence; Pearce & Gregersen, 1991); and stress (Frustration with Work; Peters et al., 1980 and Job Stress Scale; Parker and Decotiis 1983).
SUMMARY Psychosocial and demographic variables can be useful in conducting outcomes research, but their use comes with a price. The conceptualization of
34411_CH09_265_304
9/9/05
10:49 AM
Page 297
References
297
psychosocial phenomena and demographic variables can be complex. Even the seemingly simple phenomenon of male or female can create problems: Is it reproductive physiology or gender role (masculinity versus femininity) that is being conceptualized? The problem is further compounded when measurement (operationalization) is added to the task. Measuring subjective phenomena is fraught with error and the need to constantly monitor the relationship between conceptual and operational definitions is a requirement for the effective use of psychosocial factors in outcomes research. Although it may seem that a large number of instruments were reviewed in this chapter, many more measures have been created to assess psychosocial phenomena. A number of excellent resources are available to researchers when looking for measures in this area. Some focus primarily on health (McDowell & Newell, 1996); others are more general in nature (Robinson et al., 1991). Many resources can be found on the Internet. When using any instrument, it is important that regardless of the source (print or electronic), the original source material be read. Failing to look at the original intent of the instrument can lead to a serious problem in which the conceptual model behind the development of the instrument is not consistent with the conceptual definition being used in the research. As with all factors in research, it is crucial to ask basic questions prior to the inclusion of psychosocial factors. Why should it be included? Is there any reason to believe that “it” has a relationship between or to intervention, treatment, and outcome? Will it be used as a dependent variable, independent variable, or control? Answering these questions guides the process of determining whether or not to include “it” in the research. If it is included, then researchers must begin the task of ensuring that how one thinks about “it” is consistent with how “it” is measured.
REFERENCES Allen, N.J., & Meyer, J.P. (1990). The measurement of antecedents of affective, continuance, and normative committment to the organization. Journal of Occupational and Organizational Psychology, 63, 1–18. American Psychological Association. (1994). Diagnostic and statistical manual of mental disorders—DSM-IV. Washington, DC: American Psychological Association. Anderson, G. (1999). Nondirectiveness in prenatal genetics: Patients read between the lines. Nursing Ethics, 6(2), 126–136. Bacharach, S.B., Bamberger, P., et al. (1990). Work processes, role conflict, and role overload: The case of nurses and engineers in the public sector. Work and Occupations, 17, 199–228. Becker, T.E., Billings, R.S., et al. (1996). Foci and bases of employee committment: Implications for job performance. Academy of Management Journal, 39, 464–482.
34411_CH09_265_304
298
9/9/05
CHAPTER 9
10:49 AM
Page 298
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Bennett, P.H. (1999). Type 2 diabetes among the Pima Indians of Arizona: An epidemic attributable to environmental change? Nutrition Reviews, 57(5 Pt 2), S51–54. Berger, P.L., & Luckmann, T. (1990). The social construction of reality: A treatise in the sociology of knowledge. New York: Anchor Books. Blalock, H.M. (1974). Measurement in the social sciences: Theories and strategies. Chicago: Aldine. Blatt, S.J., D’Afflitti, J.P., et al. (1976). Depressive experiences questionnaire. New Haven, CT: Yale University Press. Bohrnstedt, G.W., & Borgatta, E.F. (1981). Social measurement: Current issues. Beverly Hills, CA: Sage Publications. Borgatta, E.F., & Montgomery, R.J.V. (2000). Encyclopedia of sociology. New York: Macmillan Reference USA. Bradburn, N.M. (1969). The structure of psychological well-being. Chicago: Aldine. Broadhead, W.E., Gehlbach, S.H., et al. (1988). The Duke-UNC Functional Social Support Questionnaire. Measurement of social support in family medicine patients. Medical Care, 26(7), 709–723. Bruss, G.S., Gruenberg, A.M., et al. (1994). Hamilton Anxiety Rating Scale Interview guide: Joint interview and test-retest methods for interrater reliability. Psychiatry Research, 53(2), 191–202. Campbell, A., Converse, P.E., et al. (1976). The quality of American life: Perceptions, evaluations, and satisfactions. New York: Russell Sage Foundation. Campbell, D.T., & Overman, E.S. (1988). Methodology and epistemology for social science: Selected papers. Chicago: University of Chicago Press. Campbell, D.T., & Russo, M.J. (2001). Social measurement. Thousand Oaks, CA: Sage Publications. Caplan, R.D., Cobb, S., et al. (1980). Job demands and worker health. Ann Arbor: University of Michigan, Institute for Social Research. Carroll, B.J., Feinberg, M., et al. (1981). The Carroll rating scale for depression. I. Development, reliability and validation. British Journal of Psychiatry, 138, 194–200. Cella, D.F., Tulsky, D.S., et al. (1993). The Functional Assessment of Cancer Therapy scale: Development and validation of the general measure. Journal of Clinical Oncology, 11(3), 570–579. Clancy, C., Stryer, D., et al. (2002). From publication to public action: Agency for Healthcare Research and Quality (AHRQ) perspectives on ethnicity and race-related outcomes research. Ethnicity & Health, 7(4), 287–290. Clare, A.W., & Cairns, V.E. (1978). Design, development and use of a standardized interview to assess social maladjustment and dysfunction in community studies. Psychological Medicine, 8(4), 589–604. Cleeland, C.S. (1991). Pain assessement in cancer. In D. Osoba (Ed.), Effect of cancer on quality of life (pp. 293–305). Boca Raton, FL: CRC Press. Cobb, J.L., Wolf, P.A., et al. (1995). The effect of education on the incidence of dementia and Alzheimer’s disease in the Framingham study. Neurology, 45(9), 1707–1712.
34411_CH09_265_304
9/9/05
10:49 AM
Page 299
References
299
Cohen, S., Kamarck, T., et al. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385–396. Daly, M.C., Duncan, G.J., et al. (2002). Optimal indicators of socioeconomic status for health research. American Journal of Public Health, 92(7), 1151–1157. Daut, R.L., Cleeland, C.S., et al. (1983). Development of the Wisconsin Brief Pain Questionnaire to assess pain in cancer and other diseases. Pain, 17, 197–210. Diez Roux, A.V. (2003). Residential environments and cardiovascular risk. Journal of Urban Health, 80, 569–589. Douglas, C.H., & Douglas, M.R. (2004). Patient-friendly hospital environments: Exploring the patients’ perspective. Health Expectations, 7(1), 61–73. Duncan, O.D. (1961). A socioeconomic index for all occupations. In A.J. Reiss, O.D. Duncan, P.K. Hatt and C.C. North (Eds.), Occupations and Social Status. Glencoe, IL: The Free Press. Dwyer, D.J., & Ganster, D.C. (1991). The effects of job demands and control on employee attendance and satisfaction. Journal of Organizational Behavior, 12, 595–608. Endler, N.S., Edwards, J.M., et al. (1991). Endler Multidimensional Anxiety Scales (EMAS): Manual. Los Angeles: Western Psychological Services. Evans, R.G., Barer, M.L., et al. (1994). Why are some people healthy and others not? The determinants of health of populations. New York: Aldine de Gruyter. Fairbank, J.C., Couper, J., et al. (1980). The Oswestry low back pain disability questionnaire. Physiotherapy, 66(8), 271–273. Fazio, A.F. (1977). A concurrent validational study of the NCHS general well-being schedule. Hyattsville, MD: U.S. Dept. of Health Education and Welfare Public Health Service, Health Resources Administration National Center for Health Statistics. Folstein, M.F., Folstein, S.E., et al. (1975). Mini-mental state. A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3), 189–198. Gunz, H.P., & Gunz, S.P. (1994). Professional/organizational commitment and job satisfaction for employed lawyers. Human Relations, 47, 801–828. Hamilton, M. (1959). The assessment of anxiety scales by rating. British Journal of Psychology, 32, 50. Hamilton, M. (1983). The clinical distinction between anxiety and depression. British Journal of Clinical Pharmacology, 15(Suppl 2), 165S–169S. Hamilton, M.A. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry, 23, 56–62. Hill, D.J., & Bale, R.M. (1980). Development of the mental health locus of control and mental health locus of origin scales. Journal of Personality Assessment, 44, 148–156. Hodgins, D.C. (2001). Stage of change assessments in alcohol problems: Agreement across self and clinician reports. Substance Abuse, 22, 870–896. Hollingshead, A.B. (1957). Two factor index of social position. New Haven, CT: Privately printed. Hollingshead, A.B., & Redlich, F.C. (1958). Social class and mental illness. New York: John Wiley.
34411_CH09_265_304
300
9/9/05
CHAPTER 9
10:49 AM
Page 300
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Holmes, T.H., & Rahe, R.H. (1967). The Social Readjustment Rating Scale. Journal of Psychosomatic Research, 11(2), 213–218. Impara, J.C., & Plake, B.S. (Eds.). (1998). The thirteenth mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurements. Imperatore, G., Knowler, W.C., et al. (2001). Genetics of diabetic nephropathy in the Pima Indians. Current Diabetes Reports, 1(3), 275–281. Jackson, R.J. (2003). The impact of the built environment on health: An emerging field. American Journal of Public Health, 93(9), 1382–1384. Kahn, R.L., Goldfarb, A.I., et al. (1984). Mental Status Questionnaire. In D.J. Mangen & W.A. Peterson (Eds.), Research instruments in social gerontology: Vol. 3. Health, program evaluation, and demography. Kammann, R., & Flett, R. (1983). Affectometer 2: A scale to measure current level of general happiness. Australian Journal of Psychology, 35, 259–265. Kane, R.L., Bershadsky, B., et al. (2002). Efforts to standardize the reporting of pain. Journal of Clinical Epidemiology, 55(2), 105–110. Kaplan, G.A., & Keil, J.E. (1993). Socioeconomic factors and cardiovascular disease: A review of the literature. Circulation, 88(4 Pt 1), 1973–1998. Kozma, A., & Stones, M.J. (1980). The measurement of happiness: Development of the Memorial University of Newfoundland Scale of Happiness (MUNSH). Journal of Gerontology, 35, 906–912. Lawton, M.P. (1975). The Philadelphia Geriatric Center Moral Scale: A revision. Journal of Gerontology, 30, 85–89. Leavitt, F., & Garron, D.C. (1979a). The detection of psychological disturbance in patients with low back pain. Journal of Psychosomatic Research, 23(2), 149–154. Leavitt, F., & Garron, D.C. (1979b). Validity of a Back Pain Classification Scale among patients with low back pain not associated with demonstrable organic disease. Journal of Psychosomatic Research, 23(5), 301–306. Levenson, H. (1981). Differentiating among internality, powerful others, and chance. In H.M. Lefcourt (Ed.), Research with the locus of control construct (pp. 15–63). New York: Academic Press. Levenstein, S., Prantera, C., et al. (1993). Development of the Perceived Stress Questionnaire: A new tool for psychosomatic research. Journal of Psychosomatic Research, 37(1), 19–32. Lieberson, S. (1985). Making it count: The improvement of social research and theory. Berkeley: University of California Press. Lillioja, S., & Bogardus, C. (1988). Insulin resistance in Pima Indians. A combined effect of genetic predisposition and obesity-related skeletal muscle cell hypertrophy. Acta Medica Scandinavica - Supplementum, 723, 103–119. Linn, M.W., Sculthorpe, W.B., et al. (1969). A social dysfunction rating scale. Journal of Psychiatric Research, 6(4), 299–306. Lippman, A. (1991). Prenatal genetic testing and screening: Constructing needs and reinforcing inequities. American Journal of Law & Medicine, 17(1–2), 15–50. Lucy, W.H. (2003). Mortality risk associated with leaving home: Recognizing the relevance of the built environment. American Journal of Public Health, 93(9), 1564–1569.
34411_CH09_265_304
9/9/05
10:49 AM
Page 301
References
301
Madlon-Kay, D.J., Reif, C., et al. (1992). Maternal serum alpha-fetoprotein testing: Physician experience and attitudes and their influence on patient acceptance. Journal of Family Practice, 35(4), 395–400. Marx, K., Moore, S., et al. (1906). Capital: A critique of political economy. Chicago: C.H. Kerr & Company. McConnaughy, E.N., Prochaska, J.O., et al. (1983). Stages of change in psychotherapy: Measurement and sample profiles. Psychotherapy: Theory, Research and Practice, 20, 368–375. McDowell, I., & Newell, C. (1996). Measuring health: A guide to rating scales and questionnaires. New York: Oxford University Press. Meyer, J.P., & Allen, N.J. (1997). Commitment in the workplace. Thousand Oaks, CA: Sage Publications. Michener, H.A., & DeLamater, J.D. (1994). Social psychology. Fort Worth, TX: Harcourt Brace College Publishers. Millward, L.J., & Hopkins, L.J. (1998). Psychological contracts and job commitment. Journal of Applied Social Psychology, 28, 1530–1556. Moseley, J.B., O’Malley K., et al. (2002). A controlled trial of arthroscopic surgery for osteoarthritis of the knee. New England Journal of Medicine, 347(2), 81–88. Motowidlo, S.J., Packard, J.S., et al. (1986). Occupational stress: Its causes and consequences for job performance. Journal of Applied Psychology, 71, 618–629. Mowday, R.T., Steers, R.M., et al. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224–247. Nam, C.B., & Terrie, E.W. (1986). Comparing the Nam-Powers and Duncan SEI occupational scores. Tallahassee: Florida State University, Center for the Study of Population. Northridge, M.E., Sclar, E.D., et al. (2003). Sorting out the connections between the built environment and health: A conceptual framework for navigating pathways and planning healthy cities. Journal of Urban Health, 80, 556–568. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory. New York: McGraw-Hill. Oakes, J.M., & Rossi, P.H. (2003). The measurement of SES in health research: Current practice and steps toward a new approach. Social Science & Medicine, 56(4), 769–784. Parker, D.F., & Decotiis, TA. (1983). Organizational determinants of job stress. Organizational Behavior and Human Performance, 32, 160–177. Parkerson, G.R., Jr., Michener, J.L., et al. (1989). Associations among family support, family stress, and personal functional health status. Journal of Clinical Epidemiology, 42(3), 217–229. Parsons, T. (1975). The sick role and the role of the physician reconsidered. Milbank Memorial Fund Quarterly, 53, 257–277. Paulhus, D. (1983). Sphere-specific measures of perceived control. Journal of Personality and Social Psychology, 44, 1253–1265. Pearce, J.L., & Gregersen, H.B. (1991). Task interdependence and extra-role behavior: A test of the mediating effects of felt responsibility. Journal of Applied Psychology, 76, 838–844. Perdue, W.C., Gostin, L.O., et al. (2003). Public health and the built environment: Historical, empirical, and theoretical foundations for an expanded role. Journal of Law, Medicine & Ethics, 31(4), 557–566.
34411_CH09_265_304
302
9/9/05
CHAPTER 9
10:49 AM
Page 302
DEMOGRAPHIC, PSYCHOLOGICAL, AND SOCIAL
Peters, L.H., O’Connor, E.J., et al. (1980). The behavioral and affective consequences of performance-relevant situational variables. Organizational Behavior and Human Performance, 25, 79–96. Pfeiffer, E. (1975). A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients. Journal of the American Geriatrics Society, 23(10), 433–441. Pilowsky, I., & Spence, N.D. (1983). Manual for the Illness Behavior Questionnaire (IBQ). Adelaide, Australia: University of Adelaide. Pratley, R.E. (1998). Gene-environment interactions in the pathogenesis of type 2 diabetes mellitus: Lessons learned from the Pima Indians. Proceedings of the Nutrition Society, 57(2), 175–181. Prochaska, J.O., DiClemente, C.C., et al. (1992). In search of how people change. Applications to addictive behaviors. American Psychologist, 47(9), 1102–1114. Prochaska, J.O., & Norcross, J.C. (2001). Stages of change. Psychotherapy: Theory, Research, Practice, Training, 38(4), 443–448. Radloff, L.S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 685-701. Ravussin, E. (1993). Energy metabolism in obesity. Studies in the Pima Indians. Diabetes Care 16(1), 232–238. Remington, M., & Tyrer, P.J. (1979). The Social Functioning Schedule: A brief semistructured interview. Social Psychiatry, 14(3), 151–157. Robinson, J.P., Shaver, P.R., et al. (1991). Measures of personality and social psychological attitudes. London: Academic Press. Rollnick, S., Heather, N., et al. (1992). Development of a short “readiness to change” questionnaire for use in brief, opportunistic interventions among excessive drinkers. British Journal of Addiction, 87(5), 743–754. Rosenburg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press. Rotter, J.B. (1966). Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs, 80. Rubin, H.R. (1998). Status report—An investigation to determine whether the built environment affects patients’ medical outcomes. Journal of Healthcare Design, 10, 11–13. Sarason, I.G., Levine, H.M., et al. (1983). Assessing social support: The Social Support Questionnaire. Journal of Personality & Social Psychology, 44(1), 127–139. Shapiro, C.M., Flanigan, M., et al. (2002). Development of an adjective checklist to measure five FACES of fatigue and sleepiness. Data from a national survey of insomniacs. Journal of Psychosomatic Research, 52(6), 467–473. Shaver, P.R., & Brennan, K.A. (1991). Measures of depression and loneliness. In J.P. Robinson, P.R. Shaver and L.S. Wrightsman, Measures of Personality and Social Psychological Attitudes. London: Academic Press. Sherbourne, C.D., & Stewart, A.L. (1991). The MOS social support survey. Social Science & Medicine, 32(6), 705–714.
34411_CH09_265_304
9/9/05
10:49 AM
Page 303
References
303
Shryock, H.S., Siegel, J.S., et al. (1976). The methods and materials of demography. New York: Academic Press. Snipp, C.M. (2003). Racial Measurement in the American census: Past practices and implications for the future. Annual Review of Sociology, 29, 563–588. Spielberger, C.D. (1983). Manual for the State-Trait Anxiety Inventory (STAI). Palo Alto, CA: Consulting Psychologists Press. Stewart, A.L., & Ware, J.E. (1992). Measuring functioning and well-being: The medical outcomes study approach. Durham, NC: Duke University Press. Streiner, D.L., & Norman, G.R. (1995). Health measurement scales: A practical guide to their development and use. New York: Oxford University Press. Summers, G.F. (1970). Attitude measurement. Chicago: Rand McNally. Takeuchi, D.T., & Gage, S.-J.L. (2003). What to do with race? Changing notions of race in the social sciences. Culture, Medicine and Psychiatry, 27(4), 435–445. Turk, D.C., & Melzack, R. (1992). Handbook of pain assessment. New York: Guilford Press. Turk, D.C., Rudy, T.E., et al. (1985). The McGill Pain Questionnaire reconsidered: Confirming the factor structure and examining appropriate uses. Pain, 21(4), 385–397. Wallston, K.A., & Wallston, B.S. (1981). Health locus of control scales. In H.M. Lefcourt (Ed.), Research with the locus of control construct (pp. 189–243). New York: Academic Press. Weber, M., Roth, G., et al. (1978). Economy and society: An outline of interpretive sociology. Berkeley: University of California Press. Weeks, J.R. (1981). Population: An introduction to concepts and issues. Belmont, CA: Wadsworth. Weissman, M.M., & Bothwell, S. (1976). Assessment of social adjustment by patient selfreport. Archives of General Psychiatry, 33(9), 1111–1115. Wilkinson, R.G. (1996). Unhealthy societies: The afflictions of inequality. London: Routledge. Wolfe, J., & Kimerling, R. (1997). Gender issues in the assessment of Post Traumatic Stress Disorder. Assessing psychological trauma and PTSD. J.P. Wilson and T.M. Keane. New York: Guilford Press, xiv, 577. Yesavage, J.A., Brink, T.L., et al. (1982). Development and validation of a geriatric depression screening scale: A preliminary report. Journal of Psychiatric Research, 17(1), 37–49. Zuckerman, M., & Lubin, B. (1965). Manual for the Multiple Affect Adjective Checklist. San Diego, CA: Educational and Industrial Testing Service. Zung, W.W. (1965). A self-rating depression scale. Archives of General Psychiatry, 12, 63–70. Zung, W.W. (1971). A rating instrument for anxiety disorders. Psychosomatics, 12(6), 371–379. Zung, W.W. (1983). A self-rating pain and distress scale. Psychosomatics, 24(10), 887–890.
34411_CH09_265_304
9/9/05
10:49 AM
Page 304
34411_CH10_305_334
9/9/05
10:51 AM
Part IV Analysis
Page 305
34411_CH10_305_334
9/9/05
10:51 AM
Page 306
34411_CH10_305_334
9/9/05
10:51 AM
Page 307
10 Capturing the Effects of Treatment Jeremy Holtzman
INTRODUCTION: THE IMPORTANCE OF UNDERSTANDING TREATMENT Treatment lies at the very center of an outcomes research study; it is the focus of the question the study seeks to answer. Outcomes research is basically about assessing the effects of treatment. It is the “if ” of the “what-if ” question as in: “What would be the outcomes if we used medication A instead of medication B, if we told people to stop smoking, or if we enrolled everyone in HMOs?” Because the treatment defines the central hypothesis that is the impetus for the entire study, it is especially ironic that it frequently gets the least attention from those doing an outcomes research study. Defining, understanding, and implementing the treatment are critical to the successful execution of an outcomes research study, but these steps are frequently neglected. It seems most straightforward; the very presence of a study implies a treatment, yet the obvious presence of a treatment may be deceptive. Although researchers seem to appreciate the need to carefully craft definitions and measures for outcomes, comorbidities, or psychosocial variables, they seem more content to accept treatment without much insight. Quite to the contrary, the pitfalls in defining the treatment easily equal those of any other aspect of outcomes research. Many promising outcomes research projects fail to yield useful information because of a lack of understanding of this important point. WHAT IS TREATMENT? Outcomes research can be applied to an extremely broad range of questions within health care. Any potentially modifiable factor that may impact 307
34411_CH10_305_334
308
9/9/05
CHAPTER 10
10:51 AM
Page 308
CAPTURING THE EFFECTS OF TREATMENT
health might be considered a treatment within outcomes research. This could be as simple as using a single medication as in the question “Does the use of aspirin reduce mortality following heart attacks?” or it could be as complex as an entire health delivery system such as “Is the mortality after heart attacks greater for patients receiving health care on a fee-forservice basis or from an HMO?” Excluded from this definition are only those things that cannot even in theory be changed. Hence, insurance status, which could be very difficult to change for any given individual, could be a treatment whereas age, which cannot be changed, would not be.1 Although outcomes research may deal with a wide range of “treatments” from the very simple to the extraordinarily complex, all of health care ultimately may be distilled down to more basic elements. For example, when the effect of insurance status—fee-for-service versus health maintenance organization—on mortality for individuals with myocardial infarctions is examined, the treatment is in effect composed of thousands of other treatments that add up to the whole observed treatment. In both groups, individuals receive medications, procedures, nursing care, and counseling. Any differences seen in the outcome of the patients (after appropriately accounting for other patient factors) ultimately results from differences in the delivery of medications, procedures, nursing care, and all of those other elements of care that make up health care.
COMPONENTS OF TREATMENT In typical randomized controlled trials, attempts are made to narrowly define the treatment. It would not be unusual for the treatment examined in such a trial to consist of one specific drug given at one specific dosage on one specific schedule. In a typical outcomes study, the treatment is likely to be much more complex and less disciplined regardless of whether the treatment is as basic as management of a disease or as complex as insurance status. In both cases, the treatment is made up of basic components of medical care. In some outcomes studies, one may design an intervention to study. In others, one may study interventions that are already extant, such as receiving a specific surgical procedure. If one is designing an intervention from basic components of health care, one obviously needs to understand the components that will make up the intervention—yet, even if one is using an extant intervention or designing a complex intervention where the basic
34411_CH10_305_334
9/9/05
10:51 AM
Page 309
Components of Treatment
309
components do not seem important (e.g., one may not be concerned with the specifics of the use of medications when the intervention is insurance status), it is still important to understand how the basic components make up the intervention. As is discussed later, a clear understanding of the nature of the treatment generates more confidence that any associations seen are real and may help to explain variations in outcomes.
Diagnosis Versus Treatment Medical care of an individual entails making a diagnosis in order to determine the patient’s problem and then delivering a treatment to address the problem identified. When discussing outcomes studies, the terminology becomes confusing because both the diagnosis and the treatment of a medical condition may be components of the treatment. For example, in the hypothetical study of differences in outcomes seen between patients in health maintenance organizations and fee-for-service medicine, the use of preventive diagnostic testing (e.g., mammography) might be expected to play a role in differences in outcomes seen, given that one hypothesized advantage of health maintenance organizations is their more diligent use of preventive diagnostic testing. It might be argued that diagnosis has an effect only when it changes treatment; at a fundamental level, this is true. However, diagnosis and treatment are frequently so inextricably linked that attempting to focus solely on the treatment leads to less understanding rather than more. Thus, it is important to understand both the diagnostic and treatment components of the treatment in the outcome study. Because the treatment components of the treatment in the outcomes study are usually the more important, they will be discussed first and the diagnostic components after.
Treatment Components There are three basic components of treatment in allopathic (i.e., “Western”) medicine: medications, procedures, and counseling/education.2 Within each of these components, there is a number of characteristics that further define the component. There are common patterns within the characteristics. All include the type and most include some measure of amount and timing. The components and their characteristics are described later. These individual pieces then fit together as the treatment studied.
34411_CH10_305_334
310
9/9/05
CHAPTER 10
10:51 AM
Page 310
CAPTURING THE EFFECTS OF TREATMENT
Table 10–1 offers a generic approach to assessing treatment; it fits different types of treatment differently. The “x” indicates that the characteristic is relevant to that treatment type. Medications Medications include everything that a patient physically takes into his or her system that has some causal, nontrivial relationship to health status. The type of medication could be a specific prescription or nonprescription drug, but could also include nutritional supplements, herbal remedies, or even the nutritional intake itself. Any other substances that an individual takes into his or her system could be a type of medication. For example, the type of anesthesia used during surgery could be considered medication. An outcomes study could compare the outcomes of patients who were given different anesthetics during surgery. Another important aspect of medications is the dosage. Different amounts of a medication may be expected to have different effects. Likewise, the duration of treatment is important; many antibiotics are discontinued prematurely. Medications may also be characterized by the route of delivery (e.g., by mouth or intravenously) and the frequency of the medication. The point in the course of the illness when treatment begins can also be important. Table 10–1 Elements Relevant for Each Type of Treatment Treatment Type Treatment Element
Medications
Procedures
Counseling
Type
x
x
x
Dosage
x
Route
x
Frequency
x
x
x
Duration
x
x
x
Onset/Timing
x
x
x
x
x
Technical Aspects/ Provider Characteristics
x
34411_CH10_305_334
9/9/05
10:51 AM
Page 311
Components of Treatment
311
Procedures Procedures include anything physically done to the patient. Procedures share some characteristics with medications and have others that are unique. Further, some characteristics may apply to certain procedures but not to others. For example, it makes more sense to think about the frequency of physical therapy than open-heart surgery. Technical aspects of the procedure (e.g., specific technique, devices used, how well it was done) can be very important. The most important characteristic of a procedure is the type of procedure. It can range anywhere from a noninvasive massage from a physical therapist to very invasive open-heart surgery. For many procedures, the timing of the procedure may have an important effect on outcome. For example, the outcomes of total hip arthroplasty are better if it is done before patients become too disabled (Holtzman, Saleh, & Kane, 2002). Many procedures are done only one time, but some can be done multiple times and for these the frequency of the procedure is an important characteristic. Unlike medications, many of the technical aspects involved in procedures may affect outcomes. The same procedure may be done with different techniques, with different degrees of skill, and with different devices. Some of these characteristics, such as technical skill, may be directly observed or may be approximated by using characteristics of the providers (for example, comparing novice operators to those with extensive experience). Using approximation requires some careful thought. For example, years of training or practice may not always correlate with specific experience performing a given procedure. Counseling/Education The counseling and education of patients refer to an information exchange between the patient and the clinician for a therapeutic purpose. One type of counseling and education involves the clinician providing information to the patient that is intended to change his or her health environment or behaviors. This category also includes psychological or psychiatric counseling in which the information exchange itself is therapeutic. Counseling and education are broad areas. As was true for medications and procedures, the most important characteristic of counseling/education is the type of counseling or education. This can range anywhere from an educational poster on health to intensive psychotherapy. It may also in-
34411_CH10_305_334
312
9/9/05
CHAPTER 10
10:51 AM
Page 312
CAPTURING THE EFFECTS OF TREATMENT
clude instruction on other therapies such as how to take medications. As with medications, the total dose of the counseling/education may be important; a one-time short intervention may produce quite different results from multiple episodes of counseling or education over time. For counseling and education, the medium is also important. For example, effects may be different if the same information is conveyed by a pamphlet or in person from a physician. Likewise, the nature of the counselor may be important. Advice from a physician may be received differently from that given by another health professional or a lay person. Diagnosis Components As just discussed, the treatment in an outcomes study may have components that relate to the diagnosis of medical issues as well as the actual treatment of them. Diagnosis has a similar set of categories as the treatment components discussed. These categories include: • Medications • Procedures and diagnostic tests • Information exchange between patient and clinician Medications Whereas medications are one of the most common treatments in outcomes studies, they are the least common diagnostic modalities. They are sometimes used as part of a diagnostic test (such as contrast given during an X-ray), but can be used alone such as a therapeutic trial in which a diagnosis is made by whether an individual responds to treatment. One example of this would be administering nitroglycerine to an individual with chest pain in an attempt to make the diagnosis of angina. The use of medications in diagnosis could be characterized in a similar manner to that for treatment just discussed. Diagnostic Procedures and Tests Diagnostic procedures and tests include anything physically done to a patient for the purpose of making a diagnosis. It may range from the simple collection of blood or urine to X-rays to invasive diagnostic procedures
34411_CH10_305_334
9/9/05
10:51 AM
Page 313
Understanding the Components of Treatment
313
(e.g., cardiac catheterization or tissue biopsy). These may be characterized in a similar manner to procedures used for the treatments discussed. Information Exchanged Between Patient and Clinician There is no simple term for the information exchange between clinician and patient for diagnostic purposes as there is for therapeutic purposes (i.e., counseling and education), yet in some areas of medicine such as primary care, it is perhaps the activity that occupies the greatest amount of resources. How the clinician talks to the patient to make a diagnosis may have great impact on eventual outcomes of the patient (Davies & Ware, 1988). It is also possible to consider different modalities of information transfer such as the effect of patients completing information before coming to the physician’s visit. In this way, information transfer for the purpose of diagnosis may be characterized in a similar way to that for therapeutic purposes (i.e., counseling and education).
UNDERSTANDING THE COMPONENTS OF TREATMENT If one is designing a narrow intervention such as examining the outcome of one medication or one session of counseling, the components of treatment are obvious. However, when one is examining a more complex intervention such as the impact of guidelines for the treatment of a specific disease, it is tempting to ignore the fact that the treatment of interest is made up of individual components of health care. It is technically possible to perform an outcomes study while paying no attention to the components of health care and to get results that appear to answer the important questions posed by the study. The danger comes when one attempts to draw conclusions from the results. Without an understanding of the components of the treatment, one cannot understand how the treatment works—and without this understanding, one runs the risk of drawing erroneous conclusions about the true effects of the treatment. A couple of examples may illustrate this.
Clinical Pathways One of the major means of attempting to change health care delivery in the last decade has been the development and implementation of clinical
34411_CH10_305_334
314
9/9/05
CHAPTER 10
10:51 AM
Page 314
CAPTURING THE EFFECTS OF TREATMENT
practice guidelines. One popular type of guideline that has been developed for use in hospitals is the critical or clinical pathway. These guidelines are specific to patients admitted to the hospital with a particular clinical problem such as stroke or patients undergoing a specific surgical procedure such as knee replacement or coronary artery bypass grafting. The clinical pathway usually contains specific treatments that patients should receive on each day as they move through their hospitalization as well as milestones that they should be meeting during the hospitalization. The pathway is then used to guide each individual patient’s treatment in the hospital and to track his or her progress. In theory, the care of the patient will be improved because he or she will get all of the appropriate treatments— because the treatments are spelled out in the pathway—and any deviations from the expected progress will be detected early—because the expected progress is spelled out in the pathway—and appropriate steps will be taken to address the deviation from expected progress. In general, clinical pathways are based on outcomes research, but it is possible to reverse the process. The validity of pathways based on clinical insights can be tested by their ability to affect outcomes. A very reasonable outcome study would be to examine whether the use of clinical pathways did indeed improve outcomes for patients, and a number of such studies have been undertaken. A fairly representative study of this type is that of Odderson and McKenna (1993), which examined the effect of a pathway to guide the care of patients admitted to the hospital with stroke. They studied whether in their institution the outcomes of patients were better after the introduction of guidelines for stroke care. After implementing the pathway, they noted a decrease in the rate of urinary tract infections (UTIs) of nearly two thirds. One might conclude from that decrease that clinical pathways for stroke care are effective, at least in avoiding the complications of UTIs, and should be implemented broadly. However, before drawing such a conclusion one needs to consider what are the true underlying components of care. For patients with strokes, the component of care that was most directly related to the development of UTIs was early diagnosis of difficulty voiding through the use of an ultrasound of the bladder. The clinical pathway included an ultrasound of the bladder, which had not been routine care prior to the use of the pathway. Hence, the pathway appears to be effective. Yet is it? Why are patients now getting bladder ultrasounds? Can one conclude that it is from the use of the pathway? Can one attribute the improvement to one element of the pathway or the whole enterprise?
34411_CH10_305_334
9/9/05
10:51 AM
Page 315
Understanding the Components of Treatment
315
Would a more focused approach that simply emphasized bladder ultrasounds be as effective? In order to implement the clinical pathway for stroke, it was first necessary to write the pathway. This might require that representatives of all of the clinical staff get together and compose the pathway and reach consensus on the proper care of patients admitted with stroke. In this case, the conclusion was that all patients should receive a bladder ultrasound; something that had not been routine before use of the pathway. So, what is the treatment? Is it bringing nurses and doctors together to derive a plan of best care or is it actually using the piece of paper, the pathway, in clinical care? The implications are great. If it is the process of developing the pathway that is important, one can stop and need not invest the time and resources in actually using the pathway. Further, one may not expect the same results if one adopts a different pathway developed elsewhere and implements it. To draw a reasonable conclusion, one must become aware of the components of treatment and, having done so, isolate the specific treatment of interest.
Care from Cardiologists Versus Other Physicians Medicine has become more specialized over the last couple of decades. One reasonable question is whether this trend is a good thing, improving the outcomes of patients—or a bad thing, making care more disjointed and costly. Jollis and colleagues (1996) examined this question for one specific clinical problem, acute myocardial infarction. Specifically, they examined whether the outcomes of patients admitted to the hospital with acute myocardial infarction were better if a cardiologist versus another type of physician admitted the patient. They found that patients who were admitted by a cardiologist were less likely to die in the coming year than those who were admitted by other types of physicians. One possible issue with the result is that the patients admitted by cardiologists may be systematically different. They may differ in terms of severity of disease, comorbidities, and demographic factors that may independently influence the probability of death, yet the researchers found that the differences were still present when those factors were accounted for through statistical means. Can one conclude that care for patients admitted by cardiologists is better? First, one should examine the components that make up the intervention. Care of the patient with a myocardial infarction is complex with multiple treatment components. All of the types of components of treatment may be
34411_CH10_305_334
316
9/9/05
CHAPTER 10
10:51 AM
Page 316
CAPTURING THE EFFECTS OF TREATMENT
present, including medications, procedures, and counseling, but all of these components are not universally available and where available may not be of similar quality everywhere. For example, cardiac procedures such as cardiac catheterization and coronary artery bypass grafting are not available at all hospitals. Further, such services such as care in a coronary care unit might be expected to vary from a hospital that routinely cares for multiple patients with myocardial infarction at a time versus one that rarely cares for a patient with a myocardial infarction. Are these components of treatment the same in institutions where cardiologists practice versus the institutions where other physicians practice? In fact, they are not. For example, cardiac procedures are more likely to be available in institutions where cardiologists admit patients. Are the outcomes of cardiologists better or is admission by a cardiologist simply a marker of care in an institution where the individual components of care lead to better outcomes? By understanding the components of care and taking them into account in the execution and analysis of the study, one may be able to conclude whether the treatment “admission by a cardiologist” leads to better outcomes.
Isolating the Treatment of Interest Having examined the components of treatment, how does one go about isolating the effect of interest to permit reasonable conclusions? As with the similar issue of accounting for differences in the population’s comorbidities, severity, and so forth, there are two possible solutions: One may design the study in such a way that the results directly reflect the treatment of interest or one may use statistical methods to isolate the effect of the treatment of interest. Designing a Study to Isolate the Effect of the Treatment The most straightforward way to isolate the effect of the treatment of interest is to use a comparison group that differs from the group that gets the treatment only by not getting the treatment. Again, it is important to understand the elements of the treatment to facilitate making fair comparisons. In the example of clinical pathways, one might consider that after developing the pathway, it is used for some patients and not others. All of the patients benefit from the development phase of the pathway, but only those for whom the pathway is actually used benefit from the use of the
34411_CH10_305_334
9/9/05
10:51 AM
Page 317
Understanding the Components of Treatment
317
pathway. If a positive effect is seen, one might conclude that the use of the pathway itself is beneficial. One disadvantage to doing the study in this way is that any beneficial effects of the development of the pathway would not be observed. For treatments that occur over time, another possible means of isolating the effect of the treatment would be to examine the effect at different points in time. In the clinical pathway example, one might examine the change in outcome between the time before the pathway project is undertaken and the time when the pathway is developed but not yet not implemented. A second analysis could focus on changes after pathway development but before and after it is implemented. Such an approach could isolate the effect of developing the pathway from the use of the pathway. (This technique for examining treatment would be much stronger if one can also include a group that is not exposed to the development or implementation of the pathway to account for any changes occurring over time.) One study that used this approach for a renal transplant pathway found that the positive changes that occurred with the development of the pathway were of about the same magnitude as those that occurred with the implementation of the pathway, suggesting that both aspects of the treatment were important (Holtzman, Bjerke, & Kane, 1998). No recipe for a study design will work to isolate the treatment effect for every outcomes study. The important process is to examine the components of the treatment and with that knowledge design the study to isolate the effect. Statistically Isolating the Effect of Treatment Just as not every study can be done as a randomized controlled trial to disentangle the effect of treatment from the effect of differences in the study groups, not every treatment can be isolated through the design of the study. In these cases, it is necessary to use statistical methods to isolate the treatment effect. The example of the effect of admission by cardiologists on the outcomes of acute myocardial infarction is a case in point. The hospitals where cardiologists practice may be an important element in the treatment, but the investigators have no power over that factor and therefore cannot isolate the effect of admission physician through the study design. However, it is possible to statistically adjust for those other differences in treatment and thereby isolate the effect of admission by a cardiologist. For example, one may measure and then statistically adjust for the availability
34411_CH10_305_334
318
9/9/05
CHAPTER 10
10:51 AM
Page 318
CAPTURING THE EFFECTS OF TREATMENT
of cardiac procedures such as coronary angiography and bypass surgery.3 Then, an observed effect on outcome can be more reasonably interpreted as the effect of the physician specialty and not the institution in which the physician practices. When Jollis and colleagues (1996) employed this approach, they found that the outcomes of cardiologists’ patients were still superior. Hence, understanding the components of treatment and taking appropriate steps to isolate the treatment of interest through statistical means permit more confident conclusions.
NEED FOR VARIATION IN TREATMENT Investigating the effect of treatment requires that there be variation in the treatment within the study. In other words, it is necessary to have a comparison group. Without some sort of comparison, one cannot assess the treatment effect. The need for some variation in treatment within the study to assess the effect of treatment seems obvious when discussed in theory, but barriers stymie comparison in practice. Some of these barriers are within the investigator’s control, whereas others arise from the nature of the study. One issue that makes meaningful variation within the treatment difficult is the siren song of a new treatment. It is tempting when one has an idea for a new treatment that will give better care to want to do it for everybody. From the point of view of care, there seems little point in denying the new treatment to anybody. For example, a health plan wanted to study the outcomes of a new clinic designed to do risk assessment and treatment for patients after hospitalization for myocardial infarction. They were convinced that this new approach was better than their current method of having the patients return to their primary care providers (if they had one) for risk management. No other comparison groups such as patients not in the health plan were available. Although the health plan had some historical data on patients with myocardial infarction, they did not have sufficient detail on risk modification following myocardial infarction to use a historical comparison group. The only available option to provide variation in the treatment was to utilize the new clinic only for a portion of the patients, but the physicians planning the intervention could not accept the idea that some patients would get what they viewed as inferior care, and they could not support using the intervention for only a limited group.
34411_CH10_305_334
9/9/05
10:51 AM
Page 319
The Treatment Variation Defines the Treatment
319
Sometimes organizational constraints will not allow variation in the treatment. The creation of the risk management clinic for post–heart attack patients required a significant economic investment. It was feared that the clinic could not afford to serve only a portion of the myocardial infarction population. Hence, it was not possible to have a group of patients not use the clinic. Thus, the desire to give the new treatment to everyone and the economics that did not allow a smaller treatment group meant that there was no way to create an adequate comparison; that is, there was no way to have variation in the treatment within the study. Doing an outcomes study was impossible. Sometimes doing no study compared to doing a flawed one is the better option. If everyone waited until everything was scientifically proven, beneficial change might be more ploddingly slow than it already is, yet, in an age of evidence-based practice, such a position is heresy. Without the ability to examine the outcomes, one may never know if the treatment has improved care. There may be practical barriers to variation in treatment. For example, a surgeon wanted to compare the outcomes of revascularization to amputation for severe peripheral vascular disease—two treatments that she felt may in the long run be similar. However, almost all of the patients seen at the tertiary care hospital had been referred for revascularization. There was no way then to create a reasonable comparison.
THE TREATMENT VARIATION DEFINES THE TREATMENT Given the barriers to identifying a comparison group to test the treatment, it may be tempting to use whatever comparison proves most practical. However, the selection of the comparison group must be done as carefully as the identification of the treatment, because the comparison group defines the treatment as much as the treatment group does. The treatment is not simply what is happening to the group of specific interest. In the previous example, the interest was focused on the outcomes of patients who went to a risk management clinic after a myocardial infarction, but the treatment effect one could observe and analyze is the difference in outcomes between that group and any comparison group. The size of the effect of the new clinic differs if it is compared to, for example, patients who have no specific follow-up care or patients who are
34411_CH10_305_334
320
9/9/05
CHAPTER 10
10:51 AM
Page 320
CAPTURING THE EFFECTS OF TREATMENT
followed up in a primary care clinic with a history of aggressive prevention programs. The nature of the comparison group is particularly critical in addressing complex treatments in which interest is ultimately directed toward only a portion of the treatment. The clinical pathway example illustrates this situation. In that example, the treatment could consist of either the development and implementation of the pathway or just the implementation alone. In studying the effect of implementing the pathways, the comparison group needs to differ from the treatment group only in whether it was exposed to the implementation of the pathway; that is, both groups need to be exposed to the development of the pathway. A historical comparison group, which has been used for most of the studies of pathways, is inadequate because the group would not be exposed to the development of the pathway. Using a historical comparison group automatically defines the treatment effect as the result of both the development of the pathway and its implementation.
Types of Variation The variation in treatment does not have to be one group getting a treatment and another group not getting a treatment. There is a wide range of possible variations or comparisons depending upon the treatment effect of interest. The basis of comparison might be organized into the categories shown in Table 10–2. Although they are depicted as separate categories, they are not necessarily independent of one another. A difference in the outcomes of care across two hospitals, for example, is attributable not merely to the difference in physical setting, but also differences in the providers, the financial settings, the types and quality of treatments provided, the policies and procedures, and so on. Comparison of Different Treatment Regimens Comparing different treatment regimens means comparing what was done. As the examples in Table 10–3 show, any treatment category (medication, procedure, or counseling) can be compared across different dimensions. The cells in the matrix represent the various ways in which different treatments can be compared. The cells in the table use published studies as examples. Three dimensions of comparison are identified:
34411_CH10_305_334
9/9/05
10:51 AM
Page 321
The Treatment Variation Defines the Treatment
321
Table 10–2 Bases for Comparing Treatments 1. Comparison of different treatment regimens a. Treatment A versus no treatment b. Treatment A versus treatment B c. Treatment A B versus treatment B alone 2. Comparison of the intensity of treatment 3. Comparison of the duration of treatment 4. Comparison of the timing of treatment 5. Comparison of the setting of the treatment a. Financial setting: e.g., HMO vs. FFS b. Physical setting: e.g., Hospital A vs. Hospital B c. Social/organizational setting: e.g., Teaching hospitals vs. community hospitals Hospice vs. home care d. Geographic setting: e.g., Rural vs. urban clinics 6. Comparison of the characteristics of the provider a. Training b. Experience c. Personal characteristics
1. Treatment versus no treatment. Do elderly individuals who receive medication for isolated systolic hypertension have a lower incidence of cardiovascular disease than those receiving no treatment? (Anonymous, 1991b) 2. Treatment A versus treatment B. Is one antihypertensive medication more effective at preventing heart attack than another? (Anonymous, 2002) 3. Treatment A + B versus treatment B alone. Do beta-blockers provide additional benefit for patients with congestive heart failure already on angiotensin-converting enzyme inhibitors? (Anonymous, 1999) Investigators comparing treatment versus no treatment must account for the placebo effect inherent in these experimental designs. The placebo effect refers to the phenomenon by which the patient’s health status improves after being given a substance that has no metabolic significance. The patient reacts psychologically to the administration of the treatment,
34411_CH10_305_334
322
9/9/05
10:51 AM
CHAPTER 10
Page 322
CAPTURING THE EFFECTS OF TREATMENT
Table 10-3 Studies of Various Treatment Regimens Basis of Comparison
Treatment
Treatment A vs. no treatment
Treatment A vs. treatment B (best current practice)
Treatment A B vs. treatment A alone
Medication
CIBIS-II examined SHEP trial; investi- ALLHAT trial whether betacompared the gated whether blockers were outcomes of it is better to beneficial in different medicatreat isolated addition to tions for hypersystolic hyperACE inhibitors tension with no tension in the for congestive placebo group elderly than to heart failure (Anonymous, leave it un(Anonymous, 2002) treated 1999) (Anonymous, 1991b)
Procedure
Allen et al. examThe National Macular ined whether a Cancer Institute Photocoagulation transmyocardial conducted a Study (MPS); revascularizastudy examining investigated the tion procedure whether outuse of photoprovided addicomes were coagulation vs. tional benefit to different for no treatment patients undermastectomy for eyes with going coronary or breast conchoroidal revasartery bypass servation surcularization grafting (Allen et gery for breast (Anonymous, al., 2000) cancer (Poggi et 1991a) al., 2003)
Counseling/ Education of patient
Teri and Lewinsohn Lovibond et al. Hypertension examined examined Prevention Trial; whether more whether individaddressed the personalized ual or group effects of councounseling on treatment was seling changes exercise promore effective for in health behavvided benefit in depression (Teri iors on conaddition to gen& Lewinsohn, trolling mild eral counseling 1986) hypertension (Lovibond et al., (Anonymous, 1986) 1990)
34411_CH10_305_334
9/9/05
10:51 AM
Page 323
The Treatment Variation Defines the Treatment
323
Basis of Comparison
Treatment Combinations
Treatment A vs. no treatment
Treatment A vs. treatment B (best current practice)
Treatment A B vs. treatment A alone
Hurt et al. examThe VA cooperaThe Lung Health ined whether tive urologic Study; examined adding buprostudy group; whether medicapion to counselexamined surgitions and couning was more cal orchiectomy seling were effective than versus medicamore effective counseling alone tions for prostate for smoking for smoking cescancer cessation than sation (Hurt et (Anonymous, no intervention al., 1997) 1967) (Hughes et al., 2004)
and these psychological changes affect the patient’s physical well-being. If only the treatment group receives a treatment, then one cannot distinguish changes in health status resulting from the treatment envisioned by the researchers from those generated by a placebo effect. In many instances, investigators can control for the placebo effect by providing the no-treatment group with a placebo. In a drug outcome study, this would take the form of a benign pill that is similar in appearance to the medication in question, but has no significant metabolic effects. Developing a placebo for a procedure is more problematic, although there are instances where it can be done and has been done;4 in most situations, a surgical placebo—a sham operation—is both impractical and unethical. However, researchers should attempt to minimize the placebo effect by having the control group engage in many of the same procedures associated with the treatment in question. If exercise with a physical therapist is the treatment in question, then researchers should have the control nonexercise group make the trip to the physical therapist even though they do not receive the treatment or they should engage in exercise (or a comparable activity) led by someone with less training. Otherwise, the investigators cannot determine whether any difference in outcomes between the two groups was influenced by the fact that one group left the house and met with the therapist and the other did not.
34411_CH10_305_334
324
9/9/05
CHAPTER 10
10:51 AM
Page 324
CAPTURING THE EFFECTS OF TREATMENT
Combinations of different classes of treatments can also be compared. For example, Hurt and colleagues (1997) examined whether adding the medication bupropion to counseling for smoking cessation was more effective than counseling alone. Because the critical issue is comparing what was done, investigators should take steps to control for how well the treatment was administered. For example, if the study compares one surgical procedure to another, investigators should take steps to ensure the two procedures were performed by equally competent surgeons. A poorly executed CABG may have worse outcomes than a competently executed angioplasty regardless of the merits of the two procedures. The effect of variations in surgical skill can be disentangled from the effects of the treatments by manipulating the experimental design or by accounting for these variations statistically. Solutions based on experimental design include randomizing surgeons to surgical procedures, so that there is no systematic correlation between surgeon characteristics and type of procedure. Statistical solutions involve estimating a multiple regression that includes dummy variables to indicate which physician performed the surgery or using other results from these surgeons as a proxy for their skills. In this way, any special skills that a given surgeon brings to the experiment can be statistically disentangled from the effect of the surgery itself. Dummy variables that represent interactions between the surgeon and the surgical procedure he or she performed may also be necessary if some surgeons tend to be skilled in one surgical procedure but not the other. A more controversial element of treatment is the role of patient adherence. On the one hand, it seems intuitive that a drug not taken is unlikely to work. Hence, the rate of actually taking the drug should be considered. However, the reasons for not taking the drug may be related to the drug itself (e.g., taste, side effects). Thus, adherence is part of the picture. In formal drug trials, adherence is not allowed as an intervening variable. Indeed, there are formal rules about using intention to treat (ITT), whereby every subject assigned to a study condition must be retained in the analysis even if the subject did not complete the course of treatment. Nonetheless, examining adherence may prove useful in understanding why a particular drug regimen was not successful. Comparing the Intensity of Treatment Treatment intensity—that is, the amount of treatment per unit (i.e., per unit time, dosage, or encounter)—can affect the treatment. Table 10–4
34411_CH10_305_334
9/9/05
10:51 AM
Page 325
The Treatment Variation Defines the Treatment
325
Table 10–4 Studies of Comparing Treatment Intensity, Timing, or Duration Basis of Comparison Treatment
Intensity
Timing
Duration
Medication
Kearon et al. examined more intensive versus less intensive warfarin therapy to prevent recurrent deep venous thrombosis (Kearon et al., 2003)
Cook et al. examined the effectiveness of twice daily dosing of Augmentin compared to three times a day for childhood respiratory tract infections (Cook et al., 1996)
Elhanan et al. examined 1 day versus 5 days of antibiotics for urinary tract infections in women (Elhanan et al., 1994)
Procedure
Helmhout et al. examined whether highintensity or low-intensity training was more effective for low back pain (Helmhout et al., 2004)
Rainville et al. examined the effectiveness of twice a week versus three times a week physical therapy for back pain (Rainville et al., 2002)
Hamza et al. examined the effect of different durations of electrical stimulation to relieve low back pain (Hamza et al., 1999)
Counseling/ Education of patient/ provider
Alterman et al. examined varying intensity of counseling for smoking cessation (Alterman et al., 2001)
Joseph et al. examined the optimal timing of smoking cessation counseling in individuals undergoing alcohol dependence treatment (Joseph et al., 2002)
Gilbert et al. examined the effect of the duration of a smoking cessation program on quitting (Gilbert et al., 1992)
continues
34411_CH10_305_334
326
9/9/05
CHAPTER 10
10:51 AM
Page 326
CAPTURING THE EFFECTS OF TREATMENT
Table 10–4 continued Basis of Comparison Treatment Combinations
Intensity
Timing
Duration
The UKPDS examined whether a more intensive regimen of blood glucose control for diabetics reduced diabetic complications (Anonymous, 1998)
depicts various study questions that compare treatments according to intensity, timing, and duration. Comparisons based on the duration of treatment could examine how duration of antibiotic use affects recovery from urinary tract infections (Elhanan, Tabenkin, Yahalom, & Raz, 1994; medications), how the duration of electrical stimulation affects relief of back pain (Hamza et al., 1999; procedure), or how a longer smoking cessation program affects the likelihood of quitting (Gilbert et al., 1992; counseling). Studies may also focus on other aspects of intensity or timing of the intervention. Comparing the Setting of the Treatment The setting of the treatment (i.e., where and under what conditions the treatment is given) can affect both the treatment regimen as well as the quality or skill of the treatment. Studies that use the setting of the treatment as the basis of comparison typically treat the treatment as a black box. Although the investigators recognize that treatments may differ qualitatively across settings, they are not immediately interested in characterizing these differences. Rather, they concentrate on the implications of the differences (whatever they may be) for the health of patients in each setting. Several aspects of treatment setting have been studied in terms of their impacts on treatment and outcomes. The financial setting of the treatment refers to the method by which the providers of the treatment are paid for
34411_CH10_305_334
9/9/05
10:51 AM
Page 327
The Treatment Variation Defines the Treatment
327
their services. Financial reimbursement schemes can have a substantial impact on the type of treatments provided and on the locus of care. The restrictions that HMOs may place on access to specialists have implications for the type of care provided to HMO enrollees. Similarly, changes in the reimbursements of hospitals over the last decades moved substantial amounts of care from inpatient to outpatient facilities. The geographic and physical settings also clearly affect the type and quality of treatments offered to patients. For example, comparing surgical outcomes between rural and urban hospitals may reflect differences in care and available resources. Similarly, a comparison of outcomes between teaching and nonteaching hospitals tests the effect of the social and organizational setting on the outcomes of care. Comparison Across Providers of the Treatment Figure 10–1 provides a classification of the various aspects of the provider of the treatment that can be used to compare treatment. Orthodoxy refers to the perspective of medicine under which the provider of treatment has trained. For example, chiropractors and neurosurgeons both treat back ailments but have been trained under quite different philosophies. Consequently, the treatments they provide differ in type and perhaps quality. Another aspect of training orthodoxy is the treatment paradigm or practices of the institution in which a provider trained. Certain institutions may emphasize preventive care, or cost-conscious medicine, or other treatment paradigms that are imbued in their graduates. Table 10–5 shows how orthodoxy, as a basis of comparison, relates to the treatment regimen prescribed. The level of training refers to the amount of time a provider spends in formal education (including clinical training). A board-certified specialist has more formal training than a general practice physician, and this training may affect the type and quality of treatment provided. Table 10–6 speculates on how training may affect the quality of a consultation between a patient and physicians with different levels of training. The effect on treatment of the provider’s experience can be disentangled into the effect attributable to volume of treatment and the effect attributable to age; that is, accumulated practical knowledge and time since formal training. Volume can affect the mechanical and intellectual mechanical skill with which a treatment is performed. Low-volume surgical units are more likely to suffer from low quality than more actively used units (Hannan, Sia, Kumar, Kilburn, & Chassin, 1995; Jollis et al., 1994; Shook, Sun, Burstein,
Example: Specialst vs. General Practioner
Example: Chiropractor vs. Neurosurgeon
Example: High volume surgical units vs. Low
Volume
Figure 10–1 Aspects of the Provider of Treatment
Degree of Training
Time on the Job
Example: Female vs. Male OB/Gyn
Gender
Personal Chars.
Race
10:51 AM
Orthodoxy
Experience
CHAPTER 10
Training
328
9/9/05
Who Did It?
34411_CH10_305_334 Page 328
CAPTURING THE EFFECTS OF TREATMENT
34411_CH10_305_334
9/9/05
10:51 AM
Page 329
The Treatment Variation Defines the Treatment
329
Table 10–5 Differences in Treatment Associated with Provider Orthodoxy Orthodoxy of Provider Treatment
Chiropractor
Neurosurgeon
Medication
Over-the-counter medications
Prescription medications
Procedure
Manipulations of the spine Strong emphasis
Surgery
Counseling
Weak emphasis
Eisenhauer, & Matthews,1996). A provider’s age may have conflicting influences on treatment, because it may reflect either an accumulation of working knowledge or an unfamiliarity with the most recent findings. Personal characteristics of the provider may also affect both the quality and type of treatment provided. For example, the gender of physicians has been associated with the provision of Pap smears and mammography to their female patients (Lurie et al., 1993). Patients who were treated by female physicians were more likely to receive diagnostic procedures than were patients treated by male physicians, although this difference ameliorated with the age of the physician. These results are difficult to interpret. They could reflect either patient or physician reluctance to discuss sex-related issues with an individual of the opposite gender. On the other hand, the results may reflect a lack of vigilance on the part of male physicians to the needs of female patients. Alternatively, they may reflect the effects of a selecTable 10–6 Possible Differences in Treatment Quality Associated with the
Training of the Provider Training of Provider Quality/Skill of Treatment
Specialist
General Practice
Mechanical
High
Varied
Intellectual
High
Moderate
Interpersonal
Moderate
High
34411_CH10_305_334
330
9/9/05
CHAPTER 10
10:51 AM
Page 330
CAPTURING THE EFFECTS OF TREATMENT
tion process whereby female patients who are concerned about prevention may seek out female physicians. SUMMARY The treatment in outcomes research is the element of health care that is hypothesized to affect outcomes. The scope of that element is broad; it could include anything from one particular medication to the organization of an entire delivery system. Any potentially modifiable factor that can be studied is fair game for an outcomes study. The treatment variable can become the quicksand of an outcomes study. Unlike assessing the outcomes and the other important independent factors and conducting the analysis—all of which might seem on the surface to present potential problems that must be addressed—the treatment may appear straightforward and benign. However, lack of care with the construct of treatment in an outcomes study is as likely as any other aspect of the study to prohibit meaningful conclusions. Drawing meaningful conclusions about the effect of treatment requires attending to three things about the treatment and its assessment in the study: 1. Understand the components of the treatment and how they go together—what elements are of interest and what elements need to be addressed in some other way 2. Include variation in treatment in the study—some sort of comparison group or comparison within the treatment group (e.g., varying doses of medication) 3. Understand that the effect of the treatment observed is the difference in outcomes between the treatment group and the comparison group REFERENCES Allen, K.B., Dowling, R.D., DelRossi, A.J., Realyvasques, F., Lefrak, E.A., Pfeffer, T.A., et al. (2000). Transmyocardial laser revascularization combined with coronary artery bypass grafting: A multicenter, blinded, prospective, randomized, controlled trial. Journal of Thoracic Cardiovascular Surgery, 119(3), 540–549. Alterman, A.I., Gariti, P., & Mulvaney, F. (2001). Short- and long-term smoking cessation for three levels of intensity of behavioral treatment. Psychology of Addictive Behaviors, 15(3), 261–264. Anonymous. (1967). Treatment and survival of patients with cancer of the prostate. Surgery, Gynecology & Obstetrics, 124(5), 1011–1017.
34411_CH10_305_334
9/9/05
10:51 AM
Page 331
References
331
Anonymous. (1990). The hypertension prevention trial: Three-year effects of dietary changes on blood pressure. Archives of Internal Medicine, 150(1), 153–162. Anonymous. (1991a). Laser photocoagulation of subfoveal neovascular lesions in agerelated macular degeneration. Results of a randomized clinical trial. Archives of Ophthalmology, 109(9), 1220–1231. Anonymous. (1991b). Prevention of stroke by antihypertensive drug treatment in older persons with isolated systolic hypertension. Final results of the systolic hypertension in the elderly program (SHEP). Journal of the American Medical Association, 265(24), 3255–3264. Anonymous. (1998). Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet, 352(9131), 837–853. Anonymous. (1999). The cardiac insufficiency bisoprolol study ii (CIBIS-II): A randomised trial. Lancet, 353(9146), 9–13. Anonymous. (2002). Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs. diuretic: The antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT). Journal of the American Medical Association, 288(23), 2981–2997. Cook, R.C., Zachariah, J., Cree, F., & Harrison, H.E. (1996). Efficacy of twice-daily amoxycillin/clavulanate (“augmentin-duo” 400/57) in mild to moderate lower respiratory tract infection in children. British Journal of Clinical Practice, 50(3), 125–128. Davies, A.R., & Ware, J.E., Jr. (1988). Involving consumers in quality assessment. Health Affairs, 7(1), 33–48. Elhanan, G., Tabenkin, H., Yahalom, R., & Raz, R. (1994). Single-dose fosfomycin trometamol versus 5-day cephalexin regimen for treatment of uncomplicated lower urinary tract infections in women. Antimicrobial Agents & Chemotherapy, 38(11), 2612–2614. Gilbert, J.R., Wilson, D.M., Singer, J., Lindsay, E.A., Willms, D.G., Best, J.A., et al. (1992). A family physician smoking cessation program: An evaluation of the role of follow-up visits. American Journal of Preventive Medicine, 8(2), 91–95. Hamza, M.A., Ghoname, E.A., White, P.F., Craig, W.F., Ahmed, H.E., Gajraj, N.M., et al. (1999). Effect of the duration of electrical stimulation on the analgesic response in patients with low back pain. Anesthesiology, 91(6), 1622–1627. Hannan, E.L., Siu, A.L., Kumar, D., Kilburn, H., Jr., & Chassin, M.R. (1995). The decline in coronary artery bypass graft surgery mortality in New York State. The role of surgeon volume. Journal of the American Medical Association, 273(3), 209–213. Helmhout, P.H., Harts, C.C., Staal, J.B., Candel, M.J., & de Bie, R. A. (2004). Comparison of a high-intensity and a low-intensity lumbar extensor training program as minimal intervention treatment in low back pain: A randomized trial. European Spine Journal, 13(6), 537–547. Holtzman, J., Bjerke, T., & Kane, R. (1998). The effects of clinical pathways for renal transplant on patient outcomes and length of stay. Medical Care, 36(6), 826–834. Holtzman, J., Saleh, K., & Kane, R. (2002). Effect of baseline functional status and pain on outcomes of total hip arthroplasty. Journal of Bone Joint Surg Am, 84-A(11), 1942–1948.
34411_CH10_305_334
332
9/9/05
CHAPTER 10
10:51 AM
Page 332
CAPTURING THE EFFECTS OF TREATMENT
Hughes, J., Lindgren, P., Connett, J., & Nides, M., (2004). Smoking reduction in the lung health study. Nicotine & Tobacco Research, 6(2), 275–280. Hurt, R.D., Sachs, D.P., Glover, E.D., Offord, K.P., Johnston, J.A., Dale, L.C., et al. (1997). A comparison of sustained-release bupropion and placebo for smoking cessation. New England Journal of Medicine, 337(17), 1195–1202. Jollis, J.G., DeLong, E.R., Peterson, E.D., Muhlbaier, L.H., Fortin, D.F., Califf, R.M., et al. (1996). Outcome of acute myocardial infarction according to the specialty of the admitting physician. New England Journal of Medicine, 335(25), 1880–1887. Jollis, J.G., Peterson, E.D., DeLong, E.R., Mark, D.B., Collins, S.R., Muhlbaier, L.H., et al. (1994). The relation between the volume of coronary angioplasty procedures at hospitals treating Medicare beneficiaries and short-term mortality. New England Journal of Medicine, 331(24), 1625–1629. Joseph, A.M., Willenbring, M.L., Nelson, D., & Nugent, S.M. (2002). Timing of alcohol and smoking cessation study. Alcoholism: Clinical & Experimental Research, 26(12), 1945–1946. Kearon, C., Ginsberg, J.S., Kovacs, M.J., Anderson, D.R., Wells, P., Julian, J.A., et al. (2003). Comparison of low-intensity warfarin therapy with conventional-intensity warfarin therapy for long-term prevention of recurrent venous thromboembolism. New England Journal of Medicine, 349(7), 631–639. Lovibond, S.H., Birrell, P.C., & Langeluddecke, P. (1986). Changing coronary heart disease risk-factor status: The effects of three behavioral programs. Journal of Behavioral Medicine, 9(5), 415–437. Lurie, N., Slater, J., McGovern, P., Ekstrum, J., Quam, L., & Margolis, K. (1993). Preventive care for women. Does the sex of the physician matter? New England Journal of Medicine, 329(7), 478–482. Moseley, J.B., O’Malley, K., Petersen, N.J., Menke, T.J., Brody, B.A., Kuykendall, D.H., et al. (2002). A controlled trial of arthroscopic surgery for osteoarthritis of the knee. New England Journal of Medicine, 347(2), 81–88. Newhouse, J.P. (1993). Free for all? Lessons from the RAND health insurance experiment. Cambridge, MA: Harvard University Press. Odderson, I.R., & McKenna, B.S. (1993). A model for management of patients with stroke during the acute phase. Outcome and economic implications. Stroke, 24(12), 1823–1827. Poggi, M.M., Danforth, D.N., Sciuto, L.C., Smith, S.L., Steinberg, S.M., Liewehr, D.J., et al. (2003). Eighteen-year results in the treatment of early breast carcinoma with mastectomy versus breast conservation therapy: The National Cancer Institute randomized trial. Cancer, 98(4), 697–702. Rainville, J., Jouve, C.A., Hartigan, C., Martinez, E., & Hipona, M. (2002). Comparison of short- and long-term outcomes for aggressive spine rehabilitation delivered two versus three times per week. Spine Journal, 2(6), 402–407. Shook, T.L., Sun, G.W., Burstein, S., Eisenhauer, A.C., & Matthews, R.V. (1996). Comparison of percutaneous transluminal coronary angioplasty outcome and hospital costs for low-volume and high-volume operators. American Journal of Cardiology, 77(5), 331–336.
34411_CH10_305_334
9/9/05
10:51 AM
Page 333
Notes
333
Teri, L., & Lewinsohn, P.M. (1985). Individual and group treatment of unipolar depression: Comparison of treatment outcome and identification of predictors of successful treatment outcome. Behavior Therapy, 17(3), 215–228.
NOTES 1. For a description of an elaborate test of the effects of health insurance, see Newhouse (1993). 2. Other medical belief systems such as spiritual healing or chiropractic may have other types of components. These may be reasonable elements for an outcome study, but the specifics of such are beyond the scope of this text. The general principles of understanding those interventions on a basic level outlined in this section would still apply to those other types of care. The basic level may, however, be different. 3. In some cases, it may be necessary to use more complex analytic approaches that capture the effects of “nesting” physicians within hospitals. Statistical consultation will be needed for such analyses. 4. One of the rare examples of a surgical placebo study involved arthroscopic surgery (Moseley et al., 2002).
34411_CH10_305_334
9/9/05
10:51 AM
Page 334
34411_CH11_335_350
9/9/05
11:25 AM
Page 335
11 Cost-Effectiveness Analysis John A. Nyman
INTRODUCTION Cost-effectiveness analysis (CEA) is a procedure for determining whether a new health care “treatment” (meaning a procedure, social program, pharmaceutical, device, or a medical intervention of any type) is beneficial to society. A CEA is often used by government agencies to formally evaluate whether a certain treatment should be paid for by a public agency. The incremental cost-effectiveness ratio (ICER) is the main statistic of interest in CEA. The ICER represents the cost per unit increase in effectiveness and is calculated by dividing the marginal increase in costs by the marginal increase in effectiveness or benefits. For example, assume there are two new treatments for patients with heart attacks: a new drug and a new surgery. The drug costs $6000 and extends the life expectancy of the heart attack patient for 0.5 years, compared with no treatment. The surgery costs $30,000 and extends the life expectancy of the heart attack patient 1 year, compared with no treatment. The ICER would be constructed by comparing the increase in costs, $24,000 ($30,000 – $6000), to the increase in life years saved, 0.5 life years (1 life year – 0.5 life year). The ICER would be calculated as $48,000 ($24,000 / 0.5) per life year saved. If society cannot identify another treatment that saves a life year for less than $48,000, then the surgery is “cost effective” and should be performed.1 Cost-effectiveness analysis is based on the fundamental economic assumption that resources are scarce. Thus, if society were to spend its resources on treatments that are not cost effective, it would not be making the best use of its resources. The mind experiment that is often used to drive 335
34411_CH11_335_350
336
9/9/05
CHAPTER 11
11:25 AM
Page 336
COST-EFFECTIVENESS ANALYSIS
home this point is the case of the government agency that has a certain budget, say, $1 billion, and wants to maximize the health benefit to society from spending this budget. To spend efficiently, it would first conduct a CEA on all the possible treatments and rank them according to their ICERs. Then, it would allocate the $1 billion first to the treatment that produces a life year for the lowest cost, then the treatment that produces a life year for the next lowest cost, and so on until the entire $1 billion is spent. This approach would maximize the number of life years gained from this budget. If the government agency, in going down its list of procedures, came to the surgery for heart attack patients that achieved a life year at a cost of $48,000, it should spend the next portion of its budget on these surgeries only if there is no alternative that achieves a life year at a lower cost. If there were a cheaper alternative, spending on the surgery would represent an inefficient purchase in the sense that it could achieve the same number of life years at a lower cost by using the budget to purchase the cheaper alternative. In the jargon of economists, the “opportunity cost” of purchasing the surgery is too great. Government should spend on the cheaper alternative first, and then on the surgery, if there is any budget left to spend. Consumers make the same type of judgments in purchasing the goods and services that they purchase every day. They implicitly weigh the cost of consuming commodity A against its benefits, and then do the same for commodity B, commodity C, and so on. The commodity that produces a unit of benefit at the lowest cost is purchased first, then the next least costly, and so on, until the consumer’s income is spent. Consumers, however, are accountable only to themselves and do not need to conduct a formal study for every purchase. In contrast, the government is the agent of its citizens in a democracy and theoretically has a responsibility to spend the revenues it obtains from taxpayers in way that rationally maximizes the benefits to society. Thus, there is the need for a cost-effectiveness analysis to explicitly and transparently account for the costs and benefits, and thereby justify the spending decision on a certain treatment. The explicitness and transparency of CEAs make it easy for others to evaluate and critique government’s spending decisions.
TYPES OF COST-EFFECTIVENESS ANALYSES Three types of cost-effectiveness analyses differ according to the variable that is used to measure the effectiveness of the treatment. The various types of CEAs have different advantages and disadvantages.
34411_CH11_335_350
9/9/05
11:25 AM
Page 337
Types of Cost-Effectiveness Analyses
337
Cost-Effectiveness Analysis Although the generic term for all these types of evaluations is “costeffectiveness analysis,” the same term is also used to refer to a specific type of economic evaluation. In this type of analysis, effectiveness is measured in the natural units, such as blood pressure, cholesterol, forced expiratory volume in 1 second (FEV1), or life expectancy. Costs are measured in dollars, so the CEA would calculate the cost of, say, using new treatment A to achieve a one-unit reduction in blood pressure, compared with usual care. One disadvantage of a CEA is that the information produced by it is meaningful only relative to another similar CEA. That is, a researcher only knows that the $50 cost of using drug A to increase the FEV1 by one unit in a patient with chronic obstructive pulmonary disease (COPD) is cost effective if the alternative drug B costs $60 to achieve the same increase in effectiveness. Sometimes, this problem can be remedied by using the literature to “model” the effect of an “intermediate endpoint,” such as blood pressure or FEV1, by converting it into a more meaningful “final endpoint,” such as life years. For example, if one knew the effect of increasing FEV1 by one unit on life expectancy and also knew the value of a (qualityadjusted) life year, then a person could tell absolutely whether this drug was cost effective. Another disadvantage is that a treatment may affect many different effectiveness measures simultaneously, but there is no way to aggregate them in a CEA. For example, an exercise program may increase the maximum distance that a patient can walk and also increase FEV1 at the same time. With CEA, the ICERs for the two effectiveness outcomes would need to be calculated separately, because there is no mechanism in CEA for aggregating the measures of both these outcomes.
Cost-Benefit Analysis Cost-benefit analysis (CBA) is able to solve the aggregation problem by evaluating all the benefits in dollars and summing them. For example, it could theoretically determine the patient’s willingness to pay for being able to walk an additional unit of distance and for being able to exhale an extra unit of FEV1 capacity, and add these dollar values together to obtain a grand total value of the increases in effectiveness caused by the exercise program. Thus, CBA is able to add “apples and oranges” by first converting them into dollars.
34411_CH11_335_350
338
9/9/05
CHAPTER 11
11:25 AM
Page 338
COST-EFFECTIVENESS ANALYSIS
The evaluation of “health” in dollars also represents an important disadvantage of CBA because many aspects of health are difficult to evaluate. For example, could a researcher reliably determine the value of being able to exhale an extra unit of FEV1 capacity? Indeed, many individuals have such a visceral opposition to measuring health with dollars that they would refuse to do it. For example, many are fundamentally opposed to finding a dollar value for saving a human life or extending a life expectancy by 1 year. One advantage of CBAs is that they generate absolute information regarding the desirability of a treatment. For example, if a researcher determined that the dollar value of the benefits of a mammography outreach program for disadvantaged women exceeded the costs, then this treatment would be deemed cost effective. In other words, with CBAs, it is not necessary to construct an ICER comparing it to an alternative technology. Instead, it is simply sufficient to find a positive net benefit, such that the dollar value of benefits exceed the costs.
Cost-Utility Analysis In cost-utility analysis (CUA), effectiveness is measured in qualityadjusted life years (QALYs). QALYs are able to combine mortality and morbidity into one measure of health by weighting each of the years of life expectancy by the health-related quality of life. The quality of life is measured on a 0 to 1 scale, where 1 is the quality of life associated with perfect health and 0 is the “quality of life” associated with death. For example, a person who is expected to live an additional 10 years, with the first 5 of those in perfect health (quality of life 1) and the last 5 with a debilitating chronic disease (quality of life 0.5), would expect quality-adjusted life years equal to 7.5([5 1.0] [5 0.5]). The ability to combine mortality and morbidity in this way is an important, if controversial, advantage of CUA. Another important advantage of CUA is that it focuses on the ultimate endpoint of any health intervention: the length of life and the quality of that life. A disadvantage, however, is that it again is a relative measure. For example, one might know that a certain treatment generates an additional QALY for $50,000 compared with usual care, but cannot determine if this is desirable (that is, “cost effective”) unless one knows that an alternative treatment costs $60,000 per QALY saved.
34411_CH11_335_350
9/9/05
11:25 AM
Page 339
Costs
339
This problem has been circumvented in practice by using the value-oflife literature to determine how much consumers appear to value an additional quality-adjusted life year. For the past decade or so, $100,000 has been thought to be a reasonable estimate of the value of a QALY; however, the many recent studies have used larger values. Therefore, if treatment A generates an additional QALY for $50,000 and it is known that a QALY is worth $100,000, then the cost effectiveness of using treatment A is clear. In this chapter, we concentrate on CUAs because they are increasingly seen as the standard analysis in the economic evaluation of health treatments. Moreover, more has been written to attempt to standardize the methodology associated with CUAs than with CEAs or CBAs. In the next section, the costs to be included in CUAs are discussed.
COSTS The costs of a treatment typically have two components: (1) the cost of the treatment itself and (2) any subsequent cost savings, especially with regard to reduction of medical care expenditures that the patient with the disease in question experiences as a result of the effectiveness of the treatment. The latter can be referred to as “downstream” cost savings. There is controversy, however, regarding how broadly to cast the net for downstream costs. This is especially true if a treatment causes the patient to survive the original disease (e.g., cancer), only to die years later of another one (e.g., heart disease). Many think that cost increases due to caring for the patient with the second or “unrelated” disease should be attributed as a part of the cost of the original treatment. Some even go so far as to include the cost increases due to survival, both in terms of the additional purchases of consumer goods and services and the negative costs represented by a worker’s additional earnings. Those holding the latter view think that it is appropriate to include in the CUA any differences in costs between the state of the world with the treatment and without it. Nyman (2004) has argued, however, that it would be inappropriate to include the cost of survival consumption if the benefits from this consumption were not also included in the measure of the quality of life used to construct the QALY. Because the benefits of survival consumption are not included in any of the measures of quality of life that are currently used, the inclusion of the cost of survivor consumption (or of survivor earnings) in the ICER would be inconsistent. Moreover, it would bias the CUA against finding that treat-
34411_CH11_335_350
340
9/9/05
CHAPTER 11
11:25 AM
Page 340
COST-EFFECTIVENESS ANALYSIS
ments for diseases of the elderly and poor are cost effective. This remains a controversial area (Meltzer, 1997; Garber & Phelps, 1997). The appropriate costs to include in a CUA also depend on the stance of the study. The stance reflects from whose perspective the study is conducted. Most CUAs adopt a societal stance, which implies that all costs are candidates for inclusion, regardless of who incurs them, as long as they are consistent with the limited measure of quality of life represented by QALY, and as long as they represent real resources that have been used up. In other words, they cannot simply represent money that has been transferred from a taxpayer to another citizen. The societal stance is most consistent with economic theory (Mishan, 1971) and gives the most comprehensive view of the cost effectiveness of a treatment. Other stances are not as comprehensive. For example, a CUA done from the perspective of the health plan would not count the costs that the patient incurs in traveling to the physician for treatment. These costs are real and would be attributable to the treatment if the CUA were conducted from a societal perspective, but would be disregarded from a health plan stance because the health plan does not incur them. Other possible stances include those of the patient, the government agency or program, and the provider. The cost of the treatment itself should reflect the treatment as it would be implemented in its full-fledged version. It would include, for example, the costs of any new construction that would be required, and it would include the lower costs derived from economies of scale. The costs of the treatment would not, however, include any costs that were due to conducting the cost-utility analysis itself. Thus, it may be necessary to specify a treatment in the actual CUA that has costs that are substantially different from those of the treatment that was used in the study that forms the basis of the CUA. Both the treatment and the downstream cost savings are likely to include the use of medical care. Therefore, the direct and indirect medical care costs should be included. Direct costs include the costs of hospital care, outpatient care, physician visits, nursing home stays, hospice stays, pharmaceuticals, durable medical equipment, physical therapy, occupational therapy, and home health care aide services. Indirect costs include the costs of patient travel to receive care, such as bus fares, automobile mileage, and hotel bills if it is necessary to stay overnight in another town to receive care. They also include the costs of any informal care that takes the place of purchased care and include the costs of the patient’s time traveling to and from the care venue and receiving care. The decision about including indi-
34411_CH11_335_350
9/9/05
11:25 AM
Page 341
Costs
341
rect costs depends on the stance of the analysis. In any event, the costs of the patient’s time spent in recuperation and convalescence are not to be included in these costs. In an attempt to standardize CUA, it has been recommended that these be included as part of the quality of life measure associated with the treatment of the disease instead (Gold, Siegel, Russell, & Weinstein, 1996). This recommendation is controversial. When identifying which costs to include in the analysis, it is important to recognize that all costs can be disaggregated into the product of the number of services utilized and the unit prices of a service. For example, a cost of $500 in physician visits can come from 10 visits at a unit cost of $50 per visit. This is important because economic theory teaches that the true cost of a good or service is best represented by its “marginal cost,” which is in turn the value of what those same resources could have produced in their next most valuable use. Economic theory further teaches that the unit price that is observed in a competitive market in a competitive economy is the marginal cost. The problem for health care services is that a competitive market hardly ever exists. Therefore, to use the actual total costs that are obtained from the health provider would likely overstate the true costs. A better approach is to disaggregate costs by recording only the number of services of the various types (codes are important here) that are utilized, and then apply to them standardized unit “prices,” which better reflect marginal costs, and calculate the total cost. What unit prices are closest to marginal costs? This is also an unresolved question because as mentioned, there is probably no aspect of the medical sector of the economy that can be considered perfectly competitive. As a second best alternative, Medicare prices could be used in CUAs in the United States, just as the fee tariffs from the national health insurance programs are typically used in foreign CUAs. For one thing, some Medicare prices have been set according to an estimation of their marginal costs (for example, the Resource-Based Relative Value Scale that is behind Medicare’s Prospective Physician Fee Schedule). For another, even though they might not reflect marginal costs, Medicare prices are perhaps the best known fee tariff used in the United States. Thus, Medicare fees could be used as unit prices in U.S. CUAs. If the costs are to be incurred over many years, discounting should be considered. Discounting is a standard financial application used to bring all future costs to levels that are comparable in the present. Discounting traditionally has had two justifications. First, costs invested in the present can earn a real rate of return, making any future expenditure less costly. Thus,
34411_CH11_335_350
342
9/9/05
CHAPTER 11
11:25 AM
Page 342
COST-EFFECTIVENESS ANALYSIS
future costs should be discounted to account for this alternative rate of return. Second, people simply prefer money in the present to money in the future, perhaps because of the human condition: One could die and not be around to enjoy the money in the future. This also means that future costs or benefits are not worth as much as present ones. Some controversy surrounds discounting. For example, with discounting, any costs incurred far enough into the future would be negligible. For example, one could bury nuclear waste that would kill many thousands of people 200 years from now, but with a 5 percent discount rate, today’s discounted number of deaths would be negligible. Also, although an individual person can die and has a preference for present dollars as a result, is it reasonable to attribute the same preferences to government, especially since the government is charged with taking into account the preferences of successive generations of citizens? A controversy also surrounds whether QALYs saved in the future should also be discounted. At present, the recommendation is to discount both costs and QALYs at a 3 percent and a 5 percent rate (Gold et al., 1996). Future costs and QALYs should also be evaluated with a 0 percent discount rate for completeness.
HEALTH-RELATED QUALITY OF LIFE The effectiveness measure used in CUA is QALYs. Thus, the effectiveness in a CUA would be determined by the number of life years generated by treatment A, evaluating each by their quality-of-life weights and, compared to the number of life years generated by usual care, evaluating each according to their quality-of-life weight. The difference in quality-adjusted life years would measure the effectiveness of treatment A compared to usual care. The quality of life in a QALY is limited to “health-related” aspects of life. Because of this, it is often referred to as health-related quality of life or HRQOL. HRQOL typically does not include many of the aspects of life that individuals associate with quality. For example, HRQOL does not include anything related to the environment (such as air quality or traffic congestion), society (the crime rate), culture (availability of art museums, symphony orchestras, theater), entertainment (restaurants, bars), economic factors (income distribution, unemployment, consumption, earnings), or spiritual factors (church attendance, community spirit). Thus, it is limited in scope.
34411_CH11_335_350
9/9/05
11:25 AM
Page 343
Health-Related Quality of Life
343
HRQOL is determined by a two-step process: First, the various health states to be evaluated must be specified, and second, the health states must be evaluated. Although many approaches for specifying health states exist, typically they involve first determining the dimensions of health status that are important to HRQOL. For example, many would agree that the degree to which a person is free of pain is a dimension of health status that would be important to HRQOL. Other dimensions might include ones related to anxiety or depression, mobility, self-care, sightedness, ability to hear, ability to work, and so on. Once the dimensions are identified, then it is necessary to determine the levels that would distinguish health states within these dimensions. Therefore, for the dimension of mobility, being able to walk 1000 yards would be a different health state than being able to walk only 500 yards or 100 yards. These levels, however, would track monotonically into HRQOL because one can assume that the HRQOL for being able to walk 1000 yards is better than that of 500 yards, which exceeds that of 100 yards. The dimensions and levels are then converted into a questionnaire that can be used to determine the constellation of responses that would characterize a person’s health state. Three approaches can be used to evaluate health states: 1. The standard gamble (SG) 2. The time trade-off (TTO) 3. The visual analogue scale (VAS) To illustrate these approaches, consider the respondent who is asked to think about the health state of a person who is in excellent health with regard to all dimensions considered, except that she is blind. Thus, “blindness” would characterize the health state in question. With the SG, the researcher would tell the respondent who is evaluating this health state that there is a new treatment that results in a complete cure for blindness, but that this treatment has a chance of instantaneous and painless death associated with it. In considering this health state of blindness, the researcher would ask the respondent: At what probability of death would the respondent be indifferent to trying the cure or remaining blind? The HRQOL associated with blindness would then be 1 minus the respondent’s declared probability of death. For example, an answer of a 0.2 probability of death would indicate an HRQOL of 0.8(1 – 0.2). The worse the HRQOL that the respondent associates with blindness, the greater the probability of death the respondent would tolerate.
34411_CH11_335_350
344
9/9/05
CHAPTER 11
11:25 AM
Page 344
COST-EFFECTIVENESS ANALYSIS
With the TTO, the researcher first determines how long the respondent would expect to live. For example, say, the respondent expects to live 20 years. Then the researcher asks the respondent if he were blind, how many of those 20 years he would give up for perfect health during the remaining years, then immediate and painless death. The HRQOL would be determined by subtracting the respondent’s declared number of years sacrificed from 20 and dividing this remainder by 20. For example, an answer of 4 years would indicate an HRQOL of 0.8([20 4]/20). The worse the HRQOL that the respondent associates with blindness, the greater the number of years blind he would be willing to give up. With the VAS, the researcher presents the respondent with a graphical aid in the form of a thermometer. For example, a VAS might be 10 centimeters long, with each of the 10 centimeters and 100 millimeters marked, and with a “1” at the top and a legend indicating that the top of the thermometer is equivalent to “best imaginable health” and a “0” at the other end indicating that the bottom of the thermometer is equivalent to “worst imaginable health.” To the side, a box would indicate the health state of “blindness” and the researcher would ask the respondent who is contemplating what it would be like to be blind to draw a line from the box to the thermometer to indicate where the quality of life associated with blindness would be, relative to the health states at the top and bottom of the thermometer. Finally, because it is possible that the “worst imaginable health” state could be a health state worse than death and because it is necessary to calibrate HRQOL with 0 being the HRQOL weight associated with death (otherwise a person could die and have a positive quality of life for the indefinite future), the respondent would also be asked to indicate where the health state “death” would lie on this thermometer in a similar manner. The HRQOL measure would be the distance on the VAS between the blindness and death, divided by the distance between the top of the VAS and death. For example, if the respondent indicated that “blindness” is 0.8 of the way up from the bottom of the VAS and death is 0.2 of the way up from the bottom, then the HRQOL for blindness is 0.75([0.8 0.2]/[1 0.2]). The worse the HRQOL that the respondent associates with blindness, the smaller the distance between blindness and death on the VAS. Figure 11–1 shows the VAS from the EuroQol (EQ-5D). The VAS is used here to determine the HRQOL of the respondent’s health state. In a subsequent question and page of the survey, the respondent is asked to return to the VAS and identify the HRQOL associated with death.
34411_CH11_335_350
9/9/05
11:25 AM
Page 345
Health-Related Quality of Life
best imaginable health state 100 9 0
8 0
7 0
6 0
Rate your current state of health
5 0
4 0
3 0
2 0
1 0 worst imaginable health state Figure 11–1 The VAS Thermometer from the EuroQol
345
34411_CH11_335_350
346
9/9/05
CHAPTER 11
11:25 AM
Page 346
COST-EFFECTIVENESS ANALYSIS
A number of health status questionnaires have been developed. The three health status questionnaires that are most commonly used in the United States are 1. Quality of Well-Being Index (QWB) (Kaplan, Ganiats, & Sieber, 1996) 2. Health Utilities Index–Mark III (HUI-III) (Feeny, Forlong, Boyle, & Torrance, 1995) 3. EuroQol (EQ-5D) (EuroQol Group, 1990) Each of these has been calibrated using one or more of the three preference elicitation procedures just described (SG, TTO, or VAS). Thus, in order to determine the HRQOL of a patient, a researcher would simply ask the patient to fill out the questionnaire, and then either look up or use a formula to calculate the HRQOL level that is associated with the patient’s set of responses. Alternatively, the researcher could use the SG, TTO, or VAS of a patient to evaluate the health state directly. Another commonly used health status questionnaire is the Short Form 36 (SF-36). This questionnaire was not developed to be used to construct HRQOL values. Recently, however, a subset of the responses within this questionnaire was identified as being amenable to HRQOL conversion. This subset of responses, the SF-6D, has been calibrated and can also be used to obtain HRQOL weights (Brazier, Usherwood, Harper, & Thomas, 1998). Many CUAs already exist with calculated HRQOL weights for specific diagnoses. These weights can be obtained by referring to the CUA studies themselves or by referring to a number of studies that list the various weights associated with certain diagnoses as found in the literature (e.g., Tengs and Wallace, 2000). A HYPOTHETICAL ILLUSTRATION Consider a program to reduce falls in the elderly. A group of elderly— representing the treatment arm of the study—receives periodic visits from nurses who conduct strength-training exercises with them. A similar group of elderly—the control arm—receives visits from nurses but no strength training. The outcome of interest is the difference in falls between the treatment and control groups. The costs of the intervention is the cost of the nurse’s time spent training the elderly participant, the nurse’s travel costs, and the strength-training equipment. The costs of the control group do not include the nurse’s time
34411_CH11_335_350
9/9/05
11:25 AM
Page 347
Summary
347
because that attention was not intended to produce a benefit. The downstream cost savings would be the difference in medical care costs associated with falls, assuming that the strength training is effective in reducing falls. Although one assumes a societal stance, to make things simple, one further assumes that the indirect costs are the same for both groups. The study finds that among the treatment elderly, there are 40 fewer falls. For each person, the EQ-5D is used to assess the participants’ HRQOL weights throughout the year of follow-up in the study. Assume that the study shows that for those who do not fall, the average HRQOL for the year is 0.9, and for those who fall, the average HRQOL over the year is 0.7. Therefore, each fall is associated with a reduction of QALYs of 0.2, which also represents the QALYs gained from a fall averted. The cost of the intervention itself in $400,000, but because of the effectiveness of the intervention in averting falls, $200,000 in downstream medical care costs are saved. Therefore, the net cost of the treatment is $200,000. The ICER is calculated comparing the difference between the treatment group and the control group in costs and QALYs as: (CT CC)/(ET EC ) $200,000/(40 0.2 QALYs) $200,000/8 QALYs $25,000/QALY
where C is cost, E is effectiveness measured in QALYs, and the subscripts represent treatment and control. Because a reasonable value of a QALY from the value of life literature is $100,000, this falls prevention program for the elderly is deemed to be cost effective. SUMMARY A number of sources are available for further reading; however, the most important is the book Cost-effectiveness in Health and Medicine by Gold and associates (1996), which represents the recommendations of the Panel on Cost-Effectiveness in Health and Medicine, a body convened by the U.S. Public Health Service and charged with standardizing the conduct of cost-utility analyses. The “reference case” described in this volume represents the ideal CUA at the time the recommendations were made. The book, however, acknowledges many controversies. Another excellent text, more from the European perspective, is Methods for the Economic Evaluation of Health Care Programmes by Drummond
34411_CH11_335_350
348
9/9/05
CHAPTER 11
11:25 AM
Page 348
COST-EFFECTIVENESS ANALYSIS
and colleagues (1997). This is the second edition of what had been the standard “how-to” text in cost-effectiveness analysis for many years. It contains information about not only CUA, but CEAs and CBAs and is written by the pioneers in this field. A more recent and updated offering is the volume, Economic Evaluation in Health Care: Merging Theory with Practice, edited by Drummond and McGuire (2001). It contains a series of essays that are more up-to-date, but also more detailed discussions of the theoretical issues in this field. Sloan (1996) also has a valuable edited volume (Valuing Health Care: Costs, Benefits, and Effectiveness of Pharmaceuticals and Other Medical Technologies) that does much the same thing, but from an American perspective. REFERENCES Brazier, J., Usherwood, T., Harper, R., & Thomas, K. (1998). Deriving a preference-based single index from the UK SF-36 Health Survey. Journal of Clinical Epidemiology, 51, 1115–1128. Drummond, M.F., & McGuire, A. (Eds.). (2001). Economic evaluation in health care: Merging theory with practice. New York: Oxford University Press. Drummond, M.F., O’Brien, B., Stoddart, G.L., & Torrence, G.W. (1997). Methods for the economic evaluation of health care programmes (2nd ed.). New York: Oxford University Press. EuroQol Group. (1990). EuroQol: A new facility for the measurement of health-related quality of life. Health Policy, 16, 199–208. Feeny, D., Furlong, W., Boyle, M., & Torrance, G.W. (1995). Multi-attribute health status classification systems. PharmicoEconomics, 7, 490–502. Garber, A.M., & Phelps, C.E. (1997). Economic foundations of cost-effectiveness analysis. Journal of Health Economics, 16,1–31. Gold, M.R., Siegel, J.E., Russell, L.B., & Weinstein, M.C. (Eds.). (1996). Cost-effectiveness in health and medicine. New York: Oxford University Press. Kaplan, R.M., Ganiats, T.G., & Sieber, W.J. (1996). Quality of well-being scale, selfadministered, QWB-SA, V1.04. Meltzer, D. (1997). Accounting for future costs in medical cost-effectiveness analysis. Journal of Health Economics, 16, 33–64. Mishan, E.J. (1971). Cost-benefit analysis: An introduction. New York: Praeger. Nyman, J.A. (2004). Should the consumption of survivors be included as a cost in costutility analysis? Health Economics, 13, 417–427. Sloan, F.A. (Ed.). (1996). Valuing health care: Costs, benefits, and effectiveness of pharmaceuticals and other medical technologies. Cambridge, MA: Cambridge University Press.
34411_CH11_335_350
9/9/05
11:25 AM
Page 349
Note
349
Tengs, T.O, & Wallace, A. (2000). One thousand health-related quality-of-life estimates. Medical Care, 38, 583–637.
NOTES 1. Two (or more) treatments of the same disease are evaluated with an ICER. The evaluation of the drug compared to no treatment, or the evaluation of surgery compared with no treatment, is referred to as the average cost-effectiveness ratio (ACER) and is calculated by dividing the costs of the drug or the surgery by its effectiveness. Comparison of the ACERs shows that the drug is a less costly way to obtain a life year than surgery, so it is assumed that the first level of the decision is to proceed with the use of the drug as treatment. The second decision is whether to proceed with the use surgery instead, which will require the ICER calculation. Note that the ACER is just a special case of the ICER.
34411_CH11_335_350
9/9/05
11:25 AM
Page 350
34411_CH12_351_370
9/9/05
10:55 AM
Page 351
Part V Research Issues
34411_CH12_351_370
9/9/05
10:55 AM
Page 352
34411_CH12_351_370
9/9/05
10:55 AM
Page 353
12 Implementing Outcomes Research in Clinical Settings David M. Radosevich
INTRODUCTION Implementing outcomes research in the clinical setting is easier said than done. Without careful planning, organizational readiness, and allocation of necessary resources, the investigator is embarking on the path to failure. At its most basic level, conducting research in clinical settings is like mixing oil and water. Each activity is conducted for different purposes. Data needed for clinical care may or may not be the same data necessary for research. Frequently, data is not collected in a standard manner that supports research or is easily retrievable for research. There is no perfect outcomes research study anymore than there is a perfect epidemiological study. All research involves trade-offs between stated objectives, possible research study designs, and available resources in terms of staff and budget. Like many laudable goals, “the devil is in the details.” Successful research in clinical settings generally deals head-on with strategic issues faced in conducting all research and minimizes disruptions to clinical care. There are no hard and fast rules for how this is accomplished. Instead, this chapter provides a framework for implementing outcomes research in clinical settings. The framework is summarized in a Structured Guide for Implementing Outcomes Research in Clinical Settings, which appears in Appendix 12A. ORGANIZATIONAL CHARACTERISTICS To begin with, certain organizational characteristics support outcomes research. Successful implementation of outcomes research in clinical settings depends on the following: 353
34411_CH12_351_370
354
9/9/05
CHAPTER 12
10:55 AM
Page 354
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
• A clinical environmental committed to conducting outcomes research • Clear leadership and support for the research enterprise • The ability to leverage the research with other activities in the clinical setting • Necessary resources • A well-defined planning process Clinical Environment Commitment to conducting outcomes research depends heavily on organizational goals and the culture of the clinical setting. At the very least, the clinical environment needs to be committed to the importance of research and the knowledge gained through research to enhance patient care. Successful clinical research organizations have adopted a commitment to quality and an appreciation for the patient’s perspective in the delivery of health care services. This demands some level of expertise in the methods necessary to conduct credible research. Generally, these include measurement along all three dimensions of structure, process, and outcomes. (Donabedian, 1988, 1989). Clinical environments that are willing to learn and entertain new ideas are more successful in implementing outcomes research. This mind-set seems to increase the sense of security and trust internally and contributes to a competitive nimbleness within the health care environment. Clinical settings that learn faster hold advantages over stagnant organizations that are unwilling to listen to their patients. Leadership Successful leadership in outcomes research involves three aspects. First, there needs to be a respected person who acts as the research lead. Much research is driven by a strong investigator with a clear understanding of the study goals and methods. Leadership by committee diffuses responsibilities. Although multidisciplinary efforts are required in much research, a strong lead is critical to keeping team members on task and the project on time. A second aspect of leadership involves the support of senior management. If they convey the message to the organization that research is important, the support of others is gained and the necessary resources are mobilized.
34411_CH12_351_370
9/9/05
10:55 AM
Page 355
Organizational Characteristics
355
A strong principal investigator and the support of senior management are necessary but not sufficient to implementing a successful outcomes research study in the clinical setting. The support of other key individuals in the organization is critical. Nothing stops a project quicker than someone subverting the good work. These individuals will vary by clinical setting.
Leverage the Research Coordinating outcomes research in the clinical setting involves managing a number of interests and activities that frequently compete with one another. Frequently, outcomes research can be leveraged by coordinating with other research activities, pooling resources, or joining other organizations. One of the easiest ways to leverage research is by coordinating with other research activities. Often the outcomes research involves the same group of patients, so adding additional data or broadening inclusion criteria minimizes the duplication of outcomes measures and reduces the burden to the study subjects. Sharing personnel and other resources is a useful strategy for improving the efficiency of outcomes research. Certain types of resources make good sense to share: for example, database management activities, survey centers, and analytic resources. Unfortunately, most organizations find it easier to share hardware than to share personnel. However, if there is a clear designation of the time spent by staff on research, it is a simple accounting function to charge that time to the appropriate research project. Contract research organizations have been doing this for years. Another strategy for leveraging research is to participate in multiorganizational consortia. Coordinating centers for multisite studies perform critical functions such as database development, training, and outcomes analysis for the participants. In addition, these consortia are critical in standardizing data collection instruments and methods and the development of study protocols.
Resources Adequate resources are critical to the success of any research enterprise: the collection, the interpretations, and use of research data. Planning for outcomes research involves the careful planning for the resources required.
34411_CH12_351_370
356
9/9/05
CHAPTER 12
10:55 AM
Page 356
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
An important distinction to make when considering resources are start-up costs versus operations costs. Start-up costs are similar to research and development costs. Although these costs might appear wasted within the overall scope of the project, they increase the efficiency of study operations. Start-up costs in research are not “sunk costs,” but rather a critical part of the planning process (Davies, Doyle, David, & Rutt, 1993).What start-up costs are involved? It begins with planning and includes the following: • • • • • • • •
Literature review Networking and organizational assessment Formulating the research goals and objectives Creating the protocol for conduct of the study and data collection Hiring staff and purchasing hardware and software Creating data collection forms Designing the research database and analytical framework Contracting, if necessary, and regulatory activities such as human subjects protection application
Staff directly involved in the outcomes research requires education and training. These might be considered investments in “peopleware” to produce a key resource. Operations costs are distinct from startup costs. These include the following: personnel, telecommunications, workstation support, materials, printing, and postage. Personnel costs address a variety of activities that are critical to implementing outcomes research.
IMPLEMENTATION RESPONSIBILITIES Responsibilities required for successful implementation of outcomes research include: • • • • • •
Study leadership Study management Study coordination Database design Data management Data analysis
34411_CH12_351_370
9/9/05
10:55 AM
Page 357
Implementation Responsibilities
357
These responsibilities may be performed by one person or may require a number of people, depending on the scope of the research and available skills.
Study Leadership To be successful, at least one respected clinical leader must provide a vision for the study. In most clinical research, this is provided by the principal investigator (PI) of the study. Leadership implies more than a name or endorsement for the work. In too many studies, PI facetiously means “practically invisible.” The leader generates project support, organizes activities, and allocates time and resources required for a successful study. All successful studies have a study lead. Leadership will need to work in close collaboration with study management to build relations with clinicians in the clinical setting. The research should minimize interference with clinical activities, so that questions over patient ownership and access are addressed before conflicts escalate. Frequently, conducting outcomes in clinical settings involves work flow redesign so that data collection is built into existing clinical activities.
Study Management All studies require an individual who is responsible for the day-to-day management of the research. A study manager should have the following characteristics: • The ability to act as a liaison between clinical and administrative personnel and technical personnel (i.e., those involved in the day-to-day activities of collecting data, managing data, and analysis) • The breadth of skills necessary to understand the scientific and application sides of outcomes assessment, as well as an appreciation for the political and cultural forces within the organization • A generalist background; versed in the fields of research, information systems, database design, statistics, budgeting, forecasting, and reporting and presentation of information • The ability to synthesize feedback from reports and have communication skills needed to present results to various groups within and outside the organization
34411_CH12_351_370
358
9/9/05
10:55 AM
CHAPTER 12
Page 358
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
• A keen awareness of internal and external developments, even to the point of anticipating potential barriers to the study • An extremely organized nature; the ability to multitask Critical study management responsibilities include: • • • • • • •
Develop and implement the data collection methodology Select, train, and supervise personnel Create and supervise data collection protocols Manage the data collection schedule Analyze or supervise data analysis Prepare, interpret, and disseminate study reports Manage the budget
In most outcomes research, a study manager is responsible for the overall conduct of the work and the day-to-day operations, including the coordination of activities, team meetings, and the ability to make midcourse corrections when there are changes in the study protocol.
Study Coordination At the front of any outcome study are the individuals directly responsible for data collection. If the size and scope of the research are small with few sites collecting data, one person may be sufficient to fill this role. If the scope of research implementation is broader, however, it may be more efficient to have one or more study coordinators; possibly a coordinator designated for each site participating in the research. The role of the study coordinator is extremely important, yet it is often overlooked. This person must recruit and retain study subjects. If the coordinator is unfamiliar with the study’s purpose and is not kept informed regarding the study’s progress and problems, he or she may lack the insight and enthusiasm to participate effectively in the research. The coordination function is the most valuable resource for identifying methodological glitches in the study. In general, the responsibilities of study coordination includes the following: • Obtain and produce data collection forms and necessary supplies • Prepare and code forms for distribution
34411_CH12_351_370
9/9/05
10:55 AM
Page 359
Implementation Responsibilities
359
• Identify eligible subjects using a standard operating procedure (SOP) manual • Administer data collection forms (if necessary) • Maintain contact with study subjects and keep them interested in staying with the study • Maintain a tally of completed forms • Edit data collection forms for completeness, accuracy, and consistency • Communicate problems or errors with the project manager and data manager • Mail, transmit, or transport completed forms to the data center (if needed)
Database Design Unless a researcher purchases an off-the-shelf software package, someone will have to design a database for entry, storage, and retrieval of the data. Although this undoubtedly is a crucial role, it is often ignored or downplayed. Frequently, it is never included in the final budget or it is dropped from the budget. The following are important considerations and tasks for the individual who will be responsible for database design: • The individual must have experience and working knowledge of not only the database software, but also the software that will be used for reporting and analysis (spreadsheet, presentation, and statistical packages). Knowledge of the reporting and analysis software is crucial for determining the proper platforms within which to import and export data. • Besides designing the database, this person must create the data entry screens (if manual entry will be used) and embed edit and range checks within the data entry screens. • Another important task is designing the reports that will be used to track the data collection process, monitor data quality, remind program coordinators to follow up on subjects, profile subjects, and display outcomes.
34411_CH12_351_370
360
9/9/05
CHAPTER 12
10:55 AM
Page 360
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
Data Management The role of data management involves the day-to-day monitoring of data going in and coming out of the database. This includes the following activities: • Preparing health surveys and forms for data entry. Frequently, critical data is omitted. The data manager assures the collection of these critical data, for example, the subject’s date of birth and gender. • Undertaking or supervising data entry. Data entry may be reliably done by clerical staff or trained personnel. Data managers periodically prepare reports detailing data quality and consistency. Reports should summarize such topics as response rates, frequency of missing data, inconsistent data and the use of proxy or substitute respondents. Reports detailing the progress of the data collection efforts should also be included. These reports are typically needed by the study coordinators and the study manager for managing the research. These data errors must be detected in a timely manner in order to correct them while the study is in progress. • Maintaining the integrity of the database, which includes backing up the database each time additions or changes have been made. If paper forms are used for data collection, this person is charged with archiving the forms in a fashion that permits easy retrieval if verification and cross-check of the data are required. • Ensuring data confidentiality by enforcing the rules of data access set forth by the study and Health Insurance Portability and Accountability Act (HIPAA) guidelines.
Data Analysis As noted in the chapter on design, analysis is too often overlooked during planning. Conventional wisdom is that because reporting results of analysis occurs last, analysis need not be considered until the data is collected. However, plans for analysis should be considered during the design phase and based on the questions or objectives of the study. In fact, most analysts require their involvement in planning as a condition for directing the analysis. Study design and analysis are inexorably linked. In the words of the analyst:
34411_CH12_351_370
9/9/05
10:55 AM
Page 361
Implementation Responsibilities
361
If you want me on the plane when it lands, I need to be there for the takeoff. Data analysts make significant contributions to the overall design and methodology used for data collection. Their expertise in measurement, database design, and statistical analysis will aid in ensuring the proper study design for acquiring relevant and correct data to answer the study questions. Statisticians should provide insights into: 1. Sample size calculation. Before undertaking a study, the likelihood of detecting a true relationship needs to be assessed. This is referred to as adequate statistical power and is discussed in greater depth in the chapter on study designs. 2. The analyst assists in planning the sampling strategy because this directly affects how subject variation is determined. 3. The analyst works closely with the data manager in the design of the database and the formatting of data for analysis. Quality checks of the data should always precede a description of the data. Response distributions, frequencies, and measures of central tendency (means, modes, medians) should be generated starting early in the data collection phases. Analysts check data values for outliers, values out of range and inconsistent responses. Any of these discrepancies need to be verified and cross-checked against the original records. Next analysts, create conceptually meaningful groupings for data, such as the calculation of age from a date of birth or body mass index using height and weight (Last, 1995). Except for standard reports that summarize data quality, detail the data collection process, and describe the data, an expert may be needed to complete the analysis necessary to answer the questions important to the outcomes study. Finally, participating clinicians should receive some feedback regarding the results of the study. The analysis needs to structure the findings for the audience. Clinicians are customarily a nontechnical group who lack the sophistication of clinical researchers. The presentation of results needs to be structured for the audience (i.e., clinical practitioners, researcher, policymaker). There is never one right way to present the data because final analysis will be driven by the question and consumer of the results.
34411_CH12_351_370
362
9/9/05
CHAPTER 12
10:55 AM
Page 362
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
REGULATORY DEMANDS Regulatory demands are increasingly assuming greater importance in outcomes research studies. Institutional review boards (IRBs) are organized within universities and health care organizations to review and monitor research involving human subjects. An IRB review assures that the subjects’ rights and welfare are protected (U.S. Food and Drug Administration, 1998). Before the PI begins data collection, the IRB must have approved the research and made all required modifications to the study. IRB approval can take weeks and sometimes months to secure. As a consequence, approval for human subjects research needs to occur early in the project planning process. Researchers must assure that medical privacy protections have been made under the HIPAA privacy rule (U.S. Department of Health and Human Services, 2004). Outcomes researchers must be granted written permission by subjects to use protected health information (PHI) such as names, dates of birth, social security numbers, or date of surgery. The privacy rule is intended to provide ways to access vital data needed for research in a manner that protects the privacy of the research subject. The reader is referred to the National Institutes of Health Web site for a comprehensive discussion of the rules as they apply to outcomes research. The address for the Web site is http://privacyruleandresearch.nih.gov/. SUMMARY Successful implementation of outcomes research involves assignment of critical responsibilities: leadership, management, coordination, database design, data management, and analysis. Depending on the scope and scale of the research, an individual may be responsible for the full range of activities, or highly specialized roles may evolve depending on the size and scope of the project. Resource planning is critical because outcomes research is a complex, data-intensive process. A systematic planning process is recommended. A self-analysis is a good place to start using something like the Structured Guide for Implementing Health Outcomes Research in Clinical Settings found in Appendix 12A. REFERENCES Davies, A.R., Doyle, M.A.T., David, L., & Rutt, W. (1993). Outcomes assessment: Issues in implementation. Boston: New England Medical Center.
34411_CH12_351_370
9/9/05
10:55 AM
Page 363
References
363
Donabedian, A. (1988). The quality of care: How can it be assessed? Journal of the American Medical Association, 260(12), 1743–1748. Donabedian, A. (1989). The end results of health care: Ernest Codman’s contribution to quality assessment and beyond. Milbank Quarterly, 67(2), 233–256. Feinstein, A.R. (1987). Clinimetrics. New Haven, CT: Yale University Press. Kirshner, B., & Guyatt, G. (1985). A methodological framework for assessing health indices. Journal of Chronic Diseases, 38(1), 27–36. Last, J.M. (Ed.). (1995). A dictionary of epidemiology (3rd ed.). New York: Oxford University Press. McHorney, C.A. (1998). Health status assessment methods for adults: Past accomplishments and future challenges. Annual Review of Public Health, 20, 309–335. U.S. Department of Health and Human Services. (2004). Medical privacy—National standards to protect the privacy of personal health information. Retrieved February 14, 2005, from www.hhs.gov/ocr/hipaa. U.S. Food and Drug Administration. (1998). Guidance for institutional review boards and clinical investigators. Retrieved February 14, 2005, from www.fda.gov/oc/ohrt/irbs/ default.htm.
34411_CH12_351_370
9/9/05
10:55 AM
Page 364
Appendix 12A Structured Guide for Implementing Health Outcomes Research in Clinical Settings Name: ________________________
Date: ____________________
Organization: __________________
Phone: ____________________
Project Name: ______________________________________________
The Structured Guide for Implementing Health Outcomes Research in Clinical Settings poses questions that need to be considered before undertaking outcomes research in a clinical setting. Write a brief response to each question under the domains listed. Careful thought should be given to the issues underlying these questions before implementing a health outcomes research study. Clinical Environment 1. What is the organization’s and clinic’s commitment to research?
2. Is there recognition within the organization of the value of the patient’s perspective?
364
34411_CH12_351_370
9/9/05
10:55 AM
Page 365
Appendix 12A
365
3. Does the organization show a willingness to change? . . . to try new ideas? . . . to learn? . . . to take risks?
4. What is the level of trust within the organization? Is there significant uncertainty with the organization? What is the organization’s level of tolerance for mistakes?
Leadership 1. Is there a respected person (physician leader) willing to lead the research?
2. Is senior management supportive of outcomes assessment?
3. Are others within the organization (midlevel managers, support staff) willing to support an outcomes assessment?
Leverage 1. Are there competing research initiatives in the organization?
2. What opportunities are there for sharing resources (support staff, expertise, hardware, software) in the organization?
3. Are there external alliances that can be formed to support outcomes assessment?
Planning and Implementation 1. How do you plan to use the data from the outcomes assessment?
34411_CH12_351_370
9/9/05
CHAPTER 12
366
10:55 AM
Page 366
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
2. List the primary questions (no more than three) you expect to answer from the outcomes assessment.
a.
b.
c.
3. Who (patients, enrollees, subjects) will participate in the outcomes assessment? What inclusion and exclusion criteria will be used to determine participant eligibility?
4. What outcomes are of primary interest?
5. How will these outcomes be measured? In collecting data, will you be using an existing instrument or developing a new instrument?
a. Does the instrument measure the content needed to answer the primary question? What is the scope of the measure? Is the measure sensible for the application?
b. What is the fidelity of the measure?
c. Has the measure been shown to be reliable?
d. Has the measure been shown to be valid for this application?
34411_CH12_351_370
9/9/05
10:55 AM
Page 367
Appendix 12A
367
e. What is the relevance of responsiveness for this application? How responsive is the measure to change in health status?
f. What is the respondent and investigator burden of the instrument?
g. Is the measure suited for the implementation design?
6. How many subjects will participate in the outcomes assessment? Does this represent an adequate number for answering the primary question?
7. Describe the timeline for implementation. When do you plan to begin implementation? What is the time interval for subject recruitment? If you are planning for the continuous observation of a cohort, what is the timing for follow-up measurement?
8. What resources do you have available for the study?
a. Personnel
b. Hardware (computers, printers, optical scanners)
c. Software (word-processing database, statistical package, spreadsheet, presentation package)
d. Funding support (internal and external funding—specify)
34411_CH12_351_370
368
9/9/05
CHAPTER 12
10:55 AM
Page 368
IMPLEMENTING OUTCOMES RESEARCH IN CLINICAL SETTINGS
9. Who will assume primary responsibility for critical activities during the implementation? Indicate names(s) and department(s), as well as external support, consultants, vendors, and contract research groups.
a. Project design and protocol development
b. Project management and coordination
c. Instrument (questionnaire) development
d. Data collection process
e. Data management (data entry, database management)
f. Analysis and reporting
g. Dissemination and interpretation of results
10. Who will have access to the data?
11. Who is the primary consumer of the study findings? Who will have access to the results of the study (e.g., other outcomes researchers, providers, department managers, administrators, purchasers, health plans, patients, payers)?
34411_CH12_351_370
9/9/05
10:55 AM
Page 369
Appendix 12A
Regulatory Demands 1. Human subjects protections and IRB
2. Adherence to HIPAA rules
3. Data privacy and security
369
34411_CH12_351_370
9/9/05
10:55 AM
Page 370
34411_CH13_371_388
9/9/05
10:56 AM
Page 371
13 Practical Advice About Outcomes Research Robert L. Kane
INTRODUCTION: ORGANIZING ONE’S THINKING In an age of evidence-based medicine, the demand for solid outcomes research will grow even faster. Outcomes research can be considered as the intersection of careful clinical reasoning and strong research design. It is infeasible and impractical to expect that all clinical questions will be addressed by randomized controlled trials (RCTs). Much of the knowledge that powers the field will have to come from observational studies; it is essential that they be done well. Observational studies require more planning and analysis than do RCTs precisely because they cannot control factors by design. They demand more thoughtful and insightful interpretation. Earlier in this book, we urged the importance of developing a conceptual model that could inform the analysis. That model should indicate which variables are thought to influence the outcomes and how that effect is brought about. A conceptual model need not be equivalent to a theoretical model in the sense that it does not have to draw upon some grand general theory. Instead, it can be based on one’s clinical experience or even beliefs about what is going on and what factors are likely to influence the outcomes of interest. Its major role is to clarify the expected relationships. Often, drawing a model illuminates the analytic problems. Such diagrams can distinguish the primary risk factors from the intervening events (including both treatments and complications of treatment) and can make clear the temporal relationships among the variables. The capacity to statistically account for variance has become a passion in some quarters, but it is important to appreciate that the more temporally 371
34411_CH13_371_388
372
9/9/05
CHAPTER 13
10:56 AM
Page 372
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
proximal a variable is to the outcome, the more variance for which it will usually account. (At the extreme, cessation of respiration predicts death very well.) However, explaining variance is not always helpful. For example, absence of a heartbeat will undoubtedly explain much of the variation in mortality rates, but so what? In some cases, accounting for variation may lead to false conclusions. For example, a study of the utilization of medical care by the informal caregivers for Alzheimer’s disease patients showed that they appeared to use less care than controls. The investigators had earlier identified that depression was more common among caregivers and hence controlled for this difference in their analysis to render the two groups more comparable. However, a causal model might well suggest that the development of depression was a result of caregiving and a determinant of medical care use. Controlling for such a step would dramatically dampen the effects of giving this stressful care and hence misstate the effects of caregiving. The conceptual model needs to make clear not only the relationship between various antecedents and the outcomes but also the relationship among the various outcomes themselves. In some instances, the outcomes may be seen as parallel, but in many cases, they will assume some sort of hierarchical form, where one may be expected to lead to the next. The discussion of this measurement hierarchy and its role in relating outcomes to interventions (covered in Chapter 4 on measures) is very relevant here (Spiegel et al., 1986). In planning an outcomes study, it is important to identify in advance which outcome is the primary one; the basis on which success or failure can be ultimately judged. Many interventions can be expected to achieve a variety of results, but in the end, one outcome must be identified as the most significant. This decision will, among other things, affect the choice of what variables to use in calculating the necessary sample size to obtain statistical power. A priori identification of the major outcome does not preclude retrospective insight, but it should cast it in a different light. It is reasonable— even desirable—that greater understanding about the relationship between treatments and outcomes occurs after one has systematically studied the problem. There is nothing wrong with postulating new relationships or embellishing the prior models, but one must appreciate that such post hoc reasoning is not causal; it represents hypothesis formulation, not hypothesis testing. It should be pursued, but it should be recognized for what it is. One cannot generate a hypothesis to apply to data after it has been ana-
34411_CH13_371_388
9/9/05
10:56 AM
Page 373
The Search for Simple Measures
373
lyzed, but the insights gained from such thought (and the exploratory analyses associated with it) can be usefully applied in planning (or recommending) the next series of studies. Unfortunately, there is no absolute protection against misrepresentation. Except for those instances where hypotheses are on file (e.g., grant proposals), it is impossible to know with certainty when in the course of a study a hypothesis was generated.
THE SEARCH FOR SIMPLE MEASURES Many clinicians (and policymakers) long for a single simple outcome. Is there a number that can be said to summarize all the outcomes of concern, akin to the Dow Jones average or the percentage of GNP spent on health care or even the infant mortality rate? Most outcome measures are too complex. The best ones usually offer a profile of results, where a provider (or a treatment) can excel in one area and fall short in another. Just as some prefer a GPA as the summary of a student’s academic success, others look for the profile of areas of success and failure. However much some may wish it, most of life cannot be summed up in a single statistic. Economists are fond of using quality-adjusted life years (QALYs) as a health outcome. Such a measure weights survival with function. Some view this process of imposing external fixed values on a highly personal area an act of hubris. Is the quality of a year lived really judged by one’s functional level? Certainly one can be very active and still have a very bad year. Conversely, some people have had very meaningful lives despite severe functional limitations. Imagine Helen Keller’s QALYs. If quality of life is very intimate and subjective, perhaps each person could rate his or her own. Leaving aside the problems of retrospectively assessing those who died, such an approach would produce a great deal of inter-rater variation. Similar problems arise with so-called health-related quality of life (HRQoL). Moreover, providing a complex summary measure that represents the weighted average of many outcome variables may not be illuminating either. Such indexes are not immediately understandable. They lack a frame of reference. What does it mean to have a score of “x” or to have improved by “y” points? When such scale scores are used, it is very useful to provide some sort of guide that can enable the reader to translate the numerical value into something more tangible and familiar. For example, “someone with a score of ‘x’ cannot do the following . . . , or is similar to
34411_CH13_371_388
374
9/9/05
CHAPTER 13
10:56 AM
Page 374
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
someone who is in ___ condition.” (The importance of anchoring such measures in the context of life events is discussed more thoroughly in Chapter 4 on measurement.)
ADJUSTING FOR CASE MIX Every doctor treats the most difficult patients. Comparisons among practitioners will inevitably raise questions about comparability of caseloads. Thus, assessing severity and comorbidity is especially salient. If in doubt, it is better to err on the side of excess measurement of patient characteristics. Much of the technical literature on outcomes research addresses the search for measures of outcomes. However, particularly in observational studies, the more crucial questions pertain to measuring the variables that describe the subjects of the study. Precisely because the cases cannot be randomly allocated, credibility of the results depends on the audience believing that the groups were comparable. Those whose oxen are being gored are likely to grasp for whatever straws they can. Being unable to answer the question “Did you control for serum zinc levels?” makes the research vulnerable to such attacks. For this reason, researchers should collect even more descriptive information than they might think necessary. It is always better to be able to say that one looked at a given aspect and found it made no difference than to have to argue that it should not make a difference. At the same time, there must be some finite limits to what data can be realistically collected. One strategy to provide anticipatory protection and still remain affordable is to recruit an advisory committee that includes skeptics. Encourage them to raise as many questions as possible during the design phase. It will be annoying, but it should pay off.
DATA QUALITY The bane of many clinical studies is data quality. There is a basic conviction that real data comes from clinicians; patient reports cannot be relied on. This medico-centric bias needs to be attacked on two grounds First, patient perceptions can provide extremely useful information. In some instances, they may be the only (or certainly the best) source. They
34411_CH13_371_388
9/9/05
10:56 AM
Page 375
Data Quality
375
may not be technically accurate in areas that require clinical expertise, but they may be the most valid sources for information about how they feel and how disease affects them. Filtering such information through a clinician does not always improve the noise-to-signal ratio. Information on activities of daily living, for example, sounds more professional from a clinician, but unless there has been specific formal testing, the information is based on patient reports. Some clinicians believe that they can sort out inconsistencies in patient reports or detect areas of exaggeration, but there is little hard evidence to support such a contention. In most cases, the data is psychometrically sounder when it comes through fewer filters. Secondly, clinician reports are not universally useful or accurate. Often elaborate systems are devised to review medical records reliably. Record abstractors are trained and monitored closely to assure comparable work. However, the quality of the review can be only as good as the substrate on which it is based. Most medical records leave much to be desired. Some years ago, the creator of the Problem-Oriented Medical Record, Lawrence Weed, commented that if clinician scientists kept their lab books in the same manner that they completed clinical records, science would be in turmoil. Not only is medical information recorded in incomplete ways, it is impossible to infer what an omission means. Does the lack of a comment on a physical finding or a symptom imply that nothing of interest was found or that nothing was looked for? What does it mean when a record says a body system was “within normal limits”? What type of examination was actually done? What does “no history of ‘x’ ” mean? What questions were asked? Few clinicians systematically record all the pertinent information needed to adjust for risk factors; nor do they provide complete data on which to establish outcomes. For many variables, especially those that require judgment or interpretation, clinicians are the best source of information, even when their interrater (and perhaps even their intertemporal) reliability has not been established. The challenge is to collect such information in a systematic and consistent manner (Feinstein, 1977, 1987). Getting useful data from clinicians usually means setting up some type of prospective data collection apparatus. Some means of querying the clinicians is needed to be sure that they make the complete set of observations sought and record the information consistently. Clinicians do not generally respond well to such demands for structured information collection. Many have a strong aversion to forms, which they deprecate as some type of “cookbook medicine.” In order to gain their
34411_CH13_371_388
376
9/9/05
CHAPTER 13
10:56 AM
Page 376
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
cooperation, it is often necessary to involve them proactively in designing the data collection forms or at least identifying the relevant material to be included. This proactive strategy is especially important in the context of quality improvement projects. Although purists will worry (justifiably) that such active involvement will bias clinician behavior, such a bias is precisely what quality improvement seeks. Moreover, those being judged are more likely to accept the results of the outcomes analysis if they have had some role in developing the criteria and the way the data is collected. In some cases, the bias toward clinician-generated information may be misplaced. Clinicians serve as mere conduits; the data can be better collected directly from patients, who may provide more and better information without the clinician filter. For example, data on patient functioning is usually better collected directly from the patients (or their proxies). Even information on key symptoms can often be obtained more easily and better directly from patients. In some cases, office (or hospital) personnel can be trained (and induced) to collect questionnaires (or even conduct short interviews) with patients when they present themselves for care.
GETTING FOLLOW-UP DATA Although it is at least theoretically possible to structure baseline and risk factor data collection into ordinary clinical activities, obtaining follow-up information is more complicated. Even if clinicians were cooperative, problems would arise. Patients are not usually seen for follow-up appointments on a consistent schedule. Some patients are lost to follow-up. Because biased data can be extremely dangerous, some form of systematic data collection that does not rely on patients returning to the clinic or hospital is in order. Although such a step adds cost and represents an added dimension, its value in providing adequate unbiased coverage justifies the expense. Follow-up data can be collected by various means. The most expensive is in-person interviews. Such a step is generally not required unless the patients are especially unreachable (e.g., nursing home patients) or data must be physically collected (e.g., blood samples, blood pressure, weight). Most times the data can be reduced to patient reports of symptoms and behaviors, which can be collected by either mailed questionnaires or telephone interviews. The choice of which approach to use is based on money and expected compliance. Often a mixed approach will
34411_CH13_371_388
9/9/05
10:56 AM
Page 377
Getting Follow-up Data
377
be needed, with nonrespondents to the mailed survey contacted by phone or even visited. Response rates are very important. The basic concern is about response bias (where those responding are systematically different from those who do not). Follow-ups are like the Sherlock Holmes story, The Hound of the Baskervilles. The dog’s not barking may be more interesting than his barking. Those who return for follow-up are likely to be different from those who do not. Failure to return may reflect a great success or a great dissatisfaction. It may also simply mean that the patient came from a distant referral and is returning to his or her primary physician. In some cases, the source of the bias is obvious (e.g., those returning for follow-up care). In most cases, the reasons for nonresponse are not evident, even when interviewers ask why the patient refused to participate. The best way to avoid such a bias is to plan in advance for proactive follow-up and to go all out to collect data on all subjects who are eligible. It is a great temptation to stint at the end game; it is also a great mistake. Inevitably, the last group to be reached will require the majority of the overall follow-up effort, but they are likely the most interesting cases as well. Those not familiar with research methods sometimes confuse large numbers of respondents with adequate response rates. Whereas sample size is important in determining the statistical power of an analysis, large numbers of respondents cannot compensate for poor response rates. It is usually better to use the available resources to assure that one collects a high proportion of those targeted than just a lot of cases. A high response rate from a smaller sample is by far preferable to a larger, but more incomplete sample. In some instances, patients may not be able to respond because they are too sick or cognitively compromised. Proxy respondents can be used to obtain information, but they should be employed with caution. Often family members will try to protect patients from such contacts, even when the patients can and want to participate. In some cases, those responsible for the patient may not want them to participate or may view the data collection activity as another nuisance. For example, nursing homes may find it inconvenient to get patients to a telephone to answer questions. Persistence and creativity will be needed to counter these ploys. In some cases, proxies are inevitable. Care and common sense should be used in deciding when a proxy can accurately represent the patient. At least two criteria should apply: (1) the proxy should have recent and adequate exposure to the patient to be able to report on his or her status (e.g., family members listed as the responsible party in a hospital or nursing home may
34411_CH13_371_388
378
9/9/05
CHAPTER 13
10:56 AM
Page 378
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
not have visited frequently enough to know what is really happening); (2) the domains that proxies can address should make sense; for example, it is unrealistic to expect a proxy to report on a patient’s degree of pain or the patient’s state of mind. At a minimum, proxy responses should be identified in the analysis with a dummy variable. In some cases, it may be prudent to analyze them separately. A good first step is to compare the overall patterns of proxy responses and direct subject responses. The timing of follow-ups is important. Particularly when treatments may extend over long periods (e.g., outpatient therapy), the effects of dating follow-ups from the beginning or the end of treatment can make a huge difference. If follow-up is dated from the beginning of the treatment and some patients remain in treatment substantially longer than others, the differences in outcomes may be attributable to the time between the end of treatment and the follow-up. Thus, it is usually safer to date follow-up from the end of treatment. On the other hand, if treatment extends for a long time and there is a natural course to recovery, those with longer treatments will have had more time to recover. For example, an RCT of geriatric evaluation units found differences at discharge between those treated and not, but failed to recognize that these differences might simply be attributable to natural recovery, especially since they disappeared with time (Cohen et al., 2002). Thus a crucial data point may be the status at the end of treatment. However, even this precaution will not eliminate the need to pay careful attention to the duration of the treatment. As noted in Chapter 7, duration and intensity are important attributes of treatment that must be considered in the analysis of its effects. The duration of treatment can thus have several effects—for instance, it can mask the effects of the natural course of a disease. If the disease in question has a natural history independent of treatment, a longer duration of treatment may simply allow the problem to run its course. Remaining in prolonged treatment may reflect patients’ motivation and hence influence their recovery. If those who become discouraged drop out of treatment, it is hard to separate the effects of less treatment from inherent characteristics of the patients. Because those who drop out may also be harder to contact at follow-up, there may be a selective attrition bias.
USING EXTANT DATA SOURCES In some cases, outcomes information can make use of extant secondary data sources. Many large national studies are restricted to data that can be
34411_CH13_371_388
9/9/05
10:56 AM
Page 379
Basic Analysis Issues
379
gleaned from administrative records. Such studies have policy value but they have serious limitations. The outcomes are usually limited to death and health care utilization and the databases have little clinical information that permits strong case-mix adjustments. For example, Medicare records can be used to trace when a person is rehospitalized or when a given operation is revised. Using secondary data sources, especially administrative databases, can be limiting and challenging. It is no coincidence that the most common outcomes addressed are death and rehospitalization. They are the two items that most administrative databases can yield. In some circumstances, such follow-up data can provide useful adjunctive information. For example, a study of hip replacement may address short-term benefits in terms of pain and mobility, but it may also be useful to see how long the prostheses lasted. It is important to distinguish studies that rely exclusively on secondary data and those used in conjunction with primary data. The former face problems of limited information, especially when it comes to adjusting for risk factors, most of which cannot be found in administrative data sets. Thus, outcomes studies derived from administrative data, although they can relatively inexpensively compare the results of large heterogeneous groups of providers, are often harshly criticized as being unfair and biased. For example, the Medicare mortality studies, which found widely varied death rates among hospitals serving Medicare patients, were actively challenged as inadequately correcting for case mix (Greenfield, Aronow, Elashoff, & Watanabe, 1988; Jencks et al., 1988; Kahn et al., 1988). On the other hand, it may be possible to link clinical information from medical records and even patient interviews with administrative data. For example, a study of the outcomes of hip replacements used data from record abstracts and Medicare administrative data to link baseline severity and comorbidity with outcomes to show a difference across genders (Holtzman, Saleh, & Kane, 2002).
BASIC ANALYSIS ISSUES Although data analysis is an integral component of a successful outcomes system, this book cannot explore this topic in great depth. Many good texts and commentaries are available (see Chapter 2) (Shadish, Cook, & Campbell, 2002; Feinstein, 1977). Nonetheless a few basic suggestions and cautions are offered. The first and most important is to recognize the complexities of this area and to get the necessary assistance. Outcomes
34411_CH13_371_388
380
9/9/05
CHAPTER 13
10:56 AM
Page 380
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
analysis can often be a subtle process. The potential statistical pitfalls are numerous. It is crucial to consult with a statistician while developing the study ideas. Few statisticians like to be approached after everything has been decided. Statistical concerns can directly affect decisions about sample size, instruments, and overall design. An important step in designing an analytic strategy is to link each hypothesis or research question with a specific analysis. This translation needs to be done in considerable detail. The concepts alluded to in the hypothesis need to be translated into specific variables and the type of analysis specified. It is often helpful to create dummy tables that show just how the results will be displayed. Going through the exercise of designing those tables can help investigators (especially neophytes) clarify just how they will organize their data and what form each variable will take. Most analyses involving outcomes will eventually use some type of multivariate analysis, most of which will fall into one or another type of regression model. Many regression models provide two types of information: the amount of variance explained (R2) and the relationship of each independent variable to the dependent variable (regression coefficient). These two factors may not be related. It is possible to explain a substantial amount of the variance without a single significant coefficient, and significant variables need not contribute much to explaining the overall variance. Each item connotes something different. It is important to decide which piece of information is most salient. The explained variance can be thought of as comparable to, in epidemiological terms, attributable (or absolute) risk. Attributable risk reflects how much added risk for developing a condition a given factor poses. The individual variable coefficient is comparable to relative risk; how does the risk of developing the condition with the factor compare without it? In rare events, one may encounter a very high relative risk but a small attributable risk.1 Likewise, with outcomes, the goal may be to identify factors that influence outcomes even if they do not explain much of the overall risk of the outcome occurring.2 In one respect, treatment can be viewed as one of a series of risk factors3 that affect the outcomes of care. In most instances, the goal of the analysis is to separate the effect of treatment from the effects of the other risk factors. It is also possible, however, to use outcomes analyses to examine directly the effect of other risk factors. In some cases, one may want to see how a risk factor performs after the effects of other factors have been controlled. (i.e., what is the specific con-
34411_CH13_371_388
9/9/05
10:56 AM
Page 381
Basic Analysis Issues
381
tribution of a given factor). In other situations, the effect of other factors may be thought to have an effect on the factor of interest (i.e., there is an interaction). For example, hospital length of stay may be influenced by whether a patient is discharged to a nursing home and whether that person is covered by Medicaid. One could examine the individual contributions of each variable in a regression model. However, the effect of a nursing home discharge might be modified by the patient’s Medicaid status. To test this possibility, one could look at the effect of the interaction of these two variables on the dependent variable or one could form two subgroups (those on Medicaid and those not) and compare the effects of nursing home discharge on each. Different types of variables will require different types of analyses. The differences may be based on the assumptions about the normality of the distribution of the variables (i.e., is the curve bell-shaped). In general, researchers talk about two classes of analyses: parametric and nonparametric. The former are appropriately used when the dependent variable is assumed to be normally distributed. In some cases, it is possible to transform the dependent variable into a normal distribution to make such analyses more feasible. Some variables have unusual distributions. Variables like health care cost and utilization data may show a large peak near zero use and a thin long tail of high users (i.e., there may be a large number of cases in which there is no event during the observation period; many people use few, if any, services in a given year). For example, approximately 20 percent of Medicare enrollees will use no services in a given year. Using regular regression techniques will result in biased coefficients. Special analytic techniques have been developed to handle such cases. Categorical dependent variables (including dichotomous or polytomous) variables require different analytic techniques, usually termed “nonparametric.” Special regression techniques (e.g., logistic regression) that avoid the problem of biased coefficients are available for these circumstances. A note of caution is in order. Care should be used in interpreting the results of regression analyses. A variety of problems can haunt such efforts, including using too many variables for the number of observations, colinearity among the variables, endogeneity (i.e., reciprocal relationships among the dependent and independent variables), and unusual distributions that bias the coefficients or render them uninterpretable. Statistical assistance is invaluable, both when designing a study and in interpreting the results.
34411_CH13_371_388
382
9/9/05
CHAPTER 13
10:56 AM
Page 382
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
ETHICS Outcomes investigators may encounter some ethical issues. There is still debate about how much informed consent must be obtained from patients. In general, when an outcomes project is done as part of an ongoing quality of care process that is incorporated into a medical care system’s regular activities, no special permission is required. It is assumed that the patient who initially agreed to be treated accepts the outcomes work as part of that treatment. However, if outside agencies are used to collect the data or if it is used for more scientific purposes (or any purpose beyond direct quality improvement for the clinicians involved), then patients must first give their permission to be interviewed or even to have their records examined. Confidentiality is an important ethical consideration. Some institutional review boards (who must adjudicate the ethical aspects of a study) will not even permit an outside research agency to contact patients directly to request their permission to participate in a study. Simply releasing their names is seen as a breach of confidentiality. Instead, the patients’ physicians must first seek their permission to be contacted. Such a step rigorously enforced can put an end to outcomes research. Few care providers have the resources to persist in following up the substantial numbers of patients who simply fail to respond to an invitation to participate. Although it would be wrong to coerce a patient into participating, it is also dangerous to eliminate from a study those who simply fail to indicate whether or not they are willing to participate. Somehow the investigators need to be deputized to act in the physicians’ stead if the study is to be conducted. Both patients and providers need to be clear about how the material around an outcomes study will be used. When anonymity is promised, it must be complete. Under such cases, the results about a given patient cannot be shared with that patient’s doctors. On the other hand, some patients may want the information shared; they should be given an opportunity to indicate such a preference. In general, outcomes information is obtained under the promise that it will be used only in aggregate form and no identifiers will be attached. Providers also need to know in advance when they will be identified with the results. In cases where the outcomes information may be useful to consumers in making informed choices, anonymity would be disadvantageous. However, providers may be rightfully concerned that adequate casemix adjustments have been made before data is released. On the other hand, failure to release identifying information can be viewed with suspi-
34411_CH13_371_388
9/9/05
10:56 AM
Page 383
Quality Improvement
383
cion, as if the providers had something to hide. Careful prior arrangements need to be established about how and when data will be released.
DISEASE MANAGEMENT The rise of chronic disease has spawned enthusiasm for disease management. This term applies to various efforts to pay closer attention to the status of patients in such a way as to intervene before their deteriorating clinical course becomes a crisis. A number of commercial programs have been developed but the science underlying their efficacy, assessed either economically or in terms of actual changes in disease state, is still underdeveloped. Outcomes research has only a limited amount to offer to disease management. They share common needs for measures that reflect clinical status and a need for strong information systems. Outcomes research can make an important contribution by testing the effectiveness of disease management. By comparing groups that do and do not receive such attention, one can ascertain if the clinical trajectories are affected.
QUALITY IMPROVEMENT Quality actions can be considered at several levels. The most basic is quality assessment, which implies simply measuring the quality of care (perhaps expressed in terms of outcomes). Assessment per se assumes no responsibility for what is done with the information. By contrast, quality assurance implies a commitment to assure that at least a certain level of quality is reached. It is a much heavier responsibility. In the middle of those poles lies quality improvement (QI). In essence, QI rejects the idea of meeting some fixed standard to strive for an ever higher goal. At the same time, this philosophy can be construed as performing positively when very bad care becomes only bad care. The basic approach to QI consists of a cycle that includes assessing the situation, designing an intervention, implementing the intervention, and assessing the outcome. Outcomes research methods are thus congruent with the goals of QI; they are frameworks on which it is based. However, the standard of precision normally applied to outcomes research may be more relaxed for QI. In many instances, it is sufficient to create the sense of a problem as a spur to
34411_CH13_371_388
384
9/9/05
CHAPTER 13
10:56 AM
Page 384
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
action. Likewise, the perception of improvement may be a motivation for continued efforts to improve. Ultimately, however, stringent standards of evidence will be needed to justify sustained QI efforts.
OPERATIONAL STEPS Perhaps the best way to illustrate the various issues around conducting an outcomes study is to offer an example.4 Consider the case of a study done with a large number of Minnesota hospitals to assess the outcomes of elective cholecystectomy (Kane et al., 1995) . The study was sponsored by a consortium of hospital and medical associations at the state and county levels. The group had originally organized to develop guidelines or protocols for care management under the assumption that it was preferable to develop one’s own than to have someone else’s thrust upon one. In the course of the guidelines work, questions were raised about the quality of the database available on which to base determinations of appropriate care. It was decided that the study should be expanded to include the collection of outcomes. A study design was developed to identify the potential risk factors that should be considered in assessing the effect of treatment on the outcomes of cholecystectomy. Table 13–1 summarizes the major categories of variables used in this study according to the classification scheme used in this book. In this case, the treatment variable of interest was initially the surgeon and the hospital where the operation was performed. The question posed was not whether performing a cholecystectomy was better than using some sort of medical treatment or even watchful waiting. Rather, the question was did the operation performed by one person lead to better outcomes than if done by another and which characteristics of a case predicted better outcomes. Just as the study was being designed, a new approach to cholecystectomy, laparoscopic surgery, was introduced. The study was quickly amended to include a comparison of the results of the two approaches: conventional open surgery versus laparoscopic. Clinical teams worked with outcomes researchers to establish the conceptual model. Literature reviews and meetings with clinicians were used to identify the potential risk factors that could influence the outcomes. The outcomes themselves were derived from several sources. Condition-specific measures included symptoms that were associated with indications for performing the procedure (e.g., pain, nausea). In effect, the appropriateness
34411_CH13_371_388
9/9/05
10:56 AM
Page 385
Operational Steps
385
Table 13–1 Variables Used in Cholecystectomy Study Outcomes
Risk Adjustment
Treatment
• Concordance with classic cholecystitis pain
• Severity measures (duration, X-ray findings, symptoms)
• Open vs. laparoscopic
• Symptom score
• Comorbidity (Charlston scale)
• Hospital
• Functional status
• Demographics (age, gender)
• Surgeon
• Satisfaction (3 factors)
• Prior history
criteria were adapted as outcome measures under the assumption that the main purpose of the treatment was to alleviate the factors that suggested a need for care in the first place. In addition, pertinent generic measures of quality of life (e.g., the ability to perform daily activities and self-rating of health status) were added. These were reviewed to assure that the clinicians believed that good care would actually influence them. A series of risk factors was established based on such aspects as severity and duration of the problem. Certain physiological and laboratory tests were used as criteria. After a preliminary review of some sample charts, it became quickly evident that much of the information deemed pertinent would not be available from the hospital record. The nursing departments from each of the participating hospitals were contacted to see if they would be willing to implement a special data collection activity at the time of admission. The data collected at baseline would include both specific symptom information and more generic measures; it would also be an opportunity to obtain informed consent to be contacted subsequently to ascertain follow-up status. Although the nursing staffs proved willing to undertake this added work (some were sold on the basis that this study would prove an opportunity to demonstrate the value of good nursing care as well), logistical problems did occur. As same-day surgery increased, it proved harder to obtain the baseline data: Patients were not available in advance. The follow-up data was collected by a special survey research unit. The primary data collection mode was by mail, but telephone follow-ups were
34411_CH13_371_388
386
9/9/05
CHAPTER 13
10:56 AM
Page 386
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
used if responses were not received within the time frames allocated. This sort of follow-up plan required that patients’ names and addresses be known; special procedures were used to keep the records confidential. All completed follow-ups were coded with a specific code number linked to an index. All subsequent analyses were done with only that code number. The linking of the several components of data (the baseline interviews, the followup questionnaires, and the medical record abstracts) was done by a small team that was part of the sponsoring organization. Careful monitoring was required to assure that the 6-month follow-ups were collected within the time window. As soon as cases were enrolled and baseline data collected, a file was opened and the case tracked. The medical records of each case were reviewed using a specially designed abstraction form. Interrater reliability was checked and monitored to be sure that the same information was interpreted consistently. The results of this step were combined with the baseline and follow-up data to create a single analytic file. This file had no identifiers for patients, surgeons, or hospital. Instead code numbers were substituted for each. The analysis used regression models to examine the effects of the potential risk factors on the various outcomes of interest. When potential risk factors were shown to play an active role, they were retained in the models when the independent variables of major interest were introduced. The effects of laparoscopic surgery were examined by both using a dummy variable to represent the type of surgery and by examining the outcomes separately for those undergoing laparoscopic and open cholecystectomies.
SUMMARY The demand for outcomes information is growing. For clinicians and academics, it is needed to support and grow the underlying information base that will help the field to base its actions on strong evidence. It is potentially useful to consumers as the basis for choosing providers of care. Although it might be too optimistic to believe that health care decisions will be any more rational than most consumer purchases, at least some subset of the population will welcome better information, especially if it can be presented in a useful way. For many decisions, however, the number of cases in a given clinician’s portfolio over a finite period may be too small to allow adequate analysis.
34411_CH13_371_388
9/9/05
10:56 AM
Page 387
References
387
Outcomes research has the potential to add considerably to the empirical basis of medical practice. It will never be feasible to base all (or even a large proportion) of medical care on randomized clinical trials. On the other hand, it is irrational to assume that simply intuitively assembling the lessons of clinical experience serves as an adequate basis for scientific practice. The immediate response to the recognition of substantial variation in practice has been the institution of guidelines based largely on professional judgments about what constitutes good care. The next step is to amass the empirical database that those looking for a more scientific basis to establish guidelines have not found. Careful observations developed as part of well-designed studies will go a long way toward providing the insights needed. One may never be able to say with absolute certainty that a given treatment works in a given situation, but one will have come a lot closer to making informed statements. Simply collecting data is not the answer. Studies must be carefully designed. Conceptual models must be created that combine the best of clinical and social science insights. These models should form the basis for deciding what information is to be collected and how it will be analyzed. The technology of outcomes research has come a long way in the last decades and promises to go much further. Sophisticated analytic and measurement methods are available. Like any other powerful tools, they must be handled carefully by persons skilled in their use. The best outcomes research is likely to come from partnerships of technically proficient analysts and clinicians, each of whom is sensitive to and respectful of the contributions the other can bring. REFERENCES Cohen, H.J., Feussner, J.R., Weinberger, M., Carnes, M., Hamdy, R.C., Hsieh, F., et al. (2002). A controlled trial of inpatient and outpatient geriatric evaluation and management. New England Journal of Medicine, 346(12), 905–912. Feinstein, A.R. (1977). Clinical biostatistics. St. Louis, MO: C.V. Mosby. Feinstein, A.R. (1987). Clinimetrics. New Haven, CT: Yale University Press. Greenfield, S., Aronow, H.U., Elashoff, R.M., & Watanabe, D. (1988). Flaws in mortality data: The hazards of ignoring comorbid disease. Journal of American Medical Association, 260, 2253–2256. Holtzman, J., Saleh, K., & Kane, R. (2002). Gender differences in functional status and pain in a Medicare population undergoing elective total hip arthroplasty. Medical Care, 40(6), 461–470.
34411_CH13_371_388
388
9/9/05
CHAPTER 13
10:56 AM
Page 388
PRACTICAL ADVICE ABOUT OUTCOMES RESEARCH
Jencks, S.F., Daley, J., Draper, D., Thomas, N., Lenhart, G., & Walker, J. (1988). Interpreting hospital mortality data: The role of clinical risk adjustment. Journal of the American Medical Association, 260, 3611–3616. Kahn, K.L., Brook, R.H., Draper, D., Keeler, E.B., Rubenstein, L.V., Rogers, W.H., et al. (1988). Interpreting hospital mortality data. How can we proceed? Journal of the American Medical Association, 260, 3625–3628. Kane, R.L., Lurie, N., Borbas, C., Morris, N., Flood, S., McLaughlin, B., et al. (1995). The outcomes of elective laparoscopic and open cholecystectomies. Journal of the American College of Surgeons, 180(2), 136–145. Radosevich, D.M., Kalambokidis, T.L., & Werni, A. (1996). Practical guide for implementing, analyzing, and reporting outcomes measures. Bloomington, MN: Health Outcomes Institute. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company. Spiegel, J.S., Ware, J.E., Ward, N.B., Kane, R.L., Spiegel, T.M., Paulus, H.E., et al. (1986). What are we measuring? An examination of self-reported functional status measures. Arthritis and Rheumatism, 31(6), 721–728.
NOTES 1.
To appreciate the difference between attributable risk and relative risk, imagine two interventions: one reduces the risk of the common cold from 1 in 2 to 1 in 4; the other prevents a rare leukemia, reducing the risk from 1 in 2 million to 1 in 4 million. Both have a relative risk of 0.5, but the attributable risk for the common cold is 1 in 4, whereas the attributable risk for leukemia is 1 in 4 million. 2. To be accurate, we should note that in many studies the explained variance and regression coefficients do not precisely correspond to predictive variables because they describe associations that have been gathered retrospectively. Hence, they should be interpreted more cautiously. When a study uses what epidemiologists call a “casecontrol” model (i.e., it begins by identifying those with and without the outcome and looks for factors that are associated with one or the other), the best an investigator can do is to identify factors (i.e., treatments) that are likely to be more strongly associated with the outcome than its absence. These relationships are more appropriately expressed as odds ratios, which do not imply causal assumptions. 3. The term “risk factor” is used here in a generic sense to include all of the factors that can affect the outcomes of care. In some discussions of disease severity (see Chapter 9), risk factor is used interchangeably with “severity.” In this book, we have tried to use it consistently in the larger context. 4. For a practical guide to implementing quality improvement projects, see Radosevich, Kalambokidis, & Werni, 1996.
34411_INDX_389_404
9/9/05
10:58 AM
Page 389
Index
Page numbers followed by f, n, or t indicate figures, notes, or tables, respectively.
A ACAHPS. See Ambulatory CAHPS ACASI. See Audio Computer Assisted Self Interviewing Accountability, 4–5 ACER. See Average cost-effectiveness ratio ACGs. See Ambulatory Care Groups Acquiescent response bias, 196 Activities of daily living (ADLs), 87, 88, 109, 128, 142 instrumental (IADLs), 87, 128, 142, 144 Acute Physiology, Age, and Chronic Health Evaluation (APACHE), 252–253 APACHE II, 236t, 252–253 APACHE III, 253 Acute physiology score (APS), 252 ADGs. See Ambulatory diagnostic groups Adjustment for case mix, 374 Adjustment for severity, 219–220 Adjustment measures, 295–296 ADLs. See Activities of daily living Administration of surveys methods for, 91 mixed modes of, 93 Administrative computerized diagnosis records, 246–247 Administrative meaningfulness, 235 Affect, 278–286 Affect Balance Scale, 282t, 285 Affective, Normative, and Continuance Commitment instrument, 292t, 296 Affectometer 2, 285 Age, 266–268
Agency for Health Care Policy and Research (AHCPR), 200, 201 Agency for Healthcare Research and Quality (AHRQ), 4, 208 Aggregation, 98–100 “Agree-disagree” response scale, 195 AHCA. See American Health Care Association AHCPR. See Agency for Health Care Policy and Research AHRQ. See Agency for Healthcare Research and Quality AIMS. See Arthritis Impact Measurement Scales ALLHAT trial, 322t Allopathic (Western) medicine, 309 Alternatives, 170–171 AMA. See American Medical Association Ambulatory CAHPS (ACAHPS), 203–205, 204t Ambulatory Care Groups (ACGs), 238t, 258–259 Ambulatory care satisfaction measures, 203–206, 204t Ambulatory diagnostic groups (ADGs), 259 American Health Care Association (AHCA), 207t, 208–209 American Medical Association (AMA), 246 American Society of Anesthesiologists (ASA), 246 AMHCS. See Annual Member Health Care Survey
389
34411_INDX_389_404
390
9/9/05
10:58 AM
Page 390
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
Analysis, 305–349 basic issues, 379–381 of data, 360–361 units of, 62–63 Annual Member Health Care Survey (AMHCS), 200–201 Anxiety Adjective Checklist, 285 Anxiety measures, 281t–282t, 284–286 APACHE. See Acute Physiology, Age, and Chronic Health Evaluation APS. See Acute physiology score Area Resource File, 269 Arthritis Impact Measurement Scales (AIMS), 168–269 ASA. See American Society of Anesthesiologists Attribution, 189 Attribution bias, 65–66, 66f Attrition, 39–40, 47t Audio Computer Assisted Self Interviewing (ACASI), 92 Average cost-effectiveness ratio (ACER), 349n1
B Back Pain Classification Scale, 274t Back pain measures, 274t Barthel Index, 142, 143t, 144 Basic concepts, 1–120 BDI. See Beck Depression Inventory Beck Depression Inventory (BDI), 139, 143t, 145 Behavioral Risk Factor Surveillance Survey, 269 Berkson’s Bias, 38 Bias acquiescent response, 196 attribution, 65–66, 66f Berkson’s Bias, 38 mono-method, 44 mono-operation, 44 nonresponse, 196 in psychometric testing, 196 selection, 66, 67f controlling for, 67–68, 68–70 identification of, 229–231, 230f Brief Pain Inventory, 274t, 277
C CAHPS. See Consumer Assessment of Health Plans
Cancer measures, 274t Capital SES (CAPSES), 290t, 294 Cardiologists, 315–316 Carroll Rating Scale, 281t, 284 Case-control models, 388n2 Case mix, 374 Categorical outcomes, 34 CBA. See Cost-benefit analysis CDS. See Chronic Disease Score CEA. See Cost-effectiveness analysis Ceiling effects, 164n5 Center for Epidemiologic Studies Depression Scale (CESD), 281t, 283, 284 Center for Medicare and Medicaid Services (CMS), 208 CESD. See Center for Epidemiologic Studies Depression Scale Change interpreting, 109–110 sensitivity to, 107–108 “stages of change” research, 278 Change in Function Index, 142 Charlson Comorbidity Index, 237t, 255–256 CHE. See Chronic health evaluation Chiropractors, 327, 329t, 333n2 Cholecystectomy study (example case), 384, 385t Chronbach’s alpha coefficient, 101, 102, 135, 164n3 Chronic Disease Score (CDS), 238t, 257–258 Chronic health evaluation, 252 Chronological age, 267–268 CIBIS-II, 322t CIRS. See Cumulative Illness Rating Score Clinical environment, 354, 364–365 Clinical pathways, 313–315 Clinical relevance, 132t, 134 Clinical settings: implementing outcomes research in, 353–369 structured guide for, 364–369 Clinicians: information exchanged between patients and, 313 Clustered data, 75 “Clustering” observations by site, 74–75 CMS. See Center for Medicare and Medicaid Services Cochrane Collaborating Centers, 4 Coexisting conditions, 226 Cognitive-affect model of satisfaction, 186
34411_INDX_389_404
9/9/05
10:58 AM
Page 391
Index
Cognitive functioning, 129–130 example survey items, 127t measures of, 145–146, 286 Combination designs, 52–53 Combinations of treatment studies, 323t, 324, 326t Commitment measures, 292t Comorbid conditions, 221 Comorbidity, 219–261 adjusting for, 219–220 versus complications, 226–228 definition of, 226 equation for, 221 measurement of, 243–244 severity of illness and, 220–222 Comorbidity measures conditions or diseases included, 249–250 criteria for choosing, 233t data sources, 244–245 disease of primary interest, 249 domains of health included, 233–234 how to choose, 233–247 issues specific to, 248–251 population of interest, 240–241 prognostic endpoints, 235–240 range of scores, 244 reasons to include, 229–232 reliability of, 234 setting, 241 specific measures, 251–260 timing of measurement, 241–242 unweighted versus weighted, 250 validity of, 234–235 Comparison of treatments, 60–61 bases for, 320, 321t across providers, 327–330 Complex organization measures, 292t, 296 Complications versus comorbidity, 226–228 definition of, 226 Comprehensive Older Person’s Evaluation (COPE) scale, 144–145 Computerized diagnosis records, administrative, 246–247 Computerized Severity Index (CSI), 236t, 254 Concepts, basic, 1–120 Conceptual design, inadequate, 43–44 Conceptual models and modeling, 17–20, 60–63, 83, 171–175 examples, 18, 18f, 77–79, 77f key questions to be addressed, 60
391
Conceptual planning, 43–44 Conceptualization of measures, 87–90 Condition-specific measures, 123, 165–183. See also specific measures alternatives, 170–171 choosing, 171–177, 180–181 comparison with generic measures, 165–166 health status measures, 168–269 measurement by, 173, 173t role of, 177–180 selection of domains, 172 types of, 165 ways to use, 172t Conditions or diseases included, 249–250 Confidence intervals, 72 Confidentiality, 382 Confirmatory factor analysis, 106–107 Consistency, internal, 101 CONSORT, 13 Construct validity, 26–27, 43–45, 106, 107 threats to, 25t, 47t–48t Consumer Assessment of Health Plans Study (CAHPS), 196, 197–198, 199t, 210 Ambulatory CAHPS (ACAHPS), 203–205 CAHPS 2.0, 200 CAHPS 2.0H, 201 Hospital CAHPS (HCAHPS), 201 Nursing Home CAHPS (NH-CAHPS), 207t, 208 Consumer Satisfaction Survey (CSS), 199t, 200–201 Content validity, 105–106 Controlling for selection bias with matching or propensity scores, 68–70 with multivariate statistics, 68 through natural experiments, 67–68 through random assignment, 67 COOP. See Dartmouth COOP Charts Cooperative Information Project, 124, 138, 149t, 155–156 Coordination, study, 358–359 COPE scale. See Comprehensive Older Person’s Evaluation scale Correlation due to nonrandom survey sampling techniques, 74 Cost-benefit analysis (CBA), 337–338 resources, 347–348 Cost-effectiveness analysis (CEA), 98, 335–349 economic evaluation, 337 hypothetical illustration, 346–347
34411_INDX_389_404
392
9/9/05
10:58 AM
Page 392
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
resources, 347–348 types of, 336–339 Cost-utility analysis (CUA), 338–339 resources, 347–348 Costs, 339–342 operations, 356 start-up, 356 Costs of treatment, 339 Counseling, 311–312 differences associated with provider orthodoxy, 327, 329t generic approach to assessing, 310t studies of, 322t, 325t Cox proportional hazards regression, 36t CPT-4. See The Physician’s Current Procedural Terminology Criterion-related validity, 105 CSI. See Computerized Severity Index CSS. See Consumer Satisfaction Survey CUA. See Cost-utility analysis Cultural differences, 270 Cumulative Illness Rating Score (CIRS), 259–260 CIRS-Geriatric (CIRS-G), 260 measurement properties, 239t
D Dartmouth COOP Charts, 124, 138, 155–156 criteria, 155 domains, 149t validation of, 155 Data follow-up, 376–378 missing, 40–42 Data analysis, 360–361 Data collection, 91–93 Data management, 360 Data quality, 374–376 Data sources, 244–245 extant, 378–379 Database design, 359 Demographic variables, 265, 266–271 Depression measures, 280–284, 281t Depressive Experiences Questionnaire, 281t, 284 Design(s) combination designs, 52–53 conceptual, 43–44 critical steps, 83
database design, 359 hierarchical or nested outcomes designs, 35 to isolate effects of treatment, 316–317 quasi-experimental designs, 24, 49–53 study, 23–57 general guidelines for, 53–55 types of, 12–15, 23–24 Design controls, 55 Design process, 24 Design questions, 27–28 Diagnosis components of, 312–313 versus treatment, 309 Diagnosis records administrative computerized, 246–247 medical records, 245–246 Diagnosis-related groups (DRGs), 236t, 253 Diagnosis-specific measures, 248 Diffusion effect, 45 Dimensionality, 134 Disease management, 383 Disease of primary interest, 249 Disease Staging (DS), 254 Disease Staging (DS) (Clinical), 236t Disease Staging (DS) (Scale), 237t Diseases included, 249–250 Domains of health, 126, 127t, 131–132, 132t how to choose, 233 included, 233–234 severity of illness and, 222–224, 223f Donabedian framework for quality assessment, 191 DRGs. See Diagnosis-Related Groups Drug prescription data, 247 DS. See Disease Staging DSM-IV (Diagnostic and Statistical Manual of Mental Disorders–Fourth Edition), 283 Duke Health Profile, 125, 153 Duke Severity of Illness checklist, 237t–238t, 257 Duke Social Support and Stress Scale, 291t, 295 Duke-UNC Functional Social Support Survey, 291t, 295 Duke University Health Profile (DUHP), 149t, 153 Dummy variable models, 75 Duncan Socioeconomic Index, 289, 290t Dysfunction, 133
34411_INDX_389_404
9/9/05
10:58 AM
Page 393
Index
E EBM. See Evidence-based medicine Ecological analysis, 62 Economic evaluation, 337 Economic meaningfulness, 235 Educated guesses, 30–31 Education, 311–312 Education of patients/providers studies, 325t Education of patients studies, 322t Effect sizes, 169 Elixhauser et al. measure, 239t, 259 EMAS. See Endler Multidimensional Anxiety Scales Emotional functioning, 279 Emotional functioning measures, 129, 145 example survey items, 127t Emphasis, level of, 132t, 134 Endler Multidimensional Anxiety Scales (EMAS), 282t, 285 Epidemiological (observational) studies, 13–14 EQ-5D. See EuroQol EQ-5D Equivalent groups, 70–71 Error rate problems, 32–33, 46t methods to adjust for, 33 nonrandom errors, 100 strategies for minimizing, 33 Estimation, 63–71 Ethics, 382–383 Ethnic differences, 270 EuroQol EQ-5D, 125, 156–157, 346 domains, 149t resources, 157 role of, 179–180 VAS Thermometer, 344, 345f Evidence-based medicine (EBM), 4 “Excellent-poor” response scale, 195 Expectations, 188–190 Explanatory variables, 61–62 External validity, 26–27, 45 threats to, 25t, 48t
F Factor analysis, confirmatory, 106–107 Fatigue, Energy, Consciousness, Energized and Sleepiness, 274t Fatigue/insomnia measures, 274t Feedback loops, 186 FIM. See Functional Independence Measure
393
Fishing, 32–33, 46t Fixed- and random-effects models, 75 “Fixed effects” (term), 75 Fixed-effects models, 75 Floor effects, 164n5 Follow-up data, 376–378 Frustration with Work instrument, 293t, 296 Function measures, 173, 173t Functional Activities Questionnaire, 142 Functional Assessment of Chronic Illness Inventory, 284 Functional Independence Measure (FIM), 142–144 Functional status, 222 Fuzziness, 105
G Gallup Association, 209 Gamble method, standard, 112f General health perceptions: SF-36 examination of, 166t General linear regression analysis, 34–35, 36t General mental health: SF-36 examination of, 166t General practice providers. See also Providers of treatment treatment quality differences associated with, 327, 329t General Well-Being Schedule, 272t Generalizability theory, 104 Generic measures, 16, 123–164. See also specific measures by name choosing, 141–156 comorbidity measures, 248 comparison with condition-specific measures, 165–166 criteria for, 131–136, 132t guidelines for, 157–158 health domains typically measured in, 125–131 health utility measures, 156–157 multidimensional measures, 147–148, 149t physical functioning measures, 142–145 potential uses for, 125 practical considerations, 136–138 purposes of, 125 reasons not to use, 167–168 reasons to use, 123–125 resources, 157
34411_INDX_389_404
394
9/9/05
10:58 AM
Page 394
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
role of, 177–180 severity measures, 248 traditional, 141 unidimensional measures, 142, 143t Geriatric Depression Scale, 86, 281t, 284 GHAA. See Group Health Association of America Group Health Association of America (GHAA), 200 Group randomized trials, 76 Growth curve modeling, 76 Guttman scaling, 97, 112
H Hamilton Anxiety Scale (HAMA), 281t, 285 Hamilton Rating Scale, 284 Harvard Medical School, 200 HCAHPS. See Hospital CAHPS HCFA. See Health Care Financing Administration HCSM. See Home Care Satisfaction Measure HCTC satisfaction scale. See Home Care and Terminal Care Satisfaction Scale Health concept of, 222 range of, 132–133, 132t WHO definition of, 124, 191 Health and Psychosocial Instruments database, 180 Health Beliefs Model, 78 Health Care Financing Administration (HCFA) hospital mortality rates, 220 The International Classification of Disease (ICD-9), 246 Health care quality utility measures, 98 generic measures, 156–157 multidimensional, 147–148, 149t Health domains, 126, 127t, 131–132, 132t how to choose, 233 included in measures, 233–234 severity of illness and, 222–224, 223f typically measured in generic measures, 125–131 Health Insurance Experiment, 146 Health Insurance Portability and Accountability Act (HIPAA), 360, 362 Health perceptions, general: SF-36 examination of, 166t
Health Plan Employer Data and Information Set (HEDIS), 201 Health plan satisfaction measures, 198–201, 199t Health-related quality of life (HRQoL), 126, 275–276, 342–346, 373 approaches to evaluation of, 343 concepts of, 126 determination of, 343 domains of, 131 terminology, 164n2 Health status, initial, 66, 67f Health status measures approaches, 343 condition-specific, 168–269 generic, 167–168 multidimensional, 147–148, 149t unidimensional, 142, 143t Health status questionnaires, 346 Health Utilities Index Mark 2 (HUI-2), 149t, 156 Health Utilities Index Mark 3 (HUI-3), 125, 149t, 156, 346 Health Utilities Indices, 157 HEDIS. See Health Plan Employer Data and Information Set Hierarchical or nested outcomes designs, 35 HIPAA. See Health Insurance Portability and Accountability Act History, 42–43, 47t natural, 124 selection interactions, 38 Hollingshead Index of Social Position, 289, 290t Home Care and Terminal Care (HCTC) Satisfaction Scale, 207t, 209–210 Home Care Satisfaction Measure (HCSM), 207t, 209 Hospital CAHPS (HCAHPS), 201, 202t Hospital mortality rates, 220 Hospital satisfaction measures, 201–203, 202t HRQoL. See Health-related quality of life “HRQOL” (term), 164n2 HUI. See Health Utilities Index Hypertension Prevention Trial, 322t
I IADLs. See Instrumental activities of daily living
34411_INDX_389_404
9/9/05
10:58 AM
Page 395
Index
IAQs. See Interviewer-administered questionnaires ICD-9. See The International Classification of Disease ICED. See Index of Coexistent Disease ICER. See Incremental cost-effectiveness ratio ICUs. See Intensive care units Illness, 222 Illness Behavior Questionnaire, 275t Implementation of interventions, inconsistent, 36–37 Implementation of research in clinical settings, 353–369 issues to consider before implementing studies, 365–368 responsibilities of, 356–361 structured guide for, 364–369 Incremental cost-effectiveness ratio (ICER), 335, 349n1 hypothetical illustration, 347 Index of Coexistent Disease (ICED), 256 measurement properties, 237t subscales, 256 Index of Well-Being, 272t Inference, 71–76 Inflated type I error rate, 33 Influenza vaccination effectiveness (example analysis), 76–80, 77f conceptual model, 77–79, 77f statistical model, 79–80 Information sources, 90–91 Initial health status, 66, 67f Insomnia measures, 274t Institute for Health Care Delivery Research in Intermountain Health Care, 4 Institutional review boards (IRBs), 362 Instrumental activities of daily living (IADLs), 87, 128, 142, 144 Instrumental variables, 70–71 Intensive care units (ICUs), 252 Intention to treat (ITT), 26, 323 Internal consistency, 101 Internal reliability, 101–102 Internal validity, 26–27 threats to, 25t, 37–43, 47t Internality, Powerful Others, and Chance scales, 272t, 276 The International Classification of Disease (ICD-9) (HCFA), 246 Interrater reliability, 103, 103t
395
Interval scales, 94t, 95 Interventions. See also Treatment(s) inconsistent implementation of, 36–37 Interviewer-administered questionnaires (IAQs), 91 Introduction, 3–22 Inventory of Stressful Events, 293t IRBs. See Institutional review boards IRT. See Item response theory Item characteristic curves, 112, 112f Item response theory (IRT), 111–112 logistic models, 112–114 ITT. See Intention to treat
J Job Interdependence instrument, 293t, 296 Job Overload instrument, 293t, 296 Job Routinization and Formalization instrument, 293t, 296 Job Stress Scale, 293t, 296
K Kaplan and Feinstein measure, 238t, 257 Kaplan-Meier Life Table methods, 36t Kappa (k), 103 Katz Index, 97, 142, 143t, 144 Key Clinical Findings (KCFs), 255 “Kitchen sink” approach, 132
L Latent variable structural equation modeling, 114–116 Leadership, 354–355 considerations before implementing studies, 365 study, 357 Level of emphasis, 132t, 134 Leveraging research, 355, 365 Life satisfaction overall. See Overall wellbeing Life Stressor Checklist-Revised, 275t Life Stressors and Social Resources Inventory (LISRES), 275t, 278 Likert “agree-disagree” response scale, 96–97, 113, 195 LISRES. See Life Stressors and Social Resources Inventory
34411_INDX_389_404
396
9/9/05
10:58 AM
Page 396
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
Literature reviews, 210 Locus of control measures, 272t–273t, 276 Logistic models, 112–114 Logistic regression, 34, 36t Logistical plans, 54–55 Long-term care satisfaction measures, 206–210, 207t Longitudinal data analysis, 76 The Lung Health Study, 323t
M Macular Photocoagulation Study (MPS), 322t Magnitude estimation, 139 Maladjustment scales, 295 Management of data, 360 disease, 383 study, 357–358 MAPI Institute, 157 Marginal models, 75 Marital status, 270–271 Market decisions, 4 Matching or propensity scores, 68–70 McGill Pain Questionnaire, 273t, 277 Mean substitution, 40 Measurement and measures, 83–120. See also specific measures adjustment measures, 295–296 ambulatory care satisfaction measures, 203–206, 204t anxiety measures, 281t–282t, 284–286 back pain measures, 274t cancer measures, 274t choosing measures, 180–181, 233–247 criteria for, 233t cognitive functioning measures, 145–146 commitment measures, 292t comorbidity measures how to choose, 233–247, 233t issues specific to, 248–251 reasons to include, 229–232 specific measures, 251–260 unweighted versus weighted, 250 complex organization measures, 292t, 296 conceptualization of, 87–90 condition-specific measures, 123, 165–183, 173, 173t definition of, 84–85 depression measures, 280–284, 281t
diagnosis-specific measures, 248 emotional functioning measures, 129, 145 example survey items, 127t fatigue/insomnia measures, 274t function measures, 173, 173t generic measures, 16, 123–164 comorbidity measures, 248 role of, 177–180 severity measures, 248 guides to breadth or width of, 87–88 health plan satisfaction measures, 198–201, 199t health status measures approaches, 343 condition-specific, 168–269 generic, 167–168 multidimensional, 147–148, 149t unidimensional, 142, 143t hierarchy of, 175–177 hospital satisfaction measures, 202t as imperfect, 85 issues, 87–93 long-term care satisfaction measures, 206–210, 207t multidimensional measures, 147–148, 149t multiple-item measures, 110–116 nature of, 84–87 outcomes measures, 14–17 classes of, 123 reliability of, 35–36, 46t repeated on same individual, 75–76 selection of domains, 172 specific, 121–216 physical functioning measures, 142–145 psychological measures, 281t–283t quality-of-life measures, 126–127 risk adjusters, 217–303 satisfaction measures, 198–210 severity measures how to choose, 233–247, 233t issues specific to, 248 reasons to include, 229–232 specific measures, 251–260 simple measures, 373–374 single-item measures, 110–116 social functioning measures, 146–147 social measures, 290t–293t specific measures, 121–216 timing of, 241–242 traditional measures, 141
34411_INDX_389_404
9/9/05
10:58 AM
Page 397
Index
types of scales, 94t, 95–96 utility measures generic measures, 156–157 of health care quality, 98 multidimensional, 147–148, 149t Measuring Health (McDowell and Newell), 180 Medical Care, 203 Medical meaningfulness, 235 Medical meaningfulness test, 223–224 Medical Outcomes Study (MOS) Pain Measures, 273t, 277 Short Form General Health Survey, 171 Short Form Health Survey (SF-36), 123, 148 Social Support Survey, 143t, 291t, 295 Medical records diagnoses, 245–246 Medications, 310 diagnostic, 312 differences associated with provider orthodoxy, 327, 329t generic approach to assessing, 310t studies comparing treatment intensity, 325t studies of treatment regimen, 322t Medigap insurance, 78 Medisgroups, 255 Medisgroups (Original), 237t Memorial University of Newfoundland Scale of Happiness (MUNSH), 282t, 285 Mental health, general: SF-36 examination of, 166t Mental Health Inventory, 142, 148 Mental Health Locus of Control (MHLC) scale, 273t, 276 Mental Status Questionnaire (MSQ), 142, 143t, 145–146, 286 MHLC scale. See Mental Health Locus of Control scale MHLCS. See Multidimensional Health Locus of Control Scale Mind-body connection measures, 271–278, 272t–275t Mini-Mental State Exam (MMSE), 283t, 286 Minnesota: cholecystectomy study (example case), 384, 385t Minnesota Heart Health Program, 76 Missing data, 40–42, 47t Mixed-model methods, 34–35, 36t Mixed models, 75
397
MMSE. See Mini-Mental State Exam; Multidimensional Mental Status Examination Modeling conceptual, 17–20 latent variable structural equation, 114–116 Modified Mini-Mental State (3MS) Examination, 146 Mono-method bias, 44, 48t Mono-operation bias, 44, 48t MOS. See Medical Outcomes Study MPS. See Macular Photocoagulation Study MSQ. See Mental Status Questionnaire Multicenter trials, 74–75 Multidimensional Health Locus of Control Scale (MHLCS), 272t–273t, 276 Multidimensional measures, 147–148, 149t Multidimensional Mental Status Examination (MMSE), 146 Multiple Affect Adjective Checklist, 285 Multiple-item measures, 110–116 Multivariate logistic regression, 34 Multivariate regression analysis, 68 Multivariate statistics, 68 MUNSH. See Memorial University of Newfoundland Scale of Happiness Murphy’s Law, 40
N Nam-Powers Socioeconomic Score, 289, 290t National Cancer Institute, 322t National Center for Health Statistics: General Well-Being Schedule, 272t National Committee on Quality Assurance (NCQA), 200–201 National Health and Nutrition Examination Survey, 145–146 National Health Service (NHS): Technology Assessment Program, 210 Natural experiments, 67–68 Natural history, 124 NCQA. See National Committee on Quality Assurance “Negative emotions,” 279 Negative issues: reporting, 92 Nested outcomes designs, 35 Neurosurgeons, 327, 329t NH-CAHPS. See Nursing Home CAHPS NHP. See Nottingham Health Profile
34411_INDX_389_404
398
9/9/05
10:58 AM
Page 398
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
NHRSS. See Nursing Home Resident Satisfaction Scale NHS. See National Health Service Nominal scales, 94t, 95 Nonrandom errors, 100 Nonrandom survey sampling techniques, 74 Nonresponse bias, 196 Norming, 16 Nottingham Health Profile (NHP), 149t, 152 nQuery Advisor (software), 32 NRS. See Numeric Rating Scales Numeric rating scales (NRS), 273t, 277 Nursing Home CAHPS (NH-CAHPS), 207t, 208 Nursing Home Resident Satisfaction Scale (NHRSS), 207t, 208
O Observational studies, 13–14, 24 Observations: “clustering” by site, 74–75 Operational steps, 384–386 Operations costs, 356 Ordinal scales, 94t, 95 Ordinary least squares linear regression, 36t Organizational characteristics, 353–356 Organizational Commitment Questionnaire, 292t, 296 Organizing one’s thinking, 371–373 Orthodoxy: treatment differences associated with, 327, 329t Oswestry Low Back Pain Disability Questionnaire, 274t, 277 Oswestry Personal Care measure, 89–90, 90t Outcomes definition of, 60 independent predictors of, 231, 231f reasons for difficulties with, 6–7 and satisfaction, 193–194 and treatment, 28 unmeasured factors associated with treatment and, 65–66, 66f why look at, 3–9 Outcomes analysis baseline status information, 10 collecting information, 8–9 example, 76–80, 77f interpolation of values, 40 model for, 9 reasons for, 4–5
recommended guidelines for analyzing data, 37t units of, 62–63 Outcomes approach, 9–12 Outcomes measures, 14–17. See also specific measures classes of, 123 reliability of, 35–36, 46t repeated on same individual, 75–76 selection of domains, 172 specific, 121–216 Outcomes research designing studies, 23–57 general guidelines for, 53–55 hierarchical or nested designs, 35 implementation responsibilities, 356–361 implementing in clinical settings, 353–369 key steps, 17 Murphy’s Law for, 40 practical advice about, 371–388 structured guide for, 364–369 threats to, 25–28 Overall severity of illness, 222 Overall well-being, 130–131 example survey items, 127t Ovid database, 180
P p-values, 72 ways to get wrong, 73–76 Pain, 130 Pain and Distress Scale, 273t Pain measures, 273t, 276–277 example survey items, 127t SF-36 examination, 166t Pain Measures (MOS), 273t, 277 Panel data analysis, 76 Panel on Cost-Effectiveness in Health and Medicine, 347 Patient education studies, 322t, 325t Patient Judgments on Hospital Quality (PJHQ), 202t, 203 Patient satisfaction importance of, 185–186 model of, 186, 187f Patient Satisfaction Questionnaire (PSQ), 205–206 Form III (PSQ III), 200, 204t, 205–206 PSQ-18, 206
34411_INDX_389_404
9/9/05
10:58 AM
Page 399
Index
Patients: information exchanged between clinicians and, 313 Perceived Stress Questionnaire, 274t, 278 PF-10. See Physical Function Scale PGC. See Philadelphia Geriatric Center PHI. See Protected health information Philadelphia Geriatric Center (PGC) Extended MSQ, 146 Moral Scale, 282t, 286 Physical Function Scale (PF-10), 40 Physical functioning measures, 128, 142–145 example survey items, 127t SF-36 examination, 166t Physical Status measure (ASA), 246 Physician care, 315–316 The Physician’s Current Procedural Terminology (CPT-4) (AMA), 246 PI. See Principal investigator PJHQ. See Patient Judgments on Hospital Quality Placebo effects, 321–323 Planning issues to consider before implementing studies, 365–368 logistical plans, 54–55 resource, 362 systematic, 362 POMS. See Profile of Mood States POMS Brief, 145 Populations of interest, 240–241 Positive issues: reporting, 92 Powell, Colin, 19 Practical advice, 371–388 Practical considerations, 132t, 136–138 Preference weighting, 138–140 Prescription data, 247 Principal investigator (PI), 357 Problem-Oriented Medical Record, 375 Procedures, 311 diagnostic, 312–313 differences associated with provider orthodoxy, 327, 329t generic approach to assessing, 310t studies of, 322t, 325t Profile of Mood States (POMS), 143t, 145, 171 Profiling, 125 Prognostic endpoints categories of, 235 of severity and comorbidity measures, 235–240
399
Propensity scores, 13–14 controlling for selection bias with, 68–70 matching, 69–70 Protected health information (PHI), 362 Providers of treatment aspects of, 327, 328f comparison of characteristics of, 321t comparison of treatments across, 327–330 studies associated with education of, 325t studies associated with orthodoxy of, 327, 329t studies associated with training of, 327, 329t Proxies, 377–378 PSQ. See Patient Satisfaction Questionnaire Psychological measures, 281t–283t Psychological variables, 265, 271–286 Psychometric testing, 195–196 Psychosocial determinants, 188–190 Psychosocial variables, 266, 267f PULSES Profile, 142
Q QALYs. See Quality-adjusted life years QI. See Quality improvement Quality-adjusted life years (QALYs), 337, 373 Quality assessment, 191 Quality checks, 361 Quality control improvement, 40 Quality improvement (QI), 383–384 Quality of data, 374–376 Quality of health care: utility measures of, 98 Quality of life, health-related (HRQoL), 126, 275–276, 342–346, 373 terminology, 164n2 Quality of Life Instrument Database (MAPI Institute), 157 Quality-of-life measures, 126–127 Quality of Well-Being Scale (QWB), 125, 154, 346 clinical relevance, 134 domains, 149t Quasi-experimental designs, 24, 49–53 combination designs, 52–53 standard nomenclature for describing, 49t types of, 49–53 without control groups, 51 without pretest measures, 51–52
34411_INDX_389_404
400
9/9/05
10:58 AM
Page 400
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
Questionnaires. See also specific questionnaires interviewer-administered (IAQs), 91 self-administered (SAQs), 91, 92 Questions, study, 27–28 QWB. See Quality of Well-Being Scale
R Race, 269–270 RAND, 200 RAND Health Insurance Experiment, 155 RAND Social Health Battery, 143t, 146–147 Random assignment, 67 Random coefficient models, 75 Random-effects models, 75 Random-intercept models, 75 Randomized controlled trials (RCTs), 12–13, 13 Range of health, 132–133, 132t Range of scores, 244 Rasch model, 113 Rating, 197–198 Ratio scales, 94t, 95–96 RCTs. See Randomized controlled trials Readiness to Change instruments, 275t Reality of everyday life (REL), 289 Reference groups, 197 Reference panels, 140 Regression models, 380 Regression-selection interactions, 39 Regression to the mean, 39, 47t Regulatory demands, 362, 368 REL. See Reality of everyday life Reliability, 100–101, 101–104, 135–136 equation for, 101 impact on measure, 132t internal, 101–102 interrater, 103, 103t of outcomes measures, 35–36, 46t of psychometric testing, 195–196 of severity and comorbidity measures, 234 test-retest, 102 Reliability coefficient, 101 Repeated measures analysis, 75–76 Reporting, 197–198 Representativeness, 48t Research issues, 351–388 Research studies: designing. See Study design Research Triangle Institute, 200 Residence, 269
Resource planning, 362 Resources, 355–356 for cost-effectiveness analysis, 347–348 for generic measures, 157 issues to consider before implementing studies, 367 Responsiveness, 108, 132t, 135 Risk adjusters, 217–303 medical meaningfulness test of validity of, 223–224 Risk adjustment, 10–11 Risk-adjustment controls, 55 Risk factors, unmeasured, 66, 67f “Risk factors” (term), 22n1, 388n3 Roland Low Back Pain Rating Scale, 178 Role limitations: SF-36 examination of, 166t Role Overload instrument, 293t, 296 Rosenburg Self-Esteem Scale, 285
S Sample size calculation of, 361 threats to validity and, 29–30, 29f Sampling, nonrandom survey, 74 SAQs. See Satisfaction Assessment Questionnaires; Self-administered questionnaires Satisfaction with care, 185–216 cognitive-affect model of, 186 dimensions of, 190–193, 192f future directions, 210–211 literature reviews, 210 measures of, 198–210 methods of measuring, 194–198 outcomes and, 193–194 patient importance of, 185–186 model of, 186, 187f reporting versus rating, 197–198 theoretical models of, 186 Satisfaction Assessment Questionnaires (SAQs), 207t, 208–209 Satisfaction ratings interpreting, 188–194 psychosocial determinants, 188–190 Satisfaction surveys response rates, 196 timing of, 197
34411_INDX_389_404
9/9/05
10:58 AM
Page 401
Index
Scales and scaling, 94–100 interval scales, 94t, 95 maladjustment scales, 295 nominal scales, 94t, 95 numeric rating scales (NRS), 273t, 277 ordinal scales, 94t, 95 ratio scales, 94t, 95–96 traditional multiple-item, 111–114 types of, 94t, 95–96 visual analog scales (VAS), 273t, 277, 344 weighting/aggregation, 98–100 Scores: range of, 244 Selection, 38–39, 47t survival, 40 Selection-attrition interactions, 39 Selection bias, 66, 67f controlling for with matching or propensity scores, 68–70 with multivariate statistics, 68 through natural experiments, 67–68 through random assignment, 67 identification of, 229–231, 230f Selection-history interactions, 38 Selection-regression interactions, 39 Self-administered questionnaires (SAQs), 91, 92 Self-analysis, 362 Self-criticism, 24 Self-Rating Anxiety Scale, 281t Self-Rating Depression Scale, 281t, 283 Sensitivity, 132t, 134–135 to change, 107–108 SES. See Socioeconomic status Setting, 241 Severity, 219–261 “Severity” (term), 388n3 Severity measures criteria for choosing, 233t data sources, 244–245 domains of health included, 233–234 how to choose, 233–247 issues specific to, 248 measurement properties of, 236t–239t population of interest, 240–241 prognostic endpoints, 235–240 range of scores, 244 reasons to include, 229–232 reliability of, 234 setting, 241 specific measures, 251–260
401
timing of measurement, 241–242 validity of, 234–235 Severity of disease, 220 Severity of illness, 221–222 and comorbidity, 220–222 components of, 224–228, 226t and health domains, 222–224 measurement of, 242–243, 242f overall, 222 Sex, 268 Sexual functioning, 127t, 129 SF-36. See Short Form Health Survey SG. See Standard gamble SHEP trial, 322t Short Form General Health Survey, 171 Short Form Health Survey (SF-36), 124, 147–150, 346 alternatives, 170 clinical relevance, 134 domains covered, 149t fishing and error rate problems, 32 as generic measure, 123 at multiple points in time, 93 range of health, 133 reasons not to use, 167 repeated, 75–76 role of, 180 sensitivity, 135 time frame of, 181 variables included in examined concepts, 166t Short Portable Mental Status Questionnaire, 283t, 286 Sickness Impact Profile (SIP), 88, 150–151 alternatives, 170–171 domains, 149t, 150 fishing and error rate problems, 32 as generic measure, 123 range of health, 133 SIP68 version, 151 strengths and limitations of, 169 Signs: measurement of, 173, 173t Simple measures, 373–374 Single-item measures, 110–116 SIP68, 151 Sites, 74–75 Social Adjustment Schedule, 291t, 296 Social capital measures, 290t, 294 Social Dysfunction Scale, 292t Social functioning, 128–129 example survey items, 127t
34411_INDX_389_404
402
9/9/05
10:58 AM
Page 402
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
measures of, 146–147, 291t–292t, 295–296 SF-36 examination of, 166t Social Functioning Schedule, 291t, 295 Social Maladjustment Scale, 292t, 296 Social measures, 290t–293t Social Readjustment Rating Scale, 274t, 278 Social support measures, 291t, 294–295 The Social Support Questionnaire, 291t Social Support Survey (MOS), 147, 291t, 295 Social variables, 265, 286–296 Socioeconomic status (SES), 287 capital SES (CAPSES), 290t, 294 measures of, 289–294, 290t–292t Sociopsychological artifacts, 196 Sources of information, 90–91 Specialists, 327, 329t Spheres of Control Battery, 272t, 276 Spillover, 45 Spiritual healing, 333n2 “Stages of change” research, 278 STAI. See State-Trait Anxiety Inventory Stance of study, 340 Standard gamble (SG), 112f, 343 Standardization of Self-Reported Pain, 273t Start-up costs, 356 State-Trait Anxiety Inventory (STAI), 285 State Trait Anxiety Measure, 282t Statistical approaches, 35, 36t Statistical conclusion validity, 28–37 threats to, 25t, 46t Statistical controls, 55 Statistical model, 63–76 example, 79–80 Statistical power guidelines for analysis, 31 low, 28–32, 46t planning for, 28–29 standards for considering, 31 threats to validity and, 29–30, 29f Statistical tests inappropriate, 46t violated assumptions of, 33–35, 46t Statistical weights, 139 Statistically isolating effects of treatment, 317–318 Statisticians, 361 Statistics, multivariate, 68 Stress measures, 274t–275t, 277–278 Structural equation modeling, latent variable, 114–116
Study coordination, 358–359 Study design(s), 23–57 combination designs, 52–53 conceptual, 43–44 critical steps, 83 general guidelines for, 53–55 hierarchical or nested outcomes designs, 35 to isolate effects of treatment, 316–317 quasi-experimental designs, 24, 49–53 types of, 12–15, 23–24 Study design questions, 27–28 Study leadership, 357 Study logistical plans, 54–55 Study management, 357–358 Study managers, 357–358 Study questions, 27–28 Study stance, 340 Subgroup analysis, 232, 232f Surveys. See also specific surveys administration methods for, 91 correlation due to nonrandom techniques, 74 example items, 127t mixed modes of administration, 93 at multiple points in time, 93 timing of, 197 Survival analysis, 34, 36t Survival selection, 40 Symptom Fatigue Scale, 171 Symptoms: measurement of, 173, 173t
T t-tests, 177–178 Technology Assessment Program (NHS), 210 Terminology, 22n1 “HRQOL,” 164n2 standard nomenclature for describing quasiexperimental designs, 49t Test-retest reliability, 102 Tests and testing diagnostic tests, 312–313 measurement by condition-specific measures, 173, 173t psychometric testing, 195–196 Theoretical models of satisfaction, 186 Threats to outcomes research validity evaluating, 25–28, 53–54 impact on sample size, 29–30, 29f scheme for classifying, 25t, 46t–48t
34411_INDX_389_404
9/9/05
10:58 AM
Page 403
Index
3MS Examination. See Modified Mini-Mental State Examination Time-to-event analysis, 34, 36t Time trade-off (TTO), 98, 344 Timing of measurement, 241–242, 242f Timing of surveys, 93, 197 Traditional measures, 141 Traditional multiple-item scales, 111–114 Training of providers, 327, 329t Treatment(s), 11–12, 307–308 allocation of, 61 basic aspects of, 12 comparison of, 60–61 bases for, 320, 321t across providers, 327–330 components of, 308–313, 309–312, 313–318 definition by variation, 319–330 versus diagnosis, 309 differences associated with provider orthodoxy, 327, 329t generic approach to assessing, 310, 310t importance of understanding, 307 of interest, 316–318 and outcome, 28 quality differences associated with provider training, 327, 329t unmeasured factors associated with initial health status and, 66, 67f unmeasured factors associated with outcome and, 65–66, 66f unmeasured factors uncorrelated with, 64–65, 65f variation in, 319–330 need for, 318–319 types of, 320–330 Treatment assignments, 61 Treatment costs, 339 Treatment diffusion, 44–45, 48t Treatment effects, 330 capturing, 307–333 designing studies to isolate, 316–317 isolating, 59–81 statistically isolating, 317–318 Treatment implementation, 46t inconsistent, 36–37 Treatment intensity studies, 323–325, 325t–326t Treatment providers aspects of, 327, 328f comparison of characteristics, 321t
403
comparison of treatments across, 327–330 studies associated with education of, 325t studies associated with orthodoxy of, 327, 329t studies associated with training of, 327, 329t Treatment regimens comparison of, 320–324, 321t dimensions of, 320–321 studies of, 320, 322t–323t Treatment setting comparisons, 321t, 326–327 TTO. See Time trade-off
U UKPDS, 326t Uncertainty Principle, 120n1 Unidimensional health status measures, 142, 143t United Kingdom, 188–190 Units of analysis, 62–63 University of Washington, 150 University of Wisconsin, 209 Unmeasured factors associated with initial health status and treatment, 66, 67f associated with treatment and outcome, 65–66, 66f uncorrelated with treatment, 64–65, 65f Unweighted comorbidity measures, 250 U.S. Census Bureau, 289 U.S. Public Health Service Panel on CostEffectiveness in Health and Medicine, 347 Utility measures generic measures, 156–157 of health care quality, 98 multidimensional, 147–148, 149t Utility weighting, 138–139 Utilization patterns, 197
V The VA Cooperative Urologic Study Group, 323t Vaccination, influenza (example analysis), 76–80, 77f Validity, 25–28, 100–101, 104–107, 136 construct, 26–27, 43–45, 106, 107 threats to, 25t, 47t–48t content, 105–106
34411_INDX_389_404
404
9/9/05
10:58 AM
Page 404
UNDERSTANDING HEALTH CARE OUTCOMES RESEARCH
criterion-related, 105 external, 26–27, 45 threats to, 25t, 48t impact on measure, 132t internal, 26–27 threats to, 25t, 37–43, 47t medical meaningfulness test for, 223–224 of psychometric testing, 195–196 of severity and comorbidity measures, 234–235 statistical conclusion, 28–37 threats to, 25t, 46t threats to evaluating, 25–28, 53–54 impact on sample size, 29–30, 29f scheme for classifying, 25t, 46t–48t Variables dummy variable models, 75 instrumental, 70–71 VAS. See Visual analog scales VAS Thermometer (EuroQol), 344, 345f “Very satisfied–very dissatisfied” response scale, 195–196 VF-14, 169 Vietnam War, 19 Visit-Specific Satisfaction Questionnaire (VSQ), 204t, 205 Visual analog scales (VAS), 273t, 277, 344 VAS Thermometer (EuroQol), 344, 345f Vitality, 130 example survey items, 127t SF-36 examination of, 166t
VSQ. See Visit-Specific Satisfaction Questionnaire
W Weed, Lawrence, 375 Weighted comorbidity measures, 250 Weighting/aggregation, 98–100 preference, 138–140 statistical weights, 139 utility weights, 139 ways to calculate, 241 Well-being concept of, 133 measures of, 272t, 275–276 overall, 130–131 example survey items, 127t Western medicine, 309 Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), 170 WHO. See World Health Organization WOMAC. See Western Ontario and McMaster Universities Osteoarthritis Index Work Control instrument, 292t–293t, 296 Work control measures, 292t–293t World Health Organization (WHO), 124, 191
Z Zung Self-Rating Anxiety and Self-Rating Depression Scales, 142, 284, 285
