Evaluating Career Education and Guidance

Evaluating Career Education and Guidance offers a complete basis for evaluation judgements and decisions. It is an invaluable guide for anyone who needs to decide about the merit or worth of career programs or services. The book is aimed at a wide audience of practitioners, researchers and policy makers. It provides a useful reference and guide for anyone who is required to evaluate a career initiative.

Australian Council for Educational Research

Evaluating Career Education and Guidance

The ECCOES framework— Ethics, Coverage, Costs, Objectives, Effects, and Stakeholders—emphasises the perspectives of key stakeholders in any evaluation. It deals with fundamental issues such as the ethics of a program, the cost-benefits of a service and the extent of coverage of any new initiative. Readers are shown the importance of determining whether a program achieves its objectives and its relative effectiveness. Both qualitative and quantitative approaches are discussed.

James Athanasou ­­

Career education or guidance programs vary in their nature and purpose; each operates in a different context, time and space. Through the use of a synthetic and holistic approach, Evaluating Career Education and Guidance describes a framework developed by the author for judging the merit or worth of career programs, policies, services or initiatives.

James Athanasou ­­

Evaluating Career Education and G uidance

James A Athanasou

EVALUATING CAREER EDUCATION AND G UIDANCE ACER Press

To the Estia Foundation of Australia for disabled children

First published 2007 by ACER Press Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell, Victoria, 3124 Copyright © James Athanasou 2007 All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publishers. Edited by Carolyn Glascodine Cover and text design by Divine Design Typeset by Baseline Design Printed by Shannon Books Cover photograph by Purestock/©Getty Images National Library of Australia Cataloguing-in-Publication data: Athanasou, James A. Evaluating career education and guidance. Bibliography. Includes index. ISBN 9780864317971. 1. Vocational education - Evaluation. 2. Vocational guidance - Evaluation. I. Title.

370.113 Visit our website: www.acerpress.com.au

Contents Preface

v

Chapter one What is evaluation?

1

Chapter two What is the background of evaluation?

11

Chapter three Is there a framework for evaluating career education and guidance?

21

Chapter four What information do I need for educational evaluations?

35

Chapter five How do I assess program coverage?

47

Chapter six How do I analyse program costs?

53

Chapter seven How do I describe a program?

63

Chapter eight How do I determine effectiveness?

81

Chapter nine How do I make judgements in educational evaluations?

93

Further reading

101

References

102

About the author

105

Index

106

Foreward

Preface Welcome to the evaluation of career education or career guidance. This is a multifaceted endeavour that addresses real questions about the value of our career programs, policies or services. In this text I examine the application of evaluation to the fields of career education, career guidance, career development, as well as employment programs and services. A general framework for evaluation called the ECCOES model (ethics, costs, coverage, objectives, effects and stakeholders) is offered to you. It is designed to help you make a judgement about the value of a program, policy or service in which you are involved. Although evaluation involves investigation, it differs from research. It is an activity that involves the application of critical thinking to the solution of social, political and administrative problems. More often than not it takes place under conditions, timeframes and restraints that are less than ideal. Right from the outset, I assume that you have expertise in a variety of research methods or that you are prepared to gain the necessary expertise. It is not an onerous task. The idea for this book arose from the need to summarise some ideas on evaluation for my students and other readers. I have tried to emphasise a holistic and synthetic approach to evaluation rather than provide you with a smorgasbord of ideas and methods. It is not a handbook of evaluation methods but a guide to judgement. I hope that you find this guide useful when you come to evaluate careers programs. The content of this book comes from my classes to Masters’ students at the University of Technology, Sydney and some earlier writings on educational evaluation. The career development program at Edith Cowan University has assisted and supported me in the preparation of this manuscript. Readers should feel free to contact me and to make any comments or to point out any errors or omissions. This book developed out of my lecture notes over many years and wherever possible I have tried to provide a reference for my ideas and sources but in some cases I may have omitted specific acknowledgements. I would be grateful if this is drawn to my attention. I wish you well in your efforts to evaluate career education and guidance. James A Athanasou University of Technology, Sydney



Chapter one

What is evaluation?

Most readers would agree that evaluation is an important part of managing that has a place in every field of human activity. It is especially relevant in career education or career guidance in order to ensure the following: the needs of participants are being satisfied; or there are benefits to the community; or the guidance approach is efficient; or the most ethical methods are being used; or that the rights of all interested parties are being maintained. Evaluation is one step to improving career guidance. Its purpose is to provide the information that will help us to make judgements about the appropriateness, effectiveness and efficiency of a program. Many professionals are merely interested in knowing the tricks of the trade in evaluating the contribution of a program to their organisation. Others may want to know the most appropriate ways of justifying their programs, services or policies, especially in times of staff cut-backs or other financial restrictions on services. Some might feel a personal need to evaluate in their field of practice. All of these are worthwhile reasons and linked directly to the purpose of evaluation as the systematic process of investigating the merit or worth of a program, policy, service or initiative in career education or career guidance. So, when I talk about evaluation I mean just the quality and the value. I really mean to ask: ‘Is what I am doing positive or negative, effective or ineffective, worthwhile or not worthwhile, appropriate or not appropriate, ethical or not so ethical, and better or worse than an alternative?’



Evaluating Career Education and Guidance

Evaluation encompasses the word ‘value’. It is sometimes confused with research or inquiry because it certainly involves a variety of research methods, but unlike pure research it has an applied and policy emphasis. The greatest benefit of evaluation is that it helps you to think critically about what you are doing and maybe to look a little more objectively at your programs or services. The skills associated with evaluation are as relevant to your personal life as they are to government policy or community programs – it is really about thinking critically and rationally in order to do what is best. The word ‘evaluate’ comes from the French valuer, meaning to value; and also from the Latin e meaning from and valuere, to be strong and to be of worth. The common-sense meaning of the word is to determine the worth of something or to appraise, in the sense of calculating an amount. In social and educational contexts, evaluation has the formal meaning of determining the value of a program or service, policy or initiative. The United States Joint Committee on Standards for Educational Evaluation defined it as ‘the systematic merit or worth of some object’ (Joint Committee, 1981, 1.12). Evaluation is a natural response to improving any initiative. While informal evaluations occur frequently in the lives of individuals, evaluation as a discipline has achieved prominence as a formal exercise in the area of examining the impact of public programs, policies and initiatives. It is an example of the deeply held view that ‘the methodology of science can be harnessed for the improvement and effective management of social affairs’ (Norris, 1990, p. 11). Some of the basic ideas that supported the practice of evaluation have been summarised in the following way: (a) that research is more authoritative or trustworthy than individuals’ judgements; and (b) that the knowledge of experts will lead to better decisions. This ethos has been described as: a pervasive belief that well-conducted evaluations could, and should, constitute the single most important factor in the rendering of educational decisions. (Popham, 1993, p. 5)



What is evaluation?

The need to evaluate outcomes I have always argued that no program is complete without some formal or informal evaluation. Sometimes claims are made about the effectiveness of a technique and busy practitioners adopt this because it seems to make sense and is considered suitable for their context. There is nothing wrong with implementing reforms but it may also be worthwhile to determine the effectiveness of such approaches prior to wide-scale applications. You need to know whether the approaches you are using are more effective than other approaches and/or whether they are providing some unintended or placebo effect for participants (e.g., social enjoyment, escape from work, morale, group cohesion). You may have concerns about how long-lasting are the benefits of any service or, on the other hand, your programs may be designed to bring about longer-term changes. At the very least you should be able to justify to yourself the ethical aspect as well as the cost benefits (in both dollar, environmental, human and social terms) together with the effectiveness and efficiency of your efforts.

Career education and guidance programs Now this book is unashamedly about career education and guidance programs because this is the area in which I have practised since 1978 but I have worked and taught in other areas of evaluation and the ideas are widely applicable to other educational, social and business areas. The framework that I will outline can be transported across contexts. The evaluation of career education and guidance comes under the broad umbrella of educational evaluation. Some readers with a background in other disciplines, say psychology, might also classify career guidance under the heading of counselling psychology and that is fine for me. The principles are the same. In referring to career education and guidance I would include the following: group counselling, computer-assisted guidance, one-to-one counselling services, career advisory services, telephone career information and



Evaluating Career Education and Guidance

advice, employment services, career information, recruitment programs, supported employment programs, human resource development programs, outplacement services, rehabilitation programs, vocational assessment services, and vocational guidance. You can see that I will take a broad view.

Evaluating the scope of programs or services Evaluations can go beyond looking at outcomes to considering whether the aims of the program are worthwhile. Figure 1.1 summarises the general scope of program evaluation. For instance, you may have an effective and cost-efficient guidance service but one which is not relevant to the aims of the organisation. It is all very well to achieve an objective, but what if the objective is not really that important. For this reason, I encourage evaluators to consider the needs and views of all stakeholders in their judgements and analyses. It is unlikely, however, that any evaluation will address all the needs of groups with a vested interest in a program.

Purposes of evaluations • to improve • for decisions • as a response to controversy • to compare

Evaluations

• to judge

The objects of evaluations • programs • policies

• as a management requirement

• initiatives

• to determine effectiveness • to assess quality • to determine performance

Figure 1.1. The general scope of educational evaluations (Athanasou, 2000)



What is evaluation?

In addition to considering whether outcomes were achieved, whether the aims are appropriate and whether needs are fulfilled, evaluators are also encouraged to consider by whom the results of their evaluations will be used. Will the evaluation be conducted with some cost and effort only to have the results gather dust on some desk or find obscurity in the ‘too hard’ section of a filing cabinet? Career practitioners are encouraged to have regard for their evaluation audience, the sponsors of the program but especially the types of information needed by the decision makers. And a final point, sometimes a formal evaluation of a program or policy is not required because the answer is obvious.

Summary It is not the purpose of this chapter to outline all possible evaluation techniques. In fact there are more than 50 techniques, evaluation philosophies or approaches from which you could choose. There is even an encyclopaedia of educational evaluation. Some approaches are more suited to careers practice than others; some are suited to education, commerce, industry or government; others are useful for program evaluations; and some are related directly to the community. I think that all of them can contribute to your thinking and that they have added to the different ways in which you can go about evaluating services. In particular, as an educator I am proud of the fact that the evaluation of programs is a product of the early work in education, such as the curriculum work of Tyler (1950). In Australia and the United States of America, the evaluation of some health, education and welfare programs is now legislated. Unfortunately many evaluations are piecemeal and wishy-washy efforts to substantiate a service. We can do much better than this and we have available a wide range of research, statistical, economic and educational approaches to assist in evaluations. The conscientious professional can set in place a program of evaluation starting with the collection of routinely available data. My recommendation is that you focus on the qualitative or descriptive analyses of your efforts. If 

Evaluating Career Education and Guidance

these are conducted continuously and taken from various vantage points in your organisation, then you will have an effective means of substantiating your claims about a program. You will also have the necessary ammunition to make reforms. Ideally evaluation is an informative and ongoing process designed to improve human initiatives. While program evaluations are not straightforward, they are not really that difficult to implement. I encourage you to develop skills in this area in order to be able to determine what works and what is not helpful to you. The benefits for you are that evaluations can: • identify needs to be fulfilled • help you choose from different options • enable you to follow the progress of a program • allow you to monitor the personal development of participants • enable you to critically analyse proposals and ideas • focus your efforts on outcomes and criteria • ensure that ethical criteria are satisfied • identify the costs and benefits of a program for you. The next chapter introduces some details of the background of evaluation and its historical development. It offers an introduction to the field. I hope that you find this background of some interest. -oOo-

REVIEW QUESTIONS Here are some questions for you to consider. Do not overlook them the way I do and rush on to the next chapter. They are meant to complement your reading and allow you the chance to apply some of the ideas you have encountered. 1 Why should program evaluation be applied routinely in a careers context? 2 Should evaluation always involve comparisons? Can you think of arguments in favour of, or opposed to, comparative evaluation in a program, policy or service in which you are involved or are likely to be involved?



What is evaluation?

3 A joint committee on standards for evaluation defined it as ‘the systematic merit or worth of some object’ (Joint Committee, 1981, 1.12). Can this definition be applied to your workplace? 4 Evaluation can serve a variety of functions:

(a) a formative function for the improvement of a program or activity, or



(b) a summative function for accountability or accreditation.



If you have not come across these terms, then the following brief definitions may be helpful: formative refers to evaluations of activities that are ongoing and still capable of modification; summative refers to the evaluations of completed activities. Considering the above functions, what would be the most important purposes for evaluation in your work context? What do you think should be the main reason for an evaluation in a program, project or service in which you have an interest?



Evaluating Career Education and Guidance

CASE STUDY Here is a case study for you to consider. It describes a program of online support for students. Read the abstract and then consider some issues. Please note that the abstract is only a brief summary and may not include all the details of the full paper. Work Integrated Learning (previously known as Cooperative Education) is a valued component in undergraduate degrees delivered through the Business Portfolio at RMIT in Victoria. After completing two years at university, students move into workplaces as paid employees for a year before returning to RMIT to complete their degree. There have been recent changes in the funding of work industry placements with the Higher Education Support Act 2003 specifying that higher education providers must ‘provide support’ in order to charge a tuition fee for the placement year. As a result, the concept of support has widened from preparatory job-seeking, resumé writing and interview skills to encompass the student experience while on placement and to assist transition back to the university environment. Online resources have been developed to provide support for completing the required assessment tasks in a workplace setting. Source: Rizzetti, J. & Smith, S. (2005), WIL-ling and able: Online support for students moving between work integrated learning experience and the university. In Searle, J., Beven, F. & Roebuck, D. (Eds.), Vocational learning:Transitions, partnerships and sustainable futures (Vol. 2), pp. 124–130, Centre for Learning Research, Griffith University.



What is evaluation?

My questions to you are: • Is this a program, policy, service or initiative? • Is it important to evaluate Work Integrated Learning? • Is it important to evaluate online support for Work Integrated Learning? • What is the key evaluation question to be answered? • What factors would need to be considered in an evaluation? • How could Work Integrated Learning have been designed so that evaluation was ‘built-in’? • How could the online support for Work Integrated Learning be designed so that the evaluation is ‘built-in’? • What is your view of Work Integrated Learning? • From what you have read and can infer, what is your view of the online support for Work Integrated Learning? • Is online support for Work Integrated Learning just a cynical effort to maintain federal funds? • Is the description too specific to an institution? • Are there lessons that can be generalised to other forms of cooperative education and career education? • What evidence would be necessary to suggest that Work Integrated Learning was not worthwhile? • What evidence, if any, is provided? • What evidence would be sufficient for you to say it is worthwhile?



C h a p t e r t wo

What is the background of evaluation?

Evaluation is a common experience for program providers and is now a component of the funding for many human services. For instance, a careers teacher in a country college may want to compare a newly developed, selfdirected career development program for students with previous career education instruction. An adult migrant education service dealing with people from a professional background may be asked to justify the effectiveness of its work placement program. A trainer for the supported employment placement of persons with a developmental disability wants to determine the costs and benefits of direct instruction on job performance. A local employment network may want to review the scope and influence of its community program in a region. A firm of consultants wants to document the value of its computerassisted guidance program for senior high school students using case studies. A government department calls for a report on the effects of a labour market program. All of these scenarios have something in common. There is a need to evaluate programs in order to determine whether they are worthwhile. How should you go about evaluating such programs? What issues should you consider? Are there standard techniques for evaluations? Are there any guidelines in designing an evaluation?

11

Evaluating Career Education and Guidance

This chapter1 provides some answers by sketching a historical overview of some of the issues in educational evaluation. The development of educational evaluation as a field of practice is outlined and a survey of some evaluation approaches is provided (based on Popham, 1993, Chapter 2). I hope that you come away with the idea that evaluation approaches are continually evolving and that many of your own concerns about how to evaluate have already been addressed by the earlier developments in this field. Before considering the background and development of evaluation, it may help to clarify the use of some terms. New program evaluation standards have been developed and replace the 1981 standards (Joint Committee, 1994) and for the sake of consistency, I would like to use the definitions provided by this Joint Committee in the latest standards (1994, p. 3). Evaluation: the systematic investigation of the worth or merit of an object Object: the object of evaluation, namely educational and training programs, projects and materials Program: educational activities that are provided on a continuing basis Project: educational activities that are provided for a defined period of time Client: the evaluation contractor Stakeholder: individuals or groups that may be involved in or affected by a program evaluation Meta-evaluation: an evaluation of an evaluation.

1

An earlier version of this chapter appeared in Athanasou, J. A. (1995). Issues in the evaluation of adult education. In G. Foley (Ed.), Understanding adult education and training (pp. 75–88), Sydney: Allen & Unwin, and Athanasou, J. A. (2000), Evaluating adult education and training. In G. Foley (Ed.), Understanding adult education and training (2nd ed.), (pp. 81–93). Sydney: Allen & Unwin.

12

What is the background of evaluation?

The background of evaluation as a discipline The early history of formal program provision was not characterised by the need for evaluations as you know them today. Prior to this century there were social ideas of what was best in education. I imagine that there were fairly fixed, common-sense and a priori views about how people responded to programs. These social ideas and common-sense views together with political beliefs influenced the development of services in education and welfare. Accordingly, there was little demand for formal evaluations since pre-determined values and perspectives largely influenced the delivery of educational services. The theory and practice of evaluation emerged in the United States of America, mainly in response to the development of educational initiatives. For example, many existing ideas on education were challenged by educational reformers such as John Dewey (1916). Educational evaluation has now developed to the point where it is a field of study in its own right.

Objectives and goals as a basis for evaluation A change in the concept of evaluation from a priori ideas about what was good or best was influenced by Ralph Tyler with his comparison of progressive versus conventional schooling. Tyler was invited to direct the evaluation staff of the Eight Year Study (1932–40), which arose from concerns that some universities were not admitting high school graduates from progressive schools. These were schools that were developing student-centred education more closely linked to the interests of their students, without regard to standardised curricula. Tyler was among the first to apply evaluation to programs rather than to the mere assessment of people. At the time, this was a significant development in evaluation approaches. A program would now be judged successful according to the extent to which the objectives had been met. Tyler (1950) stated these objectives in observable terms and developed measures or collected data in order to make recommendations concerning the program goals.

13

Evaluating Career Education and Guidance

The objectives model of evaluation offered a rational approach to program evaluation. While it was suited to some programs with very specific outcomes, it was unsuitable as an all-purpose evaluation model. Michael Scriven, who is a philosopher of evaluation, emphasised the importance of rational (scientific) judgement in his paper The Methodology of Evaluation. Scriven (1967) argued that another failing of the objectives model was that there were no procedures for judging the worth of the goals themselves: ‘… it is obvious that if the goals aren’t worth achieving then it is uninteresting how well they are achieved’. He distinguished the formative and summative roles of educational evaluation. A formative evaluation is conducted during a program in order to bring about changes, if required; a summative evaluation summarises evidence and occurs mainly at the conclusion of a program.

Research and educational experiments as a basis for evaluation In the United States of America, a major impetus for the role of evaluation was the Elementary and Secondary Education Act (1965). This was among the first pieces of legislation that required the compulsory evaluation of projects, programs and instructional materials. The concern was whether local agencies utilised funds properly and at that time evaluation was based on research methods and educational testing (see Norris, 1990). The favoured approach to evaluation was some type of comparison such as the educational testing of an experimental and a control group. It reflected an empirical approach based on methods that could ideally be replicated by others. Evaluation projects were treated as social experiments with tests of hypotheses but this type of design was not always capable of dealing with the complexity and specificity of learning. As a result, many of these evaluations were largely unhelpful. These mandatory requirements created a demand for evaluation that was far in excess of supply. It led eventually to dissatisfaction with so-called

14

What is the background of evaluation?

scientific attempts to evaluate broad educational programs. In 1967, Robert Stake indicated the disparity between the purpose of the legislation and the inability of research or testing to produce the desired evaluation outcomes (Stake, 1967a, b). The field of evaluation diversified beyond educational research to such an extent that it is now difficult to characterise. Evaluation is an applied area of methodology that draws inter alia upon disciplines such as psychology, psychometrics, sociology, epidemiology, demography, ethnography, economics and business disciplines. There are some similarities between research and evaluation. They both use disciplined inquiry, assessment methods and data analysis. There are also distinctions between research and evaluation. These centre on the purpose of the activity. The focus of research is the discovery of phenomena for their own sake whereas evaluation focuses on the application of knowledge to decisions, programs and policies. The generalisability of research and evaluation findings also differs. In evaluation studies, there is no intention that results will necessarily generalise to other situations. The importance of the establishing of value or worth has been noted as a key feature of evaluation especially when compared to pure research. One aspect not often emphasised is the extent to which researchers work within abstract constructs and theoretical frameworks whereas evaluation has a more pragmatic focus.

Judgement as a basis for evaluation Judgemental models of evaluation were the next phase after the objectives, the educational testing and educational research orientations. These contributed to the further development of evaluation and moved it on from the objectives-based or effectiveness approaches to focus on merit and worth, while recognising that the outcomes of programs were multi-dimensional. In a 1963 paper Course improvement through evaluation, Cronbach argued that evaluation should deal less with program comparison than with the extent to which a program promoted its desired consequences. Cronbach made

15

Evaluating Career Education and Guidance

other points which are still relevant to evaluators, namely: (a) that there are links between evaluation and decision making; (b) that evaluation performs a number of functions such as curriculum improvement or the review of policies; (c) that follow-up or longitudinal studies are extremely useful in determining the lasting effects of educational programs; (d) that learner performance and/or experimental studies should not be the only focus and method of evaluation; and (e) that measurement approaches to evaluation should be broadened to include the sampling of specific instances and matrix sampling that avoids the need to assess every outcome with every person. Other perspectives on evaluation built on Cronbach’s insights (see Scriven, 1991). A form of goal-free evaluation was advocated (see Popham, 1993, Chapter 3 for a summary). In this approach the evaluator recorded the major effects of a program, which could then be compared with the original purpose of the program. It was a significant learner-centred innovation that acknowledged the unplanned results of education. Performance data were gathered to establish the merit or worth of educational activities, and the evaluator was responsible for judging the merit of the goals. The strength of this approach was that it encouraged educators to systematically investigate educational processes and to critically examine the value of educational goals.

User-oriented approaches as the basis for evaluations In an article ‘The countenance of educational evaluation’, Stake (1967a) went on to address the important question of what kind of evidence an evaluator should collect. This was a user-oriented approach. It involved (a) the description of a program; (b) reporting the description to groups; (c) obtaining and analysing their judgements; and (d) reporting the analysed judgements back to the audiences. User-oriented approaches can be said to reflect the beginning of more descriptive approaches to evaluations.

16

What is the background of evaluation?

Decisions as the basis for evaluation Decision-management approaches to evaluation offered a third dimension of evaluation in addition to objectives-based and judgemental approaches. To Stufflebeam (1971) the evaluation of a program was linked to decision making about four dimensions of the program: Context (information about the aims, desired outcomes), Input (the resources and materials), Process (procedures and strategies used) and Product (the achievements). This approach encouraged educators to systematically generate criteria by which educational programs could be evaluated: context criteria for planning; input criteria for programming; process criteria for implementation; and product criteria for decisions. The value of this model lay in its recognition of the need for holistic evaluation. A program needed to be evaluated in its entirety and through its different phases. Objectives-based, judgemental and decision-making models are only some of the evaluation approaches but they were important in the development of evaluation as a discipline. They give you an indication that previous generations of evaluators have considered program objectives; they have tried comparisons between programs; and they have analysed programs from the perspectives of decision makers or users but no single approach per se has been sufficient for educational evaluations. Although these approaches contributed to an awareness of the merit of programs, they did not always provide the comprehensive basis required for program evaluation.

Summary There are now many ways in which educational evaluations can be approached, including: • objectives-based evaluation (Tyler, 1950) • evaluation for decision making (Cronbach, 1963) • context-input-process-product (Stufflebeam, 1971) • community impact of a program (Stake, 1967a, b) • cost-benefit analysis (Levin, 1983)

17

Evaluating Career Education and Guidance

• evaluation as a judgement of worth and merit (Scriven, 1967) • outcome or output-oriented models (Kirkpatrick, 1975) • illuminative approaches (Parlett & Dearden, 1977). Such a large number of evaluation models is a positive sign in the sense that it reflects the diversity of the contexts in which we work and the variety of views we hold about education. The complexity of career issues and their uniqueness is such that cookbook approaches to evaluation are not always helpful. It is doubtful that one approach is sufficient for all circumstances and a general strategy is required. Experience in evaluations and research quickly reveals that these activities can be highly subjective, political and ideological. The orientation taken in this text is towards an integrated and multidimensional view of evaluation where information from various sources is combined. This approach is more likely than others to capture the complexity of career education processes and guidance services. Another aspect that is relevant is the fact that many evaluation approaches seem to consider issues of effectiveness, usefulness and efficiency from the perspective of an administrator or an organisation rather than from the perspective of the consumers of services. The standards which an educator uses to judge a program may emphasise completely different aspects in the values or the merit of some policy, program or initiative. -oOo-

REVIEW QUESTIONS 1 What major elements of the different models would you emphasise in your evaluation work? 2 Describe/define these models of evaluation briefly:

(a) educational research model of evaluation



(b) objectives-based evaluation



(c) goal-free evaluation



(d) judgemental model of evaluation.

18

What is the background of evaluation?

CASE STUDY Here is another case study for you to consider. It describes a career education program for gifted students. Read the abstract and then consider some issues. Please note that the abstract is only a brief summary and may not include all the details of the full paper. This career education program was developed for gifted high school students in a New South Wales selective high school. This program replaced a previous approach, which was demand-responsive, generic in approach, and not aligned with student needs. A needs analysis of the school and formative evaluations (through a survey of participants, participant observation, stakeholder input and outcome reviews) were undertaken to ensure a successful program re-development. These evaluations resulted in the following changes: earlier and more flexible career awareness opportunities; increased scope for values clarification; inclusion of psychological, psychocreative and social elements; integration of career education and student welfare; lifelong learning emphases; aspiration enhancement for particular students; strategy employment for females; and an expansion in community learning opportunities. The implications for school personnel, students and parents are considered with the aim to deliver a ‘best practice’ career education program for gifted high school students. Source: Boyd, G., Hemmings, B., & Braggett, E. (2000). The development of a career education program for gifted high school students. Paper presented at the Australian Association for Research in Education. http://www.aare.edu.au/00pap/hem00464.htm. Retrieved 16 November 2006.

19

Evaluating Career Education and Guidance

My questions to you are: • What is your view of the career education program? • From what you have read and can infer, what is your view of the responsive and generic approach? • Is the description too specific to an institution? • Are there lessons that can be generalised to other forms of career education? • What evidence would be necessary to suggest that the responsive and generic approach was not worthwhile? • What evidence, if any, is provided? • What evidence would be sufficient for you to say it is or is not worthwhile?

20

Chapter three

Is there a framework for evaluating career education and guidance?

Although people pride themselves on the ability to act rationally, it is not always the case that social programs or educational services are reviewed adequately. As far back as 1987, Fitz-Gibbon and Morris highlighted the neglect of evaluation. They referred to a review of 2000 projects in which there was no acceptable evidence regarding project success or failure (Fitz-Gibbon & Morris, 1987, p. 12). Unfortunately, there is a common view that almost any educational program is beneficial. Well it might be but it will be more certain when you determine standards for the merit or worth of a program. It is not the case that people are reluctant to evaluate but what are some of the barriers? Some of the reasons have centred on a lack of standards, the difficulty in measuring learning, insufficient staff, difficulties in quantifying or measuring, as well as a lack of expertise in evaluation methods (Morris, 1994). In this chapter, I outline a general framework for evaluating educational programs.1 1

An earlier version of this chapter appeared originally in Athanasou, J. A., 1997, ‘Testing competency in a high tech learning environment’, International Quality & Productivity Centre (ed.) Technology-Assisted Learning Conference, Sydney and also in Athanasou, J. A. (1998a), A framework for evaluating the effectiveness of technology-assisted learning, Industrial and Commercial Training 30(3), 96–103, as well as Athanasou, J. A. (2000), Evaluating adult education and training. In G. Foley (Ed.), Understanding adult education and training (2nd ed.), (pp. 81–93). Sydney: Allen & Unwin.

21

Evaluating Career Education and Guidance

Evaluation as a way of deciding worth and merit The introduction of any program or service reflects a personal, social and organisational investment that needs to be reviewed. How can one be a professional if there is no built-in monitoring or some form of program evaluation? Without this, you can end up with sloppy thinking, with poor judgements about the efficacy of services and below average performance. Evaluation can serve as a useful component of all programs and services, ideally in a formative sense to make amendments and improvements as well as a general monitoring role. In a field such as career education or career guidance, which has a proud tradition of theory and practice, there could be nothing more consistent with professionalism than to mandate the evaluation of major programs and services. As you saw in the previous chapter, evaluation was developed largely by educators and has an applied emphasis. It has been refined over many years to become a separate discipline and area of study in its own right, growing from developments in social science research methodologies and the healthy respect that we have for evidence and facts. For the purposes of this section I am concerned with public, structured, formal and summative evaluations and I shall focus on the key steps for evaluating a program or service, mainly in the area of career education and guidance. The aim of this section is to outline a holistic or synthetic approach that you can use to evaluate educational programs (see Figure 3.1).

22

Is there a framework for evaluating career education and guidance?

Ethics

Stakeholders

Coverage

Effects

Costs

Objectives

Figure 3.1. The ECCOES model as a holistic evaluation framework for career education and guidance I do not dwell on the specific plans or procedures (i.e., methodology) in this section but focus on six key issues. Some of the examples that I am using will come from various fields of education or training but the ideas can readily be transferred to all areas of learning within career education and guidance.

Steps in an evaluation of a career education or career guidance program Let me now direct your attention to the criteria that I think should be used in an evaluation. An attempt has been made to provide you with a framework and a structure for your decision making. Six steps are summarised by the ECCOES acronym (ethics, costs, coverage, objectives, effects and stakeholders) and I have called this a holistic approach to evaluation. It is up to you, however, to decide how to combine the answers to these six pieces of information in deciding upon the merit or worthiness of any program or service.

23

Evaluating Career Education and Guidance

Step 1 – Is the program or service ethical? This is the first and foremost question prior to any program evaluation. It takes into account questions such as: • ethics/morality • trust • conflicts of interest • social and political implications • legality or vested interests • any harmful ecological issues • any impact on privacy or confidentiality • any abuse of privilege • whether human ethical guidelines have been satisfied. You may wish to question the ways in which programs are applied or the rationale for services or policies. The latter should probably involve valid social, economic, vocational or pedagogic reasons. To my mind the issue of ethics is paramount; there can be no merit or worth in any program or service if ethical aspects are transgressed. I would not presume to tell you what is appropriate but your own professional ethics and conscience might serve as a starting point.

Step 2 – To what extent do the programs or services cover those who are most in need? The second issue here is the coverage of the program (e.g., coverage efficiency, specificity and sensitivity). While it is important to establish that large numbers of people are capable of being served, it is also important to relate it to questions of: • the incidence of the educational or career need • the prevalence of the educational or career need • the sensitivity of the program • the specificity of the career education or career guidance • the attendance and completion rates for any course. The issue of program coverage is dealt with in greater detail in Chapter 5.

24

Is there a framework for evaluating career education and guidance?

Step 3 – What are the costs, benefits and utilities of the program or service? Most education and guidance programs are labour-intensive in delivery. My initial estimates of the general break-even cost level of face-to-face career guidance (without additional resources) was around A$85 in 2006 per person per hour including on-costs but excluding overheads and capital costs. This can be compared with more flexible forms of delivery where the bulk of the costs are transferred to the production stages rather than to delivery. It is a helpful first step to take into account all the likely costs associated with the production and delivery of any service. These include salaries, on-costs, participant wages, rental and all the other overheads that would be required if the program was to be developed or offered independently or externally. Many development and delivery costs are buried or cross-subsidised and the real expenses are hardly ever considered. See Chapter 6 for a discussion of cost issues.

Step 4 – Did the program or service achieve its key objective(s)? The focus of this question is to concentrate attention on the key objectives of your program. To what extent were you able to achieve these outcomes? Here I am focusing on the gross outcomes or effects of the program. A peculiar difficulty in evaluations is separating out the short- from the long-term effects of career education. You may have the experience that participants evaluate some programs positively but the program has few longterm effects while other programs might not be evaluated well in the shortterm but have significant long-term effects. Moreover, the placebo effect of any program should not be overlooked. A multi-dimensional approach that offers a straightforward, pragmatic and initial basis for an educational evaluation is Kirkpatrick’s hierarchy of evaluation (Kirkpatrick, 1996). This is a comprehensive approach to evaluation that has been around for many years and which is popular in adult education and training. It focuses on four levels of program objectives and is illustrated

25

Evaluating Career Education and Guidance

in Figure 3.2. It is especially suited to training contexts where specific outputs are of interest right from the outset. It is a pity, therefore, that the four levels of program outcome are not used more often.

Results

Behaviour Learning Reaction

Figure 3.2. An evaluation hierarchy (Athanasou, 2000) 1 Reaction – program participants’ estimates of satisfaction provide an immediate level of evaluation 2 Learning – the extent of learning (i.e., skill acquisition, attitude change) that has been achieved is also of interest 3 Behaviour – the extent to which learning has generalised to the work situation or there has been a transfer of skills is assessed at this level through follow-up after a program 4 Results – the wider impact of a program in the community or the organisation is observed at this level in the hierarchy. Most people undertake the evaluations of program objectives only at Level 1 through the use of end-of-course or follow-up questionnaires. This is helpful but is only part of the picture. At the very least, the questionnaires need to be standardised, comparable with other databases of results, anonymous and processed independently. My opinion is that all course participants should be assessed at Level 1. Depending on the size of the group, course participants could evaluate either (a) the quality of the delivery; (b) the curriculum; or (c) the overall course. Further random samples should be taken to assess the

26

Is there a framework for evaluating career education and guidance?

effect of training at Level 2 (say 80 per cent of course participants over time where numbers are less than 300), with smaller group samples at Levels 3 and 4. In conducting these surveys I recommend the benefits of matrix sampling (i.e., asking different questions to individuals and combining results for the group) rather than asking every person every question. The simplicity of the Kirkpatrick model masks some real complexities in the relationships between reaction, learning, behaviour and results. You should also be aware that there are documented criticisms of this approach relating to the validity of the hierarchy and the relationship between the levels (e.g., Tamkin, Yarnall & Kerrin, 2002) but it has served as an influential catalyst for training evaluation.

Step 5 – What is the net effect of the program or service? A challenge for any evaluator is to determine the effectiveness of his or her program or service and to contrast this with comparable programs. This means that you need to be able to describe the impact of a program, its significance and the size of any learning effect. Firstly, the net outcome must be established. You also might want to analyse the differential impact that a program is having for various groups and some analytical framework is helpful. I would refer you to almost any textbook on research designs for some help in selecting the most appropriate method of analysis or design. It is difficult to make any statement about value, however, without a benchmark or some basis for comparison. This is because program evaluation is largely a relative process.

Step 6 – To what extent have the perspectives or interests of all stakeholders been considered and met? Programs and services may inadvertently overlook the interests or needs of users. Consulting clients and obtaining feedback makes you aware of factors that you may not have considered important. Stakeholder perspectives are many and varied (e.g., participants, staff, management, community, government) and some of the questions that you may wish to consider could include:

27

Evaluating Career Education and Guidance

• Does the program, policy or initiative meet the needs of users? • Are intended outcomes useful? • How do people feel about the program? • Have all the key stakeholders been consulted? • What are the perceived costs and benefits of the program? • Have all the stakeholders been treated fairly? • What is important to stakeholders in any evaluation findings of the program or service?

Summary These six questions comprise ethics, coverage, cost, objectives, effect and stakeholder issues. The approach is holistic. These issues synthesise the educational evaluation process for me and I hope that they also provide a concise but comprehensive framework within which you can operate. I think that they cover the most important issues when you are ready to undertake an evaluation of any program or service. The six aspects of the ECCOES model and some of the criteria are summarised in the checklist in Table 3.1. These steps, however, do not provide an automatic answer. The results form the basis for a decision about the worthiness of the program or service. With all the information from these six steps, you then have to decide whether the program has merit. In some cases a program may be costly but very effective; or it may be effective in the short-term and wellliked but may not be meeting long-term needs; or it may be less than perfectly ethical but have wide coverage and popularity; and so on. You will bring to the evidence a set of priorities, which must ultimately influence any decision. That is fine and it is exactly the way it should be – there is no right answer. Some criteria will be more relevant to you than others but all need to be weighed appropriately for your decision. Such decisions involve complex judgements about each career program or service. Maybe the best that you can hope for is to provide information that confirms your ideas about the potential of a program or that refutes claims about the value of a program. In this way evaluations will add to our store of knowledge about what works and what does not work in education and training and how well it works. 28

Is there a framework for evaluating career education and guidance?

Table 3.1. Checklist for holistic evaluation of programs Steps and criteria – ✓or ✗ Step 1 – Is the program or service ethical? • • • • • • • • •

ethics/morality trust conflicts of interest social and political implications legality or vested interests any harmful ecological issues any impact on privacy or confidentiality any abuse of privilege whether human ethical guidelines have been satisfied.

Step 2 – To what extent does the program or service cover those who are most in need? • • • • •

the incidence of the educational or career need the prevalence of the educational or career need the sensitivity of the program the specificity of the career education or career guidance the attendance and completion rates

Step 3 – What are the costs, benefits and utilities of the program or service? • • • •

cost feasibility cost-effectiveness cost utility cost benefit

Step 4 – Did the program or service achieve its key objective(s)? • • • •

reaction learning behaviour results

Step 5 – What is the net effect of the program or service? • comparison with other programs • determine statistical effect size (if relevant)

Step 6 – To what extent have the perspectives or interests of all stakeholders been considered and met? • • • • • • •

does the program, policy or initiative meet the needs of users? are the intended outcomes useful? how do people feel about the program? have all the key stakeholders been consulted? what are the perceived costs and benefits of the program? have all the stakeholders been treated fairly? what is important to stakeholders in any evaluation findings of the program or service?

Evaluating Career Education and Guidance © James Athanasou 2007

29

Evaluating Career Education and Guidance

To return to the perceived barriers for evaluation, firstly it is clear that evaluation does require expertise but it can be acquired. It is not overly complex and is an inter-disciplinary field in which I would encourage your greater involvement (e.g., through your own profession or specialist organisations such as the Australasian Evaluation Society). Secondly, many of the difficulties in evaluation, such as the lack of standards or the difficulty in isolating behaviours often relate to the ways in which programs were established and conceived at the outset. You may need to set your own standards and define the behaviours that you consider most important. Thirdly, the difficulties in measurement or assessment can be overcome by having well-stated and specific learning outcomes which are few in number. Qualitative outcomes can be described using case studies that yield valuable descriptions. Fourthly, the issue of insufficient staff might be considered from the viewpoint of building evaluation into every service component from the outset. The easiest evaluations to conduct are those in organisations that keep excellent records or have set up mechanisms for program monitoring. It is not possible to make career education or career guidance perfect but over time it is possible to reduce errors through evaluations and to improve quality. -oOo-

REVIEW QUESTIONS 1 Early approaches to evaluation focused only on the results or outcomes. The scope of evaluation studies has been extended in recent years to goals, design, implementation process, outcomes (such as Stufflebeam’s (1971) CIPP model). What kinds of information do you think should be collected in an evaluation of a program or service in which you are involved? 2 The achievement of important goals is one possible basis for evaluation criteria. Other criteria include needs of actual and potential clients, ideals, social values, known standards set by experts, or the quality of alternatives. What criteria should be used to judge the merit and worth of a program or service?

30

Is there a framework for evaluating career education and guidance?

3 Show how an evaluator might select criteria for educational programs that are easily assessed but miss the central point of the program. 4 List the stakeholder groups for an educational program in which you might have an interest. 5 Consider information that might reflect the quality of an educational program. Arrange the information in two columns (a) factors that might show the program is effective and (b) factors that might show the program is not effective. 6 Consider the Kirkpatrick approach to the evaluation of training. Indicate its strengths and weaknesses for use in career programs and services.

31

Evaluating Career Education and Guidance

CASE STUDY Assume that you have been asked to evaluate the careers service at a university. The unit helps students who need assistance with career planning and job-seeking. Many of them are young students with little experience of tertiary study and its requirements. The service operates as an individual self-help centre offering one-to-one counselling with some small group programs covering common issues. The unit relies on self-referrals and advertising to encourage students to attend. The staff of the unit comprise a full-time Director, a full-time careers counsellor and three support staff. The Registrar has been concerned about the viability of such programs that are not considered as core areas of activity. The Registrar has asked for an evaluation and analysis of the operations of the careers service. Some of the initial concerns are: • What kinds of students are using the services? • Do they really need the service? • Do students in-need differ from others? • What is the drop-out rate from the counselling, career planning and job-seeking programs? • How are the services being delivered? In your first discussion with the Director you have discovered that some information is available, such as basic demographic information on participants and attendance records. You have been

32

Is there a framework for evaluating career education and guidance?

given permission to interview the Director, counsellors, support staff and students. What questions would you want answered? Which questions deal with service delivery? Is it more important to focus on inputs, processes or outcomes? Outline the structure of a report that you would provide to the Registrar.

33

Chapter four

What information do I need for educational evaluations?

Any evaluation or problem to be solved is socially defined and can be viewed from different perspectives. For instance, job-finding difficulties in unemployed adults might be identified as due to the following: (a) poor knowledge of job-search processes, or (b) the result of a disability; or (c) a deficiency in schooling; or (d) as a wider symptom of socioeconomic disadvantage; or (e) as a result of structural problems in the labour market. Depending on the vantage point that is assumed, then this will have implications for the evaluation process of a labour market program. Identifying the program, policy, product or initiative to be evaluated is a political type of process. It begins even before the client for the evaluation and the evaluator meet. This reflects the notion that both bring to a situation their own preconceptions and values. Sometimes in the process of an evaluation you will come to see more clearly the dimensions of the problem and you may often realise that the client’s perspective might need to be revised in the light of the data you have collected. This is the advantage of an approach to evaluation where the data allow you to answer questions and revise your ideas. You should realise that others would disagree with my view, saying that the whole process is subjective and already constructed within an ideology.

35

Evaluating Career Education and Guidance

In practice, there will be some important pieces of information that an evaluator will need to know about any program. For instance, a key aspect of any evaluation is to be very clear about the main questions to be answered in the evaluation. These should be framed as simple unequivocal questions. The questions should be such that they could be understood by a layperson. I have listed some details that you might want to obtain in order to set the parameters for your evaluation. These details help you to frame the evaluation question. The parameters have been categorised under two headings and can encompass: The nature of the program • the description of the program • details of program participants • an outline of services provided • the program goals, and • the number of sites involved. The purpose of the evaluation • the client’s concerns • the underlying question(s) to be answered by the evaluation • the overt reasons for the evaluation • the need for an outside evaluator • the constraints on the evaluation • the groups likely to have a stake in the outcome of the evaluation, and • the resource commitment to the evaluation. The purpose of this section is to indicate ways in which an evaluator can approach a project. The emphasis is on evaluating an established program. I emphasise formal evaluations of an external nature but much of what I have to say also relates to other contexts. External evaluations are helpful where there is a need to prove a point to others and to have this done by persons for whom there is no conflict of interest in the outcomes. Internal evaluations might be conducted on your own initiative or as an internal requirement. In those cases many of the steps that I describe below and also some of the questions that I list will be quite redundant. 36

What information do I need for educational evaluations?

Where you are seeking to fine-tune established programs you may be aware already of the context but you may still wish to obtain background information or comparative data. In all cases, an understanding of the program, and knowing the expectations of the client are fundamental for the progress of an evaluation.

Collecting and analysing information A first step is to review the information that is publicly available on the organisation, program or policy that you are about to evaluate. This background information helps you to establish a perspective. You then develop some tentative hypotheses, which can be revised in the light of additional information. Use available data from annual reports, or from official publications to help you establish the dimensions of the problem. In searching published data be aware of any trends and ensure that you examine historical data series where they are available. Agency data and files can also be a useful source of information. For instance, an agency may record the training provided to clients and the shortterm employment outcomes. A search of published articles through online databases (e.g., ERIC, Proquest, Academic ASAP) can also provide additional insights but there is no substitute for pre-existing expertise in the field to be evaluated. I am not convinced that there are all-round evaluators; my opinion is that expertise in a field is invaluable and can be supplemented by knowledge of evaluation methods. If you are planning an evaluation in an area in which you are not familiar and/or one in which there is no source of data or publications, then it may be necessary to interview key persons, such as branch heads in a bureaucracy, program managers or directors of services. They will be able to alert you to key issues and their vested interests. A disadvantage of relying only on key persons is the potential bias that might be introduced. In other words, the validity of the information that you obtain might be compromised. These processes also enable you to define the stakeholders and the target groups for the policy or program. In the evaluation of training programs 37

Evaluating Career Education and Guidance

for unemployed persons it may be necessary to divide the target population into those with varying lengths of unemployment since there is a qualitative difference in the experiences of short-term, long-term and very long-term unemployed. There will also be scope at the planning stage for you to consider how the evaluation might be conducted or how it is required to be conducted. All this presumes that evaluation is proceeding under ideal conditions. My experience has been that clients, stakeholders and available resources can place constraints on an evaluation. There are many possible extraneous influences. A steering committee for a project may not be able to meet at the required time to give approval for the next stage and this can seriously delay progress in a project. Stakeholders with vested political interests can lobby for their perspectives to be included in any contracted evaluation. In many ways the evaluator is a type of social program detective, unearthing evidence and making sense of clues. Some programs will be easier to evaluate than others. Some will have specific and clearly stated objectives while others will have been implemented in an ad hoc way. In my experience there has nearly always been some vested interest. The comments I made refer mainly to established services rather than proposed programs. I am using programs and services to refer to any object of evaluation such as a careers initiative, an educational product, an organisation, a policy or a program. A further qualification that I would like to make is that the process of preparing for the evaluation may need to operate within a tendering and submission process so that many of these desirable steps may not be possible for an evaluator to undertake until after the approval for the evaluation to proceed has been granted. The evaluator often has to make a submission so they can then be shortlisted from the set of tenderers and in this submission the evaluator indicates the proposed methodology on the basis of limited information. This may pre-empt many of the steps I have described or they may be included in a tender as initial steps in the formal evaluation process. The details of the evaluation will depend on your approach and it is recommended that the evaluator seek answers to a range of questions. Three overarching questions have been recommended (Stecher & Davis, 1987, 38

What information do I need for educational evaluations?

p. 44): what is the program?; why is it being evaluated?; and what constraints limit the evaluation? These three questions are a useful start to providing you with information about the size of the problem or the key issue, the types of clients served by a program and the social context in which it occurs. The aim is to generate a clear statement for the evaluation. What is the key question? In many ways it can be helpful to focus the evaluation by phrasing it in terms of a specific question(s) that can be answered ‘yes’ or ‘no’ (e.g., Has this system of instruction enhanced job acquisition?). This is the same aspect that I teach to doctoral students about to embark on a research thesis – the phrasing of the question is a critical step, as it will focus your thinking and investigation. The analysis in the following sections is based on Stecher and Davis (1987, Chapter 3), Owen and Rogers (1999) and Herman, Morris and Fitz-Gibbon (1987).

What is the program? This first question covers basic familiarity with the details of the program. There are a number of sub-questions about the clients, the goals and the processes (see Table 4.1 for a classification of some questions adapted to career education and guidance). All of these provide some insight into the extent and size of the career problem being addressed by a particular program, policy or initiative. By asking these questions you start to sort out what are the important facts for your evaluation of the program. You come to understand some of the factors influencing the evaluation problem. These questions are especially useful when official and published information is lacking, when agency records are inadequate or inaccessible to you or there is little previous research upon which to rely. There is no substitute, however, for experience and knowledge in a field and if an evaluator has these qualities, then they know which questions to ask. This range of questions can be asked of a wide range of people from those who designed or developed the program, from those who have to implement it as well as those who are most affected by it in the community. You can ask 39

Evaluating Career Education and Guidance

these questions of people within an organisation as well as of people who are outside the organisation. Some of the groups you may need to contact include the following: government departments; bureaucrats at all levels; managers; client groups; staff; or community organisations. The questions being asked centre on the goals and objectives for the program. It is helpful to clarify these objectives when assessing the potential impact of any service. Sometimes selective interviews or small pilot studies can be helpful to give you a better understanding of the dimensions of a problem. Throughout this process a type of hypothesis generating and testing process is beginning to take shape. You are also seeking to determine the size and extent of the problem addressed by the program or service as well as the resource implications and the characteristics of the clientele. Some specific evaluation benchmarks have been listed by Rossi, Freeman and Lipsey (1999, Chapter 3). To summarise: Incidence: the flow of new clients or the number of new people with a problem (e.g., new recruits, newly retrenched) Prevalence: the rate of occurrence of a problem (e.g., rate of unemployment, extent of persons with lower than average literacy skills) At risk: the potential group of persons who might be likely to encounter a problem (e.g., social and economic disadvantages leading to poor completion rates) Sensitivity of a program: the adequacy of the criteria used to identify the appropriate persons covered by a program (e.g., sensitivity of a screening test to identify persons with learning disabilities) Specificity of a program: the adequacy of the criteria used to exclude or reject those unlikely to benefit from a program (e.g., the value of a selection test to diagnose those unlikely to benefit from training). Taken together, these five benchmarks help to establish the quantitative dimensions of the career issue or problem. In asking all of these questions, it soon becomes apparent that different groups will have perceived the issues and the career problem quite differently. 40

What information do I need for educational evaluations?

Why is the program being evaluated? The answers to this question help the evaluator establish his or her role and client expectations. In most cases evaluators aim to provide evidence of accountability or to justify the existence of a program. The questions summarised in Table 4.1 indicate whether the evaluator needs to focus on the coverage of the program, the services it provides, its impact, efficiency and/ or financial aspects for particular groups. These basic criteria are the ones upon which most people will base their decisions, either rightly or wrongly, for or against a program.

What constraints limit the evaluation? The evaluator also needs to know to what extent there are constraints that limit the process (e.g., deadlines, budgets). Some potential questions that are relevant to career education and guidance are also summarised in Table 4.1 on page 42.

41

Evaluating Career Education and Guidance

Table 4.1. Evaluation questions for career education and guidance

What is the program? Clients • Details of the program participants and non-participants • Ostensible reasons for involvement in the program or service

Goals • The scope of the program’s goals and objectives • An indication of the desired outcomes

Process • A description of the career education or guidance program • An outline of the specific services provided • The underlying career theory or guidance rationale for the program • The schedule of activities

Organisation • Information on the staffing and the sites for program delivery • Details of funding • Information on organisation and funding • Background on any changes in the program or its organisation • The nature of the information that is publicly available about the program

Why is the program being evaluated? Immediate purposes • The reasons for the evaluation of the program or the service • Expectations about the evaluation process and the outcome • Indication of any special concerns about the program or the service

Audiences for the evaluation • Type of information that will be of most value to stakeholders • Type of information that will be of most value to decision makers

What constraints limit the evaluation? Budget • Details of the funding (if any) available for the evaluation • The extent of staffing or other support (if any) available for the evaluation

Schedule • Details of the final and interim reporting dates • Availability of information • The extent to which the findings of the evaluation are confidential • The extent of access to the individuals, organisations and sites related to the program

Source: Adapted from Stecher & Davis (1987, Chapter 3), Owen & Rogers (1999) and Herman, Morris, & Fitz-Gibbon (1987). Evaluating Career Education and Guidance © James Athanasou 2007

42

What information do I need for educational evaluations?

Summary Some fundamental steps in the planning stage are now summarised below in Table 4.2. Table 4.2. Proposed steps in planning for evaluations of programs Proposed steps in planning for evaluations of programs 1 Check background information and details. 2 Specify the question the evaluation will seek to answer. 3 Identify the aims and objectives of the program. 4 Ensure you are familiar with the content of the program. 5 Describe the stakeholders and client groups. 6 Consult key decision makers. 7 Consider how the program ought to be evaluated. 8 Prepare submission or tender or the evaluation plan. 

It is not necessary to follow each of these steps religiously or in a specific sequence. They offer a roadmap and broad guide as to the direction in which your evaluation planning might travel. With this background and information you would now be in a position to address any ethical concerns. I do not write about these matters in a separate section because I find them difficult to describe in advance. Nor am I sure that the topic benefits greatly from a lengthy description. Ethical concerns can be quite straightforward where they reflect matters of conscience and what is right or wrong. At this point you should have enough information to decide about the ethics or morality of the program or service, the social and political implications of your evaluation, whether there are issues of legality in what you are doing, whether you have a vested interest in some outcome, as well as broader issues relating to your own values. Once again, I would suggest that your personal conscience serves as a guide. If you have an intuition that your involvement in a program or your evaluation of it has ethical problems then my recommendation is that it is best to walk away from it. -oOo-

43

Evaluating Career Education and Guidance

REVIEW QUESTIONS 1 Evaluations can have more than one client. These may have different needs. Who should be served by an evaluation? 2 Document your answers to the following three questions for a program, service, policy or initiative in which you are involved or are likely to be involved.

(a) What is the program?



(b) Why should the program be evaluated?



(c) What constraints would limit any evaluation?

3 How do answers to some of the following questions about a program or service affect your evaluation?

(a) What are the components of the service?



(b) How was the service provided?



(c) What was the frequency and duration of the service?



(d) How many people are being serviced by the program?

4 Do you agree that people who are personally interested and involved in an evaluation are more likely to use evaluation findings? 5 Under what conditions would a needs assessment be threatening to an organisation?

44

What information do I need for educational evaluations?

CASE STUDY The career education quality framework was designed as a useful document for internal evaluation of a career education program. The Careers Education Quality Framework is a ‘guide for those who guide’. It provides teachers, careers specialists, school leaders and school communities with a practical tool for self-assessing Careers Education and developing action plans for improvement … The Framework is a development tool for a school’s Careers Education program. It is most effectively used when stakeholders work through it together and jointly produce an Action Plan for the coming 12 months. This group could include parents, school staff and members of the community (p. 2). This document can be downloaded from http://www.dest.gov.au/directory/ceqfwholebook.pdf. Source: Department of Education, Science and Training, 1999. Note that some aspects of the Careers Education Quality Framework have been superseded by later developments. Both the career and work preparation outcomes framework and the Australian Blueprint for Career Development also contain ideas for evaluating career education programs.

Indicate how this Careers Education Quality Framework approach might be applied in your work context and to what extent it covers the relevant issues for an evaluation.

45

CHAPTER FIVE

How do I assess program coverage?

Questions always arise in my mind whenever someone boasts about their career education program or service. Before I even ask about impact I often want specific answers to questions about the nature of the program, the clientele and the coverage of the service. Answers to questions about the coverage of a program require meaningful information about its constituents. Some of the questions have already been answered in the planning stage of the evaluation but some more may remain to be answered by further investigations. In the last chapter I considered how you go about determining the focus of an evaluation. This included obtaining some details about the scope of a program and its operations. This chapter emphasises data on a program’s coverage. These data are required to describe a program’s operations so that its implementation can be monitored or some aspects of its coverage can be assessed. An evaluation of a program or service cannot proceed in the absence of data and observations. These can range from quasi-standardised personal reflections to systematic analyses of existing records in an organisation (see the continuum in Figure 5.1 on page 48).

47

5813-05-Chapter 5.indd 47

16/02/2007 11:57:30 AM

Evaluating Career Education and Guidance

Personal reflection

Observation

Data collection

Management information system

Figure 5.1. A continuum of information for evaluation (Athanasou, 2000)

How many people are being serviced by the program? Participation in a program is a convenient index of the performance of a program. The question to ask is: ‘How many people participated in the program?’ One of the first factors that needs to be considered in any analysis of the answer to the question includes the extent to which the program is voluntary in nature. If it is voluntary then you get an idea about how acceptable the service is in proportion to the target population. A second factor that needs to be considered is whether the program is servicing only a select group such as a minority, or only those able to access a service, or those who are aware of a service rather than the entire population. For instance, a labour market program might service only those who were most likely to be employed in any instance rather than those who are at the highest risk of being long-term unemployed. Programs and services by their nature often develop selectivity over time. Different groups may come to participate at varying levels with inequities developing in meeting the original aims. You will need to be aware of this factor and assess whether there is any bias in the delivery of a service. Such selectivity or bias occurs when client selection is not directed towards the target population. A service can be operating profitably and to everyone’s satisfaction but also providing programs to those who are not in need. The first pieces of information required to support any claim about a service are to determine aspects of the target population. You need to know: • the size of the target population (e.g., those of non-English speaking background in a local government area)

48

5813-05-Chapter 5.indd 48

16/02/2007 11:57:31 AM

How do I assess program coverage?

• the proportion of people to whom the service is delivered • the proportions of people in need and/or not in need to whom the service is distributed. Two aspects of a program’s operations need to be discerned with respect to coverage. The first is what proportion of those in need are being serviced. The second is how many of those who are being serviced are really in need. people in need who are serviced by a program Program coverage = total number of people who are in need The coverage of a program gives you an indication of the extent to which the service is meeting needs. It is an index of participation. The coverage of the population is important because a program or service may only be reaching a minority of the eligible participants. An important political question to be answered is whether it should be discontinued if it is not reaching a significant proportion of the population. A vocational guidance service claimed that it served 50,000 per year but counted services and not clients; moreover it did not consider the clientele as a proportion of the population of some 750,000. There was no analysis of whether the service was really needed by clientele or whether other groups were excluded. It is important to distinguish both the target population and the entire population for the purposes of program coverage. The following formula has been suggested. people in need who Extent of needs serviced correctly

are serviced by a program = total number of people serviced by a program

49

5813-05-Chapter 5.indd 49

16/02/2007 11:57:31 AM

Evaluating Career Education and Guidance

This leads to considering a measure for whether particular groups within a community are being serviced. Rossi, Freeman and Lipsey (1999, p. 211) have suggested that the efficiency of coverage can be calculated by this formula: Number in Coverage efficiency = 100 ×

need served ____________

Number not _

in need served _____________

Total number

Total number

in need

served

Source: Rossi, Freeman & Lipsey (1999, p. 211)

Example: Population

Program

50 in need

20 in need

150 not in need

30 not in need

Coverage efficiency =

20 30 – = 0.4 – 0.6 = - 0.2 or –20% 50 50

Negative values occur when inappropriate groups are being serviced. The conceptual basis for this index can be difficult to interpret because there are no common units in the formula. It should be used with caution as the index will be subject to fluctuation depending on the proportions involved and may not always apply to particular groups. Finally, I would always want to know how many people continue in a program. Attrition rates for many social and educational programs are very high and some official sources are embarrassed to provide data on drop-outs from programs.

Sources of information Where do you obtain the information for these calculations? Obviously program records are important. If these do not exist (and you should be

50

5813-05-Chapter 5.indd 50

16/02/2007 11:57:31 AM

How do I assess program coverage?

asking why they are non-existent), then you may need to establish a record system for the duration of the evaluation. If a record system is established for the evaluation, then there are many practical considerations about the quality of the data obtained. For this reason, it may be necessary to collect only a minimum amount of data, (a) so that agency staff is not opposed to the procedures established, and (b) also to ensure that aspects of client privacy and confidentiality are maintained. When programs and services are diverse in nature and/or it is difficult to obtain information directly from an organisation, then you may need to consider community surveys as a source of information. Such surveys can encompass participation rates as an index of awareness; drop-out rates as an index of dissatisfaction; and failures as an index of program effectiveness. This brief section has emphasised the initial analysis of program operations. It acts as a basis for the next step in an evaluation. In some instances the information obtained at this stage may already be sufficient to answer the evaluation questions. For instance, a program may not be achieving efficient coverage, may have high drop-out rates during its operation and poor success rates after program completion. Accordingly any further analysis may be redundant. -oOo-

REVIEW QUESTIONS 1 Describe and analyse the coverage and efficiency of coverage of a program or service in which you are involved or likely to be involved. Comment on your findings. 2 How does program monitoring fit into the function of program evaluation? 3 How does monitoring relate to formative and summative evaluation? 4 What types of program evaluation questions can be answered using an information system?

51

5813-05-Chapter 5.indd 51

16/02/2007 11:57:31 AM

Evaluating Career Education and Guidance

CASE STUDY 1 The Career Horizons Program is located at a large Mid-western university [in the USA]. It provides at-risk students going into seventh grade with an intensive summer program combining career classes with exposure to a college campus and recreational team-building activities to enhance students’ career selfconfidence and vocational considerations. Source: O’Brien, K., Dukstein, R. D., Jackson, S. L., Tomlinson, M. J., & Kamatuka, N. A. (1999). Broadening career horizons for students in at-risk environments. The Career Development Quarterly, 47, 215–228.

What program coverage questions would you raise in conjunction with this program for students who are considered at risk? 2 This table shows the pattern of services requested from a university careers service. Reasons for attendance Per cent Job search 28 Interview skills 4 Resumé writing 13 Course issues 13 Course change 8 Career direction 28 Prospective student 3 Industry placement 1 Other 2 What program coverage questions would you ask about this service?

52

5813-05-Chapter 5.indd 52

16/02/2007 11:57:32 AM

Chapter six

How do I analyse program costs?

Efficiency in career education and guidance programs is linked to their costeffectiveness and/or the cost of the benefits that they provide. This is the way these terms are defined in the second edition of the Program Evaluation Standards (Joint Committee, 1994): Cost: the total value, social and monetary of all the human and physical resources used to carry out the program (p. 77) Benefits: the value of all the results and outcomes (p. 77) Cost-effectiveness: the extent to which one program, project or instructional material produces equal or better results than competitors’ programs that cost about the same amount of time, effort and resources; or the extent to which an object produces the same results as competitors’ but is less costly (p. 204).

How can you determine the efficiency of a program or service? This aspect of many programs is not always determined and in many cases it is overlooked. My experience is that some large-scale programs or services are 53

Evaluating Career Education and Guidance

well below standard in their level of efficiency. What do I mean by this? Firstly, the cost of the benefits (human, economic, ethical, social) is greater than comparable programs. Secondly, the effectiveness might be much less than what is stated. Certainly, long-term effectiveness is rarely established. This does not mean that career education and guidance should be measured only in terms of cost. Nor does it mean that effectiveness or efficiency is the only criterion. Education and guidance are not production lines. Sometimes less effective approaches might suit a community or costly methods might have long-term advantages for some individuals. This is fine provided you know about it. You should also consider the social costs and efficiency of any program. At the present time it is not clear to me how this could be done in a way that would have overall consensus. What I am saying is that you should know at least what the costs and benefits are and whether the efficiency is as high as is claimed. You should know which program produces the same impact for a given level of expenditure. This chapter covers issues that relate financial costs to program results. The ideas are derived from business contexts but also relate to any reasonable evaluation where financial expenditures are involved.

Efficiency Efficiency is a relative concept. There is no maximum level of efficiency so standards need to be established. The standard would involve best practice in career education or career guidance or some other yardstick. You should also realise that efficiency will depend on resources, equipment, size of program and quality of management. In this chapter I look at costs and efficiency as a static concept. In part this is misleading since it fails to consider the development of a program and the fact that these indices change over the natural history of a program. The advantage of measuring efficiency is that it makes you think about a program’s advantages and disadvantages. It disciplines your evaluation of claimed benefits. You can undertake an efficiency analysis in the planning of a program or after a service has been offered for some time. 54

How do I analyse program costs?

It is also relevant to determine costs and benefits (financial, personal, community and social) prior to establishing programs. Too often governments establish programs and services and people then build their careers and working lives around these services, only to have them withdrawn at a later stage. Both clients and agency staff are greatly dislocated by such policy inefficiencies. It is difficult to recall any major human services that have not been subject to policy whims or changes. Secondly, some excellent policies are often implemented in less than effective ways. Prior analysis of likely efficiency is relevant where services cannot be easily withdrawn or where capital investments are involved. Post-program costbenefit analyses are relevant in order to examine the effectiveness of a program or to compare the efficiency of a service or to record its impact. I realise that converting outcomes to money terms is not straightforward and that not all inputs into a program can be costed. Effectiveness and efficiency, however, do not always require you to estimate financial costs. There are human, social and material outcomes and costs that can be considered. You could compare training in terms of the time required to learn; or you could look at the efficiency of an educational service in terms of long-term social effects on individuals; or you could estimate the time required to implement a new system of instruction.

How to calculate costs Costs can be expressed in monetary values but they can also be compared in terms of alternatives. They can be calculated in resources required or in terms of the value or benefit derived. Costs include labour, rent, facilities, materials, capital equipment, resources and services. You may need to add any costs undertaken by participants in an employment context (e.g., textbooks, uniforms, travel, tools). There are different methods of cost analysis including (a) cost feasibility; (b) cost-effectiveness; (c) cost benefit; and (d) cost utility.

55

Evaluating Career Education and Guidance

Cost feasibility = cost of program – funds or resources available Cost effectiveness = cost of program or the effect of one program compared with another program on some common measure Cost feasibility: Cost feasibility answers the question of whether I have sufficient funds available. It is useful for eliminating options that are not viable. If the costs of three employment training programs are $1248, $4589 and $2512 per participant and the budget available is $1.8m, then only the first and third alternatives are feasible if you want to train at least 600 people. Cost-effectiveness: Cost-effectiveness is useful when programs or services have the same outcomes or goals. There must be a common measure for costeffectiveness. Effectiveness allows you to rank programs and services in terms of their costs for achieving the same outcomes. A hypothetical example for calculating effectiveness is shown in Table 6.1. Table 6.1. Example showing how to calculate cost-effectiveness Job-finding services

Additional cost to college

No. of cases 2007

Effectiveness per person helped

Direct instruction Individual counselling Supported placement No treatment

$88,000 $92,000 $71,000 na

47 32 18 7

$1872 $2875 $3944 –

Cost benefit: Cost-benefit analysis allows you to compare different alternatives when their inputs and outcomes are measured in dollar terms. If money values have not been assigned to inputs and outputs, then these must be determined in some fashion. The cost-benefit analysis requires you to subtract the cost from the benefit. It can also be expressed as a ratio. A hypothetical and simplified example that compares the effectiveness of two programs for an organisational career development program is shown in Table 6.2. Note the larger net benefits for one program over another but also that the benefit to cost ratio of one over the other is markedly different. This highlights one weakness of the costbenefit approach.

56

How do I analyse program costs?

Table 6.2. Example showing how to calculate cost benefit for a corporate career development program

Computer-based recruitment service

Career information publication

Benefits Costs Net benefits Benefit–cost ratio

$5.6m $0.8m $4.8m 7.0

$3.4m $0.3m $2.9m 11.3

Discounting: Career education and guidance are considered to have shortterm as well as long-term benefits. In order to estimate the present value of future benefits, you can discount these future benefits. For instance, a program may claim to provide future benefits of $200,000 per year to a corporation and over a five-year period it is stated that around $1 million will be saved. This amount should be reduced to its present value. In order to do this, you discount it by any amount you choose. If you allow for an inflation rate of around 5 per cent over the period, then the value in present dollars would be equivalent to $865,893. Table 6.3. Example showing how to calculate present value Present value of an amount = Amount/(1 + r)t Example: Year 1 $200,000 ÷ (1 + 0.05)1 = $190,476 Year 2 $200,000 ÷ (1 + 0.05)2 = $181,405 Year 3 $200,000 ÷ (1 + 0.05)3 = $172,767 Year 4 $200,000 ÷ (1 + 0.05)4 = $164,540 Year 5 $200,000 ÷ (1 + 0.05)5 = $156,705 Where t = period of time

Cost utility: Programs and services can be analysed in terms of their costs and monetary benefits but they might also vary in their usefulness and in other values. To undertake a utility analysis, a program or service must be rated in terms of its utility. A scale of values can be determined, say from 1 to 5: 5 – Very high utility (top 10%) 4 – High utility (next 20%) 3 – Average utility (middle 40%) 2 – Low utility (next 20%) 1 – Very low utility (bottom 10%)

57

Evaluating Career Education and Guidance

Note: I prefer to take the logarithm of these numbers as the units from 1 to 5 are not equal; the spacing between the logarithms is for me a better indication of real differences; for example: log(5) = .69, log(4) = .60, log(3) = .47, log(2) = .3, log(1) = 0. I can now apply cost-utility analysis to the previous two examples (see Table 6.3). In the case of the job-finding services (Table 6.4), the high value placed on the individual counselling means that this program will be favoured in a cost-utility analysis even though it is not as cost-effective. Similarly, in the case of the corporate career development program (Table 6.5), the greater utility is for career information over computer-based recruitment. In many instances the cost-utility approach provides you with contradictory findings because educational phenomena cannot always be weighted and costed in the same way as other inputs and outputs. Table 6.4. Example showing how to calculate cost utility of job-finding services Program

Direct instruction

Individual counselling

Supported placement

No treatment

Additional cost No. of cases Effectiveness per person helped Utility to college policy Cost-utility

$88,000 47

$92,000 32

$71,000 18

nil 7

$1872 2 $44,000

$2875 4 $23,000

$3944 3 $23,666

– 1 na

.6 $38,333

.47 $50,353

0 na

Using logarithms of utilities Utility to college policy Cost-utility

.3 $29,333

58

How do I analyse program costs?

Table 6.5. Example showing how to calculate cost utility of corporate career development

Computer-based

Career information

Benefits Costs Net benefits Benefit–cost ratio Utility Cost-utility

$5.6m $0.8m $4.8m 7.0 3 $266,666

$3.4m $0.3m $2.9m 11.3 5 $60,000

.47 $1,702,127

.69 $434,782

Using logarithms of utilities Utility Cost-utility

-oOo-

REVIEW QUESTIONS 1 Which cost-analysis procedure do you believe offers the most useful basis for analysis: (a) career education programs; (b) career counselling services; (c) computer-assisted guidance? 2 What would you identify as some of the costs associated with the provision of a careers service at a secondary college? 3 What would you identify as the benefits of (a) an employment training program for the long-term unemployed; (b) an interactive multimedia career education package; (c) an associate diploma course in career development at a regional technical college; (d) a job training program for long-term unemployed? 4 For a service or program in which you are involved (or plan to be involved or interested), undertake an analysis of the costs and benefits. Apply those cost-analysis procedures that you believe offer the most useful basis for analysis. 5 Compare cost benefit and cost-effectiveness in terms of the decisions that might be made for a career education or guidance program in which you are involved or interested.

59

Evaluating Career Education and Guidance

CASE STUDY 1 Consider the table below, which is reproduced from ‘Restoring Full Employment’. In your opinion, which of the labour market programs described is (a) most cost effective; (b) has the greatest cost benefit; and (c) appears to have the greatest cost utility? Justify your response and/or indicate what further information is required to make a conclusion. Outcomes and net costs of labour market programs 1992–93 Program

Proportion of Proportion of Net unit cost Net cost to long-term clients assisted of assistance each positive unemployed with a positive outcome targeted by outcome the program % %

JOBSTART

62.0

58.9

$1,519

$2,572

Job Clubs

58.0

47.0

$625

$1,460

Skillshare

58.2

41.8

$747

$1,791

JOBTRAIN

67.4

36.4

$1,213

$3,226

Special intervention

65.1

40.7

$1,088

$2,572

100.0

44.0

$9,055

$20,768

53.7

73.0

$7,218

$9,888

JOBSKILLS Self-employment assistance

Notes: A positive outcome is defined as unsubsidised employment some three months after assistance ceases. Net costs take account of direct offsets and take into account that some job-seekers would have found jobs without assistance. Source: © Commonwealth of Australia, reproduced by permission from Committee on Employment Opportunities, Restoring Full Employment, Canberra, AGPS, 1993.

60

How do I analyse program costs?

2 Now consider the estimated return in Social Security spending and the tax paid for each dollar of program spending which is listed in the following table. Which of the three programs (JOBTRAIN, JOBSTART, or JOBSKILLS) would you support as being most cost-efficient, cost-effective and having the greatest cost utility?

Source of saving for each dollar spent

JOBSTART wage subsidies (cents)

JOBTRAIN courses (cents)

JOBSKILLS work experience (cents)

Social security savings during the life of the program

26

47

26

Social security savings after the end of the program

30

1

5

Tax paid by participants during the life of the program

4

< 1

6

Tax savings after the end of the program

4

< 1

<1

Tax savings for $ spent

64

50*

37

*The original table (p. 103) showed 60 cents.

Source: © Commonwealth of Australia, reproduced by permission from Committee

on Employment Opportunities, Restoring Full Employment, Canberra, AGPS, 1993.

61

Chapter seven

How do I describe a program?

Although there has been considerable research in career education and some has documented the experiences of participants, there have not been as many major program evaluations of education and guidance programs as one might expect. Maybe there is a sense that any form of career education or guidance is intrinsically good and therefore should not require evaluation. Furthermore, there are aspects of the ECCOES framework (such as ethical dimensions and stakeholder perspectives) that demand a descriptive approach to evaluation. Career education settings, however, can vary considerably in content, purpose, location, funding and organisation. They can include formal structured programs as well as informal services. The heterogeneous scope of programs means that varied forms of evaluation are required. Accordingly, the more qualitative or descriptive evaluations may also find application in such settings.

Qualitative versus quantitative approaches to evaluation The descriptive methods used by evaluators contrast markedly with experimental and quasi-experimental designs; they stand in stark contrast with the statistical analyses of some survey research. This is because the 63

Evaluating Career Education and Guidance

emphasis is on the analysis of a specific situation in relation to the evaluation question. It does not mean that descriptive approaches are less rigorous or objective. They allow greater scope for speculation, explanation and interpretation because they tend to study only a single setting (or a small number of settings) and focus on depth rather than breadth of analysis. Unlike other evaluation approaches they involve the personal presence of an evaluator. There is an emphasis on (a) processes; (b) interpretations; (c) constructing the social world of the program participant; and (d) an understanding of the individual that arises from the situation in which he or she behaves. The distinction between qualitative and quantitative approaches has a long history in social science methodology. Much of the discussion is metatheoretical (i.e., the theory about theory). Support for descriptive techniques has emerged as a reaction against approaches in which theory is inferred as a result of testing hypotheses and the mindless application of statistical methods; qualitative approaches emphasise phenomenological and interpretive paradigms where theory is generated from the data collected. I am not sure, however, that labels such as quantitative versus qualitative or interpretive versus positivist will be particularly helpful to you or for your client groups when you undertake evaluations. The key issue is to decide pragmatically what information will aid in your judgement of the merit or worth of a program. No doubt, there will always be choices available to any evaluator. Ideally, the approach you take should be more a function of the problem than a function of the evaluator but there is no guarantee that this will be the case. What I have found is that our ideology and our research training tend to influence the perspectives that we take or our preference for one evaluation method over another. The methods used should provide you with details about ethics, costs, coverage, objectives, effects and stakeholders. For this reason, it is essential first to phrase every evaluation in terms of a basic question to be answered and for you to be as familiar as possible with the widest range of research and evaluation approaches. Descriptive approaches are a natural place to start your evaluation of career education and guidance. You will find that these 64

How do I describe a program?

approaches have many advantages and the purpose of the following sections is to highlight what I call these detective-like approaches to evaluation.

Descriptive approaches – naturalistic emphasis Descriptive approaches in evaluation have a naturalistic emphasis. They permit you to investigate issues as a participant or observer. They focus on the documentation of program details as they occur, such as describing program processes or participant experiences. Descriptive approaches can also include relevant numerical details such as the number of experiences across time periods, the cost in monetary terms or the levels in achievement on a task. Counting and quantification occurs as it is needed and when it makes sense to count. In many ways it reminds me of scientific descriptions and discoveries in fields such as astronomy, ecology or zoology. Naturalistic methodology is a generic term which has come to include a family of comprehensive techniques: narrative descriptions, explanatory approaches, ethnographic research and field observations. These methods are useful for generating new vantage points and are suitable when any or all of the following conditions are present: • there is a need to quickly understand and portray events • the evaluation occurs in a naturalistic setting • in-depth information is required • there is a need for an integrated view of a situation • stakeholders’ perspectives need to be emphasised • evaluation resources are limited • the evaluator is the main source of both data gathering and analysis • the evaluation is exploratory • the evaluation is participative • the evaluator is not an uninvolved bystander • the in-built subjectivity of the evaluation is accepted from the outset • there is a need for flexibility.

65

Evaluating Career Education and Guidance

In some evaluations, there is no need to make statistical comparisons. Descriptive approaches can suffice and they can have considerable benefits. The aim is to develop an integrated understanding of a program, policy or initiative. Where programs are evolving, then an evaluation that is descriptive may provide valuable details for its further improvement.

Descriptive approaches – participatory emphasis The involvement of participants in evaluations can increase the accuracy of the data and promote ownership of the findings. The vehicle of participation may be a community task force, a steering committee, a working group or individual representations. These offer a responsive feature that seeks to understand an issue from the multiple perspectives of stakeholders. As noted above, the principal methods of descriptive approaches involve fieldwork and have included interviews, observations, questionnaires, program reports or records. Case studies and ethnographic methods feature as components of qualitative approaches. These approaches are particularly suited to making initial inferences about events as well as for testing ideas. There is a greater degree of personal involvement and potential for valuable subjective input into methods and interpretations. Descriptive evaluations are enhanced by direct personal contacts with a program, policy or initiative. The issues to be analysed are guided by the evaluation question(s) and the key issues that emerge. A relevant factor in deciding about merit or worth may well be the personal experience of the evaluator.

When to use samples in descriptive approaches A descriptive approach may involve purposeful sampling in order to assess the impact of a program or service. There are no a priori guidelines as to the size of sample required. This is because you will be using the sample (a) to test

66

How do I describe a program?

the hypothesised merit or worth of a program; or (b) to provide support for the merit or worth of a program. If you are assured by others that a program has merit, then even one negative instance would challenge that idea. If you are uncertain about a program, then each positive case or instance will lend support to a hypothesis that the program has merit or worth but of course it will not be conclusive. In this case you accumulate evidence about the value of a program. This aspect is also discussed in Chapter 9. Samples in a descriptive evaluation can involve extreme cases; cases of great diversity; small homogeneous groups; typical cases; critical cases; chain sampling of individuals through consecutive referrals; sampling cases which meet some criterion; sampling politically significant cases; or convenience sampling. Such purposeful sampling is helpful when there is a need for a deeper explanation of what is happening in education and guidance than that which can be provided by surveys or experiments. The personal and subjective aspects of situations such as the responses of a middle-aged participant; the impact on an adolescent of the move from high school to TAFE; or the behaviour of a participant in a labour market program can be grasped by a descriptive approach. This means looking at the context within which people work, learn and live. It uses their expertise as participants in the situation to provide a source of data. This is a valuable approach to finding out about situations because it takes into account the personal and social context within which education and guidance occur. It looks in depth at the beliefs, the values, the interests, the misconceptions, and the persons involved. As I mentioned earlier, it is particularly helpful when you want to disprove a commonly held belief or positive assertions about programs. In these cases the evaluation goes beyond being descriptive in nature or explanatory in focus and starts to tests hypotheses and ideas. The implications for you are that it means writing about people, gathering original information from people involved; creating a picture of their life; and providing descriptions of what actually happens. The evaluator using such methods describes the expectations, the intentions, the perceptions and the interpretations of participants. The value 67

Evaluating Career Education and Guidance

and power of such case studies in testing your ideas and influencing the opinions of others should not be underestimated. It then becomes possible to develop policies and solutions, which take into account the realistic and accurate descriptions provided.

Information sources for descriptive evaluations Four major sources of information or forms of evaluation are commonly used in descriptive evaluations (see Figure 7.1): observations, questioning (i.e., interviews, questionnaires), assessments (i.e., tests) and records (i.e., documentary and background sources). Within each of these are subsumed a variety of specific research techniques which can include context-based observations, life histories, journals, letters, records or open-ended interviews. Each evaluator needs to select the appropriate forms and techniques of evaluation required for particular tasks. There is no single correct way in which to undertake an evaluation.

68

How do I describe a program?

Observations – fieldwork – ethnography

Assessments – skills tests – simulations – quasi-experiments

Information sources for evaluations

Questions – interviews – questionnaires

Records – evidence – statistical data

Figure 7.1. Multiple sources of information for evaluations (Athanasou, 2000) After defining the topic or problem for evaluation, it is important to consider whether it lends itself to a descriptive approach. The opportunity to observe or participate in a situation also needs to be approved for ethical and legal reasons. This might involve official permissions to observe and record. Consideration also needs to be given to the timing of observations (e.g., the commencement or end of a semester). Assuming that this issue has been resolved, you can then go on to consider how observations can be made and how data might be collected. Some steps might include: • start by making broad descriptive observations • get an overview of the learning situation • analyse the first observations • write up the observations and analysis continuously • formulate working hypotheses. The advantage of observational techniques is their immediacy. There is no need to rely upon past reports or verbal statements of intention, such as in a questionnaire. Observational techniques can also be supplemented by

69

Evaluating Career Education and Guidance

interviews (formal and informal) and survey questionnaires. It may be helpful at the start to define the categories under which observations and data can be organised. The findings of descriptive evaluations can have high internal validity because of the fact that they are focused on observations in natural settings. The validity of observations can be reduced, however, by reactions between the observer and other participants and/or the selection of categories for classifying observations and data. To enhance the validity of descriptive approaches, multiple methods of data collection (i.e., triangulation) on the same topic can be used to (a) replicate findings and (b) overcome reli-ance on any one method for conclusions. In a classroom setting this could involve instructor’s ratings, educational records, participants’ results and structured observations. Such checking of one source of data against another offers a means of building checks and balances into designs and is an expression of the wellknown psychological concepts of discriminant and convergent validity in multitrait multi-method measurement approaches (Campbell & Fiske, 1959). The same factors assessed by different methods should be substantially related (i.e., there should be convergent validity) and the relationship should be greater than that held with other factors (i.e., there should be discriminant validity). Triangulation utilises multiple data sources, multiple observers, multiple viewpoints and multiple methods to confirm findings (Mathison, 1988). It is a solution to concerns about the extent to which you rely upon any one single data source in an evaluation. Factors such as the following can affect the results of evaluations in career education and guidance: • other events occurring at the same time (e.g., redundancies, retrenchments) • prior experiences (e.g., previous information about a career)

70

How do I describe a program?

• coverage and participation rates in the sample (e.g., access to advice may depend upon the stage of schooling) • evaluator bias in judgements and problem solving (e.g., an evaluator with a preference for specific career counselling approaches) • the difficulty of generalising findings from one situation to another (e.g., generalising from rural to city contexts) • a Hawthorne-type effect1(i.e., merely paying attention to a group is rewarding). Your aim is to make a judgement based on the information that can be collected reasonably within the circumstances and resources available to you. Accordingly you would acknowledge any limitations or shortcomings.

Reporting descriptive evaluations The report in a descriptive evaluation can be organised around the themes which guided the fieldwork rather than the traditional model for evaluation reports. It could encompass: • evaluation question(s) • design and methods as well as the justification for these • participants, settings and context • data collection strategies • findings of the study • interpretation and application of the findings. In the next section I want to focus upon one descriptive approach to evaluation, namely illuminative evaluation. This provides a comprehensive descriptive approach, especially for adult contexts and is based largely on Athanasou (1995).

1

The Hawthorne effect draws its name from a factory in which studies were done on illumination and worker productivity; since that time, it has been linked with situations where people supposedly improve their performance merely as a result of being observed.

71

Evaluating Career Education and Guidance

Illuminative evaluation Illuminative evaluation (Parlett & Hamilton, 1977) offers an evaluation model which is derived from a social-anthropological research paradigm. It encourages a focus on the entire program and allows the problem to dictate the choice of methods. I have emphasised this approach for you because it identifies the importance of the context within which education and guidance occur, the processes as well as the outcomes. It provides you with a basic and flexible framework for career education evaluations, especially in adult and community settings (see Figure 7.2). My intention is also to give you a framework from which you should feel free to depart as the need arises. Outcomes

Rationale

Background

Program policy initiative

Limitations

Consequences

Implementation

Figure 7.2. A holistic framework for illuminative evaluations The illuminative emphasis was firstly a reaction against formal research designs in educational evaluation. Here, the concern is with description and interpretation in human settings rather than prediction and measurement. This type of evaluation has an emphasis on increasing the overall knowledge and understanding of a program and represented a substantial shift in the development of formal evaluation approaches. As its name implies it seeks to ‘light up or throw light upon’ key issues. According to Parlett and Dearden (1977):

72

How do I describe a program?

Characteristically in illuminative evaluation there are three stages: investigators observe, inquire further, and then seek to explain … Obviously the three stages overlap and functionally interrelate … Within this three-stage framework, an information profile is assembled using data collected from four areas: observation; interviews; questionnaires and tests; and documentary and background sources. (pp. 17–18)

In the fourth edition of the Evaluation Thesaurus, Scriven (1991, p. 190) described illuminative evaluation as ‘A type of pure process naturalistic evaluation … Congenial to responsive evaluation supporters … Considerable emphasis is placed on “progressive focusing”’. The first stage in an illuminative evaluation is one of familiarisation (see Figure 7.3). There is a concern here to understand the key issues and the milieu being evaluated. The second or observation stage involves selecting a number of topics and ideas for closer study and inquiry. The third inquiry stage consists of information gathering. In the fourth stage the evaluator makes judgements about the program based on the evidence that has been collected. These evaluation findings are then reported to interested parties.

Reporting Explanation (Judgement) Inquiry (Data gathering, Analysis) Observation (Design) Familiarisation

Figure 7.3. Indicative steps in an illuminative evaluation study

73

Evaluating Career Education and Guidance

The illuminative approach is a common-sense but critical mode of evaluation. It involves intensive study of a program and asks questions such as the following (see also Table 4.1): (a) Why was the program, policy or initiative introduced?; (b) What are its intended career education or guidance outcomes?; (c) How do people feel about the program?; (c) Who are the key stakeholders?; (d) What are the social costs and benefits of the program?; (e) What constraints are operating?; and (f) How will the evaluation findings be used? A cycle of inquiry occurs with continuous revision after data collection. New questions may arise as different information comes to hand. The illuminative approach takes advantage of serendipitous findings. Ideas are maintained until they are confirmed or disconfirmed by the available observations. Such a naturalistic methodology permits the evaluator to investigate issues, cases or events descriptively without traditional experimental manipulation, or control. This approach focuses on the documentation of program details and can include program processes, participant experiences and program outcomes. Programs with complex goals, initiatives that are influenced greatly by their community context as well as evaluations that are restricted by resources and opportunities are suitable for the flexible methodology of illuminative evaluation. In this approach there is scope for new evaluation questions to arise in the course of an inquiry. This heuristic aspect of illuminative evaluation means that you are also able to respond directly to the needs of practitioners, participants and/or policy makers. As noted, its methodology has a naturalistic orientation, which can involve a variety of techniques such as observation, ethnography or case studies.

Techniques for illuminative evaluations An illuminative approach documents the idiosyncrasies of the contexts within which people work and learn using a variety of techniques. Observations covering topics such as ethnic identity, students’ interests, counsellor

74

How do I describe a program?

perceptions or the everyday activities of guidance help us to understand the complexity of participation and involvement. They allow experience of the intricacies of programs and thereby avoid the simplification of situations and ideas which occurs in large-scale survey approaches to evaluation. Connell, Ashenden, Kessler and Dowsett (1982) in a major interview study of Australian school students, noted some of the weaknesses of conventional approaches to research and evaluation: Very often researchers had never even laid eyes on the people being researched; at best they saw them briefly while handing out questionnaires. Normally they communicated only via ticks on answer sheets handed in by research assistants or part-time interviewers, then processed through computers. Such research was more like manufacturing margarine than like meeting people and learning about their problems. (p. 29)

In-depth interviews, on the other hand, tap the knowledge of participants in the situation and provide a valuable source of information. This is as important an approach as observation is in the physical and biological sciences. Case studies and ethnographic techniques are suited to making inferences about events. Evaluations are enhanced by valuable personal contacts with the participants in the program. Such contact allows greater scope for explanation and interpretation. The next section outlines how qualitative and observational approaches to evaluation need to be carefully reported in order to increase their usefulness.

Reporting the illuminative evaluation A final issue is the reporting of evaluation findings. A typical report might encompass the background to a program; the key issues in the evaluation; participant details and their involvement; details of how the program was implemented; the direct impact of the program; other outcomes; analysis of advantages and disadvantages of the program; and/or recommendations and conclusions.

75

Evaluating Career Education and Guidance

A useful strategy in an illuminative evaluation is for those acquainted with the program, policy or initiative to check the evaluator’s observations, to comment on the draft report and to critique its conclusions or recommendations. However, the reporting of evaluations may well be compromised when they are client-focused and conducted within the scope of organisational pressures or policy agendas. Potential conflicts of interest and ethical dilemmas in the reporting or conduct of an evaluation cannot be overlooked.

Summary The heterogeneous nature of career development and the diversity of career interventions support in-depth investigation and the use of descriptive methodologies. So too do the humanistic and person-centred ideologies that underpin most career education and guidance practice. This is consistent with the ECCOES framework outlined in Chapter 3. Criteria by which evaluations can be considered were outlined by Rachal and Courtney (1986). They proposed three criteria for judging the worth of findings: (a) practical application; (b) contribution to understanding; and (c) universality. At this point, it seems worthwhile also to return to the standards for evaluations. The Joint Committee on Standards for Educational Evaluation (1981, 1994) identified four primary criteria by which evaluations should be judged: utility, feasibility, propriety and accuracy. It should now be obvious that there are multiple approaches to evaluating education programs. There are no prescribed procedures for career education or guidance evaluations. Techniques can come from a variety of disciplines but they must be capable of providing authentic findings and confirming or disconfirming your ideas. Finally, the two fundamental guidelines in designing an evaluation are that (a) the results must be reproducible, and (b) the findings must make some statement about the content and process of the program and its present and future impact. With regard to (b), follow-up or longitudinal studies are extremely important in determining the lasting effects of career education programs.

76

How do I describe a program?

This chapter has argued that evaluations of career education or guidance programs, policies and initiatives require approaches that are sensitive to the subtleties and complexities of human learning. As a general rule it is useful to allow the circumstances of an evaluation to define the methods. If there is to be a definitive approach to career education and guidance evaluation, it could probably be summed up in the eclectic and multi-dimensional concept of an integrated evaluation. This would incorporate the illuminative approaches and naturalistic methodologies outlined above. The aim of such approaches is to promote valid and reproducible evaluations which will advance the theory and practice of career education and guidance in Australia. -oOo-

REVIEW QUESTIONS 1 The evaluator must often balance the demands for internal validity against external validity. Which of the two types do you consider to be most important in qualitative valuations? 2 Which factors do you think are most likely to contaminate the results from most descriptive evaluation studies?

77

Evaluating Career Education and Guidance

CASE STUDY Career education elective for pre-service teachers Victoria University was contracted by the Department of Education, Science and Training to develop a career education elective for pre-service teachers undertaking a Bachelor of Education or a Bachelor of Teaching degree. The project was guided by an Advisory Committee consisting of a representative from the Australian Council of Deans of Education and Career experts. The goal of this elective is to provide graduates with the skills necessary to educate secondary students in how to: • access and use a variety of available career information • access and take advantage of the wide range of post-school pathways open to them • develop lifelong learning and lifelong career management skills • understand the world of work, including the interactions between society, work and the local/national/global economy. The Career Education Elective Curriculum comprises three curriculum modules, each with a number of topics fully developed. They are: • Personal Management • Learning and Work Exploration • Career Building This elective is a unique undertaking for Australian teacher education. It is an attempt to provide graduating secondary teachers with an opportunity to assist secondary school students to make connections between specific subject discipline and content areas and life, learning and work beyond school.

78

How do I describe a program?

Curriculum structure MODULE 1: Personal Management Topic A: Introduction to Learning about Career Development Topic B: Career over the Life Span Topic C: Developing Self-understanding and Self-confidence Topic D: Managing Change and Transition MODULE 2: Learning and Work Exploration Topic A: Connecting Learning and Work Topic B: Lifelong Learning and its Relationship to Career Development Topic C: Teachers as Facilitators of Learning about Career Development Topic D: Engaging the Disengaged MODULE 3: Career Building Topic A: Career Development Theory Topic B: Career Decision Making and Management Topic C: Career Building and Serendipity Topic D: Lifestyle, Career Goals and Work Learning outcomes Pre-service teachers will be able to: • demonstrate an understanding of the place of career education in the broad context of lifelong learning • explore and discuss the connections between theoretical frameworks and real life, and school-to-work transition experiences • design a careers unit for delivery in a secondary school.

79

Evaluating Career Education and Guidance

Furthermore pre-service teachers will be expected to: • analyse the basic elements of an idea, experience or theory • synthesise and organise ideas, information or experiences • make judgements about the value of information, arguments or methods • apply theories or concepts to practical problems or new situations • restructure their existing knowledge in terms of new ways of thinking about careers. How would you evaluate the career education elective using a descriptive approach? Source: Cherednichenko, B., Malloch, M., Martino, J., Tyler, J., Simpson, J. & Brooks, J. (2005). Career Education Elective Curriculum. School of Education,Victoria University.

80

Chapter eight

How do I determine effectiveness?

If you are asked whether students in a career education setting are satisfied with such learning, you may resort to a survey of participants to determine their levels of satisfaction. Such survey and questionnaire approaches to evaluation are common in the social sciences and offer useful indications about what might work or what does not work. Unfortunately, it is not always possible to draw cause-and-effect conclusions from survey data based on a single group. Here is another example. Suppose you are asked whether a new system of job-finding instruction is useful for persons with disabilities. In order to test this idea an instructor tries it with a group. How would you interpret the results if all the participants obtained work? Would you say this was caused by the new instructional method? Well, it is not straightforward. For a start, there is no way of knowing whether the participants would have produced the same levels of job achievement even by chance or even if the new system of instruction was not used. Chance is an important consideration. It involves all the unknown factors and their potential effects. There might also have been other factors – the group might have been exposed to other beneficial influences such as sympathetic employers, or benefited from a positive labour market at that time, or merely responded to some positive attention in the training or any one of a number of other serendipitous factors.

81

Evaluating Career Education and Guidance

To deal with this problem, you really need a comparison or control group. If there was another similar group that undertook a related course or maybe did not receive the instruction, then it might be possible to make some judgement about the program. In this way you start to control the situation and to vary the conditions to find out if what you are doing has any real effect. Otherwise, it just might be chance factors that are operating. Both these examples might seem tedious or unnecessarily too theoretical but they do raise the issue of how you determine whether a program or service has a positive effect. This complements the descriptive analyses of the earlier sections. There, I considered what should be the focus and scope of your evaluation, the extent of coverage of the program or service and how much it cost and what benefits it offered. The purpose of this stage is to assess the impact of services in a more controlled way. To do this you can use a number of standard research designs and in this chapter I will outline some experimental and quasi-experimental approaches. These rely mainly upon comparisons. So, then you can say things like ‘there was an improvement from time A to time B’ or that ‘service A was more effective than B’. The use of such designs is a standard part of most courses on research methods and it highlights one of the strong historical links between evaluation and research methodologies. Without wishing to pre-empt conclusions about effectiveness, it is unlikely that many of these designs can be readily applied in social and education settings. In some sense they reflect an idealised evaluation. You may find that this section is not easy to understand as it condenses a considerable amount of material into one chapter. I apologise for the compressed presentation and for the fact that some sections may require rereading. It may also be helpful to look at some texts on research method and research design. Any of the standard texts should suffice. Look at different explanations to help you understand the alternatives available to you. Read different accounts as these reinforce the ideas and make you more familiar with this area of research design. In this chapter I have tried to summarise the key ideas and as I mentioned you may find the presentation somewhat overwhelming at first glance – you can skip those sections that are not useful to you and maybe revisit them at a later date. 82

How do I determine effectiveness?

Designs and procedures for data collection The focus on formal evaluation designs and procedures is appropriate only for those situations where the issues can be resolved by empirical means. It should also be emphasised, however, that these designs may not help an evaluator assess the value of a set of objectives (e.g., increasing retention in education or training) and that rational and analytic approaches are required to determine such fundamental issues. I think that it is important to understand what options are available to you for evaluation studies. At the outset, practical circumstances may dictate that these approaches are not applicable but you should at least be familiar with a range of alternatives. The purpose of an evaluation should dictate what data is collected. One problem that occurs in this respect is that evaluators try inappropriately to implement one of a set of standard research designs to a question. There is a trade-off between designing a study that will provide the answer you want and ensuring that the results have external application. One tendency has been to design evaluations that are artificial, experimental or quasi-experimental and they may not be appropriate in cases of great social need.

Internal and external validity of evaluation designs Validity deals with the truth or accuracy of your observations. Validity is not measured but is inferred. The judgement of validity is based on the evidence available. There are many aspects of validity (content validity, construct validity, criterion validity, discriminant validity, convergent validity) but this section deals only with the internal and external aspects of validity. The issue of whether your results can extend to other situations is addressed by external validity. A well-conducted experiment might demonstrate clearly that participants’ learning about a specific occupation is related to their interest but you might doubt that these findings can be generalised to other occupations. Your evaluation method needs to have some external 83

Evaluating Career Education and Guidance

validity if you ever want to say that your findings can be applied to other circumstances. Does this mean that formally designed experiments are best for career education and guidance? While these experimental designs may be useful in scientific fields, they are rarely appropriate in education because of difficulties in obtaining representative samples. It is also the case that you cannot infer from group results to the behaviour of a single person. What you may need are experimental designs that study the behaviour of a person intensively, so that you can generalise to other situations in that person’s life. Internal validity is the certainty that the formal evaluation approach has been designed correctly. It would be pointless to compare two career teaching approaches and then fail to control for other factors affecting learning such as ability or interest. Internal validity is best satisfied with some type of comparison in which a number of factors can be controlled. Once you start to control factors, however, you move away from the reality of typical career education or guidance situations. The trade-off is that typical situations may not offer us enough certainty to answer the problem. Campbell and Stanley (1966) have identified factors that reduce internal validity (see also Popham, 1993, p. 222). • There may be other events occurring around the same time. • There may be natural growth, maturation or development. • There may be some carry-over improvement from any pre-test. • There may be social or demographic differences in groups. • There may be test-retest reliability limitations in the measures used. • The results may only apply to specially selected groups. • There may be an order effect in the sequence of tasks or activities. • There may be a high attrition rate. • There may be what is called a regression towards the mean.

84

How do I determine effectiveness?

Table 8.1. Formal evaluation designs for guidance situations CASE STUDY Guidance ----- Observation

ONE GROUP (EQUIVALENT TO ONE-GROUP PRE-TEST – POST-TEST DESIGN) Observation 1 ----- Guidance ----- Observation 2

GROUP COMPARISON (NON-EQUIVALENT CONTROL GROUP) Guidance ---------- Observation on Group 1 No Guidance ----- Observation on Group 2

EXPERIMENTAL DESIGNS Post-test only control group Random allocation ----- Guidance ----- Observation on Group 1 Random allocation ------------------------ Observation on Group 2

Pre-test post-test control group Random allocation ----- Observation ----- Guidance ----- Observation Group 1 Random allocation ----- Observation ------------------------ Observation Group 2

Factorial designs Random ---- Observation ---- Guidance ---- Special Group 1 ---- Observation Random ---- Observation ---------------------- Special Group 1 ---- Observation Random ---- Observation ---- Guidance ---- Special Group 2 ---- Observation Random ---- Observation ---------------------- Special Group 2 ---- Observation

QUASI-EXPERIMENTAL DESIGNS Interrupted time series Observation 1 – Observation 2 – Observation 3 – Guidance – Observation 4 – Observation 5 – Observation 6 Guidance 1 – Observation 1 – Guidance 2 – Observation 2 – Guidance 1 – Observation 3 – Guidance 2 – Observation 4

Non-equivalent groups Observation on Group 1 ----- Guidance ----- Observation 2 on Group 1 Observation on Group 2 ------------------------ Observation 2 on Group 2

85

Evaluating Career Education and Guidance

Formal evaluation designs For the purposes of describing different evaluations I shall focus on guidance interventions but you could just as easily substitute teaching or counselling for guidance in Table 8.1 on page 85. I would like to emphasise that the designs described in Table 8.1 are only some of the many designs available to you. They are some of the more straightforward and widely used approaches and you should consult a text on research methods or research design before planning a formal evaluation. It is also helpful to obtain some professional statistical advice. Some aspects of the research designs in Table 8.1 are outlined below. Control group: the control group acts as a comparison group. It comprises classes or schools or individuals who do not receive a program or intervention. Ideally the control group should be as similar in age, sex, socioeconomic status or education as the group(s) receiving the program, treatment, teaching or training. This emphasis on controlling is relevant if you want to decide on effectiveness. For instance, if you wanted to test a new pharmaceutical medicine you would also use a placebo, such as a sugar tablet, as a comparison. It would be given to people who would think they were receiving the real medicine. In a sense they would be operating ‘blind’ to the real treatment. To ensure that the people involved in the evaluation also did not influence the results indirectly, they would not be advised whether each pill is the real medicine or the placebo and they would also be ‘blind’ as to the real treatment. This is commonly referred to as a double blind experiment. Randomisation: groups that are made as identical as possible by random assignment are termed ‘equivalent’. Equivalent control groups use randomisation to eliminate alternative explanations for differences between two groups. Of course, in some settings (e.g., classrooms), it will not be feasible to consider randomisation. The issue, however, is that without a control or comparison it is difficult to claim that a program is effective. You need to have some basis for comparison.

86

How do I determine effectiveness?

Non-equivalent control groups: these are comparison groups which have been used where random assignment is not considered feasible. For instance, in investigating computer-based guidance it may not be possible to randomly assign people to two different programs. In this case, you could use two existing scheduled groups and (a) apply the new guidance techniques to one of the two groups and/or (b) compare the new guidance to a group waiting for the service and which has not received any service. Usually these control or comparison groups are similar in some way to the group receiving guidance. In some instances, the non-equivalent control group might receive a comparable treatment; in other cases, they may receive a competing program. As a last resort, the control group may receive no formal intervention. The effectiveness of the different levels of control is summarised in Table 8.2. Table 8.2. The effectiveness of different levels of control Control

➝ Effectiveness* 1 Randomised comparison with a ➝ Tightly controlled form of evaluation competing program and ensuring that the evaluator and participants are unaware of which group is the comparison and control Useful form of evaluation

2 Comparison with a competing program

➝

3 Comparison with a similar program

➝ Moderate basis for comparison

4 No treatment group comparison

➝ Least effective comparison

5 Case study – single observation

➝ Only able to disprove a theory

*

Effectiveness is calculated by the effect size (d). See the appendix to Chapter 8.

Post-tests: these are measurements or assessments that are made at the completion of a program. The post-test focuses on the dependent variable (i.e., the outcome) in a research design. Questions arise as to the timing of post-test assessments of outcomes. In some instances, assessments will be made at the conclusion of a program some short time later or some designs may allow for long-term follow-up evaluations.

87

Evaluating Career Education and Guidance

Pre-tests: pre-tests focus on assessments that are made prior to training or instruction or program delivery. Pre-tests can be used to assess people on the intended outcome, such as present values, initial knowledge or existing skills. The group can then be assessed on these same factors at the conclusion of a program to test its effectiveness. Another use of a pre-test is to determine the comparability of individuals in the two groups on factors such as pre-existing knowledge, attitudes or skills. Comparisons of pre-test with post-test results are not always straightforward. Firstly, there is the issue of the unreliability of differences. That is, the differences or gains made in a program have some unreliability attached to them. Secondly, there is the problem of what is called regression toward the mean. High and low scorers on a pre-test tend to have scores on a post-test which move closer to the average. In one sense, this is because the high and low scorer’s results have nowhere to go on a post-test except to stay where they are (i.e., high or low) or go down for a high scorer and go up for a low scorer. To summarise, a pre-test can be used to select people for a program; check the comparability of groups; and check the gains made during a program. Some people might argue that trying to reduce evaluations to systematic analysis under fixed situations eliminates many of the personal aspects of human services. They are correct. Moreover, there are substantial technical limitations in making inferences from experimental designs in career education and guidance.

Summary The determination of effectiveness is a key aspect of the ECCOES framework. In this chapter the focus has been on experimental comparisons. The strengths and weaknesses of such comparisons have been highlighted for you. The least effective basis for experimental comparison was described as that which does not have a comparison group. As a general observation, it is also unwise to rely just on survey or questionnaire evaluations which are common in career education and 88

How do I determine effectiveness?

guidance. Many of these surveys are post hoc (after the fact) and offer no benchmark for comparison. More often than not, they are based on nonrepresentative groups. Typically, they are small surveys with usually less than 300 non-randomly selected respondents from an adult or school population. Consequently, they have large sampling errors (see Streiner, 1994, for an indication of the large sample sizes required for various confidence intervals). Often there are no grounds for generalisation; as I mentioned there is little if any basis for comparison; and as a result any conclusions are of limited validity. In the preceding sections I have also emphasised limitations in the relevance of randomised designs for educational evaluation. Examples from the practice of evaluation highlight the difficulties of applying experimental design to evaluation. For instance, there is a practical conflict between the convention of randomisation and the selective nature of established career education practices. Rather than focusing on double-blind randomised control studies or quasiexperimental designs or surveys, a more useful approach for evaluations is to use time series designs in which individuals act as their own control. These designs have in-built internal and external validity. They could just as easily have been included under the heading of descriptive approaches. Time series designs incorporate both pre- and post-test measures. This design for an evaluation is very powerful because it plots changes before and after interventions. It is based on a series of measures and provides a stable basis for evaluating some effects. The limitations of research designs also affect any inferences that you might wish to make about your program and its effectiveness. Firstly, there is the question of the extent to which empirical methods and experimental design can be meaningfully applied to many career issues. Furthermore, the size of an effect between groups in an experimental design is important since it is not enough to show that there are statistically significant differences. For instance, large samples are capable of showing statistically significant but quite small differences. Moreover, the results from experimental designs that are based on group data hardly ever apply to an individual case. Averages hide 89

Evaluating Career Education and Guidance

as much as they reveal. A career guidance program may be demonstrated as effective for decision making of most Year 10 students but there is no guarantee that it will be effective for the next Year 10 student on whom it will be used. Finally, in many cases, all that people note is the gross effects of a program or service. Some people then go on to assume that this means that a program or service is working if any effects are positive. They may fail to distinguish between gross and net effects or disregard some of the intricacies of formal evaluation designs. -oOo-

90

How do I determine effectiveness?

APPENDIX: EFFECT SIZES Effect sizes (d) have been defined as small, d = .2, medium, d = .5, and large, d = .8 d = M1 – M2 / SDpooled where M1 is the mean of group 1 M2 is the mean of group 2 SD1 is the standard deviation of group 1 SD2 is the standard deviation of group 2 spooled = √[(SD1² + SD2²) / 2] and spooled is the pooled standard deviation (this is found as the square root of the two standard deviations squared and then divided by 2)

REVIEW QUESTIONS 1 Traditional experimental and quasi-experimental designs, naturalistic methods as well as case studies are among the most common methods of inquiry used in evaluation. Which approaches do you favour in your work context? 2 Using parallel columns, list the advantages and disadvantages in program evaluations of:

(a) experimental designs



(b) quasi-experimental designs



(c) randomisation



(d) control groups.

3 Prepare a hypothetical example of a pre-test, post-test evaluation that would appear favourable because of threats to internal validity.

91

Chapter nine

How do I make judgements in educational evaluations?

In career education and guidance, the processes that are used to evaluate a program or service are designed to lead to information on which a decision is based. The results, however, may be equivocal or only provide an indirect answer or more likely partial support for a program, policy or initiative. Even the most thoroughly designed evaluation methodology with results of undoubted reliability and near perfect validity with respect to the evaluation questions still needs to be interpreted. It is not often the case that the results will speak directly for themselves. The purpose of this final chapter is to outline a number of ways in which decisions are affected by errors and to describe some principles that might be used to govern the judgements from an evaluation process. The reason for considering judgement in the context of evaluation is because there is considerable evidence that human judgement is quasi-rational; combines intuition and analysis; is prone to bias; and may introduce distortion into educational decisions. In the study of evaluation I have focused so far on the nature, purpose and methods of evaluations. In this final chapter I want to go beyond methods and see how evaluation decisions are made when you need to combine information on ethics, coverage, costs, objectives, effects and stakeholders for an evaluation. Judgement is important because the ECCOES

93

Evaluating Career Education and Guidance

framework is holistic and synthetic in its approach. It combines information from these six aspects to arrive at a judgement of merit or worth.

Background influences that limit judgements There are a number of factors that will limit your judgements of the merit or worth of a program or service. Some sources of error in judgements are summarised in Figure 9.1. Limited information

Lack of respect for evidence

Judgements Value systems

Inappropriate heuristics

Decisions

Limited information processing

Error

Figure 9.1. Potential sources of error in judgements Limited information: Firstly, most decisions from evaluations are based on limited information about situations. Even when the data available appear to be incontrovertible, such as when extensive databases on costs and outputs are available, there are still aspects of the problem that will remain undefined, or inaccessible. So, the first issue affecting any decision is the incomplete basis of information on which you might rely. Value systems: A second influence is in the value system that you bring to an issue. This will influence your perception of the evidence and the weight that you give to different factors. One tendency of people when they come to make any evaluation of evidence is to seek information that will confirm their pre-existing beliefs. This is complemented by the rating of evidence (for and against) in a biased manner. Moreover, you may be unaware that the same evidence can be considered from different viewpoints and often lead to markedly different conclusions.

94

How do I make judgements in educational evaluations?

Error: One influence that has been discussed in previous sections has been that any information or data upon which you base your decisions is itself subject to error. The most important errors for an evaluation are a degree of unreliability and imperfect predictive validity in the information available to you. The first type of error is random error. This is the chance variation in the results that will always occur when observations are made. It is the probability that a finding or result is too large as the chance that it is too small. It is something over which you as evaluator have no control but it can be estimated and reduced through sampling procedures. The second type of error was dealt with in Chapter 7 and Chapter 8. This is systematic error and it has to do with the design of your evaluation and your methodology. Systematic errors cannot be reduced statistically but can be eliminated by altering the design of an evaluation. In actual fact though, you may never be aware that the error exists. Chance and systematic errors can lead you to make to two types of fundamental judgemental errors, either false positive or false negative decisions. These are called Type I or Type II errors. A false positive decision would be inferring that something is when it reality it is not. A Type II or false negative decision would be inferring that something is not when in reality it is. Lack of respect for evidence: Well, it is not always the case that people respect evidence and consequently they may or may not rely on the information provided by an evaluation. This is also highlighted in the subjectivity associated with the acceptance of evidence. On the one hand, an evaluation may cease at an early point because of the fact that the evidence appears consistently favourable, but on the other hand, an evaluation might continue until some support for a position is obtained. In the absence of any decision rules, the ideas and evidence from an evaluation might be scrutinised unfairly. Using inappropriate judgemental heuristics: A good starting point for thinking about problem solving in an evaluation is to consider the idea of a heuristic. A heuristic is like a general rule that is used in a situation to solve a problem or

95

Evaluating Career Education and Guidance

make a decision. Research psychologists Amos Tversky and Daniel Kahneman (Tversky & Kahneman, 1974) found that people rely on a number of common heuristics in their ordinary thinking. • A typical heuristic is the heuristic of representativeness. Decisions are often made about something on the basis that it is typical or like other situations. • Another heuristic is that of availability. This occurs when an example from memory becomes the basis for decision making. • A third heuristic is that of anchoring. With this heuristic, judgements are based on the information framework or context that is provided. These three heuristics are useful in everyday life; however, one of the aspects of decision making that has been investigated is what happens when these heuristics are not applicable to situations and what are the common types of errors that people make in their thinking and judgement. The inappropriate use of such heuristics relates especially to the users of evaluation results such as policy makers and decision makers, who may misinterpret complex and probabilistic findings. Limited information processing: The accuracy of reasoning is limited when there are uncertainties in a situation, and probabilistic decisions are required. Our judgements also reflect our use of specific cues or criteria and it has been shown that there is a great deal of subjectivity. This is especially the case when judgements are made under conditions of uncertainty and where there are probabilities involved. Some sources of judgement bias are listed below: • Some specific details are easier to recall than others. • People selectively perceive details or information that is consistent with their ideas or they may downplay conflicting evidence. • Single experiences are more likely to dominate our decision making than, say, base rates of performance. • There are primacy and recency effects in judgement. • There are inconsistencies in applying judgement strategies. • Sometimes we rely on small samples of performance and give them too much weight in our decision making.

96

How do I make judgements in educational evaluations?

The available evidence indicates that the ability to judge is limited by our information processing capacity. People seek to reduce the complexity of the mental effort involved. There is a tendency to rely on rules of thumb and judgement heuristics in decision making.

Approaches to making judgements in evaluations How then can evaluators reduce the systematic errors in the judgement process? One way is to define clearly the criteria for judgement and to state that certain combinations of evidence will be accepted whereas others will be incomplete justification. There are three main evidence strategies (derived from Hager, Athanasou & Gonczi, 1994, p. 71; see also Athanasou & Lamprianou, pp. 277–279). These are depicted in Figure 9.2 and summarised below. Testing or confirming an idea

Aggregating information to support an idea

Evaluation judgements

Balancing the evidence

Figure 9.2. Approaches to making judgements in evaluations Hypothesis testing or confirming: that is, assuming that a career education or guidance program has merit or is effective until there is standards-based evidence to the contrary. On this basis any agreed evidence of failure would be sufficient to negate merit. This provides a very strict but universal yardstick for judgements and

97

Evaluating Career Education and Guidance

decision making. In some ways this is typical of scientific research and testing a hypothesis or an idea. Aggregation of information and collecting evidence: or assuming that a career education or guidance program does not have merit or is not effective until there is sufficient evidence in accordance with pre-determined criteria. This is a more flexible approach to judgement. It assumes that a program or service does not have sufficient merit or effectiveness at the outset until our evaluation commences to produce evidence to the contrary. On this basis the evidence would be accumulated until such time as an agreed stable judgement of merit is reached. Further evidence would then continue to confirm that it meets the evaluator’s criteria. This appears to be the most practical solution to making judgements in career education contexts. The only technical caution that will be mentioned here is that accumulating evidence is not like the operation of addition in arithmetic. In this case there are no standardised units of evidence that can simply be added. What we are doing here is aggregating information or classes of information in a qualitative manner. Judicious approaches and balancing the evidence for and against the idea that something has merit or is effective, where the pros and cons are weighed and a decision made on the balance of the evidence available and the probability that something is or is not true. Situations will arise where the evidence is (a) entirely consistent with merit; (b) entirely consistent with no merit; or, (c) partly supporting a judgement of merit. There do not appear to be obvious judgement errors associated with the first two options (a) or (b). Option (c) however is a common scenario. In this case the evaluator will need to weigh up both the quantity and the quality of the evidence. This judgement will have regard for the consistency of the evidence. The essence of this approach might be to ask: would a reasonable person be prepared to overlook failure on a number of elements and still consider the program or service worthy of merit or as being effective? Such a qualitative judgement might offer one of the most practical solutions for the evaluation decisions that need to be made in career education and guidance.

98

How do I make judgements in educational evaluations?

Summary Despite attempts to objectify evaluation processes, it is clear that they still involve some complex judgements. Evaluation is not simply a matter of collecting one piece of evidence and then deciding that a program has merit or worth. It involves a range of information. In this book I have introduced you to the ECCOES framework as a way of consolidating your thinking for evaluation in career education and career guidance. These six detailed evaluative criteria of ethics, coverage, costs, objectives, effects and stakeholders serve as the foundation for your evaluation judgement. Ethics

Stakeholders

Coverage

Effects

Costs

Objectives

Figure 9.3. A hexagonal evaluation framework for education and training Your journey in the evaluation of career education and guidance has brought you a long way from our introduction. I have tried to provide you with a framework rather than a cookbook of methods or sure-fire recipes for career program evaluations. My aim has been to say that you need to seek out and combine the six types of evidence and then you have to make a judgement. You are not locked into a particular method or approach but you have substantial freedom to use a variety of approaches to consider the merit or worth of your career education or guidance services.

99

Evaluating Career Education and Guidance

The earlier sections provided you with a large-scale road map to describe how you might traverse these six pieces of evidence. Underlying this approach is the view that evaluation is a pragmatic activity that focuses on a judgement and decision. Accordingly, evaluation of career education and guidance is holistic and at the same time a synthetic process that combines six pieces of information about career programs, policies, initiatives or services. I have great regard for the formal definition of evaluation as the systematic investigation of merit or worth. I hope that you also found this approach and orientation of some interest and value. -oOo-

REVIEW QUESTIONS 1 The Joint Committee on Standards for Educational Evaluation (1981) identified four primary criteria by which evaluations should be judged: utility, feasibility, propriety and accuracy. Evaluators differ on the emphasis they place on these criteria.

(a) How much preference should be given to utility in your area of work?



(b) Do you believe that technical elegance or absolute accuracy is important for judgement of value?

2 Contrast the problems that Type I and Type II errors create for program evaluation.

100

How do I make judgements in educational evaluations?

Further reading Adelman, C. (1984). The politics and ethics of evaluation. London: Croom Helm. Baumgartner, L., Stenersen, S., & Campbell, K. (1996). Overcoming barriers to evaluating health education and training programs. Evaluation and the Health Professions, 19, 292–310. Bernthal, P. R. Evaluation that goes the distance. Training & Development. 49, 41–45. Brinkerhoff, R. O., Brethower, D. M., Hluchyj, T., & Nowakowski, J. R. (1983). Program evaluation – sourcebook. Boston: Kluwer-Nijhoff Publishing. Deshler, D. (1984). Evaluation for program improvement. San Francisco: Jossey-Bass. Guba, E. G., & Lincoln, Y. (1983). Effective evaluation. San Francisco: Jossey-Bass. Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluator’s handbook (2nd ed.). Newbury Park: Sage. Kemmis, S., & Stake, R. (1988). Evaluating curriculum. Geelong: Deakin University. Leviton, L., & Hughes, E. (1981). Research on the utilization of evaluations: a review and synthesis. Evaluation Review, 5, 525–548. Lincoln, Y., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Owen, J. M. (1993). Program evaluation. Sydney: Allen & Unwin. Patton, M. Q. (1990). Qualitative evaluation and research methods. (2nd ed.). Newbury Park: Sage. Phillips, J. J. (1996). Was it the training? Training & Development. 50, 28–32. Smith, E. R., & Tyler, R. W. (1942). Adventure in American Education (Vol. 3). Appraising and recording student progress. New York: Harper & Row. Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In G. Madaus, M. Scriven, & D. L. Stufflebeam (Eds.), Evaluation models: Viewpoints on educational and human services evaluation. Boston: Kluwer-Nijhoff Publishing. Stufflebeam, D. L., & Shinkfield, A. J. (1985). Systematic evaluation. Boston: KluwerNijhoff Publishing. Trow, M. A. (1970). Methodological problems in the evaluation of innovation. In Wittrock, M. C. & Wiley, D. E. (Eds.) The evaluation of instruction, New York: Holt. Walberg, H. J., & Haertel, G. D. (1990). The international encyclopaedia of educational evaluation. Oxford: Pergamon Press. Wholey, J. S. (1983). Evaluation and effective public management. Boston: Little Brown & Co.

101

Evaluating Career Education and Guidance

References Athanasou, J. A. (1995). Issues in Adult Educational Evaluation. In G. Foley (Ed.), Understanding Adult Education and Training. Sydney: Allen & Unwin. Athanasou, J. A. (1998a). Testing competency in a high tech learning environment, Training and Development in Australia, 25, 18–22. Athanasou, J. A. (1998b). A framework for evaluating the effectiveness of technologyassisted learning. Industrial and Commercial Training, 30(2–3), 96–103. Athanasou, J. A. (2000). Evaluating adult education and training. In G. Foley (Ed.), Understanding Adult Education and Training. (2nd ed.), pp. 81–96. Sydney: Allen & Unwin, 2000. Athanasou, J. A., & Lamprianou, I. (2002). A teacher’s guide to assessment. Tuggerah: Social Science Press. Australian Bureau of Statistics. (1994). Training Practices – Australia 1994 (No. 6356.0). Australian Bureau of Statistics. (1996). Employer Training Expenditure, Australia July– September 1996 (No. 653.0). Boyd, G., Hemmings, B., & Braggett, E. (2000). The development of a career education program for gifted high school students. Paper presented at the Australian Association for Research in Education. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally. Cherednichenko, B., Malloch, M., Martino, J., Tyler, J., Simpson, J. & Brooks, J. (2005). Career Education Elective Curriculum. School of Education, Victoria University. Committee on Employment Opportunities. (1993). Restoring Full Employment. Canberra: Australian Government Publishing Service. Connell, R. W., Ashenden, D. J., Kessler, S., & Dowsett, G. W. (1982) Making the difference. Sydney: George, Allen & Unwin. Cronbach, L. J. (1963). Course improvement through evaluation. Teachers College Record, 64, 672–683.

102

References

Department of Education, Science and Training. (1999). Careers education quality framework. Canberra: Author. Dewey, J. (1916). Democracy and education. New York: Macmillan. Dixon, N. M. (1996). New routes to evaluation. Training & Development, 50, 82–85. Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to design a program evaluation. Newbury Park: Sage. Hager, P., Athanasou, J. A., & Gonczi, P. (1994). Assessment technical manual. Canberra: Australian Government Publishing Service. Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluator’s Handbook (2nd ed.). Newbury Park: Sage. Joint Committee on Standards for Educational Evaluation (1981). Standards for evaluation of educational programs, projects and materials. New York: McGraw-Hill. Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards. Newbury Park, CA: Sage. Kirkpatrick, D. L. (1975). Evaluating training programs. Washington: American Society for Training and Development Inc. Kirkpatrick, D. L. (1996). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berret-Koehler Publishers. Levin, H. M. (1983). Cost-effectiveness: A primer. Beverly Hills, CA: Sage. Mathison, S. (1988). Why triangulate? Educational Researcher, 17, 13–17. Morris, L. (1994). To evaluate or not to evaluate. Training & Development, 48, 62. Norris, N. (1990). Understanding educational evaluation. London: Kogan Page. O’Brien, K., Dukstein, R. D., Jackson, S. L., Tomlinson, M. J., & Kamatuka, N. A. (1999). Broadening career horizons for students in at-risk environments. Career Development Quarterly, 47, 215–228. Owen, J. M., & Rogers, P. J. (1999). Program evaluation: Forms and approaches (2nd ed.). Sydney: Allen & Unwin. Parlett, M., & Dearden, G. (1977). Introduction to illuminative evaluation: Studies in higher education. Cardiff-by-the-Sea, CA: Pacific Soundings Press. Parlett, M., & Hamilton, D. (1977). Evaluation in illumination: A new approach to the study of innovative programs. In D. Hamilton et al. (Eds.), Beyond the numbers game. London: Macmillan Education.

103

References

Popham, W. J. (1993). Educational evaluation. (3rd ed.). Boston: Allyn & Bacon. Rachal, J., & Courtney, S. (1986). Response: Focus on adult education research questions. Adult Education Quarterly, 36, 157–65. Rizzetti, J., & Smith, S. (2005), WIL-ling and able: Online support for students moving between work integrated learning experience and the university. In J. Searle, F. Beven, & D. Roebuck (Eds.), Vocational learning: Transitions, partnerships and sustainable futures (Vol. 2), pp. 124–130, Centre for Learning Research, Griffith University. Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation. A systematic approach (6th ed.). Newbury Park: Sage. Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagne, & M. Scriven (Eds.). Perspectives of curriculum and evaluation. Chicago, IL: Rand McNally. Scriven, M. (1991). Evaluation Thesaurus (4th ed.). Newbury Park, CA: Sage. Stake, R. (1967a). The countenance of educational evaluation. Teachers College Record, 68, 523–540. Stake, R. (1967b). Towards a technology for the evaluation of educational programs. In Tyler, R. et al. Perspectives on curriculum evaluation, AERA Monograph Series on Curriculum Evaluation No. 1 Chicago: Rand McNally. Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation. Newbury Park, CA: Sage. Streiner, D. L. (1984). Sample size formulae for parameter estimation. Perceptual and motor skills, 78, 275–284. Stufflebeam, D. L. (1971). Educational evaluation and decision making. Itasca, IL: FE Peacock Publishers. Tamkin P., Yarnall J., & Kerrin M. (2002). Kirkpatrick and Beyond: A review of models of training evaluation, IES Report 392, Brighton, UK: Institute for Employment Studies. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131. Tyler, R. W. (1950). Basic principles of curriculum and instruction. Chicago, IL: University of Chicago Press.

104

References About the author

About the author James A Athanasou was born in Perth in 1948 and went to the Maroubra area of Sydney in 1953, where he still lives. Jim obtained his PhD from the Centre for Behavioural Studies in Education at the University of New England. He spent most of his career in the New South Wales Public Service and came to the University of Technology, Sydney in 1991 as a lecturer in Measurement and Evaluation. He is now Associate Professor in the Faculty of Education. Jim is the author of A Teacher’s Guide to Assessment (Social Science Press), Adult Educational Psychology (Social Science Press) and other texts including Selective Schools and Scholarships Tests (Pascal Press), Basic Skills Tests (Pascal Press), Opportunity Class Tests (Pascal Press). He is the editor of the Australian Journal of Career Development published by the Australian Council for Educational Research and PHRONEMA, the Annual Review of St Andrew’s Greek Orthodox Theological College. He is on the editorial board of Perspectives in Education and the Australian Educational and Developmental Psychologist. He has been a visiting fellow at the Universität der Bundeswehr, München, the University of Illinois at Urbana-Champaign, the University of Hong Kong and the Vrije Universiteit, Brussels. Jim is a registered psychologist, a member of the Australian Psychological Society and member of the Australian Association of Career Counsellors. He developed the Career Interest Test which is used by the Federal government through www.myfuture. edu.au and his research centres on vocational interests, Rasch measurement and the Brunswick lens model. He is married with four children and his interests include religion, reading, walking, gardening and crosswords.

105

References

Index anchoring  96 Athanasou  4, 12, 21, 26, 48, 69, 71, 97 attrition rates  50 Australasian Evaluation Society  30

Kahneman  96 Kirkpatrick  18, 25, 27

client  12, 27, 30, 35–44, 48, 49, 51, 55, 76 control group  14, 82, 85, 86, 87 cost feasibility  29, 55, 56 cost-benefit  3, 17, 29, 55, 56 coverage  23, 24, 28, 41, 47–51, 71, 82 Cronbach  15, 16

meta-evaluation  12 Morris  21, 39

educational evaluation  3–5, 12–13, 16, 25, 72, 89 effect size  29, 87, 91 effectiveness  4, 15, 27, 56, 81–88 effects  16, 23, 25, 55, 64, 76, 81, 89 efficiency  53–55 ethics  23, 24, 29, 43, 64, 93 evaluation descriptive methods  63 holistic  17, 22, 23, 28, 29, 94, 100 object of evaluation  12, 38 program evaluation  4, 12, 14, 22, 27, 53, 63, 99 qualitative  5, 30, 38, 63, 64, 66, 75, 98 synthetic  5, 22, 94, 100 Fitz-Gibbon  21, 39, 42 Hawthorne effect  71 heuristic  94–97 availability  96 representativeness  96 illuminative evaluation  18, 71–77 incidence  24, 29, 40

Levin  17

Norris  2, 14 objectives  13–15, 17, 23, 25, 26, 38, 40, 42, 43, 83 Parlett  18, 72 Popham  2, 12, 16, 84 post-tests  87 present value  57 pre-tests  84, 85, 88 prevalence  24, 29, 40 randomisation  86, 89 regression toward the mean  84, 88 representativeness  96 Scriven  14, 16, 18, 73 sensitivity  24, 29, 40 specificity  14, 24, 29, 40 Stake  15–17 stakeholder  4, 12, 27–29, 37–38, 42–43, 65, 74 Stufflebeam  17, 30 summative  7, 14, 22, 51 triangulation  70 Tversky  96 Tyler  5, 13, 17 utility  57–59

Joint Committee  2, 7, 12, 53, 76 judgement  2, 14, 15, 17, 18, 22, 64, 71, 73, 93–99 aggregation of information and collecting evidence  98 judicial approaches and balancing the evidence  97, 98

validity discriminant and convergent validity  70, 83 external validity  77, 83, 89 internal validity  70, 84

106

Evaluating Career Education and Guidance offers a complete basis for evaluation judgements and decisions. It is an invaluable guide for anyone who needs to decide about the merit or worth of career programs or services. The book is aimed at a wide audience of practitioners, researchers and policy makers. It provides a useful reference and guide for anyone who is required to evaluate a career initiative.

Australian Council for Educational Research

Evaluating Career Education and Guidance

The ECCOES framework— Ethics, Coverage, Costs, Objectives, Effects, and Stakeholders—emphasises the perspectives of key stakeholders in any evaluation. It deals with fundamental issues such as the ethics of a program, the cost-benefits of a service and the extent of coverage of any new initiative. Readers are shown the importance of determining whether a program achieves its objectives and its relative effectiveness. Both qualitative and quantitative approaches are discussed.

James Athanasou ­­

Career education or guidance programs vary in their nature and purpose; each operates in a different context, time and space. Through the use of a synthetic and holistic approach, Evaluating Career Education and Guidance describes a framework developed by the author for judging the merit or worth of career programs, policies, services or initiatives.

James Athanasou ­­

Evaluating Career Education and G uidance