These pages look best viewed in Netscape!
© Copyright 1997 by the Wyoming Department of Employment, Research & Planning

The 1990’s are an era of both fundamental economic changes and a time of accountability. "Ways in which resources are used--especially public resources--are being questioned. Are the resources being invested wisely?" (Evaluation Practice, Volume 17, Spring 1996). Researchers are using labor market information (LMI) to try to measure both the process and the outcome of some programs. The U.S. Department of Labor is proposing some visionary goals and commitments to lifelong learning, welfare to work, and building partnerships between public service and the private sector. Evaluation research (EVR) can help to address some questions about the return on investment in employment and educational programs. For example, some research sponsored by the Wyoming Occupational Coordinating Council (WOCC) is directly relevant to the School-to-Work (STW) program and the Department of Employment is actively participating in Workforce Development (WD).

The Wyoming Workforce Development Council was established by Executive Order 95-15. One of the Council's duties is to "assess the adequacy of existing workforce development activities and services ... and make recommendations to the Governor and Legislature ... regarding the need and effectiveness of such services, and changes which could improve services provided; and to develop a performance measurement system which will provide for a consistent and fair evaluation." EVR is expanding into new frontiers, measuring organizational change, job and career advancement, and some close ties between education and future careers.

Some Definitions

Evaluation research is defined in this article as the measurement of outcomes, processes, and utility, using a reliable and scientifically valid methodology. Evaluation refers to the statistical measurement of programs, not to a personal assessment of individuals or practices. The definitions of some terms are:

1. outcomes: measurable results
2. processes: on-going developments and improvements
3. utility: rewards minus costs

To understand the foundation and assumptions of EVR, one needs to have a well defined conceptual framework. An article in Workforce (1996)^*, by Harrington and Sum, establishes the conceptual framework for a discussion of EVR and LMI. The article, "Key Human Resource, Supply, Demand Issues Face Nation’s LMI System", stresses the necessity for a conceptual framework which then shapes effective employment, training and workforce development strategies. Several key issues are summarized in Table 1.

Inherent in the discussion by Harrington and Sum is recognition of the critical role of increasing international competition in the global marketplace. For example, trade agreements such as the General Agreement on Trade and Tariffs (GATT) and the North American Free Trade Agreement (NAFTA) will reduce barriers to trade and may also influence the competitiveness of American businesses and workers. On the state level, the direct impact of this paradigm shift is seen in the allocation of funds to training programs for dislocated workers. The authors capture the impact of rapid change: "Most states are likely to develop mission statements that incorporate multiple objectives for workforce development systems, including programs to reduce the incidence of poverty problems among working-age residents--especially the working poor." (Workforce, 1996, p. 17).

A review of the literature in EVR is necessary to understand the future direction of this relatively new and emerging field. The current debate over research focusing on outcomes versus research emphasizing processes has financial implications for current programs. This debate is presented in the literature entitled, "New Directions for Evaluation", a publication of the American Evaluation Association, and in Evaluation Practice (Spring 1996 edition).

To summarize this lively yet serious discussion, one can focus on the current research of Michael Scriven entitled, "Types of Evaluation and Types of Evaluator". Dr. Scriven is a strong and articulate advocate of "bottom line research" and would probably feel right at home in a management-by-objectives environment and measuring results of programs. His EVR focuses on outcomes. For example, he questions, "was the project a wise investment in terms of financial costs versus the number of people assisted?" In contrast to this outcome-based EVR, other researchers focus more on the processes of social and governmental programs: a holistic approach. Their EVR research is used primarily in the developmental sense to help policymakers and administrators to improve on-going programs.

This author favors a balanced position of outcome- and process-oriented research, similar to the position advocated in several current publications. Some uses of EVR are summarized in Table 2.

EVR can be used to measure beginning and intermediate goals (processes) and final results (outcomes). Instead of an "either-or" dichotomy, this moderate approach could integrate the best of both worlds. This balanced approach acknowledges the insistence of Dr. Michael Scriven that at some point, one must evaluate the success or failure of a program to accomplish its mission and provide a reasonable return on taxpayer or stockholder investment of funds. This integrated approach seems essential to evaluation in applied settings where evaluators have the opportunity to serve both purposes of program improvement and public accountability. Some contributions of balanced EVR are summarized in Table 3.

Current applications of EVR are of increasing importance in educational and employment settings. EVR is a systematic approach to addressing fundamental questions about social interventions, training programs and transitions from school to work. For example, in welfare reform, participants now have maximum time limits for receiving public benefits. Questions might include: Which programs of intervention are most successful in placing these applicants in full-time jobs? In employment settings, which training programs offer displaced workers the best opportunity to obtain jobs which pay above the minimum wage? In educational settings, how can we measure a program’s success in helping students to make the transition from school to work?

School-to-Work Study

A research study on the school-to-work program in Wyoming provides a highly relevant example of balanced EVR. First, this research project exemplifies what can be accomplished with the mutual cooperative efforts of the Departments of Employment, Public Instruction and the Northwest Regional Educational Laboratory. The final report is entitled, State of the School-to-Work Program in Wyoming and the Role of Career Information. The research was sponsored by WOCC. The purpose of the study was to survey all high schools and post-secondary institutions in Wyoming to provide state leaders and planners with an understanding of how STW programs are operating and about current and future needs. The project was an ambitious research undertaking. It established a foundation for future research studies which can compare programs, costs, improvements and outcomes.

This article will summarize some of the major results and recommendations of the study:

1. Approximately 69 percent of enrolled high school students are engaged in STW activities.
2. There is a strong need to integrate academic and vocational learning and to align school curricula with the demands of the business community.
3. More than 50 percent of the responding secondary schools and postsecondary institutions have career centers.
4. Given Wyoming’s rural nature, it makes sense to take full advantage of available technology. The Internet may help students link to sources of career information.
5. Since the goal of STW is to make a difference for students in their career and educational choices, the state needs to set benchmarks or expectations for STW programs and provide assessment tools for local use.

Job Changers Study

Another research project used wage records in an employment setting to study Job Changers in Wyoming’s Economy (refer to the October 1995 and October 1996 issues of TRENDS). These wage records contain the wages of employees who are covered by Unemployment Insurance and are submitted quarterly by employers. This report studied people in Wyoming who changed jobs (job changers) at least once between 1992 and 1995. This study focused on the process of job change and there are important consequences for employment training programs. The results indicate that "most people stay in their industry when changing jobs. Those who do change industries typically migrate to the Services industry (first choice) or Retail Trade (second choice) ... These job paths distinctly show patterns which, if used correctly by training and educational program administrators, could help focus training dollars on logical clusters that already exist."

A Longitudinal Study

A different longitudinal approach to EVR was presented at the 1997 Careers Conference, Education and Work Center, University of Wisconsin, about the process of career advancement. The study is entitled "A Longitudinal Study of the Career Development and Aspirations of Women Managers in Business Firms". This study by Mary Wentling, Ph.D., illustrates the advantages of longitudinal research encompassing a five-year time span. The participants in the study were thirty women managers in middle management positions in a variety of organizations. The objectives of the study were to explore these questions:

1. What factors helped these women managers to progress toward higher level positions?
2. What barriers did they encounter in progressing toward higher level positions?

The major conclusion of the longitudinal study is that several factors helped these managers: good people skills, producing high quality work, perseverance and the active support of their organizations. The major barriers encountered by these managers included: lack of opportunity, downsizing, reorganizations and lack of support from their bosses. While the sample size is limited due to the five-year time period, these conclusions merit future research. The information obtained in this study can be used by organizations to develop methods of organizational change and to identify better ways to prepare women for management positions. The impact of this research is useful in an increasingly important specialty area in EVR: implementing organizational change.

Conclusions

In an era of rapid changes such as technological advances, international competition and reform of internal social policies, the issues of evaluation research are decisions which impact the lives of individuals, employees, corporations and nations. Modern technology such as sophisticated computer databases can be combined with the content from applied fields of industrial psychology, economics and business management. The technical challenges of integrating and analyzing information are being addressed with the tools of the statistician. An important challenge is to carefully define the validity of what we want to measure: a social question of values and philosophy. As we move into the year 2000, evaluation research is embarking on an exciting new frontier.

Fay Walther is a Principal Statistician, supervising Labor Market Information (LMI) projects with Research & Planning.

^* If you are interested in a Workforce subscription or just want to see what it's about, you can look at their online page at Workforce Online.

Go Back To:

Table of Contents

Labor Market Information

Employment Resources

Send Us Mail