© Copyright 1999 by the Wyoming Department of Employment, Research & Planning

Performance Accountability in the Workforce Investment Act: An Application with Division of Vocational Rehabilitation Data Part Two
by: Tony Glover, Senior Analyst

"The problem with employment evaluation research lies in isolating the economic factors that influence the population, control group and the performance of clients in a training program."

P art One of this series addressed the performance management principles of the "Performance Accountability System" specified by the Workforce Investment Act (WIA)1 using Division of Vocational Rehabilitation (DVR) program data. Where performance management techniques focus on the measurement of some value (i.e., earnings) before and after training for a group of clients, they fail to account for external factors that influence program outcomes. By contrast, when the design of the analysis shifts to performance evaluation, it means we also measure the earnings for representative groups of individuals over the same period and within the same conditions as the client group to assess the impact of a training program. These control groups (theoretically) represent all of the contextual changes affecting the client group as well as such things as maturation. Part Two examines evaluation methods, bringing into focus the shortcomings of exclusive reliance on management techniques and showing the value of a combined approach.

This article finds that two of the three core indicators for workforce investment activities, from Part One--entered employment rate and earnings gained in employment--are sensitive to influences from surrounding economic conditions. Economic conditions directly affect the opportunity structure of the labor market, impacting prospects of finding and retaining employment and increasing earnings. Thus, interpreting the performance outcomes of a workforce training program requires an awareness of local economic conditions.

This study contrasts DVR clients with two groups, a matched control group and a population group. Simply defined, a matched control group is a representative subset of the population similar to DVR clients on a number of characteristics (i.e., age, sex, prior income) that operates in the same environment as DVR clients. The population group represents the activity of the Wyoming labor force in the context of surrounding conditions.

A true population criterion design would designate a control group from the population that does not differ significantly from the client group analyzed. However, we lack an indicator to select a control group mirroring the DVR population on the variable of disability. Therefore, this method does not lend itself to a true impact analysis of whether and to what extent the DVR program affects its clients. However, building a longitudinal baseline of the differences between the DVR and control groups permits us to assess future changes in the DVR program itself.

Differences in labor market outcome between DVR clients, the control group, and the population discussed in the next section, are consistent across time. The hypothetical examples shown introduce the idea that success of a program is discovered through changes in the differences between cohorts (common groups of individuals) of the clients, control and population groups.

Consider a hypothetical situation from Table 1, where for the past seven years, DVR clients’ entered employment rate on average fell 10 percent below (plus or minus 2 percent) that of the control group. If in the eighth year the program’s performance declines to 20 percent below the earnings of the control group, consequently falling outside the eight to 12 percent range (explained by DVR clients’ disabilities), the performance of the program should be investigated to assess changes in program administration or other internal factors.

In the three hypothetical examples (Figures 1, 2 and 3), each point on a line corresponds to a cohort’s performance on some indicator for that respective year; for example, the entered employment rate discussed in Part One. These examples contrast DVR clients’ program performance with a proportionally matched control group, displaying the relationship between performance and economic factors affecting the entire labor market.

Figure 1: DVR clients perform consistent with the economy in this example (as represented by the control group). Knowing the performance of DVR clients, without knowing how the environment and similarly situated individuals are changing, does not allow us to determine whether the performance decreases occurring after 1994 stemmed from changes in the program or economic conditions.

Figure 2: The difference between DVR and the control group’s performance decreases over time. This positive result shows continuous improvement. Although DVR clients show a performance level below the control group, the example illustrates DVR performance consistent with the economy.

Figure 3: This example represents a worst case scenario. The program displays a continuous decline in DVR performance relative to the control group, while the control group improves. In this case, some aspect of the program (i.e., training provided, participant selection) creates a situation where the program fails to meet expectations.

As these examples reveal, tracking DVR program performance over time without the benefit of a control group, would lead to incorrect conclusions based entirely on management techniques. Interpreting a performance management system demands a frame of reference to compare the effects of external factors such as the economy on performance.

Population and Control Group Selection

We selected the population group by compiling the Wyoming Department of Employment’s (DOE) administrative databases and determining if individuals resided in Wyoming in the reference years (1994, 1995 or 1996). The years correspond to available DVR client data. To determine residency, we relied on the following databases: Unemployment Insurance Claims (UI), Wyoming’s Driver’s License (DL), Employment Services (ES) and wage records (WR). A determination of Wyoming residency required a valid Wyoming zip code in the reference year from the first three databases listed above. Residency was further determined by whether a person had wages in WR during the reference year.

The largest possible control group for the reference year was selected from the available population group of the corresponding reference year. To accomplish this, the three cohorts of DVR clients were proportionally matched to a subset of the population on sex, age and total wages of the four quarters prior to the reference year. Table 2 shows the number of DVR clients along with the persons in the population and control groups for each of the three cohorts of analysis.

In Table 2, only those DVR clients with no wages in WR the quarter prior to program participation are used to calculate the entered employment rate. Similarly, the entered employment rate of the control group reflects those with no wages in the quarter prior to the reference year. Only those DVR clients or control group members who entered employment or who had wages in the quarter prior to the reference period were used to calculate retention in employment and earnings gained. Due to the large number of those with wages in the quarter prior to the reference period among the population group, the number of people increases for core indicators 2 and 3.

As mentioned earlier, the matching criteria did not use disability status. Despite the omission, differences between DVR clients and the control group are consistent over time: DVR program eligibility requires a disability, and the number of disabled persons in the control group and the population remains unknown. The performance of the DVR program is not assessed by whether its clients achieve better results than the control group. Instead, assessment of the DVR program considers program performance relative to the economy as represented by control group performance.

Analysis of DVR’s database revealed that the duration of services for clients corresponded on average to a four-quarter period. Where the reference year for DVR clients represents training periods of various duration, the four quarters of a given year define the reference year for the population and control groups. The progress of DVR clients or members of the control or population groups are measured using quarterly data from wage records, prior to and following program participation or the equivalent reference year.

The entered employment rate for DVR clients represents the number of clients with no wages the quarter prior to application (Q A-1) and wages the quarter following closure (Q C+1) divided by all clients with no wages the quarter prior to application (Q A-1 - see Formula 1). The control group and population consist of individuals with no wages in the quarter prior to the reference year (Q RY-1) and wages in the quarter following the reference year (Q RY+1) divided by all participants with no wages the quarter prior (Q RY-1 - see Formula 2). The retention in employment rate and earnings gained in employment discussed in Part One are based on different formulas using wage records.

Formula 1: Entered Employment Rate of DVR Clients

No Wages Q(A-1) with Wages Q(C+1)
Entered Employment =
Rate All Individuals with No Wages Q(A-1)

Formula 2: Entered Employment Rate for Population and Control Groups

No Wages Q(RY-1) with Wages Q(RY+1)
Entered Employment =
Rate All individuals with No Wages Q(RY-1)

Results

Table 3 shows the three cohorts’ year-to-year entered employment rate, retention in employment rate and earnings gained in employment for DVR clients, the control group and the population. As mentioned in Part One, the DVR program showed a continuous improvement in the entered employment rate (Figure 4) from 35 percent for the 1994 cohort to 37 percent for the 1995 cohort. The control group’s entered employment performance decreased about four percent from 35 percent for the 1994 cohort to 31 percent for the 1995 cohort, and the population’s entered employment rate also decreased (-5%). The decrease in the performance of these two groups suggests economic change that had no apparent impact on the DVR clients’ performance.

Wyoming experienced a slowdown in economic growth 2 in 1995 that corresponds to the decrease in the control group and population’s performance. Figure 4 clearly illustrates the manner in which the control and population groups move in tandem with the economy. Figure 5 graphically represents the economic slowdown’s impact on the growth in the total number of jobs based on the ES-202 database. The 1994 cohort responded to the creation of 9,584 jobs in the economy from 1993 to 1995, while the 1995 and 1996 cohorts experienced a situation whereby only 4,250 and 4,809 jobs were created in those years, respectively. While the total number of jobs continued to increase from 1993 to 1997, the slowdown in jobs created for the 1995 cohort decreased the opportunity to find employment, thus impacting the entered employment rate.

The type of industry employing individuals following the reference year contributed to the difference in results among the three groups. For the most part, DVR clients and the other groups, across all three cohorts, were placed in the Retail Trade and Services industries. However, the population and control groups of the 1994 and 1996 cohorts showed more even distribution among higher paying industries.

The combination of the slowdown in job growth and the shift in available jobs to industries generally paying lower wages contributed to the decline in performance for the control and population groups. Perhaps, when faced with economic conditions such as these, individuals in the control group and population left the state. Although DVR clients participated in a program designed to place them in employment (most likely in Wyoming), the analysis cannot infer motivation to work in Wyoming to the population and control groups. Currently, Research & Planning has not yet gained the means--interstate agreements for sharing wage records--to track the labor movement beyond state borders.

Figure 6, based on data from Table 3, compares the year-to-year six month earnings gains for DVR clients, the control group and the population. The earnings gained by all three groups decreased for the 1995 and 1996 cohorts. To demonstrate the impact of the economic slowdown on earnings gained, Figure 7 represents the economic conditions the cohorts faced, measured by the average annual wage (AAW) per job from the ES-202 database. The AAW increased $1,495 for the 1994 cohort, $24 for the 1995 cohort and $1,243 for the 1996 cohort.

We expected to see the decrease in DVR performance for earnings gained in employment for the 1995 and 1996 cohorts, considering the control group and population both experienced a corresponding decrease in performance for these cohorts. Further evidence generated from the ES-202 database supports the conclusion that the decrease in earnings gained stemmed from deteriorating economic conditions, not DVR program changes. This conclusion illustrates the benefits realized by the combined application of performance management and performance evaluation techniques. If, as mentioned in Part One, only DVR performance across the three cohorts is known, the program would be at a loss to justify the decrease.

Conclusions

The two indicators discussed, entered employment rate and earnings gained in employment, show the descriptive concepts surrounding population criterion performance evaluation. The analysis presented economic factors influencing the performance of DVR clients, the control group and the population. These factors offer a limited explanation of economic influences on performance.

The hypothetical examples in the introduction compensate for a lack of historical data for the DVR program. With more data available, researchers can better determine the relationship between the factors discussed (data generated from the ES-202 databases) and program performance. Access to a longer time series of data would have informed and enriched the processes and results addressed in this article. Further, the process must be applied across other workforce investment activities. Currently, we do not know if clients of the Job Training Partnership Act (JTPA) programs behaved similarly to the DVR clients under the same economic conditions.

The research presented in this series is exploratory in nature. While the content is difficult, we will continue to familiarize readers with the concepts of performance management and performance evaluation. The problem with employment training evaluation research lies in isolating the economic factors that influence the population, control group and the performance of clients of a training program. After isolating the factors involved, evaluation research attempts to understand the relationships between the factors and the subsequent impact the interrelationships have on program performance.

1 The Workforce Investment Act of 1998, Pub. L. No. 105-220 (1998). Sec. 136.
2David Bullard, "Total Payroll as a Tool for Identifying Business Cycles in Wyoming," Wyoming Labor Force Trends, May 1999.


Table of Contents | Labor Market Information | Wyoming Job Network | Send Us Mail

These pages designed by Gayle C. Edlin.
Last modified on by Valerie A. Davis.