© Copyright 2004 by the Wyoming Department of Employment, Research & Planning

WYOMING LABOR FORCE TRENDS

Vol. 41 No. 4    

 

 

Examining Workforce Information Act Programmatic Outcomes Using the Wyoming Wage Records Universe as a Statistical Comparison Group

by: Sylvia D. Jones, Statistical and Research Analyst

WIA participation appears to be very effective in improving the economic lives of participants. For example, 65 percent of Dislocated Workers and 61 percent of Adults experienced at least one positive outcome from participation; many of those experienced more than one positive outcome. 

The U.S. Department of Labor, Employment and Training Administration (ETA) currently uses employment, job retention, and earnings increase as performance measures for programs it funds (U.S. Department of Labor, ETA, 2003). Section 171 of the 1998 Workforce Investment Act (WIA) requires the Secretary of Labor to prepare a five-year research plan for pilot, demonstration, research, evaluation, multi-state, and multi-service initiatives every two years. ETA's research plan provides for the evaluation of existing programs but it also covers applied research, including labor market research. Section 171 requires the continuing evaluation of programs and activities using appropriate methodologies and research designs, especially experimental research designs (U.S. Government Accounting Office, 2004). Unfortunately, experimental research is expensive, time consuming, and often raises legal and ethical issues. This article addresses the current performance measures used by ETA and provides alternative measures that could improve program outcomes.

A basic limitation with ETA’s current performance measurement strategy is the failure to measure outcomes in the context of the surrounding labor market. Without the labor market context, neither the state nor ETA can determine whether changes in wages and retention are effects of WIA participation or simply a reflection of the local economy. 

ETA's current evaluation strategy also uses a limited number of outcome measures, preventing a thorough understanding of program outcomes. Research emphasis has been placed on finding new ways to evaluate the current outcomes without determining whether or not those outcomes represent program success. While performance measures can drive program development and management, it is not clear that federally required measures capture relevant information pertinent to employers, workers, or program managers.

Methods
A broader evaluation of current performance measures was conducted by putting Wyoming WIA training participants in the context of the Wyoming labor market. New outcome measures are investigated as potential enhancements to the strategy already in place. 

The study group included Wyoming WIA participants for the Program Year 2001 and 224,217 individuals selected from the Wyoming Unemployment Insurance Wage Records file. There were 568 individuals who participated and exited WIA training during the program year, including 217 Adults, 110 Dislocated Workers, and 241 in Youth programs. The Youth groups were removed from the analyses because their outcome measures differ significantly from Adults and Dislocated Workers. The WIA participant file was combined with UI Wage Records in order to obtain wage and employment information. Turnover, employment industry, UI claims rate, and success were investigated. 

Turnover was defined as the number of exits in employment divided by the number of jobs worked during the defined time period (Glover, 2002). If an individual exited all jobs worked, the turnover rate would be 1.0. 

The researcher defined employment industry as the North American Industry Classification System (2004) sector of the primary employer in each quarter. Primary employer was defined as the employer which paid the individual the most wages in a quarter, regardless of the number of employers. For purposes of this study, living-wage industries were defined as those which pay, on average, wages higher than 130 percent of the federal poverty guideline for a family of four (Harris, 2003). The UI claims rate reflects the number of participants who filed a UI claim at any time during a quarter, not those who actually received UI benefits.

The final outcome investigated in this study was a two-category measure that reflects "success." Individuals received a success rating if they met any of the following criteria:  

Discussion / Suggested Measures for ETA Outcomes
Study findings were mixed. Other studies which showed short-term earnings gains were replicated because wages of both WIA program segments increased after termination from the program. However, their wages did not increase more than the statistical control group. 

While WIA participants initially showed a rather large increase in earnings after program termination (see Figure 1); those high earnings started to decline after the second quarter. This could occur if participants stayed with their On-the-Job Training employer for two quarters after program termination and then needed to look for new employment, or if participants faced retention barriers, such as childcare issues. 

Contrary to expectations, changes in turnover rates cannot be predicted by WIA participation either, even though turnover does significantly decrease after program exit for both participant groups used in the study (see Figure 2).

The industry measure revealed interesting results. The industrial distribution of Adult participants changed considerably after training (see Figure 3). The percentage of individuals employed in low-paying industries decreased. Alternatively, employment in living-wage industries increased. The implications of this are very positive. WIA participation for Adults appears to result in a shift from no or low-paying employment, to employment in higher paying industries. 

Dislocated Workers did not fare as well as Adult participants in the industry measure (see Figure 4). More Dislocated Workers were employed six quarters before program termination than eight quarters post-program. Employment in living-wage industries declined. However, employment in low-paying industries also declined. It is possible that Dislocated Workers are leaving the state for employment elsewhere, becoming employed in non-UI covered jobs, or becoming self-employed. Enrollment in a community college or university without concurrently working a UI covered job would also remove them from Wage Records. Outside influences could also affect Dislocated Workers' outcomes. For instance, Dislocated Workers tend to be firmly established in a job before they become eligible for the WIA program. After program completion, they are encouraged to find employment in an area in which they have no experience. They face competition from applicants with more relevant experience for jobs that often pay less than what they originally earned. Frustration with these situations could lead to discouragement and an eventual withdrawal from the labor market. This hypothesis could be tested using longitudinal and inter-state data.

Before program termination, the WIA participation groups were less likely to be employed than the Comparison group (see Figure 5). After program termination, there is no noticeable difference between the groups. The WIA program appears to be allowing participants to become indistinguishable from the average Wyoming worker, and that is an important accomplishment. 

Findings in the success measure are most interesting. WIA participation appears to be very effective in improving the economic lives of participants. For example, 65 percent of Dislocated Workers experienced at least one positive outcome from participation; many of those experienced more than one positive outcome. The same is true for Adults. Except for Adults who enrolled in a Wyoming community college, who tended to withdraw from the labor market during their education, very few participants had only one positive outcome. The majority of Adults with a successful outcome increased their wages, decreased their turnover rate, and gained employment in a living-wage industry. In essence, they epitomize the unwritten goals of the WIA program. 

Summary
While the capacity to conduct true experimental design and replication is limited, this study offers a powerful alternative analysis. By comparing human capital and work experience between WIA participants and a comparable segment of Wyoming's workforce, it was determined that WIA participants basically became indistinguishable from the rest of Wyoming's workforce, an arguably favorable outcome. However, the Wyoming study shows that no program effect exists for wages or turnover, at least in the 24 months following program completion. In the absence of a strong economy, participants may have failed to become indistinguishable. Training programs likely have difficulty showing success in economically challenged environments. Additionally, the federally mandated outcome measures fail to reflect all of the successes of program completers. ETA should consider changing the focus of WIA program evaluation to measures that capture relevant information pertinent to employers, workers, and program managers.

References
Glover, T. (2002). Market dynamics from administrative records; Seven state project report for December 2002. Retrieved April 16, 2004, from http://doe.state.wy.us/lmi/w_r_research/MarketDynamics1202.pdf  

Harris, M.A. (2003). Identifying potential living wage employment growth opportunities in Wyoming. Wyoming Labor Force Trends. Retrieved on April 6, 2004, from http://doe.state.wy.us/lmi/0503/a1.htm 

North American Industry Classification System. (2004). North American Industry Classification System (NAICS) at BLS. Retrieved April 14, 2004, from http://www.bls.gov/bls/naics.htm

United States Department of Labor, Employment and Training Administration. (2003, December). Training and employment guidance letter No. 15-03. Retrieved April 16, 2004, from http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1535 

United States General Accounting Office.(2004). Workforce training. (GAO-04-282). Retrieved April 16, 2004, from http://www.gao.gov/ 

 

Table of Contents | Labor Market Information | Wyoming Job Network | Send Us Mail

These pages designed by Julie Barnish.
Last modified on by Susan J. Murray.