Measuring the Impact of Wyoming's Workforce Development Training Fund: Part Two

Measuring the Impact of Wyoming's Workforce Development Training Fund: Part Two

by: Mark A. Harris, Sociologist, Ph.D.

"There is a good reason to believe the differences we observe between the matched and participant samples are due to program effects."

The Wyoming Workforce Development Council (WWDC), created by Executive Order 1998-1, is responsible for coordinating a workforce development system that serves the needs of all Wyoming residents, students, and employers by integrating economic development, training, education, and employment opportunities. The Council also has oversight responsibilities for the workforce programs within the workforce development system such as Workforce Investment Act (WIA) youth, adult, and dislocated worker programs, Adult Education, and Vocational Rehabilitation. A strategy the WWDC proposes for reaching its goal of
increasing economic opportunity for Wyoming workers¹ is to support programs that demonstrate success in wage progression.²

A major training program for Wyoming workers supported by the WWDC is the Workforce Development Training Fund (WDTF). The May 2002 issue of Wyoming Labor Force Trends contained part one of this article, which described the wage experience of WDTF completers. Part two examines the wage experience of WDTF completers within the context of a matched control group and multi-variate statistical analysis. Such a strategy allows us to compare wage progression of program participants with individuals who did not participate in WDTF training. Results indicate that WDTF participants have higher wages after training than those who did not participate.

Goals

The goal of the quasi-experimental research presented here is to determine whether program participation has a net effect on wages above what happens to a matched control group. As such, Research & Planning (R&P) constructed matched control groups for the WDTF participant groups. This was done by stratifying the participant groups on relevant theoretical characteristics and then selecting control groups with identical (or matched) characteristics. In principle, having matched control groups allows us to determine whether the outcome variables (i.e., wages) are different between the control and participant groups. Equivalence between the two groups is not assured; however, in the absence of random assignment to control and experimental groups, the proposed strategy is superior to non-experimental design, especially if employed longitudinally.

Theoretically, we expect the WDTF group will have higher subsequent wages than its matched control group. If this is the case, our assumption is that the training programs had a net effect on wages beyond what took place in the control group. However, if the control and participant groups are not adequately matched (i.e., if there are theoretically important differences between the two groups), the differences, rather than the explanatory variable of interest (i.e., program participation), may have caused the variation in the outcome variables. For example, if program participants are highly motivated workers, but members of the control group are not, differences in outcome variables could be attributed to this personality characteristic rather than participation in training. In other words, highly motivated workers would likely have better wages with or without participating in training.

We only have a limited number of theoretical variables with which to stratify our sample and select a control group with similar characteristics. If the stratification variables we have selected are theoretically relevant, then there is good reason to believe the differences we observe between the matched and participant samples are due to program effects.

Methods

Participant groups for this study consist of WDTF participants who finished their training in fiscal years 1999 (FY99) and 2000 (FY00). To be included in the study, participants had to have wages in R&P's Wage Records³ database for at least two quarters in the year prior to the year training ended. The WDTF group was stratified by gender, five age categories, and wage quintiles⁴ for the average quarterly wage in the year prior to the year training ended. Wage Records for this study included data from Wyoming, Colorado, Idaho, Nebraska, New Mexico, South Dakota, Texas, and Utah. Using regional wage data increases the likelihood of capturing wages for participants both before and after training (i.e., it increases sample size). Regional data are also theoretically relevant because skills gained from training should be related to a wage increase regardless of whether or not the participant remains in Wyoming.

A matched control group of individuals who did not participate in WDTF training (during the period of interest) was then selected from Wage Records. The selection was accomplished by constructing strata with the same age, gender, and prior wage characteristics as the participant groups, then randomly selecting a proportionate sample from the different strata.

Quasi-Experimental Results

Figure 1 presents average quarterly wages by year for the FY99 WDTF participant group. As can be seen in Figure 1, the FY99 participant group experienced an increase in wages subsequent to training. We pose the question “How does the earnings capacity of the participant group compare to the experience of the matched control group?” Figure 2 adds the control group to the graph. As shown in Figure 2, the WDTF participant group's average quarterly wage was about $250 dollars more than the WDTF control group two years after the year training ended.⁵

Figures 3 and 4 present similar data for the FY00 group. The major difference between Figures 1 and 2, and Figures 3 and 4, is for the latter two we only have one year of wage data after training. As shown in Figure 4, the WDTF participant group had higher wages (approximately $400) than the matched control group in the year training ended and one year after training ended.

Taken together, Figures 2 and 4 indicate that, relative to their respective control groups, the FY00 participant group performed better than the FY99 participant group. Basic differences in the participant groups between the two fiscal years may account for this, or it may be that WDTF program effectiveness is increasing over time.

In sum, the results presented here seem to indicate that, as a group, WDTF participants experience wage progression relative to a matched control group. Assuming the stratification variables we have selected are the appropriate theoretical controls, the data show that training associated with the WDTF may be effective in increasing wages above and beyond the experience of a matched control group.

Multi-Variate Tests

The following sections present Ordinary Least Squares (OLS) regression results for the WDTF FY99 and FY00 groups. In an effort to stimulate comparison and replication, we present the results in tabular form (see Tabular Regression Results) for readers who are familiar with regression analysis. For readers unfamiliar with tabular regression results, we present the same data in graphical form (see Graphical Regression Results).

Matched control group designs are useful in examining group differences, but they have limitations. Specifically, it becomes impractical to build more than a few controls into the selection process when participant groups are small. The more variables one desires to stratify or control simultaneously (e.g., age, gender, wages), the larger the participant group required to have sufficient cell sizes. As such, matched control designs are inherently limited in the number of factors that can be controlled simultaneously. For instance, in our control group construction process we used five age categories, two gender categories, and five wage categories. Thus, we divided our participant group by all logical combinations of these three variables, and then created a matched control group based on this stratification. Including an additional variable (e.g., industry) adds additional categories. When doing so, the likelihood of having very small, or zero, cell sizes for any of the logical combinations of stratification variables increases and makes the stratification process unworkable.

One solution to the problem of small cell sizes is to employ multi-variate statistical tests that control for various characteristics statistically rather than building them into the control groups manually through the stratification process. This is a technique suggested by Rossi, Freeman, and Lipsey.⁶

To this end, we utilize Ordinary Least Squares (OLS) regression techniques. The basic logic behind OLS regression is to statistically control for relevant theoretical variables that could explain higher wages and then, after these factors have been accounted for, examine whether program participation is a significant predictor of wages subsequent to training. For instance, once we control for age, gender, and other factors, does program participation explain or account for variation in subsequent wages? This is substantively similar to the theory behind creating matched control groups. The advantage here is that it provides a measure of the “net” effect of program participation on subsequent wage outcomes and tells us whether this effect is statistically significant. Cell size problems are less of an issue for multi-variate techniques such as OLS than the stratification process in matched control group designs.

Tabular Regression Results

Table 1 presents OLS unstandardized regression coefficients (b's)⁷ for average quarterly wages subsequent to training regressed on age, age-squared,⁸ a dummy variable⁹ for gender (males compared to females), prior average quarterly wage, a dummy variable for industry (goods producing compared to services producing industries), and a dummy variable for program participation (participants compared to non-participants). With the exception of program participation, all independent variables are measured one year prior to the year training ended. Prior industry is an additional variable being controlled for that was not accounted for in the matched control group design presented earlier. Individuals employed in a goods producing industry prior to training may have a different level of wages subsequent to training relative to those employed in a services producing industry. The sample utilized for these regressions is the same as that used in the design presented earlier, with the additional criteria that those included in the OLS model have at least two quarters worth of wages in the year after WDTF training ended. This additional criteria creates a more reliable estimate of average quarterly wages for individuals in the regression equations.

As shown in Table 1, being male is positively and significantly related to subsequent wages for all individuals under study. In other words, males have significantly higher quarterly wages than females (approximately $790 for FY99 and $717 for FY00). Age has a significant curvilinear effect for both FY99 and FY00 as noted by the significant age-squared term - indicating that wages peak near the middle of the age distribution and then decline. Working in a goods producing industry is also positively and significantly related to subsequent wages. Those who work in a goods producing industry, on average, have significantly higher wages (approximately $102 for FY99 and $189 for FY00) than those who work in a services producing industry. Prior wages are also positively and significantly related to subsequent wages - indicating that those with higher prior wages have higher ending wages.

Of central theoretical importance, the WDTF participant dummy variable is significant. As a group, WDTF participants have significantly higher average quarterly wages than non-participants subsequent to training (approximately $359 for FY99 and $391 for FY00). Thus, R&P cannot rule out the possibility that participation in the WDTF training, in fact, increased wages of WDTF participants above the wages of those who did not participate in the training when controlling for age, gender, industry, and prior wages. It appears that we have found a program effect for WDTF participants.

Graphical Regression Results

We illustrate predicted average quarterly wage outcomes for WDTF participants in Figure 5. As an example, we present the results for a 24-year-old, in a services producing industry, with average wages one year prior to training for their specific group (i.e., the average quarterly wages for the FY99 group were $2,769 and $3,459 for the FY00 group). Figure 5 shows results for males and females separately. As can be seen in Figure 5, WDTF male and female participants for both FY99 and FY00 have significantly higher wages than those who did not participate in training.

Conclusions and Directions for Future Research

Although the results presented here are supportive of the argument that WDTF participants experience wage progression relative to a matched control group, we can only speculate on the source of the difference. The addition of industry to the OLS model shows that this factor does not “explain away” the significant relationship between program participation and subsequent wages for those involved in WDTF. Some theoretical possibilities that are not controlled for in our matched control group or OLS designs include firm characteristics, such as size, geographic location, and progressive compensation packages. An additional step is to conduct further OLS analyses that measure and test important firm characteristics on wage progression. More variables could be added to the multi-variate models without the difficulty associated with adding variables to the matched control group design.

Beyond these steps, additional research is needed to determine long-term training effects. In particular, we would like to determine whether the WDTF group continues to experience higher wages. There is some indication of this as noted by the different lag times in the study results (i.e., one or two years after training completion). Even though unanswered questions remain, this article demonstrates the advantages of matched control and multi-variate design for exploring the effects of employment training programs. Without comparative control groups or multi-variate statistical controls, we have no context within which to place the wage experience of training participants.

¹Alfrieda Gonzales, Strategic Plan Vision Statement, Wyoming Workforce Development Council, June 2001.

²Alfrieda Gonzales, Goals of the Wyoming Workforce Development System, Wyoming Workforce Development Council, June 2001.

³Wage Records is an administrative database. Each employer in the State that has employees covered under Unemployment Insurance, by law, must submit quarterly tax reports to the State showing each employee's Social Security Number and wages earned in the quarter. Wage Records has a two-quarter time lag (e.g., wage information for first quarter 2001 employees is not available until third quarter 2001). For more information, see Wayne M. Gosar, “Insurance Wage Record Summary: A New Way to Look at Wyoming,” Wyoming Labor Force Trends, May 1995, pp. 4-8.

⁴Outliers on the top and bottom of the wage distribution were removed before the groups were broken into wage quintiles.

⁵Statistical tests of program and control group differences are presented in the multi-variate tests.

⁶Peter H. Rossi, Howard E. Freeman, and Mark W. Lipsey, Evaluation: A Systematic Approach, 2002.

⁷The coefficients can be interpreted as increases or decreases in average quarterly wages (depending on a positive or negative sign) for a one-unit increase in the variable of interest. To illustrate, males in the Workforce Development Training Fund Fiscal Year 1999 group earn, on average, approximately $790 more in average quarterly wages than females in the same group.

⁸We also include an age-squared term due to the strong possibility that wages peak near middle age and then decrease with time. If this is the case, then the appropriate functional form for age is curvilinear. A significant age-squared term in OLS regression models indicates the relationship is curvilinear.

⁹The term dummy variable is a standard statistical term in which the members of the group of interest are coded as 1 and the members of the comparison or “dummy” group are coded 0.

Table of Contents | Labor Market Information | Wyoming Job Network | Send Us Mail

These pages designed by Julie Barnish.
Last modified on by Susan Murray.