Mark Harris, Sociologist, Ph.D.

Determining Whether There is a Net Effect of Training Programs on Wages Through the Use of Control Groups (Quasi-Experimental Design) and Multi-Variate Analyses

The goal of the quasi-experimental research presented here is to determine the net effect of various training program participation on worker wages.  To do this we constructed a matched control group to compare outcome measures.  The control groups are constructed by stratifying the experimental group on relevant theoretical characteristics (age, gender, prior wages).  We then select a control group with identical (or matched) characteristics.  Control groups were constructed using administrative data (in this case Wage Records and the Department of Motor Vehicle’s Drivers License Data Base) that span the same time frame of interest and incorporate similar demographic and wage characteristics.  In principle, having a matched control group allows us to determine whether the outcome variables (i.e., wages) are different between the control and experimental groups.  We also examine OLS regression equations to determine or identify program effects.  Theoretically, we expect that our experimental groups will have higher wages than a matched control group.  If we find this to be the case, our assumption is that the experimental training program had a net effect on wages above and beyond other important characteristics. 

St. Paul Copy Draft Copy, 4/15/02

By: Mark A. Harris, Sociologist, Ph.D.

Determining Whether There is a Net Effect of Training Programs on Wages Through the Use of Control Groups (Quasi-Experimental Design) and Multi-Variate Analyses

Overview

One of the primary objectives of any employment training program should be to increase the earnings of participants.  This is the case with programs supported by the Wyoming Workforce Development Council (WWDC). 

The WWDC was created by Executive Order 1998-1 and is responsible for developing a workforce development system that serves the needs of all Wyoming residents, students, and employers by integrating economic development, training, education, and employment opportunities. The Council has oversight responsibilities for workforce programs.

One of the stated goals of the WWDC is to “increase the economic opportunity and self sufficiency for all Wyoming workers ….” [1]   One strategy the WWDC proposes for doing this is to “take advantage of programs with demonstrated success in wage progression.”[2] 

A major training program supported by the WWDC for Wyoming workers is the Workforce Development Training Fund (WDTF).  This paper is the first to examine the wage experience of WDTF completers and a matched control group.  Such a quasi-experimental strategy allows us to determine whether WDTF participants show “wage progression” in comparison to a matched set of individuals who did not participate.  In addition to a quasi-experimental examination, this paper presents a multi-variate statistical examination of program effects.  Results indicate that WDTF participants have higher wages after training relative to a matched control group.

This study also examines wage results for Workforce Investment Act (WIA) and Job Training Partnership Act (JTPA) participants (hereafter referred to as WIA).  Results for this group do not demonstrate wage progression in comparison to a matched control group.  However, there is some evidence to suggest potential wage convergence over time. 

Goals

The goal of the quasi-experimental research presented here is to determine whether program participation has a net effect on wages above and beyond what happens to a matched control group.  As such, Research & Planning (R&P) constructed matched control groups for the WDTF and WIA groups.  This was done by stratifying the participant groups on relevant theoretical characteristics and then selecting control groups with identical (or matched) characteristics.  In principle, having a matched control groups allows us to determine whether the outcome variables (i.e., wages) are different between the control and participant groups.  Equanimity is not assured — however, in the absence of random assignment to control and experimental groups the proposed strategy is superior to non-experimental design, especially if employed longitudinally.

Theoretically, we expect that the WDTF and WIA groups will have higher wages than their matched control groups.  In other words, training should provide some benefit that does not accrue to a control group.  If we find this to be the case, our assumption is that the training programs had a net effect on wages above and beyond what took place in the control group.  However, if the control and participant groups are not adequately matched (i.e., if there are theoretically important differences between the two groups), these differences, rather than the explanatory variables of interest (i.e., program participation), may have caused the variation in the outcome variables.  For example, if program participants are highly motivated workers (a personality characteristic), but members of the control group are not, this could explain differences in outcome variables rather than participation in training.  In other words, highly motivated workers would likely have better wages with or without participation in training. 

We only have a limited number of theoretical variables with which to stratify our sample and select a control group with similar characteristics.  If the stratification variables we have selected are theoretically relevant then there is good reason to believe that the differences we observe between the matched and participant samples are due to program effects.  However, if we have missed important theoretical selection criteria, we lose confidence that differences in outcome variables are due to program effects.

Methods

Participant groups for this study come from WDTF and WIA participants who finished their training in fiscal years 1999 and 2000.  To be included in the study, participants had to have wages in R&P’s Wage Records database for at least two quarters in the year prior to the year that training ended.  The WDTF and WIA groups were stratified by gender, five age categories, and wage quintiles [3] for the average quarterly wage in the year prior to the year that training ended.  Wages Records for this study include data from Wyoming, Colorado, Idaho, Nevada, New Mexico, South Dakota, Texas, and Utah.  Using this regional wage data increases the likelihood that we will capture wages for participants both before and after training (i.e., it increases our sample size).  It is also theoretically relevant because skills gained from training should be related to a wage increase regardless of whether the participant remains in Wyoming or not. 

Matched control groups were then selected from Wage Records for individuals who did not participate in WDTF or WIA training (at least during the period of interest for each group).  This was done by constructing strata with the same age, gender, and prior wage characteristics as the participant groups and then randomly selecting from the different strata a proportionate sample.

Results

Figure 1 presents average quarterly wages by year for fiscal year 1999.  Average quarterly wages are first calculated for the individual and then for the group.  Only the WDTF and WIA participant groups are included.  As can be seen in the graph, both the WDTF and WIA participant groups experience an increase in wages subsequent to training.  The question that we pose is, “how does this compare to the experience of the matched control groups?”  Figure 2 adds the control groups to the graph.  As shown in Figure 2, the WDTF participant group’s average quarterly wage is about $250 dollars more two years after the year training ended than the WDTF control group.   In comparison to the WIA control group, the WIA participant group has a lower average quarterly wage (by about $250) two years after the year training ended. [4]   However, the slope of WIA participant line, if continued for an additional year, may converge with the WIA control line — suggesting similar average quarterly wage levels at a later time.

Figures 3 and 4 present similar WDTF and WIA data for fiscal year 2000.  The major difference between Figures 1 and 2 and Figures 3 and 4 is that for the latter two we only have one year of wage data after training.  As shown in Figure 4, the WDTF participant group had higher wages in the year training ended and one year after the training ended (approximately $400).  Interestingly, the WIA group had lower wages in the year training ended but closed the gap one year after training ended. 

Taken together, Figures 2 and 4 indicate the WDTF and WIA participant groups performed better relative to the control groups in 2000 than in 1999.  Basic differences in the cohorts between the two fiscal years may account for this, or it may be that program effectiveness is increasing over time. 

In sum, the results presented here would seem to indicate that, as a group, WDTF participants experience wage progression relative to a matched control group.  Assuming that the stratification variables we have selected are the appropriate theoretical controls, the data show that training associated with the WDTF may be effective in increasing wages above and beyond the experience of a matched control group. 

The same conclusions cannot be drawn for the WIA group.  WIA average quarterly wages were either lower or equal to the WIA control group.  However, given that the slopes of the lines for the WIA participants suggest convergence, we cannot conclude that the WIA group will have lower wages than the WIA control group at a later point in time.  Additional study over time is required to determine this. 

Multi-Variate Tests

Although quasi-experimental design is useful in examining differences between groups it does have limitations.  Specifically, it becomes impractical to build more than a few controls into the selection process when participant groups are small.  The more variables one desires to stratify or control simultaneously (e.g., age, gender, wages) the larger the participant group size required to have sufficient cells sizes.  As such, quasi-experimental designs are inherently limited in the number of factors that can be controlled simultaneously.  For instance, in our control group construction process we used five age categories, two gender categories, and five wage categories.  Thus, we divide our participant group by all logical combinations of these three variables and then create a matched control group based on this stratification.  Adding an additional variable, say for example SIC industry, adds additional categories.  When doing so, the likelihood of having very small, or zero, cell sizes for any of the logical combinations of stratification variables increases and makes the stratification process unworkable.

One solution to this problem is to employ multi-variate statistical tests that control for various characteristics statistically rather than building them into the control groups manually through the stratification process.  This is a technique suggested by Rossi, Freeman, and Lipsey. [5]  

To this end, we utilize Ordinary Least Squares (OLS) regression techniques.  The basic logic behind OLS regression is to statistically control for relevant theoretical variables that could explain higher wages after training and then, after these factors have been accounted for, examine whether program participation is a significant predictor of subsequent wages after training is completed.  For instance, once we control for age, gender, and other factors does program participation explain or account for variation in subsequent wages?  This is substantively similar to the theory behind creating matched control groups.  The advantage here is that it provides a measure of “net” effect of program participation on subsequent wage outcomes and whether this effect is statistical significant.  Multi-variate techniques such as OLS are not constrained to the same cell size problems that are faced with the stratification process in quasi-experimental designs.

Table 1 presents OLS regression coefficients for average quarterly wages subsequent to training regressed on age (one year prior to training), age-squared[6], a gender dummy variable, prior wages (one year prior to training), a dummy variable for goods producing industry location (one year prior to training), and a dummy variable for program participation.  The additional variable being controlled for, that is not accounted for in the quasi-experimental design presented earlier, is goods producing industry location.  The comparison group for the regression equation is services producing industry location.  Theoretically, it is likely that individuals employed in a goods producing industry prior to training have higher wages subsequent to training relative to those employed in a services producing industry.  The sample utilized for these regressions is the same as that used in the quasi-experimental designs presented earlier, with the additional criteria that those included in the model have at least two quarters worth of wages in the year after training being evaluated.  This was done to create a more reliable estimate of average quarterly wages for individuals in the regression equations.

As can be seen in Table 1, gender (being male), is positively and significantly related to wages subsequent to training.  In other words, males have significantly higher wages after training.  Age has a significant curvilinear effect as noted by the significant age-squared term — indicating that wages peak some time in the middle of the age distribution and then decline afterwards. Goods producing industry location is also positively and significantly related to subsequent wages.  In other words, those whose are located in a goods producing industry prior to training have significantly higher wages after training than those who are located in a services producing industry.   Prior wages are also positively and significantly related to subsequent wages — indicating that those with higher wages before training have higher wages after training. 

Of central theoretical importance, the participant dummy variable is significant for WDTF participants but not for WIA participants.  The OLS models presented here indicate that WIA participation is not significantly related to subsequent wages when controlling for age, gender, industry location, and prior wages.  Participation in WIA training, based on these OLS results, does not appear to be related to wage outcomes after training.  For WDTF participants the dummy variable remains a significant predictor of subsequent wages.  Thus, we cannot, as of yet, rule out the possibility that participation in the WDTF training did in fact increase wages of WDTF participants above and beyond those of those who did not participate in the training when controlling for age, gender, industry location, and prior wages.  In other words, it appears that we have found a program effect for WDTF participants. 

To illustrate these differences we solved the regression equations and present predicted average quarterly wage outcomes for WDTF and WIA participants in Figures 5 and 6.  The equations were solved for a 24 year old, in a services producing industry, with average wages one year prior to training (i.e., for their specific group).  Figure 5 presents results for males and Figure 6 presents results for females.  As can be seen in the Figure 5, male WDTF participants have significantly higher wages for both the FY99 and FY00 cohorts than males in the associated control group.  Male WIA participants for both the FY99 and FY00 cohorts do not have significantly higher wages.  However, the graph does indicate that the WIA participants change from having lower average quarterly wages than the control group in FY99 to slightly higher average quarterly wages for the FY00 cohort.  The same general patterns hold true for the female groups shown in Figure 6.  The primary difference between the two graphs is that females have lower average quarterly wages than males.     

Conclusions and Directions for Future Research

Although the results presented here are supportive of the argument that WDTF participants experience wage progression relative to a matched control group, we can only speculate on the source of the difference.  The addition of industry location to the OLS models shows that this factor does not “explain away” the significant relationship between program participation and subsequent wages for those involved in the WDTF.  Some theoretical possibilities that are not controlled for in our quasi-experimental or OLS designs include firm characteristics.  Potentially, WDTF participants are employed in aggressive firms that seek to improve the wage level for participants.  This is likely given that the WDTF application process is competitive.  An additional step would be to conduct further OLS analyses that measure and test important firm characteristics on wage progression.  More variables can be added to the multi-variate models without the difficulty associated with adding variables to the quasi-experimental design.

Beyond these steps, additional research is needed to determine long-term training effects.  In particular, we would like to determine whether the WDTF group continues to experience higher wages and whether the WIA group truly converges with the WIA control group.  There is some indication of this as noted by the differences lag times in the study results (i.e., one or two years after training completion).  Even though unanswered questions remain, this paper demonstrates the advantages of quasi-experimental and multi-variate design for exploring the effects of employment training programs.  Without such knowledge we have no comparative or contextual data within which to place the wage experience of training participants.

Wage Records Research Index  |  Labor Market Information