Impact Evaluation of a Wyoming Employment Assistance Program
In January 2016, the Research & Planning section of the Wyoming Department of Workforce Services published the results of an impact evaluation study of the Wyoming Workforce Development Training Fund (WDTF). According to that research, the mean quarterly wage of WDTF participants from second quarter 2007 was significantly higher statistically than the mean quarterly wage of non-WDTF participants in a comparison group (Manning, 2016). This article presents a similar evaluation of Wyoming’s JobAssist program conducted in 2013.
This evaluation reports on the effectiveness of the recently discontinued JobAssist Program on increasing wages and decreasing the use of government assistance usage. Over the past decade, the federal government has issued several memos regarding the need to evaluate government funded programs for effectiveness and the necessity of using scientific methods to make conclusions about such programs.
The key to this evaluation was employing matching techniques to form a comparison group of individuals with similar characteristics but who did not participate in the program. The analysis focused on the difference in quarterly wages, number of employers, and unemployment claims pre- and post-enrollment for both groups. Further, analyses were also completed for the use of government assistance after enrollment in the program.
The findings indicate that the JobAssist participants fared no better on either wages or the use of government assistance compared to the comparison group. For example, the second year after enrollment in the program, JobAssist participants and comparison group individuals did not have significantly different quarterly wages or reduced government assistance.
It is important to note that several limitations to the data collection techniques by Department of Workforce Services (DWS) coordinators and the private contractor involved in the JobAssist program have impacted the effectiveness of rigorous program evaluation. Several recommendations are given to help facilitate effective collection of data and the reasons why systematic data collection is necessary. These recommendations are available in the full research publication, available at http://doe.state.wy.us/LMI/w_r_research/JobAssist_evaluation.pdf.
The results of this evaluation provide useful information and insight into the effectiveness of the JobAssist program. However, it should be noted that an overall conclusion, in terms of policy implementation, may not be completely justified based on the results. This is discussed in the conclusion section of this article.
Introduction and Background
In an effort to reduce the cost of public assistance programs and increase resident self-sufficiency, various Wyoming state agencies – including the departments of Health, Corrections, Family Services, Workforces Services, State Insurance Pool, and Administration and Information – began contributing data in 2003 to a data warehouse called the Wyoming Health Information Network (WHIN). The data provided information on certain populations that received services from one or more of the state agencies. Using these data to identify individuals using multiple public assistance programs, the private contractor began the Healthy Families Succeed (HFS) program in 2005, starting with the HealthAssist component. In conjunction with the Department of Workforce Services, the private contractor added the JobAssist component in 2008. In order to be eligible for JobAssist, individuals needed to be able to work, be 18 to 64 years old, and be utilizing at least two state services. Healthy Families Succeed began in Laramie County and with funds from the American Recovery & Reinvestment Act (ARRA); the program expanded to cover Natrona, Sheridan, Park, Sweetwater, and Teton counties. The goal of the HFS program was to link personal and family health, job skills, and employment in order to move toward self-sufficiency.
The process of enrolling in JobAssist began when the contractor selected potential participants from the WHIN data warehouse who met eligibility criteria using integrated risk analytics. Once a participant was selected by the contractor, his or her contact information was given to DWS coordinators, who contacted the individual to obtain written consent to participate in the program. If consent was given, the participant’s confidential information was sent to the contractor. Because consent was required, the program should be considered voluntary.
In the spring of 2012, DWS program management approached the Research & Planning (R&P) section to evaluate the effectiveness of the JobAssist program using administrative databases currently in place. R&P currently has access to administrative databases from multiple state agencies, including information on public assistance programs.
Results
|
One of the main objectives of this evaluation was to discover whether the JobAssist program increased employment and wages for participant. The number of individuals employed during a particular quarter has a significant effect on wages earned. As seen in Figure 1, the percentage of JobAssist and comparison group participants peaked three quarters prior to enrollment and then began to decline (Ashenfelter dip) and remained at about 60% for the fourth through eighth quarters after enrollment.
|
To test the difference in JobAssist participants’ wages pre- (-8 to -5 quarters) and post- (5 to 8 quarters) enrollment, a paired-sampled t-test was performed with mean quarterly pre- and post-wages. The t-test revealed no significant differences between pre- (M = $2,264, SD = $2,095) and post- (M = $2,088, SD = $2,384) enrollment earnings, t(230) = .958 (two-tailed), p = .40. A statistically significant difference in the means (averages) of the two time periods would have resulted in a p-value of less than .05. JobAssist participants saw a decrease in wages of $176, but this decrease was not statistically significant. Figure 2 presents the average quarterly earnings for JobAssist and comparison group participants by quarter after propensity score matching. After the dip at enrollment, wages increased over the first six quarters post-enrollment then leveled off for the JobAssist participants, while the comparison group’s wages showed an increasing trend beginning with the fifth quarter after enrollment.
|
Comparing the mean wages pre- and post-enrollment between the JobAssist group and the comparison group allowed for a determination of the effectiveness of the JobAssist program on participant wages. If the JobAssist program was effective, R&P expected to find that their post-program wages were significantly higher statistically than those in the comparison group. A one-way Analysis of Variance (ANOVA) was conducted with mean quarterly wages post-enrollment (5 to 8) as the dependent variable with group membership (JobAssist vs. Comparison) as the independent variable. An ANOVA is a statistical technique used to test for differences in means between groups. The ANOVA revealed that JobAssist participants’ wages (M = $2,088, SD = $2,384) were not significantly different from the comparison group wages (M = $1,819, SD = $2,150), F (1, 460) = 1.62, p = .204.
|
To understand the trends of government assistance over the course of the two years after enrollment, individuals were selected based on whether they had any government assistance during a given quarter. For example, an individual was included if he had some sort of government assistance (greater than zero) and excluded if the government assistance total was zero. Figure 3 presents the percentage of individuals who used government assistance during a given quarter. Over the course of five quarters after enrollment, both groups showed a steady increase in government assistance usage. In quarter eight, both groups showed a decline in usage. The percentage of individuals in the comparison group who used government assistance was higher compared to the JobAssist participants for all but the first quarter. Figure 4 shows the average amount of government assistance usage during a given quarter. Comparing the two figures, it can be concluded that even though the number of comparison group participants using government assistance increased (see Figure 3), the actual dollar amount did not differ significantly compared to JobAssist participants. It should be noted that data from the eight quarters prior to enrollment were not available, so pre-post comparisons could not be conducted.
Conclusion
This evaluation examined the effectiveness of the JobAssist program in regards to wages and the use of government assistance for individuals who participated in the program. In this section, the implication of the findings and recommendations for future JobAssist evaluations are discussed. For both females and males, the average wage increase from eight quarters before (pre-enrollment earnings) and eight quarters after (post-enrollment earnings) were not significantly different statistically. Further, when the JobAssist participants were matched to the comparison group individuals, there was no significant difference in post-enrollment wages between the two groups. These findings show that after two years of enrolling in the JobAssist program, participants showed no significant gains in wages from two years prior to enrollment. Further, JobAssist participants fared no better compared to individuals who were also economically disadvantaged who did not participate in the program (comparison group).
JobAssist participants were also less likely to be employed both years after enrollment compared to the comparison group. However, a small percentage did transition from not being employed the first year after enrollment to being employed the second year after enrollment. The number of employers and the number employed in each group were similar in the post-enrollment quarters (4 to 8). JobAssist participants also had higher unemployment insurance claims filed starting the second year after enrollment compared to the comparison group. This may be explained by the high level of turnover seen at the year anniversary of enrollment when many JobAssist participants exited the program. The number of individuals who were actually paid UI benefits was small for both groups. This decrease may be due to means testing for other programs.
In terms of government assistance usage, there was no significant difference statistically between the JobAssist participants and the comparison group in the post-enrollment period. The number of JobAssist participants who used government assistance was lower than the comparison group; however, the actual dollar usage of government assistance was similar across groups. This finding shows that for those who participated in the JobAssist program, there was no difference in actual dollar usage compared to the comparison group.
The goal of the JobAssist program was to increase participant wages and employment outcomes while decreasing participants’ reliance on government assistance programs. The motivation to conduct this evaluation was to answer the question of whether the JobAssist program did in fact accomplish these goals. Overall, there was no indication that the JobAssist program participants fared any better compared to similar individuals who did not participate in the program. However, several cautions should be recognized. First, no analyses of the various services provided by the JobAssist program were conducted. The data were incomplete for many participants regarding which services they received, the duration, and what the exact purpose was during the service. For example, if a JobAssist coordinator documented one hour of a home/office visit, there was no indication of what actually occurred during the visit. Second, the amount of time a JobAssist coordinator spent referring a participant to a specific service was also largely unknown. Of the 3,702 hours of time JobAssist coordinators documented, 2,071 were unknown. The amount of time that is unknown makes analysis of the time spent coordinating certain services almost impossible.
The results of this evaluation provide useful information and insight into the effectiveness of the JobAssist program. However, it should be noted that an overall conclusion, in terms of policy implementation, may not be completely justified based on the results. The results suggest that those services provided by the JobAssist program did not have the intended effects of reducing government assistance usage and increasing wages compared to those who applied for UI benefits. Before any useful conclusions can be made regarding the effectiveness of the services provided by JobAssist, an in-depth evaluation of the services provided to each client will be needed. Some of the services that JobAssist provided may have had an effect on wages and government assistance outcomes, but since these were not well documented, no conclusions should be drawn on service effectiveness at this time.
Reference
Manning, P. (2016). Higher wages and more work: Impact evaluation of a state-funded incumbent worker training program. Wyoming Labor Force Trends, 53(1). Retrieved June 21, 2016, from http://doe.state.wy.us/LMI/trends/0116/a1.htm