ISSUES AND PRACTICES

Chapter 7:
Evaluation, Funding, and Technical Assistance

Key Points

Monitoring program activities is essential for convincing administrators to support the program and for learning how it may need to be improved.

Conducting an impact evaluation is also important for learning whether the program is achieving its goals and for convincing administrators to continue to fund--or increase funding for--the program.

  • An evaluation by the Connecticut Department of Correction involves surveying 4,100 employees to compare before-and-after self-reported stress levels, absenteeism, drinking behavior, intentions of resigning, and other attitudes and behaviors.
  • Using department records, the Collier County (Florida) Sheriff's Office compares before-and-after rates of absenteeism, sick time, job-related accidents, and disciplinary reports among officers, and compares the findings with those for officers in another sheriff's department that has no stress program.

Logic suggests that stress programs save departments of corrections and sheriff's departments money by reducing sick time and stress-related disability retirements. A few empirical studies add concrete evidence to this hypothesis.

Most stress programs share certain cost elements, such as personnel and trainer fees.

  • However, program costs among programs vary considerably, depending on how much programs rely on volunteers and inkind services.
  • Some programs have obtained outside funding from State and Federal agencies and from officer unions.

A number of sources of help are available for setting up or improving a stress program, including program and government materials, articles, public and private agencies and associations, and stress program coordinators and correctional officer stress experts.

This chapter examines how to monitor and evaluate stress programs, how to estimate program costs, and where to get help in planning or improving a program.

Monitoring

Administrators of any program are often reluctant to monitor or evaluate their efforts because of lack of time, lack of expertise, or concerns about confidentiality. However, as two very different correctional staff explain, tracking program performance is essential:

"The important thing [with providing stress services] is: How successful was the program?"
--Maria Houser, deputy commissioner for administration, Connecticut Department of Correction

"Jim Hollencamp [the Massachusetts DOC's Stress Unit coordinator] makes the [stress] program work because he keeps all the statistics which validate the program. Without him, the DOC doesn't know it's getting its money's worth."
--Dick Gould, member of the Stress Unit

Programs can collect and report several types of information related to professional counseling and peer support without breaching confidentiality, including--

  • Number of clients or contacts.
  • Nature of the clients' problems (e.g., domestic violence, alcohol abuse, marital difficulties, suicidal tendencies).
  • Whether assistance was provided on or off duty.
  • Type of client (correctional officer, civilian employee, family member, child).
  • Number and types of referrals.

Unit staff support team members in Texas complete an incident log (see appendix F) on which they record--

  • Date, shift, number, and gender of employees affected by a critical incident.
  • The type of critical incident.
  • Any medical attention that was required.
  • The time involved in providing support.
  • Whether the regional team was notified.

Unit team members turn in their logs to their unit team leaders. Unit team leaders submit the information to the regional team coordinator. Exhibit 3-5 in the case study describing the Texas Post Trauma Program in chapter 3 shows the annual report Elaine Smith, the program coordinator, submitted to the department for 1999 that incorporates these data.

Evaluation

Programs can implement two types of evaluations: a process (or formative) evaluation and an impact (or outcome) evaluation.

Process evaluation

When information gathered by program monitoring is used to judge the quality, adequacy, or appropriateness of program operations, the assessment is referred to as a process evaluation. The focus of a process assessment is the implementation of the program, not program impact. When Elaine Smith scans the Texas DOC's Emergency Action Center daily printout of serious incidents to make sure wardens activate unit teams whenever one of eight mandated types of incidents occurs, she conducts a process evaluation. Smith also sends questionnaires each year to the regional teams and facility administrators asking for comments on the program's effectiveness. The regional teams, in turn, obtain comments from officers who have participated in debriefings.

Another process evaluation tool is to determine whether supervisory training results in increased referrals to the stress program or employee assistance program (EAP). Exhibit 7-1 displays data provided by the New York State Department of Correctional Services' EAP program indicating the dates on which it provided training to frontline supervisors (sergeants) in eight prisons along with the number of officers and supervisors referred to the EAP during the 3 months before and 3 months after the training (excluding the month in which the training was conducted). By comparing the two sets of data, it is possible to determine whether referrals increased after--and presumably because of--the training. As shown, there was a slight overall increase from 20 referrals before the training to 25 after the training. Anecdotal information--often a valid source of data for a process evaluation--also suggests that referrals may have increased after the training. According to the State EAP assistant program manager, "Anecdotally, it appears that after supervisor EAP training, referrals increase dramatically, but then they fall back." According to an EAP coordinator who has conducted the training many times, "I've seen a big increase in referrals [since the training was first offered]. I've been [EAP] coordinator for several years and never had any before, and now I've had six in the last year."

1

Impact evaluation

An impact evaluation is designed to show the effects a program has had on its clients and the department as a whole. Two examples of impact evaluations follow.

Connecticut Department of Correction evaluation

In 1999, the Connecticut Department of Correction planned a new stress program funded by the National Institute of Justice to offer officers seminars addressing five topics, such as parenting skills, financial management, and conflict resolution. Two evaluators from the University of Connecticut designed a questionnaire for measuring what the seminars achieve (see appendix G).

The evaluators will survey 4,100 employees in five facilities before the seminars begin. A followup survey will be given to the 1,000 officers who attend the seminars and to 300 randomly selected officers from among the 3,100 officers who fill out the initial questionnaire but do not attend the seminars. The surveys will compare before-and-after self-reported stress levels, absenteeism, drinking behavior, intentions of resigning, and other attitudes and behaviors.

The evaluators will ask the officers to record a number on their answer sheet on both the pre- and post-questionnaire--the last four digits of their Social Security number, the last four digits of their home telephone number, and their birthday. These identifiers will not only enable the evaluators to compare individual officers in terms of initial and followup stress but also to compare stress levels among facilities.

The evaluators will not make adjustments for officers who answer the initial questionnaire but not the follow-up questionnaire--for example, because they were transferred to other prisons, retired, or were sick. These unavailable officers may be personnel who benefited from the seminars more or less than other officers (for example, if they retired because of stress). As a result, the survey findings may not be entirely valid.

Collier County (Florida) Sheriff's Office evaluation

The Collier County Sheriff's Office used a Corrections and Law Enforcement Family Support (CLEFS) grant from NIJ to develop and evaluate a stress prevention and reduction program for its correctional and law enforcement officers. The evaluators, two researchers from Florida Gulf Coast University, use agency records to examine changes before and after the program in terms of

  • absenteeism,
  • sick time,
  • tardiness,
  • job-related accidents,
  • civilian complaints (primarily for road deputies),
  • disciplinary reports, and
  • positive supervisory reports.

The evaluators have already used three other before-and-after measures of program effectiveness:

  • Stress measured by the Occupational Stress Inventory (OSI), Index of Family Relations (IFR), and Index of Clinical Stress (ICS).
  • Alcohol- and drug-related problems as measured by the Michigan Alcoholism Screening Test (MAST).
  • Overall personal health and job and family satisfaction as measured by a general health survey the evaluators developed.

A psychologist hired under the grant administered the tests to 33 law enforcement and 44 correctional officer recruits during their preacademy training and again at the end of the recruits' 16-week field training experience. The same battery of tests was also administered to 262 law enforcement officers and 97 correctional officers who received inservice training in stress reduction and stress resolution training.

The psychologist is also collecting data from a voluntary group of correctional officers and their spouses who agreed to be tracked annually over a 3-year period. Each pair has been administered the Hilson Spouse/Mate Inventory and the Hilson Relationships Inventory for Public Safety Personnel.

The evaluators are using two comparison groups:

(1) Treatment groups of law enforcement and correctional officers in preacademies and law enforcement and correctional officers in a 40-hour inservice training program.

(2) A group of 63 officers from the Charlotte County Sheriff's office, a comparable county in southwest Florida 50 miles from Collier County, who have not received stress education.

More modest evaluations

Program coordinators can also implement less comprehensive, but still useful, evaluations.

For 3 years, the South Carolina DOC tracked 28 employees who were held hostage or injured during a riot in 1986 and compared their retirement and sick time usage with other employees who were not involved in the riot. There was less turnover and sick time among the riot-involved employees-all of whom had received individual or group debriefings from Post Trauma Resources-than among the uninvolved employees.

Edward Stelle, coordinator of the Multnomah County Sheriff's Department Peer Support Program, compared the number of correctional officers on sick leave in July 1997 with those on sick leave in July 1998, a year after the program had substantially increased its effort to provide peer support to correctional officers. Stelle found there were 40 officers on sick leave in 1997, 24 of whom had problems related to stress. In 1998, there were 20 officers on sick leave, only 2 of whom had stress-related problems.

Managing Program Costs and Securing Funding

Most DOCs and sheriff's departments have incomplete information about how much their stress programs cost because of difficulty estimating expenses: separate budget line items for the operation of in-house programs generally do not exist; staff, office space, and equipment may be shared with other department units; and in-kind contributions of space, supplies, and personnel are often used.

Independent mental health professionals who consult to correctional agencies may also have difficulty estimating their counseling costs because officers typically constitute only a part of their practice, reimbursement varies according to a given officer's insurance coverage, and staff may provide pro bono services to the departments they serve by way of free counseling, training, or crisis intervention.

Identifying cost elements

Many programs, however, share certain cost elements:

  • personnel;
  • consultants (e.g., trainers, clinicians);
  • rent and utilities;
  • vehicles (or mileage reimbursement if personal cars are used) for responding to emergencies and meeting clients away from the program's offices;
  • equipment and supplies (e.g., beepers, business cards, program brochures, training materials, coffee);
  • telephone, fax, and postage; and
  • travel (e.g., to conferences and seminars).

Newly begun programs will incur some one-time startup costs that established programs typically no longer have to pay for, such as the purchase of office furniture, computers and duplicating machines, and initial staff training.

Actual program budgets

As shown in exhibit 7-2 and explained below, there is considerable variation in program budgets.

  • In 1985, the Rhode Island DOC signed a 1-year $6,000 contract with Family Service Society as a test. Under the most recent 3-year contract (July 1, 1997, through June 30, 2000), the State reimbursed Family Service Society at the rate of $55 per counseling and training hour not to exceed $16,500 per fiscal year.
  • The San Bernardino Counseling Team's contracts with correctional agencies provide for charging $60 to $125 for each hour of critical incident response time and $175 for each officer who attends the program's peer supporter training course. The contracts provide for free short-term counseling for employees on an as-needed basis with caps that depend on the size of the department.
  • With one exception, the Massachusetts DOC does not have to pay additional salaries for its stress unit because the five full-time peer supporters are correctional officers who were transferred into the unit and not replaced by new hires in the institutions from which they came. (With more than 4,000 correctional officers, the department has no difficulty transferring existing officers to take the place of the transferred peer supporters. Smaller departments that need to hire new staff to replace transferred officers or to pay current officers overtime to cover for them will, of course, incur a considerable cost in adopting this type of peer support program.)1 Stress unit members receive no overtime, and they are all at maximum salary of $44,000. Members receive $90 a week for carrying a beeper. Furthermore, the department called a former correctional officer out of retirement to coordinate the unit, paying him $12,240 for a 20-hour week with no fringe benefits. Assuming four of the five unit members carry a beeper in any 1 week for 48 weeks a year, the total expenses for the stress unit come to $29,520.
  • The Multnomah County Sheriff supports the Peer Support Program with $50,000 for outside psychologists to provide free counseling to officers (capped at four visits per officer), $10,000 for training, $20,280 for one-third of the coordinator's 30-hour week (he receives no fringe benefits), and about $7,000 for a part-time secretary. The total program budget comes to about $87,280. The money comes out of the sheriff's administrative budget.
  • With the exception of the coordinator, who spends 80 percent of her time on the Post Trauma Program, and a secretary, who spends half time on the program, Texas' Post Trauma Program consists entirely of volunteers. In 1999, the director's and secretary's salaries pro rated for the time they spent on the program was $36,966. When regional and unit team members travel for training, individual wardens arrange transportation in State vehicles and meals in the prison nearest the training site. When overnights are needed, regional Institutional Division coordinators pay for hotels and food from their security operations budgets.
  • The New York State Department of Correctional Services' all-volunteer program has no expenses. The department has twice secured Federal grants for training.
  • The State workers' compensation carrier reimburses Post Trauma Resources $70 per hour for each approved counseling hour. The carrier reimburses PTR $127 per staff hour for critical incident debriefings. The DOC pays the bill if the carrier rejects PTR's claim. The DOC reimburses PTR separately for execution team debriefings, awarding the contract after a competitive procurement.

2

The program costs identified above do not reflect the cost of paying officers overtime to cover for peer supporters during training. Some departments pay peers overtime if they are called in from home or remain on the job after their shift ends to provide support after a critical incident.

Funding sources

Many programs are funded entirely by the correctional agency or agencies they serve. However, some programs have secured supplemental funding from other sources. The three unions representing correctional officers in Connecticut each offered to contribute $3,000 to the department's new stress program in return for extending the stress survey of 4,100 officers from four to five institutions (see the section "Collaborating With the Union" in chapter 5).

Several DOCs have secured funds from the National Institute of Corrections (NIC) for training peer supporters and other staff. For example, the Texas Institutional Division secured a $6,000 NIC training grant, and the New York State Department of Correctional Services twice arranged for NIC to pay up to $10,000 to experts to provide critical incident debriefing training for several of its employees. Finally, sheriffs in California can obtain reimbursement from the California Standards and Training for Corrections Commission for The Counseling Team's fee to train their peer supporters.

Sources of Help in Setting Up or Improving a Stress Program

There are several resources for learning more about setting up, expanding, or improving a stress program. The list of resources below is based on a limited search and is therefore not comprehensive.

Materials, agencies, and associations

Stress programs have prepared a number of materials, some of which have been included in the appendixes to this report, that can be useful in planning or improving a stress program for correctional officers. Other program materials, along with additional materials of assistance in developing a stress program, are listed in appendix H.

Publications

The notes to chapter 2, "The Extent and Sources of Correctional Officer Stress," reference literature that discusses correctional officer stress. Woodruff, L., "Occupational Stress for Correctional Personnel: Part One," American Jails 7 (4) 1993: 15-20; "Part Two," 7 (5): 71-76, presents recent research findings on the effects of stress on correctional officers.

Individuals with experience in correctional officer stress programming

The individuals identified in the chart below are available to provide technical assistance related to stress programming by telephone. In addition, chapter 3, "Program Case Studies," provides the names of program coordinators after each case study who are also available to provide telephone consultation.
3

4

Focus Groups Can Help Identify Correctional Officer Stress

Departments of corrections and sheriff's departments can tailor their inservice stress training to their employees' particular needs if they first determine what the most serious sources of stress are for their employees--institution by institution.

As part of its new stress program, the Connecticut Department of Correction has asked the researchers who will evaluate the program's effectiveness to run 10 focus groups to identify what officers consider to be their primary sources of stress. John Rogers, one of the researchers, suggests that the focus groups will also serve as a marketing tool for the program: "As members share information about the program with their coworkers, this will increase officers' buying into the survey that will follow." The researchers developed a sampling plan for deciding which officers to include in the focus groups based on gender, race, shift, facility, rank (officers and sergeants), and length of service. The DOC agreed to arrange for the focus group members to participate on released time. The unions will hand out and collect the questionnaires.

---------------------------

Do Stress Programs Save DOCs and Sheriffs Money?

Common sense suggests that stress programs save DOCs and sheriff's departments money by reducing the need for overtime to cover for officers who take sick time, disability leave, or early retirement because of stress. While no one has demonstrated empirically that a stress program can save a DOC or sheriff's department money, a study conducted by the San Bernardino Sheriff's department suggests it can.

In the early 1980s, Deputy Chief James Nunn of the San Bernardino Sheriff's Department was in charge of internal affairs and a member of the county retirement board. After every critical incident, Nunn would see officers' names on the retirement list, with each premature retirement costing $750,000 to $1 million in unfunded liability to the retirement fund.* As a result, the department contracted for stress services with The Counseling Team in San Bernardino.

Nunn looked at the number of stress-related retirements that officers and deputies took after critical incidents for a 10-year period before the contract and found they cost the county $20 million. For 6 years after the program had been in place, there were none. The sheriff's department estimated that it had saved $13 million by avoiding increases in its annual fee for membership on the retirement board.

* Most correctional officers in California work for the State and are part of the Public Employee Retirement System, a huge retirement fund. Employers and employees contribute money into the fund. When employees retire, the fund sends them a monthly check for the rest of their lives. However, the employers' contributions are calculated under the assumption that their employees will, on average, retire after 20 to 40 years of work. When an officer retires after, say, 8 years because of a work-related, stress-related disability, the extra money for the added 12 years of checks is not available in the fund-creating a so-called unfunded liability. The fund raises the employer's contribution to the fund for the next year to make up for this loss. Reducing early retirements precludes the need for the sheriff's department to pay more into the fund each time an officer retires prematurely.

---------------------------

Note

1. Of course, by not replacing officers transferred to the stress unit, it is possible that the stress levels of remaining personnel may increase as they struggle to take on the work of the departed officers.

Glossary

Critical incident--An event that affects performance, exceeds the individual's coping skills, and creates stress for even normal individuals.

Debriefing--A specially structured meeting, usually in a group setting, for employees who have been involved directly or indirectly in a critical incident. Sessions are led by individuals trained to conduct debriefings, with a licensed mental health worker present or acting as a coleader. Generally held within 72 hours after the incident, the 1- to 8-hour session allows employees to discuss their behavior during the incident and the thoughts and feelings the incident caused.

Defusing--A meeting held immediately following a critical incident with employees who have been affected directly or indirectly by the incident. A defusing provides information about possible stress reactions, emphasizes that most reactions are normal, identifies resources for further help, and assesses the need for debriefing.

Educational debriefing--Workshops in which participants are given information about the facts of the event and told psychological help is available if needed.

Employee assistance program (EAP)--A program that provides referrals for counseling or, less commonly, actual counseling for employees experiencing problems that may affect their ability to perform their jobs.

Peer supporters--Correctional employees, uniformed or civilian, who typically listen to coworkers' problems and decide whether to refer them for professional counseling. Because they have no training as clinicians and no protection against requests to disclose conversations with other officers, peer supporters should never engage in counseling.

Previous Page - Page 1 2 3 4 5 6 7 8 9 10 11 12 - Next Page
Page 12 of 12

Return to Top of Page