Appropriate staffing, funding, integrity vital to effective program evaluation

In an ideal world, recreation programs would be continually and thoroughly evaluated. Yet, without a systematic plan for evaluation, you and your organization will be left with only the best of intentions.

Many details must be considered when developing such a plan. Foremost are the elements of staffing, funding, and timing, as well as considerations for integrity and ethics. Overall, these are managed through a set of guidelines for establishing an evaluation system (see the checklist below).

• Link evaluation projects. Specific efforts to evaluate specific programs should be connected in an interwoven system that focuses on a joint and coordinated effort. This requires open communication among programmers and a transparent vision for organization-wide evaluation. For example, information collected in a summative evaluation for one program might be helpful when planning an evaluation project for another program. Or questionnaires prepared for one program’s evaluation might also be used to evaluate another program. Individual program evaluations should answer questions about program service success for the entire organization. Overall, are you achieving your mission?
• Gain administrative support. Your supervisors should be involved in planning specific evaluation projects as well as the entire evaluation system. Also, an advisory committee may be useful in helping with individual evaluation projects and overseeing the overall evaluation system for the organization. Communication with administrators or an advisory group can help you develop insights and gain commitment to the study. For example, organization directors may be crucial for getting the cooperation of other program staff for an evaluation effort. Involving them will also mean a greater likelihood that the results from the evaluation will be used to improve programs and upgrade program policies.
• Designate staff. Your organization should designate a person who is ultimately responsible for program evaluation. This is essential for any coordinated and systematic evaluation effort. Sometimes the programmers themselves may fill this role, or when more sensitivity is needed, outside consultants or research specialists may be called in. As well, sometimes a staff member is appointed to oversee the organization’s entire evaluation system, or an individual is designated for a specific program evaluation project. 

Ideally, persons responsible for evaluation should be well versed in both qualitative and quantitative inquiry methods, should be able to analyze and interpret data, and should have excellent written and verbal communication skills (Russell, 2005). They should also be able to work objectively, not trying to prove a particular finding or distort or promote a specific conclusion. As well, they should be thorough, meticulous, and able to design and carry out evaluation studies using precise methods in order for the results to have any real value to the organization.

• Involve program participants. There are many reasons for involving program participants in evaluation. Involving them early and often can help them gain understanding of what the evaluation is all about, ultimately increasing their support for the study and the program. Program participants can learn why the evaluation is being conducted and how it will proceed. This may help them feel less threatened. It will also help them understand what the program is trying to accomplish. Best of all, including participants in program evaluation can vastly improve the evaluator’s understanding of the worth of the program.
• Define roles and clarify authority structures. At the outset, those involved in an evaluation project-administrators, staff, and participants-should be very clear with each other about what is expected from them. The scope and limits of their roles need to be declared, ideally in written form. This not only helps ensure completeness-the elimination of gaps in the evaluation study-but also helps solve disagreements if they arise.

• Provide adequate funding. Costs of evaluation should be included in organizational operations budgets. Depending on their scope and method, evaluations can potentially be expensive. Personnel costs, in particular, must be considered in an evaluation budget because they are often the major expense category. Also, some supplies can be purchased inexpensively or are free, but others will require specifically earmarked funds. For example, some equipment, such as computers, may already be available, but specialized software, such as statistical and graphing programs, may need to be purchased. To augment organization-provided general operations budgets, grants can be a source of evaluation funding. In fact, most grants that fund programs require an evaluation component in the grant proposal. Regional foundations and state, provincial, and federal government agencies will sometimes fund entire evaluation studies. Evaluation expenses that should be included in an organization’s budget or grant proposal include the following:

• Sample selection costs
• Travel expenses for interviewer(s)
• Materials costs for pretesting the evaluation procedures
• Supervisory costs for hiring an outside consultant and overseeing his or her work
• Staff training costs
• Labor and material costs for data entry
• Analyst costs for preparing tabulations and special data analyses
• Labor and materials costs for report preparation
• Telephone charges, postage, reproduction, and printing costs

• Ensure adequate time. Evaluation takes time. Some projects require several months to complete at a certain time of year, while others may take only a day and can be done during any season. For example, if you are interested in assessing downhill skiers’ satisfaction with instructional programs at the ski resort, then administering a questionnaire to participants may take only a couple of days, but it will need to be done during the winter.
  As well, since evaluators and decision makers have different time schedule demands, you should establish an agreed-upon time frame for the completion of an evaluation project (Riddick & Russell, 1999). Estimate how much time is needed for recruitment of study participants, data collection, data analysis, preparation of the final report, and so on. Planning for evaluation therefore requires development of a timetable, or time budget (Riddick & Russell, 1999). Both calendar and clock time need to be budgeted. A mistake many evaluators make is underestimating the time needed, resulting in late delivery of the report or missing the deadline altogether. We recommend that the time budget include a 20 percent "extra time" factor to compensate for this.
• Provide useful feedback. Programmers are most appreciated by supervisors, advisory and policy groups, participants, and other program staff when the information gathered in the evaluation is perceived to be valuable. For example, suppose you wish to determine guest satisfaction with state park cabins. Measuring overall satisfaction is important, but even more useful information is possible by measuring specific satisfaction with such details as cabin design, cleanliness, linen service, the registration system, and proximity to park activities. These more targeted results increase the evaluation’s value to a wider array of staff.
• Prepare an evaluation project proposal. An evaluation plan benefits from documented decisions about the purpose of a project and how it will be carried out. A formal proposal-a written description of the intentions of the evaluation-is useful for communicating your plans to others in the organization. In addition, the process of writing a proposal helps you think through the research steps you’ll follow, helping you be more efficient in carrying out the study. It is much easier to find flaws in the evaluation project and correct them before beginning to collect information than it is to discover and correct problems after you have started.
• Keep evaluation projects going. Successful evaluation projects start out well organized and stay on track through the finish. Usually the person who develops the project proposal is the one who has the responsibility of seeing it through to completion. This might mean taking care of such logistics as recruiting respondents to questionnaires, printing and distributing questionnaires, monitoring the evaluation budget, and seeing that the final report is timely.

• Conduct evaluations ethically and with integrity. Integrity and ethics are essential for conducting a valid evaluation. Integrity refers to the validity of the information that is collected to determine the value of a program. Ethics is concerned with the treatment of the participants in the evaluation process.

First, how can you maintain evaluation integrity? Those who use the evaluation results want the information to be accurate; they apply what Weiss and Bucuvalas (1980) call truth tests in deciding how seriously to pay attention to an evaluation outcome-or its validity. Typically, a program evaluation is valid if it is "true, credible, and right" (House, 1980, p. 250). This means the nature of the evaluation process, the study design, the data gathering, and the way results are reported are honest. Both the evaluation (action) and the evaluator (person) must be perceived as trustworthy for the evaluation to have high validity (Russell, 2005). Only an honestly conducted evaluation can provide useful information and help an organization achieve its goals. The box on this page provides two illustrations of the negative effect an evaluation that lacks integrity can have.

Second, what are sound ethical practices in evaluation? Conducting ethical evaluations requires doing everything possible to treat the people, programs, and organizations studied graciously and honorably, including providing an environment that is safe and free of discrimination. One way to accomplish this is through informed consent-communication that ensures a potential evaluation respondent has sufficient information to decide whether to participate in the study. Typically, the respondent reads and signs a statement agreeing to participate in the study. The statement should describe the essential details of the study, including an explanation of the procedures, the potential benefits and risks of participation, the participant’s rights to confidentiality, and the choice to participate. When children are involved in an evaluation, parents or guardians should sign an informed consent. Government-sponsored evaluations, as well as most professional associations, require informed consent.

Another factor that helps ensure an ethical evaluation is an approving authority. An approving authority, such as a special committee within or outside the organization, can review evaluation project proposals for fair treatment of the people and programs studied as well as for honorably collected information. For example, universities all have institutional review committees that must give approval for studies that use human subjects. Elementary and secondary school systems have committees to review studies that outside groups, such as park and recreation agencies, want to conduct within the schools. Similarly, hospitals have review committees that must give approval for studies done with patients.

This is an excerpt from Leisure Program Planning and Delivery.