June 2009 // Volume 47 // Number 3 // Tools of the Trade // 3TOT4
Using Planning and Evaluation Tools to Target Extension Outputs & Outcomes: The New England Private Well Symposium Example
Increasingly, the success of Extension programming is evaluated based on achieved outcomes. Here, we report on the use of the ADDIE model as a tool to plan, implement, and evaluate a specific activity within the New England Private Well Initiative's regional efforts. Using this tool, we have successfully identified outcomes and objectives for the New England Private Well Water Symposium.
The past decade has witnessed a revolution in the expectations for Extension programming. The success of outputs, like workshops and factsheets are viewed through a lens that focuses on measureable objectives and achieved outcomes (Hoffman & Grabowski, 2004; NOAA, 2003). In this article we illustrate the use of an instructional systems design model, the ADDIE model (NOAA), to design and evaluate a specific output, the New England Private Well Water Symposium (Arnold, 2002; Peterson, 2003).
The goal of the symposium is to improve information sharing, networking, and collaboration among professionals working in the field of private well water protection by communicating current research, educational approaches, and materials, and providing opportunities for interaction. The work emerged from a CSREES competitive grant that funded the New England Regional Water Program <www.usawaterquality.org/nesci>. Extension-led interagency teams formed to address important regional water resource issues.
Based on input from Extension water quality staff and communications with the Environmental Protection Agency (EPA) Region 1, the New England Private Well Initiative (Initiative) formed in 2001. Extension has a history of private well programming (Lemley, 1993; Swistock, 2001; Clemens, 2007). Although 2.3 million New Englanders rely on private wells as their source of potable water, regular testing is not regulated (US EPA).
The ADDIE Model
We used the ADDIE model as a framework to develop and evaluate all the Initiative's efforts as well as a specific Initiative output - the New England Private Well Water Symposium. The ADDIE model's five-step approach includes Assessment, Design, Development, Implementation and Evaluation. With its clearly defined steps, ADDIE allows for effective implementation of activities that are learner-centered and outcome based (Peterson, 2003).
Each of the model's five steps contains sub-steps, summarized in Table 1.
Below, we discuss each of ADDIE's five components in reference to the symposium.
In 2003, we conducted a regional needs assessment meeting with Extension faculty and staff, the EPA, state drinking water agencies, and nonprofit organizations to address the following questions.
- What are the most serious risks to well water quality and human health?
- What support and educational programs are in place to assist private well owners to address these risks?
- What are the critical gaps in educational programming between what is already provided and what is needed?
Prior to this meeting, Extension representatives distributed a pre-planning questionnaire to stakeholders within their states. Eighty-nine questionnaires were returned and represented the responses of public health officials, educators, well owners, scientists/researchers, environmental regulators, analytical lab professionals, and well drillers. The responses served as a cornerstone for discussion during the assessment meeting.
Following the needs assessment, the initiative developed a logic model for regional coordination and collaboration efforts. The symposium was identified as one activity within this larger effort that would enable us to achieve several outcomes as listed in Table 2.
|Short-Term Outcomes||Mid-Term Outcomes||Long-Term Outcomes|
|Increased knowledge within
the region of research, education and Extension programs.
Increased ability of the regional team to provide integrated research, education and Extension programming.
Increased ability to strengthen state-based and regional program efforts as a result of coordination.
Professional sector groups will have an increased understanding of the programs and resources available to private well owners.
and joint programming among Initiative to coordinate and work
Established framework for regular communication and sharing of resources among Initiative members.
New partner agencies and organizations engaged in Initiative efforts.
|Increased integration of
research, education and Extension within Initiative.
Reduced health risk associated with private well water users.
High quality and sufficient quantity of groundwater resources for drinking water supplies.
The needs assessment established that key partners shared many goals but were not coordinating and communicating well, resulting in program gaps, redundancies, and missed opportunities for delivering relevant research and information to private well owners. An interagency planning committee, coordinated by Extension, was formed to develop the New England Private Well Water Symposium to address this identified gap. We selected private and public sector professionals as an audience for this effort. We believed that building capacity and collaboration within this group of professionals would eventually improve educational content and delivery to private well owners.
Via conference calls and e-mails, we first used the symposium logic model to define outcomes (Bennett, 1975; UWEC, 2002; NOAA). Following the ADDIE approach, the outcomes were then restated as SMART objectives (Specific, Measurable, Audience-focused, Realistic, Time-bound) (NOAA). SMART objectives provide a basis for conducting a meaningful evaluation and documenting impacts. For example, one mid-term outcome was the integration of symposium information by our audience. Several objectives related to this outcome, including the integration of knowledge gained from the event into participants' work efforts (Objective #4, Table 3).
We then developed a project timeline and evaluation plan, linking the outcomes with the project inputs and outputs. Only outputs that contribute to achieving an outcome were pursued (NOAA). Objectives were shared with all planning team members and repeatedly referenced in discussions.
|Inputs||Activities||Audience||SMART Objectives||Short-Term Outcomes||Mid-Term Outcomes||Long-Term Outcomes|
England water quality Extension staff
3.Equipment & supplies
-Work with conference organizer to distribute RFP to area chamber of commerce
-Solicit presentations and workshops that meet identified educational needs.
-Hotel site visit and selection
-Develop symposium website
-Develop and distribute save the date and call for abstracts cards
-Develop and continually update mail and email lists for marketing.
-Solicit symposium sponsors & vendors
-Develop final agenda
and/or well water related organizations.
1. 85% will agree that the symposium was a great platform to exchange ideas.
2. 70% will report a significant increase in knowledge.
3. 30% will contact one colleague/expert they met at the symposium.
4. 60% will integrate knowledge gained from the symposium into their policy, science or educational efforts.
5. 90% will express a positive opinion about the benefit of attending the event in the future.
greater exchange of ideas among scientists, regulators, technical
professionals, and educators about private well water concerns.
Increased knowledge of:
analysis and occurrence
High degree of satisfaction with Symposium as a regular event.
will apply something they learned into their own work.
Increased collaboration among symposium participants.
of health risks associated with ground water use to private well
High quality and sufficient quantity of groundwater resources for drinking water supplies.
We decided to pilot a one-day symposium. We would use participants' feedback to evaluate the structure, content, and satisfaction level and also help us improve a future event if results warranted. Based on our SMART objectives, we created an agenda that allowed participants to:
- Learn about current research relating to groundwater and well water quality and availability through invited talks by research scientists.
- Learn about techniques and approaches to effectively communicate this research to well owners, such as social marketing.
- Interact with each other.
- Take advantage of different educational formats (e.g. presentations, small and large group discussions).
The 1-day pilot was held in Portsmouth, New Hampshire, in November 2005 for 95 people. Questionnaire results indicated that respondents found the event useful and overwhelmingly desired a follow-up symposium. Responses also indicated that the event should be held biennially, that a longer format would be preferred, and that additional topics be included.
As a result, a second symposium was held in 2007 in Newport, Rhode Island. This expanded 2-day event attracted 123 participants. For the 2007 symposium, we broadened the range of topics by including a mix of keynote speakers and presenters who addressed the science, management, and legal issues surrounding groundwater wells. We selected presenters based on abstract submissions. Based on participants' comments from 2005, the second symposium included several structured opportunities for breakout sessions that enabled networking.
Participants completed a questionnaire at the end of the event to help us evaluate outcomes (Table 4). The questions reflected the SMART objectives. An end-of-session questionnaire is a useful way to obtain immediate feedback about the event and to assess progress towards achieving short-term outcomes (Arnold).
|Objectives||Percent of Respondents Agreeing or Strongly Agreeing|
|At least 85% of respondents will agree or agree strongly that the symposium provided an effective avenue for exchanging ideas about private well water issues.||91%||100%|
|At least 70% of respondents will report a significant increase in knowledge in at least one of the symposium topic areas.||95%||100%|
|At least 30% of respondents will report that they anticipate contacting at least one expert/colleague they identified through the event within the year.||84%||89%|
|At least 60% of the respondents will report the integration of some knowledge gained from the symposium into their educational efforts within 1 year.||98%||91%|
|At least 90% of the respondents will express a positive opinion about the utility of this event as a regular event in the future.||97%||98%|
|Note: Thirty-eight% of the evaluations were returned in 2005 and 46% in 2007.|
In 2008, 6 months after the 2007 event, we posted a Web-based questionnaire for participants who attended the 2005 and 2007 events. The purpose of conducting this inquiry was to begin to assess the impact of the symposium on achieving mid-term outcomes. We focused on behavior changes that may have resulted from the learning or contacts that occurred at the symposia.
We received 48 responses, a 37% return rate (Table 5). Virtually all respondents wanted to participate in any future symposium. Attendee responses included a number of our logic model outcomes, for example:
- Using information and approaches to inform the process of developing state policies for private well testing.
- Enhancing collaboration with partners.
- Developing new content and communicating water quality and safety techniques to private well owners.
- Increasing networking opportunities that are resulting in collaborative grant proposals and projects for both research and Extension projects.
|1. Please select the best
description of your position.
Private, non-profit organization
|2. After attending the symposium, I contacted a fellow attendee for information, programmatic resources, research results, or other materials pertaining to topics discussed at the symposium.||Yes: 50%|
|3. After attending the symposium, I provided information to another attendee on a topic discussed at the symposium.||Yes: 48%|
|4. I used what I learned at the symposium to improve my work.||Yes: 92%|
|5. After attending the symposium, I coordinated/collaborated with a fellow attendee to enhance an existing program or develop a new program/project focused on private well water protection, education or research.||Yes: 39%|
NOAA's ADDIE model has provided us with a systematic way of developing the symposia and documenting impacts. We have evaluated the program against the defined measurable objectives and have been able to point to the success of this effort. As a result, we have raised sponsorship funds to help support these efforts, offset costs to our participants, and coordinated a strong and committed network of Extension personnel and partners who want to continue to carry out extensive programming for private well water protection.
Planning for the 2009 Symposium is currently underway.
We would like to thank the Symposium Planning Committee for their dedication to this event <www.usawaterquality.org/nesci/2009Symposium>. USDA CSREES National Integrated Water Quality Program provides funding for the New England Regional Water Program and the Initiative. Additional funding for the symposia has been provided by EPA - New England, Water Systems Council, University of Rhode Island, College of the Environment and Life Sciences, University of Maine Cooperative Extension, University of Massachusetts Extension, Barnstable County Cooperative Extension, University of New Hampshire Cooperative Extension, Senator George J. Mitchell Center for Environmental and Watershed Research, University of Maine, and USDA Healthy Homes and Dartmouth College Toxic Metals Research Program<http://www.dartmouth.edu/%7Etoxmetal/>. This is contribution number 21361 of the College of the Environment and Life Sciences, University of Rhode Island.
Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line], 40(3) Article 3FEA4. Available at: http://www.joe.org/joe/2002june/a4.php
Bennett, C. F. (1975). Up the hierarchy. Journal of Extension [On-line], 13(2). Available at: http://www.joe.org/joe/1975march/index.php
Clemens, S., Swistock, B., & Sharpe, W. (2007). The Master Well Owner Network: Volunteers educating Pennsylvania well owners. Journal of Extension [On-line], 45(4). Article 4RIB7. Available at: http://www.joe.org/joe/2007august/rb7.shtml.
Hoffman, B., & Grabowski, B. (2004). Smith Lever 3(d) Extension evaluation and outcome reporting—A scorecard to assist federal program leaders. Journal of Extension [On-line], 42(6) Article 6FEA1. Available at: http://www.joe.org/joe/2004december/a1.php
Lemley, A., & Wagenet, L. (1993). Rural water quality database. Journal of Extension [On-line], 31(3) Article 3FEA2. Available at: http://www.joe.org/joe/1993fall/a2.php
National Oceanic and Atmospheric Administration (NOAA) Coastal Services Center. (2003). Product design and evaluation. Course Training Manual. Charleston, SC.. Updated periodically. Course information retrieved June 18, 2009 from: www.csc.noaa.gov/training
Peterson, C. (2003). Bringing ADDIE to life: Instructional design at its best. Journal of Educational Multimedia and Hypermedia, 12(3) pp. 227-241.
Swistock, B. R., Sharpe, W. E., & Dickison, J. (2001). Educating rural private water system owners in Pennsylvania using satellite versus traditional programs. Journal of Extension [On-line], 39(3) Article 3FEA7. Available at: http://www.joe.org/joe/2001june/a7.php
U.S. EPA Private Well Owners. Retrieved June 18, 2009 from: http://www.epa.gov/region1/eco/drinkwater/private_well_owners.html
University of Wisconsin Cooperative Extension. Evaluation logic model (2002). Retrieved June 18, 2009 from: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html