February 2007 // Volume 45 // Number 1 // Commentary // 1COM1
 |  |
 |
 |
 |
Can We Define and Measure Excellence in Extension?
Abstract
In 2005, Chester Fehlis challenged the Extension system to define and measure Excellence in Extension. The Extension Committee on Policy responded with a task force and work group that identified a matrix based on an academic perspective of universities and an Extension-familiar perspective. Seven criteria were identified as most important, for which definitions and measures were developed. A pocket card was created that lists the uniqueness of Extension and qualities for excellence. Also proposed is a national database for entering, aggregating, and sharing the measures, as well as other data that will standardize comparisons between different Extension institutions.
Are We Interested in Our Egos or Excellence?
Chester Fehlis in his 2004 Ruby Lecture, "A Call for Visionary Leadership," stated:
In Extension, every institution and its faculty have their own personally defined metrics. . . . I have discovered that every Extension Director believes that his or her state's Extension program is in the top five or ten in the country. That may be good for our egos, but it is not good for Extension. . . . If we are truly to have visionary Extension leaders and administrators . . . then we should consider establishing accepted criteria and metrics that clearly define excellence in Extension (Fehlis, 2004; Fehlis, 2005).
In a follow-up presentation to the Extension Committee On Policy (ECOP) in February 2005 (Fehlis, 2005), Fehlis stated that because there were no established criteria that define "Excellence in Extension," we do not capitalize on the synergy of our national Extension system, but rather have 70 Extension programs going in many different directions, each believing theirs is the most effective and best Extension program in the nation.
Fehlis challenged the Extension system to take the initiative to define excellence and to identify ways to measure it. In the summer of 2005, ECOP responded by appointing a task force representing all five regions to focus on measuring Excellence in Extension. The group was charged with:
- Identifying the criteria that defines Excellence in Extension
- Determining the measures of those criteria
- Delineating the necessary collection methods
- Determining how to gain acceptance of the criteria and measures by the Extension system
What Do Extension Administrators Do Best? They Delegate!
The task force, along with a work group of Extension evaluators, set about defining the criteria of excellence and identifying appropriate measures. They:
- Brainstormed what constitutes Excellence in Extension
- Developed a matrix to display the criteria of excellence
- Obtained feedback on criteria from directors and administrators at the regional level
- Engaged directors and administrators in determining priorities.
Who Do We Serve? A Matrix Evolves
The work group struggled in early months with how to identify criteria of excellence that are meaningful to all stakeholders, given the diversity of institutions and Extension systems around the country. The product was a 4" by 5" matrix based on two conceptual frameworks. One axis represents the traditional academic perspective of universities. The four columns in the matrix are: Teaching and learning; Discovery and scholarship; Engagement; and Management. This approach reflects and values the language of research and teaching faculty and university administrators.
The second axis represents a perspective that is more familiar to Extension personnel and supporters. The five rows in the matrix are: University commitment to Extension; Relevance of Extension programs; Quality of Extension programs; Outcomes and impacts; and Funding and external support.
These two dimensions illustrate some of the variability among Extension's many partners and stakeholders as to how they perceive the value of Extension. The primary audiences for this information are persons making funding decisions, university administrators, and Extension decision makers.
Identifying criteria of overall Excellence in Cooperative Extension is a valuable exercise for the Extension system as well as for individual institutions. These criteria will enable Extension to describe its strengths and to differentiate itself from other agencies and organizations. (See "Excellence in Extension: Two Products for Definition and Measurement" in this issue for a fuller description of the Matrix of Criteria of Excellence in Cooperative Extension.)
Can a System to Measure Excellence in Extension System Actually Be Established?
The task force and work group encountered and resolved some significant problems: agreeing on the criteria for excellence, developing a widely accepted definition for each criterion that was deemed highly important, and identifying measures for each criterion. Several challenges remain: gaining acceptance of the measures by the system, collecting the data, and developing, maintaining, and updating a national database.
Through a yearlong interaction of task force members, work group members, and periodic interaction with Extension administrators at the regional and national levels, 61 criteria were identified. Based on the priorities expressed by Extension directors and administrators, the number of criteria was reduced to 19, and seven were identified as most important. The original 61 were retained because some may be useful to individual institutions.
For the seven most important criteria, uniform definitions were developed. This was vital because individual institutions currently use unique definitions that are rooted in history and tradition. Without uniform definitions, aggregating measures and comparing across institutions is problematic.
Measures were also identified for the seven highest priority criteria. The focus was on measures that are reasonable to collect without placing an undue burden on institutions; however, to implement some of the measures, there is a need to conduct more in-depth studies.
A national database will be established and patterned after one developed by Texas Cooperative Extension for use by the Association of Southern Region Extension Directors. It will focus on institutional-level indicators for internal decision-making. This does not duplicate the Cooperative States Research, Education, and Extension Service (CSREES) USDA reporting system, which focuses on program results of individual programs. It capitalizes on the work of individual states, regional and national subject matter work groups developing measures of program impact. Institutions will self report data through password-protected access. The database will require maintenance, and a mechanism will be needed to make decisions about changes in indicators that can be expected to evolve over time.
It is important to understand the distinction between two separate elements of this product. One element is the seven criteria for excellence and associated measures and definitions. The other element is the set of measures to be included in the national database. These are related, but they are not the same.
The seven criteria are better employed as an internal mechanism for individual Extension systems to determine how well they are doing on the things that would make their own systems excellent. Only a portion of the criteria measures will be included in the national database. Measures from the criteria that are included in the national database have the greatest chance of being collected in an easily quantifiable and uniform manner. The national database measures will be used more for comparison and aggregation purposes across the entire Extension system.
Who Should Use the Products? All Extension
A national effort will give credibility to the criteria of excellence and will encourage uniformity in measurement throughout Extension. The criteria and measures identified will assist institutions as they identify indicators; consistent criteria will provide a basis for standardized comparisons with other institutions; and measures reported by all institutions can be aggregated to represent the entire Cooperative Extension System
This information can be used to describe the overall scope of Extension and to market it as a national system. Such information could also be used to document Extension's strengths or to focus on areas that could be improved.
The matrix provides a two-dimensional perspective on criteria for Excellence in Cooperative Extension. It describes Excellence in Extension in a concise, yet comprehensive way.
A "pocket card" provides a list of five attributes that make Extension unique, six summary points of qualities for Excellence in Extension, and the seven criteria for Excellence in Cooperative Extension. This card summarizes what can become every Extension worker's or administrator's talking points about Extension. (See "Excellence in Extension: Two Products for Definition and Measurement".)
The database provides the mechanism for entering, aggregating, sharing, and archiving the identified measures, as well as other data that will help standardize comparisons between different Extension institutions. How well this works will depend greatly on how individual institutions of the Extension System embrace the need.
Are There Flaws? You Decide
This has been an internal effort. Extension directors and administrators established the priorities, and Extension evaluation specialists outlined the detail. Would the matrix, criteria, definitions, and measures be different if rated by alternative audiences, e.g., legislators, other government officials, university administrators or clients?
We think not much. Over a period of 18 months, Extension has taken a thoughtful and critical look at itself and proposed a process to measure its internal progress, make comparisons among institutions, and at the same time provide a succinct description of Excellence in Extension to the world.
What Now? Gain Acceptance
The final charge of the ECOP Task Force on Measuring Excellence in Extension is to determine how to gain acceptance of the criteria and measures by the Extension System. Extension administrators and directors have been updated several times during the 18 months of development. The products have been, or soon will be, published in a variety of venues to share with the Extension System.
To be effective, this process and these products for understanding and measuring Excellence in Extension must be widely embraced. This may be the greatest challenge of all. Institutions may need to develop or modify data collecting and reporting routines to fit with the uniform measures that are proposed.
But the potential payoff can be huge. If this approach gains acceptance throughout the Extension System, then perhaps we will have responded adequately to Dr. Fehlis' call to "establish accepted criteria and methods that define Excellence in Extension."
References
Fehlis, C. P. (2004). A call for visionary leadership, Ruby Lecture, National Epsilon Sigma Phi Annual Meeting, September 17, 2004. Available at: http://espnational.org/historybyyear/2004/ruby2004.htm
Fehlis, C. P. (2005). A call for visionary leadership. Journal of Extension [On-line], 43(1) Article 1COM1. Available at: http://www.joe.org/joe/2005february/comm1.shtml
Fehlis, C. P. (2005) Defining excellence in Extension. Presentation to Extension Committee on Policy (ECOP), February 15, 2005.
While not a part of your 'target audience,' I had a comment on one of your articles,
"Can We Define and Measure Excellence in Extension?" [Article No. 1COM1]
The task the authors have set out for themselves and others is well known to me, as it is common within the world of quality management and leadership as sought by industry, education, health care, and non-profits.
If one would go to the link below, they will find information that may shorten their time to answer the questions raised within their referenced article.
http://baldrige.nist.gov/Criteria.htm
with best regards,
Bill
The items in the outline of the Criteria on page 11 of the
"Education Criteria" document do align with our Criteria of Excellence in Cooperative Extension Matrix in the Tools of the Trade section of this issue of JOE, especially if one looks at the column axis with the headings of: Land-Grant University Commitment to Extension (Baldrige = Leadership, Strategic Planning); Relevance of Extension Program (Baldrige = Student, Stakeholder & Market Focus); Quality of Extension Programs; Outcomes & Impacts (Baldrige = Workforce Focus, Results); and Funding & External Support (Baldrige = Process Management)
The Baldrige Criteria = Measurement, Analysis, and Knowledge Management really appears to be what our Excellence in Extension effort is about. On page 45 of the Baldrige document, it states, "The Measurement, Analysis and Knowledge Management Category is the main point within the Criteria for all key information about effectively measuring, analyzing, and improving performance and managing organizational knowledge to drive improvement in student and operational performance."
In our manuscript, we stated, "The seven criteria are better employed as an internal mechanism for individual Extension systems to determine how well they are doing on the things that would make their own systems excellent. Only a portion of the criteria measures will be included in the national database. Measures from the criteria that are included in the national database have the greatest chance of being collected in an easily quantifiable and uniform manner. The national database measures will be used more for comparison and aggregation purposes across the entire Extension system."
These two "goals" are similar, if not the same.
The biggest differences in the Baldrige efforts and the Excellence in Extension appear to be that the Excellence in Extension effort: [1] Includes a level at which the collected data goes beyond self-improvement to compare similar institutions; [2] Exists outside the traditional formal education framework (informal education, more variety of size of Extension organization, wider spectrum of financial resources); [3] Addresses the higher education perspective; and [4] Creates a system that would not take a great deal of additional time or extra effort to identify, enter, and query the data - It is desired that ALL Extension systems participate, not just those which are inspired to do so.
To determine "outcomes" of an educational program is very different from highlighting the "inputs" (facilities, personnel), the "outputs" (activities, people involvement), or the "reactions" (quality of teaching) of our Extension efforts. Most of us in Extension were hired for our subject matter, programming and people skills - NOT our evaluation skills. Then too, "good science" evaluation skills take additional resources of time and money, which subtract from what we can do in the programming arena. So we already have two strikes against us: [1] Limited evaluation skills; and [2] Less programming if we do evaluation right!
But we no longer live in a funding world that gives us credit for just "doing good". We must prove that what we do makes a difference.
So, what is the answer? I think there are at least a few things that we can do:
[1] Pick the low hanging fruit - We can do things relatively easy as a part of education programming that can at least get us to the short term outcome level of knowledge, attitude, skill or aspiration change in program participant. Simple pre-post assessments based on our teaching objectives, or retrospective pre-post assessments.
[2] Plan outcome evaluations for one or two of our educational efforts every year, and gather good, in-depth outcome information. The next year, do outcome evaluations for one or two different programs, and so on for the next 2-3 years. Then, you will have good impact data on 8-10 of your programs that is relatively current. (No person in our organizations can impact evaluate every program, every year. That would require one or two evaluators for every programming person in Extension!)
[3] I know that there are a few people in every Extension organization who are what I call "Closet Evaluators" - Those Extension professionals who either already possess great evaluation skills, or who are interested in learning more about evaluation, BUT who do not want to admit it, because the word "Evaluation" has a bad connotation in our culture. If you are a "Closet Evaluator", come out; If you know of a colleague who is a "Closet Evaluator", take advantage of their skills/ potential. It always amazes me that Extension people are not afraid to arrange the best subject matter person they know to contribute to an educational effort, but they do not look or take advantage of those resources which may be able to assist with the outcome evaluations.
The bottom line is that it WILL take extra effort to gather outcome data, and the need for good outcome information is not going to go away.