|
|
|
|
In January of 2005, the Leading for Quality Initiative launched a pilot coaching initiative whereby on-site support was offered to seven programs/collaborative groups. Interested programs were invited to respond to a Request for Proposal by stating their desire for a coach and intended program improvement goals.
Between February and June a Leading for Quality coach worked with administrators and site staff from each group in an effort to help guide and support each program/group's individualized program improvement efforts.
Evaluation of the Coaching Initiative:
During June of 2005 the project evaluator conducted (phone and in-person ) interviews with twenty-one individuals that had received direct or indirect coaching during FY05. This included eleven Site Coordinators & organizational administrators and ten site staff representing the seven groups that had received coaching. In addition, two phone interviews were conducted with the Leading for Quality Coach.
The purpose of the interviews and focus groups was to answer the following questions:
- What was the coaching model?
- How should/could it be strengthened?
- Did coaching make a difference at participating sites?
- What coaching model is viable for the future?
All coaching recipients were asked the same set of open-ended questions regarding their experiences & perceptions of the coaching model, as well as their ideas for how it might be improved. In addition, all recipients were also asked to assign a rating to "how effective" they felt coaching had been, and to answer whether they felt the current duration and frequency of coaching had been "just right" or needed adjustment.
While interviews were not intended to directly assess the performance of the coach, given that respondents had to answer questions based on their own experience with the L4Q coach, Erica Lindamood, recipients reflections' on her strengths, aspects of her coaching style or approach that were seen as most contributing to changes, etc. were an inevitable piece of the data collection process.
Findings:
Coaching Recipient Responses:
All coaching recipients were asked, "What was it like to have a coach? Was it what you expected?" About two thirds of those interviewed felt that, overall, it had been a positive experience to have a coach come work with their site. For some, it was not necessarily what they expected-but for this group of recipients, it was still found to be positive nonetheless. Respondents had a range of different reasons for their positive assessments-as well as varying levels of enthusiasm for the experience.
Sample Comments: What was it like to have a coach?
- It pointed me in a good direction
- Yes, lots of observation and feedback and goal setting. Was helpful to my staff in that way.
- Very comfortable, she wasn't overbearing, easy to engage with and talk to.
- Her energy was comforting.
- Liked having a person come in with ideas, affirmation for what we were doing well.
- Actually been more than what I expected.
- A lot of communication lines have been open as a result of the coaching.
About one third of the respondents, representing (a portion of those from ) four of the agencies, felt that the coaching was not what they expected. These recipients expressed varying levels of disappointment, confusion or frustration regarding their coaching experience.
What did the coach do? What aspects of coaching were felt to be most effective?
When asked, "What did the coach do that was helpful?", four very strong trends emerged in respondents comments regarding the coach's style, and approach. These included (in descending order)
Coaching Approaches and Characteristics Perceived to Be Helpful or Effective
- Her ability to provide programs with an overall assessment of strengths and areas needing improvement (i.e. her ability to help pin-point priorities).
- Her thoughtfulness in giving feedback and ideas. (Able to deliver constructive feedback in a way that could be heard and felt constructive)
- The wide range and quality of concrete ideas she provided for addressing areas in need of strengthening. (She was found to be knowledgeable-have good ideas)
- The coach was felt to be trustworthy, non-threatening, dependable, and non-judgmental (felt to be desirable characteristics in a coach)
In addition, many felt they had benefited from one or more of the coach's areas of expertise, including:
- Ways to set up the physical environment
- Promoting youth autonomy and ownership
- How to improve the daily schedule
- Creating kits and interest centers
- Facilitating youth discussions and meeting times.
Sample Comments-Effective Coaching Approaches or Characteristics
- Her evaluation of the program was very effective--right on.
- She remained neutral, she wasn't taking one person's side, that way no one felt threatened.
- She has a calm nature about her. She was thoughtful
- She helped us realize that we needed to step back and look at the basics.
- She was very effective in giving feedback that might not be received so positively, good at pointing out strengths and successes
- She taught us simple things, how to organize the space, how to get kids' feedback.
- She would reflect and follow up on issues and her observations and come back with concrete suggestions for things for us to try.
- Her feedback was honest. She would affirm what we were doing well.
- She pushed us a bit.
When asked "what did the coach do that was less helpful?", recipient responses indicated several areas that could be clarified or improved in future coaching models.
- Would have liked more explicit direction and "solutions" .
- Would have preferred more frequent and direct communication and feedback.
- Would have liked a faster pace to change efforts.
- Would have preferred more support with staying motivated and on-track.
- Would have preferred more action (e.g. modeling, observations) and fewer meetings
When asked, "What have been the challenges with working with a coach?" most respondents said there had been none. However, for those who had experienced challenges, a few minor trends emerged, including (in descending order)
- Not enough information regarding the coach's specific areas of expertise-what was she good at? How could we best use her help?
- Less effective having her coach just one person or dyad-would have been better to work with the whole organization or players together.
- Too few coaching visits to site or coaching visits at wrong times of day.
When asked, "what do you feel you did to help make coaching successful?", the strongest trend in coaching recipients responses were : "being available for meetings" and "being open to change or hearing what the coach had to say". Not as many respondents stated (without prompting) that they had taken the time to read materials the coach brought or implement suggested ideas. In a related question, coaching recipients were asked "do you feel that you (or your staff) were able to make good use of coaching?". Respondents were evenly split with half saying "yes" and half saying "no". Interestingly, a few of the respondents felt that they had learned through this first experience how they could be more pro-active by being better about articulating their needs asking for help--and in general, making better use of coaching the next time around.
Did coaching make a difference at sites?
Both coaching recipients and coach were asked to reflect on the perceived "effects" that coaching had had on them and their sites. First, all coaching recipients were asked to rate how effective they felt coaching had been in helping them or their site to make desired changes. Ratings were made on a 4 point scale, where 1=Not at all effective, 2=Somewhat effective, 3=Fairly Effective and 4=Extremely Effective. Similar to ratings for many other L4Q components, the average rating to this question was that coaching had been "somewhat effective" (2.24).
Staff and Site Administrators appeared to perceive the coaching experience slightly more positively than Upper Level Administrators. This might be due to the, on average, greater time spent with the coach developing a 1:1 relationship and experiencing changes in individual practice.
Perceived Changes at Sites:
An important component of the coaching model was allowing each site to identify their own unique needs and goals for coaching. As a result, it was not possible to utilize a consistent or standardized approach for gaining baseline information for each site prior to receiving coaching. Given the absence of baseline information, the evaluator sought to determine the nature and extent of change at sites through coach and recipient perceptions. To increase the validity of perceptual data, a standard of having three independent sources (triangulation) stating that change of a similar nature had taken place, was applied.
The primary source of data from coaching recipients (related to program change) was responses to the interview question: "do you feel you have made any progress in meeting your goal(s), what changes do you feel have come about as a result of coaching?". In one case, information generated through a pre and post Self Assessment completed by an additional organizational source was also utilized. Perceptions of site/staff changes were also gathered from the coach who was asked to reflect on the extent of progress each site had made in implementing desired changes. The coach communicated whether she thought each site/group had made "a lot" of progress, "some" or "little or none".
An examination of all sources indicates that "change" had occurred at slightly more than half of the seven sites/groups. For some sites/groups, it was felt that the many of the goals of coaching had been met. For other organizations, it was felt that positive progress (some changes) had been made towards their goals. For some it was felt that no changes had occurred.
Examples of the nature of changes:
- Improvements to the lay-out or organization of physical space/materials
- New program or organizational rules or procedures
- Staff have improved planning skills
- Staff have new skills in managing/facilitating youth
- Staff have established stronger relationships with youth
- Organizational decision to increase staffing to better support site
For the organizations that felt that "change" had in fact taken place, only one organization was confident across the board ("yes" by all staff, Site Coordinator, and administrator) that the change was "sustainable". For some organizations, a belief of "sustainable" change was a view only held by the administrator --not necessarily shared by all staff and Site Coordinators. Administrators for other organizations felt that "sustainable" changes would be likely with continued coaching support in FY06.
For organizations that did not report any change through coaching, only one felt that change was not possible through the current coaching model. The others embraced the prospect of continued coaching, and expressed a belief that change was indeed possible at their sites.
Potential Factors to Consider in Maximizing the Effectiveness of Coaching:
Interviews with the coach, recipients and an intra-analysis of site progress revealed that there were tremendous differences across sites/groups with respect to a number of factors. This variability allowed some preliminary look at possible connections with coaching success and how the coaching model might be modified to best meet the differing needs of sites and organizations.
Information from coach and recipient interviews, as well as an intra-analysis of coaching sites suggests that for those sites that made the most progress there had been: (in descending order)
- Frequent communication and planning time between coach and administrator. (i.e. administrator was pro-actively involved in coaching)
- Some level of flexibility in establishing goals for coaching between administrator, site staff, and coach. (All parties discussed program needs and negotiated a refined/modified goal for coaching work)
- Minimum of once per week of on-site coaching
- Coaching offered to both administrators and site staff
- Compatibility of work and personal style between coach, administrator and site staff (including desired levels of directness, pace of change, communication styles)
- Pre-existing organizational features (low staff turnover, interest in coaching by staff, smooth daily functioning of program)
While it appears that all of these factors (and of course potentially other factors) play some role in influencing the success of coaching, the one factor that was present for all sites that experienced change was a "high level of pro-active administrative support of the coaching". In other words, even when other factors listed above were low or not present, it was possible to see change if there was a high level of administrative engagement/support of coaching. It is also possible that the "level of administrative engagement" is influenced to a certain extent by some of these other factors (i.e. administrators may not become engaged unless they feel their work style is compatible with the coach, for instance).
How could the coaching model be strengthened and improved?
It is clear from recipient and coach reports that it is possible to produce positive effects from coaching. However, it is also clear that the provision of coaching to sites is by no means a guarantee that change will occur.
To enhance the potential positive effects from coaching three steps are suggested:
- Make adjustments to the model based on recipient and coach input
- Pay special attention to the factors/ingredients that appear to be linked to greater success in order to make more explicit to coaches and coaching recipients "how to best achieve an effective coaching experience".
- Create a support and accountability structure to maximize site progress with coaching
Coaching recipients and the coach were asked to express their views regarding the current coaching model, and how they would like to see it changed or improved.
Time of Year, Frequency and Duration of Coaching:
When asked, "do you feel that the current duration of time that coaching is offered is just right-or needs changes?"-only 25% of coaching recipients said it was "just right". All but two of the organizations felt that the length of coaching should be longer than the typical 8 weeks sites felt they had received. All but one organization felt that the coaching would be more effective if offered earlier in the year-preferably in the fall.
When asked, "do you feel that the frequency/intensity of coaching is just right-or needs changes?" more than a third of the respondents felt that the current frequency of coaching visits was "just right". However, sites did not always receive the same frequency of coaching so responses to this question did not yield clear results. During interviews respondents expressed differing views on the desired frequency/intensity of coaching. More than half felt that once a week would be effective, however, two organizations felt that twice a week was needed, and two organizations felt that every other week would have been more effective.
What other changes would you like to see in the coaching model?
When the coaching recipients were asked for their ideas on how the coaching model could be strengthened, three clear trends in suggestions for desired changes emerged (in descending order)
- Having a variety of coaches from which to choose enabling programs to best match style, areas of expertise and cultural characteristics to each program. Explicit and detailed information of coach's areas of expertise (what they are knowledgeable about or good at) was felt by many to be an important piece of information that would be needed. (One suggested that there be an "executive coach" to help assess needs, followed by a variety of coaches to help with differing aspects of the program)
- Greater clarity regarding the goals of coaching (at each site) and process that would be undertaken to get there. Creation of a system for setting goals and benchmarks to track progress. (While some organizations felt that the great flexibility and organic nature of the process had worked fine for them, all felt that more structure, clarity, and communication about "What are we trying to accomplish through coaching? What will we do to accomplish this-by when?" would have helped to get more done and at a faster pace)
- Intensive coaching should be followed by a "checking-in" phase to ensure continued progress
Other ideas and suggestions by coaching recipients and coach, included (in descending order)
Suggestions Regarding Basic Coaching Design:
- Begin coaching earlier in the year
- Increase duration and/or intensity
- Minimum of once per week on-site coaching
- Reduce number of sites per coach (e.g. from 7 to 4) to allow for greater focus, more timely feedback to sites and staff.
- Change orientation process to be more personalized rather than formal
- Clarify process, guidelines for accessing $1500 site coaching funds.
- Allow programs to have an assessment by the coach before submitting the request for specific coaching support.
Suggestions Regarding Coaching Style/Approach:
- More explicit direction (e.g. potential approaches, solutions) offered to staff and administrators (such as scripting, modeling, suggested follow up tasks)
- More direct communication and feedback.
- Faster pace of work/change.
- Higher energy, more motivating and dynamic coaching style.
- Fewer meetings discussing ideas--more demonstrations and observations with immediate feedback.
- More information on coach's background and particular areas of coach's areas of expertise. (what is she/he good at, what can he/she offer us?)
- More tangible examples of products and practices.
- 1:1 coaching prior to engaging with groups/partnerships
- Coach(s) hosts monthly group discussion/training session on topics of mutual interest to differing coaching sites.
Next Steps for the Coaching Model:
The results from the coaching evaluation suggest that program change is possible through coaching. Clearly, finding ways to ensure that administrators are highly engaged and supportive of coaching appears to be a critical step in contributing to site success. Once the administrative engagement can be obtained, it is clear that coaching must be tailored to best meet the needs, goals, levels and styles of staff and administrators.
Many program factors necessitate differing coaching approaches and the capacity for change. Programs at differing levels of organizational development, structure, focus, age groups, gender, staff skill levels, philosophies, financial resources, etc. might differ in their capacity to make positive changes, and will also require differing coaching approaches, styles, and intensity in order to benefit. For instance, programs with new or inexperienced staff that want to affect practice, might need more direct instruction, pre-prepared activities/curriculum and scripting & modeling for staff with a slow progression to staff initiated changes and experimentation. Programs with more seasoned staff might benefit more from reflective discussions with the coach.
Hypothetically, it is possible that one coach who is highly flexible and versatile in adapting approaches, pace of change, teaching style, variety of content expertise, etc. could meet the diverse needs of staff and sites, however, this is a tall order for any one person. It is more likely that a small cadre of coaches will be needed to work most effectively with the wide range of needs and interests held by Cambridge sites. It may also be necessary in some cases for multiple coaches to work with sites, with perhaps one Primary Coach who may help sites identify areas of need, and then help enlist the help of others with particular expertise to work towards specific goals. The matching process between sites and coaches should be participatory with administrators, staff, L4Q, and coach making decisions on matches together.
Some matching factors to consider might include:
- Coach's experience and expertise in working with the ages, genders, cultural/linguistic background of youth to be focus of changes
- Program's interest (and coach's preference) for doing general program assessment/improvement versus specific curricular or activity goals
- Degree of flexibility (by all parties) around establishing coaching goal and willingness to address issues raised by coach.
- Area(s) of expertise by coach including both content as well as coaching approaches (e.g. skills in modeling versus meetings with staff)
- Staff/Sites preference for level of directness in coaching and feedback, pace of change, types of accountability, (e.g. "be tough and hold me to this" versus "let me go at my own pace")
- Compatibility of communication style and educational philosophy (values and beliefs about how to work with youth -and staff)
- Coach's experience working with program/sites at this level of development
Accountability and Support Structure for Coaching:
To ensure that sites make progress and that coaching is going smoothly, L4Q may want to consider instituting more structured supports and check-points to the coaching process. These would be utilized to help support communication, identify and resolve issues in a timely fashion, and/or discontinue coaching for those sites that are not benefiting from coaching support. For instance, perhaps coaches and sites should be expected to agree and submit their agreed upon "goal" and basic coaching approaches that will be utilized, sometime in the first 4 weeks of coaching. During the ensuing months, perhaps coaches and sites might be asked to do some reporting or participate in a "check-in" meeting with L4Q to discuss progress/benchmarks. Organizations might also be asked to fulfill certain obligations in following through with coaching progress. For instance, checking in with staff to be sure they have the time and supports -or understand what they need to follow through with changes. Or, if the primary recipient of coaching leaves their position, the organization might be expected to propose how progress will be maintained in their absence. Coaches might also be encouraged to help sites document progress related to coaching through creating a notebook illustrating changes. For instance, it could include pre and post photos of physical space, photos of projects implemented, statements from staff, youth and kids reacting to new approaches, etc. This type of documentation could serve multiple purposes from accountability, to boosting staff morale, to communicating progress to others.
|
|
|
