LIFE News issue 11: project evaluation
Welcome to LIFE News – September 2010. Earlier this year LIFE hosted program evaluation workshops in conjunction with the Austalian Psychological Society. Continuing our focus on the importance of evaluating your projects, this edition of LIFE News includes sector comments from two experts in the evaluation fileld, the latest research and two evaluation profiles of projects funded under the National Suicide Prevention Strategy.
By Dr Kairi Kõlves
|
Above: Dr Kairi Kõlves
|
Dr Kairi
Kõlves is a Senior
Research Fellow (currently Acting Director) at the Australian Institute for Suicide Research and Prevention (AISRAP). Dr Kõlves specialises in epidemiology and medical sociology. She has been working in suicide research and prevention since 1998. Between 1999 and 2008, she worked at the Estonian-Swedish Mental Health and Suicidology Institute in Estonia. In this sector comment, Dr Kõlves discusses evaluation of suicide prevention, with a particular focus on what has been published in suicidology research.
One of the challenges in suicide prevention is finding evidence for the effectiveness of different interventions, activities, projects or programs. An evaluation analyses the quality of the evidence demonstrating the effectiveness of an activity, project or program. Different levels of evaluation have different aims.
A process evaluation focuses on the quality of activities. An impact evaluation measures the immediate effects of a program on recipients and the achievement of objectives. Finally, an outcome evaluation examines the long-term effects of a project/program on a wider population and the achievement of goals.
Plans for all levels of evaluation should be incorporated in the initial stages of any project. In order to create a strong evidence-base in suicide prevention, our evaluations have to be as objective and systematic as possible. We should only use reliable information and scientific methods for the collection of information and analysis.
Current state of suicide prevention evaluation research
During the last two decades, the evaluation of suicide prevention projects and programs has become more frequent. However, our knowledge about what works and what does not work in suicide prevention remains limited as only a few projects have included an evaluation component that is made publicly available. Even when included, evaluations have been poorly designed with critical questions incorrectly stated. Other methodological issues have also been raised where evaluators have not been independent, evaluation levels have been left uncompleted, and it has been impossible to measure initiatives within a project’s timeframe.
Recently, a few papers have systematically analysed the effectiveness of different suicide prevention initiatives around the world (Mann et al. 2005). There have been specific analyses made in the Western Pacific Region (WHO 2010), New Zealand (Beautrais et al. 2007) and the US (Rodgers et al. 2007). These reviews have classified suicide prevention activities into categories based on levels of evidence: effective; promising; and, insufficient current evidence.
Suicide prevention activities proven to be effective:
- Training GPs to recognise and treat depression and suicidal ideation has been shown to be effective in several countries. However, despite improvements in GPs’ and other health-care professionals’ knowledge, there is not enough evidence to prove long-term changes or reduced suicide rates in Australia.
- Restricting access to means has been proven to reduce suicide in different countries. In Australia, restricting access to firearms and barbiturates, and the placement of barriers at suicide jumping sites has proven to be effective. However, method substitution still remains a problem.
Promising suicide prevention activities (there is some evidence, positive or contradicting results):
- Training gatekeepers (pharmacists, teachers, clerks, etc.) to recognise signs of mental health problems and suicidality has improved their knowledge. However, there is no clear evidence that it has helped to reduce suicidal behaviours.
- There have been no systematic evaluation of activities which raise public awareness, and improve mental health literacy and help-seeking behaviours. However, promising results have been shown in some countries, including Australia.
- School-based programs targeting students have shown contradicting results. Programs focussing on coping skills and competency-building have shown positive or no results. However, awareness about suicide may potentially create harmful effects.
- The ways in which suicides are reported in the media can influence suicidal behaviours. However, the creation of media guidelines and education for journalists has not given clear results in reducing suicidal behaviours.
- There has not been enough evaluation publicly available of crisis centres and phone counselling services. However, it has been shown that Kids Help Line in Australia has improved the mental health state of those children who access it.
- Some psychotherapies, such as cognitive therapy, problem-solving therapy, and interpersonal therapy, have shown reduced suicidal behaviours and feelings of hopelessness when compared to treatment-as-usual. However, they are often only tested on small groups of subjects.
- Follow-up care for suicide attempters has shown positive effects. Further, brief intervention care has proven to be effective in some countries in the Western Pacific Region.
- Restricting alcohol availability has been shown to reduce the number of deaths from suicide and other external causes in some countries.
- Activities supporting suicide survivors and communities bereaved by suicide have shown positive or no results.
Despite several studies in the area, the effectiveness of pharmacotherapy in reducing suicidality remains unclear.
Current reviews indicate that suicide prevention programs have not been rigorously evaluated using proper methodologies. There is a need to encourage the evaluation of existing and innovative suicide prevention activities. Evaluations should be integrated into every suicide prevention project and, subsequently, be a natural part of everyday project management. An important part of continuous evaluation is the distribution of results, within a reasonable time and among the relevant people, which should help to improve existing activities.
LIFE Fact sheet 14: Project evaluation: gives a brief overview about project evaluation and also helps with references to find more detailed instructions.
References:
Beautrais A, Fergusson D, Coggan C, Collings C, Doughty P, et al. (2007). Effective strategies for suicide prevention in New Zealand: a review of the evidence. New Zealand Medical Journal 120, U2459.
Mann JJ, Apter A, Bertolote J, Beautris A, Currier D, Haas A, et al. (2005). Suicide prevention strategies: a systematic review, Journal of the American Medical Association 294, 2064-2074.
Rodgers PL , Sudak HS, Silverman MM, Litts DA (2007). Evidence-based practices project for suicide prevention, Suicide and Life-Threatening Behavior 37, 154-164.
World Health Organization (2010). Towards evidence-based suicide prevention programmes. World Health Organization: Western Pacific Region.
The evaluation experience: StandBy Response Service
By Geoff Walters and Jill Fisher
The StandBy Response Service is a community based active suicide postvention program developed by United Synergies Ltd in Tewantin, Queensland.
The StandBy program has undergone several evaluations since it commenced in March 2002. In 2007, the Department of Health and Ageing funded a longitudinal evaluation of the StandBy Response Service which was completed in 2009. This independent evaluation was linked to the federally funded roll out of the StandBy program in four diverse sites across eastern Australia. Since that time the StandBy Response Service has further expanded with StandBy now operating in 9 sites across Australia all of which are managed by local agencies.
The role of the evaluation was to demonstrate whether community benefits of the StandBy program were evidenced and whether the primary target group (people bereaved by suicide) who had experienced the StandBy service had reduced physical and mental morbidity; a greater sense of connectedness to and within the community; improved understanding of government and non-government services and a greater willingness to engage with those services.
This evaluation provided a unique opportunity to demonstrate the efficacy and effectiveness of the StandBy program and to ultimately further expand the evidence base. We felt this was important given there is limited evidence available to inform suicide postvention practice.
To do this effectively, an independent evaluator was identified at an early stage of the StandBy replication project so the evaluation design could be incorporated.
Two pronged approach to evaluation
Driven by the evaluation aims, it was decided that a two pronged approach would be used to evaluate the replication trial. This meant that there was a focus on measuring the capacity of the local community to respond to suicide events as well as assessing the impact of the StandBy program on people bereaved by suicide. A pre and post implementation measurement of the community’s capacity to respond to suicide events was conducted focussing on emergency services and other first response community agencies. Four specifically designed survey instruments were developed to monitor and measure the key concepts being evaluated. The different groups for comparison with those who had used StandBy were:
- Those recently bereaved by suicide who did not use Standby Support Service
- Those who had experienced a non-suicide related trauma in past five years
- Those who has not experienced a recent life trauma (in the past ten years)
ChallengesWe found that the timeframe for the client needs assessment had to be shortened as the setting up of the services in the different areas took longer than originally anticipated due to the great variations within the different regions. We ultimately saw this as serving to enrich the data that was collected as it meant that both the clients and communities served were more diverse.
An additional challenge was how to collect and utilise data that was of a sensitive nature, so clients provided informed consent and were reassured that the information they provided was confidential and secure. All participants were also provided with contact details of support services in their local communities as well as national services such as Lifeline. All research staff collecting the data were extensively trained and they were adequately supported in their role (including the provision of debriefing).
Responses to the findings and recommendationsWe were very pleased with the outcome of the independent evaluation. Firstly, the StandBy Response Service was found to be a very effective mechanism for reducing the potential for suicide and adverse health reactions following suicide bereavement. The evaluator's report clearly showed StandBy has had a wide-ranging and positive impact on the regions where it operates. Taking the lead from this evaluation, we are keeping the basic components of the program whilst refining it on the basis of experience and new evidence as it becomes available.
Secondly, as StandBy is the first type of program of its kind in the world, the findings of the report provide an invaluable contribution to the knowledge base regarding what can work in the area of postvention. With the view to sharing our learnings from the evaluation, we sent the Executive Summary to a number of stakeholders, nationally and internationally. The responses included comments such the evaluation demonstrating that StandBy, along with the other postvention programs in Australia, are delivering world-class services and that StandBy was the postvention program with the strongest empirical support internationally.
Benefits of evaluation
One of the major benefits of evaluation is that you can ascertain whether your program may be doing harm or is in fact making a positive difference to the health and wellbeing of your target group and their community.
StandBy continues to be committed to evaluation and is currently commencing to evaluate a new area exploring the economic impact and outcomes of StandBy to determine the program’s economic contribution and cost-effectiveness.
Tips for evaluation
The most important points to keep in mind are:
- Consult widely with various stakeholders regarding the evaluation design
- Include the evaluation design in the project planning phase
- Be prepared to be thorough
- Communicate your reasons for evaluation to all those involved, emphasising the importance of this process
A full list of the agencies running StandBy in the various locations, and copies of the Executive Summary and full report are all available from the bottom of this page here.
Suicide Call Back Service: a perspective on evaluation
By LIFE Communications
This is an edited interview with Laura Kennan, General Manager Clinical Services Manager, Crisis Support Services which manages the Suicide Call Back Service.
The Suicide Call Back Service (SCBS) is currently engaged in the second round of evaluation. The first round of National Suicide Prevention Strategy (NSPS) funding and evaluation ran from May 2007 to April 2009; the second round will end in April 2011.
Planning for evaluation was incorporated in the overall project plan at the very beginning of each round. To assist with this we employed an external evaluator from the beginning of the project planning phase. The external evaluator was selected by interviewing three different evaluators who were chosen from a short list, and our decision was guided by both what each evaluator said they would do and the quote given for the evaluation.
In terms of deciding what to measure, and how, we were guided by the key performance indicators set out by the funding but we also chose to include some variables that we as an organisation considered important to assess. For instance, we were interested in hearing about the experience of the service from multiple perspectives. This included surveying consumers, external health professionals, the telephone counsellors who provide the call backs, the program leader and one of the executive management team members. We also wanted to explore gaps in service delivery pertinent to our service.
Responding to recommendations of first evaluation
The first round of evaluation provided us with valuable information when it came to applying for the next round of funding. The recommendations from the first evaluation report included the suggestion that we broaden our service to include a 24 hour crisis counselling component and offer call backs for professionals working with people who are at risk of suicide. We also now manage a moderated bulletin board for carers and those bereaved by suicide and e-therapy modules
for people caring for someone who is suicidal. Additionally we have started to use SMS for appointment reminders and forwarding referral information to clients.
ChallengesOne of the biggest challenges we experienced was getting in contact with some of the clients and professionals who referred to our service. To manage this we scheduled appointments with everyone who was to provide feedback about the service across two full days with the evaluator.
Another challenge that we experienced in the first round was missing data in some areas due to the fact that very few of the fields in the client database used by our staff were mandatory. There was also a lot of qualitative information that we received as part of the evaluation that could not be substantiated by the quanitative data as we had not collected data on these variables. We responded to that by including mandatory fields in almost all sections of the new client database and thinking more broadly about data that would be interesting to collect and evaluate as part of the second evaluation beyond the contract requirements.
Aside from these two challenges, we have overall found the evaluation to be a pretty smooth process due to the thorough planning, which was assisted by involving the external evaluator in each step of planning.
Benefits of evaluation
The greatest benefit of evaluation from our perspective is that we were able to see the positive impact we were having on those who came into contact with the service. The process also allowed us to identify gaps that we as a service could potentially fill.
Tips for evaluation
- Include designing your evaluation in the project planning phase, as it is much easier and will ensure that you have richer data for the evaluation
- Get feedback about the services from various perspectives (we included clients, referring agencies and SCBS staff)
- Use a combination of qualitative and quantitative measures. The quantitative data provides the big picture view whilst the qualitative data elicits more detailed information that might be otherwise missed. For example we would not have picked up on the need to provide a 24 hours crisis service without the qualitative measures included.
For more information visit the Suicide Call Back Service website.
Project evaluation mini-literature review
By LIFE Communications
This literature review highlights some of the program evaluations that have been published in peer-reviewed journals during the last two years. Whilst there is a large amount of literature published on risk factors for suicide, focus on what works in terms of suicide prevention remains in relative infancy. However a journal database search indicates that the publications of evaluations on specific suicide prevention programs are increasing.
The majority of the articles located for this literature review evaluate programs that aim to prevent suicide in the school-aged population. The articles reported here have used a number of models to prevent suicide, such as peer-support, raising community awareness, and developing professional support relationships. Additionally, they have used a number of different approaches to evaluate this work.
Life-Savers was a US program that aimed to both raise awareness of suicide within the adolescent population and assist them to develop the skills to support their peers when facing suicide (Walker, Ashby, Hoskins & Greene, 2009). This program was found to be effective in increasing awareness of suicide and increased a sense of commitment to prevent suicide after adolescent attendees completed the three day course designed for those based in non-metropolitan areas. This was assessed by administering the measures immediately before and after attendance.
A Queensland-based program targeting older men (Bartlett, Travers & Cartwright, 2008) had the same aims as Life-Savers, raising awareness of suicide within attendees and providing them with the skills to support peers as needed. The evaluation of this program was likewise conducted using pre- and post-attendance measures of knowledge, attitudes about, and confidence in providing support regarding suicide. What was different however, was that the latter study included an extra post-attendance evaluation phase undertaken 6 months after the training had been delivered. Both of the post-attendance measures indicated that overall participants felt better resourced, and that they continued to feel more confident in supporting others and raising awareness six months down the track.
The above two studies used individually applied measures of knowledge increase and attitude change over a fairly short time frame, both of which are indirect measures of potential suicide prevention/reduction.
However, another study (Zenere & Lazarus, 2009) used the population-based measure of reduction in completed and attempted suicides in the 15-19 year olds during the period of 1989 to 2006 to assess the effectiveness of a school district program. The Youth Suicide Prevention and Intervention Program is a comprehensive school-based district-wide model that aims to raise awareness amongst both students and staff about suicide risk, whilst also aiming to increase protective factors within students. Additionally, more targeted components of the program are aimed at providing support to individuals who are identified as being at potential suicide risk (including a crisis hotline for concerned staff to consult). The analysis indicated that there was a reduction in youth suicide within the school-district following the implementation of the intervention program, particularly in the first few years following the program’s implementation compared to the most recent years (the authors report that there was no information available on suicide attempts prior to 1989). The authors acknowledge that the difficulty of assessing which of the individual components may have been more effective in preventing suicide given that some components were latterly introduced.
Another comment not acknowledged by the authors (Zenere & Lazarus, 2009) is that with a population based evaluation, it is not possible to ascribe the suicide prevention program as the direct cause of suicide prevention within the population (Department of Health and Ageing, 2007b). Hopefully, some more specific evaluations about the programs, or ones that use individually-based surveys will be published in the future.
Another form of outcome measure that has been used in evaluations is the rate of participation. One example is the use of an interactive web-based method of outreach as a means of developing a pathway to ongoing professional support for college students deemed as high risk for suicide (Hass, Koestner, Rosenburg, Moore, Galow, Sedway, et al., 2008). The students of two US colleges were invited (via an email) to complete an online mental health survey that auto generated a depression and suicide risk report which was then sent to a campus counsellor. The counsellor then sent a detailed report to each participant with an introduction about herself and the services offered. For those respondents whose results indicated moderate to high suicide risk, the detailed report addressed specific concerns raised by the responses. All students were invited to engage in online counselling, but those deemed to be of higher risk were encouraged to make an appointment as well. All of the contact maintained anonymity via an ID and password. The results indicated that 62% of those deemed as being high risk and 38% of those rated as being at moderate risk saw a counsellor following the survey completion.
The studies cited above have used a combination of outcome measures and participant types. However, they have not included one of the designs considered the most rigorous within experimental research design; the inclusion of both ‘treatment’ group and a ‘control’ group. Whilst this type of research design is not always ethically possible in suicide prevention (Department of Health and Ageing, 2007a), the use of comparing those who have and have not received training could be a possible means for such a research design. Reis and Cornwell (2008) used this design to compare the effectiveness of a suicide prevention training program with school counsellors and teachers (the 'control' group was to receive the training at a later stage). The findings suggested that, as would be perhaps anticipated, the attendees demonstrated improved knowledge and confidence in applying this knowledge than those who were not trained. Additionally however the counsellors were found to be more likely than the teachers to use this knowledge following the training.
The above brief exploration of the existing research around program evaluation shows it is possible to appreciate that this aspect of suicidology is growing, with both the types of programs and the methods used to evaluate these programs being quite diverse.
References
Bartlett H, Travers C, Cartwright C (2008). Evaluation of a project to raise community awareness of suicide risk among older men. Journal of Mental Health 17, 388-197.
Department of Health and Ageing (2007a). A framework for suicide prevention in Australia. Department of Health and Ageing: Canberra.
Department of Health and Ageing (2007b). Research and evidence in suicide prevention. Department of Health and Ageing: Canberra.
Haas A, Koestner BS, Rosenberg J, Moore D, Garlow S, Sedway J, et al. (2008). An interactive web-based method of outreach to college students at risk for suicide. Journal of American College Health 57, 15-22.
Reis C, Cornell D (2008). An evaluation of suicide gatekeeper training for school counselors and teachers. Professional School Counselling 11, 386-394.
Zenere FJ, Lazarus PJ (2009). The sustained reduction of youth suicidal behaviour in an urban, multicultural school district. School Psychology Review 38, 188-189.
Comment: the importance of evaluation
Above: Rosemary McKenzie |
By LIFE Communications
Rosemary McKenzie is a Research Fellow with the Centre for Health Policy, Programs and Economics, School of Population Health at the University of Melbourne and Coordinator and teacher of Health Program Evaluation in the Victorian Consortium Master of Public Health program.
This sector comment is an edited interview Rosemary McKenzie, where McKenzie discusses the importance of evaluation of suicide prevention programs. She highlights key points to consider throughout the evaluation process, resources available to assist evaluation and why sharing evaluation experience and knowledge is so important.
The importance of evaluation
For all of us working in public health, including in suicide prevention, it is important to know that as service providers we are making a difference and that we are on track with our program activities.
There are still a lot of unanswered questions in the field of suicide prevention. This is because we do not have as much knowledge as we would like on what works to reduce suicide. However, undergoing evaluation can help us to ensure that there is a high level of quality in what we are delivering, that we are producing the outcomes we intend, and also give us confidence about our contribution within this complex field.
Often we seek to evaluate programs for a variety of reasons, including wanting to evaluate a program to demonstrate to our organisation or funder that they are investing their money wisely, and wanting to contribute to the broader evidence or knowledge base about suicide prevention.
Key points to evaluation
No matter what our reasons for conducting evaluation, when beginning evaluation of a suicide prevention program the most important thing is to clearly define what is being evaluated, why it is being evaluated and understand the specific questions to be answered.
A common pitfall often associated with evaluation is focussing only on finding out what works. Often it is not just what works that we are interested in, but how and why. This leads to using multiple methods of evaluation, and drawing upon literature and other studies which can offer great guidance when designing both a program and its evaluation.
Often it is not feasible for service providers to conduct evaluation programs individually. Scoping and developing a plan that matches your resources and intentions is very important in this respect. Hence, external evaluators can be an option for those wanting a professional evaluation that can be done quickly and skillfully. I recommend the Australasian Evaluation Society website, which has a list of evaluation consultants and examples of evaluation stories and reports.
Sharing evaluation experience is important
Because the area of suicide prevention is so complex there is never going to be just one intervention that reduces the suicide rate, it will be a whole array of prevention interventions and strategies that maybe over some years will start to have an impact on suicide rates. This reinforces how important it is that everyone tries to do some evaluation. Even the smallest of evaluation will contribute to a bigger body of evidence in which we might start to see trends and patterns, ultimately encouraging prospects for change.
By sharing evaluation outcomes, people can learn from others’ mistakes or identify where there will be real challenges in their own programs. It is certainly possible also to share program findings in an ethically sound manner by reporting sensitive material in a way that shares useful knowledge without identifying individual sources. It is absolutely paramount that personal confidentiality (of all participants) is protected. The omission or the use of pseudonyms in place of identifying details, such a people’s names, location names and social settings are common practices that aim to protect those who have participated in the program.
Resources to help guide the evaluation process
The evaluation process doesn’t have to be daunting. There are many excellent free online resources which can give local programs the added security needed when undergoing evaluation for the first time or when looking for tried and true evaluation methods. Some of the generic resources available include:
Ultimately the field of evaluating the effectiveness of suicide prevention interventions and programs is complex and finding all the answers is never going to be an overnight phenomenon, it’s going to be a long term task. This is why I would like to encourage people to view evaluation as an integral part of all suicide prevention interventions.
For more information email Rosemary or visit the Centre for Health Policy, Programs and Economics website.
Tomorrow is World Suicide Prevention Day. This year's theme is: Many Faces, Many Places: Suicide Prevention Around the World. Many events are happening across the country to raise public awareness of one of the world’s largest causes of premature and unnecessary death – suicide. To find an event near you, visit the World Suicide Prevention Day site.
On September 21 LIFE will be hosting an online chat session with one of Australia’s leading voices on youth suicide prevention – Jo Robinson. Robinson is a research fellow at Orygen Youth Health Research Centre and leads a research program specialising in youth suicide prevention. To register for the chat, email the LIFE team or call us on 03 8398 8422. More information.
The 12th edition of LIFE News will feature projects, people and comment focusing on community support services that work to prevent suicide. LIFE Communications encourages input and contributions. If you would like to contribute to this edition, please contact us with your idea.
Feedback and evaluation of LIFE News
In the spirit of evaluation and sharing findings, the LIFE team invites you to help us evaluate LIFE News. Please fill out our five minute anonymous online evaluation survey, the results of which will be shared here on the LIFE website. Alternatively, please contact us with any comments or suggestions.
We strongly encourage readers to use the LIFE site as a place to share information about suicide prevention program evaluation. If your suicide prevention program has been evaluated, then we’d like to hear about it – email us or upload your report or report summary to the LIFE Library. Remember you can also use the Professional Development Network discussion forum to post questions or share learnings from your program evaluation.