Action For Child Trauma International (ACTI) Measurement, Evaluation, and Learning Framework

Approved December 2021

Introduction

The goal of this document is to outline a framework for the measurement, evaluation, and learning (MEL) of the action (in terms of training activity) of ACTI. This framework can be applied to multiple contexts. This allows for uniformity in ACTI evaluation, allowing retroactive and lateral comparison. 

The key point of this document is that, when evaluating the actions of ACTI, the scope of evaluation should not be limited to the action itself (eg a CATT training course) and its direct output (eg participants trained in CATT) as this would not capture the entire scope of the impact of the action. ACTI needs to not only evaluate itself for its direct output, ie evaluating the effectiveness of the training, but also the broader outcomes of this training. The value of the former is to improve the training and become more effective, while the value of the latter is the impact on the child's wellbeing and that of their family and community. 

The terminology that is used in this document is specific to MEL and will be expanded. Action refers to the work or operations of ACTI (ie the training that ACTI provides). Outputs are the direct outcome of the actions of ACTI. Outcomes are important but this is not the true value of ACTI’s actions. The Outcomes, which are the effects of the Outputs, is where ACTI can see the impact the actions have on the wellbeing of individuals.

This principle is displayed below: 
1. Action - CATT Training ➡️  2. Outputs - Participants trained in CATT ➡️ 3. Outcomes - Improvement in the wellbeing of children in the context of their families and communities

Summary

Each step follows a similar pattern. It ensures that ACTI evaluates both the effectiveness of its actions through measurement of the output and the impact of its actions through its outcomes. Both of the evaluations can be accomplished effectively through surveys completed by the participants, producing both quantitative and qualitative data which can contribute to a cycle if improvement. Therefore surveys carried out by the participants is an adequate measure of the impact on the wellbeing of the children treated. If there was a need for data to be produced for external consumption this would need more rigorous data collection. 

The data from each level of the process informs both future projects at that level and follow-up work. For example, output data highlighting areas of success and improvement in TRT training may also provide data for the selection of individuals for further training, such as CATT.

Action Output Outcomes
TRT Training Participants are trained to …XXX

Output measured: How effective the training was from the perspective of the trainees 		Participants take this knowledge into the environment in which they work and use their new skills to help children. This leads to …
Outcome measured:
Reduction of anxiety symptoms in children and improved level of understanding in the community about anxiety and its effect on the children of this community
Feedback Immediate feedback Goal
(Output measured):
Understanding the effectiveness of the mechanics of the training, relating only to the immediate action of ACTI.

Core Question: How to improve the training? 		Extended Feedback (Outcomes Measured):
This feedback will focus on the impact that the training had on the participant and their work. Specifically, what was the impact of the training on children and the community? (eg: The impact on children in school settings, the change in understanding within families and across the community.
Core Question: What is the impact on the wellbeing of children, their families and communities?
How it is produced:
Through feedback of the participants about their experience and the efficacy of the training from their perspective.
Metrics: Quantitative and Qualitative
The direct feedback, potentially in the form of surveys, of the participants. The focus being on the individual's experience, what they gained from it and how the training could be improved. This is currently being produced through feedback forms. 		How it is produced:
This will also be through feedback forms, completed by the participants, that produce data on the impact of the training.
Metrics:
This data would be produced through audit reporting with data submitted by the trainees and analysed. May include hard data on clinical outcomes.

Audits & surveys produce both qualitative and quantitative data that can get answers to specific questions while giving the participant to expand on their experience in a guided way.
Value:
This can lead to an improvement of the training to be more effective. 		Value
  • Demonstrates the effect of TRT training on the community
  • Serves as an indications of those who should continue to another type of training
    • Action Output Outcomes
    CATT Training Participants, chosen by Training Partners or based on the feedback from earlier training, are trained to …XXX

    Output measured: How effective the training was from the perspective of the trainees?     		Participants are capable of providing CATT to individuals which are suffering from PTSD …
    Outcome measured:
    Reduction of symptoms among children treated, and long-term improvement in their wellbeing. Also the impact of greater knowledge of PTSD on the wellbeing of families and communities.
    Feedback Immediate feedback Goal
    This mirrors that of TRT training. Only focusing on the mechanics of the training itself.     		Extended Feedback (Outcomes Measured):
    This will examine the impact of CATT on the children through the use of recognised measurement scales eg CRIES-8
    Has it changed behaviour, inclusion, or wellbeing and how?
    How it is produced:
    Please see Feedback for TRT
    Metrics: Quantitative and Qualitative
    Please TRT training Immediate feedback.     		How it is produced:
    CATT Counsellors produce qualitative and quantitative data based on the children that they have treated, using measurement scales and family/community reporting.
    Follow-up reports from trainers/supervisors
    Qualitative interviews conducted with new CATT counsellors
    Metrics:
    This data would be produced through audit reporting with data submitted by the trainees and analysed. May include hard data on clinical outcomes.
    Value:
    This can lead to an improvement of the training to be more effective.     		Value
    1. Demonstrates the impact that training participants in CATT for children, their families and communities.
    2. Allows for ACTI to determine who has displayed the capacity and understanding required to become a trainer
    Action Output Outcomes
    Training of Trainers CATT practitioners learn how to become CATT trainers

    Output measured: How effective the training was from the perspective of the trainees?     		Cost effective in-coutry rollout of training of all kinds
    Outcome measured:
    Can local people deliver effective training with only remote support/funding
    Feedback Immediate feedback Goal
    This mirrors that of TRT training. Only focusing on the mechanics of the training itself.     		Extended Feedback
    This will evaluate the outcomes of the training courses delivered by local trainers
    How it is produced:
    Please see Feedback for TRT. In addition, observation of teaching practice and reports from those trained by new trainers
    Metrics: Quantitative and Qualitative
    SEE TRT training Immediate feedback.     		How it is produced:
    Reports from trainers who follow up/supervise trainees regularly External reviews/Social Impact Assessments
    Metrics:

    This data would be produced by a range of methods depending on local conditions and accessibility.
    Value:
    This can lead to an improvement of the training to be more effective.     		Value
    1. Demonstrates the impact that training participants in CATT for children, their families and communities.
    2. Allows for ACTI to determine who has displayed the capacity and understanding required to become a trainer