Reflection research shines light on decolonising evaluation

A unique reflective practice project offers new insights into decolonising the practice of development evaluation, in a partnership between GraceWorks Myanmar (GWM) and two independent evaluation consultants.

The work is based on a three-year evaluation of GWM’s community development education (CDE) and peacebuilding program in Rakhine State, engaging with 24 villages.

Initially led by Dr Leanne Kelly, a respected evaluation specialist, she began collaborating part-way through with an emerging evaluator from Myanmar, Ms Phyo Pyae Thida (Sophia) Htwe, who has temporarily been in Australia undertaking advanced tertiary studies in international development.

Their work recognises that, while the development sector seeks to ‘do itself out of a job’ by strengthening local leadership and action, evaluation has typically remained a foreign-led and highly colonising exercise where priorities, methods and quality are judged by people who are culturally, and usually physically, removed from where the development work is taking place.

They are showcasing why actions such as translated surveys, local data collection and local capacity building, while positive, are insufficient in light of complex entrenched barriers and assumptions.

Their work is challenging what ‘good’ evaluation is between local priorities and donor/non-government organisation (NGO) perspectives.

The three-year evaluation comprised three reports, the most recent of which was designed around decolonising evaluation as well as managing unexpected constraints from COVID-19 and Myanmar’s military coup.

The final two reports include in-depth interviews with in-country NGO staff, conducted by Ms Htwe; interviews with the CDE program’s 12 village facilitators, conducted with local Rakhine researchers; and focus groups engaging dozens of village residents, facilitated by the local Rakhine researchers.

Moving toward locally led evaluation

For years, Dr Kelly’s evaluation practice has been shifting toward community-led evaluation and supporting community representatives as a facilitator and ‘critical friend’, more than an evaluator, to craft locally appropriate methods for evaluation.

“GWM’s work has been emphasising locally led solutions for locally identified problems for years, so it only made sense to transform its evaluation approach to match those community-focused values,” Dr Kelly said.

“People can be overly impressed by academic language, consistent formatting and author accolades,” she said.

“These tricks can cause people to dismiss less academic-looking work and frame it as ‘less trustworthy’.

“We need to value local voices and opinions, and note that their knowledge on the subject of NGO effectiveness in action is deeply experiential and valid, and whatever way they wish to share that insight is worthy of our attention.

“While evaluation can probably never be fully decolonised, we need to unlearn some of the rules and assumptions of evaluation and go on an authentic journey of discovery with an open-heart.

“I’d love to see development evaluation transformed from a focus on upward accountability benefiting donors and executives, to downward accountability benefiting communities and nurturing and promoting local ways.”

Balancing power and building capacity

Ms Htwe describes working with Dr Kelly as rewarding, having been encouraged to think outside the box with her own evaluation practice and raise ideas for improving the process. It’s given her the confidence to lead future CDE evaluation.

“Decolonising evaluation is a very new concept even to people working in local community service organisations (CSOs) and NGOs in Myanmar, to the point where something that seems harmless, like writing findings and reports in English, might be overlooked as being colonising,” Ms Htwe said.

“I feel like sometimes foreign people do not quite understand, and are not trying enough to understand, the local context or be willing to give up their power,” she said.

“Western evaluation methods and frameworks are not perfect. Local people need to create evaluation that is suited to their local culture and lead the change.

“Shifting power dynamics globally might also help to encourage two-way learning – not only local people learning from Western education, but Westerners learning from local people.

“That will be a harder shift. Westerners have not had to sit in the position of learner, and local people often don’t believe in their ability to work their own way or in the value of sharing their experiences and opinions.

“Decolonisation is very much about balancing power. There is a long way to go, and a lot of capacity building needed, to help people at a local level start to see why it matters.

“Academics and practitioners need to work on this together and local people need to raise their voices.”

Reflecting on a path forward

Decolonising evaluation improves accountability, meaningfulness, usefulness and value to all stakeholders, most importantly communities, by providing more accurate and insightful local analysis of the value and effectiveness of an NGO’s approach – in this case GWM’s approach.

GWM’s Chief Executive Officer, Peter Simmons, sees the work as encouraging an exciting shift in the thinking and doing of community development.

“Through Dr Kelly and Ms Htwe’s research, we’re seeing how evaluation can become part of the way communities reflect, learn and move closer to their own goals,” Peter said.

“Our CDE work already includes village-level monitoring and evaluation, using ActionAid’s village books, so pairing that with a decolonising approach to external evaluations will increasingly build local ownership and empowerment,” he said.

“The standards we evaluate ourselves by should first be those defined and prioritised by the communities we exist to serve, so this work is, in a way, starting from where we normally finish.”

GWM’s evaluations have already advanced in areas such as local data gathering.

Key findings and recommendations from an interim evaluation report were also turned into a visual poster in Burmese for the CDE villages, created with low literacy in mind and supported by discussions with village facilitators.

Future improvements include participatory analysis to examine data and explore priorities and recommendations through a local lens, and broadening poster translations to include Rakhine and Rohingya languages.

Dr Kelly and Ms Htwe’s work is championing local people as active participants in determining what is evaluated, when, by whom and how, privileging local values and culture, and local processes led by local perspectives.

This includes working with communities to identify their goals and success indicators, making the work of decolonising evaluation one of decolonising project design and delivery.

The act of writing together about decolonising evaluation has itself helped Dr Kelly and Ms Htwe identify ideas for the future they may not have considered alone – a template for co-creating evaluation and shifting historical roles of experts to those of facilitators and critical friends.

For GWM, we see this as a powerful step toward holistic local ownership of community development.

---

For more information

• An academic article, titled Decolonizing Community Development Evaluation in Rakhine State, Myanmar, is forthcoming.

• To find out more about this subject, please contact Leanne Kelly at Deakin University on kelea@deakin.edu.au or Sophia Htwe at University of Melbourne on phtwe@student.unimelb.edu.au