A Participatory Evaluation HOW TO: tips and tools for sensemaking, storytelling, and more

It was college reunion weekend at my alma mater (wahoowa!), and in addition to taking a few (literal) walks down memory lane, I thought a lot about feeling a sense of community and what magic combination of ingredients is needed to create that.

Shared experiences - especially those where you gain something of value and even more so those where you create something of value - are part of that secret sauce.

And participatory evaluation, when done well, checks all those boxes.

If you missed the last issue of Community Threads, be sure to check it out first, as it provides an overview of PoP Health’s participatory evaluation approach. With that approach in mind, today’s issue is diving into HOW we actually do participatory evaluation. Let's jump right in.

Can you share some tips for participatory evaluation?

Begin with the end goal in mind, and design your evaluation and learning process accordingly. What do community members, coalition members, and the evaluation and learning team most want to learn and why?

Consider how community members can co-own - and meaningfully benefit - from every stage of the evaluation and learning process, from initial brainstorming and planning through data collection and analysis through sensemaking and storytelling. How is each stage of the process structured to allow community members to drive or co-lead the process? How is each stage of the process designed to ensure community members walk away with new capacity, connections, resources, and supports?

Develop infrastructure for responsive feedback and continuous quality improvement. You’ve heard this tip before from me and I'm repeating it here because it can't be overstated and hardly anyone truly does it! Yes, a key goal of evaluation is to understand impact but an equally if not more important goal is to improve the work. It's vital to set up infrastructure (time, resources, systems) from the outset so you are flexible and nimble enough to implement course corrections and improvements in real time based on analysis of monitoring data.

Diversify and tailor your evaluation deliverables, and make them modular where possible. Here is a slide from PoP Health’s Evaluation 101 workshop that includes a range of possible deliverables through which to share evaluation results and stories (and there are many more beyond what's listed here).

Don’t limit yourself here, get creative! Data dashboards are all the rage these days, but I love Stephanie Evergreen’s take on them, especially for something community facing - make it a webpage instead. You can also tailor content to different audiences - we often do this via briefs/two-pagers, one for community members, one for policymakers, one for funders, and so on. They each care about different things. We have also had success with making our briefs modular - having “modules” (short sections of the brief) that can be toggled in or out of a brief depending on who needs X background information or who cares about Y data.

What are some specific strategies for participatory evaluation?

There are many, but here are a few I especially appreciate for the meaningful role community and coalition members play:

Community based system dynamics modeling: A key part of evaluation and learning in public health is understanding dynamic, complex, messy systems. In the case of our work on school mental health in DC, students, family members, teachers, school administrators, policymakers, and others may have completely different understandings of the school mental health system. So we engaged in what’s called Community Based System Dynamics, in partnership with the Social System Dynamics Lab – this process uses participatory group model building approaches to explore the system in question. We held modeling workshops with students, with caregivers, with teachers, and with our multisector Stakeholder Learning Community. During the workshops, groups huddled around large sheets of paper, discussing, writing things down, crossing things out, drawing arrows, and so on. They produced a series of causal loop diagrams, which I then synthesized and integrated into this version of our systems map. Much more on this in a prior PoP Health newsletter here.

Data placemats for collective sensemaking: Community members and coalition members have experiences, expertise, and perspectives that lead them to insights your evaluation team, program team, and funders are apt to miss entirely. So don't make the mistake of leaving them out of the conversation. I love using highly visual data placemats and data posters to bring coalition members and community members into the process of making sense of data and drawing insights - about what we've learned so far, how to continually improve the initiatives we work on, and what other information we need to gather moving forward. We’ve recently used data placemats (during a coalition-wide data sensemaking session) and data posters (during a community-wide symposium) to share initial data from evaluation of the BIRTH Plan with our community and coalition partners in Pittsburgh, pairing the visuals with discussion questions that help elicit their thoughts on what they take away from the data, their insights about how to improve the work, and what additional information they’d most like to see in the future.

Sharing personal narratives: Nothing is quite as powerful as a story in someone’s own voice. Capturing personal narratives of community members/program participants/those influenced by a policy is a vital participatory evaluation strategy. In addition to focus groups and interviews, there are many creative ways to do this, including video journals, audio diaries, photo voice, and more. I especially love this idea of a participatory video process focused on stories of significant change. Participants are given the chance to take part in a Participatory Video process at baseline; stories of Most Significant Change are collected via structured story circles at midline; each circle selects one story to record on video. A participatory analysis identifies themes and recommendations. Given consent, videos can be shared so stakeholders learn directly from participants’ stories.

Community collaboration strategies: The community collaboration strategies we have featured previously (i.e., focus groups in a box, data walks, and street stalls) can also all be used in the context of evaluation and learning.

What are some resources to help me engage in participatory evaluation?

I’d like to leave you with a few resources I’ve found especially practical and useful in terms of participatory evaluation. As always, drop me a note to share other helpful resources or tools you’ve come across!

• ​Participatory Evaluation: Theories + Methods for Remote Work(a guide I refer to regularly for meeting facilitation ideas, whether virtual or in-person!) from Evaluation + Learning Consulting

• ​Community-Driven Data and Evaluation Strategies to Transform Power and Place (hot off the presses from Build Healthy Places)

• ​Participatory Evaluation Toolkit from Health Nexus (which provides a great overview of 7+ specific techniques)

As always, we share these tips, strategies, and resources in the hopes that they help you understand HOW to engage in participatory evaluation.

Participatory evaluation is likely going to be messier, slower, and more expensive than more traditional evaluation approaches. But on the flip side… You’ll be driven by community. You’ll learn more. You’ll create greater and more sustained change in your community.

The pros definitely outweigh the cons in my book - what about for you?

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!

The 'P' in our CAPE: Participatory Evaluation - by the community + for the community

A recent book club discussion about the challenge of adult friendships in the present day has had me thinking a lot about the importance of a sense of community.

On that note, a portion of a recent conversation between two of my favorite thinkers, Surgeon General Vivek Murthy and writer Anand Giridharadas, really struck me. Surgeon General Murthy shared that statistics show that people now are working more - “and parents are parenting more, even though they’re also working more.” He asks, “Where is that time coming from, that extra time? You put all this together and that time that is eroding is a time that we spend in person with family and friends, the time we spend for ourselves, and the time we spend for our communities.”

On the heels of a community event I helped plan, I’ve also specifically been thinking a lot about what it means to do something “by the community, for the community” (our event tagline).

What does all this have to do with the focus of today’s newsletter, participatory evaluation (aka the ‘P’ in PoP Health’s CAPE)? Let’s get into it!

What do you mean by participatory evaluation?

Simply put, participatory evaluation is monitoring, evaluation, and learning - by the community, for the community.

As more formally defined in this guide, “Participatory evaluation is not top-down or expert-led. It is a bottom-up framework that stimulates and utilizes the wealth of experiences and wisdom that participants have to create more meaningful, productive, and engaging discussions and debates.”

When it comes to the work of community coalitions and collaboratives, we want coalition members as well as community members integrally involved in every stage of monitoring, evaluation, and learning (or MEL, as it’s often called), to the point where they co-own the process, alongside their MEL team.

The idea that participatory evaluation encompasses monitoring, evaluation, and learning is key. Monitoring gets at the idea of continually examining whether things are implemented as planned, and applying what is learned to make improvements along the way. Evaluation gets at the ultimate question of “did it work”. And Learning underscores that the point of all of this is to learn together, and apply what we learn to make things better moving forward.

What are some ways to think about participatory evaluation?

As with action planning, there are hundreds of evaluation frameworks out there.

So instead of sharing or dissecting all of those, I’m going to share PoP Health’s approach to participatory evaluation + one evaluation framework specific to participatory evaluation that we have found helpful.

PoP Health’s Approach to Participatory Evaluation

Community-Driven and Co-Creative: Ensure participants share ownership of the evaluation process. Involve coalition and community members, centering and amplifying their voices in every stage of the process, from developing the plan, evaluation questions, and data collection approaches through interpretation and dissemination of results.

• Equity-Focused and Inclusive: Engage voices that have been historically excluded, emphasizing collective strengths, and maintaining a focus on upstream, root causes of health inequities. Build belonging and civic muscle through an evaluation process that helps participants develop their power to shape their world.

• Taking a Systems-Level Lens: Recognize the powerful role of policy, systems, and environmental change, and make every effort to capture and learn from the impact of changes at those levels. Ensure your evaluation process is also reflective of the dynamic behavior, complexity, and interconnectedness of systems.

• Value-Adding: Build upon ongoing activities and utilize the wealth of existing data and efforts, taking care not to waste time or resources reinventing the wheel. Identify where there is unique value to be added and focus evaluation efforts accordingly.

• Actionable: Generate relevant evidence and translate that evidence into key takeaways and concrete steps that can be taken to continually improve. Build upon assets and facilitators, address challenges, and proactively pursue transformative change that is guided by the experiences, stories, and voices of participants.

CoAct's Principles of Co-Evaluation

CoAct is focused on Citizen Social Science, which they define as follows:

Citizen Social Science combines equal collaboration between citizen groups (co-researchers) that are sharing a social concern and academic researchers. Such an approach enables [us] to address pressing social issues from the bottom up, embedded in their social contexts, with robust research methods. We aim to co-create socially robust knowledge.

They offer six principles of co-evaluation, each of which is further defined and paired with practical recommendations here.

What questions should I be asking myself about how we engage in participatory evaluation?

• What will coalition members and community members gain through the process? What new understanding, capacity, connections, resources, supports, etc. might they walk away with? ​

• How can we meaningfully co-own each stage of the monitoring, evaluation, and learning process with community members, while also being respectful of their time and other constraints?

• How can we make sure our continuous monitoring is part of a feedback loop that feeds directly into making concrete improvements?

• How do we best capture the impact that matters most to each of our audiences - community members, coalition members, policymakers, funders, and so on?

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!

A non-cringey survey that helps coalitions understand themselves

Today, I’m going to talk about surveys.

Does that make you cringe?

I understand if it does. Surveys get a bad rap.

We’re often trying to get participants in a program or members of a community to take a survey, and - often - we don’t provide enough context for them to buy into the value of the information they’re sharing or for them to believe any changes will result from their input. To be fair, and to make matters worse, oftentimes there aren’t changes made as a result of survey responses.

There is a time and place for participant/community surveys, we certainly use them in our work here at PoP Health, and we have some thoughts on how to make them less painful and more impactful - more to come on that in a later newsletter.

But today, I’m here to talk about a kind of survey that doesn’t have quite as many potential pitfalls. A non-cringey survey, if you will.

This is a survey that coalitions and collaboratives can use to reflect on themselves. For lack of a better name, we call them partnership surveys.

These surveys are solely for members of the coalition, and the results help coalition members understand who they are and what they bring to the table, where they already have a lot of energy and expertise to move things forward, and where they may have gaps that need to be filled.

As per the latest guidance from the National Association of City and County Health Officials (NACCHO)’s, by way of their Mobilizing Action through Planning and Partnerships 2.0 process, local health departments are encouraged to begin their community health improvement assessment process with exactly this kind of survey (they call it a Community Partners Assessment).

I mentioned this briefly in the last issue of Community Threads, but we used this kind of survey in advance of our kick-off meeting with the Perry County Health Coalition in Pennsylvania earlier this month. At the meeting, we shared the survey results - which painted a picture of:

• Top action planning priorities;

• What members hope to accomplish via participation in the coalition;

• Current coalition composition, sectors and topics represented, populations touched by coalition member organizations, and suggestions around organizations, individuals, and communities to engage (either directly as coalition members or at other levels of engagement);

• Alignment between member organization focus areas and focus areas identified in the 2022 health assessment the coalition completed;

• Current and desired levels of engagement in the coalition;

• Activities and tasks members are most interested in supporting;

• Desired meeting format, frequency, and scheduling.

As you can imagine, by gathering this information ahead of the meeting and then sharing the synthesized responses and drawing out the key takeaways, we were able to facilitate a much more meaningful, nuanced discussion about where to go from here.

And since we were only surveying members of the coalition - who were already bought into the work of the coalition and wanted to actively use the results to inform their work - this survey rated pretty high on the non-cringey scale.

Have I convinced you that partnership surveys are non-cringey? Feel free to disagree with me (or share other examples of non-cringey surveys!) - drop me a line!

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!

Leveraging the power of in-person gathering

I’ve been thinking a lot about the power of in-person gathering on the professional end as well. After a long summer vacation, this Fall has been a period of reconvening in person.

Last month, we had an in-person kick off for a DC School Behavioral Health Consortium that drove home the energy and sense of community that can come simply from sitting around the same table. I also loved the icebreaker the meeting conveners used to set the stage for our time together - they shared a collage of different images representing collaboration and asked each person, as we went around the room for introductions, to share which image resonated most with them and why. What a great way to orient the group to the reason we had come together and unearth some key distinctions and nuances in how each person at the table might approach this work. We’ve gotten used to “setting the stage” activities in virtual spaces, but they are just as valuable in in-person gatherings.

Last week, we had our second annual set of Ignite presentations and data sensemaking sessions for our work (in partnership with the Center for Health and Healthcare in Schools at GW) as the Monitoring, Evaluation, and Learning partner for an antiracist, whole child development initiative in DC schools. Although this kind of sharing of our work and making sense of data visuals could, technically speaking, be done virtually, there is something about the power of being seated in a circle and lending our full attention to the discussion at hand that is hard to recreate on Zoom. There is also perhaps something even more powerful about the “sideline” conversations that occur between partner organizations during an in-person gathering, especially one that includes time to share meals together and catch up informally, as we did on both days of this convening. One of our partners emailed us after the event, “I wanted to thank you all for such an impactful convening this week. It was really inspiring to be around such a thoughtful group of educators/advocates and to be able to more deeply connect the dots across the entire project. I was even able to leave with immediate steps to further collaborate with [two of the other partner organizations on particular projects].” This is exactly what we hope can arise from bringing people together.

And finally, later this week, we are kicking off our partnership with the Perry County Health Coalition in Pennsylvania with an in-person convening of the coalition - the first time they’ll be gathering in person since 2019! So we’re trying to be thoughtful about how we make this brief time together as meaningful as possible. One thing we did is send around a survey and conduct a few interviews several weeks in advance of the meeting. We’ll be sharing our findings from the survey and interviews at the meeting as a launchpad for further discussion - this approach allows us to get the more obvious and consensus-y ideas shared quickly, leaving more of our precious in-person time to navigate the areas where there is complexity, uncertainty, or disagreement.

None of this is to say there is not unique power in virtual gathering - here at PoP Health, we love facilitating virtual convenings and have lots of tips and tools to make that time meaningful, engaging, and productive.

It’s just that this past month in particular, we’ve also been thinking a lot about the power of in-person gathering and how we can leverage that.

Reader, curious to hear your thoughts - in your experience, what’s the unique value-add of gathering in person? And how can we best leverage the power of those (sometimes rare) opportunities to be together in the same physical space? Drop me a note!

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!

How to keep people engaged in community health improvement efforts

Have you ever struggled to keep people engaged in a community health improvement effort (or any effort, for that matter)?

I work with a lot of community coalitions and collaboratives, and most of them are volunteer-based, which makes this question of how to maintain engagement especially important. But even those that compensate their members for their involvement need to consider the question of how to keep people involved and inspired and motivated to participate and act in meaningful ways.

In my line of work, with public health focused coalitions, people are generally at the table because they care - they care about the health of the community and they want to help make it better. But there are so many services and supports to coordinate and so many partners to collaborate with; it’s hard to know where to start; and there is simply not enough time in the day, especially since most coalition members have their actual day jobs to do.

Whether you are recruiting members or trying to hold onto the ones you already have, what’s the secret to keeping folks engaged?

This week, I came across an interview with Arthur C. Brooks (a Harvard professor who studies, teaches, and writes about happiness) about “the secrets to happiness at work,” and I think it’s spot on and extremely relevant to this question of how we keep coalition members engaged in the work of the coalition.

Here’s what he says on this topic: “The two parts of work that really matter [for happiness at work] are not title, not money, not admiration, not power, [but] earned success and serving other people.”

Here’s how he describes these two concepts among workers: “They feel like they’re earning their success, which is to say that they’re creating value with their lives and with their work lives, that their accomplishments are moving the needle and they’re being recognized for those accomplishments. And number two, they feel like they’re serving people so that they’re needed.”

Do these two concepts resonate with you?

They did for me - when I think of the jobs I’ve been least satisfied with and the ones I’ve loved the most (including what I do now!), earned success and serving other people do seem like distinguishing factors.

In this interview, Brooks goes on to say, “The number one thing that you can do for recruitment, for retention, the ultimate rewards that go far beyond money are making sure that you have a system where people are earning their success through their merit and personal accomplishment. They know it, they see it, and so do their friends. And they actually feel like they’re serving other people and they can see the faces of the people for whom they’re creating value.”

And this, I think, is what community coalitions and collaboratives (and public health organizations and agencies more broadly) should be thinking about. How can we give people clear roles and responsibilities, where accomplishments they are striving for are clearly defined and celebrated once achieved? How can we make meaningful connections between the members/employees of our coalitions/organizations and the community members we serve, so the value we’re creating can be truly seen and felt?

These are the questions I leave you with today- I would love to hear your thoughts!

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!

Hearing vs. Listening to community

You want to transform health in your community. And you recognize that can’t be done without communities in the center and in the lead.

But how do we actually DO that?

We’ve talked about meeting community where they literally, physically are and these strategies are great to hear what they have to say, but today, I want to push us a little further. Because there is a HUGE difference between hearing and truly listening.

Merriam-Webster defines hearing as “the process, function, or power of perceiving sound, specifically: the special sense by which noises and tones are received as stimuli”.

On the other hand, the definition of listening reads, “to pay attention to sound; to hear something with thoughtful attention: give consideration”.

There’s a lot of lip service being paid to community voice these days, as well as genuine efforts to elevate community voice. But either way, community voice isn’t enough if we merely hear it, but don’t listen to and act on it.

The “act on it” piece is vital - I’d take the definition of “listening” and argue for us to take it one step further than paying thoughtful attention and giving consideration to actually internalizing and acting on what community has to say.

I’ve had the pleasure of working with the Allegheny County infant health equity coalition over the last couple years, and we’ve been incredibly intentional about trying to do this. What does this look like in practice?

• The coalition is composed of moms, doulas and other birth workers, nutrition advocates, community leaders, as well as nonprofit, healthcare, and government leaders;

• We used strategies like focus groups in a box that put community members and leaders in the driver’s seat of leading discussions with community members;

• As we continue to gather community input (as we just did a couple months ago during a community kick off event for implementation of the action plan), we reflect back what we’re hearing to community in ways that are easily digestible (you can see our summary of input from the kick off here);

• We directly link the input we receive through these strategies to our action, implementation, and learning/evaluation plans and we make these connections explicit (for example, you’ll see we quoted community members directly throughout our action plan to demonstrate the connections between what we heard from community and what ended up in the action plan);

• We’ll be capturing community feedback in learning and evaluation activities that directly ask them whether they felt listened to and their input acted upon - and we’ll modify our approach to reflect what we learn.

How are you and your coalitions/organizations working to go beyond hearing from community members to actually listening to them? Drop me a line and let me know!

Sign up to receive future newsletters directly in your inbox at www.pophealthllc.com!