Data x Policy x Outcomes: A Conversation with Aaron Maniam, Eleanor Carter and Leon Lim

ETHOS Digital Issue 12, Apr 2024

PARTICIPANTS:

Dr Aaron Maniam is Fellow of Practice and Director, Digital Transformation Education at Oxford’s Blavatnik School of Government

Dr Eleanor Carter is Research Director for the Government Outcomes Lab (GO Lab) in Oxford’s Blavatnik School of Government

Leon Lim is a Research Fellow in the Governance & Economy department at the Institute of Policy Studies

[see full bios below]

---

Tell us about your current work and how it relates to evidence-based policymaking.

AARON: As Director for Digital Transformation Education at the Blavatnik School of Government in Oxford, part of my job is to think about how governments use, regulate and enable technology more effectively.

There are three particularly strong intersections between the digital space and evidence-based policymaking. First, digitalisation is in many ways synonymous with being more data-driven. Digitalisation means we have access to far more data, both qualitative and quantitative, than we have ever had before. There is both a lot of noise, and more evidence that we can use. So governments need to learn how to harness this abundance and have the literacy to sift through the noise and discern substance from it.

Second, governments want to know what actually works or does not work in the digital transformation process that public agencies go through. We need evidence-based approaches to determine this more clearly.

The third aspect has to do with what competencies are needed in government in order to digitise well: I based my PhD on this! For instance, the more we outsource digital transformation, the more we need the right skills inside government to use and discern what is going on in the digital space. A key set of skills here involve data literacy and analytic literacy—understanding what is out there and what we can use to inform decision-making in more effective ways.

ELEANOR: I am Research Director at the Blavatnik School’s Government Outcomes Lab (GO Lab). We are a research centre that investigates how government partners with the private and not-for-profit sectors, in particular to improve public service delivery. Our work is very much applied research—we produce scholarly work, but we also serve as a bridge to government agencies, supporting them in partnering better.

After over 30 years of using different forms of public-private partnership, what we can say about what works is actually still very limited. So part of the GO Lab’s mission is to develop the evidence base around government partnerships, and developing an understanding of how the government can appropriately partner. This includes thinking about using an outcomes focus in a way that improves cross-sector partnerships.

An outcomes-oriented framework, either within government or in the way the government works with partners, could energise the ways in which different forms of data and evidence intersect with decision-making across the full policy-development process, in a positive feedback loop.

LEON: The Institute of Policy Studies (IPS) strives to be a trusted research partner for the Singapore Government. We are starting an initiative called SCOPE (Social Compact, Outcomes, Partnerships and Engagement), which aims to encourage the non-profit sector in Singapore to become more outcomes-oriented, by fostering a virtuous cycle where the various parties in the non-profit ecosystem—funders, sector developers, non-profit organisations, etc. continuously learn from one another, turning new insights into better implementation and improved social impact. In this way, we can encourage more participation between people, private, and public sectors to tackle social issues as part of a refreshed social compact.

What is evidence-based policymaking to you? What constitutes evidence in the context of policymaking?

AARON: To me, evidence-based policymaking is grounded in testing and experimentation. It is finding out what really works in practice. Ideas should be proven to work not just in one or two places, but across a whole range of different settings. For example, we should not assume that just because something has worked in New Zealand or Estonia that it will work elsewhere.

Related to this is finding out which are generalisable principles, and which are the more context-specific ideas that may have to be adjusted for culture, tradition and different locations. A lot of this boils down to actual practice, that is well documented and well coded. And where we do not know something, we test things out first through experiments, pilots, or talking to people.

I do find that there is a bias in favour of quantitative, hard, tangible evidence in policy circles. There is going to be bias in quantitative evidence also, of course. Sometimes the best thing we can do is to identify broad trends illuminated by quantitative data, and then also test them with smaller scale stories and individual narratives that come out of ethnographic and qualitative data. I do hope we as researchers can help address this bias, and that governments start to broaden the repertoire of data they use.

ELEANOR: For me, evidence-based policymaking is about working in a way that integrates and absorbs research and evidence. As Aaron has pointed out, there needs to also be a deep contextualisation and appreciation around what works. A lot of evidence-based policymaking is often skewed to the top of the Maryland Scale,¹ and we end up inappropriately mimicking evidence-based medicine by being fixated on trial evidence and intervention.

What we understand as evidence in policymaking does need to be more well-rounded and refined, rather than taking it as meaning only data from randomised controlled trials.

LEON: I agree on what Eleanor and Aaron have said about evidence, and I am concerned about how we can marry both qualitative and quantitative approaches. For example, a big issue with using quantitative methods in the non-profit sector in Singapore is that many interventions have very small sample sizes, which limits the usefulness of the results from such methods. On the other hand, qualitative methods give you rich detail but also raise questions about whether findings can be generalised. I believe what we need are new syntheses of quantitative and qualitative methods. We have the range of technical expertise to do these in Singapore, but there needs to be more platforms and opportunities for people from different parts of the ecosystem to come together and talk about how to combine these approaches to create useful, practical evidence in cost-effective ways.

There is a real hunger for more information and better evidence of what is going on, and what is working. That said, what works for government may actually be quite different from what works for the non-profit’s point of view.

In the day-to-day firefight of policymaking, it can seem that acquiring the depth and diversity of data to improve policymaking is a luxury. How can governments strike a good balance?

ELEANOR: Policy work will always feel time crunched, and there will always be a gap between the policy decision timeline and the research timeline. That said, I am interested in having policy teams in government think about the questions that keep coming back up in a particular domain. What are the research assets we could be cultivating towards these questions, so they are ready to be drawn on when needed? It is important to clarify whose role it is to be evidence-based within government. It is not enough to have a few standalone initiatives or centres diligently compiling, distilling and articulating high quality evidence in a particular domain if it does not connect with broader stakeholders and policy teams.

One idea is to have systematic reviews that are continually updated, and available as an easily navigable resource to tap on when teams need to pull evidence together quickly to respond to urgent issues. This is where machine learning could be a powerful tool to use to help rapidly sort through the data, particularly when the evidence landscape around a particular theme is extensive and fast moving. At GO Lab, we attempted to do this with the massive corpus of papers on outcomes-based contracting, for instance, to surface say the top five relevant papers published in the past 10 years.²

AARON: As more people use ChatGPT or its counterparts, we need them to be accessing far better-quality data than they are doing now. Systematic reviews can be the basis for such AI tools to draw on.

There is also a question on how we organise this work: as Elle points out, to clarify who needs to be doing good work on data, in order for the system to do well. My current view on this is informed by data analytics and also the foresight work I used to do.

We all know a system needs these skills and these capacities. But it does not work to just vest these in a single entity. If you put them into a Ministry, they will only do that Ministry’s sectoral work. If you put them at the centre of government, no domain agency is going to do any deeper work. So the question is how the centre and the periphery of government end up balancing with each other in providing these “public goods for public service” as I like to call them. And what I would say is, some people must be doing this work all the time. And all the people must be doing it some of the time.

When it comes to evidence, this means you do need dedicated teams. This is not a random project one does for six months and then sets aside. Someone has to be doing it all the time.

But then everyone must also be doing it some of the time, which means they must learn to be literate consumers. They do not need to be experts at producing or churning the data, but they have to be literate enough to know how to discuss or make use of it. I think this balance gives us a much healthier system overall.

LEON: At IPS we want to set up processes for ongoing systematic reviews as well, and what we have in mind is some fusion of technology and human effort.

We are focusing on building relationships and collaborations first, because research is more likely to be shared and used if it comes from or is interpreted by someone you already know and trust. Let me here make a pitch for government to work more with trusted partners like IPS.

Generally, Singapore would benefit from closer and more frequent ties between the technical people, the researchers, and those making policy. Datasets need to be interpreted, and interpretation often needs people with tacit knowledge about context, experience etc. that is difficult to codify. This is something very experienced public officers can bring to the table. So how can we build better systems to harness our collective knowledge and experience across different domains?

In your experience, are there any organisations or projects that seem to have achieved this balance?

AARON: My two examples are not quite mainstream yet, relative to the rest of the system, but they are interesting prototypes.

First, Singapore’s Ministry of Trade and Industry (MTI) has always had an Economics Division, with a deep literacy for quantitative analytical work, but they have also been learning to conduct semantic analysis and work with narratives as well. Then-Director and Principal Economist Kuhan Harichandra set up what was called the DAU (Data Analytics Unit) within the Economics Division. He recognised that there were many more types of data they were not literate in, and he was trying to bring these on board. This initial effort laid the groundwork for the subsequent establishment of MTI’s Digitalisation Office, dedicated to catering to the evolving needs across economic agencies. Central to this was MTI’s flagship product, NERVE, which stand for the National Economic Research and Visualisation Engine. NERVE serves as the Sectoral Data Hub for the economic domain, seamlessly integrating data from both governmental and private sources. This amalgamation not only augments economic surveillance, but also informs strategy formulation, facilitates policy evaluation and enhances service delivery.

Then there is the National Data Campus in the UK, which is linked to the Office of National Statistics. You can tell they are not only doing data push work. Instead, they are trying to ask themselves: “What do we want the country to be like, and what do we want the government to be like within that? Where does data fit to answer these questions better?” And that leads them to questions such as: “What do we learn about citizen preferences? What can we do in terms of proactive policymaking?”

ELEANOR: I would point to some of the smaller scale pilots, such as the Life Chances Fund,³ that are using an outcomes-based commissioning arrangement, where central government teams and local government partners commit to jointly pay for outcomes. In some situations, this leads to a model that moves from a data push approach to a hunger for data. We see an appetite for being more data-led and adaptive in the way that programmes are running, in real time.

This is because when we insist on more of an outcomes orientation, we are not just encouraging teams to identify interventions and services that are likely to do well to unlock those outcomes. Instead, we bring a space that allows for iteration and adaptation, and for much richer datasets that are in the hands of the teams that are doing the service delivery. This outcomes orientation can be a way to develop muscular capacity around the use of evidence and data throughout the full implementation journey of a particular programme.

This is exciting because it means the evidence is being used in a different place than usual—not just at the initial design stage, and not just at the end evaluation stage. It brings everyone to the development of data and evidence throughout the programme process, which means there can be self-reinforcement and self-learning, with the integration of different types of evidence along the way.

LEON: The hunger for data resonates with me, and I relate this to our earlier points about bringing different people in the ecosystem together. First-order discussions should be about building relationships and trust. These elements also encourage people to think about what they really want to know, which then leads to discussions about how to go about collecting data to meet those needs.

Once we establish common goals, it would be easier to have second-order discussions about who should resource it, who should house it and so on. It’s too easy to get distracted by these. If we skip the first-order discussions, we end up centring discussions around the second-order questions, and then collaboration dies.

How do you hope to see evidence-based policymaking evolve in future?

ELEANOR: There is always a risk when we are talking about data and evidence that we centre on the technical aspects, and not the more relational and personal issues. Yet so much of whether data is trusted, or evidence is seen as credible, is informed by these relational aspects. It is also incredibly subtle work to determine if and how we can use particular data sets, whether we can read across them, or mash them together for analysis.

This is why it will be important to have data stewards: people who can translate and navigate this space. But we need to think hard about where the capability for this is, both in government and also spanning outside government.

It is also not enough to see more evidence being generated. What is key is people coming together around it. So I would like to see more spaces for people to convene, and to have these conversations about evidence; to think about evidence, not in a straight line way, but acknowledging the messiness and the need to keep revisiting the conversations.

The risk, if we do not do this, is that we overlook how to improve. Not only is this damaging to a government trying to do the best it can, but it is also potentially corrosive to trust in government. If citizens do not feel that a range of good perspectives and bodies of evidence have been considered in policymaking, it will affect confidence in the way policy decisions are made, and public services deployed.

LEON: What I hope to see are pilot initiatives where policymakers, ops people, and researchers come together to tackle a problem, start working out the processes and building relationships for collaboration, collecting the necessary data and evidence, and communicating findings in ways that are relevant for policy and operations teams.

AARON: A lot of this work will boil down to a number of success factors. We will need the right compact—the trust that government has to cultivate. This work needs to be seen as part of a broader political and social compact, which also demands that the centre and periphery have a better-defined set of relationships.

Then different types of capabilities need to be developed, both at the centre of government and in the other agencies. While they may share some capabilities, they all need to develop different overall sets of expertise. For this to happen, we also need better convening capacity, and better conversations within that convening.

As has been discussed, we need to cultivate a craving—a hunger—for data.

The hope I have is that we also build this up as a community, in a collaborative way. If governments try to do all of this by themselves, they will fall prey to new sets of biases. There will just be a big echo chamber across government instead of a small echo chamber in one agency. The more we broaden the lens when we start to frame the questions by bringing different people on board, the better our questions become. And we will not be starting the policy learning cycle from zero.

NOTES

1. For more information on the Maryland Scale, see: https://whatworksgrowth.org/resource-library/the-maryland-scientific-methods-scale-sms/.
2. https://golab.bsg.ox.ac.uk/knowledge-bank/indigo/syrocco-ml-tool/.
3. https://golab.bsg.ox.ac.uk/knowledge-bank/indigo/fund-directory/INDIGO-FUND-0012/.