PETER ZHANG – DECEMBER 2ND, 2020
Fernando Hoces de la Guardia is a Postdoctoral Fellow at the Berkeley Initiative for Transparency in the Social Sciences (BITSS) hosted by the Center for Effective Global Action (CEGA). His research interest is evidenced-based policy, particularly in increasing the transparency and reproducibility of policy analysis. Dr. Hoces wishes to highlight the role of CEGA and BITSS in making Open Policy Analysis (OPA) a team effort. BER Staff Writer Peter Zhang interviewed Dr. Hoces over Zoom on October 27th, 2020.
Peter Zhang: How were you initially drawn to economics and public policy?
Fernando Hoces: A long time ago, in Chile—I did my undergrad and master’s in Chile, where you don’t just go to college, you go to a career right from the start—I started on a track that’s called Commercial Engineering, which is basically a dual major between business and economics. I started thinking of going along the business track because I was seeking financial stability. But, when I was two years into it, I discovered economics and realized that you can use this rigorous language to talk about social issues— and you can make a career out of it! So that was my first encounter.
Peter Zhang: How has learning in Chile influenced how you think of economics today?
Fernando Hoces: The career I had in Chile was what you would call a traditional track in economics—doing a master’s and following a standard path. Then I spent a total of four years working in the public sector. That’s what heavily influenced my thinking about public policy. I describe myself as a policy economist; my training plus my experience was in doing economic analysis for public policy, and that heavily shaped the way that I think about how to bring more rigor into policy debates.
One of the main things is that when I was in Chile, I was thinking that basically economics and public policy is this exercise of doing rigorous analysis to answer policy questions. But I was somewhat surprised that I had the opportunity to serve under two different governments—one from the left and one from the right. When I was doing pretty much the same job, the perceived effectiveness of my work or the perceived rigor of my work was very different under different administrations. That was a motivating factor.
On top of that, I migrated to the US, 7, 8 years ago, permanently because my wife is here. When I came here, I was like, “ok, being a policy economist is pretty much a translatable skill.” I thought of this as something that’s pretty tradable; you can move from one country to another and pretty much the same thing.
And I realized it’s really not the case. There’s a lot, especially in policy economics, that entails credentialing and the authoritativeness of how you say things and how you’re perceived. In Chile, I had some that authority, some of those credentials, and here I did not have the eloquence; I did not have the connections; I did not have anything. I was sort of stripped from that.
I realized “Wow! There’s a big premium around that,” and that means that there is a lot of noise around how policy economists are conducted. There’s a lot of space for opaqueness. There’s a lot of space of ambiguity, and that was one of my biggest motivations to say, “this whole exercise can be way more transparent, and I suspect that there will be big benefits out of it.”
Peter Zhang: That’s a really interesting journey. Given this background that you have in policy economics, looking at both the left and right in Chile and then moving to the United States, how has all of this influenced the work you do right now?
Fernando Hoces: After realizing that there’s a large amount of opaqueness in policy analysis, there’s something a little bit choking or discouraging. When I was drawn to economics, I was drawn by this idea of bringing rigor to public points. I was not driven to bring in rigor to academia. I think academia has rigor, and that’s great.
But when I started looking at how this idea of rigor to public policy is brought in practice, I saw that there’s a little bit of contempt coming from the academic world to the policy world. It’s like, “ah, that’s like less rigorous.” It’s totally fine that it’s less novel, but there’s no reason why it should be less novel.
And this contempt, I’ve seen it in Chile. I’ve seen it here, from the academic world and toward policy analysis world, is what heavily motivates me to say: “wait a minute, to say these analyses are as important, if not more important, than academic work, and we should have exactly the same standards of rigor—how do we do that?
Then, I saw what was going on in the reproducibility crisis in science and all the discussion about how to bring more transparency and reproducibility in science, and I saw the parallel. I draw a lot from the language that they were using, and the tools and the solutions. That’s what influenced my decision to bring these ideas into policy analysis.
Peter Zhang: Could you talk a bit about open policy analysis? What is it and how does it connect to that goal?
Fernando Hoces: With the Open Policy Analysis initiative, what we’re doing is that we’re trying to promote the use of open science practices into policy analysis. It’s basically asking for policy analysis to be more transparent and reproducible in a systematic way. We’re doing that by building a framework, building a conceptual idea of how to carry out open policy analysis.
Another part is to carry out some of these open policy analyses in practice. We call these open policy analysis projects. So we did one with the wealth tax, with [Professors] Emamual Saez and Gabriel Zucman. We’re finishing one with deworming interventions, with [Professor] Edward Miguel and Evidence Action, [an NGO]. We’re building a pipeline, one about unemployment insurance in a collaboration with the Berkeley Initiative for Young Americans.
Next, we’re looking to build a community of practice around open policy analysis. We’re not the only ones working on this idea. There’s other people who are doing great work. We want to bring everyone together to create a critical mass of people interested in these ideas.
Peter Zhang: Could you dive in a little bit more into one of these projects?
Fernando Hoces: The way that we articulate the framework of Open Policy Analysis is to have 3 high-level principles. One principle is open output. Another principle is open analysis. Another one is open materials.
By open output, what we mean is to move away from the traditional report where you put several results in a report and you give the policymakers 10 scenarios. This is the famous economist that says, “on the one hand, this thing, on the other hand, another thing.” We want to give the policymakers what from the analyst’s perspective is the best representation of the facts. There should be one scenario. That’s the idea of open output: to commit to one output, but also to make a clear connection between how that output changes when you change the underlying assumption. So that the open output will be at one fixed output but also in an interactive way such that you can change the underlying assumptions and see how the output changes. That first principle results in an application where you can change the underlying assumption and see how the results change.
The second component is to have an open analysis. By that we mean to have an exhaustive description of how the analysis was carried out a) in narrative form but also b) with equations to add more clarity as to how this analysis is carried out, but c) on top of that to add code, so [sic] combine everything using principles of computer science of literate programming in what’s called dynamic documents, to basically combine everything in one place such you can see all the analysis, all the code, and all the equations in a transparent way.
The last one, open materials, is that all these materials should be in an open repository that allows external viewers to reproduce those components with minimal effort. So just do a few clicks, and you get the analysis using open-source software with a minimal number of restrictions like having a computer.
Peter Zhang: In a lot of fields, open-sourcing data and methodologies is becoming more common. You’re trying to build this practice—why isn’t this already commonplace?
Fernando Hoces: I thought quite a bit about that and I can only speculate. I think that we’re in a suboptimal equilibrium in policy analysis.
First of all, there’s a good amount of policy analysis out there that is credible, that is rigorous . . . serious, let’s say. But a large fraction is much more advocacy, painting the target right after you shoot the bullet. You justify the conclusion ex-post. I would call all those policy analyses noise.
In the unfortunate equilibrium we’re in, those who produced credible policy analysis have little individual incentives to go with a fully open, transparent and reproducible analysis, because locally they are perceived as credible. The tough part is that the larger community—particularly policy analysts that have different ideologies—is less likely to view them as credible.
We end up having parallel worlds, where we have a think tank on the left and a think tank on the right who address the same empirical question and conduct the same analysis to address the same quantitative issues and they get radically different answers. This is what Charles Manski started writing about in the early 2010s.
The unfortunate component of the equilibrium, based more on what I’ve seen in Chile, is that there’s a premium around credentials and authority. The people who have the authority, who have the perceived seriousness, do not want to relegate so easily. The people who are considered serious policy analysts or policy economists have little incentive to disclose everything.
Peter Zhang: You’ve painted a world where ideology divides and institution incentives keep in place this poor equilibrium, so what are the next steps to upset that?
Fernando Hoces: I see a parallel between that and the open science movement. The open science movement has gained a lot of traction over the last 5-10 years. The incentives are somewhat similar. The incumbent, prestigious institutions have little incentive to open up materials and open up their work because they’re perceived as credible. But if there’s a critical mass of discontent with the current practice of science, you will have innovators in the open science space who will do it differently. That’s what has happened in the last 5-10 years: a push towards open data, open code, computational reproducibility, replicability of experiments. That’s something that has changed radically in the last 10 years.
After the creation of the new principles of open science, there was a push to get the buy-in of funders, to get the buy-in of journals, and that spurred the movement. This is where CEGA—the largest is the Center for Effective Global Action and the initiative is Berkeley Initiative for Transparency in the Social Sciences [BITSS]—plays a key role. The center is focused on bringing the best evidence possible to alleviate poverty around the globe. With that motivation, CEGA created BITSS to bring more credibility to social science. The next logical step is to move from evidence policy change to policy analysis to improve the credibility of policy analysis. That’s where CEGA and BITSS are playing a key role.
Peter Zhang: Let’s say that we create this shift in policy analysis. How do you envision this shifting policy?
Fernando Hoces: We want to create an environment where we only have these credible policy analyses around. Now, there are a multitude of facts lying around. That ultimately creates a situation where policymakers can choose their own facts.
Peter Zhang: Alternative facts!
Fernando Hoces: It has been accentuated in the U.S. over the last four years, but this is not something specific to the last four years. I think it relates this contempt from academia to policy analysis. It’s kind of agreed that policymakers can choose whatever fact they want, when discussing minimum wage, wealth tax, funding higher education, or whatever issue is under debate. It’s kind of agreed that now policymakers can choose whatever they want. What I see is that as open policy becomes a norm, policymakers will be forced to face the best representation of the facts. If the policy analyst community, supported by the academic community, can say this is “the best representation of the facts as we know them today,” then policymakers will be forced to say, “I support the minimum wage, and someone opposes the minimum wage, and we both agree on what the facts are.” That will shed light on the normative preferences of the different policymakers. What do they stand for?
That’s our agenda for the next 10 years.
Peter Zhang: Suppose I’m a pessimist and I think that, even with consensus among policy analysts, someone like President Donald Trump still won’t care.
Fernando Hoces: I thought quite a bit about that and that is a possible scenario. My response would be that the way I see things—and this is obviously painted by my current initiative—is that under a world where there is open policy analyses, it’s much harder for people like Donald Trump to exist. Some people out there might still give some credibility to the different facts laid on the table. And it’s precisely because we do not have 100% agreement of what’s the best representation of analysis.
I don’t think OPA is going to prevent something like Donald Trump from happening in the future. I think it’s going to make it less likely. It will shed light on when there is a blatant divergence from the facts.
This is not a magic bullet. But, I think it is something of a step in the right direction to prevent the spread of misinformation.
Peter Zhang: If I am a policy researcher who is interested in your project and I want to get involved or follow your work, where could I do that?
Fernando Hoces: Definitely check out the Berkeley Initiative for Transparency in the Social Sciences. That’s the initiative that is supporting open policy analysis. Also, check out the Center for Effective Global Action.
If you want to be particularly involved in open policy analysis, you can visit the background on our webpage and you can be involved as an undergraduate research assistant each semester. You can suggest a policy analysis for us to open up or for us to collaborate with others who open up. You can also subscribe to BITSS news so you’re up to date on everything we’re doing.
Peter Zhang: Great. Thank you for your time!
Featured Image Source: Center for Effective Global Action
Disclaimer: The views published in this journal are those of the individual authors or speakers and do not necessarily reflect the position or policy of Berkeley Economic Review staff, the Undergraduate Economics Association, the UC Berkeley Economics Department and faculty, or the University of California, Berkeley in general.
