Geographical region: Latin America & the Caribbean
Preparer: Alexander Singal
This research project was undertaken as part of the second round of the Metaketa Initiative. The principal investigators were Darin Christensen, an Assistant Professor of Public Policy & Political Science at UCLA and Francisco Garfias, an Assistant Professor of Political Science at UCSD. They worked with Sebastián Leiva Molano, a Research Associate at Innovations for Poverty Action – Colombia (IPA), who in turn collaborated with a local non-profit in Bogotá to test whether that partner’s interventions were effective at increasing the rate of business formalization for small businesses in a suburb south of the city.
The experiment took place between February and April of 2019, with employees of the local partner reaching out to business owners and providing information on the process of formalization and, in a second treatment arm, inviting owners to attend a workshop related to business development. The experiment was completed successfully, with the researchers evaluating the effectiveness of the intervention based on administrative data showing which businesses had formalized. While information and facilitation did not increase the rate of business formalization, the study found a modest increase in formalization rates among businesses invited to the workshop.
Decisions to Partner with the Research Team
The collaboration between the investigators and IPA was originally intended to study land titling, however that project had to be shelved due to administrative and legal reasons. The preparation for the land titling project had already given them the opportunity to explore the area of southern Bogotá, though, and the team realized this location would also serve well for an experiment on business formalization. IPA had contacts with the local partner from previous work, and the partner had experience working with businesses on formalizing, so they were happy to participate in this project.
Leiva Molano felt that the collaboration with the local non-profit was a natural fit, saying, “It’s a program that they’re used to doing, that they’ve been doing for quite some time, so when we proposed doing an evaluation, they said sure, why not, it’s aligned exactly with that we do.” Leiva Molano said there was particular enthusiasm from one of the higher ups at the local organization, whose background in economics gave him familiarity with the benefits of such a rigorous review. “He knows what an impact evaluation is, and he knows how helpful it can be to have an impact evaluation, so they saw it as a free evaluation of the work they were already doing.” Additionally, the organization had been skeptical of the results of a previous evaluation that had been performed earlier in the decade, and they were optimistic that a second evaluation would affirm the value of their work.
Working with the Team
From the outset, IPA and the researchers attempted to make clear to the local partner what the costs and benefits of the partnership would be, and that the process would require close monitoring to ensure that experimental standards were being adhered to. Specifically, IPA provided guidance on best practices around survey methods and data collection, and they followed up persistently to make sure that guidance was being followed.
The importance of the control group not receiving any treatment was also stressed, since this marked a departure from the organization’s normal procedure, and it may not have been clear to everyone why this was necessary. Leiva Molano explained, “They have a team of people who do the business outreach. They have the knowledge and experience, but they’ve never done it this way, where they are instructed to not reach out to certain businesses. This was the biggest accommodation they had to make.”
And while there had been support from some in the partner organization for the importance of this type of evaluation, Leiva Molano felt that others seemed to bristle at the perception that they were now being monitored constantly. “The other thing—and to be clear this is how I feel—I believe that they were working under the idea that we were always scrutinizing their work and they had to be extra careful around us, and that sometimes made for a more difficult relationship, because it’s hard when you feel like you have people looking over your shoulder all the time.”
Additionally, while the local partner had an office near the area being studied, they hadn’t actually done any previous outreach to that area, as they would more typically be promoting formalization among larger businesses in other parts of the city. This caused some concern, as members of the organization felt like they were now being judged for their performance among a set of smaller businesses that they had not previously targeted. There was a feeling of discouragement that the outreach to the smaller, mom-and-pop type businesses in southern Bogotá were not going to prove effective, and that this would potentially reflect poorly on the organization.
Dissemination of Findings
A presentation of the preliminary results was held for key stakeholders in the local organization. This was followed by a reflection on what happened and why the results didn’t show the effect that the partner had hoped for. Discussion turned to next steps, including the possibility of a follow up evaluation on how the local partner can more effectively target its outreach.
The local partner felt that the study’s null effect was due to the outreach being targeted at different kinds of businesses than they are used to working with, and they are eager for an evaluation of their more typical operations. Reflecting on the meeting, Leiva Molano said, “Maybe not having positive results was disappointing to them, but they understand why the results were what they were.” The partner was also able to acknowledge the lessons they could take from the null result, rather than just interpreting it as a disappointment. “They said, at least we can confirm that we shouldn’t be wasting our efforts on these businesses. These businesses are important, and they need other kinds of help, but what we do is not what they need.”
While the partnership was successful in seeing the project through to its conclusion, Leiva Molano acknowledged that there were areas in which the collaboration could have been improved. Though he felt IPA and the investigators were very clear upfront about what the process would entail, he wonders if those details weren’t always taken at face value, and whether those processes could have been better formalized. “We did say this would happen, and they said ‘sure go ahead’, but maybe they would have wanted to know that when we said it we actually meant it, and if they were going to go into the field twenty times then that meant we would go with them twenty times, to make sure everything is going the way it was described.”
Nonetheless, Leiva Molano was optimistic that the partner ultimately benefitted from their collaboration and was starting to internalize the best practices that IPA and the researchers recommended. “At the end of this, we had a session where they did tell us what they learned from working us. They told us they tried to be more organized and tried to be a little bit more rigorous, because they knew of our standards and they tried to get to that.”