SPEAKER_01 0:12

You're listening to the Flat Your Business podcast with Roger Davis. Hello, you're listening to the Philanthropisms Podcast. This is the podcast where we try to put philanthropy in context. I'm your host, as ever, Rod Davis, and this week we are in conversation with Aaron Horvath. Now, Aaron is a postdoctoral research fellow at the Centre on Philanthropy and Civil Society at Stanford University. And he writes a lot about various issues to do with uh organizational structures and kind of measurement um within the nonprofit sphere. Um and I uh sat down particularly to talk with Aaron about a really interesting paper that he recently published, um which was titled Organizational Supererogation and the Transformation of Nonprofit Accountability, which is a sort of unusual uh catchy title for a journal article. Um but essentially this paper is about a really interesting um bit of research uh that Aaron's done looking at uh data on uh what happened when new measurement requirements were imposed on nonprofits um in the San Francisco Bay Area between uh starting in 2002 but going all the way through to to 2016. Um and essentially the finding that actually contrary to what you might expect, um organizations rather than sort of doing the bare minimum, went above and beyond what was required of them, hence the idea of supererogation, which essentially means kind of uh doing something that is above and beyond GC. Um, I'll put a link in the show notes this way you can have a look at the paper and a shorter version of an article summarising that Aaron's done. Um but yeah, we had a really great conversation kind of talking around the paper and also broader issues around uh accountability and measurement in the non-profit sphere. So we talked a bit about uh the idea of supererogation in the paper and why it was perhaps that non-profits were kind of going above and beyond when it came to measurement and reporting. Uh we talked about what sort of form that took. Was it that they were measuring new things, or was it that they were using kind of more narrative or qualitative tools to tell a story about the work that they did? And we talked a bit about whether there were any risks involved in that, whether actually kind of measuring more than you're required to uh kind of present some challenges for non-profit organizations. Um we also talked a bit about the kind of wider context on why there's been uh you know an increased emphasis on measurement and metrics in the non-profit world and certainly over the last few decades. Um we talked about whether there's you know a danger that externally imposed metrics that are that are kind of um uh presented for nonprofits to comply with, whether they uh reinforce a tendency for nonprofits to see themselves as accountable not to the their recipients or even to the people supporting them, but instead accountable to regulators um um and whether that was kind of uh a problem. Uh we talked about whether the imposition of rankings and ratings and metrics brought with it the danger of a greater degree of um homogeneity and more sort of everything being the same uh in civil society and might undermine the kind of uh pluralism and diversity that that is often seen as one of the strengths of civil society. Um and we talked about whether the kind of fundamental idea of of measurement and the goal of measurement being that we're working towards some kind of universal measure that allows us to uh accurately assess the value of uh all work in civil society, regardless of cause area, whether that is still an idea that people yearn for or whether it's kind of fundamentally uh misguided. And and if there are elements in the work of a civil society that we can't make apature in our current measurement systems, what's the conclusion there? Do we need to get better at measuring and and more sophisticated, or do we need to become more comfortable with the idea that there will always be some things that aren't measured? Um we also talked a little bit towards the end of the conversation about another article that Aaron had um out relatively recently, a joint article with Michael McElroy, um about uh limited liability companies and their use as a kind of new structure for philanthropy, particularly in the US, and some of the challenges that that uh brings around accountability and transparency within big money philanthropy. So it was really interesting as well. Anyway, enough of me talking. Um without further ado, let's go into the conversation. Um, I will be back at the end for the usual bit of housekeeping and tidying up. Okay, great. Well, I'm here with Aaron Horvath. Hi there, Aaron.

SPEAKER_02 5:06

Hello.

SPEAKER_01 5:07

Oh, well, great to have you on the podcast. Uh and for people uh listening, I'm sure I've done some uh biographical information up front, but Aaron is a postdoctoral fellow at uh Stamford Center on Philanthropy and Civil Society. Um and I wanted to talk to you about a whole bunch of things, but I guess the the centerpiece was a paper that you uh recently had out as a journal article, and you had a version, I think, on Histville as well, which kind of centers all around some work you've done looking historically at a group of nonprofits um in the US, but in terms of how they responded to changes in what was demanded of them when it comes to measurement and some sort of slightly surprising uh findings about the way in which they they did respond. So I don't know if the best place is for you to just kind of outline what it was you looked at and what the kind of key thing that you found was.

SPEAKER_00 5:54

Yeah, maybe the the best place to start is just to give a little background on the project itself. Um, because it's it's basically the ground I stand on to even be able to make some of these claims. So um the the project is uh is is called the Civic Life of Cities Lab, and it's it's housed at the Stanford Center on Philanthropy and Civil Society. Um and with my collaborators there, I've been following uh a random sample of 200 nonprofits in the San Francisco Bay Area for uh, I guess coming up on two decades. Um I inherited this project as a grad student um and have carried it on, and we've we've published a bunch of papers out of it. Um but uh in this in this particular paper, um I was looking at this this whole slate of evaluative pressures that uh emerged in the nonprofit sector in the US um around the late 1990s and and early 2000s. There's just a an explosion of evaluative and oversight demands and all sorts of formal oversight um that there'd been inklings of in the sector before, but maybe not to this extreme of a degree. And and I can get into uh the history of why these things even popped up. Um, but the the main kind of theme here is that it was a new set of evaluative pressures that um caught a lot of organizations off guard and that they had to they had to deal with. Uh, just to give a taste of what these pressures were, um, there's everything from Congress mandating that nonprofits make their annual tax returns. And these are in the US, the uh Form 990 available on demand. There was also an effort to digitize these forms and make them publicly accessible online. Um, with this idea that once you put uh these financial forms out in the open, the uh the rogues will quote think twice before uh before they commit some act of financial or charitable malfeasance. Um so simply getting the information out there was was kind of a concern on a regulatory front. But there uh there were also these rating agencies that popped up. The the best known ones would be like Charity Navigator, uh, the Better Business Bureau, Wise Giving Alliance, um, and a handful of others that they they were mining these financial forms to uh you know use their financial data to then compute ratios, like the infamous administrative overhead ratio. And they were using that to assign all sorts of accountability scores based on cutoffs, like if you spent too much on your staff, uh, you might be penalized because um, according to this fairly simplistic metric, you weren't spending enough on direct service provision. So there was that set of pressures, just kind of a financial oversight and accountability set of pressures. And then foundations and funders, philanthropists uh added another dimension into this kind of space of evaluation and formal oversight, accountability by promoting uh impact evaluation and outcome measurement to the kind of the front of nonprofit discourse. It's really hard to even have a discussion about nonprofits these days without talking about impact and measurement. So there's major players like Gates, Hewlett, Kellogg, um, and these foundations built metrics into their logic models and to the very core of their grant-making activities. And they even established dedicated impact evaluation units. And they were very, you know, they weren't quiet about doing this. They very much promoted these ideas um to other funders and to the broader public as the right way to do philanthropy. And there was a handful of consulting firms and philanthropic advisors and funder coalitions that were all helping to spread this gospel of measurable impact. You know, we we could talk more about the fact that these are highly funder-centric uh evaluative demands, often because they're evaluating their own portfolios and trying to compare across grantees. So the idiosyncrasies of uh you know, grantees were kind of uh kicked out sometimes and more simplistic measures were used. Um, but there are also these kind of grand ambitions of uh trying to establish causal links, which anyone who's dabbled in the world of causality uh or causal inference knows is is quite difficult and quite resource intensive. So oftentimes uh nonprofits were were hit with all sorts of metrics that maybe seemed a bit bizarre uh or didn't quite make sense for things that they cared about, and yet they're having to report in these terms. So, given this whole you know, onset of evaluative pressures that around the turn of the century, proponents are looking at these things as yes, like finally we have a more objective uh set of tools to make decisions because you know humans are very fallible in their in their judgment and decision making. And you know, that's something that obviously uh behavioral psychologists delight in telling us is just how fallible our our decision making is. And and so the antidote to a lot of folks was if we use you know quantification and evaluation, um, it will help to discipline all this fallibility and the mechanical objectivity of measurement will uh will kind of allow us to make better decisions. And that's all well and good. Um, we are very fallible creatures, but that antidote of evaluation and quantification comes with pretty serious complications. So when economic and organizational sociologists look at the kind of stuff that was happening in the nonprofit sector, and more generally to how non uh organizations respond to intensive scrutiny, evaluation, regulatory pressures, et cetera, uh, they tend to find one of two things. One of those is what we can just call the kind of decoupling account. That's frankly what it's known as in sociology, um, which is that organizations will do the bare minimum to satisfy all these public demands. Um, and they'll demonstrate that everything's on the up and up. There's no need for you know looking any further, uh no need for further scrutiny. Basically, it's it's filing the requisite forms, you know, checking the right boxes, adopting the right policies. It's a lot of CYA, it's symbolic stuff, uh, and kind of an effort to maintain legitimacy. So, but that's not always possible. So sociologists have arrived at you know, another account, um, which is the disciplining account, where when organizations can't kind of compartmentalize all these external pressures and scrutiny to these more peripheral or low-cost practices, they might end up actually internalizing the criteria by which they're judged. And in doing that, they end up forsaking uh their original goals. So the you know, the classic case of this is law schools that are subjected to the US News and World Report rankings end up kind of forsaking some of their more idiosyncratic goals in order to move up on the rankings based on the criteria that are set in the rankings. So the fears that nonprofits exposed to all these things would end up kind of becoming uh more like the criteria by which they're being judged. So, from the perspective of the early 2000s, uh it didn't look good. Um, and in fact, you know, critics were very hit to this. Um, they're worried that these narrow and simplistic accountability demands would uh would lead to a whole reorientation um of the sector and possibly even homogenization of like what a sector that is known for being so pluralistic and varied and diverse. Um that seemed like a scary outcome. Of course, for some proponents of all this evaluation, that was the goal that you could reorient how nonprofits collectively thought about their work and you can make them more business-like, more efficient, more effective, et cetera. And so this kind of disciplining perspective, as sociologists would see it, was for some proponents of this stuff. It was the point of doing all of this. And then the concern was that the emphasis, you know, even if you didn't have this kind of disciplining effect going on, that folks would become more oriented around short-term, easily measurable indicators. And this would crowd out all the you know important stuff of civil society. Social movements are awfully hard to measure. Uh, civic goals like community building, citizen engagement, um, you know, those things are very real, but it's really, really, really hard to like identify a specific point of intervention and then uh you know measure the effect that has had because there's so many diffuse and knock-on effects of those activities. So that's kind of setting the stage of where this paper begins. It takes a lot of kind of setup to understand why what I ended up finding um was so surprising.

SPEAKER_01 14:02

Yeah, I mean, it's it's really interesting because I think I mean the thing just to chip in and say is I think even though we're talking about it in the context of looking back to the you know, the start of the 2000s or whatever, the the that narrative and all of those issues around measurement and both the kind of the drive for it, but also the unintended consequences of it and the questions about well, who decides what's getting measured, you know, to what purpose, how does it relate to questions of accountability and stuff, feel very live right now. So I think it's you know, these these are things that people are really grappling with right now. Um, I guess, yeah, it'd be really interesting to hear because like the at the core of the paper is um the the idea of supererogation, which is sort of an idea that's been borrowed from elsewhere, but you've applied it to kind of explain how actually lots of the nonprofits that you tracked didn't do either of those things that you might have expected from the sort of standard sociological accounts. And actually they did something quite surprising. So um, yeah, do you want to say a bit about that?

SPEAKER_00 14:59

Yeah, of course, of course. And and this is where I apologize for being a sociologist and introducing such an infelicitous term as supergation into uh into people's vocabulary. So um my my apologies for that. Um, but I I I'll try to justify why why I use that word, where it comes from, and and that sort of thing. So when I started looking at the data that I'd collected uh along with my collaborators, at first launch it kind of seemed like the typical sociological story was going to hold. Um, nonprofits basically detested a lot of these new demands that were put upon them. Um they saw them as pretty time consuming, they saw them as kind of, as I mentioned earlier, just bizarre and not necessarily in line with what they're actually up to, um, and that they often missed really important things. So it seemed like you would get some sort of decoupling, like this doesn't make sense for us, we'll check the boxes and move on. But then I realized something else that was quite surprising. So these organizations weren't just doing pro forma compliance to satisfy external demands. They were also disclosing far more information and detail than was ever required by anyone. Uh, and they were doing so in ways that were quite idiosyncratic to their missions and their values. Uh, just to give a quick sense of that, they were doing things like on top of the evaluations they were required to do, they were devising their own bespoke evaluative criteria and even seeking out uh independent analysts to evaluate them in those terms. They were uh publishing unvarnished details of fairly contentious and often very awkward uh board meetings and the decision-making processes, and then establishing channels for feedback from beneficiaries and the communities they serve, et cetera. Um and they were doing things like elevating the voices of staff and beneficiaries whose, in their view, meaningful experiences were obscured by ratings and other reductive metrics. And so it's important to note that really no one was requiring any of this. And even some of the nonprofit sector auditors that I interviewed, because I reached out to them, say, what's going on here? Um, they were nervous that some of these organizations were actually creating liabilities in the process of being so radically transparent. So none of this fit with the theoretical expectations uh that I was working with as a student of sociology. So I set about trying to find um, you know, a better account for for this pretty wild explosion of disclosure. And one of the concepts I quickly uh was drawn to was this idea of superarrogation. And I don't even know why I know that word, other than the fact that I operate in a fairly interdisciplinary environment with a bunch of uh philosophers. And they use a bunch of fancy words, and I occasionally write them down. Um, and when I looked this one up, I was like, oh, gee, that that fits quite well. This this is a term for uh going above and beyond. And when I looked at my data, it was nonprofits not only complying with what they're being asked to do, but going above and beyond beyond what they're being asked to do. So on the surface, it seemed to fit the broader pattern. But the more I dug into it, the more I realized this this actually really helps to explain the story. So in moral philosophy, um, supererogation is uh is basically a complement to a three-part classification of moral action. And here's where I pretend to be a moral philosopher for a moment, which I'm decidedly not, but um they have some great ideas. So this three-part classification of action would be things that you do because uh they're good to do, things that you things that you have to do because they're good to do, to be clear. Um, things that you cannot do because they're bad, um, and things that you're permitted to do because they're neither good nor bad. Um, superarrogation offers a fourth category, which is things that are good to do, but that you're not in any strict sense required to do. So to supererogate is to voluntarily perform good acts that exceed the demands of moral obligation. Now, digging into that even more deeply, because philosophy knows no bounds and will pick everything apart, um, the philosophical use of superarrogation lies at the intersection of the deontic and the axiological, or in plain English, which is what I try to speak, obligations and ideals. Um, and so just to parse that out, obligations are the things that you absolutely have to do. They're prescriptive, they're minimal requirements of moral action. Ideals, on the other hand, are more open-ended. They're indeterminate conceptions of what is good or what is virtuous. And, you know, obviously philosophers work themselves into knots over these things, like if it's good to do, why don't you have to do it, et cetera? But as a sociologist, this is a quite useful set of concepts because begin to realize its component parts of obligations and ideals map onto social life really well. Translated into sociological terms, obligations are the things we have to do. So the forms we have to fill out, the taxes we have to pay, the rules we have to follow, et cetera. And ideals signify these more open-ended evaluative stances of what we think to be good and virtuous, but often kind of ambiguous things. Accountability is one of those things where it has a word that we all know and we have a general sense of what it means, but in practice, it can mean all sorts of different things. Um, and working out those things in practice can often be quite contentious. So these things of obligations and ideals in social life are very linked. Um, laws, regulations, standards, evaluations, they're all, you know, they reflect broader cultural values. And in the nonprofit case, uh, these mandatory reporting protocols, um, these things aren't just, you know, the requisite forms and checklists and questionnaires. They're they're kind of idealized projections of what a good nonprofit should look like and ought to be. The challenge here, of course, is that different people and organizations can have very different practical interpretations of what are nominally the same values. You know, it's we we see this all the time in politics, where people who vehemently disagree with each other are both, you know, touting the ideas of freedom and justice. Um, but in practice, those things mean quite different things. If we put it in the nonprofit case, it's I don't I think it's pretty common to see conversations between philanthropists and their grantees where it's clear that everyone wants to have an impact. I mean, who wouldn't want to have an impact? But the word impact is doing a lot of work coordinating between folks who have very different conceptions of what it actually means in practice to have an impact. So this idea of supererrogation highlights there's a simultaneous connection between obligations and ideals and a kind of slipperiness between them. Uh, and that that push and pull is really the central motor to my argument. So the the idea that the argument that I advanced, this kind of sociological adaptation of super irrigation as applied to the nonprofit case, is that these externally imposed evaluative demands ended up clashing with how organizations you know operated day to day. And it superimposed this very thin approach to To accounting, you know, what are the numbers to a longstanding, thick set of practices about accountability to the people and the communities they serve. But nonprofits weren't able to dodge these new requirements. And they drew on these new tools and discourses around evaluation as basically a legitimated vocabulary through which they could pursue more meaningful forms of accountability, but in terms that others might understand. So it kind of lent legibility to their ideas, their distinctive understandings of accountability. And that's something I can dig into more deeply, the findings that support this, but um I've spoken for quite a bit there.

SPEAKER_01 22:42

No, it's really, it's really interesting. And I guess I mean one thing that I sort of wanted to ask is you kind of you've outlined there this, you know, the these two different poles there of uh the um uh you know having these these obligations that are imposed on you, but then also having these ideals. And it feels as though uh the this act of supererogation that's taking place is because you know, they're even where organizations are resistant to the obligations that are imposed on them, they don't feel as though the metrics are the right ones, they do share the underlying ideals of accountability and the importance of of measurement. Um I guess what I what I wondered was it feels like there would be an alternative to to choosing to supererogate and kind of go beyond using those existing measures, which would be to more fundamentally just question or challenge the measures. And whilst you know, that might be difficult in some practical senses, in the organizations were just required for regulatory reasons to provide a certain amount of that information. Did you see any signs that as well as as organizations choosing to supererogate, there were others who perhaps went in a different direction and and you know, more fundamentally said, you know, we think these measures are the wrong ones and that actually you're kind of um, you know, you're failing to capture the the value of the work that we do. And so, you know, rather than sort of trying to work within the system as as it is, they decided to kind of more fundamentally challenge that system.

SPEAKER_00 24:04

Yeah, I think I think that was going on. A lot of the story I feel like I'm telling is is maybe something of the Davids versus the Goliaths. And I I read a I read an idea recently that uh rich people have peers and uh poor people have have patrons. And and so in these situations, a lot of these organizations uh were the poor people. They weren't they weren't able to um you know necessarily challenge the the fundamentals of what they were being required to do because they're they're simply trying to get by um and and they needed resources to do that. But there are cases where um where we do see that resistance. First of all, we see small acts of resistance, which is uh you know, that you'd be required to fulfill out a quantitative report, but then the the report would ask just for numbers, and the person filling out the report would say, sure, I'll give you the numbers, but here's here's a little story too. I had one director you know explain like, you know, to me, that number is meaningless without the story. To them, that story is meaningless without the number. Um, so you see these little acts of subversion, and I and I do wonder, you know, the effect that seeing that uh over time has had on people who are imposing the kind of evaluative pressures. The there are larger efforts, though, um, to change some of the narrative. There's there's a moment that I I think is pretty interesting. In in 2007, I believe, uh, the IRS was proposing a new form uh or a new version of the Form 990 that would put the overhead, and just to be clear, the Form 990 is the annual financial return or tax return for um nonprofits in the US. And um it proposed to put the uh you know, the well-known overhead ratio pretty much at the top of the form and demote the space that is allowed for narrative basically to the bottom of the form. Um and you know, with with such changes like this, there's a period for public comment. And the letters that the uh IRS received after they proposed these changes do reveal pretty much exactly what you're talking about, which is this idea of like this would this would be hugely disruptive to how people understand our organizations. It would be enshrining a very reductive metric at the very top of a government form through which people come to know nonprofits. Uh, and so there are these more collective acts of resistance against some of these these reductive metrics. Um, but with the organizations I was studying, I wasn't seeing so much of that as I was seeing what I think of as a creative politics of evaluation, where uh they're they're kind of working with and repurposing the tools they're handed um and and using them in ways that that resist these very simplistic narratives about what nonprofits are um or ought to be.

SPEAKER_01 27:04

And and in terms of the the audience for for the information that they're going above and beyond in providing, presumably there's no intention or they're not there's no expectation that the audience for that is the same one that is demanding the information that's provided as you know as a uh as a result of obligation. So the IRS doesn't care about this narrative information. So are are those nonprofits that are doing it, are they are they kind of essentially speaking to an entirely different audience when they're providing that information? And who is that audience? I mean, is it their supporters or the public or funders?

SPEAKER_00 27:36

Yeah, I think it's a whole bunch of audiences. Uh it's whenever I hear the word accountability or accounting, I think there's always the question of to whom. And when when there are all these evaluative pressures put onto nonprofits, uh the there are cases where rhetorically it was, you know, this is accountability for the broader public. We see that in you know kind of government regulations where that is their goal. Um, but when it comes to uh funders imposing evaluative requirements, it was largely accountability to the funder itself. And and that wasn't the the entity that a lot of these organizations thought of themselves as needing to be most accountable to. Uh traditionally, they they saw themselves as being accountable to their beneficiaries, to the communities that they were products of and that they served. Um, and so what I see a lot of this activity uh as doing is trying to reclaim and kind of redirect some of that uh that accountability to whom question. So there's some efforts to do what seems like continuing a dialogue with a community that um that had been going on for quite some time uh with a recognition that um, yeah, maybe in our current era we are a bit more geographically dispersed. The organizations we interact with aren't necessarily in our neighborhood. Um and so to kind of maintain uh the similar crime of a face-to-face uh interaction, maybe you do uh instead of you do doing town halls, um, you do surveys or you do you know online blogs that allow people to respond to uh decisions that are being made. So there's this kind of effort to reclaim some of the ideas of accountability uh for those audiences and redirect it in that direction.

SPEAKER_01 29:23

Yeah, it's really it's really interesting. And and one thing we've kind of touched on before, you know, in in saying that actually the information that's being provided when these organizations are choosing to superarrogate and go above and beyond tend tends to be more uh to some extent narrative, although you did say that they some organizations got into using kind of alternative metrics and employing consultants to do that. I guess this this feels like something that's still very relevant at the moment. And I wonder, W Williamson, your views, do you think there is when it comes to the question of kind of measuring the value of what it is that nonprofits and civil society do, that there's uh there's this sort of inherent asymmetry between the stuff that we have hard metrics for, which tends to be the kind of things regulators are asking for, and it's financial information, and then all of the stuff that nonprofits really feel is probably their actual quote unquote value, but but it's we don't have good measurements for that, or they're not the same kind of measurements, and you have to be more willing to accept soft measures, but but often, you know, from the point of view of regulators, or often, you know, maybe even from funders, they're not seen as comparable. So they have less worth. So if you're contrasting hard financial metrics, you know, that say one thing on the one hand, and then softer measures of you know, that are narrative or talk about outcomes or you know, the value to communities, it's apples and pears, and you can't really compare them. Do you think that is, you know, something that we're still struggling with in this whole question of metrics?

SPEAKER_00 30:48

Absolutely. I think it's always been apples and pears, and and in some sense, folks that were trying to impose more standard or universalizing metrics, um, were saying, sure, they're both apples and pears, but you know, they're both fruit, they both grow in trees. And while that might be true, it kind of misses the, well, yeah, there's a value of analyzing the apple as an apple and the pear as a pear in their own right. So, so sometimes it feels like their efforts to um, you know, as opposed to do the universalizing metrics kind of elaborate the criteria by which we assess organizations. And that seems like you can just go to no end where you can just, you know, well, this kind of organization or this kind of organization, they all need to set, uh, they all need to satisfy, you know, this or that criteria. And it can be, you know, there's these kind of elaborate typologies that the Urban Institute I know worked on in the early 2000s that were kind of common standards for excellence. So there's this view that you just have a bunch of metrics. But um, I I kind of lost the thread on the question there.

SPEAKER_01 31:50

Um it's interesting because you're talking about the ideal of a universalized uh metric. Because I guess we started um for the point of talking about whether there was a an asymmetry between kind of hard financial metrics and and other types of softer metrics. But I'd I'd be really interested. I mean, just to follow up on what you were saying there, do you I mean that that ideal that it does feel was there more maybe kind of 15 years ago or a decade ago, that actually what we need to do is find, you know, the kind of the perfect UR metric that can compare across all nonprofits and give you one single measure of value. I mean, do you think that is an entirely mistaken ideal? And have we maybe moved beyond it a little bit? It feels like that's you know, there's less talk of that kind of thing these days, maybe.

SPEAKER_00 32:32

Yeah, I do think it's a mistaken ideal because it assumes that the nonprofit sector is a thing, um, and that it's a clearly bounded thing, and that organizations, because uh at least in the US case, they uh they are 501c3s under the Internal Revenue Code, inherently have something uh about them or within them that is comparable or worth comparing, or that it makes sense to try and compare. And and that's uh that often leads to pretty crazy things where you end up comparing, you can compare, you probably shouldn't compare Stanford University to a soup kitchen because they are both 501c3s. They're both they're both nonprofits. There's all sorts of diversity in the sector. So thinking of the kind of universal UR metric um does lead to a lot of really odd conclusions. Um, so so that's definitely, I think that was more of a tendency in the past, and and there's been a lot of recognition of of the follies of trying to think this way. I'm not saying that people were actually trying to compare Stanford to soup kitchens, but the metrics do allow for that. Uh, and that was always an odd thing. There, there's a much more uh, there seems to be a divergence right now in the ideas around evaluation. Some folks seem to have given up on the idea that uh you should uh you know try to have unifying metrics for the sector, but have instead focused their their charitable or philanthropic efforts on things that you can very clearly measure and compare. Uh so we see a lot of this concern with um, you know, are there you know clear, clearly identifiable effects of some given intervention? Um, so you know, the effective altruism movement is is very good about this. They they love being able to find uh metrics that that convincingly demonstrate something and then you know be able to put money behind that. But the the cost of that is ignoring the the whole you know chunk of activity, a massive chunk of activity that doesn't easily fall under this kind of you know easily measurable category. So there's a lot of civic stuff that we that we want um that doesn't necessarily isn't necessarily conducive to to clear measurement. Um so I think that's one, there's one direction going that way of this kind of uh more hard-nosed, do we have a randomized control trial that shows this works? Uh, direction of funding. On the other hand, we have this kind of recognition that maybe the metric obsession was um was the wrong way to go. And and there's something uh Hal Harvey wrote uh a few years back where he basically apologizes for promoting strategic philanthropy and advises that people maybe should just uh drop their hubris, uh, write the check, and get out of the way. And I think we've seen that picked up by a handful of high net worth donors, uh, where there's more of this trust-based thing, um, or where people are saying, we're gonna take the reins off a bit and just allow you a little more freedom. So I think there is a movement in that direction as well. Um, the the challenge, of course, is you know, metrics have all sorts of problems, but so too does a more relational or trust-based form of giving. Um, because you know, metrics have this emancipatory potential where, uh, or democratizing potential where, you know, people are compared in terms of of their outcomes. Um, relational methods of who do you know or who do you trust really, you know, moves things toward, you know, what are your biases? Who do you know? Like, did you go to business school together? Um, do you look similar to each other? Do you sound like each other? And and that also is prone to all sorts, uh introducing all sorts of inequalities and biases and giving.

SPEAKER_01 36:23

Yeah, I think this is such a fascinating question because I think, I mean, I'm absolutely I think in terms of the question of the shift towards trust-based approaches and away from measurement. And there's there's definitely the version of it which is it's hugely positive because you're getting away from largely kind of top-down funder-driven metrics that are you know inappropriate and are a means of kind of exerting power over recipient organizations. But then, as you say, the other way of framing uh you know trust-based approaches, which sounds like an inherently positive thing, is that it can be potentially you know kind of exclusionary. And you are right, you that actually if you are limiting um the way that funding's going to existing circles of trust, then that in itself becomes problematic if there are no means for people who are outside the circle of trust to to break into it. And actually in in that sense, it's potentially more meritocratic um, you know, to allow to have some form of measurement or to you know reinstitute things like open grant applications, which is you know, but interestingly, um, I think Mackenzie Scott has has done that with a portion of her giving more more recently in kind of recognition of precisely that that challenge. So um, yeah, it feels like a I mean a really interesting, but a kind of a difficult time for the whole question of of you know where we go in terms of measurement, because there is the potentially this this divergence. And one thing I wanted to ask actually, this kind of feels like a an additional dimension to that whole challenge is is uh you know, when we're talking about who is imposing these measures, I guess traditionally we would have been talking about regulators or uh at an individual level, you know, maybe funders, and then uh we kind of have the emergence of non-profit specific intermediaries uh kind of coming in uh and establishing them themselves as offering uh some version of measurement. As giving increasingly feels like it's happening more and more online through digital platforms, do you think that's gonna have an impact? Because again, it feels as though you could just offer this stuff in an entirely unstructured way and just let people make their choices however they want. But but probably if you're a platform, you're gonna recognize that there's a consumer demand to kind of inform choices and have those choices shaped somehow. And that's gonna lead you back to this question of well, how do we kind of point people towards things they're gonna like or that are good? But then that's potentially now in the hands of entirely for-profit companies whose, you know, whose kind of incentives maybe are not at all aligned with those of civil society. So do you think this is kind of a new front on which we're gonna have to think about some of these issues as well?

SPEAKER_00 39:00

Absolutely. And I and I think in some ways that that's been going on for a while with uh, you know, in the US, Charity Navigator is a major one of these. Their website um in some ways feels like you're shopping for an impact. You go on there and you know, it's like, what cause are you interested in? Um, okay, what organizations satisfy, you know, these criteria that we at Charity Navigator have, you know, determined to be uh important markers of a good, trustworthy organization that has an impact. And then you know, you you can click and donate. And in some ways that's that's great because it allows access to things like uh I know uh it's it's fairly common now when there's uh maybe a natural disaster, which there are way too many of, um, that people will want to give uh to you know the place they just heard about, you know, going through something horrible. Um and you know, charity navigator offers a great service for that where you can go on there and say, hey, like where's the local uh food pantry? What's you know the local you know health clinic or whatever that can contribute to? Um at the same time, these kind of platforms are really framing in some ways how we even think about the act of charity or engaging with these organizations. I I don't even think they need to be for-profit to have maybe some problematic effects in those ways, because simply they're not they're not you know mere conduits of the world that is out there. They present the information in a particular way. Um, you know, they prioritize and rank and sort things in particular ways, and that informs what we see, that informs how we think about these organizations, that informs what we think charities are even for. Um, and I think the whole kind of construct of you know shopping for impact is is very different than the idea of civic engagement, which is the other story about these organizations. Um, you know, there is there's a long, you know, decades-long concern about civic decline and people not being engaged in in community organizations. And I do wonder sometimes um if uh if these kind of online platforms are a product of that or or in some ways actually help to um propel some of that by reframing how we even think about engaging with uh civil society.

SPEAKER_01 41:11

Yeah, that's that's really interesting because I've I've thought before, I've had conversations with people about whether the you know the decline in participation in more traditional forms of giving and the you know, whether there is some element of that being that people are shifting to other models of giving, whether that's giving to kind of non-traditional organizations or or just sort of peer-to-peer, and and whether that is a reflection of the traditional non-profit sphere having become too transactional. And an exact, you know, this would be an absolute case in point, isn't it? As you say, the presentation of um supporting civil society, as you say, in it framing it in terms of shopping for impact absolutely emphasizes that it's a kind of monetary transaction almost on a on a consumer basis rather than something where you are asked to to genuinely kind of participate and to give your financial resources and time and and whatever else, um, you know, uh on a voluntary basis to kind of support organizations. So um yeah, I think that question of whether it's uh which way around the causation runs there is really interesting. For for sure.

SPEAKER_00 42:18

And and I I I do I do wonder um, you know, if if in some ways the nonprofit responses that I've seen to these kind of this earlier wave of evaluative pressures has actually helped to uh propel this mindset and and and normalize it a bit. Um so the the really positive spin on the super arrogation story is that these organizations were able to reclaim what accountability meant to them. And uh they were able to do it in basically a new, often very quantified language, but they were able to maintain their distinctiveness and idiosyncrasy in the face of these kind of homogenizing pressures. But I've I've wondered if you know these organizations have upped the ante on quantitative evaluation if they've ended up uh crowding out other forms of knowledge or ways of thinking about engaging with these organizations. So it's like, yes, I I I know you first hand, but I'd rather see what the numbers say. You know, in some cases, maybe that makes sense, in others, you know, that's that's kind of a ridiculous thing. Uh, you know, if you trust to the daycare center where you send your kid, but you also want to see the numbers, it's it's like almost learning, unlearning to trust our gut. Um, so I do wonder if if in some ways that's a negative uh long-term effect of of this kind of change in the sector's uh you know, basic language of accountability toward a more quantified front, as opposed to a more face to face relational sort of uh accountability.

SPEAKER_01 43:50

Yeah, that's that's really interesting. Um and uh I kind of yeah, I wondered whether the fact that these things are intermediated, even if the even if the information that you're uh providing kind of through you know through super irrigation is intended to be for those supporters and uh you know whether it's the general public or funders actually because it's happening in an intermediated way through this this kind of a ratings agency or or a reporting structure like that you're not you're even though you're providing the information that you want to get to that audience you're not necessarily building the relationship of trust that might happen if you were speaking directly to that person or interacting directly with them. Um yeah it's I mean it's really interesting. Um I just just wanted to take a slight uh left turn here I'm in danger of taking up far too much of your time but I just I wanted to ask because you had another piece out uh recently on a different topic but I think the two do do link together a bit um which is about the the rise of um LLCs so limited liability companies as a kind of alternative model for a particular um type of philanthropist kind of elite philanthropist particularly often ones from a sort of tech uh technology and Silicon Valley background um and you kind of outline in that some of the narrative about why people you know what people say about why they're using this and what the potential benefits are and what some of the potential downsides are in terms of uh accountability and and uh and kind of transparency um and I guess I mean maybe to to link that to what we've been talking about already as a starting point in that you talk about you know the potential that actually maybe it might be that we see that that people using these LLCs even though the the regulatory requirements around transparency and declosure are are lower can we be optimistic that there will be a similar process of supererogation where they will decide to use them but then go above above and beyond in terms of how much uh you know information that they publish is there any you know should we be optimistic about that is there any sign of that happening as a process uh yeah I mean you're talking to a sociologist so I'm uh you know dispositionally a pessimist I guess but um yeah so in in in some ways uh you know there there's there's plenty to laud about a handful of LLCs because they they really don't have to be transparent at all that is what the law permits um and in fact that is part of the you know the attraction of the LLC to uh to philanthropists it's you know um unlimited for-profit investing so you're not kind of really stuck in a nonprofit sphere um unlimited political lobbying uh funding political campaigns there's no spending requirements um you can have cozy relationships with uh your for-profit uh you know partners which so like uh the Chan Zuckerberg initiative and uh and and meta and and you can dissolve it and take your money back but and you know above all in terms comes to uh you know accountability in terms of reporting things there are no reporting requirements as opposed to with a uh traditional philanthropic LLC or sorry philanthropic foundation so um you know the the LLC proposition in some ways is is basically saying um take me at my word um I'm you know trust me I'm rich and take me at my word I'm doing philanthropy and and hoping that their idea of philanthropy lines up with uh with your idea of philanthropy.

SPEAKER_00 47:18

Um so you know it's interesting when there are a handful of very prominent LLCs that are are going above and beyond what they're required to do. And again to be clear what they're required to do is basically nothing um and and you know reporting there is um I know the uh Mid Yard network uh publishes its investments financials uh various perspectives it has on on you know society on its website um doesn't have to do that uh CZI um has a searchable grants database which uh granted is only nonprofit recipients and at that it's not all of them and uh Laura and John Arnold um have been quite committed to openness and transparency and also very critical of DAFs and and other forms of giving that are possibly even less transparent. And we've seen Jack Dorsey with his you know infamous spreadsheet Start Small initiative where you know it's a Google sheet listing all the money he's given. So yeah, these organizations are going above and beyond in terms of transparency. But my critique is that in doing that, they are helping to legitimize an organizational form, the LLC, that basically leaves the question of whether or how billionaires should be held accountable to the billionaires themselves. And so I wonder if our attention to the good guys is allowing some of the uh less good guys to kind of um get by uh without any criticism. So it is a possible implication of of superarogation in this case is that it may actually allow others to sub-arrogate if you will or or simply uh you know completely skip on on all commitments to uh society and being accountable for for what they're doing their wealth.

SPEAKER_01 49:08

Yeah no that's I think that that is really interesting because um yeah it's true I guess you know interesting to note if there are people going above and beyond but if if the fundamental point is that you there's been a change in the nature of of accountability from one where it is something that is you know demanded as a matter of of duty or obligation um by you know a regulatory authority to one where we're just sort of hoping that the wealthy people themselves decide to do it. That that is a pretty fundamental difference and one that we should probably be pretty uh concerned about.

SPEAKER_00 49:41

I I think it creates a sort of uh almost public ambivalence about the the methods through which people are giving um so long as they're giving and I and I think my message in some ways is to is to say that the methods actually very much matter here um because some of them are just frankly unaccountable.

SPEAKER_01 49:58

And just I mean just on on LCs I mean it's a probably a topic for an entire podcast uh in itself I mean one of the things that gets highlighted as the you know the the the positive benefit of it because most donors don't come out and say oh I want to use it because I I find you know the reporting requirements of a foundation too too onerous and I want to be less transparent about my philanthropy it tends to be a narrative about not wanting to impose arbitrary boundaries upon themselves about the types of organizations they can invest in or whether you want to kind of blur the lines between for profit investing and and philanthropy. I mean do you think that that we should be concerned even about that positive narrative I mean do you do you think that there is something to be said for carving out a specific sector or a set of acts used to support it that are distinctly non-profit and to sort of get away from the idea that we should all be kind of free and easy about just blurring the lines between between the commercial sector and the nonprofit sector?

SPEAKER_00 50:58

So so that idea has kind of sent me for a loop recently and myself and a collaborator Michael Michael McElroy um have been thinking about if if this is the case and the lines are truly blurred between what is for profit and nonprofit because the ultimate question is social good, then in some ways the LLC approach to philanthropy is inviting a critique not just of philanthropy but of wealth and that these people even have the right to make these decisions at this scale about what society should look like. So yeah I I think I think there's definitely a line blurring thing um I I don't want to say like I don't think that businesses are you know inherently incapable of doing social good um or that uh you know nonprofits inherently do social good. And so I you know the line the lines are very blurry but I think I think the fact that they're blurring does open it up to a new line of critique that those of us who are critical of philanthropy um maybe should be you know setting our sights on an even larger target which is the the right of wealthy people to make these decisions in the first place.

SPEAKER_01 52:11

Yeah yeah absolutely and I guess I mean in terms of whether I think that distinction there between I think the yeah then the the sort of naive distinction sorry not to call you naive I'm trying to use it to to capture to capture the sense but not certainly not the idea that that doing social good would be the sole preserved of the nonprofit sector and you know businesses can can never have a social purpose because that clearly you know there are clearly many counterexamples that would that would disprove both of those I I guess the thing I I can't uh decide myself but I think is really interesting is uh is there something is there some unique value to the idea of doing social good in some context using models and methods that that have particular characteristics whether that's the kind of not no expectation of financial return or having you know some sort of legal um embodiment of the principle that you know once money's given away it can't be returned or that that kind of thing because or is it that actually over time we could replace all of traditional philanthropy with social purpose businesses or would something get lost in that process I I feel like the answer is yes definitely yes but but being clear about exactly what it is I think I need to think about it a lot more.

SPEAKER_00 53:26

Yeah so I I do think that there is something lost in that process and I think some of the fact that there is this blurring is that so much of you know as you were mentioning earlier the the nonprofit sector in philanthropy has come to feel very commodified and transactional that that makes it harder to distinguish from for-profit activity. It makes grant making an act of investing um and and I think some of that you know reveals a deeper shift from a focus on the act of giving and what that entails and and all the the social stuff around that the act of participating in community the act of you know identifying and and using your money in these particular ways and and refocused our attention on the effect of giving we've become consequentialists. And I think in some sense that has shifted our expectations of nonprofits um where they're not you know bulwarks of community or civic organizations, but they are low-cost service vendors that we expect them to do a thing. So I think uh in some ways maybe we've already lost some of what is uh worth redeeming about the distinction between charity and uh and more for-profit you know approaches um but I do think it's worth it's it's still there and it's worth redeeming.

SPEAKER_01 54:43

Yeah yeah absolutely um and you know positive no so much Steve said um listen I I could talk about this stuff for a lot lot longer um but I think it'll probably test people's patience but maybe I'll I'll you know twist your arm and get you back on at some point in the future and we can uh pick up there's a whole load of notes on here about things that we didn't even get around to talking about um but yeah no just just remains to say thanks ever so much Aaron for finding time to do it it's great to to have the chance to to chat really interesting uh stuff and um yeah I mean just a final call if you've got anything coming up or anything you want to sort of draw people's attention to uh now's the opportunity. I'll put links in the show notes to the stuff we've been discussing and the paper and the articles and things so that people can find that. But if you've got any other things that you that you want to plug uh please do take the opportunity.

SPEAKER_00 55:31

Uh right now I'm just kind of hard at work and thinking through um a a book uh so so I'm deep deep in that hole um but uh in the in the meantime if people want to check out my work uh the my my website is uh is the easiest repository of it so I just post everything I write there and um if people have questions about it email me I'm happy to chat.

SPEAKER_01 55:56

Cool and I'll yeah I'll put a link in the show notes to that as well so people can find you easily um yeah well thanks ever so much for doing it um great to have a chance to talk. Absolutely thank you so much. Okay great well my thanks again to Aaron for coming on the podcast. It was great to have a chance to talk to him get into some really kind of substantive questions about the the research that he's done and the kind of wider issues around accountability and measurement. As I promised I'll put links in the show notes to where you can read the paper in full and also um uh a shorter article on his fill um where Aaron's kind of summarised what the paper's about and what what the findings are and I'll put links to places where you can find more information on on Aaron uh his work and also some things that I've done that you might be interested in that are relevant. Um if you're interested more broadly in issues around philanthropy and civil society which I don't know if you've made it to this point in the podcast I'm guessing you are uh do check out the uh the website at whyphilanthropymatters dot com where you can find uh all of the episodes of this podcast you can also find lots of articles and bits of news and bits of video and things about philanthropy and civil society um you can follow me on social media I'm on Twitter at Rodri underscore h underscore davis you can find me on LinkedIn uh I'm on various other places as well with varying degrees of enthusiasm. If you've got ideas for things that we can talk about on the podcast or people that we could interview um do drop me a line. You can find my contact details on the Y Philanthropy Matters website and I know some of you have uh taken advantage of that recently and got in touch to suggest ideas or um people that I could talk to so thank you very much for doing that. And if you have the time and and the inclination please do uh go to wherever it is so you get your podcasts and leave a nice review so that people know that this is worth listening to I think that stuff all helps. Other than that um I will see you next time by the microphone