79017 THE WORLD BANK GROUP ARCHIVES ORAL HISTORY PROGRAM Transcript of interview with CATHERINE GWIN January 20, 2010 Washington, D.C. Interview by: Charles Ziegler 1 [Begin Tape 1, Side A] ZIEGLER: Today is January 20, 2010. My name is Charles Ziegler, a consultant with the World Bank Group Oral History Program. I have with me today in the Archives of the World Bank Group Catherine Gwin. As an outside policy analyst, she was employed briefly as a short-term consultant with the Economic Development Institute in 1994 through 1998. She began her career with the World Bank as a short-term consultant, and then a lead evaluation officer in the Operations Evaluation Department, which is now the Independent Evaluation Group [IEG]. From 2001 to 2007, she was the lead evaluation officer in IEG, and then since 2007, she has again been a short-term consultant there. Well, Catherine, I‟m glad you could come and participate in the World Bank Group Oral History Program. We‟ll start with some easy questions, some basic background. When and where were you born? GWIN: I was born in New York City in June 1945. ZIEGLER: Please relate something of your early life and education, again, just in general terms for background purposes. GWIN: Well, early on my family moved to Washington, D.C., because my father worked briefly for the State Department during the Marshall Plan years, and then we moved back up to New York State. I grew up in the suburbs of New York City, but something rubbed off in Washington because I was always after interested in politics, and then international affairs and international politics. ZIEGLER: Where did you go to university? GWIN: I went first to Mount Holyoke College in Massachusetts, and then I got a Ph.D. from the University of California, Berkeley. ZIEGLER: What were some of your work experiences before you joined the World Bank? GWIN: Well, I often have young people, and mostly young women, come to me for career advice, and my answer is, don‟t follow my line. It‟s sort of nomadic. I worked in government, briefly, during the Carter administration, but before that I worked with the Council on Foreign Relations, which is a U.S. foreign policy center. So I went from the Council on Foreign Relations, into government, out of government, to another policy research center called the Carnegie Endowment for International Peace, and then for a period of time I worked as a consultant for a variety of organizations in New York when I had a young child, foundations, the United Nations, universities. So, as I‟ve said, I‟ve been quite nomadic, but the basic thrust is on the policy research side rather than on the operational side. Catherine Gwin January 20, 2010 – Final Edited 2 I came down to Washington to take over as vice president for research for something that was then called the Overseas Development Council, and it was from there that I moved into doing the evaluation work at the Bank. Most of my writing in the policy area was around the issue of international development assistance, and U.S. foreign policy in particular. And I had done a fair amount of watching and writing about the Bank before I ever came here. ZIEGLER: Prior to your work at the World Bank, you wrote a book on U.S. relations with the World Bank, and it was entitled U.S. Relations with the World Bank, 1945 to 1992. It was a Brookings Occasional Paper [Gwin, Catherine. U.S. Relations with the World Bank, 1945 to 1992. Washington, D.C.: The Brookings Institution, 1994]. What occasioned the writing of this book? GWIN: It actually wasn‟t what I would call a book. It was a Brookings Institution “Occasional Paper� and a chapter that I was asked to write by the people who wrote the second major World Bank history published by Brookings, John Lewis and colleagues [Lewis, John, Richard Webb and Devesh Kapur. The World Bank: Its First Half Century. Washington, D.C.: The Brookings Institution, 1997]. They decided when they did that history to have two volumes: one that they as a team would write, and a second volume where they asked people from different countries to write about their country‟s engagement with the Bank, and this included both donor countries and recipient countries. Since I was one of the people on the outside writing about the World Bank, and writing about U.S. development assistance at the time, they asked me to write the chapter on the U.S., which I found very interesting. ZIEGLER: So that . . . GWIN: But it got published as a separate paper because the history took so long to be written that I said to them, "Look, this thing is going to be out of date, and I'll have to update it three times by the time you guys ever get finished. So let me publish it separately, and then you can include it as a chapter." That's the only reason that it stands alone. But the nice thing about it standing alone is, I have had several people who work in the U.S. executive director's office, most particularly Jan Piercy, who was the U.S. ED [executive director] for a good number of years, tell me that they found it a very good primer as they came to the ED‟s office because it does give a history of the United States' engagement with the Bank, as founder and as major donor over all these years. ZIEGLER: All too often executive directors don't do their research on the Bank, and just come in and here they are. GWIN: I don't think that's true of U.S. EDs. I can't speak for other countries' EDs. I have engaged with several of them, and I think they're pretty well briefed. They may not pay attention to it, but I think they read it. ZIEGLER: Well, that's encouraging to hear, anyway. What were some of the major conclusions that you arrived at in writing this book, or this work? Catherine Gwin January 20, 2010 – Final Edited 3 GWIN: Since it‟s a history, really, I was just trying to tell the history of the evolving engagement of the U.S. from what is pretty well known about the founding days, some little interesting wrinkles that not too many people know about, such as the argument between [John Maynard] Keynes and the U.S. as to where the Bank would be located, and how the executive board would work. The U.S., of course, won out because it had the money. ZIEGLER: Then. GWIN: I then talked about the evolving policy focus of the U.S., and because people need to understand it, the way U.S. policy towards the World Bank, and particularly its resource contributions to the World Bank, is decided. This includes the very important role that Congress plays, which is quite different than in other donor countries; and also the ebb and flow reflects the party in power in the United States. So I traced the U.S. engagement beyond the early days, where I think nobody would disagree that the United States was enormously constructive in its founding of the Bank, bringing in other countries and leading it, to later years where U.S. policy got more contentious and controversial, the ‟70s, when we had a secretary of the treasury, [William E.] Simon, who was not all that friendly towards the World Bank, in a Republican administration. Then we had a Democratic administration, which I came down on quite hard. ZIEGLER: This would be the [Jimmy] Carter administration? GWIN: The Carter administration, and Mr. [C.] Fred Bergsten, who was in Treasury, and in the policy position to be influencing Bank policy. It was at this time that the United States began to use IDA replenishments to push their policy issues because they had the power in IDA, as the lead donor, where they didn‟t necessarily always get their way on moving issues through the ongoing processes of the Bank. I think in the ‟80s and ‟90s, the whole IDA replenishment process became highly politicized. Maybe it was inevitable because it is a donor replenishment process, and now I think many donors view it as the place where they can push hard on the particular issues they‟re interested in. And then we have the interesting period of the [Ronald W.] Reagan administration, which came in as very anti-World Bank until the Mexican debt crisis. ZIEGLER: I do remember that, yes. GWIN: The administration turned around 180 degrees because it had to deal with the debt crisis in that period, and there was no other institution to rely on but the Bretton Woods institutions. ZIEGLER: I gather from what you were saying earlier that there were no major revisions made in the initial pamphlet in terms of what was published in the Bank history. That is pretty much the same? Catherine Gwin January 20, 2010 – Final Edited 4 GWIN: Well, my chapter was my chapter. ZIEGLER: Okay. GWIN: It was written. It was finished. It had gone through the peer review processes and so forth, and it was just sitting there, and other chapters hadn‟t been written. The Bank history hadn‟t been written, and I was very frustrated, and I said, "This won‟t do. I need to get this out. I‟m moving on to other things." So that‟s why it‟s the only piece that I think came out as a separate occasional paper. ZIEGLER: Yes. You were a member of the advisory group for this World Bank‟s 50th anniversary history. What, in your opinion, were the highlights of the discussions and recommendations of the advisory group? GWIN: I don‟t have a very good recollection of that. It was an interesting group of three men who wrote that history, different ages and different backgrounds. I think the advisory group was particularly focused on them to move on. They wrestled with some very big issues. They were the kinds of issues you could wrestle with for years, and the authors wrestled for too many years. Honestly, it‟s too far in the past for me to tell you what the advisory group‟s primary contribution was. I know at the time, because of the time that the book was written, the study gave a major focus to the Bank‟s perspective on and role in poverty reduction, in part because that was a major interest of John [P.] Lewis and the other . . . ZIEGLER: Richard [C.] Webb? GWIN: Richard Webb, and there‟s Devesh Kapur as well. We didn‟t disagree with that emphasis, but I think sometimes we thought they were in danger of drowning in their intellectual interest in this. So—but I really can't tell you, it's too far in the past. ZIEGLER: What is your considered opinion of the quality and value of this 50th anniversary history? GWIN: I think it's valuable. It's valuable for people who have a real deep research or historical interest in this institution because it's dense. It's not a light read. It wasn't meant to be a light read. We have a book much more recently about Mr. [James D.] Wolfensohn's era called The World’s Banker [Mallaby, Sebastian. The World’s Banker: A Story of Failed States, Financial Crises, and the Wealth and Poverty of Nations. New York: The Penguin Press, 2001]. You can practically buy that in the airport store and pick it up and read it. A very different purpose in that book, and probably much more widely read. Catherine Gwin January 20, 2010 – Final Edited 5 A history is a history, and I remember as a student and then as a researcher using the prior history that was done written in the 1970s. ZIEGLER: This is the World Bank Since Bretton Woods . . GWIN: Yes, yes. ZIEGLER: . . the 25th anniversary history [Mason, Edward S. and Robert E. Asher. The World Bank Since Bretton Woods. Washington, D.C.: The Brookings Institution, 1973]. GWIN: The 25th anniversary one. I have always found that useful. I have gone back to it over time. It covers the earlier years. The second one was really meant to pick up where they left off and do the next 25 years, but it‟s a very dense history, and I think it is, therefore, valuable for those who are part of the universe that knows that you can learn from history. ZIEGLER: A small universe, sometimes I think. GWIN: Well, I think institutions—you know, I think institutions need that. It's a way of trying to put an objective and historical lens on what's going on, and we don't do enough of that. ZIEGLER: In 1994, you became a consultant to the Economic Development Institute [EDI], which is now called the World Bank Institute [WBI]. What was the nature of the work that you did at that point? GWIN: It was a very short consultancy, very limited. There was a group pulled together because there was a new head [Vinod Thomas] coming into what was then called EDI, who wanted to re- think the mandate, strategy, purpose. We served as a sounding board. We were an outside group of development practitioners, development researchers, First World, Third World, however you want to call it. ZIEGLER: How large was the group? GWIN: I don't remember. I think it was pretty small, which meant that we could have some really good conversations. We did that over a short period of time. We‟d be presented with some papers, sort of think pieces to reflect on, and as I said, it was limited and I think it was useful, but not more than that, to the EDI folks, who were in the midst of coming up with a new kind of strategic plan. ZIEGLER: And so they actually did implement a new strategic plan at that point? GWIN: Over time, yes, they did. I mean, it was Vinod Thomas who was the one who took it from EDI to WBI, changed the name and broadened the purpose and built it up. I think EDI, up until that point, had had a very specific purpose. The Rockefeller Foundation helped with the creation of EDI, which is interesting. The idea was to have a place where you could bring policy makers from the Third World for sort of short term, very short-term, very Catherine Gwin January 20, 2010 – Final Edited 6 focused, very sort of practically-oriented training, discussion, interaction, and I guess over time people decided that it just somehow wasn't constructed as usefully as it needed to be, so it was re-done. ZIEGLER: While you were a senior vice president at the Overseas Development Council, you co-authored a book on the role of the World Bank in improving status of women. It was entitled Investing in Women: Progress and Prospects for the World Bank [Buvinic, Mayra, Catherine Gwin, and Lisa M. Bates. Investing in Women: Progress and Prospects for the World Bank. Washington, D.C: Overseas Development Council, 1996]. In the book, the authors, you included, of course, write, "Three concepts are salient in the Bank‟s current�—that is to say 1996— “discourse that supports efforts on behalf of women: a shift from women in development to gender; the need to mainstream gender issues in operations; and the importance of participatory project lending strategies. All three hold the potential for strengthening investments in women, but in reviewing project experience, this study finds reason to be concerned about the translation of these concepts into practice." Could you please elaborate on the three concepts that you wrote about? GWIN: Sure. I think our point was they were all good shifts, but there were significant caveats or potential downsides to the shifts, and that we wanted to bring out, partly, a need for the shift itself to be well understood, and then the need for it to be operationalized, and the potential trips along the way. The main aspect of the shift from women in development to gender was to introduce the issue of equality between sexes with particular focus at the household level, rather than just reduction of poverty, or increases in opportunities for women. The gender focus is really a focus on the idea of reducing inequalities between men and women based on understanding that the policies and actions needed to improve the lives of women had to take into account underlying structural, cultural inequalities. But there were potential downsides, we found, when you shifted to this focus on the introduction of equality between sexes, which initially focused primarily at the household level. The potential downsides were underplaying policy, institutional, and in the case of the Bank operations, project sources of gender inequality independent of the family or household origins, and that affect poor women rather than all women. So we were trying to make people mindful of that. Also, there was, we felt, at the time a misinterpretation by a lot of staff that to deal with both men and women would result in lost attention to women, which wasn't the point. The study's point was that gender analysis is an essential tool to complement—it doesn't replace—the focus on women. The gender approach gets at the underlying origins of women's position, if you will, in public and private life. But there‟s a need for focus on both women and gender equality. The second concept about mainstreaming was very much around at the time. It wasn't anything original that we were saying. However, in relation to the Bank's work on women, at the time what had been called WID—women in development—focused primarily on helping education, and the argument was there was a need to incorporate women's issues into many other aspects and policy programs, and not just stand-alone women's projects, and not just stand-alone Catherine Gwin January 20, 2010 – Final Edited 7 women's projects focused on maternal and child health, and girls‟ education. It went beyond that into areas of economic opportunity. Again, participation was an emerging notion at the time, and our caveat on that was not to argue against the importance of participatory projects, but as it was being applied—particularly in the case of poor women—there were political risks to women stemming from how they were being engaged. For example, women contributing voluntary labor to community-based programs, and on that we were trying to say, "That's not the good side of participation." So those were the three concepts. ZIEGLER: How was the study received by the World Bank? GWIN: My recollection is dim, but my recollection is that it got fairly little attention at the time because that was a time when the Bank was, overall, less open to and acknowledging of the value of work done outside the institution. Anything done inside was right; anything done outside was, “We don't have time to pay attention to it.� That‟s a bit of an overstatement, but that's my recollection. We're pretty far back in time. ZIEGLER: Has this attitude changed from then to now, in your perception? GWIN: The inside/outside? Yeah, and I think we might want to come back and talk about it. I think it‟s changed a lot. We can come back and talk about it. But has the gender stuff changed? Has the practice changed over time? And can we claim any contribution to that? I think it's a pretty interesting question. I think the Bank has moved in the right direction over time, but I would particularly highlight the fact that my co-author Mayra Buvinic subsequently became sector director and chair of the sector board in the Gender in Development Department of the Poverty Reduction and Economic Management Network in the World Bank. So I can't claim that short monograph did a lot, but Mayra Buvinic has done a lot. ZIEGLER: Well, that may have improved her thinking on the subject, also. GWIN: It certainly gave her an opportunity to sharpen up her thinking on the subject. She was coming at this from a career in research on women in development. I was bringing my knowledge of the World Bank, and so the combination was strong. I would say intellectually the manuscript really built on her basis of knowledge and experience of the subject matter, and so practice over time ran her way. ZIEGLER: Well, what's your perception of the role and status of women within the World Bank? How have they evolved over time in your perception? GWIN: Well, if you go back to—what did you say, '94 when I was first an STC [short-term consultant]?—it's changed dramatically, a major advance in my time as evidenced by women in high-level management positions in the Bank. I haven't done the number work, but I'm sure it's less than 50 percent still, but it's a whole lot different from 1994, ‟95, „96. This reflects what was happening in the world at large, and I don't think that the Bank gets any credit for being a leader Catherine Gwin January 20, 2010 – Final Edited 8 on this. I don't think the Bank was ahead of the curve in any way. I just think that women forced themselves on the Bank and proved their worth, and undeniably. So, yes, there‟s been change, but, no, the Bank wasn't a leading institution on this. ZIEGLER: But it is doing better now, in your . . . GWIN: Just count the number of high-level senior management women, country directors, vice presidents, et cetera. It‟s changed; certainly it‟s changed, but as I said, I haven't done the count myself. ZIEGLER: You were hired as a consultant by the Operations Evaluation Department, which is now called the Independent Evaluation Group, to be the task manager for an independent evaluation of IDA Replenishments 10, 11 and 12. What was the genesis of this evaluation, and what were its objectives? GWIN: It's interesting. This was absolutely atypical for what was then OED, the Operations and Evaluation Department. Let's just use IEG from now on, okay? ZIEGLER: Yes. GWIN: Atypical for the Evaluation Department, which sets its own agenda in agreement with the Board. What's very special about the Evaluation Department in the Bank—and it was the first in any major multilateral institution—is that it reports to the Board. It's not a creature of management. So it's considered independent because it sets its own agenda, and arrives at its findings and recommendations independently. ZIEGLER: And as I recall, the director general . . . GWIN: Yes, the director general. Go ahead. ZIEGLER: He, he or she—we have had any she‟s yet. GWIN: Okay. ZIEGLER: But that's their last job in the Bank. GWIN: That's right. ZIEGLER: So that gives them a degree of independence also. GWIN: Yes, and there's been some talk that it should be the same for senior management in the department as well, but it‟s always been the case for the director general. In this case of the IDA [International Development Association] 10, 11 and 12 review, the IDA deputies, i.e., the U.S. in particular, had demanded—you can call it commissioned, to put it politely—an independent review of IDA's performance in implementing the undertakings of the Catherine Gwin January 20, 2010 – Final Edited 9 IDA 10, 11 and 12 replenishments as background for the IDA 13 negotiations. The notion was, this review of 10, 11, and 12 and the Bank's compliance with what it had committed itself to do under those three replenishment periods had to be done before the conclusion of the IDA 13 negotiations. Why the focus on IDA 10, 11 and 12 instead of just 12 is because that period of time, those three replenishment periods were really a kind of new phase in IDA replenishments, a period of broadening the agenda, an emphasis on aid effectiveness, a strong donor push on specific issues, and the key question was the continued relevance of the policy framework, which for 10, 11 and 12 had been pretty similar. If you compare IDA 10 to IDA 9, you‟ll see a dramatic change. The IDA replenishment agreements up to that point involved the policy commitments, which were pretty broad and general, and the negotiations were previously around the amount of money, and the shares of participating donor governments. In IDA 10, when the United States felt a need to get specific policy commitments into the replenishment agreement to get the U.S. replenishment contribution approved by Congress, the IDA 10 process took on a very different character, and since then it has been like that. So, 10, 11 and 12 were a set, in a way, and the policy framework didn‟t change very much over that period. Ten was the first that really had a policy framework of any . . . ZIEGLER: Was that one politically motivated, or . . . GWIN: Totally politically motivated. So, that was the genesis of the IEG evaluation. The objective and the approach was consistent with what donors commissioned, which was to examine IDA‟s compliance with the major substantive undertakings of the three replenishments, of which there were, when we counted them up, some 150 substantive undertakings—that is to say, what the Bank committed itself to in substantive terms— that fell into ten broad issue areas. Six were programmatic areas, and four were process areas. The six major program areas were: poverty reduction, social sectors, private sector development, gender, environment, and governance; and the four process areas were a performance-based allocation system, country assistance strategies, aid coordination and the participation of beneficiaries—operations. ZIEGLER: I understand that this was a high stakes evaluation . . GWIN: Sure. ZIEGLER: . . in that Bank management felt, or professed to feel, that the very future of IDA depended on the outcome. Also, it was a very complex evaluation to design, as is clear from what you just said. How did you cope with the pressure to maintain your independence? GWIN: The high-stakes nature of it and the broad scope were both well understood and made manageable for me by Bob [Robert] Picciotto, who was the director-general of OED at the time. I was very much helped by the reputation that he had created for OED as a serious independent evaluation department, and also very much helped by an early comment that he made to me which I'll never forget, which was, "Keep your head down, do you work, and I'll see that you're protected." And that made a huge difference. Catherine Gwin January 20, 2010 – Final Edited 10 So we were in good standing, in addition to which the unique arrangement we made whereby— this was, again, atypical—we did some interim reporting to the donors as well as the Board, so there would be no big surprises at the end because this was very high profile, and because people, certainly Bank management, felt that it would have a bearing on IDA 13. It seemed important to us to try to keep relevant stakeholders informed, and to not wind up at the end just unloading our findings. Also, I had a very seasoned team of people to complement my newcomer status because, after all, I was brought in essentially from the outside. I wasn‟t in the Bank before. I had never worked at the Bank at the time. I was brought in from the outside to head up this monstrous, high profile evaluation, and I was very grateful for a strong team of people who knew the Bank, had experience with the Bank, had guts, and were good researchers. ZIEGLER: Were there any other particular challenges or difficulties you encountered in managing this evaluation, and, if so, how did you address them? GWIN: There are many challenges, but the first and foremost was starting with the scope of it. We were supposed to evaluate all of IDA, all of what it had committed itself to do, which meant to a very large extent it was a meta-evaluation. It wasn‟t a typical evaluation where we could drill down deeply into a particular area of focus. So we had to rely on a lot of existing evaluative material from IEG and from outside. Then we were faced with the additional challenge of the unevenness of the quality of the existing evaluation evidence across the ten issue areas that we covered. In addition to that was the varying degrees of established understanding and practice in those different areas, such as governance and gender, which were relatively new. Notably, governance was much on the radar screen by IDA 12, but it was pretty new, and there was not clarity on what one donor meant by governance versus what another donor meant, what was written into the replenishment, what anybody in practice meant by it. So I would say all of those things—the scope, the quality of existing evaluation, the variation in the understanding and practice in different areas—were major challenges. A fourth challenge, in this case and in all of IEG's evaluations, is that our reports end up speaking to multiple audiences. That's always an issue, but especially for this evaluation because it had actually been commissioned by the donors, intended by them to serve their policy environment. I mean, they wanted this done because they wanted to be able to say in their parliaments and their legislatures, "Okay, there's all this yelling and screaming about the World Bank. We commissioned an evaluation. Here's what it says about what it does well." So there was a policy audience for the report right from the get-go, but at the same time it had to speak to a practical audience. It wasn't just written for policy-makers; IEG always writes for the Board. That is its first and foremost audience for everything it does. It reports to the Board. But beyond the Board, it always intends that its evaluations speak to the management and staff side of the Bank, as well as the outside audience of researchers, practitioners, NGOs. So that was a real challenge, to figure out the useful level of analysis across this very broad scope of issues. Catherine Gwin January 20, 2010 – Final Edited 11 ZIEGLER: Was the independence of your group generally accepted? GWIN: By the outside world? ZIEGLER: Yes, yes. GWIN: Oh, probably not. I remember getting some complimentary reviews in various places; what nowadays we call it blogs, but some that said, in effect, "Oh, this is an establishment job." I think I had a very good team. The fact that all of us, except for me, had had internal Bank experience was useful, while I was the one responsible for the report. ZIEGLER: Yes, yes. GWIN: So we were all the authors of the report. But I didn‟t pick . . . ZIEGLER: Who were some people? GWIN: Who were some of the people? ZIEGLER: Yes, yes. GWIN: Well, Stephen [D.] Eccles, who you know well, looked at the issue of performance- based allocations, and I think that was one of the most creative and one of the most controversial parts of the report, which we‟ll come back to. The person who worked on governance was Pierre [M.] Landell-Mills, who was really, I think, a leading former Bank staff person on the whole area of public sector governance, anti-corruption, and he did a very good piece of work for me. A woman named Alison [Margaret] Evans, who had been in the Bank and had worked in IEG. Right before we got started, she had finished an IEG evaluation on the Bank‟s poverty reduction strategy, which was a very good piece of work. Because this was a meta-evaluation, I drew very heavily on Alison‟s work, but she also contributed a few additional pieces of analysis that were specific to our task of assessing IDA compliance. She was very strong analytically, and she had a lot of credibility with the outside world. I don‟t need to go through all the other people, but we did also have an advisory committee. Part of the reason for that was to say, okay, we‟re a team. Now we want an outside advisory committee to be a sounding board for us, to beat up on us if they think we‟re not doing a good job. ZIEGLER: Yes, I wanted to ask you what the main contributions of that committee were to this evaluation. GWIN: You know, that idea about an advisory committee for evaluation is more standard practice now than then. I think you were starting to ask me something before about what my experience coming in from the outside . . Catherine Gwin January 20, 2010 – Final Edited 12 ZIEGLER: Yes. GWIN: . . brought. I think one of the things I brought to IEG were ideas like this. It wasn‟t so much standard practice at the time, and I said when I was doing this IDA review, I have to have an outside group to give me additional credibility because I am using a team of experienced people. There‟s no way you can do this job without that, but I want there to be one more arm‟s length, pair of eyes on us. So I saw the value in getting reactions to our work from expert representatives of different stakeholder groups: researchers, donor representatives, client representatives, and especially if you think back to when that was done, this was a climate in which there was a lot of NGO [nongovernmental organization] criticism of the World Bank. It was a noisy, noisy period of time. ZIEGLER: Fifty Years Is Enough, for instance. GWIN: Yeah, all of that. So we had also some people from some thoughtful NGOs. It was a good mix of people. In addition to the advisory group, though, we also did some other things that were useful to help with getting sure that we weren‟t too much of just an insider look. We had various country consultations, and we held a couple of international workshops at both the design stage of the evaluation, and an international workshop when we had emerging findings, and we asked our advisory committee to be part of that. The advisory committee was kept pretty small, to make it manageable. We asked them to participate in what was a larger international workshop. So what were the main contributions? In thinking back I‟m not going to remember all of them, but the two impacts I clearly remember were that they toughened our spine on what we were saying about IDA shortfalls, particularly in regard to poverty and the role of donors, and the way in which donors tried to push the Bank around in their own interests, and at the same time they helped us avoid over-claiming on what we found. So they really kind of added rigor to the process. Then we would claim we found the Bank has done very well in this . . ZIEGLER: Right. GWIN: . . or pretty well on that, and they‟d say, "What‟s the evidence?� You know, “Are you over-claiming here?" ZIEGLER: Yeah. GWIN: "Can you really say that so firmly?" So in that sense the advisory committee really helped. I can't say that we said anything where I heard, “You're dead wrong.� It wasn't so much that; it was more, as I said, the toughening our spine, making us say what we were saying with solidity. Catherine Gwin January 20, 2010 – Final Edited 13 ZIEGLER: With some empirical evidence. GWIN: Well, of course we had empirical evidence. That's what evaluations are. It's full of empirical evidence, but looking at the empirical evidence, to both tell it tough enough and at the same time to say maybe in this area you think you're saying the empirical evidence says more than we think it says. So both ways. ZIEGLER: Interpretation of that evidence, in other words. GWIN: Yeah, yeah. I mean, evaluations are full of evidence. That‟s what they‟re about. Okay. ZIEGLER: I also understand that there was a divergence of views between management and IEG on the outcome ratings. How did the respective views differ and how, if at all, were they reconciled? GWIN: There are always differences between IEG and management over evaluation ratings. It‟s a game—it's not a game, but it's standard operation. This was pretty serious. We had some pretty long, drawn out, serious back-and-forth with them. It is spelled out in detail, and I think, worth kind of looking at in writing, in the footnote of the overview of the report, and again in the text. ZIEGLER: Who was the . . . GWIN: Who spoke for management? ZIEGLER: Yes. GWIN: Standard practice in the way an evaluation is done in IEG is we write the evaluation. When it's in draft, it goes to OPCS [Operations Policy and Country Services], and OPCS has the responsibility of getting and synthesizing comments from whomever in the Bank they decide are the relevant reviewers, depending on the subject matter. While that is fairly obvious for topic- specific evaluations, on the environment or this or about that, for the IDA review every VPU [vice-presidential unit] made comments. OPCS is responsible for combining their responses and giving them back to us, and we will respond back to them and say, "This is a good comment. You're right. We need adjust the facts," or, "We don't agree with that. The facts speak otherwise, and we won't change that." They have the opportunity at that point to say, "If you don't change that, we will write a footnote saying management disagrees with this point." Ninety-nine percent of the time the back-and-forth ends up with little footnotes. Well, maybe not 99 percent, but I'll bet you almost every evaluation has at least one footnote. This one had two, and one of them was on the issue of “partially satisfactory� on the outcome rating. I quite honestly had argued that unlike other evaluations, this was just too broad to use our sort of practice of grading, and dear Bob Picciotto—and I give him credit for this—said, "No, that's chickening out. We're going to rate this, and it matters, and we're going to do that." Catherine Gwin January 20, 2010 – Final Edited 14 So, what was the difference? In brief—because it's an interesting difference. The rating could have been “highly satisfactory,� “satisfactory,� “partially satisfactory,� “partially unsatisfactory,� “unsatisfactory,� or “highly unsatisfactory,� a six-point scale, so that you can't wind up in the middle because if you have a five-point scale, everything always comes to the middle. So we didn't say “partially unsatisfactory.� We felt we were pretty generous. Well, we didn't feel generous. We thought we were absolutely dead right. “Partially satisfactory.� Management's view was that that rating did not capture either the progress over the period covered by the review, and that that rating confused the quality of IDA's performance from the development outcome intended. And it was management's view that we were rating things that were outside IDA‟s control. But that‟s exactly right. That is what those ratings do, and it‟s spelled out—it's carefully spelled out that we're not rating IDA's performance. It‟s not a performance rating. It's an outcome rating. The rating was the cause of a knock-down, drag-out fight with management. I think they were worried about how our partiality sat would influence the IDA 13 replenishment. The point that we were not rating the quality of IDA's performance, but rather the development outcome, they just seemed to not get for the longest time. Several years later I was walking the corridors—and I won't say who it was—but a very senior management person in the position of arguing with us over that came up to me and said, "Catherine, I finally understand what you guys were getting at." So that was progress. That was at least two or three years later. The donors understood completely, as we actually were told, but let me give you two examples of where this happens. Do you want me . . . ZIEGLER: Yeah. Can I stop you just for a minute? [End Tape 1, Side A] [Begin Tape 1, Side B] GWIN: We were talking about IEG management disagreement on how we came out on the IDA rating of partially satisfactory, and I said to you it was their thinking we were rating the quality of IDA‟s performance, rather than the development outcome. It‟s a little bit of an esoteric point, so let me say this. Management emphasized the emergence, over the period we were looking at, of the low income business models grounded in PRSP—the Poverty Reduction Strategy Paper— and the CAS [Country Assistance Strategy], and it's true. Those processes emerged importantly in this period of time. IEG's conclusion was that greater progress was needed in links between the CAS and poverty reduction to claim achievement of objectives. At that time, the Bank would say this country program is all about poverty reduction, but the link was not really there in the country, not so much in the strategy paper, but in the actual strategy at work. So, yes, they were saying, "Well, look. There's this new emphasis." And we were saying, "Yeah, but actual progress is still limited." Secondly, on gender, another example: We said there was substantial increase in support for gender activities in health and education, especially maternal health and girls‟ education, but concluded that over this review period, which is, after all, quite a number of years, IDA hadn't Catherine Gwin January 20, 2010 – Final Edited 15 made sufficient progress in incorporating gender into other areas of activity, particularly supportive with these economic advances. So they were saying, "Look at all we have done." And we were saying, "No, look at the development outcome." So those are two examples. Okay? ZIEGLER: Are there other, just in general, important conclusions to this evaluation? GWIN: Yes. In findings, in each of the ten areas there were main findings, and I certainly don't want to summarize them. ZIEGLER: And they're available in the report? GWIN: In the report. ZIEGLER: Yes. GWIN: But there are some over-arching points at the more general level that I think are kind of interesting, and I will mention three. On the general point of IDA‟s development effort and processes, we said very clearly that IDA, at the midpoint of IDA 12, which is where we were when we did the evaluation, was different in important ways from where IDA was at the start of IDA 10. No doubt about it. Country programs, lending instrument adjustments, repositioning of staff to the field; portfolio performance had steadily improved, though there was room for more. But it could have done better in a number of ways, which we identified. Again, our findings are in the report, so I will just mention one or two. First, if it had moved more quickly in the mid- ‟90s to increase attention to governance and the institutional impediments to country development. We were quite hard on the Bank; though it had acknowledged the importance of good governance, it was still moving very slowly. Second, if it had directed more of its analysis and dialogue to identifying ways to improve rural development; that was a long discussion. Some ways to better improve capacity building in client countries, and some other points. Okay. So those are some of the areas we cited in which better efforts were needed. Another thing we spoke about, which I think was terribly important, was that the donor‟s approach to replenishment agreements needed to be improved; that when you look at three IDA replenishments and find 150 things that the Bank was in some sense forced to say, "We will do this. We will do that. We will do the other." It‟s ridiculous, and we were saying that the donors needed to consolidate the IDA mandates, develop a longer-term vision focused on results, engage developing countries in setting those replenishment commitments, and define commitments much more in terms of what was monitorable and achievable within a realistic costing, because a lot of stuff that‟s written in there is just out of line. I think we had some, though limited, impact on the donors, in fact. Certainly the idea of engaging developing countries in discussions around IDA replenishments happened subsequently. I think it began in IDA13, and has continued on since then. ZIEGLER: And didn't Mr. Wolfensohn have the view of putting the countries in the driver's seat? Catherine Gwin January 20, 2010 – Final Edited 16 GWIN: Yes, but that wasn't around the replenishment. That was around country programs, and that did come along at this time. The third thing—the broad area that we talked a lot about that I think was a major push in our report was IDA‟s role in promoting greater donor harmonization and coordination. IDA needed to play a much more proactive role. It was the big boy, and somebody had to take the lead on it, and it needed to change its own practices, and it needed to engage the donors much more proactively. If I said that now, it would sound silly to you because we talk about it all the time. Back then it wasn't happening nearly enough. ZIEGLER: So those are really some of the long-term effects of this evaluation. GWIN: The long-term effects, I would say it‟s always hard to say with IEG evaluations, in part because we‟re always reviewing a moving target, and management can always say to us, "Oh, we‟re already doing that." And with this, it was mainly in the areas of synthesis of existing evaluations, so how much can I claim is long-term effect? I think there were four. First of all, the way of doing business inside of OED. As I mentioned to you, I used advisory committee, I did outside workshops. I think that it was happening a little bit, but I think I gave a big push to that. Second, in one particular area where we, again, had a huge fight with management: performance- based allocations. The paper we did proposed reforming the system, and they fought and they fought and they fought, and subsequently they reformed it in the way we were pushing for because it was needed. That‟s very specific, and it isn‟t going to make headlines in The New York Times, but it was very important, actually, because it affected allocation of resources to recipient countries. And I would say we added some wind to the sails of some emerging issues. Governance and the replenishment process, as I‟ve already mentioned, were two of those. Also, maybe most importantly, the credibility of doing independent evaluation work was a long-term effect of this. What I mean by that is because this has been asked for by the donors, the reporting on this particular evaluation was dual. We reported both to the Board at the Bank and the IDA deputies; completely atypical. When the report was finished and we took it to an IDA deputies meeting, every single IDA deputy around the table said, "I have to tell you, this will help us in our policy discussions back at our capital. It‟s very unique to have an independent, objective, and honest evaluation like this." And management, of course, had been worrying all the time that we were going to kill IDA, you know, this was going to be the death to IDA. Absolutely to the contrary, every single one of the donors said the same thing. There was a growing emphasis by donors on results, impact evaluation, and independent evaluation, everything bubbling up at the time, and I think—I'm not saying this is the only IEG evaluation that proved this—but I think it was clear that the credibility of independent evaluation .. ZIEGLER: Was enhanced by this. Catherine Gwin January 20, 2010 – Final Edited 17 GWIN: . . was shown, was enhanced, yes. ZIEGLER: Hmm. In 2001 you joined the World Bank . . GWIN: Yes. ZIEGLER: . . as a lead evaluation officer in OED—or IEG. GWIN: Yes. ZIEGLER: What were the circumstances that led to your acceptance of this post? You were a, quote, unquote, “outsider� before this, for the most part. GWIN: Well, I was actually asked by OED if I wanted to become a regular staff member while I was still in the midst of doing the IDA review, and I said, "Wait a minute, guys. You can‟t ask me that. You brought me in from the outside. I‟m not going to even answer—I don't even know yet. Let me see how this turns out." I did know. I was quite fascinated by the work, but I wasn't going to say. So, when I finished the IDA evaluation, I decided to stay. Why? My long-term interest in the processes and effectiveness of development assistance, and special interest in the World Bank. I had certainly seen the value of this kind of independent evaluation work. I hadn't done evaluation before, but I had done certain policy work, however frustrating evaluation work is, and it is frustrating because you never really know what effect you have. You can just give it your best shot, and hope that you present it in a way that makes sense. ZIEGLER: The specific unit of IEG that you joined was Corporate Evaluation and Methods. It's an interesting title, I think. Please describe, in general terms, the mandate of that unit and your role in it. GWIN: Yes, it has evolved over time. The corporate has stayed, the methods—at that point in time, Bob Picciotto was director general, and he was really working on strengthening the evaluation methodology, and the several different units in OED were substantively oriented, so some were doing primarily country evaluation work, some were doing sector thematic evaluation, and corporate was looking at cross-cutting issues like this kind of, like PRSP, like things that neither were sector nor country specific. But we also parked the work on how to improve our methodology there, and in addition to that, that unit in OED always did the annual report that used to be called ARDE—the annual review of development effectiveness—again, an invention of Bob Picciotto‟s, which was an attempt every year to kind of synthesize findings, show trends in whether things were improving or not, and usually focus on particular sub-issue along the way. So that sat in that corporate unit, so it was cross-cutting, not single country, and not single-sector or thematic work for the most part. And my own—you asked me about my own role, I guess. ZIEGLER: Yes, and your role in it. Catherine Gwin January 20, 2010 – Final Edited 18 GWIN: I was not a methodology person. I had no evaluation background when I came in, and I learned it on the job. So my role was not on the methodology side, for sure. I led the IDA review, then I led three other very broad-ranging evaluations, each time on an entirely new subject and with a team of people, and each time on a subject that people would sort of shake their heads and say, "How are you going to evaluate that?" I would say, "I don't know. It's not like what are they doing in the area of transportation.� I mean, you can evaluate that, but I would have these crazy subjects. ZIEGLER: Clearly, you're very adaptable then. GWIN: [Laughter] Or foolish; I‟m not sure which. ZIEGLER: In 1996, at the behest of President James D. Wolfensohn, the World Bank launched a comprehensive knowledge-sharing initiative, based on the understanding that development knowledge is a global public good that belongs to everyone, and from which everyone should benefit. The main objectives were to develop a world-class knowledge management system, and to improve and expand the sharing of development knowledge with clients and partners, as well as among Bank staff, in order to improve the quality of Bank operations, and to enhance the capacity of client countries to achieve their development goals. You were the task manager in an evaluation entitled Sharing Knowledge - Innovations and Remaining Challenges, completed in 2003 [Gwin, Catherine, Sharing Knowledge - Innovations and Remaining Challenges. Washington, D.C.: The World Bank, 2003], which evaluated the World Bank‟s knowledge- sharing initiatives. How did you proceed to evaluate such non-traditional World Bank products? And of course, this is, I imagine, just what you were talking about a moment ago. GWIN: Yes, that‟s an example of the corporate unit‟s cross-cutting subject matter. How did we proceed to evaluate this? There were three steps that we took because it was kind of tricky. First, we spent time learning about evolving knowledge practice in the corporate world and other large institutions, and this included actually hiring a consultant for a short period of time who was a knowledge management officer at a very major corporation, and another consultant who had worked in this field to do a literature review for us, and so we would understand what this practice was at large, in order to look at the Bank and see how it was taking this evolving practice and adapting it to its own use. The second thing we decided was that we needed to do the evaluation in two phases, which is sometimes, but not often, done. The first phase was to develop an evaluation framework and to write that up as a paper, and to get buy-in on that from the management side of the Bank, particularly from the Knowledge Management crowd in the Bank, so that it was then understood that we would undertake our assessments using that framework because there wasn't an existing Bank framework on which to base our evaluation. So we created the framework for the assessment, spent a lot of time, good time, very productive time, talking with people about what they thought they were doing, what they thought the framework was; very congenial and cooperative interaction on that, got the buy-in on the framework, and then undertook the evaluation. Catherine Gwin January 20, 2010 – Final Edited 19 The third step of this evaluation was a very, very, very active consultation process because we needed to understand what they thought staff were trying to do and the problems they were encountering, and at times I almost felt like a therapist. We do consultations every time we do an evaluation, but I would say we did more in this one. Sometimes, you know, the practice is so well established you don't need to really spend a lot of time; you just go in to looking at what was done, what the Bank did, what were the outcomes. ZIEGLER: This was so unusual, or so new in terms of practice. GWIN: It was new, and it was—no, there wasn't a shared understanding of it, and it meant some things to some people and some things to other people. So—what is that phrase about what part of the elephant you touch? You know. ZIEGLER: Yes, yes. GWIN: So we really needed to get both an understanding for ourselves and an understanding of the views of the people who we were going to be evaluating on the aims of this Knowledge Bank and how it was supposed to operate. There were big questions because of the way Mr. Wolfensohn presented it, the way he presented lots of things, sort of just throw a ball up in the air and say, "Okay. Here‟s an idea. Now you go figure it out." It was, on the one hand, intended, as you said just a minute ago, to much improve the management of the knowledge that the Bank had on hand, so that staff was actually doing their work in a way that was as evidence based as possible. They always intended to do this, but with all the technology improvements, it was possible to make it so much richer than it had been. So, there were objectives aimed at staff; objectives aimed at sharing Bank knowledge with the outside world; and objectives specific to building knowledge and knowledge capacities of client countries. All of those were aims of the Knowledge Bank initiative, and there were all kinds of activities going, with some people focusing on one thing, and some people focusing on another, and the big question of how do you cohere that in a way that has real effects on development outcomes. I think what has happened over time—and I haven't continued to follow this closely, so I can't tell you in detail—the word "knowledge services" came into play somewhere along the line. The Bank was not only in the business of providing financing, but knowledge services. This is still much in flux; there's still a lot of grumbling about what we exactly mean, but I think it's gotten somewhat more strategic. ZIEGLER: What were the major conclusions of this study? GWIN: We said that we thought the Bank had done the right thing in committing itself to a major knowledge initiative, the rationale being to improve development outcomes by fully exploiting both the revolution in IT [information technology] and the Bank's comparative advantage as a source of development knowledge; both were highly relevant. From the start, the Catherine Gwin January 20, 2010 – Final Edited 20 Bank moved quickly to upgrade its IT capacities. It still does that constantly now, but it was a big improvement quite quickly; establishing new knowledge-sharing programs and activities and technological capacities and so forth; and created a much more open knowledge-sharing organizational culture so that as a result both staff and clients gained easier access to knowledge and information. The change in the organizational culture was not unimportant. In talking to key people early on, any number of staff—and I was really fascinated by this—would say, "You know, it's not been the culture of the Bank. The intentions aren't to share the knowledge you have, but to keep it close to the chest, and you use it and you benefit.“ ZIEGLER: I've heard that. GWIN: Now look, if anything, staff are overloaded with knowledge, but as much as the IT was important, the IT wasn‟t anything without the organizational culture change, and I think staff were very happy with that. I think they were much happier to be in that kind of an environment. ZIEGLER: Do you have any idea what would have engendered such a change in the culture? GWIN: The whole initiative depended on it. I can‟t say that staff incentives are yet wholly adequate, but there were some steps taken to actually change incentives, including performance measures. ZIEGLER: Performance evaluation? GWIN: Yeah, and written into that became things like team building and knowledge sharing and so forth. So there was a push inside. There was a recognition by the Bank that if they were going to be like any other major institution out there in the world, they had to get much better at this, and to get much better at that the staff had to understand that this was a good thing. So a lot contributed to that. Do the incentives go far enough? I don‟t know. All I know is that staff told me that it is different now. So that was the second finding, but the third finding, which we really emphasized, was the existence of two major shortcomings of the strategy. First, there was a failure to take the strategy of knowledge aggregation and dissemination to the next level providing direct support to task teams in their operational work. The networks‟ knowledge management sharing activities tended to focus on writing papers and putting them up on the website, and having good websites and making them accessible to everybody. We said, "That's fine, but the Bank needs to devote much more time and attention to working directly with frontline staff." This is a much bigger matrix management problem. I still think this is a huge problem. Operational teams need to do more to capture and feed back systematically the lessons and experiences from on the ground because, after all, this is where the Bank‟s comparative advantage comes from; and the networks‟ Bank- wide efforts need to improve the content management of the distributive knowledge to improve quality, timeliness, and operational relevance. Catherine Gwin January 20, 2010 – Final Edited 21 And so, all three of those things—more focus on front line staff; more effort by operational staff to capture and feedback systematically the lessons; and more rigorous content management of the shared knowledge—were needed. Otherwise, you're putting junk up on the website, and the poor outside viewer would have to determine what was good and what wasn‟t. The second shortcoming was the failure of units, country teams, and task managers to develop more strategic approaches to the knowledge dimensions of the Bank services to clients. I think that‟s come along a lot since we wrote our report, but we were looking at the knowledge initiative about five years out, and this dimension was not evident yet. It wasn't really connecting to clients. It was very focused on serving us. The main objective is to improve the development outcomes, so serving us is only one step in that. Now you've got to connect to clients. For that we said operational staff needs more incentives, and relates to the subject of another evaluation I did subsequently on capacity building, but we can come to that. ZIEGLER: I was going to ask—well, you‟ve touched on it, but just to clarify it—in the ensuing years, do you think the World Bank has made further progress in sharing knowledge to achieve development goals? GWIN: You can look at the technology and see an explosion of ITC-enabled ways of enhancing access to Bank knowledge and bringing global knowledge into the Bank. No question about that. The other aspect of knowledge services for clients has come along. But I honestly haven't stayed close enough in touch to say how far that's come along. The little bit I know from afar suggests continuing progress. ZIEGLER: In . . . GWIN: But let me just say one last thing. ZIEGLER: Oh, sure, sure. GWIN: I think there is still a big issue of where the Bank's particular comparative advantage lies in terms of knowledge for whom, and how the Bank helps countries to link knowledge to practice. I don't think the Bank has really got a sharp enough perspective of its own on its role that way. ZIEGLER: In 2005, you were the task manager for a study of World Bank support for capacity building in Africa [Capacity Building in Africa - an OED Evaluation of World Bank Support. Washington, D.C: The World Bank, 2005]. This evaluation assessed Bank support for public sector capacity building in Africa over the previous ten years, based on six country studies, assessments of country strategies, and operations across the Region, and review of the work of the World Bank Institute, the Institutional Development Fund, and the Bank-supported African Capacity Building Foundation. What were the main results of the evaluation, and what were the principal recommendations? It‟s another one of your complicated—or complex, I should say— evaluations. Catherine Gwin January 20, 2010 – Final Edited 22 GWIN: Not clear cut, yes, that‟s for sure. Our main finding was that capacity building is not a practice that is well defined nor well understood after years and years and years of talking about, and it is not an aspect of development in which the World Bank has done well, except in some quite specific domains. Nor has the Bank determined where it does and where it does not do capacity building well. More specifically, we found that in recent years country assistance strategies and programs have done better in identifying capacity needs to make Bank operations work. But there is lack of a strategic or comprehensive approach to capacity building that addresses the three—call them if you will—nested dimensions of capacity building: institutional, organizational, and human. Rather, the Bank has often provided fragmented support for these three dimensions of capacity building. Some of the least effective support we found involved isolated training programs for government staff. Is it really effective to send some government people off to a five-day training program, and then bring them right back into an organizational context where they don't have the resources to apply what they‟ve learned, they don't have the incentives, their jobs change after they got back to something else? So the training is irrelevant, but they go because they get paid to go, and they like it. Hey, why not? Recognition that capacity building is a complex process—involving institutional, organizational, and human capacity has those dimension—had not been incorporated into a comprehensive strategic approach at the time of the evaluation. I doubt that such an approach has yet been established—within the Bank or other development institution. Another thing we said was that capacity building objectives and individual projects were often not well defined, not well tracked, except when capacity building was the core project objective—an exception in an increasing, but still few, cases. Usually capacity building wasn't the core project objective. Something about, say, infrastructure was the core objective, but there was some capacity building component to it. So only where it was identified as the core objective was it well defined and well tracked. We also said that how you get success in an enhancing public sector capacity varies tremendously across sectors as well as across countries. This point elicited a lot of argument, but I think we've been proven right. The reaction was to emphasize variation in capacity building success across categories of countries, such as fragile, poor states, and middle-income countries. However, we examined efforts in several sectors and found that the variation was greater across sectors than it was across countries. That is, where the Bank had had greater success at capacity building could be identified in certain sectors and not in others. So one of our recommendations was to develop more sector-specific guidance on capacity building. Another main finding was that the Bank hadn't developed a body of knowledge—knowledge or tools and approaches—to be used in different country and sector circumstances, and was lagging in having the knowledge base for doing this work. So we made four recommendations: 1) at the corporate level, to strengthen the Bank‟s knowledge base and operational framework; 2) develop sector-specific guidance on diagnosing and responding to capacity needs; 3) use country Catherine Gwin January 20, 2010 – Final Edited 23 assistance strategies more effectively to help the countries identify high-priority capacity building needs; and 4) undertake a serious review of how it supports training because . . . ZIEGLER: Training for? GWIN: Especially public sector training; it throws money at training workers in the public sector. ZIEGLER: Oh. GWIN: And we recommended that the Bank review its record of support for training. You really need to—not the history, the experience on this entry. Well, the Bank didn't do that, but IEG did a training evaluation two years later. Did we have any effect? I haven't followed it, so I can't say. The focus of the evaluation was capacity building in Africa. At the time, the Region had a task force on capacity building. I don't know if that withered on the vine or not. I really didn't follow it, but at the time it had some “oomph,� and it had some effect on giving some people inside the Region some responsibility to promote improved capacity building assistance. My current view, but from afar, is that capacity building remains one of the least successful areas of Bank work, in part because of lack of incentives for staff, and in part because the practice remains very badly defined. ZIEGLER: How were the results of this evaluation received within the Bank? Did you find levels of agreement or disagreement? GWIN: I think we found lots both. Well, I pointed out one thing where we got a lot of pushback, and that was on sectors, surprising to me. But I think over time people started to say, "You know, you're really right." So I think I would say disinterest more than disagreement. The Africa Region had its own framework going, and it had its own spin, so I was brought into lots of discussions. We were working simultaneously, but I was often brought into discussions with them, and we‟d give them emerging findings as we were going along. I think we were more convergent than divergent—their framework and our evaluation. I think the bigger problem is disinterest and disincentives. Many in the Bank conveniently say, "Well, that's WBI's [World Bank Institute] problem.� Well, it isn't WBI's problem. WBI is not the answer to how the Bank does capacity building. So. ZIEGLER: So there really hasn't, in your view, from your . . . GWIN: I don't know. I really don't want to say because I moved on. The ink wasn't dry on my final report when I was asked to start another evaluation, and got a team started before I could even spend time on it. So I had no time between the one and the next one I did, and I never really had a chance to look back. I know that IEG did a training study which built on the capacity building study. I don't know that anything‟s happened with that, either. It's not—what can I say?—it's not a good—I don't think it's Bank's best area of work. Catherine Gwin January 20, 2010 – Final Edited 24 ZIEGLER: In 2006, you led a complex—another very complex evaluation that produced a report entitled The Development Potential of Regional Programs: an Evaluation of World Bank’s Support of Multi-country Operations [The Development Potential of Regional Programs: an Evaluation of World Bank’s Support of Multi-country Operations. Washington, D.C: The World Bank, 2007]. I understand that this evaluation was essentially your idea. What are the reasons behind your, I believe, ultimately successful advocacy of this evaluation? GWIN: It was my idea. I pushed for it hard because it was obvious that cross-border development challenges were on the rise, and yet the Bank had implemented few operations of regional scope. Cross-country programs accounted for a very small share of overall ODA [Official Development Assistance] as well as World Bank assistance. There was a notion that regional programs were a comparative advantage, or were a distinct role, of the regional development banks—Asian Development Bank, African Development Bank, Inter-American Development Bank—but their amount of assistance wasn't a lot bigger than the World Bank's. Certainly the whole didn't look anything like what seemed to be necessary, given the rise in cross-border development challenges. What do I mean? Notably water management, control of communicable diseases, environment and other kinds of natural resource management, trans- border infrastructure development, and more; issues which a single country couldn‟t deal with on its own. So I think on those grounds it was not too hard to make the case. I argued that it would be good to know, even though the experience was limited, what the experience had been, what had worked well, when and why, with the aim of learning what the record might suggest about the potential development gains of scaling up World Bank assistance for multi-country regional projects and programs. I did this report at a time when we‟d already moved into global programs. These programs had been increasing in number for over a decade. Most provided support for analytical work and advocacy on global issues and grant-based partnership programs. ZIEGLER: And an example would be? GWIN: Well, a large and early example would be the CGIAR [Consultative Group on International Agricultural Research]. ZIEGLER: Yeah. GWIN: Other more recent examples are the Water and Sanitation Program, and PEFA, which is Public Expenditure Financial Assessment. Essentially, these are programs that provide technical assistance. ZIEGLER: Yeah. GWIN: But the difference in the regional area is that the Bank engages in both partnership programs and lending operations. And regional investment lending is complicated because Catherine Gwin January 20, 2010 – Final Edited 25 you've got to have agreement with each of the countries that are in this regional program. For example, we reviewed a hydro development project aimed at increasing power availability in Senegal and two other countries; a trade and transport facilitation project in Southeast Europe involving eight countries in standardizing and improving custom procedures at the borders to speed trade and reduce corruption; and projects to reverse deteriorating conditions in Lake Victoria and the Aral Sea, in both cases involving all of the countries that surrounded those bodies of water. ZIEGLER: Yes. GWIN: These programs involved investment lending by the Bank. In addition, the Bank is engaged in regional partnership programs supported by multiple donors, and in which the Bank is one of a number of partners. It made the regional projects evaluation very interesting. ZIEGLER: How did you proceed to implement this very complex evaluation? GWIN: This wasn't complex in the way the other ones were, which were just unwieldy and hard to define and so forth. This was pretty straightforward. It was a matter of selecting. We drew on an existing OED large global partnership evaluation to give us something of the evaluation framework because there were a fair amount of overlapping issues in evaluating global programs and regional programs, such as how to handle governance arrangements and accommodate the multiple countries‟ different interests. For example, an upstream country and a downstream country will have different interests when it‟s a water project, and so forth and so on. Then we selected nineteen regional programs, twelve projects and seven partnerships that reflected the overall active regional programs‟ portfolio in terms of geographical location and substantive focus—be it environment, or natural resources management, or something else. In other words, we drew a purposive sample of both projects and partnerships, and then we followed OED evaluation methods. We did evaluations of each of the 19 programs using a common framework—involving a template of evaluation questions—and developed means of rating the programs. We carried out field missions in the case of seven programs; the others were desk reviews, but those seven programs took us to twenty-four countries. In evaluating each program we assessed the standard OED criteria of relevance and effectiveness, but beyond that we tried to focus on what were the factors of success in this multi-country type of program. ZIEGLER: And your major conclusions in this evaluation? GWIN: Well, there‟re too many for me to list for you. However, we presented our findings at a major international conference organized by IEG at the time of the 2007 Bank-Fund Annual Meetings, so I can give you the presentation, and speak to a few key points. ZIEGLER: Okay. Yes, just for the tape. GWIN: Okay. Here are the main points. One, regional programs can deliver strong development results. Two, regional programs have worked well in building knowledge and infrastructure and less well in reforming policies. But, three, the development potential of regional programs is Catherine Gwin January 20, 2010 – Final Edited 26 underexploited. Opportunities are overlooked in a country-focused development strategy. I would say opportunities are overlooked in the World Bank, which is a country-focused institution. Four, demand and support is limited because of the complexities in arranging programs in ways that share benefit-to-cost equitably among countries. That's a complicated point. One of the toughest things about regional programs is how to design them so both the benefits and the costs are shared reasonably among the participating countries. Five, coordination capacities for achieving among the countries, as well as among the multiple donors, tend to be under-developed. So regional programs have development potential, but they're under-exploited. Then we drew on the evaluations of the 19 programs to identify what makes a regional program successful. Some of what we found was obvious, like strong ownership by all participating countries. We always say ownership of countries is essential for program success, but here you have, say, three or four countries with different interests in the problem being addressed, and all of them have to see something to gain from cooperating. In some of the programs we found strong ownership by some and not others, so the program never did work. [End Tape 1, Side B] [Begin Tape 2, Side A] GWIN: We were talking about what makes a regional program successful, and there were a number of different points. A second one I‟ll highlight is the importance of the governance arrangements. This requires clarity at two levels. There has to be clarity about the responsibility and accountability at the national level, and the responsibility, accountability of the regional institutions, because in most cases some form of regional institution is put into place, or used in the implementation of a regional program. More often than not we found that weak or poorly- defined governance arrangements undermined programs. A third factor of success is the financing of regional programs. It‟s tricky. The World Bank and other development partners really need to help countries figure out how to put together the kind of special financing requirements, the special financing packages required by regional programs—upfront in the design stage and for program implementation. Regional programs take a lot of time and effort up front, and need grant money at that stage. They require analysis and dialogue to get the countries to understand how they will benefit from a multi-country arrangement, how to define their interests clearly, and coordinate with the interests of others. So the financing package is more complicated than in the case of most Bank lending operations. ZIEGLER: I understand that the Nile Basin Initiative, which is intended to provide a framework through which its ten member states can cooperatively develop the resources in the Nile Basin to fight poverty and promote socioeconomic development in the region, was to be included in this evaluation. However, IEG management came under tremendous pressure from Bank management not to include it because of alleged sensitivities, and so the Nile Basin Initiative was not included in the evaluation. What conclusions did you draw from this experience? GWIN: Yes, the IEG management was asked not to include it, and the reason given was that the initiative—that the timing wasn't good; the initiative was at a critical negotiation point. IEG has always said when it's doing a country evaluation, or when it‟s thinking of doing a country Catherine Gwin January 20, 2010 – Final Edited 27 evaluation or something else, that if there are real world, real reasons for postponing, it's not averse to doing that. It's not trying to cause trouble by doing evaluations. It's trying to be informative and constructive. So the decision by the head of IEG was to say, "Okay. We take your point on timing. We won't include this one." I will only say that IEG has still not evaluated the Nile Basin, and I think it's long overdue, and as far as I know there has been no overall evaluation of the Nile Basin Initiative inside or outside the Bank, though that may have occurred by now. ZIEGLER: Would you care to speculate on the reasons, or . . . GWIN: Well, I was enormously annoyed at the time, and I thought we were being pushed around, but we were presented with the point of view that they were at an important negotiating point. ZIEGLER: That was effective? GWIN: Our evaluation would have been completed quite a bit later. Objectively, what they were saying to us was true in terms of ongoing activity. Would we have had any impact on that by the time our evaluation was finished is another matter. The Nile Basin Initiative had been ongoing for many, many years at that point, and it was always saying it was at a very critical point in getting launched. I think it now is launched, and I think it‟s a fascinating initiative, something the Bank will ultimately be quite proud to have assisted. I don‟t know that for a fact, but just from our initial examination we could see its importance and complexity. I‟d love to evaluate it at some point. ZIEGLER: You may get the chance. GWIN: Yes, who knows? ZIEGLER: You spent most of your career outside the World Bank, but as this interview has indicated, you were often closely involved with it and—as well as with other outside aid agencies. How did the World Bank look from the outside, compared with how it looked once you had been on the inside for a few years? GWIN: From the outside it looked monolithic, formulaic, and imperious. And the best in the business in many regards. From the inside, it looked highly talented. I would say that‟s one of its most impressive aspects. I was highly impressed with staff at all levels that I met at the Bank. At the same time, it looked sprawling and absent of clear priorities, driven by incentives to lend, and procedurally encumbered. It also looked unable or unwilling to withstand political pressure from major donors that affect its agenda and its ways of doing business. Very dedicated, very imaginative, often. I mean, it's a tremendous place to work. That's for sure because of the skills, the talents, and the dedication of the people here. That you don't see from the outside. Catherine Gwin January 20, 2010 – Final Edited 28 How does it compare to others? I mean, from the outside I said it looked like the best in the business in many regards. From the inside it looks like the best in the business in many regards. But as I‟ve also heard other people say it's very good, and often one of the best, in taking up new ideas and putting them to work, but not often the best in coming up with new ideas in the first place. ZIEGLER: In other words, the ideas are generated outside the institution. GWIN: Yes, often. And I'm not talking about little technical approaches. I'm talking about— and that's . . ZIEGLER: About larger issues. GWIN: . . why should it be? It's not—that's not exactly surprising, but I don't think the Bank often acknowledges that. Compared to others, it obviously has strong convening power due to a combination of resources, analytical strength, and global experience. It's also a standard setter in many regards on operational policies and processes, certainly for the other multi-lateral development banks. ZIEGLER: Given the—how should I put it?—the nice things you say about the Bank, there's often a great deal of criticism of the Bank. I mean, some of it's justified and some of it is not. GWIN: I'm not saying all nice things. The things I said before about it from the insider point of view, especially the incentives to move money, are real issues. ZIEGLER: Yeah. GWIN: I think that's where a lot of the criticisms—if you traced it back to the source, the incentive is to move money at the expense of insuring successful implementation and capacity built on the ground. You name the criticism that‟s made of the Bank, and you can trace it back, I think, to the incentive to move the money. Not every one of the criticisms—a second one is, it is either unwilling or unable to withstand political pressure from donors that affect its agenda, and therefore does too much. Then there are the issues at an operational level, the criticisms of the operational level, it ties itself up in knots about procurement matters—not that you shouldn't pay attention to procurement and lack of corruption and so forth and so on, but all the processes and procedures. You hear this all the time from everybody. Other donors that try to coordinate with the Bank certainly find the Bank is cumbersome to work with. Now, you know, I'm of two minds on that because you say, "Yeah, we're cumbersome because we're rigorous," but we're cumbersome because we're cumbersome, too, and we're constantly, I think, telling ourselves we need to be more agile, less cumbersome, more responsive, less arrogant. I think that thing about arrogance is so—it's said all the time, but it's really a problem. Some people are very arrogant, and some people in this Bank are very collaborative, and not the least bit arrogant. Catherine Gwin January 20, 2010 – Final Edited 29 ZIEGLER: I've met both. In the context of the IEG work, how did you find that management and the Board interacted with IEG, and did they really make the most use of such potentially . . GWIN: No. ZIEGLER: . . valuable resource? GWIN: Yes and no. The use of the evaluations has varied quite considerably. I think it sometimes depends on the quality of the evaluation. No management likes to be evaluated. Do you know the history of how this got set up? Very interesting. ZIEGLER: OED, originally? GWIN: Yes. ZIEGLER: All I know is that it was an initiative of Mr. [Robert S.] McNamara around 1969 or 1970. GWIN: Yes, but Mr. McNamara wanted an evaluation department that reported to him. The U.S. at that time was pushing for an independent, outside evaluation department, if I remember correctly, but it was certainly pushing for an evaluation department, and in effect, the compromise was an OED that reported to the Board, not to Mr. McNamara, and was very weak for many, many years. I mean, limited largely to project-level assessment. ZIEGLER: Un-huh. GWIN: What I have heard is that McNamara paid little attention to OED because it wasn‟t what he wanted. He wanted something that would answer questions for him. ZIEGLER: Right. GWIN: We report to the Board, so management is under our microscope. So, of course, they‟re going to be defensive. I think evaluations have the greatest effect when they are focused on an issue that is unsettled. That is when there is a re-thinking and an open-mindedness about an area of work because the Bank has had mixed or poor experience with it. An evaluation can be particularly useful at that point. On the other hand, an evaluation of country programs or sectoral areas thought to have been successful may draw out lessons of what has worked well, what hasn‟t, and why. That isn‟t always well received. And management‟s response is to take issue with the evaluation‟s Catherine Gwin January 20, 2010 – Final Edited 30 methodology. For example, to assert that the sample of projects or cases reviewed was inadequate. But acceptance of the evaluation findings and recommendations tend to grow over time because the quality of the work is generally high. I told you before about the experience with the IDA review. Management wants assessments to come out as positive as possible, and IEG aims to produce objective, hard hitting, useful reports, so the tension is always there, and I don't think it's unhealthy. The level of analysis is a problem that hinders the usefulness of the evaluations. IEG often hears from operational people that its evaluations aren't as operationally useful as needed. But IEG‟s primary audience is the Board, whose focus is on a higher policy level. It's very hard to do an evaluation that fully works on both levels, though IEG aims to be relevant to both concerns. ZIEGLER: Yes. GWIN: So, fair enough, we get criticism sometimes from operational people. They wish the recommendations had been more specific. But there's a difference between the Bank‟s operational reviews and IEG evaluations. While IEG makes use of the Bank‟s project reports and CAS reviews, those are discrete self-evaluations, and limited in the lessons that can be learned from them. IEG evaluations focus on a whole theme or a whole country and derive findings on operational effectiveness at a higher level of analysis. Sometimes this creates frustrations for operational staff. ZIEGLER: I understand that one of your biggest contributions to the work of IEG was to significantly increase the role of outside scholars and experts in the design of their evaluations. Are there any other methodological innovations that you introduced during your time at the IEG that you would like to relate? GWIN: No. I really wasn‟t an innovator of methodology. One of the key innovations is the use of impact evaluation, and I have never done that type of evaluation. So I think my contributions to IEG was as someone had spent time engaged in bringing together external views of researchers, NGOs, and governments with views within institutional organizations. ZIEGLER: Did you—the evaluation methods that you employed, did they evolve in any . . . GWIN: No, I got better. I think I got better at using them. As I said, I came in and I learned on the job. I think I really—you know, each evaluation I understood better what the methods were. ZIEGLER: In your view, what is the role of evaluation in maintaining World Bank legitimacy? GWIN: Fundamental. It's a great strength of the Bank to have an evaluation department. It adds important knowledge for the Bank to draw on, and it gives the Bank credibility in the outside world. I know the Bank itself gets unhappy when IEG writes a critical review and it gets into a newspaper, but I actually think, on the whole, it does the Bank a great deal of benefit. Catherine Gwin January 20, 2010 – Final Edited 31 Over the last ten years there's been more and more focus on the part of donors particularly, but also developing countries, on getting to results, and knowing that things would work. In fact, some have said that IEG is not independent enough and that there should be instead an entirely outside evaluation mechanism. I don't think that would work. I mean, you have a lot of independent think tanks around, and they do a lot of good work and they can do whatever they want, but I think IEG is independent of management, and yet can know how this institution works, and how it should work. So I think an entirely outside group would be just like a great big international think tank, and not a substitute for IEG or its equivalent in other international organizations. I don't object to it, but I don't think we really need it. There are plenty of places able to do outside, independent evaluations. The bigger problem is ensuring that programs include results frameworks, and monitoring and evaluation procedures so there is adequate data on which to base good, independent evaluation. ZIEGLER: Are the lessons and results of the evaluation sufficiently integrated into ongoing World Bank operations? I know you've touched on this. GWIN: I touched on that. I don't think I have much more to say on that. Sometimes, yes. It's hard to generalize. As I said to you, it depends on a lot of things. It depends on the state of play of the issue itself; it depends on the quality of the evaluation. So, yes, sometimes. ZIEGLER: What were the most important lessons you learned, both from a personal and institutional point of view, during your career at the World Bank? GWIN: Mmm, that‟s a tough question. A lot of clichés could be thrown at that question, like “People matter.� ZIEGLER: It doesn't mean they're not true, though. Clichés can be true. GWIN: Ultimately it's people who matter. Teams work better than individuals, and knowing how to design and motivate is really important; you know, those kinds of things. I mean, as for me personally, things that have both personal and institutional bearing is tricky. There are always reasons for why things work out, and so you have to look for those reasons and try to understand the underlying circumstances. So it's not only the what, but the why. I think sometimes that's better handled in some evaluations than in others. A clear framework from the outset, and spending the time up front to get that clear. Managing expectations and risks. It's good for more than just PR, and again, it gets back to the up-front work. I think a lot of that, the up-front work is highly critical, and often under resourced. The difference between evaluation and operational learning, I don't know; that's a tough question. I'm not really sure I can come up with much on that. ZIEGLER: What would you consider to be your greatest success, and what would you have liked to have done better? Catherine Gwin January 20, 2010 – Final Edited 32 GWIN: Well, in a funny way, I think the first thing I did was one of my better, more successful efforts. A sound, objective IDA review in a fraught period of time. It wasn't newsworthy, but I think it was very valuable in the context of IDA 13, and we talked about why; the credibility that that had. More concrete success, and I think much more fun, was the regional program review. It was my own idea, and I really enjoyed doing it. Also, I think we managed to draw attention and contribute useful findings at a time when cross-border problems were beginning to expand. I've been told that that evaluation actually fed into a decision of IDA deputies to allow IDA grants to be used for more up-front regional work, so that there was a direct link between that report and an actual outcome. You don‟t always have the satisfaction of seeing a direct link between your evaluation work and a major decision. ZIEGLER: Yeah. GWIN: What I would have liked to have done better, no doubt about it, the weakest effort was the capacity-building evaluation. I don't really think there was any change in the discourse or the practice in any way that I'm aware of. I still see it as one of the weakest aspects of the World Bank's work. Maybe the whole subject needs fundamental re-thinking, and the evaluation didn‟t foster that. We didn't get that. It really didn't get to the fundamentals of why this is such a weak area of work, not just in the Bank. I'm least happy with that piece of work. ZIEGLER: But it was a difficult subject to tackle. GWIN: Maybe, but the point is I don‟t think we‟ve changed anything in the discourse or the practice, and I think if I were to do it again, I‟d have to think harder about what the fundamental questions are. Why? Why has this experience not been more successful? We‟d have to dig much deeper on that one. ZIEGLER: Well, is there anything else that we haven‟t covered that you would like to discuss? GWIN: No, I don‟t think so. I hope this was useful. ZIEGLER: Yeah, I believe so. You‟ve been very generous with your time, and I greatly appreciate it, and I really enjoyed it. I‟ve learned a lot from this interview. So thank you . . GWIN: Thank you. ZIEGLER: . . very much, Catherine. GWIN: You‟re very welcome. [End of Tape 2, Side A] [End of interview] Catherine Gwin January 20, 2010 – Final Edited