Problems and proposals summary
Summary of problems and proposals from the International Working Group report
Proposal 1: Assess offers of technical advice against standards for the provision of information and evidence justifying the approach and for the planned dissemination of data about impact
The report proposes the use of a standardized framework for data collection and sharing. Current strategies for learning about the effectiveness of development interventions are of limited utility in the face of provider agencies’ incentives to restrict information about their activities and to make excessive claims about their impact, and of client governments’ incentives not to challenge that behavior. Exposing more clearly the basis on which offers of technical advice are made and the claims that they entail could initiate a virtuous circle. Governments would have a basis for making more informed choices among providers of technical advice and potential approaches, and providers of technical advice would have greater incentive to share data and justify claims. Such a framework could include standardized data categories to allow for comparison among design options and among results achieved.
The PFM discipline could develop a tool to assess how well PFM support interventions conform to open data standards on program design and results. Such a tool would ask how clearly the proposals developed by technical advice providers are elaborated along two dimensions. First, an efficiency justification dimension would rate proposals according to the degree to which they show the potential costs involved, such as financial and transactional costs, and the likely returns. Second, an empirical justification dimension would rate proposals on the availability of evidence on which they draw to make their case. Development of this tool could draw from cross-country and multisector studies about how this type of reform proposal has worked out in general, and from impact evaluations or other research that can test for the salience of distinctive contextual conditions.
The data assessment tool could be used as part of “quality at entry” processes within international agencies. While the tool would assess the transparency with which project proposals are justified financially and empirically, it would not be used to collect actual project data. Use of the tool could be combined with some form of independent, non-binding review or validation—possibly by an arm’s-length agency or entity of some form—that the appropriate procedures for using the assessment tool had been followed. The “PEFA Check” provided by the Public Expenditure and Financial Accountability (PEFA) Secretariat is one such example.
Proposal 2: Establish practitioner groups to share sensitive performance information on PFM interventions on a voluntary and confidential basis to identify ways of improving performance
The report suggests opening up a smaller dataset for more in-depth review and lesson sharing – in addition to the use of a standardized framework for data collection and sharing. Robust and comparable external measurement of results can seem risky for providers of technical advice that need to make claims about the benefits of their interventions. Provider agencies face strong incentives to demonstrate to their shareholders or funders that they are highly effective, which means that rigorous, verifiable, and comparable measurement of results by third parties may not be a high priority. Political pressures on provider agencies instead emphasize the need to demonstrate visible, short-term results to shareholders, taxpayers, or other principals. Risk aversion is especially high among providers in fragile states. Open data, deep knowledge, and sharing of learning are not likely to be the primary concern.
In some other sectors in countries of the Organisation for Economic Co-operation and Development (OECD), these concerns have been addressed through the creation of “data clubs,” or private networks for sharing sensitive data, which operate along reciprocal lines. This approach could be considered for the PFM sector as well. Submission of a base set of data and the commitment by all participants to treating all data shared in strict confidentiality would be conditions of joining a PFM data club. Such an approach has been shown to overcome government concerns and to facilitate the sharing of sensitive safety data. Membership would provide access to an aggregate, anonymized summary of the data that have been shared by others, and would allow individual members to approach each other for more tailored sharing of information. Club membership could be offered to entities or individuals, whether from client governments, international organizations, or donors and other assistance providers. Opting into the club would entail careful specification of the parties or types of parties with whom the data may be shared as well as whether and when the data must be anonymized.
Members, serving as representatives of their respective agencies, would thus be able to obtain authoritative technical support and data and have the credibility to speak critically about their agency’s approach. They would decide which data mattered and how to obtain information for sharing, with an emphasis on practical and implementable actions and explicit protocols about how the data could be used in their respective agencies. Members would establish the governance arrangements for the club. Agencies would be able to opt out of the club at any time, with all references to their data removed.
Proposal 3: Experiment with “managed choice” in the provision of technical advice, with a coalition of funders and client governments and the creation of a dedicated funding facility
The report proposes a “managed choice” model of technical advice. Providers of PFM support would prepare more than one intervention design option as the basis for comparison, discussion, and ultimately choice by the client government. Each option would be subjected to a quality assurance process, for example through robust peer review, and would explicitly harness available research evidence. Offering multiple intervention options is more expensive than asserting that there is just one correct or superior way forward. However, given the lack of certainty about what works and the need for greater experimentation, it would be a more honest and open-minded approach. When offering options to a client government, any differences in the intervention logic or technical design would be emphasized and discussed openly. Accordingly, each option would need to make explicit the technical content and intervention logic that underpinned it and the logical and empirical justification for that approach.
The potential benefits of this approach stretch beyond creating more experiments to assist in learning about what works in PFM support interventions. Other benefits could include the incentives that such arrangements would provide for strengthening relationships among different parts of government as they explore the implications of options together, the increased motivation inherent in giving key actors the opportunity to express their views and influence reform design, and the possibility that the problems to be addressed would incorporate locally identified issues more centrally.
While this proposal is still a long way from creating a competitive market for PFM support, it would move the PFM discipline toward greater contestability of ideas. It may not be known whether, for example, the International Monetary Fund, the World Bank, the United Kingdom’s Department for International Development, or the European Commission has the best track record in supporting a particular dimension of PFM reform. Equally, it may not be known what degree of balance between standard technical design on the one hand and problem-driven and adaptive assistance on the other hand is likely to be most effective. The research evidence is simply not good enough to provide clear, definitive, or general answers. The benefit of introducing options and choices more strongly into PFM support is that it would provide more data to support choices among different types of interventions and more information on which approaches are most credible from the perspective of the client government. It should also strengthen the quality of the dialogue and relationships between advice providers and client governments.
Proposal 4: Review the current conventions embedded in standard PFM assessment frameworks to test evidence of results and the degree of applicability in advanced economies
The report suggests that the currently tacit criteria by which PFM conventions are established and become embedded in standardized PFM assessment frameworks be made more explicit. It proposes three criteria for judging whether it is appropriate to treat any particular PFM process and its operational performance as a “basic,” or foundational, convention that might then be relevant across countries. The first criterion is that they are rational and based on accepted theory or other logic. For example, budgets should be comprehensive so that all claims are considered together to avoid logrolling. The second criterion is that they are universal, in the sense that they are found in all advanced economies. Universality is a proxy for assuming that the particular PFM process specifications are achievable, practicable, and strongly associated with successful socioeconomic development—even if the exact causal link cannot be ascertained. If particular specifications of PFM processes pass both these tests of rationality and universality, then it is reasonable to treat them as PFM conventions without requiring further evidence. Meeting these two criteria would therefore be enough to justify inclusion in standard PFM assessment frameworks as a working convention.
Where a particular specification of a PFM process is not universally observed, then a third criterion becomes important for assessing suitability—namely, systematic evidence of results across a range of relevant settings. In other words, if proposed PFM process specifications pass the first test of rationality but not the second test of universality, then to achieve the status of recommended conventions they must be backed up by systematic evidence of results. Meeting those two tests would suggest that the PFM process specification can be treated as a recommended arrangement across country settings. The challenge here is that the external validity of empirical findings—that is, their generalizability to other settings—is always limited.
In cases where the specification of a PFM process is neither universally observed nor supported by systematic evidence on results, then much greater care would be needed with regard to its application in practice. Examples include medium-term expenditure frameworks and performance-based budgeting. Pending clearer empirical evidence, such PFM process specifications would not be promoted as conventions through standardized assessment frameworks.
Along with establishing which particular specifications of PFM processes may be suitable for inclusion in standard frameworks, a corollary is to consider which established PFM conventions might be “retired” from the assessment frameworks. That would apply if the evidence on their applicability turns out to have been overstated or if there is concern about excessive promotion in contexts where they are not well suited. The same three criteria would apply in both instances: considering potential new conventions and reviewing existing conventions.
Proposal 5: Use PFM assessment frameworks to collect data and evidence “at the frontier” to build insights about which aspects of PFM operational performance matter for service delivery
The report notes that potential future conventions could be found through data collection “at the frontier” by scoping out new ideas about how the performance of PFM processes affects service delivery, as a prominent example of government actions. Almost all country-level assessments of PFM performance are currently both formal and official. They are formal in the sense that they rely on the application of one or more assessment tools and accompanying guidelines about their methodology and use. They are official in the sense that international organizations, donors, or official secretariats are typically responsible for running, authorizing, validating, or assuring the assessments and their results. These standard assessments could be complemented by use of the same tools—or components of them—for informal or experimental inquiry with the aim of generating knowledge rather than measuring performance.
The dimensions of PFM operational performance that are most relevant to service delivery are likely to be found in the intermediate functions of well-performing PFM systems such as those posited by Andrews et al. (2014, p.6): “prudent fiscal decisions, credible budgets, reliable and efficient resource flows and transactions, and institutionalized accountability”. They break down these result areas into sixteen more specific “functional concerns” of PFM that seem to matter for service delivery. Those range from “cash is provided to spending agencies when agreed, in agreed amounts” to “concerns raised by independent assurance exercises are transparently discussed by citizens’ representatives, receive timely follow-up and redress by the executive.”
These intermediate result areas represent attempts to find further stepping stones in the results chain, in this case moving from PFM conventions toward service delivery and development results. Other candidates might include government staff perceptions of budget credibility, the reliability of resource flows and the accuracy of transactions, sequencing of reforms, and external assessments of accountability arrangements. Such metrics would focus on ideas that are of interest but have not achieved the status of conventions accepted by the PFM discipline.
Existing PFM assessment approaches can be accompanied by institutionalized collection of data that are more speculative and draw on a wider array of frameworks. This could involve adapting or honing assessment frameworks or other analytical tools such as public expenditure reviews, public expenditure tracking surveys, and service delivery surveys to learn more about the relationships between PFM processes and operational performance on the one hand and service delivery on the other. The mechanism for achieving this institutionalized data collection could be a virtual and open-access “beta” or “sandbox” space for experimentation that would be independent of the funding and affiliation pressure of major assistance providers.
Proposal 6: Investigate how PFM conventions and processes could better support non-incremental policy change through strategic reprioritization in budget policies
The report notes that better understanding of how PFM processes could be used to support changes in policy direction is a priority. Some investigation of examples and lessons regarding how PFM process and conventions have accommodated or supported instances of major reallocation could shed light on how innovations in PFM may be helpful. Given the limited reference points or case examples of successful strategic reprioritization, a broader review might need to include episodes of significant spending increases or cases of major subsidy reductions. The focus would be on PFM dimensions, although the dominant policy dimensions and political factors would also need to be recognized and understood.
The starting point would be to take stock of the lessons learned through many previous attempts to overcome incrementalism and increase policy responsiveness in budgeting. The review would identify the incentives toward incrementalism, the historical track record of attempts to move away from incrementalism in countries of the Organisation for Economic Co-operation and Development (OECD), and the lessons emanating from past and current tools and initiatives. Examples may include program budgeting, performance-based budgeting, pro-poor budgeting, spending reviews, medium-term expenditure frameworks, and country-specific approaches such as Public Service Agreements in the United Kingdom. Examining the success factors in cases involving the politically difficult task of explicitly transferring funds from a relatively low-priority policy area to fund increased spending in an area of higher priority will be critical.
Another way to approach this question is to consider the fit between particular policy directions and the PFM processes and conventions that underpin them. Judgment will be needed to determine which processes and conventions are versatile across multiple policy objectives and which are more bespoke elements that will support a particular policy orientation. For example, there may be concurrent but incompatible needs or pressures to constrain spending and reduce the deficit, to boost the volume and flow of funds to infrastructure development and service delivery, and to increase redistributive transfers to marginalized constituencies facing severe hardship and lack of opportunity. A strong foundational set of PFM processes can help with good information and analysis across any of these priorities. However, different fiscal policy orientations may indicate the need for different processes to try to make those policies effective.
It would be helpful if the work were not limited to one country income group but conducted instead as a collaborative initiative across the income spectrum. Many of the policy challenges and political pressures are common across countries—whether as a result of the links to global public goods or owing to the increasingly convergent reference points and aspirations among citizens. Many existing tools, such as spending reviews or their equivalents, are used across groups of countries. Some innovation in the types of partnerships that are formed to investigate this agenda would be valuable, reaching beyond the usual organizational suspects and bringing in former political leaders to help with insights about better connecting technocratic PFM processes with policy and political considerations.
Proposal 7: Expand the toolkit of PFM to analyze the trade-offs among policy objectives and to consider the socioeconomic impact of fiscal and budgetary policy choices
The dominance of aggregate fiscal concerns has arguably biased PFM toward giving insufficient weight to wider socioeconomic values and objectives. Yet aims such as reducing poverty, improving sector outcomes, narrowing income disparities, and strengthening citizen voice in governance are increasingly on the radar in discussions about PFM. On the other hand, there may be objections from within the PFM discipline, and especially from fiscal policy experts, about the risk of overreach into policy matters.
A useful first step would be to launch a reappraisal of the standard objectives in PFM to make the case for their expansion and to propose specific new formulations for debate. Expanding the three long-standing PFM objectives would make those objectives more relevant to contemporary priorities and policy concerns. It would also re-focus attention on the tensions and trade-offs that exist among some of these objectives, which had gradually become obscured over the past two decades.
There is a particular need to develop and test new PFM conventions and supporting processes that can be used to analyze the distributional implications of policy choices, as this is a significant omission in the current setup of conventions and frameworks. Several tools and approaches could be tested, including greater emphasis given to the revenue side of PFM. Publication of socioeconomic impact statements could be required or recommended when a government proposes a specific fiscal course—for example, requiring the assessment of impacts on public services, income distribution, and poverty levels. The government or an independent entity could be required to produce assessments of the distributional effects of fiscal and budget policy—for example, impacts on distinct income quintiles or on young or old populations in particular. Governments could be required to publish a pre-budget statement to encourage public debate and analysis of policy options, thereby helping to strengthen the role of the demand side in fiscal governance.
Links could be made to the separate proposal to introduce greater contestation of ideas as part of the external technical advice provided to client governments. The main intention would be to equip governments themselves with better analytical tools and operating conventions as part of their PFM system. These tools could also be used by external actors to inform dialogue with governments and technical advice. The core idea would be to make explicit the trade-offs among different objectives in managing public finance as the basis for informed debate and government decision making.