top stories

Share

Comments from the Association of Chartered Certified Accountants
October 2002

Executive Summary

The Association of Chartered Certified Accountants (ACCA) is pleased to have this opportunity to provide comments to the HM Treasury on the draft guidance on Appraisal and Evaluation in Central Government (The Green Book). These comments have been developed with our Public Sector Technical Issues Committee, a group of experienced accountants who work in the public sector.

ACCA welcomes the reduction of the discount rate which is to be used to calculate the net present value of capital projects from 6 per cent to 3.5 per cent. This should help to ensure that project appraisals take an effective long-term view of the relative costs and benefits of projects.

We believe, however, that the draft guidance is more complex and less transparent. In addition, we do not consider that it will improve the openness of the appraisal processes for major capital schemes.

The draft guidance should provide more detail on the different types of appraisal process which are undertaken and non-financial aspects, such as the environmental impact of a project, need to be given greater prominence in the appraisal process.

The draft guidance puts significant emphasis on the adoption of a cost benefit approach to project appraisal. A major criticism of this approach, recognised by the guidance, is that it ignores inequalities in income distribution and the declining marginal value of consumption. Greater thought is needed on how a practical distributional adjustment can be developed for use in all appraisals to ensure that these problems can be overcome.

The examples provided in the text on reports of appraisal exercises should be expanded to ensure that these reports contain details of all the major assumptions made and at least an outline of the logic of the appraisal process. This will allow an independent reader to undertake an intelligent critique of the appraisal process and to explore adequately the main options.

We believe that the guidance should be further developed to ensure that appraisals do not provide spurious impressions of accuracy. One way of achieving this would be to recommend the much wider and more general use of sensitivity and scenario analysis.

We have some concerns over that part of the guidance which was produced by external consultants. We believe that this guidance may have significantly over emphasised the adjustments which should be made to the public sector comparator when comparing this to the costs of the PFI option. We also believe that, while the guidance may include some important insights, it will need significant extra work if it is to be sufficiently clear and practical.

Detailed Comments

Objectives of the guidance

1. The draft guidance aims to provide advice on how to undertake appraisals to underpin three different types of decision. These are:
   • whether a project or programme is worth undertaking on the basis of an economic appraisal
   • whether the private finance option or the direct public procurement route appears to provide better value for money
     and
   • if the project is to be privately financed, which consortium appears to provide the most advantageous option.
     
     
2. These different types of appraisal require significantly different approaches and methodologies. ACCA believes that this should be made clear in the guidance and three sets of guidance should, therefore, be produced. We appreciate that, for example, the Office of Government Commerce (OGC) provides additional guidance on the third type of appraisal, but the draft guidance should still distinguish between the different approaches to be adopted for each of the main types of appraisal.
   
   Systematic and thorough appraisals
   
   
3. The draft guidance intends to encourage users to adopt a systematic and thorough approach to appraisal. The draft guidance, however, provides a series of pre-set values for, for example, the risks of optimism bias in the cost and time taken to complete major capital projects. This approach may lead to the routine use of these pre-set values rather than a systematic critical evaluation of the risks involved with each individual project and the identification of appropriate steps to manage these risks.
   
   Level of detail provided in the worked examples
   
   
4. The level of detail currently provided in the examples in the text does not allow a reader to follow the logic of the appraisal nor are all assumptions explicitly stated. This level of detail may reduce the openness and accountability of the appraisal process and would make the critical evaluation of such appraisals difficult to undertake. As a result, the output from the appraisal would dictate the decision which should be taken, rather than being an aid to the decision-making process. In addition, the examples encourage a spurious sense of accuracy as many of the figures are shown to at least five significant figures.
   
   
5. In some cases, the examples given in the text do not follow the guidance which is provided. For example, on page 19, the draft guidance recommends that the objectives of any proposal should be specific, measurable, achievable, realistic and timed. The example on the next page clearly does not follow this advice. In some cases, the text suggests that the examples just provide a summary of the appraisal (for example at the top of page 25), but this is not consistently applied.
   
   
6. The level of detail provided in an appraisal will depend on:
   • the size and complexity of the project
     and
   • whether, for example, a strategic, outline or full business case is being made.
     
     
7. In contrast, the draft guidance aims to cover "projects of all types and size" (page 4 of the main text) and does not differentiate between the level of detail which should be provided for appraisals undertaken at different stages of the project's life cycle.
   
   Depth of practical guidance provided in annexes
   
   
8. The draft guidance is supplemented by a series of annexes. We believe that these annexes are too theoretical, rather than providing guidance on the practical application of the suggested methodology. This is particularly true for the first two annexes.
   
   Justification for cost benefit analysis
   
   
9. The adoption of the cost-benefit approach to appraisal "is a central part of the new guidance" (page 7 of the invitation to respond). Despite recognising that "placing a monetary value on a non-market benefit can appear artificial" and that these adjustments "are largely of a subjective nature" (page 27 of the main document), HM Treasury has provided little justification for the adoption of this approach.
   
   
10. In addition, there are a number of criticisms of this approach, which are not adequately addressed in the draft guidance.
    • Cost benefit analysis does not adequately address the question of income distribution nor the declining marginal value of consumption. The draft guidance does have a section on the distributional effects of the recommended approach, but admits that the information needed to undertake such adjustments is "most unlikely to be available at acceptable cost for many applications" (page 46 of annexes).
    • The methodology does not adequately address the synergies and other interactions between projects and interventions, but provides a methodology which merely analyses individual projects in isolation, without indicating how joined-up government can be implemented in practice. A clear example of this problem is indicated on page 48 in annex 6. Here, particular adjustments are recommended "if the objective of a policy is aimed specifically at tackling inequality". If the Government has the objective of tackling inequality, this should be taken into account in all appraisals, not just those for particular projects or programmes.
    • Cost benefit analysis may provide a spurious sense of accuracy. This could be overcome if a range of possible values were provided; this approach - sensitivity analysis - is, however, not routinely recommended, being suggested only for situations where "uncertainty exists" (page 12 of the main document).
    • When undertaking a cost benefit analyses there is a tendency only to measure those costs or benefits which are relatively easy to measure.
      
      
    Financing of the public sector comparator
    
    
11. The draft guidance does not recommend any change to the current approach of assuming that, for the Public Sector Comparator, the capital costs of the scheme will be disbursed at the start of the project. In practice, the Government or the local public sector entity would borrow to finance a major capital scheme and then make interest and capital payments over the life of the project. The PFI option includes the financing costs of the project so, to ensure comparability, the Public Sector Comparator should also include the way in which the project would actually be financed.
    
    Distributional adjustment
    
    

    The draft guidance includes an annex on distributional impacts. This annex is particularly complex, will not be easily understood by most readers and contains a number of errors. For example:

    • the first table could be simplified as the figures for pensioners are the same as those in some other columns
    • the column headings for the gross income table are incorrect and this table refers to "Quintile of Equivalised net (sic) income"
      and
    • the example in the main text (page 50) refers to a single adult on 80 per cent of average earnings having a marginal utility of income relative to the mean of 0.63, while the table in the annex gives a figure of 0.6. This also suggests a significant flaw in the methodology as someone on only 80 per cent of average earnings is placed in the second highest quintile of gross earnings.

    As a result, we do not believe that the draft guidance provides an appropriate mechanism for making distributional adjustments to cost benefit analyses to overcome the inherent problems of this methodology.
    
    

    Economic or comprehensive appraisals
    
    
12. The introduction to the revised Green Book states that its "process of appraisal and evaluation assesses the economic, social, environmental and financial impacts" of a proposal. The draft guidance provides little guidance, however, on anything other than economic impacts. Thus, the diagrams on pages 15, 16, 21 and 56 of the main text make no mention of environmental appraisals and these are not included in the glossary provided with the guidance. ACCA believes that greater priority should be placed on undertaking comprehensive environmental assessments for all major capital projects.
    
    Risk and uncertainty
    
    
13. The draft guidance makes an unusual distinction between risk, where the "probabilities are known", and uncertainty, where "there is very little information at all even on the probabilities of a particular event occurring" (page 40 of the main text). In practice, probabilities are never "known"; they can only be estimated with lesser or greater degrees of accuracy. The approach recommended by the draft guidance will lead to spurious levels of precision rather than an admission that an appraisal process can only produce a probable range of values. The draft guidance cites the example of the risk of floods as one which can be "estimated objectively by statistical techniques" (page 40, annex 5). Given the increasing risk of floods in recent years resulting from global warming, this does not appear to be an appropriate example of a risk with a known probability.
    
    
14. Almost all appraisals are subject to greater or lesser degrees of uncertainty (as is recognised on page 46 of the main guidance). As a result, most appraisals, and certainly all those for major projects or programmes, should be subject to
    
    
    •  sensitivity analysis to assess the effects of minor changes to the main variables
      and
    • scenario analysis to assess the combined impact of these uncertainties.
      
      
15. This approach is recommended in the current edition of the Green Book which states that in "most central government appraisals the analysis of risk can best be handled by sensitivity analysis" (paragraph 4.48).
    
    Comprehensive risk analysis
    
    
16. Care should be taken to ensure that all aspects of risk are adequately covered by appraisals. This should include an analysis of risks to the success of both the public sector comparator and to the PFI option. The draft guidance considers the risks to the success of direct public sector procurement but the risks to the PFI option are not rigorously considered. Thus, for example, in the section of the guidance on risk and uncertainty, the risks of the private sector partner failing to provide quality services or financially are not considered.
    
    Irreversibility and variability
    
    
17. The section of the draft guidance on risk and uncertainty discusses four types of risk which "commonly arise in appraisal" (page 44 of the main guidance). These are listed as:
    • optimism bias
    • project specific risk
    • irreversibility
      and
    • the cost of variability in outcomes.
      
      
18. The draft guidance does not provide adequate guidance on either of the last two items in this list. Irreversibility may be considered to apply in PFI projects "where implementation of a proposal might rule out later investment opportunities or alternative uses of resources" (page 40, annex 5) because of the long-term nature of these contracts. The draft guidance does not, however, provide any practical guidance on how such risks should be assessed or how the value of PFI contracts should be adjusted to take account of the risks.
    
    
19. The draft guidance states that "adjusting for the impact of variability will be unnecessary for the majority of appraisals" (page 41, annex 5). No practical guidance is, however, provided on when such adjustments should be made or how they should be calculated.
    
    Present value and discounting
    
    
20. The section in the draft guidance on present value and discounting (paragraph 4.13.1 of the main text) includes some guidance which is not clear. For example, it is suggested that, for projects with "very long-term" impacts, a lower discount rate may be appropriate; there is no definition of what the "very long-term" is in this context. Indeed, on page 53, annex 7, the reference is merely to the "long term". The main text then continues with an inconclusive discussion on infrastructure assets (road and rail schemes) which gives no guidance on either the discount rate which should be used or the length of the appraisal period. This section finishes with a brief indication of other cases where alternative discount rates may be more appropriate, without suggesting what alternative rates should be used. The text at the end of annex 7 provides very little additional guidance on this problem.
    
    Internet links to other guidance
    
    
21. The references, throughout the documents, to other sources of guidance, including references to relevant documents on the Internet, is most useful. The problem is that these references change quickly. So, for example, the Internet addresses for documents produced by the OGC and the Cabinet Office on page 57 of the draft guidance are already out of date.
    
    Post-project evaluations
    
    
22. The draft guidance does not clearly differentiate between the approach to be taken for project appraisals, before a project is begun, and the approach to be taken for evaluations undertaken after the project has been completed.
    
    
23. The importance of post-project evaluations has been under-played in the past and there has been widespread recognition that such evaluations are not routinely undertaken. This situation is likely to be perpetuated as a result of the relative emphasis placed on the two processes in the draft guidance. This includes 48 pages of guidance on undertaking pre-project appraisals and only two and a half pages on post-project evaluations. In addition, no clear mechanism is proposed to ensure that these post-project evaluations are rigorously undertaken for all programmes and projects. The table comparing appraisal and evaluation quotes the OGC's Gateway 5 and individual "institutional structures within departments" as the mechanisms for ensuring that appropriate post-project evaluations are undertaken. The OGC's Gateway 5 provides the methodology for under-taking post-project evaluations, but the OGC does not have responsibility for ensuring that these are actually undertaken. Thus, the draft guidance will maintain the current situation where post-project evaluations are not routinely completed.

    Concerns over the Out-Sourced Guidance on Optimism Bias and Taxation Effects

    Introduction
    
    
24. We note that HM Treasury has out-sourced guidance on optimism bias and taxation effects to the consultants Mott MacDonald and KPMG. Both of these organisations are major players in the PFI market and have a commercial interest in ensuring that this approach is used more extensively.
    
    
25. The adjustments which are being suggested by the consultants for optimism bias and the scale of the adjustments for corporation tax may mean that the whole appraisal process will lose credibility. This could reinforce the scepticism which is now widely held about PFI and the value for money test. In a recent survey of ACCA members working in the public sector, for example, fewer than 1 in 7 respondents felt that PFI schemes are currently being objectively tested for value for money.
    
    Review of large public procurement in the UK
    
    
26. The Mott MacDonald study does, however, usefully emphasise the problem of optimism bias and, if implemented, the recommendations would help to ensure that:
    • budgets for large procurement projects are developed to take account of this bias and so would be more realistic
      and
    • risks involved in large scale public procurement projects are more rigorously identified and managed.
      
      
27. The Mott MacDonald study recommends that, for standard build contracts at the first stage of appraisal, an uplift of 28 per cent should be added to budget estimates for all major capital schemes. The Mott MacDonald study recognises that the adjustments "tend to be based prudently on the higher levels of optimism bias" found by their study (page 69 of their study) compared to recent studies undertaken by government departments.
    
    
28. The study does not directly recommend adjustments to be made when comparing the costs of potential PFI projects with those for traditional procurement. The uplift of 28% for standard build contracts is, however, the estimate which the study made for traditional procurement while the estimate for cost and time over-runs for PFI projects is only 5%.
    
    
29. There are a number of aspects of the Mott MacDonald study which, at least implicitly, it recognises would understate the relative cost and time over-runs for PFI projects. These include that:
    • its sample is not a random sample of large public procurement projects
    • the sample of projects is skewed towards larger projects
    • the sample of projects is chosen from the last 20 years while recognising that there have been significant improvements in recent years
      and
    • the optimism bias for PFI projects is understated as this was measured at a later stage in the project life cycle.
      
      
30. The sample of 50 projects was initially chosen from a list provided by HM Treasury "evenly spread across departments" (page 7), but only a maximum of 10 were selected from each department. This would result in the sample being biased against those experienced departments with large programmes of capital projects. As a result, the cost and time over-runs for the projects chosen would be higher than for an average project.
    
    
31. The study looked only at capital projects with "costs exceeding £40m in 2001 prices" (page 4) and then the larger projects were chosen (see footnote on page 7). This again could be expected to result in higher estimates of cost and time over-runs as these would be expected to be proportionally higher for larger and more complex projects.
    
    
32. The sample of projects covers the last 20 years without making adjustments for improvements in project management which have taken place in recent years. For example, in appendix G to the study, details of a study by HM Treasury Central Unit of Procurement showed a reduction in cost and time over-runs of nearly 60 per cent over the four years from 1990-91 to 1994-95. In addition, the comparison with the cost and time over-runs with PFI projects is skewed as most of these have only been completed in the last five years and the study states that the PFI projects assessed "are all early PFI projects, so many of the issues identified are no longer significant sources of optimism bias".
    
    
33. The study recognises that it is not surprising that the average optimism bias identified for PFI projects is much lower than for traditionally procured projects. This is because more project risks are identified and mitigated at the full business case stage, when the optimism bias is measured for PFI projects, than at the strategic outline and the outline business case stages, when the optimism bias is measured for traditionally procured projects (pages 14 and 15 of the study).
    
    
34. Finally the study is long and complex, running to 84 pages compared with the main document of the revised Green Book which is only 75 pages long. Additional work will be needed to produce guidance which clearly demonstrates the logic and justification for the figures which the Mott McDonald study recommends should be used for optimism bias.
    
    Adjustments for differential tax receipts
    
    
35. The KPMG report recommends that at least 2% and up to 22% should be added to the net present value of the public sector comparator to account for "the differential UK tax receipts from PFI schemes for the purpose of evaluation against a PSC" (page 3). This contrasts with the last edition of the Green Book which estimated an appropriate adjustment for tax for private sector firms "is typically of the order of 1 percentage point or less" which is equivalent to a tax adjustment of approximately 10%. In addition, the adjustment recommended by KPMG for the tax effect would suggest a profit rate for the PFI contractor in excess of 60%.
    
    
36. HM Treasury's summary of the KPMG guidance is inconsistent with that provided by the consultants. For example, its description of the primary and secondary tax effects differs from that of KPMG. This suggests that KMPG's guidance has not yet been adequately or critically assessed by HM Treasury.
    
    
37. The KPMG report states that the choice of whether there is a tax effect is "essentially an economic judgement based on the additionality of the tax receipts" (page 4). It then goes on to say that the "choice of which [effect] to use falls outside the scope of this study" (page 4). Thus it does not accept responsibility for the choice of whether or not its model is actually adopted.
    
    
38. The profit rate for the PFI scheme, indicated above, which would be necessary to justify KPMG's recommended adjustments, appears to be based on the assumption that the Public Sector Comparator does not include any out-sourced work, so that all the building work is undertaken by direct labour with no contractors. This is not the approach which is adopted for publicly financed capital schemes.
    
    
39. The KPMG report does not state what its input assumptions were for developing its model. As a result, it is difficult critically to assess this model.
    
    
40. KPMG also states that: "VAT, employee taxes, and business rates should not normally lead to differences between publicly and privately financed options" (page 9). It does, however, accept that, "if hard evidence of radically different levels of employment taxes between the two" options is available, these differences should be taken into account.
    
    
41. In the IPPR study of prisons, however, it was found that "staff costs accounted for all the differences in costs" between the PFI and conventional prisons (Public Finance 6-12 September 2002). Thus, any increase in the Corporation Tax paid by PFI contractors will be matched by a reduction in the value of the income tax paid by their employees. This explains why the previous edition of the Green Book accepted that any taxation effect would be accounted for in the discount rate of 6 per cent which was used and that there may be cases when a greater tax effect would occur although "such cases are rare and complicated".

Back to top