How GDP Data Quality Ratings Are Produced
8 July 2020
MEASURING GDP QUALITY
World Economics has developed the Global GDP Data Quality Ratings to review the utility of official GDP data of individual countries. The Ratings currently cover five factors to determine data quality. Each factor is evaluated to provide country scores which are then normalised using the standard deviation of the data for each factor and combined into the DQR score using a weighted aggregate to reflect the importance of each of the individual factors.
These five factors used to judge data quality are:
- Base Year used to calculate the GDP data (years out of date)
- Standard of National Accounts (SNA) applied
- Estimated Size of the Informal Economy
- A Proxy for Resources Devoted to Measuring Economic Activity
- A Proxy measure for likely Government Interference in Economic Data production
It should be noted that there is not infrequent variation between what the World Bank and IMF list as the most recent Base Year and/or most recent SNA in use, and what countries themselves claim to be using. This is sometimes caused by often unavoidable time lags in the International organisations being informed of changes that have taken place locally and sometimes simple error is involved. Whatever the reasons, World Economics takes some trouble to find out what is the on-the-ground reality behind the figures. If we also fail to reflect the latest changes occasionally, we apologise in advance, but hope the data in this report is as correct and timely as is possible to achieve.
KEY VARIABLES AND METHODOLOGY
Base Year (Range from 1984 to Chained)
Constant price estimates of GDP use the inflation adjusted price of goods and services relative to a particular year, known as a base year, to weight the volume components of output. But since the structure of production and relative prices over time are dynamic, the structure of the prices of products and the industries surveyed in the base year become less relevant over time. For some rapidly changing products (such as the smartphone in recent years) rapid technological change and relative price falls make any kind of comparison fraught with difficulty.
What is clear is that using data from 10 or 20 years ago (as many countries do) as a basis for calculations of the size and shape of economic activity, is unlikely to produce reliable estimates of GDP. In countries that revise base dates, very significant increases are usually recorded, highlighting just how inaccurate data is that has been produced using out of date base years.
When GDP is revised and the base year is updated, it allows the statistician to reweight the relative importance of the different sectors of economic activity, and further change or reconsider the methods and data sources.
The United Nations recommends updating base years every five years, although, most developed countries now adopt the practice of chaining, where relative prices are updated every year. The more out of date a country’s base year, the more inaccurate are estimates of GDP and the lower a country’s score in the World Economics Data Quality Ratings (DQR).
The base year score for each country in the DQR is a number between 0 and 100, with 100 indicating that a country is using a chaining system, where base years (or relative prices) are updated every year). Information on individual country’s base year is taken from the World Bank’s World Development Indicators (WDI), IMF’s World Economic Outlook Report, United Nations and National Statistics Offices. Base year points for use in the data ratings are then calculated by taking the range of base years used and applying a sliding scale based on the number of years out of data. 100 indicates Chained or the latest possible base year where the oldest base year used (Madagascar, 34 years) is assigned the lowest score of 0. All variations between these years are deducted multiples of 2.9 points for each year out of date.
System of National Accounts: (SNAs used range from 1968 to 2008)
National income measurement is governed by a global standard: the United Nations System of National Accounts (SNA) - an internationally agreed standard set of recommendations on how to compile and measure economic activity and facilitate international comparability of economic statistics. The first SNA was published in 1953 and there have been three revisions SNA 1968, SNA 1993, and SNA 2008.
The longer it takes a country to update its SNA the less reliable the data becomes, particularly when used for economic comparisons to a country with a more recent SNA version. In the World Economics Data Quality Ratings, the newer the SNA version, the higher a country’s score.
The score for the SNA component is based on a scale of 0-100, with 100 points given to countries using the latest SNA version; 50 points going to countries using the 1993 version; and no points for use of earlier versions. Information on individual country’s SNA is taken from the World Bank’s World Development Indicators (WDI), IMF World Economic Outlook Report, United Nations and National Statistics Offices.
Informal Economy: (ranges from less than 7% to over 65% of GDP)
In many poorer countries, a very large swathe of activity can remain uncounted and even in wealthy countries, some informal activities remain outside the national accounts. But due to the nature of much informal work, ranging from housework, farming through to gambling, prostitution, drug dealing, and smuggling, calculations of the value of such activities are extremely difficult. The existence of such large amount of informal activity is so economically important that to leave it unrecorded in the official national accounts is unsatisfactory.
There have been many attempts to estimate the size of parts of the informal economy. The World Economics Data Quality Ratings employs estimates for 2015 provided by the IMF Working Paper: WP/18/172.
In constructing the data, the higher the size of the informal economy, the lower a country’s factor score. A DQR score of 100 means that a country has the lowest rate of informal sector activity as percentage of GDP.
Resources Available for Producing National Accounts Data
The quality of national income estimates depends to some extent on the statistical capacity and the resources available to national statistics offices. The United Nations System of National Accounts has put a global standard in place but the challenge for a local national statistics office is to produce a measure of the economy, usually with limited resources. Statistical capacity, or the ability to adhere to the global standard, depends critically on the resources and information available at any given time and place.
All other things being equal, there are a priori grounds to believe that poorer economies will have lower-quality statistics. The statistical capacity and economic resources in national statistics offices therefore matter a great deal in terms of data availability and quality of economic statistics. Data availability is subject to the number of trained staff and the level of resources available for collecting, processing and analysing the data.
As an illustration of the importance of resources in the collection of data, few have shown with greater clarity the nature of the problem than Morten Jerven in his book Poor Numbers, 2013, based on actual visits to statistics offices in Africa. To quote Jerven: “This book has shown that the most basic metric of development , GDP, should not be treated as an objective number but rather as a number that is the product of a process in which a range of arbitrary and controversial assumptions are made. As a result the metric should be used with the utmost care. The quality of this number depends on the state of the system that produces the statistics and this system is deficient in many poor countries.” This problem is not confined to Africa but is evident in countries on all continents.
The score for this component of the DQR is derived from the World Bank Statistical Capacity Index. We use this index as a proxy for assessing the availability of economic resources in national statistics offices. In theory, the larger the resources devoted to statistics offices, the better the quality of statistics. That is, the higher the index, the higher the country’s score. This is a proxy measure.
Governments interfere with the production and dissemination of basic economic data in many ways. Attempts in Greece to prosecute and potentially jail the man hired by the IMF to sort out the corrupt mess of Greek economic data is perhaps the most egregious recent example. See: A Statistician’s Ordeal: The Case of Andreas Georgiou.
The Greek instance might appear to be an extreme special case. But unfortunately there are also many occurrences of serious Government interference in the production of economic data in the Americas.
For example, a recent paper (On Measuring Hyperinflation; Venezuela's Episode, by Hanke), records the following:
The Banco Central de Venezuela (BCV), like many central banks, has followed a pattern that Oskar Morgenstern elegantly documents in his classic work On the Accuracy of Economic Observations. Indeed, the BCV has failed to report data that would reflect poorly on the government, and when it has reported inflation statistics, it has lied and doctored the data. Instead of reporting Venezuela’s ‘real’ open rate of inflation, the BCV has attempted to measure suppressed inflation.
Venezuela imposes a thick blanket of price controls and a maze of subsidies over the economy. List prices are artificially held down. Yet these suppressed prices are the ones that, in principle, the BCV attempts to measure and use to construct a price index for calculating the inflation rate. But this metric misses the mark. Arbitrage opportunities prevail under the Venezuelan regime of price controls and subsidies, because there is a gap between the items under price controls and the prices of those goods and services that are actually exchanged on the black market. And it is in the black market and underground economy that most of Venezuela’s economic activity occurs. In consequence, there is a huge gap between the official inflation rate, which is based on artificially suppressed prices, and the ‘real’ open inflation rate.
And similarly, a paper by Ariel Coremberg "Measuring Argentina's GDP Growth; Myths and Facts" makes the following points:
- Since 2007, official economic statistics in Argentina, particularly on consumer inflation and GDP, have been subject to political manipulation.
- This paper reproduces Argentine national income from 2007 using standard methods and original sector data and finds that declared GDP is 12.2% higher in 1993 prices due to political intervention.
- The paper finds that the distortion is mainly due to changes in accounting methodology across industries and not to changes in inflation estimates.
- The reproduced GDP data dispels the myth that Argentina has been the fastest growing South American economy in recent years.
Governments and Government agencies manipulate GDP data directly in many ways, for example through the calculation of price indexes such as the GDP deflator which impact on GDP per capita data. They can and do stop publishing important data prior to elections. They try to abolish independent statistics bureaus. They try to add questions that will bias responses to Census data. They leave in place price indexes known to be unreliable and impacting heavily and negatively on crucial pensions systems.
This is not only a problem evident in poor countries, although countries with autocratic systems probably suffer to a greater extent. Sometimes the transgressions are deliberate, and sometimes due to incompetence or lack of resource.
Government corruption also infect all parts of an economy and its accurate measurement in systematic ways. Often a direct result of the government’s concentration of economic or political power, corruption manifests itself in many forms such as bribery, extortion, nepotism, patronage, embezzlement, and graft.
For example, excessive and redundant government regulations provide opportunities for bribery or graft. In addition, government regulations or restrictions in one area may create informal markets in another. As a result, corruption and the informal economy are often correlated.
All these potential ways of corrupting data are difficult to measure directly. We have adopted a general measure of corruption as a proxy for Government interference .The score for this component of the DQR is derived directly from Transparency International’s Corruption Perceptions Index (CPI), which measures the level of perceived corruption in 175 countries.
The CPI score is based on a 100-point scale in which a score of 100 indicates very little corruption and a score of 0 indicates rampant corruption. That is, in the DQR, the lower the level of corruption, the higher a country’s score. Similarly, the higher the level of corruption, the lower a country’s score. This factor varies from the United States of America with a score of 77 to Haiti scoring only 9 when standardised into the Data Quality Ratings.
CALCULATING THE GDP DATA RATINGS
Differences in reliability are highlighted by weighting and combining the five factors discussed. The three variables given the most weight are the objective Base year and SNA Indexes, with 30% and 20% respectively, together with the research based Informal Economy values with a weighting of 15%. The Capacity indicator is weighted at 20% while Corruption is weighted 15%.
Having weighted the five factors, they are then combined into the GDP Data Quality Index. The data are then ranked by quartile and each country receives a summary rating from A to D grade.
The summary ratings are described below:
Grade A: As Good as it Gets
Countries in this grade all have up-to-date Base Years or use the chaining methodology and all employ the most up to date SNA 2008 standard for measuring GDP. They have lower levels of unmeasured informal economic activities with a median estimate of just over 14 per cent of official GDP and generally low levels of corruption. The quality of the economic data in ‘A’ graded countries is “as good as it gets” and can be used for most purposes, although the usual caution must be taken given less than full implementation of the latest standards, difficulties in estimating government output and the services sector.
Grade B: Use With Caution
Nearly half of the countries in this grade have out of date base years ranging from 3 to 8 years and nearly one in five still employ the outmoded SNA 1993 standard for measuring GDP. The average level of unmeasured informal economic activities as a proportion of official GDP at 28 per cent is over twice the estimate for ‘A ‘ranked countries with a range from 12 to 53 per cent. Corruption levels on average are twice as high as in category ‘A’ countries. The quality of economic data in ‘B’ graded countries should be used with caution means that it provides a reasonable guide of GDP for some purposes should as growth and approximate size, but it should not be used to make direct comparisons to countries ranked with A grade data. Particular attention should be made to the quality of government data and price indexes.
Grade C: Unreliable Guide for Many Purposes
The vast majority of the countries in this grade have out of date base years apart from six which use chaining. The lag ranges from 4 to 18 years with an average of 7 years out of date. Two-thirds of the countries still employ the SNA 1993 standard for measuring GDP which means that the average level of unmeasured informal economic activities as a proportion of official GDP is high at 32 per cent with a range from 15 to 60 per cent. Corruption levels remain high. The quality of economic data in ‘C’ graded countries provides a generally unreliable guide for many purposes, particularly investment, but some attempts are being made by many countries to improve accuracy.
Grade D: Extremely Poor
All of the countries in this grade have out of date base years with a lag ranging from 5 to 32 years with an average of 15 years out of date over twice the figure for category C countries. Only two countries use the latest standard for measuring GDP with virtually all of the other countries using SNA 1993, apart from seven which still use SNA 1968. The average level of unmeasured informal economic activities as a proportion of official GDP is very high at 37 per cent and ranges from 15 to 67 per cent. Corruption is endemic with levels almost twice as high as in category C countries. The quality and reliability of economic data in ‘D’ graded countries is extremely poor and official GDP data should not be used for any purpose.