Data, Models, and Standards

Once a model supporting energy planning activities in a developing country has been completed, it can be difficult to build on in the future. It may not ‘play nice’ with other efforts. It may not be reproducible. Most knowledge, especially of the tacit type, rests with the expert(s) who applied a model for a particular case study. Model development and application on the ground are usually performed by external consultants with counterpart assistance in data collection. Consultants are generally not requested to perform the full data documentation, there are no templates or standards to follow and, above all, consultants would have to be remunerated for any additional requirements.

All of this results in wasted efforts and resources, as duplication and splintered capacity occur. Data, models and processes are not shared (as they are not accessible). Neither their quality or origin can be easily traced. The outcome is inefficient coordination between donors, practitioners and consultants in any followed-up planning activity.

To address these issues, the Roundtable Initiative has outlined the rationale for the introduction of standards related to the documentation of data and data sources, methodologies applied, and assumptions used in energy system planning studies and projects. Such standards would:

• Support transparency and repeatability;
• Allow successive analyses and planning projects to build on each other in a coherent manner;
• Enable subsystems and sector analyses to coalesce;
• Improve donor efficacy and quality;
• Remove aspects that ‘lock-in’ or limit service providers;
• Have the potential to sustain long-term national planning capacity.

An ad-hoc Working Group has been set up within the Roundtable to develop standards that aim to make all steps in the modelling and data management process aligned with Ubuntu, Retrievability, Repeatability, Reconstructability, Reproducibility, Interoperability and Auditability (U4RIA):

• Ubuntu (community engagement): There should be clear plans and commitments by the modellers to engage with the national energy stakeholders to transfer knowledge, capacity and ownership of  the data, tools and models. The involvement of national stakeholders should go beyond requesting data and their interpretation. Plans for capacity building should be incorporated in all strategic energy planning support activities according to the possibilities of the project.
• Retrievability: Data should be easily retrievable, with good metadata, clear archiving and formats that allow for interoperability. However, currently data is often not easily retrievable. For instance, even widely cited datasets like the IEA-developed “Projected Cost of Generating Electricity” is not in a format retrievable by data search engines such as Google Data Set Search. Without systematic access to the data and other elements of the energy planning ecosystem, public transparency is greatly reduced. Poor retrievability and inability to test and audit outputs can easily result in lack of trust in modelling. Therefore, clear standards for enhancing the retrievability of/easy access to datasets are required.
• Repeatability: In theory, models should be showing the same results every time they are run. In practice, this is not always true. For instance, different technical specifications of the machine models are run on may produce the software to crash. Moreover, changes in subsequent versions of the same model could produce different results even if the input data are unchanged, as model calculations or default value (e.g. emission factors) have been amended. This may affect the trust of users in the model and any generated output using it. Therefore, it is important that (a) when results are recorded and disseminated, modellers specify the model’s version and machine specifications they have used; and (b) any changes between model versions and minimum technical requirements are clearly specified in new version releases and made the info publicly available.
• Reconstructability: There is the need for modellers from donors, academia, and recipient of energy planning services to develop minimum reporting standards for each element of the energy planning ecosystem. These standards will ensure that data (including metadata, assumptions, methodology, and outputs/results) behind energy planning analysis can, as far as possible, be subsequently reconstructed.
• Reproducibility: Once the reporting of model results is completed, a study is often published and archived. However, as models change – for example because of new software platforms, updates and bug fixes, changes in formulation – and/or data change, re-running old analysis might not be possible. The definition and application of best practices in data management and storage are therefore an important requirement of a sustainable energy planning ecosystem.
• Interoperability: The data available from statistical bureaus, administrations, industries or other stakeholder are rarely in the format needed by the energy planner. At the same time, models themselves often require data that are similar in substance, but different in form and/or specific computing requirements (e.g. a specific operating system). The consequence is that models often require extensive data manipulation that is not only lengthy and inefficient, but also multiplies the chances of errors. This risk of errors in existing data often makes it easier for modellers to start from scratch by constructing their own datasets. What is needed to begin with is a guide or manual on ‘best practices’ to facilitate the interoperability of datasets and models, including the definition of interchange policies with open standards, and interoperable and vendor neutral software.
• Auditability: Accountability to the public is essential for every government entity – including funders and bilateral partners.Therefore, it is important that all the previous principles are followed, so that a successful audit of the data and deliverables produced under energy modelling activities can be carried out.

The Roundtable’s Secretariat prepared a template that can be customised and finally included in the Terms of Reference of energy planning and modelling assignments to align the data, metadata and models produced and treated with the U4RIA goals. The purpose of this document is to provide donors, international organisations, development partners, and whoever else is going to commission activities involving strategic energy planning and modelling support to developing countries with a practical way to embed sound data and modelling transparency and management practices.

For further info or to be involved in the development of the U4RIA standards please refer to the U4RIA online forum.