Artificial intelligence in scientific research: ‘you need a certain basic scepticism’

‘One thing must be clear: AI is really only ever an assistant. And one that cannot be trusted blindly. You have to question it, accompany it throughout the process and maintain a certain basic scepticism. Also because chatbots hallucinate when they don’t know the answer to a question, meaning they make up answers,’ explains Carmen Loschke, researcher at Oeko-Institut.

Loschke uses artificial intelligence all the time, including for personal purposes. ‘I wouldn’t use ChatGPT for simple search queries – also because AI consumes significantly more energy than a classic search engine. But if I want to learn something about a specific topic, it’s a valuable tool because I can ask questions and influence the complexity of the answers, for example.’ Following a barista course, the researcher learnt a lot about different types of coffee and their acid and bitter notes, the harvesting process and the different ways of preparing coffee. But she also uses AI for her scientific work, even in her master's thesis. ‘I evaluated the discussions on X, then still Twitter, during the German energy crisis on the subject of saving energy. Various AIs helped me with the programming and analysis."

Using AI to evaluate scientific publications

How can energy poverty in the EU be combated? A policy guide created by the researchers Dr. Sibylle Braungardt and Carmen Loschke from the Energy & Climate Division provide answers to this question with the help of a Large Language Model (LLM). ‘The quality of life of low-income households can be significantly improved through energy efficiency, as this reduces energy costs,’ says Loschke.

There are many ways to achieve energy efficiency, which were discussed at the eceee Summer Study, a conference of the European energy efficiency community, in 2024. Organised by the European Council for an Energy Efficient Economy (eceee), over 400 experts exchanged ideas in presentations and specialist articles. ‘However, due to time and personnel constraints, it is not really possible to read all the publications. That’s a great pity, because a lot of exciting analyses are being produced,’ says the Oeko-Institut researcher. ‘This is why we set ChatGPT the task of evaluating the publications that were produced as part of the summer study.’ GPT-4.0 was chosen because the chatbot is easily accessible and customisable. By setting up a RAG architecture (Retrieval Augmented Generation), through which the model accesses a defined knowledge database, it was possible for the chatbot to create well-founded articles on the topic of energy poverty. ‘This required a step-by-step process in which the input and results were repeatedly checked and the prompts, i.e. the questions asked of the AI, were adjusted as necessary.’ In some cases, the experts had to use workarounds. This is because ChatGPT does not allow more than 20 uploaded documents, each of which must not exceed a certain file size and number of tokens – but there were more than 130 publications. ‘We reformatted them and combined them into one document, which also brought challenges because the AI initially focused more on the first part of a document.’

The resulting policy guide summarises the latest research findings and policy developments and is intended to help policy makers strengthen energy efficiency, implement EU directives in this area and thus support the transition to a sustainable but also socially just energy system. ‘It addresses the challenges of energy poverty, discusses measurable indicators and presents positive examples from Europe.’ With the help of artificial intelligence, the Oeko-Institut researchers have also summarised numerous recommendations for combating energy poverty. ‘In our view, it makes sense, for example, to support disadvantaged households with financial aid for efficient heating systems and to set up local energy advice points,’ says the expert. ‘The policy guide also emphasises that even small and inexpensive measures such as LED lighting and draught-proof windows can be very useful for short-term savings.’

The talking database

Despite all the hurdles and challenges, Carmen Loschke sees great potential in turning ChatGPT into a research assistant. ‘Without AI, it would have taken much longer to gather the relevant information and write the paper. Although the first attempt took quite a long time overall, it will be quicker in future thanks to the learning effects.’ There is no question in the expert’s mind that she wants to use artificial intelligence again and again for her research work. Her idea: there would need to be an eco-chat bot that is fed with all the studies, analyses and other publications by Oeko-Institut and that people can then talk to. ‘It’s impossible to know all of your colleagues' work. So you could search very specifically for what has already been researched in the different areas.’

Using AI to evaluate heat plans

There are also already plans to evaluate municipal heat plans in 2025 as part of a research project under the 8^th Energy Research Programme of the Germany Federal Ministry for Economic Affairs and Climate Action. ‘There aren’t too many of them yet, but according to the Heat Planning Act, all municipalities have to produce them by 2026 or 2028, depending on the size of the municipality. That would mean about 12,000 heat plans. Artificial intelligence can play an important role in analysing them, especially since there are no guidelines for the approaches and the heat plans can therefore vary greatly.’ The aim of the project is to create a database based on all heat plans, from which different information can be extracted, e.g. how often the use of biomass or hydrogen is planned or how much CO₂ is to be reduced. ‘The project also aims to analyse how the heat plans fit in with the energy transition scenarios and where there are still differences that need to be addressed,’ says the Oeko-Institut expert.

Carmen Loschke is a researcher in the Energy & Climate Division. Since 2023, she has been working on the analysis of text data sets with the help of artificial intelligence. She has already used it to evaluate social media posts for her Master thesis.

Further information

Policy Guide “Leveraging Energy Efficiency to Combat Energy Poverty in the European Union”

Publication “What motivates and demotivates energy savings in times of crisis? - An argument mining analysis using X/Twitter data”

Publication “Has the energy crisis polarized citizens views on energy efficiency policy? - An analysis on the German discourse on X/Twitter”

Municipal heat planning for all of Germany - on the German government's website