Breakthrough or boast? The quest for comparable research results

In late 2019, physicist Dr Lorenzo Pattelli was part of an Italian-Chinese scientific team working on a cooling technology that is fast gaining attention as the Earth gets hotter from climate change.

Called passive daytime radiative cooling, or PDRC, the technology uses engineered materials to reflect away the sun’s radiation. The idea is that, amid heat waves, PDRC panels would cool buildings without the need for energy-intensive air conditioning.

Desert surprise

Right before Pattelli’s team was ready to publish the results of its study into the cooling effects of a particular material, another research group released a paper on a very similar material that exhibited double the cooling effect. 

‘At first, we were a bit disappointed as it looked like we were beaten to the punch,’ Pattelli said. ‘But these things happen in science.’

Welcome to the world of metrology – the scientific field that studies measurements. When gauging the performance of new technologies, scientists need to use common standards so that results are comparable.

Initially, the Italian-Chinese team of researchers couldn’t explain the discrepancy between the findings about the cooling effect of the materials. How could the two experiments produce such different results? 

The answer from the other group was found in its paper’s supporting information, which said that the testing was done in the Atacama desert in Chile.

‘The Atacama plateau is located at high altitude with lower atmospheric pressure and very low humidity,’ he said. ‘We, on the other hand, tested our material in downtown Beijing. Neither study was wrong, but the other group tested its material in much more favourable conditions.’

Trust test

This highlights why metrology is so important. Without it, research results can vary dramatically and, as a result, public trust in science can be eroded.

“

Metrology is very much needed.

International institutions are beginning to pay more attention to metrology. UNESCO for the first time will celebrate World Metrology Day on 20 May 2024 and, through the European Partnership on Metrology, the EU and its Member States are investing in the field.

A researcher at the National Metrology Institute of Italy, Pattelli now leads a project that received partnership funding to determine the best measurement standards for PDRC materials. Called PaRaMetriC, the project began in 2022 and runs until 2025. 

Excitement around PDRC has prompted scientific teams around the world to hunt for the greatest possible cooling effects for particular materials, resulting in some doubtful results, according to Pattelli. 

‘Some of the claims we’re seeing in more recent papers are clearly biased,’ he said. ‘Metrology is very much needed here to establish good practice guidelines.’

New benchmarks

PaRaMetriC is setting standards and measurement protocols for PDRC research. 

The project is seeking, among other things, to measure cooling through water instead of air temperature. 

The researchers have built a device to gauge the flux and temperature of water that goes in and out as it is being cooled by the material. The water touches the material here and is cooled by it, enabling better measurements of how much cooling takes place.

‘Measuring temperature in air is subject to high uncertainty, especially considering that the PDRC effect occurs outdoors,’ Pattelli said. ‘Measuring the temperature of liquids is a lot more reliable.’

He said that, while ambient temperature will still necessarily be measured in air, using liquids offers a more robust measurement of the actual cooling power provided by the different materials.   

PaRaMetriC plans to set benchmark measurements on a set of PDRC materials. These benchmarks can then be used by other projects as a common baseline to evaluate the cooling performances of newly proposed materials – similar to how members of an orchestra tune their instruments together so they all sound in line with one another.

Methane measurements

Metrology is useful not only to assess claims about new technologies, but also to spot the advance of climate change. 

Another project funded by the European Partnership on Metrology is seeking to improve measurements in the atmosphere of methane, a potent greenhouse gas. Called isoMET, the project also runs from 2022 until 2025.

“

Trust in research is limited if everyone measures things in their own way.

One method of determining the origin of methane in the atmosphere is called isotope ratio measurement. This can be done in several ways. 

In one – laser absorption spectroscopy – a reservoir is filled with air, after which laser light passes through it and measures the concentration and isotope ratio in the sample. This allows researchers to assess methane concentrations in a specific place – and even where it came from.

‘It’s like the fingerprint of the methane,’ said Dr Javis Nwaboh, research scientist at the National Metrology Institute of Germany and part of isoMET. ‘We can determine the source of it, for example agriculture or the burning of fossil fuel.’

Here again, however, a lack of common measurement standards or procedures prevails. 

‘Measurements aren’t harmonised between different laboratories,’ said Nwaboh. ‘So you might get different results between different studies. This makes it more difficult to tackle climate change.’

The isoMET researchers aim to establish one set of standards that can be used internationally so results become comparable.

Come together

After they come to an end in 2025, both isoMET and PaRaMetriC will try to get the scientific community as a whole to adopt their standards. The projects will do this by presenting their results to scientists and showing them why existing measurement standards may be inadequate.

For Nwaboh and Pattelli, such steps are essential to underpin trust in science and counter the possibility that scientists under pressure to publish interesting results will present their research as more spectacular than it actually is.

‘PRDC is a very promising field of research,’ said Pattelli. ‘It would be a shame if this technology gets hindered by papers whose results are difficult to compare. We have a better chance to advance science if everyone follows reproducible practices.’

Nwaboh echoed the point.

‘Trust in research is limited if everyone measures things in their own way,’ he said. ‘The right procedures can alleviate that.’

Research in this article was funded by the EU’s Horizon Programme. The views of the interviewees don’t necessarily reflect those of the European Commission. If you liked this article, please consider sharing it on social media.