PDF Basket
Earth is under the permanent scrutiny of hundreds of satellites. Their purpose: Gathering data on the likes of climate change, meteorology, natural resources, the oceans’ health or natural disasters. The EU’s own fleet of Earth Observation (EO) satellites, known as Copernicus Sentinels, provides over 20 000 gigabytes of new data every day to just about anyone interested in it. And it’s all free of charge, thanks to dedicated open data policies.
There is a problem though. As invaluable to researchers as it may be, open EO data comes in such enormous amounts that it’s becoming increasingly difficult to deal with. “We now have to store and process big EO data at a petabyte scale (Editor’s note: 1 petabyte is equivalent to 1 million gigabytes),” says Matthias Schramm, coordinator of the openEO project on behalf of TU Wien. One solution is cloud storage and processing. Many new market players have stepped into the breach, with different offerings for an increasingly large panel of users from the scientific, industrial and governmental sectors. They were very quick to do so, perhaps even too quick.
“The speed and momentum for the development of new cloud platforms prevented the development of widely accepted and used standards. This lack of comparable access points results in technical challenges for users, who can’t easily switch between service providers to compare results,” Schramm explains.
That’s where the EU-funded openEO project comes in. From October 2017 to November 2020, project partners have been developing a new communication interface that enables standardised access to EU data and processing capabilities across distinct cloud services providers. This new interface works in two steps. First, openEO libraries on users’ local machines prepare their workflows to send them to the service providers of choice. There, these standardised processing requests are automatically translated into the syntax understood by the provider.
More untapped potential
“We have openEO libraries for the programming languages Python, R and JavaScript. We also created a QGIS plug-in, a web editor and mobile applications to serve as access points. Various European cloud platforms are already connected to the openEO interface for standardised communication, and all aspects of the open-source interface have been designed to enable future connection with additional platforms and programming languages,” Schramm notes.
To ensure the comparability of different cloud providers’ results, the project team developed five use cases for pilot users, testing workflows with the openEO syntax on several platforms. They found that the only possible limitation to guaranteeing the comparability of cloud services was a lack of similar EO data (e.g. different sensors, resolution, sampling interval, provided area, pre-processing). Aside from that, differences have been found only in the scale of computational accuracy, which goes to show that the interface works particularly well.
“Several cloud platform providers external to the project, along with EO users and the climate research sector have already shown interest in embedding the openEO interface in their new services,” Schramm adds. “Right now, the openEO standard is a theoretical opportunity to jointly access cloud providers and compare them. This joint access still needs to be realised, for instance with real communication between cloud platforms using our new common language. We submitted several proposals for scientific projects to address this gap, some of which, like the Horizon 2020 C-SCALE project, have already received funding.”