Together in the cloud

In many fields of research there is talk of a “data explosion”, and the term is used particularly often in relation to satellite remote sensing. How did this come about?

Peter Zellner: The number of satellites orbiting the Earth, continuously scanning its surface with various technologies has increased dramatically in recent years. The Sentinel 1, 2 and 3 missions of ESA, the European Space Agency (ESA) in particular, were a game changer here; since then, the volume of data has increased exponentially and continues to rise.

How does this change the way researchers work?

Zellner: You can no longer work the way you did ten or fifteen years ago. Bath then, you were dealing with one or two satellites and the amount of data was so small that you could save it on your computer and continue working with it there. With today’s data volumes, this is impossible. That’s why the trend is heading towards cloud computing: the data is stored in a cloud, where the specialized software is also offered to process it in a fraction of the time it used to take. The calculations that are created can then also be made freely available via the cloud platform - so others can continue working with it.

Does this also lead to a more collaborative way of doing research?

Zellner: Yes, that is the open science approach. The individual processing steps are visible and reproducible, so you can see how the data was created and you can continue working with it directly in the same system. Let’s say someone develops an indicator for the greenness of vegetation from satellite data, then others can continue working with the calculation that has already been created. So you don’t have to start all over again. That saves resources.

Does this mean that computer science is playing an increasingly important role in the research process?

Zellner: The experts who develop these cloud platforms are working intensively on this of course. At our Institute for Earth Observation, it is the Advanced Computing research group, of which I am a member. However, the aim is actually to streamline the complexity and offer simple processes to those who use the software. That way the specialists in certain subject areas should have more time for their analysis and content, and not have to struggle with data formats and servers, which has always been a big part of the work up to now. The next step is to generate data products from which people without specialist knowledge can derive recommendations for action, for example for politics or agriculture. We have developed something like this with the Alpine Drought Observatory.

How does it work?

Zellner: It is an online tool that provides an overview of drought in the Alpine region in maps and graphics. We have harmonized the data from all Alpine countries and optimized it with high resolutions. The large amount of satellite data, meteorological and hydrological data that has been incorporated and continues to be incorporated has been prepared in process chains to such an extent that the result is completely self-explanatory: you can see at a glance what the drought situation is like in the Alps, you can click on a time series and see how much drier this year is compared to last year ...

Who participated in the project?

Zellner: It was a transnational and interdisciplinary team of meteorologists, agronomists, hydrologists, environmental engineers, remote sensing experts and technical specialists.
The scientists developed algorithms to derive drought indicators from satellite data and weather data and validated them, and then it was the job of the technical experts to implement this for the entire Alpine region and update it on a daily basis.

Do you know who uses the observatory’s data?

Zellner: Not in detail, because access to the portal is completely free and you don't have to register. We only know the number of hits, which is currently around 200 every week. During the major drought in the summer, we often had inquiries from journalists. And there is a lot of interest from other scientific platforms. For example, the “Green Transition Information Factory” which is an ESA pilot project, uses our data. By the way, not only is the data openly available, so is the developed software that makes up the ADO Platform; it is open source. Anyone can reuse it, for example to apply it to other regions of the world

You and your colleagues also developed the Massive Open Online Course (MOOC) “Cubes and Clouds” for ESA – what’s it about?

Zellner: This is related to the major change in working methods I mentioned earlier, which is necessitated by the enormous volume of satellite data. In order to effectively exploit this wealth of data, for example for the major research topic of climate change, we need expertise in cloud technologies, which to date, too few scientists have. Because development has been so rapid, there is still hardly any teaching material available. That’s why ESA commissioned us to develop this course. Interested researchers should have the chance to learn how to work effectively on cloud platforms, the concepts behind them and how to share data. In other words, how to actually do open science in the context of cloud computing. The task at the end of the course is to calculate the snow cover for a square kilometer area in the Alps for a specific year, issue an open license for this map, describe it precisely with metadata and make it available in a web map. Those who complete the course have contributed to a joint project of mapping the snow cover of the Alps and can also provide proof of the skills they have acquired.

And they have learned to conduct science openly, cooperatively and transparently.

Zellner: Cloud software close to the data, analyses that can be reproduced with just a few clicks and openly accessible results all enable effective cooperation between researchers. This is essential to tackle the burning issues of our time. The amount and variety of data currently available to us is unprecedented in the history of science. Consequently, individual scientists can no longer derive results from this complexity alone. The breakthrough in research is only possible through cross-border collaboration, across countries and disciplines. At Eurac Research and in our Institute for Earth Observation, we are trying to lay the building blocks for this and set an example of how to put them together.