Data infrastructure

Online seit Fri 19 March 2021 in Infrastructure

In the data managment post I described how data was going to be processed and the initial schema for the InfluxDB database. I will post an update on that reflect the actual situation (not very different). In this post I provide a birds-eye view of the data infrastructure:

Data infrastructure

Logger

The data travels from the data-logger to our WABEsense server. Currently this transfer involves a manual step, in which the operator (often the Brunnenmeister) carries the data from the logger to the Raspberry Pi node. From that point on, everything is automated.

The manual step can be automatized, for example by wireless link between the logger and the node. Our project is constrained here by the fact that municipalities are not yet convinced of the value of the data we are collecting, hence they keep their investment to the minimum. Follow-ups on this project will focus on exploiting the data to generate scientific and societal value. This will eventually open up larger investments by the municipalities and/or the Cantons, and allow us to install the required communication infrastructure.

There is also another alternative to automatize the data transfer (and actually to reduce the overall complexity of this project!). That is, to utilize the currently existing infrastructure, installed by control and automation companies. However, the price tags these companies put to setup an open interface made this option unfeasible. The WABEsense project deliberately puts out into the market an open-hardware data-logger in an attempt to disrupt the current situation. The users of the WABEsense data-logger, e.g. the water utilities, will be able to retrieve data from the project in an automated way, we are still deciding on the interface. See InfluxDB section below for more information. We hope this will bring us to a better situation in which the societal stakeholders (municipalities and water consumers) can customize, adapt, and optimize their infrastructure without being held back by private interests.

InfluxDB

Once the data arrived to the WABEsense server we make a raw data back-up, process the data, and upload it to the influxDB OSS instance running on that server.

InfluxDB allows other parties to query for data using its HTTPS API. Therefore, it is possible to offer the data in an automated way. However, we are still discussing what is the best way to share this data publicly. We need to consider hardware investments and make best use of the project's resources. The options being considered are:

Grafana

The Grafana OSS that is also running in the server picks the data from InfluxDB and visualizes it in several dashboards. We are currently polishing the dashboards to include feedback we get from the α-sites users and the WABEsense team members. Some of these dashboards will be publicly available soon, so stay tuned for more news regarding public data visualization.

If you have never seen a Grafana dashboard, here is an screenshot of the one showing the discharge from the Ulrika spring in Oberriet (more about Ulrika here)

Ulrika discharge till 13-04-2021

Stay tuned!