Let op: Tweakers stopt per 2023 met Tweakblogs. In dit artikel leggen we uit waarom we hiervoor hebben gekozen.

Push data to Luftdaten and OpenSenseMap API with Home Assistant

Door Hmmbob op maandag 24 augustus 2020 16:38 - Reacties (13)
Categorie: Home Assistant, Views: 5.944

Over the last few weeks I've been working on a nice tinkering project with my oldest son (10). We wanted to create a little bird house containing a few environmental sensors and get that data into Home Assistant. But because sharing = caring, we also wanted to push the data to several Citizen Science projects such as Luftdaten (also called sensor.community) and OpenSenseMap. This blog will describe the steps we took to make this happen. All Home Assistant code is also available in my GitHub repo.

Building the sensor

We started building the sensor with the sensor.community building instructions, using the following components:
  • NodeMCU ESP8266
  • SDS011 Particle Sensor
  • BME280 Temperature/Pressure/Humidity Sensor
Time to let the kid hook everything up (after soldering headers to the BME280):
https://tweakers.net/i/kTWXkl4ykBmeAn92nf4XhxR2f0I=/100x75/filters:strip_icc():strip_exif()/f/image/fCLoylqM5pT90NaM6f5eugOk.jpg?f=fotoalbum_smallhttps://tweakers.net/i/E-1R2L995bXOUrcn0DHLVn9TU1Y=/100x75/filters:strip_icc():strip_exif()/f/image/4BbKyzrIfKujfWSqPLUE1eQ4.jpg?f=fotoalbum_smallhttps://tweakers.net/i/PiruREnHdHO8OiWufBNN-IUcqlg=/100x75/filters:strip_icc():strip_exif()/f/image/mwYJin33iqNLruoBcoV7ZxPR.jpg?f=fotoalbum_small

Firmware

After plugging all the components together, it was time to flash the firmware. We initially started using the default Luftdaten firmware, but overnight I was thinking that this did not match the "local first, cloud when I want to" philosophy that I use for my Home Assistant. That is why we decided to flash the NodeMCU with the ESPHome software and push data directly into my Home Assistant instance and, from there, push it towards different cloud APIs. This opposite to letting the device push it to the cloud first and then collect it from the cloud into Home Assistant. As added benefit the device now lives in a different, restricted, subnet without direct internet access.

The ESPHome firmware was built as follows:

YAML:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
esphome:
  name: weerhuisje
  platform: ESP8266
  board: nodemcuv2

wifi:
  networks:
  - ssid: "<<My SSID>>"
    password: "<<My WiFi Password>>"
  ap:
    ssid: "Weerhuisje AP"
    password: "<<Backup AP Password>>"
  manual_ip:
    static_ip: 10.0.1.2
    gateway: 10.0.1.1
    subnet: 255.255.255.0
  power_save_mode: none

captive_portal:
web_server:
logger:

# Enable Home Assistant API
api:
  reboot_timeout: 5min

ota:

time:
  - platform: homeassistant
    id: homeassistant_time
    timezone: Europe/Amsterdam

# Enable UART for SDS011
uart:
  rx_pin: D1
  tx_pin: D2
  baud_rate: 9600

# Enable i2c for BME280
i2c:
  sda: D5
  scl: D6

sensor:
  - platform: sds011
    pm_2_5:
      name: "Particulate Matter <2.5µm Concentration"
      accuracy_decimals: 2
    pm_10_0:
      name: "Particulate Matter <10.0µm Concentration"
      accuracy_decimals: 2
    update_interval: 30min
  - platform: bme280
    temperature:
      name: "BME280 Temperature"
      accuracy_decimals: 2
    pressure:
      name: "BME280 Pressure"
      accuracy_decimals: 4
    humidity:
      name: "BME280 Humidity"
      accuracy_decimals: 2
    update_interval: 5min
    address: 0x76
  - platform: uptime
    name: "Weerhuisje uptime"
    update_interval: 60s
  - platform: wifi_signal
    name: "Weerhuisje WiFi signaal"
    update_interval: 60s

binary_sensor:
  - platform: status
    name: "Weerhuisje Status"

text_sensor:
  - platform: wifi_info
    ip_address:
      name: Weerhuisje IP Address
    ssid:
      name: Weerhuisje Connected SSID
    bssid:
      name: Weerhuisje Connected BSSID
  - platform: version
    name: Weerhuisje Version

Initial upload was really easy by using the ESPHome Flasher tool - just connect the device with a USB cable, install the correct drivers (see sensor.community building instructions) and you are good to go.

Integrating with Home Assistant

When the device was done booting, I found a bunch of new sensors in my Home Assistant:

https://tweakers.net/i/JH3hOjslkTIXYyjDblOFhIIMgWM=/100x75/filters:strip_exif()/f/image/Eq9xx6ht7I6tU9JLEHWuIPO3.png?f=fotoalbum_small

So that works! The device is now pushing data to Home Assistant, ready for us to play with. Let's start playing.
REST commands
In order to upload the data to the various APIs, one needs to create a few rest commands in HA. We would like to push to Luftdaten (and their associates at Madavi.de) and OpenSenseMap. I'll leave it up to you to navigate to their websites and set-up your accounts there (Madavi uses the same Sensor-ID as Luftdaten). These accounts will need to be set up first to make this work. Source at my GitHub
YAML:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
# rest_command.yaml, included from configuration.yaml

##################################
## Push values to Air Quality APIs
##################################

# The BME280 pressure value in HA is in hPa - the APIs expect it in Pa, hence the 
# multiplication of the value in the templates below.

# Push to Luftdaten API. Luftdaten uses headers to distinguish between different sensor
# types, so we need to push twice. The X-Sensor header contains the sensorID from Luftdaten,
# typically formatted as esp8266-12345678 or similar.
send_luftdaten_pm:
  url: https://api.sensor.community/v1/push-sensor-data/
  method: POST
  content_type: 'application/json'
  headers:
    X-Pin: 1  ## This tells Luftdaten that it is SDS011 data
    X-Sensor: !secret luftdaten_x_sensor
  payload: >
    {
      "software_version": "HomeAssistant-{{ states('sensor.current_version') }}",
      "sensordatavalues":[
        {"value_type":"P1","value":"{{ states('sensor.particulate_matter_10_0um_concentration') }}"},
        {"value_type":"P2","value":"{{ states('sensor.particulate_matter_2_5um_concentration') }}"}
      ]
    }
send_luftdaten_tph:
  url: https://api.sensor.community/v1/push-sensor-data/
  method: POST
  content_type: 'application/json'
  headers:
    X-Pin: 11  ## This tells Luftdaten that it is BME280 data
    X-Sensor: !secret luftdaten_x_sensor
  payload: >
    {
      "software_version": "HomeAssistant-{{ states('sensor.current_version') }}",
      "sensordatavalues":[
        {"value_type":"temperature","value":"{{ states('sensor.bme280_temperature') }}"},
        {"value_type":"pressure","value":"{{ states('sensor.bme280_pressure') | float * 100 }}"},
        {"value_type":"humidity","value":"{{ states('sensor.bme280_humidity') }}"}
      ]
    }

# Push to Madavi. This is related to Luftdaten and stores data for use in Grafana.
send_madavi:
  url: https://api-rrd.madavi.de/data.php
  method: POST
  content_type: 'application/json'
  headers:
    X-Pin: 0
    X-Sensor: !secret luftdaten_x_sensor
  payload: >
    {
      "software_version": "HomeAssistant-{{ states('sensor.current_version') }}", 
      "sensordatavalues":[
        {"value_type":"SDS_P1","value":"{{ states('sensor.particulate_matter_10_0um_concentration') }}"},
        {"value_type":"SDS_P2","value":"{{ states('sensor.particulate_matter_2_5um_concentration') }}"},
        {"value_type":"BME280_temperature","value":"{{ states('sensor.bme280_temperature') }}"},
        {"value_type":"BME280_pressure","value":"{{ states('sensor.bme280_pressure') | float * 100 }}"},
        {"value_type":"BME280_humidity","value":"{{ states('sensor.bme280_humidity') }}"}
      ]
    }

# Push to OpenSenseBox / OpenSenseMap. The url !secret contains the openSenseBox API url,
# which looks like https://api.opensensemap.org/boxes/<<yoursenseboxid>>/data
# The input_text items contain the sensor-IDs you need to publish the data to the API.
# You can find those on your SenseBox page on https://opensensemap.org/account
post_opensensebox:
  url: !secret opensensebox_api_url
  method: post
  headers:
    content-type: "application/json; charset=utf-8"
  payload: >-
    {
      "{{ states('input_text.opensensebox_sensorid_temp') }}": "{{ states('sensor.bme280_temperature') }}",
      "{{ states('input_text.opensensebox_sensorid_press') }}": "{{ states('sensor.bme280_pressure') | float * 100 }}",
      "{{ states('input_text.opensensebox_sensorid_hum') }}": "{{ states('sensor.bme280_humidity') }}",
      "{{ states('input_text.opensensebox_sensorid_pm25') }}": "{{ states('sensor.particulate_matter_2_5um_concentration') }}",
      "{{ states('input_text.opensensebox_sensorid_pm10') }}": "{{ states('sensor.particulate_matter_10_0um_concentration') }}"
    }
Secrets
I'm using secrets above to keep the API endpoints for my sensors off GitHub. As the template payload for OpenSenseMap doesn't support the use of !secret, I worked around this by using input_text items.

Change the following items to show the data from your account:
YAML:
1
2
3
4
5
6
7
8
9
# secrets.yaml

luftdaten_x_sensor: esp8266-99999999 
opensensebox_api_url: https://api.opensensemap.org/boxes/<<YourSenseBoxID>>/data
opensensebox_sensorid_temp: 123456789012345678901231
opensensebox_sensorid_press: 123456789012345678901232
opensensebox_sensorid_hum: 123456789012345678901233
opensensebox_sensorid_pm25: 123456789012345678901234
opensensebox_sensorid_pm10: 123456789012345678901235

And the work-around to get the !secret data into the rest template (Source at my GitHub):
YAML:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# input_text.yaml,  included from configuration.yaml

##################################
## Input text configuration
##################################

# I am using this workarond to hide sensor-IDs needed in the rest_command 
# payloads. These templates cannot handle secrets, but you can set these 
# input_text items with !secret, so in this way the IDs are obfuscated.
# I don't mind having these in my HA instance, I just don't want them on 
# GitHub to prevent copy-paste errors.

opensensebox_sensorid_temp:
  initial: !secret opensensebox_sensorid_temp

opensensebox_sensorid_press:
  initial: !secret opensensebox_sensorid_press

opensensebox_sensorid_hum:
  initial: !secret opensensebox_sensorid_hum

opensensebox_sensorid_pm25:
  initial: !secret opensensebox_sensorid_pm25

opensensebox_sensorid_pm10:
  initial: !secret opensensebox_sensorid_pm10
Automations
Now all data is available and the REST commands are ready for use, let's start using them! Source at my GitHub

YAML:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#air_quality.yaml, included from configuraiton.yaml automations

##################################
## Push values to Air Quality APIs
##################################
# Sensor uses a NodeMCUv2 and SDS011+BME280 sensors. NodeMCU is flashed with ESPHome
# software. SDS011 updates every 30 mins, the BME280 every 5 minutes. As all BME280
# sensors update at the same time, we only need to monitor one sensor to trigger
# the automation. Temperature is the sensor that changes most often.
- alias: 'Weerhuisje: Update Air Quality Sensors'
  initial_state: true
  mode: queued
  trigger: # Trigger on every update of the temperature
    platform: state
    entity_id: sensor.bme280_temperature 
  action:
    ## Send to Luftdaten / Sensor.Community API
    ## Split over two calls as the headers differ
    - service: rest_command.send_luftdaten_tph
    - service: rest_command.send_luftdaten_pm
    ## Send updates to Madavi.de too for statistics.
    - service: rest_command.send_madavi
    ## Send data to OpenSenseMap. 
    - service: rest_command.post_opensensebox

- alias: "Weerhuisje: Notify if weerhuisje Offline"
  initial_state: true
  trigger:
    - platform: state
      entity_id: binary_sensor.weerhuisje_status
      to: 'off'
      for:
        seconds: 60
  action:
    service: notify.notify
    data_template:
      title: "Weerhuisje is Offline"
      message: "Weerhuisje is offline"

End result

https://tweakers.net/i/cNpmTjBhJE3Q0XiTwImK3NzimR0=/100x75/filters:strip_icc():strip_exif()/f/image/nMhbeOU67HaKRQTTDJJ8sm4b.jpg?f=fotoalbum_smallhttps://tweakers.net/i/cMq8FSx2TeLSmxPx6v0oTK0vXFM=/100x75/filters:strip_icc():strip_exif()/f/image/G6Qj7JWlIuISXJu97vApmldS.jpg?f=fotoalbum_smallhttps://tweakers.net/i/S1dSGydVxX-TWLVdKipdKkihtjk=/100x75/filters:strip_icc():strip_exif()/f/image/3Jq03oCT9S1RlrN77Zm2S5jg.jpg?f=fotoalbum_small
(Still need to put it on the wall - scheduled for later this week)

At OpenSenseMap:
https://tweakers.net/i/QL1U7tpeDiAFdo0_qEVM7ZO3jqQ=/100x75/filters:strip_exif()/f/image/aepLIJ9PqOwFMm8mVfJW4RMO.png?f=fotoalbum_small

Luftdaten info at Madavi:
https://tweakers.net/i/Jif_lGPxK4GTrrRZTfXlsEvmrDs=/100x75/filters:strip_exif()/f/image/aG2O3erYx2Km2wdGH8xkkuR4.png?f=fotoalbum_small

Volgende: NS Reisinformatie in Home Assistant 11-'17 NS Reisinformatie in Home Assistant

Reacties


Door Tweakers user m-vw, maandag 24 augustus 2020 19:36

Ziet er leuk uit, maar wat als er effectief een vogel in gaat nestelen? Het gat is open.

Door Tweakers user Hmmbob, maandag 24 augustus 2020 19:40

m-vw schreef op maandag 24 augustus 2020 @ 19:36:
Ziet er leuk uit, maar wat als er effectief een vogel in gaat nestelen? Het gat is open.
Goed punt, daar kan ik nog wel iets voor maken.

Door Tweakers user naftebakje, dinsdag 25 augustus 2020 10:03

m-vw schreef op maandag 24 augustus 2020 @ 19:36:
Ziet er leuk uit, maar wat als er effectief een vogel in gaat nestelen? Het gat is open.
Ventilator toevoegen om als CO² te hoog wordt verse lucht toe te voegen? Verwarming zodat de vogel lekker warm zit?
Gewoon een tussenschot toevoegen zodat het nog steeds voor een vogel te gebruiken is lijkt me wel zo fijn.

Door Tweakers user himlims_, dinsdag 25 augustus 2020 14:17

is domoticz niet kleiner/rapper voor deze toepassing?

Door Tweakers user Hmmbob, dinsdag 25 augustus 2020 14:35

himlims_ schreef op dinsdag 25 augustus 2020 @ 14:17:
is domoticz niet kleiner/rapper voor deze toepassing?
Ik gebruik al Home Assistant voor andere zaken.

Door Tweakers user Hmmbob, dinsdag 25 augustus 2020 14:35

naftebakje schreef op dinsdag 25 augustus 2020 @ 10:03:
[...]
Ventilator toevoegen om als CO² te hoog wordt verse lucht toe te voegen? Verwarming zodat de vogel lekker warm zit?
Gewoon een tussenschot toevoegen zodat het nog steeds voor een vogel te gebruiken is lijkt me wel zo fijn.
het is niet bedoeld als vogelhuisje, enkel voor de looks. We hebben meer vogelhuisjes waar het vogeltje dan maar naar toe gaat :)

Door Willem, zondag 18 oktober 2020 14:21

Ik heb gisteren eindelijk ook mijn sds011 aangesloten via esphome.
Het viel me op dat `update_interval: 30min` geen geldige setting is.
De sds011 gaat dan in de default modus, wat ervoor zorgt dat er iedere seconde een update wordt uitgelezen uit de sensor.
Dit verkort de levensduur van de sensor aanzienlijk!
De sds011 heeft een levensduur van ongeveer 8000 uur.

Dus misschien waard om nog te controleren of 30min echt werkt.

Door Hmmbob, zondag 18 oktober 2020 15:03

Willem schreef op zondag 18 oktober 2020 @ 14:21:
Ik heb gisteren eindelijk ook mijn sds011 aangesloten via esphome.
Het viel me op dat `update_interval: 30min` geen geldige setting is.
De sds011 gaat dan in de default modus, wat ervoor zorgt dat er iedere seconde een update wordt uitgelezen uit de sensor.
Dit verkort de levensduur van de sensor aanzienlijk!
De sds011 heeft een levensduur van ongeveer 8000 uur.

Dus misschien waard om nog te controleren of 30min echt werkt.
Hmm, in Home Assistant krijg ik toch echt maar iedere 30min een nieuwe waarde tegen. Zal ESPHome eens aanzetten en de log mee kijken.

Door Hmmbob, zondag 18 oktober 2020 15:51

Hmmbob schreef op zondag 18 oktober 2020 @ 15:03:
[...]
Hmm, in Home Assistant krijg ik toch echt maar iedere 30min een nieuwe waarde tegen. Zal ESPHome eens aanzetten en de log mee kijken.
Gaat toch echt prima, Willem. Weet je zeker dat je je config goed hebt?

code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[13:04:42][C][sds011:054]: SDS011:
[13:04:42][C][sds011:055]:   Update Interval: 30 min
[13:04:42][C][sds011:056]:   RX-only mode: OFF
[13:04:42][C][sds011:057]:   PM2.5 'Particulate Matter <2.5µm Concentration'
[13:04:42][C][sds011:057]:     Unit of Measurement: 'µg/m³'
[13:04:42][C][sds011:057]:     Accuracy Decimals: 2
[13:04:42][C][sds011:057]:     Icon: 'mdi:chemical-weapon'
[13:04:42][C][sds011:058]:   PM10.0 'Particulate Matter <10.0µm Concentration'
[13:04:42][C][sds011:058]:     Unit of Measurement: 'µg/m³'
[13:04:42][C][sds011:058]:     Accuracy Decimals: 2
[13:04:42][C][sds011:058]:     Icon: 'mdi:chemical-weapon'
[13:16:22][D][sensor:092]: 'Particulate Matter <2.5µm Concentration': Sending state 16.80000 µg/m³ with 2 decimals of accuracy
[13:16:22][D][sensor:092]: 'Particulate Matter <10.0µm Concentration': Sending state 18.70000 µg/m³ with 2 decimals of accuracy
[13:46:16][D][sds011:154]: Got PM2.5 Concentration: 22.4 µg/m³, PM10.0 Concentration: 24.7 µg/m³
[13:46:16][D][sensor:092]: 'Particulate Matter <2.5µm Concentration': Sending state 22.40000 µg/m³ with 2 decimals of accuracy
[13:46:16][D][sensor:092]: 'Particulate Matter <10.0µm Concentration': Sending state 24.70000 µg/m³ with 2 decimals of accuracy

Door Eagle, zondag 20 december 2020 01:58

Hele mooie oplossing, ik probeer deze ook werkend te krijgen maar de SDS011 sensor werkt niet (wel als ik deze stappen volg https://sensor.community/en/sensors/airrohr/). De BME280 werkt wel. Enig idee wat dit zou kunnen zijn? Hieronder de yaml die ik gebruik:

esphome:
name: fijnstof_sensor
platform: ESP8266
board: nodemcuv2

wifi:
ssid: "xxxxxxxx"
password: "xxxxxxxx"

# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "xxxxxxxx"
password: "xxxxxxxx"

captive_portal:

# Enable logging
logger:

# Enable Home Assistant API
api:
password: "xxxxxxxx"

ota:
password: "xxxxxxxx"

time:
- platform: homeassistant
id: homeassistant_time
timezone: Europe/Amsterdam

# Enable UART for SDS011
uart:
rx_pin: D1
tx_pin: D2
baud_rate: 9600

# Enable i2c for BME280
i2c:
sda: D3
scl: D4

sensor:
- platform: sds011
pm_2_5:
name: "Particulate Matter <2.5µm Concentration"
accuracy_decimals: 2
pm_10_0:
name: "Particulate Matter <10.0µm Concentration"
accuracy_decimals: 2
update_interval: 30min
- platform: bme280
temperature:
name: "BME280 Temperature"
accuracy_decimals: 2
pressure:
name: "BME280 Pressure"
accuracy_decimals: 4
humidity:
name: "BME280 Humidity"
accuracy_decimals: 2
update_interval: 3min
address: 0x76
- platform: uptime
name: "Fijstof sensor uptime"
update_interval: 60s
- platform: wifi_signal
name: "Fijstof Sensor WiFi signaal"
update_interval: 60s

binary_sensor:
- platform: status
name: "Fijstof sensor Status"

text_sensor:
- platform: wifi_info
ip_address:
name: Fijnstof sensor IP Address
ssid:
name: Fijnstof sensor Connected SSID
bssid:
name: Fijnstof sensor Connected BSSID
- platform: version
name: Sensor Version

Door Tweakers user Hmmbob, zondag 20 december 2020 09:12

Eagle schreef op zondag 20 december 2020 @ 01:58:
Hele mooie oplossing, ik probeer deze ook werkend te krijgen maar de SDS011 sensor werkt niet (wel als ik deze stappen volg https://sensor.community/en/sensors/airrohr/). De BME280 werkt wel. Enig idee wat dit zou kunnen zijn? Hieronder de yaml die ik gebruik:
YAML:
1
2
3
4
5
# Enable UART for SDS011
uart:
  rx_pin: D1
  tx_pin: D2
  baud_rate: 9600
Het enige wat ik me kan bedenken is dat je SDS011 misschien niet op D1 en D2 aangesloten zit, en/of verkeerd om (dus draadjes even van pin wisselen)?

Wat zeggen de (debug) logs?

[Reactie gewijzigd op zondag 20 december 2020 09:12]


Door Eagle, zondag 20 december 2020 12:25

Het werkt nu, eerste meting komt na een half uur binnen. De tx pin van de SDS011 heb ik nu aangesloten op D1 en de rx op D2. En in de YAML:
# Enable UART for SDS011
uart:
rx_pin: D1
tx_pin: D2
baud_rate: 9600
Hmmbob schreef op zondag 20 december 2020 @ 09:12:
[...]
Het enige wat ik me kan bedenken is dat je SDS011 misschien niet op D1 en D2 aangesloten zit, en/of verkeerd om (dus draadjes even van pin wisselen)?

Wat zeggen de (debug) logs?

Door Tweakers user pmvdbroek, zaterdag 2 januari 2021 17:15

Erg leuk project!
Ik heb vandaag mijn sensor node omgebouwd op basis van jouw voorbeeld.
Ik had vaak last dat de website eruit lag en heb inderdaad liever mijn eigen data in beheer en had al mijn andere esp nodes al op esphome platform staan.

Bedankt voor dit leuke voorbeeld, weer eea geleerd!

Reageren is niet meer mogelijk