Measure & Report Ambiance Condition using Raspberry Pi 3, DHT11 Sensor and AWS IoT

10 min readJun 23, 2021

Introduction

This is my side hustle project in IoT domain relevant for Industry 4.0. This project deals with Sensors (DHT11), Cloud (AWS), IoT edge compute device (Raspberry Pi 3) and CI-CD for end-to-end application delivery. As part of this project room ambiance (temperature and humidity) has been measured using DHT11 sensor attached to Raspberry Pi 3. The ambiance measures are published to AWS IoT from Pi and then simple event analysis is performed to create email notification when the ambiance conditions are out of line.

Usecase

It is always vital and critical to business and operations to keep track of the ambiance conditions (eg. Temperature and Humidity) in Factories, Retail stores, Storage Plants, Logistics, etc. The accurate ambiance conditions have to be measured continuously all the time and take necessary actions when the conditions are volatile and deviates from the pre-defined tolerance and range limits. Without IoT application, it always requires continuous manual inspection and businesses and operations many times miss the chance of taking appropriate measures and actions before the loss happens.

Predictive Maintenance and Pro-active Monitoring are the crucial and essential features that many businesses are looking for nowadays in order to reduce the wastage, identify the faults early, analyse the trends, reduce the time to fix the issues, remote maintenance, continuous automated monitoring, increase productivity and ultimately increase the revenue and operations margins.

Architecture

The following picture explains the overall end-to-end application architecture as developed in the prototype.

Components used in Prototype

Network/Connectivity

Edge: Home Ethernet LAN
Cloud to Edge: Open Internet (No VPN is used for prototype)

Field Edge

Raspberry Pi 3 Model B v1.2
Temperature and Humidity Sensor (DHT11)

Gateway Edge

None (No explicit Storage and No additional Compute is used at the edge for prototype)

Protocols

MQTT (for IoT messages on Port 8883)
TCP & HTTPS (for general systems communication)
REST/HTTPS (for Microservice API calls)
Email (for notification alert)
SSH (to access Raspberry Pi for administration and pipeline)
SCP (to transmit the device certificates to Raspeberry Pi from AWS EC2 Pipeline Agent server)

IoT Platform

AWS IoT Core

AWS Serverless

AWS SNS (For Notifications)
AWS CloudWatch (For Monitoring and Dashboards)

Security

Identity Credentials: GUID has been used as a unique identity for Raspberry Pi device. AWS IoT Core will be accessed using AWS provided account credentials. AWS is accessed using AWS SDKs.
Access Policies and Permissions: AWS best practices are used for policies and permissions to ensure appropriate authorization is provided for the resources created in AWS landscape.
Firewalls: Firewall is enabled in Raspberry Pi and only specific ports are enabled as required for MQTT, TCP, SSH and other ports as required for the applications. Port forwarding is enabled.
Device Certificates: X.509 Client Certificate issued by AWS IoT Core & Amazon Root CA Certificate (RSA 2048) are is used to authorize the Raspberry Pi3 device for sending messages over MQTT to AWS IoT Core. Device Certificates stored in device ephemeral storage and they have to be copied into Pod everytime it is created in Kubernetes cluster within Pi. (Certificates are transmitted to Pi over TLS connection using SCP protocol as part of CI-CD Pipeline from AWS EC2 Jenkins pipeline agent server)
Secure Keys: RSA 2048 asynchronous Key pair (public, private) is used for SSH communication with Raspberry Pi for administration access
Data Protection: Data is encrypted in transit over TLS connection between AWS Cloud and IoT Device (for both MQTT and TCP). Data stored within AWS IoT Core is encrypted at rest by default. Data stored elsewhere in other AWS components (eg. AWS SNS) is not encrypted at rest. No data is stored in edge device (Pi). No protected data is present in the application/system logs produced within Raspeberry Pi and as well as in AWS Cloudwatch.

End-to-end Application aspects covered in Prototype

Data Ingestion/Collection: Temperature and Humidity sensors are used to measure the ambiance (room) condition and they are attached to Raspberry Pi3. Sensors will continuously measure the parameters in intervals and the measured data will be collected by Pi.

Data Transmission: Data is transmitted directly from field edge device (i.e Raspberry Pi) and no intermediate Gateway Hub is used in the prototype. Ethernet LAN is used within edge scope whereas Home Broadband is used for sending the data to AWS IoT Core using Internet.

Data Processing: No major processing considered once the data is available in AWS IoT Core. Data is sent as JSON messages and no other transformation or enrichment is considered in scope of the prototype. Data is not stored anywhere else apart from AWS IoT Core.

Data Analysis: Simple analysis is considered where the temperature and humidity parameters are checked in real time within AWS IoT Core to identify any anomaly (values outside the prescribed limits).

Data Visualization: When there is any anomaly in the data, then a AWS SNS notification will be sent to Operator via email mechanism.

Operations Monitoring: If there are any errors in the process they will be visible in monitoring dashboards within AWS Cloudwatch. No monitoring aspect is considered within the Edge scope.

Application Delivery: Jenkins CI server is used to enable CI-CD pipeline for building, deploying and perform sanity test of the applications end-to-end

Raspberry Pi 3 & Sensor Integration

AWS IoT

AWS IoT Thing setup

Verification of messages published from Raspberry Pi 3 in AWS IoT Console with MQTT Test Client

IoT Rules & Actions

Publishing of specific events to AWS SNS and subscription of those events to email notification when certain rule condition matches

Publishing of specific events to AWS CloudWatch Event Logstream when certain rule condition matches

Publishing of Errors to AWS CloudWatch Error Logstream when there is error while sending to SNS or CloudWatch event stream

CI-CD Pipeline

Jenkins has been used as CI server here to perform build, configure, deploy and test end-to-end.

CI Server: Jenkins (Job Type: Pipeline)
Build & Integration Environment: AWS VPC + Public Subnet + EC2 (T2.Large: 4 GB RAM, default SSD volume of 8 GB, Ubuntu 20.04 LTS)
Target Deployment: Raspberry Pi 3 Model B v1.2 or AWS EC2 [ARM64, t4g.small: 2 GB RAM, default SSD volume of 8 GB, Ubuntu 20.04 LTS)

Pipeline Flow

Pipeline Execution

Complexity Factors

Things that are easy & Straightforward

AWS IoT Platform
AWS SNS and AWS CloudWatch
Raspberry Pi3 & Sensor Integration
Python Microservice development & AWS IoT SDK Usage
AWS VPC and other resource setup for e2e pipeline
Ubuntu OS and Bootstrap setup in Raspberry Pi 3 & AWS EC2

Things that are Not straightforward and Time consuming

Jenkins CI/CD Job without external plugins (type= Pipeline): SSH access to Edge devices & Docker BuildX for multi OS architectures (amd64 & arm64)
Making MicroK8s or K3S to work in constrained environments with memory = 1GB and CPU = 2 cores (I am not completely successful with this yet as they both are not working as they are claimed)
Securing Raspberry Pi 3 device (Password less authentication, RSA certificate SSH access & Firewalls)
Inbound access of Raspberry Pi 3 from AWS using Home network: Port forwarding and IPv6 setup wont work unless ISP allows it. It requires static IP and as well as dedicated network line like that of business lines.

Cost of Prototype

AWS Cost: ≤ $15 (98% of it is for EC2 usage over a spanned period of 7 days. AWS IoT, AWS SNS, AWS CloudWatch and all other AWS VPC resources are free of cost)
Raspberry Pi 3 + DHT11 Sensor: ≤ $40

Key Technical Aspects Unearthed

Default AWS IoT SDK contains python2 commands as per its latest version as on 22-Jun-2021. Hence start.sh script within the sdk code base has to be modified to use python3 and pip3 in respective commands.
When the device is registered using AWS IoT Console Onboard menu, it will create new certificates again and attach a default restricted policy to it eventhough an existing Thing (with certificate) is associated. Hence after the device onboarding steps are finished, detach the new certificate that would be created by AWS Console onboard process and retain the previous certificate and custom policy that would have been associated to it already. Or otherwise create a new policy with required permissions and attach it to the new certificate that is created by Onboard process and detach the old certificate that already exists.
By default the sample basicPubSub.py program in AWS SDK subscribes to topic “sdk/test/Python” for which the publish and subscribe operations are allowed by default from the IoT device as per the default policy attached to the respective thing certificate. If the test has to be done for a custom topic, then that topic have to be created within AWS IoT space and also make sure the policy attached to the thing’s certificate allows to publish and subscribe to that custom topic. The default policy attached to the certificate will not allow all actions on all resources
Ubuntu Core OS is light weight and created for IoT devices only. But the drawback is to use only snap and all apps have to be available in snap store which is a serious constraint and limitation. Hence I have chosen Ubuntu Server 20.04 LTS 64 bit ARM for my Raspberry Pi 3 eventhough it occupies more space but gives full control.
With Ubuntu OS (any flavor), HDMI display will not work after the micro SD is inserted with OS. Pi outputs a relatively weak HDMI signal. Some devices may not immediately notice the Pi’s HDMI or may not do the negotiation. Setting the hdmi_force_hotplug=1 within config.txt file in micro SD OS directory will makes sure the Pi believes the monitor/TV is really there and display will work.
It’s easy, flexible and performant to use SSH route to access Raspberry Pi 3 from local computer linux machine. That will avoid connecting HDMI screen, keyboard and mouse controls to Pi and hence reduces the load on Pi processing and at the same time it doesn’t need any GUI to be installed in Pi so that more than 1 GB storage in micro SD will be saved and CPU and memory resources will be saved
When accessing Pi using SSH, then it is absolutely recommended to use SSH certificate access rather than mere username-password access so that no one can hack the edge device if IP and password are obtained
When Publish and Subscribe have to happen at different times or different frequency separately using separate programs from the same client device on same topic, then different Client ID has to be used when connecting to AWS IoT MQTT broker for Publish and Subscribe. If the same Client ID is used then AWS IoT MQTT broker will misbehave and terminate the two sessions in case disconnect event is sent from Publish program
General Docker Build images will not work for Raspberry Pi 3. General Docker Build images will be created for amd64 (in case of linux environments) whereas Raspeberry Pi 3 will be arm64 architecture. Hence Docker BuildX experimental feature has been used in this case to generate the Docker image for ARM64 architecture as suitable for Raspberry Pi 3.
Microk8s frequently hangs up in Raspberry Pi 3 due to low memory constraint (1 GB). [I have replicated the issue with AWS EC2 (arm64, Ubuntu 20.04, t2g.micro with 1 GB and 2 CPU) as well] I have tried K3S as well in Raspberry Pi 3 and K3S is worse than MicroK8s when there is less memory on edge devices. All CPU resource and memory will be consumed when K3S is installed in Raspberry Pi 3 and whole Pi will be hanged up completely. With MicroK8s, Pi will not hang up but it will run very slow. Microk8s has a flexible feature where the required kubernetes components can be chosen as needed for the given usecase. Effectively, both MicroK8S and K3S will not work as needed in constrained edge devices as they are claimed by respective companies.
In case of Home network, Inbound external traffic will not be allowed by most ISPs. I have tried with IPv6 by enabling port forwarding but my ISP (Jio Fiber) doesn’t allow that either. So, if Raspberry Pi has to be accessed from external network (for eg. CI-CD flow), then necessary arrangement (eg. static IP, port forwarding, etc.) has to be done to get external inbound traffic.