Face mask detection system using Deep learning

Motive

 

As COVID-19 virus are spread widly, nothing's important than wearing mask for prevection.

So I made a device to detect whether mask is on to place it at an entrance

 

 

 

Architecture

 

• ESP32 and PC are connected as server-client.
• based on ESP32 camera web server example.
• starts server and manipulated on PC by http request
• ESP32 communicates with Arduino by pin output.
• Added arduino nano due to lack of GPIO from ESP32.

 

 

Components

 

 

• ESP32 cam
• Arduino nano
• PIR sensor
• piezo buzzer
• RGB LED
• Resistor
• etc(breadboard, battery)

 

for arduino board, uno or mega are good to use but it has not much thing to do and not much pins are needed so I chose arduino nano

 

resistor is 33ohm. it's used for LED.

 

Other than that breadboard, jumper cable, battery, zip ties are used. There's no need for soldering.

 

Ovarall process are:

 

1. create training model
2. confirm mask detection on tensorflow
3. connect ESP-32 cam
4. connect to arduino nano
5. set up hardware

I had two ways to create training model.

 

Create Model

https://github.com/ghchoi0427/FacemaskDetection/blob/master/createModel.py

 

 

tensorflow, openCV libraries and keras_processing.image API are used for this code.

noise is added for image augmentation

set directories for training data, validation data.

batch size means number of pictures that can be loaded at a time.

this sets augmentation for images. it reinforces model by distorting images intentionally. no augmentation used for validation data.

MobileNetV2 is light weight train model that can be implemented on mobile environments.

Adding layers

save model

 

 

Test trained model on PyCharm

- import openCV library

- receive webcam video from PyCharm and detect with trained model

- put model to the designated directory

 

After this, Every process from PC side including http request will be carried out on PyCharm

 

testing trained model. it works.

 

Receive ESP32 streaming video through pycharm.

 

As we seen eariler openCV used embedded webcam for video information.

 

But as my goal in this project is to get video through ESP32 and implement alarm system, I'm getting video from ESP32 to Pycharm.

 

 

 

video from ESP32 is encodded and transmitted as MJPEG type

 

 

Pycharm decoded those bytes and shows it as image(video)

 

Now, I can get video through pycharm.

 

next, let us put flash light here

 

Miscellaneous - flashlight

 

When you look through camera_index.h file which is one of ESP32 examples, you can find hexadecimal array from source file.

 

that's a compressed html due to memory problems. You can find it after decoding it.

 

pin 4 is in charge of flashlight of ESP32.

 

Add a function which put pin4 on HIGH when "flash" switch is turned on and

add a button on html source as well.

 

Compress the modified html source. copy paste it in where it was and change the length.

 

On the upper right picture you'll see a switch named "Flash" is added.

 

 

Flash works good. On the right picture Flash lamp is it. 

 

 

add alert_handler,normal_handler,blank_handler

 

Each handlers responds to person without of with inappropriate use of mask/person with mask on/no one

 

vary each pin output from GPIO12, GPIO13

 

alert_handler : 12 HIGH 13 LOW

normal_handler : 12 LOW 13 HIGH

blank_handler : 12 LOW 13 LOW

 

 

On port 80 you'll find control panel.

Control panel is valid while video is broadcasted. 

Also the custom function sends request to port 80.

Start stream button is not used here.

 

port 81 is a port only for video streaming.

In order to analyze an image in Pycham tensorflow, there must be nothing on the screen other than a streaming image.

 

Wiring

 

When finding no mask GPIO 12, with mask GPIO 13 turn on HIGH. when no one is detected both of them are LOW.

 

Arduino nano gets digital input from 12, 13 to 6, 5 pin

 

 

Arduino programming

 

arduino nano's codes are simple.

 

if sensor value of PIR is 1 and

pin 5 is HIGH siren goes off and turns on red LED

pin 6 is HIGH siren goes out and turn on green LED

 

both of them are off when no one is detected

 

Testig by sending http request.

 

I used postman here but any kind of browsers are allowed.

 

As the response from server is set to null, I can get nothing but blank.

 

 

On Pycharm I used a library called "requests" to send http request.

 

URL is determined according to whether the mask is on or not.

 

To keep it from sending http request too rapidly, the requests are sent only when the determination has changed.

 

Test HTTP Request

 

Each lights are on according to requests

 

 

Hardware set up

 

The battery was connected to the Arduino Nano, and the breadboard and battery were tied and mounted on the cradle.

 

It works wherever there is wifi.

 

Final test

Test is carried out after installation.

 

red light is on, siren goes off when mask isn't on

 

green light is on, siren goes out when mask is on.

 

Sometimes even when there are no people but still determines as no-mask, the alarm does not go off because the PIR sensor does not respond.

 

 

reference

“Adding features to Examples->ESP32->Camera->CameraWebServer“, https://www.esp32.com/viewtopic.php?t=11190

“Editing Camera Web Server HTML Source Code for the ESP32-CAM”, https://youtu.be/bIJoVyjTf7g

“ESP32-CAM Python stream” https://gist.github.com/youjunjer/79e5dad5f47ee5757fcb9d401a95e76b

“CyberChef”, http://icyberchef.com/

“deepLearning start guide”, https://github.com/JeiKeiLim/dl_start_guide/tree/master/02_TensorFlow/06_application/01_mask_nomask

 

+Source code: Github