Bernd's development stuff

I was looking into a solution to send push notifications from an ESP8266 module to an Android application. I know that this is possible with the Google Cloud Messaging service.

But what I found on the internet is using an external webserver, able to run PHP (or Python) scripts and using a MySQL server to store registration IDs. So the information flow looks like:

The developer opens a project in the Google Developer Console, gets an API key and uses this API key in the Android application to register an Android phone or tablet (or watch or TV) on the Google GCM server.

In case you were not already aware, an API (application programming interface) key is a unique identifier used to authenticate a user, developer, or calling program to an API. They are typically used to authenticate a project with the API rather than a human user and different platforms (like those you can click here to learn more about) may implement and use API keys in different ways. With any and all APIs, constant testing is highly important to make sure they are doing their jobs properly and not slowing down a website/webpage. To help achieve this, a synthetic monitoring test can be run to check performance and how it affects the customers.

Moreover, APIs facilitate interactions between different software programs and therefore it is crucial that regular software testing takes place. You can learn more about the importance of conducting an SOAtest and other common types of API software testing here: https://www.parasoft.com/products/parasoft-soatest/.

That said, in this case, the GCM server sends back a registration ID to the Android device. The Android app then needs to send this registration ID to (your own) external webserver where it is stored and used to send push messages to the Android device.

The device that wants to send a message to the registered Android devices then uses a PHP script to send data to the GCM server, which then creates and sends a push message to the Android devices.

So far so good, but why do I need to setup an external server with MySQL and PHP just to send messages to the Android devices. The ESP8266 has server capabilities build-in and has enough storage to save registration IDs in the Flash memory.

The solution would look like:

The registration IDs would be stored in the ESP8266 and the ESP8266 would contact the GCM server directly to request push messages to the registered Android devices.

As I could not find any tutorial or example on the Internet how to achieve this, I started to write the code by myself.

The result is not (yet) a library that you can just link to your ESP8266 code, it is more a collection of functions that you can copy into your code to use the ESP8266 to send push messages via GCM.

Requirements:

ESP8266

This code is written for the Arduino IDE, it is not usable for LUA or other coding solutions for the ESP8266 modules. But of course it can be used as a base to write something similar.

This code requires an additional library to handle JSON objects. I used ArduinoJSON, a nicely written and well working library to deal with JSON objects, JSON arrays, encoding and decoding. All credits for the ArduinoJson library go to Benoit Blanchon. You can get it from:
Blog on Good Code Smell – Github sources
Install the library as explained in the ArduinoJson Wiki: Using the library with Arduino

I used an Adafruit HUZZAH ESP8266 module to test this code. This breakout is a little bit more expensive than other available ESP8266 breakouts, but I like it because I can power it direct with 5V and it has a 5V compatible serial connection. But the functions will work on any ESP8266 module as long as you program it with the Arduino IDE.

For testing I connected the module to the local WiFi and my Android phone and tablet were connected to the same WiFi while doing the registration process. As my local internet provider I doesn’t allow to forward port 80 to a server on my local lan, I could not test it with an public domain name. But it should be working the same.

Android

Android Studio (or Eclipse) is required to build the test application.

At least one Android phone or tablet to test the functions.

The small application I used to test the functions is a slightly changed version of the application provided by Android Push Notification using Google Cloud Messaging (GCM) – Part 2

Limitations of this article

I will not explain

• how to setup the IDEs for Arduino, ESP8266 or Android.
• how to install the ESP8266 board in the Arduino IDE
• how to program the ESP8266
• how to add libraries to the Arduino IDE or Android studio
• how to get the necessary API needed for GCM.

There are a lot of blogs/articles/tutorials and examples on the Internet about these basic requirements.

But I strongly recommend to read the articles Android Push Notification using Google Cloud Messaging (GCM) – Part 1 and Part 2. They are a very good explanation and tutorial how GCM works. Even in this article a external web server with PHP scripts is used, the explanations about GCM, how to get a API key and how to setup the Android test app is very easy to understand and helpful.

Function references

Beside of the webserver, only 5 functions are needed for the GCM functionality and handling of the registration ids:

boolean gcmSendMsg()
Description

Signatures

Arguments

Used global variables

Return value

Example

boolean writeRegIds()
Description

Arguments

Used global variables

Return value

boolean getRegisteredDevices()
Description

Signatures

Arguments

Used global variables

Return value

Example

boolean addRegisteredDevice()
Description

Signatures

Arguments

Used global variables

Return value

Example

boolean delRegisteredDevice()
Description

Signatures

Arguments

Used global variables

Return value

Examples

boolean gcmSendOut()
Description

Signatures

Arguments

Used global variables

Return value

Examples

ESP8266 code

I splitted the ESP8266 code into 4 files

• wifiAPinfo.h -> contains the name and login credentials for the local WiFi (not shared for privacy reasons)
• gcmInfo.h -> contains the API key for the GCM (not shared for privacy reasons)
• gcm_esp.ino -> functions for GCM and handling the registration IDs
• gcm_esp_test.ino -> web server function, setup and main loop

wifiAPinfo.h

This file contains the name and login credentials for my local WiFi and of course I am not sharing this information here. You have to create this file and enter your own WiFi informations:

gcmInfo.h

This file contains the API key given by Google Developer Console. Obviously I am not sharing this information here. You have to create this file and enter your own API key:

gcm_esp.ino

This file contains the necessary includes, global variables and functions for the GCM functions. THIS IS THE PART YOU WANT TO INCLUDE INTO YOUR OWN CODE TO USE THE GCM FUNCTIONALITY!

Includes:

Defines:

For the testing I limited the maximum number of Android devices that can be served to 5. Of course this value can be changed to your personal needs.

Global variables:

serverName[] and hostName only differ in the additional “http://”. But saving it as a char array and as a string makes it easier to handle in the function gcmSendMsg().

The array regAndroidIds holds the Android registration IDs. Here the IDs are stored as Strings for easy use in the function gcmSendMsg(). As the registration IDs have a fixed length of 140 characters this could be as well defined as an array of character arrays.

The byte regDevNum holds the number of currently registered devices.

The functions:

boolean gcmSendMsg()

This function can be found two times, but with different arguments.

gcmSendMsg (JsonArray& pushMessageIds, JsonArray& pushMessages) Message keys and message values are received as 2 Json arrays.

gcmSendMsg (JsonObject& pushMessages) Message keys and messages are already combined in an Json object

gcmSendMsg can be used with separate arguments for the keys and the messages itself. First one is a Json array with the keys (aka names of the messages), second one is a Json array with the messages itself. I chose to use Json arrays here to make it possible to have mixed type messages. e.g. some messages can be string, others be integer or float. Possible message formats:

The following example would send a push notification with three messages the “power” (as a float), “light” (as an integer) and “status” (as a string) to the registered Android devices:

gcmSendMsg will send the push notification to all registered devices!

gcmSendMsg can be used as well with only one argument which must be a correct formatted Json object which is directly used to build up the request to the GCM server.

The first signature of the function:

First step in the function is to update the list with the registered devices and do some basic checks.

Next we build up the Json object that needs to be sent to the GCM server. Inside the Json object is one Json array which holds the IDs we want the message to be sent to, the message itself is hold within another Json object that can contain The format is:

The two key “registration_ids” and “data” are mandatory, the ids must be valid 140 character strings, the key-value pairs must be string:value, where value can be any type as listed above. It is task of the Android app that receives the push notification to parse this key-value pairs.

After the Json object with the registration IDs and the data is built successfully gcmSendOut is called to connect to the GCM server and post the push notification request:

The second signature of the function:

First step in the function is to update the list with the registered devices and do some basic checks.

Next we build up the Json object that needs to be sent to the GCM server. Inside the Json object is one Json array which holds the IDs we want the message to be sent to, the message itself is hold within another Json object that can contain The format is:

The two key “registration_ids” and “data” are mandatory, the ids must be valid 140 character strings, the key-value pairs must be string:value, where value can be any type as listed above. It is task of the Android app that receives the push notification to parse this key-value pairs.

After the Json object with the registration IDs and the data is built successfully gcmSendOut is called to connect to the GCM server and post the push notification request:

boolean writeRegIds()

This function is writing the content of the array with the registration IDs to the file gcm.txt in the file system of the ESP8266. The registration IDs are written as a Json formatted string to the text file.

First we try to open the file for writing

Then we prepare the Json object for the registration IDs and add the content of the regAndroidIds array to the Json object

Before writing to the file we convert the Json object into a string

boolean getRegisteredDevices()

This function reads the IDs from the file gcm.txt and stores them into the global variables regAndroidIds[]. The number of IDs is stored in regDevNum.

First step is to open the file for reading

If successful we read the content into a string and parse it to a Json object

If the parsing was successful the elements of the Json object are stored in the variable regAndroidIds[]

And some error handling at the end of the function

boolean addRegisteredDevice()

This function adds a new ID to the array regAndroidIds and then calls writeRegIds to save them in the file system. In the example I limited the number of devices to 5, but with 1Mb storage in the file system a much higher number of registered devices could be handled.

First three checks are done

• check if the ID has the correct length
• available space to store the ID
• ID is not yet registered

If all checks are ok we add the new ID to the array regAndroidIds and call writeRegIds to update the stored file

boolean delRegisteredDevice()

This function can be found three times, but with different arguments.

delRegisteredDevice() deletes all stored registration IDs

delRegisterDevice(String delRegId) deletes only the ID given in the string delRegId

delRegisterDevice(int delRegIndex) deletes only the ID at the position (index) delRegIndex

boolean delRegisteredDevice()

First refresh the list of registered IDs

Then we delete the file gcm.txt and set the number of registered devices to 0

boolean delRegisteredDevice(String delRegId)

First refresh the list of registered IDs

Then we are searching the ID we want to delete in the regAndroidIds array

If the ID was in the array we delete and fill the gap by shifting the other IDs towards the top of the array. Then we store the new list in the file system

And some error handling at the end

boolean delRegisteredDevice(int delRegIndex)

First refresh the list of registered IDs and make a

Then we check if the index is within the limits, delete the entry at the index delRegIndex and fill the gap by shifting the other IDs towards the top of the array. Then we store the new list in the file system

And some error handling at the end

boolean gcmSendOut(String data)

gcmSendOut is used by gcmSendMsg to send the prepared data to the GCM server. gcmSendOut is used internally only,

First we stop eventually still running clients

Second we connect to the GCM server and send the POST header and body

Then we wait for the response from the GCM server to check if the request was accepted:

The result is somewhere hidden in the response. Best thing is to first check if the response contains “200 OK”, because that means that the GCM server accepted our request. The detailed response from the GCM server is a Json object which looks like

Important here are the keys “success” and “failure”. They tell us to how many Android devices the message was successfully transferred (or not!).

For now I do not parse this result in the code but I return it to the calling code as “failReason”.

Some final householding:

gsm_esp_test.ino

This code is only an example for the implementation of the GCM functions into your own code.

Two parts are required to get the GCM functions to work:

• within setup() you mount the file system by adding

you need to start the server to handle incoming registration messages

Therefor I will only go into details of the webServer function which will handle the registration, listing and deletion of IDs.

void webServer()

Description

Arguments

Used global variables

Return value

Example

URL arguments

Commands are send together with the URL. Lets assume your ESP8266 is connected with the IP address 192.168.0.148. Then the URL together with a command would look like http://192.168.0.148/?l for the command to list all registered device IDs. Supported commands are:

• Register a new id
  • /?regid= with length 140 letter, no leading/trailing <">
• List registered ids
  • /?l
• Delete all registered ids
  • /?da
• Delete specific registered id by 140 letter registration id
  • /?di= with length 140 characters, no leading/trailing <">
• Delete specific registered index
  • /?dx=
• Send broadcast message to all registered devices
  • /?s={“msgkeys”:[“key1″,”key2″],”msgs”:[“msg1″,”msg2”]}

URL examples

In the example code I implemented a simple webserver that handles all 6 commands, here are some comments about the code:

The declaration:

Preparation of JSON object that will be used to send a response to the client. Here we wait 3 seconds to get the request body from the client. If there is no request body sent we respond with a timeout failure and close the connection

Read the request body from the client, which should have the command and parameters.

We reduce the received data to the command and parameters only by finding the first occurence of “/” which is the start of the command and deleting the last 9 characters.

Check if the command is “Register new device”. If yes, then we isolate the Android registration ID from the string and save it in regID

If the length of the Android registration ID is correct we add it to the list of registered devices. If addRegisterDevice() was successfull we return “success” together with an updated list of registered devices. If addRegisterDevice() returns with a failure, the failure description is hold in the variable failReason. This could be one of the following reasons:

• Maximum number of devices already registered
• Device ID was already registered
• File IO error (which would be a fatal error and the sign of a hardware problem with your ESP module)

Next we check if the command is “List registered devices” and return the list of registered devices or an error message. An error could be:

• No devices registered yet
• File IO error (which would be a fatal error and the sign of a hardware problem with your ESP module)

Next we check if the command is “Delete one or all registered devices” and which type of delete request it is. The first letter after the command “/?d” is the type of delete command

• a – delete all registered IDs
• i – delete specific ID by given ID
• x – delete ID at specific index

If the request is “delete all”, we simple call delRegisteredDevice() with the first parameter set to true.

If the request is “delete specific ID”, we extract the ID from the command line and then call delRegisteredDevice() with the second parameter containing the ID we want to delete

If the request is “delete ID at index”, we extract the index from the command line, check if it is valid and then call delRegisterDevice() with the third parameter containing the index we want to delete

And at last we add the updated list of devices to the response

The last part of the webServer function handles requests to send push messages to the registered devices. I chose a Json formatted string for the key names and messages. Key names must be always strings, but the messages can be all kind of variables. You can use Boolean, Integer, Float, Char, String, … for a message.

Extract the JSON formatted string from the request, reformat if necessary and parse the string into a Json object

Check if the parsing was successful and if the Json object has the required keys “msgkeys” and “msgs”.

If all is ok create the Json arrays with the key names and the message values, then call gcmSendMsg to send the push request to the GCM server.

Finally some error handling:

If we couldn’t find any valid command we respond with an error

Last step is to send the response back to the connected client and close the connection.

The response is again a Json formatted string. It always has the key “result”, which tells whether the command was successful or failed. If there was a problem the key “reason” will give more information about the failure.

The Android application to test the GCM notification system.

For testing purposes I slightly changed the code from Android Push Notification using Google Cloud Messaging (GCM) – Part 2, an excellent tutorial how to setup the GCM notification system with an external web server.

The changed code is available on my Github repository.

The Android code is split into 3 classes

• MainActivity – handles the UI and buttons
• GCMBroadCastReceiver – receives the push notifications from the GCM server
• GCMIntentService – handles the received push notifications and displays them as notifications in the top bar of the device

The changes I did to the original code are

• in MainActivity add more buttons to test the unregister function and to list all registered devices
• in MainActivity added handling of the response from the ESP8266 to display error messages
• in GCMIntentService changed the way the received push notifications are shown
• Broadcast sender and receiver to display GCM push notifications on the main UI

When the application is freshly installed and started it shows only a button to register to the GCM service:

When the button is pressed, the device registers itself at the GCM service and sends his registration ID to the ESP8266. There it is stored.

From this moment on the Android device will receive all push notifications that are initiated by the ESP8266 and displays them.

If the button “GET LIST OF REGISTERED DEVICES” is pressed the list is sent from the ESP8266:

Android code

I will not go into every detail of the Android code and focus only on a few code parts that are important to the functionality.

AndroidManifest.xml

It is important to add here the permissions that the application needs to connect to the Internet and receive the GCM messages:

The receiver class for the GCM push notifications needs to be registered here as well:

MainActivity.java

In onCreate() we ask for permission to access the internet and check if the device is already registered to the GCM service:

In onResume() we register to the broadcast message from the GCM push receiver :

In onPause() we unregister from the broadcast messages:

In onClick I added the handling of the different buttons. The communication itself is handled in an AsyncTask in the background by calling

This example is to delete the registration ID from the ESP8266. The command is given to the AsyncTask as an string argument. For listing the registered devices the command looks like

The webServerTask contacts the ESP8266 (or the GCM server for the initial registration) in the background without blocking the UI. As the AsyncTask cannot update the UI directly it calls activityUpdate which will run on the UI thread and can do the updates.

In activityUpdate the response from the ESP8266 (which is a JSON object) is analyzed and the results are shown on the UI below the buttons.

activityReceiver is a broadcast receiver which is triggered by GCMIntentService if a GCM push notification has arrived.

Similar to the webServerTask the broadcast receiver cannot update the UI directly. To do this the updateUIHandler is used to call updateUI.

GCMBroadcastReceiver.java

This broadcast receiver is called by the Android OS when a GCM push notification arrives. It activates the GCMIntentService to handle the incoming notification. I used here the original code from Android Push Notification using Google Cloud Messaging (GCM) – Part 2 without any changes.

GCMIntentService.java

This intent service handles the incoming push notifications. I added to the original code a broadcast sender to inform the MainActivity.

That’s it. Hope this small code example helps you to get the Google Cloud Messaging service to work directly with your ESP8266 module without the need of an external webserver, PHP and MySQL databases.

Just as a reminder, the complete code is available on my Github repository.