Introduction
The ESP82661 is a cheap hardware platform for WiFi-enabled devices: you can put something on the internet for well under a fiver2.
However, we need to consider the software too. The default ESP8266 firmware makes the device into something like a WiFi modem: it connects to the main processor over a serial link, and accepts AT commands3. This isn’t ideal, because you’ll need a second processor in the box to handle the application code, even though in many cases the ESP8266 can do it all.
Mongoose OS4 is one alternative. It provides a replacement firmware for the ESP8266 which includes a psuedo-Javascript interpreter and webserver. So, a typical ESP8266 Mongoose OS project contains:
- The OS, by which I mean both the traditional OS which handles networking and the like, plus extentions to the Javascript engine to handle hardware. Like other operating systems, it includes a number of daemons, some open to the network.
- ‘User-space’ files which include can include Javascript files containing ‘application logic’.
- C code compiled at build time, and linked into the firmware.
Mongoose claim support for other processors too: more of this anon.
Installation
The Mongoose website includes installation instructions5, which in the modern style amount to piping the output from curl into bash! There’s also some more technical documentation6.
I’m wary of such a plan so I looked at the script. As of today (July 2017), on the Mac, the main action is to download a single executable, mos
, from,
https://mongoose-os.com/downloads/mos/mac/mos
The script then checks for libusb
and libftdi
, which I had already installed via Homebrew7.
Once installed you can run mos
to get a list of options:
$ ./mos --help
The Mongoose OS command line tool, v. 20170706-152142/master@871c1644.
Checking updates... Up to date.
Usage:
./mos <command>
Commands:
ui Start GUI
init Initialise firmware directory structure in the current directory
build Build a firmware from the sources located in the current directory
flash Flash firmware to the device
flash-read Read a region of flash
console Simple serial port console
ls List files at the local device's filesystem
get Read file from the local device's filesystem and print to stdout
put Put file from the host machine to the local device's filesystem
rm Delete a file from the device's filesystem
config-get Get config value from the locally attached device
config-set Set config value at the locally attached device
call Perform a device API call. "mos call RPC.List" shows available methods
aws-iot-setup Provision the device for AWS IoT cloud
update Self-update mos tool
wifi Setup WiFi - shortcut to config-set wifi...
Global Flags:
--verbose Verbose output. Optional, default value: "false"
--logtostderr log to standard error instead of files. Optional, default value: "false"
You’ll see that the program is very keen to check we’re running the most recent version. Again following the modern fashion, new versions are published frequently.
The code for mos
lives on GitHub8. It’s written in Go, so you’ll need to install a Go toolchain to compile it.
Blinky
If you attach a suitable ESP8266 board to a USB port e.g. the NodeMCU devkit9, then making a LED blink is simply a case of:
abc(small).. $ ./mos flash esp8266
Fetching https://mongoose-os.com/downloads/esp8266.zip...
Loaded default/esp8266 version 1.0 (20170706-161740/???)
Using port /dev/cu.SLAB_USBtoUART
Opening /dev/cu.SLAB_USBtoUART...
Connecting to ESP8266 ROM, attempt 1 of 10...
Connected
Running flasher @ 460800...
Flasher is running
Flash size: 4194304, params: 0x0240 (dio,32m,40m)
Deduping...
128 @ 0x3fc000 -> 0
Writing...
4096 @ 0x0
4096 @ 0x7000
262144 @ 0x8000
675840 @ 0x100000
4096 @ 0x3fb000
Wrote 950272 bytes in 20.89 seconds (355.47 KBit/sec)
Verifying...
2592 @ 0x0
4096 @ 0x7000
262144 @ 0x8000
673824 @ 0x100000
4096 @ 0x3fb000
128 @ 0x3fc000
Booting firmware...
All done!
The code above downloads a blob containing the OS and a file-system from https://mongoose-os.com/downloads/esp8266.zip, then flashes it to the board. At this point a LED should start flashing.
Some gotchas:
- If you have a directory called esp8266,
mos
will try to find a firmware blob locally instead. - If you have multiple plausible serial devices, you might need to tell
mos
which one to use with the --port option. - If your LED is on a different GPIO, you’ll need to edit the code.
WiFi
At this point, you might wish to configure the WiFi connection:
$ ./mos wifi SSID PASSWORD
In practice, it is often easier to do this from the browser-based IDE.
Mac Flashing
On my Mac I found it impossible to flash boards which use the CH340 USB-serial bridge. Once flashed, everything worked well though. Rather than explore this particular rabbit hole, I flashed these boards from a Linux box instead.
The integrated IDE
Mongoose contains a full web-based UI, which allows you to all the things you can do from the command line and more. For example, you can flash the ESP8266 or configure the WiFi with the IDE instead of at the command line. To invoke the IDE:
$ ./mos
At this point your web-browser should open something like http://127.0.0.1:1992/#files, which of course is a web server embedded into mos
.
You should see a pretty UI which lets you explore the device. For example, you can browse files on the ESP8266 by clicking on the ‘Device Files’ link on the left-hand-side.
init.js is a key file: it’s essentially what gets run at boot, and so by looking at it, we can tell what the device is going to do. You can either use the in-browser file manager, or the command line:
$ ./mos get init.js
load('api_config.js');
load('api_gpio.js');
load('api_mqtt.js');
load('api_sys.js');
load('api_timer.js');
// Helper C function get_led_gpio_pin() in src/main.c returns built-in LED GPIO
let led = ffi('int get_led_gpio_pin()')();
let getInfo = function() {
return JSON.stringify({total_ram: Sys.total_ram(), free_ram: Sys.free_ram()});
};
// Blink built-in LED every second
GPIO.set_mode(led, GPIO.MODE_OUTPUT);
Timer.set(1000 /* 1 sec */, true /* repeat */, function() {
let value = GPIO.toggle(led);
print(value ? 'Tick' : 'Tock', 'uptime:', Sys.uptime(), getInfo());
}, null);
// Publish to MQTT topic on a button press. Button is wired to GPIO pin 0
GPIO.set_button_handler(0, GPIO.PULL_UP, GPIO.INT_EDGE_NEG, 200, function() {
let topic = '/devices/' + Cfg.get('device.id') + '/events';
let message = getInfo();
let ok = MQTT.pub(topic, message, 1);
print('Published:', ok ? 'yes' : 'no', 'topic:', topic, 'message:', message);
}, null);
So we can see that besides flashing a LED, the NodeMCU has also been configured to make a MQTT10 request when a push-button is pressed.
The MQTT server is configured in the confN.json
files: these form a crude overlay database when things defined in e.g. conf9.json
override things in conf0.json
.
You can also configure MQTT from the ‘Device Config’ section of the UI, or by using mos config-set
.
The filesystem
Having got to this stage, it is easy to edit files on the ESP8266. You can either get and put files with the mos
tool, or just edit them live in the UI. Either way, you’ll probably have to reboot the ESP8266 for the changes to take effect.
For example, having flashed the default firmware, you can change the LED’s period by just editing the 1000ms interval between timer events. You can then save this new init.js, reboot, and see the change. There is no need to reflash the ESP8266 from scratch. Instead when you save the file, you make an RPC call to the Mongoose OS on the ESP8266, which puts the data into the filesystem.
So, one approach to building a new Mongoose OS system from scratch is to:
- Flash some generic firmware.
- Upload whichever files you want on the device.
- Set config variables as needed: it seems better to use
mos
for this rather than writing aconfX.json
file directly.
For example, once flashed, I run the following script:
#! /bin/sh
for f in fs/*
do
./mos put $f
done
./mos config-set \
wifi.sta.enable=true \
wifi.ap.enable=false \
wifi.sta.ssid='XXXX' wifi.sta.pass=XXXX \
aws.shadow.thing_name='XXXX' \
mqtt.enable=true \
mqtt.server=xxx.iot.us-east-1.amazonaws.com:8883 \
mqtt.ssl_cert=cert.pem \
mqtt.ssl_key=private.pem \
mqtt.ssl_ca_cert=ca_cert.pem
It will probably become obvious quite quickly that Mongoose has a flat filesystem i.e. there are no directories.
Network access
It’s probably worth emphasizing that once you’ve flashed the basic firmware, all the subsequent interactions need only to exchange blobs of data. So, although it might be convenient to keep using the USB-serial bridge, once the ESP8266 is on the network, you can do most of this remotely too.
You can see some examples of this in the RPC documentation11 on the Mongoose website.
This flexibility obviously has some security implications: if you do something over the network without any kind of access control, so can anyone else!
Building an application
If you step back a bit, it is clear that all this configuration malarky just changes bits in the filesystem, so we could avoid doing the configuration step if we flashed a correctly configured image.
Mongoose makes this easy, and probably even encourages such a development model. In Mongoose jargon, a customized blob like this is called an ‘app’, and the process for building them is well documented12.
Abstracting away from details, executing mos build
, takes the app recipe specified in mos.yml
, and turns it into a blob suitable for flashing to the ESP8266.
The toochain for building this is supplied as Docker container. You can either run it in the cloud or locally.
There are advantages to this approach over flashing random firmware then tweaking it:
- Conceptually it is nice to have a single blob containing all the code and data.
- You can configure the kernel as you wish, in particular disabling daemons you don’t want to both reduce the attack surface and save disk space.
- Given that you have to flash the device anyway, it is quicker to flash it with the correct data than to flash a generic image and then update it.
However, there is a penalty to pay too. It takes a while to build the image, and you have to flash the whole device. So, if you’re just changing a few things in e.g. init.js, it is much faster to edit the files on the ESP8266.
As mentioned above, flashing CH340 boards didn’t work from my Mac which made the process even slower for me. YMMV13!
Javascript
Although Mongoose talks about writing code in JavaScript, this isn’t quite true. You actually write code in mJS14, a limited subset of JavaScript.
I think it would be better for all concerned if the Mongoose documentation made this clearer. I wasted ages trying to get some code to work, only to discover that mJS does not support closures. The mJS GitHub repo does include a list of restrictions15, but I didn’t know that at the time.
The lack of closures mean that many callbacks take a void *
pointer to userdata16.
AWS IoT
One of Mongoose’s headline features is support17 for Amazon Web Services IoT18. You can see examples both on the Mongoose19, and AWS20 websites.
As AWS IoT supports MQTT, the marginal work to get this working is to create the relevant certificates for access control, and configure objects on AWS.
mos
provides a helpful aws-iot-setup
command which makes a series of AWS calls on your behalf.
I was slightly wary of this, and so did things manually. To do the job properly you could study the mos source21, but I just did this:
$ aws --region us-east-1 iot create-keys-and-certificate \
--set-as-active \
--certificate-pem-outfile=fs/cert.pem \
--public-key-outfile=fs/public.pem \
--private-key-outfile=fs/private.pem
and then used to AWS console to connect a suitable policy and thing to these certificates.
You’ll need to configure the ESP8266 too:
$ mos config-set \
wifi.sta.enable=true \
wifi.ap.enable=false \
wifi.sta.ssid='XXXXX' wifi.sta.pass=XXXX \
aws.shadow.thing_name='xxxxxxxx \
mqtt.enable=true \
mqtt.server=a2uxxxxxxxxxxxx.iot.us-east-1.amazonaws.com:8883 \
mqtt.ssl_cert=cert.pem \
mqtt.ssl_key=private.pem \
mqtt.ssl_ca_cert=ca_cert.pem
I think the main difference from calling mos
is that I made RSA certificates, but it might be better to use ECDSA instead. As this forum post22 explains, ECDSA will be a lot faster if you have a ATECC508A crypto-chip. On the other hand, I don’t have such a chip!
It is worth pointing out that connecting to AWS IoT does take ages: about half-a-minute in my experience. This isn’t Amazon’s fault: the ESP8266 is slow23.
Once the AWS IoT stuff is working, you can use it to manage the device remotely24 as well.
Whilst clever and potentially interesting, I note that connecting a Mongoose device to AWS IoT does not mean you’re opening a connection purely for data.
STM32
Another headline Mongoose feature is support for other chips. Although I’ve mentioned ESP8266 a lot above, I’d hoped I could substitute ESP32 and STM32 without a problem.
However, whilst Mongoose stand by their ESP32 support, the STM32 stuff is presently ‘really flaky’25.
Conclusions
Overall I’ve been both delighted and disappointed by Mongoose OS.
On the plus side, it makes it very easy to build a certain class of projects based around the ESP8266. You can get source code from GitHub26, you don’t have to go far to get reasonable documentation27, and the forums28 are great.
On the other hand I think my expectations were set rather too high:
- You can’t program in JavaScript but instead a subset of it.
- The STM32 support doesn’t work.
Also I think the ESP8266 just isn’t fast enough to let you write even slow device handlers in mJS. For example, I wanted to connect a rotary encoder, but it was easy to turn it too fast to track.
In practice I think anything which goes up to the JavaScript layer takes a randomish time of about a few ms. For example, I hooked a scope to the blinky example and found that although the average flashing period was pretty accurate the standard deviation was about 1ms, and after a thousand iterations I had about 4ms spread between the minimum and maxiumum periods. I think that means that if you want to handle frequencies higher than a few Hertz, you’ll need to use C.
Writing in C isn’t a problem, but I’m not sure I want to write lots of it against the Mongoose APIs or in the Mongoose environment. I think I’d prefer to work in a more traditional setting, but that might just be ignorance.
There’s also a subjective, aesthetic, issue. I feel there’s quite a lot of magic built into mos
so you either have to take things on trust, or spend time and thought working around mos
.
Take the AWS credential issues: I wasn’t particularly keen to divulge my AWS secrets to Mongoose, nor am I particularly keen to outsource certificate creation to Mongoose. I would have been much happier if mos
had generated a script which I could eyeball before executing.
I think this is a particularly serious issue with anything security related, so I see a dark side to many of the clever remote management features.
My tentative conclusion is that Mongoose is a great way to build things which:
- are based on the ESP8266;
- live on safe networks;
- contain only hardware which is supported by existing APIs.
Happily, that’s a reasonably interesting subset! It’s probably also fair to say that with more experience, I expect I’d be happy to enlarge this domain.
References
- 1. https://en.wikipedia.org/wiki/ESP8266
- 2. https://en.wikipedia.org/wiki/Bank_of_England_£5_note
- 3. https://www.espressif.com/sites/default/files/documentation/4a-esp8266_at_instruction_set_en.pdf
- 4. https://mongoose-os.com
- 5. https://mongoose-os.com/software.html
- 6. https://mongoose-os.com/docs/quickstart/setup.html
- 7. https://brew.sh
- 8. https://github.com/cesanta/mongoose-os/tree/master/mos
- 9. https://github.com/nodemcu/nodemcu-devkit-v1.0
- 10. https://en.wikipedia.org/wiki/MQTT
- 11. https://mongoose-os.com/docs/overview/rpc.html
- 12. https://mongoose-os.com/docs/overview/apps.html
- 13. https://en.wiktionary.org/wiki/your_mileage_may_vary
- 14. https://github.com/cesanta/mjs
- 15. https://github.com/cesanta/mjs#restrictions
- 16. https://github.com/cesanta/mjs#callbacks
- 17. https://mongoose-os.com/aws_integration.html
- 18. https://aws.amazon.com/documentation/iot/
- 19. https://mongoose-os.com/aws-internet-button.html
- 20. https://aws.amazon.com/blogs/apn/aws-iot-on-mongoose-os-part-1/
- 21. https://github.com/cesanta/mongoose-os/blob/master/mos/aws.go
- 22. https://forum.mongoose-os.com/discussion/1224/connection-to-aws-iot-without-aws-iot-setup
- 23. https://forum.mongoose-os.com/discussion/1150/esp8266-cpu-frequency-aws-iot-connection-time
- 24. https://mongoose-os.com/blog/secure-remote-device-management-with-mongoose-os-and-aws-iot-for-esp32-esp8266-ti-cc3200-stm32/
- 25. https://forum.mongoose-os.com/discussion/1132/installing-on-stm32
- 26. https://github.com/cesanta
- 27. https://mongoose-os.com/docs/quickstart/setup.html
- 28. https://forum.mongoose-os.com