Introduction

The ESP82661 is a cheap hardware platform for WiFi-enabled devices: you can put something on the internet for well under a fiver2.

However, we need to consider the software too. The default ESP8266 firmware makes the device into something like a WiFi modem: it connects to the main processor over a serial link, and accepts AT commands3. This isn’t ideal, because you’ll need a second processor in the box to handle the application code, even though in many cases the ESP8266 can do it all.

Mongoose OS4 is one alternative. It provides a replacement firmware for the ESP8266 which includes a psuedo-Javascript interpreter and webserver. So, a typical ESP8266 Mongoose OS project contains:

Mongoose claim support for other processors too: more of this anon.

Installation

The Mongoose website includes installation instructions5, which in the modern style amount to piping the output from curl into bash! There’s also some more technical documentation6.

I’m wary of such a plan so I looked at the script. As of today (July 2017), on the Mac, the main action is to download a single executable, mos, from,

https://mongoose-os.com/downloads/mos/mac/mos

The script then checks for libusb and libftdi, which I had already installed via Homebrew7.

Once installed you can run mos to get a list of options:

$ ./mos --help										
The Mongoose OS command line tool, v. 20170706-152142/master@871c1644.			
Checking updates... Up to date.								
Usage:											
  ./mos <command>									
											
Commands:										
  ui             Start GUI								
  init           Initialise firmware directory structure in the current directory	
  build          Build a firmware from the sources located in the current directory	
  flash          Flash firmware to the device						
  flash-read     Read a region of flash							
  console        Simple serial port console						
  ls             List files at the local device's filesystem				
  get            Read file from the local device's filesystem and print to stdout	
  put            Put file from the host machine to the local device's filesystem	
  rm             Delete a file from the device's filesystem				
  config-get     Get config value from the locally attached device			
  config-set     Set config value at the locally attached device			
  call           Perform a device API call. "mos call RPC.List" shows available methods	
  aws-iot-setup  Provision the device for AWS IoT cloud					
  update         Self-update mos tool							
  wifi           Setup WiFi - shortcut to config-set wifi...				
											
Global Flags:										
  --verbose      Verbose output. Optional, default value: "false"			
  --logtostderr  log to standard error instead of files. Optional, default value: "false"

You’ll see that the program is very keen to check we’re running the most recent version. Again following the modern fashion, new versions are published frequently.

The code for mos lives on GitHub8. It’s written in Go, so you’ll need to install a Go toolchain to compile it.

Blinky

If you attach a suitable ESP8266 board to a USB port e.g. the NodeMCU devkit9, then making a LED blink is simply a case of:

abc(small).. $ ./mos flash esp8266
Fetching https://mongoose-os.com/downloads/esp8266.zip...
Loaded default/esp8266 version 1.0 (20170706-161740/???)
Using port /dev/cu.SLAB_USBtoUART
Opening /dev/cu.SLAB_USBtoUART...
Connecting to ESP8266 ROM, attempt 1 of 10...
Connected
Running flasher @ 460800...
Flasher is running
Flash size: 4194304, params: 0x0240 (dio,32m,40m)
Deduping...
128 @ 0x3fc000 -> 0
Writing...
4096 @ 0x0
4096 @ 0x7000
262144 @ 0x8000
675840 @ 0x100000
4096 @ 0x3fb000
Wrote 950272 bytes in 20.89 seconds (355.47 KBit/sec)
Verifying...
2592 @ 0x0
4096 @ 0x7000
262144 @ 0x8000
673824 @ 0x100000
4096 @ 0x3fb000
128 @ 0x3fc000
Booting firmware...
All done!

The code above downloads a blob containing the OS and a file-system from https://mongoose-os.com/downloads/esp8266.zip, then flashes it to the board. At this point a LED should start flashing.

Some gotchas:

WiFi

At this point, you might wish to configure the WiFi connection:

$ ./mos wifi SSID PASSWORD

In practice, it is often easier to do this from the browser-based IDE.

Mac Flashing

On my Mac I found it impossible to flash boards which use the CH340 USB-serial bridge. Once flashed, everything worked well though. Rather than explore this particular rabbit hole, I flashed these boards from a Linux box instead.

The integrated IDE

Mongoose contains a full web-based UI, which allows you to all the things you can do from the command line and more. For example, you can flash the ESP8266 or configure the WiFi with the IDE instead of at the command line. To invoke the IDE:

$ ./mos

At this point your web-browser should open something like http://127.0.0.1:1992/#files, which of course is a web server embedded into mos.

You should see a pretty UI which lets you explore the device. For example, you can browse files on the ESP8266 by clicking on the ‘Device Files’ link on the left-hand-side.

init.js is a key file: it’s essentially what gets run at boot, and so by looking at it, we can tell what the device is going to do. You can either use the in-browser file manager, or the command line:

$ ./mos get init.js
load('api_config.js');								
load('api_gpio.js');								
load('api_mqtt.js');								
load('api_sys.js');								
load('api_timer.js');								
										
// Helper C function get_led_gpio_pin() in src/main.c returns built-in LED GPIO	
let led = ffi('int get_led_gpio_pin()')();					
										
let getInfo = function() {							
  return JSON.stringify({total_ram: Sys.total_ram(), free_ram: Sys.free_ram()});
};										
										
// Blink built-in LED every second						
GPIO.set_mode(led, GPIO.MODE_OUTPUT);						
Timer.set(1000 /* 1 sec */, true /* repeat */, function() {			
  let value = GPIO.toggle(led);							
  print(value ? 'Tick' : 'Tock', 'uptime:', Sys.uptime(), getInfo());		
}, null);									
										
// Publish to MQTT topic on a button press. Button is wired to GPIO pin 0	
GPIO.set_button_handler(0, GPIO.PULL_UP, GPIO.INT_EDGE_NEG, 200, function() {	
  let topic = '/devices/' + Cfg.get('device.id') + '/events';			
  let message = getInfo();							
  let ok = MQTT.pub(topic, message, 1);						
  print('Published:', ok ? 'yes' : 'no', 'topic:', topic, 'message:', message);	
}, null);

So we can see that besides flashing a LED, the NodeMCU has also been configured to make a MQTT10 request when a push-button is pressed.

The MQTT server is configured in the confN.json files: these form a crude overlay database when things defined in e.g. conf9.json override things in conf0.json.

You can also configure MQTT from the ‘Device Config’ section of the UI, or by using mos config-set.

The filesystem

Having got to this stage, it is easy to edit files on the ESP8266. You can either get and put files with the mos tool, or just edit them live in the UI. Either way, you’ll probably have to reboot the ESP8266 for the changes to take effect.

For example, having flashed the default firmware, you can change the LED’s period by just editing the 1000ms interval between timer events. You can then save this new init.js, reboot, and see the change. There is no need to reflash the ESP8266 from scratch. Instead when you save the file, you make an RPC call to the Mongoose OS on the ESP8266, which puts the data into the filesystem.

So, one approach to building a new Mongoose OS system from scratch is to:

For example, once flashed, I run the following script:

#! /bin/sh								
									
for f in fs/*								
do									
   ./mos put $f								
done									
									
./mos config-set \							
   wifi.sta.enable=true \						
   wifi.ap.enable=false \						
   wifi.sta.ssid='XXXX' wifi.sta.pass=XXXX \				
   aws.shadow.thing_name='XXXX' \					
   mqtt.enable=true \							
   mqtt.server=xxx.iot.us-east-1.amazonaws.com:8883 \			
   mqtt.ssl_cert=cert.pem \						
   mqtt.ssl_key=private.pem \						
   mqtt.ssl_ca_cert=ca_cert.pem						

It will probably become obvious quite quickly that Mongoose has a flat filesystem i.e. there are no directories.

Network access

It’s probably worth emphasizing that once you’ve flashed the basic firmware, all the subsequent interactions need only to exchange blobs of data. So, although it might be convenient to keep using the USB-serial bridge, once the ESP8266 is on the network, you can do most of this remotely too.

You can see some examples of this in the RPC documentation11 on the Mongoose website.

This flexibility obviously has some security implications: if you do something over the network without any kind of access control, so can anyone else!

Building an application

If you step back a bit, it is clear that all this configuration malarky just changes bits in the filesystem, so we could avoid doing the configuration step if we flashed a correctly configured image.

Mongoose makes this easy, and probably even encourages such a development model. In Mongoose jargon, a customized blob like this is called an ‘app’, and the process for building them is well documented12.

Abstracting away from details, executing mos build, takes the app recipe specified in mos.yml, and turns it into a blob suitable for flashing to the ESP8266.

The toochain for building this is supplied as Docker container. You can either run it in the cloud or locally.

There are advantages to this approach over flashing random firmware then tweaking it:

However, there is a penalty to pay too. It takes a while to build the image, and you have to flash the whole device. So, if you’re just changing a few things in e.g. init.js, it is much faster to edit the files on the ESP8266.

As mentioned above, flashing CH340 boards didn’t work from my Mac which made the process even slower for me. YMMV13!

Javascript

Although Mongoose talks about writing code in JavaScript, this isn’t quite true. You actually write code in mJS14, a limited subset of JavaScript.

I think it would be better for all concerned if the Mongoose documentation made this clearer. I wasted ages trying to get some code to work, only to discover that mJS does not support closures. The mJS GitHub repo does include a list of restrictions15, but I didn’t know that at the time.

The lack of closures mean that many callbacks take a void * pointer to userdata16.

AWS IoT

One of Mongoose’s headline features is support17 for Amazon Web Services IoT18. You can see examples both on the Mongoose19, and AWS20 websites.

As AWS IoT supports MQTT, the marginal work to get this working is to create the relevant certificates for access control, and configure objects on AWS.

mos provides a helpful aws-iot-setup command which makes a series of AWS calls on your behalf.

I was slightly wary of this, and so did things manually. To do the job properly you could study the mos source21, but I just did this:

$ aws --region us-east-1 iot create-keys-and-certificate \		
    --set-as-active \							
    --certificate-pem-outfile=fs/cert.pem \				
    --public-key-outfile=fs/public.pem    \				
    --private-key-outfile=fs/private.pem

and then used to AWS console to connect a suitable policy and thing to these certificates.

You’ll need to configure the ESP8266 too:

$ mos config-set \							
   wifi.sta.enable=true \						
   wifi.ap.enable=false \						
   wifi.sta.ssid='XXXXX' wifi.sta.pass=XXXX \				
   aws.shadow.thing_name='xxxxxxxx \					
   mqtt.enable=true \							
   mqtt.server=a2uxxxxxxxxxxxx.iot.us-east-1.amazonaws.com:8883 \	
   mqtt.ssl_cert=cert.pem \						
   mqtt.ssl_key=private.pem \						
   mqtt.ssl_ca_cert=ca_cert.pem						

I think the main difference from calling mos is that I made RSA certificates, but it might be better to use ECDSA instead. As this forum post22 explains, ECDSA will be a lot faster if you have a ATECC508A crypto-chip. On the other hand, I don’t have such a chip!

It is worth pointing out that connecting to AWS IoT does take ages: about half-a-minute in my experience. This isn’t Amazon’s fault: the ESP8266 is slow23.

Once the AWS IoT stuff is working, you can use it to manage the device remotely24 as well.

Whilst clever and potentially interesting, I note that connecting a Mongoose device to AWS IoT does not mean you’re opening a connection purely for data.

STM32

Another headline Mongoose feature is support for other chips. Although I’ve mentioned ESP8266 a lot above, I’d hoped I could substitute ESP32 and STM32 without a problem.

However, whilst Mongoose stand by their ESP32 support, the STM32 stuff is presently ‘really flaky’25.

Conclusions

Overall I’ve been both delighted and disappointed by Mongoose OS.

On the plus side, it makes it very easy to build a certain class of projects based around the ESP8266. You can get source code from GitHub26, you don’t have to go far to get reasonable documentation27, and the forums28 are great.

On the other hand I think my expectations were set rather too high:

Also I think the ESP8266 just isn’t fast enough to let you write even slow device handlers in mJS. For example, I wanted to connect a rotary encoder, but it was easy to turn it too fast to track.

In practice I think anything which goes up to the JavaScript layer takes a randomish time of about a few ms. For example, I hooked a scope to the blinky example and found that although the average flashing period was pretty accurate the standard deviation was about 1ms, and after a thousand iterations I had about 4ms spread between the minimum and maxiumum periods. I think that means that if you want to handle frequencies higher than a few Hertz, you’ll need to use C.

Writing in C isn’t a problem, but I’m not sure I want to write lots of it against the Mongoose APIs or in the Mongoose environment. I think I’d prefer to work in a more traditional setting, but that might just be ignorance.

There’s also a subjective, aesthetic, issue. I feel there’s quite a lot of magic built into mos so you either have to take things on trust, or spend time and thought working around mos.

Take the AWS credential issues: I wasn’t particularly keen to divulge my AWS secrets to Mongoose, nor am I particularly keen to outsource certificate creation to Mongoose. I would have been much happier if mos had generated a script which I could eyeball before executing.

I think this is a particularly serious issue with anything security related, so I see a dark side to many of the clever remote management features.

My tentative conclusion is that Mongoose is a great way to build things which:

Happily, that’s a reasonably interesting subset! It’s probably also fair to say that with more experience, I expect I’d be happy to enlarge this domain.