Continuous Delivery on a Spark.io

by Mike Linnen 15. February 2015 23:11

I have always wanted to remotely update my IoT devices when they are in need of a software change.  It is really a pain when I have to disassemble a dedicated device just because I need to update the software that is embedded in it.  It has been so much of a pain for me I have gone as far as trying to make the device as dumb as possible and leave the smarts in a service located on a computer with an OS that supports remote access.  Doing so seems to make the device just a remote I/O extension and not an actual Smart Device. 

Continuous delivery takes the remote update one step further.  If I have a source control repository that I commit some new code to, then my automated process should be smart enough to pick up that change, compile, test and deploy the new bits to it’s final destination.  Wow wouldn’t that be cool!!!

In order to do this the device would need to be smart enough to support a remote update process.  That means a device would have to be able to check for updates and automatically apply them.  You could write your own remote update process and I have done this many times for desktop applications, but doing this on an embedded device is little more tricky.  Sometimes it would require you to actually change the device firmware especially if the device doesn’t support dynamically loaded software.  Basically the process to do so would require you to do the following:

  • Detect if an update exists on some remotely accessible resource
  • Download the update and unpack it
  • Unload the current version of the software
  • Load the new version of the software
  • Start executing the new version
  • Do this reliably without bricking the device

Fortunately the Spark.Io platform does all this for you so you don’t have to.  The platform even supports an open source backend that will take source code, compile and deploy it.  Now that makes it very possible to do continuous delivery for IoT devices.  The rest of this blog post will explain how easy it is to set this up.

There are several things you will need to have in order to accomplish this:

  1. A Spark.io account and a device that is connected/registered with the account.
  2. A source control repository that is supported by your CI tool of choice (I am using GitHub)
  3. A build script that can either call restful APIs or can shell out to curl.exe (I am using psake with curl.exe).
  4. A CI tool that watches the source control for commits and kicks off the build process (I am using Team City on a Windows Azure VM).

You can see my source repository over at SparkContinuousDeliveryExample. Feel free to fork it and use it if you like.  I used psake as my build scripting language of choice as it is supported by Team City and I find it very easy to understand. 

Build script

My build script is called default.ps1.

properties { 
  $deviceId = $env:SparkDeviceId 
  $token = $env:SparkToken 
  $base_dir = resolve-path . 
  $curl = "$base_dir\lib\curl\curl.exe" 
}

task default -depends Deploy

task Clean { 
}

task Deploy { 
  $url = "https://api.spark.io/v1/devices/" + $deviceId + "?access_token=" + $token 
  exec { 
     .$curl -X PUT -F file=@src\helloworld.ino "$url" -k 

  } 
} -depends Clean

task ? -Description "Helper to display task info" { 
   Write-Documentation 

The first thing you will notice is that the script takes 4 parameters that are defaulted but also can be overridden by Team City.  The $deviceId represents the spark.io device Id and can be found in the spark.io build website after you have logged in and picked one of your devices.  The $token is your spark.io access token and can be found in the spark.io build website after you have logged in and selected the settings menu option.  You really only have 1 access token for an account but you could have multiple device id’s for an account.  The other 2 parameters are really only for executing the curl.exe program that interfaces with the spark.io platform.  Since the repository has the curl.exe you wont have to worry about installing it on your build server, but if you don’t like putting EXE’s in your repository you could override these parameters to launch the curl.exe that is installed on your build agent. The Deploy task is the only task in this build script that does anything.  Basically it launches curl.exe passing in the device id, token and the file name to upload to the spark.io platform for compilation and deployment.  In this case the program source file is helloworld.ino.

Testing the build script locally

You can test the psake build script on your local machine by first setting the environment variables for the $env:SparkDeviceId and $env:SparkToken.

SetEnvVariables

Of course I didn’t include my real Device ID and Token in the example above so make sure you set them using your own Spark.io settings.

Make sure you are in the directory where you cloned the repository and use the Invoke-psake keyword to kick off the build.

Build

Your code will be uploaded to spark.io, compiled and then deployed to your device.  As you can see here the process completed successfully.

Setting up Team City:

You will have to follow the instructions on the Team City web site on how to install Team City.  It is very easy to do but I wont detail it here.  I am also going to ignore the step of setting up Team City to connect to your Github repository. Assuming you have a project created in Team City and you have assigned the VCS root to the project you need a build step added to your project.  You want to make sure you select Powershell as the runner type.  See the other options I selected in the following screen snapshot:

TC_1

Make sure you select Source Code as the script type and then type in the powershell script but make sure you replace the YourDeviceId and YourToken.

TC_2

Here is the script so you can copy and paste it easier.


import-module .\lib\psake\psake.psm1
invoke-psake .\default.ps1 -properties @{"deviceId"="YourID";"token"="YourToken"}
if ($psake.build_success -eq $false) {exit 1} else { exit 0}

Here are the rest of the options to finish off the build step:

TC_3

And that is all there is to it.  You will want to add a Trigger to your build project so that it fires off a build when source is committed to the repository.  Also make sure you have the VCS Root pointed to the “master” branch.  Now whenever you commit changes to the master branch the build will pick it up and send it off to Spark.io for compile and deploy.

Limitations:

  • Works on single file source solutions at this time.
  • If a compile error exists the build does not fail (but the device does not update).
  • If you edit the readme or any other non-source files and commit it then the device gets updated even though no source changes were made.

Enhancements:

  • Get this working on a free hosted build platform
  • Use a Raspberry PI as a build server
  • Remove the dependency on curl.exe and use powershell to make the REST api calls
  • Add in multi-file support
  • If the Spark.io platform is down fail the build
  • Make sure a compile error fails the build

Conclusion:

That is all I wanted to cover on this blog post.  Of course there a million other ways you could set this up and I would also like to hear how others might do this.  I have always dreaded pulling devices apart just to upgrade them and therefore I tended to not update them as frequently as I would like.  I just need to get the hardware nailed down on some projects so that I can get them installed and hooked up to a build.

Garage door and the Spark Core on github

by Mike Linnen 25. December 2014 21:11

I just posted my initial code for my Garage door project using the Spark Core on github.  This is a work in progress and will evolve over time.

Garage door and the Spark Core

by Mike Linnen 20. December 2014 01:21

I recently moved to a new home and I now have 2 garage doors to control instead of one.  So I decided to revamp my garage door home automation project by using the Spark Core.  This is a fascinating device as it is designed to connect to the Spark.IO cloud service without doing a lot of coding to maintain the connectivity to the cloud.  The default firmware in the device allows you to remotely connect to it and invoke functions, expose variables to the cloud service as well as perform pub/sub between devices.  Make sure you check out their website to gain a better understanding of all the capabilities of this small packaged IoT controller.

My goals in this project was to achieve the following:

  • know when either garage door is opened or closed.
  • remotely close or open the garage door.
  • have the door automatically close when I go to bed.
  • keep track of how long it takes to open/close the garage door.
    • if the door starts to take longer to open and close this might be a sign that it needs maintenance.
  • automatically open the garage door when I arrive home.
  • automatically close the garage door when I leave my home.
  • notify me when the door needs maintenance because it has been opened/closed so many times.
  • monitor the temperature and humidity in the garage.
  • monitor the garage for motion when the security system is on.
  • notify me that I left the door open after a specific time of night.

That is certainly a large number of goals and I don’t intend to complete all of them initially but you kind of get the idea of what the possibilities are.

Initially I intended to do all of this for 2 garage doors with one Spark Core, but after thinking about it a bit it made more sense to use at least two Spark Cores

One thing I decided to do right off the bat is to make sure I have enough sensors that could determine when the door was opened and when it was closed.  I have seen other remote garage door projects that simply have one sensor that detects if the door is closed or not.  I wanted more inputs so that I could time how long it took for a door to complete it’s open or close command.  I want to keep track of this in order to determine if the door will need maintenance when it starts to take longer to open or close.  I can also gather a little more analytics around the timing of the door command and the temperature in the garage.  I don’t know if I will use this more detailed information for anything or not but I thought it would be fun to play around with.

So I will have 2 magnetic reed switches that I plan on placing on the door track to determine if the door is closed or opened.  The status of the door will be 5 different states: opened, closed, opening, closing and unknown.  The unknown state will only be for when neither of the sensors are triggered and the device doesn’t know if the door was previously opened or closed.  I have most of the code written to handle the basic door operations and I will be sharing that code in a future post.

So stay tuned on future posts on this topic as I move forward with it.  Please feel free to give me feedback or ask questions on items I haven’t clarified very well.  I am very interested in anyone’s thoughts on the Spark Core as well as home automation in general.

BuilderFaire track at Raleigh Code Camp 2014

by Mike Linnen 8. November 2014 10:04

I gave a talk at the Raleigh Code Camp called "What can I do with a Raspberry PI". The slide deck is attached to this post for those of you that attended.

Raspberry PIv2.pptx (6.67 mb)

Raleigh Code Camp 2013 Netduino Azure Session

by Mike Linnen 30. October 2013 21:31

I am presenting two sessions in the Raleigh Code Camp 2013 Builder Faire track on November 9th.  The first session is called Building a cloud enabled home security system Part 1 of 2 (the presentation).  The second session is Building a cloud enabled home security system Part 2 of 2 (the lab).  You really need to come to both sessions as the first session explains what you will be building in the second session.  Yes I said that right, if you attend the second session you will be building a Netduino based security system that connects to Windows Azure.  Check out the project website for more details at Cloud Home Security.

I hope to see you there!!

Carolina Code Camp 2013 Netduino Azure Session

by Mike Linnen 3. May 2013 21:39

I am presenting 2 sessions in the Carolina Code Camp 2013 Builder Faire track.  The first session is called Building a Home Security System – The Introduction.  The second session is Building a Home Security System – The Lab.  You really need to come to both sessions as the first session explains what you will be building in the second session.  Yes I said that right, if you attend the second session you will be building a Netduino based security system that connects to Windows Azure.  Check out the project website for more details at Cloud Home Security.    

Demo connecting 11 Netduinos to Windows Azure Service

by Mike Linnen 14. February 2013 23:28

I put together a talk that includes a lab on building a security/home automation system using 11 netduinos communicating over MQTT with a broker located in Windows Azure.  The attendees of this talk will walk through the lab and build out various components of a security system.

Here is a video demonstrating the various components of the system.  

The source for the project can be found on github:

The Security System website is hosted on a Web Role and it contains all the documentation for the lab.

Announcing: Hands on Hacknight connecting Netduino's to an Azure cloud service

by Mike Linnen 27. October 2012 00:00

In December I will be presenting this talk for the Charlotte Alt.Net users group. This talk is less about presenting and more about actually coding up a device that connects to the cloud.

This is not a sit back and watch the speaker meeting. As a participant in this project you will be building the devices that complete a Simulated Home Security system. There will be some basic code that is written for you but for the most part it will be your job to complete the code and make the device functional. The cloud service that connects all the devices via a message bus will already be completed and deployed to Windows Azure for you to use. Your device will publish and subscribe to messages on the bus.

Come out to the event and learn how to connect a Netduino Plus to Windows Azure.

Head on over to the meeting invite and sign up now.

Getting Really Small Message Broker running in Azure

by Mike Linnen 1. June 2012 23:34

In my previous post I talked about changing my home automation messaging infrastructure over to MQTT.  One of my goals was to also be able to remotely control devices in my house from my phone while I am not in my home.  The good news is that this is easily done by setting up a bridge between two brokers.  My on-premise broker is configured to connect to the off-premise broker as a bridge.  This allows me to publish and subscribe to topics on the off-premise broker which in turn get relayed to the on-premise broker. Well we need to host the off-premise broker somewhere and that somewhere can be an Azure worker role.

Really Small Message Broker (RSMB)  for windows is simply a console application that can be launched in a Worker Role.  In this blog post I will be showing you how to do just that.  One thing to note here is make sure you read the License agreement of RSMB before you use this application for your purposes.

Of course to actually publish this to Azure you will need to have an Azure account but this will also run under the emulator.  If you don’t have the tools to build windows azure applications head on over to the Windows Azure Developer portal  and check out the .Net section to get the SDK bits.  Also the following instructions assume you have downloaded RSMB and installed it onto your windows machine.

Create a new Cloud Windows Azure Project

image

Once you press the Ok button you will be asked what types of roles you want in the new project.  Just select a Worker Role and add it to the solution.

image

To make things easier rename the role as I have done below.

image

After selecting the Ok button you need to set up an endpoint for the worker role that will be exposed through the load balancer for clients to connect to.  Select the worker role and view the properties of the role.  Select the Endpoints tab and add a new endpoint with the following settings:

  • Name: WorkerIn
  • Type:
  • Protocol: tcp
  • Public Port: 1883

image

Add a new folder under the RSMBWorkerRole project called rsmb

image

Copy the following RSMB files to the new folder and add them to the RSMBWorkerRole project with Copy to Output Directory set to Copy Always

  • rsmb_1.2.0\windows\broker.exe
  • rsmb_1.2.0\windows\mqttv3c.dll
  • rsmb_1.2.0\windows\mqttv3c.lib
  • rsmb_1.2.0\messages\Messages.1.2.0

image

Add a class level declaration as follows:

Process _program = new Process();

Make sure you have a using statement for system.io at the top of the class.

using System.IO;

Add code to the OnStart Method as follows:

public override bool OnStart()
{
    // Set the maximum number of concurrent connections
    ServicePointManager.DefaultConnectionLimit = 12;

    string rsbroot = Path.Combine(Environment.GetEnvironmentVariable("RoleRoot") + @"\\", @"approot\\rsmb");
    int port = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WorkerIn"].IPEndpoint.Port;

    ProcessStartInfo pInfo = new ProcessStartInfo(Path.Combine(rsbroot, @"broker.exe"))
    {
        UseShellExecute = false,
        WorkingDirectory = rsbroot,
        ErrorDialog = false,
        CreateNoWindow = true,
    };
    _program.StartInfo = pInfo;
    _program.Start();

    return true;
}

You should be able to launch the project under the Azure emulator and then use an MQTT client to connect to a topic like $SYS/# and the client should connect without error and start receiving notifications for the published messages.  If you need to setup some additional broker configurations such as the broker.cfg then just add it to the project under the rsmb folder and make sure it is set to copy to the output directory always.  You might want to enhance the code in the OnStart method to redirect output of the RSMB console to the azure diagnostics to make troubleshooting issues easier.  You also need to setup the on-premise broker to connect to the remote broker as a bridge.  The instructions to set up the local broker as a bridge can be found in the README.htm where you installed RSMB.

Switching out my home automation messaging infrastructure for MQTT

by Mike Linnen 1. June 2012 21:09

I am re-vamping my home automation strategy from a home grown publish/subscribe messaging system to use MQTT instead.  I was using Azure Service Bus to connect remote devices such as my phone with devices in my home such as my lawn irrigation system.  This worked well as a messaging infrastructure for remote devices but I wanted to have a more standard messaging infrastructure that could work in my home network without connectivity to the outside world. 

A few reasons why I switched to MQTT:

  • Light weight
  • Many Clients already exist for many platforms and languages
  • Support for on-premise message broker
  • Support for off-premise message broker
  • Support for bridging brokers (on-premise to off-premise)
  • Fast
  • Used by companies like COSM (was Pachube) and Github
  • Simple topic subscription model that is also powerful
  • I don’t want to write a message broker

For the most part I am moving toward having devices in the home that are relatively dumb and having services running on a home server that add the smarts behind the devices.  This will give me the flexibility to change the behavior of the system a lot quicker without the hassle of tearing apart a device to upgrade the software on it.  This means I needed to have my services available all the time.  Placing these services in the cloud for mission critical things would mean I am left with devices in the home that cannot function while my internet connectivity is down.  This was the biggest reason I moved to an off the self pub/sub infrastructure like MQTT.

Like most messaging protocols, MQTT works on the notion of a topic and a message.  The topic is just a unique way of addressing a message.  I struggled a lot and I probably will continue to struggle on what my topic structure for my home automation should look like.  One thing I wanted to do is try to make the topics readable so that troubleshooting message problems would be easier.  Hear are a few standards I am trying to settle on:

  • When a device or service changes state and wishes to notify interested parties the topic will end with /event 
  • When a device or service wants to know the status of another device or service the topic will end with /getstatus
  • When a device or service receives a topic /getstatus the response topic it generates will end with /status
  • When a device or service needs to set the state of another device or service the topic will end with /set

Here are a few examples of topics and messages for my irrigation system:

Description Topic Message
Zone 1 turned on irrigation/zone/event z1 On
Request the current schedule from the irrigation service. The service will respond to this request by publishing various status topics irrigation/schedule/getstatus  
Set the time that the irrigation service should start watering irrigation/schedule/starttime/set 09:00 AM
Status of the schedule start time in response to the getstatus request irrigation/schedule/starttime/status 09:00 AM
Set the days of the week that the irrigation system will water irrigation/schedule/days/set MON WED FRI
Status of the scheduled days of the week in response to the getstatus request irrigation/schedule/days/status MON WED FRI
Set the zones that the irrigation system will run and how long irrigation/schedule/zones/set z1 10 z2 8 z3 10
Status of the scheduled zones in response to the getstatus request irrigation/schedule/zones/status z1 10 z2 8 z3 10
Sets the zones to run on the irrigation device and how long irrigation/zones/run z1 10 z2 8 z3 10

MQTT does have a concept of publishing messages with a retain bit. This just tells the broker to hang onto the last message for a topic and when a new subscription arrives the client will receive the last message.  I could have used this concept instead of the /getstatus standard that I have show above.  I might change over to using the retain bit but for now the /getstatus works for me. I am also making my messages a little verbose as they tend to contain multiple values that could have been broken down into more granular topics.

Overall I really like how simple MQTT is and it is very easy to get a device like the Netduino to understand MQTT messages.  I am sure I will make modifications on how I actually define my topics and message body over time as I develop more and more devices and services that do useful stuf in my home.    

About the author

Mike Linnen

Software Engineer specializing in Microsoft Technologies

Month List