iKommunicate

People in Tha Middle

Last friday a post on the Signal K google group announced a kickstarter campaign for the first commercial NMEA-to-signal-K gateway to be announced.  The iKommunicate has been prototyped by Digital Yacht, a company whose track record includes being one of the earliest to market with NMEA-to-wireless-IP converters.  iKommunicate features an NMEA-2000 interface, two NMEA-0183 interfaces and an ethernet interface.  NMEA data are converted to Signal K format which is then available for Signal K client applications to query.

Paul Sumpner is Digital Yacht's CTO.  We had talked about Signal K at the London Boat Show in January and when I read the iKommunicate announcement I mailed him to ask if he'd mind answering a few questions about it.  He very kindly agreed.....

Digital Yacht and Signal K

Paul, Digital Yacht were instrumental in popularising the use of “NMEA-0183-over-IP” with their NMEA-to-wifi bridges. What do you believe will be the most significant driver for Signal K supplanting this as data format of choice for app developers?

NMEA-0183 over IP was a good initial solution and for some people will be quite satisfactory, but for the developer there is still the issue of getting the NMEA0183 information, the cost of the spec and NMEA membership puts a lot of developers off and also they have to support a relatively specialist and old protocol.  

The driver for Signal K will be the NMEA2000 data that is not supported in NMEA0183, such as Engine data which until now has been inaccessible to all but a very few developers and once developers get their hands on this data, I think we will see some really interesting and innovative apps.

There are currently very few apps which can use Signal K data and almost none which can only use Signal K as a data source. Who do you see as the early adopters who will buy the first tranche of iKommunicates?

I think once a couple of apps that have a large user base, such as Active Captain or OpenCPN fully release with Signal K support (both have beta releases) then I think iKommunicate will be in demand. Michael Zapf the developer of the popular NMEARemote Instrument App is the first iOS developer to support SK and we will be contacting other iOS and Android App developers about Signal K and iKommunicate.

A kickstarter campaign is an unusual method for an established business like Digital Yacht to fund a new product and your $20,000 target surely can't cover development costs. Did you take this approach to mitigate risk in building a product with unknown demand, to create a pre-release buzz or for another reason?

It is very easy to look at Kickstarter and just see it as a way of getting funding for start-ups but I think any company can benefit from using this medium. If you are developing a completely new and unique product or trying to break in to a new market, then Kickstarter is a great way to promote your new initiative, gain feedback on whether there is a market for your new product/service and to get directly in contact with the early adopters in that market 

Apart from the time and effort in setting up the campaign, there are no other costs and if it turns out that no one likes your new product/service and you do not achieve funding, then you have avoided wasting a lot more time and effort developing a product/service that no one will buy. This is our first Kickstarter campaign and we would certainly use it again.

The kickstarter campaign was launched only days ago but you are more than a quarter of the way to achieving funding (I will be surprised if it's not at least “half” by the time you receive this). In the unlikely event that you don't meet your target, what would be the trigger for Digital Yacht to revisit iKommunicate in future?

You are right, as I am writing this answer, we have just broken the 50% funding target, so fingers crossed we will reach and hopefully exceed our original $20,000 target. Even if we do not reach our target, we hate wasting development time and effort, so we will probably regurgitate the work we have done in to some other new product in the future, but it would be a real shame as I think what Signal K is trying to achieve would be good for our industry.

Even at the full $299 SRP, the iKommunicate is competitively priced compared with some other NMEA-to-IP bridge products. Is it worth buying for its N2K/NMEA-0183 ethernet bridging and multiplexing capabilities alone?

Absolutely, we have really tried to make iKommunicate competitively priced and straight out of the box it will be a very powerful NMEA2000 and NMEA0183 to ethernet bridge, supplying data to many existing apps and software packages over TCP or UDP. Then as Signal K becomes more and more established, iKommunicate can simultaneously supply other devices on the network with the Signal K data.

iKommunicate Product Details

Does iKommunicate implement a complete Signal K server, and if so were compromises needed to make it run in only 1MB memory?

The differences between a Signal K server and a Signal K gateway are somewhat blurred and some Signal K developers would argue that there should be no difference between them. I don't entirely agree with this and designed iKommunicate to be primarily a gateway but also capable of being a basic Signal K server. 

My own view is that in a smaller boat, you will probably just have an iKommunicate and some Apps (Signal K "consumers") that will use the data. So in this instance, iKommunicate is the Signal K server. 

On a larger boat, you may have an iKommunicate and a Signal K server where iKommunicate is just a gateway converting the NMEA data in to Signal K but the server is taking this data, managing it and then serving it up to the apps or sending it to the cloud. 

iKommunicate has been designed to be a very efficient gateway and a low power server, capable of hosting some simple web apps, but not capable of running sophisticated apps, with charts, messaging, logging, etc. We are not running LINUX on iKommunicate, just native C code on an ARM M4 Cortex processor and currently we are not even using half of the 1MB of memory, even when receiving 300+ AIS targets. 

So rather than compromises, we feel that a number of new and different design decisions were taken with iKommunicate and I am very happy with how our "proof of concept" has performed and been received. 

I think a full blown Signal K server needs to run an OS, which is an overhead that requires a faster processor and more memory, but you get access to loads of software services and utilities, making it so much easier to do really powerful and innovative functions and features. That is why the Raspberry Pi, Beagle Board and other COTs hardware solutions have been initially used as the basis for Signal K servers.

Does iKommunicate store vessel state (gleaned from NMEA inputs) in an internal data model, or simply dynamically translate NMEA PGNs/sentences into delta format with Signal K context which are then supplied to all websocket clients? And are there any static data supported/query-able in the initial release?

iKommunicate has a Signal K namespace memory in which it retains the latest values of all the NMEA data that has been received including AIS data (other Vessels). When a NMEA sentence of PGN is received, it updates the namespace and sends out a Signal K delta file. If an HTTP: restAPI GET request comes in, then this is extracted from the namespace and output as SIgnal K HTTP.

We will have an option in the iKommunicate web config page called "Manage Own Data" or similar and this will make the iKommunicate responsible for populating all of the Signal K "Self" data. The static Own/Self data will be populated where possible automatically from the NMEA data or the user can enter the data manually like Boat Name, Dimensions, MMSI, etc.

What (if any) are the major differences between Digital Yacht's Signal K server implementation and the reference open source server?

The main differences are:

1. iKommunicate connects directly to the NMEA2000 network, rather than via a 3rd party NMEA2000 to USB gateway.
2. iKommunicate is a LAN rather than a WAN device and another server or app is required to handle the external transmission and reception of the Signal K data in to the internet.   
3. iKommunicate (at least in Version 1) does not support subscription websockets, it simply sends all "delta" files through the websocket i.e. every time a new NMEA PGN or Sentence is received it transmits a Signal K "delta" file. The idea with subscriptions is that the app asks for specific data and the server remembers what data the app is interested in and only sends this data when it changes

Is the flow of information through iKommunicate “one-way” or can data be passed back to the NMEA side, e.g. to control an autopilot?

In Version 1 of iKommunicate there is only data flowing from the NMEA networks to Signal K, apart from the minimum NMEA2000 network
interaction i.e. address claiming, heartbeat PGN, etc. We will be adding bi-directional NMEA communication though in a future firmware release, which by the way is designed to be really easy to update; either by uploading an update via its built-in web interface, downloading directly from the internet or copying it to the internal SD card.

Security is something that has been discussed more than directly implemented in Signal K. What (if any) security challenges will iKommunicate address in its initial implementation and what do you intend to address in future firmware updates?

The security of the "Internet of Things" in general is being discussed ad infinitum by many, many technology companies and organisations and still has some way to go before the procedures, tools and mechanisms are in place for a secure "IoT" world. The Signal K team could have wasted a lot of time and effort on security and it was felt that, at least for Version 1 of Signal K and iKommunicate, that the issue of security should be left with the Apps and Services that were taking Signal K data from the boat and transmitting it in to the internet. 

In reality every boat has a large and very good security "firewall", the open sea they are sailing on, and for the vast majority of Signal K users, the data will stay firmly on the boat, at least until more apps and services become available. On larger commercial vessels and super yachts, there will already be a secure and protected network onboard which Signal K can hide behind, as after all it is just normal network data. 

We are following closely how the security of the "IoT" is progressing outside of the marine world, in particular open source initiatives such as Let's Encrypt (https://letsencrypt.org/) to create free SSL certificates. As and when a clear security solution for "IoT" appears, we will aim to include it in Signal K.

Future Direction

At this stage Signal K complements rather than replaces NMEA protocols, with products like iKommunicate providing the necessary bridge. How much harder / more expensive (if at all) would it be to implement a sensor (e.g. GPS, Heading) outputting “native” Signal K (in its current form) than one which outputs NMEA-0183 or N2K (or, dare I mention, OneNet?)

Any "smart" sensor that has a microprocessor, rather than just a transducer inside, could in theory be a Signal K sensor. In component cost terms there is a bit of a step up from NMEA0183 to Signal K, but the difference between an NMEA2000 interface and a Signal K interface is very small and in fact it might actually be cheaper to do a Signal K interface. 

You would really want the Signal K sensor to use PoE (Power over Ethernet) which will add some additional component cost but it is not completely out of the question, I am not sure it will be Digital Yacht, but I am sure we will see Signal K sensors before too long.

If iKommunicate proves successful, what other Signal K products are Digital Yacht considering?

We intentionally did not include Wi-Fi in the first version of iKommunicate, as we felt that most early adopters would be pretty technical users and most likely have their own wireless network onboard already, but a wireless iKommunicate would be useful for smaller boats, so that is definitely on our future product roadmap, as is a J1939 version for direct connection to engines that do not natively use NMEA2000. As for anything else, assuming Signal K is the success we hope it will be, let us just say "watch this space" !

As well as implementing Signal K in its products, do you see Digital Yacht (and indeed other commercial organisations) taking an active role in Signal K development?

I hope so and Digital Yacht certainly intend to be an active contributor to Signal K. There are challenges for commercial companies to be involved in "Open Source" projects and I am sure, a certain level of dis-trust from the purist Open Source community members, but there are benefits for both sides and as long as commercial companies put in as much (or more) as they take out, then successful results can be achieved.

Many thanks to Paul for taking the time to share this.  As I write this, just 5 days into the kickstarter campaign and with 47 days still to go iKommunicate has reached nearly 4 times its funding target with more than 340 units spoken for and the number climbing all the time.  Seems like iKommunicate will be a real product and Signal K does have a commercial market.

Signal K revisited (part 2)

Talking 'Bout a Revolution

Signal K isn't what I was looking for last year when seeking an "NMEA protocol alternative". The problem to be solved is how to interface software with sensors and devices on a boat when those sensors and devices use a closed communications protocol whose owners want a (to many independent developers prohibitively high) fee for its use and forbid its use in free software. Wireless networks, tablets and smart phones are now commonplace on boats.  A good solution opens the gateway to even more applications, potentially revolutionising the way many people interact with their vessels.

I believed the solution was a new protocol satisfying the constraints of current NMEA implementations (compact, friendly to resource-limited devices and low-powered networks, easily implemented in low level languages) but running over Internet Protocol and free and open rather than proprietary.  I was initially disappointed that Signal K isn't that thing but I started to like it a lot more once I accepted that and appreciated it for what it is.

Communications Breakdown

Signal K addresses the same problem in a different way.  It's a way of representing the state of a vessel and a format for transfer of all or part of that state to a client application.  It isn't oriented to minimal resource on low powered devices.  Instead it is built to provide data to applications running on consumer devices over full-fat wifi or ethernet networks and is geared towards functionality rather than minimalism.

A "Signal K server" is an entity which acts as an aggregator for information inputs which can come from a variety of sources using any protocols which the Signal K server understands. It uses these data and static information which the user has manually input to create a model of the "state" of the boat and potentially other vessels around it.  In early iterations of Signal K, client applications would update themselves on boat data by regularly transferring the server's internal data model.  As Signal K progressed a method of transferring only new data was adopted (the "delta format") and a method of "subscribing" to updates of particular data.  The transport by which data are transferred is not mandated, although there is now a method of querying the available transports (over http).  A reliable transport is occasionally assumed and the de facto standard transport as used in the reference server implementation seems to be websockets.

What back-end protocols does a Signal K server understand? This is implementation dependent.  The Signal K project defines the architecture and provides a reference implementation (in java and javascript) but the architecture could be implemented by others differently.

The reference server includes providers which can interpret NMEA-0183 and the output of the canboat  analyzer, the latter enabling those with a method of reading NMEA-2000 PGNs (the Actisense NGT-1) to interpret some N2K data.

You're Not the One (I was looking for)

I don't care for monolithic central server architectures, nor of putting extra steps between data producers and data consumers but the data model I initially disliked has many practical advantages.  Although the security implementation is a little sketchy at present it offers the potential for sophisticated access control (like a directory server) allowing detailed specification of who has access to what.  It allows clients to specify how often they want to update their data rather than being passive consumers.  The reality is also that for a while yet our sensors and other marine devices are still going to be talking (a variety of) proprietary protocols rather than open ones so translation will be needed, and the Signal K server provides a straight forward pluggable architecture for that.  I'm not a fan of the (to my mind) unnecessary overhead of http / websockets but this is not a concern in the typical Signal K use case, it simplifies coding in the higher level languages that most app developers use, and the Signal K developers would argue that those transports are not required (despite them being de facto standards).

Picture This

There's currently plenty of details which aren't currently filled in with Signal K.  The most common data have standard representations but much does not.  In many cases this is just a matter of time and work which but much of this needs to be done for Signal K to realise its potential. It is frequently argued that Signal K's JSON format makes it infinitely extensible but it is not without some challenges.

A problem for current NMEA protocols is that they have no standard way of representing imaging data from RADAR and Sonar, relying instead on individual manufacturer's proprietary protocols.  Signal K doesn't yet solve this.  Representing image data in a text-based protocol generally requires encoding which makes high bandwidth data even more bloated.  One proposal to address this has been to use Signal K only to transport the URL of a binary imaging data source which would then be connected to by other means.

Signal K has had many of its rough edges smoothed in the past six months, but it is still a work in progress.

Safe From Harm

Security doesn't feature much in traditional marine data communications.  Most people are aware that that has to change as marine devices become accessible from outside the closed environment of a boat's internal wiring.  There are many aspects to "security" including authentication (proving something says what it says it is), authorisation (saying who can do or see what) and encryption (concealing things from prying eyes).  Using a Signal K server rather than devices publishing data directly to the network allows for a great deal of control over who has access to what.  This potential is not yet well realised, but the architecture should be well suited to it.

Authentication and encryption are the more glamorous aspects of "security" and have been extensively discussed on the Signal K google group but with no definitive mechanisms yet mandated.  The use of web protocols should allow a huge range of well-proven systems to be leveraged but possibly due to a reticence to mandate transports for Signal K, reliance hasn't been put on readily available security mechanisms associated with those transports. I can't help but wonder if standardising on the transports which people are currently using Signal K with and restricting the problem space might result in a more full-featured solution for the common use case rather than an unconstrained but less well defined product.

Welcome to the Machine

Signal K still has a few details to be filled in but it certainly has great potential in its primary use case: providing data from NMEA devices to mobile and web applications and enabling such applications to communicate with each other using the same, extensible format. I feel confident that it will gain considerably popularity over the next year.  It lacks compactness and the tight specification to be ideal as a replacement for NMEA protocols but that's not where it is targeted. It is a complement to those protocols.  The biggest initial driver to adoption is likely to be the ability to represent NMEA-2000 data, much of which can't be represented in the currently popular "NMEA-0183-over-IP" format.  Due to CAN interfaces (as used by NMEA-2000) being less trivial than RS-422 interfaces (as used by NMEA-018) and only part of NMEA-2000 having been reverse engineered, this is where commercial NMEA-to-Signal K gateways (whose vendors have access to the complete NMEA-2000 specification) come in as part of the architecture now embraced by the Signal K team and the NMEA alike.  The average user will need to buy such a device to link their Signal K applications to their boat network.

The big news this week is that Digital Yacht have just announced potentially the first commercial NMEA-to-Signal-K gateway: the iKommunicate   Even more exciting (well, for me anyway), Digital Yacht CTO Paul Sumpner has kindly agreed to answer some questions on the new product and Signal K.  Stay tuned!

Marine Data Transports Re-visited

Last year I wrote a post in which I extolled the virtues of UDP multicast for transmission of sensor information within a boat network.  Whilst that conclusion still stands in theory, real world practice is somewhat different when it comes to 802.11 wireless networks.

In my post on transmission of RADAR data over wireless networks, I mentioned that at layer 2 (the "datalink layer" in the OSI network model) "wifi" multicast packets (of which broadcast are a special case) are not acknowledged by the receiving end whereas unicast packets (those explicitly sent to one receiving station) are, so the sending station knows when it must retransmit a dropped packet.  Some may say "but isn't UDP always unreliable and unacknowledged?".  At the transport layer yes but the 802.11 ("wifi") frames that these packets are transported in don't care about the transport layer.   "Wifi" is far more "lossy" than transmission over copper cable, so 802.11 addresses this where it can by acknowledgements and retransmissions.  This is not possible for multicast (/broadcast) packets, but is for unicast.

In using my homebrew Raspberry Pi 2-based access point/data server I was noticing the GPS fix regularly dropping out in OpenCPN and AIS targets taking a long time to appear.  Some testing showed appallingly bad reliability of transmission using UDP broadcast.

The test was simple.  A MacBook Pro (the receiving station) was a wireless client of the Pi access point and located 2 metres from it in the saloon of my GRP boat.  The Pi was running kplex and, at that point, transmitting only GPS information from a directly attached Adafruit Ultimate GPS at a rate of 6 sentences per second.  The access point was not connected to the Internet and thus other traffic was minimal.  There are few access points visible on the Mac, and only the Pi transmitting on its chosen channel.

tcpdump was run simultaneously on the Pi and the Mac to record the number of "NMEA-0183" packets transmitted and received respectively over UDP broadcast during a 10 minute period.  Packet loss was then calculated: 96%!

The test was then repeated but with kplex transmitting packets via UDP unicast.  This time packet loss was, as expected, 0%.

It should be remembered that this was a home-brew access point solution using a low cost USB dongle and I'd be interested to hear of other people's experiences with commercial access points, especially expensive "marine" units like Navico's Wifi-1.  I note that Brookhouse Online, manufacturers of the "iMux" commercial multiplexer, state that "UDP is unsuitable for transmission of navigation data" (although I'll argue that actually they mean "UDP broadcast over wifi").

Meantime though, 96% packet loss clearly makes broadcast/multicast over wifi non-viable.  The alternatives?

• Use TCP.  Easy. Well supported
• Use UDP unicast.  Client addresses must be known in advance or some (currently non-existent) subscribe/publish protocol needs to be developed for marine data
• Use an access point which implements UDP multicast over 802.11 unicast (as discussed in the RADAR post).  Support for this is rare, as is support for NMEA-0183 data over multicast.

For most people the solution for now would seem to be to use TCP for delivering marine data over wireless networks.

Navionics SonarChart™ Live on Android

Skip to the good bit

For a quick hack to get SonarChart™ Live on android working with a home-brew or "unsupported" serial-to-wifi solution without the background, jump here.

How Deep Is Your Love?

A few months back a kplex user made me aware of a new feature in Navionics Boating: SonarChart™ Live.  This is a new feature in Navionics which enables users to generate bathymetric maps using current (hence, presumably, "Live") depth information from their own instruments.  More specifically,  data provided via selected partners' products. 

Anthony was trying to work out how to feed depth information via kplex to Navionics.  He'd done some investigation of the iOS version of Navionics and determined that the app was trying to contact fixed IP addresses and ports associated with the default set-up of the various "supported" partner products.  In subsequent analysis of the app I also noticed references to the bonjour services associated with GoFree (_nmea-0183._tcp) and Vesper (_vesper-nmea0183._tcp).

By setting up his pi to emulate the Digital Yacht Sonar Server, Anthony managed to feed depth information into Navionics from kplex but I thought that there must be a better way to achieve this so I emailed Navionics to ask how Navionics Boating users with existing serial-to-wifi solutions could realise the benefits of this exciting new feature. Not possible came the reply, and no they wouldn't reveal how their "automatic" connection worked:

We have developed at code level this features together with our partners. We do not open it externally at the moment and there's not any future plan to do it.

I asked for this to be escalated to product management as surely they'd want as many people as possible to enjoy this feature. I was promised a reply but despite several prompts over the following weeks one never came.

Echoes

A few days ago my android phone prompted me to upgrade Navionics Boating. The android version previously didn't support SonarChart™ Live but now it does, so despite my disinclination to allow the draconian new permission requirements (access to all my accounts? Make phone calls? Access to my contacts?  Why?) I upgraded.  With my Raymarine plotter bridging my seatalk-1 depth sounder's data to NMEA, converted to wifi by kplex running on my raspberry pi access point, I started experimenting.

Say Hello

First up...would advertising _nmea-0183._tcp as Navico's GoFree does tell Navionics to look for depth data in the right place?  I installed avahi (the linux bonjour implementation) and configured a service, creating a file /etc/avahi/services/kplex.service as follows:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->

<!DOCTYPE service-group SYSTEM "avahi-service.dtd">

<service-group>

  <name replace-wildcards="yes">%h NMEA</name>

  <service>

    <type>_nmea-0183._tcp</type>

    <port>11010</port>

  </service>

</service-group>

No joy.  For good measure I also tried configuring the _vesper-nmea0183._tcp service. Still nothing.

Listen Like Thieves

Some network snooping showed the phone attempting to connect to 192.168.0.10 on port 10110.  This was the only address I could see it trying apart from an immense volume of traffic to MarkMonitor (apparently a "brand protection" company), Flurry and Amazon Web Services (presumably Navionics's servers).  Good to know how they're using your bandwidth and exploiting all those security privileges then.

My pi is set up as a router (so will move traffic from one interface to another) so I ran up a secondary interface, assigned it the address 192.168.0.10, restarted kplex (which was implementing a tcp server on all interfaces), restarted Navionics boating and Bingo!  SonarChart™ Live functionality.

My Name Is

By curious co-incidence 192.168.0.10 is the first DHCP address handed out by the Navico GoFree Wifi-1 in default configuration and would often be the address handed out to a plotter (distributing GoFree data) on a Navico network.  Further experimentation whilst attempting to emulate a Navico network revealed something bizarre.  Without the 192.168.0.10 address configured, if the SSID of the network the phone connects to contains the string "GoFree" (e.g. "GoFree-1234") and a data service on 192.168.0.10 is not available, the Navionics app throws up a screen asking the user to input the address for the source of depth information.  Port is not configurable and seems to be fixed at 10110 but inputting the pi NMEA server's address once again gave me SonarChart™ Live functionality.

This seems to be a very crude method of attempting to limit the data servers which can be used with SonarChart™ Live given that SSID is configurable and some GoFree wifi-1 users will have doubtless changed theirs from default.  Knowing less about the set-up of other supported data servers (e.g. Vexilar T-Box, Raymarine Dragonfly) I have not investigated whether default SSID plays a part in presuming them to be sources of data.

Disco 2000

So what of UDP?  For some reason Digital Yacht's products have used 2000 as a port for their wireless NMEA servers in the past.  I've had a hard time finding a manual for their SonarServer (one of the Navionics "SonarChart" supported products).  I tried UDP broadcasts from kplex on port 10110 to no avail but changing the port to 2000 once again yielded success.

Where's Your Head At?

Interestingly Navionics Boating seems to be parsing GPS data it is being sent over wifi.  Turn off location services on android and the app doesn't know your position when it fires up.  Turn on wireless data which includes GPS as well as depth sentences and your position is plotted.   This should be of particular interest to people who bought duff mobile devices which don't include an integrated GPS chipset.

Different Trains

It seems to be the case that the iOS version of Navionics may well function differently from the android one in how it determines where its data comes from.  Bonjour is well established on Apple platforms but Network Service Discovery was only added in android 4.1.  On the other hand "wifi multicast reception" is one of the (many, many) privileges Boating requests on android.  Sadly I don't have an iOS device with Navionics to play with in order to learn more.

As Navico's use of bonjour as one of the two ways they advertise their GoFree data service isn't a bad plan, I've decided to retain my avahi configuration on the pi for the time being.

The First Cut is the Deepest

So how best to get SonarChart™ Live on android to use our home-brew data server?  You could run up an interface on 192.168.0.10 but the easiest solution would appear to be one of the following.

If you prefer to access your data over TCP, change your SSID to something with the string "GoFree" in it, e.g. "GoFree-1234" and run an nmea server on port 10110.  When you open Navionics Boating it will ask you for the address of your server, tell it, and you can start using SonarCharts™ Live.

Otherwise broadcast your data over UDP port 2000.

If anyone has a better suggestion on how to achieve this with either the android or iOS versions of Navionics Boating, please do post a comment below.

Signal K Revisted (Part 1)

NMEA Press Release

A Panbo article a few days ago announced a statement on the NMEA's website "recognizing" Signal K.  The text is as follows: 

NMEA Recognition of Signal K Open Source Project  NMEA recognizes the Signal K Open Source project as it serves the need and creates a method that allows both large and small mobile app developers to get involved with NMEA networking. NMEA will recognize this through the means of a NMEA 2000 certified gateway as it does for all other protocols. Signal K developers will need to use a NMEA 2000 certified and fire-walled gateway for the physical interface- either stand-alone or built within a display. This protects the NMEA Intellectual Property and the investments of years of development by many manufacturers worldwide. This method allows the NMEA Intellectual Property to be locked by the gateway as stated in the NMEA 2000 License Agreement. Only this style gateway can perform the translation from NMEA 2000 to Signal K. The gateway requirements are defined in NMEA 2000 Appendix H "3RD Party Gateway Requirements”. This potential new gateway is treated by NMEA no differently than any other certified gateway on the market, such as a NMEA 2000 to J1939 as an example. If NMEA members wish to get involved in Signal K, more information is available at http://signalk.org 

Supporters of Signal K including Bill Bishop  are very excited about this "recognition" but what is actually being said here?  There has never been a barrier to commercial organisations buying the NMEA 2000 specification and producing an NMEA-to-SignalK (or any other data format) gateway and stumping up the cash to get it certified.  The NMEA don't seem to be doing anything new to facilitate production of such a gateway.  If anything, this statement appears to be a warning against producing "uncertified" bridges.  A memo on the NMEA's web site to the Signal K team clarifies this somewhat.  Following discussions with the Signal K team they had agreed to make the web site statement but 

not do a press release or a member mailing regarding Signal K as previously discussed. NMEA will not use the words “Support”, “Endorse”, or “Partner With” in this web announcement. 

The NMEA will be providing the core Signal K developers with a platform at METS to discuss Signal K with NMEA members.

What does it all mean?  Given the NMEA's reputed affinity for the legal profession, although this web posting doesn't actually say anything it might be as close to the "endorsement" they say it definitely isn't as we might hope to see.  Both Actisense (reported in the Panbo article) and Digital Yacht (from conversations at the London Boat Show earlier this year) have been following the Signal K project with interest and tacit acceptance of it by the NMEA is hoped to be the trigger for more concrete product development from such companies.

For companies making gateways this presents the possibility of new products and hence revenue. For the NMEA this represents revenue from certification of new products and arguably protection of intellectual property as open source and small commercial developers will have an "official" alternative to reverse-engineered data formats.

For the average boater?  Perhaps a hope would be stimulus to app development but development of boating applications doesn't seem to have been completely hindered to date by the legal grey area surrounding NMEA-0183-over-IP which is almost universally used.  Commercial availability of NMEA-2000-to-SignalK gateways would doubtless encourage adoption of SignalK by developers but will consumers care if their data arrives on their iThing in a different format?

Probably not if it's the same data, but possibly if it's data that weren't previously available.  NMEA-0183 has no standard representation for many types of data that may exist on an NMEA-2000 bus.  NMEA 2000-to-0183 gateways don't tend to convert much of the engine management data, tank level data etc. available on an NMEA-2000 network.  Signal K is not so constrained and a commercial NMEA-2000-to-SignalK gateway could pass all these data.  More data availability opens the door to more applications.

More widespread adoption of Signal K may also promote development of apps which exploit novel features of it, such as the "social boating" aspect and sharing of data between craft.  Is that a good thing which the punters will be queuing up for?  Don't ask me: I'm so anti-social I don't even have a twitter account.

Although initially sceptical that the announcement really says nothing at all,  I do concur that this "recognition" does indeed represent a significant advancement of the  Signal K cause.

Since my previous post about Signal K last year have I concluded that it's the best thing since sliced bread?  More in part two although I confess that I'm not all that keen on sliced bread.

NMEA-0183 over- IP: The unwritten rules for programmers

An earlier post presented a subjective overview of the current non-standardised use of the application layer of NMEA-0183 by independent and open source developers.  As discussed there, NMEA-0183 is a proprietary "whole stack" protocol running over serial lines.  While a proprietary standard method for transporting the highest layer of this protocol over IP networks does exist, it is not a standard used by open source programmers or indeed the vast majority of people writing marine data applications today.

This post is aimed at people writing applications which produce or consume marine data and wondering how to maximise their interoperability with other applications.  To re-emphasise: there are no agreed rules.  The basis for this post is 3 years of observing "how everyone else does it" and hacking and debugging networking with both kplex and OpenCPN in the context of other data sources and consumers.

The Basics

The canonical publicly-available reference for NMEA-0183 is Eric Raymond's "NMEA Revealed".  This post will not attempt to repeat the contents of that document and familiarity with the structure of  NMEA-0183 as explained there will be assumed for the remainder of this post.  Here we focus on how other application developers commonly put that information over Internet Protocol.

For open source programmers, purchasing the NMEA-0183 protocol document is probably not an option: publishing source code based on the standard may be considered a violation of copyright in some countries. I attempted to clarify this with the NMEA's legal department but they declined to reply to my email.

Application Layer Structure

Application layer structure tends to be almost identical to the documented serial line standard.

Start

The starting "$" or "!" is always included as part of the transmitted sentence. You should expect it when receiving and produce it when transmitting.

Body

Not all applications enforce correct line length but there should be no need to exceed what many sources cite as the maximum of 80 characters including the initial "!" or "$" but excluding terminating line delimiters.  Applications should expect longer lines to be rejected by receiving applications and there is generally no reason not to reject longer lines.

At time of writing neither OpenCPN nor kplex makes specific checks for "illegal" characters in sentence bodies.  Legal vs. illegal NMEA-0183 characters do not seem well documented in publicly available texts.  "$", "!", "*", the line feed character <LF> and carriage return <CR> are all likely to be used by parsers to divide up sentences. 

Line termination

Significant variation seems to exist in what applications transmit.  The "correct" NMEA-0183 sentence termination sequence <CR><LF> (carriage-return linefeed, 0x0D0x0A) is most common (and used by OpenCPN), should be accepted by all parsers, and should be used to terminate sentences when transmitting data.  Some applications terminate sentences with just a <LF> and I've encountered one android application which transmitted GPS sentences terminated with just a <CR>.  For maximum compatibility, receiving applications should accept lines terminated with <CR> or <LF> and ignore subsequent characters until one marking the start of new data of interest (i.e. "$", "!" or (if supporting TAG blocks) "\").  

Parsing

The wrong way to parse data is to do what kplex did in early iterations:  Assume everything between two <CR><LF> sequences constitutes a sentence and and then check this string for "correct" structure.  A better way (used by current (at time of writing) versions of OpenCPN and kplex) is to ignore all characters until a "!" or "$" marking the beginning of a sentence (or "\" denoting the beginning of a TAG block if supporting them) then read characters until the end of the sentence and ignore everything after a terminating <CR> or <LF> delimiter until a new start of data character is seen.

Publicly available documents describing the introduction of TAG blocks in version 4 of the standard state that TAG blocks will be ignored by pre-existing NMEA-0183 parsers.  This tends to imply that the latter method is "correct".  Under that scheme TAG blocks will be ignored if not specifically supported.  Use of the latter scheme will also mean sentences are not rejected if inter-sentence line noise results in serial-to-IP converters inserting spurious characters into a data stream and will make accommodation of multiple line terminators (i.e. any combination of <CR> and <LF>) easier to code.

TAG blocks

TAG blocks as detailed here and here, are not widely supported but it has been reported that at least one hardware multiplexer aimed at recreational boaters produces them. OpenCPN's parsing strips TAG blocks. kplex recognises them, validates them, then discards them but is also capable of producing some TAG blocks.  If not explicitly supporting them, applications should ensure their parsing routines will silently discard TAG blocks without discarding the associated sentence, as discussed above.   If your application produces TAG blocks, providing an option to disable them may be useful in case the user has another application whose parsing rules have problems with them.

Transport and Network Layers

Network Layer

A well-structured post would work its way down the stack (or up) but let's get the network layer out of the way first.  It's IPv4.  With the exception of kplex I am not aware of any marine devices or applications which explicitly support NMEA-0183 over IPv6.  IPv6 may be implicitly supported on some platforms where the development framework takes care of the dirty networking details. It is not supported by OpenCPN.  To date out of hundreds of kplex users I have corresponded with, only one was using IPv6.  Do support IPv6 just because it's the decent thing to do.  Just don't expect anyone to use it.

Transport Layer

The majority of applications and devices expect data to be transmitted and received over either UDP broadcast or TCP.  For maximum interoperability both of these methods should be supported.  In a previous post I advocated UDP multicast as the optimum transport for NMEA-0183-style data.  OpenCPN supports this.  Kplex supports it.  Very little else seems to.  Please do implement it: it really isn't hard (the update to OpenCPN to support it was trivial) but as with IPv6, don't expect people to thank you for Doing The Right Thing.

UDP unicast is supported by some devices and applications.  It seems to be the preferred method for sending data to some AIS consolidation sites although most of these (including marine traffic) also seem to support TCP connections.

Packetization

Is that even a word?

TCP is a stream-based protocol so we simply write to it as we would a serial line and let TCP worry about dividing up the data (with a small caveat discussed later).  For UDP the question arises how we should break sentences between datagrams.  Should we write one sentence per datagram or fill a datagram with sentences before sending?  This question is generally not relevant to interoperability.  Receivers I have examined simply read data from a socket without being concerned about packet boundaries.  The choice of how to send is usually one of expediency.  Sending one sentence per packet incurs a higher degree of network protocol overhead relative to the amount of data sent.  Buffering packets to send multiple packets in a single datagram introduces delay.  The latter approach is also more difficult as it requires awareness of the maximum size of a datagram if fragmentation is to be avoided.  Assuming that a datagram can accommodate 82 bytes of a single NMEA-0183 sentence is not unreasonable.  The low data rates generally associated with NMEA-0183 mean that additional protocol overhead from one sentence per packet should not put undue load on a network. As this is easier to code and involves less delay this approach is my choice.

One exception to this is transmission of multi-sentence AIS data.  As sentences have to be reassembled, no additional delay is introduced by buffering parts of a multi-sentence AIS message.  Some AIS data consolidation sites such as localizatodo.com which do not support separate ports for each client rely on all parts of the message being transmitted in a single packet for the message to be correctly reassembled.  Few applications seem to concern themselves with buffering multi-sentence AIS messages to send in a single datagram.

In summary: for maximum compatibility ignore packet boundaries when reading.  With the exception mentioned above, how you packetize over UDP shouldn't affect compatibility but one sentence per packet would be my preferred choice.

Port

10110 is the port the NMEA have registered with the IANA for data over both UDP and TCP.  10110 is used as the default port by OpenCPN but there is a wide range of ports used by other devices and applications so this should always be user configurable.  Kplex's approach is first to use a user-defined port.  If none is specified it looks for "nmea-0183" in the system services database and if not found falls back to a default of 10110.

Network Addresses

Broadcast

Some devices and applications seem to follow the often-frowned-upon practice of sending broadcast data to the broadcast address of the zero network, i.e. to 255.255.255.255.  Better practice is to use the sending system's subnet broadcast address.

Applications and devices receiving UDP are rarely coded to care about the address to which data it receives was sent and with the exception of kplex, every application I have seen simply binds a receiving UDP socket to INADDR_ANY.  kplex can of course be told to do things like listen on a particular network interface which OpenCPN cannot.  To my knowledge, no-one has raised this as an issue for OpenCPN and in most environments end users simply won't care.

Multicast

Ignoring the proprietary IEC-61162-4 standard, there is no default or commonly agreed multicast group for NMEA-0183 over IP used by open source applications or devices aimed at recreational boaters.  This should therefore be end-user configurable. If picking a default I would choose one from the IPv4 organisation-local range (239.192.0.0/14 as described in RFC 2365) or the site-local IPv6 range (ffx5::/16 as described in RFC 4291)

TCP Considerations

Nagle Algorithm

If you accept the arguments given for sending one sentence per UDP packet, you'll also want to set TCP_NODELAY on sending TCP sockets.  kplex and OpenCPN do this.  Network analysis of some other devices suggests that they don't.  To disable the Nagle algorithm or not makes no difference to interoperability.

Service Discovery

UDP broadcast does not require service discovery.  There is no subscribe/publish mechanism for a client to request NMEA-0183 data over unicast UDP from a server so there is no need for service discovery for UDP.

For TCP, finding a server for NMEA-0183 data on the network can be an issue in end users' boat networks where devices are configured with dynamic addresses.

Products intended for use under Navico's "GoFree" brand are the only ones I am aware of making use of service discovery for locating NMEA-0183 data sources.  GoFree uses two service discovery mechanisms.  One is Apple's bonjour mechanism with services announced as "_nmea-0183._tcp". The other is Navico's own JSON-based service announcements sent to multicast group 239.2.1.1 port 2052 as detailed in their "tier 1" specification document. 

Applications should not assume that any programs other than those specifically designed to work with GoFree support such service discovery.  OpenCPN does not (either as client or server). kplex can listen for Navico service announcements in order to locate a server, but does not advertise itself using them.

As a more generic and widely supported protocol, applications which do wish to leverage service discovery should probably opt for bonjour with the service "_nmea-0183._tcp"

Miscellaneous

What about baud rate?

The answer may be obvious but I have seen this question asked so it is worth addressing.  This isn't an issue.  Baud is simply the transmission rate used on a serial line. It is not a property of the data which is transmitted.  The slowest rates commonly used on IP networks today, wired or wireless, are many times faster than the fastest speeds NMEA data are commonly transmitted over serial lines. Generally speaking programmers don't have to worry about transmission rates for this kind of data over IP.

Is ethernet different from wireless?

As far as the average application programmer is concerned?  No.  In some cases the physical medium over which some kinds of marine data is transmitted can matter as discussed in the previous post.  At the kinds of data rates at which NMEA-0183 is transmitted the programmer should not need to care whether their application is running on a wired or wireless network.

Summary

For maximum interoperability with other applications, applications using the application layer of NMEA-0183 over IP should:

• Use UDP broadcast and TCP over IPv4 as transports
• Use ASCII strings as they would be sent over a serial line with no additional encapsulation, starting from an initial "!" or "$", ending with the sequence <CR><LF>.
• Use NMEA-0183 checksums
• Accept sentences terminated with <CR> or <LF> and ignore all subsequent data until the start of a new sentence
• Observe a maximum sentence length of 80 characters excluding the line termination characters
• Use a default but configurable port of 10110 for both UDP and TCP
• Ensure that TAG blocks, if not supported, are gracefully ignored without discarded their associated sentences

But it would also be nice if applications:

• Support  UDP multicast
• Support IPv6 as well as IPv4
• Send data as soon as usable information is available for sending: One sentence or all sentences forming part of a multi-sentence message in one datagram in the case of UDP, Nagle algorithm disabled in the case of TCP
• Use subnet broadcast addresses rather than 255.255.255.255 when sending IPv4 broadcast

RADAR over wifi: Not always a pretty picture

OpenCPN plugins support overlay of imaging from some Garmin and Navico RADAR units.  Unfortunately some users attempting to use these plugins over wireless networks on their boats have encountered issues. Not only does it not work very well, it can make the whole wireless network unusable.  Moreover where people bridge a wireless segment to their wired network (e.g. with a simple domestic access point/switch/router) the wireless network can become unusable when the radar is switched on even if nothing on the wireless side is using it.  What's going on and what can be done to resolve it?

Why RADAR imaging might not mix well with wireless

The problem is with the way images are transmitted by the types of RADAR which OpenCPN plugins support (and probably others).  These RADAR images are not transmitted using standard NMEA protocols: The data rate is too high even for NMEA 2000. Instead manufacturers transmit RADAR images using proprietary methods over Internet Protocol running on 100Mb/s copper ethernet. Crucially manufacturers tend to use UDP multicast, sending data to multiple receivers at once (as opposed to "unicast", or sending to a single receiver).  In a previous post I extolled the virtues of multicast for transmission of boat data.  It allows all computers who have subscribed to a given multicast group to see the same data without them being replicated to each client.  Sending RADAR data over multicast on an ethernet network as the manufacturer intends works well.  Sending it over wifi is another matter.

The issue is not simply that the data rates are too high to be handled by wireless networks.  Navico state that their RADAR transmits at "up to 8 Megabits per second" which seems to concur with what users have observed.  Surely this is within the capabilities of most modern wireless networks?

Unfortunately a wireless access point has a number of "known unknowns" to deal with when transmitting to associated stations on a wireless network including the distance of the station from the transmitter, objects in between and interference in the environment which makes the task more complex than it is with copper wires.  The IEEE 802.11 standards on which common wireless networks are based each define a number of data rates.  Transmissions at lower data rates tends to be more robust than those at higher rates. Typically an access point will dynamically adjust the rate at which it sends data to each associated station based on packet loss (where transmitted frames are unacknowledged by the receiver at the link layer).  The connection between a "150 Mb/s" access point and a "150 Mb/s" laptop may be considerably lower than 150 Mb/s if the laptop is far away from the access point with intervening obstacles and/or there is interference from other devices in the environment.

Multicast (and broadcast) data must be sent at a rate which all listeners can be assumed to reliably receive.  Standards dictate that multicast should generally be sent using a rate in the "basic rate set" (the set of rates which all clients are assumed to support). Although not universally the case,  many retail access points will use the lowest of these rates, often 1Mb/s.  This of course is the physical data rate.  Various overheads mean that the rate of transmission of useful information will be be even less than that.

On top of this, multicast over wireless is more prone to packet loss than unicast.  Unlike ethernet, unicast 802.11 frames are normally acknowledged by receivers allowing them to be re-sent if they are lost in what is an inherently less reliable medium than copper wires.  Multicast frames are not usually acknowledged so aren't retransmitted if a receiver loses them.  Another potential source of packet loss  with high data rate multicast is the buffering that occurs on an access point when some stations on the network are in power saving mode, although without information on the sizes of buffers involved in a given access point it is difficult to assess the likely impact of this.

Note that these issues don't mean that multicast is a bad choice for "normal" NMEA-0183-style data. In those cases data rates are far lower, should not significantly impact the wireless network, and by the repeated nature of transducer data, occasional dropped packets are not necessarily a problem.  Remember also that broadcast data suffers from the same problems on wireless as multicast.

Why does RADAR stuff up all your other wireless data too?

If it's only the multicast which is constrained to be transmitted at 1Mb/s, why is your other wireless traffic badly impacted?

When transmitting multicast data the access point is operating at the multicast rate (e.g. 1Mb/s). Not only is there likely much more RADAR data than is involved in the average browsing session, the rules of contention for the radio give the multicast data an unfair advantage.  The radio in the access point is likely spending most of its time at the lower data rate.

Why does wired RADAR stuff up your wireless data when only wired devices are listening to it?

Your boat network may use an access point with an integrated switch for wired devices or you may have a wireless access point connected to switch to which your wired devices are also connected.

A simple switch has no knowledge of what devices on the network need to receive multicast data, so it will send multicast (and broadcast) data out of all ports.  This will include an attached wireless network, even if there are no wireless devices interested in receiving the multicast data.

Preventing multicast getting onto your wireless segment

Some switches do offer a method of ensuring that they only send multicast data out of ports to which a device which wants to receive it is attached: "IGMP snooping".  Whenever a program tells a computer that it wants to receive a particular group of multicast data, the computer will send an Internet Group Management Protocol (IGMP) "report" message on the local network.  Such messages are intended for routers which forward Internet Protocol (IP) packets between networks (e.g. your local network and your Internet connection) but some switches (which are generally link layer devices so know little of the higher level protocols carried in their data link frames) can listen for these reports and use the information to decide which switch ports to send multicast data out of.

Many high-end switches will have this functionality. Many low-end devices won't.

Making RADAR work over wireless

Wireless networks are also a problem for those wanting to deliver IPTV (Television over Internet Protocol)  using multicast so the problem has received extensive attention.  A number of solutions have been proposed and implemented in various pieces of hardware.  Retail access points are less likely to offer a solution than "Enterprise Class" (i.e. unaffordable) devices.  Two relevant solutions which require only support from the access point are:

• Multicast-to-unicast translation.  Access points send multicast packets to clients requesting them via link layer unicast.  In other words, IP packets to multicast groups are delivered in unicast 802.11 frames to the stations interested in receiving them and as such these frames are delivered reliably and at maximum reliable speed.  This technique is variously implemented in some enterprise class access points using various methods.  It is one of the subjects in the (currently not commonly implemented) IEEE 802.11aa and related 802.11v standards where it is referred to as "Directed Multicast Service" (DMS).  Some implementations use IGMP snooping to determine which clients are interested in which multicast data.  With 802.11v the client can request multicast to be delivered over unicast with DMS: a cleaner solution as the OSI layers aren't so mixed up but I haven't been able to find any hint of an API for this.  DMS does not scale well which can be an issue for IPTV delivery, but on board where no more than two clients are likely to require RADAR data it is ideal.
• Setting the multicast rate. Some devices will allow you to explicitly set the "multicast rate".  Setting this to a high value might make using RADAR over wifi viable.  The downside is that as this is the minimum rate at which clients will be allowed to connect to the access point, the range at which the access point is usable will be reduced.  In the average GRP yacht this will be a non-issue. Once again, this is more likely to be a feature on higher end equipment but is more likely to feature in consumer units than multicast-to-unicast translation.  It may also be possible to implicitly alter the multicast rate by disabling certain modulation types.

Which access points will work?

Without a RADAR unit to test or a multiplicity of access points to test I can only report what might work based on Internet research and reader feedback.  Please do let me know if you have any relevant information.  This section will be updated as new information is received.

"Enterprise Class" products capable of multicast-to-unicast are not generally found on boats so high end devices from the likes of Ruckus and Cisco are not considered here.

OpenWRT

The popular open source alternative firmware for a number of routers implemented multicast-to-unicast in the recent "Barrier Breaker 14.07" release.

Apple Airport Extreme

This apparently allows the user to set the multicast rate.  I will endeavour to test the effectiveness of this using simulated data.

Navico wifi-1

The manual for Navico's wifi-1 module describes an option of interest:

Multicast-to-Unicast:
IGMP snooping. Convert Multicast data to Unicast one.The default setting is "Enable". 

I contacted Navico to clarify this.  It seems that the "Multicast to Unicast" statement is a mistake.  The wifi-1 does do IGMP snooping but only to keep unwanted multicast data off the wireless segment. According to Navico's product manager for the wifi-1 it does not do multicast to unicast translation at layer 2 and Navico do not support RADAR data over wireless.