Signal K revisited (part 2)

Talking 'Bout a Revolution

Signal K isn't what I was looking for last year when seeking an "NMEA protocol alternative". The problem to be solved is how to interface software with sensors and devices on a boat when those sensors and devices use a closed communications protocol whose owners want a (to many independent developers prohibitively high) fee for its use and forbid its use in free software. Wireless networks, tablets and smart phones are now commonplace on boats.  A good solution opens the gateway to even more applications, potentially revolutionising the way many people interact with their vessels.

I believed the solution was a new protocol satisfying the constraints of current NMEA implementations (compact, friendly to resource-limited devices and low-powered networks, easily implemented in low level languages) but running over Internet Protocol and free and open rather than proprietary.  I was initially disappointed that Signal K isn't that thing but I started to like it a lot more once I accepted that and appreciated it for what it is.

Communications Breakdown

Signal K addresses the same problem in a different way.  It's a way of representing the state of a vessel and a format for transfer of all or part of that state to a client application.  It isn't oriented to minimal resource on low powered devices.  Instead it is built to provide data to applications running on consumer devices over full-fat wifi or ethernet networks and is geared towards functionality rather than minimalism.

A "Signal K server" is an entity which acts as an aggregator for information inputs which can come from a variety of sources using any protocols which the Signal K server understands. It uses these data and static information which the user has manually input to create a model of the "state" of the boat and potentially other vessels around it.  In early iterations of Signal K, client applications would update themselves on boat data by regularly transferring the server's internal data model.  As Signal K progressed a method of transferring only new data was adopted (the "delta format") and a method of "subscribing" to updates of particular data.  The transport by which data are transferred is not mandated, although there is now a method of querying the available transports (over http).  A reliable transport is occasionally assumed and the de facto standard transport as used in the reference server implementation seems to be websockets.

What back-end protocols does a Signal K server understand? This is implementation dependent.  The Signal K project defines the architecture and provides a reference implementation (in java and javascript) but the architecture could be implemented by others differently.

The reference server includes providers which can interpret NMEA-0183 and the output of the canboat  analyzer, the latter enabling those with a method of reading NMEA-2000 PGNs (the Actisense NGT-1) to interpret some N2K data.

You're Not the One (I was looking for)

I don't care for monolithic central server architectures, nor of putting extra steps between data producers and data consumers but the data model I initially disliked has many practical advantages.  Although the security implementation is a little sketchy at present it offers the potential for sophisticated access control (like a directory server) allowing detailed specification of who has access to what.  It allows clients to specify how often they want to update their data rather than being passive consumers.  The reality is also that for a while yet our sensors and other marine devices are still going to be talking (a variety of) proprietary protocols rather than open ones so translation will be needed, and the Signal K server provides a straight forward pluggable architecture for that.  I'm not a fan of the (to my mind) unnecessary overhead of http / websockets but this is not a concern in the typical Signal K use case, it simplifies coding in the higher level languages that most app developers use, and the Signal K developers would argue that those transports are not required (despite them being de facto standards).

Picture This

There's currently plenty of details which aren't currently filled in with Signal K.  The most common data have standard representations but much does not.  In many cases this is just a matter of time and work which but much of this needs to be done for Signal K to realise its potential. It is frequently argued that Signal K's JSON format makes it infinitely extensible but it is not without some challenges.

A problem for current NMEA protocols is that they have no standard way of representing imaging data from RADAR and Sonar, relying instead on individual manufacturer's proprietary protocols.  Signal K doesn't yet solve this.  Representing image data in a text-based protocol generally requires encoding which makes high bandwidth data even more bloated.  One proposal to address this has been to use Signal K only to transport the URL of a binary imaging data source which would then be connected to by other means.

Signal K has had many of its rough edges smoothed in the past six months, but it is still a work in progress.

Safe From Harm

Security doesn't feature much in traditional marine data communications.  Most people are aware that that has to change as marine devices become accessible from outside the closed environment of a boat's internal wiring.  There are many aspects to "security" including authentication (proving something says what it says it is), authorisation (saying who can do or see what) and encryption (concealing things from prying eyes).  Using a Signal K server rather than devices publishing data directly to the network allows for a great deal of control over who has access to what.  This potential is not yet well realised, but the architecture should be well suited to it.

Authentication and encryption are the more glamorous aspects of "security" and have been extensively discussed on the Signal K google group but with no definitive mechanisms yet mandated.  The use of web protocols should allow a huge range of well-proven systems to be leveraged but possibly due to a reticence to mandate transports for Signal K, reliance hasn't been put on readily available security mechanisms associated with those transports. I can't help but wonder if standardising on the transports which people are currently using Signal K with and restricting the problem space might result in a more full-featured solution for the common use case rather than an unconstrained but less well defined product.

Welcome to the Machine

Signal K still has a few details to be filled in but it certainly has great potential in its primary use case: providing data from NMEA devices to mobile and web applications and enabling such applications to communicate with each other using the same, extensible format. I feel confident that it will gain considerably popularity over the next year.  It lacks compactness and the tight specification to be ideal as a replacement for NMEA protocols but that's not where it is targeted. It is a complement to those protocols.  The biggest initial driver to adoption is likely to be the ability to represent NMEA-2000 data, much of which can't be represented in the currently popular "NMEA-0183-over-IP" format.  Due to CAN interfaces (as used by NMEA-2000) being less trivial than RS-422 interfaces (as used by NMEA-018) and only part of NMEA-2000 having been reverse engineered, this is where commercial NMEA-to-Signal K gateways (whose vendors have access to the complete NMEA-2000 specification) come in as part of the architecture now embraced by the Signal K team and the NMEA alike.  The average user will need to buy such a device to link their Signal K applications to their boat network.

The big news this week is that Digital Yacht have just announced potentially the first commercial NMEA-to-Signal-K gateway: the iKommunicate   Even more exciting (well, for me anyway), Digital Yacht CTO Paul Sumpner has kindly agreed to answer some questions on the new product and Signal K.  Stay tuned!

Marine Data Transports Re-visited

Last year I wrote a post in which I extolled the virtues of UDP multicast for transmission of sensor information within a boat network.  Whilst that conclusion still stands in theory, real world practice is somewhat different when it comes to 802.11 wireless networks.

In my post on transmission of RADAR data over wireless networks, I mentioned that at layer 2 (the "datalink layer" in the OSI network model) "wifi" multicast packets (of which broadcast are a special case) are not acknowledged by the receiving end whereas unicast packets (those explicitly sent to one receiving station) are, so the sending station knows when it must retransmit a dropped packet.  Some may say "but isn't UDP always unreliable and unacknowledged?".  At the transport layer yes but the 802.11 ("wifi") frames that these packets are transported in don't care about the transport layer.   "Wifi" is far more "lossy" than transmission over copper cable, so 802.11 addresses this where it can by acknowledgements and retransmissions.  This is not possible for multicast (/broadcast) packets, but is for unicast.

In using my homebrew Raspberry Pi 2-based access point/data server I was noticing the GPS fix regularly dropping out in OpenCPN and AIS targets taking a long time to appear.  Some testing showed appallingly bad reliability of transmission using UDP broadcast.

The test was simple.  A MacBook Pro (the receiving station) was a wireless client of the Pi access point and located 2 metres from it in the saloon of my GRP boat.  The Pi was running kplex and, at that point, transmitting only GPS information from a directly attached Adafruit Ultimate GPS at a rate of 6 sentences per second.  The access point was not connected to the Internet and thus other traffic was minimal.  There are few access points visible on the Mac, and only the Pi transmitting on its chosen channel.

tcpdump was run simultaneously on the Pi and the Mac to record the number of "NMEA-0183" packets transmitted and received respectively over UDP broadcast during a 10 minute period.  Packet loss was then calculated: 96%!

The test was then repeated but with kplex transmitting packets via UDP unicast.  This time packet loss was, as expected, 0%.

It should be remembered that this was a home-brew access point solution using a low cost USB dongle and I'd be interested to hear of other people's experiences with commercial access points, especially expensive "marine" units like Navico's Wifi-1.  I note that Brookhouse Online, manufacturers of the "iMux" commercial multiplexer, state that "UDP is unsuitable for transmission of navigation data" (although I'll argue that actually they mean "UDP broadcast over wifi").

Meantime though, 96% packet loss clearly makes broadcast/multicast over wifi non-viable.  The alternatives?

• Use TCP.  Easy. Well supported
• Use UDP unicast.  Client addresses must be known in advance or some (currently non-existent) subscribe/publish protocol needs to be developed for marine data
• Use an access point which implements UDP multicast over 802.11 unicast (as discussed in the RADAR post).  Support for this is rare, as is support for NMEA-0183 data over multicast.

For most people the solution for now would seem to be to use TCP for delivering marine data over wireless networks.

Navionics SonarChart™ Live on Android

Skip to the good bit

For a quick hack to get SonarChart™ Live on android working with a home-brew or "unsupported" serial-to-wifi solution without the background, jump here.

How Deep Is Your Love?

A few months back a kplex user made me aware of a new feature in Navionics Boating: SonarChart™ Live.  This is a new feature in Navionics which enables users to generate bathymetric maps using current (hence, presumably, "Live") depth information from their own instruments.  More specifically,  data provided via selected partners' products. 

Anthony was trying to work out how to feed depth information via kplex to Navionics.  He'd done some investigation of the iOS version of Navionics and determined that the app was trying to contact fixed IP addresses and ports associated with the default set-up of the various "supported" partner products.  In subsequent analysis of the app I also noticed references to the bonjour services associated with GoFree (_nmea-0183._tcp) and Vesper (_vesper-nmea0183._tcp).

By setting up his pi to emulate the Digital Yacht Sonar Server, Anthony managed to feed depth information into Navionics from kplex but I thought that there must be a better way to achieve this so I emailed Navionics to ask how Navionics Boating users with existing serial-to-wifi solutions could realise the benefits of this exciting new feature. Not possible came the reply, and no they wouldn't reveal how their "automatic" connection worked:

We have developed at code level this features together with our partners. We do not open it externally at the moment and there's not any future plan to do it.

I asked for this to be escalated to product management as surely they'd want as many people as possible to enjoy this feature. I was promised a reply but despite several prompts over the following weeks one never came.

Echoes

A few days ago my android phone prompted me to upgrade Navionics Boating. The android version previously didn't support SonarChart™ Live but now it does, so despite my disinclination to allow the draconian new permission requirements (access to all my accounts? Make phone calls? Access to my contacts?  Why?) I upgraded.  With my Raymarine plotter bridging my seatalk-1 depth sounder's data to NMEA, converted to wifi by kplex running on my raspberry pi access point, I started experimenting.

Say Hello

First up...would advertising _nmea-0183._tcp as Navico's GoFree does tell Navionics to look for depth data in the right place?  I installed avahi (the linux bonjour implementation) and configured a service, creating a file /etc/avahi/services/kplex.service as follows:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->

<!DOCTYPE service-group SYSTEM "avahi-service.dtd">

<service-group>

  <name replace-wildcards="yes">%h NMEA</name>

  <service>

    <type>_nmea-0183._tcp</type>

    <port>11010</port>

  </service>

</service-group>

No joy.  For good measure I also tried configuring the _vesper-nmea0183._tcp service. Still nothing.

Listen Like Thieves

Some network snooping showed the phone attempting to connect to 192.168.0.10 on port 10110.  This was the only address I could see it trying apart from an immense volume of traffic to MarkMonitor (apparently a "brand protection" company), Flurry and Amazon Web Services (presumably Navionics's servers).  Good to know how they're using your bandwidth and exploiting all those security privileges then.

My pi is set up as a router (so will move traffic from one interface to another) so I ran up a secondary interface, assigned it the address 192.168.0.10, restarted kplex (which was implementing a tcp server on all interfaces), restarted Navionics boating and Bingo!  SonarChart™ Live functionality.

My Name Is

By curious co-incidence 192.168.0.10 is the first DHCP address handed out by the Navico GoFree Wifi-1 in default configuration and would often be the address handed out to a plotter (distributing GoFree data) on a Navico network.  Further experimentation whilst attempting to emulate a Navico network revealed something bizarre.  Without the 192.168.0.10 address configured, if the SSID of the network the phone connects to contains the string "GoFree" (e.g. "GoFree-1234") and a data service on 192.168.0.10 is not available, the Navionics app throws up a screen asking the user to input the address for the source of depth information.  Port is not configurable and seems to be fixed at 10110 but inputting the pi NMEA server's address once again gave me SonarChart™ Live functionality.

This seems to be a very crude method of attempting to limit the data servers which can be used with SonarChart™ Live given that SSID is configurable and some GoFree wifi-1 users will have doubtless changed theirs from default.  Knowing less about the set-up of other supported data servers (e.g. Vexilar T-Box, Raymarine Dragonfly) I have not investigated whether default SSID plays a part in presuming them to be sources of data.

Disco 2000

So what of UDP?  For some reason Digital Yacht's products have used 2000 as a port for their wireless NMEA servers in the past.  I've had a hard time finding a manual for their SonarServer (one of the Navionics "SonarChart" supported products).  I tried UDP broadcasts from kplex on port 10110 to no avail but changing the port to 2000 once again yielded success.

Where's Your Head At?

Interestingly Navionics Boating seems to be parsing GPS data it is being sent over wifi.  Turn off location services on android and the app doesn't know your position when it fires up.  Turn on wireless data which includes GPS as well as depth sentences and your position is plotted.   This should be of particular interest to people who bought duff mobile devices which don't include an integrated GPS chipset.

Different Trains

It seems to be the case that the iOS version of Navionics may well function differently from the android one in how it determines where its data comes from.  Bonjour is well established on Apple platforms but Network Service Discovery was only added in android 4.1.  On the other hand "wifi multicast reception" is one of the (many, many) privileges Boating requests on android.  Sadly I don't have an iOS device with Navionics to play with in order to learn more.

As Navico's use of bonjour as one of the two ways they advertise their GoFree data service isn't a bad plan, I've decided to retain my avahi configuration on the pi for the time being.

The First Cut is the Deepest

So how best to get SonarChart™ Live on android to use our home-brew data server?  You could run up an interface on 192.168.0.10 but the easiest solution would appear to be one of the following.

If you prefer to access your data over TCP, change your SSID to something with the string "GoFree" in it, e.g. "GoFree-1234" and run an nmea server on port 10110.  When you open Navionics Boating it will ask you for the address of your server, tell it, and you can start using SonarCharts™ Live.

Otherwise broadcast your data over UDP port 2000.

If anyone has a better suggestion on how to achieve this with either the android or iOS versions of Navionics Boating, please do post a comment below.

Signal K Revisted (Part 1)

NMEA Press Release

A Panbo article a few days ago announced a statement on the NMEA's website "recognizing" Signal K.  The text is as follows: 

NMEA Recognition of Signal K Open Source Project  NMEA recognizes the Signal K Open Source project as it serves the need and creates a method that allows both large and small mobile app developers to get involved with NMEA networking. NMEA will recognize this through the means of a NMEA 2000 certified gateway as it does for all other protocols. Signal K developers will need to use a NMEA 2000 certified and fire-walled gateway for the physical interface- either stand-alone or built within a display. This protects the NMEA Intellectual Property and the investments of years of development by many manufacturers worldwide. This method allows the NMEA Intellectual Property to be locked by the gateway as stated in the NMEA 2000 License Agreement. Only this style gateway can perform the translation from NMEA 2000 to Signal K. The gateway requirements are defined in NMEA 2000 Appendix H "3RD Party Gateway Requirements”. This potential new gateway is treated by NMEA no differently than any other certified gateway on the market, such as a NMEA 2000 to J1939 as an example. If NMEA members wish to get involved in Signal K, more information is available at http://signalk.org 

Supporters of Signal K including Bill Bishop  are very excited about this "recognition" but what is actually being said here?  There has never been a barrier to commercial organisations buying the NMEA 2000 specification and producing an NMEA-to-SignalK (or any other data format) gateway and stumping up the cash to get it certified.  The NMEA don't seem to be doing anything new to facilitate production of such a gateway.  If anything, this statement appears to be a warning against producing "uncertified" bridges.  A memo on the NMEA's web site to the Signal K team clarifies this somewhat.  Following discussions with the Signal K team they had agreed to make the web site statement but 

not do a press release or a member mailing regarding Signal K as previously discussed. NMEA will not use the words “Support”, “Endorse”, or “Partner With” in this web announcement. 

The NMEA will be providing the core Signal K developers with a platform at METS to discuss Signal K with NMEA members.

What does it all mean?  Given the NMEA's reputed affinity for the legal profession, although this web posting doesn't actually say anything it might be as close to the "endorsement" they say it definitely isn't as we might hope to see.  Both Actisense (reported in the Panbo article) and Digital Yacht (from conversations at the London Boat Show earlier this year) have been following the Signal K project with interest and tacit acceptance of it by the NMEA is hoped to be the trigger for more concrete product development from such companies.

For companies making gateways this presents the possibility of new products and hence revenue. For the NMEA this represents revenue from certification of new products and arguably protection of intellectual property as open source and small commercial developers will have an "official" alternative to reverse-engineered data formats.

For the average boater?  Perhaps a hope would be stimulus to app development but development of boating applications doesn't seem to have been completely hindered to date by the legal grey area surrounding NMEA-0183-over-IP which is almost universally used.  Commercial availability of NMEA-2000-to-SignalK gateways would doubtless encourage adoption of SignalK by developers but will consumers care if their data arrives on their iThing in a different format?

Probably not if it's the same data, but possibly if it's data that weren't previously available.  NMEA-0183 has no standard representation for many types of data that may exist on an NMEA-2000 bus.  NMEA 2000-to-0183 gateways don't tend to convert much of the engine management data, tank level data etc. available on an NMEA-2000 network.  Signal K is not so constrained and a commercial NMEA-2000-to-SignalK gateway could pass all these data.  More data availability opens the door to more applications.

More widespread adoption of Signal K may also promote development of apps which exploit novel features of it, such as the "social boating" aspect and sharing of data between craft.  Is that a good thing which the punters will be queuing up for?  Don't ask me: I'm so anti-social I don't even have a twitter account.

Although initially sceptical that the announcement really says nothing at all,  I do concur that this "recognition" does indeed represent a significant advancement of the  Signal K cause.

Since my previous post about Signal K last year have I concluded that it's the best thing since sliced bread?  More in part two although I confess that I'm not all that keen on sliced bread.

NMEA-0183 over- IP: The unwritten rules for programmers

An earlier post presented a subjective overview of the current non-standardised use of the application layer of NMEA-0183 by independent and open source developers.  As discussed there, NMEA-0183 is a proprietary "whole stack" protocol running over serial lines.  While a proprietary standard method for transporting the highest layer of this protocol over IP networks does exist, it is not a standard used by open source programmers or indeed the vast majority of people writing marine data applications today.

This post is aimed at people writing applications which produce or consume marine data and wondering how to maximise their interoperability with other applications.  To re-emphasise: there are no agreed rules.  The basis for this post is 3 years of observing "how everyone else does it" and hacking and debugging networking with both kplex and OpenCPN in the context of other data sources and consumers.

The Basics

The canonical publicly-available reference for NMEA-0183 is Eric Raymond's "NMEA Revealed".  This post will not attempt to repeat the contents of that document and familiarity with the structure of  NMEA-0183 as explained there will be assumed for the remainder of this post.  Here we focus on how other application developers commonly put that information over Internet Protocol.

For open source programmers, purchasing the NMEA-0183 protocol document is probably not an option: publishing source code based on the standard may be considered a violation of copyright in some countries. I attempted to clarify this with the NMEA's legal department but they declined to reply to my email.

Application Layer Structure

Application layer structure tends to be almost identical to the documented serial line standard.

Start

The starting "$" or "!" is always included as part of the transmitted sentence. You should expect it when receiving and produce it when transmitting.

Body

Not all applications enforce correct line length but there should be no need to exceed what many sources cite as the maximum of 80 characters including the initial "!" or "$" but excluding terminating line delimiters.  Applications should expect longer lines to be rejected by receiving applications and there is generally no reason not to reject longer lines.

At time of writing neither OpenCPN nor kplex makes specific checks for "illegal" characters in sentence bodies.  Legal vs. illegal NMEA-0183 characters do not seem well documented in publicly available texts.  "$", "!", "*", the line feed character <LF> and carriage return <CR> are all likely to be used by parsers to divide up sentences. 

Line termination

Significant variation seems to exist in what applications transmit.  The "correct" NMEA-0183 sentence termination sequence <CR><LF> (carriage-return linefeed, 0x0D0x0A) is most common (and used by OpenCPN), should be accepted by all parsers, and should be used to terminate sentences when transmitting data.  Some applications terminate sentences with just a <LF> and I've encountered one android application which transmitted GPS sentences terminated with just a <CR>.  For maximum compatibility, receiving applications should accept lines terminated with <CR> or <LF> and ignore subsequent characters until one marking the start of new data of interest (i.e. "$", "!" or (if supporting TAG blocks) "\").  

Parsing

The wrong way to parse data is to do what kplex did in early iterations:  Assume everything between two <CR><LF> sequences constitutes a sentence and and then check this string for "correct" structure.  A better way (used by current (at time of writing) versions of OpenCPN and kplex) is to ignore all characters until a "!" or "$" marking the beginning of a sentence (or "\" denoting the beginning of a TAG block if supporting them) then read characters until the end of the sentence and ignore everything after a terminating <CR> or <LF> delimiter until a new start of data character is seen.

Publicly available documents describing the introduction of TAG blocks in version 4 of the standard state that TAG blocks will be ignored by pre-existing NMEA-0183 parsers.  This tends to imply that the latter method is "correct".  Under that scheme TAG blocks will be ignored if not specifically supported.  Use of the latter scheme will also mean sentences are not rejected if inter-sentence line noise results in serial-to-IP converters inserting spurious characters into a data stream and will make accommodation of multiple line terminators (i.e. any combination of <CR> and <LF>) easier to code.

TAG blocks

TAG blocks as detailed here and here, are not widely supported but it has been reported that at least one hardware multiplexer aimed at recreational boaters produces them. OpenCPN's parsing strips TAG blocks. kplex recognises them, validates them, then discards them but is also capable of producing some TAG blocks.  If not explicitly supporting them, applications should ensure their parsing routines will silently discard TAG blocks without discarding the associated sentence, as discussed above.   If your application produces TAG blocks, providing an option to disable them may be useful in case the user has another application whose parsing rules have problems with them.

Transport and Network Layers

Network Layer

A well-structured post would work its way down the stack (or up) but let's get the network layer out of the way first.  It's IPv4.  With the exception of kplex I am not aware of any marine devices or applications which explicitly support NMEA-0183 over IPv6.  IPv6 may be implicitly supported on some platforms where the development framework takes care of the dirty networking details. It is not supported by OpenCPN.  To date out of hundreds of kplex users I have corresponded with, only one was using IPv6.  Do support IPv6 just because it's the decent thing to do.  Just don't expect anyone to use it.

Transport Layer

The majority of applications and devices expect data to be transmitted and received over either UDP broadcast or TCP.  For maximum interoperability both of these methods should be supported.  In a previous post I advocated UDP multicast as the optimum transport for NMEA-0183-style data.  OpenCPN supports this.  Kplex supports it.  Very little else seems to.  Please do implement it: it really isn't hard (the update to OpenCPN to support it was trivial) but as with IPv6, don't expect people to thank you for Doing The Right Thing.

UDP unicast is supported by some devices and applications.  It seems to be the preferred method for sending data to some AIS consolidation sites although most of these (including marine traffic) also seem to support TCP connections.

Packetization

Is that even a word?

TCP is a stream-based protocol so we simply write to it as we would a serial line and let TCP worry about dividing up the data (with a small caveat discussed later).  For UDP the question arises how we should break sentences between datagrams.  Should we write one sentence per datagram or fill a datagram with sentences before sending?  This question is generally not relevant to interoperability.  Receivers I have examined simply read data from a socket without being concerned about packet boundaries.  The choice of how to send is usually one of expediency.  Sending one sentence per packet incurs a higher degree of network protocol overhead relative to the amount of data sent.  Buffering packets to send multiple packets in a single datagram introduces delay.  The latter approach is also more difficult as it requires awareness of the maximum size of a datagram if fragmentation is to be avoided.  Assuming that a datagram can accommodate 82 bytes of a single NMEA-0183 sentence is not unreasonable.  The low data rates generally associated with NMEA-0183 mean that additional protocol overhead from one sentence per packet should not put undue load on a network. As this is easier to code and involves less delay this approach is my choice.

One exception to this is transmission of multi-sentence AIS data.  As sentences have to be reassembled, no additional delay is introduced by buffering parts of a multi-sentence AIS message.  Some AIS data consolidation sites such as localizatodo.com which do not support separate ports for each client rely on all parts of the message being transmitted in a single packet for the message to be correctly reassembled.  Few applications seem to concern themselves with buffering multi-sentence AIS messages to send in a single datagram.

In summary: for maximum compatibility ignore packet boundaries when reading.  With the exception mentioned above, how you packetize over UDP shouldn't affect compatibility but one sentence per packet would be my preferred choice.

Port

10110 is the port the NMEA have registered with the IANA for data over both UDP and TCP.  10110 is used as the default port by OpenCPN but there is a wide range of ports used by other devices and applications so this should always be user configurable.  Kplex's approach is first to use a user-defined port.  If none is specified it looks for "nmea-0183" in the system services database and if not found falls back to a default of 10110.

Network Addresses

Broadcast

Some devices and applications seem to follow the often-frowned-upon practice of sending broadcast data to the broadcast address of the zero network, i.e. to 255.255.255.255.  Better practice is to use the sending system's subnet broadcast address.

Applications and devices receiving UDP are rarely coded to care about the address to which data it receives was sent and with the exception of kplex, every application I have seen simply binds a receiving UDP socket to INADDR_ANY.  kplex can of course be told to do things like listen on a particular network interface which OpenCPN cannot.  To my knowledge, no-one has raised this as an issue for OpenCPN and in most environments end users simply won't care.

Multicast

Ignoring the proprietary IEC-61162-4 standard, there is no default or commonly agreed multicast group for NMEA-0183 over IP used by open source applications or devices aimed at recreational boaters.  This should therefore be end-user configurable. If picking a default I would choose one from the IPv4 organisation-local range (239.192.0.0/14 as described in RFC 2365) or the site-local IPv6 range (ffx5::/16 as described in RFC 4291)

TCP Considerations

Nagle Algorithm

If you accept the arguments given for sending one sentence per UDP packet, you'll also want to set TCP_NODELAY on sending TCP sockets.  kplex and OpenCPN do this.  Network analysis of some other devices suggests that they don't.  To disable the Nagle algorithm or not makes no difference to interoperability.

Service Discovery

UDP broadcast does not require service discovery.  There is no subscribe/publish mechanism for a client to request NMEA-0183 data over unicast UDP from a server so there is no need for service discovery for UDP.

For TCP, finding a server for NMEA-0183 data on the network can be an issue in end users' boat networks where devices are configured with dynamic addresses.

Products intended for use under Navico's "GoFree" brand are the only ones I am aware of making use of service discovery for locating NMEA-0183 data sources.  GoFree uses two service discovery mechanisms.  One is Apple's bonjour mechanism with services announced as "_nmea-0183._tcp". The other is Navico's own JSON-based service announcements sent to multicast group 239.2.1.1 port 2052 as detailed in their "tier 1" specification document. 

Applications should not assume that any programs other than those specifically designed to work with GoFree support such service discovery.  OpenCPN does not (either as client or server). kplex can listen for Navico service announcements in order to locate a server, but does not advertise itself using them.

As a more generic and widely supported protocol, applications which do wish to leverage service discovery should probably opt for bonjour with the service "_nmea-0183._tcp"

Miscellaneous

What about baud rate?

The answer may be obvious but I have seen this question asked so it is worth addressing.  This isn't an issue.  Baud is simply the transmission rate used on a serial line. It is not a property of the data which is transmitted.  The slowest rates commonly used on IP networks today, wired or wireless, are many times faster than the fastest speeds NMEA data are commonly transmitted over serial lines. Generally speaking programmers don't have to worry about transmission rates for this kind of data over IP.

Is ethernet different from wireless?

As far as the average application programmer is concerned?  No.  In some cases the physical medium over which some kinds of marine data is transmitted can matter as discussed in the previous post.  At the kinds of data rates at which NMEA-0183 is transmitted the programmer should not need to care whether their application is running on a wired or wireless network.

Summary

For maximum interoperability with other applications, applications using the application layer of NMEA-0183 over IP should:

• Use UDP broadcast and TCP over IPv4 as transports
• Use ASCII strings as they would be sent over a serial line with no additional encapsulation, starting from an initial "!" or "$", ending with the sequence <CR><LF>.
• Use NMEA-0183 checksums
• Accept sentences terminated with <CR> or <LF> and ignore all subsequent data until the start of a new sentence
• Observe a maximum sentence length of 80 characters excluding the line termination characters
• Use a default but configurable port of 10110 for both UDP and TCP
• Ensure that TAG blocks, if not supported, are gracefully ignored without discarded their associated sentences

But it would also be nice if applications:

• Support  UDP multicast
• Support IPv6 as well as IPv4
• Send data as soon as usable information is available for sending: One sentence or all sentences forming part of a multi-sentence message in one datagram in the case of UDP, Nagle algorithm disabled in the case of TCP
• Use subnet broadcast addresses rather than 255.255.255.255 when sending IPv4 broadcast

RADAR over wifi: Not always a pretty picture

OpenCPN plugins support overlay of imaging from some Garmin and Navico RADAR units.  Unfortunately some users attempting to use these plugins over wireless networks on their boats have encountered issues. Not only does it not work very well, it can make the whole wireless network unusable.  Moreover where people bridge a wireless segment to their wired network (e.g. with a simple domestic access point/switch/router) the wireless network can become unusable when the radar is switched on even if nothing on the wireless side is using it.  What's going on and what can be done to resolve it?

Why RADAR imaging might not mix well with wireless

The problem is with the way images are transmitted by the types of RADAR which OpenCPN plugins support (and probably others).  These RADAR images are not transmitted using standard NMEA protocols: The data rate is too high even for NMEA 2000. Instead manufacturers transmit RADAR images using proprietary methods over Internet Protocol running on 100Mb/s copper ethernet. Crucially manufacturers tend to use UDP multicast, sending data to multiple receivers at once (as opposed to "unicast", or sending to a single receiver).  In a previous post I extolled the virtues of multicast for transmission of boat data.  It allows all computers who have subscribed to a given multicast group to see the same data without them being replicated to each client.  Sending RADAR data over multicast on an ethernet network as the manufacturer intends works well.  Sending it over wifi is another matter.

The issue is not simply that the data rates are too high to be handled by wireless networks.  Navico state that their RADAR transmits at "up to 8 Megabits per second" which seems to concur with what users have observed.  Surely this is within the capabilities of most modern wireless networks?

Unfortunately a wireless access point has a number of "known unknowns" to deal with when transmitting to associated stations on a wireless network including the distance of the station from the transmitter, objects in between and interference in the environment which makes the task more complex than it is with copper wires.  The IEEE 802.11 standards on which common wireless networks are based each define a number of data rates.  Transmissions at lower data rates tends to be more robust than those at higher rates. Typically an access point will dynamically adjust the rate at which it sends data to each associated station based on packet loss (where transmitted frames are unacknowledged by the receiver at the link layer).  The connection between a "150 Mb/s" access point and a "150 Mb/s" laptop may be considerably lower than 150 Mb/s if the laptop is far away from the access point with intervening obstacles and/or there is interference from other devices in the environment.

Multicast (and broadcast) data must be sent at a rate which all listeners can be assumed to reliably receive.  Standards dictate that multicast should generally be sent using a rate in the "basic rate set" (the set of rates which all clients are assumed to support). Although not universally the case,  many retail access points will use the lowest of these rates, often 1Mb/s.  This of course is the physical data rate.  Various overheads mean that the rate of transmission of useful information will be be even less than that.

On top of this, multicast over wireless is more prone to packet loss than unicast.  Unlike ethernet, unicast 802.11 frames are normally acknowledged by receivers allowing them to be re-sent if they are lost in what is an inherently less reliable medium than copper wires.  Multicast frames are not usually acknowledged so aren't retransmitted if a receiver loses them.  Another potential source of packet loss  with high data rate multicast is the buffering that occurs on an access point when some stations on the network are in power saving mode, although without information on the sizes of buffers involved in a given access point it is difficult to assess the likely impact of this.

Note that these issues don't mean that multicast is a bad choice for "normal" NMEA-0183-style data. In those cases data rates are far lower, should not significantly impact the wireless network, and by the repeated nature of transducer data, occasional dropped packets are not necessarily a problem.  Remember also that broadcast data suffers from the same problems on wireless as multicast.

Why does RADAR stuff up all your other wireless data too?

If it's only the multicast which is constrained to be transmitted at 1Mb/s, why is your other wireless traffic badly impacted?

When transmitting multicast data the access point is operating at the multicast rate (e.g. 1Mb/s). Not only is there likely much more RADAR data than is involved in the average browsing session, the rules of contention for the radio give the multicast data an unfair advantage.  The radio in the access point is likely spending most of its time at the lower data rate.

Why does wired RADAR stuff up your wireless data when only wired devices are listening to it?

Your boat network may use an access point with an integrated switch for wired devices or you may have a wireless access point connected to switch to which your wired devices are also connected.

A simple switch has no knowledge of what devices on the network need to receive multicast data, so it will send multicast (and broadcast) data out of all ports.  This will include an attached wireless network, even if there are no wireless devices interested in receiving the multicast data.

Preventing multicast getting onto your wireless segment

Some switches do offer a method of ensuring that they only send multicast data out of ports to which a device which wants to receive it is attached: "IGMP snooping".  Whenever a program tells a computer that it wants to receive a particular group of multicast data, the computer will send an Internet Group Management Protocol (IGMP) "report" message on the local network.  Such messages are intended for routers which forward Internet Protocol (IP) packets between networks (e.g. your local network and your Internet connection) but some switches (which are generally link layer devices so know little of the higher level protocols carried in their data link frames) can listen for these reports and use the information to decide which switch ports to send multicast data out of.

Many high-end switches will have this functionality. Many low-end devices won't.

Making RADAR work over wireless

Wireless networks are also a problem for those wanting to deliver IPTV (Television over Internet Protocol)  using multicast so the problem has received extensive attention.  A number of solutions have been proposed and implemented in various pieces of hardware.  Retail access points are less likely to offer a solution than "Enterprise Class" (i.e. unaffordable) devices.  Two relevant solutions which require only support from the access point are:

• Multicast-to-unicast translation.  Access points send multicast packets to clients requesting them via link layer unicast.  In other words, IP packets to multicast groups are delivered in unicast 802.11 frames to the stations interested in receiving them and as such these frames are delivered reliably and at maximum reliable speed.  This technique is variously implemented in some enterprise class access points using various methods.  It is one of the subjects in the (currently not commonly implemented) IEEE 802.11aa and related 802.11v standards where it is referred to as "Directed Multicast Service" (DMS).  Some implementations use IGMP snooping to determine which clients are interested in which multicast data.  With 802.11v the client can request multicast to be delivered over unicast with DMS: a cleaner solution as the OSI layers aren't so mixed up but I haven't been able to find any hint of an API for this.  DMS does not scale well which can be an issue for IPTV delivery, but on board where no more than two clients are likely to require RADAR data it is ideal.
• Setting the multicast rate. Some devices will allow you to explicitly set the "multicast rate".  Setting this to a high value might make using RADAR over wifi viable.  The downside is that as this is the minimum rate at which clients will be allowed to connect to the access point, the range at which the access point is usable will be reduced.  In the average GRP yacht this will be a non-issue. Once again, this is more likely to be a feature on higher end equipment but is more likely to feature in consumer units than multicast-to-unicast translation.  It may also be possible to implicitly alter the multicast rate by disabling certain modulation types.

Which access points will work?

Without a RADAR unit to test or a multiplicity of access points to test I can only report what might work based on Internet research and reader feedback.  Please do let me know if you have any relevant information.  This section will be updated as new information is received.

"Enterprise Class" products capable of multicast-to-unicast are not generally found on boats so high end devices from the likes of Ruckus and Cisco are not considered here.

OpenWRT

The popular open source alternative firmware for a number of routers implemented multicast-to-unicast in the recent "Barrier Breaker 14.07" release.

Apple Airport Extreme

This apparently allows the user to set the multicast rate.  I will endeavour to test the effectiveness of this using simulated data.

Navico wifi-1

The manual for Navico's wifi-1 module describes an option of interest:

Multicast-to-Unicast:
IGMP snooping. Convert Multicast data to Unicast one.The default setting is "Enable". 

I contacted Navico to clarify this.  It seems that the "Multicast to Unicast" statement is a mistake.  The wifi-1 does do IGMP snooping but only to keep unwanted multicast data off the wireless segment. According to Navico's product manager for the wifi-1 it does not do multicast to unicast translation at layer 2 and Navico do not support RADAR data over wireless.

SignalK

I'm one of the many people who believe that there is a desperate need for an open marine communications protocol.  Basing communications on an unofficial description of a closed protocol running over lower layers the original was never intended for is a far-from-ideal situation.  There are incompatibilities in implementations and no open, solid base to build other features of a protocol stack, like service discovery to eliminate much of the manual set-up involved in the current crop of ethernet/wifi based marine applications.

The GPSd project has developed a protocol for transmission of GPS and AIS data.  It is based on JSON (Javascript Object Notation) which offers a number of advantages.

• It is human readable which is widely regarded as being beneficial when debugging systems
• It is extensible
• It is easily assimilated into many modern high-level languages, facilitating rapid application development

On the downside it requires more space to transmit than a binary format, requires more processing to parse and most significantly requires dynamic memory allocation which is resource inefficient and complicates processing, although much of that will be hidden from programmers in higher level languages.  GPSd imposes some additional constraints on its JSON messages to allow for compact and efficient processing, notably a fixed maximum array size and disallowing the use of the NULL character: legal in JSON but traditionally a string termination character in C and some derived languages.

GPSd is currently maintained by Eric Raymond, one of the biggest names in the Open Source movement whose work in publicising the details of NMEA-0183 has facilitated much of the software development which has leveraged that still-technically-closed protocol.  It has a very large existing user base and would seem ideal as a starting point for a general NMEA-alternative protocol.

GPSd as it stands is not a general purpose marine data communications protocol and direct OneNet alternative.  Most significantly the protocol only documents GPS and AIS related information although extending to other types of data would seem entirely feasible.   The protocol is transport-agnostic.  The gpsd program itself implements a tcp server although an add-on program (gps2udp) can distribute gpsd data via UDP.  Being unconcerned by lower layers is certainly good software architecture design and poses no problem for the general use case of gpsd but it may not be ideal in a network of autonomous sensors providing updates over multicast UDP as advocated in a previous post.   Here the imposition of limits on message length in order to fit into a single datagram may increase reliability and simplify processing.

Other features not relevant to the core functionality of GPSd which might be considered desirable in a complete marine protocol suite would include a standard authentication method which does not require a connection-oriented link and some kind of service discovery mechanism.

Looking further to investigate whether anyone was working on a more generalised communications protocol I was pointed in the direction of the SignalK project.  SignalK, named after the single-flag signal Kilo ("I wish to communicate with you") is a concept being developed by a group which includes the developers of some popular open source marine projects.  It is, however, not a communications protocol but a data model for representing all information relating to a vessel and its environment.  It proposes the use of JSON to convey information between SignalK systems but conceptually this is not a standalone protocol but a native method of sharing data in the common model.  Although an example of the data representation envisaged for SignalK in the form of a GNSS object is given, the bulk of detail in the model has yet to be filled in with much of the discussion currently focusing on higher-level concepts.  The project seems geared towards the idea of a central server holding a data representation which can be communicated to other servers locally or on other vessels as well as client applications.  Project participants have argued that this does not preclude the concept of a distributed network of sensors providing independent asynchronous data updates but that area does not seem to be of primary concern.

Does SignalK contain the protocol I'm looking for?  It's too early to tell.  The project is intended to be far more than a drop-in replacement for proprietary NMEA protocols but could this ultimately be a hinderance to adoption?  Marine data communications has traditionally been very engineering-oriented.  Implementation of high level object-oriented data models are facilitated by high level languages and communications abstractions (typically over connection-oriented links) but are such concepts going to be readily adopted by those programming limited-resource embedded systems in C?  The future would appear to contain sensors with direct network attachment.   Not all anemometers or GPS units are on boats.  Should the way they communicate with the network be based on their place in a marine-specific data model?

SignalK is certainly an interesting project with some very capable programmers as participants but it is not a simple OneNet alternative and perhaps this may ultimately be a hinderance to wider adoption.