Translate

Saturday 6 September 2014

SignalK

I'm one of the many people who believe that there is a desperate need for an open marine communications protocol.  Basing communications on an unofficial description of a closed protocol running over lower layers the original was never intended for is a far-from-ideal situation.  There are incompatibilities in implementations and no open, solid base to build other features of a protocol stack, like service discovery to eliminate much of the manual set-up involved in the current crop of ethernet/wifi based marine applications.

The GPSd project has developed a protocol for transmission of GPS and AIS data.  It is based on JSON (Javascript Object Notation) which offers a number of advantages.

  • It is human readable which is widely regarded as being beneficial when debugging systems
  • It is extensible
  • It is easily assimilated into many modern high-level languages, facilitating rapid application development
On the downside it requires more space to transmit than a binary format, requires more processing to parse and most significantly requires dynamic memory allocation which is resource inefficient and complicates processing, although much of that will be hidden from programmers in higher level languages.  GPSd imposes some additional constraints on its JSON messages to allow for compact and efficient processing, notably a fixed maximum array size and disallowing the use of the NULL character: legal in JSON but traditionally a string termination character in C and some derived languages.

GPSd is currently maintained by Eric Raymond, one of the biggest names in the Open Source movement whose work in publicising the details of NMEA-0183 has facilitated much of the software development which has leveraged that still-technically-closed protocol.  It has a very large existing user base and would seem ideal as a starting point for a general NMEA-alternative protocol.

GPSd as it stands is not a general purpose marine data communications protocol and direct OneNet alternative.  Most significantly the protocol only documents GPS and AIS related information although extending to other types of data would seem entirely feasible.   The protocol is transport-agnostic.  The gpsd program itself implements a tcp server although an add-on program (gps2udp) can distribute gpsd data via UDP.  Being unconcerned by lower layers is certainly good software architecture design and poses no problem for the general use case of gpsd but it may not be ideal in a network of autonomous sensors providing updates over multicast UDP as advocated in a previous post.   Here the imposition of limits on message length in order to fit into a single datagram may increase reliability and simplify processing.

Other features not relevant to the core functionality of GPSd which might be considered desirable in a complete marine protocol suite would include a standard authentication method which does not require a connection-oriented link and some kind of service discovery mechanism.

Looking further to investigate whether anyone was working on a more generalised communications protocol I was pointed in the direction of the SignalK project.  SignalK, named after the single-flag signal Kilo ("I wish to communicate with you") is a concept being developed by a group which includes the developers of some popular open source marine projects.  It is, however, not a communications protocol but a data model for representing all information relating to a vessel and its environment.  It proposes the use of JSON to convey information between SignalK systems but conceptually this is not a standalone protocol but a native method of sharing data in the common model.  Although an example of the data representation envisaged for SignalK in the form of a GNSS object is given, the bulk of detail in the model has yet to be filled in with much of the discussion currently focusing on higher-level concepts.  The project seems geared towards the idea of a central server holding a data representation which can be communicated to other servers locally or on other vessels as well as client applications.  Project participants have argued that this does not preclude the concept of a distributed network of sensors providing independent asynchronous data updates but that area does not seem to be of primary concern.

Does SignalK contain the protocol I'm looking for?  It's too early to tell.  The project is intended to be far more than a drop-in replacement for proprietary NMEA protocols but could this ultimately be a hinderance to adoption?  Marine data communications has traditionally been very engineering-oriented.  Implementation of high level object-oriented data models are facilitated by high level languages and communications abstractions (typically over connection-oriented links) but are such concepts going to be readily adopted by those programming limited-resource embedded systems in C?  The future would appear to contain sensors with direct network attachment.   Not all anemometers or GPS units are on boats.  Should the way they communicate with the network be based on their place in a marine-specific data model?

SignalK is certainly an interesting project with some very capable programmers as participants but it is not a simple OneNet alternative and perhaps this may ultimately be a hinderance to wider adoption.

Friday 7 February 2014

Raymarine Reply

kplex user Cardo posted a link to the previous article on Raymarine's Facebook page.  Not only did this elicit a response from Raymarine's product support manager, he also copied the response to me in email.

The main part of the reply was a defence of Raymarine's wireless security, which I didn't criticise, saying only:
All wireless security was off on the MFDs, presumably for show convenience as according to the manuals this is not the default. 
My actual criticism was of the apparently redundant services running on the MFDs, but Raymarine's response is to assure us that these are all either part of "show mode" or deliberately enabled as part of the system's operation.  The services have been tested and are all "benign".  This does slightly miss the point that the objective of running  fewest possible services is not to avoid those with known problems but to minimise the threat from the problems which you don't know about yet and can't test for.

The other part of my criticism and what actually prompted me to write in the first place was the way in which mails to Raymarine simply weren't replied to.  Did they really have to wait for someone to post something on social media to get back to me rather than just replying to me in private in the first place?

I'll have to wait to see how one of these systems looks different in "normal" operation, but if anyone has one of these MFDs please do sick nmap on it and post the results as a comment.

My guess about this being an off-the-shelf OS with services accidentally left running was incorrect.  These services were deliberately enabled.  I still find the idea that telnetd and sshd and ftpd are all required intriguing.  Perhaps also whatever processes require NFS file sharing could enable the processes only when they have something to share and leave them disabled the rest of the time?

As well as my Facebook reply, I also replied (at slightly greater length) by email and got a response re-iterating that all running services were necessary and that reasons why could not be discussed as the workings of Raymarine's software were confidential.  The issue of default passwords which I raised in both replies was not addressed.

So how much of a problem is this?  Recent reports have publicised "smart" devices such as fridges and TVs being targeted to be used as spam relays.  A Raymarine MFD isn't a dumb piece of electronics, it's a full blown computer.

To a large extent though Raymarine are right: this is a lot of fuss about nothing.  The effect of idle, unused services like this on performance will be negligible.  On the security front their MFDs have been engineered as a gadget which is not designed to be deployed in a "hostile" environment and in the main this is a fair assumption.  The wireless networking on Raymarine's MFDs are not designed to play well with an existing wireless network which may be connected to the network: They create an isolated one for the specific reason of communicating with devices running Raymarine's own and partners' apps.  Bad news for owners who then need to flip between wireless networks to view their instruments and browse the web, but also bad news for potential malware writers wanting to turn your MFD into a spam bot.  Malware which first infects a user's PC, then targets the MFD (probably a different OS and architecture) then adjusts the routing table on the off chance that the user has connected their MFD's ethernet to an Internet connected network is, frankly, tenuous.  Most consumers would doubtless prefer rich functionality over unnecessarily paranoid security.  Really this is not something to get worked up about.

It's only a matter of time however before connecting a boat's network to the Internet becomes the norm and MFDs start to be able to leverage data from the Internet in their operation (Grib downloads? Software updates?).  Time will tell if the MFD of tomorrow looks a little more secure to the casual observer.

Thursday 30 January 2014

Raymarine: Security FAIL, Customer Services FAIL

Three weeks ago I visited the London boat show with my laptop.  My aim was try and test the GoFree extensions to kplex but sadly Navico weren't at the show.  Stopping by the Raymarine stand I tried to confirm what I'd been told that the new Raymarine MFDs don't have an IP-based nmea-0183 server like the competition.  Most of my boat's electronics are Raymarine, but my plotter is one of their older models which I may soon consider updating.  Sadly there was no-one on the stand with any knowledge in this area so I asked the rep if he'd mind if I connected my laptop to an MFD to see for myself what services were available.  I was told to help myself.

All wireless security was off on the MFDs, presumably for show convenience as according to the manuals this is not the default.  After connecting up to an e165 I listened for traffic (none), so with the consent of the rep on the stand I did a basic port scan (tcp only) to check for services.

PORT      STATE SERVICE
21/tcp    open  ftp
22/tcp    open  ssh
23/tcp    open  telnet
111/tcp   open  rpcbind
2049/tcp  open  nfs
6000/tcp  open  X11
6668/tcp  open  irc
8080/tcp  open  http-proxy
50000/tcp open  ibm-db2 
 
I  couldn't help but wonder whether services running on assigned ports were really what those ports are officially assigned to. The ftp port does indeed seem to have a copy of tnftpd running which supports anonymous accesss.  Dropbear sshd is running on 22 and both it and the telnet daemon running on 23 give a login prompt.  There is indeed something serving http on 8080.  I didn't poke the (apparent) X11 or irc ports but rpcbind is really a portmapper:

   program vers proto   port  service
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  52498  mountd
    100005    1   tcp  35944  mountd
    100005    2   udp  52498  mountd
    100005    2   tcp  35944  mountd
    100005    3   udp  52498  mountd
    100005    3   tcp  35944  mountd
    100024    1   udp  44953  status
    100024    1   tcp  52967  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  49439  nlockmgr
    100021    3   udp  49439  nlockmgr
    100021    4   udp  49439  nlockmgr
    100021    1   tcp  40926  nlockmgr
    100021    3   tcp  40926  nlockmgr
    100021    4   tcp  40926  nlockmgr  

Although running mountd, no network file systems are apparently exported.

If you installed a standard Linux or other UNIX system from CD 15 years ago it would have come with all the services you would probably ever need running and waiting to be used.   In a secure or business environment it would then be someone's job (often mine as it happens) to customise the installation process so that everything that wasn't needed was turned off.  These days manufacturers are more security conscious and deliver systems with most services turned off by default and leave it to the customer to enable those which they den necessary.

There's a lot of apparently unnecessary things here.  Everything necessary for sharing file systems but no files shared.  Notably, the telnet service is running. Telnet used to be the predominant protocol for command line login to systems up until the turn of the last century.  Its problem is that everything, including passwords, is sent unencrypted so passwords are easily snooped by anyone listening in.  Consequently it was superseded by the "secure shell" (ssh).  ssh encrypts all communications and also performs other functions like file transfer and some nifty port forwarding.  Use of telnet is deprecated these days so it is a surprise to see it running, especially when the ssh service is also running: Hard to imagine what telnet is needed for when ssh is there.  Ssh also performs file transfer functions, so the ftp service (also running) might also appear to be unnecessary.

Maybe there was a reason for all of this, but it seemed a sound hypothesis that Raymarine had simply concentrated on their app and forgotten to turn off unnecessary OS services.  This is generally considered bad practice both from an efficiency point of view (idle services don't cause much system overhead, but they're still unnecessary) and a security perspective.  They may not have any known security flaws, but they may have flaws which will be publicised at some point in the future.  The fewer services you run, the less chance that you are running something that could give access to an attacker.

I posted a question on Raymarine's support forum saying that I'd noticed apparently superfluous services running on their MFDs and was this perhaps an oversight.  The post was immediately removed and I was contacted by a moderator who told me to post the question privately using their "Ask Raymarine" contact form.
This I did and received a mail telling me that the query had been forwarded to their engineering department and I would receive a reply in due course.

At this time  I started thinking about writing a post about the differing approaches of the established computer industry and the marine electronics industry to software development and network security but thought it appropriate to wait for a response from Raymarine.  If there was a sound explanation for these services, or if the issue was simply an acknowledged oversight which would be corrected in the next release then I didn't really have a story: errors happen in all organisations and don't necessarily indicate a problem of ethos.  Failure to react on the other hand would indicate a culture that is worth questioning.

A week went by with no reply so I mailed again, asking when I could expect a reply as I was considering writing an article related to my observations.  This got the attention of their press office and I was quickly contacted to ask about my writing qualifications, what I was writing about and who I was writing for. I gave them the information they asked for and asked about the possibility of interviewing their head of product development about the approach to security in current future network-connected products, including any proposed OneNet security strategy.

I received no reply.  Another week passed.  I mailed the press office back to ask how things now stood and was told that as my query did not relate to a system I had installed on my boat, I could not expect to receive a reply.

I left it another week before posting this, just in case they decided to answer my original query. They didn't.

Clearly I am personally disappointed.  I wasted time following a procedure I was told to follow by a Raymarine representative, was told to expect a reply, was grilled for information about my background and then simply ignored.  Not encouraging on the customer services front.  Not good PR either, as a simple reply wouldn't have encouraged me to write about the company negatively.

It does make me question their procedure for handling security reports, which large established software vendors will have.  Do they have one? Did I not receive a reply because they believed that this was a non-issue but couldn't be bothered to reply to me, or because the question never reached the right people?

The marine electronics industry has traditionally dealt in closed systems which people did not try to deliberately break for fun and profit.  Are they now blundering like naifs into the connected world with the same approach?  Unfortunately the system is no longer guaranteed to be closed.  It may be connected to the Internet.  A customer might unwittingly connect a malware infected PC to it.  The person on the next boat may simply enjoy cracking wireless systems.

For the past quarter century the mainstream computing industry has operated with the constraint that security cannot be simply an afterthought.  What customer-impacting events will it take to bring the marine electronics industry into the 1990s?

Thursday 16 January 2014

We come OneNet

After hopefully clarifying IPv6 for the uninitiated in yesterday's post we can move on to OneNet.

OneNet is the NMEA's next-generation protocol for marine data "over ethernet".  Although the standard is both closed and not yet published, publicly available descriptions suggest that it has an application layer similar to NMEA 2000 but uses an Internet Protocol network layer.   The use of Internet Protocol means that OneNet data will be easily transportable over computer networks to PCs, tablets, phones etc.. Given that NMEA standards include the entire stack from the physical to application layers, use of non-ethernet networks, the Internet or wired networks not using OneNet connectors might be outside the standard but will almost certainly be an option available to consumers.

Use of Internet Protocol was almost a given for a new NMEA data communications standard.  The interesting thing is the version used: It's IPv6.

Why IPv6?

In many respects IPv6 is the ideal protocol for this application. A boat network of transducers and other devices is classic "Internet of Things" territory.  IPv6 provides a method for autoconfiguration thereby obviating the need for the NMEA to develop one.  IPv6 mandates supports for many features of relevance here which although now widely supported are strictly speaking add-ons to IPv4: multicast, security enhancements and quality of serviceIPv6 flow labels might potentially be leveraged by the NMEA for interaction with the underlying datalink layer to provide reserved bandwidth for lossless delivery in control links.  NMEA 2000 took many years to become widely adopted.  With the first OneNet products at least 2 years away, widespread market penetration might not occur until such time as IPv4 is starting to retreat towards obsolescence.    Using IPv6 for a product which may not be mainstream for more than 5 years avoids the pain of having to change the network layer at a later date.

Some people may have concerns that OneNet, being IPv6, will not interact with their existing computing devices.  As mentioned in the previous post, almost all modern consumer computing devices support IPv6.  Marina wifi and mobile providers are rarely configured to transport IPv6, but this is a good thing.  Ideally people's boat data networks would be entirely independent of the network they use for checking the weather and looking at LOLcats.  In practice it won't be, because the consumer will want to use their PC/tablet to look at both transducer data and cat videos.  A logical separation of networks is possibly the next best thing to a physical one.

On the money-spinning front, while IPv4 remains the predominant global network protocol, use of IPv6 on boats creates a market for "marine" NAT64 products and other devices to facilitate working with both IPv4 and IPv6.  "NMEA certified" networking equipment can be sold to those wary of purchasing cheap domestic alternatives with IPv4-only user interfaces.

My opinion is that IPv6 is undoubtedly the right solution to this problem, but that doesn't make it without challenges

The Challenges

Current marine software rarely supports IPv6

Many people would like to be able to just connect their laptop to their instruments over the network. Their marine software however (e.g. chartplotter program) will likely only take data from an IPv4 source.

Actually most marine laptop/tablet software uses an NMEA 0183 application layer data which is not what OneNet would be providing.  To use OneNet, the application would need updating whatever the network layer used. Adding support for IPv6 should be easy by comparison.

Embedded IPv6 stacks are not as mature as IPv4

Although the Linux and *BSD network IPv6 stacks are in widespread use, some people have concerns about the maturity of IPv6 stacks for microcontroller-based devices.  A lack of widespread use allows the possibility of as-yet unknown problems which can severely impact product development times.  Moreover selecting a 3rd party stack to use will involve a repeat of the research which a manufacturer would have already done in bringing IPv4-based products to market. I have insufficient experience in embedded product development to know how much of a problem this will be.  Use will obviously bring the maturity of IPv6 stacks into line with that of IPv4 stacks.

Staff skills

IPv6 expertise exists in many areas of IT but it is the exception rather than the rule.  Programmers working for marine technology companies are possibly less likely than those working for network hardware vendors to posses extensive IPv6 experience.  None of the large marine electronics companies can be emailed over IPv6 or have IPv6 accessible web sites, which might suggest that their IT infrastructure staff also lack experience in this area. Lack of experience implies increased training and development costs, greater risk and a longer time to market.

The unknown

An IPv4 network layer for OneNet looked like a safe choice. Challenges would be readily anticipated from existing development work and experiences from existing NMEA-over-IPv4 work with IEC 61162-450 and various vendor-proprietary solutions.  IPv4 would be familiar and readily accepted by consumers and manufacturers alike. IPv6 is not only new to the marine industry, practical use in an application like this is ground breaking.  As such the risks may be harder to quantify and in business, unquantifiable risk is bad.  What makes an excellent technical decision may not be such a strong business decision.

Conclusion

Two things I seem to spend a lot of time doing are IPv6 network coding and Sailing.  It's no surprise that I'm delighted that IPv6 will form the network layer for OneNet.  From a technical perspective I strongly believe it to be a brilliant decision and a bold one.

The implications of this choice extend beyond the boating industry.  If OneNet comes to fruition it will be one of the first true industry-led steps towards the much discussed Internet Of Things.  Demonstrable success would diminish risk associated with similar initiatives in other industries and drive adoption.  IPv6-connected marine devices would create business opportunities for network connected services which could give impetus to adoption of IPv6 connectivity and change the way in which we all interact with the Internet.

From a business perspective it seems like a brave move.  It would appear to carry more business risk and short-term cost than the safer IPv4 alternative.
Technology is more my department than business, but an industry taking the long view to the ultimate benefit of the consumer makes a refreshing change.

Another two years or more is a long time to wait for OneNet, even assuming things stay on schedule.  Not only do the risks imply delays in themselves, but caution may delay development efforts in some organisations.  The CTO of one well-known marine electronics company told me that his organisation were adopting a "wait and see" policy towards the new standard.

It will be interesting to see what proprietary or open standards might fill the gap between then and now and whether one of those regents may ultimately pretend to OneNet's throne.

Wednesday 15 January 2014

WTF is IPv6?

The NMEA have announced that their next generation data protocol, OneNet, will run over IPv6 (Internet Protocol, version 6).  Before considering what that means for boaters, what the heck is IPv6 anyway?

"Internet Protocol" is a method of transporting data across computer networks. It's like a shipping container (or "packet") with a label on it for information such as the address of the sender and that of the recipient.  For the past three decades we've mostly been using Internet Protocol version 4, or "IPv4" (though more commonly just "IP").  As the number of computers connected to the Internet expanded rapidly during the 1980s, it became apparent that the "address" fields of this shipping container's label (or "header") were soon not going to be big enough to give a unique address to every new potential sender and recipient of Internet data.  A new type of shipping container with a different label with space for much longer addresses was required so the IETF started work on a new version of Internet Protocol, eventually dubbed IPv6 (there was no IPv5).

The differences between IPv4 and IPv6 extend to more than just address size.  IPv6 incorporates new mechanisms for auto-configuration (devices assigning addresses to themselves) and standardises many of the features which have been bolted on to IPv4 over the years.  Ultimately though  these protocols both perform much the same core function: shipping data across networks.  However they behave like two independent courier services which although they may put their containers onto the same trucks, trains, ships and planes in the course of delivering a packet, each courier uses postal codes on the address labels which are meaningless to the other courier.  You can do tricks like putting an IPv6 packet inside an IPv4 packet for part of the route where the devices connecting networks ("routers") only understand IPv4, but both sender and receiver must ultimately use the same addressing scheme.  A single device may have an IPv6 address and an IPv4 address, in the same way that a company can have an account with two courier companies and use whichever one delivers to the whoever they need to send something to.

The first IPv6 standard appeared in 1995.  During that time we've regularly seen news items predicting that we would soon be running out of IPv4 addresses.  Google tracks the percentage of users accessing their services over IPv6 (drilling down by country is interesting and occasionally surprising). Today it's only 2.3%. So why is IPv4 still ubiquitous and IPv6 almost unknown to anyone other than network engineers?

In the past 20 years a number of effective mechanisms have been adopted to slow the rate at which IPv4 addresses are being depleted.  Despite the hype to the contrary, IPv4 addresses are still obtainable and will remain so for a few years yet.  The bottom line is that for most Internet-using businesses there simply isn't a business case for using IPv6: IPv4 does what they want, so why endure the cost and risk of change?

Due in part to US government mandates that all computer equipment they purchase must be capable of running IPv6, all mainstream operating systems (Windows, MacOS, Linux, Android, iOS etc.) are capable of using it.  This does not mean other software is.  Your Mac may be quite capable of sending and receiving IPv6, but at time of writing, OpenCPN doesn't understand if you tell it to take a data feed from an IPv6 address. Users in many countries (including the UK) will have a hard time finding a domestic Internet Service Provider whose routers have been configured to correctly forward IPv6 packets even though the machines themselves are capable of it.  Without a bit of trickery, home computers can't normally speak to remote web servers using IPv6.  Even if they could, practically all web sites are reachable using IPv4.  Only a small minority can be reached via IPv6. Most computers which are configured to use IPv6 are also configured to use IPv4 to communicate with sites with IPv4-only connectivity.

Internet Of Things

The buzzphrase "The Internet of Things" is used to describe the interconnection of many different kinds of uniquely identifiable objects which are able to send and/or receive data.  Those objects might be toasters, fridges, transducers or cows.  Although a concept rather than an implementation and therefore independent of any particular technology, the Internet Of Things is often associated with IPv6 because of the very large number of objects (many billion) potentially requiring identifiers on the network.  Other properties of IPv6, particularly its self-configuration mechanisms, make IPv6 particularly appropriate for an internet of things.

The phrase has recently become so over-used as to be deeply annoying.

Key points for the average leisure boater

  • IPv6 does the same kind of things as IPv4 and end users shouldn't really notice or care about the difference
  • Yes your Mac/PC/phone/tablet can use it
  • Yes it can probably use IPv6 on your existing boat network at the same time as IPv4
  • Devices can only talk to each other using IPv6 if they both have an IPv6 address. Devices can only talk to each other using IPv4 if they both have an IPv4 address
  • A device may have an IPv4 address and an IPv6 address and use either protocol
  • No your Mac/PC/phone/tablet probably isn't configured to use IPv6
  • No, your marina wifi / mobile data service probably hasn't been configured to transfer IPv6 between you and the Internet
  • No, apart from "big" applications (e.g. firefox) and those produced by Microsoft, Apple, Google etc., the application you're using doesn't necessarily understand IPv6 addresses you type in.
  • No your MFD or other marine devices don't work with IPv6: Those companies struggle to get IPv4 right. There is nothing preventing updated firmware from changing that in the future though
  • You can, of course, put your nmea-0183 data on your network using IPv6 with kplex. Just thought I'd mention it
  • No, unless you really want to you probably shouldn't waste your time doing it.
The only reason you really need to know anything at all about IPv6 right now is so I can write about NMEA OneNet in the next post.

Friday 10 January 2014

Cape Multicast: The Logical Route

UPDATE September 2015.  Practical layer 2 considerations have lead me to change my conclusion on the optimum practical data transmission method over wireless networks as described in this later post.

In my last post I touched on the lack of standardisation in the way NMEA-0183 data (strictly speaking, data in a format based on the application layer of NMEA-0183, but we'll use the shorter phrase for readability) is transported over IPv4 networks with current marine software and devices.

In this post I argue that the way in which most devices and software currently do this using TCP or broadcast UDP is not optimal, propose an alternative solution and discuss the viability of its implementation.

For the sake of this discussion, the network is assumed to be the de facto standard of IPv4.

For the sake of brevity I will assume some familiarity with TCP/IP and UDP/IP.  If anyone comments to request it I will elaborate on terms more extensively in a later post.

How things stand

Most devices and software which are designed to send or receive NMEA-0183 data over an IP network do so by one or both of unicast TCP or broadcast UDP.  Devices which are data sources on the network (e.g. NMEA-to-network converters) often implement a TCP server though in some cases programming is rudimentary and the server can only support a single client.  Data consumers (e.g. mobile apps) using TCP normally act as a client, initiating a connection to the server.  Consumers expecting UDP can normally receive unicast or broadcast data as programmers of these apps rarely bind sockets to a particular address.

Very few devices or programs explicitly designed to send or receive NMEA-0183 data do so using UDP multicast.  kplex will, and as of beta version 3.3.1303, so will OpenCPN.  Software and devices able to send to a configurable UDP address can normally send to a multicast address so long as the underlying network stack supports multicast, which it will in the case of a modern operating system (Linux/Windows/MacOS/*BSD).  Software and devices will not be able to receive NMEA-0183 over UDP multicast without explicit support in their code, although such support is normally trivial.

TCP

Why you want it

TCP has the following advantages in this application
  • With no clients connected, a TCP server consumes no bandwidth.  If your server sends unsolicited UDP it neither knows nor cares if any clients are listening: it sends it anyway.  This increases power consumption on many types of network if no clients require this data.  If your network only intermittently has a client device wanting to receive NMEA-0183 data, TCP may be the most power-efficient solution.
  • For Internet NMEA-0183 servers such as ones streaming AIS info for various ports, TCP is the most viable option.  Multicast over the Internet is rarely an option due to configuration required on intervening routers.  Broadcast or unicast UDP would require a separate protocol to request and terminate a datastream from the server and would generally encounter difficulties with firewalls.
  • Reliably implemented in embedded stacks
  • Currently supported by most applications

Why you don't

  • That reliable delivery you thought was a good idea? In this application it probably isn't.  Reliable transmission is not a feature of native NMEA-0183 over serial lines or AIS over VHF and no application should break as a result of a lost sentence. A reliable transport mechanism is not required if reliability is provided by the application layer protocol.  In this case, transducers are constantly transmitting their status with the most recent data.  If a UDP packet is dropped the result is that an application will not be updated with the information it contained but will be updated with more recent information the next time that sentence is transmitted, possibly 1-5 seconds later.  If a TCP packet is dropped, no subsequent packets (sentences of any kind) will be delivered on that data stream to the application until the packet loss has been detected by TCP and the dropped packet has been successfully retransmitted. Instead, any data sent between the dropped packet and the retransmitted one will be queued on the receiver pending the ability to deliver data in the order it was sent: an unnecessary requirement here.  The duration of the delay will be influenced by many factors but if it exceeds the frequency of transmission of the data type in the lost packet, the result could be that a subsequent update is delayed waiting for TCP to retransmit the (now obsolete) dropped packet.  This will happen in the case where a network cable is disconnected for more than a second, interrupting the flow of data from a transducer outputting at 1Hz.  Another case where it should happen is if that 1Hz data is the only thing being sent over a data stream.  RFC 6298 gives a minimum retransmission timeout of 1s so in the absence of other packets being sent to trigger a fast retransmit, TCP will not time out waiting for the dropped packet to be ACKed until after the subsequent packet has been sent.  In practice, most OSes implement a lower minimum RTO and in any case, the average NMEA-0183 stream may carry more traffic, triggering fast retransmits.  It could be argued that in the case of AIS data where a dropped packet containing a report from a vessel may not be updated for minutes, TCP's retransmission may be a net benefit but in general, TCP's "reliability" is not beneficial in this application.  If a LAN is heavily congested for long periods, it is far better to randomly drop data leading to graceful degradation of performance of the end user application than to queue a backlog of stale data.
  • Protocol overhead.  In addition to the actual NMEA data transmitted a TCP connection requires packet exchange in order to set up a connection, packet exchange to take it down and acknowledgements of data received to be sent. Loss of the latter may lead to unnecessary retransmission.  In practice protocol overhead is not excessive: data streams are long lived meaning that connection establishment and termination overhead are negligible and not every packet needs ack-ing.
  • Data are prone to additional delays if programmers neglect to disable Nagle algorithm on sending sockets.  There is no need for this to be an issue, but marine electronics companies do not have a good track record of software excellence. UPDATE: In testing this with OpenCPN I realised that most modern operating systems seem to have delayed ACK timers of 200ms or less rather than the traditional 500ms, reducing any impact from this.
  • Data replication.  Any unicast protocol (as TCP inherently is) requires each piece of data to be copied to each client.  Two clients necessitate double the data on the network as opposed to a single client. Three clients triple etc.  In practice the small number of network NMEA-0183 clients likely to be required on the average pleasure boat is unlikely to exceed the multiple by which network capacity exceeds the capacity of a 38.4k serial line, so whilst wasteful, should not present an obvious problem.
  • If a server is ungracefully shut down (e.g. the power removed) and restarts, a receive-only TCP client may not realise for some time (or ever if keepalives are not enabled) and data will simply seem to stop, where a UDP-based solution would continue to function as soon as the server came back on line.
  • Requires users to know the network address of a server in order to configure a client, which can be problematic on links with dynamically assigned addresses.  This might be overcome by use of a service discovery protocol (e.g. bonjour, ssdp or a custom protocol).

Broadcast UDP

Why you want it

  • Doesn't suffer delays or protocol overheads associated with TCP
  • No requirement to replicate data: Many clients read the same data packet
  • Available on the types of network used in this application
  • Reliably implemented in embedded stacks
  • Currently supported by many applications

Why you don't

  • Not a good solution when crossing network boundaries as will be dropped by routers unless they are configured to do otherwise. Not usually a problem since common applications use a single link.
  • Inappropriate for Internet use
  • Can be challenging to program correctly for bi-directional communication with standard APIs.  Applications need a method to avoid re-reading their own broadcasts and possibly creating broadcast storms. OpenCPN does this using interface priorities.
  • Explanation of netmask/broadcast concept needed in documentation if UI  requires user to input destination address manually
  • Requires some processing by all devices on a link regardless of whether any are interested in the data being sent.  This is the big problem with broadcast and why it is generally not considered a good solution in modern networking applications.

Unicast UDP

Why you want it

  • For streaming data to a single, known, always-connected destination it is ideal.  None of the protocol overhead or timing issues of TCP yet easily traverses networks and is universally implemented in IP network stacks.  An example might be streaming data to Marine Traffic servers.

Why you don't

  • Worst of all worlds in most other respects
  • Data duplication required with multiple clients
  • Server must know addresses of all clients
  • Additional protocol required for clients to dynamically request a data stream
  • Wasted traffic if server is pre-configured to stream data to fixed unicast addresses which may have no attached client

Multicast UDP

Why you want it

  • Doesn’t suffer delays or  protocol overheads associated with TCP
  • No requirement to replicate data: Many clients read the same data packet
  • Available on the types of network used in this application
  • Unlike broadcast, sending requires setting of no special flags (assuming a correctly functioning network stack)
  • Easier to implement bi-directional many-to-many communication with standard networking APIs than with broadcast
  • Can be efficiently routed across networks
  • Unlike broadcast, doesn't (in theory) cause processing overhead on systems not interested in receiving those particular multicast data
  • Reliably implemented in all modern operating systems (Linux/MacOS/Android/IoS/Windows)

Why you don't

  • Not currently supported for receiving by many applications.  kplex and OpenCPN are two which do
  • Multicast routing across networks requires support on routers which will likely be unavailable across the Internet or on a small home/boat network, limiting effectiveness to a single link (like broadcast) for many users.
  • Not a required part of IPv4, although implemented for more than a decade in almost all operating systems
  • May not be implemented in some basic embedded/microcontroller stacks

So what's the answer?

For the most part, TCP is the wrong protocol for delivery of NMEA-0183 data over a boat LAN.  The protocol's bandwidth throttling and reliability mechanisms are not required by the nature of the application level data and can have a negative impact on the timeliness of data delivery to client. The data required on the network is a linear function of the number of clients using those data as opposed to a fixed amount as with UDP broadcast and multicast.

TCP is undoubtedly the best choice for requesting data from a server on a remote network as UDP would require an additional protocol for initiation and termination requests.  Although the best choice for servers streaming port AIS information which in turn is often useful for debugging marine data applications without another source of live or simulated data, this facility should not normally be required for navigational purposes.

Unicast UDP only has utility when sending data to a known endpoint on a remote network.  For navigation data within a boat's network, it has little to recommend it over the alternatives.

Of the current widely implemented solutions, broadcast UDP is perhaps the most appropriate.  Clients don't need to know a source address (only a port) so it is useful where servers are dynamically configured. However, its only advantages over UDP multicast are:
  • Current acceptance in the market
  • Possible lack of multicast support in some embedded network stacks
Manufacturers will at some point need to address the latter point (if it even turns out to be a problem) with the 3rd party network stacks they currently use.  Service discovery protocols which are increasingly being used in marine applications (e.g. in GoFree, OneNet) require multicast to function correctly.  This leaves only the problem of acceptance in the market which will be addressed in a later section.

Multicast would be used for this application in any commercial computer networking solution and indeed is reportedly the choice for data delivery in formal marine data networking standards IEC-61162-450 and NMEA OneNet. It is possibly only unfamiliarity with IP networking concepts in the marine electronics world which has led to broadcast being adopted.  Multicast has the advantages of broadcast but without the unnecessary processing overhead imposed on devices attached to the network which are not interested in receiving data of this sort.  The receiving group is a simple address which the user does not need knowledge of netmasks to determine.  It could be argued that multicast does not always remove unnecessary processing overhead: The IP-multicast to ethernet-multicast address mappings are not one to one and coarseness of multicast filters on various network cards, particularly at the lower end, may lead to unnecessary CPU interrupts caused by irrelevant multicast traffic.  Processing load is, however, reduced by using multicast rather than broadcast.

Does any of this even matter?

Not much.  Boat networks are small with few attached devices, perhaps the majority of which will be interested in  NMEA-0183 data and the rest will be negligibly affected by using broadcast rather than multicast.  It is unlikely that users will notice a difference no matter which delivery method is used.  This doesn't mean that it's is a good reason to choose a theoretically inferior solution because in practice there is no observable difference.

Conclusion and Stratgey

Multicast UDP is the most technically correct of the transports considered here for NMEA-0183.  The main issue is acceptance in the market place.

From the  product development angle this is easily addressed for products currently supporting unicast or broadcast UDP: Supporting multicast, in addition to unicast and/or broadcast is simple. If the sending/receiving UDP address is not already configurable, the UI needs to be updated to make it so.  After this only trivial code changes are required if a user specified address is multicast: When the data connection is initialised if the user-supplied UDP address is a multicast address, the program simply doesn't set the broadcast option on the connection (SO_BROADCAST for the sockets API) if it needs to send data.  If it needs to receive data, it simply joins the relevant multicast group, a single function call in most APIs.

For multicast to be useful, all clients requiring NMEA-0183 data should support it.  There is no point in running multicast and broadcast together on the same network: this has no advantage over broadcast alone.

For new applications all it requires for developers to implement multicast as a transport is awareness as the additional development cost is negligible.  For modification of existing applications there is only justification if there is demand, and there will only be demand if other apps support it. We've started the ball rolling with openCPN, perhaps it's time to spread the word.

But obviously an open standard would be nice.

Wednesday 8 January 2014

The Curious World of Marine Data Networking

It's a rum do this marine networking business. For someone who has spent the past quarter century dealing with open Internet protocols the strangeness is not so much in the archaic technology of the standards themselves but the way in which they are defined and administered.  The IETF has a variety of funding sources but one of them isn't selling the standards which they define.   Their biggest potential cost, the time of those involved, is funded by the organisations which employ them.  Corporate altruism aside, it makes good business sense to fund a seat at the decision making table.

In the world of marine data networking, standards are controlled by the US-based National Marine Electronics Association (NMEA).  In data networking, these standards cover the entire stack in the OSI model, from physical to application layers.  The International Electrotechnical Commission (IEC) formalises standards for the transmission of marine data based on the NMEA protocols (IEC 61162).  These are aimed at commercial vessels legally bound to conform to IMO requirements.

Unlike Internet standards, marine networking standards are not publicly published.  Anyone wishing to understand or implement the protocols has to buy the standards documents.  Not only this but the NMEA keep strict control of their intellectual property.  The documents aren't redistributable in any form which reportedly includes software source code.  Consequently, NMEA protocols cannot in theory be implemented in open source software.

Open source evangelist Eric Raymond has frankly expressed his opinions about this method of standards control.  Not being familiar with US intellectual property laws and keen to not be blinkered by possible hyperbole from those with hard line views on software freedom, I emailed the NMEA in an attempt to clarify their position on the use of legitimately purchased copies of their standards in open source software.  I never received a reply.

The NMEA has other sources of revenue besides selling standards documents: membership fees, conferences, product certification, training etc.  Whether revenue from standards is a necessary cornerstone of their funding or whether standards remain closed because of entrenched and ideas about how the marine electronics industry does business is not something I wish to speculate on here.

State of the Ark

The NMEA's most up to date data communications protocol is NMEA 2000.  This is based on CAN bus, the vehicle networking protocol from the 1980s which has only recently started to gain widespread use in the leisure boating market.  Still to be found on most marine devices today is the ability to communicate via an older and more widely deployed protocol, NMEA-0183 (a version of which is formalised as IEC 61162-1).  NMEA-0183 encodes data from sensors at the application layer as lines of carriage-return-line-feed-terminated ascii characters and transports them over a serial line (EIA-422 in later standards).

Due to the need to support higher data rates for connections between devices such as radar, sonar, cameras and the multifunction displays they talk to, manufacturers have implemented proprietary protocols (not using NMEA standards) running over ethernet networks.

There also exists an IEC standard for the transport of IEC 61162-1 (NMEA-0183) data over IPv4 networks known as IEC 61162-450. However I have never seen an implementation of IEC 61162-450 in any software or device available in the marine leisure market.

The NMEA is currently working on a new standard for transport of marine data over IP-based ethernet networks.  "Onenet" is due for publication in late 2014.  At the 2013 Southampton boat show and again at the 2014 London boat show I asked on the stands of Raymarine, Garmin and (at Southampton) Navico (who own the B&G, Simrad and Lowrance brands) when we were likely to see OneNet enabled products.  Sadly none of the staff were even aware of OneNet, let alone possible products.

If you don't build it, they will come anyway

PC applications which take marine data from an Internet Protocol ("IP") network using proprietary protocols are not new: Raytech used data from Raymarine equipment and MaxSea could use Furuno data.  Some of the newer tablet and phone apps continue to use proprietary protocols:  The Navionics tablet app communicates with Raymarine plotters (or "Multifunction Displays" (MFDs) as they are increasingly being called) using an undisclosed method agreed between the two companies.  Most of the current crop of mobile and PC apps using marine data from a network are not tied to a particular manufacturer.  The most common method for the transport of marine data over IP (including wireless) networks surprisingly follows no defined standard. 

NMEA-0183 over IP: The Non-Standard

The majority of current apps sending or receiving data over IP do so using the application layer of NMEA-0183.  In other words, the data an application will send or receive will look like a string of ascii characters, starting with a "!" or "$" and ending with an ascii carriage return and line feed.  Very few implement the new features in NMEA-0183 version 4 (e.g. TAG blocks).

There is no standard transport layer but in practice the options are limited anyway.  Most software apps and hardware devices use either TCP, UDP or offer the user a choice of either.  There is no standardisation for maximum UDP datagram size or whether one or several sentences may be carried in a single datagram.  There is no common statement that Nagle algortihm should be disabled for tranmission of marine data over TCP.

There is no standardisation for data transfer at the network layer except a de facto standard that IPv4 is used.  My own multiplexer program kplex is the only marine data application I am aware of which supports data in NMEA-0183 format over IPv6.  TCP is inherently unicast.  UDP is most commonly used for broadcast.  Even there there is no standardisation: Some applications use the broadcast address of the zero network (255.255.255.255), others the network broadcast address.  Most applications don't care which broadcast or unicast address is used as they simply listen for any UDP packets the system receives (ie bind to INADDR_ANY).  Very few applications use UDP multicast.  The only ones I know of are the ones I have put the code in to support it myself: kplex and OpenCPN.

There is no common standard for device addressing and no standard for service discovery.  Navico have developed and released a service discovery protocol described in their "GoFree Tier 1" specification.  This is a combination of bonjour and service information announced using JSON strings send over UDP multicast.  Few of the major applications described as working with "GoFree" currently make use of the service announcements: instructions for iNavX and similar applications involve getting the (dynamic) IP address of the MFD via the MFD's menus then connecting the mobile device to that.  Service discovery in OneNet is rumoured to be similar to this.  Perhaps unsurprising given that Navico is one of the biggest players in the industry and none of their major competitors has proposed such a protocol.

So why use a 30 year old protocol instead of the "current" NMEA 2000 at the application layer?  I am open to suggestions but here is my speculation.
  1. Prior to disassociating the application layer from the lower layers of the protocol stack, it was easier and cheaper to present NMEA-0183 to a PC over a serial port (or via a serial to USB converter) than it was to present NMEA-2000 (via hardware incorporating a CAN controller).  Many applications developed prior to the widespread use of IP networks on boats were written to take NMEA-0183 from a serial interface.  Simply using the application layer of the already-implemented data communications protocol involves the least implementation effort when network communication is added.
  2. NMEA-0183 is the most common format for data generally available on a pleasure boat.  Although NMEA 2000 has become more common over the past 5 years the ubiquity of NMEA-0183 means that most MFDs will have an NMEA-0183 interface in addition to any NMEA 2000 interface.  Many boats have older equipment which don't support NMEA 2000.  NMEA 0183 interfaces are still more common on some types of new equipment (VHF radios, AIS receivers).
  3. Many of the details of the application layer of NMEA 0183 are known.  The details of NMEA 2000 are considerably less widely known.
The latter point needs discussing further.

NMEA 0183: An Open and Shut case

As previously mentioned, the NMEA's protocols are "closed".  To legitimately use them they must be purchased and not re-distributed in any way, including by open sourcing code written to implement them.  However many of the details of NMEA-0183 up to and including version 3 have been widely publicised on the Internet.  The document I consulted in writing kplex and possibly the best known publicly available source of information on NMEA-0183 is the document assembled by Eric Raymond.  In this he states that none of the content was directly derived from NMEA standards but was taken from a variety of other publicly available sources.  It was written as part of the gpsd project, but also in part as a political protest against the NMEA's closed standards.

As a consequence of the details of NMEA-0183 being publicised it has been widely implemented in apps where the developers have not purchased or cannot use (because the application is open source) the official standard.  NMEA 2000 has not been publicly revealed in this way.  The features added to NMEA-0183 in version 4, although referred to in some publicly accessible presentations and errata are not publicly described in detail and remain largely unsupported by most apps using NMEA-0183 style data over a network.

Are we there yet?

So what are we left with? The leisure market at least has ignored IEC-61162-405.  Products implementing OneNet look to be 2 years away.  We are left with the application layer of an antiquated and arguably poorly defined protocol which is generally implemented based on non-primary unofficial sources transmitted over an unspecified transport layer in an IPv4 network with whatever method of addressing happens to exist.

The quite remarkable thing is that it mostly seems to work.