SignalK development ?

[barnakiel]

Originally Posted by Auspicious

(...)

N2K isn't fast enough.

Ethernet is 10 Mbps, 100 Mbps, or 1 Gbps.

So assuming CAN bus is 1Mbps (which is limiting) vs. Ethernet 10Mbps or faster, if I build a wifi based sensor, can this one reach Ethernet-like speeds then? I mean skip the CAN bus part, work in wifi only.

Sorry if I sound naive. I am using TCP passively but have low understanding of the technical side of ethernet / wifi world.

Not that it is necessary for sensors like depth or fuel but maybe then a common platform for the sensors and the radar (and other) imagery could be had?

Does SK have a 'sending an img file' sentence that we could use to send radar data (or other image data) in SK over WiFi?

We may after all end up with small boats and big boats using two wildly different methods to create, transfer and display the nav data. Maybe the big guys need a new version of N2K that will manage radar and other big data well.

b.

[Auspicious]

Originally Posted by barnakiel

So assuming CAN bus is 1Mbps (which is limiting) vs. Ethernet 10Mbps or faster, if I build a wifi based sensor, can this one reach Ethernet-like speeds then? I mean skip the CAN bus part, work in wifi only.

CANbus is 1/4 of 1 Mbps. BT is 1-3 Mbps(ish). WiFi can achieve 54 Mbps pretty easily. Gigabit Ethernet is really fast. *grin*

I don't know what you have in mind. I expect your biggest issue will be traffic management and scaling. What works for three sensors may not work for ten.

With all due respect you sound like your background is like that of the digital EEs as data rates got faster. I lived through the time when we had to go out and hire analog EEs out of retirement as clock speeds got faster and traces on circuit boards behaved like transmission lines and we had to worry about characteristic impedance and termination resistance. This is relevant because as software people become systems people you have to understand - really understand - the hardware you run over. Consider the bottom of the OSI model. The physical layer is important. You have to worry about latency and buffering. If you include autopilot--just as an example--you are in the realm of near real time and that means more than faster hardware. It means determinism. By the way - don't confuse near real time with real time. Real time is really hard. Ask the people doing remote medicine.

Originally Posted by barnakiel

Does SK have a 'sending an img file' sentence that we could use to send radar data (or other image data) in SK over WiFi?

I'm a little dated on some of this stuff and the newer radar systems are not something I've done more than install and set up. I suspect that there is no imagery in the conventional sense of the word. I know a lot of processing that used to be "downstairs" is now in the dome. You'll have to do the research yourself (please report back) that what is passed down the cable from the dome is rϴ arrays. You have to keep up, stay in order, and render it not only fast enough but at a consistent rate. Frame rates aren't relevant. On the other hand I may just be old. *grin* Given that the radars I am familiar with have a dome to display connection and subordinate displays must network with the primary makes me think I'm correct.

While more processing is happening at the sensor (consider the movement of GPS receivers into the "antenna" as an analogy) I suspect there is still a lot of processing in the display for radar and sonar, especially fishfinders and side-scan.

Homework will be graded. *grin* Seriously, if you find credible data to the contrary (footnotes please) of my speculation especially at the interface I will be grateful.

[muttnik]

Originally Posted by Lake-Effect

I don't see the necessity of having every single transducer on a single buss... but it's still very possible. One option would be small wifi boards like the ESP8266 ($5) plus a regulator/interface board ($?) that can easily take the output of most transducers and put out as SK on wifi.

With a genuine motive of discussion around more effective promotion of Signal K rather than being combative, I think this illustrates some opportunities for improvement.

Signal K has been developed with a particular use case in mind: Apps on powerful devices written in high level languages across high bandwidth IP networks with a variety of protocols feeding a single server on the backend.

- Acknowledge that there are other use cases that it's not optimal for
- DOTADIW
- Stop the hand waving and concentrate on making the format you have properly defined so people can stop saying "yeah we'd implement it but it's a moving target".

On the last point...a bus-oriented protocol over IP such as some of us with experience in other industries might expect is based on multicast (or broadcast if it's braindead). *Signal K does not support this*. You have a mechanism for connecting to a single server (nice to see that WS and http are now defined as required given that in the early days there was a refusal to define a standard session/transport layer). You have no standard mechanism for accepting data from multiple devices. Signal K has no application level mechanism for loss detection and retransmission so it relies on an underlying reliable transport. Fine for WS/http. Not fine for multicast UDP. Plus there's no limit on message length so you have no limits on fragmentation. I'm similarly reminded of the hand-waving discussions about security on slack.

DOTADIW. You have something that was designed for and works for an important use case. Shore that up. Don't evangelise for use cases which it doesn't work for: it doesn't do the brand any favours.

[Lake-Effect]

Originally Posted by barnakiel

Does SK have a 'sending an img file' sentence that we could use to send radar data (or other image data) in SK over WiFi?

I don't know... but what would be the use-case? What "data" produced by a boat radar would be consumable and useful to another device?

There are other solutions to a wired or wireless remote display for radar.

Originally Posted by muttnik

With a genuine motive of discussion around more effective promotion of Signal K rather than being combative, I think this illustrates some opportunities for improvement.

First, I'm a SK fanboy, I like where they're going, but I'm not yet a serious user and certainly not on the team, or positioned to be a knowledgeable evangelist for it. Sorry if I gave that impression. I'll try harder to not speculate, and to stick to the facts as printed, or let the actual SK experts speak to important points.

You have a mechanism for connecting to a single server (nice to see that WS and http are now defined as required given that in the early days there was a refusal to define a standard session/transport layer). You have no standard mechanism for accepting data from multiple devices. Signal K has no application level mechanism for loss detection and retransmission so it relies on an underlying reliable transport. Fine for WS/http. Not fine for multicast UDP.

As you know, a network server by definition can handle simultaneous requests, so if each of a few simultaneous requests contain a data payload... you just accepted data from multiple devices. The spec also anticipates separately wired sensors in simpler protocols as appropriate (eg serial/NMEA0183, I2S, IIS). So, the potential is there. I think we would also agree that the TCPIP layer is a good choice as transport mechanism. I haven't dug deeply into the UDP/sockets part of the spec, except to read that it's a means for broadcasting the SK data.

Again, I'm a SK fan, not a knowledgeable spokesperson. Don't let my errors reflect on the good people who are actually working on this.

[muttnik]

Originally Posted by Lake-Effect

As you know, a network server by definition can handle simultaneous requests, so if each of a few simultaneous requests contain a data payload... you just accepted data from multiple devices. The spec also anticipates separately wired sensors in simpler protocols as appropriate (eg serial/NMEA0183, I2S, IIS).

Indeed the way Signal K works at the moment is via a central server with multiple protocols on the back end and signal K over web sockets, http or some other reliable protocol not officially specified on the front end, but this is not the bus protocol I think barnakiel is imagining (correct me if I'm wrong barnakiel): That would be something more akin to IEC 61162 part 4, what I guess OneNet will turn out to be (if it ever turns up :-), or something more akin to a market data protocol in the financial world.

[seandepagnier]

My repository for a free autopilot using signalk is here:

https://github.com/pypilot/pypilot

[rgleason]

I like the idea of 0183 data via TCP to instruments. Sean, I am always impressed, way above my head. I recall your early git repos about boat automation.

I am guessing it was this one
https://github.com/seandepagnier/rpi_autopilot

[barnakiel]

Originally Posted by rgleason

I like the idea of 0183 data via TCP to instruments. Sean, I am always impressed, way above my head. I recall your early git repos about boat automation.

I am guessing it was this one
https://github.com/seandepagnier/rpi_autopilot

Why to the instruments?

Do you say this meaning a sensor / instrument pair?

If you have a good link to creating a multi client TCP solution for many sensors talking TCP to a single server (processor - e.g. one in a laptop or a tablet) I will be most obligated if you share.

It seems so simple when our onboard devices all talk to the same AP. But somehow I cannot replicate this for our sensors to talk to the same tablet.

I managed to build BT and TCP sensors but I am stumbling when it comes to having them all talk to one tablet. Most boats have many sensors and I would like to be able to handle this now.

I am on BT hardware now but for the above reasons considering a move to TCP. My own boat is wired TCP via kplex run on a wifi router. Simple and works but a bit high Amper footprint (about 0.5A).

Best regards,
b.

[rgleason]

Do you say this meaning a sensor / instrument pair? yes

[conachair]

Originally Posted by barnakiel

If you have a good link to creating a multi client TCP solution for many sensors talking TCP to a single server (processor - e.g. one in a laptop or a tablet) I will be most obligated if you share.
.

Openplotter running on a Raspberry Pi. Though to be fair I've just pushed it a bit using node-red to simulate sensor data going into the Pi over wifi. One udp test was messages going out of a laptop at 100Hz, average time to return to the laptop with all the other gps etc data was around a few mS with around 1% lost messages. This was with everything talking to a mobile phone as an access point, not using the openplotter access point.
This is a snippet of results of a few minutes, 48,727 nmea messages sent.

Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=6mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=11mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=3mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=4mS, Lost sentences =425 Time=5mS, Lost sentences =425 Time=3mS, Lost sentences =425

Might give it a go soon with half a dozen tcp sources going out.

[teppokurki]

Originally Posted by muttnik

- Acknowledge that there are other use cases that it's not optimal for

There are other use cases that Signal K is not optimal for.

I don't want to force Signal K on anybody. If it works for you great, if not then that's ok too.

I just hope that people would start building more stuff together and making it interoperable. The world seems to be full of private marine data projects.

I am 100% guilty of this myself, but at least I am trying to be a better person.

Originally Posted by muttnik

- Stop the hand waving and concentrate on making the format you have properly defined so people can stop saying "yeah we'd implement it but it's a moving target".

Agreed. What would be the best next step in this regard from your point of view?

Originally Posted by muttnik

DOTADIW. You have something that was designed for and works for an important use case. Shore that up. Don't evangelise for use cases which it doesn't work for: it doesn't do the brand any favours.

I can only speak for myself, but it is just too bad if the eagerness to explore different solutions in this area looking through Signal K glasses comes across as pushing or evangelising Signal K.

Please try to see it as a wish to build more and better things together rather than pushing a particular solution on everybody.

Originally Posted by muttnik

On the last point...a bus-oriented protocol over IP such as some of us with experience in other industries might expect is based on multicast (or broadcast if it's braindead). *Signal K does not support this*. You have a mechanism for connecting to a single server (nice to see that WS and http are now defined as required given that in the early days there was a refusal to define a standard session/transport layer). You have no standard mechanism for accepting data from multiple devices. Signal K has no application level mechanism for loss detection and retransmission so it relies on an underlying reliable transport. Fine for WS/http. Not fine for multicast UDP. Plus there's no limit on message length so you have no limits on fragmentation. I'm similarly reminded of the hand-waving discussions about security on slack.

All of these points are true.

However both java and node SK server accept data from multiple devices, but it is not documented.

About loss detection and retransmission: people keep referring to this, but since neither NMEA0183 or N2K support this I feel this criticism is a bit unwarranted. SK has it over TCP. Why is this a problem? What practical case would you like to solve that SK is not solving and since SK is not the solution what are you using now?

I believe Signal K would work well with IPv6 multicast, but this is just me hand waving since I haven't done it nor have the expertise to say anything more. Again, can you point an example in the marine field that Signal K should learn from? For example NMEA0183 over TCP is architecturewise exactly the same as SK over http/ws.

I believe we have stated that security remains an issue largely unaddressed, apart from the universal mechanisms available for securing http and ws connections in general. For interoperability we do need to address this in the spec.

[conachair]

Originally Posted by barnakiel

Why to the instruments?

Do you say this meaning a sensor / instrument pair?

If you have a good link to creating a multi client TCP solution for many sensors talking TCP to a single server (processor - e.g. one in a laptop or a tablet) I will be most obligated if you share.

It seems so simple when our onboard devices all talk to the same AP. But somehow I cannot replicate this for our sensors to talk to the same tablet.

I managed to build BT and TCP sensors but I am stumbling when it comes to having them all talk to one tablet. Most boats have many sensors and I would like to be able to handle this now.

I am on BT hardware now but for the above reasons considering a move to TCP. My own boat is wired TCP via kplex run on a wifi router. Simple and works but a bit high Amper footprint (about 0.5A).

Best regards,
b.

Some more testing, 6 different xdr nmea messages sent to different ports via TCP created n node-red, openplotter multiplexes and sends back out on port 10109 for opencpn (or anything else wanting to listen)

About 0.3A @12v and does *so* much more as well.

Can't see how you'd do this without something acting as a multiplexer sending over an access point, how could your tablet hear anything if it isn't connected to an AP?

[sbender]

Originally Posted by barnakiel

If you have a good link to creating a multi client TCP solution for many sensors talking TCP to a single server (processor - e.g. one in a laptop or a tablet) I will be most obligated if you share.

First off, apologies for being an A-hole.

Second. Take a look at SignalK node server. It does exactly this

[muttnik]

Originally Posted by teppokurki

About loss detection and retransmission: people keep referring to this, but since neither NMEA0183 or N2K support this I feel this criticism is a bit unwarranted. SK has it over TCP. Why is this a problem? What practical case would you like to solve that SK is not solving and since SK is not the solution what are you using now?

I believe Signal K would work well with IPv6 multicast, but this is just me hand waving since I haven't done it nor have the expertise to say anything more. Again, can you point an example in the marine field that Signal K should learn from? For example NMEA0183 over TCP is architecturewise exactly the same as SK over http/ws.

To address those points out of order :-)...I think it's perfectly valid to leave transmission reliability to lower layers : however the nature of the default delta update mechanism as I understand it (yes I know the subscription model has options to work differently) means that reliability is required. Unlike a system where transducers repeatedly send their state and missing one may not matter, miss a delta and you could be stuck with the wrong state for a while. You've now specified WS and HTTP as "standards" which is great, but the team refused to do that a couple of years ago insisting that Signal K was good for running over anything.

Also by refusing to bound the length of a Signal K message you make it necessarily stream-oriented: There are limits on a UDP packet size (arguably quite big with IPv6 jumbograms) so putting a Signal K message in UDP may generally work but it's "wrong". Moreover given that your standard use case is transmission over wifi where multicast is notoriously lossy, fragmentation is going to cause nightmares. If you're proposing IPv6 multicast you want to limit message size to some sensible value less than or equal to 1232 bytes (1280-40-8). Also a bounded message size makes things easier for developers in lower level languages.

N2K has reliability at the datalink layer. NMEA-0183-over-IP isn't a real protocol but works ok over multicast because the message sizes are small and generally constantly repeated.

I don't think you need to do what those things do: You've design Signal K for a particular use case and it's working: stick with that.

I do use nmea-0183 over IP simply because i have the tools to do so and sharing them is useful to others. I don't *like* it.

Originally Posted by teppokurki

I believe we have stated that security remains an issue largely unaddressed, apart from the universal mechanisms available for securing http and ws connections in general. For interoperability we do need to address this in the spec.

I've been slow in responding because slack doesn't keep enough history to reference the relevant security discussions (probably over a year ago now) and I really didn't want to resort to a hand-waving explanation of my hand-waving statement, but sadly it seems I must. I wonder if the enthusiasm of some of your more confident contributers can lead to false hope that a particular technology which is not in their core area of expertise can solve problems without thinking through the practicalities. A better approach might be to ask what the use case is and whether limiting scope can help address the problem. If your concern is privacy and integrate in transit, leave it to the transport. Drop connectionless transports from your scope and things become much more manageable. Integrity of a message which may traverse multiple hops, resting at each (producer->server A->server B -> client) is something you might like to address in the spec (some kind of json signing?) but is a much smaller scope.

Originally Posted by teppokurki

What would be the best next step in this regard from your point of view?

Ask your commercial parters what their pain points are and prioritise those. Their concerns are probably other potential partners' concerns.

Limit the scope and explicitly state what it is: few interfaces to support means you can focus on solutions for the ones you do.

Have a look through the spec and think if anything could be more pinned down, e.g. what standard you're following for json interoperability. Yes it doesn't worry most people but it doesn't hurt to be explicit.

[barnakiel]

Originally Posted by conachair

(...)

Can't see how you'd do this without something acting as a multiplexer sending over an access point, how could your tablet hear anything if it isn't connected to an AP?

An android tablet can create its ow AP. Newer Windows (I think since 7.0) too.

At this point my boat sensors (gps, ais, depth, etc.) (nmea, serial) are connected thru an old wifi router that becomes the AP for all and any listeners. But this method is lame as it requires the router to be there. I can convert these sensors to TCP but I do not know how to get many TCP sensors to talk to the same soft AP. Wifi talking sensors are not too common at all, I think.

I tried the same gimmick with BT sensors but on my testing platform (Arduino) BT gives as many headaches as it does under Windows, so I am a pessimist in respect of my long term patience for BT. I only use it because it is so simple to deploy and cheap to test. I am sure aware of benefits of BT LE but man how I wish BT were as simple and reliable as TCP is.

And so, TCP could be where I will vest my future experiment time and effort. It is not as easy to implement as BT sensors (are) but it is way more reliable, easier to set up and trouble free.

Thanks again for all the tests you have run - most appreciated.

b.