Signal K

[teppokurki]

Originally Posted by conachair

1 wire temperature sensors wait until you ask before sending data, you can't tell them to send once a second or whatever.

You are using one example to say that this is all wrong.

If I combine a DS18B20 with some logic that reads it every x seconds and sends it over udp as SK then that device looks like a sensor whose sampling period I can control (I actually have this setup on my boat).

So it is all about abstraction level. If you look deep enough into practically any digital sensor it is pull (sampling on demand) and you can make it push by sampling on timer.

Originally Posted by conachair

Never heard of subscribing to a web socket feed at a certain 'frequency'.

Then you have not read the Subscriptions part of the specification, see https://signalk.org/specification/1...._protocol.html

[conachair]

Originally Posted by teppokurki

You are using one example to say that this is all wrong.

If I combine a DS18B20 with some logic that reads it every x seconds and sends it over udp as SK then that device looks like a sensor whose sampling period I can control (I actually have this setup on my boat).

How i do it as well, the sensor waits for something to ask it to send data then sends it, which in my case is a esp32 which then sends the data over udp. Haven't come across any sensors which just send on their own, is that6 even possible on 1wire or i2c?

Then you have not read the Subscriptions part of the specification, see https://signalk.org/specification/1...._protocol.html

Yep, need to have a read and play with that, thanks

sailwing, for day to day practical things you'd probably be better off asking here >
OpenMarine
Quite a lot of posters on there with actual systems working on boats, openplotter v2 will hopefully be released before too long with good documentation.

[sailwing]

thank you everyone. My bad that I should have mentioned it was the DS18B20. I posted the question here because I wasn't sure whether it is with signalK or something in the openplotter. Just looking for some direction.

[conachair]

Originally Posted by sailwing

thank you everyone. My bad that I should have mentioned it was the DS18B20. I posted the question here because I wasn't sure whether it is with signalK or something in the openplotter. Just looking for some direction.

1wire / ds18b20 should be soon in openplotter it seems, it was here before in v1.X



End of thread drift...

[Tehani]

The use of One wire is very compromised because the high and low state times are strict and there are no specialized peripherals (USART, I2c, SPI, etc.) that handle it. It is all by force of CPU timings. It can compromise the operation of a TCP or UDP stack. In the case of ESP32, care must be taken to initialize the service task of ds18b20 in core 0.

......

I represent here something approximate to what would be the time slots in each "-":

Single packet frames (header 29 bits + 8 data bytes):
X : Heading, rapid update.
Y : COG / SOG, rapid update.
Z : AWA / AWS, rapid update.
Fast packet frames (multi packets: several header 29 + 8 bytes each)
N : AIS diverse messages.
M : GNSS position
P : Satellite info.

<--------------------------   1S   --------------------------------->
N2k: Single packets capture:
X---YZPPNNXPP-YZ----X-NNYZGGGGXNNGYZ----X---YZ----XNN-YZMMM-X---YZ...

TCP:
--------------------PK1-----------------PK2----------------------PK3...
PK1 = XYZPPNNXPPYZX
PK2 = NNYZGGGGXNNGYZX
PK3 = YZXNNYZMMMXYZ

We can observe several things in this simplified representation:
1) X, Y and Z are spaced at exact time intervals in N2k.
2) A fast-packet is interrupted by a frame (single or fast-packet) of higher priority. Then it has just been completed.
3) This is not seen in the graph: PGN's have no time stamps, their position in the time domain is what matters. In some, only one counter is added in the 1st byte of data, in others, only an ID that associates the nature of the issuer, in others, nothing ...
4) Indeed, TCP packets last much less time.

In the TCP part, I have represented a transmission through a router. Since the N2k packets are very small, there is a possibility that they will be sent in burst. Although timestamps are added, those messages really arrive expired at their destination. A pilot could not handle them, and would result in a yaw.

What happens in TCP, we can observe it even in the human perception of time. Surely everyone will have been able to see how a Vo-IP conversation is severely cut or delayed significantly.
In our case, it doesn't matter if the data is only for a dashboard.

I know the N2k section in depth, but please correct me in the TCP part if I'm wrong, I only use libraries ...

José Luis.

[motu42]

Originally Posted by motu42

Any discussion of this kind gets very complex quickly as each participant vocalises their views, from user to deep tech. (me included). So I want to summarize the main benefits of signalk to opencpn again.

1) An open-source language for data transfer, easily extendable by any-one, avoiding the restrictions of NMEA0183, and removing dependency on it.

2) Compatibility with the internet, send and receive off-board data as easily as on-board data. Opens up data sharing and social use cases.
3) Provides a solution to multi-user OpenCPN chartplotters and web browser solutions, and scalability options for power users.

4) Provides better integration with NMEA2000 (N2K), and OneNet if it ever arrives :-)
5) Offloads the work in maintaining NMEA* data integration, and provides a simple data model to work with in OpenCPN.


I see a few must-haves:
1) Maintain compatibility with current users and devices.
2) At least as easy to install and use
3) At least as performant


Sure there will be problems, compromises, and sweat to make it work, but thats software.



Do you see these benefits as positive ?
Have I missed any must-haves?

Are there any blockers Ive missed ?


I guess at a high level I'm trying to gauge the mood on this integration?

Yep, underway again after drifting into the shallows of one-wire bay

[motu42]

While looking for an efficient cross-language binary protocol for the internal distribution of data I came across https://google.github.io/flatbuffers/index.html


It would also work well as a transport for key:value over the TCP/IP port (local subnet). Has the advantage of zero-copy, so minimal memory/resources.

[Axel_G]

Originally Posted by Tehani

I represent here something approximate to what would be the time slots in each "-":

Single packet frames (header 29 bits + 8 data bytes):
X : Heading, rapid update.
Y : COG / SOG, rapid update.
Z : AWA / AWS, rapid update.
Fast packet frames (multi packets: several header 29 + 8 bytes each)
N : AIS diverse messages.
M : GNSS position
P : Satellite info.

<--------------------------   1S   --------------------------------->
N2k: Single packets capture:
X---YZPPNNXPP-YZ----X-NNYZGGGGXNNGYZ----X---YZ----XNN-YZMMM-X---YZ...

TCP:
--------------------PK1-----------------PK2----------------------PK3...
PK1 = XYZPPNNXPPYZX
PK2 = NNYZGGGGXNNGYZX
PK3 = YZXNNYZMMMXYZ

We can observe several things in this simplified representation:
1) X, Y and Z are spaced at exact time intervals in N2k.
2) A fast-packet is interrupted by a frame (single or fast-packet) of higher priority. Then it has just been completed.
3) This is not seen in the graph: PGN's have no time stamps, their position in the time domain is what matters. In some, only one counter is added in the 1st byte of data, in others, only an ID that associates the nature of the issuer, in others, nothing ...
4) Indeed, TCP packets last much less time.

In the TCP part, I have represented a transmission through a router. Since the N2k packets are very small, there is a possibility that they will be sent in burst. Although timestamps are added, those messages really arrive expired at their destination. A pilot could not handle them, and would result in a yaw.

What happens in TCP, we can observe it even in the human perception of time. Surely everyone will have been able to see how a Vo-IP conversation is severely cut or delayed significantly.
In our case, it doesn't matter if the data is only for a dashboard.

I know the N2k section in depth, but please correct me in the TCP part if I'm wrong, I only use libraries ...

José Luis.

The N2K message of 27 Bytes from above at 250kBit/s will take over 1 milliseconds to arrive at the destination. The same data in TCP at 100 MBit/s will take 6.4 mikroseconds (because the shortes frame is by defintion 64 Bytes.

So whatever timestamp there is is quick enough to be good enough on a boat.

And there is significant factor to make sure the TCP data arrives on time, even through a switch. In addition the TCP traffic is probably routed much quicker within your PC or Raspberry or whatever. These tools are optimised for TCP traffic.



And the ViIP is an issue over many kilometers, not some meters, for which N2K is only good for. With TCP you could even Wifi-transfer to another boat and still be on time.

[Tehani]

Originally Posted by Axel_G

The N2K message of 27 Bytes from above at 250kBit/s will take over 1 milliseconds to arrive at the destination. The same data in TCP at 100 MBit/s will take 6.4 mikroseconds (because the shortes frame is by defintion 64 Bytes.

So whatever timestamp there is is quick enough to be good enough on a boat.

And there is significant factor to make sure the TCP data arrives on time, even through a switch. In addition the TCP traffic is probably routed much quicker within your PC or Raspberry or whatever. These tools are optimised for TCP traffic.



And the ViIP is an issue over many kilometers, not some meters, for which N2K is only good for. With TCP you could even Wifi-transfer to another boat and still be on time.

Thank you very much for the information.
This is a real N2k chronogram, taken with logic analyzer, which contains data from HDG, STW, AWA, AWS, COG, SOG, DPT, and several more navigation and satellites. The green dots indicate the moments where HDG is sent.
Of course ethernet is much faster, but the fact that the shorter frames are 64 bytes implies that 6 N2k singles are needed for the shorter TCP / UDP transmission. As you can see, sometimes it is necessary to wait more than 50mS to gather 64 bytes.

We can think that the latency of the data in TCP is reduced if the traffic is increased in N2k (64 bytes are collected faster). But if there is only wind and heading data (12 bytes per single packet with the 29-bit identifier), almost 300mS will be required to complete 64 bytes of TCP.
Now I see it clearer: A larger JSON block (> 64 bytes) to send only HDG ensures instant transmission. I'm right?

[Axel_G]

Originally Posted by Tehani

Thank you very much for the information.
This is a real N2k chronogram, taken with logic analyzer, which contains data from HDG, STW, AWA, AWS, COG, SOG, DPT, and several more navigation and satellites. The green dots indicate the moments where HDG is sent.
Of course ethernet is much faster, but the fact that the shorter frames are 64 bytes implies that 6 N2k singles are needed for the shorter TCP / UDP transmission. As you can see, sometimes it is necessary to wait more than 50mS to gather 64 bytes.
Attachment 204036
We can think that the latency of the data in TCP is reduced if the traffic is increased in N2k (64 bytes are collected faster). But if there is only wind and heading data (12 bytes per single packet with the 29-bit identifier), almost 300mS will be required to complete 64 bytes of TCP.
Now I see it clearer: A larger JSON block (> 64 bytes) to send only HDG ensures instant transmission. I'm right?

Actually one does not need to fill up the 64Bytes, the Ethernet driver or even the Ethernet hardware will handle that for you and fill it with zeroes. So from software point of view it can be sent out when it's ready. When the bus gets crowdy it is better to group the data before transmission but on a boat that is not very likely to happen unless you run a couple of videos on te same line.



For Signal-K it wouldn't have to fill up the frame with zeroes, so nothing lost.



Let's assume a DPT frame needs to be transmitted. This requires 19 Bytes in NMEA0183. So on a NMEA0183 line at 4800bd that would take 40 ms, on N2K it would take 1 ms, on Ethernet TCP it would need 6.4 microseconds (so 0.006ms) and using Signal-K it would also need 6.4 microseconds. Now I havn't taken all the bits and pieces into account, bit I think that gives an overview of the speed and latencies.



That is the reason json protocols are so popular nowadays, the performance of the hardware is so abundant that it it is not useful to implement crypric algorithms to reduce size any more. With Signal-K all the boat data and so on can be transmitted with every frame without hurting performance.

[rgleason]

I haven't been following this recently, but the contributions are quite useful, thinking about latency, timing, viability and approaches.

AxelG: Actually one does not need to fill up the 64Bytes, the Ethernet driver or even the Ethernet hardware will handle that for you and fill it with zeroes. So from software point of view it can be sent out when it's ready. When the bus gets crowdy it is better to group the data before transmission..

Tehani: I represent here something approximate to what would be the time slots in each "-":
Single packet frames (header 29 bits + 8 data bytes):
X : Heading, rapid update.
Y : COG / SOG, rapid update.
Z : AWA / AWS, rapid update.
Fast packet frames (multi packets: several header 29 + 8 bytes each)
N : AIS diverse messages.
M : GNSS position
P : Satellite info.
...Great knowledge of N2K and Excellent questions and concerns about filling packets and timing.

Motu42: While looking for an efficient cross-language binary protocol for the internal distribution of data I came across https://google.github.io/flatbuffers/index.html It would also work well as a transport for key:value over the TCP/IP port (local subnet). Has the advantage of zero-copy, so minimal memory/resources.

To my totally inexperienced eye it appears to me that you have established that it can work, in some way.
I was cleaning up my downloads folder and found some SignalK sketches by Petri from some time ago. I think it would be good to share them so you guys can think about it
Thanks for all the good work and contributing your knowledge.

[rgleason]

Here are Petri's sketches.

[rgleason]

Perhaps it would be helpful to create a short list of recommendations.

I've been trying to find a link to those diagrams, and have not yet succeeded, but there are some other links that might be useful.
Petri implemented a signalK streamer in Dashboard-Tactics.

http://www.cruisersforum.com/forums/...ml#post3017032
https://canne.github.io/dashboard_ta...amerUsage.html

and here is a PDF he made.
https://canne.github.io/dashboard_ta...utStreamer.pdf
He did tests on Windows systems and Linux and recorded timing.

He reached some conclusions about reducing jitter in the system.


Now the "jitter" noticed may not be of the type that a signalK implementation would have, but it might provide some insights.
I have to go back and read this in more detail, but Schnapsy was noticing jitter and different instrument updates in standard "Dashboard" and started asking questions about it.

This is the original post http://www.cruisersforum.com/forums/...ml#post2984268
You'll have to read the thread and link to Petri's [Canne] extensive work to track the problem down.

[rgleason]

And here is a Github Issue about it https://github.com/OpenCPN/OpenCPN/issues/1464
You can see some more results from Petri's testing using USB, a signalK converter via TCP and signalK direct USB.

https://github.com/canne/dashboard_t...ps_win10.ipynb

[motu42]

Originally Posted by rgleason

Motu42: While looking for an efficient cross-language binary protocol for the internal distribution of data I came across https://google.github.io/flatbuffers/index.html It would also work well as a transport for key:value over the TCP/IP port (local subnet). Has the advantage of zero-copy, so minimal memory/resources.

To my totally inexperienced eye it appears to me that you have established that it can work, in some way.

I havent tried flatbuffers, but the library is mature and presumably works as advertised. It will be simpler to use than a roll-your-own protocol, and avoid problems like endian format, escapes, etc. at no cost to efficiency. Plus zero copy is faster and doesnt waste RAM.