Openplotter hangs because disk full

[AndersG]

Linux openplotter 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

Running on a Raspberry Pi 2. Thing is that it works absolutely beautifully, but if I leave it running overnight and then shutdown and restart. It will boot up and show the ordinary boot screen, but when it goes into graphic mode I just get a black screen with a blinking cusrsor.

I can get in through ssh. Problem seems to be that the disk is full. Question is: What in OpenPlotter is filling up the disk?

pi@openplotter:/var/log $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.4G  5.2G     0 100% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   18M  446M   4% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   22M   21M  52% /boot
tmpfs            93M     0   93M   0% /run/user/1000


pi@openplotter:/ $ sudo du -a /var | sort -n -r | head -n 10
432952  /var
194016  /var/cache
187212  /var/cache/apt
135308  /var/cache/apt/archives
119888  /var/lib
102400  /var/swap
61720   /var/lib/apt
61632   /var/lib/apt/lists
60048   /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_stretch_main_binary-armhf_Packages
52580   /var/lib/dpkg

[AndersG]

OK. A fresh install is a bit close for comfort as well. I guess I should expand the root partition to use all of the 16Gb on the SD-card:

pi@openplotter:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.4G  4.3G  832M  85% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   18M  446M   4% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   22M   21M  52% /boot
tmpfs            93M     0   93M   0% /run/user/1000
pi@openplotter:~ $

[AndersG]

OK. Used GParted on another Linux box to expand. Will let it sit and see what happens:

Sat  4 Aug 07:07:33 UTC 2018
pi@openplotter:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  4.4G  9.6G  32% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M  6.3M  458M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   22M   21M  52% /boot
tmpfs            93M     0   93M   0% /run/user/1000

[AndersG]

Sat  4 Aug 17:41:43 UTC 2018
pi@openplotter:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  6.4G  7.6G  46% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M  6.3M  458M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   22M   21M  52% /boot
tmpfs            93M     0   93M   0% /run/user/1000

pi@openplotter:~ $ sudo du -a /var | sort -n -r | head -n 10
427484  /var
199516  /var/cache
192712  /var/cache/apt
140808  /var/cache/apt/archives
119884  /var/lib
102400  /var/swap
61720   /var/lib/apt
61632   /var/lib/apt/lists
60048   /var/lib/apt/lists/raspbian.raspberrypi.org_raspbian_dists_stretch_main_binary-armhf_Packages
52580   /var/lib/dpkg

[AndersG]

pi@openplotter:~ $ sudo du -a / | sort -n -r | head -n 10
6632924 /
3494792 /usr
2380952 /home
2380948 /home/pi
2098576 /home/pi/.cache
2097164 /home/pi/.cache/lxsession
2097160 /home/pi/.cache/lxsession/LXDE-pi
2097156 /home/pi/.cache/lxsession/LXDE-pi/run.log
1755264 /usr/lib
1310732 /usr/share

[AndersG]

OK, so culprit seems to be SignalK filling up a log file:

pi@openplotter:~ $ tail /home/pi/.cache/lxsession/LXDE-pi/run.log
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files

Question is: Do I even need signalK if I just have an USB GPS?

[conachair]

Originally Posted by AndersG

OK, so culprit seems to be SignalK filling up a log file:

pi@openplotter:~ $ tail /home/pi/.cache/lxsession/LXDE-pi/run.log
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files
accept failed for connection to signalk: Too many open files

Question is: Do I even need signalK if I just have an USB GPS?

ISTR something going on between sigk and kplex if kplex was getting no data, what's kplex showing in htop? I'll dig deeper later, i made node red send a dummy sentence to kplex to keep it all happy when nothing else was going into kplex.
Might be worth asking on the openplotter forum, developers are on there.

Edit. Found it, maybe this? Not sure if it's fixed yet..

http://forum.openmarine.net/showthre...id=1216&page=2

[AndersG]

Thanks! Was not aware of that forum. Must register there as well Anyway, I stopped SignalK and that stopped the file from growing. OpenCPN still works and gets input from GPS.

[conachair]

Originally Posted by AndersG

Thanks! Was not aware of that forum. Must register there as well Anyway, I stopped SignalK and that stopped the file from growing. OpenCPN still works and gets input from GPS.

Sailoog on the forum would certainly be grateful for any details of what sounds like a bug so they can fix it.

Interesting opencpn is still getting nme gps data, it all goes through signalk to get there ..
Would be a shame not to have signalk running, it's really powerful with many apps these days
https://docs.sailoog.com/openplotter...w-does-it-work

[AndersG]

Yes. But I was under the impression that it was not the case if you used a direct port connection for the gps. I am fully aware of the beauty of SignalK, but I would like to keep it simple for now

How shall I report this?

[AndersG]

@RGleason

Issue seems fixed now. Perhaps that should be in the FAQ as well?

[conachair]

Originally Posted by AndersG

@RGleason

Issue seems fixed now. Perhaps that should be in the FAQ as well?

Was it the signalk / kplex issue that was causing you problems?

Provisional fix here if anyone else is having problems >
*newbie* /dev/root full after 24hrs

[AndersG]

Yes. I changed the line as per the instructions in the forums and it has been stable since. As I said, I have no need for SignalK at this point.

[conachair]

Originally Posted by AndersG

Yes. I changed the line as per the instructions in the forums and it has been stable since. As I said, I have no need for SignalK at this point.

Signalk updated to prevent the problem already, opensource is great

https://github.com/SignalK/signalk-server-node/pull/581