#Feeder Has Odd Dropouts of ADS-B Tracks and CPU Utilization

1 messages ยท Page 1 of 1 (latest)

rotund vault
#

My feeder is a recent addition and the first one I have assembled, UID QLvS_Der7bJk. It is built from a Pi 4, Nooelec Flycatcher, Nooelec SAWBird+, and self built antennas with wired ethernet. I had been running the antennas initially in the basement for about the first week, moved the 1090 to the first floor for about a week, and then both into the attic during Jan. 11th. Overnight, my feeder experienced a complete halts of all CPU activity that required hard power of and on reset to get going again. I had tried a "save and restart" via the http://local_ip_addr/config/ link, but it was not enough to restore feed status. There were 2 short halts before I went to sleep and the feeder stopped overnight while I was asleep for 3 hours and again required a hard power reset.

Since that reset, I have made sure to close all local feeder web server pages and to not tax the Pi. I think the issue has to do with keeping multiple instances of the http://local_ip_addr/tar1090/ page open on multiple devices in my house. Overall CPU utilization less than 10%. Keeping the local page serving traffic to the Pi at minimums seems to allow the feeder to keep running.

What I am seeing now are odd dropouts of ADS-B 1090 Tracks and ADS-B CPU Utilization at about 20 minute intervals. I spoke too soon, while I have been writing this, the feeder completely halted again. With the power off reset, the feeder seems to have lost 9 hours of Performance Graph data.

I have a lot of screen capture and some log data captured. What would be most useful for me to post for diagnosing this trouble?

Thank you!

orchid wadi
#

The first question we always ask in cases like this is; what power supply are you using for the Pi? If it's not an official Pi one, or at least one designed for the purpose (i.e. not a phone charger, etc), it could cause issues.

Assuming the ADSBx pi image, graphs data is only written to the SD card once a day, to minimise "wear", so it's normal to lose data since the last write in the event of an "ungraceful shutdown".

rotund vault
#

It is a qualified 3A wall wort that was previously supplied to me with a rather power hungry LTE hotspot. I did measure the voltage supplied amperage draw by the Pi and receivers. From the current System Info...

#

Voltage Drops:

Status: 0x0
Undervolted:
Now: NO
Run: NO
Throttled:
Now: NO
Run: NO
Frequency Capped:
Now: NO
Run: NO
Softlimit:
Now: NO
Run: NO

CPU temp=54.5'C
Redline=80'C

#

The dropouts and CPU halting only started when I moved the 1090 & 978 antennas to their final intended location. The activity level should not have been a problem overnight as traffic seems to be much less at night.

orchid wadi
#

Load should never make it completely lock up, although I'm not entirely sure what you mean by "complete halt". You said you "Save & restart services" ... if it was completely halted, how did you even get to that page? Anyway, it sounds like a hardware issue.

rotund vault
#

Complete CPU halt.

#

Generally, I try to use 'sudo shutdown -h now', but the time I did not, I lost 6 hours of local log data. Understandable, but it shows my growing frustration.

#

Dseven makes a good point, it cannot be a complete CPU halt as the Pi responds to SSH. As a feeder, it is doing nothing at those times.

#

tar1090.service log has things like this in it that seem to coincide with the 20 minute cycle.
Jan 12 16:10:46 adsbexchange tar1090[647]: No aircraft.json found in /run/readsb during the last 30 seconds! Try restarting dump1090 or reinstalling tar1090 if you switched dump1090 to readsb!
Jan 12 16:10:50 adsbexchange tar1090[647]: Found aircraft.json in /run/readsb, continuing operation as per usual!
Jan 12 16:10:50 adsbexchange tar1090[17691]: /usr/local/share/tar1090/tar1090.sh: line 104: 978.json: No such file or directory
Jan 12 16:11:00 adsbexchange tar1090[647]: No aircraft.json found in /run/readsb during the last 30 seconds! Try restarting dump1090 or reinstalling tar1090 if you switched dump1090 to readsb!

#

The short (recoverable) dropouts seem to coincide with the big spikes of Disk I/O and the above message in the tar1090.service log.

orchid wadi
#

Might be worth trying a new SD card too. Assuming you're not doing anything else on the Pi4, this doesn't seem normal, so still suspecting a hardware issue of some sort.

rotund vault
#

Sigh! I had a tough time getting 978 to work. Yes, this Pi is dedicated to only ADS-B/UAT feeding. The hesitation is rebuilding the software stack from scratch. For some that is trivial, not so trivial for me. How do I keep from causing a new feeder identity?

orchid wadi
#

Keep a copy of /boot/adsbx-uuid. It shouldn't be complicated to setup - the whole purpose of the image is to make it simple. I've never used a Flycatcher, but it looks like its pretty much two SDRs (with LNAs?) duct-taped together?

rotund vault
#

Yes, the Flycatcher is only on the Pi-Hat format to make it stupid simple to mount. It still has and requires separate USB power cables to each of 1090 and 978. Back to SSH to find the UUID file. Thank you. I'll report back to this thread when I get that accomplished.

fierce radish
#

I have had nothing but problems with even the "official" pi wallwarts, I recomend a 25W PS, I got mine from aliexpress... 5V 5A 5$. ๐Ÿ˜„

rotund vault
orchid wadi
rotund vault
# orchid wadi There are two types of feeder ID - the full UUID (the one you saved), and the sh...

I reported the short ID to make it easier to find the statistics. I did use the UUID from the old SD card and put it on the new SD card. I had to Nano edit the long ID into the file as I did not have permissions set to FTP the old UUID file into the new card. I don't believe either is causing a problem. The statistics seem to be updating on the ADS-B Exchange website for both 1090 and 978 data and under the expected short feeder ID.

orchid wadi
#

oh, so you don't have 978 at all in graphs1090, or graphs with no data?

rotund vault
orchid wadi
#

hmm. I don't have 978, but, so less experience with it, but... maybe if you just enabled it, graphs1090 might need a restart to pick it up, or something. Reboot might do it?

rotund vault
#

I'll try another reboot. Seems like I have done many. To me it feels like an enable statement is missing in the graphs1090 config, but I have not found the file I think I am looking for.

orchid wadi
#

if that doesn't do it, show the output of grep 978 /etc/collectd/collectd.conf

rotund vault
#

Reboot (actually a shutdown, power off, power on and normal startup) has not given me 978 graphs.

orchid wadi
#

OK... does /run/adsbexchange-978/aircraft.json exist?

rotund vault
#

yes

orchid wadi
#

then I would try running sudo bash /usr/share/graphs1090/install.sh

rotund vault
#

Odd result...

#

pi@adsbexchange:/ $ sudo bash /usr/share/graphs1090/install.sh
16406 (process ID) old priority 0, new priority 10
/ /
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 27 (delta 18), reused 9 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (27/27), 14.02 KiB | 398.00 KiB/s, done.
From https://github.com/wiedehopf/graphs1090

  • branch master -> FETCH_HEAD
  • 639ac71...c6cf415 master -> origin/master (forced update)
    HEAD is now at c6cf415 airspy: DF counts, add DF20 somehow this got missed
    /

Install in progress, this shouldn't take longer than a minute or two ..........
/usr/share/graphs1090/install.sh: line 161: Plugin: No such file or directory
[ERROR] Error in line 161 when executing: face "'$iface'"' /etc/collectd/collectd.conf && continue
# only add interface starting with et en and wl
case $iface in
et*|en*|wl*)
sed -i -e '/ "interface" Interface "'$iface'" < Plugin > /{a
pi@adsbexchange:/ $

GitHub

Graphs for readsb / dump1090-fa / dump1090 (based on dump1090-tools by mutability) - wiedehopf/graphs1090

#

Just in case something useful happened, I rebooted. Still no 978 graphs.

orchid wadi
#

it's like the script got corrupted somehow

#

what's the output from grep -C3 Plugin /usr/share/graphs1090/install.sh (please enclose it in triple backticks for readability)

rotund vault
#

Not sure what you meant by "backticks". The output in the native SSH under Windows is no joy to read.

orchid wadi
#

I mean ``` then the output then ``` again

rotund vault
#

is this right?'''

#

'''pi@adsbexchange:~ $ grep -C3 Plugin /usr/share/graphs1090/install.sh
# only add interface starting with et en and wl
case $iface in
et*|en*|wl*)
sed -i -e '/<Plugin "interface">/{a
Interface "'$iface'"
}' /etc/collectd/collectd.conf
;;
pi@adsbexchange:~ $'''

orchid wadi
#

those are forward ticks - backtick is top-left corner of the keyboard (usually)

rotund vault
#

Yes, finally figured it out - ```

#
    # only add interface starting with et en and wl
    case $iface in
        et*|en*|wl*)
sed -i -e '/<Plugin "interface">/{a\
    Interface "'$iface'"
}' /etc/collectd/collectd.conf
        ;;
pi@adsbexchange:~ $```
orchid wadi
#

that's the one ๐Ÿ‘

#

That looks normal. The error you got is bizarre. Trying to think what would cause that....

#

but I reproduced it.... investigating....

#

meanwhile, try this instead: sudo bash -c "$(curl -L -o - https://github.com/wiedehopf/graphs1090/raw/master/install.sh)"

rotund vault
#
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11109  100 11109    0     0  38233      0 --:--:-- --:--:-- --:--:-- 38233
14302 (process ID) old priority 0, new priority 10
/home/pi /home/pi
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
From https://github.com/wiedehopf/graphs1090
 * branch            master     -> FETCH_HEAD
HEAD is now at c6cf415 airspy: DF counts, add DF20 somehow this got missed
/home/pi
------------------
Install in progress, this shouldn't take longer than a minute or two ..........
---------
write reducing measures already enabled, no need to do anything for this script!
---------
--------------
--------------
All done! Graphs available at http://192.168.1.45/graphs1090
It may take up to 10 minutes until the first data is displayed
pi@adsbexchange:~ $```
orchid wadi
#

Good... now grep 978 /etc/collectd/collectd.conf again?

rotund vault
#

UAT Graphs Now Report! Outstanding! Thank you!

orchid wadi
#

Sweet! ๐Ÿ‘

rotund vault
#

No data in them, yet. Cautiously optimistic.

#

Data has begun. Nice!

#

It will take a few days to see if I have achieved the goal of stability. The original trouble I eventually realized that the whole feeder software was halting and exiting. Very few clues as to why.

rotund vault
#
URL_978 "file:///usr/share/graphs1090/978-symlink"
pi@adsbexchange:~ $```
orchid wadi
#

That's not the same ๐Ÿ˜‰

#

before it was #URL_978 http://localhost/skyaware978

#

that's "commented out", so doesn't do anything - it's the default when there's no 978 data found

rotund vault
#

You're right. I missed the minor change - details are everything in programming

orchid wadi
#

indeed!

rotund vault
#

Is your feeder dseven_wl ?

orchid wadi
#

yes

rotund vault
#

It does well

orchid wadi
#

about as well as I can do from this location. I'm at sea level, in a low traffic area, with hills most of the way around me

rotund vault
#

Outside antenna?

orchid wadi
#

Yes, colinear on the roof of an outbuilding

rotund vault
#

My 1090 and 978 are separate, simple ground planes and in the attic. The only way I can put any outside would be to hide them in a tree. Not optimum either way.

orchid wadi
#

1090 spider is tiny - shouldn't be too noticable.... weatherproofing may be an issue, though

rotund vault
#

For now, I am hoping it is stable. I have another project coming up where I plan to build another feeder into a large RV (caravan?) I will make sure to add a USB GPS to take advantage of the MLAT switching (off when moving.)

orchid wadi
#

Cool. Fingers crossed on the stability ๐Ÿคž

rotund vault
#

Thanks again for your help!