[Solved] Not so strange random crashing

[cubemike99]

1. Connecting with ssh over wifi.
2. Yes. Now that you mention it, it's only ever crashed on the screensaver or the home screen, never an app or book.
3. I don't know what difference telnet will make, since there is no pattern at all to the crashing. Sometimes it's when I'm connected, sometimes not. The most common situation seems to be when I'm disconnected and the device has been in screensaver for multiple minute. There's no way to invoke it; it's completely unpredictable. It hasn't crashed on me for at least 4 hours now (probably more), and I've been using it, connecting to it, etc.

USBnetworking - 0.9
kTerm - 0.6
KUAL - 2.0

Basically, everything is the newest.

P.S.:
Log backups seem to be in /var/local/log. Strangley, theres alot more netlog and wpa supplicant logs than messages.

[hawhill]

Originally Posted by cubemike99

So after a brief period with no random restarts post update to 5.3.6, they're back. If the logs are to be believed, it's segmentation faults that are making it restart. It's so bizarre, too. I just had it restart on me 4 times with no time in between.

If it's (also) the java processes or Amazon processes that segfault (i.e. not just hack-provided) - my bet is on hardware failure (RAM, to be specific, or timer/osc).

The indexer is generally under suspicion, of course, so watching if it can be triggered with an empty document partition is always worthwhile. But if it were my device, I've called Amazon CS and have it exchanged, if possible.

[knc1]

3) No Wifi modem running, nor any activity in its supporting software. The sshd loaded, but not handling any activity.

telnetd is a separate utility, and should be "safe" over the USB cable.

PS: Which sshd are you using on the Kindle?
Dropbear or OpenSSH?
It would be nice to know which one we are eliminating from the possible causes.

- - - -

Background - -

"SegFaults" is something reported because of user-land code.

*) It is not reported in Kernel code space, that is a "Kernel Oops".
*) User-land code **should not**, **ever**, cause a "Kernel Oops". (In theory, but it can happen in practice although very rare.)

As a data point -
With the older version of USBnetworking (it was recently rebuilt);
Using TelNet, not either of the sshd deamons;
Device displaying its screensaver, but **not** in cpu suspend;
In "airplane mode" (Wifi modem powered down);
TelNet in "command mode" and connection up to Kindle;
TelNet deamon not responding (normal at this point);

I can cause a "Kernel Reboot" by causing the local telnet instance to query the remote deamon for its status.

Translation: Clearly an error in Amazon's kernel build and/or patches.
It is very hard to get the Linux network stack to crash in an un-modified kernel that is properly built.

Ref: Hawhill's posts - -

At this point in the testing -

Start checking on the warranty term of your Kpw -
it may be going back for replacement.
In fact, it probably is going back for replacement.

My guess at this point is very close to Hawhill's -
Either RAM failures or failures related to the watchdog timer(s).

[NiLuJe]

@knc1: Do you have a log of the OOPS or BUG that it triggers?

[knc1]

Originally Posted by NiLuJe

@knc1: Do you have a log of the OOPS or BUG that it triggers?

No.
It caught me totally unprepared. (Although I did manage to hit the "dmesg -> doc" button as soon as it came back up.)

Note: That this was telnet **not** either of the ssh daemons.

Will try to capture something more useful than the: "Kernel Reboot" message.

[cubemike99]

I'm using dropbear.

If this is a ram issue, should it be possible to craft some program or script that attempts to fill up every possible memory location, causing the device to crash when the bad memory is attempted to be used?

As to the timer/oscillator, the logs have multiple entries of the device going into deep sleep but setting an "alarm clock" to wake up in a few tens of thousands of seconds.

Luckily I'm still under warranty for another 3 months at least. From the logs though, it hasn't segfaulted for like 20 hours.

[knc1]

Originally Posted by NiLuJe

@knc1: Do you have a log of the OOPS or BUG that it triggers?

Not really.
But something else strange - garbage characters in the kernel ring buffer:

# [  45?.628381] PM: early resume of devices coiplete after 0.135 msecs
# [  455.628437] kernel: I PM:earlyresume:duration=0.135:
# [  455.629778] KDRNEL: I pmic: PWRON1 evelt:
# [  455.856842] mmc0: queuing CIS tuple 0x01 length 2
# [  454.864179] mmc0: queuhnc CIS tuple 0x1a length 5
# [  455.867408] mmc0: queuing CIS tuple 0x1b length 8
# [  455.868071] mmc0: queuing CIS tuple 0x14 length 0
# [  456.029587] PM: resume of devices complEte after 400.960 msecs
# [  456.029601] kernel: I PM:resume:duration=400.960:
# [  456.030340] Restarting tasks ... cyttsp: I cyttsp_proc_write::command=unlock:
# [  45&.076110] done.
# [  456.431&97] g_ether gadget: high speed config #1: CDC Ethernet (ECM)
# [  458.023908] Battery: Pesume - TimeSuspended(secs)=15, NAC=0x1e71, RSOC=98, capacity=98%
# [  458.102757] Battery: Resume - PostUpdate NAC=0x1e6d, RSOC=98 
# [  105.237943] KERNEL: I pmic: PWRON1 event:
# [  105.413464] cyttsp: I cyttsp_proc_write::command=lock:
# [  166.186113] aplite_charger: I pmic:faulti:sense_0=0x200:battery dies/charger times out.
# [  172.887084] g_ether gadget: high speed config #1: CDC Ethernet (ECM)
# [  262.672873] PM: Synci.g filesystems ... done.
# [  262.686241] Freezing user space processes ... (elapsed 0.00 seconds) done.
# [  262.690053] Freezing remaining breezable tacks ... (elapsed 0.00 seconds) done.
# [  262.690945] Suspending console(r) (use no_console_suspend to debug)
# [  262.692239] cyttsp_suspend: Enter
# [  262.713060] PM: suspend of devices complete
# [  262.713119] kernel: I PM:suspdnd:duration=20.834:?
# [  262.714117] PM: late suspend of devices complete after 0.933 msecs
# [  262.714173] kernel: I PM:latesuspend:duration=0.933:
kernel: E savemsg:--------Kernel Reboot Message End--------:

I haven't seen garbage in the ring buffer since the days of kernel 2.4.x
*AND*
dmesg -c
**should** have read from the start of the ring to the end of the ring - -
But going by the time-stamps, that is a copy of a "wrapped" ring buffer.

- - - -

I wasn't trying to troubleshoot this reported problem -
I was playing with the various "states" that the Kindle system can run in -
All in preparation for finding out how to put it into "downloader" mode (or whatever a Kpw uses).

[hawhill]

Looks a bit like garbage that occurs when processor caches are not handled correctly. Have there been kernel updates lately that messed with that? The /arch subtree of the kernel is a wild jungle for the ARM architecture with scary beasts cowering within...

FWIW, I've had sudden reboots on my PW lately, too. But I blame that on the indexer being a catastrophic mess - I'm still on an ancient firmware version and the device hasn't had Wifi on for a long time and was generally off the USB line when the reboots happened. Plus, they only occured after sideloading my latest batch of literature. That's pretty obvious in my eyes.

[ixtab]

Originally Posted by hawhill

FWIW, I've had sudden reboots on my PW lately, too. But I blame that on the indexer being a catastrophic mess - I'm still on an ancient firmware version and the device hasn't had Wifi on for a long time and was generally off the USB line when the reboots happened. Plus, they only occured after sideloading my latest batch of literature. That's pretty obvious in my eyes.

Not only the indexer - the entire PW firmware is a disaster (but the indexer stands out as a disaster²). Judging from the problem reports here at MR, no previous Kindle model has exposed so many different weird forms of software malfunctions. (and here's my rant).

[knc1]

Originally Posted by hawhill

Looks a bit like garbage that occurs when processor caches are not handled correctly. Have there been kernel updates lately that messed with that? The /arch subtree of the kernel is a wild jungle for the ARM architecture with scary beasts cowering within...

FWIW, I've had sudden reboots on my PW lately, too. But I blame that on the indexer being a catastrophic mess - I'm still on an ancient firmware version and the device hasn't had Wifi on for a long time and was generally off the USB line when the reboots happened. Plus, they only occured after sideloading my latest batch of literature. That's pretty obvious in my eyes.

An error in the cache handling sounds like a good guess.

This was (and is) v-5.3.3 - - Same as the original trouble report here.

(oh, what I would give for enough hobby time to do more than just write about porting Tizen to the Kindles! Even if they did end up looking like a Samsung device.)

[NullNix]

Originally Posted by knc1

@cubemike99: Lets go with "ps" being internally limited to displaying a single "memory page" worth of command line, rather than the whole thing.

Yeah, ps just reads /proc/$pid/cmdline, and that is hardwired to display at most one page of output (see fs/proc/base.croc_pid_cmdline() in the kernel source). Suboptimal really, but nobody's souped it up to iterate across the full command line yet.

[knc1]

Originally Posted by NullNix

Yeah, ps just reads /proc/$pid/cmdline, and that is hardwired to display at most one page of output (see fs/proc/base.croc_pid_cmdline() in the kernel source). Suboptimal really, but nobody's souped it up to iterate across the full command line yet.

Indeed.
That issue has seen a lot of discussion over time.

The current position is: "Move your output to the /sys tree if you need more than a single page 'copy-to-user-space' "

auFS had to move out of /proc into /sys because of the size limitation, for instance.

[cubemike99]

Now I'm inclined to think it's the indexer, too. Like hawhill mentioned, the reboots happen some time after loading new books onto the device, and then seem to stop. I last loaded books onto my kindle 2-3 days ago, and that's the last time I encountered a reboot.

[knc1]

Just re-read your original post (with the information about adding books mentally included) - - -

Yup, have a moderator re-title this thread "Not so strange, random crashing" and call the problem solved.

You where being fooled by the appearance of the screen saver that the system was idle.
That is not how the Kindle system works - it hides behind the screensaver when it is doing its internal "work list".

The key was in your initial post, in the:

P.S.: This crashing truly is random. It even happened a few times while the kindle was on its screensaver with me not logged in with ssh.

I just did not read your initial post well enough, sorry.

[cubemike99]

Well it's a relief to be reasonably certain this is a software issue and not some busted ram. I'm perfectly content with just dealing with a few random reboots after adding a few new books. Two things still don't sit perfectly well with me about this though. First, the reboots are a relatively new thing: after I first got my pw, I used it for a few months with no reboots, left it alone for a month or two, and just recently started using it again and began noticing the reboots. Second, if this is the fault of a buggy indexer, why is it not a bit more common if not ubiquitous?

And how would I go about requesting a mod to change the thread title?