Background/scheduled downloader daemon

[dumky]

When you subscribe to RSS feeds or newspapers via Amazon's service, the Kindle automatically downloads them. There must be some kind of scheduler "daemon" on the device, which presumably has a queue of content that it needs to refresh.

It would be interesting to figure out if it is possible to make use of this daemon to refresh content outside of Amazon's service.

For example, FeedBooks.com offers a catalog of their ebooks, in a mobi format. The user is supposed to use the browser to go and download that catalog. From there the user can pick and choose from the catalog and download individual items.
Would it be possible for the FeedBooks.com catalog to get automatically refreshed once a week?

This is just an example of course and there are many applications beyond that, if someone could find a hack to do it

Cheers,
Julien

[snookums]

According to Igorsk blog article on hacking the kindle command line, this is the result of a check of all running processes. Since he probably wasn't downloading a book at the time, the daemon may not be one of these processes unless it polls amazon frequently to check for new purchases.

[root@kindle root]# ps -A f
  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:01 [swapper]
    2 ?        SN     0:00 [ksoftirqd/0]
    3 ?        S<     0:00 [events/0]
    4 ?        S<     0:00  \_ [khelper]
   20 ?        S<     0:10  \_ [kblockd/0]
   87 ?        S      0:02  \_ [pdflush]
   89 ?        S<     0:00  \_ [aio/0]
   86 ?        S      0:00  \_ [pdflush]
   10 ?        S      0:00 [sleepd]
   33 ?        S      0:00 [khubd]
   88 ?        S      0:00 [kswapd0]
  676 ?        S      0:12 [voltd]
  678 ?        S      0:02 [pnlcd_animate]
  681 ?        S      0:00 [kseriod]
  710 ?        S      0:00 [wantph]
  709 ?        S      0:00 [wanend]
  721 ?        S      0:00 [mmcdd]
  727 ?        S      0:00 [hpdetd]
  740 ?        Ss     0:00 init    
 1116 tts/2    Ss     0:00  \_ -sh
 2344 tts/2    R+     0:00      \_ ps -A f
  831 ?        S      0:00 [kjournald]
  884 ?        S      0:03 /sbin/syslogd -m 0 -b 1 -S -s 250
  887 ?        S      0:01 /sbin/klogd
  976 ?        S      0:00 [eink_fb_apt]
  974 ?        S      0:04 [eink_fb_udt]
  975 ?        S      0:00 [eink_fb_sst]
 1023 ?        S      0:07 [f-s-gadget]
 1024 ?        S      0:00 [f-s-activity]
 1063 ?        S      0:00 [wdtpmd]
 1071 ?        S      0:00 /usr/sbin/watchdogd -k 9 -t 30
 1079 ?        S      0:00 /usr/sbin/netwatchd -d 20 -t 5 -p www.amazon.com
 1086 ?        S      0:03 /usr/sbin/nomkd -v 80 -r 44 -d 23 cvm
 1092 ?        S      0:00 crond -l 9 -c /etc/crontab             
 1097 ?        S      0:00 /bin/sh /usr/sbin/tphmonitor
 1101 ?        S      0:00  \_ /usr/sbin/tphserver -f
 1119 ?        S      0:00 /bin/sh /usr/sbin/execmonitor
 1128 ?        S      0:00  \_ /usr/sbin/execserver
 1123 ?        S      0:00 /bin/sh /opt/amazon/ebook/bin/run_framework
 1169 ?        S      0:00  \_ /bin/sh /opt/amazon/ebook/bin/start.sh
 1173 ?        SL     0:18      \_ /usr/java/bin/cvm -Xmx16m -Dsun.boot.library.path=/opt/usr/java/lib:/usr/java/lib -cp :/opt/amazon/ebook/lib/MobiCore-impl.jar:/opt/amazon/ebook/lib/MobipocketCoreReader.jar:/opt/amazon/ebook/lib/ReaderSDK.jar:/opt/amazon/ebook/lib/SearchSDK.jar:/opt/amazon/ebook/lib/framework-api.jar:/opt/amazon/ebook/lib/framework-impl.jar:/opt/amazon/ebook/lib/jdbm.jar:/opt/amazon/ebook/lib/json.jar:/opt/amazon/ebook/lib/kxml2.jar:/opt/amazon/ebook/lib/xyml.jar:/opt/amazon/ebook/booklet/AudiblePlayer.jar:/opt/amazon/ebook/booklet/AudioPlayer.jar:/opt/amazon/ebook/booklet/Browser.jar:/opt/amazon/ebook/booklet/ContentManager.jar:/opt/amazon/ebook/booklet/Demo.jar:/opt/amazon/ebook/booklet/Experimental.jar:/opt/amazon/ebook/booklet/Home.jar:/opt/amazon/ebook/booklet/MobiReader.jar:/opt/amazon/ebook/booklet/PictureViewer.jar:/opt/amazon/ebook/booklet/PrefBooklet.jar:/opt/amazon/ebook/booklet/Search.jar:/opt/amazon/ebook/booklet/XymlBooklet.jar:/opt/amazon/ebook/booklet/msp.jar:/opt/usr/java/lib/libjnisystem.jar -Ddebug=1 -Dcheck_comm_stack=true -Dhttp.keepalive.timeout=60000 -Dhttp.maxConnections=16 -Dallow_demo=false -Dawt_fb_enable=0 -Dextkeyboard=false -Dconfig=/opt/amazon/ebook/config/framework-unix.conf -DENABLE_SEARCH_INDEXING_THREAD=true -Dprintdebugtime=false com.amazon.ebook.framework.Main
 (around 30 cvm copies skipped)
 2298 ?        S      0:00 [mmcqd]

netwatchd appears to be a daemon that is checking to see if www.amazon.com is available. That may mean that the software doesn't know when the network switch is turned off. It may only know when it can't reach amazon. According to the source I found, netwatchd can run some scripts every time the network is found to be connected.

Have to go now. Will finish this comment when I have more time.

[snookums]

watchdogd is a daemon that triggers watchdog at regular intervals. watchdog is a program that causes the computer to reboot if the kernel halts. That's what I've gleaned. So it can't be our perp.

to be continued...

[igorsk]

I believe that's managed by "tphserver" process which probably gets events from the CDMA module. Search for "tph" in your logs.

[snookums]

[f-s-gadget] and [f-s-activity] are probably related to the File-backed Storage Gadget entry in the log file.

[mmcdd] probably has something to do with the sd slot. Kindle has sd instead of mmc cards, but mmc disk drive/daemon sounds right. There seems to be a asm/arch/mmc.h file in the kernel code. I couldn't be dead certain what it is, but my guess, for what it is worth, is media card driver.

We could go through everything in the list, but I'm betting that netwatchd checks amazon every couple minutes to see if the network connection is available. If amazon is there, say if the network switch is on and you're in a reception area, it probably updates the content list and downloads what it needs to download.

I managed to pluck this out of google cache. It was one of the only content links I found when I searched for netwatchd. As I'll explain below, this is not the same program as is running on the Kindle. I'm merely posting this because I think that it may give an indication of what netwatchd may do on the Kindle.

A daemon process to monitor a network connection and take action when a remote host is accessible.
Netwatch - Schedule Jobs To Run Depending On Network State

One of the eternal annoyances of using a laptop that is only occasionally connected to a network is how to schedule jobs so that they only run when connected. Typically, as soon as I connect to my home network, I want to run fetchmail, switch wwwoffle to online mode, and synchronise the system date with my local server. While connected, I want to run fetchmail periodically. In addition, I'd like to ensure that some jobs I would normally schedule with cron, anacron, or at, are run only when connected. For example, once a day or thereabouts I run the fetchnews program for leafnode. All of this I can now achieve with the aid of my perl script, netwatchd.

The netwatch deamon

The netwatch daemon, netwatchd runs continuously in background and periodically performs an arping to a server on the local network. The ip address and interface for the arping are set in the TEST_HOST and TEST_IF variables in netwatchd. If the arping ws successful, netwatchd assumes the network is now connected, otherwise it assumes disconnected. Depending on the current and previous connection status, netwatchd executes scripts found in the following directories:

/var/spool/netwatch/up

Scripts here are run each time the network is found to be connected

/var/spool/netwatch/down

Scripts here are run each time the network is found to be disconnected

/var/spool/netwatch/transition_up

These scripts are run if the network is now connected but was disconnected

/var/spool/netwatch/transition_down

These scripts are run if the network is now disconnected but was connected

/var/spool/netwatch/up_once

These scripts are run and then deleted if the network is connected

/var/spool/netwatch/down_once

These scripts are run and then deleted if the network is disconnected

netwatchd will also perform the network check and run the appropriate scripts if it receives a SIGUSR1, so it can be induced to check the network every time the laptop is awakened from suspend mode by adding the following to /etc/sysconfig/apm-scripts/apmcontinue:

if [ $1 == "resume" ] ; then
if [ -f /var/run/netwatchd.pid ] ; then
kill -USR1 `cat /var/run/netwatchd.pid`
fi
fi
For this to work correctly you should get into the habit of plugging in your network cable before switching the laptop back on

Using Netwatch With cron, anacron, Or at

To run a command the next time the laptop is connected, or the next time after a specified time, use at, cron, or anacron to copy or cat a script into /var/spool/netwatch/up. I keep the most commonly used scripts in /var/spool/netwatch/template and simply copy them from there when required.

Controlling The Daemon

The supplied init script, when copied into /etc/sysconfig/init.d allows the daemon to be controlled by /sbin/service. Link into /etc.sysconfig.rc*.d in the usual manner for automatic start and stop.

Security

It doesn't take a great deal of imagination to realise that on an untrusted system this utility could be a glaring security hole. So please pay attention to the following safegaurds
Make sure that the /var/spool/netwatch/* directories are writeable only by root
Make sure that any scripts you place in the /var/spool/netwatch directories are owned by root and read only
Patch netwatchd to setuid to the owner of a script file if you don't do the above, then mail me your patch ;-)

I checked the file listing the igorsk placed on his blog. The script folders listed here aren't present on the Kindle's file system. There also isn't a perl install which means that this isn't the perl script described above. That doesn't mean that it couldn't have a similar function. The boot log calls it network connection watchdog. it also lists a copyright to Lab126. It is listed in the Kindle boot log as starting at boot.