Firmware 2.1.4 Plugin Alpha Release

[davidfor]

Originally Posted by textchimp

Anything in particular? I'm setting margin and padding already...

I mean that the paragraphs may be using a specific style rather than just the default for "p". Check the html files for styles on the paragraphs.

[textchimp]

I'd like to use this plugin to add a custom download option to the 'Tools' menu on the home screen. I guess this would involve using executeSystemCmd to run a shell script on the device which would (ideally) download from a remote HTTP server a text file containing a list of file URLs to be downloaded, then download each file individually.

On a Linux-based computer this is a relatively easy task, using a command line tool like 'wget' to download the remote files, but on the Kobo I'm not sure what command to use to retrieve remote URLs. Does anyone know if there is any way to do this by running such a command on the device?

[sven]

The Kobo is also running a Linux-based OS and has the wget command available. Thus, you can implement your script like on any other Linux box.

[textchimp]

Originally Posted by sven

The Kobo is also running a Linux-based OS and has the wget command available. Thus, you can implement your script like on any other Linux box.

Ah, good to know! However I've just tried to run wget without apparent success. Should something like

Plugin::executeSystemCmd("/bin/wget URL")

do what I want? Will wget be in /bin ? Is there some way to log output from these commands?

Also, where will this downloaded file be saved? I guess I want to save it to the root directory (how do I refer to this internally for a system command?), but then, will the Glo notice that a new file has appeared there and needs to be added to my library, or do I have to manually call some kind of library update?

[Koboyashi]

Try /usr/bin/wget - and here's a man page of wget.

[textchimp]

Okay, I've got a successful custom sync working, at least as a proof-of-concept version. It currently requires you to have access to your own web server, running PHP (I'm running this on my own Linux desktop) - this is because we need to generate a list of file URLS for the script on the Kobo to retrieve. If anyone knows how to e.g. get the built-in Calibre HTTP server to generate a plain text file of new files in the last X days, that would be even better. This has been tested only on the Kobo Glo with firmware 2.1.5.

Here's how it works:

Add to .kobo/koboplugins.ini:

CSync=Plugin::connectWifi()>>Plugin::executeSystemCmd("/mnt/onboard/run.sh &")

Note that this seems to force this entry in the menu to appear as "WiFi" - it looks like there's a bug in the plugin that forces the label to be "WiFi" whenever it's used, in a command sequence like this, or on its own. If you already have a WiFi entry in your menu, remember they appear alphabetically, so make sure you select the correct one.

Now create the following scripts in the Kobo root folder (i.e. the root folder when you access it as a USB drive):

run.sh

#!/bin/sh
touch /mnt/onboard/started_sync
nohup /mnt/onboard/sync.sh &

(I use this intermediate file to call the actual sync script, because if I try to call it directly, even with Plugin::executeSystemCmd("nohup sync.sh &"), the menu seems to freeze until the whole thing finishes.)


sync.sh

#!/bin/sh

# Add your HTTP server details here
LIST_URL="http://192.168.0.123/books/filelist.php";
PING_IP="192.168.0.123"

ATTEMPTS=30

#Uncomment the next line to always download files in the list (i.e. ignore 
existing downloaded file, save with unique name)
# NOTE: for testing only! Will create duplicate books on your device
#ALWAYS_DL=1

# wait for wifi network to come online; give up after $ATTEMPTS tries
tries=1
while ! ping -c 1 -q $PING_IP; do 
   if [ "$tries" -gt "$ATTEMPTS" ] ; then 
     timestamp=`date`
     echo "sync: aborted after $ATTEMPTS connection attempts ($timestamp)" >>/mnt/onboard/sync.log
     break; 
   fi
   sleep 2
   tries=$(( $tries + 1 ))
done

   
successes=0
LIST=`/usr/bin/wget -q -O - $LIST_URL`
IFS=$'\n'  # make the for loop break up the list on newlines

for url in $LIST; do
  filename=`echo $url|sed -e 's|.*/||g'`
  if [ $ALWAYS_DL ] ; then
    timestamp=`date +%Y%m%d-%k%M%S`
    if /usr/bin/wget "$url" -O "/mnt/onboard/$timestamp-$filename" >>/mnt/onboard/wget.log 2>&1 ; then
      successes=$(( $successes + 1 ))                                                                                                                 
      echo "Downloaded (forced) $url to $timestamp-$filename" >>/mnt/onboard/sync.log
    else
      timestamp=`date`                                                                                                                                
      echo "Failed to force download $url ($timestamp)" >>/mnt/onboard/sync.log                                                                
    fi
  else
    if /usr/bin/wget "$url" -O "/mnt/onboard/$filename"  >>/mnt/onboard/wget.log 2>&1 ; then 
      successes=$(( $successes + 1 ))
      echo "Downloaded $url" >>/mnt/onboard/sync.log
    else
      timestamp=`date`
      echo "Failed to download $url (file exists? $timestamp)" >>/mnt/onboard/sync.log
    fi
  fi
done

if [ "$successes" -gt "0" ] ; then 
  timestamp=`date`
  echo "Downloaded $successes files ($timestamp)" >>/mnt/onboard/sync.log
  nohup /mnt/onboard/reindex.sh &
fi

(You will need to put the correct address for your HTTP server at the appropriate place at the top of the script. Note that this script first checks that it can reach the server on the network. This is because the command sequence we defined in the menu turns on the Wifi but does not wait for the connection to be established before running the next command, i.e. this sync.sh script. So the script itself must keep checking and waiting until the network is up. It will give up after 30 attempts, about 60 seconds.

reindex.sh

#!/bin/sh
echo usb plug add >> /tmp/nickel-hardware-status 
sleep 10 
echo usb plug remove >> /tmp/nickel-hardware-status

This forces the Kobo to add any files downloaded to the Library by faking a USB connection and disconnection (found here: http://a3nm.net/blog/fnacbook_kobo_more_hacking.html). It works but requires you to tap 'connect' on the dialog it brings up on the screen, and then automatically disconnects and indexes a few seconds later. It works, but if anyone knows how to trigger a reindex in a less clunky and interactive way, please let me know.

These files all need to be set as executable, i.e. "chmod 755".

Here is the PHP script which will create the expected list of file URLs to download; the epub files must in the same directory as the script.

<?php
$path = substr($_SERVER[PHP_SELF], 0, strrpos($_SERVER[PHP_SELF], '/'));
$find = 'find . -iname \\*.epub'; //-mtime 0 ';

$ar = array();
exec($find, $ar);

foreach($ar as $file){
 $url = "http://" . $_SERVER['SERVER_NAME']  . $path . '/' . str_replace(' ', '%20', substr($file, 2));
 print "$url\n";
 //print "<a href=\"$url\">$url</a><br>";

}
?>

When you test this PHP script in your browser you should get a list of Epub file URLS, separated by newlines (they will all appear to be on the same line in your browser). If you don't see them, make sure you have some files in the same directory as the script, and check that they're accessible - uncomment the second "print" statement at the end to test that the links work.

Whew. Ok, once all that is in place, you should be able to automatically download files to your Kobo and have them automatically indexed (after the fake USB-connect sequence). The script will not re-download files that it has already saved to the Kobo unless you uncomment the ALWAYS_DL line in sync.sh, but note that this creates unique filenames each time it downloads a file, so you'll get multiple versions of the same book in your library.

The script does some logging to wget.log and sync.log in the same root directory as the scripts themselves.

TODO:

- Less interactive way to force checking/reindexing of new files. Anyone know if this can be done via a shell command?

- Neater way to make sure wifi is available before doing download

- Fix label bug that causes menu item to appear as "WiFi"

- Ideally it would be good to either serve the files from the Calibre server, if it can be made to generate a simple enough list of URLs of new files. Alternately, it would be good to be able to remotely host the files so they can be synced even when your own computer is off, i.e. Dropbox or similar; however with Dropbox there's no way to generate the list of file URLs, and no way to list the contents of a web folder to get all the files. If anyone can think of a generic alternative that would work in this case, that would be good.

- Text updates at the top of screen - currently it gets stuck on "Connecting", I think this is the Wifi command. Not sure how to change the this text.

- A way to extract emailed articles and save them as epub files, so they can be downloaded using this process, i.e. make use of browser extensions that allow you to "email this page to your kindle"... any ideas?


Attached is a zip file with the above scripts that need to be extracted into the root folder, and the PHP script for your server. The Kobo shell scripts will probably still need to have the correct execute permissions set - this will be hard if you haven't enabled telnet for the device.

Let me know how this works if you try it out.

[vlad59]

Great work !

If you want I can add a special page in Cops (http://blog.slucas.fr/en/oss/calibre-opds-php-server) that produce the correct file format.

It's been a while since I've used the plugin, but you had a goo idea.

[benjaminries]

Got a Kobo Glo last weekend. Running firmware 2.1.5. People on this thread seem to confirm that the plugin interface works on the Glo with this firmware.

I followed the instructions - connected my Glo via USB, unzipped both files to the .kobo directory, and ejected the device from my PC. I think the .tgz file is being read, because when I do this, the screen says that the software is updating.

My issue is that after that, the old stock interface is still there, with no changes. I open a book, and close it. Still no change. I shut down the Kobo, boot it back up, open and close a book, and still nothing.

I've tried this with my own *.ini content, and with an unmodified *.ini file downloaded from the first post of this thread. Neither has worked.

What am I doing wrong?

[textchimp]

Does anyone know if there's a command available on the embedded Linux running on the Kobo Touch/Glo which could send a file to a remote host? Ideally something like an automated FTP file send (i.e. ncftpput - but clearly that's not available), although even making an HTTP POST would probably be enough.

I'm trying to write a script to send all the annotations/highlights on the device to a remote server.

[SeigneurAo]

Originally Posted by textchimp

Does anyone know if there's a command available on the embedded Linux running on the Kobo Touch/Glo which could send a file to a remote host? Ideally something like an automated FTP file send (i.e. ncftpput - but clearly that's not available), although even making an HTTP POST would probably be enough.

I'm trying to write a script to send all the annotations/highlights on the device to a remote server.

Maybe wget ?
Quite widespread and basic command on Linux systems.

[koenieee]

Do we have curl access to our Kobo's?
Then we could use this: http://superuser.com/questions/86043...part-form-data (Second post)

I was thinking to create some sort script that synces epubs booksmarks (not the offical kobo books) with my android smartphone.

Does anyone know where the Kobo saves the bookmarks?
In the sqllite database I guess?

[textchimp]

Ha, ok yes, wget again. I've got this to work, again as a proof-of-concept, but the version of wget on the Kobo is limited - it does not have the --post-file=FILE command line option, but it does have (undocumented) the --post-data="DATA" option. This means you can send some post data, but only via the command line, and it seems to limit how much data you can send. (Anyone got a better idea?)

Anyway, here's a working example to grab all the annotation data from /mnt/onboard/Digital\ Editions/Annotations and post it to a PHP script on a server.

First create a menu entry in koboplugins.php like:

CSync=Plugin::connectWifi()>>Plugin::executeSystemCmd("/mnt/onboard/run-notesync.sh &")

Then as with the sync script above, we use an intermediate script to prevent the menu freezing up:

run-notesync.sh:

#!/bin/sh
touch /mnt/onboard/started_notesync
nohup /mnt/onboard/notesync.sh &

notesync.sh:

#!/bin/sh

ATTEMPTS=40

# change the addresses below for your server
REMOTE_URL="http://serveraddress/path-to/post.php"
PING_ADDRESS="serveraddress"
POSTDATA=""

# wait for wifi network to come online; give up after $ATTEMPTS tries
tries=1
while ! ping -c 1 -q $PING_ADDRESS; do
   if [ "$tries" -gt "$ATTEMPTS" ] ; then
      timestamp=`date`
      echo "sync: aborted after $ATTEMPTS connection attempts ($timestamp)" >>/mnt/onboard/notesync.log
      exit; 
   fi
   sleep 2
   tries=$(( $tries + 1 ))
done


POSTDATA=""
FILES=`find "/mnt/onboard/Digital Editions/Annotations/" -iname \*.annot -exec grep -l '<text>' {} \;`
IFS=$'\n'
for file in $FILES; do
  POSTDATA=${POSTDATA}`awk '/<dc:title>/ ||  /<dc:creator>/ || /<text>/ || /<\/text>/' "$file"`
  POSTDATA="${POSTDATA}\n\n"
done

#echo -e "$POSTDATA"

wget $REMOTE_URL -O /mnt/onboard/notesync.log --post-data="content=$POSTDATA"
timestamp=`date`
echo -e "\n\nSent on $timestamp \n" >>/mnt/onboard/notesync.log

This will extract just the title, creator and text tags from all the annotation data, in the interest of saving upload size via the limited wget command line option. As with the download script in my sync script post, it will first test the network connection until it successfully pings the remote host, otherwise it will quit.

(NOTE: it looks like the way the notes are saved in these .annot files ignores paragraph breaks - it just runs text from different paragraphs together. Annoying! In any case, the awk filtering done in the script will ignore lines without a <text> or </text> tag; needs a better multiline regex...)

On the server end, you need a simple PHP script like this:
post.php

<?php
if(isset($_POST['content'])) {
  file_put_contents('notes.html', $_POST['content']);
  exit;
}
?>

Then you can see your notes at http://server-address/path-to/notes.html.

[davidfor]

Originally Posted by koenieee

I was thinking to create some sort script that synces epubs booksmarks (not the offical kobo books) with my android smartphone.

Does anyone know where the Kobo saves the bookmarks?
In the sqllite database I guess?

The bookmarks are in three places depending on what you want.

The annotations and highlights are in the annot file that textchimp is using. These are also in a database table called "Bookmark". Hopefully any ADE based reader could reuse the file. Also, the file on the Kobo's takes precedence. If you delete the records in the table, the annotations are loaded from the file. If you delete the file, the annotations are removed.

The current place in the book is stored in the "content" table. Each book has a row with ContentType=6. Books that are being read have a value in the column "ChapterIDBookmarked". This has a reference to the current chapter. The reference is the key to the content table for a ContentType=9 row. This row has a value in the column "adobe_location". This is the current location in the chapter. The following SQL will show the current position for all sideloaded books:

SELECT c1.Title, c1.Attribution, c1.DateLastRead, c1.ChapterIDBookmarked, c1.ReadStatus, c1.___PercentRead, c2.ContentId, c2.Title, c2.adobe_location
FROM content c1 join content c2 on c1.ChapterIDBookmarked = c2.ContentID 
where c2.bookid like 'file:///mnt/onboard%'

Decoding the value in adobe_location is the only problem in this data. An example for me is "OEBPS/138508.xhtml#point(/1/4/278/1:222)". The first part is the name of a file inside the epub. The point must translate to some position in the file, but I haven't bothered to work out exactly what it means.

[sensoria]

Just wanted to say many, many thanks for this plugin. It's making my Kobo experience so much better. :-) Love the enhanced reading footer especially.

[jkgeyti]

textchimp: I'm also using the same approach to "syncing" books. You may want to use the plugin's command chaining and its sync mechanism instead of simulating a usb connecting. Also, the busybox wget will not work with secure connections. So you may want to use curl instead.

I'm currently also using a simple php script with a simple interface to add URLs that I want the kobo to download. When I get some free time, I hope to do Dropbox syncing using their REST api.