[Kindle Touch] Calibre Kindle Collections Plug-in Hooks

[sumpin]

This extension converts the collections in the Content Catalog (/var/local/cc.db) on the Kindle Touch to a JSON file (/mnt/us/system/collections.json) that can be read and updated by the Calibre Kindle Collections Plug-in. It then inserts/updates the collections managed by Calibre in Content Catalog. The script is compatible with collections and content that only exist on the Kindle and not in Calibre per the plug-in and books may exist in multiple collections.

Requirements:
1. Jailbreak - https://www.mobileread.com/forums/sho...d.php?t=160454
2. Launcher - https://www.mobileread.com/forums/sho...d.php?t=161263
3. Calibre - http://calibre-ebook.com/download
4. Calibre managed data stored on the Kindle in documents/calibre (i.e., under Calibre Preferences > Sending books to devices, Save template starts with: calibre)
5. Sqlite3 built with loadable extension support to run on the Kindle (included)

Install:
1. Extract both tar files to /mnt/us/extensions (If your zip file extractor has a setting like "TAR file smart CR/LF conversion", make sure it is disabled or the scripts cannot run)
2. Restart Kindle

To Use:
1. Select “Create input for Calibre” to export the existing collections
2. Connect Kindle to computer, run Calibre Kindle Collections Plug-in, eject Kindle
3. Select “Import data from Calibre and reboot” to import the new/modified collections

Note:
By default, when the Calibre data is loaded, the access times of all books is updated (avoid by changing –uat to +uat in menu.json) and the database is cleaned up by deleting the Collections Journal and by issuing an SQL VACUUM command (avoid by changing –vacuum to +vacuum in menu.json). To disable removing collections with no entries on import, add +rec to the command line of json2db.sh in menu.json.

By default, 5 copies of cc.db are maintained in /mnt/us/system. This can be changed by modifying the command line params of the json2db.sh script in menu.json.

If you run the import procedure too soon after loading data via Calibre, you may find that some books that are in a collection show up as not in a collection. Simply run the import again.

This supersedes my input to https://www.mobileread.com/forums/sho...d.php?t=160855 as that thread has taken the directory structure driven approach of CollectionSync https://www.mobileread.com/forums/sho...1&postcount=16.

I've split the release up into two parts. One tar file contains the sqlite3 binary and loadable extension needed and the other contains the Launcher hooks and scripts that do all the work.

I added a "progress bar" to the non-verbose import/export screens to give feedback that it's ticking along when importing large databases and I added verbose import/export to the menu to aid in trouble shooting and boredom reduction.

Updated for firmware 5.1.0 (use dynamic launcher menu, eips command line changes). Add menu item to update access times and remove this as default behavior.

Attachment d2j2d.tar.gz last updated on April 26th, 2012.

[dave2008]

nice hack :-)

[JustAMan]

Small advice - it would be better to store resulting .db somewhere else (or at least store copies not in /var/local as it easily fills up and thus breaks KT).

[sumpin]

Good point. I updated the script to store the cc.db backups in /mnt/us/system as well.

[JustAMan]

And another idea - for those who don't install SSH it would be impossible to restore backup... Maybe add some menu item that would do exact that?

[nynaevelan]

I've tried running this but it does not seem to be working to put the books into collections. Perhaps I am misunderstanding something. If I put new books on the kindle, then run the kindle collections plugin, then run your script and reboot. Shouldn't it put the new books into the collections based on what the kindles plugin did?

[sumpin]

Originally Posted by nynaevelan

I've tried running this but it does not seem to be working to put the books into collections. Perhaps I am misunderstanding something. If I put new books on the kindle, then run the kindle collections plugin, then run your script and reboot. Shouldn't it put the new books into the collections based on what the kindles plugin did?

This is exactly what should happen. I've noticed that every time a new book gets added to cc.db, indexing and who knows what else kicks off. I've noticed that if I import the collections.json from calibre into cc.db before the cpu settles down, some of the books don't show up as in collections. I just re-run the import and all is dandy.

Also, if you're not storing your calibre content in documents/calibre, the script will skip those collections.

Just to prove what a masochist I am, I deleted my cc.db file. A zero length file was immediately created and it took a *long* time before all the content was re-added and re-indexed. When the indexing was done, I ran the export/calibre/import and got all my collections back.

Since stdout and stderr go to /dev/null it's hard to determine what's going wrong. The best bet is to ssh into the Kindle and run json2db.sh -v to see the output.

[wuschel83]

doesn´t work for me, all my folders are invisible now . how can i get them back to my screen???

[wuschel83]

i have delete the cc.db, after this i could start new:
rm /var/local/cc.db

[sumpin]

Originally Posted by JustAMan

And another idea - for those who don't install SSH it would be impossible to restore backup... Maybe add some menu item that would do exact that?

Done.

[nynaevelan]

I took some time away from this and now I am back. I have run the import process twice and still the books are not showing up in collections. The wheel is not turning, how do I know if it doing anything?

[pookakitten]

I'm not sure if this is caused by this or the Kindle Collections plugin, but it won't delete empty collections. The plugin reports less collections, but they never disappear when I reboot the kindle. The only way to remove the collections is to do a factory reset which is a pain, to say the least.

For now, I'm going to stick to creating tags based on tags which is genre for most books and fandom for fanfiction.

[lsi]

I tried to implement this, but I cannot connect using SSH.

The first to steps were OK:
1. Jailbreak - https://www.mobileread.com/forums/sho...d.php?t=160454
2. Launcher - https://www.mobileread.com/forums/sho...d.php?t=161263
I was able to set up the USBnetworking, and my computer (Linux Mint) sees the network and connects. I use static IP 192.168.15.1.

Then when I run "sudo ssh root@192.168.15.244", I get "ssh: connect to host 192.168.15.244 port 22: Connection refused".

What else could I try?

[knc1]

Originally Posted by lsi

I tried to implement this, but I cannot connect using SSH.

The first to steps were OK:
1. Jailbreak - https://www.mobileread.com/forums/sho...d.php?t=160454
2. Launcher - https://www.mobileread.com/forums/sho...d.php?t=161263
I was able to set up the USBnetworking, and my computer (Linux Mint) sees the network and connects. I use static IP 192.168.15.1.

Then when I run "sudo ssh root@192.168.15.244", I get "ssh: connect to host 192.168.15.244 port 22: Connection refused".

What else could I try?

nmap

Let it search for the address of the Kindle usbnet, which may not have recognized your entry of the address 192.168.15.244 in its configuration.

nmap 192.168.15.0/24
will report any addresses in that 256 address subnet range actively in-use and if port 22 is open on any of them.

"man nmap" for lots more details.

[lsi]

Originally Posted by knc1

nmap

Let it search for the address of the Kindle usbnet, which may not have recognized your entry of the address 192.168.15.244 in its configuration.

Hi knc1! Thanks for your suggestion. This is what nmap says:
nmap 192.168.15.0/24

Starting Nmap 5.21 ( http://nmap.org ) at 2012-04-04 16:46 CEST
Nmap scan report for 192.168.15.1
Host is up (0.00022s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
139/tcp open netbios-ssn
445/tcp open microsoft-ds

Nmap scan report for 192.168.15.244
Host is up (0.015s latency).
All 1000 scanned ports on 192.168.15.244 are closed

How should I word my ssh?