Issue with Calibre 0.7.1 & Kindle 2.5.2

[kovidgoyal]

I've already implemented support for displaying the collections made on the Kindle when you connect it to calibre. There appears to be no way to actually create/edit/delete collections using the USB connection. Here are my notes on the subject:

A collections cache is stored at system/collections.json
The cache is read only, changes made to it are overwritten (it is regenerated)
on device disconnect

A log of collection creation/manipulation is available at
system/userannotationlog

collections.json refers to books via a SHA1 hash of the absolute path to the
book (prefix is /mnt/us on my Kindle). The SHA1 hash may or may not be prefixed
by some characters, use the last 40 characters.

Changing the metadata and resending the file doesn't seem to affect collections

Adding a book to a collection on the Kindle does not change the book file at all
(i.e. it is binary identical). Therefore collection information is not stored in
file metadata.

[eboyhan]

Kovid,
In reading the post of the tests done by johan1fs at Amazon, I got the impression that if you place a collections.json file at the root of your kindle file system, and then restart the kindle, the collections are rebuilt from the contents of the json file. Note that whispernet has to be off or else the json file will be overwritten by the copy on the Amazon archive before the rebuild is done.

This is clumsy I admit, and you are probably right that there is no convenient way to create a collection from the Calibre side. Displaying the collections works for me.

I wasn't sure where the asin numbers in the book titles came from -- it seems that you have found the asin's are only in the filesystem names (and only for Amazon books). I still wonder what happens to my collections if I change the book title metadata on Calibre to remove the asin's? Will this break collections on the kindle side -- as the collections expect the asin's to be present.

[Steven_m64]

Interesting, I tested what eboyhan said as to editing the json file and placing it in the root dir before a restart.

that has no effect. but my kindle did rebuild the collections using a edited /system/collections.json after a restart.

this is proven by the fact that if the /system/collections.json file is missing on a kindle when its restarted and wireless is off all collections will be lost.

its annoying but at least it IS possible to edit collections externally.

[kovidgoyal]

@eboyhan: I don't know about books from amazon as I dont have any, but I tried changing the metadata on a side loaded book and resending it to the kindle and the collection weren't affected. Judging by collections.json the collections are keyed to absolute paths, so changing file metadata should not affect anything as long as the path does not change. Of course if you change the title/author, and resend using calibre, the path will change.

[eboyhan]

Kovid,
That is my point exactly. Amazon is placing the asin# (and some other gobbledygook) in the filename for their books -- presumably to guarantee that the collections feature can distinguish among the many different kindle versions of the same title that Amazon sometimes sells. When an Amazon book is brought over to Calibre all this extraneous stuff appears in the title. My question is: if I remove the extraneous stuff (it's ugly) on Calibre, and then transfer the book back to the kindle, will the filename put back on the kindle still be unchanged or not? Later today I'll run a few experiments with Amazon books, and let you know the results.

[kovidgoyal]

If you change the title, then no, it wont be the same.

[NiLuJe]

@eboyhan: AFAICT, when referring to an amazon book with intact DRM, they're not keyed to the sha1 hash of the absolute path like DRM-free mobipocket books, but to their ASIN number.

And that's still true for books that've been handled by calibre at some point, so the asin number isn't in the filename anymore. So I'm assuming it's also in the metadata somewhere, and that calibre doesn't touch it, even if you change the title/author metadata.

Note that I didn't try this with a book downloaded directly on to the Kindle from Whispernet with the 2.5 fmw and the 'new' drm, so that may not be true anymore, but I didn't have any problem at all with my 'old' books, whether they were first downloaded to the kindle or a computer, and they're all handled by Calibre.

[kwharrigan]

I whipped up a quick Python script to aid in listing/editing the collections.json file. I find it convenient so maybe you will too.

gist linky

I've had success placing this script on the Kindle volume itself and just running it straight from there. Only dependence is simplejson, though if you have 2.6 you can just use built-in json.

As indicated in previous posts, I can add any number of entries and following a reboot they show up in the list. I can also remove entries. Saves a little bit of time versus hand jamming the json.

The util does not yet associate documents to collections for you but that is the next step.

Would be nice if we could at least trigger a reboot from the USB..

[eboyhan]

@kovid

I have some questions about how metadata is created during the add books process from a kindle. I also have some Q's about what you do with the metadata as viewed in Calibre when a send to device is done back to the kindle. I did some experiments the other day with PDF files that were all members of various collections. I found that what Calibre came up with for title and author was very different from what the kindle had. I changed some of the Calibre-side metadata, and then did a send to device, and those documents which were in collections on the kindle before I add them to Calibre were not in any collections after I sent them back to the KDX.

Before I ask my questions let me describe my understanding of how Calibre gets the metadata it displays. Metadata can come from 3 possible sources: the content of the eBook, the filename of the eBook, or from three places on the internet: ISBNDB.com, google books, and/or Amazon.com. The Fetch-ebook-metadata command (and its GUI equivalent) are used to augment the metadata gotten from inside the eBook, or from the filename (there is an option in Calibre to only create metadata from the filename). There does not appear to be any user control over how the 3 internet sites are used. I took a quick look at the code, and I got the impression that isbndb.com was the primary metadata source, and that amazon was only queried for social networking metadata (I could be wrong -- my Python skills are not of the first water).

So now the questions:
1. Is there anything wrong with the description in the preceding paragraph?
2. When the "only create metadata from filename" is NOT checked, is the filename yet ever used to form title/author metadata in Calibre? If yes, when is the filename used?
3. How do you determine what the title/author metadata for PDF files is during an add books process? Do you have any idea why what Calibre gets seems to be different from what the kindle has?
4. If I make a change to title/author metadata on Calibre how are those changes propagated back to a kindle during a send device?
5. Will the "create metadata only from filename" option have any effect on the filename when a send to device is done?
6. Are there any circumstances upon which a metadata change on Calibre will cause a change in the filename on the kindle?
7. if #6 is yes, is there anyway to prevent a title/author metadata change on Calibre from changing the filename on the kindle? How?
8. You said you will display the collections of kindle books on Calibre. Where are you getting the collections info from? How are you handling books that are members of multiple collections?
9. When you do a send to device, you provide the ability to configure the files/folders in which the document will be placed. You use metadata variables to assist in this configuration. Would it be possible to create a collection metadata variable? Even though you cannot change collections from the kindle side, it would be useful to organize the book files on the kindle side in folders that mirror the collections -- especially if one has hundreds or thousands of books on your kindle along with many collections. I realize there is an issue with multiple-collection books, but being able to specify one to used for a folder name would still be useful.

I have found one subtle bug in 0.7.2. I run Windows XP SP3; when I click on the down arrow next to the preferences menu item, and then select Preferences from the pull down menu, the preferences dialog never appears. If I get impatient and try to force the dialog open with the ctrl+p key (the pulldown menu still being down), Calibre crashes. If I just don't pull down the preferences menu, but just hit the ctrl+p key directly, the dialog opens correctly. If you haven't already seen this, I'll open a ticket.

[kovidgoyal]

1. no

2. if the there is no internal metadata

3. Look at the info dict

4. yes, though metadata i not always successsfully written to PDF files

5. no

6. if the title or the author are changed

7. set a send to device template that doesn't include title or author

8. collections.json. what's the problem with books that are part of multiple collections

9. not really, the send to device template uses metadata available in the calibre library. Before you send a file to the kindle it is not in any collection, since it hasn't been sent to the kindle yet

[eboyhan]

Thanks for the quick response -- this will be very helpful. You didn't comment on the preference bug. Do you need a ticket?

3a. Suppose I have "create metadata from filename" "on", will you then bypass the info dict in PDF files?
3b. If I change the title & author fields in Calibre for a PDF file, will you rewrite that into the info dict?

8. no problem with multiple collections books per se -- are you displaying the names of collections that a book might be in as a comma separated list (similar to tags) or what?
9. One of us is not communicating well here -- it's probably me. Let me try again. I've got a bunch of books on my kindle -- all in collections. Let's suppose the calibredb is empty. I do an add books and bring these books over to calibre; presumably their collections membership comes along too, and is displayed (or perhaps optionally displayed in a custom column). Now I want to do a send to device of all these books back to the kindle -- only now the metadata in the send to device preferences governs where the books are sent to. I have tested this and if the metadata template refers to folder/filenames which don't exist, you create them. What I am asking for is to add "collection" to the list of metadata variables so that the books are placed into folder/filename where "folder" is the collection name and "filename" is the title of the book. Since you already have a book's collection information, it should be possible to do this. Or am I missing something?

10. There have been a number of reports today of users losing their collections. There is an Amazon provided backup facility, but it requires whispernet to be on. I'm wondering if it would make sense for calibre to capture a copy of the collections.json file for backup purposes. It could be restored from an option on the send to device menu. It would still require a reboot on the kindle side; don't know whether you could deliver a message to do that -- perhaps when the KDK goes live later this year. Even if the reboot requirement could only be in the documentation, a user controlled collections backup facility away from the vagaries of Amazon could prove useful. See below:

Today, over in the Amazon communities, there have been some reports of kindle users losing their collections. Amazon customer service is telling these users to deregister/reregister their kindles -- in about half the cases their collections are then recovered from the copy of the collections.json file kept on the Amazon archive.

Since many users run with whispernet off most of the time, this recovery doesn't always work. There is also an "Annotations backup" flag on the settings page of the kindle which governs automatic backup of annotations and collections on the Amazon archive. The annotations part of this is involved with some of the social networking stuff -- so there have been some privacy concerns raised. I only point this out because both calibre and Amazon will be accessing the collections.json file. You will be reading it; Amazon will be copying it to the archive and restoring it from the archive -- it can also be written to another kindle on the amazon account when the user asks for collections to be merged.

[kovidgoyal]

Already fixed.

yes

maybe, depends on the pdf in question.

yes

when you add books from the kindle, collections are not migrated to calibre (since they are not part of book metadata). Use the tags for this. Though really since tags (or collections) are many valued, there's no good way to automatically convert them into a folder hierarchy

Feel free to open an enhancement ticket, but I'm not overly motivated to do this. It's error prone and messy and I personally would never use the collections feature on a Kindle anyway.

[kcarfi]

Do you have instructions for using this or a GUI version?

Originally Posted by kwharrigan

I whipped up a quick Python script to aid in listing/editing the collections.json file. I find it convenient so maybe you will too.

I've had success placing this script on the Kindle volume itself and just running it straight from there. Only dependence is simplejson, though if you have 2.6 you can just use built-in json.

As indicated in previous posts, I can add any number of entries and following a reboot they show up in the list. I can also remove entries. Saves a little bit of time versus hand jamming the json.

The util does not yet associate documents to collections for you but that is the next step.

Would be nice if we could at least trigger a reboot from the USB..

[guiyoforward]

hi, anyone know if there's been progress on this - i.e. Calibre-PC side creation of collecitons? thanks!

[eboyhan]

As far as I know Kovid stated it was not currently possible -- maybe when (if) KDK gets released -- the KDK documentation seems to imply that an app could be created on the kindle side which can manage network (3G and USB) connections.