Issue with Calibre 0.7.1 & Kindle 2.5.2

[eboyhan]

@kovid:
I'm running 0.7.1 so it might be something there, but I don't think so.

Two weeks ago on 6.55 (and kindle 2.3.4)I cleared my calibre db to zero, attached my kindle and told calibre to add all the books in the kindle's documents folder (I download my books to kindle and then push them to Calibre). Then (about 2 weeks later) I upgraded Calibre to 0.7.1 and I got the 2.5.2 kindle update -- as I'm putting my books into collections (on the kindle) I notice I have 10 more books on my kindle than in Calibre -- so I attach my kindle and ask Calibre to add all the books from the kindle's documents folder (I no longer knew what those 10 new books were).

In the past Calibre would do this, flag the duplicates and ask me whether I wanted to add the dupes, I would say "no" and all would be fine. This time it asked about the dupes, and I dutifully answered "no", but this time virtually all the kindle books were added again. Upon looking at the duplicates in the calibre db, I could see that all the titles coming from the kindle (now on 2.5.2) had had the ASIN# added to their titles -- so calibre didn't see them as duplicates. This addition of ASIN# makes the titles less cosmetically pleasing, but I think you have to leave them be as I think they have something to do with collection synchronization with the Amazon archive on the kindle side.

Removing the ASIN # might break the collection behavior on the Amazon/kindle side.

I hope someone with more familiarity with 2.5 can explain what is going on in some detail.

I realize that people on the Calibre side of things may not have access to the 2.5.2 update yet -- so I pass this along as just something to be aware of.

I think I will hold off on aggressive adoption of 0.7.1 until you can look at the kindle 2.5.2 update in detail and figure out how Calibre will accommodate it.

[kovidgoyal]

Do you have the option set to read metadata only from file names?

[eboyhan]

"read metadata only from filename" was unchecked both times I did the add books.

Amazon is adding stuff to title metadata inside their ebooks to support collections. They use the added metadata (ASIN#) to link books to the collections they belong in so that when books move from kindle-to-archive-back-to-another-kindle it can put them back in the correct collections.

My understanding is that when collections are saved on the archive (and perhaps as implemented on the kindle as well), the ASIN#s of the members are what are preserved -- so that when a new book is presented to the add-to-collection mechanism, if the ASINs match, it is added otherwise not.

Stated more clearly (I hope): a collection is a list of ASIN#s. This, of course, begs the question as to how the non-amazon material is handled vis a vis collections membership.

[kovidgoyal]

I'll have to reserve comment until I get the 2.5 update and play with.

[eboyhan]

thought you would

[eboyhan]

After your question about options, I got to thinking; and I attached my kindle DX to USB to look at documents folder file names -- they too now contain the ASIN# in the name for Amazon-acquired books.

Also I noticed in the root of the kindle device a file called "metdata.calibre". What is this?

In the system folder there are two interesting files: "userannotlog" and (of more interest to you I'm sure) "collections.json"

[Steven_m64]

Originally Posted by eboyhan

In the system folder there are two interesting files: "userannotlog" and (of more interest to you I'm sure) "collections.json"

i have attempted editing the collections.json file and it never had any effect.

[kovidgoyal]

See https://www.mobileread.com/forums/showthread.php?t=83548

[Steven_m64]

The format is not a issue, its quite simple. the propagation of the changes you make to the file having any effect is the problem. any changes i have made to the file are ignored by the kindle in my experience so far.

[kovidgoyal]

You mean if you change the collections defined in the json file, the changes don't take effect?

[Steven_m64]

Originally Posted by kovidgoyal

You mean if you change the collections defined in the json file, the changes don't take effect?

Correct.

i fear that the file only may be used as a cache for whispernet sync.

[kovidgoyal]

Originally Posted by Steven_m64

Correct.

i fear that the file only may be used as a cache for whispernet sync.

So what happens if wireless is turned off?

[Steven_m64]

Originally Posted by kovidgoyal

So what happens if wireless is turned off?

having it on or off has no effect, changes are ignored.

changes to the file will be removed/lost as well when/if you click "sync and check for items" setting.

[kovidgoyal]

Hmm not looking good, if there's no way to manipulate the collections using the filesystem, then I'm afraid there's never going to be support for it in calibre

[eboyhan]

Kovid:

First, Amazon has released/posted the 2.5.2 update you can find it at the links here:
http://www.amazon.com/gp/help/custom...4143443&sr=1-1

Second, I got interested in how Amazon is implementing the collections feature, and so I did some research. There's not much available to a Google search, but there is some minimal information in the Kindle community forum threads at Amazon.com.

It appears that a kindle collection is a data structure consisting of a collection name plus a list of its members. A member is a SHA-1 hash of the title (at least) and perhaps additional metadata. Amazon has added its internally unique ASIN number to the titles of all its kindle books (both in the metadata in the book; and in the filename as seen ihn the kindle documents folder).

The collections.json file is a serialization of this data structure, and is used to convey a copy of a device's collections (and the names of their contents) to the Amazon archive.

The exact interaction among the file, the archive, the whispernet status, and the state of the kindle is a bit obscure. So I thought I would copy a couple of useful posts here.

C. Stoneham says:
Loading…
So, I'm poking around a bit in the "system" folder and I see a file named "collections.json". Looking closer, I see the following organizational scheme:

{"collection1name@en-US":{";items":["#B000R8YMMM^EBOK";,etc, etc,etc],"lastAccess":1276105657406},"collection2name@e n-US":{"items":["#B001D23Q00^EBOK",etc,etc,etc],"lastAccess";:1276106033495},

and so on.

Well, it just so turns out that, at least for azmn books, the "#B001D23Q00^EBOK" is a shortened form of the filename of the book with just the serial and ^ used in place of "-type_"

So, has anyone tested if we can edit this json file directly and speed up the process of making collections for large numbers of books?

Johan1FS says:
I did some testing ... deleted the .JSON file with wireless off ... my collections were still there. However, after I soft reset the Kindle, the collections were gone, so they probably stay in memory until such time as you restart the Kindle.

Copied the file back, no collections. Soft-reset again, and hey, the collections are back, but it took a while for them to update with the actual content.

So, I hope this information will prove to be of use, and that a way to accommodate the collections feature and the metadata changes it apparently requires can be found.

In the meantime, now that my kindle collections are stable, I'm going to clear my Calibre DB back to zero, and start over, and see what results. I will pass on any interesting discoveries that may appear.