Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-10-2010, 03:58 PM   #1
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Question Issue with Calibre 0.7.1 & Kindle 2.5.2

@kovid:
I'm running 0.7.1 so it might be something there, but I don't think so.

Two weeks ago on 6.55 (and kindle 2.3.4)I cleared my calibre db to zero, attached my kindle and told calibre to add all the books in the kindle's documents folder (I download my books to kindle and then push them to Calibre). Then (about 2 weeks later) I upgraded Calibre to 0.7.1 and I got the 2.5.2 kindle update -- as I'm putting my books into collections (on the kindle) I notice I have 10 more books on my kindle than in Calibre -- so I attach my kindle and ask Calibre to add all the books from the kindle's documents folder (I no longer knew what those 10 new books were).

In the past Calibre would do this, flag the duplicates and ask me whether I wanted to add the dupes, I would say "no" and all would be fine. This time it asked about the dupes, and I dutifully answered "no", but this time virtually all the kindle books were added again. Upon looking at the duplicates in the calibre db, I could see that all the titles coming from the kindle (now on 2.5.2) had had the ASIN# added to their titles -- so calibre didn't see them as duplicates. This addition of ASIN# makes the titles less cosmetically pleasing, but I think you have to leave them be as I think they have something to do with collection synchronization with the Amazon archive on the kindle side.

Removing the ASIN # might break the collection behavior on the Amazon/kindle side.

I hope someone with more familiarity with 2.5 can explain what is going on in some detail.

I realize that people on the Calibre side of things may not have access to the 2.5.2 update yet -- so I pass this along as just something to be aware of.

I think I will hold off on aggressive adoption of 0.7.1 until you can look at the kindle 2.5.2 update in detail and figure out how Calibre will accommodate it.

Last edited by eboyhan; 06-10-2010 at 05:38 PM. Reason: add more info
eboyhan is offline   Reply With Quote
Old 06-10-2010, 04:13 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Do you have the option set to read metadata only from file names?
kovidgoyal is offline   Reply With Quote
 
Advertisement
Old 06-10-2010, 05:36 PM   #3
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
"read metadata only from filename" was unchecked both times I did the add books.

Amazon is adding stuff to title metadata inside their ebooks to support collections. They use the added metadata (ASIN#) to link books to the collections they belong in so that when books move from kindle-to-archive-back-to-another-kindle it can put them back in the correct collections.

My understanding is that when collections are saved on the archive (and perhaps as implemented on the kindle as well), the ASIN#s of the members are what are preserved -- so that when a new book is presented to the add-to-collection mechanism, if the ASINs match, it is added otherwise not.

Stated more clearly (I hope): a collection is a list of ASIN#s. This, of course, begs the question as to how the non-amazon material is handled vis a vis collections membership.

Last edited by eboyhan; 06-10-2010 at 05:46 PM.
eboyhan is offline   Reply With Quote
Old 06-10-2010, 05:55 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I'll have to reserve comment until I get the 2.5 update and play with.
kovidgoyal is offline   Reply With Quote
Old 06-10-2010, 05:59 PM   #5
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
thought you would
eboyhan is offline   Reply With Quote
Old 06-10-2010, 07:17 PM   #6
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Lightbulb

After your question about options, I got to thinking; and I attached my kindle DX to USB to look at documents folder file names -- they too now contain the ASIN# in the name for Amazon-acquired books.

Also I noticed in the root of the kindle device a file called "metdata.calibre". What is this?

In the system folder there are two interesting files: "userannotlog" and (of more interest to you I'm sure) "collections.json"
eboyhan is offline   Reply With Quote
Old 06-10-2010, 07:39 PM   #7
Steven_m64
Insane Programmer
Steven_m64 began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Mar 2010
Device: Kindle2
Quote:
Originally Posted by eboyhan View Post
In the system folder there are two interesting files: "userannotlog" and (of more interest to you I'm sure) "collections.json"
i have attempted editing the collections.json file and it never had any effect.
Steven_m64 is offline   Reply With Quote
Old 06-10-2010, 10:05 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
See http://www.mobileread.com/forums/showthread.php?t=83548
kovidgoyal is offline   Reply With Quote
Old 06-10-2010, 10:48 PM   #9
Steven_m64
Insane Programmer
Steven_m64 began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Mar 2010
Device: Kindle2
The format is not a issue, its quite simple. the propagation of the changes you make to the file having any effect is the problem. any changes i have made to the file are ignored by the kindle in my experience so far.
Steven_m64 is offline   Reply With Quote
Old 06-10-2010, 10:51 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You mean if you change the collections defined in the json file, the changes don't take effect?
kovidgoyal is offline   Reply With Quote
Old 06-10-2010, 10:54 PM   #11
Steven_m64
Insane Programmer
Steven_m64 began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Mar 2010
Device: Kindle2
Quote:
Originally Posted by kovidgoyal View Post
You mean if you change the collections defined in the json file, the changes don't take effect?
Correct.

i fear that the file only may be used as a cache for whispernet sync.
Steven_m64 is offline   Reply With Quote
Old 06-10-2010, 11:25 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by Steven_m64 View Post
Correct.

i fear that the file only may be used as a cache for whispernet sync.
So what happens if wireless is turned off?
kovidgoyal is offline   Reply With Quote
Old 06-11-2010, 01:03 AM   #13
Steven_m64
Insane Programmer
Steven_m64 began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Mar 2010
Device: Kindle2
Quote:
Originally Posted by kovidgoyal View Post
So what happens if wireless is turned off?
having it on or off has no effect, changes are ignored.

changes to the file will be removed/lost as well when/if you click "sync and check for items" setting.
Steven_m64 is offline   Reply With Quote
Old 06-11-2010, 01:14 AM   #14
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,329
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm not looking good, if there's no way to manipulate the collections using the filesystem, then I'm afraid there's never going to be support for it in calibre
kovidgoyal is offline   Reply With Quote
Old 06-14-2010, 05:52 PM   #15
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Kindle Update 2.5.2 avaialable for download

Kovid:

First, Amazon has released/posted the 2.5.2 update you can find it at the links here:
http://www.amazon.com/gp/help/custom...4143443&sr=1-1

Second, I got interested in how Amazon is implementing the collections feature, and so I did some research. There's not much available to a Google search, but there is some minimal information in the Kindle community forum threads at Amazon.com.

It appears that a kindle collection is a data structure consisting of a collection name plus a list of its members. A member is a SHA-1 hash of the title (at least) and perhaps additional metadata. Amazon has added its internally unique ASIN number to the titles of all its kindle books (both in the metadata in the book; and in the filename as seen ihn the kindle documents folder).

The collections.json file is a serialization of this data structure, and is used to convey a copy of a device's collections (and the names of their contents) to the Amazon archive.

The exact interaction among the file, the archive, the whispernet status, and the state of the kindle is a bit obscure. So I thought I would copy a couple of useful posts here.

Quote:
C. Stoneham says:
Loading…
So, I'm poking around a bit in the "system" folder and I see a file named "collections.json". Looking closer, I see the following organizational scheme:

{"collection1name@en-US":{";items":["#B000R8YMMM^EBOK";,etc, etc,etc],"lastAccess":1276105657406},"collection2name@e n-US":{"items":["#B001D23Q00^EBOK",etc,etc,etc],"lastAccess";:1276106033495},

and so on.

Well, it just so turns out that, at least for azmn books, the "#B001D23Q00^EBOK" is a shortened form of the filename of the book with just the serial and ^ used in place of "-type_"

So, has anyone tested if we can edit this json file directly and speed up the process of making collections for large numbers of books?

Quote:
Johan1FS says:
I did some testing ... deleted the .JSON file with wireless off ... my collections were still there. However, after I soft reset the Kindle, the collections were gone, so they probably stay in memory until such time as you restart the Kindle.

Copied the file back, no collections. Soft-reset again, and hey, the collections are back, but it took a while for them to update with the actual content.
So, I hope this information will prove to be of use, and that a way to accommodate the collections feature and the metadata changes it apparently requires can be found.

In the meantime, now that my kindle collections are stable, I'm going to clear my Calibre DB back to zero, and start over, and see what results. I will pass on any interesting discoveries that may appear.
eboyhan is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Kindle, Calibre, & Fetch News sacoward Amazon Kindle 6 11-16-2010 10:50 PM
Periodicals generated by calibre not using kindle back issue archiving alessandro Calibre 6 08-09-2010 02:33 PM
Calibre & Kindle DX Collections deborakaye Calibre 4 08-05-2010 04:13 PM
Using calibre for Sony Pocket & Kindle grizedale Calibre 3 08-04-2010 03:00 PM
Newbie question about Kindle & Calibre mdumas Calibre 3 05-26-2010 05:30 PM


All times are GMT -4. The time now is 10:22 PM.


MobileRead.com is a privately owned, operated and funded community.