Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-14-2010, 05:06 PM   #16
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,803
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I've already implemented support for displaying the collections made on the Kindle when you connect it to calibre. There appears to be no way to actually create/edit/delete collections using the USB connection. Here are my notes on the subject:

A collections cache is stored at system/collections.json
The cache is read only, changes made to it are overwritten (it is regenerated)
on device disconnect

A log of collection creation/manipulation is available at
system/userannotationlog

collections.json refers to books via a SHA1 hash of the absolute path to the
book (prefix is /mnt/us on my Kindle). The SHA1 hash may or may not be prefixed
by some characters, use the last 40 characters.

Changing the metadata and resending the file doesn't seem to affect collections

Adding a book to a collection on the Kindle does not change the book file at all
(i.e. it is binary identical). Therefore collection information is not stored in
file metadata.

Last edited by kovidgoyal; 06-14-2010 at 05:09 PM.
kovidgoyal is offline   Reply With Quote
Old 06-15-2010, 01:09 AM   #17
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Kovid,
In reading the post of the tests done by johan1fs at Amazon, I got the impression that if you place a collections.json file at the root of your kindle file system, and then restart the kindle, the collections are rebuilt from the contents of the json file. Note that whispernet has to be off or else the json file will be overwritten by the copy on the Amazon archive before the rebuild is done.

This is clumsy I admit, and you are probably right that there is no convenient way to create a collection from the Calibre side. Displaying the collections works for me.

I wasn't sure where the asin numbers in the book titles came from -- it seems that you have found the asin's are only in the filesystem names (and only for Amazon books). I still wonder what happens to my collections if I change the book title metadata on Calibre to remove the asin's? Will this break collections on the kindle side -- as the collections expect the asin's to be present.
eboyhan is offline   Reply With Quote
Old 06-15-2010, 02:04 AM   #18
Steven_m64
Insane Programmer
Steven_m64 began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Mar 2010
Device: Kindle2
Interesting, I tested what eboyhan said as to editing the json file and placing it in the root dir before a restart.

that has no effect. but my kindle did rebuild the collections using a edited /system/collections.json after a restart.

this is proven by the fact that if the /system/collections.json file is missing on a kindle when its restarted and wireless is off all collections will be lost.

its annoying but at least it IS possible to edit collections externally.
Steven_m64 is offline   Reply With Quote
Old 06-15-2010, 08:50 AM   #19
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,803
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@eboyhan: I don't know about books from amazon as I dont have any, but I tried changing the metadata on a side loaded book and resending it to the kindle and the collection weren't affected. Judging by collections.json the collections are keyed to absolute paths, so changing file metadata should not affect anything as long as the path does not change. Of course if you change the title/author, and resend using calibre, the path will change.
kovidgoyal is offline   Reply With Quote
Old 06-15-2010, 12:53 PM   #20
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Kovid,
That is my point exactly. Amazon is placing the asin# (and some other gobbledygook) in the filename for their books -- presumably to guarantee that the collections feature can distinguish among the many different kindle versions of the same title that Amazon sometimes sells. When an Amazon book is brought over to Calibre all this extraneous stuff appears in the title. My question is: if I remove the extraneous stuff (it's ugly) on Calibre, and then transfer the book back to the kindle, will the filename put back on the kindle still be unchanged or not? Later today I'll run a few experiments with Amazon books, and let you know the results.
eboyhan is offline   Reply With Quote
Old 06-15-2010, 01:10 PM   #21
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,803
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
If you change the title, then no, it wont be the same.
kovidgoyal is offline   Reply With Quote
Old 06-15-2010, 06:06 PM   #22
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 4,805
Karma: 4461973
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW & PW2
@eboyhan: AFAICT, when referring to an amazon book with intact DRM, they're not keyed to the sha1 hash of the absolute path like DRM-free mobipocket books, but to their ASIN number.

And that's still true for books that've been handled by calibre at some point, so the asin number isn't in the filename anymore. So I'm assuming it's also in the metadata somewhere, and that calibre doesn't touch it, even if you change the title/author metadata.

Note that I didn't try this with a book downloaded directly on to the Kindle from Whispernet with the 2.5 fmw and the 'new' drm, so that may not be true anymore, but I didn't have any problem at all with my 'old' books, whether they were first downloaded to the kindle or a computer, and they're all handled by Calibre.

Last edited by NiLuJe; 06-15-2010 at 06:14 PM.
NiLuJe is offline   Reply With Quote
Old 06-17-2010, 12:36 AM   #23
kwharrigan
Junior Member
kwharrigan began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jun 2010
Device: Kindle
Utility for reading/writing collections.json

I whipped up a quick Python script to aid in listing/editing the collections.json file. I find it convenient so maybe you will too.

gist linky

I've had success placing this script on the Kindle volume itself and just running it straight from there. Only dependence is simplejson, though if you have 2.6 you can just use built-in json.

As indicated in previous posts, I can add any number of entries and following a reboot they show up in the list. I can also remove entries. Saves a little bit of time versus hand jamming the json.

The util does not yet associate documents to collections for you but that is the next step.

Would be nice if we could at least trigger a reboot from the USB..
kwharrigan is offline   Reply With Quote
Old 06-17-2010, 11:41 PM   #24
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Question Metadata Creation Questions

@kovid

I have some questions about how metadata is created during the add books process from a kindle. I also have some Q's about what you do with the metadata as viewed in Calibre when a send to device is done back to the kindle. I did some experiments the other day with PDF files that were all members of various collections. I found that what Calibre came up with for title and author was very different from what the kindle had. I changed some of the Calibre-side metadata, and then did a send to device, and those documents which were in collections on the kindle before I add them to Calibre were not in any collections after I sent them back to the KDX.

Before I ask my questions let me describe my understanding of how Calibre gets the metadata it displays. Metadata can come from 3 possible sources: the content of the eBook, the filename of the eBook, or from three places on the internet: ISBNDB.com, google books, and/or Amazon.com. The Fetch-ebook-metadata command (and its GUI equivalent) are used to augment the metadata gotten from inside the eBook, or from the filename (there is an option in Calibre to only create metadata from the filename). There does not appear to be any user control over how the 3 internet sites are used. I took a quick look at the code, and I got the impression that isbndb.com was the primary metadata source, and that amazon was only queried for social networking metadata (I could be wrong -- my Python skills are not of the first water).

So now the questions:
1. Is there anything wrong with the description in the preceding paragraph?
2. When the "only create metadata from filename" is NOT checked, is the filename yet ever used to form title/author metadata in Calibre? If yes, when is the filename used?
3. How do you determine what the title/author metadata for PDF files is during an add books process? Do you have any idea why what Calibre gets seems to be different from what the kindle has?
4. If I make a change to title/author metadata on Calibre how are those changes propagated back to a kindle during a send device?
5. Will the "create metadata only from filename" option have any effect on the filename when a send to device is done?
6. Are there any circumstances upon which a metadata change on Calibre will cause a change in the filename on the kindle?
7. if #6 is yes, is there anyway to prevent a title/author metadata change on Calibre from changing the filename on the kindle? How?
8. You said you will display the collections of kindle books on Calibre. Where are you getting the collections info from? How are you handling books that are members of multiple collections?
9. When you do a send to device, you provide the ability to configure the files/folders in which the document will be placed. You use metadata variables to assist in this configuration. Would it be possible to create a collection metadata variable? Even though you cannot change collections from the kindle side, it would be useful to organize the book files on the kindle side in folders that mirror the collections -- especially if one has hundreds or thousands of books on your kindle along with many collections. I realize there is an issue with multiple-collection books, but being able to specify one to used for a folder name would still be useful.

I have found one subtle bug in 0.7.2. I run Windows XP SP3; when I click on the down arrow next to the preferences menu item, and then select Preferences from the pull down menu, the preferences dialog never appears. If I get impatient and try to force the dialog open with the ctrl+p key (the pulldown menu still being down), Calibre crashes. If I just don't pull down the preferences menu, but just hit the ctrl+p key directly, the dialog opens correctly. If you haven't already seen this, I'll open a ticket.
eboyhan is offline   Reply With Quote
Old 06-18-2010, 01:24 AM   #25
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,803
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
1. no

2. if the there is no internal metadata

3. Look at the info dict

4. yes, though metadata i not always successsfully written to PDF files

5. no

6. if the title or the author are changed

7. set a send to device template that doesn't include title or author

8. collections.json. what's the problem with books that are part of multiple collections

9. not really, the send to device template uses metadata available in the calibre library. Before you send a file to the kindle it is not in any collection, since it hasn't been sent to the kindle yet
kovidgoyal is offline   Reply With Quote
Old 06-18-2010, 01:46 AM   #26
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Thanks for the quick response -- this will be very helpful. You didn't comment on the preference bug. Do you need a ticket?

3a. Suppose I have "create metadata from filename" "on", will you then bypass the info dict in PDF files?
3b. If I change the title & author fields in Calibre for a PDF file, will you rewrite that into the info dict?

8. no problem with multiple collections books per se -- are you displaying the names of collections that a book might be in as a comma separated list (similar to tags) or what?
9. One of us is not communicating well here -- it's probably me. Let me try again. I've got a bunch of books on my kindle -- all in collections. Let's suppose the calibredb is empty. I do an add books and bring these books over to calibre; presumably their collections membership comes along too, and is displayed (or perhaps optionally displayed in a custom column). Now I want to do a send to device of all these books back to the kindle -- only now the metadata in the send to device preferences governs where the books are sent to. I have tested this and if the metadata template refers to folder/filenames which don't exist, you create them. What I am asking for is to add "collection" to the list of metadata variables so that the books are placed into folder/filename where "folder" is the collection name and "filename" is the title of the book. Since you already have a book's collection information, it should be possible to do this. Or am I missing something?

10. There have been a number of reports today of users losing their collections. There is an Amazon provided backup facility, but it requires whispernet to be on. I'm wondering if it would make sense for calibre to capture a copy of the collections.json file for backup purposes. It could be restored from an option on the send to device menu. It would still require a reboot on the kindle side; don't know whether you could deliver a message to do that -- perhaps when the KDK goes live later this year. Even if the reboot requirement could only be in the documentation, a user controlled collections backup facility away from the vagaries of Amazon could prove useful. See below:

Today, over in the Amazon communities, there have been some reports of kindle users losing their collections. Amazon customer service is telling these users to deregister/reregister their kindles -- in about half the cases their collections are then recovered from the copy of the collections.json file kept on the Amazon archive.

Since many users run with whispernet off most of the time, this recovery doesn't always work. There is also an "Annotations backup" flag on the settings page of the kindle which governs automatic backup of annotations and collections on the Amazon archive. The annotations part of this is involved with some of the social networking stuff -- so there have been some privacy concerns raised. I only point this out because both calibre and Amazon will be accessing the collections.json file. You will be reading it; Amazon will be copying it to the archive and restoring it from the archive -- it can also be written to another kindle on the amazon account when the user asks for collections to be merged.

Last edited by eboyhan; 06-18-2010 at 03:00 AM. Reason: more info
eboyhan is offline   Reply With Quote
Old 06-18-2010, 11:12 AM   #27
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,803
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Already fixed.

yes

maybe, depends on the pdf in question.

yes

when you add books from the kindle, collections are not migrated to calibre (since they are not part of book metadata). Use the tags for this. Though really since tags (or collections) are many valued, there's no good way to automatically convert them into a folder hierarchy

Feel free to open an enhancement ticket, but I'm not overly motivated to do this. It's error prone and messy and I personally would never use the collections feature on a Kindle anyway.
kovidgoyal is offline   Reply With Quote
Old 06-18-2010, 01:35 PM   #28
kcarfi
Junior Member
kcarfi began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Dec 2009
Device: Kindle
kindle script

Do you have instructions for using this or a GUI version?

Quote:
Originally Posted by kwharrigan View Post
I whipped up a quick Python script to aid in listing/editing the collections.json file. I find it convenient so maybe you will too.

I've had success placing this script on the Kindle volume itself and just running it straight from there. Only dependence is simplejson, though if you have 2.6 you can just use built-in json.

As indicated in previous posts, I can add any number of entries and following a reboot they show up in the list. I can also remove entries. Saves a little bit of time versus hand jamming the json.

The util does not yet associate documents to collections for you but that is the next step.

Would be nice if we could at least trigger a reboot from the USB..
kcarfi is offline   Reply With Quote
Old 07-28-2010, 08:41 PM   #29
guiyoforward
Connoisseur
guiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the roughguiyoforward is a jewel in the rough
 
Posts: 97
Karma: 7104
Join Date: Jul 2010
Device: Kindle DX Graphite
hi, anyone know if there's been progress on this - i.e. Calibre-PC side creation of collecitons? thanks!
guiyoforward is offline   Reply With Quote
Old 07-29-2010, 01:44 AM   #30
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
As far as I know Kovid stated it was not currently possible -- maybe when (if) KDK gets released -- the KDK documentation seems to imply that an app could be created on the kindle side which can manage network (3G and USB) connections.
eboyhan is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Kindle, Calibre, & Fetch News sacoward Amazon Kindle 6 11-16-2010 09:50 PM
Periodicals generated by calibre not using kindle back issue archiving alessandro Calibre 6 08-09-2010 01:33 PM
Calibre & Kindle DX Collections deborakaye Calibre 4 08-05-2010 03:13 PM
Using calibre for Sony Pocket & Kindle grizedale Calibre 3 08-04-2010 02:00 PM
Newbie question about Kindle & Calibre mdumas Calibre 3 05-26-2010 04:30 PM


All times are GMT -4. The time now is 06:19 AM.


MobileRead.com is a privately owned, operated and funded community.