Kindle device problem with both library and collection

misling · 08-01-2011, 10:14 AM

I'm having a weird problem with Calibre and my Kindle.

My Kindle has ~2000 books on it. About half are mobi and the other half are azw with a few azw1 (topaz). All the books on the device are in a collection.
I have the latest version of Calibre (I download the new version every Saturday). I have the Kindle collections plugin installed (I have others as well, but probably unrelated to this problem)

1. I create a new empty calibre library 'Kindle Device Stuff' and switch to it.
2. I connect my Kindle via USB
3. I wait until the device button appears and all the books on the Kindle are listed under it. The library is still empty at this point.
4. In the 'device' view screen use the "add to library" function to add all books from the device to the empty library.
5. I switch over to the library to see the books that have been added (they are all added). Here is where the FIRST problem shows:
PROB 1: All the azw/azw1 books are marked as being on the device in Main memory with a checkmark in the column. All the mobi books are NOT marked as being on the device (even though that is where they were loaded into the library from).
6. I run the collections plugin option to import the collections from the Kindle. Here is where the SECOND problem shows.
PROB 2: All the mobi books are shown as being in a Collection. All the azw/azw1 books are NOT shown as being in a collection.

End result from the library view:
all Mobi books are 'not on the device' but 'in a collection'
all azw/azw1 books are 'on the device' but 'not in a collection'

Real situation: all books of all types are 'on the device' AND 'in a collection'

My purpose is to move books around between collections. This situation makes that impossible.

Anybody got any ideas?

misling · 08-05-2011, 09:40 AM

Anybody?
Not being able to tell what's on my Kindle is really messing things up.
I keep accidentally adding duplicate books to my Kindle from Calibre since half of the books on the device don't show as being on the device.
And of course the collections plugin is useless too when it doesn't report half of the collections on the device.

pchrist7 · 08-05-2011, 11:26 AM

Hi misling.
I'm no expert, but heard your cry for help.
First of all, I need more info.
Secondly I suggest we concentrate on getting Kindle and Calibre in sync since that's the basis for it all.

This is a new problem that suddenly occured, did it use to work (with other libraries, etc etc ) ?

Which Kindle do you have ?
Kindle FW version ?
OS version ?

Mobi books on Kindle: Are they DRM'ed ? (copy protected) ?
Your azw and azw1 DRM'ed ?

What's the size of you metadata.calibre on Kindle ?

Are ALL books on your Kindle transferred via Calibre ?!!

It might also be possible that your metadata.calibre on Kindle is corrupt, but hard to say. I had what might be a similar situation in february, and ended up biting the bullet (start from scratch) BUT YOU shouldn't do that yet !!.

I'll try to do some tests tonight/tomorrow to see if I can recreate your scenario, I hope.

Brds Per

pchrist7 · 08-06-2011, 06:05 AM

Hi again
Did some tests, as far as I dared to go

I use send to device plugboard to add series info to title when sending to my K3. When importing books into new, empty library, the books are imported, but title now includes series info if book is on a series.
If book is in a series (and included in title), then calibre does not show the book as being on device.
If i change the book title back to the original (remove series info), restart calibre, the book shows as being on device !

Does this help ?

Waiting for info from you, misling

Brgds Per

DoctorOhh · 08-06-2011, 06:33 AM

Quote:

Originally Posted by pchrist7

I use send to device plugboard to add series info to title when sending to my K3. When importing books into new, empty library, the books are imported, but title now includes series info if book is on a series.
If book is in a series (and included in title), then calibre does not show the book as being on device.

I don't have a Kindle but I always thought the books placed on your device using plugboards would show up in calibre's library as on device. At least they always show as being on device on all Sony devices.

pchrist7 · 08-06-2011, 07:26 AM

Quote:

Originally Posted by dwanthny

I don't have a Kindle but I always thought the books placed on your device using plugboards would show up in calibre's library as on device. At least they always show as being on device on all Sony devices.

Hi
They do show up as on device in calibre.
This issue here is when importing/copying books from kindle to a new calibre library.
Some books show correct status, some don't.
Need more info from OP !

Bgrds Per

chaley · 08-06-2011, 08:35 AM

Quote:

Originally Posted by pchrist7

I use send to device plugboard to add series info to title when sending to my K3. When importing books into new, empty library, the books are imported, but title now includes series info if book is on a series.
If book is in a series (and included in title), then calibre does not show the book as being on device.
If i change the book title back to the original (remove series info), restart calibre, the book shows as being on device !

There is probably an interaction between the setting of "Read metadata from files on device" in the Kindle plugin, "read metadata from file contents rather than file name" in preferences -> adding books, and importing from the Kindle. If the one in Add Books is set, then calibre will read the metadata from the book and get the plugboard-modified data. If the one in the device plugin is not set, then the device system will read metadata from the path, which is not affected by the plugboard. Result: titles do not match, so ondevice is blank.

Book matching between a book on the device that calibre did not put there and calibre's library is approximate. It depends on very-close-to-exact matches between authors and title, something that is often not the case, especially with purchased DRMed books.

If calibre put the book on the device, and if no one has messed with the .metadata.calibre file or changed any part of the name of the book file on the device, then book matching is done using a unique ID. I have no idea whether putting a book into a collection on the Kindle changes some part of the file path. If it does, then book matching by UUID will fail and calibre drops back to title/author matching.

pchrist7 · 08-07-2011, 12:04 AM

Quote:

Originally Posted by chaley

There is probably an interaction between the setting of "Read metadata from files on device" in the Kindle plugin, "read metadata from file contents rather than file name" in preferences -> adding books, and importing from the Kindle. If the one in Add Books is set, then calibre will read the metadata from the book and get the plugboard-modified data. If the one in the device plugin is not set, then the device system will read metadata from the path, which is not affected by the plugboard. Result: titles do not match, so ondevice is blank.

Book matching between a book on the device that calibre did not put there and calibre's library is approximate. It depends on very-close-to-exact matches between authors and title, something that is often not the case, especially with purchased DRMed books.

If calibre put the book on the device, and if no one has messed with the .metadata.calibre file or changed any part of the name of the book file on the device, then book matching is done using a unique ID. I have no idea whether putting a book into a collection on the Kindle changes some part of the file path. If it does, then book matching by UUID will fail and calibre drops back to title/author matching.

Thanks chaley !
I think you explained it all.
Adding books to collections changes nothing besides collections.json file AFAIK.

Brgds Per

meme · 08-07-2011, 07:28 AM

As pchrist7 says, adding books to collections only changes the collections.json file. It does not update the metadata in books or change their titles or pathnames, etc. It does not change what is stored in calibre.

Whether a book shows up as on-device in Calibre has nothing to do with the plugin. If some books are not showing up as on-device that are really on the device, then I suggest you look at resolving this first before trying to resolve any collections issue.

The plugin gets the information about what is on device directly from calibre these days, and doesn't use the metadata.calibre file.

I wonder if renaming metadata.calibre (on the Kindle) and then restarting calibre to force it to rebuild metadata.calibre will help. Of course with 2000 books this could take a very long time! And if the calibre options are set as chaley described it might not change anything anyway.

misling · 08-07-2011, 09:09 AM

Sorry had to travel off on business. I'll try to answer the questions as far as I understand them.

Quote:

This is a new problem that suddenly occured, did it use to work (with other libraries, etc etc ) ?

I'm not sure if this just started happening. My original library is about 8000 books and until I created the new empty library I may not have noticed if all the books didn't show as being on the device. I'd just added the collections plugin and my purpose in creating the new library was to have a place to work with just the books on the device and their collections, so of course I noticed it then

Before this point I was doing collections by hand.

Quote:

Which Kindle do you have ?

Kindle 3 3G

Quote:

Kindle FW version ?

3.1

Quote:

OS version ?

Windows Vista Ultimate

Quote:

Mobi books on Kindle: Are they DRM'ed ? (copy protected) ? Your azw and azw1 DRM'ed ?

Half and half. The azw/azw1 probably about 75% copy protected, the mobi stuff is not.

Quote:

What's the size of you metadata.calibre on Kindle ?

4,131 KB

Quote:

Are ALL books on your Kindle transferred via Calibre ?!!

no. The amazon azw/azw1 stuff downloads from Amazon. The mobi stuff is sideloaded via USB from the main Calibre library. I imported them all from the Kindle (USB) to the Calibre empty library in order to do collections on them.

misling · 08-07-2011, 09:14 AM

OK so if I'm following this:

Quote:

There is probably an interaction between the setting of "Read metadata from files on device" in the Kindle plugin, "read metadata from file contents rather than file name" in preferences -> adding books, and importing from the Kindle.

then what I need to do is check where the preferences are set to see if they are all doing the same thing for add books and the Kindle plugin (and maybe the collection plugin?). I'll try that next.

Thanks for the help!

I'm tempted to just redo the kindle from scratch, as somebody said above.

meme · 08-07-2011, 10:49 AM

Renaming the metadata.calibre file on the Kindle doesn't redo your kindle from scratch - it just removes the cache file that Calibre uses. By renaming it, you force calibre to read through all the books on your device to get their information so it can rebuild the cache file to save time each time your device is connected. But if the settings below aren't in sync I don't think it will help.

Check Preferences -> Adding Books - is Reading metadata from file checked or unchecked

Check Preferences -> Plugins, then expand Device Interface Plugins, then click Kindle 2/3 Device Interface, then click Customize Plugin. Is Read metadata checked or unchecked?

misling · 08-08-2011, 11:00 AM

I'll check those settings when I get home if I can find time to boot the laptop - I don't spend much time on the computer at home, and right now I'm at work.
Thanks for noting the path to the device settings under the plugin, don't know if I would have found that without instructions.

Oh, and I wasn't talking about removing the metadata or whatever. I was talking about doing a device restore to factory default, and reloading my books. Wouldn't fix this problem, I don't think, but I've been considering it because the books on the device are all f'd up due to duplicates being loaded on.

misling · 08-09-2011, 12:04 PM

OK. The 2 preferences are set the same and apparently that is not the problem.
I took screenshots but I don't see how to put them in here. It just asks for a url. My pictures aren't on the web, they are on my desktop. I don't have time to find a picture upload service and sign up for it, have to get back to work.

Here is the whole sequence.
Create an empty library.

Check Preferences -> Adding Books - is Reading metadata from file checked or unchecked? CHECKED

Check Preferences -> Plugins, then expand Device Interface Plugins, then click Kindle 2/3 Device Interface, then click Customize Plugin. Is Read metadata checked or unchecked? CHECKED

Plug in the Kindle. Wait for the Device screen to show up and fill.
Go to Device screen.
Highlight all.
Rightclick and 'Add to Library'.

In the library page, and also on the device page, Half the books, the azw/azw1 show "Main" under "on device" and half, the mobi, do not.

I hadn't noticed it the last time, but this time in the library page a few of the mobi were recognised - about 10 of them.

Also, on the library page the azw/azw1 are in a collection and the mobi are not. On the device page, the mobi are in a collection and the azw/azw1 are not. This is just weird.

All the books are on the Kindle device - that's where I built the library FROM. Both are really in a collection on the Kindle.

DoctorOhh · 08-09-2011, 10:12 PM

Quote:

Originally Posted by misling

I took screenshots but I don't see how to put them in here. It just asks for a url. My pictures aren't on the web, they are on my desktop. I don't have time to find a picture upload service and sign up for it, have to get back to work.

Upload service not required. Edit your post, go to the Advanced editor, click the paperclip to attach a file and upload your screenshots as attachments.

08-01-2011, 10:14 AM	#1
misling Zealot Posts: 106 Karma: 510000 Join Date: Nov 2010 Device: Kindle	Kindle device problem with both library and collection I'm having a weird problem with Calibre and my Kindle. My Kindle has ~2000 books on it. About half are mobi and the other half are azw with a few azw1 (topaz). All the books on the device are in a collection. I have the latest version of Calibre (I download the new version every Saturday). I have the Kindle collections plugin installed (I have others as well, but probably unrelated to this problem) 1. I create a new empty calibre library 'Kindle Device Stuff' and switch to it. 2. I connect my Kindle via USB 3. I wait until the device button appears and all the books on the Kindle are listed under it. The library is still empty at this point. 4. In the 'device' view screen use the "add to library" function to add all books from the device to the empty library. 5. I switch over to the library to see the books that have been added (they are all added). Here is where the FIRST problem shows: PROB 1: All the azw/azw1 books are marked as being on the device in Main memory with a checkmark in the column. All the mobi books are NOT marked as being on the device (even though that is where they were loaded into the library from). 6. I run the collections plugin option to import the collections from the Kindle. Here is where the SECOND problem shows. PROB 2: All the mobi books are shown as being in a Collection. All the azw/azw1 books are NOT shown as being in a collection. End result from the library view: all Mobi books are 'not on the device' but 'in a collection' all azw/azw1 books are 'on the device' but 'not in a collection' Real situation: all books of all types are 'on the device' AND 'in a collection' My purpose is to move books around between collections. This situation makes that impossible. Anybody got any ideas?

08-05-2011, 11:26 AM	#3
pchrist7 Addict Posts: 385 Karma: 6514 Join Date: Aug 2010 Location: Denmark Device: Kindle 3 3G+Wifi, Oasis	More info needed, please Hi misling. I'm no expert, but heard your cry for help. First of all, I need more info. Secondly I suggest we concentrate on getting Kindle and Calibre in sync since that's the basis for it all. This is a new problem that suddenly occured, did it use to work (with other libraries, etc etc ) ? Which Kindle do you have ? Kindle FW version ? OS version ? Mobi books on Kindle: Are they DRM'ed ? (copy protected) ? Your azw and azw1 DRM'ed ? What's the size of you metadata.calibre on Kindle ? Are ALL books on your Kindle transferred via Calibre ?!! It might also be possible that your metadata.calibre on Kindle is corrupt, but hard to say. I had what might be a similar situation in february, and ended up biting the bullet (start from scratch) BUT YOU shouldn't do that yet !!. I'll try to do some tests tonight/tomorrow to see if I can recreate your scenario, I hope. Brds Per Last edited by pchrist7; 08-05-2011 at 01:02 PM. Reason: Forgot ....

08-06-2011, 06:05 AM	#4
pchrist7 Addict Posts: 385 Karma: 6514 Join Date: Aug 2010 Location: Denmark Device: Kindle 3 3G+Wifi, Oasis	Using plugboards ? Hi again Did some tests, as far as I dared to go I use send to device plugboard to add series info to title when sending to my K3. When importing books into new, empty library, the books are imported, but title now includes series info if book is on a series. If book is in a series (and included in title), then calibre does not show the book as being on device. If i change the book title back to the original (remove series info), restart calibre, the book shows as being on device ! Does this help ? Waiting for info from you, misling Brgds Per

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Free (Kindle/Nook) Investor's Library (Collection)	arcadata	Deals and Resources (No Self-Promotion or Affiliate Links)	4	08-15-2011 03:21 AM
calibre removes the collection on my device that I made for my library books	LadyHeather	Calibre	5	01-06-2011 10:01 AM
Library Name as a Collection on Device	averyml	Calibre	3	11-30-2010 02:20 PM
Kindle DX US problem: Device Password Unavailable	steve.chan.bj	Amazon Kindle	2	07-26-2010 12:22 PM
Collection Problem	JeffElkins	Calibre	6	11-03-2008 06:33 AM

08-05-2011, 09:40 AM	#2
misling Zealot Posts: 106 Karma: 510000 Join Date: Nov 2010 Device: Kindle	Anybody? Not being able to tell what's on my Kindle is really messing things up. I keep accidentally adding duplicate books to my Kindle from Calibre since half of the books on the device don't show as being on the device. And of course the collections plugin is useless too when it doesn't report half of the collections on the device.

08-07-2011, 07:28 AM	#9
meme Sigil developer Posts: 1,274 Karma: 1101600 Join Date: Jan 2011 Location: UK Device: Kindle PW, K4 NT, K3, Kobo Touch	As pchrist7 says, adding books to collections only changes the collections.json file. It does not update the metadata in books or change their titles or pathnames, etc. It does not change what is stored in calibre. Whether a book shows up as on-device in Calibre has nothing to do with the plugin. If some books are not showing up as on-device that are really on the device, then I suggest you look at resolving this first before trying to resolve any collections issue. The plugin gets the information about what is on device directly from calibre these days, and doesn't use the metadata.calibre file. I wonder if renaming metadata.calibre (on the Kindle) and then restarting calibre to force it to rebuild metadata.calibre will help. Of course with 2000 books this could take a very long time! And if the calibre options are set as chaley described it might not change anything anyway.

08-07-2011, 10:49 AM	#12
meme Sigil developer Posts: 1,274 Karma: 1101600 Join Date: Jan 2011 Location: UK Device: Kindle PW, K4 NT, K3, Kobo Touch	Renaming the metadata.calibre file on the Kindle doesn't redo your kindle from scratch - it just removes the cache file that Calibre uses. By renaming it, you force calibre to read through all the books on your device to get their information so it can rebuild the cache file to save time each time your device is connected. But if the settings below aren't in sync I don't think it will help. Check Preferences -> Adding Books - is Reading metadata from file checked or unchecked Check Preferences -> Plugins, then expand Device Interface Plugins, then click Kindle 2/3 Device Interface, then click Customize Plugin. Is Read metadata checked or unchecked?

08-08-2011, 11:00 AM	#13
misling Zealot Posts: 106 Karma: 510000 Join Date: Nov 2010 Device: Kindle	I'll check those settings when I get home if I can find time to boot the laptop - I don't spend much time on the computer at home, and right now I'm at work. Thanks for noting the path to the device settings under the plugin, don't know if I would have found that without instructions. Oh, and I wasn't talking about removing the metadata or whatever. I was talking about doing a device restore to factory default, and reloading my books. Wouldn't fix this problem, I don't think, but I've been considering it because the books on the device are all f'd up due to duplicates being loaded on.

08-09-2011, 12:04 PM	#14
misling Zealot Posts: 106 Karma: 510000 Join Date: Nov 2010 Device: Kindle	OK. The 2 preferences are set the same and apparently that is not the problem. I took screenshots but I don't see how to put them in here. It just asks for a url. My pictures aren't on the web, they are on my desktop. I don't have time to find a picture upload service and sign up for it, have to get back to work. Here is the whole sequence. Create an empty library. Check Preferences -> Adding Books - is Reading metadata from file checked or unchecked? CHECKED Check Preferences -> Plugins, then expand Device Interface Plugins, then click Kindle 2/3 Device Interface, then click Customize Plugin. Is Read metadata checked or unchecked? CHECKED Plug in the Kindle. Wait for the Device screen to show up and fill. Go to Device screen. Highlight all. Rightclick and 'Add to Library'. In the library page, and also on the device page, Half the books, the azw/azw1 show "Main" under "on device" and half, the mobi, do not. I hadn't noticed it the last time, but this time in the library page a few of the mobi were recognised - about 10 of them. Also, on the library page the azw/azw1 are in a collection and the mobi are not. On the device page, the mobi are in a collection and the azw/azw1 are not. This is just weird. All the books are on the Kindle device - that's where I built the library FROM. Both are really in a collection on the Kindle.

Advert

Advert