Device <=> Library handling

Loeffel · 05-29-2011, 11:18 AM

Hi,

how it is evaluated which book is on the device and in the library and which isn't?
I have the problem that some books are marked correctly and others aren't. I will make a detailed list to find out when books are marked correctly and when they aren't.
The strange thing is, when I delete them from the library and reinsert them, then nothing changes. They are still marked as not part of the library/device.

If I know the way the program evaluates this flag, I know why this happens.

chaley · 05-29-2011, 11:44 AM

The check consists of several parts. Which part depends on whether the book supports metadata (e.g., EPUB) and whether the metadata.calibre file is accurate.

First step: when you connect the device, the file metadata.calibre (on the device) is read and verified. If the books mentioned in metadata.calibre are still located where they were last time calibre saw them, and if the size of the book has not changed, then the metadata in the metadata.calibre file is used. If either check fails, then calibre scans the book for metadata. If it cannot read metadata, then it attempts to get metadata from the file path. The 'read metadata from path' option might play a role here, I don't remember.

Second step: for each book on the device, compare the metadata for the book with the books in calibre's library. There are several ways a book can match.

The first is the UUID. Calibre checks if the book's unique ID (UUID) is in the library. The UUID of the book on the device comes from the metadata.calibre file or the metadata in the file (I think this works only for EPUB). If the UUIDs match, the book as marked as ondevice.

If the UUID does not exist or matches no book, then calibre looks for matches using book metadata. Calibre first computes a shortened version of the book's title. All letters are converted to lower case and all spaces are removed. If these titles for a book in the library and a book on the device match, then a series of sub-checks are applied to see if the book really matches.

Subcheck 1: does the book have an the application id (calibre database ID) and if so, does it match? If so, the book is marked as ondevice. This is rare.
Subcheck 2: do the authors match? Similar processing is done to authors as to title before the check. If they match, then book is marked as ondevice.
Subcheck 3: does author_sort match the book's authors? If so, the book is marked as ondevice.

otherwise the book matches nothing and is not marked as ondevice.

If the metadata.calibre file is being written to the device and is not being corrupted, then it should be the case that any book that calibre puts on the device will be matches. As noted above, this match will fail if the book is moved or changes name on the device, or if the book changes size.

If books you send with calibre are not being marked as ondevice, then one of three things is happening. All of these are on the device:

the metadata.calibre file is being corrupted.
the book is changing names/folders
the book is changing size

I don't know how or why this could be happening. I don't have a kindle, and have no idea what magic it does.

pchrist7 · 05-30-2011, 02:55 PM

Quote:

Originally Posted by Loeffel

Hi,

how it is evaluated which book is on the device and in the library and which isn't?
I have the problem that some books are marked correctly and others aren't. I will make a detailed list to find out when books are marked correctly and when they aren't.
The strange thing is, when I delete them from the library and reinsert them, then nothing changes. They are still marked as not part of the library/device.

If I know the way the program evaluates this flag, I know why this happens.

Hi Loeffel
This used to work for you for fine, for along time, and then .. it's just out of sync ?
Had that, luckily only once !
Please tell us more about your OS, K3 firmware, Calibre version.
Most importantly : HOW is your work sequence ! ( start calibre, connect K3, bl.a. bla., send books to K3, HOW DO YOU DISconnect K3 ?? From Calibre; shutdown calibre disconnect form OS; or ? )
No help I know so far, but .. I might be able to help.
(I had just recieved a replacement K3, transferred/indexed/collectioned everything, and boom... out of sync. Tried to delete the metadata.calibre file as it is supposed to recreate it self. Didn't work. I then decided to start from scratch again ( DON'T do this yet !!! ) That worked. Have personally changed my "rutines" so once Calibre is done with whatever has to be done, I wait for a minute or 2, then use the OS to disconnect the USB connection, and maybe even a safely remove .. but that depends on your OS.)

@chaley: THANX !
I might be too old, but I can't remember reading so detailed a description as you just posted.
That could help us all to debug if anything happens.

Loeffel · 05-31-2011, 04:29 AM

@pchrist7
Yes it was just the way you said it. It worked fine and suddenly it was out of sync without any reason. I disconnect the Kindle different ways at last. In the beginning always after closing Calibre and disonnecting it from windows and even then the Kindle was connected for some time for loading.
I have already deleted the Calibre metadata file, but that doesn't change anything.

My Kindle is a European Version Kindle 3 3G with software 3.1 jailbreaked.
I use Win7 Ult 64bit, but lately I had the problem that I cannot disconnect the Kindle within Calibre or while Calibre is opened. It started suddenly, I can't say if this starts with a version change of Calibre or with installing a plugin.

All you write sounds like I need to do it by scratch, but at the same time I am not sure if this would help.
After reading the post from Chaley I am not sure if the problem isn't with some of the files. Some newer books have no correct metadata when Calibre reads it (within the Kindle everything is fine). On the other side some books where tagged correctly and now this tagging is gone. So now I'm a little confused where the error is to find.

On the other side I don't really sure how to start from scratch as there is no way to format the Kindle or reset it to delivery status. I would say a format would kill it, so this would be an absolute no go.

pchrist7 · 05-31-2011, 10:25 AM

@Loeffel
Hmm - sounds a lot like I experienced.
However I run Vista64, but ... not important I think.
In january I didn't really want to start over (again), but renaming metadata.calibre, reconnect etc. having Calibre re-create the file, it was just way too small, still missing imformation.
Ran fw 3.0.3/run fw 3.1, no mods/jailbreaks.
I suggest you do a backup of what you've got on your K3 first off all.
Then try to re-send a problem book to if anything changes.
If not, try to delete the book on K3, send it from calibre and check.
I tried to fix it this way, but didn't work out (can't remember the calibre version)

I'm NOT recommending you to do the following, unless you're sure:
I did a full backup of my K3 just in case.
Then I did a factory reset of K3
Send books from calibre ( not too many at once ) and wait for indexing to finish.
Collections I setup using meme's Kindle Collections plugin.
I did have challenges with the old annotation files, and never figured a smart way to fix.
Since then I've had no problems.

BUT ... I've changed my rutines since then.
Before plugin in K3, I make sure Calibre is not running.
Then I connect K3, make a backup copy of *.calibre files and maybe the collections.json.
Now I dare to start Calibre, wait, and then wait. Do what ever I want to do (send books, fetch annotations, manage collections etc.)
Once done in calibre, I wait at least 1 min. Shutdown Calibre, wait 1 min, from Vista Explorer, eject K3, wait a bit and then safely remove (not in win7)
This has worked for me fine since january,
Why did it happen ? Well, I might have been too fast disconnecting at some point (incredible I know

) or a windows update did something strange.
I don't the calibre windows eject function has changed in ages. Kovid is not too impressed with windows usb functions

Hope this makes sense, and might help you along.
I suspect you're like me - Calibre and device info MUST be in sync and correct !
Cheers, Per

Loeffel · 05-31-2011, 11:09 AM

I'm still thinking about one special thing. Perhaps there the error comes up, but I don't know and it is hard to evaluate. If I am right soon all would be messed up again! Nothing I would like.

Like you I want that the data I see is correct, so if I have such an indicator I want that it is right what is shown. What use do I have if it isn't that way?

I have two ways of transporting.
1. From Calibre to my Kindle (that should be no problem at all, but by now even this information isn't always right)
2. From my Kindle to Calibre (books I have purchased and for sure those I want to have them in my library too)

Perhaps a full reset would help, but what must be done else?
I had imported books into Calibre and corrected their data (title, author, publisher and so on). Can this mess up everything? What happens If I merge the data with my books already in the library?
First I thought merging was the problem, but my tests show otherwise. It doesn't matters. Sometimes the book aren't marked directly after transfer, sometimes after disconnecting and reconnecting.

Now you can say: "Hey by now he could have reseted the Kindle and done the work" but it is a lot of work. I have filled all the data of the book after importing them from my Kindle, sometimes even changed the picture and I just don't want to do it all again. ;-)

So I still try to figure out the failure and try to repair it and even if this isn't possible I want to make sure it doesn't happen again.
Calibre is a fine program and I really appreciate it and so I put a lot of work into building up my library. So I don't want to throw away a little bit of this work and redo it.

As far as I understand by now there is only the metadatafile from Calibre which is used to evaluate this tagging. Is there also something on the Calibre side?

chaley · 05-31-2011, 11:22 AM

Quote:

Originally Posted by Loeffel

As far as I understand by now there is only the metadatafile from Calibre which is used to evaluate this tagging. Is there also something on the Calibre side?

The calibre library database, metadata.db.

Loeffel · 05-31-2011, 11:58 AM

You mean on the Kindle? That is the file I meant. There is no other file involved, is there?

chaley · 05-31-2011, 12:16 PM

Quote:

Originally Posted by Loeffel

You mean on the Kindle? That is the file I meant. There is no other file involved, is there?

I am confused. You asked if there was anything on the "calibre side" involved with matching, while referring to the metadata.calibre file on the kindle. I thought that "calibre side" meant "not kindle".

If you are asking whether other files on the kindle can play a role in matching, the answer is yes. Calibre might read the book files during the 'first step' I described in post 2.

If you are asking what files on the computer running calibre can play a role, the answer is only the calibre DB, metadata.db.

Or are you asking something else?

Loeffel · 05-31-2011, 12:30 PM

Ah ok, that was what I wanted to know. But the metadata.db on my computer I am not allowed to delete if I am right.

theducks · 05-31-2011, 01:29 PM

Quote:

Originally Posted by Loeffel

Ah ok, that was what I wanted to know. But the metadata.db on my computer I am not allowed to delete if I am right.

metadata.db IS your 'library' and the link to the books filed in the folders found there.

Loeffel · 05-31-2011, 06:33 PM

Just as I thought. Ok I can't find any way to repair it. Nothing is working. So I have resetted my Kindle as Per did. Now it is downloading all my books, that will last some time as my WLAN access point just give up his life and I have only a very bad GPRS connection here at home. :-(

Afterwards I will try everything again and will report what happens. The Kindle already downloaded all collections from Amazon.

pchrist7 · 06-01-2011, 01:16 PM

@Loeffel
Sorry everything else didn't work out.
Really hope you'll get a sync'ed setup after this .. it does take quite some time
So far no new ideas from me, i'm stuck in a labour conflict against CSC, so this was my first day in "strike school" and Im done, over & out
Maybe in a couple of days,
Hope it works for you !

Loeffel · 06-04-2011, 03:15 AM

I've made know a full reset of the device, downloaded everything once again. Sorted the books on my Kindle, made a restart of my Kindle, connected it to Calibre and books are seen as part of the library. Not all, but the most.
Imported one of those not found and nothing changes. Calibre says the books isn't on the device/in the library. Just imported the book, nothing more!

chaley · 06-04-2011, 04:51 AM

Importing a book downloaded from Amazon to the kindle into calibre does not guarantee that calibre will see it as on the device. The book won't have a UUID because it didn't come from calibre, and there is no guarantee that the scan to build the metadata.calibre has the same information and calibre constructs during the import. Look for the book in the metadata.calibre file and visually compare the title and author to those in calibre's library.

If you send the book *from* calibre *to* the device, then it will have a UUID in the metadata.calibre file and should appear as ondevice.

05-29-2011, 11:18 AM	#1
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	Device <=> Library handling Hi, how it is evaluated which book is on the device and in the library and which isn't? I have the problem that some books are marked correctly and others aren't. I will make a detailed list to find out when books are marked correctly and when they aren't. The strange thing is, when I delete them from the library and reinsert them, then nothing changes. They are still marked as not part of the library/device. If I know the way the program evaluates this flag, I know why this happens.

05-29-2011, 11:44 AM	#2
chaley Grand Sorcerer Posts: 11,742 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	The check consists of several parts. Which part depends on whether the book supports metadata (e.g., EPUB) and whether the metadata.calibre file is accurate. First step: when you connect the device, the file metadata.calibre (on the device) is read and verified. If the books mentioned in metadata.calibre are still located where they were last time calibre saw them, and if the size of the book has not changed, then the metadata in the metadata.calibre file is used. If either check fails, then calibre scans the book for metadata. If it cannot read metadata, then it attempts to get metadata from the file path. The 'read metadata from path' option might play a role here, I don't remember. Second step: for each book on the device, compare the metadata for the book with the books in calibre's library. There are several ways a book can match. The first is the UUID. Calibre checks if the book's unique ID (UUID) is in the library. The UUID of the book on the device comes from the metadata.calibre file or the metadata in the file (I think this works only for EPUB). If the UUIDs match, the book as marked as ondevice. If the UUID does not exist or matches no book, then calibre looks for matches using book metadata. Calibre first computes a shortened version of the book's title. All letters are converted to lower case and all spaces are removed. If these titles for a book in the library and a book on the device match, then a series of sub-checks are applied to see if the book really matches. Subcheck 1: does the book have an the application id (calibre database ID) and if so, does it match? If so, the book is marked as ondevice. This is rare. Subcheck 2: do the authors match? Similar processing is done to authors as to title before the check. If they match, then book is marked as ondevice. Subcheck 3: does author_sort match the book's authors? If so, the book is marked as ondevice. otherwise the book matches nothing and is not marked as ondevice. If the metadata.calibre file is being written to the device and is not being corrupted, then it should be the case that any book that calibre puts on the device will be matches. As noted above, this match will fail if the book is moved or changes name on the device, or if the book changes size. If books you send with calibre are not being marked as ondevice, then one of three things is happening. All of these are on the device: the metadata.calibre file is being corrupted. the book is changing names/folders the book is changing size I don't know how or why this could be happening. I don't have a kindle, and have no idea what magic it does.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Library Name as a Collection on Device	averyml	Calibre	3	11-30-2010 02:20 PM
Best Device for getting ebook from library	fatlard	Which one should I buy?	10	12-06-2009 11:58 AM
Best device for public library use?	applecore	Which one should I buy?	10	03-01-2009 12:27 PM
Synchronising library with device	jbjb	Calibre	4	11-29-2008 05:19 AM
Large book library handling on prs-505	acidzebra	Sony Reader	5	07-23-2008 02:48 AM

05-31-2011, 04:29 AM	#4
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	@pchrist7 Yes it was just the way you said it. It worked fine and suddenly it was out of sync without any reason. I disconnect the Kindle different ways at last. In the beginning always after closing Calibre and disonnecting it from windows and even then the Kindle was connected for some time for loading. I have already deleted the Calibre metadata file, but that doesn't change anything. My Kindle is a European Version Kindle 3 3G with software 3.1 jailbreaked. I use Win7 Ult 64bit, but lately I had the problem that I cannot disconnect the Kindle within Calibre or while Calibre is opened. It started suddenly, I can't say if this starts with a version change of Calibre or with installing a plugin. All you write sounds like I need to do it by scratch, but at the same time I am not sure if this would help. After reading the post from Chaley I am not sure if the problem isn't with some of the files. Some newer books have no correct metadata when Calibre reads it (within the Kindle everything is fine). On the other side some books where tagged correctly and now this tagging is gone. So now I'm a little confused where the error is to find. On the other side I don't really sure how to start from scratch as there is no way to format the Kindle or reset it to delivery status. I would say a format would kill it, so this would be an absolute no go.

05-31-2011, 10:25 AM	#5
pchrist7 Addict Posts: 385 Karma: 6514 Join Date: Aug 2010 Location: Denmark Device: Kindle 3 3G+Wifi, Oasis	@Loeffel Hmm - sounds a lot like I experienced. However I run Vista64, but ... not important I think. In january I didn't really want to start over (again), but renaming metadata.calibre, reconnect etc. having Calibre re-create the file, it was just way too small, still missing imformation. Ran fw 3.0.3/run fw 3.1, no mods/jailbreaks. I suggest you do a backup of what you've got on your K3 first off all. Then try to re-send a problem book to if anything changes. If not, try to delete the book on K3, send it from calibre and check. I tried to fix it this way, but didn't work out (can't remember the calibre version) I'm NOT recommending you to do the following, unless you're sure: I did a full backup of my K3 just in case. Then I did a factory reset of K3 Send books from calibre ( not too many at once ) and wait for indexing to finish. Collections I setup using meme's Kindle Collections plugin. I did have challenges with the old annotation files, and never figured a smart way to fix. Since then I've had no problems. BUT ... I've changed my rutines since then. Before plugin in K3, I make sure Calibre is not running. Then I connect K3, make a backup copy of *.calibre files and maybe the collections.json. Now I dare to start Calibre, wait, and then wait. Do what ever I want to do (send books, fetch annotations, manage collections etc.) Once done in calibre, I wait at least 1 min. Shutdown Calibre, wait 1 min, from Vista Explorer, eject K3, wait a bit and then safely remove (not in win7) This has worked for me fine since january, Why did it happen ? Well, I might have been too fast disconnecting at some point (incredible I know ) or a windows update did something strange. I don't the calibre windows eject function has changed in ages. Kovid is not too impressed with windows usb functions Hope this makes sense, and might help you along. I suspect you're like me - Calibre and device info MUST be in sync and correct ! Cheers, Per

05-31-2011, 11:09 AM	#6
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	I'm still thinking about one special thing. Perhaps there the error comes up, but I don't know and it is hard to evaluate. If I am right soon all would be messed up again! Nothing I would like. Like you I want that the data I see is correct, so if I have such an indicator I want that it is right what is shown. What use do I have if it isn't that way? I have two ways of transporting. 1. From Calibre to my Kindle (that should be no problem at all, but by now even this information isn't always right) 2. From my Kindle to Calibre (books I have purchased and for sure those I want to have them in my library too) Perhaps a full reset would help, but what must be done else? I had imported books into Calibre and corrected their data (title, author, publisher and so on). Can this mess up everything? What happens If I merge the data with my books already in the library? First I thought merging was the problem, but my tests show otherwise. It doesn't matters. Sometimes the book aren't marked directly after transfer, sometimes after disconnecting and reconnecting. Now you can say: "Hey by now he could have reseted the Kindle and done the work" but it is a lot of work. I have filled all the data of the book after importing them from my Kindle, sometimes even changed the picture and I just don't want to do it all again. ;-) So I still try to figure out the failure and try to repair it and even if this isn't possible I want to make sure it doesn't happen again. Calibre is a fine program and I really appreciate it and so I put a lot of work into building up my library. So I don't want to throw away a little bit of this work and redo it. As far as I understand by now there is only the metadatafile from Calibre which is used to evaluate this tagging. Is there also something on the Calibre side?

05-31-2011, 11:58 AM	#8
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	You mean on the Kindle? That is the file I meant. There is no other file involved, is there?

05-31-2011, 12:30 PM	#10
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	Ah ok, that was what I wanted to know. But the metadata.db on my computer I am not allowed to delete if I am right.

05-31-2011, 06:33 PM	#12
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	Just as I thought. Ok I can't find any way to repair it. Nothing is working. So I have resetted my Kindle as Per did. Now it is downloading all my books, that will last some time as my WLAN access point just give up his life and I have only a very bad GPRS connection here at home. :-( Afterwards I will try everything again and will report what happens. The Kindle already downloaded all collections from Amazon.

06-01-2011, 01:16 PM	#13
pchrist7 Addict Posts: 385 Karma: 6514 Join Date: Aug 2010 Location: Denmark Device: Kindle 3 3G+Wifi, Oasis	@Loeffel Sorry everything else didn't work out. Really hope you'll get a sync'ed setup after this .. it does take quite some time So far no new ideas from me, i'm stuck in a labour conflict against CSC, so this was my first day in "strike school" and Im done, over & out Maybe in a couple of days, Hope it works for you !

06-04-2011, 03:15 AM	#14
Loeffel Connoisseur Posts: 58 Karma: 10 Join Date: Mar 2011 Device: Kindle 3 3G	I've made know a full reset of the device, downloaded everything once again. Sorted the books on my Kindle, made a restart of my Kindle, connected it to Calibre and books are seen as part of the library. Not all, but the most. Imported one of those not found and nothing changes. Calibre says the books isn't on the device/in the library. Just imported the book, nothing more!

06-04-2011, 04:51 AM	#15
chaley Grand Sorcerer Posts: 11,742 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Importing a book downloaded from Amazon to the kindle into calibre does not guarantee that calibre will see it as on the device. The book won't have a UUID because it didn't come from calibre, and there is no guarantee that the scan to build the metadata.calibre has the same information and calibre constructs during the import. Look for the book in the metadata.calibre file and visually compare the title and author to those in calibre's library. If you send the book from calibre to the device, then it will have a UUID in the metadata.calibre file and should appear as ondevice.

Advert

Advert