Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 05-29-2011, 11:18 AM   #1
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
Device <=> Library handling

Hi,

how it is evaluated which book is on the device and in the library and which isn't?
I have the problem that some books are marked correctly and others aren't. I will make a detailed list to find out when books are marked correctly and when they aren't.
The strange thing is, when I delete them from the library and reinsert them, then nothing changes. They are still marked as not part of the library/device.

If I know the way the program evaluates this flag, I know why this happens.
Loeffel is offline   Reply With Quote
Old 05-29-2011, 11:44 AM   #2
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,618
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
The check consists of several parts. Which part depends on whether the book supports metadata (e.g., EPUB) and whether the metadata.calibre file is accurate.

First step: when you connect the device, the file metadata.calibre (on the device) is read and verified. If the books mentioned in metadata.calibre are still located where they were last time calibre saw them, and if the size of the book has not changed, then the metadata in the metadata.calibre file is used. If either check fails, then calibre scans the book for metadata. If it cannot read metadata, then it attempts to get metadata from the file path. The 'read metadata from path' option might play a role here, I don't remember.

Second step: for each book on the device, compare the metadata for the book with the books in calibre's library. There are several ways a book can match.

The first is the UUID. Calibre checks if the book's unique ID (UUID) is in the library. The UUID of the book on the device comes from the metadata.calibre file or the metadata in the file (I think this works only for EPUB). If the UUIDs match, the book as marked as ondevice.

If the UUID does not exist or matches no book, then calibre looks for matches using book metadata. Calibre first computes a shortened version of the book's title. All letters are converted to lower case and all spaces are removed. If these titles for a book in the library and a book on the device match, then a series of sub-checks are applied to see if the book really matches.
  • Subcheck 1: does the book have an the application id (calibre database ID) and if so, does it match? If so, the book is marked as ondevice. This is rare.
  • Subcheck 2: do the authors match? Similar processing is done to authors as to title before the check. If they match, then book is marked as ondevice.
  • Subcheck 3: does author_sort match the book's authors? If so, the book is marked as ondevice.
otherwise the book matches nothing and is not marked as ondevice.

If the metadata.calibre file is being written to the device and is not being corrupted, then it should be the case that any book that calibre puts on the device will be matches. As noted above, this match will fail if the book is moved or changes name on the device, or if the book changes size.

If books you send with calibre are not being marked as ondevice, then one of three things is happening. All of these are on the device:
  1. the metadata.calibre file is being corrupted.
  2. the book is changing names/folders
  3. the book is changing size
I don't know how or why this could be happening. I don't have a kindle, and have no idea what magic it does.
chaley is offline   Reply With Quote
 
Enthusiast
Old 05-30-2011, 02:55 PM   #3
pchrist7
Addict
pchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animals
 
pchrist7's Avatar
 
Posts: 385
Karma: 6514
Join Date: Aug 2010
Location: Denmark
Device: Kindle 3 3G+Wifi
Stupid question

Quote:
Originally Posted by Loeffel View Post
Hi,

how it is evaluated which book is on the device and in the library and which isn't?
I have the problem that some books are marked correctly and others aren't. I will make a detailed list to find out when books are marked correctly and when they aren't.
The strange thing is, when I delete them from the library and reinsert them, then nothing changes. They are still marked as not part of the library/device.

If I know the way the program evaluates this flag, I know why this happens.
Hi Loeffel
This used to work for you for fine, for along time, and then .. it's just out of sync ?
Had that, luckily only once !
Please tell us more about your OS, K3 firmware, Calibre version.
Most importantly : HOW is your work sequence ! ( start calibre, connect K3, bl.a. bla., send books to K3, HOW DO YOU DISconnect K3 ?? From Calibre; shutdown calibre disconnect form OS; or ? )
No help I know so far, but .. I might be able to help.
(I had just recieved a replacement K3, transferred/indexed/collectioned everything, and boom... out of sync. Tried to delete the metadata.calibre file as it is supposed to recreate it self. Didn't work. I then decided to start from scratch again ( DON'T do this yet !!! ) That worked. Have personally changed my "rutines" so once Calibre is done with whatever has to be done, I wait for a minute or 2, then use the OS to disconnect the USB connection, and maybe even a safely remove .. but that depends on your OS.)

@chaley: THANX !
I might be too old, but I can't remember reading so detailed a description as you just posted.
That could help us all to debug if anything happens.
pchrist7 is offline   Reply With Quote
Old 05-31-2011, 04:29 AM   #4
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
@pchrist7
Yes it was just the way you said it. It worked fine and suddenly it was out of sync without any reason. I disconnect the Kindle different ways at last. In the beginning always after closing Calibre and disonnecting it from windows and even then the Kindle was connected for some time for loading.
I have already deleted the Calibre metadata file, but that doesn't change anything.

My Kindle is a European Version Kindle 3 3G with software 3.1 jailbreaked.
I use Win7 Ult 64bit, but lately I had the problem that I cannot disconnect the Kindle within Calibre or while Calibre is opened. It started suddenly, I can't say if this starts with a version change of Calibre or with installing a plugin.

All you write sounds like I need to do it by scratch, but at the same time I am not sure if this would help.
After reading the post from Chaley I am not sure if the problem isn't with some of the files. Some newer books have no correct metadata when Calibre reads it (within the Kindle everything is fine). On the other side some books where tagged correctly and now this tagging is gone. So now I'm a little confused where the error is to find.

On the other side I don't really sure how to start from scratch as there is no way to format the Kindle or reset it to delivery status. I would say a format would kill it, so this would be an absolute no go.
Loeffel is offline   Reply With Quote
Old 05-31-2011, 10:25 AM   #5
pchrist7
Addict
pchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animals
 
pchrist7's Avatar
 
Posts: 385
Karma: 6514
Join Date: Aug 2010
Location: Denmark
Device: Kindle 3 3G+Wifi
@Loeffel
Hmm - sounds a lot like I experienced.
However I run Vista64, but ... not important I think.
In january I didn't really want to start over (again), but renaming metadata.calibre, reconnect etc. having Calibre re-create the file, it was just way too small, still missing imformation.
Ran fw 3.0.3/run fw 3.1, no mods/jailbreaks.
I suggest you do a backup of what you've got on your K3 first off all.
Then try to re-send a problem book to if anything changes.
If not, try to delete the book on K3, send it from calibre and check.
I tried to fix it this way, but didn't work out (can't remember the calibre version)

I'm NOT recommending you to do the following, unless you're sure:
I did a full backup of my K3 just in case.
Then I did a factory reset of K3
Send books from calibre ( not too many at once ) and wait for indexing to finish.
Collections I setup using meme's Kindle Collections plugin.
I did have challenges with the old annotation files, and never figured a smart way to fix.
Since then I've had no problems.

BUT ... I've changed my rutines since then.
Before plugin in K3, I make sure Calibre is not running.
Then I connect K3, make a backup copy of *.calibre files and maybe the collections.json.
Now I dare to start Calibre, wait, and then wait. Do what ever I want to do (send books, fetch annotations, manage collections etc.)
Once done in calibre, I wait at least 1 min. Shutdown Calibre, wait 1 min, from Vista Explorer, eject K3, wait a bit and then safely remove (not in win7)
This has worked for me fine since january,
Why did it happen ? Well, I might have been too fast disconnecting at some point (incredible I know ) or a windows update did something strange.
I don't the calibre windows eject function has changed in ages. Kovid is not too impressed with windows usb functions

Hope this makes sense, and might help you along.
I suspect you're like me - Calibre and device info MUST be in sync and correct !
Cheers, Per
pchrist7 is offline   Reply With Quote
Old 05-31-2011, 11:09 AM   #6
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
I'm still thinking about one special thing. Perhaps there the error comes up, but I don't know and it is hard to evaluate. If I am right soon all would be messed up again! Nothing I would like.

Like you I want that the data I see is correct, so if I have such an indicator I want that it is right what is shown. What use do I have if it isn't that way?

I have two ways of transporting.
1. From Calibre to my Kindle (that should be no problem at all, but by now even this information isn't always right)
2. From my Kindle to Calibre (books I have purchased and for sure those I want to have them in my library too)

Perhaps a full reset would help, but what must be done else?
I had imported books into Calibre and corrected their data (title, author, publisher and so on). Can this mess up everything? What happens If I merge the data with my books already in the library?
First I thought merging was the problem, but my tests show otherwise. It doesn't matters. Sometimes the book aren't marked directly after transfer, sometimes after disconnecting and reconnecting.

Now you can say: "Hey by now he could have reseted the Kindle and done the work" but it is a lot of work. I have filled all the data of the book after importing them from my Kindle, sometimes even changed the picture and I just don't want to do it all again. ;-)

So I still try to figure out the failure and try to repair it and even if this isn't possible I want to make sure it doesn't happen again.
Calibre is a fine program and I really appreciate it and so I put a lot of work into building up my library. So I don't want to throw away a little bit of this work and redo it.

As far as I understand by now there is only the metadatafile from Calibre which is used to evaluate this tagging. Is there also something on the Calibre side?
Loeffel is offline   Reply With Quote
Old 05-31-2011, 11:22 AM   #7
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,618
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by Loeffel View Post
As far as I understand by now there is only the metadatafile from Calibre which is used to evaluate this tagging. Is there also something on the Calibre side?
The calibre library database, metadata.db.
chaley is offline   Reply With Quote
Old 05-31-2011, 11:58 AM   #8
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
You mean on the Kindle? That is the file I meant. There is no other file involved, is there?
Loeffel is offline   Reply With Quote
Old 05-31-2011, 12:16 PM   #9
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,618
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by Loeffel View Post
You mean on the Kindle? That is the file I meant. There is no other file involved, is there?
I am confused. You asked if there was anything on the "calibre side" involved with matching, while referring to the metadata.calibre file on the kindle. I thought that "calibre side" meant "not kindle".

If you are asking whether other files on the kindle can play a role in matching, the answer is yes. Calibre might read the book files during the 'first step' I described in post 2.

If you are asking what files on the computer running calibre can play a role, the answer is only the calibre DB, metadata.db.

Or are you asking something else?
chaley is offline   Reply With Quote
Old 05-31-2011, 12:30 PM   #10
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
Ah ok, that was what I wanted to know. But the metadata.db on my computer I am not allowed to delete if I am right.
Loeffel is offline   Reply With Quote
Old 05-31-2011, 01:29 PM   #11
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,839
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Loeffel View Post
Ah ok, that was what I wanted to know. But the metadata.db on my computer I am not allowed to delete if I am right.
metadata.db IS your 'library' and the link to the books filed in the folders found there.
theducks is offline   Reply With Quote
Old 05-31-2011, 06:33 PM   #12
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
Just as I thought. Ok I can't find any way to repair it. Nothing is working. So I have resetted my Kindle as Per did. Now it is downloading all my books, that will last some time as my WLAN access point just give up his life and I have only a very bad GPRS connection here at home. :-(

Afterwards I will try everything again and will report what happens. The Kindle already downloaded all collections from Amazon.
Loeffel is offline   Reply With Quote
Old 06-01-2011, 01:16 PM   #13
pchrist7
Addict
pchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animals
 
pchrist7's Avatar
 
Posts: 385
Karma: 6514
Join Date: Aug 2010
Location: Denmark
Device: Kindle 3 3G+Wifi
@Loeffel
Sorry everything else didn't work out.
Really hope you'll get a sync'ed setup after this .. it does take quite some time
So far no new ideas from me, i'm stuck in a labour conflict against CSC, so this was my first day in "strike school" and Im done, over & out
Maybe in a couple of days,
Hope it works for you !
pchrist7 is offline   Reply With Quote
Old 06-04-2011, 03:15 AM   #14
Loeffel
Connoisseur
Loeffel began at the beginning.
 
Loeffel's Avatar
 
Posts: 58
Karma: 10
Join Date: Mar 2011
Device: Kindle 3 3G
I've made know a full reset of the device, downloaded everything once again. Sorted the books on my Kindle, made a restart of my Kindle, connected it to Calibre and books are seen as part of the library. Not all, but the most.
Imported one of those not found and nothing changes. Calibre says the books isn't on the device/in the library. Just imported the book, nothing more!
Loeffel is offline   Reply With Quote
Old 06-04-2011, 04:51 AM   #15
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,618
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
Importing a book downloaded from Amazon to the kindle into calibre does not guarantee that calibre will see it as on the device. The book won't have a UUID because it didn't come from calibre, and there is no guarantee that the scan to build the metadata.calibre has the same information and calibre constructs during the import. Look for the book in the metadata.calibre file and visually compare the title and author to those in calibre's library.

If you send the book *from* calibre *to* the device, then it will have a UUID in the metadata.calibre file and should appear as ondevice.
chaley is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Library Name as a Collection on Device averyml Calibre 3 11-30-2010 02:20 PM
Best Device for getting ebook from library fatlard Which one should I buy? 10 12-06-2009 11:58 AM
Best device for public library use? applecore Which one should I buy? 10 03-01-2009 12:27 PM
Synchronising library with device jbjb Calibre 4 11-29-2008 05:19 AM
Large book library handling on prs-505 acidzebra Sony Reader 5 07-23-2008 02:48 AM


All times are GMT -4. The time now is 10:14 PM.


MobileRead.com is a privately owned, operated and funded community.