How does Calibre recognise books on device / Kobo Kepub-Epub (not) linked

[Mrs_Often]

I'm not sure if this should go in the Calibre or Kobo (plugin) forum... sorry.

I'm trying to figure out something weird. Here's the situation:

• On my Kobo Glo, I have both kepubs directly bought from Kobo, synced straight to my device from Kobo (I'll call them kepubs); and sideloaded epubs that I send to my device with Kobo using the KoboUtilities plugin to make kepubs out of them (I'll call them 'epubs' to make things less confusing).
• I have downloaded all kepubs to the device (from my Kobo account).
• In Calibre, I have free epubs, epubs bought from several different locations, and epub versions of all the books I bought/got for free from Kobo (either acsm downloaded from Kobo and epub moved into Calibre after downloading through ADE; or DRM-free epubs downloaded and imported into Calibre).

Here's the 'weird':
When connecting my device to Calibre, some epub versions of Kobo books are recognised as "on the device" while not being sideloaded (i.e. the kepub is on the device through my Kobo account), and some epub versions of Kobo books are not recognised as "on the device" even if the kepub is on the device and author and title (and other details) of epub and kepub are identical. (And yes I checked the device directory to make sure that any books being recognised weren't accidentally also sideloaded.)

In my ideal world Calibre would recognise all the kepubs on the device as the same books as the corresponding epubs in the Calibre library, so I know I have the book and I don't have to sideload it. So I'm trying to figure out what makes Calibre recognise a book... I've compared books that are and aren't recognised but I can't figure it out. Both DRM and DRM-free books can be/not be recognised.

Some examples (see screenshot as well):

• The (bought) Tawny Man Trilogy is not recognised, but it has very ugly titles that I corrected in the Calibre library. So I re-downloaded the Tawny Man Trilogy from Kobo and re-imported it into Calibre to see if the books weren't recognised because I'd corrected the really ugly titles... but even with the original ugly Kobo titles they're not recognised.
• I have put two identical epubs ('Emma'; free) in my Calibre library: one downloaded from the Kobo account that I have my device sync to (this one is not recognised; though I did add a cover image to the epub), and one downloaded from a different Kobo account (this one is recognised).
• 'Bleak House' (free) kepub is recognised as identical to the DRM-free epub in Calibre library, downloaded from the same Kobo account I sync my device to. I added a cover image to this epub too...
• 'The Help' (bought) is also recognised as identical to the epub with DRM in Calibre library.

What's going on? Is there a magic field that I have to make sure is set to a certain something for my ideal world to come into existence?

[kovidgoyal]

Side-loaded books are always matched. Other books are speculatively matched based on metadata.

IIRC if you run calibre in debug mode the In Library column in the device view shows some text indicating the type of match.

[Mrs_Often]

Thanks for your quick reply, Kovid! You recalled correctly... all matched books in the 'In Library' column in the Device view all state "UUID"... I don't really understand. Why would the two Emma books, both downloaded from Kobo (from different accounts) have different UUIDs?

Assuming I can't mess with UUIDs to match books up, what metadata is used for speculative matching?

[kovidgoyal]

title and author is used. If you want to manually match books simply right click on them in the device view and choose "match to library"

[davidfor]

Originally Posted by Mrs_Often

Thanks for your quick reply, Kovid! You recalled correctly... all matched books in the 'In Library' column in the Device view all state "UUID"... I don't really understand. Why would the two Emma books, both downloaded from Kobo (from different accounts) have different UUIDs?

Assuming I can't mess with UUIDs to match books up, what metadata is used for speculative matching?

The match with "UUID" means that calibre has either previously matched the books and stored this in the metadata.calibre file, or that there is an id in the book that matches with an id in calibre. Odds are a book downloaded from Kobo directly to the device won't have a UUID that matches a book downloaded by other methods and added to the calibre library.

Otherwise, as Kovid said, the matching is on the title and the authors. These need to be an exact match. The extra stuff put in the titles by all the publishers and stores cause a problem with this. Plus the way Kobo handles multiple authors causes problems with the matching especially for sideloaded books.

There have been a couple of recent threads on this where I made some suggestions on how to handle this. Beyond those suggestions, I haven't been able to come up with a way to improve the matching and have to manually match the books. After that, the books stay matched on each connection.

[Mrs_Often]

Originally Posted by kovidgoyal

title and author is used. If you want to manually match books simply right click on them in the device view and choose "match to library"

Oh, I'm so ignorant, I should have expected this functionality and looked for it. Thank you so much!

Originally Posted by davidfor

The match with "UUID" means that calibre has either previously matched the books and stored this in the metadata.calibre file, or that there is an id in the book that matches with an id in calibre. Odds are a book downloaded from Kobo directly to the device won't have a UUID that matches a book downloaded by other methods and added to the calibre library.

Aaahhh that makes sense.

Originally Posted by davidfor

Otherwise, as Kovid said, the matching is on the title and the authors. These need to be an exact match. The extra stuff put in the titles by all the publishers and stores cause a problem with this. Plus the way Kobo handles multiple authors causes problems with the matching especially for sideloaded books.

Good to know. I wonder what is different in the books in my library that have exact same author and title and no additional authors or silly info in the title (like 'Emma') but still don't match... extra spaces in the field or something?

Originally Posted by davidfor

There have been a couple of recent threads on this where I made some suggestions on how to handle this. Beyond those suggestions, I haven't been able to come up with a way to improve the matching and have to manually match the books. After that, the books stay matched on each connection.

Oops, I couldn't find those. Probably because I had no idea which search terms to use I'll hunt them down and try stuff out and otherwise use the wonderful manual match option.

Thank you so much for taking the time to explain this to me!

[Mrs_Often]

Originally Posted by davidfor

There have been a couple of recent threads on this where I made some suggestions on how to handle this. Beyond those suggestions, I haven't been able to come up with a way to improve the matching and have to manually match the books. After that, the books stay matched on each connection.

Originally Posted by Mrs_Often

Oops, I couldn't find those. Probably because I had no idea which search terms to use ☺️ I'll hunt them down and try stuff out and otherwise use the wonderful manual match option.

[s]Hmmm I still don't seem able to find them Could you please point me in the right direction, or maybe just suggest some useful keywords to search with?[/s]

Never mind, I think I just found the two threads you meant by chance. (Mainly https://www.mobileread.com/forums/sh...d.php?t=296482 and in the other you suggest to look there.)

Thank you!

[Mrs_Often]

Originally Posted by davidfor

[...] After that, the books stay matched on each connection.

After matching books manually to library, you have to "Update cached metadata on device". And then it matches on each connection.
-> Would it be possible to save this cached metadata within calibre files instead of/as well as onto the device? That way I don't have to re-match all the books for other devices that have the same books that aren't matching... and it would be easier to re-match after factory reset. (Is there a file I can backup from the device to make sure I don't have to re-match after factory reset?)

Thanks!

[davidfor]

Originally Posted by Mrs_Often

After matching books manually to library, you have to "Update cached metadata on device". And then it matches on each connection.
-> Would it be possible to save this cached metadata within calibre files instead of/as well as onto the device? That way I don't have to re-match all the books for other devices that have the same books that aren't matching... and it would be easier to re-match after factory reset. (Is there a file I can backup from the device to make sure I don't have to re-match after factory reset?)

In order, the automatic matching is done based on:

1. Details already in the metadata.conf file. This effectively means that the book was matched before and most books sent from calibre will fall do this. And the manual matching process does this.
2. An id in the library and in the metadata of the book.
3. The title and the author sort.

I don't know exactly where the id that is used comes from. I think calibre adds when a book is added to the device or changed somehow, but I'm not sure. And I have books that have been sent from calibre that didn't match this way, so I don't know what is happening.

The title and author are the main problem I see. The title is always OK if the book goes through calibre, but books downloaded directly to the device will have some strange titles and I have to do the manual match for those. The authors are usually a problem for multiple author books sent from calibre. The best solution I have for that is the metadata plugin I mentioned in the other thread. With that, the books usually match.

Overall, I haven't come up with a satisfactory solution. I tried a couple of different things when those earlier threads came up, but, they didn't help. If I think of something, I'll try and see if it helps.

[Mrs_Often]

Thank you for the additional information, very interesting! When I manually matched a load of books most of them indeed seemed to have something wrong or slightly different in the titles or authors (except for the "Emma" anomaly in the example above).

However what I'm after now (because of course, how can one ever be satisfied), is a more permanent match after manual matching is completed, like a local save of the matched metadata, or possibility to backup the matched IDs. Currently the metadata/matched information is saved on the device (right?), which means I have to match everything manually for every single device that has the same books on with the same matching problems (e.g. same Kobo account logged in on the device), and as well after factory reset...

[davidfor]

As I said, I don't have a good solution. If you have the same books on all devices in the same locations, then doing the match on one and copying the metadata.calibre file to the others will probably work. If you have one device that has all the books and the others have a subset, then copying from the main device to the others should work as well.

Backing up the metadata.calibre and restoring it after all the books have been restored after a factory reset should also work. I've been planning to add this into the backup done by the Kobo Utilities plugin for a while. I'll do this the next time I work on the plugin.

[Mrs_Often]

Originally Posted by davidfor

As I said, I don't have a good solution.

Ah sorry, I didn't realise you were aiming at the permanent matching request, I thought you were talking about the first time matching issues.

If you have the same books on all devices in the same locations, then doing the match on one and copying the metadata.calibre file to the others will probably work. If you have one device that has all the books and the others have a subset, then copying from the main device to the others should work as well.

Backing up the metadata.calibre and restoring it after all the books have been restored after a factory reset should also work. I've been planning to add this into the backup done by the Kobo Utilities plugin for a while. I'll do this the next time I work on the plugin.

Thank you and thank you! Yay!

[davidfor]

Originally Posted by Mrs_Often

Ah sorry, I didn't realise you were aiming at the permanent matching request, I thought you were talking about the first time matching issues.

No, I was talking about any solution. If I had a one-off solution, it would probably work as a permanent solution.