[KOBO] books on the device that are not recognised by calibre

[anacreon]

I have a brand new Forma since Thursday, in exchange for the one I bought October last which had battery problems, and I've been suffering through loading some 4300 books (epubs converted to kepubs on the go).

There was the usual rigmarole of having to split the books in small parcels, and after a few parcels Calibre closes down, with no way to see what went wrong since when I reopen it, it is in the state it was in when closed down properly.

Now that I have finished transferring, some 100 books are on the device, but are not recognized by Calibre. I can read them, but the metadata are not updated in the details of the book, though they are in the proper collections. And I can't synchronise the reading positions (I have 3 devices active).

I have tried several times cancelling those books from the device and loading them again, with the same results.

I verified the database, which is ok, and my PC and devices are on the latest F/W for everything.

[davidfor]

Originally Posted by anacreon

I have a brand new Forma since Thursday, in exchange for the one I bought October last which had battery problems, and I've been suffering through loading some 4300 books (epubs converted to kepubs on the go).

There was the usual rigmarole of having to split the books in small parcels, and after a few parcels Calibre closes down, with no way to see what went wrong since when I reopen it, it is in the state it was in when closed down properly.

What? I've never had calibre "close down" when transferring any number of books to my Kobo devices. I don't usually do it in big batches, but, I have done more than a 1000 in the not to distant past. Were there any error messages from calibre?

If something does go wrong, if you just restart calibre, it won't see the books that were transferred to the device in the previous session. You need to disconnect the device and let it import the books. Then calibre will see the books when you next connect.

Now that I have finished transferring, some 100 books are on the device, but are not recognized by Calibre. I can read them, but the metadata are not updated in the details of the book, though they are in the proper collections. And I can't synchronise the reading positions (I have 3 devices active).

I have tried several times cancelling those books from the device and loading them again, with the same results.

Are you saying that none of the books on the device are matched to the calibre library? Or just some of them? If the latter, the most common reason I have found for the automatic matching to fails is if the book has multiple authors. You will probably need to do a manual match for these. There are other reasons, but this is the most common.

If you mean all the books, I think I'd need to see a debug log from calibre to try and guess what is going on.

[theducks]

Are the file name EXACTLY the same between devices?
Did ALL the books get ON TO the devices via Calibre?

In Calibre, switch to 'device view' (device is connected): For each book that is NOT ticked, right click Match to Library <you select the entry in the Library>

This procedure does not solve the 'sync' device issue. That relies on Kobo servers doing the match (I don't know what that needs to work . I only have a Kobo and a Kindle that do not even use the same formats )

[anacreon]

Originally Posted by davidfor

What? I've never had calibre "close down" when transferring any number of books to my Kobo devices. I don't usually do it in big batches, but, I have done more than a 1000 in the not to distant past. Were there any error messages from calibre?

If something does go wrong, if you just restart calibre, it won't see the books that were transferred to the device in the previous session. You need to disconnect the device and let it import the books. Then calibre will see the books when you next connect.

I've often had this problem when loading a new device, and I do disconnect the device when Calibre shuts down.

Originally Posted by davidfor

Are you saying that none of the books on the device are matched to the calibre library? Or just some of them? If the latter, the most common reason I have found for the automatic matching to fails is if the book has multiple authors. You will probably need to do a manual match for these. There are other reasons, but this is the most common.

If you mean all the books, I think I'd need to see a debug log from calibre to try and guess what is going on.

The problem concerns some 100 books on a total of more than 4000.
I did try the match to library like theducks says yesterday, but probably didn't do it right.

In any case, I couldn't try again today because I had this problem when connecting (after a long time trying)

Spoiler:

calibre, version 3.44.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 89, in run
File "site-packages\calibre\gui2\device.py", line 512, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1985, in books
File "site-packages\calibre\devices\usbms\driver.py", line 457, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 455, in write_prefix
File "site-packages\calibre\ebooks\metadata\book\json_codec.p y", line 135, in encode_to_file
File "json\__init__.py", line 251, in dumps
File "json\encoder.py", line 210, in encode
MemoryError.

When I tried it again, after more than 6 minutes Calibre disconnected.

[anacreon]

This morning I could connect the device, after 8 minutes (usually about 1 minute), and harmonized 111 books. And the next time I connected, it took under 1 minute.
Thanks to theducks who taught me about harmonization.

[davidfor]

Originally Posted by anacreon

I've often had this problem when loading a new device, and I do disconnect the device when Calibre shuts down.

Calibre just shuts down with no warnings? No messages from calibre or Windows?

The problem concerns some 100 books on a total of more than 4000.
I did try the match to library like theducks says yesterday, but probably didn't do it right.

What theducks missed was that after matching the books, you need to right click on the device button and choose the option "Update cached metadata on device". This updates the calibre.metadata on the device so that the matched book will be remembered. But, I think you have worked this out.

In any case, I couldn't try again today because I had this problem when connecting (after a long time trying)

Spoiler:

calibre, version 3.44.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 89, in run
File "site-packages\calibre\gui2\device.py", line 512, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1985, in books
File "site-packages\calibre\devices\usbms\driver.py", line 457, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 455, in write_prefix
File "site-packages\calibre\ebooks\metadata\book\json_codec.p y", line 135, in encode_to_file
File "json\__init__.py", line 251, in dumps
File "json\encoder.py", line 210, in encode
MemoryError.

This is while generating the metadata.calibre. I can't think of any reason to be running out of memory there, unless you have a lot of custom columns with a lot of data in them. And I suppose if you are running the 32bit version of calibre, that could contribute to it as well.

When I tried it again, after more than 6 minutes Calibre disconnected.

What do you mean here? Disconnects from the device? Or calibre closes like you say at the top?

And the first connection after putting a lot of books on a device will probably take longer. There is more work involved in matching the first time, plus the Kobo drivers will probably do more work updating metadata and collections on the device. The next connection should be a lot quicker.

[anacreon]

Originally Posted by davidfor

Calibre just shuts down with no warnings? No messages from calibre or Windows?

Yep!

What theducks missed was that after matching the books, you need to right click on the device button and choose the option "Update cached metadata on device". This updates the calibre.metadata on the device so that the matched book will be remembered. But, I think you have worked this out.

Yes.

This is while generating the metadata.calibre. I can't think of any reason to be running out of memory there, unless you have a lot of custom columns with a lot of data in them. And I suppose if you are running the 32bit version of calibre, that could contribute to it as well.

I have only 3 or 4 custom columns, and I use the portable version of calibre, I don't know whether that is the 32 or 64 bits. My PC is a surface book 2, which can do both.

What do you mean here? Disconnects from the device? Or calibre closes like you say at the top?

Close like I said above.

And the first connection after putting a lot of books on a device will probably take longer. There is more work involved in matching the first time, plus the Kobo drivers will probably do more work updating metadata and collections on the device. The next connection should be a lot quicker.

Which is why I disconnect every few hundreds of books - when Calibre doesn't close down first, which is rare.
And now the problem is solved, the connection is much quicker, as normal.

[BetterRed]

@anacreon - FTR: portable is 32 bit.

BR

[anacreon]

Originally Posted by BetterRed

@anacreon - FTR: portable is 32 bit.

BR

Thank you for the information.
However, the advantages of the portable version are such that I'll chose to remain with that version.