[Kobo] Transfer to Clara HD dumps

[Syana344]

Hi everybody,

I face an issue I never had with Calibre and Kobo.

When I try to transfer my library (yes... the full one) to my brand new Clara HD the transfer fails.

What I did
- First I have to say that I changed the SD card of the Clara to be able to have my full library with me. So I exchanged the existing 8GB with a brand new 128GB (which works already on my Glo HD - so I think that's not the issue).
- I had the configuration Kobo Glo HD FW 3.18 + Calibre 2.68 with the same SD card, and never had an issue.
- Now I bought a Clara HD and I'm working with Calibre 3.32 and a patched version of Kobo 4.10.11655. (I tried the unpached version too, but still have the same issue).
- I reset several times the Clara to factory status - does not change the issue
- I checked my library, all the ebooks seem fine.
- I changed the SD card to make sure the card is working well (and it's working fine on other devices).

What is happening
- I'm used to transfer my full library in batches. So, I launch e.g 5000 books, which works well (transfer in about 25mn + import to Kobo +/- 20min). Takes time, but it's a "one shot" action, as once the basic set up of the Kobo is done, I "just" add the new books, which does not take me more than 15mn.
- Now, what's happening, and absolutely new for me: my first batch works well, the second one is quite long, but still works... and whatever I do, the third one fails. I get an error message from Calibre:

calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique



Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 540, in _sync_booklists
  File "site-packages\calibre\devices\kobo\driver.py", line 947, in sync_booklists
  File "site-packages\calibre\devices\usbms\driver.py", line 448, in sync_booklists
  File "site-packages\calibre\devices\usbms\driver.py", line 446, in write_prefix
  File "site-packages\calibre\ebooks\metadata\book\json_codec.py", line 135, in encode_to_file
  File "json\__init__.py", line 251, in dumps
  File "json\encoder.py", line 210, in encode
MemoryError

After this error, my pc still recognizes the Kobo, and Kobo even does a proper import of the data, shows me the correct number of books... all seems ok. However, when plug it the next time to Calibe, Calibre starts up, starts to read the content, and after a long long time... closes without any error message or anything...

I'm not able to understand where the issue is coming from. I understand a "memory" error. But it can hardly be a device memory issue, as I have enough free space (particularly after the second batch).

Can anybody help me to understand what is the problem here?

Thanks a lot, and wish you a very nice day

[davidfor]

That error is happening when generating the calibre.metadata file. And it looks like the process is running out of memory. Are you running the 32bit version of calibre? If so, try the 64 bit version. That is the only thing I can think of with what you are seeing.

[Syana344]

Hi Davidfor

That's a good tip... I install Calibre for years, but it's true, I didn't check which version I installed this time... Maybe I downloaded the wrong one...

I'll try this evening and give you a feedback.

Thanks a lot and have a nice day.

[Syana344]

Hi David,

Yes in fact... stupid me... I changed the version yesterday evening.

Thanks a lot and have a nice day

[davidfor]

Not a stupid thing, you apparently just hit a limit. And for my curiosity, how many books have you put on the device? And how large is the calibre.metadata file?

[Syana344]

My Kobo GloHD runs with Calibre 2.68 and Firmware 3.18.
I changed my SD card with a 128 GB one an I have my full library on it...
Means around 53000 books.

And it runs really fine.

When I connect the device to Calibre, it takes 10mn to connect and make the update. When I export the new books on it, it does not take more than 10mn (not more than 200 books, I update regularly).
When I use the device, absolutely no dump or running time.

I have another Kobo with "only" 1000 books on it, and the "big" one behaves the same as the "small" one.

Regarding the file size, I don't know where to find it... If you can tell me where I have to look for, I'll check it, but the library size itself is around 32GB.

[davidfor]

Originally Posted by Syana344

My Kobo GloHD runs with Calibre 2.68 and Firmware 3.18.

You are sticking with older firmware? If so, I don't think 3.18 is a good choice. The second 3.19 is a better choice.

I changed my SD card with a 128 GB one an I have my full library on it...
Means around 53000 books.

And it runs really fine.

Wow. I would have thought that this number of books would have been noticeably slower.

When I connect the device to Calibre, it takes 10mn to connect and make the update.

The initial connect takes 10 minutes? I'm not surprised. It might speed things up if you turn off the "Read metadata from files on the device" in the driver configuration, it might be faster. I think but it's been ages since I looked at exactly how that option worked in the driver.

When I export the new books on it, it does not take more than 10mn (not more than 200 books, I update regularly).
When I use the device, absolutely no dump or running time.

I have another Kobo with "only" 1000 books on it, and the "big" one behaves the same as the "small" one.

Regarding the file size, I don't know where to find it... If you can tell me where I have to look for, I'll check it, but the library size itself is around 32GB.

The calibre.metadata file is in the root directory of the device when it is connected. On my Glo HD with about 4500 books, it is 45MB. So, 500MB would be reasonable yours. Calibre rewrites it as part of the connection, so that could take a few minutes.

[Syana344]

Hi David

In the meantime I checked the calibre.metadata file. The size is 395MB.

Ok, I have to precise that I use the ereader only to read... no games, no internet, no wifi. And of course, with this type of library, I take specially care of each single book - not corrupted, size, polish, classification, shelves etc - to avoid any corruption of the DB. On top, I have an full image to be able to restore the device if something breaks.

Thanks for the tip to turn off the "Read metadata from files on the device" feature. I'll try it out the next time I connect the device and have a look if it runs faster. (by the way, when I connected this evening, took exaclty 8mn45sec to synchronise with Calibre, which is already a good score for 53000 books on the device).

Regarding the FW version. In fact, I tried out the FW 3.19 but faced the issue that the device returned a "fatal error" around 40000 books. The only way out was a factory reset... and to start again. I have several GloHDs (not all mine, but I take care of) all set up in the same way, which excluded a HW issue on one single device. Therefore, I came to the conclusion, that the FW may be somehow limited.

So this was a kind of no-go for me to upgrade.

Now, as I like to explore the limits of my devices , I'm making tests with the Clara... and the Clara is not compatible with the 3.18... so I must use a 4.x version. which I find "sexy". Just curious to see if I'll be able to cross the 40000, or if it's limited also.

[davidfor]

Originally Posted by Syana344

In the meantime I checked the calibre.metadata file. The size is 395MB.

Smaller than I thought. But, if you don't use many custom columns, it does keep the size down.

And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB.

Ok, I have to precise that I use the ereader only to read... no games, no internet, no wifi. And of course, with this type of library, I take specially care of each single book - not corrupted, size, polish, classification, shelves etc - to avoid any corruption of the DB. On top, I have an full image to be able to restore the device if something breaks.

Thanks for the tip to turn off the "Read metadata from files on the device" feature. I'll try it out the next time I connect the device and have a look if it runs faster. (by the way, when I connected this evening, took exaclty 8mn45sec to synchronise with Calibre, which is already a good score for 53000 books on the device).

Yes, I'm surprised at it being so fast. And I'm not sure how much help turning the "Read metadata from files on the device" off will be. I just checked the code and it will only help if the book on the device hasn't previously been matched to the book in calibre. Or if the calibre.metadata file is missing.

Regarding the FW version. In fact, I tried out the FW 3.19 but faced the issue that the device returned a "fatal error" around 40000 books. The only way out was a factory reset... and to start again. I have several GloHDs (not all mine, but I take care of) all set up in the same way, which excluded a HW issue on one single device. Therefore, I came to the conclusion, that the FW may be somehow limited.

That is something I hadn't heard of before. I can't think of what the changes in that time frame would have been that caused this. It isn't deliberate on Kobo's part, more likely something changed that used more memory. I'm curious about when it displayed the error. What was happening on the device.

So this was a kind of no-go for me to upgrade.

Now, as I like to explore the limits of my devices , I'm making tests with the Clara... and the Clara is not compatible with the 3.18... so I must use a 4.x version. which I find "sexy". Just curious to see if I'll be able to cross the 40000, or if it's limited also.

One thing that has changed somewhere around version 4.3, is the speed of the collections list. Before this, if you had a lot of collections and any large collections, then it could take minutes to open the collection list. Afterwards, it was done to 10 seconds or so. There have been a lot of other performance improvements as well. Nothing as big as that, but generally it fells snappier. And a few of the other changes make me like the updates (series sorting, author list) but, I prefer the older tiled home screen.

[Syana344]

Originally Posted by davidfor

Smaller than I thought. But, if you don't use many custom columns, it does keep the size down..

I have 5 custom colums

Originally Posted by davidfor

And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB.

I'll check and tell you

Originally Posted by davidfor

Yes, I'm surprised at it being so fast. And I'm not sure how much help turning the "Read metadata from files on the device" off will be. I just checked the code and it will only help if the book on the device hasn't previously been matched to the book in calibre. Or if the calibre.metadata file is missing.

In fact, I tested it out yesterday evening, and it does not change.

Originally Posted by davidfor

That is something I hadn't heard of before. I can't think of what the changes in that time frame would have been that caused this. It isn't deliberate on Kobo's part, more likely something changed that used more memory. I'm curious about when it displayed the error. What was happening on the device.

Basically, Calibre behaved well, exported the epubs and the metadata. I could plug off the device normally from Calibre. After a while, the Kobo started the import, and after a moment, the screen turned to black screen only with a text "Fatal error happened, to recover your device you have to perform a factory reset", with two possibilities either approve or reject, and the two of them triggered a factory reset.

Originally Posted by davidfor

One thing that has changed somewhere around version 4.3, is the speed of the collections list. Before this, if you had a lot of collections and any large collections, then it could take minutes to open the collection list. Afterwards, it was done to 10 seconds or so. There have been a lot of other performance improvements as well. Nothing as big as that, but generally it fells snappier. And a few of the other changes make me like the updates (series sorting, author list) but, I prefer the older tiled home screen.

Yes, this I saw. Basically, I was interested in exactly the two features you are listening: series sorting and author list. It's why I was interested in the new FW. But I just made tests with the Clara, and it seems that the limit is even lower. Because it collapses at around 20000 epubs. I finished yesterday evening, with a 64bit version, and I faced exactly the same issue as with the 32bit version... Same error code and same detail message... So it seems, that I'm stucked with my Clara, as I cannot downgrade it to a 3.18 version.

[davidfor]

Originally Posted by Syana344

In fact, I tested it out yesterday evening, and it does not change.

I was afraid of that.

Basically, Calibre behaved well, exported the epubs and the metadata. I could plug off the device normally from Calibre. After a while, the Kobo started the import, and after a moment, the screen turned to black screen only with a text "Fatal error happened, to recover your device you have to perform a factory reset", with two possibilities either approve or reject, and the two of them triggered a factory reset.

After my post earlier, I was chatting with someone about this, from his comments, I suspected that was were the problem would be. And I'm pretty sure this won't be a problem in the recent firmware. And it is a lot faster. I had to reload all books on my Glo HD on the weekend with 4.10.something. I don't know the exact time as I wasn't watching closely, but the import was less than 30 minutes for 4000 books. With 3.18, I think that would have been a couple of hours. I don't

Yes, this I saw. Basically, I was interested in exactly the two features you are listening: series sorting and author list. It's why I was interested in the new FW. But I just made tests with the Clara, and it seems that the limit is even lower. Because it collapses at around 20000 epubs. I finished yesterday evening, with a 64bit version, and I faced exactly the same issue as with the 32bit version... Same error code and same detail message... So it seems, that I'm stucked with my Clara, as I cannot downgrade it to a 3.18 version.

If you are seeing the error in the first post, that shouldn't have anything to do with the firmware version. That is calibre generating a file on the device from the list of books it has detected. Is it exactly the same error? I can't think of any reason this would be happening at half the number of books.

At the point that error happens, the device should be connected, and the list of books should show in the device list. Sending books should work, but, you will probably see the error after the send finishes.

I would be interested in seeing exactly when this is happening. There is something about when this is happening that is bugging me. Could you run calibre in debug mode and send me the log? Do this by right-clicking on the preferences button and selecting "Restart in debug mode". Calibre will restart with a message about the mode. Connect the device, let the error happen and close calibre. The debug log will be displayed. Post that or put it somewhere and send me a PM with the link to download it. I'll look at it and see if I can think of anything to improve this.

[Syana344]

Hi David

First thanks a lot for your interest. I'm absolutely aware that this is not really a common issue :-)

Originally Posted by davidfor

If you are seeing the error in the first post, that shouldn't have anything to do with the firmware version. That is calibre generating a file on the device from the list of books it has detected. Is it exactly the same error? I can't think of any reason this would be happening at half the number of books.

Yes it was exactly the same error which happened not once, not twice, but at least five or six times. After this happens, I have to start again the full process, means factory reset, customizing (patched version) and then the transfer of books into batches.

Originally Posted by davidfor

At the point that error happens, the device should be connected, and the list of books should show in the device list. Sending books should work, but, you will probably see the error after the send finishes.

This was exactly what happened when I tried some months ago to upgrade from 3.18 to 3.19 my GloHD. And at the end it finished in a fatal error on Kobo side. But now, with the Clara, I'm even not able to reach this point as the dumps happen on Calibre side before the "fatal error" can happen.

Originally Posted by davidfor

I would be interested in seeing exactly when this is happening. There is something about when this is happening that is bugging me. Could you run calibre in debug mode and send me the log? Do this by right-clicking on the preferences button and selecting "Restart in debug mode". Calibre will restart with a message about the mode. Connect the device, let the error happen and close calibre. The debug log will be displayed. Post that or put it somewhere and send me a PM with the link to download it. I'll look at it and see if I can think of anything to improve this.

Of course I can! I just need some time to do it. As the last transfer crashed, I have to restore the full device. Will take a couple of days as I'm absent this weekend, but I think Monday or Tuesday I'll be done with it. I'll send you a PM when done.

Have a very nice day

[Syana344]

Hi David,

Just to keep you posted: I'm transferring the books, and try to let the error happen. I don't know if it comes from the debug mode as I never worked with it, but it takes really ages.

I transferred 21000 books in different batch sizes, but it took me around 11 hours to have it completed.

However, for the moment Calibre behaves well except from a time point of view.

I'll come back to you as soon as I see the error popping up.

Have a nice day

[Syana344]

Dear David

Last but not least...

Now it happened...
Device is the Clara HD, Version of calibre is 3.32 (64bits), FW is 4.10.11655 patched version.

So... what I did:
Batch 1 : 9000 books, took 4h48min., Reconnection time (for synch) 1m44s
Batch 2 : 1000 books, took 37min, Reconnection time (for synch) 1m35s
Batch 3 : 5000 books, took 2h58min, Reconnection time (for synch) 2m26s
Batch 4: 6000 books, took 3h24min, Reconnection time (for synch) 3m37s
Batch 5: 6000 books – error after 1h59min, not same error code

This time I have following:

calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique
Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 591, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub
  File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__
  File "site-packages\calibre\utils\localunzip.py", line 231, in extractall
  File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall
ValueError: Not a ZIP file

Big temp file created in the AppData/Local/temp folder => calibre_ta0k3r with 39.1 GB

Kobo did not start the import of the books => I tried to switch off, switch on again without result. I tried to repair the account without result, I tried to import just “one book” (sometimes it’s enough to relaunch the import process), but did not work. I tried to force a start up of the device, but this did not work. It's writen in the user manual that it should happen when you switch off your reader and release the power button and press it again for about 30 seconds, but nothing happened. The reader just restarted "normally" withtout starting the import of anything. At the end, I tried to re-import the same 6000 books. Import again stucked with the error code.

calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique

Not a ZIP file

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 591, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub
  File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__
  File "site-packages\calibre\utils\localunzip.py", line 231, in extractall
  File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall
ValueError: Not a ZIP file

The reader is now on the home page, fully charged and shows 20999 imported items (which makes me think that I hit a limit at 21000). I think I'm able to reconnect it (I did not try, but the last time it worked).

I have the debug log file done, and will send it to you per PM. I did not empty the temp folder - which I will definitely have to do, but not yet.

So... that's it from my side. I can restart a trial with the Glo HD for which I have a 4.8.11073 patched version, but I'm quite sure we'll end up with the same result (at least the last 2-3 times I tried, it behaved the same).

Have a very nice day

[Syana344]

Originally Posted by davidfor

And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB.

This I had some time to check: 2.84GB (with all 53000 books on the Glo HD)