10-09-2018, 02:35 AM | #1 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
[Kobo] Transfer to Clara HD dumps
Hi everybody,
I face an issue I never had with Calibre and Kobo. When I try to transfer my library (yes... the full one) to my brand new Clara HD the transfer fails. What I did - First I have to say that I changed the SD card of the Clara to be able to have my full library with me. So I exchanged the existing 8GB with a brand new 128GB (which works already on my Glo HD - so I think that's not the issue). - I had the configuration Kobo Glo HD FW 3.18 + Calibre 2.68 with the same SD card, and never had an issue. - Now I bought a Clara HD and I'm working with Calibre 3.32 and a patched version of Kobo 4.10.11655. (I tried the unpached version too, but still have the same issue). - I reset several times the Clara to factory status - does not change the issue - I checked my library, all the ebooks seem fine. - I changed the SD card to make sure the card is working well (and it's working fine on other devices). What is happening - I'm used to transfer my full library in batches. So, I launch e.g 5000 books, which works well (transfer in about 25mn + import to Kobo +/- 20min). Takes time, but it's a "one shot" action, as once the basic set up of the Kobo is done, I "just" add the new books, which does not take me more than 15mn. - Now, what's happening, and absolutely new for me: my first batch works well, the second one is quite long, but still works... and whatever I do, the third one fails. I get an error message from Calibre: Code:
calibre, version 3.32.0 ERREUR : Erreur: Erreur pendant la communication avec le périphérique Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 87, in run File "site-packages\calibre\gui2\device.py", line 540, in _sync_booklists File "site-packages\calibre\devices\kobo\driver.py", line 947, in sync_booklists File "site-packages\calibre\devices\usbms\driver.py", line 448, in sync_booklists File "site-packages\calibre\devices\usbms\driver.py", line 446, in write_prefix File "site-packages\calibre\ebooks\metadata\book\json_codec.py", line 135, in encode_to_file File "json\__init__.py", line 251, in dumps File "json\encoder.py", line 210, in encode MemoryError I'm not able to understand where the issue is coming from. I understand a "memory" error. But it can hardly be a device memory issue, as I have enough free space (particularly after the second batch). Can anybody help me to understand what is the problem here? Thanks a lot, and wish you a very nice day |
10-09-2018, 03:04 AM | #2 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
That error is happening when generating the calibre.metadata file. And it looks like the process is running out of memory. Are you running the 32bit version of calibre? If so, try the 64 bit version. That is the only thing I can think of with what you are seeing.
|
Advert | |
|
10-09-2018, 03:14 AM | #3 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Hi Davidfor
That's a good tip... I install Calibre for years, but it's true, I didn't check which version I installed this time... Maybe I downloaded the wrong one... I'll try this evening and give you a feedback. Thanks a lot and have a nice day. |
10-10-2018, 12:24 AM | #4 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Hi David,
Yes in fact... stupid me... I changed the version yesterday evening. Thanks a lot and have a nice day |
10-10-2018, 12:32 AM | #5 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Not a stupid thing, you apparently just hit a limit. And for my curiosity, how many books have you put on the device? And how large is the calibre.metadata file?
|
Advert | |
|
10-11-2018, 12:49 AM | #6 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
My Kobo GloHD runs with Calibre 2.68 and Firmware 3.18.
I changed my SD card with a 128 GB one an I have my full library on it... Means around 53000 books. And it runs really fine. When I connect the device to Calibre, it takes 10mn to connect and make the update. When I export the new books on it, it does not take more than 10mn (not more than 200 books, I update regularly). When I use the device, absolutely no dump or running time. I have another Kobo with "only" 1000 books on it, and the "big" one behaves the same as the "small" one. Regarding the file size, I don't know where to find it... If you can tell me where I have to look for, I'll check it, but the library size itself is around 32GB. |
10-11-2018, 01:14 AM | #7 | |||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
You are sticking with older firmware? If so, I don't think 3.18 is a good choice. The second 3.19 is a better choice.
Quote:
Quote:
Quote:
|
|||
10-11-2018, 02:47 PM | #8 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Hi David
In the meantime I checked the calibre.metadata file. The size is 395MB. Ok, I have to precise that I use the ereader only to read... no games, no internet, no wifi. And of course, with this type of library, I take specially care of each single book - not corrupted, size, polish, classification, shelves etc - to avoid any corruption of the DB. On top, I have an full image to be able to restore the device if something breaks. Thanks for the tip to turn off the "Read metadata from files on the device" feature. I'll try it out the next time I connect the device and have a look if it runs faster. (by the way, when I connected this evening, took exaclty 8mn45sec to synchronise with Calibre, which is already a good score for 53000 books on the device). Regarding the FW version. In fact, I tried out the FW 3.19 but faced the issue that the device returned a "fatal error" around 40000 books. The only way out was a factory reset... and to start again. I have several GloHDs (not all mine, but I take care of) all set up in the same way, which excluded a HW issue on one single device. Therefore, I came to the conclusion, that the FW may be somehow limited. So this was a kind of no-go for me to upgrade. Now, as I like to explore the limits of my devices , I'm making tests with the Clara... and the Clara is not compatible with the 3.18... so I must use a 4.x version. which I find "sexy". Just curious to see if I'll be able to cross the 40000, or if it's limited also. |
10-11-2018, 08:51 PM | #9 | ||||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB. Quote:
Quote:
Quote:
Last edited by davidfor; 10-11-2018 at 08:59 PM. Reason: Added question about database size. |
||||
10-11-2018, 11:00 PM | #10 | |||||
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Quote:
Quote:
Quote:
Quote:
Quote:
|
|||||
10-12-2018, 12:52 AM | #11 | |||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
Quote:
At the point that error happens, the device should be connected, and the list of books should show in the device list. Sending books should work, but, you will probably see the error after the send finishes. I would be interested in seeing exactly when this is happening. There is something about when this is happening that is bugging me. Could you run calibre in debug mode and send me the log? Do this by right-clicking on the preferences button and selecting "Restart in debug mode". Calibre will restart with a message about the mode. Connect the device, let the error happen and close calibre. The debug log will be displayed. Post that or put it somewhere and send me a PM with the link to download it. I'll look at it and see if I can think of anything to improve this. |
|||
10-12-2018, 03:01 PM | #12 | |||
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Hi David
First thanks a lot for your interest. I'm absolutely aware that this is not really a common issue :-) Quote:
Quote:
Quote:
Have a very nice day |
|||
10-17-2018, 01:00 AM | #13 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Hi David,
Just to keep you posted: I'm transferring the books, and try to let the error happen. I don't know if it comes from the debug mode as I never worked with it, but it takes really ages. I transferred 21000 books in different batch sizes, but it took me around 11 hours to have it completed. However, for the moment Calibre behaves well except from a time point of view. I'll come back to you as soon as I see the error popping up. Have a nice day Last edited by Syana344; 10-17-2018 at 01:06 AM. |
10-18-2018, 12:41 AM | #14 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
Dear David
Last but not least... Now it happened... Device is the Clara HD, Version of calibre is 3.32 (64bits), FW is 4.10.11655 patched version. So... what I did: Batch 1 : 9000 books, took 4h48min., Reconnection time (for synch) 1m44s Batch 2 : 1000 books, took 37min, Reconnection time (for synch) 1m35s Batch 3 : 5000 books, took 2h58min, Reconnection time (for synch) 2m26s Batch 4: 6000 books, took 3h24min, Reconnection time (for synch) 3m37s Batch 5: 6000 books – error after 1h59min, not same error code This time I have following: Code:
calibre, version 3.32.0 ERREUR : Erreur: Erreur pendant la communication avec le périphérique Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 87, in run File "site-packages\calibre\gui2\device.py", line 591, in _upload_books File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__ File "site-packages\calibre\utils\localunzip.py", line 231, in extractall File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall ValueError: Not a ZIP file Kobo did not start the import of the books => I tried to switch off, switch on again without result. I tried to repair the account without result, I tried to import just “one book” (sometimes it’s enough to relaunch the import process), but did not work. I tried to force a start up of the device, but this did not work. It's writen in the user manual that it should happen when you switch off your reader and release the power button and press it again for about 30 seconds, but nothing happened. The reader just restarted "normally" withtout starting the import of anything. At the end, I tried to re-import the same 6000 books. Import again stucked with the error code. Code:
calibre, version 3.32.0 ERREUR : Erreur: Erreur pendant la communication avec le périphérique Not a ZIP file Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 87, in run File "site-packages\calibre\gui2\device.py", line 591, in _upload_books File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__ File "site-packages\calibre\utils\localunzip.py", line 231, in extractall File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall ValueError: Not a ZIP file I have the debug log file done, and will send it to you per PM. I did not empty the temp folder - which I will definitely have to do, but not yet. So... that's it from my side. I can restart a trial with the Glo HD for which I have a 4.8.11073 patched version, but I'm quite sure we'll end up with the same result (at least the last 2-3 times I tried, it behaved the same). Have a very nice day |
10-18-2018, 01:19 AM | #15 |
Member
Posts: 22
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
|
This I had some time to check: 2.84GB (with all 53000 books on the Glo HD)
|