Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Devices

Notices

Reply
 
Thread Tools Search this Thread
Old 10-09-2018, 02:35 AM   #1
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
[Kobo] Transfer to Clara HD dumps

Hi everybody,

I face an issue I never had with Calibre and Kobo.

When I try to transfer my library (yes... the full one) to my brand new Clara HD the transfer fails.

What I did
- First I have to say that I changed the SD card of the Clara to be able to have my full library with me. So I exchanged the existing 8GB with a brand new 128GB (which works already on my Glo HD - so I think that's not the issue).
- I had the configuration Kobo Glo HD FW 3.18 + Calibre 2.68 with the same SD card, and never had an issue.
- Now I bought a Clara HD and I'm working with Calibre 3.32 and a patched version of Kobo 4.10.11655. (I tried the unpached version too, but still have the same issue).
- I reset several times the Clara to factory status - does not change the issue
- I checked my library, all the ebooks seem fine.
- I changed the SD card to make sure the card is working well (and it's working fine on other devices).

What is happening
- I'm used to transfer my full library in batches. So, I launch e.g 5000 books, which works well (transfer in about 25mn + import to Kobo +/- 20min). Takes time, but it's a "one shot" action, as once the basic set up of the Kobo is done, I "just" add the new books, which does not take me more than 15mn.
- Now, what's happening, and absolutely new for me: my first batch works well, the second one is quite long, but still works... and whatever I do, the third one fails. I get an error message from Calibre:

Code:
calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique



Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 540, in _sync_booklists
  File "site-packages\calibre\devices\kobo\driver.py", line 947, in sync_booklists
  File "site-packages\calibre\devices\usbms\driver.py", line 448, in sync_booklists
  File "site-packages\calibre\devices\usbms\driver.py", line 446, in write_prefix
  File "site-packages\calibre\ebooks\metadata\book\json_codec.py", line 135, in encode_to_file
  File "json\__init__.py", line 251, in dumps
  File "json\encoder.py", line 210, in encode
MemoryError
After this error, my pc still recognizes the Kobo, and Kobo even does a proper import of the data, shows me the correct number of books... all seems ok. However, when plug it the next time to Calibe, Calibre starts up, starts to read the content, and after a long long time... closes without any error message or anything...

I'm not able to understand where the issue is coming from. I understand a "memory" error. But it can hardly be a device memory issue, as I have enough free space (particularly after the second batch).

Can anybody help me to understand what is the problem here?

Thanks a lot, and wish you a very nice day
Syana344 is offline   Reply With Quote
Old 10-09-2018, 03:04 AM   #2
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,244
Karma: 24732480
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE, Clara HD
That error is happening when generating the calibre.metadata file. And it looks like the process is running out of memory. Are you running the 32bit version of calibre? If so, try the 64 bit version. That is the only thing I can think of with what you are seeing.
davidfor is offline   Reply With Quote
Advert
Old 10-09-2018, 03:14 AM   #3
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Hi Davidfor

That's a good tip... I install Calibre for years, but it's true, I didn't check which version I installed this time... Maybe I downloaded the wrong one...

I'll try this evening and give you a feedback.

Thanks a lot and have a nice day.
Syana344 is offline   Reply With Quote
Old 10-10-2018, 12:24 AM   #4
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Hi David,

Yes in fact... stupid me... I changed the version yesterday evening.

Thanks a lot and have a nice day
Syana344 is offline   Reply With Quote
Old 10-10-2018, 12:32 AM   #5
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,244
Karma: 24732480
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE, Clara HD
Not a stupid thing, you apparently just hit a limit. And for my curiosity, how many books have you put on the device? And how large is the calibre.metadata file?
davidfor is offline   Reply With Quote
Advert
Old 10-11-2018, 12:49 AM   #6
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
My Kobo GloHD runs with Calibre 2.68 and Firmware 3.18.
I changed my SD card with a 128 GB one an I have my full library on it...
Means around 53000 books.

And it runs really fine.

When I connect the device to Calibre, it takes 10mn to connect and make the update. When I export the new books on it, it does not take more than 10mn (not more than 200 books, I update regularly).
When I use the device, absolutely no dump or running time.

I have another Kobo with "only" 1000 books on it, and the "big" one behaves the same as the "small" one.

Regarding the file size, I don't know where to find it... If you can tell me where I have to look for, I'll check it, but the library size itself is around 32GB.
Syana344 is offline   Reply With Quote
Old 10-11-2018, 01:14 AM   #7
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,244
Karma: 24732480
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE, Clara HD
Quote:
Originally Posted by Syana344 View Post
My Kobo GloHD runs with Calibre 2.68 and Firmware 3.18.
You are sticking with older firmware? If so, I don't think 3.18 is a good choice. The second 3.19 is a better choice.
Quote:
I changed my SD card with a 128 GB one an I have my full library on it...
Means around 53000 books.

And it runs really fine.
Wow. I would have thought that this number of books would have been noticeably slower.
Quote:
When I connect the device to Calibre, it takes 10mn to connect and make the update.
The initial connect takes 10 minutes? I'm not surprised. It might speed things up if you turn off the "Read metadata from files on the device" in the driver configuration, it might be faster. I think but it's been ages since I looked at exactly how that option worked in the driver.
Quote:
When I export the new books on it, it does not take more than 10mn (not more than 200 books, I update regularly).
When I use the device, absolutely no dump or running time.

I have another Kobo with "only" 1000 books on it, and the "big" one behaves the same as the "small" one.

Regarding the file size, I don't know where to find it... If you can tell me where I have to look for, I'll check it, but the library size itself is around 32GB.
The calibre.metadata file is in the root directory of the device when it is connected. On my Glo HD with about 4500 books, it is 45MB. So, 500MB would be reasonable yours. Calibre rewrites it as part of the connection, so that could take a few minutes.
davidfor is offline   Reply With Quote
Old 10-11-2018, 02:47 PM   #8
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Hi David

In the meantime I checked the calibre.metadata file. The size is 395MB.

Ok, I have to precise that I use the ereader only to read... no games, no internet, no wifi. And of course, with this type of library, I take specially care of each single book - not corrupted, size, polish, classification, shelves etc - to avoid any corruption of the DB. On top, I have an full image to be able to restore the device if something breaks.

Thanks for the tip to turn off the "Read metadata from files on the device" feature. I'll try it out the next time I connect the device and have a look if it runs faster. (by the way, when I connected this evening, took exaclty 8mn45sec to synchronise with Calibre, which is already a good score for 53000 books on the device).

Regarding the FW version. In fact, I tried out the FW 3.19 but faced the issue that the device returned a "fatal error" around 40000 books. The only way out was a factory reset... and to start again. I have several GloHDs (not all mine, but I take care of) all set up in the same way, which excluded a HW issue on one single device. Therefore, I came to the conclusion, that the FW may be somehow limited.

So this was a kind of no-go for me to upgrade.

Now, as I like to explore the limits of my devices , I'm making tests with the Clara... and the Clara is not compatible with the 3.18... so I must use a 4.x version. which I find "sexy". Just curious to see if I'll be able to cross the 40000, or if it's limited also.
Syana344 is offline   Reply With Quote
Old 10-11-2018, 08:51 PM   #9
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,244
Karma: 24732480
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE, Clara HD
Quote:
Originally Posted by Syana344 View Post
In the meantime I checked the calibre.metadata file. The size is 395MB.
Smaller than I thought. But, if you don't use many custom columns, it does keep the size down.

And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB.
Quote:
Ok, I have to precise that I use the ereader only to read... no games, no internet, no wifi. And of course, with this type of library, I take specially care of each single book - not corrupted, size, polish, classification, shelves etc - to avoid any corruption of the DB. On top, I have an full image to be able to restore the device if something breaks.

Thanks for the tip to turn off the "Read metadata from files on the device" feature. I'll try it out the next time I connect the device and have a look if it runs faster. (by the way, when I connected this evening, took exaclty 8mn45sec to synchronise with Calibre, which is already a good score for 53000 books on the device).
Yes, I'm surprised at it being so fast. And I'm not sure how much help turning the "Read metadata from files on the device" off will be. I just checked the code and it will only help if the book on the device hasn't previously been matched to the book in calibre. Or if the calibre.metadata file is missing.
Quote:
Regarding the FW version. In fact, I tried out the FW 3.19 but faced the issue that the device returned a "fatal error" around 40000 books. The only way out was a factory reset... and to start again. I have several GloHDs (not all mine, but I take care of) all set up in the same way, which excluded a HW issue on one single device. Therefore, I came to the conclusion, that the FW may be somehow limited.
That is something I hadn't heard of before. I can't think of what the changes in that time frame would have been that caused this. It isn't deliberate on Kobo's part, more likely something changed that used more memory. I'm curious about when it displayed the error. What was happening on the device.
Quote:
So this was a kind of no-go for me to upgrade.

Now, as I like to explore the limits of my devices , I'm making tests with the Clara... and the Clara is not compatible with the 3.18... so I must use a 4.x version. which I find "sexy". Just curious to see if I'll be able to cross the 40000, or if it's limited also.
One thing that has changed somewhere around version 4.3, is the speed of the collections list. Before this, if you had a lot of collections and any large collections, then it could take minutes to open the collection list. Afterwards, it was done to 10 seconds or so. There have been a lot of other performance improvements as well. Nothing as big as that, but generally it fells snappier. And a few of the other changes make me like the updates (series sorting, author list) but, I prefer the older tiled home screen.

Last edited by davidfor; 10-11-2018 at 08:59 PM. Reason: Added question about database size.
davidfor is offline   Reply With Quote
Old 10-11-2018, 11:00 PM   #10
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Quote:
Originally Posted by davidfor View Post
Smaller than I thought. But, if you don't use many custom columns, it does keep the size down..
I have 5 custom colums

Quote:
Originally Posted by davidfor View Post
And what I should have asked is how big the database is. That is "KoboReader.sqlite" in the ".kobo" directory. Just a curiosity, but it must be getting past a GB.
I'll check and tell you

Quote:
Originally Posted by davidfor View Post
Yes, I'm surprised at it being so fast. And I'm not sure how much help turning the "Read metadata from files on the device" off will be. I just checked the code and it will only help if the book on the device hasn't previously been matched to the book in calibre. Or if the calibre.metadata file is missing.
In fact, I tested it out yesterday evening, and it does not change.

Quote:
Originally Posted by davidfor View Post
That is something I hadn't heard of before. I can't think of what the changes in that time frame would have been that caused this. It isn't deliberate on Kobo's part, more likely something changed that used more memory. I'm curious about when it displayed the error. What was happening on the device.
Basically, Calibre behaved well, exported the epubs and the metadata. I could plug off the device normally from Calibre. After a while, the Kobo started the import, and after a moment, the screen turned to black screen only with a text "Fatal error happened, to recover your device you have to perform a factory reset", with two possibilities either approve or reject, and the two of them triggered a factory reset.


Quote:
Originally Posted by davidfor View Post
One thing that has changed somewhere around version 4.3, is the speed of the collections list. Before this, if you had a lot of collections and any large collections, then it could take minutes to open the collection list. Afterwards, it was done to 10 seconds or so. There have been a lot of other performance improvements as well. Nothing as big as that, but generally it fells snappier. And a few of the other changes make me like the updates (series sorting, author list) but, I prefer the older tiled home screen.
Yes, this I saw. Basically, I was interested in exactly the two features you are listening: series sorting and author list. It's why I was interested in the new FW. But I just made tests with the Clara, and it seems that the limit is even lower. Because it collapses at around 20000 epubs. I finished yesterday evening, with a 64bit version, and I faced exactly the same issue as with the 32bit version... Same error code and same detail message... So it seems, that I'm stucked with my Clara, as I cannot downgrade it to a 3.18 version.
Syana344 is offline   Reply With Quote
Old 10-12-2018, 12:52 AM   #11
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,244
Karma: 24732480
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE, Clara HD
Quote:
Originally Posted by Syana344 View Post
In fact, I tested it out yesterday evening, and it does not change.
I was afraid of that.
Quote:
Basically, Calibre behaved well, exported the epubs and the metadata. I could plug off the device normally from Calibre. After a while, the Kobo started the import, and after a moment, the screen turned to black screen only with a text "Fatal error happened, to recover your device you have to perform a factory reset", with two possibilities either approve or reject, and the two of them triggered a factory reset.
After my post earlier, I was chatting with someone about this, from his comments, I suspected that was were the problem would be. And I'm pretty sure this won't be a problem in the recent firmware. And it is a lot faster. I had to reload all books on my Glo HD on the weekend with 4.10.something. I don't know the exact time as I wasn't watching closely, but the import was less than 30 minutes for 4000 books. With 3.18, I think that would have been a couple of hours. I don't
Quote:
Yes, this I saw. Basically, I was interested in exactly the two features you are listening: series sorting and author list. It's why I was interested in the new FW. But I just made tests with the Clara, and it seems that the limit is even lower. Because it collapses at around 20000 epubs. I finished yesterday evening, with a 64bit version, and I faced exactly the same issue as with the 32bit version... Same error code and same detail message... So it seems, that I'm stucked with my Clara, as I cannot downgrade it to a 3.18 version.
If you are seeing the error in the first post, that shouldn't have anything to do with the firmware version. That is calibre generating a file on the device from the list of books it has detected. Is it exactly the same error? I can't think of any reason this would be happening at half the number of books.

At the point that error happens, the device should be connected, and the list of books should show in the device list. Sending books should work, but, you will probably see the error after the send finishes.

I would be interested in seeing exactly when this is happening. There is something about when this is happening that is bugging me. Could you run calibre in debug mode and send me the log? Do this by right-clicking on the preferences button and selecting "Restart in debug mode". Calibre will restart with a message about the mode. Connect the device, let the error happen and close calibre. The debug log will be displayed. Post that or put it somewhere and send me a PM with the link to download it. I'll look at it and see if I can think of anything to improve this.
davidfor is offline   Reply With Quote
Old 10-12-2018, 03:01 PM   #12
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Hi David

First thanks a lot for your interest. I'm absolutely aware that this is not really a common issue :-)

Quote:
Originally Posted by davidfor View Post
If you are seeing the error in the first post, that shouldn't have anything to do with the firmware version. That is calibre generating a file on the device from the list of books it has detected. Is it exactly the same error? I can't think of any reason this would be happening at half the number of books.
Yes it was exactly the same error which happened not once, not twice, but at least five or six times. After this happens, I have to start again the full process, means factory reset, customizing (patched version) and then the transfer of books into batches.

Quote:
Originally Posted by davidfor View Post
At the point that error happens, the device should be connected, and the list of books should show in the device list. Sending books should work, but, you will probably see the error after the send finishes.
This was exactly what happened when I tried some months ago to upgrade from 3.18 to 3.19 my GloHD. And at the end it finished in a fatal error on Kobo side. But now, with the Clara, I'm even not able to reach this point as the dumps happen on Calibre side before the "fatal error" can happen.

Quote:
Originally Posted by davidfor View Post
I would be interested in seeing exactly when this is happening. There is something about when this is happening that is bugging me. Could you run calibre in debug mode and send me the log? Do this by right-clicking on the preferences button and selecting "Restart in debug mode". Calibre will restart with a message about the mode. Connect the device, let the error happen and close calibre. The debug log will be displayed. Post that or put it somewhere and send me a PM with the link to download it. I'll look at it and see if I can think of anything to improve this.
Of course I can! I just need some time to do it. As the last transfer crashed, I have to restore the full device. Will take a couple of days as I'm absent this weekend, but I think Monday or Tuesday I'll be done with it. I'll send you a PM when done.

Have a very nice day
Syana344 is offline   Reply With Quote
Old Today, 01:00 AM   #13
Syana344
Member
Syana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enoughSyana344 will become famous soon enough
 
Posts: 18
Karma: 546
Join Date: Dec 2015
Device: Kobo Glo, Aura, AuraHD, H2O, GloHD, Clara
Hi David,

Just to keep you posted: I'm transferring the books, and try to let the error happen. I don't know if it comes from the debug mode as I never worked with it, but it takes really ages.

I transferred 21000 books in different batch sizes, but it took me around 11 hours to have it completed.

However, for the moment Calibre behaves well except from a time point of view.

I'll come back to you as soon as I see the error popping up.

Have a nice day
Attached Thumbnails
Click image for larger version

Name:	Calibre.PNG
Views:	5
Size:	90.4 KB
ID:	166968  

Last edited by Syana344; Today at 01:06 AM.
Syana344 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump


All times are GMT -4. The time now is 02:32 PM.


MobileRead.com is a privately owned, operated and funded community.