[Kobo] Transfer to Clara HD dumps - Page 2

davidfor · 10-18-2018, 08:58 AM

Quote:

Originally Posted by Syana344

This I had some time to check: 2.84GB (with all 53000 books on the Glo HD)

WOW! That would basically fill the standard storage of the Glo HD.

davidfor · 10-18-2018, 09:39 AM

Quote:

Originally Posted by Syana344

Dear David

Last but not least...

Now it happened...
Device is the Clara HD, Version of calibre is 3.32 (64bits), FW is 4.10.11655 patched version.

So... what I did:
Batch 1 : 9000 books, took 4h48min., Reconnection time (for synch) 1m44s
Batch 2 : 1000 books, took 37min, Reconnection time (for synch) 1m35s
Batch 3 : 5000 books, took 2h58min, Reconnection time (for synch) 2m26s
Batch 4: 6000 books, took 3h24min, Reconnection time (for synch) 3m37s
Batch 5: 6000 books – error after 1h59min, not same error code

I don't think you mentioned before that you were using the KoboExtendedDriver. That explains the time taken to send the books. Each is being modified to be a kepub during the send. I've been doing some experimenting and calibre is sending 1175 books using the KoboTouch driver in a little over 10 minutes. The device imported these in about 15 minutes.

Quote:

This time I have following:

Code:

calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique
Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 591, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub
  File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__
  File "site-packages\calibre\utils\localunzip.py", line 231, in extractall
  File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall
ValueError: Not a ZIP file

That error is from the extended driver trying to modify a book. Apparently it isn't a valid zipfile and cannot be unpacked. There is an option to ignore errors like this and send the book as an epub. But, that would cause a problem on the device.

Quote:

Big temp file created in the AppData/Local/temp folder => calibre_ta0k3r with 39.1 GB

Kobo did not start the import of the books => I tried to switch off, switch on again without result. I tried to repair the account without result, I tried to import just “one book” (sometimes it’s enough to relaunch the import process), but did not work. I tried to force a start up of the device, but this did not work. It's writen in the user manual that it should happen when you switch off your reader and release the power button and press it again for about 30 seconds, but nothing happened. The reader just restarted "normally" withtout starting the import of anything. At the end, I tried to re-import the same 6000 books. Import again stucked with the error code.

Code:

calibre, version 3.32.0
ERREUR : Erreur: Erreur pendant la communication avec le périphérique

Not a ZIP file

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 591, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 396, in upload_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2021, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 200, in _modify_epub
  File "site-packages\calibre\ebooks\oeb\polish\container.py", line 1143, in __init__
  File "site-packages\calibre\utils\localunzip.py", line 231, in extractall
  File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall
ValueError: Not a ZIP file

The reader is now on the home page, fully charged and shows 20999 imported items (which makes me think that I hit a limit at 21000). I think I'm able to reconnect it (I did not try, but the last time it worked).

I have the debug log file done, and will send it to you per PM. I did not empty the temp folder - which I will definitely have to do, but not yet.

So... that's it from my side. I can restart a trial with the Glo HD for which I have a 4.8.11073 patched version, but I'm quite sure we'll end up with the same result (at least the last 2-3 times I tried, it behaved the same).

20999 is a weird number to stop on. Using kepubs could be dropping the maximum books the device can see as they use more space in the database. But, I have been discussing this elsewhere and that person is using an account with 60K purchased books in it. I could see issue during the import if you tried to do them all in one go.

For sideloading, if there is a problem with a book, then it can block the rest from being imported. If you are seeing the import screen after ejecting without adding a book, then this is probably what is happening. Unfortunately, there isn't any indication of which book causes this. You have to use trial and error to find it.

If there is an error happening, there could be stack log in the .kobo directory. This will have the name stack_0n.log where the n is a digit. This can give a clue as to the problem, but is usually more use

I'm curious about what is going on, so I'm doing some tests. Unfortunately, the biggest spare card I have is 16GB. I'll see how far I get with that, but I might get a bigger card on the weekend. I'm just adding the same 1145 books multiple times. I know they load OK, so they should work for this.

Syana344 · 10-23-2018, 02:30 PM

Hi David,

Thank you very much for your reply. I really appreciate, and your message "turned on some lights".

Quote:

Originally Posted by davidfor

I don't think you mentioned before that you were using the KoboExtendedDriver. That explains the time taken to send the books. Each is being modified to be a kepub during the send. I've been doing some experimenting and calibre is sending 1175 books using the KoboTouch driver in a little over 10 minutes. The device imported these in about 15 minutes.

Guess what... I was not aware of, it's why I didn't mention...

Basically, I made some tests, I think one year ago, about the difference epub, Kepub, and the added-value a kepub may provide. Finally, I decided that's not really worth... At that time, I activated the extension and... I forgot to deactivate it. So... first action when I read your message: I deactivated the extension and checked all the other ones

Maybe this was also the issue I faced when I tried to upgrade 3.18 to a newer version... Let's check.

Quote:

Originally Posted by davidfor

That error is from the extended driver trying to modify a book. Apparently it isn't a valid zipfile and cannot be unpacked. There is an option to ignore errors like this and send the book as an epub. But, that would cause a problem on the device.

Here, I have to say, I'm surprised. In fact, I read the code, but was not able to understand. In my library, I see one single format, which is epub. No Zip format at all... How can it be that Calibre tries to unpack a zip, which is not available in the format?

Quote:

Originally Posted by davidfor

20999 is a weird number to stop on. Using kepubs could be dropping the maximum books the device can see as they use more space in the database. But, I have been discussing this elsewhere and that person is using an account with 60K purchased books in it. I could see issue during the import if you tried to do them all in one go.

For sideloading, if there is a problem with a book, then it can block the rest from being imported. If you are seeing the import screen after ejecting without adding a book, then this is probably what is happening. Unfortunately, there isn't any indication of which book causes this. You have to use trial and error to find it.

So, I know that the error is between 20999 and 22000. Not bad for a start

Quote:

Originally Posted by davidfor

If there is an error happening, there could be stack log in the .kobo directory. This will have the name stack_0n.log where the n is a digit. This can give a clue as to the problem, but is usually more use

I'm curious about what is going on, so I'm doing some tests. Unfortunately, the biggest spare card I have is 16GB. I'll see how far I get with that, but I might get a bigger card on the weekend. I'm just adding the same 1145 books multiple times. I know they load OK, so they should work for this.

On my side, I'll start a new test with the Glo HD (same config), which is already ready to start... just to see if I face the same error... or a similar one... or no issue at all :-)

I'll need some time to get there, but anyhow, will keep you posted.

Thanks again for your support and I wish you a very nice day.

davidfor · 10-23-2018, 10:19 PM

Quote:

Originally Posted by Syana344

Hi David,

Thank you very much for your reply. I really appreciate, and your message "turned on some lights".

Guess what... I was not aware of, it's why I didn't mention...

Basically, I made some tests, I think one year ago, about the difference epub, Kepub, and the added-value a kepub may provide. Finally, I decided that's not really worth... At that time, I activated the extension and... I forgot to deactivate it. So... first action when I read your message: I deactivated the extension and checked all the other ones

Maybe this was also the issue I faced when I tried to upgrade 3.18 to a newer version... Let's check.

There are a couple of nice things in the kepub format. And somethings that people hate. It's about even as to which people use.

Quote:

Here, I have to say, I'm surprised. In fact, I read the code, but was not able to understand. In my library, I see one single format, which is epub. No Zip format at all... How can it be that Calibre tries to unpack a zip, which is not available in the format?

An epub is a zip file. Calibre opens them with Python zip functions. Hence, if an epub is an invalid zip file, you see this error.

Quote:

So, I know that the error is between 20999 and 22000. Not bad for a start

I have been slowly adding books to my old Glo which I put a 16GB card on the weekend. All I did was to use calibre to send a batch of 924 epubs to it. After they were imported, I connected to the PC and copied the directory. I'm doing it in batches like this to get an idea of when it stops working. Last night I was up to about 25000 books. This is with the 4.11.11911 firmware that has just been released (but not to the Glo).

I also tried this with my Aura H2O. But, in that case, I accidentally used a mix of epubs and PDF and had to use the external SD card. That failed at about 12000 books using the same firmware.

In both cases, one of the problems I am seeing is that it takes a long time after connecting to the PC before the PC sees the device. That is slowing things down a lot.

Syana344 · 10-24-2018, 01:04 AM

Quote:

Originally Posted by davidfor

An epub is a zip file. Calibre opens them with Python zip functions. Hence, if an epub is an invalid zip file, you see this error.

I just had an idea: instead of trial and error process, which may take quite some time, what if I try to convert the 1000 books? If Calibre is not able to unpack the zip for the transfer, it should show me which one it's not able to unpack to convert. Am I right? At least, it would give me an idea of which one I have to have a closer look on.

Quote:

Originally Posted by davidfor

In both cases, one of the problems I am seeing is that it takes a long time after connecting to the PC before the PC sees the device. That is slowing things down a lot.

Sees the device or sees which books are already on the device? I never faced the problem about having the device on explorer, but in fact, the sync time with Calibre took me up till 25-30mn, whereas, with my "old" configuration, it takes me 10mn.

davidfor · 10-24-2018, 02:16 AM

Quote:

Originally Posted by Syana344

I just had an idea: instead of trial and error process, which may take quite some time, what if I try to convert the 1000 books? If Calibre is not able to unpack the zip for the transfer, it should show me which one it's not able to unpack to convert. Am I right? At least, it would give me an idea of which one I have to have a closer look on.

Yes, that would work. Embed metadata or Polish books would do the same. I'd probably use Embed metadata as I wouldn't have to clean up afterwards.

Quote:

Sees the device or sees which books are already on the device? I never faced the problem about having the device on explorer, but in fact, the sync time with Calibre took me up till 25-30mn, whereas, with my "old" configuration, it takes me 10mn.

Just for the Windows machine to mount the device. I'm not running calibre when I do this. That's a later step.

Advert

Advert