[Kindle] Very slow to transfer

[theducks]

Originally Posted by ilovejedd

If you do on-the-fly conversion, does Calibre keep the converted books in its library or are they gone as soon as they're sent to the device?

If the converted copies don't ever get stored in the Calibre library, that would explain why it does a send immediately after conversion. Imagine Calibre crashing at book 1999 of a 2000 book transfer and needing to start all over again.


One trick I use to speed up sending thousands of books to a device (assuming embedded book metadata does not need to be updated):

1. Use "Save to disk" with "Update metadata in saved copies" unchecked to save books to a folder on SSD.
2. Exit Calibre.
3. Copy the save to disk folder to Kindle using FastCopy, TeraCopy, Robocopy, etc.
4. (optional) Start Calibre again to update device metadata cache (metadata.calibre and covers?).

This shaves off around 40% from the transfer time for me (practically pure text, fairly small file sizes so relatively large overhead). I also find it useful being able to skip the metadata cache update when I'm in a hurry.

OTF conversion (format was not there), leaves it as part of the Library.
I think, the in book (format exists) metadata happens fairly quickly.

[alexter]

Originally Posted by ilovejedd

If you do on-the-fly conversion, does Calibre keep the converted books in its library or are they gone as soon as they're sent to the device?

It keeps the book on the library. Not using the on-the-fly conversion anymore and its all good. Just sent a batch of 357 books in 3:40 minutes and the metadata took 2:15. All done in 5:55 minutes. Using on-the-fly conversion and upload method would take more than 12 hours.
As it is the on-the-fly conversion is only usefull when sending a single book to the device.

[ilovejedd]

Originally Posted by theducks

OTF conversion (format was not there), leaves it as part of the Library.
I think, the in book (format exists) metadata happens fairly quickly.

Typical usage, sure. Alas, I once did a batch transfer of 5,500 books that crashed on me so instead of spending an hour doing smaller batches, I just found a different method.

[mst]

In my case, transferring files from Calibre to a device does take much longer than just copying the files from the HD to the device, even without sending and updating metadata. Sending 2k epubs from Calibre to the Kobo Clara HD takes about 10 hours. Copying them over with Fastcopy takes 15 minutes.

This is not a question of the speed of sending 2,000 1MB files vs. one file of 2GB. Something going on in the background between Calibre and the device is probably slowing down the transfer considerably. It also seems like sending smaller batches of hundreds of books takes less time than sending one large batch of thousands.

Has anybody additional ideas how to optimize this transfer process from Calibre to the device without using third-party tools like Fastcopy?

[davidfor]

Originally Posted by mst

In my case, transferring files from Calibre to a device does take much longer than just copying the files from the HD to the device, even without sending and updating metadata. Sending 2k epubs from Calibre to the Kobo Clara HD takes about 10 hours. Copying them over with Fastcopy takes 15 minutes.

This is not a question of the speed of sending 2,000 1MB files vs. one file of 2GB. Something going on in the background between Calibre and the device is probably slowing down the transfer considerably. It also seems like sending smaller batches of hundreds of books takes less time than sending one large batch of thousands.

Has anybody additional ideas how to optimize this transfer process from Calibre to the device without using third-party tools like Fastcopy?

It has been a while since I sent that number of books to one of my Kobo device, but, it definitely didn't take that amount of time. And to check, I sent 100 books and it took less than 1 minute to send and another 12 seconds to update the metadata. I then tried the KoboTouchExtended driver and that took 1 minute 35 seconds for the send plus the same time for the metadata update. This included sending the covers to the device. These were fairly small book, but, it agrees with my experience. This was my Aura H2O that doesn't have many books on it. One of the ones with a lot of books would take longer for the metadata update, but, the actual book send wouldn't be much longer.

Calibre does do a lot of work when sending books to the device. It updates the metadata in the book before sending it. If needed, it does a conversion of the book to a format the device supports. For Kobo devices, you have the option to send the cover separately which means resizing the cover for three sizes and copying them. You should expect it to take longer than a simple copy, but, it shouldn't take that much longer.

If there is a problem, we need to see where it is. Run calibre in debug mode and get the debug logs. From that, you should be able to see where the time is being taken. Is it spread evenly over the actions? Or is there something taking a large chunk of time?

[JSWolf]

delete post

[sricochet]

I've found that if you make a queue of several jobs that involve the kindle, the task at hand gets really slow. When I made the only job involving the Kindle, with no other jobs queued up, the upload is as fast as a kindle can be. I don't know if it's kindle or calibre. I think the Kindle can't process more than one thing at a time though, because I've tried deleting and uploading at the same time with the finder and that slows things down quite a bit. Just a tip!