[Kindle] Very slow to transfer

[alexter]

Having this problem as well with slow metadata uploads to my kindle. Also It updates the metadata after each book wich makes it take a very long time to upload many books to my device. Few seconds for the book upload, 3 minutes for the metadata to update. Is there a way to send all the books and update the metadata just once? It takes 12 hours to send 330 books to my device. It's a pain. Any advice?

[theducks]

Originally Posted by alexter

Having this problem as well with slow metadata uploads to my kindle. Also It updates the metadata after each book wich makes it take a very long time to upload many books to my device. Few seconds for the book upload, 3 minutes for the metadata to update. Is there a way to send all the books and update the metadata just once? It takes 12 hours to send 330 books to my device. It's a pain. Any advice?

AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.
Tame your anti-virus (exclude the Kindles drive on the PC).
you might need to Tame your Kindle (tablet) too.

I send, only a few books (novels..not manga) at a time. (I can only read so fast ) A USB transfer takes about 2 minutes to do a half-dozen books and be ready to eject on my K4NT.
Consider Calibre Companion (Wireless Device mode) it may be faster.
You are also limited by how slow your SD card is (lots slower than a HD).

[alexter]

Originally Posted by theducks

AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.
Tame your anti-virus (exclude the Kindles drive on the PC).
you might need to Tame your Kindle (tablet) too.

No Antivirus here. Nothing out of order runing in the background. I can also transfer files between HDs on my box at 140MB/s. Using an i7 with 16GB ram. Also I'm using a USB2.0 port wich gives me 30MB/s transfer rate to any external drive I use on it. Also I'm sending the books as a batch, selecting all the books I want and using the "send to main memory" option.
Am I doing something wrong here?

[ilovejedd]

Originally Posted by theducks

AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.

Same behavior on my PW3/PW4 (metadata updated once per batch send).

That said, if one has 13,000 books on the device, I can understand how metadata update can take a very long time regardless considering how slow flash memory on SD cards and devices can be. Per robocopy, copy speed for my 500K-2MB books to a PW3 4GB is ~1-2MB/s. A bit faster on my 32GB Kindles.

[ilovejedd]

Originally Posted by alexter

No Antivirus here. Nothing out of order runing in the background. I can also transfer files between HDs on my box at 140MB/s. Using an i7 with 16GB ram. Also I'm using a USB2.0 port wich gives me 30MB/s transfer rate to any external drive I use on it. Also I'm sending the books as a batch, selecting all the books I want and using the "send to main memory" option.
Am I doing something wrong here?

That's probably 140MB/s large file, sustained (or RAM cached).

Try doing very short sequential and random write (1 pass 50MiB, Q31T1 and 4KiB Q1T1) tests on your ereader's storage or SD card with CrystalDiskMark. That's most likely where the bottleneck is.

[alexter]

The real problem for me is the multiple tasks to send the books and the metadata. It's sending each book and metadata individually... If I could convert from epub to mobi and send all the books at once and then update the metadata just once it would solve my problem. As it is it's taking 3 minutes to send each book to device, and I'm on the process to send 2.200 books to my kindle 32GB.
Ps. My drives are not overloaded, I have a pretty fast computer with my OS on a SSD and 6 7200 rpm Seagate HDs connected on SATA ports. I have absolutely no problems transferring files on my computer, both between HDs and to my external drives.
Ps2. I'm on Windows 7 64bit using Calibre 64bit 3.40.1

[ilovejedd]

Originally Posted by alexter

The real problem for me is the multiple tasks to send the books and the metadata. It's sending each book and metadata individually... If I could convert from epub to mobi and send all the books at once and then update the metadata just once it would solve my problem. As it is it's taking 3 minutes to send each book to device, and I'm on the process to send 2.200 books to my kindle 32GB.
Ps. My drives are not overloaded, I have a pretty fast computer with my OS on a SSD and 6 7200 rpm Seagate HDs connected on SATA ports. I have absolutely no problems transferring files on my computer, both between HDs and to my external drives.
Ps2. I'm on Windows 7 64bit using Calibre 64bit 3.40.1

We're not saying your PC storage is slow.

We're saying the e-reader/Kindle onboard storage is slow (see attached benchmark results for my PW3 and PW4).

Also, transferring 2,200 2MB ebooks takes longer than transferring a single 4.4GB file.

Calibre doesn't divide my send to device jobs into multiple tasks. Are you converting ebooks as you send? If I were you, I'd just run a batch conversion job and do the send to device afterwards.

[alexter]

Yes, I'm converting as I send. I'll try to batch convert the books first and then send to device. Will report here when done. Thanks for the help

[alexter]

Yep, after converting calibre sends all the books in one task and updates the metadata just once. Maybe this behavior could be added to the next version? If a convert and send is requested calibre would first convert and then send all the books as a batch? Thanks for the help, it solved my problems

[theducks]

Originally Posted by alexter

Yes, I'm converting as I send. I'll try to batch convert the books first and then send to device. Will report here when done. Thanks for the help

Even if you let Calibre convert (needed) books on the fly, a Batch that includes thos, still only sends the metadata file ONCE.
note: 2K books is going to have a HUGE metadata.calibe to re-write (this is an XML file, so the whole thing gets redone)
I just had to restore a 8G SD card (~4G used) from the backup on my HD. almost 5 times as long to restore, as it was to back it up (and that was using the local card reader)

It is not about CAN you do it. But SHOULD you do it. I did not read 1% last year (and I got a lot of reading done ) of the number of books you are transferring.

[crane3]

I think the slowdown is at the Kindle. Remember that each book sent needs a folder creation for the author, a sub-folder for the book then the files for the book in the sub-folder. Each step may require a security/permissions check depending on the tablet os version & cpu.

I was delighted that my Mediapad M5 was able to write to the external sd nearly as fast as the Fire HDX, sometimes. Then the Tab S2 & ASUS Zenbook have the slowness I expected in adding books to the CC book lib on the external sd.

The Likebook Mars is even "slower than molasses" in adding books to the CC lib on the external sd..

edit: Also Calibre does some stuff before actually sending the book as seen when selecting the "job" view. about 1 or 2 mins past after several steps before the book is sent. Do not know the additional time/steps needed when converting on the fly.

Haven't tried to see how fast it is on the ipad 6th gen for adding books but that is a different CC & everything is sandboxed.

[theducks]

Originally Posted by alexter

Yep, after converting calibre sends all the books in one task and updates the metadata just once. Maybe this behavior could be added to the next version? If a convert and send is requested calibre would first convert and then send all the books as a batch? Thanks for the help, it solved my problems

Calibre sends books as soon as they are ready. The metadata.calibre is held off until the CURRENT batch is done. Open the CALIBRE task manager (click the task wheel) and you will get a idea of what is going on.
There is nothing to chang. It has worked this way for YEARS

[alexter]

Yeah. Ill not read all these books of course. The idea is having my kindle as a library so I can chose whatever I want to read. Actually, I have 13.401 books on my library and 11.200 are already on my kindle. I'm just finishing to load the full library on my device. Have also a kindle 7th gen but I bought this new 10th gen 32GB kindle just to load all my collection. Also, I have no problems with the device beeing slow. It is a bit slow if you compare with a USB 3.0 device but my whole problem was the multiple tasks taken to convert from epub to mobi and send the ebooks and metadata to my device. Converting the epubs to mobi beforehand solved all my problems.

[alexter]

Originally Posted by theducks

The metadata.calibre is held off until the CURRENT batch is done.

Not what is happening here. If the book is not pre converted to mobi it sends the metadata after each book convert and upload, just take a look at the screenshot I've posted earlier. After I convert the books as a batch it sends all the books at once and then it sends the metadata.

[ilovejedd]

Originally Posted by alexter

Not what is happening here. If the book is not pre converted to mobi it sends the metadata after each book convert and upload, just take a look at the screenshot I've posted earlier. After I convert the books as a batch it sends all the books at once and then it sends the metadata.

If you do on-the-fly conversion, does Calibre keep the converted books in its library or are they gone as soon as they're sent to the device?

If the converted copies don't ever get stored in the Calibre library, that would explain why it does a send immediately after conversion. Imagine Calibre crashing at book 1999 of a 2000 book transfer and needing to start all over again.


One trick I use to speed up sending thousands of books to a device (assuming embedded book metadata does not need to be updated):

1. Use "Save to disk" with "Update metadata in saved copies" unchecked to save books to a folder on SSD.
2. Exit Calibre.
3. Copy the save to disk folder to Kindle using FastCopy, TeraCopy, Robocopy, etc.
4. (optional) Start Calibre again to update device metadata cache (metadata.calibre and covers?).

This shaves off around 40% from the transfer time for me (practically pure text, fairly small file sizes so relatively large overhead). I also find it useful being able to skip the metadata cache update when I'm in a hurry.