Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Devices

Notices

Reply
 
Thread Tools Search this Thread
Old 04-15-2019, 01:15 PM   #1
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
[Kindle] Very slow to transfer

Having this problem as well with slow metadata uploads to my kindle. Also It updates the metadata after each book wich makes it take a very long time to upload many books to my device. Few seconds for the book upload, 3 minutes for the metadata to update. Is there a way to send all the books and update the metadata just once? It takes 12 hours to send 330 books to my device. It's a pain. Any advice?
Attached Thumbnails
Click image for larger version

Name:	metadata.jpg
Views:	37
Size:	276.4 KB
ID:	170715  
alexter is offline   Reply With Quote
Old 04-15-2019, 01:36 PM   #2
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 22,946
Karma: 23982242
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: K4NT, Galaxy Tab A, Kobo Aura2
Quote:
Originally Posted by alexter View Post
Having this problem as well with slow metadata uploads to my kindle. Also It updates the metadata after each book wich makes it take a very long time to upload many books to my device. Few seconds for the book upload, 3 minutes for the metadata to update. Is there a way to send all the books and update the metadata just once? It takes 12 hours to send 330 books to my device. It's a pain. Any advice?
AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.
Tame your anti-virus (exclude the Kindles drive on the PC).
you might need to Tame your Kindle (tablet) too.

I send, only a few books (novels..not manga) at a time. (I can only read so fast ) A USB transfer takes about 2 minutes to do a half-dozen books and be ready to eject on my K4NT.
Consider Calibre Companion (Wireless Device mode) it may be faster.
You are also limited by how slow your SD card is (lots slower than a HD).
theducks is offline   Reply With Quote
Advert
Old 04-15-2019, 02:17 PM   #3
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
Quote:
Originally Posted by theducks View Post
AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.
Tame your anti-virus (exclude the Kindles drive on the PC).
you might need to Tame your Kindle (tablet) too.
No Antivirus here. Nothing out of order runing in the background. I can also transfer files between HDs on my box at 140MB/s. Using an i7 with 16GB ram. Also I'm using a USB2.0 port wich gives me 30MB/s transfer rate to any external drive I use on it. Also I'm sending the books as a batch, selecting all the books I want and using the "send to main memory" option.
Am I doing something wrong here?
alexter is offline   Reply With Quote
Old 04-15-2019, 02:39 PM   #4
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 3,703
Karma: 9002933
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, iPad 9.7, iPad 12.9
Quote:
Originally Posted by theducks View Post
AFAIK Calibre updates the metadata.calibre ONCE per batch.
If you 1-sies your send, then it does this after each book.
Same behavior on my PW3/PW4 (metadata updated once per batch send).

That said, if one has 13,000 books on the device, I can understand how metadata update can take a very long time regardless considering how slow flash memory on SD cards and devices can be. Per robocopy, copy speed for my 500K-2MB books to a PW3 4GB is ~1-2MB/s. A bit faster on my 32GB Kindles.

Last edited by ilovejedd; 04-15-2019 at 02:41 PM.
ilovejedd is offline   Reply With Quote
Old 04-15-2019, 02:47 PM   #5
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 3,703
Karma: 9002933
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, iPad 9.7, iPad 12.9
Quote:
Originally Posted by alexter View Post
No Antivirus here. Nothing out of order runing in the background. I can also transfer files between HDs on my box at 140MB/s. Using an i7 with 16GB ram. Also I'm using a USB2.0 port wich gives me 30MB/s transfer rate to any external drive I use on it. Also I'm sending the books as a batch, selecting all the books I want and using the "send to main memory" option.
Am I doing something wrong here?
That's probably 140MB/s large file, sustained (or RAM cached).

Try doing very short sequential and random write (1 pass 50MiB, Q31T1 and 4KiB Q1T1) tests on your ereader's storage or SD card with CrystalDiskMark. That's most likely where the bottleneck is.
ilovejedd is offline   Reply With Quote
Advert
Old 04-15-2019, 02:49 PM   #6
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
The real problem for me is the multiple tasks to send the books and the metadata. It's sending each book and metadata individually... If I could convert from epub to mobi and send all the books at once and then update the metadata just once it would solve my problem. As it is it's taking 3 minutes to send each book to device, and I'm on the process to send 2.200 books to my kindle 32GB.
Ps. My drives are not overloaded, I have a pretty fast computer with my OS on a SSD and 6 7200 rpm Seagate HDs connected on SATA ports. I have absolutely no problems transferring files on my computer, both between HDs and to my external drives.
Ps2. I'm on Windows 7 64bit using Calibre 64bit 3.40.1

Last edited by alexter; 04-15-2019 at 03:07 PM.
alexter is offline   Reply With Quote
Old 04-15-2019, 03:20 PM   #7
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 3,703
Karma: 9002933
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, iPad 9.7, iPad 12.9
Quote:
Originally Posted by alexter View Post
The real problem for me is the multiple tasks to send the books and the metadata. It's sending each book and metadata individually... If I could convert from epub to mobi and send all the books at once and then update the metadata just once it would solve my problem. As it is it's taking 3 minutes to send each book to device, and I'm on the process to send 2.200 books to my kindle 32GB.
Ps. My drives are not overloaded, I have a pretty fast computer with my OS on a SSD and 6 7200 rpm Seagate HDs connected on SATA ports. I have absolutely no problems transferring files on my computer, both between HDs and to my external drives.
Ps2. I'm on Windows 7 64bit using Calibre 64bit 3.40.1
We're not saying your PC storage is slow.

We're saying the e-reader/Kindle onboard storage is slow (see attached benchmark results for my PW3 and PW4).

Also, transferring 2,200 2MB ebooks takes longer than transferring a single 4.4GB file.

Calibre doesn't divide my send to device jobs into multiple tasks. Are you converting ebooks as you send? If I were you, I'd just run a batch conversion job and do the send to device afterwards.
Attached Thumbnails
Click image for larger version

Name:	CrystalDiskMark PW3.png
Views:	6
Size:	35.1 KB
ID:	170716   Click image for larger version

Name:	CrystalDiskMark PW4.png
Views:	7
Size:	35.0 KB
ID:	170717  
ilovejedd is offline   Reply With Quote
Old 04-15-2019, 03:31 PM   #8
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
Yes, I'm converting as I send. I'll try to batch convert the books first and then send to device. Will report here when done. Thanks for the help
alexter is offline   Reply With Quote
Old 04-15-2019, 03:47 PM   #9
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
Yep, after converting calibre sends all the books in one task and updates the metadata just once. Maybe this behavior could be added to the next version? If a convert and send is requested calibre would first convert and then send all the books as a batch? Thanks for the help, it solved my problems
alexter is offline   Reply With Quote
Old 04-15-2019, 03:54 PM   #10
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 22,946
Karma: 23982242
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: K4NT, Galaxy Tab A, Kobo Aura2
Quote:
Originally Posted by alexter View Post
Yes, I'm converting as I send. I'll try to batch convert the books first and then send to device. Will report here when done. Thanks for the help
Even if you let Calibre convert (needed) books on the fly, a Batch that includes thos, still only sends the metadata file ONCE.
note: 2K books is going to have a HUGE metadata.calibe to re-write (this is an XML file, so the whole thing gets redone)
I just had to restore a 8G SD card (~4G used) from the backup on my HD. almost 5 times as long to restore, as it was to back it up (and that was using the local card reader)

It is not about CAN you do it. But SHOULD you do it. I did not read 1% last year (and I got a lot of reading done ) of the number of books you are transferring.
theducks is offline   Reply With Quote
Old 04-15-2019, 04:01 PM   #11
crane3
Addict
crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.crane3 ought to be getting tired of karma fortunes by now.
 
Posts: 388
Karma: 2021584
Join Date: Sep 2014
Location: Calif
Device: Fire hdx 8.9, Tab S2, Tab Pro 8.4, Aura ONE
I think the slowdown is at the Kindle. Remember that each book sent needs a folder creation for the author, a sub-folder for the book then the files for the book in the sub-folder. Each step may require a security/permissions check depending on the tablet os version & cpu.

I was delighted that my Mediapad M5 was able to write to the external sd nearly as fast as the Fire HDX, sometimes. Then the Tab S2 & ASUS Zenbook have the slowness I expected in adding books to the CC book lib on the external sd.

The Likebook Mars is even "slower than molasses" in adding books to the CC lib on the external sd..

edit: Also Calibre does some stuff before actually sending the book as seen when selecting the "job" view. about 1 or 2 mins past after several steps before the book is sent. Do not know the additional time/steps needed when converting on the fly.

Haven't tried to see how fast it is on the ipad 6th gen for adding books but that is a different CC & everything is sandboxed.

Last edited by crane3; 04-15-2019 at 04:07 PM. Reason: additional comment:
crane3 is offline   Reply With Quote
Old 04-15-2019, 04:04 PM   #12
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 22,946
Karma: 23982242
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: K4NT, Galaxy Tab A, Kobo Aura2
Quote:
Originally Posted by alexter View Post
Yep, after converting calibre sends all the books in one task and updates the metadata just once. Maybe this behavior could be added to the next version? If a convert and send is requested calibre would first convert and then send all the books as a batch? Thanks for the help, it solved my problems
Calibre sends books as soon as they are ready. The metadata.calibre is held off until the CURRENT batch is done. Open the CALIBRE task manager (click the task wheel) and you will get a idea of what is going on.
There is nothing to chang. It has worked this way for YEARS
theducks is offline   Reply With Quote
Old 04-15-2019, 04:13 PM   #13
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
Yeah. Ill not read all these books of course. The idea is having my kindle as a library so I can chose whatever I want to read. Actually, I have 13.401 books on my library and 11.200 are already on my kindle. I'm just finishing to load the full library on my device. Have also a kindle 7th gen but I bought this new 10th gen 32GB kindle just to load all my collection. Also, I have no problems with the device beeing slow. It is a bit slow if you compare with a USB 3.0 device but my whole problem was the multiple tasks taken to convert from epub to mobi and send the ebooks and metadata to my device. Converting the epubs to mobi beforehand solved all my problems.
alexter is offline   Reply With Quote
Old 04-15-2019, 04:18 PM   #14
alexter
Junior Member
alexter began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Apr 2019
Device: Kindle Paperwhite 7th and 10th gen
Quote:
Originally Posted by theducks View Post
The metadata.calibre is held off until the CURRENT batch is done.
Not what is happening here. If the book is not pre converted to mobi it sends the metadata after each book convert and upload, just take a look at the screenshot I've posted earlier. After I convert the books as a batch it sends all the books at once and then it sends the metadata.
alexter is offline   Reply With Quote
Old 04-15-2019, 05:49 PM   #15
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 3,703
Karma: 9002933
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, iPad 9.7, iPad 12.9
Quote:
Originally Posted by alexter View Post
Not what is happening here. If the book is not pre converted to mobi it sends the metadata after each book convert and upload, just take a look at the screenshot I've posted earlier. After I convert the books as a batch it sends all the books at once and then it sends the metadata.
If you do on-the-fly conversion, does Calibre keep the converted books in its library or are they gone as soon as they're sent to the device?

If the converted copies don't ever get stored in the Calibre library, that would explain why it does a send immediately after conversion. Imagine Calibre crashing at book 1999 of a 2000 book transfer and needing to start all over again.


One trick I use to speed up sending thousands of books to a device (assuming embedded book metadata does not need to be updated):
  1. Use "Save to disk" with "Update metadata in saved copies" unchecked to save books to a folder on SSD.
  2. Exit Calibre.
  3. Copy the save to disk folder to Kindle using FastCopy, TeraCopy, Robocopy, etc.
  4. (optional) Start Calibre again to update device metadata cache (metadata.calibre and covers?).

This shaves off around 40% from the transfer time for me (practically pure text, fairly small file sizes so relatively large overhead). I also find it useful being able to skip the metadata cache update when I'm in a hurry.
ilovejedd is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Transfer of files on Android is incredibly slow, but same files super quick on iPad? lealla Calibre Companion 6 01-30-2018 06:26 AM
[Android] slow transfer rate in my new Smartphone Pierre-Olivier Calibre Companion 11 12-29-2016 11:12 AM
Kindle Voyage metadata transfer very slow HarryT Devices 6 11-08-2014 12:36 PM
Kindle slow vinny3777 Amazon Kindle 1 08-20-2014 10:11 AM
calibre - very slow conversion, very slow on PRS cremofix Calibre 3 06-10-2009 04:21 PM


All times are GMT -4. The time now is 06:36 PM.


MobileRead.com is a privately owned, operated and funded community.