Suggestion for improving the efficiency of loading new readers

[HarryT]

Hi Kovid,

Could I make a suggestion, which I think would greatly ease the ease of using Calibre to load books onto a new device? (Apologies in advance - this is a rather long-winded post!)

I've just bought a new Kindle Oasis 2, with 32GB of storage space, which allows me to have a much larger proportion of my Calibre library on the device than I could previously. I buy books from all sorts of different places, so have a mixture of formats in my library, but always want to send AZW3 to the device (which is set as the device's preferred format).

So, I highlight or mark a block of books in my library, right click, and select "Send to Device". If all the books are in Mobi or AZW3 format, no problem: I get two jobs executed. "Upload 53 books to the device" (or whatever the number is), followed by "Send metadata to the device". Within about 40s I've got all 53 books on my Kindle.

The issue occurs when, let's say, 15 of the books are only in ePub format. I get a "No available formats" warning, and Calibre asks me if I want to auto-convert the 15 books to AZW3 format. I say "Yes", and 15 conversion jobs get sent to the job queue. The books get converted to AZW3 format one at a time, and there are then 15 "Upload one book to the device" jobs in the queue.

The problem occurs when you send, say, 1000 books in total to the device, and you end up with a few hundred conversions. Conversion is fast, but unfortunately sending books isn't. The book gets sent to the device essentially instantly, but after each book there's a "Send metadata" step which takes 30+ seconds.

So in the job queue you end up with a sequence of steps that's something like:

Upload one book to the device (0s)
Send metadata to the device (35s)
Upload one book to the device (0s)
Send metadata to the device (35s)
Upload one book to the device (0s)
Send metadata to the device (35s)
Upload one book to the device (0s)
Send metadata to the device (35s)

etc etc.

As I'm writing this, my Calibre job queue, having completed all the conversions for my authors starting with the letters "J" and "K" has 298 "Upload one book to the device" steps waiting in the queue. With each "Send metadata" step taking on average 35s, that means it'll take about 298 x 35s = 10,430s, or 2h 54m to send all the books, even though actually transferring those 298 books could probably be done in under a minute.

My suggestion is this:

If Calibre's job "engine" finds that it has a whole series of sequential "Upload one book to the device" jobs in the queue, is there any way to make it "smart" and merge all the "upload" jobs into a single upload operation? That way there would be a single "upload" job followed by a single "send metadata" job, and what currently takes hours to complete could be done in a couple of minutes.

Hope you've had the patience to read through all this, and I hope you'll give the idea some consideration.

Many thanks,

[kovidgoyal]

Certainly, its something that the jobs manager could do, although it is a fairly involved implementation.

Or you could just search for

not format:=azw3


then schedule a conversion for all the books. That should be quick and you can then send all the books in a single batch.

[HarryT]

Thanks, Kovid. Yes, I realise that separating the conversion and sending steps is a workaround for the issue, and that's what I'll do with the rest of my library. It just occurred to me that this would be a beneficial optimisation in some future release of Calibre.

Thanks for considering the request, anyway .

[Strether]

This was very helpful. Thanks Harry and Kovid. I was converting mine one at a time.

Jim.

[HarryT]

Originally Posted by Strether

This was very helpful. Thanks Harry and Kovid. I was converting mine one at a time.

Jim.

Whatever you do, don't send books one at a time . Always select a block of books, and send the block - that way the block gets sent as a single unit, which is enormously faster, because there's only one "Send metadata" step for the entire block, and that's the slow part of the operation.

[Strether]

Originally Posted by HarryT

Whatever you do, don't send books one at a time . Always select a block of books, and send the block - that way the block gets sent as a single unit, which is enormously faster, because there's only one "Send metadata" step for the entire block, and that's the slow part of the operation.

I'd already seen that on books I'd converted by hand, though hadn't understood why it was so much faster. But I would never have known that I could mass convert the books, nor how to accomplish it. Calibre is busily converting 501 books.

Jim

[HarryT]

Originally Posted by Strether

I'd already seen that on books I'd converted by hand, though hadn't understood why it was so much faster. But I would never have known that I could mass convert the books, nor how to accomplish it. Calibre is busily converting 501 books.

Jim

Sure - just select the books, right click, and "Bulk convert".

[kovidgoyal]

Originally Posted by HarryT

Sure - just select the books, right click, and "Bulk convert".

You dont even need to do that. Just select multiple books and click convert. calibre will automatically show you the bulk conversion dialog.

[HarryT]

Originally Posted by kovidgoyal

You dont even need to do that. Just select multiple books and click convert. calibre will automatically show you the bulk conversion dialog.

That's useful to know, thanks. I must admit I never use the buttons, only the right-click menu, so doing it that way I have to choose either individual or bulk conversion explicitly. Calibre often offers more than one way of doing something!

[theducks]

Originally Posted by HarryT

That's useful to know, thanks. I must admit I never use the buttons, only the right-click menu, so doing it that way I have to choose either individual or bulk conversion explicitly. Calibre often offers more than one way of doing something!

I just select ant type "C". The menu is only for non-default conversions

[compurandom]

Actually, it would help me a lot of the "sending metadata" jobs could be all collected at the end. My normal work flow ends up generating 4 or 5 of those for a single connect.

Would it be easy to change it so that the metadata update is collected and postponed until it is the last device job in the queue?

[kovidgoyal]

It's not that easy. The problem is other device jobs can depend on the results of updating metadata on the device. So you cannot just move the update maetadata job past all other device jobs.

[davidfor]

Originally Posted by kovidgoyal

It's not that easy. The problem is other device jobs can depend on the results of updating metadata on the device. So you cannot just move the update maetadata job past all other device jobs.

I was thinking that when adding a "Send metadata to device" job, that all "Send metadata to device" jobs that are in waiting status in the queue could be cancelled. I know this would be safe for the Kobo devices. I don't remember seeing anything in the other drivers that would mean it is not safe. I would have said that any that are really just the USBMS driver would have been safe.

Or am I missing some device jobs? From what I see, there are four started by calibre when the device connects: Get device info, Set library, Get list of books and Send metadata. Then there are upload and delete jobs. The Kobo Utilities plugin can start two, but they can be run after the get list job. The Reading List plugin can trigger upload jobs, but that is upload all on a list. I can't think of any other plugins that do or any other device jobs.

[kovidgoyal]

The problem is that if you cancel job 1 the job 2 which could depend on the result of job 1 would be broken. Sending metadata to the device is actually a fairly important job -- it is called when uploading books, when auto-updating metadata, after metadata is *downloaded* from the device. Some device drivers update the list of collections in the sending metadata job, so the display of the collections and any operations on collections depend on that job.

[HarryT]

The "Send metadata" step only seems to be created after an "upload" job runs, though. If the "upload" jobs could somehow be merged prior to running, the "Send metadata" would look after itself, wouldn't it?