Enhacement request: Merge similar jobs in the job queue

HarryT · 11-06-2014, 03:41 AM

This is a follow-up to a post I made yesterday in the "Devices" sub-forum.

If you're sending a number of books to a device from a Calibre library, you can end up with a job queue that looks something like:

Upload 1 book
Upload 3 books
Upload 2 books
Upload 1 book
etc.

The problem is that, as things stand at the moment, Calibre does a "Send metadata to device" step at the end of each one of these jobs, and although uploading a book to a device might only take 2s, the subsequent "Send metadata" step (which, as Kovid kindly explained, sends metadata for all the books on the device) can easily take 30-40s on my Kindle Voyage. This can take literally hours if sending books in a large number of separate jobs.

I was thinking, and it occurred to me that perhaps there could be potential for serious optimisation here. Suppose that, when an "Upload Books" job was created and queued, the job queue manager were to scan the current queue and, if it found another "Upload Books" job in the queue, simply added the list of books in the new job to that of the old? That way, the slow "Send metadata" step would only be done once when the combined job ran.

Is this a feasible idea? It really could save a huge amount of time for the user if it were feasible to implement.

kovidgoyal · 11-06-2014, 04:35 AM

It's certainly possible, but IMO not worth the effort. Patches are welcome. In the meantime if you find yourself sending large numbers of books in multiple jobs, instead use the mark books tool to mark the books temporarily and only send all the marked books in one go after you are done with all of them.

HarryT · 11-06-2014, 04:38 AM

That's an excellent idea, thank you. Can you tell me how to use "Mark Books", please? I don't seem to see it anywhere.

EDIT: Ignore - found it.

itimpi · 11-06-2014, 05:08 AM

You can also use the excellent Reading List plugin to queue files for later sending to your ereader device.

HarryT · 11-06-2014, 05:16 AM

"Mark Books" does exactly what I needed - thanks. Another Calibre facility I didn't know existed until now.

HarryT · 11-06-2014, 05:32 AM

Sometimes - but not always - when I highlight a book in the library and hit "Ctrl+M", the "pushpin" marker doesn't appear alongside the book until I click on a different book. Unfortunately I can't find a consistent way to reproduce the problem: sometimes it works, sometimes not. The book gets marked - it's purely a visual issue. 32-bit Calibre 2.8 on 64-bit Windows 7.

Ripplinger · 11-06-2014, 05:49 AM

I didn't know about the mark books function either. Is there a way to sort books by whether they're marked or not? I can't find a way to do that.

HarryT · 11-06-2014, 06:20 AM

Quote:

Originally Posted by Ripplinger

I didn't know about the mark books function either. Is there a way to sort books by whether they're marked or not? I can't find a way to do that.

I don't know about sorting them, but you can certainly show them: on the "Mark Books" submenu, there's a "Show only marked books" option.

kovidgoyal · 11-06-2014, 06:26 AM

Quote:

Originally Posted by HarryT

Sometimes - but not always - when I highlight a book in the library and hit "Ctrl+M", the "pushpin" marker doesn't appear alongside the book until I click on a different book. Unfortunately I can't find a consistent way to reproduce the problem: sometimes it works, sometimes not. The book gets marked - it's purely a visual issue. 32-bit Calibre 2.8 on 64-bit Windows 7.

Hmm, I cant reproduce this. Difficult to fix without being able to reproduce.

Ripplinger · 11-06-2014, 06:43 AM

Quote:

Originally Posted by HarryT

I don't know about sorting them, but you can certainly show them: on the "Mark Books" submenu, there's a "Show only marked books" option.

I found the "show marked books" choice once I added it to the top menu, so that'll do for sorting them.

I used an extra column where I'd put a checkmark in the column, which almost seems to provide the same function. I use it to compare what's on the reader against what I want on the reader, and being able to sort lets me find books I forgot to delete from the reader after I've read them.

HarryT · 11-06-2014, 07:15 AM

Quote:

Originally Posted by kovidgoyal

Hmm, I cant reproduce this. Difficult to fix without being able to reproduce.

I completely appreciate that. If I do find a way to consistently reproduce it, I'll raise a separate bug report. It's a purely cosmetic issue, at worst.

PeterT · 11-06-2014, 10:25 AM

SLightly off topic, but what I've done for ages is to use the Reading List plugin for this purpose. I have one list that is flagged as the default within the plugin that is defined as:

Code:

Name: Sync Later
List Type: Manually add/remove items
Device to Sync To: Kobo Glo (main) <-- This is my standard eReader; others exist but only for testing purposes
When Syncing: Add new list items to device
Sync to this device as soon as it is connected: Checked
Clear this list after a sync to this device: Checked

The Reading List plugin has also been added to both Context Menu's and both Main Toolbar's. This way, when I see book I want to sync, I can right click on it, select the Reading List option, and then the Add to Sync Later List option. After flagging all books I want ifmy Kobo is not plugged in, plug it in and let the sync happen automagically, or if it IS plugged in, select the Sync Now option.

eschwartz · 11-08-2014, 09:42 PM

Quote:

Originally Posted by Ripplinger

I didn't know about the mark books function either. Is there a way to sort books by whether they're marked or not? I can't find a way to do that.

Search:

Code:

marked:true

11-06-2014, 03:41 AM	#1
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383099 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Enhacement request: Merge similar jobs in the job queue This is a follow-up to a post I made yesterday in the "Devices" sub-forum. If you're sending a number of books to a device from a Calibre library, you can end up with a job queue that looks something like: Upload 1 book Upload 3 books Upload 2 books Upload 1 book etc. The problem is that, as things stand at the moment, Calibre does a "Send metadata to device" step at the end of each one of these jobs, and although uploading a book to a device might only take 2s, the subsequent "Send metadata" step (which, as Kovid kindly explained, sends metadata for all the books on the device) can easily take 30-40s on my Kindle Voyage. This can take literally hours if sending books in a large number of separate jobs. I was thinking, and it occurred to me that perhaps there could be potential for serious optimisation here. Suppose that, when an "Upload Books" job was created and queued, the job queue manager were to scan the current queue and, if it found another "Upload Books" job in the queue, simply added the list of books in the new job to that of the old? That way, the slow "Send metadata" step would only be done once when the combined job ran. Is this a feasible idea? It really could save a huge amount of time for the user if it were feasible to implement.

11-06-2014, 04:38 AM	#3
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383099 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	That's an excellent idea, thank you. Can you tell me how to use "Mark Books", please? I don't seem to see it anywhere. EDIT: Ignore - found it. Last edited by HarryT; 11-06-2014 at 04:43 AM.

11-06-2014, 10:25 AM	#12
PeterT Grand Sorcerer Posts: 13,509 Karma: 78910112 Join Date: Nov 2007 Location: Toronto Device: Libra H2O, Libra Colour	SLightly off topic, but what I've done for ages is to use the Reading List plugin for this purpose. I have one list that is flagged as the default within the plugin that is defined as: Code: Name: Sync Later List Type: Manually add/remove items Device to Sync To: Kobo Glo (main) <-- This is my standard eReader; others exist but only for testing purposes When Syncing: Add new list items to device Sync to this device as soon as it is connected: Checked Clear this list after a sync to this device: Checked The Reading List plugin has also been added to both Context Menu's and both Main Toolbar's. This way, when I see book I want to sync, I can right click on it, select the Reading List option, and then the Add to Sync Later List option. After flagging all books I want ifmy Kobo is not plugged in, plug it in and let the sync happen automagically, or if it IS plugged in, select the Sync Now option.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Possible to queue a Send job until device is attached?	ArchGoodwin	Library Management	2	12-02-2012 03:55 PM
Free (Kindle) How To Find A Job: When There Are No Jobs by Paul Rega	ozron	Deals and Resources (No Self-Promotion or Affiliate Links)	0	08-18-2012 09:05 AM
Jobs Queue, Merging, Metadata, I think that's it....	rabidrobot	Calibre	2	08-17-2010 07:31 PM
Merge feature request (different merge)	Tarran	Calibre	1	05-24-2010 10:57 AM

11-06-2014, 04:35 AM	#2
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's certainly possible, but IMO not worth the effort. Patches are welcome. In the meantime if you find yourself sending large numbers of books in multiple jobs, instead use the mark books tool to mark the books temporarily and only send all the marked books in one go after you are done with all of them.

11-06-2014, 05:08 AM	#4
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	You can also use the excellent Reading List plugin to queue files for later sending to your ereader device.

11-06-2014, 05:16 AM	#5
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383099 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	"Mark Books" does exactly what I needed - thanks. Another Calibre facility I didn't know existed until now.

11-06-2014, 05:32 AM	#6
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383099 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Sometimes - but not always - when I highlight a book in the library and hit "Ctrl+M", the "pushpin" marker doesn't appear alongside the book until I click on a different book. Unfortunately I can't find a consistent way to reproduce the problem: sometimes it works, sometimes not. The book gets marked - it's purely a visual issue. 32-bit Calibre 2.8 on 64-bit Windows 7.

11-06-2014, 05:49 AM	#7
Ripplinger 350 Hoarder Posts: 3,574 Karma: 8281267 Join Date: Dec 2010 Location: Midwest USA Device: Sony PRS-350, Kobo Glo & Glo HD, PW2	I didn't know about the mark books function either. Is there a way to sort books by whether they're marked or not? I can't find a way to do that.

Advert

Advert