Scheduling jobs and waiting for other activities

[meme]

kiwidude has kindly updated his Book Sync plugin to call my Kindle Collections plugin in order to streamline users adding books to their Kindle.

But I'm having an issue in that my Kindle Collections plugin reads the data from the Kindle before the Book Sync plugin has synced/sent all the books to the device. As a result, books just synced don't get put into collections because they aren't on the Kindle at the time the plugin is called.

I'm looking for the best was to deal with this.

One option is to change where in Book Sync the Kindle Collections plugin is called. But since I believe Book Sync just calls Send Devices code at some point, its the Send Devices that I'm really dependent on.

Another option suggested by kiwidude is to use jobs. However, when I set up a test of this, I can get the job added to the queue - but the job runs immediately - before all other jobs are complete (I'm testing with a long running count pages).

Is there a way to schedule a job but have it put at the end of the queue to only run when all other jobs are finished? Or a way to check the job queue to see if anything is on the queue before starting my code? Or a way to have the Book Sync code call the Kindle Collections plugin only after all its related jobs are complete?

I already deal with device connected and books on device available to read thanks to the updates that were made in calibre, but it looks like there might be another dependency.

[kovidgoyal]

There is no way to add dependencies to scheduled jobs. The only way to implement this that I can see is to add code to the relevant send to device methods to support a callback on completion and then have book sync set that callback and schedule the collections update job in the callback. Let me know what method in send to device you need modified and I'll modify it.

[kiwidude]

@Kovid - sync_to_device() is the function that BookSync calls.

However as I have said to meme previously I would have thought that this would only solve one of the problem cases. Because forgetting the Book Sync plugin for a moment, imagine the common scenario of a user connecting their Kindle, selecting a bunch of books and then choosing Send to Device. If that user were to click on meme's plugin before that Send job has completed, we have exactly the same issue as with Book Sync. However in that scenario, there appears no way for meme's plugin to be "called back".

Hence why I suggested that a more generic solution to the problem might need to be found if possible.

[meme]

Is there any way to tell if a sync to device is running?

Or a way to setup a callback both when sync_to_device starts and finishes - then I could setup a callback on both to set/unset a flag similar to what I do when checking if all book information has been read from the Kindle. Although if sync is called a second time without finishing it gets a bit messy when one finishes and the other doesn't.

[kovidgoyal]

You can easily query gui.jobs_manager to see if any jobs taht communicate with the device are running. For an example of doing that see the shutdown() method in gui2.ui

[meme]

Ok, thanks for that. I've put in a simple check:

    while parent.gui.job_manager.has_device_jobs():
        debug_print('Device is busy, sleeping')
        time.sleep(3)

If nothing is running, and I run Create, it shows its not busy, which is what I expect.

If I kick off Book Sync and separately run Create I get the output that the device is busy - but it stays busy 'forever' even after the Book Sync should be done.

Any idea why this is?

If I use Book Sync to start off the create, it looks like the Create runs before the sub-process for sending books even starts so it still runs Create before the books are on the device Maybe I'll have to try a delay in starting - but only for the function called by Book Sync and not for the interactive one.

And its clear I'll need to convert over to using jobs since the interface freezes while the plugin sleeps.

[kovidgoyal]

The function is only called once if you put it as the test condition for a while loop. Also pass also_queued to has_device_jobs so that you can wait for waiting device jobs as well (I just committed that).

[meme]

Ah, that explains the infinite. (I should add loop there, but it looks more zen this way )

But now I've got another oddity that I'll have to test a bit more.

So I change my loop to a while true with a break, and found that it still stayed forever busy.

So I changed it to a countdown of 6 times to continue even if busy after 6 iterations, and it now runs - but it seems like it runs one part of the upload but not the final part:

2011/06/06 20:52:56  Device is busy, sleeping
DEBUG:   22.0 USBMS: uploading 6 books
2011/06/06 20:52:59  Device is busy, sleeping
DEBUG:   26.8 USBMS: finished uploading 6 books
2011/06/06 20:53:02  Device is busy, sleeping
2011/06/06 20:53:05  Device is busy, sleeping
2011/06/06 20:53:08  Device is not busy, continuing  (MAX OF 6 ITERATIONS REACHED)

LOTS OF KINDLE COLLECTION OUTPUT

Job: 3 Upload 6 books to device:Sense and Sensibility, Persuasion, Northanger Abbey, Mansfield Park, Emma, Pride and Prejudice finished
No details available.
DEBUG:   37.4 USBMS: adding metadata for 6 books
DEBUG:   37.5 USBMS: finished adding metadata
Job: 3 Upload 6 books to device:Sense and Sensibility, Persuasion, Northanger Abbey, Mansfield Park, Emma, Pride and Prejudice finished
No details available.
DEBUG:   38.8 USBMS: starting sync_booklists
DEBUG:   39.7 USBMS: finished sync_booklists
Job: 4 Send metadata to device finished
No details available.

It appears to upload the books ok, but then device stays busy forever and never uploads the metadata (I can see 1 Job sitting on the queue which must be the metadata upload, but of course the gui is frozen). If I use a max of 6 iterations as shown above its clear the metadata upload is done after the kindle collection code runs.

Its as if running the kindle collections is keeping the device busy (even though I'm not using jobs yet, and I haven't starting reading the book info from the kindle yet, although I have got the device id and root path from calibre).

[chaley]

Originally Posted by kovidgoyal

The function is only called once if you put it as the test condition for a while loop. Also pass also_queued to has_device_jobs so that you can wait for waiting device jobs as well (I just committed that).

If that is true, then python is seriously broken. There is no scenario where the value of a non-inlined function that is used in a loop condition should be cached or optimized away.

To test whether python value caching/loop optimization is broken, I ran:
--
x = lambda y: 0 if y > 10 else y + 1
z = 0
while x(z):
z += 1
print 'here'
---
and got 11 'here's, exactly what is expected.

The GIL shouldn't be an issue because of the sleep.

Is this (incorrect) python behavior documented somewhere?

[kovidgoyal]

I could well be wrong It just seemed the most likely explanation.

@meme: Look at the jobs list in the GUI to see what is going on. Stick a print statement in job_done in the DeviceJob class to see if it is being called.

[meme]

Well, something has changed in the calibre code

If I modify my plugin to include a check for device busy that exits after 15 iterations, I get different behaviour with 0.8.2 and the current source.

Using 0.8.2 if I run Book Sync's sync now (without auto creating collections) and then Run Kindle Collections Create as soon as I can click it, things stay busy until the maximum iterations are reached.

Using current calibre source if I run sync now and then run Create as soon as I can, the busy signal is there for a few iterations and then it continues ok. No change in my code/plugin between runs. And its seemed, subjectively that I could click on Kindle Collection's menu just a bit quicker with the new source, but that could just be me.

So lets ignore the forever busy aside, and get back to the main event.

I've added the print statements to the DeviceJob class for start/end. And if I run Book Sync with automatic collections what I see is that Create starts running before the Job is even queued so it never sees the device as busy and just keeps running.

I could add a delay to Create - but what would it be 5 seconds, 10? I think maybe the original suggestion of a callback that Book Sync uses to only call Kindle Collections Create when the sync is really done is the best approach.

I know that if users use send to device for a lot of books and run create too soon they'll miss books. But that seems reasonable since they started it they should monitor it. But if code or another plugin calls create, it should be done its updates before it makes the call. That doesn't mean to say I shouldn't look into changing the plugin to use jobs and check for queued device usage, but even with that it still wouldn't solve the Book Sync integration.

[kovidgoyal]

I'm a little confused. I am making the following assumptions about your code:

1) You have implemented update collections as a DeviceJob i.e. an object of class DeviceJob

2) The BookSync code is also implemented as a DeviceJob.

3) The BookSync auto collections code first schedules a sync and then schedules a collections update job

In this case calibre will first run the sync and only after it is completed will it run the collections job.

[kiwidude]

@Kovid - not sure whether this is relevant but to be clear here - Book Sync just calls "self.gui.sync_to_device()". It doesn't do this call from a job itself, it lets sync_to_device() create any jobs as required.

[meme]

This test was using Create without running it as a job.

I was just going to update my branch that has the job code, but I don't know if it would make any difference. If I put my check for the queue before I kick off the job, then its in the same situation as the above code - it will run before the sync device gets going. If I put my check for the queue within the job, well then it would be using the device itself and things get a bit more confusing. But I could be wrong...

Oh, and in my job code I use this call, not a subclass of DeviceJob directly (mostly because I just copied from another plugin...)

        job = self.gui.job_manager.run_job(
                self.Dispatcher(self._create_collections_completed), func, args=args,
                    description=desc)

[kovidgoyal]

@kiwidude: That's fine, sync_to_device creates and queues the job for you.

@meme: You have to create a DeviceJob, not a normal job. If you do that, your problem should automatically be taken care of as calibre will run your job after any previously queued device jobs have completed. The interface is self.gui.device_manager.create_job.