Scheduling jobs and waiting for other activities

[kovidgoyal]

Hmm perhaps one way is to allow passing of a previous job parameter to the DeviceJob class. Then the function that launches a job chain should schedule all n jobs at once so that they are atomic. The nth job should check if the n-1th job finished successfully, if so, it should run, if not it should abort.

Sounds reasonable?

[chaley]

Yes, but I think the complexity of knowing the entire chain in advance would lead to bugs. In addition, it doesn't deal well with job forks (alternate jobs later in the chain), although I don't think we currently have any of those.

Something that would work and be easier to live with (I think) would be to add a second queue to the job manager, a 'job chain' queue, and another parameter to create_job indicating that this job is part of a chain. The job dispatcher would take jobs from that queue until empty, then take jobs from the normal queue. FD semantics ensure that the next job is queued before the first job terminates. Thus, if one uses FD around done and sets the job_chain parameter, then the chain will be atomic.

[kovidgoyal]

Works for me, though what happens if there need to be two sets of atomic jobs? How would a later job in the chain know which chain to pass to create job?

[meme]

As a note on my plugin - I don't actually create any other jobs in my plugin once my create_job starts. I used to accidentally call the get_device information job (to get device uuid) but realized that this messed up my percentage done indicator, and I could just get the values directly from calibre since get_devices had already run.

Out of curiousity, I'm currently calling the get_device_information immediately after my create job, but should I be calling it in my completed job function - since I don't need it to run until after my job completes anyway? I couldn't get this to work at first and so just moved it up to the main function where it worked and I left it. (I just use the completed function to present a dialog to the user to show them the report and nothing depends on them closing the dialog).

[chaley]

Originally Posted by kovidgoyal

Works for me, though what happens if there need to be two sets of atomic jobs? How would a later job in the chain know which chain to pass to create job?

The currently running job wins. The second job won't get started until the first job either finishes the chain, schedules a subsequent job in a Dispatcher, or schedules a subsequent job without the chain flag set.

[chaley]

Originally Posted by meme

Out of curiousity, I'm currently calling the get_device_information immediately after my create job, but should I be calling it in my completed job function - since I don't need it to run until after my job completes anyway? I couldn't get this to work at first and so just moved it up to the main function where it worked and I left it. (I just use the completed function to present a dialog to the user to show them the report and nothing depends on them closing the dialog).

Since your job doesn't spawn further jobs, then there isn't any difference as far as you are concerned. The job will run after yours. However, any jobs spawned by kiwidude's job will probably run after both of these, which is probably wrong and is the reason for the discussion that Kovid & I are having.

[kovidgoyal]

Originally Posted by chaley

The currently running job wins. The second job won't get started until the first job either finishes the chain, schedules a subsequent job in a Dispatcher, or schedules a subsequent job without the chain flag set.

OK by me, I have higher priority stuff to work on at the moment, so I'll leave this alone. When you return, will you implement it, since it is your idea?

[chaley]

OK. Won't be for this week's release though.

[kovidgoyal]

Originally Posted by chaley

OK. Won't be for this week's release though.

No rush.

[chaley]

The create job subsequence changes are in source.

These changes ensure that under certain conditions, follow-on jobs scheduled by a running job run before jobs scheduled in other ways. For example, two-step jobs like upload_books will run both steps before another job can be run.

The conditions are:
- The sub-job must be created while the creating job is running. In practice, this means that the sub-job must be created in the done function of the creating job.
- The done function of the sub-job must be encapsulated with FunctionDispatcher().

All the standard jobs in device.py respect these conditions.

If anyone has uses job sequences and wants to use this feature, call create_job_step passing in the job value. This is the first parameter passed to a done function. See books_uploaded, the done function for upload_books, for an example of scheduling upload_booklists as a sub-job.

@meme: could you try two things? 1) See if it still works with the code you have, and whether the job sequencing changes (according to the log). 2) Switch back to device_manager.books and check if it still works.

My strong suspicion is that both 1 and 2 will work. You can change it back to get_device_info for your production version.

[meme]

I'll give it a test tomorrow.

[meme]

I'm not sure of the results at this point - will need more time to test. On the surface the collections seem to be getting created ok in the current release and the new source, but the job output seems mixed.

It might help if the Job printed a 'job started' as well as a 'job finished' line in the debug output (I can add one for me, but it might be helpful in general.


This output is from starting calibre with device collected and already having some new books in the Reading List ready to sync.

0.8.6:

Job: 1 Get device information finished
Job: 2 Get list of books on device finished
Job:[output messed up, interleaved with other text]
Job: 5 Get device information finished
Job: 4 Create Kindle Collections finished
Job: 6 Send metadata to device finished
Job: 7 Get list of books on device finished

Dowloaded Source:

Job: 1 Get device information finished
Job: 2 Get list of books on device finished
Job: 3 Upload 5 books to device:Life, the Universe and Everything, So Long, and Thanks For All the Fish, Mostly Harmless, The Hitchhiker’s_Guide_to the Galaxy, The_Restaurant_at the End of the Universe finished
Job: 6 Send metadata to device finished
Job: 6 Send metadata to device finished
Job: 4 Create Kindle Collections finished
Job: 5 Get device information finished
Job: 7 Get list of books on device finished

[kovidgoyal]

I committed code to print out job started notifications.

[meme]

Thanks.

First test

Job: 1 Get device information started
Job: 2 Get list of books on device started
Job: 1 Get device information finished
Job: 3 Upload 5 books to device:Life, the Universe and Everything, So Long, and Thanks For All the Fish, Mostly Harmless, The Hitchhiker’s_Guide_to the Galaxy, The_Restaurant_at the End of the Universe started
Job: 2 Get list of books on device finished
Job: 6 Send metadata to device started
Job: 3 Upload 5 books to device:Life, the Universe and Everything, So Long, and Thanks For All the Fish, Mostly Harmless, The Hitchhiker’s_Guide_to the Galaxy, The_Restaurant_at the End of the Universe finished
Job: 4 Create Kindle Collections started
Job: 6 Send metadata to device finished
Job: 5 Get device information started
Job: 7 Get list of books on device started
Job: 4 Create Kindle Collections finished
Job: 5 Get device information finished
Job: 7 Get list of books on device finished

Does started actually mean running or just queued?

It seems odd to see the get list of books (job 2) finish before Reading List (job 3) starts uploading books. I would think it would need to wait until the list is fully gotten in order to know what's on the device.

I'll see if I can figure out how to revert my copy of the source to 0.8.6 and then add in the job started print that you had to see if there's really any difference. Since I didn't study the original output, just the effect, I can't tell if what I'm seeing is an issue or has always been that way.

[kovidgoyal]

started means that it has actually started. The Finished print out may not be very accurate however, as it is run in another thread. I've committed code to print out exactly when it finishes and the callback is called.