Scheduling jobs and waiting for other activities

[meme]

Duh. Its so obvious when you know what it is

Of course, the issue was run_job is running ParallelJob, while create_job for the device is running in order. It clearly answers my original question of how to get it added to the queue at the end, even if we went round the houses for a while. Not that I would have found it myself.

So now book sync runs upload books, create_collections is queued behind it, and when book sync finishes it queues the send metadata behind my create collections (which isn't an issue as I read from the books not the cache file).

Now I just have to re-arrange my dialogs so they are separate from the device job (temporarily removed and just using prints for this test), and work out how to get he progress indicator to say 0% not 100% done until its finished (may not try to get it to show progress as it runs as there's no clear way to count, but who knows.), and see if I should make some other parts into jobs.

Thanks again.

[meme]

Hmmm. I seem to be back at the beginning again...

Create Collections is still running before the books are recognized as being on the device - even though they have been sent to the device!

I have modified the code to use jobs, and I've used the latest 0.8.5 code. I clearly see the Book sync job get kicked off, then the create job, and at some point the send metadata job. The send to device definitely finished before the create runs, but the books are not seen on the device by create when I run a query to check

Job starting  Get device information
Job starting  Get list of books on device
Job starting  Upload 4 books to device:Life, the Universe and Everything, So Long, and Thanks For All the Fish, The Hitchhiker’s_Guide_to the Galaxy, The_Restaurant_at the End of the Universe
Job starting  Create Kindle Collections
Job starting  Send metadata to device

I've added a debug line in my calibre code to indicate if a book is seen as on device or not on the device. I use this code to check if on device:

query = 'ondevice:True'
        self.ids_on_device = self.db.search_getting_ids(query, None)
        query = ''
        self.all_ids = self.db.search_getting_ids(query, None)
        debug_print('%d ids, %d on device' % (len(self.all_ids), len(self.ids_on_device)))

Execution shows that if I mark a new book for syncing, the book gets sent to the device but when Create runs it sees the book as not on the device. When I run create again just after, its seen as on the device.

Is the db updated with on device status at the end of send to device, during send metadata, or in a parallel job kicked off from send to device? Is there a way to make sure the ondevice info is correct before finishing send to device?

I've tried changing the Send to Device manual/on send/automatic, but all that did on automatic is kick off an extra send metadata before book sync. Oh, and I found I had a call to get device info in my code that was setting my percent job done line to 100% - I changed it to just use the current values for the device since calibre already called the get device info routine.

Oh look, Ubuntu 11.04 hasn't crashed in a few hours - guess that proposed change for the X server might have worked (hope I haven't jinxed it )

[kovidgoyal]

I dont recall the exact sequence of when the ondevice column is updated, that bit is Charles' bailiwick. Hopefully he'll pop in with some useful suggestions.

A generic solution to this might be to have a "job registry". Plugins can register the job there and all registered jobs are run one the on_device column is updated. Or something like that, lets wait for Charles.

[meme]

Ok, we'll wait for Charles. I did some poking around looking for where the device info is update, but haven't unravelled it yet. Either the sync could do the update before returning, or I need to call a refresh of the info before I start - though it might be that the data that's used to indicate if its on device (e.g. metadata.calibre) isn't written out yet. In which case calling the send metadata first might do, though it duplicates the call.

[chaley]

Sorry, but I haven't been following this thread. I am up to &% with an opportunistic contract in Java of all things.

If you are looking at the values in db2's cache, then it is updated just before the device_signals.device_metadata_available.emit(), which may happen while a sync_booklist is still running. If you are looking at the values in the device booklists, then it happens before the books models set_database calls, which happens before the library_view model refresh and a fair ways before device_signals.device_metadata_available.emit().

If you are looking at metadata.calibre, then there isn't an indication. The sync_booklists job that does that work is kicked off before the device_signals.device_metadata_available.emit(), but there is no guarantee that it has finished. Also note that this last job is run only if metadata management is set to automatic.

EDIT: If you are looking for when the values are correct after uploading books, then it is when the termination function books_uploaded is called. This happens when the job finishes. There are no notification signals. Note that a sync_booklists is kicked off when the books finish uploading, and will still be running when books_uploaded returns.

Did I answer the question?

[meme]

Possibly, but it might take a bit to figure out if you did

kiwidude's plugin calls sync_to_device and then calls my plugin, which queues right behind it as a device job. when my plugin starts it runs

query = 'ondevice:True'
self.ids_on_device = self.db.search_getting_ids(query, None)

to get the ids. The problem is that the search_getting_ids is not returning the ids of the new books that were just sent to the device. I need to either have calibre modified to ensure the db's view of what is on the device is updated before sync_to_device returns, or I need a function to call from my plugin to refresh the data before I continue. Something is running after I get the ids that then updates them with more recent information (since it works if I immediately manually run a create after).

[chaley]

@meme: This is complicated. From what I see, in all DeviceJobs a call to the 'done' function is scheduled (signal emitted on a queued connection), then the job terminates. This allows the next job to start, without necessarily having run the done function. Because of this, your plugin gets called before the database is updated. The done function is queued behind you, assuming that you are running on the GUI thread.

@kovid: I think that FunctionDispatcher should be used instead of Dispatcher. That way the done function will be finished before the job terminates.

[kovidgoyal]

Originally Posted by chaley

@kovid: I think that FunctionDispatcher should be used instead of Dispatcher. That way the done function will be finished before the job terminates.

OK by me, I cant think of anything that would break by having job done called before the next job is started.

[chaley]

Originally Posted by kovidgoyal

OK by me, I cant think of anything that would break by having job done called before the next job is started.

Changes pushed. I tested them with the folder device and with my Sony. Nothing broke. I don't know if the changes help meme's problem -- we will see.

[kovidgoyal]

I'm a little concerned about deadlocks, I don't see any obvious ones since all the methods to run jobs on devices simply put things in queues and then return, but I'm just throwing this out there in case you haven't considered the possibility.

[kovidgoyal]

@meme: I've merged the code, let us know if it helps.

[chaley]

I did think about it, but couldn't find a case where it would happen. To deadlock, the done function would need to wait for the result of another job, and I didn't find a case of that.

One the other hand, I did find several cases where the done function does things to the device, such as messing with the booklists and then starting a sync_booklists job. These changes happen on the GUI thread while possibly another job is running. I don't think that simultaneous access to the booklists is a good thing, and this change will prevent that.

[kovidgoyal]

I agree, having the device thread wait on the done function is a cleaner architecture.

[meme]

Sorry - out enjoying the one day of sunshine in a while

I'll give this a try.

I just gave it a quick run, but its not returning from getting device information when plugging in the Kindle. This may be my copy of the code so I'm going to remove and redownload the current source to make sure I haven't edited something while poking around.

[meme]

I've cleared my source tree and re-downloaded - same issue.

If I start without the Kindle connected, and then I connect the Kindle:

I see the notification pop up: "Device: Amazon Kindle detected" (Ubuntu 11.04)

In the job scheduler I see:
Job= Get device information, Status= Get device information, Progress= 100%

In debug I see
"Job: 1 Get device information finished"
"No details available"

And calibre just hangs and requires a ctrl-\ to stop, not just a ctrl-c.

Re-running with 0.8.5 works ok and the job continues on to get books from device.

If I can insert any debug statements, just let me know where.