Forcing Total Cache & GUI Refresh Without Restarting

[DaltonST]

I would like to change a lot of jobs from requiring a restart due to the large number of custom columns being updated in native sqlite(via apsw) in the background. From a user-friendliness perspective, a restart is annoying. To that end, I have been attempting multiple methods to obviate the need to restart Calibre after a job finishes. So far, none of the methods shown below refresh the entire GUI. Is there a fast and safe way to refresh the entire GUI?

#----------------------------------------------------------------------
def force_refresh_of_cache_and_gui(self, list_of_ids, row_list):

backend = self.gui.library_view.model().db.backend

#test the apsw connection to ensure 'backend' is valid...
backend.execute('SELECT id FROM custom_column_16')
#debug shows an apsw cursor object...so, 'backend' is valid

# method 1
mydbcache = dbcache(backend)
mydbcache.reload_from_db(clear_caches=True)

# method 2
self.gui.library_view.model().refresh_ids(list( list_of_ids ))
self.gui.tags_view.recount()

# method 3
m = self.gui.library_view.model()
m.refresh_rows(row_list)
self.gui.tags_view.recount()


#above completes with no errors or warnings, but gui visible columns do not refresh.
#so, have to restart anyway.

self.gui.quit(restart=True)

#----------------------------------------------------------------------

[aleyx]

If such a method existed, it would be used by the GUI when adding/removing/modifying custom columns, as well as installing/removing plugins. Since all of those require a restart, I fear that no such method exists yet.

Would be really cool, tho. I've been working on some custom columns as well for the last few days, and restarting at every template change adds to a looong time.

On the other hand, it's times like those where you stand in awe before the true AWESOMENESS of SSD TECHNOLOGY.

N.

[chaley]

Originally Posted by aleyx

Would be really cool, tho. I've been working on some custom columns as well for the last few days, and restarting at every template change adds to a looong time.

You can change the template of a custom composite column without restarting. Simply click in any cell in the column and press F2 (or whatever key "edits" the cell). A template editor will open, allowing you to change it however you want.

[aleyx]

...That's a truly excellent tip. Thank you very, very much!

[DaltonST]

I can totally understand and fully support requiring a restart after changing underlying sqlite database structures, such as adding new tables, indexes, views and triggers when a typical custom column is created. It sounds reasonable, though, that a function to merely refresh all custom column data for all books in the GUI without touching any other columns (or anything else) would be quite reasonable.

[chaley]

Originally Posted by DaltonST

I can totally understand and fully support requiring a restart after changing underlying sqlite database structures, such as adding new tables, indexes, views and triggers when a typical custom column is created. It sounds reasonable, though, that a function to merely refresh all custom column data for all books in the GUI without touching any other columns (or anything else) would be quite reasonable.

If all you are doing is changing custom column values then you should be able to refresh the GUI using the same code that bulk metadata edit uses. That code is:

        if changed:
            refresh_books |= dialog.refresh_books
            m = self.gui.library_view.model()
            if gprefs['refresh_book_list_on_bulk_edit']:
                m.refresh(reset=False)
                m.research()
            else:
                m.refresh_ids(refresh_books)
            self.gui.tags_view.recount()
            if self.gui.cover_flow:
                self.gui.cover_flow.dataChanged()
            self.gui.library_view.select_rows(book_ids)

If that code doesn't work then perhaps you are changing things other than column values?

[DaltonST]

Many different custom_column and book_custom_column_link tables are updated in a background job. When the job returns to the callback in ui.py, the methods shown in my original post do nothing visible to the naked eye in the gui. There is nothing being updated within the Calibre gui itself. If there were, it would refresh automatically. The sqlite tables are correct, but the gui has stale data. Hence, my post.

[chaley]

Originally Posted by DaltonST

Many different custom_column and book_custom_column_link tables are updated in a background job. When the job returns to the callback in ui.py, the methods shown in my original post do nothing visible to the naked eye in the gui. There is nothing being updated within the Calibre gui itself. If there were, it would refresh automatically. The sqlite tables are correct, but the gui has stale data. Hence, my post.

And the code I pointed at works fine in bulk edit, which updates anything and everything in the GUI. I can only conclude that you are using the APIs in some way that isn't supported, such as in a non-GUI thread.

Best of luck ...

[DaltonST]

My jobs don't use any Calibre api's. Pure SQLite and Python. Which is why a restart is required. However, after the callback to ui.py, one would think that a function might exist to refresh the gui without doing any foreground data updates first.

[chaley]

Originally Posted by DaltonST

Which is why a restart is required.

Are you directly updating the database without using the calibre DB APIs? If so, then as far as I know there is no hope, because calibre believes that its in-memory structures are correct. I can't imagine this changing, but that is up to Kovid.

You might be able to force a refresh by changing libraries to something else then changing back, but I wouldn't hold my breath.

[DaltonST]

Hence the required restart. And thus my original post. To quote it: "I would like to change a lot of jobs from requiring a restart due to the large number of custom columns being updated in native sqlite(via apsw) in the background."

[chaley]

And to quote my post, "best of luck".

[DaltonST]

I don't need luck. I have 3 plugins in production that do exactly the same thing. I use custom sqlite objects (tables and views) that Calibre knows nothing about. In fact, my Derive Genres plugin uses its own derivegenres.db sqlite database for most of its work. The job connects to that db (not Calibre), and later attaches to metadata.db, reads the data it needs, and immediately detaches from metadata.db. It treats metadata.db as "read-only". Later, it does send a dict back to its callback in ui.py for a "normal" gui-style update of the genres that were derived using the standard set_user_metadata function.

So, it appears that a restart is still, and will always continue to be, required. C'est la vie.

[DaltonST]

I have answered my own question posed in the original post, and have documented it here. Be careful.

from calibre.db.cache import Cache as dbcache
#------------------------------------------

def force_refresh_of_cache(self, books_to_refresh):

# Use with *great discretion* to avoid bad things happening.

# Do this only when an annoying restart would otherwise be indicated and required
# due to the update of custom column metadata done *outside* of the Calibre API, i.e.,
# db = self.gui.current_db.new_api #see: http://manual.calibre-ebook.com/db_api.html

# NO jobs or other background tasks should be running when you execute this.

# 'books_to_refresh' is a simple list of integers (i.e., books), and is not a list of dicts (e.g., {'calibre_id': 12345} )
# To get a set of all books in the library (required as a parameter to this function):
# db = self.gui.current_db.new_api #see: http://manual.calibre-ebook.com/db_api.html
# books_to_refresh_frozenset = db.all_book_ids()
# Then, you must create books_to_refresh as a simple list from the frozenset and pass it as the only parameter to here.
# After this executes, the next time the user clicks on the gui, it will show the refreshed metadata.

backend = self.gui.library_view.model().db.backend
mydbcache = dbcache(self.gui.library_view.model().db.backend)
mydbcache.init() #refreshes the cache from the physical metadata.db. refer to: >src>calibre>db>cache.py
self.gui.library_view.model().refresh_ids(list(boo ks_to_refresh)) #refreshes the gui from the cache
# class EditMetadataAction in src>calibre>gui2>actions>edit_metadata.py
self.gui.tags_view.recount() #refreshes the tag browser
# class TagsView in src>calibre>gui2>tag_browser>view.py


#-------------------

Best of luck.