[GUI Plugin] Kobo Utilities

[Terisa de morgan]

Originally Posted by ownedbycats

Suggestion: I wonder if these would be within scope to implement in Kobo Utilities:



Both of them can be done by editiing the SQLite database directly, but manually gets tiresome. If it's technically feasible, perhaps a KU menu option for selected books to run a SQLite query to overwrite, which is run prior to eject.

I can tell you that run a sqlite query to overwrite the database is totally feasible, as I have exactly that, an action chain which run the query (as an option for Kobo Utilities) and Smart Eject after that. The issue is that I think it could be dangerous if you're not really, really aware of what you're doing. I know too that is only in my version because davidfor didn't want to include something like that (I requested it).

[Majutsushi]

Originally Posted by ownedbycats

Suggestion: I wonder if these would be within scope to implement in Kobo Utilities:

Both of them can be done by editiing the SQLite database directly, but manually gets tiresome. If it's technically feasible, perhaps a KU menu option for selected books to run a SQLite query to overwrite, which is run prior to eject.

That's an interesting idea, although I'm not sure how it could be implemented without either being too specific or effectively reinventing plugboards. Or do you mean actually allowing you to run an arbitrary SQL query? That would certainly be possible but also very dangerous like Terisa said. It would also have to be run in a similar manual way to what Terisa described since I don't think there's a way to hook into an eject event.

I'm curious how it's implemented in the version that Terisa uses, particularly how you can run predefined queries in an automated fashion. I can create an issue for it, but I can't promise anything especially since I'm currently in the middle of some big internal refactorings.

[Terisa de morgan]

Originally Posted by Majutsushi

That's an interesting idea, although I'm not sure how it could be implemented without either being too specific or effectively reinventing plugboards. Or do you mean actually allowing you to run an arbitrary SQL query? That would certainly be possible but also very dangerous like Terisa said. It would also have to be run in a similar manual way to what Terisa described since I don't think there's a way to hook into an eject event.

I'm curious how it's implemented in the version that Terisa uses, particularly how you can run predefined queries in an automated fashion. I can create an issue for it, but I can't promise anything especially since I'm currently in the middle of some big internal refactorings.

In my case, as it's a specific query, they query is built inside the code and there is a menu option to run it. So, it uses all the plugin mechanism to run it. It's not a "generic query builder" to run it, that I'm not sure how to implement it and, for sure, I wouldn't include as it's going to give more problems than adventages (that's my POV, of course).

[Majutsushi]

I just had an idea: since you can write custom modules for the ActionChains plugin in Python, it should be possible to write a module that calls into the KoboUtilities plugin so that it gives you a database connection, and then the module can use that connection to run any query it wants. Currently the relevant method would be "calibre_plugins.koboutilities.action.KoboUtilitie sAction.device_database_connection()". You need the running instance of that action though, and I'm not sure how to get access to that from a module. That's a question for people who are more familiar with that plugin.

Note that the above method will probably move and/or change a bit as part of my current refactoring, but there should still be a way to access it once I'm done.

[chaley]

Originally Posted by Majutsushi

That's an interesting idea, although I'm not sure how it could be implemented without either being too specific or effectively reinventing plugboards. Or do you mean actually allowing you to run an arbitrary SQL query? That would certainly be possible but also very dangerous like Terisa said. It would also have to be run in a similar manual way to what Terisa described since I don't think there's a way to hook into an eject event.

Originally Posted by Majutsushi

I just had an idea: since you can write custom modules for the ActionChains plugin in Python, it should be possible to write a module that calls into the KoboUtilities plugin so that it gives you a database connection, and then the module can use that connection to run any query it wants. Currently the relevant method would be "calibre_plugins.koboutilities.action.KoboUtilitie sAction.device_database_connection()". You need the running instance of that action though, and I'm not sure how to get access to that from a module. That's a question for people who are more familiar with that plugin.

It seems to me that the "right" way to do this is to provide a field in KU to set the text form of the series index. That field would be a combination checkbox and template, just as is used in the kobo driver for "collections columns", "collections template", etc. The user would check the box and supply a template that returns the desired string for the series index, which KU would write to the device database. You would need an open database connection to pass to the template processor. Yes, this is specific to series, but that makes it understandable and safe.

I strongly advise you not to pass around database connections. There are all sorts of ways where these can lead to disaster, for example when threading. It would also permit a user to rather easily trash the database.

[DNSB]

A couple of issues that I see with the suggested change. One being that I have quite a few series indexes such as 1.25, 1.33 or 1.50 where there is one or more novella/short story/whatevers that are part of the series but is not part of the mainline. Another potential issue is that there are more than one location for the SeriesNumber, there is a SeriesNumber field that is a text field but there is also a SeriesNumberFloat field that is defined as Real. Another potential issue being that when I import a book from my Kobo ereaders to calibre, the series index code would be need to be able to handle either case and convert back to the x.yy format unless Kovid is willing to change the series index code in calibre to handle either case. I suspect that the majority of the changes are going to need to be implemented in the KoboTouch driver.

I do have a couple of items that are done by directly accessing the Kobo database but nothing that I would consider as being safe enough for general distribution. They are definitely backup the database, perform the action and check that the database is still valid before continuing. At least twice, they were broken by updates to the Kobo database.

[Majutsushi]

Originally Posted by chaley

It seems to me that the "right" way to do this is to provide a field in KU to set the text form of the series index. That field would be a combination checkbox and template, just as is used in the kobo driver for "collections columns", "collections template", etc. The user would check the box and supply a template that returns the desired string for the series index, which KU would write to the device database. You would need an open database connection to pass to the template processor. Yes, this is specific to series, but that makes it understandable and safe.

This is the kind of change that felt to me would be almost reimplementing plugboards, just with a different output type. I have no idea how the plugboards are implemented or whether the drivers have any influence over them, so I don't know how difficult it would be to make them work in the desired Kobo-specific way, or whether this kind of separate implementation would really be the only way.

I strongly advise you not to pass around database connections. There are all sorts of ways where these can lead to disaster, for example when threading. It would also permit a user to rather easily trash the database.

My idea wasn't really about "passing around" database connections as such, at least not in any new way. The method I mentioned already exists for internal use by the plugin, I was just pointing it out. Any use of it by something else would certainly be unsupported and at your own risk, but I would think that someone who knows how to write an ActionChains module would also know how to be careful with database modifications.

As for safety of the connection, if you use the returned connection object as a context manager then it takes a lock and creates a transaction, so it should be safe from that standpoint. I actually suggested the lock feature to Kovid because it was necessary for safely handling the case in Calibre 8 where the database might have been copied to a different location due to filesystem limitations.

So this idea wasn't about creating some official API, just something that people who know what they're doing might use for things that aren't available in the plugin.

[Terisa de morgan]

Originally Posted by Majutsushi

So this idea wasn't about creating some official API, just something that people who know what they're doing might use for things that aren't available in the plugin.

The part about "who know what they are doing" when you're creating something that everybody can use is complicated. Even creating an action chain means that you know how to handle some aspects that you can copy/paste, but have no idea of what you're really doing (I say it through my own experience).

[compurandom]

Originally Posted by Majutsushi

This is the kind of change that felt to me would be almost reimplementing plugboards,

Agree. This would be really complicated and possibly dangerous to implement with a database edit, but would be less than a line to do just by filling in the right stuff in the existing plugboard.

[ownedbycats]

I will open a ticket and see what Kovid thinks.

EDIT: https://bugs.launchpad.net/calibre/+bug/2112564

EDIT: wontfix.

[DNSB]

And again, what effect would this change have on the SeriesNumberFloat field. Both fields are used internally by Kobo's firmware.

[Quoth]

Originally Posted by Majutsushi

I just had an idea: since you can write custom modules for the ActionChains plugin in Python, it should be possible to write a module that calls into the KoboUtilities plugin so that it gives you a database connection, and then the module can use that connection to run any query it wants. Currently the relevant method would be "calibre_plugins.koboutilities.action.KoboUtilitie sAction.device_database_connection()". You need the running instance of that action though, and I'm not sure how to get access to that from a module. That's a question for people who are more familiar with that plugin.

Note that the above method will probably move and/or change a bit as part of my current refactoring, but there should still be a way to access it once I'm done.

This sounds like a bad idea. Think about it for a month or so.

[chaley]

Originally Posted by DNSB

And again, what effect would this change have on the SeriesNumberFloat field. Both fields are used internally by Kobo's firmware.

I'm curious; how do you know this? Is the firmware source available? What does the kobo use the float value for? My guess is sorting.

I'm tempted to add setting the text index using a template to the driver just to see what happens. I think the driver doesn't read/use that field (not near my computer so can't check). The change would be easier than it was to add collection management via template.

[ownedbycats]

Originally Posted by chaley

I'm curious; how do you know this? Is the firmware source available? What does the kobo use the float value for? My guess is sorting.

I'm not sure it does anything. On my few tests modifying the sqlite database, I think it read just SeriesNumber.

[DNSB]

Originally Posted by chaley

I'm curious; how do you know this? Is the firmware source available? What does the kobo use the float value for? My guess is sorting.

I'm tempted to add setting the text index using a template to the driver just to see what happens. I think the driver doesn't read/use that field (not near my computer so can't check). The change would be easier than it was to add collection management via template.

From a note from a while back when I asked @davidfor why the two fields, his response was that, as far as he knew, one was used for display purposes and the other was used for sorting when sorting by series. I'm not sure about the next part of what I jotted down which seems to say the text field is converted and placed in the real field by the Kobo firmware. If this is correct, it might make a bit of a mess of converting to numeric for say, 1 - 4. I never got around to testing this.