[GUI Plugin] Entities Manager

[DaltonST]

[GUI Plugin] Entities Manager

Summary:

• 'Entities Manager' allows Links and Notes to be created for any 'Entity', and to 'Associate' those Entities with other Entities without reference to any book.
• An 'Entity' is the name of a Calibre Standard or Custom Column plus a value of that Column. The name is its 'Entity Type'.
• Standard Columns: Authors, Series, Publisher, Tags and Title are valid 'Entity Types'.
• Custom Columns: All are valid as 'Entity Types' as long as they are textual and not composite/built-from-others.
• 'User Defined' Entity Types may be created, the values for which are manually assigned.
• Associations themselves may be created as a valid Entity, inheriting all of the functionality normally available to all Entities.
• Any existing value in any valid Entity Type may be used to create a new Entity.
• An Entity may be attached to/linked with/associated with any other Entity or Entities.
• Entity 'Column Icon Rules' may be automatically created, maintained, and removed.
• Entity 'Column Color Rules' may be automatically created, maintained, and removed.
• Refer to the attached images plus the plug-in's ToolTips for more information and definitions.

Documentation:

• All of the ToolTips in EM comprise its "User Guide", including Menu ToolTips. ToolTips are required reading.
• Refer to the attached images which describe and explain its uses.

Requires Calibre Version: 6.0.0+


Version History:

Spoiler:

Version 1.0.19 - 2023-01-17 Qt.core.
Version 1.0.18 - 2022-10-08 Qt6 QUrl.ParsingMode.
Version 1.0.17 - 2022-04-14 Qt6 Compatibility. Minimum Calibre Version 6.0.0
Version 1.0.15 - 2020-12-21 Miscellany.
Version 1.0.14 - 2020-11-27 Miscellany.
Version 1.0.13 - 2020-07-21 Qt tweaks.
Version 1.0.12 - 2019-12-29 Technical changes after Python 3.8 testing with Calibre 4.99.2
Version 1.0.11 - 2019-09-13 Calibre 4 compatibility with backwards compatibility to Calibre 3 due to Qt changes.
Version 1.0.10 - 2019-04-25 Python 3 Compatibility. Minimum Calibre version now 3.41.3. Main EM dialog: scroll-bars added around the entire window to accommodate small screens.
Version 1.0.9 - 2019-02-06 Enhancement to 'Quick Execute Links' when multiple Links exist for a single Entity.
Version 1.0.8 - 2019-01-21 Miscellaneous.
Version 1.0.7 - 2019-01-15 Enhancements.
Version 1.0.6 - 2019-01-14 New functionality available on the Main Menu with "binoculars" icons.
Version 1.0.5 - 2019-01-13 Entity 'Column Color' Rules. Enhancements to Entity 'Column Icon' Rules.
Version 1.0.4 - 2019-01-08 Entity Column Icon Rules automatic creation. '@Entity Types' converted to '~Entity Types'.
Version 1.0.3 - 2019-01-04 Miscellaneous.
Version 1.0.2 - 2019-01-03 Arbitrary, user-defined "@" Entity Types. @Association Entity Type special functionality
Version 1.0.1 - 2019-01-02 'Title' added as a valid 'Entity Type'.
Version 1.0.0 - 2019-01-01 Initial Version

[DaltonST]

For future use.

[BetterRed]

Trying to work out how to associate related works, e.g

1. Marcel Proust - Remembrance of Things Past
2. Marcel Proust - À la recherche du temps perdu
3. Terence Kilmartin - A Reader's Guide to Remembrance of Things Past
4. Patrick Alexander - Marcel Proust's Search for Lost Time

I've been doing it with Reading Lists, but I am thinking if Titles were available as a Type, I would associate items 2, 3, and 4 with item 1. Or perhaps there could be a way to create arbitrary entities without resort to arbitrary custom columns.

IMO Associations need a Name, like Links and Notes, e.g. Mary Wollstonecraft and Mary Shelley are Mother and Daughter, whereas J.K. Rowling and Robert Galbraith are Joanne Rowling's pseudonyms.

BR

[DaltonST]

I am thinking if Titles were available as a Type, I would associate items 2, 3, and 4 with item 1

During development of Version 1.0.0 I had Title as a valid Entity Type for a second, but then had misgivings since I was viewing it from the perspective of Notes, which is in Standard Calibre as book (title) Comments. However, your Proust Association example is an entirely different perspective that convinced me to add Title back as an Entity Type just now for Version 1.0.1.

Or perhaps there could be a way to create arbitrary entities without resort to arbitrary custom columns.

Yes, there is a way, and which I thought about doing last month, but deferred until someone decided that it was truly needed after they had time to use the initial EM functionality. That didn't take long (just a few hours). So, I will add that for Version 1.0.2.

IMO Associations need a Name, like Links and Notes, e.g. Mary Wollstonecraft and Mary Shelley are Mother and Daughter, whereas J.K. Rowling and Robert Galbraith are Joanne Rowling's pseudonyms.

This is a mixed bag. Pseudonyms really should be its own Custom Column (as in Job Spy's Pseudonym functionality). However, the relationship between 2 Authors is certainly an attribute of the EM Association for them, much like a Tag is for a Title. It is clear that Associations themselves could be Entities. This is for Version 1.0.2. (updated)

Version 1.0.2 has been released.

DaltonST

[BetterRed]

Thanks, no hurry. I have several other potential uses of EM I want to explore over the next week or so.

How do I add multiple notes, each with a different name, to the same entity? I somehow managed to add two with the same name (Bibliography) which I didn't expect to work, but I can't see how to add two notes with different Names - e.g. Bibliography and Biographies.

BR

[DaltonST]

To add additional notes, 'clear' the existing note editing box, put in a new name and a new note text, and click 'add note'. The dropdown box above with the names of all of the existing notes for the current entity will now include the new name. The names in the dropdown are sorted alphabetically, so you might have to click it to select what you just added if you need to review it after it was created.

Yes, you can have multiple notes and links with the same name. I allowed that since I figured if I disallowed it, someone would come up with a good reason to allow it. I skipped to the end. There is a "link id" and "note id" internally that makes each link and note unique, and permits you to "change" a specific note among many notes having the identical name.


DaltonST

[DaltonST]

Version 1.0.2 - 2019-01-03 Arbitrary, user-defined "@" Entity Types. @Association Entity Type special functionality.

Associations themselves may now be created as a proper Entity, with an Entity Type of "@Association" and a value that reflects the Entity Types and Values of the 2 other Associated Entities. @Association is automatically created by EM for this purpose.

Refer to the image in the Original Post.



DaltonST

[BetterRed]

As far as I can see there is no way to see in the Book List, that something (e.g. Publisher:Random House) has been been selected as an Entity.

Any thoughts of making that available, I'm thinking of something which I could use in a Column icon rule to show the stylised purple 'e' on cells that are entities.

The use of '@' to distinguish User Defined Entities could be confused with the use of '@' to distinguish User Categories. How about making it something else.

BR

[DaltonST]

@UserEntityTypes are very similar to @UserCategories, but just not in the Tag Browser. UETs are psuedo-columns, just like UCs. Look in table preferences key field_metadata, and you will see that UCs are next to Custom Columns that are next to Standard Columns. In theory (but not in practice since the book-level values would be missing), I could add UETs to the Tag Browser where all of the Standard/Custom Column ETs already are. That is why I like the "@".

As far as I can see there is no way to see in the Book List, that something (e.g. Publisher:Random House) has been been selected as an Entity.
Any thoughts of making that available, I'm thinking of something which I could use in a Column icon rule to show the stylised purple 'e' on cells that are entities.

I am not a Template person, do not ever use them, and do not know if they are limited to the GUI Cache, or can read metadata.db directly via APSW. A Template function in general program mode that could read a SQLite database directly would work. EM is outside of the GUI Cache entirely, since book ID is not anywhere within EM. That is its point; it is designed to be "above" book-level. So, the general program mode function would be reading from the physical database instead of the in-memory Cache for every book and every potential ET column at startup. Slow for large Libraries with many ET columns. If Templates can do it, I will write the general program mode function and make it available to copy-and-paste. However, if they cannot access APSW, then I know of no way to do what you ask. I am interested to know if they can.



DaltonST

[BetterRed]

Originally Posted by DaltonST

@UserEntityTypes are very similar to @UserCategories, but just not in the Tag Browser. UETs are psuedo-columns, just like UCs. Look in table preferences key field_metadata, and you will see that UCs are next to Custom Columns that are next to Standard Columns. In theory (but not in practice since the book-level values would be missing), I could add UETs to the Tag Browser where all of the Standard/Custom Column ETs already are. That is why I like the "@".

I can see how UETs and UCs are similar, but I see no advantage in using the same distinguishing leading character. The main significance of the '@' on a UC is its usage in search terms e.g. @Favourite Authors:true. Its usage on UETs might lead a user to think they are similarly searchable.

Originally Posted by DaltonST

I am not a Template person, do not ever use them, and do not know if they are limited to the GUI Cache, or can read metadata.db directly via APSW. A Template function in general program mode that could read a SQLite database directly would work. EM is outside of the GUI Cache entirely, since book ID is not anywhere within EM. That is its point; it is designed to be "above" book-level. So, the general program mode function would be reading from the physical database instead of the in-memory Cache for every book and every potential ET column at startup. Slow for large Libraries with many ET columns. If Templates can do it, I will write the general program mode function and make it available to copy-and-paste. However, if they cannot access APSW, then I know of no way to do what you ask. I am interested to know if they can.

My rough thinking is running along these lines:

• In a library where I wanted to use the Entities Manager I would have to define a custom column to store a list of columns and values that have a corresponding entity: e.g.#entity_cells/Entity Cells, viz:
  
  
  
  I chose that particular definition because I would NOT want it in the tag Browser. A individual book would only have one of these with an id that matched the book table item id. If each lookup_name:value pair had a separate row in the table with its own unique id, then it would probably make maintenance easier, but may make the icon template more difficult - so I'm assuming one per book.
  
• When an entity is created, the column lookup name and value (e.g. 'publisher:Random House', '#illustrator||Clement Hurd') would be added to the book's #entity_cells list. Maintaining the list for changes and deletes would be bit tricky if its stored a single row with multiple items.
  
• Then I would need to write an icon template for each column likely to have entities, that interrogated the #entity_cells column for an entry that had lookup_name and lookup_name value - i.e contains "publisher" and contains {publisher). Not sure if it would be possible to write a single template to use on any entity-able column.
  
  The Entity Cells column could be displayed in Book details panels. Be nice if each item appeared a separate line in a list, should be feasible with a single long text column. Should the list be sorted one asks - someone's bound to ask
  
  I haven't tried to map this scheme to the Entity Manager tables - my guess is that's there is considerable overlap.

BR

[DaltonST]

See the attached image for a yes/no column easily updated by an EM program manually and/or automatically at Startup if a particular book has any Entity Type with a matching Entity Value already created as an Entity.

#18 is the yes/no CC table.

The code is just for Author, but checking for all Entity Types that exist in the the _entities table is trivial.

UPDATE custom_column_18 SET value = 1
WHERE custom_column_18.book IN
(SELECT book FROM books_authors_link WHERE author IN
(SELECT id FROM authors WHERE authors.name IN (SELECT value FROM _entities WHERE type = 'Author')));

UPDATE custom_column_18 SET value = 0
WHERE custom_column_18.book NOT IN
(SELECT book FROM books_authors_link WHERE author IN
(SELECT id FROM authors WHERE authors.name IN (SELECT value FROM _entities WHERE type = 'Author')))

The only decision that each user would need to make is whether they want a single yes/no table for each Entity Type, or just one per book if there is any Entity Match among any of each book's metadata.


DaltonST

[BetterRed]

I was looking for indication at the cell level rather than the book level.

Having a Y/N column for every column that might have entity-info would add a lot of clutter to Preference settings, the Book List, View Manager settings etc. And the Y/N column won't move with its corresponding value column when the latter's header is dragged around in the book list - which I do quite frequently.

Whilst my preference would be to use column icon rules to indicate a cell has some entity-info, other folks might prefer to use column colour rules.

Unfortunately icons and colour rules are not applied in the Book Details panels. That was one of the things that attracted me to having a single long text cell per book containing a list of column type:value pairs that have entity-info.

BR

[BetterRed]

This is the sort of thing I was aiming for



Imagine I have associated the four titles with the icon to a user entity named 'Search for Lost Time', and I've added a link to 'Princeton press', and a note defining what I mean by 'Medieval History'.

For completeness sake these are the dummy Column icon rules I had to set up



BR

[DaltonST]

Perhaps automating the creation and maintenance of the various rules is the best way to go forward.

Could you Export your column rules and attach them to your previous post so I have a "control" data set to compare to while testing?



DaltonST

[compurandom]

If I try to use the tags column or author column for entities, the value pulldown essentially opens a full screen window, because both of these fields contain metadata imported from the books, and I've got at least one book for each that imported a full line of garbage with either no delimiter or an incorrect delimiter.

Rather than hunting through my library for extra long authors or tags, can you put a limit on the width of the pulldown or something?