Options for summarizing the book list

[chaley]

I am considering submitting to @kovid a change to add a new search term to calibre: summarize_booklist:field_name,count, where 'field_name' is the lookup name and 'count' is optional.

This search term does the same thing as the template in this post. Specifically it scans the current book list in order, searching for values in "field_name". If the number of value(s) already seen is less than "count" then it adds the book to the search results.

Because it scans the book list in order, you have control over which value(s) it finds first by sorting the book list before doing the search. Because it uses the book list, subsequent "summarize_booklist" searches further summarize what you see. You get back to the "full" book list (subject to VLs) by erasing the "summarize_booklist" search.

Here is an example of summarizing the book list by series. The first capture shows several series on the book list, sorted by series.


This capture shows the result of the "summarize_booklist:series,1" search:


This capture shows the result if the book list was sorted series descending before the "summarize_booklist" above was applied.


And finally, this capture shows the result of applying "summarize_booklistublisher,1" to the result of the previous summarize.


It isn't clear that I should propose this code. The biggest issue is that it is similar in concept to making a pivot table, but there is no way (and never will be a way) to interactively open and close the sublists. In other words, there is no "button" to show the books in theory hidden behind the displayed book.

A second issue is that as mentioned above, the summarization is cumulative. To start over one must erase the current search, or do something like open a VL that does the same thing. This behavior is different from normal calibre searches, which start over each time, ignoring what is displayed. I can't repeat that behavior so there will be confusion, and therefore posts and bug reports.

Finally, this search term works only in the GUI. Using it from the command line or calibre server won't work.

Still thinking. Your thoughts are welcome.

[Katja_hbg]

I hope I will find more time to have a deeper look into this but here is my immediately comment:

Some months ago I adjust your "first book of a serie" so that it shows "the oldest (download date) unread book of each author". I assume with your new search I can do that as well.

About the second issue, if I understand that correct. This effect is also visible using this new search when you enter any other filter before. E.g. search "David". Search "summarize...". If so, that may confuse.

Only as a quick idea: adding parameter-field for sort and sort-order.
Could that avoid the cumulative issue as you do not need "to look at the screen"?

[chaley]

Thanks for your thoughts.

Originally Posted by Katja_hbg

Some months ago I adjust your "first book of a serie" so that it shows "the oldest (download date) unread book of each author". I assume with your new search I can do that as well.

Yes. Search for #read:no (or #unread:yes or whatever). Sort the book list by download date descending. Summarize by authors.

About the second issue, if I understand that correct. This effect is also visible using this new search when you enter any other filter before. E.g. search "David". Search "summarize...". If so, that may confuse.

I'm not sure what you are saying. Filters before the summarize don't affect the book list so they won't have the same cumulative effect. Instead they affect the 'candidate' books that the summarize can show, still in the same order as the book list.

The easiest way to use it would be to use search to filter the book list then save that search as a temporary VL. This permits you to get back to the search result quickly. Next sort the book list in the order you want. Then summarize.

Finally, it is reasonable to summarize twice, that is summarize a summary, as I did in the example in the first post.

Only as a quick idea: adding parameter-field for sort and sort-order.
Could that avoid the cumulative issue as you do not need "to look at the screen"?

It could, but it isn't practical. Sorts can be arbitrarily complex. We would need to create some way for the user to express multiple nested sorts on the search bar, which would be a bit wierd.

However, it might be possible to get the current sort and apply that before doing the summarize. This would eliminate the cumulative effect, with the downsides that

• performance would be worse. Perhaps much worse.
• it would no longer be possible to summarize a summarized list.
• it might end up more-or-less requiring using a temp VL (not clear about this).

[kovidgoyal]

I'm not the greatest fan of the cumulative behavior. It's surprising, different from all the rest of the searching mechanism and likely to generate why does this work this way type bug reports.

If I understand correctly, the use case is basically generating a "report" of some kind over your books? Like series with the last book or whatever? Might be better to have a separate reports tool like there is in the editor, in a popup rather than trying to shoehorn it into the search + book_list view.

[chaley]

Originally Posted by kovidgoyal

I'm not the greatest fan of the cumulative behavior. It's surprising, different from all the rest of the searching mechanism and likely to generate why does this work this way type bug reports.

The lastest experiment doesn't have cumulative behavior other than expected, where search terms in the search, including summarize_booklist, affect subsequent search term evaluation. Said another way, search expressions, including summarize_booklist, are reproduceable. Running the expression again gives the same answer, assuming no context changes such as VLs, sorts, and underlying data changes.

If I understand correctly, the use case is basically generating a "report" of some kind over your books? Like series with the last book or whatever? Might be better to have a separate reports tool like there is in the editor, in a popup rather than trying to shoehorn it into the search + book_list view.

I will look at this.

The "pro" is avoiding adding "stuff" to existing mechanisms.

At first blush the "cons" are:

• Being required to create a query language for the report tool that duplicates the existing search, sort, and VL mechanisms.
• Not having the rest of calibre easily available, such as book details, pin views, column ordering, column icons, and plugins.

[kovidgoyal]

The rest of calibre would still be available. You could pre-apply any VL or have a button to apply a new one. The popup would function like the existing browse notes or browse annotations popups that are maeant to be used alongside the main calibre window. The main con is its a lot more work to implement. But usage wise it would probably be easier to use.

[chaley]

Originally Posted by kovidgoyal

The rest of calibre would still be available. You could pre-apply any VL or have a button to apply a new one. The popup would function like the existing browse notes or browse annotations popups that are maeant to be used alongside the main calibre window. The main con is its a lot more work to implement. But usage wise it would probably be easier to use.

One possibility within the amount of time I am willing to put into this would be a dialog where the user would construct the query. The dialog would have a "Show" button (or some such) that would update the book list with the results of the query. Each operation would have its own line in the dialog. The dialog would non-modal. The gui would refresh when the dialog is closed.

There would be 2 query operators:

• search: a standard calibre search expression
• filter: reduce the result to the first value(s) in some field. Parameters: field name, acceptable count, and sort order (asc, desc, not changed).

For example, this query expression:

• search: series:true and #read:no
• filter: series, 1, ascending
• press "Show"

would show unread books with the lowest series index.

One possible modification is not to put the "sort" in the filter but instead to have a separate sort operator. In this case the expression would be

• search: series:true and #read:no
• sort: series, ascending
• filter: series, 1
• press "Show"

The query for unread books with the highest series index would be

• search: series:true and #read:no
• sort: series, descending
• filter: series, 1
• press "Show"

We would probably want to be able to save & load querys.

It isn't clear to me that something like this is easier to use then embedding the filtering into the standard search expression. It does have the advantage that all the steps are explicit and allows the sort to change in the middle of the query. That in itself might make it (or something like it) preferable to embedding the filtering.

[kovidgoyal]

I'd certainly be in favor of having *some* UI for this. The search language is complex enough as is, having expression builders helps. Ideally this would be a new tab in the existing advanced search dialog, but it may not work there given that is a modal meant only for build expression type operations.

[chaley]

Originally Posted by kovidgoyal

I'd certainly be in favor of having *some* UI for this. The search language is complex enough as is, having expression builders helps. Ideally this would be a new tab in the existing advanced search dialog, but it may not work there given that is a modal meant only for build expression type operations.

I agree that the advanced search dialog is a good place for it. I can see how to make it fit in. One would build the query in the dialog. There would be a button to "apply" the query, which would cause the book list to reflect what it says to that point. The OK button applies the query and closes the dialog.

Assuming I go ahead with this, my goals are:

• No calibre behavior changes if you don't use it.
• The above-mentioned Advanced search dialog behavior works.
• A "query" is repeatable.
• A query is reusable.
• Queries leave the book list in an understandable state. In particular, any sorting that was done in the query is reflected in the GUI and in the sort history.
• Operations on the book list after a query do what one expects. In particular, the interface is stable when adding books and subsorting the results.

However, given that the calibre community is showing almost no interest in this, I am putting it aside to relect on whether to do it, and if so how.