Calibre always freezes with tag browser shown

[Noughty]

I had this problem with version 0.8.18 and updating to 0.8.22 didn't solve the issue.

Calibre works smoothly without tag browser with it not so much. Basically if I edit metadata (not matter how minor) for more than one book, copy more than one book to another library Calibre always freezes up for a couple mins (time seems to be the same always). It happens if I edit even two books so it makes it impossible to edit metadata if tag browser is open.
It works perfectly even with cover browser opened as long as tag browser is closed. So to manage my library I have to always close tag browser which is a little annoying. And even so, after all edits if I show tag browser it freezes, so there is no avoiding it completely just diminishing the amount of freezing up.

What could be the cause of this or is this normal (I don't remember it always freezing up like that)? Could it be because of the number of columns I created or categories in tag browser I have? How many is too much for calibre? Does hiding columns/categories in tag browser make it easier for calibre (somehow I doubt it, since it still writes all metadata even if column is hidden)?

I have up to 40 columns total (with all default columns). All are never shown (msotly a few in different views) and in tags browser I have 16 (counting everything even searches) categories shown out off 28 possible (with all default created and mine).

I also have one column for making spreadsheat so it is very long as includes fields like title, author, formats and comments (summary).

I didn't increase my library so it can't be caused because of too many books.

Also, might be related, I seem to have all (or at least some) of my previously had columns in preferences>Look and feel>book details. They are deleted but they are listed here (as far as I know only there). Is it possible to delete them somehow? Could they cause problems?

[DoctorOhh]

Originally Posted by Noughty

So to manage my library I have to always close tag browser which is a little annoying. And even so, after all edits if I show tag browser it freezes, so there is no avoiding it completely just diminishing the amount of freezing up.

I'll let someone else fill in the specifics of why this is so, but seeing a slow down when the tag browser is open has been par for the course for a long time.

How annoying can it be to click the tag browser icon in the lower right corner or drag the window closed or open?

[Noughty]

Originally Posted by dwanthny

How annoying can it be to click the tag browser icon in the lower right corner or drag the window closed or open?

Very little, but sometimes I need to use tag browser actively while editing metadata so hiding/ showing it won't help as it will freeze every time I show it again after editing. You can't cheat the freeze lol.

This solution (hiding/showing) works only if I needed to make a lot of edits and can manage it without using tag browser and show tag browser after all edits since it will freeze only once. Now it only helps to diminish the number of freezes if I make a few edits without the use of tag browser.

If this is a known issue and without a solution I can live with that and adjust my use of library. But I'd rather make sure it's not my library problem. Maybe I'm doing something too much, abusing some features or something and cause this. And this started for me recently. It would freeze up a little after big bulk edits but not so precisely and not after adding a tag for two books.

It's also interesting that freeze is always for the same time. Haven't counted seconds, but it's always two minutes. No exceptions.

Update: Interesting that there is no freeze when using goodreads sync plug into add values to columns. Tag browser opened and the category which gets edited even is in tag browser, it is updated in a few seconds. I had added tags just now and it froze only for like 10 seconds (it's a very popular tag so many books in it)

[DoctorOhh]

Originally Posted by Noughty

Very little, but sometimes I need to use tag browser actively while editing metadata so hiding/ showing it won't help as it will freeze every time I show it again after editing. You can't cheat the freeze lol.

You know it's not freezing but updating the database. Periodically Chaley and Kovid review this area and occasionally optimizations occur from those reviews. Keep your fingers crossed they may find a hiccup that can be optimized.

Originally Posted by Noughty

Update: Interesting that there is no freeze when using goodreads sync plug into add values to columns. Tag browser opened and the category which gets edited even is in tag browser, it is updated in a few seconds. I had added tags just now and it froze only for like 10 seconds (it's a very popular tag so many books in it)

Interesting, I believe this plugin generates its own calibre-parallel.exe worker process and updates the database from this process.

Here's hoping a fresh review might find something.

[Noughty]

Thanks, fingers crossed. Hope the work load will be reduced somehow if editing only a few books.

If there are any suggestions to make it easier for my calibre to update, they are welcome. Maybe fewer columns could help a little. Since I started to get the template language I might be able to reduce column number by combining columns. This did started when I created new columns so I'll combine some.



What about

I seem to have all (or at least some) of my previously had columns in preferences>Look and feel>book details. They are deleted but they are listed here (as far as I know only there). Is it possible to delete them somehow?

[kovidgoyal]

The database rewrite I am doing eliminates the Tag Browser slowdown almost completely, but given all else that is going on, it will be a while before that sees the light of day.

[chaley]

One thing to note: using composite columns as tag browser categories (the box "show in tags browser" is checked) is extremely cpu-intensive. It is even more expensive if the composite column is tags-like. Hiding the columns in either the library view or the tags browser does not change this.

Unfortunately, some time back I broke editing that check box in custom column setup, but didn't notice until today. Turning off that feature for an existing column should wait until the next calibre release (0.8.23).

Another thing that you can't do much about: using even one composite column as a tag browser category will cause all the composite columns for every book to be evaluated. The more complex your templates, the longer it will take. One thing you might try: general program mode templates ("program: ...") evaluate faster than the other modes because they are lexically analyzed once instead of on each use. If translating a template to GPM isn't too hard, then you might consider doing that.

[Noughty]

Thanks for clarifying why it happens. That has to be it since I just made quite a few columns from other columns and some of them are custom columns from others behaving like tags. The more complicated ones aren't checked to show in tag browser.
I have like 4 columns which are categorized tags: I extract genres, character qualities and similar from tags and shelves to make more manageable lists. The most complicated one combines two lists and extracts 3 previous made lists. So I get that it might take a lot of cpu.

Originally Posted by chaley

]One thing you might try: general program mode templates ("program: ...") evaluate faster than the other modes because they are lexically analyzed once instead of on each use. If translating a template to GPM isn't too hard, then you might consider doing that.

Damn I just yesterday started getting the hang of templates . Will have to see the program mode templates, hope it isn't much harder. For me the "grammar" is hardest to get, after that it's not hard.
If it's too hard I can live with freezing up a little . Can be smart and use search instead of tag browser while editing metadata.

So making a lot of columns is ok, as long as they aren't composite columns from other columns, behaving like tags? Only these make it harder? Just don't want to redo a lot to have the same result . I already removed many columns as I thought they could be the cause (didn't lose any data, just changed how it is shown). Simple columns from other columns are ok too?

And is it possible to remove items from preferenes>look and feel>book details? It's a mess, I have trouble finding things. It seems that even if I just rename a column, the old title remains in there, so renaming column leaves me with 2 columns in that list. I have almost 30 unnecessary columns in there.

Update. Damn. I just read Using general program mode and have no idea how to use it. I'm useless without examples . And is it an example in there http://manual.calibre-ebook.com/temp...l-program-mode ? Like you post all that and get a result? Where do you even post it?

I just checked and I think that 3 of the 4 my mentioned columns are made with GPM (if indication is the word 'program:"). I'm not familiar enough to separate them yet by anything else. So only one column (behaving like tags) made with simple template. Will see if it's possible to remake.

Here how they look (all 3 basically the same only values different):

program:
	list_union(
		list_intersection(field('tags'), 'Action, Adventure, Contemporary, Crime', ','),
		list_intersection(field('#shelf'), 'Action, Adventure, Contemporary, ', ','), 
		',')

So maybe I do have an example


I think I did it (not neatly but it works):

program:
list_sort(
          list_difference(list_union(field('#shelf'), field('tags'), ','),
                              list_union(list_union(field('#genre'), field('#length'), ','), field('#character'), ','), ','),0,',')

So this is GPM? Or no? Where else could I find info on it? Since it looks almost completely the same, just word program in beginning and a few tweaks.

Is this better than:

 {:'list_sort(list_difference(list_union(field('#shelf'), field('tags'), ','), list_union(list_union(field('#genre'), field('#length'), ','), field('#character'), ','),','),0,',')'}

Performance doesn't seem to have changed. Might be slower now . Definitely slower, up to 3-4 mins instead of 2... But moving to another library works smoothly now... It used to freeze too. Now freezes only for metadata changes but longer

Update. I edited the code, there was no space in original, just here

[chaley]

Regarding making columns: every column you add will add some work, somewhere. However, the work added for non-composite columns is minimal.

Regarding GPM: you got it right. GPM starts with the word program. Do note that the word "program" *must* start in column one. Your example has a space before program. That shouldn't do anything good, but it might be attempting to evaluate it somehow.

One advantage is you can have more than one line, which can make things easier to read. Using your example, you could have:

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(
         list_union(field('#genre'), field('#length'), ','), 
         field('#character'), 
         ','), 
      ','),
   0, ',')

This lines up the arguments for a function.

As for it running 2 times slower, that makes no sense at all to me. If and when you have some time, could you verify it? First make sure that your templates are working (e.g. no space before program, correct results). Then switch the one that you changed back to template program mode as you had before, and see if it gets faster. If it does, then something very mysterious is going on.

[Noughty]

I tried but it still seems slower. Might be because of other columns. I'll try to modify some, to remove one unnecessary made into complicated column. And will test more. But first I want to finish this template.

Now I am editing it to remove one tag but it fails to remove a comma. So I get a list with one extra comma. I use function re and tried to simply write the tag, also tried with list_intersection., list item and so on I still remain with a comma. What function to use to remove one tag without leaving comma?
The closest I got was writing re function three times and listing tag with comma before, after and without comma.

program:
re(
re(
re(
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(
         list_union(field('#genre'), field('#mode'), ','), 
         list_union(field('#character'), field('#length'), ','),
         ','), 
      ','),
   0, ','),
'tag,', ''),
', tag', ''),
'tag', '')

I can't simply delete that tag from tags since I need it, just not in this list, only in tags list.

Now will do some testing.
After changing it to old template it froze for 6 mins. So total work load increased and having the new program is definitely faster. That's one boring testing to stare at the screen while it unfreezes

Will modify some other columns so the freeze time decreases more.

[chaley]

Originally Posted by Noughty

Now I am editing it to remove one tag but it fails to remove a comma. So I get a list with one extra comma. I use function re and tried to simply write the tag, also tried with list_intersection., list item and so on I still remain with a comma. What function to use to remove one tag without leaving comma?

You are saying that list_sort without the re functions returns a list with a comma at the end? Or that 'tag' is in one or both of #genre or #character, but it is not being removed from the first list? Or that you want to unconditionally remove 'tag' from the result, even if it is not in the union of '#shelf' and 'tags'?

If it is the latter, then I would try something like the following:

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(
         "tag", 
         strcat(field('#genre'), ',',
                field('#mode'), ',', 
                field('#character'), ',',
                field('#length')),
         ','), 
      ','),
   0, ',')

This works by doing a union of the list "tag" and a manually constructed large list consisting of items in #genre through #length. It is OK that there might be duplicates in this constructed list, because the list_union will remove them from the final result. In the current release, only the second parameter to list_union can have duplicates. In the next release, either parameter can have duplicates.

[Noughty]

I meant that that one tag is in tags column. So I need to remove with some extra function. In ways I tried to remove it I always removes only the tag leaving a comma behind. So I get

, tag1, tag2

or

tag1, tag2,

or

,

instead of

tag1, tag2

I'll try your solution - works like a charm. Thank you.

It's nice to actually start to understand what a template does while just looking at it without calibre showing it to me
But I still need to learn a lot of functions...

Will try to transform other columns

[Noughty]

I decided to "cheat" the program. I deleted all programs in four custom columns I made (except the four made for tag categorizing since I always need them) and left them with

(default)

. (In total I have 8 columns made from other columns, 4 of them act like tags)
When I need these values, I'll just paste the program into calibre from txt file I saved. Since I'll need these columns rarely it improves performance speed twice.

So I am again with 2 mins freeze instead of 4. I guess I'll have to live with it. Since everything is in GPM now, nothing else I can do.

Or deleting the columns would make it even better? With that little I know, I don't think so as these columns have no program in them. I have four columns which I left without program, I could leave only one and paste all programs in it when I need it. I just don't want to delete 3 columns without good reason since I already have like 50 trash columns in preferences for book details.

Thanks for all the help

[chaley]

Originally Posted by Noughty

Or deleting the columns would make it even better?

The answer should be "no".

Could you either post or PM me the 4 remaining templates that are used for categories? I want to see if there is something in the template functions that is taking a lot of time.

How big is your library? How many items on average are in one of these composite column categories?

[Noughty]

Basically these 4 columns manipulate shelves and tags. In tags I have 102 items, shelves about 118. That's why I created these other columns to make it neater. I categorize all items in tags and shelves into a few categories:
genre (40 items),
character (58 items),
mode (22 items),
and whatever is left (84 items).


GENRE:

program:
	list_union(
		list_intersection(field('tags'), '40 tags&shelves listed', ','),
		list_intersection(field('#shelf'), '40 tags&shelves listed', ','), 
		',')

CHARACTER:

the same as above just different tags,shelves listed. Total 58 items

MODE:

the same as above just different tags,shelves listed. Total 22 items

LEFTOVERS :

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(
         "delete", 
         strcat(field('#genre'), ',',
                field('#mode'), ',', 
                field('#character'), ',',
                field('#length')),
         ','), 
      ','),
   0, ',')

I just changed this one to, #length wasn't needed here, only word Short:

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(strcat('Short', ',',
        'delete'),
         strcat(field('#genre'), ',',
                field('#mode'), ',', 
                field('#character')),
         ','), 
      ','),
   0, ',')

And I have a few more columns like tags but they are simple columns.
Length has 4 values, status-6,managing-22, anthology-18 and a few more. But nothing major and with very few values.