Crash when adding a custom (built) column

[MidwestJen]

I've not had this happen before, and I'm not sure if it's a limitation with the program or my system/setup.

I wanted to add a new custom column (built from other columns and behaves like tags) that would tell me what virtual library each entry is in. I was able to add the column fine (with {:'virtual_libraries()'} as the template code - thanks chaley!), but when I tried to restart Calibre it locked up. Eventually Windows broke in after an hour and fifteen minutes (or so) and told me that the program had crashed and would I like to look for a fix, etc. The only way I could open Calibre at that point (since I hadn't backed up the metadata file like an idiot) was to run a command line to remove the new column - Calibre started right up with no problems then. I repeated this a few different times (after first backing up the db), but got the same result each time.

I know it's not how I'm adding the column because I CAN add it to an empty copy of this same library (and copying books over makes the column fill in appropriately).

I do have a fairly large library - 11,819 entries as of writing this - so I did expect the addition to take awhile (especially considering that it's building the information) - but with it actually crashing it's hard to know what else I can do to troubleshoot it.

As a 'slow' workaround (because this isn't really quick) I'm going to copy books over to the new library that I've already added the field to, but I guess I need to know if this is something that is fixable in the future - or should I sit down and think really hard about any other custom built columns before I migrate everything so that I don't have to do this again? Or is there a faster way to move things that I haven't seen (I'm using the basic copy to another library feature).

---
I'm running Calibre (32bit) v2.7 on a Win7 (64bit) computer with 4GB RAM and 4 - 2.4GHz i5 processors. The calibre.exe installation and both libraries are all on the same hard drive with plenty of room. The metadata.db file is 48,597kb and I have 5 virtual libraries in addition to the default that contains everything. I can't recall any other custom columns that are building off of others. I have no issue upgrading to the 64bit version of Calibre, I just started with this version and haven't bothered to change it. I do not have any crash reports, but I'll happily go digging for them if you tell me where to look.

Thanks in advance for any advice!

[chaley]

This came up only a few days ago in the thread https://www.mobileread.com/forums/sho...d.php?t=248762 (relevant posts start at 10 or so). We are still looking at the best way to ameliorate things.

Bottom line: the problem is a combination of the size of your library and the virtual libraries column. For the moment, don't use the virtual_libraries() function and you should be fine.

EDIT: You really should be running the 64bit version of calibre, given you have a 64bit OS.

[MidwestJen]

Bummer - I did see that post after I posted this, but I suppose I was hoping that because my db wasn't quite as complex that it would be a simpler fix (plus, you know, already posted this).

On the other hand if I didn't need this info in Calibre Companion I wouldn't need to make the change at all ... *wink*.

I can wait and see if it's possible to adjust things to make it work - I'm pretty sure it'll be faster than porting all of my stories over since doing them in 1k batches is going to take days I'd rather not spend. =)

Thank you for your prompt response!

EDIT: I'll upgrade today, promise.

[chaley]

@MidwestJen: one thing you can do as a temporary fix is to "hard code" the virtual library names into a custom column "text, behaves like tags". The steps:

1) Create the new column. For these steps, lets call it #vls.
2) (only if recomputing the values) Select all books in your library. Use bulk metadata edit to clear all values from #vls
3a) Select one of the virtual libraries.
3b) Select all books.
3c) use bulk metadata edit to add the VL name to #vls (tags to add)
3d) repeat 3a - 3c for the remaining virtual libraries
4) Connect CC as a wireless device and let the metadata transfer. It will transfer metadata for all the books on the device.
5) (One time) add #vls to the grouping drawer.

With 5 VLs, this process isn't too terrible. The worst part is that it forces transferring all the metadata.

[Tanjamuse]

I have now done as Chaley suggests The other custom column was too hard on my computer.

[MidwestJen]

I did end up creating a hard coded column for my virtual library entries for now, but I did it with a 'Search and Replace' using {:'virtual_libraries()'} as a template and popping the result in my new column (oddly enough I was able to do all 11k entries in one batch).

My virtual libraries are fairly fluid, depending on what I'm doing with them, which is why I wanted to use the custom built column, but until/if that ever gets adjusted, this should work since I only need to select the entries that have been modified since my last run. It's not automated, but close enough so that I'm not growling at the screen every time I remember I need to adjust things. I'm a tad OCD about keeping things updated. *blush*

I'll keep an eye out for any updates to this in the future and I really do appreciate the help.

[chaley]

Originally Posted by MidwestJen

I did end up creating a hard coded column for my virtual library entries for now, but I did it with a 'Search and Replace' using {:'virtual_libraries()'} as a template and popping the result in my new column (oddly enough I was able to do all 11k entries in one batch).

That was a good idea. It incidentally helps confirm what the problem is and that the solution I proposed to Kovid will help.

[chaley]

Several changes have been made to calibre to help with this problem. Tanjamuse's library with the virtual_libraries() custom column now opens in 52 seconds (on my machine). Without the VL column it opens in 26 seconds instead of 52 seconds. These changes are available in source now and will be generally released next week.

It is worth noting that the changes are not specific to using the virtual_libraries() function. They improve the performance of all composite columns (columns built from other columns), especially if a composite column is shown in the tag browser.

[Tanjamuse]

Would I need to have them shown in the tag browser or just available there?

[chaley]

Originally Posted by Tanjamuse

Would I need to have them shown in the tag browser or just available there?

Performance will be improved if a composite can be shown in the tag browser (has the box checked), regardless of whether it is hidden or not.

[Tanjamuse]

Originally Posted by chaley

@MidwestJen: one thing you can do as a temporary fix is to "hard code" the virtual library names into a custom column "text, behaves like tags". The steps:

1) Create the new column. For these steps, lets call it #vls.
2) (only if recomputing the values) Select all books in your library. Use bulk metadata edit to clear all values from #vls
3a) Select one of the virtual libraries.
3b) Select all books.
3c) use bulk metadata edit to add the VL name to #vls (tags to add)
3d) repeat 3a - 3c for the remaining virtual libraries
4) Connect CC as a wireless device and let the metadata transfer. It will transfer metadata for all the books on the device.
5) (One time) add #vls to the grouping drawer.

With 5 VLs, this process isn't too terrible. The worst part is that it forces transferring all the metadata.

I've been wondering... What if the story belongs in two different virtual libraries or more?

[chaley]

Originally Posted by Tanjamuse

I've been wondering... What if the story belongs in two different virtual libraries or more?

Enter all the libraries in which the story belongs, separated by commas. That is why I suggested a column type "text, behaves like tags".

That said, the changes to calibre should have made using the custom column showing the virtual libraries have acceptable performance.

[Tanjamuse]

I thought I could use the hierarchical setting? I have different libraries (not just VL) in my Calibre on the computer.

[chaley]

Originally Posted by Tanjamuse

I thought I could use the hierarchical setting? I have different libraries (not just VL) in my Calibre on the computer.

I don't know what question you are asking.

If you are asking whether CC can show hierarchy, then the answer is "Yes, to one level." If you are asking whether calibre can show hierarchy, the answer is "yes". If you are asking whether CC can show values from multiple non-virtual libraries, the answer is "yes, but you might want to use the same lookup names for columns that have the same purpose." If you are asking something else, then you need to make the question more clear.