HELP..Problems with permanent Column Views

jecilop · 10-14-2015, 01:00 AM

I'm posting inquiries here to see if anyone knows of this before submitting a bug.

I cannot move my custom columns into a view position and have it be permanent. Every restart goes back to the view I've used for a long time. I do not use View Manager. The plug-in was installed, but I removed it to check if its presence affected this, but there was no change. I update regularly so cannot pinpoint a coinciding update and start of the problem. I've only just recently started making more adjustments to column positions. I am tired of re-positioning, re-sizing, and hiding these columns now. I use Caliber a lot, so it is significant.

EXAMPLE: Columns Viewed are A,B,C,D,E,F,G
* Move G and hide F such that A,B,C,D,G,E
* Close then restart PC or just restart Calibre; it always opens to the original.

I've tried this 2 ways, both of which worked in the past when I positioned my screen view:
1- The intuitive column drag to position and right click to hide,
2- The "Add custom columns" option where I use the up/down green arrows to move the column position,

Tech Info including testing environment:
Wdw 7

Calibre 2.40
No View Manager use or plug-in presence.
I do have my customized Calibre settings in a custom location that backs up to the cloud; however, the cloud syncing program is always inactive during Calibre use to avoid corrupting files.
Only basic background programs and virus software is running at the same time as these Calibre test.
There is plenty of memory.

BetterRed · 10-14-2015, 03:34 AM

@jecilop - column position, alignment, show/hide etc working fine here on calibre 2.40, Windows 10 64 bit.

Book List relative column positions, alignments, widths etc are kept in the database not the configuration data. They can be seen in the metadata_db_prefs_backup.json file in the library, look for "library_view books view state" - best viewed with 'format aware' text editor like Notepad++, TextPad, Text Wrangler etc.

I suggest you create a small test library and see what happens with it. Monitor the changes in metadata_db_prefs_backup.json. I can't recall if that file changes dynamically or when calibre closes.

If you forget to pause cloud syncing before starting calibre, even just once, then that could result in database corruption, my understanding that calibre sources the book list layout data from the database. That .json is a backup for use by the calibedb --restore_database command

BR

BetterRed · 10-14-2015, 03:42 AM

Addendum : jecilop - if you copy a few books into a fresh library that includes your custom columns and if the book list layout options are OK there, then move all of the books into the fresh library. Scrap the old library, assuming of course that it's now empty.

BR

jecilop · 10-14-2015, 10:56 PM

When I have a little time, I might play with this suggestion.

Right now the idea of waiting for my internet to re-upload my massive library when copied gto a new locations is more than daunting with a basic level cable internet. It took weeks to upload everthing the last time....weeks!

I will, however, read through your technical notes again (and again) so I understand them. First easy test, however, will be to create a new library with the custom columns filled (ie, copy from the main) and see if I can change the view the intuitive ways on that one and have it stay.

Thanks.

jecilop · 11-17-2015, 01:33 AM

Quote:

Originally Posted by BetterRed

@jecilop - column position, alignment, show/hide etc working fine here on calibre 2.40, Windows 10 64 bit.

Book List relative column positions, alignments, widths etc are kept in the database not the configuration data. They can be seen in the metadata_db_prefs_backup.json file in the library, look for "library_view books view state" - best viewed with 'format aware' text editor like Notepad++, TextPad, Text Wrangler etc.

I suggest you create a small test library and see what happens with it. Monitor the changes in metadata_db_prefs_backup.json. I can't recall if that file changes dynamically or when calibre closes.

If you forget to pause cloud syncing before starting calibre, even just once, then that could result in database corruption, my understanding that calibre sources the book list layout data from the database. That .json is a backup for use by the calibedb --restore_database command

BR

So I've tried the new library and such and see that works. I discovered a new problem while testing this about logic custom columns not copying (separate thread), but I've also learned more about .json files.
I do see that my problem library's metadata_db_prefs_backup file last updated in April! It is not updating when I make changes or close my library.

So......
Is that file one that can be DELETED, then allowed to restore? Does it store EVERYTHING that I don't want to lose. I've considered copying it to make a backup of the not-updating "backup" then deleting it to see what happens. Then, I think I could replace it with that messed up one if needed.

I fear trying that without someone who's in the know saying it's not a problem.
Obviously I've been adding new columns and settings since then that are not in the backup. They are still there when I go back.
I have another library that has the same setup, but it DOES update. Could I copy that metadata_db_prefs_backup file to this one if it's just about saving the views and/or configuration. I would assume based on what's happening that new columns in the problem database would not be affected. I'd just have to turn on their view since they aren't in the other library.

Any more input you cold offer???

kovidgoyal · 11-17-2015, 01:37 AM

You can always delete the json files, they are there purely as a backup. And if you find they are not being updated/re-created, then run calibre in debug mode and you might get more info on why. Though, if the reason is that you are using some sync software that is rolling back changes, then debug mode will not help.

BetterRed · 11-17-2015, 01:57 AM

@jecilop - I responded to your other thread on 'logical columns not being copied'

The metadata_db_prefs_backup.json file is NOT used at run time, it only comes into play if you do a database restore.

So, if you want to 'impose' the preferences from library B on library A, then rename metadata_db_prefs_backup.json in library A to say *.json.save, then copy the metadata_db_prefs_backup.json from library B to library A, and then do a database restore on A.

Oh yes, make sure you have backups of your libraries before you start

BR

jecilop · 11-17-2015, 04:00 AM

Ok, I went ahead with testing using my newer libraries to learn some of what you both posted while you were posting it :-)
BUT...could you please educate me on the database. I've re-read your posts, but I'm still unclear on where this information is stored and then accessed each time the library opens.

Specifically, where do the column view settings store? Is it the metadata file stored in the library? Per Kovid, the backup json is just for restoring.

Also, what is preventing the backup json from writing when I exit this library? I noticed when looking at a test library that the backup file was either created or updated when I exited the test library. That is NOT happening in this one. The first time after deleting, opening Calibre, making changes, then closing, I noticed a file named something like metadata_db_journal or a variation thereof appeared briefly then disappeared. I didn't see it again after repeating the open, change, close. Still, the backup json isn't writing here also; it obviously cannot update if it's not writing.
And still, I do not have any cloud syncing running during this.

Anyone willing to help me better understand this?
I did notice the metadata database updates when I make changes. I think that's probably a "duh" comment. If this stored the views, I don't understand why that isn't the only part not being saved when it updates.

My cloud syncing only runs when Calibre isn't. This is how I backup my library. It can't be pulling an old file(s). I can make changes to the view, close or restart Calibre, even restart the PC...Go right back to the Calibre library I'd just edited....and the view changes are back to the old. There is NO CLOUD SYNCING run between this.

My library check returns no errors, by the way. The deeper check of files and folders matching the database didn't have any major issues.
I'm working on the debug mode to see what that returns.

jecilop · 11-17-2015, 04:28 AM

OK...this is what the debug file returned after making the column view changes and position changes and closing...

*****************.....
Traceback (most recent call last):
File "site-packages\calibre\gui2\ui.py", line 948, in closeEvent
File "site-packages\calibre\gui2\ui.py", line 882, in shutdown
File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 587, in set_pref
File "site-packages\calibre\db\backend.py", line 126, in set
File "site-packages\calibre\db\backend.py", line 115, in __setitem__
File "site-packages\calibre\db\backend.py", line 806, in execute
File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 231, in resetcursor
CorruptError: CorruptError: database disk image is malformed

*****************************

So, disk image...hmmm...is there anything I can do about this. Perhaps, can I use the old backup json and run it, or will that wipe all I've done since that file was last updated (new columns, new library entries, new tags, tag changes, etc....)?

BetterRed · 11-17-2015, 04:55 AM

I suggest you restore (rebulld) the database - it's in the Library Maintenance menu, it runs in two phases - the first phase is usually quite quick, but the second phase can take a while if its a big library.

You may lose preferences you've defined since April, but you probably won't lose book metadata because that's restored from the metadata.opf files in the book folders.

The library specific preference data - (custom columns, view manager, reading list, virtual library definitions) is contained in table called 'preferences', there's not much point in trying to look at the table in the database itself - more useful to look at the backup json file.

BR

jecilop · 11-18-2015, 04:34 PM

Okay, I am doing some testing - testing that will also help me learn how Calibre works behind the scenes.

Friday's update to come (2.45?) is supposed to include the fix to the custom logic column copy per a reply I received. That would allow me to copy everything to a new library, but that would still require a massive upload to back the files up...MASSIVE on std to slow internet mean effectively never fully backed up.
While this would solve the database and thus column view issues, I would like to avoid this step if possible.

I'd really like to know where the custom preference info such as custom color columns, custom icon rules, customer searches, and virtual libraries is stored. Specifically, I'm looking for a file that I can open in it's raw or code form an manipulate if desired (such as copy/paste from old to new). I've done extensive changes to these since my April'15 prefs backup json, so the info there is only a partial help. Where can I access that active file to attempt to get that info from the current information that IS updating and staying when I add new stuff?

I've done this type of things with Tag Mapper when doing A LOT of stuff because sometimes interacting with plug-ins is very tedious and time consuming. It would be that way to "fix" a new library on a rebuild I think.

Still, I'm trying to run some more tests to learn about the last, but copying the entire old library to a new folder on the laptop still takes forever for 30gb and loads of folder and files....

And thanks, BR.

eschwartz · 11-18-2015, 07:28 PM

The info that is supposed to be stored in "metadata_db_prefs_backup.json"?

It's "home" is in metatdata.db somewhere. Should be no problem at all, if you are willing to do a lot of spelunking with an SQL browser.

jecilop · 11-18-2015, 09:06 PM

Quote:

Originally Posted by eschwartz

The info that is supposed to be stored in "metadata_db_prefs_backup.json"?

It's "home" is in metatdata.db somewhere. Should be no problem at all, if you are willing to do a lot of spelunking with an SQL browser.

Ah, so it is that kind of file.

Spelunking can take you to all kinds of new discoveries

Thanks.

jecilop · 12-17-2015, 12:18 AM

Just noting this problem is solved as of now...best fix was to move all entries to a new library although I cheated a little with mashing some database stuff together, but all is good now.

10-14-2015, 01:00 AM	#1
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	HELP..Problems with permanent Column Views I'm posting inquiries here to see if anyone knows of this before submitting a bug. I cannot move my custom columns into a view position and have it be permanent. Every restart goes back to the view I've used for a long time. I do not use View Manager. The plug-in was installed, but I removed it to check if its presence affected this, but there was no change. I update regularly so cannot pinpoint a coinciding update and start of the problem. I've only just recently started making more adjustments to column positions. I am tired of re-positioning, re-sizing, and hiding these columns now. I use Caliber a lot, so it is significant. EXAMPLE: Columns Viewed are A,B,C,D,E,F,G * Move G and hide F such that A,B,C,D,G,E * Close then restart PC or just restart Calibre; it always opens to the original. I've tried this 2 ways, both of which worked in the past when I positioned my screen view: 1- The intuitive column drag to position and right click to hide, 2- The "Add custom columns" option where I use the up/down green arrows to move the column position, Tech Info including testing environment: Wdw 7 Calibre 2.40 No View Manager use or plug-in presence. I do have my customized Calibre settings in a custom location that backs up to the cloud; however, the cloud syncing program is always inactive during Calibre use to avoid corrupting files. Only basic background programs and virus software is running at the same time as these Calibre test. There is plenty of memory. Last edited by jecilop; 10-14-2015 at 01:01 AM. Reason: grammar

11-17-2015, 04:00 AM	#8
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	Ok, I went ahead with testing using my newer libraries to learn some of what you both posted while you were posting it :-) BUT...could you please educate me on the database. I've re-read your posts, but I'm still unclear on where this information is stored and then accessed each time the library opens. Specifically, where do the column view settings store? Is it the metadata file stored in the library? Per Kovid, the backup json is just for restoring. Also, what is preventing the backup json from writing when I exit this library? I noticed when looking at a test library that the backup file was either created or updated when I exited the test library. That is NOT happening in this one. The first time after deleting, opening Calibre, making changes, then closing, I noticed a file named something like metadata_db_journal or a variation thereof appeared briefly then disappeared. I didn't see it again after repeating the open, change, close. Still, the backup json isn't writing here also; it obviously cannot update if it's not writing. And still, I do not have any cloud syncing running during this. Anyone willing to help me better understand this? I did notice the metadata database updates when I make changes. I think that's probably a "duh" comment. If this stored the views, I don't understand why that isn't the only part not being saved when it updates. My cloud syncing only runs when Calibre isn't. This is how I backup my library. It can't be pulling an old file(s). I can make changes to the view, close or restart Calibre, even restart the PC...Go right back to the Calibre library I'd just edited....and the view changes are back to the old. There is NO CLOUD SYNCING run between this. My library check returns no errors, by the way. The deeper check of files and folders matching the database didn't have any major issues. I'm working on the debug mode to see what that returns. Last edited by jecilop; 11-17-2015 at 04:13 AM. Reason: accidental enter to post..continue typing

11-18-2015, 04:34 PM	#11
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	How to access the "active" prefs file...... Okay, I am doing some testing - testing that will also help me learn how Calibre works behind the scenes. Friday's update to come (2.45?) is supposed to include the fix to the custom logic column copy per a reply I received. That would allow me to copy everything to a new library, but that would still require a massive upload to back the files up...MASSIVE on std to slow internet mean effectively never fully backed up. While this would solve the database and thus column view issues, I would like to avoid this step if possible. I'd really like to know where the custom preference info such as custom color columns, custom icon rules, customer searches, and virtual libraries is stored. Specifically, I'm looking for a file that I can open in it's raw or code form an manipulate if desired (such as copy/paste from old to new). I've done extensive changes to these since my April'15 prefs backup json, so the info there is only a partial help. Where can I access that active file to attempt to get that info from the current information that IS updating and staying when I add new stuff? I've done this type of things with Tag Mapper when doing A LOT of stuff because sometimes interacting with plug-ins is very tedious and time consuming. It would be that way to "fix" a new library on a rebuild I think. Still, I'm trying to run some more tests to learn about the last, but copying the entire old library to a new folder on the laptop still takes forever for 30gb and loads of folder and files.... And thanks, BR.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
problems with custom date column	Crueltyfre	Library Management	4	12-31-2014 05:20 PM
0.9.44 problems: doesn't remember window size/position; no On Device column	meeera	Calibre	13	08-17-2013 09:26 AM
Using "Build column from other column" custom-column type	HFC3	Library Management	3	07-07-2013 03:11 AM
Views on the DX?	bobulbous	Amazon Kindle	15	06-15-2010 02:34 PM
Seriously thoughtful Views on IE8?	HarryT	Lounge	41	06-01-2009 10:54 AM

10-14-2015, 03:34 AM	#2
BetterRed null operator (he/him) Posts: 21,660 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	@jecilop - column position, alignment, show/hide etc working fine here on calibre 2.40, Windows 10 64 bit. Book List relative column positions, alignments, widths etc are kept in the database not the configuration data. They can be seen in the metadata_db_prefs_backup.json file in the library, look for "library_view books view state" - best viewed with 'format aware' text editor like Notepad++, TextPad, Text Wrangler etc. I suggest you create a small test library and see what happens with it. Monitor the changes in metadata_db_prefs_backup.json. I can't recall if that file changes dynamically or when calibre closes. If you forget to pause cloud syncing before starting calibre, even just once, then that could result in database corruption, my understanding that calibre sources the book list layout data from the database. That .json is a backup for use by the calibedb --restore_database command BR

10-14-2015, 03:42 AM	#3
BetterRed null operator (he/him) Posts: 21,660 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	Addendum : jecilop - if you copy a few books into a fresh library that includes your custom columns and if the book list layout options are OK there, then move all of the books into the fresh library. Scrap the old library, assuming of course that it's now empty. BR

10-14-2015, 10:56 PM	#4
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	When I have a little time, I might play with this suggestion. Right now the idea of waiting for my internet to re-upload my massive library when copied gto a new locations is more than daunting with a basic level cable internet. It took weeks to upload everthing the last time....weeks! I will, however, read through your technical notes again (and again) so I understand them. First easy test, however, will be to create a new library with the custom columns filled (ie, copy from the main) and see if I can change the view the intuitive ways on that one and have it stay. Thanks.

11-17-2015, 01:37 AM	#6
kovidgoyal creator of calibre Posts: 45,251 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You can always delete the json files, they are there purely as a backup. And if you find they are not being updated/re-created, then run calibre in debug mode and you might get more info on why. Though, if the reason is that you are using some sync software that is rolling back changes, then debug mode will not help.

11-17-2015, 01:57 AM	#7
BetterRed null operator (he/him) Posts: 21,660 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	@jecilop - I responded to your other thread on 'logical columns not being copied' The metadata_db_prefs_backup.json file is NOT used at run time, it only comes into play if you do a database restore. So, if you want to 'impose' the preferences from library B on library A, then rename metadata_db_prefs_backup.json in library A to say .json.save, then copy the metadata_db_prefs_backup.json* from library B to library A, and then do a database restore on A. Oh yes, make sure you have backups of your libraries before you start BR

11-17-2015, 04:28 AM	#9
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	OK...this is what the debug file returned after making the column view changes and position changes and closing... ***************..... Traceback (most recent call last): File "site-packages\calibre\gui2\ui.py", line 948, in closeEvent File "site-packages\calibre\gui2\ui.py", line 882, in shutdown File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock File "site-packages\calibre\db\cache.py", line 587, in set_pref File "site-packages\calibre\db\backend.py", line 126, in set File "site-packages\calibre\db\backend.py", line 115, in __setitem__ File "site-packages\calibre\db\backend.py", line 806, in execute File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 231, in resetcursor CorruptError: CorruptError: database disk image is malformed *************************** So, disk image...hmmm...is there anything I can do about this. Perhaps, can I use the old backup json and run it, or will that wipe all I've done since that file was last updated (new columns, new library entries, new tags, tag changes, etc....)?

11-17-2015, 04:55 AM	#10
BetterRed null operator (he/him) Posts: 21,660 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	I suggest you restore (rebulld) the database - it's in the Library Maintenance menu, it runs in two phases - the first phase is usually quite quick, but the second phase can take a while if its a big library. You may lose preferences you've defined since April, but you probably won't lose book metadata because that's restored from the metadata.opf files in the book folders. The library specific preference data - (custom columns, view manager, reading list, virtual library definitions) is contained in table called 'preferences', there's not much point in trying to look at the table in the database itself - more useful to look at the backup json file. BR

11-18-2015, 07:28 PM	#12
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	The info that is supposed to be stored in "metadata_db_prefs_backup.json"? It's "home" is in metatdata.db somewhere. Should be no problem at all, if you are willing to do a lot of spelunking with an SQL browser.

12-17-2015, 12:18 AM	#14
jecilop Addict Posts: 260 Karma: 139980 Join Date: Mar 2014 Device: Android	Just noting this problem is solved as of now...best fix was to move all entries to a new library although I cheated a little with mashing some database stuff together, but all is good now.

Advert

Advert