Calibre Freezing with Adding Books and Saving to Disk

[jamesgl]

Good morning BR and , cybmole

https://www.dropbox.com/sh/6gl1jyutm...vzoUcjQia?dl=0

The above link goes to the zip file that contains the executable file itself, the help file, and the license statement for autoruns.

Autoruns delineates every thing that is loaded when Windows starts. It gives you the option of searching through the entire list - which may be incredibly lengthy - using control-F for find, and function key three in order to continue looking for more iterations of your search.

Un-Checking the box on the extreme left hand side prevents that particular application from loading when Windows next starts.

If you use this application, be certain of exactly what you are un-checking.

It is entirely possible to uncheck an application that is actually mandatory for Windows to run.
This results in Windows not being able to run and sometimes the problem not being fixed with System File Checker .

By the way, I just finished running Windows System File Checker [through an administrative command prompt window ] just to be sure, and no errors were found.

Okay… Next I created a simple Windows text file, opened up the sixty-four bit version of Calibre, and dropped the file onto a book. The program immediately froze and the usual window came up. Interestingly enough, the files appeared on my desktop, which was theoretically immediately underneath the Calibre 64-bit window. They were moved from their original location, to the underlying desktop.

I use total Commander in place of the Windows Explorer. Although that should not make any difference.

Now going to try creating the temporary directory and then create the environmental variable. Will report back when I have done this.

Okay… I followed the directions for creating system variables in Windows. I created a variable with the name of CALIBRE_TEMP_DIR

I made the variable value for that
C:\CalTemp

And then I also separately tried using the variable value
C:\CalTemp\

[I did it both ways, since I was not sure whether or not you needed a trailing…\… In order to properly represent the directory] Both methods yielded the same result. I did not do this in the safe mode.

I don't know if I have to tell Calibre somehow what to set the value for CALIBRE_TEMP_DIR within the program itself. Or does Calibre simply look to existing variables that preexist in Windows?

I guess I'm not quite understanding if I need to tell the Calibre 64-bit program anything specific about this particular variable. I don't seem to find the CALIBRE_TEMP_DIR variable when I search for it under tweaks.

In any event, the 64-bit program locked up per usual.

I will await further instructions on this.


Okay… cybmole… Thank you very much for making the suggestions! Here is how they have worked for me.

Your first suggestion – and by the way a very good suggestion – About going to a prior restore… I cannot do this easily. the change probably occurred at some point over a three – to – six month period and I was not – during those somewhere between three and six months – doing anything with the 64-bit version other than reading existing books, with the beautiful internal reader.

Therefore I have no idea when the conflicting program or condition arose.

My computer is set to do an automatic backup to a separate hard drive at 2 AM every morning on a daily basis. There are well over a hundred backups over that three-month period. I have no idea which backup if any, would work. It may well be that the problem occurred five months ago.

If there is no other way, then I will try going back six months, doing a complete system restore, and seeing whether or not Calibre sixty-four bit runs normally. I'm dragging my feet with that because then I have to restore email databases etc.

I have run ccleaner a couple of times since the problem occurred, but there was no difference in performance.

As far as doing a clean install of the 64-bit Calibre, I have done exactly that. I renamed my existing Calibre library to something else. Next I uninstalled the 64-bit version of Calibre using the standard Windows uninstall, then did a clean install of the 64-bit version, and when I attempt to add books to the now empty library, the program locks up.

This occurs by the way in BOTH the safe mode, and the normal Windows mode.

I uninstalled the Logitech driver for keyboard and mouse, but since it did not seem to make a difference I'm planning on reinstalling them at some point later on today.

I am not running roboform.

I removed SUPERAntiSpyware from the system completely using the normal Windows uninstall. This program was running earlier when the problem first occurred but has since been uninstalled.

I can randomly create, rename, delete, and manipulate folders at will. In fact this is done on a daily basis using Photoshop, Lightroom, and Bridge.

Many, many thanks for taking the time to read this and make suggestions!

jamesgl

[jamesgl]

duplicate posting – content deleted – sorry.

[cybmole]

quick note on clean install- the windows uninstall does not remove calibre's app data folder(s) so if the problem is with something in those...

something like Revo uninstaller will do additional searches for related files / folders or you can manually locate and remove what is in app data.

the app data stuff is , for me, the most likely culprit given your reported facts.

it seems that every program on your PC is working just fine,except for calibre, so i't s unlikely to be some other program causing this issue. I would, in your shoes though, do 2 x through malware checks, e.g. run both trend micro house calls and malwarebytes scans, just to be sure that the PC is clean.

and run microsoft disc cleanup ( chkdsk ) on its most thorough run-at-startup settings, in case you have orphan file fragments or bad disc sectors that are somehow causing this- my computer: right click on C:\ drive -> properties -> tools- error checking...

the suggestion with Ccleaner is that you go to tools -> startup -> inspect all 4 tabs for any strange program or task running at start up (& google anything dubious)

i'd also try having calibre place its library on a separate hard drive ( or on a USB stick if you don't have a 2nd drive) & see if you get the same behavior. if so , then it is not an issue with your specific hard drive file allocation tables.

it's all about patiently running overlapping tests so that you can eliminate suspects....

[BetterRed]

@jamesgl - I'm putting each idea in its own spoiler because it was rather long, I hope it helps rather than hinders

AutoRuns

Spoiler:

The above link goes to the zip file that contains ...

James, I did not want autoruns itself, I wanted to see what you see.

Run it, make sure Options->Filter Options->Hide Windows entries is checked. Then do a File->Save, it will offer to save AutoRuns.arn, save it as James.arn, upload to dropbox and post a link - then we can see what you're starting at boot time and make suggestions regarding anything 'suspicious'.

==============================
Drop Text File into a Book

Spoiler:

Okay… Next I created a simple Windows text file, opened up the sixty-four bit version of Calibre, and dropped the file onto a book. The program immediately froze and the usual window came up.

What usual window, I think of a 'frozen' program as one whose window goes milky with Not Responding in the Window title bar. Did you drag drop to the Book Details or Book List? The following image indicates what I hoped you'd do.



If you create the Text file on the desktop and drag/drop from there that will (hopefully) take Total Commander out of the equation. Do it in 32bit so that you have a benchmark - there shouldn't be any windows popping up.

BTW you can delete a format for a book by right clicking it in Book Details and selecting Delete XXXX format.

Interestingly enough, the files appeared on my desktop, which was theoretically immediately underneath the Calibre 64-bit window. They were moved from their original location, to the underlying desktop.

That is truly, truly bizarre - but potentially important. I'll assume that you drag/dropped the TXT file into the Book List, by 'usual window' do you mean this one ?



If not that, then 'what usual window'? Can you capture it and post it. To post an image here, click on Manage Attachments in Advanced Mode (scroll down), Browse to image and Upload, once its uploaded then Close Window (top right corner).

==============================
Total Commander

Spoiler:

I use Total Commander in place of the Windows Explorer. Although that should not make any difference.

It shouldn't, but... Does TC have a 'mode' where it "allegedly replaces Windows Explorer". My file manager, xplorer², (X2) has that option, but I avoid it like the plague. Directory Opus (Dopus) is another one that has it. I have seen both X2 and Dopus create problems when the replace Windows Explorer option is used.

==============================
Temporary Directory

Spoiler:

Now going to try creating the temporary directory and then create the environmental variable. Will report back when I have done this. Okay… I followed the directions for creating system variables in Windows. I created a variable with the name of CALIBRE_TEMP_DIR

<snip>

In any event, the 64-bit program locked up per usual. I will await further instructions on this.

That all seems fine to me.

Calibre looks for the CALIBRE_TEMP_DIR environment variable and uses it if its set, otherwise it uses the standard Windows Temp location. IIRC some of the previous 'inexplicable' problems that have been solved by using 32bit rather than 64 bit, were manifesting as permission problems on the Temp folder - which is also bizarre. Here's what I have



When you run calibre you should see a folder in the CalTemp folder with a randomly weird name (e.g. calibre_ya8u_p) - it may or may not have content, and it should get deleted when calibre exits normally. It probably won't get deleted when calibre crashes. Might be an idea to move it somewhere 'safe' in case we want to have a look at it, it might have some clues.

==============================
Manually Create Folders

Spoiler:

Did you try doing this

Originally Posted by BetterRed

....go into the library folder and create an author folder (eg ZZZ Author) if that works create folder in there (eg ZZZ Book), if that works copy a couple of files into the ZZZ Book folder. That's essentially what happens when you add a new book for a new author - if it works then ??? then how come calibre has a problem. If it doesn't work then at least we'll know its not a specific problem with calibre. Don't forget to delete XXX Author etc.

==============================
System Restore

Spoiler:

Your [cybmole's] first suggestion – About going to a prior restore… I cannot do this easily.

<snip>

I'm dragging my feet with that because then I have to restore email databases etc.

Given all that then IMO you would probably be better off to try an Inline Reinstall of Windows, it will replace Windows whilst leaving all your data and and non Windows software in place.

If that didn't work then I'd backup all my data, save or take screen shots of applications settings and make sure I knew how & where to install everything and then reinstall Windows to a freshly formatted disk.

I've found that doing a restore to a long-time-ago point in time (eg several months) on someone's system, results in the owner spending the next six months gradually getting the system back to as it was as they find things have gone missing or are mis-configured. And they will blame me for the inconvenience having to do so.

But in this case, replacing windows inline or in its entirety are last resort options. You have a working solution with the 32 bit version, and if push comes to shove there's the calibre portable option.

==============================
Configuration Data

Spoiler:

cybmole mentioned another possibility- the configuration data (appdata) - this is worth checking out.

But cybmole suggests you remove config data with the Revo Uninstaller, the problem with doing that is that the 32bit and the 64bit versions of calibre share the same configuration data, Revo will trash it. You have no problems with 32bit, but without the current configuration data it would be potentially 'crippled'. Hence I suggest you do the following.

You can get to the configuration data via Preferences->Miscellaneous->Open calibre configuration directory it should open in Windows Explorer, leave the window open and exit calibre**.

Then in that Explorer window, go up a level (select first item and hit backspace). You should be at somewhere like 'C:\Users\James\AppData\Roaming', with the folder 'calibre' selected, rename it to 'calibre_save'.

Now start calibre 64 bit, it will take you through the Welcome Wizard, which creates a fresh 'C:\Users\James\AppData\Roaming\calibre' folder with default contents. If your library is not in the default location with the default name (I think that's something like "C:\Users\James\Documents\Calibre Library") then you won't see it, but no matter, you can do a test with the default library the Wizard created - just try adding a book to it in your normal way.

If it doesn't crash, then something in the configuration data (in calibre_save) is causing the 64 bit version to crash but not the 32 bit - which is very strange.

If 64 bit crashes then we have eliminated the config data as a possible cause.

Delete the 'C:\Users\James\AppData\Roaming\calibre' the Wizard created, and rename the 'calibre_save' folder back to 'calibre' - that will re-establish the existing configuration which the 32bit version can use.

==============================
Close all other programs

Spoiler:

Have you tried closing all other programs, I don't mean uninstall, just close them all normally. For applications in the tray right click on the icon will expose an Exit option, if there is one. Also go into Tray Items Customise and make sure there are no items with the Hide Icon and Notification option - I would suggest they be changed to Only Show Notifications, exit these too.

Go into Task Manager - calibre should be the only thing in Applications, and in Processes (with the Show Processes from all users unchecked - bottom left corner) you should not see much - if you take a screen shot and post it I'll have a look to see if there's anything 'suspicious'.

BTW I run PS and LR alongside calibre with no issue. Not Breeze though, not even sure what it does these days.

Is your camera tethered to your PC via USB or WiFi? As a matter of principle I wont run calibre if I have a camera attached - no particular reason, just that camera's are no good for reading books

==============================
Ignore Plugins

Spoiler:

There's another option we haven't tried, which is to run calibre 64 bit with no optional PI's. You can do this by running calibre with the --ignore-plugins switch.

Open a console window, click Start, click Run. enter cmd into its Open box, and hit enter. In the console window type "C:\Program Files\Calibre2\calibre.exe" --ignore-plugins and hit enter. NB the quotes are required.

Try adding a Book.

==============================
** Replacing Windows Explorer

Spoiler:

If Prefs->Misc->Open config it doesn't open in a Windows (File) Explorer window looking something like this



then that would suggest that you've used an option in Total Commander that TC replace Windows (File) Explorer.

Given your problem appears to be related to creating folders and files can you do whatever is necessary to turn that feature off. I can't imagine why that could affect the 64bit and not the 32 bit version of calibre, but Windows Explorer is so fundamental I think its best not to tinker with it too much - IMO augmenting WE is fine, but one should avoid subverting it and trying to pretend it doesn't exist.

BR

[jamesgl]

good morning BR,

First, thank you, thank you, thank you, for taking the time to reply and organizing your reply beautifully.

It's going to take me a day or so before I can go through this list and give you the results. The reason for the delay is that I'm processing photographs from Christmas, and need to get them up in the next day or so.

I hope you had a very, very Merry Christmas, and I hope your new year will be a happy one.

We'll get back to you as soon as I have results.

Again many thanks.

jamesgl

[jamesgl]

Good morning again BR it's a little after midnight, and what you read below has taken a couple of hours to accomplish.

I do not have a way of posting the screenshots online so you can view them in this forum.
However… My email address is:
incoming [at] POBoX.com

If you send me an email, I will reply with the text below plus the screenshots from my computer. I composed the entire thing as an RTF file, which would email nicely, but will not allow me to post the screenshots embedded in the RTF file, in here. Everything else is indicated in text.

For item number two, I opened the 32-bit version of Calibre, and dropped the text file into the area you indicated. The files were appended to the book and when I clicked on text, my text editor opened and I was able to read the file. When I tried the exact same thing in the 64-bit version, the program locked up and I got the great out window that you described as being normal when the program stops working.

For item number three, it does not have a mode which allegedly replaces Windows Explorer. You can always use Windows Explorer and total commander simultaneously.
In fact you can drag-and-drop files between Windows Explorer and total commander.

Total commander operates as a standalone program that gives you infinitely more flexibility, the least of which is having to Windows as being part of it.
Anyway… When I use Windows Explorer in order to drop the text file into the 64-bit program, it locks up. The 32-bit version seems to work flawlessly.

This is a screenshot from item number four - on my computer using the standard Windows environmental editor which admittedly is a bit weak.
But I think you can see the necessary information.

Also, I do get directories created in that CALIBRE_TEMP directory when the 64-bit and/or the 32-bit version start. I have not looked at them in detail.
I checked the couple of times after the directory and variable were created. There's always stuff in there. Provided that you have run either versions of Calibre.
And by the way this stuff does not disappear when you close the programs. Let me amend that.
The directories don't disappear but I'm not certain there is always material left in the directories, when the programs are closed.

Okay… Next is item number five. The short answer is yes I can create folders, and books. I'm going to post a screenshot from total commander

The window on the left shows the sorted author directories [sorted by name] and the bottom one you will note is zzz Author make-believe folder.
The right window shows you the actual text file that I created which is called Make-Believe Book.txt
and finally… The insert window is the Lister which shows the contents - only one sentence fragment that actually is the entire text file.

As far as the configuration data suggestion goes, we did this. You suggested that I zip up the entire configuration data directory which was located in user/my name/AppData/roaming/documents/Calibre etc. I did zip up that file and copied the zip file to a separate hard drive. I deleted the directory in its entirety. Then I reinstalled Calibre 32-bit version, went through the wizard, and it worked normally. Not knowing any better, I uninstalled the 32-bit version, re-deleted the orphan configuration data directory, and then reinstalled the 64-bit version of Calibre. I did go through the wizard again as you noted I would. However the 64-bit version still locked up. So I thought that eliminated the configuration data directory as being the problem. By the way I chose - in both cases the 32-bit and the 64-bit re-install - a new vacant directory to be the library - that had zero books in it.

I then attempted to close every running program that I could. That had no effect. I right clicked every small icon pack was open and active, and exited where possible. I was unable to close Avast anti-viral. And there are something like seventy-odd process is still running in Windows task manager so I'm sure it's one of them that must be interfering. :-(

As far as the ignore plug-ins go, I took a screenshot of the command window that I used to invoke Calibre 64-bit. It still froze up. Here is the screenshot.

It did start normally, but it simply froze up when I clicked on add a book and then told it to add a book expecting that there would be one book in each directory etc.

And finally, preferences/miscellaneous/open config in the 64-bit works normally. Here is the screenshot:

It's the 64-bit version – with the preferences window opened up, and you can see the Windows Explorer indicating the location of the config information files.

Going to head for bed. But… I want to take the opportunity to thank you very, very, very much for spending the time and putting in the effort that you have so far!

I sincerely wish you a very happy new year.

I'm certain that if I did a new Windows install, that would aabsolutely take hours and hours to complete, the 64-bit version of Calibre would run normally.

At this point I'm leaning towards using the 32-bit version and simply switching to the 64-bit version if there any heavy-duty book conversions – which I honestly don't envision my having to do.

Once again, many, many thanks for all your time and effort!

Warmest regards,

Jim.

[cybmole]

i am wondering if the lock up is at the point where calibre needs to display a pop-up window to say "adding books", and maybe it's the action of creating the window that is at the heart of this, not the underlying file/folder activity.

which would point at some issue with video drivers ?
what is the relevant part of your set up ?

there is no way to say in calibre don't display the pop up window so it's back to laborious compare of 32 & 64 bit to see if this idea has any legs.

one other thing that I don't think has been tried yet is booting in safe mode & repeating the various tests. AFAIK, safe mode will bypass your usual video driver if that's an nvidia / AMD and substitute a basic microsoft driver

and , scraping the barrel now because BR has already listed the most likely tests: create a new windows user, try calibre 64 when logged on as that user. do that for both a new admin user and a normal one with no admin rights

[BetterRed]

@jamesgl - I sent you an email

AutoRuns

If you're minded email me the ARN

Drop Text File into a Book (Details)

This is OK on 32bit and not OK in 64bit - that's what I expected

Total Commander

Red Herring - shouldn't cause problems, besides same behaviour if file is drag 'n' dropped from Windows Explorer - works in 32bit but not in 64bit

Temporary Directory

Same behaviour 32bit OK, 64 bit crash - so its not a Temp permission error masquerading as something else

Manually Create Folders

Pseudo author and book folders and files can be created - eliminates the possibility of a corrupt MFT, which was a highly unlikely cause - NTFS is NTFS, is NTFS no matter what the bittedness of an application

System Restore

To repeat : replacing windows inline or in its entirety are last resort options. You have a working solution with the 32 bit version, and if push comes to shove there's the calibre portable option.

Configuration Data

This had already been eliminated - also I can't recall seeing an instance of 64bit crashing that was fixed by resetting the config data back to defaults. Occasionally there are glitches that can be fixed by editing or deleting a JSON file, but I can't recall a freeze/wedge situation - but even my memory sometimes gets parity errors

Close all other programs

60 processes is about right for the 'minimal set' if you include the Windows processes

There's a check box at the bottom left of Task Manager Show processes for all users. if you turn that off then after Exiting all programs there shouldn't be much, some applications have 'background' processes, eg Evernote Clipper, after terminating them I have 15 processes (bluetooth, video adapter, explorer, disk health monitor, AV etc).

Your tests to date, and the fact the 64bit fails in Safe Mode makes me doubt it is a 3rd party program

Ignore Plugins

Well it looks like its not a plugin.

** Replacing Windows Explorer

This is a non-issue with Total Commander - its probably a non-issue with X2 and Dopus.

=============================================

@james - can you confirm that conversions are OK with 64bit ?

There's one other thing you might like to try. There's an Environment Variable, CALIBRE_NO_NATIVE_FILEDIALOGS, that if set to 1 will cause calibre NOT to use the native (Windows) file open/save dialogues (it will use some built in dialogues). See Customizing calibre — Environment Variables

Happy New Year - BR

[BetterRed]

Originally Posted by cybmole

one other thing that I don't think has been tried yet is booting in safe mode & repeating the various tests. AFAIK, safe mode will bypass your usual video driver if that's an nvidia / AMD and substitute a basic microsoft driver

@cybmole - see post #8

Originally Posted by jamesgl

I started Windows in the SAFE MODE.

The 64-bit version did not run – actually crashed – and disappeared – in Windows safe mode. By disappeared, I mean the program opens normally, and when I try to add a book, and I choose "add book from single directory" – either one book per directory or multiple books per directory – either one of those choices – the program simply seems to stop running, and then the window literally disappears.
In other words, it remains inactive for probably five or ten seconds, and then the entire window disappears in the safe mode.

The 32-bit version seems to run completely NORMALLY in the safe mode. The add book second and third choices produce an Explorer window in order for you to choose a directory. No problems.

Originally Posted by cybmole

and , scraping the barrel now because BR has already listed the most likely tests: create a new windows user, try calibre 64 when logged on as that user. do that for both a new admin user and a normal one with no admin rights

That's worth a shot, a new user will get a fresh home directory, AppData. Desktop etc.

I can't see why video drivers would only effect some dialogues related to some file i/o operations, but stranger things have happened. I wonder if load plugin from file creates the problem?

BTW thanks for helping out on this one. I suspect like me, you find these these occasional 64 bit failing reports 'irritating'.

BR

[cybmole]

sorry, I missed that safe ode had been tested once already - but safe mode + having the task manager window open would tell us if the program "disappears" or actually closes itself. if ask manager says it is still running but you can't see it / tab to it then that's new info.

if I open my copy of 64 bit calibre and drag + drop a book i.e. add book, the very first think I see is the add book pop up information window. . I think calibre creates that window before creating any files or folders.
So when the OP does that , the program crashes before that window appears ? suggesting that crash is when calibre calls whatever asks windows to create a new window: in safe mode its a quiet fail, in normal mode it is a more obvious fail, but it's always at that point ?

have we considered whether any non standard system fonts or font sizes are in use ?. I distantly recall having some game controller software which crashed on my PC but not on the authors, & we eventually traced that to me running with 125% font sizing for desktop.

[jamesgl]

Hello again.

In the normal windows operating mode, the 64-bit Calibre seems to convert books entirely normally. I took an EPUB book and converted it to Mobi.
Then I opened both the EPUB and the Mobi versions up and went in the middle of the EPUB version book and selected some EPUB text.
I search for that in the Moby reader.
And all seem to work normally.
The paragraphs and everything else seemed to be perfectly okay.

So conversion is normal in the 64-bit.

*********************************************

Next up was the variable that you recommended I set to a value of 1.

CALIBRE_NO_NATIVE_FILEDIALOGS

I set this variable to a value of 1.

The 64-bit program still froze in the same way when I attempted to add books.
The 32-bit version worked normally.

By the way – I assumed that it was not necessary to restart Windows in order to get the environmental variable operational. In other words as soon as you complete the edit it's done. That may not be the case. Please let me know if I need to restart Windows after changing and environmental variable. This is beyond my expertise.

jamesgl

ps I restarted windows, and the 64bit still freezes with the env varib set as show above.

[PeterT]

When you set the environment variable and start calibre, how exactly are you doing it?

Setting the variable from a command prompt, and then starting calibre from the Desktop won't work; the environment variable set is local to that command prompt (and to any programs started from that command prompt).

You need to launch calibre by typing calibre into the command prompt where you have just set the environment variable.

[BetterRed]

Originally Posted by PeterT

When you set the environment variable and start calibre, how exactly are you doing it?

Setting the variable from a command prompt, and then starting calibre from the Desktop won't work; the environment variable set is local to that command prompt (and to any programs started from that command prompt).

You need to launch calibre by typing calibre into the command prompt where you have just set the environment variable.

I suspect the OP is using Control Panel->System->Advanced->Environment Variables which sets them in the registry so they persist, from an earlier post by James

Originally Posted by jamesgl

"on my computer using the standard Windows environmental editor which admittedly is a bit weak. "

James if you are doing it at the command line, and you want to make the setting persist then use this command SetX - Set environment variables

Can you confirm that when CALIBRE_NO_NATIVE_FILEDIALOGS is set to 1 that in the 32bit (and the 64 bit) the file dialogues look like this:



rather than this:



BR

[kovidgoyal]

Did you miss my post, before about the last point in the FAQ, which deals with this precise issue -- namely the File Open dialog causing a crash?

There are as, the FAQ notes, two techniques for working around it, uninstall SpiderOak (if you have it installed) or use the environment variable, as BetterRed explained.

[cybmole]

Originally Posted by jamesgl

Hello again.

In the normal windows operating mode, the 64-bit Calibre seems to convert books entirely normally. I took an EPUB book and converted it to Mobi.
Then I opened both the EPUB and the Mobi versions up and went in the middle of the EPUB version book and selected some EPUB text.
I search for that in the Moby reader.
And all seem to work normally.
The paragraphs and everything else seemed to be perfectly okay.

So conversion is normal in the 64-bit.

*********************************************

...

this is consistent with my last hypotheses, that only the operations which require calibre to pop up a window are causing the crash