Calibre taking 30+ minutes to start with 20K book library

[kovidgoyal]

Move it off onedrive and you will be fine. Be as suspicious as you like, not going to change the fact that onedrive and calibre DO NOT get along. Windows has mandatory file locking, onedrive is scanning files inside caltrash which prevents calibre from deleting those files. calibre waits some time and retries each file delete to workaround this, hence the slow startup. Again, MOVE YOUR LIBRARY OFF onedrive.

[JSWolf]

Originally Posted by Luna

I found something else:

/Applications/calibre.app/Contents/MacOS/calibre (through Terminal):
Fast startup (2-3 seconds)

Open /Applications/calibre.app (through Finder or launching via Spotlight/Alfred):
Slow startup (30+ minutes)

The app bundle launcher clearly behaves differently than the direct executable. Is there an explanation for this?

Delete Spotlight/Alfred and try again via the MacOS GUI and see if calibre opens properly.

[JSWolf]

Originally Posted by OokOok

So in my case I ran in debug mode. When I get slow startup, I see all the startup delay is accompanied by spewage of error messages claiming access errors in .caltrash - and it simply takes time to get through all the entries. HOWEVER, . I will note that I've storage my library on OneDrive for (at least) four years w/o issue, so to me this isn't obviously a case of "you can't store on OneDrive" and I'll be suspicious of any workaround that implies this w/o also explaining why it has always worked up until sometime in the last 90 days.

Nevertheless, I'll try excluding my AV from looking at .caltrash to see if there's a relationship. I also note that the folders in .caltrash are all empty - which I'd expect, because I typically use the GUI to empty the trash. So I might, at some point, manually delete the contents of .caltrash to see if that also prevents long startup.

Traceback (most recent call last):
File "calibre\db\backend.py", line 449, in rmtree_with_retry
File "shutil.py", line 787, in rmtree
File "shutil.py", line 638, in _rmtree_unsafe
File "shutil.py", line 636, in _rmtree_unsafe
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dstei\\OneDrive\\Books-CalibreLibrary\\.caltrash\\b\\2295'

During handling of the above exception, another exception occurred:

Spoiler:

Traceback (most recent call last):
File "calibre\db\backend.py", line 2243, in expire_old_trash
File "calibre\db\backend.py", line 455, in rmtree_with_retry
File "shutil.py", line 787, in rmtree
File "shutil.py", line 638, in _rmtree_unsafe
File "shutil.py", line 636, in _rmtree_unsafe
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dstei\\OneDrive\\Books-CalibreLibrary\\.caltrash\\b\\2295'
Traceback (most recent call last):
File "calibre\db\backend.py", line 449, in rmtree_with_retry
File "shutil.py", line 787, in rmtree
File "shutil.py", line 638, in _rmtree_unsafe
File "shutil.py", line 636, in _rmtree_unsafe
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dstei\\OneDrive\\Books-CalibreLibrary\\.caltrash\\b\\2298'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\db\backend.py", line 2243, in expire_old_trash
File "calibre\db\backend.py", line 455, in rmtree_with_retry
File "shutil.py", line 787, in rmtree
File "shutil.py", line 638, in _rmtree_unsafe
File "shutil.py", line 636, in _rmtree_unsafe
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dstei\\OneDrive\\Books-CalibreLibrary\\.caltrash\\b\\2298'
Traceback (most recent call last):
File "calibre\db\backend.py", line 449, in rmtree_with_retry
File "shutil.py", line 787, in rmtree
File "shutil.py", line 638, in _rmtree_unsafe
File "shutil.py", line 636, in _rmtree_unsafe
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dstei\\OneDrive\\Books-CalibreLibrary\\.caltrash\\b\\2304'

Move your calibre libraries out of users. You don't want any libraries in there.

[Sirtel]

Originally Posted by JSWolf

Move your calibre libraries out of users. You don't want any libraries in there.

Why? All my libraries have always been in Users. True, I don't use OneDrive, if that's the reason.

[JSWolf]

Originally Posted by Sirtel

Why? All my libraries have always been in Users. True, I don't use OneDrive, if that's the reason.

The reason is that most permission errors are because of libraries in users. So it's just easier to create Calibre Library in the root of C and put all the libraries in there. Sure it can be because of OneDrive. But whatever the problem, it's users that's the problem.

I've removed OneDrive from my computers. I don't like what it does. I still use OneDrive in a way that works for me. I use it with rclone and winfsp to use OneDrive like it was an external harddrive. Works very well and I don't get the awful two-way sync.

[theducks]

I am with Jon.
You are GIVING these site LEGAL access (read the TOU) to your data. That is why Free is a double Red flag. The money is from data mining.

Doubt it? I view a manufacturers site on my main PC and later go to my laptop.

I now see ads relevant to the site I visited. My MS accounts are NOT the same (and theoretically, I HAVE NOT LOGGED INTO THEM.)

Another 'no' reason for W11 is the mandatory account.

[JSWolf]

Originally Posted by theducks

I am with Jon.
You are GIVING these site LEGAL access (read the TOU) to your data. That is why Free is a double Red flag. The money is from data mining.

Doubt it? I view a manufacturers site on my main PC and later go to my laptop.

I now see ads relevant to the site I visited. My MS accounts are NOT the same (and theoretically, I HAVE NOT LOGGED INTO THEM.)

Another 'no' reason for W11 is the mandatory account.

I have Windows 11 Home on the laptop and Windows 11 Pro on the mini-PC and I have no adverts on either.

[theducks]

Originally Posted by JSWolf

I have Windows 11 Home on the laptop and Windows 11 Pro on the mini-PC and I have no adverts on either.

The adverts are in the Edge browser as you surf. (but I remember even if I used FF on one, and Edge/Explorer on the other, they still harvested my browse patterns.

[JSWolf]

Originally Posted by theducks

The adverts are in the Edge browser as you surf. (but I remember even if I used FF on one, and Edge/Explorer on the other, they still harvested my browse patterns.

So that's why I don't see the adverts. I only use Edge when I want a bare browser. Yes, I see the ads in Edge. I usually close it with one site open so I don't see the mess and when I open another tab, I just past in were I want to go and gone is the mess. So I don't really notice it.

[theducks]

Originally Posted by JSWolf

So that's why I don't see the adverts. I only use Edge when I want a bare browser. Yes, I see the ads in Edge. I usually close it with one site open so I don't see the mess and when I open another tab, I just past in were I want to go and gone is the mess. So I don't really notice it.

You seem to miss my point. It is not ADs. It is ADs that know what I browsed on other systems. Ads on the same system might be from cookies (many sites ban cookie blocking by not showing the content YOU wanted to see. Their site. Their rules as long as you are not paying for that denied service ).

[Sirtel]

I don't see any ads because I use browsers still allowing adblockers. As to data mining, can't say I really care. I use various Google and Microsoft services (although not their cloud drives - I have Dropbox for that, and no longer their browsers either) because they're convenient. Let them data mine me to their heart's content. I don't see their ads and any stuff they may gather about me is super boring.

[Sirtel]

Originally Posted by JSWolf

The reason is that most permission errors are because of libraries in users. So it's just easier to create Calibre Library in the root of C and put all the libraries in there. Sure it can be because of OneDrive. But whatever the problem, it's users that's the problem.

Well, I've never had a single problem on any of my computers (and there have been 6 or 7 of them) dating back to 2011 when I started to use calibre. But I'll keep it mind in case of any future issues. For now, they can stay where they are.

[theducks]

Originally Posted by Sirtel

Well, I've never had a single problem on any of my computers (and there have been 6 or 7 of them) dating back to 2011 when I started to use calibre. But I'll keep it mind in case of any future issues. For now, they can stay where they are.

FAT (MSDOS era) had a limit of the number of files in root.
We don't use FAT on windows system drives for a long time now.

[dunhill]

Combine several libraries into one to aggregate files and test it with a script on Linux Mint.
walter@walter-Cyborg-15-A12VE:~/Desktop$ python3 calibre.py
Exact opening time: 11.68 seconds
Number of books in the library: 180259
walter@walter-Cyborg-15-A12VE:~/Desktop$

[DNSB]

Just for the heck of it, I moved a copy my calibre library to a VM running Windows 11 26220.7271 fully updated. Updated calibre to 8.15, connected to the library copy and checked the startup time restarting in debug mode which clocked in at 16.23 seconds (16.9K books, 46.1GB) By comparison, on the PC itself, calibre displays Started up in 3.78 seconds with 16930 books. I then connected the library to OneDrive and allowed it to sync. After the sync, I started calibre and killed the process after 15 minutes. I felt that was long enough to say the OneDrive connection was the issue. I removed the calibre library from OneDrive syncing and the startup time dropped back to ~17 seconds.