Problems with large folders (with lots of files) such as book collection

[rebl]

Hello,
I was wondering anyone else experiences problems when trying to work with a very large folder (tens of thousands of files) in windows 8(.x).
I have just exported my Calibre library (but this is not relevant to Calibre itself) and the folder is over 20GB (about 24.000 sub-folders and 42.000 files of which 22.500 .opf files - small metadata files).
When I first performed a right-click on the folder, I had to wait such a long time (window was not responding) that I had to force close explorer. During this time, 100% I/O disk activity could be seen by explorer.exe, scanning all the subfolders (this could be seen in windows' Resource Monitor)
After several more tries, I managed to see the right click menu if waiting over 10 minutes.
I don't have many shell extensions, though I did tried in the past some shell extensions to show PDF properties in explorer. I've disabled all non-microsoft extensions (right click context menu handlers) using nirsoft shellexview, including the PDF shell extensions by Tracker Software but this did not speed the things considerably.
Yet, I was manage to get a right click menu after 10 minutes or so (this worked after disabling non-MS extensions) -- while before disabling them I had to restart explorer because it was not responding.
Could the PDF shell extensions cause any trouble (by Tracker Software, optionally installed with pdf x-change viwer)? - I doubt it, because those shoud bt needed for pdf files, not for folders.

I don't really understand why would need explorer.exe to "scan" all the files and subfolders of a folder, just in order to show a right click menu.
It's like... if I have a right click menu for WinRar (e.g. "add to archive"), it would show no matter what's inside the folder. What shell extension would need to "know" what's inside the folder? I can't think of any. Why would explorer need to scan the folder? In order to get the file properties eg count and size? But this should be done only when / if I click on the Properties entry of the menu...

I do have a slow hard drive (I suppose 5400 rpm, - it's on a laptop) but that should not matter. When I right click on a folder I should have a menu, not explorer counting folders and files or doing other uneeded operations.
The laptop has 3GB of ram, running windows 8.1 32-bit, and a dual core Intel processor (T4400 iirc).

The funny thing is that after a first wait of 10 minutes, any subsequent right clicks on the same folder took only 3-5 seconds - it's like the information was already "cached" and now I don't need to waite the whole 10 miuntes again. Also, even if I went inside the folder, deleted one file and renamed another, the right click time for the folder was still 3-5 seconds (I was afraid any change I would do would need the whole 10 minutes again).
I bet it's some kind of functionality of windows - those technobabble things like Volume Shadow Copy or USN Journaling (the partition is NTFS if that matters, I'm also using Everyting Search). But it still doesn't seem normal to take 10 minutes of explorer disk I/O just for right clicking a folder.

Any ideeas? Thank you.

L.E.: Two more things.
1. On a later try, it didn't took more than 1 minute to display the right click menu on another folder, as large as the first one (the initial Calibre library). So maybe I was just not waiting enough and loosing my patience too soon. This 1 minute try is indeed valid after I've disabled the non-ms shell extensions... hmmm. So maybe one of those was causing the problem?
2. If I manually enter the path in Everything Search, I can search inside the folder instantly, so it's not Everyting Search that is causing the problem. It's just that I am used to launch Everyting Search from a right click on a folder... and that wasn't possible before seing the right click menu, obviously. The stalling was definitelly caused by Explorer.exe, as I coudl see in Resource Monitor - Disk activity, where it was "scanning" all sub-folders in multiple threads.
Also, I've excluded the whole Calibre folder including the two big folders which are inside, from AVG's resident shield.

[HarryT]

I think it has to be something on your computer. My eBook library folder is a little larger than yours (29.3GB, with 24665 folders and 84328 files). It takes about 20s for Explorer to display the contents of the folder when I first open it, but a right-click is instant. As you say, it sounds as if you have some sort of shell extension installed which is scanning the folder when you right click, but I'm afraid I don't know what it might be.

I'm running 64-bit Windows 7 on a laptop with 4GB RAM, so I suppose it's conceivable it's something specific to Windows 8, but that doesn't seem very likely.

[GERGE]

Try to reset indexing.

[rebl]

Yes, I assume it might have been a shell extension - google drive or PDF x-change... maybe?
Of course those extensions inter-relate with explorer maybe that's why I was seeing explorer in resource monitor.
Now it takes 8 seconds to see the right click menu on the whole 60GB "Portable" folder containing some programs including calibre portable and both the library and the export folders...
Ha!... while using "Free disk analyzer" on partition D and Showman on partition C to see largest folders explorer just crashed and restarted . .. so typical of Windows 8!
A right click on the largest folder on C: (Users, 17GB) is instant.
A right click on another folder (13 GB) containg few large files is also istant.
Maybe the problem appears for folders with lots of small files - maybe if there are some properties handlers for them.

Originally Posted by GERGE

Try to reset indexing.

What do you mean?

[fjtorres]

Originally Posted by rebl

What do you mean?

The Windows search engine indexes your drive and folders to make content searches faster.

Depending on your settings, it (or an extension) may be trying to index the folder in question. If the computer has limited resources to work with , it might be bogging down.

One thing you can do is tell it not to index that specific folder, especially if it changes often or if you don't use Windows search there.

[kacir]

Originally Posted by rebl

Any ideeas? Thank you.

I personally use Total Commander when on Windows, not explorer. Lots of plugins, fantastic functionality. And ... I began my career working in DOS, using Norton Commander so I consider a two-panel file manager THE right way to do things ;-)
When I want to copy huge Calibre Library folder, or an archive with lots of photos on Windows I use program robocopy.exe
Robocopy is a utility provided by Microsoft - it is improved version of program called treecopy found in DOS and early versions of Windows.
Robocopy is a commandline utility and can considerably speed up copying of huge trees of folders. You can even resume copying or tell it to refresh your copy (not to copy over existing files if they are not older, for example). There are some graphical front-ends for robocopy, but they are not much of an improvement on the commandline.

I use Calibre on Linux and there I use Krusader as my file manager. Not those explorer-lookalike programs provided with KDE or Gnome or whatever Desktop environment.
For backup of my Calibre Library folders and other things that contain I use rsync. There is nice graphical front-end for rsync called grsync.

One possibility of how to drastically improve speed of such operations is to buy an SSD disk.

[theducks]

If you flatten a Calibre Library structure, you may end up with a huge number of files in a SINGLE folder. Calibre has 3 files for a Single Format Title (The format, cover, OPF)

The normal structure only has folders for a few thousand authors, those contain even less Per Folder

Your problem may be that the directory gets so big it will not fit into memory

[Katsunami]

I'm running Windows 7 x64 on a computer with 8GB of RAM.

I'm scanning my D:\Games folder at this very moment, while I'm posting this. It contains 66.7 GB, 203,122 files, and 46,170 folders. No problems...

[hapmas77]

Originally Posted by rebl

What do you mean?

- Right click on the folder. On General tab, click the Advanced button.
- Untick the box said 'Allow files in this folder to have contents indexed in addition to file properties'.
- click Ok.

[webroot]

I think you will be helped by installing a local solr based search engine and thereafter do not access the folder directly, let solr indexer run as a daemon in the background and do searching from its browser based interface, I guess not much tweaking possible from os level even I have trouble with my Linux too. Here is some starting point http://www.webwash.net/tutorials/how...ver-search-api

[rebl]

thank you all for ideas.
meanwhile I've switched from windows 8.1 to windows 7 and most of my problems (if not all) are gone.
right clicking the same huge folder takes about 2-3 seconds for the context menu to show (at first sight of the folder by the OS).
the second right click produces and instant context menu with zero delay.

I guess it was either windows 8's blame (my guess) or some shell extension or a combination of both.

I don't plan on working directly on the calibre library folder but sometimes I just want to check its properties of direclty search for a book without opening calibre (using everything search - in case I didn't specified: this is a search program diferent form window's search! my windows's search indexing usually get disabled within the few next days after a fresh install)
Using right click -> Search Everything it takes no longer than 1 second to find a book by file name and extension partial match.
If I do this within calibre I first have to wait for it to start (if it's not started) then every search takes a little longer (about 3-5 seconds; less if i filter by some criteria first)