MobileRead Forums - View Single Post - Problems with large folders (with lots of files) such as book collection

rebl · 05-26-2015, 04:35 AM

Hello,
I was wondering anyone else experiences problems when trying to work with a very large folder (tens of thousands of files) in windows 8(.x).
I have just exported my Calibre library (but this is not relevant to Calibre itself) and the folder is over 20GB (about 24.000 sub-folders and 42.000 files of which 22.500 .opf files - small metadata files).
When I first performed a right-click on the folder, I had to wait such a long time (window was not responding) that I had to force close explorer. During this time, 100% I/O disk activity could be seen by explorer.exe, scanning all the subfolders (this could be seen in windows' Resource Monitor)
After several more tries, I managed to see the right click menu if waiting over 10 minutes.
I don't have many shell extensions, though I did tried in the past some shell extensions to show PDF properties in explorer. I've disabled all non-microsoft extensions (right click context menu handlers) using nirsoft shellexview, including the PDF shell extensions by Tracker Software but this did not speed the things considerably.
Yet, I was manage to get a right click menu after 10 minutes or so (this worked after disabling non-MS extensions) -- while before disabling them I had to restart explorer because it was not responding.
Could the PDF shell extensions cause any trouble (by Tracker Software, optionally installed with pdf x-change viwer)? - I doubt it, because those shoud bt needed for pdf files, not for folders.

I don't really understand why would need explorer.exe to "scan" all the files and subfolders of a folder, just in order to show a right click menu.
It's like... if I have a right click menu for WinRar (e.g. "add to archive"), it would show no matter what's inside the folder. What shell extension would need to "know" what's inside the folder? I can't think of any. Why would explorer need to scan the folder? In order to get the file properties eg count and size? But this should be done only when / if I click on the Properties entry of the menu...

I do have a slow hard drive (I suppose 5400 rpm, - it's on a laptop) but that should not matter. When I right click on a folder I should have a menu, not explorer counting folders and files or doing other uneeded operations.
The laptop has 3GB of ram, running windows 8.1 32-bit, and a dual core Intel processor (T4400 iirc).

The funny thing is that after a first wait of 10 minutes, any subsequent right clicks on the same folder took only 3-5 seconds - it's like the information was already "cached" and now I don't need to waite the whole 10 miuntes again. Also, even if I went inside the folder, deleted one file and renamed another, the right click time for the folder was still 3-5 seconds (I was afraid any change I would do would need the whole 10 minutes again).
I bet it's some kind of functionality of windows - those technobabble things like Volume Shadow Copy or USN Journaling (the partition is NTFS if that matters, I'm also using Everyting Search). But it still doesn't seem normal to take 10 minutes of explorer disk I/O just for right clicking a folder.

Any ideeas? Thank you.

L.E.: Two more things.
1. On a later try, it didn't took more than 1 minute to display the right click menu on another folder, as large as the first one (the initial Calibre library). So maybe I was just not waiting enough and loosing my patience too soon. This 1 minute try is indeed valid after I've disabled the non-ms shell extensions... hmmm. So maybe one of those was causing the problem?
2. If I manually enter the path in Everything Search, I can search inside the folder instantly, so it's not Everyting Search that is causing the problem. It's just that I am used to launch Everyting Search from a right click on a folder... and that wasn't possible before seing the right click menu, obviously. The stalling was definitelly caused by Explorer.exe, as I coudl see in Resource Monitor - Disk activity, where it was "scanning" all sub-folders in multiple threads.
Also, I've excluded the whole Calibre folder including the two big folders which are inside, from AVG's resident shield.

05-26-2015, 04:35 AM	#1
rebl r.eads e.njoys b.ooks lol Posts: 76 Karma: 580748 Join Date: Mar 2010 Location: It's time to get this Book a Rest Device: Kindle 4 NT	Problems with large folders (with lots of files) such as book collection Hello, I was wondering anyone else experiences problems when trying to work with a very large folder (tens of thousands of files) in windows 8(.x). I have just exported my Calibre library (but this is not relevant to Calibre itself) and the folder is over 20GB (about 24.000 sub-folders and 42.000 files of which 22.500 .opf files - small metadata files). When I first performed a right-click on the folder, I had to wait such a long time (window was not responding) that I had to force close explorer. During this time, 100% I/O disk activity could be seen by explorer.exe, scanning all the subfolders (this could be seen in windows' Resource Monitor) After several more tries, I managed to see the right click menu if waiting over 10 minutes. I don't have many shell extensions, though I did tried in the past some shell extensions to show PDF properties in explorer. I've disabled all non-microsoft extensions (right click context menu handlers) using nirsoft shellexview, including the PDF shell extensions by Tracker Software but this did not speed the things considerably. Yet, I was manage to get a right click menu after 10 minutes or so (this worked after disabling non-MS extensions) -- while before disabling them I had to restart explorer because it was not responding. Could the PDF shell extensions cause any trouble (by Tracker Software, optionally installed with pdf x-change viwer)? - I doubt it, because those shoud bt needed for pdf files, not for folders. I don't really understand why would need explorer.exe to "scan" all the files and subfolders of a folder, just in order to show a right click menu. It's like... if I have a right click menu for WinRar (e.g. "add to archive"), it would show no matter what's inside the folder. What shell extension would need to "know" what's inside the folder? I can't think of any. Why would explorer need to scan the folder? In order to get the file properties eg count and size? But this should be done only when / if I click on the Properties entry of the menu... I do have a slow hard drive (I suppose 5400 rpm, - it's on a laptop) but that should not matter. When I right click on a folder I should have a menu, not explorer counting folders and files or doing other uneeded operations. The laptop has 3GB of ram, running windows 8.1 32-bit, and a dual core Intel processor (T4400 iirc). The funny thing is that after a first wait of 10 minutes, any subsequent right clicks on the same folder took only 3-5 seconds - it's like the information was already "cached" and now I don't need to waite the whole 10 miuntes again. Also, even if I went inside the folder, deleted one file and renamed another, the right click time for the folder was still 3-5 seconds (I was afraid any change I would do would need the whole 10 minutes again). I bet it's some kind of functionality of windows - those technobabble things like Volume Shadow Copy or USN Journaling (the partition is NTFS if that matters, I'm also using Everyting Search). But it still doesn't seem normal to take 10 minutes of explorer disk I/O just for right clicking a folder. Any ideeas? Thank you. L.E.: Two more things. 1. On a later try, it didn't took more than 1 minute to display the right click menu on another folder, as large as the first one (the initial Calibre library). So maybe I was just not waiting enough and loosing my patience too soon. This 1 minute try is indeed valid after I've disabled the non-ms shell extensions... hmmm. So maybe one of those was causing the problem? 2. If I manually enter the path in Everything Search, I can search inside the folder instantly, so it's not Everyting Search that is causing the problem. It's just that I am used to launch Everyting Search from a right click on a folder... and that wasn't possible before seing the right click menu, obviously. The stalling was definitelly caused by Explorer.exe, as I coudl see in Resource Monitor - Disk activity, where it was "scanning" all sub-folders in multiple threads. Also, I've excluded the whole Calibre folder including the two big folders which are inside, from AVG's resident shield. Last edited by rebl; 05-26-2015 at 04:47 AM.