12-27-2011, 10:55 PM | #16 |
creator of calibre
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Once again:
Exclude the calibre library directory from your anti-virus scan |
12-28-2011, 09:42 AM | #17 |
Addict
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
|
Once again I checked my MSE, and still have my library excluded
Is there anything else I can do to debug, or help resolve this issue? Starting up with a small library is real fine. Its only the first startup after a reboot with a large library, that takes a very long time. Don't know if my issue and lugopt's are related - may be able to narrow down an issue if so. |
Advert | |
|
12-28-2011, 10:34 AM | #18 |
Grand Sorcerer
Posts: 12,124
Karma: 73448616
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
|
Disk defragmentation?
|
12-28-2011, 11:26 AM | #19 |
Addict
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
|
I am a little short of space (190GB left) but disk defrag runs auto, and drive is 0%fragmented.
Nice idea though. I Think I have a space small drive with a fresh window install on it, I may just boot with it and install calibre and library and see if that makes a difference Edit: - Just done that, and same result - first start of Calibre takes almost 2minuets It seems to get to the "initialising user Interface" and then just churn through the harddrive for that time. Totally clean install of win7 on 64bit sys, 4GBram 3GHz CPU (is it doing some indexing or something?) Not sure what else to try. Last edited by transmitthis; 12-28-2011 at 01:23 PM. |
12-28-2011, 01:44 PM | #20 |
Junior Member
Posts: 4
Karma: 10
Join Date: Mar 2011
Device: none
|
I think we could split this issue in two:
1) the slow start; 2) the open/close of EVERY .mobi file; My eBook folder is non-defragmented and now Calibre takes 25 seconds to start. Much better since I started this thread. But Calibre still opens all .mobi/.epub file in my library. I excluded the eBook library from the anti-virus but the process that is open EVERY .mobi file is Calibre-debug or Calibre and not the anti-virus process. So something inside Calibre wants to open every .mobi files whenever it starts or library is changed. If I changed the library in the middle of the program it still opens EVERY .mobi file and it's very slow. C:\Program Files (x86)\Calibre2>calibre-debug -g Starting up... Started up in 25.0870001316 with 9453 books C:\Program Files (x86)\Calibre2>calibre-debug -g Starting up... Started up in 41.0629999638 with 14041 books I would like to ask other people having the same issue to download Sysinternal Process Monitor to see either calibre opens every .mobi file whenever it starts. I add a filter option in Process Monitor: Collumn: Process Name Relation: contains Value: calibre Action: Include I do have custom collumns in the main view: ISBN and Genre; but this data is (should be) inside metadata.db or .opf and not inside .mobi file. Thank you. |
Advert | |
|
12-28-2011, 02:31 PM | #21 | |
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Anti-virus Tag browser open at start-up Waiting for composite columns to fill-in Any of these apply? |
|
12-28-2011, 02:33 PM | #22 | |
Grand Sorcerer
Posts: 11,713
Karma: 6690879
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
My guess is that you have installed a plugin that is wandering through the formats for some reason. |
|
12-29-2011, 02:59 AM | #23 |
Groupie
Posts: 156
Karma: 10001
Join Date: Feb 2011
Device: sony
|
A while ago I was dealing with excruciatingly slow library open/startup that turned out to be, reproducibly, due to sorting on composite custom columns.
But en route to that discovery I, inconsistently, observed that calibre appeared to be walking through the library on startup -- which might have been a problem I fixed, or might have been a red herring. At different times it seemed to be accessing the cover images, or the .opfs. My guess is that it wasn't really significant, and that it was at various times either populating the cover browser, or backing up metadata changes to the opfs. (After changing metadata in the database, the changes are queued up to be saved with each book as a low-priority background process, if this isn't completed before calibre is closed, it will resume when calibre restarts.) But I could well be wrong, and on the off chance I fixed a problem like yours, here's most of what I did: Upgraded to the most recent version of calibre [always a good place to start] Set the anti-virus to exclude files in the library folders Set the anti-virus to exclude the calibre.exe and calibre-parallel.exe processes (I'm using MSE, same as you mentioned) Hid the tag browser [this was definitely a big problem in older versions, I'm not convinced it makes much difference now] Hid the cover browser [I suspect this was a red herring - yes calibre was accessing cover files on startup, but probably not enough to noticeably slow things down] Defragged the drive and the MFT [This did noticeably improve bulk metadata edits, but I don't think it did much else noticeable] Stopped sorting on composite columns at startup (as described here (I mentioned this a few posts ago -- and in my case, this was the real culprit) I also sped up slow searching by excluding the comments field -- this shouldn't have anything to do with startup, but I'll mention it anyway (Preferences-Searching-Limit the searched metadata) I was just using Win7's Resource monitor, not logging/debugging so I never really figured out why calibre seemed to be walking through the library, but when it went away I just guessed that it wasn't really significant and that I only noticed it because the library open process was so slow for other reasons. But maybe the answer to your problem is buried in what I did .... Also, I see that you mention having a custom column ISBN -- which I'm guessing is a composite column, so if you've ever sorted on it or your Genre column -- that could well be what's causing the slowdown. |
12-29-2011, 05:22 AM | #24 |
Junior Member
Posts: 1
Karma: 10
Join Date: Sep 2011
Device: none
|
I had the same problem. It occurred after creating a user-defined column (ISBN).
I first removed the checkmark for not showing the column. This didn't work. So I deleted the entire column and now the startup time was ok. |
12-30-2011, 02:55 AM | #25 |
Guru
Posts: 695
Karma: 822675
Join Date: May 2010
Device: Kobo Aura, Nokia Lumia 920 (Freda)
|
For the procmon vs. built-in "open file" discrepancy, keep in mind that procmon shows you a lot more information than anything else. For example, I just ran procmon tracing file system activity on calibre.exe startup and did indeed see "CreateFile" operations on epub files. On the surface that looks damning, so let's dig a little deeper. Right-click, Properties. Now we're getting somewhere. On the "Event" tab you can see that the Desired Access is "Read Attributes". So this isn't actually opening the file. This is getting the file attributes, a much lighter file IO access than actually opening (and reading) the file. Just to confirm, click on over to the Stack tab and you can see that calibre is trying to get the length of the file (PyObject_Length), which makes an underlying GetFileAttributesExW call, which never actually calls a CreateFile or OpenFile method. So all is good.
That said, querying attributes on 14,000 files isn't going to be instantaneous. In my case, with the files residing on a network share (server is on a gigabit wired connection but being accessed from a laptop on wifi), the duration of the call was 0.0061760 seconds, or 6 milliseconds. Honestly, this is the problem with tools like procmon, where too much information can lead to an incorrect diagnosis. Because procmon categorizes querying for file attributes as a "CreateFile" operation, it looks like a ton of CreateFile spew in the output. It's only when you dig in deeper that you realize what's going on. Edit: Just for further clarification, I verified that the only book file that calibre "opens" from procmon is whatever the first selected book is on startup. So not really a "ton" of calls, but it's still worth pointing out that you have to dig deeper into procmon's output to understand exactly what's happening. Last edited by toddos; 12-30-2011 at 03:01 AM. |
12-30-2011, 03:34 AM | #26 |
creator of calibre
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
@toddos: intersting, thanks for the info. I'm somewhat surprised even at the stat calls, as IIRC they should not be happening en masse on startup, but that is a lot more believable. I think they only happen if you have a custom column defined. That causes get_metadata to be called which in turn reads the sizes and last modified times for the ebook files. That info is cached, so future calls will not actually hit the filesystem.
Last edited by kovidgoyal; 12-30-2011 at 03:38 AM. |
12-30-2011, 04:13 AM | #27 | |
Guru
Posts: 695
Karma: 822675
Join Date: May 2010
Device: Kobo Aura, Nokia Lumia 920 (Freda)
|
Quote:
|
|
12-30-2011, 04:27 AM | #28 |
Grand Sorcerer
Posts: 11,713
Karma: 6690879
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
To add to Kovid's analysis, get_metadata is called in the following situations connected with this issue:
- if a coloring rule exists, get_metadata is called for all the books shown on the gui pane. - if there is a composite column, get_metadata is called for all the books shown on the gui pane. - if a composite column has the 'show in tag browser' box checked, get_metadata is called for every book in the library whenever the tag browser pane is rebuilt. My guess is that it is the last one that would cause grief. It is worth noting that using composite columns as tag browser categories can be the root of several performance slowdowns, because the tag browser values must be built by scanning the library and recomputing the composite column values. There is caching, which helps. |
12-30-2011, 08:57 AM | #29 |
Addict
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
|
I have to admit, I'm not that up on on reading the output of process monitor, but I had a look as suggested.
Here are a few images I did see a lot of buffer overflows when covers were queried For books, there were entries 6times for each one. And also noticed when opening a reg key, it kept trying but, not found? Anyway in last image is my plugin list, wondering if I have smiler ones as you lugopt? I do use several custom columns. If I'm understanding the latest posts - some extra calls could be due to whats showing in the gui when the program is closed, requiring that info to be read when opening up the gui again. If this is done after a reboot then the info wont be cached...I'll have a play with that. If any of that info helps, or you want me to try anything let me know. ps - some plugins are for research use only obviously Last edited by transmitthis; 12-30-2011 at 09:01 AM. Reason: image uplode |
01-15-2012, 08:24 AM | #30 |
Junior Member
Posts: 4
Karma: 10
Join Date: Mar 2011
Device: none
|
@chaley,
I do have custom columns: genre: Comma separated text, like tags, shown in the tag browser quality: Ratings, shown with stars isbn: custom build from other columns and Show in tags browser is unchecked @trasmitthis, I just have two plugins: FindDuplicates and Extract ISBN. Right now I added a few more books and I've "Started up in 200.747999907 with 19133 books". It continues to scan every file and altough it's pretty fast on wach file, it scans them all. I admit it might be related with custom columns I just want to know which ones. Thank you, |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre won't start up | hwhovis | Calibre | 3 | 02-04-2011 03:45 AM |
Way to speed up Calibre? | johnnyr | Calibre | 14 | 11-24-2010 02:21 AM |
calibre program launch speed | cybmole | Calibre | 44 | 10-16-2010 10:57 PM |
Calibre 0.7.9 won't start | HarryC | Calibre | 4 | 07-20-2010 04:45 PM |
Calibre won't start | nathancabopino | Calibre | 14 | 06-30-2009 11:40 AM |