Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 12-27-2011, 10:55 PM   #16
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Once again:

Exclude the calibre library directory from your anti-virus scan
kovidgoyal is offline   Reply With Quote
Old 12-28-2011, 09:42 AM   #17
transmitthis
Addict
transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.
 
transmitthis's Avatar
 
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
Once again I checked my MSE, and still have my library excluded

Is there anything else I can do to debug, or help resolve this issue?

Starting up with a small library is real fine.
Its only the first startup after a reboot with a large library, that takes a very long time.

Don't know if my issue and lugopt's are related - may be able to narrow down an issue if so.
transmitthis is offline   Reply With Quote
Advert
Old 12-28-2011, 10:34 AM   #18
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,124
Karma: 73448616
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
Disk defragmentation?
PeterT is offline   Reply With Quote
Old 12-28-2011, 11:26 AM   #19
transmitthis
Addict
transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.
 
transmitthis's Avatar
 
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
I am a little short of space (190GB left) but disk defrag runs auto, and drive is 0%fragmented.
Nice idea though.

I Think I have a space small drive with a fresh window install on it, I may just boot with it and install calibre and library and see if that makes a difference

Edit: - Just done that, and same result - first start of Calibre takes almost 2minuets
It seems to get to the "initialising user Interface" and then just churn through the harddrive for that time. Totally clean install of win7 on 64bit sys, 4GBram 3GHz CPU (is it doing some indexing or something?)

Not sure what else to try.

Last edited by transmitthis; 12-28-2011 at 01:23 PM.
transmitthis is offline   Reply With Quote
Old 12-28-2011, 01:44 PM   #20
lugopt
Junior Member
lugopt began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2011
Device: none
I think we could split this issue in two:
1) the slow start;
2) the open/close of EVERY .mobi file;

My eBook folder is non-defragmented and now Calibre takes 25 seconds to start. Much better since I started this thread. But Calibre still opens all .mobi/.epub file in my library.

I excluded the eBook library from the anti-virus but the process that is open EVERY .mobi file is Calibre-debug or Calibre and not the anti-virus process. So something inside Calibre wants to open every .mobi files whenever it starts or library is changed.

If I changed the library in the middle of the program it still opens EVERY .mobi file and it's very slow.

C:\Program Files (x86)\Calibre2>calibre-debug -g
Starting up...
Started up in 25.0870001316 with 9453 books

C:\Program Files (x86)\Calibre2>calibre-debug -g
Starting up...
Started up in 41.0629999638 with 14041 books

I would like to ask other people having the same issue to download Sysinternal Process Monitor to see either calibre opens every .mobi file whenever it starts. I add a filter option in Process Monitor:
Collumn: Process Name
Relation: contains
Value: calibre
Action: Include

I do have custom collumns in the main view: ISBN and Genre; but this data is (should be) inside metadata.db or .opf and not inside .mobi file.

Thank you.
lugopt is offline   Reply With Quote
Advert
Old 12-28-2011, 02:31 PM   #21
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by lugopt View Post
I think we could split this issue in two:
1) the slow start;
2) the open/close of EVERY .mobi file;
There are three things that may cause a large library to open slowly

Anti-virus
Tag browser open at start-up
Waiting for composite columns to fill-in

Any of these apply?
DoctorOhh is offline   Reply With Quote
Old 12-28-2011, 02:33 PM   #22
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,713
Karma: 6690879
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by lugopt View Post
I would like to ask other people having the same issue to download Sysinternal Process Monitor to see either calibre opens every .mobi file whenever it starts. I add a filter option in Process Monitor:
Collumn: Process Name
Relation: contains
Value: calibre
Action: Include
I used Win7's process builtin process monitor that does the same thing. Not one mobi, epub, or other book format was opened by calibre-debug. No other *calibre* process was started.

My guess is that you have installed a plugin that is wandering through the formats for some reason.
chaley is offline   Reply With Quote
Old 12-29-2011, 02:59 AM   #23
capnm
Groupie
capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'
 
Posts: 156
Karma: 10001
Join Date: Feb 2011
Device: sony
A while ago I was dealing with excruciatingly slow library open/startup that turned out to be, reproducibly, due to sorting on composite custom columns.

But en route to that discovery I, inconsistently, observed that calibre appeared to be walking through the library on startup -- which might have been a problem I fixed, or might have been a red herring. At different times it seemed to be accessing the cover images, or the .opfs.
My guess is that it wasn't really significant, and that it was at various times either populating the cover browser, or backing up metadata changes to the opfs. (After changing metadata in the database, the changes are queued up to be saved with each book as a low-priority background process, if this isn't completed before calibre is closed, it will resume when calibre restarts.)
But I could well be wrong, and on the off chance I fixed a problem like yours, here's most of what I did:

Upgraded to the most recent version of calibre [always a good place to start]

Set the anti-virus to exclude files in the library folders
Set the anti-virus to exclude the calibre.exe and calibre-parallel.exe processes
(I'm using MSE, same as you mentioned)

Hid the tag browser [this was definitely a big problem in older versions, I'm not convinced it makes much difference now]

Hid the cover browser [I suspect this was a red herring - yes calibre was accessing cover files on startup, but probably not enough to noticeably slow things down]

Defragged the drive and the MFT [This did noticeably improve bulk metadata edits, but I don't think it did much else noticeable]

Stopped sorting on composite columns at startup (as described here (I mentioned this a few posts ago -- and in my case, this was the real culprit)

I also sped up slow searching by excluding the comments field -- this shouldn't have anything to do with startup, but I'll mention it anyway (Preferences-Searching-Limit the searched metadata)

I was just using Win7's Resource monitor, not logging/debugging so I never really figured out why calibre seemed to be walking through the library, but when it went away I just guessed that it wasn't really significant and that I only noticed it because the library open process was so slow for other reasons.

But maybe the answer to your problem is buried in what I did ....

Also, I see that you mention having a custom column ISBN -- which I'm guessing is a composite column, so if you've ever sorted on it or your Genre column -- that could well be what's causing the slowdown.
capnm is offline   Reply With Quote
Old 12-29-2011, 05:22 AM   #24
Myrrah
Junior Member
Myrrah began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Sep 2011
Device: none
I had the same problem. It occurred after creating a user-defined column (ISBN).
I first removed the checkmark for not showing the column. This didn't work. So I deleted the entire column and now the startup time was ok.
Myrrah is offline   Reply With Quote
Old 12-30-2011, 02:55 AM   #25
toddos
Guru
toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.
 
toddos's Avatar
 
Posts: 695
Karma: 822675
Join Date: May 2010
Device: Kobo Aura, Nokia Lumia 920 (Freda)
For the procmon vs. built-in "open file" discrepancy, keep in mind that procmon shows you a lot more information than anything else. For example, I just ran procmon tracing file system activity on calibre.exe startup and did indeed see "CreateFile" operations on epub files. On the surface that looks damning, so let's dig a little deeper. Right-click, Properties. Now we're getting somewhere. On the "Event" tab you can see that the Desired Access is "Read Attributes". So this isn't actually opening the file. This is getting the file attributes, a much lighter file IO access than actually opening (and reading) the file. Just to confirm, click on over to the Stack tab and you can see that calibre is trying to get the length of the file (PyObject_Length), which makes an underlying GetFileAttributesExW call, which never actually calls a CreateFile or OpenFile method. So all is good.

That said, querying attributes on 14,000 files isn't going to be instantaneous. In my case, with the files residing on a network share (server is on a gigabit wired connection but being accessed from a laptop on wifi), the duration of the call was 0.0061760 seconds, or 6 milliseconds.

Honestly, this is the problem with tools like procmon, where too much information can lead to an incorrect diagnosis. Because procmon categorizes querying for file attributes as a "CreateFile" operation, it looks like a ton of CreateFile spew in the output. It's only when you dig in deeper that you realize what's going on.

Edit: Just for further clarification, I verified that the only book file that calibre "opens" from procmon is whatever the first selected book is on startup. So not really a "ton" of calls, but it's still worth pointing out that you have to dig deeper into procmon's output to understand exactly what's happening.
Attached Thumbnails
Click image for larger version

Name:	procmon_calibre_trace.png
Views:	208
Size:	26.6 KB
ID:	80528   Click image for larger version

Name:	procmon_calibre_properties.png
Views:	227
Size:	45.1 KB
ID:	80529   Click image for larger version

Name:	procmon_calibre_stack.png
Views:	210
Size:	62.5 KB
ID:	80530  

Last edited by toddos; 12-30-2011 at 03:01 AM.
toddos is offline   Reply With Quote
Old 12-30-2011, 03:34 AM   #26
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@toddos: intersting, thanks for the info. I'm somewhat surprised even at the stat calls, as IIRC they should not be happening en masse on startup, but that is a lot more believable. I think they only happen if you have a custom column defined. That causes get_metadata to be called which in turn reads the sizes and last modified times for the ebook files. That info is cached, so future calls will not actually hit the filesystem.

Last edited by kovidgoyal; 12-30-2011 at 03:38 AM.
kovidgoyal is offline   Reply With Quote
Old 12-30-2011, 04:13 AM   #27
toddos
Guru
toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.toddos ought to be getting tired of karma fortunes by now.
 
toddos's Avatar
 
Posts: 695
Karma: 822675
Join Date: May 2010
Device: Kobo Aura, Nokia Lumia 920 (Freda)
Quote:
Originally Posted by kovidgoyal View Post
@toddos: intersting, thanks for the info. I'm somewhat surprised even at the stat calls, as IIRC they should not be happening en masse on startup, but that is a lot more believable. I think they only happen if you have a custom column defined. That causes get_metadata to be called which in turn reads the sizes and last modified times for the ebook files. That info is cached, so future calls will not actually hit the filesystem.
I do have a yes/no "Read" custom column and a page count (using a plugin) column, and the book is the first in the list so it's selected when calibre starts up. It doesn't seem like the behavior is bad to me
toddos is offline   Reply With Quote
Old 12-30-2011, 04:27 AM   #28
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,713
Karma: 6690879
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
To add to Kovid's analysis, get_metadata is called in the following situations connected with this issue:
- if a coloring rule exists, get_metadata is called for all the books shown on the gui pane.
- if there is a composite column, get_metadata is called for all the books shown on the gui pane.
- if a composite column has the 'show in tag browser' box checked, get_metadata is called for every book in the library whenever the tag browser pane is rebuilt.

My guess is that it is the last one that would cause grief.

It is worth noting that using composite columns as tag browser categories can be the root of several performance slowdowns, because the tag browser values must be built by scanning the library and recomputing the composite column values. There is caching, which helps.
chaley is offline   Reply With Quote
Old 12-30-2011, 08:57 AM   #29
transmitthis
Addict
transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.transmitthis ought to be getting tired of karma fortunes by now.
 
transmitthis's Avatar
 
Posts: 288
Karma: 1003542
Join Date: May 2011
Device: Google Nexus 7 16GB
I have to admit, I'm not that up on on reading the output of process monitor, but I had a look as suggested.

Here are a few images
I did see a lot of buffer overflows when covers were queried
For books, there were entries 6times for each one.
And also noticed when opening a reg key, it kept trying but, not found?


Anyway in last image is my plugin list, wondering if I have smiler ones as you lugopt?


I do use several custom columns.

If I'm understanding the latest posts - some extra calls could be due to whats showing in the gui when the program is closed, requiring that info to be read when opening up the gui again. If this is done after a reboot then the info wont be cached...I'll have a play with that.

If any of that info helps, or you want me to try anything let me know.


ps - some plugins are for research use only obviously
Attached Thumbnails
Click image for larger version

Name:	hrm not found.jpg
Views:	223
Size:	679.6 KB
ID:	80541   Click image for larger version

Name:	image buffer overflow.jpg
Views:	228
Size:	752.6 KB
ID:	80542   Click image for larger version

Name:	readthembooks.jpg
Views:	215
Size:	685.5 KB
ID:	80543   Click image for larger version

Name:	plugins.jpg
Views:	223
Size:	137.9 KB
ID:	80544  

Last edited by transmitthis; 12-30-2011 at 09:01 AM. Reason: image uplode
transmitthis is offline   Reply With Quote
Old 01-15-2012, 08:24 AM   #30
lugopt
Junior Member
lugopt began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2011
Device: none
@chaley,

I do have custom columns:
genre: Comma separated text, like tags, shown in the tag browser
quality: Ratings, shown with stars
isbn: custom build from other columns and Show in tags browser is unchecked

@trasmitthis,

I just have two plugins: FindDuplicates and Extract ISBN.

Right now I added a few more books and I've "Started up in 200.747999907 with 19133 books".

It continues to scan every file and altough it's pretty fast on wach file, it scans them all. I admit it might be related with custom columns I just want to know which ones.

Thank you,
lugopt is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre won't start up hwhovis Calibre 3 02-04-2011 03:45 AM
Way to speed up Calibre? johnnyr Calibre 14 11-24-2010 02:21 AM
calibre program launch speed cybmole Calibre 44 10-16-2010 10:57 PM
Calibre 0.7.9 won't start HarryC Calibre 4 07-20-2010 04:45 PM
Calibre won't start nathancabopino Calibre 14 06-30-2009 11:40 AM


All times are GMT -4. The time now is 07:19 AM.


MobileRead.com is a privately owned, operated and funded community.