Large Database Issues

[phil_ga]

Originally Posted by itimpi

Any time you change the metadata for a book, the database is updated and the OPF file is also updated. If it was only done when users requested it the object of having it in the first place would be largely defeated. It is meant to allow you to recover from an unpredictable event (database damage) so the OPF files need to be kept as up-to-date as possible.

I will look into this, but i remain unconvinced at this point that it actually changes the opf content rather than the file date.

Originally Posted by itimpi

In terms of the performance issue - do you have the "Search as you type" option set under Preferences->Interface->Searching. I could see that severely impacting perceived performance. I much prefer searching to only occur when I press Enter so I always have it switched off. I am not sure what the default is on a new installation.

No I do not have that feature set. ...BUT there is a switch in the same menu called "Limit the search metadata". I tried that. It makes a huge difference to response time when searching direct fromt he search bar with free text! Wow. Thanks for the tip!! Now I will look into the sluggish behaviour in other areas - to see if any of them are also sped up (later - bedtime now...). But this tip is a big help with responsiveness for vanilla searching. Thanks!!!

[Starson17]

I didn't see it here, but has anyone suggested that he exclude the Calibre library from his antivirus? That will sometimes do wonders for Calibre speed problems.

My library is 17K books. My experience is just like chaley's. It's fast and responsive and I've never seen a corrupted database despite running from source and bleeding edge for the last 1.5 years.

[PeterT]

Also, maybe some hard drive maintenance? A defrag possibly??

[theducks]

Originally Posted by PeterT

Also, maybe some hard drive maintenance? A defrag possibly??

Check the: Path variable for reasonableness. Move Calibre forward in the list
here is the path on my XP system. as you can see, Calibre is currently in the lowest position.

%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\Sy stem32\Wbem;C:\Program Files\Support Tools\;C:\WINDOWS\system32\WindowsPowerShell\v1.0; C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Calibre2\

BTW I have seen systems wit corrupted pats that loop throug the same EXACT folders many times

[Censkar]

I got a library with 13k+ books and I noticed a large slowdown when I updated from 0.8.8 (I tried all three versions .9 .10 and .11) so I had to revert to 0.8.8. Especially at startup. .8 starts in 15-20 seconds but the later versions takes minutes to start.
I checked in task manager at startup and .8 allocates/reads 10 Mb at a time but the later versions allocates/reads kbytes at a time?

Using Win 7 X64 and db on a NAS.

[Pinecone]

Originally Posted by phil_ga

@Chaley - I get nothing like the response times you indicate (apart from the program start time), with similar specc'ed PC (but Vista).

That could be a BIG part of it.

[Starson17]

Originally Posted by Pinecone

That could be a BIG part of it.That could be a BIG part of it.

If you're pointing to Vista, I've got nearly identical response times to chaley's with 17K book library on a Vista install.

[lbik]

My guess is that not searching the comments field will do wonders for you.

Great. It does allso wonders for me.

thanks Chaley.

[phil_ga]

Originally Posted by phil_ga

I will look into this, but i remain unconvinced at this point that it actually changes the opf content rather than the file date.

No I do not have that feature set. ...BUT there is a switch in the same menu called "Limit the search metadata". I tried that. It makes a huge difference to response time when searching direct from the search bar with free text! Wow. Thanks for the tip!! Now I will look into the sluggish behaviour in other areas - to see if any of them are also sped up (later - bedtime now...). But this tip is a big help with responsiveness for vanilla searching. Thanks!!!

Just to summarise the resolution of this problem:
My slow and sluggish calibre is showing consistently faster performance in all areas I previously complained about. itimpi 's tip did the trick!

In the preferences under: Preferences -> Interface -> Searching
Choose: Searching -> Limit the searched metadata
I set my values to: title, authors, series
Not only is searching from the search bar much much zippier, but the sluggishness in other areas, such as editing fields directly also disappears.

Gee, I would think this setting should be "on" out of the box - but then again, not a lot have databases this size (but i am betting there will be more and more).
Thanks calibre, thanks itimpi and jackastor!.

[kiwidude]

You might want to add tags to that list, depending on how and if you use them.

Many months ago I was at least one of the original 'complainers' that triggered chaley to add this feature which is a must have setting for me too. My rationale was not for performance reasons, it was to do with the quality of the search results. You no longer have to prefix every search term with the likes of title: etc just to quickly find the book you are after as you no longer get all the false positives from random word matches in comments etc. That this feature also improves performance is just an added bonus to me!

Note that you can still search other fields with your list restricted, you just need to prefix them like comments:xxx

[transmitthis]

I always have problems searching - took me ages trying to discover the proper format to do a date search one day. (forgotten it now)

I would like to suggest there is a key modifier added that would automatically put the correct string into the search bar, ready for you to search...

ie,
Ctr+"Author column" (Left click in Column heading)
Would place authors:"="

Ah, wait a mo, I bet thats not possible, and it would leave the curser outside of the " so people would have to remember to type in the right place.

Still there should be a more simple one click way of searching within a particular column without having to remember these search strings with : and "" and = etc., as most users are not going to type that, they are just going to free text search


PS (C2D 3Ghz 4GBRam, Win7,64bit
Calibre starts in:
5 seconds - if the library only contains one book.
with 9000 books
60 seconds to open
14 seconds if its previously been opened, ie still in cache
I know some programs open, while loading required data in the background, to speed up load times.

[transmitthis]

Maybe a dropdown list in the search bar, that will limit search to each individual column heading. That way if you want to search authors, it would be click on drop down, select Author, and type.
Anyway just another of my not really code-able suggestions

[dkelly701]

Maybe I am missing something here, but isn't there already an easy way to do this by clicking on the "search binoculars" beside the search bar. Click on the Title, Author, Series tab and type in the search criteria. Click on the down arrow at the bottom if you want to search in other data sets.

Originally Posted by transmitthis

I always have problems searching - took me ages trying to discover the proper format to do a date search one day. (forgotten it now)

I would like to suggest there is a key modifier added that would automatically put the correct string into the search bar, ready for you to search...

ie,
Ctr+"Author column" (Left click in Column heading)
Would place authors:"="

Ah, wait a mo, I bet thats not possible, and it would leave the curser outside of the " so people would have to remember to type in the right place.

Still there should be a more simple one click way of searching within a particular column without having to remember these search strings with : and "" and = etc., as most users are not going to type that, they are just going to free text search


PS (C2D 3Ghz 4GBRam, Win7,64bit
Calibre starts in:
5 seconds - if the library only contains one book.
with 9000 books
60 seconds to open
14 seconds if its previously been opened, ie still in cache
I know some programs open, while loading required data in the background, to speed up load times.

[transmitthis]

dkelly701 - Er, no your not, I was missing that.. .lol

It only has four column heading to choose from though and would be nice if it was more integrated into the search bar,
Still thanks for the heads up at least some searches will be easier