Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 07-24-2011, 08:21 AM   #1
phil_ga
Member
phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.
 
Posts: 10
Karma: 280
Join Date: Feb 2011
Device: kindle
Unhappy Large Database Issues

I have now over 9k books in both epub and mobi format. The metadata db is 52 MB in size. The library itself is 36k files and 16.2 GB.

1. Well, calibre (v0.8.11) is sluggish, to say the least. All commands that rely on the db seem to take about 5-30 seconds. Some functions are painful in the extreme. I assume this all comes from the metadata.db being so large. I simply DO NOT recommend this program for anything larger than this unless you have incredible patience and are very happy to risk a lot of heartache.

- can the metadata.db be compacted? how?
- what is the metadata.db size limit for calibre?
- is the sluggishness due to the db or some other issue?
- is there some way to get back the program responsiveness?

It seems to me there are plenty of other people out there with db of this size, and this one is still growing. I would expect it to be doubled in 2 years. Is this just simply the limit of calibre? Its quite disappointing if it is as the memory limit in windows is quite extraordinary by comparison. We are relatively early days with the popularity of ebooks just at the start of a log phase in growth - and this is the only available app out there of any quality.

2. I have used this for about 9 months. The program has crashed three times in that period. Each time, the program was unable to recover the database. So I put in some self-protection mechansms, and the latest crash (2 days ago, after a program update) was recovered with an older database. However I lost all the new files I had added in that period (about 20) and no simple way to work out what was missing. The process of getting going again is stressful, clunky and unreliable. I did get going again, but its such a time waster.
So, am I missing something, or should the author seriously consider a more effective backup/restore procedure? At the very very simplest why not implement a system of rolling copies of the metadata.db file? Its so simple (well of course its easy for me to say so) - every time the app opens, the app could save the db as a rolling copy such as metadata.01, metadata.02, then when reaching .09, start deleting .01 and keep the cycle going. This way any of 10 different databases can be restored more simply and easily. This strategy is clearly rudimentary, but hey, it sounds so easy.

I hugely appreciate everything done to make this program the joy it currently is, but these two issues are really highly limiting (large db size, and the high risk of dependence on a single file that gets easily corrupted). I am hoping I am simply wrong and there is already a solution out there? Please?
phil_ga is offline   Reply With Quote
Old 07-24-2011, 09:00 AM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,768
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by phil_ga View Post
1. Well, calibre (v0.8.11) is sluggish, to say the least. All commands that rely on the db seem to take about 5-30 seconds. Some functions are painful in the extreme. I assume this all comes from the metadata.db being so large.
My library has 11,400 books and I don't see any delays that aren't associated with my external drive taking time to wake up after going to sleep.

Quote:
Originally Posted by phil_ga View Post
I simply DO NOT recommend this program for anything larger than this unless you have incredible patience and are very happy to risk a lot of heartache.

- can the metadata.db be compacted? how?
- what is the metadata.db size limit for calibre?
- is the sluggishness due to the db or some other issue?
- is there some way to get back the program responsiveness?
Are you running your database from a network or flash drive?

Quote:
Originally Posted by phil_ga View Post
2. I have used this for about 9 months. The program has crashed three times in that period. Each time, the program was unable to recover the database.
I have had calibre crash many times in 2.5 years of use, as I would expect of an app in beta with constant development. I have never once had any of the crashes corrupt my database.

Quote:
Originally Posted by phil_ga View Post
So, am I missing something, or should the author seriously consider a more effective backup/restore procedure?
The backup/restore of the main library is up to each of us, but the metadata.db file can be rebuilt/restored via the Library Maintenance - Restore database feature.

Quote:
Originally Posted by phil_ga View Post
At the very very simplest why not implement a system of rolling copies of the metadata.db file? Its so simple (well of course its easy for me to say so) - every time the app opens, the app could save the db as a rolling copy such as metadata.01, metadata.02, then when reaching .09, start deleting .01 and keep the cycle going. This way any of 10 different databases can be restored more simply and easily. This strategy is clearly rudimentary, but hey, it sounds so easy.
The restore database feature is more comprehensive. Although it is quite easy to start calibre in a batch file and do exactly what you suggest. I previously used the batch file to quick copy my metadata.db file, but I have my library in Dropbox and it keeps 30 days worth of file version changes so the batch file backup is not needed. Add to that calibre's restore database feature and I think I'm covered.

I'll let others focus on why you may be seeing the delays you're experiencing.
DoctorOhh is offline   Reply With Quote
Old 07-24-2011, 09:02 AM   #3
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,022
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
There has not been a dependency on the metadata.db not being corrupted and has not been for some time. It is always possible to rebuild the database using the files in the library. That is the purpose of the metadata.opf file stored with each book that stores a copy of the metadata information that is in the database relating to that book.
itimpi is offline   Reply With Quote
Old 07-24-2011, 10:22 AM   #4
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,212
Karma: 5495470
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by phil_ga View Post


I hugely appreciate everything done to make this program the joy it currently is, but these two issues are really highly limiting (large db size, and the high risk of dependence on a single file that gets easily corrupted). I am hoping I am simply wrong and there is already a solution out there? Please?
Like Walt, crashes are few and far between and have been limited to specific releases.
Calibre conversions can stress marginal PC hardware. If you edit metadata with a background conversion running, it is possible that you PSU or cooling fans/dirty fins may not keep up with the load.

Mains power quality is suffering. Use of a UPS model with AVR is highly suggested.

As to DB size. I believe that running the maintenance checks might 'pack' the DB as a side effect.

Last edited by theducks; 07-24-2011 at 10:24 AM.
theducks is offline   Reply With Quote
Old 07-24-2011, 11:21 AM   #5
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,968
Karma: 802238
Join Date: Jan 2010
Location: France
Device: Many android devices
My test library is 20,000 books, 6000 authors, 10,000 tags. On my Intel P5 2.6GHz, 4GB mem, 7200 rpm SATA drive, Win7 Pro/32, Calibre starts in under 7 seconds. Edit metadata (single, not changing author or title) completes in under 2 seconds. Searching from the tag browser is close to instantaneous. Change tag browser sort from name to popularity takes under 2 seconds. Not sure if these are the operations you are referring to.

Regarding corruption of the database: in 18 months I have never lost a database, other than when I purposely broke it to test the recovery mechanism. This is true even though I upgrade on a daily basis and use bleeding edge calibre versions that never see public use (and sometimes don't work).

My conclusion: given that your experience does not match that of several people who have very large libraries, there seems to be something odd about your configuration. Exactly what, I can't say. Any chance that you overclock? Is your library on a NAS? Have you tried defragging? (Some people have reported that defragging helped tremendously.)
chaley is offline   Reply With Quote
Old 07-24-2011, 06:01 PM   #6
phil_ga
Member
phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.
 
Posts: 10
Karma: 280
Join Date: Feb 2011
Device: kindle
Thanks for all this feedback. Firstly I am not new to the PC. Its not underpowered, dirty, fragged nor overclocked. It does not run unusually hot. I am referring only to response time after the drive is already awake. If do realise that editing metadata with a background conversion running will be slow - I am not referring to that. No other programs have issues or delays, including 500 mb files in photoshop. Does not matter if its a fresh reboot with nothing else running. I run the library from the HDD, a 1 TB internal drive. I have never used a network or flash drive for the library. Maybe I could try a faster drive? but one would think its a memory and chip speed issue, not a slow HDD issue. I moved the library to the primary HDD and the response rates are the same... Thats why I posted - nothing seems to indicate its my PC (of course I am always open to this!). I have had more program crashes than 3, but the 3 refers to when the database could not be read upon restarting everything. (Actually this program does not crash much - its other things that mostly take my PC down and calibre with it. I am only concerned about the restarts and inability to open the database again).

@Chaley - I get nothing like the response times you indicate (apart from the program start time), with similar specc'ed PC (but Vista). Searching from the tag browser is anything from 5 secs (which is ok) to 30 seconds - with the program appearing to freeze while it does so.

Secondly it seems the sluggishness and lack of responsiveness is just me? Yet I have read posts of others with this problem and have assumed its widespread. It still happens if its the portable version of the app. Same problem of sluggishness occurs on my other PC with my same library, and on my work PC. Does all this suggest the problem IS the metadata.db file?

The opf files are confusing themselves. When exactly are they updated? eg if i add AU information to the comments section - when does this get copied into the opf file? I am assuming never, untill you manually do a library maintenance. There appears to be no function to update only the opf files from the books you first select, you seem to need to do the whole data base (several hours). A deterrent. "The backup/restore of the main library is up to each of us...via the Library Maintenance". Maybe its just the wording, but it is NOT clear what any of these functions do. It took me a lot of reading to assume that backup merely means copying the db information into 10,000 opf individual files. Its not a backup of the library, but only of metadata.db file? Taking 6 hours to do this is still a deterrent (despite that its a background operation).

@itimpi "There has not been a dependency on the metadata.db not being corrupted and has not been for some time. It is always possible to rebuild the database using the files in the library.". This makes many assumptions that are not clear. If this is true, why does the program report to me a corrupt database, then say it gives up and starts a new empty one? This is a confusing response.

Should I perhaps "backup the database" THEN delete the metadata.db file and deliberately rebuild it? Is that what "restore database" does? Is this worth testing?
Ok then, I should rebuild - but a) how do you know the opf files are updated so that you are rebuilding all the changes you made in the comments etc? Many of them are probably old? I frequently edit groups of 10-20, adding AU biography to each for example. How does one "backup" those ones specifically? and b) how long would it take for 10k books - again presumably many hours? I guess I am just scared to manually delete this pesky file and rebuild it....

...again, thanks for so many responses.
phil_ga is offline   Reply With Quote
Old 07-24-2011, 10:42 PM   #7
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,565
Karma: 24978608
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
AFAIK the opf files are updated on edit/conversion so the should be up to date assuming you have backup to opf turned on. ( Wish I could turn it off)

If you are worried about rebuilding, simply rename the metadata.db before rebuilding it or copy it to another location.

I had a corrupted database, which Chaley fixed for me. However after backing it up, I decided to reimport all 9000+ books (about 16 Gig) This took 3 hours which I thought was amazingly fast.

If it takes 2 or 3 hours, or even 10, it is not like you should have to do it every day.

I went through several panic attacks and I did lose about 50 books (not duplicates) somehow, but if and when I looked for them they were in the backup and easily recovered.

My current main library is 11422 books and I am sadly using a single core laptop from Romania (not a joke) and an old external hard drive. Calibre takes less than 20 seconds to load and be fully functional.

Helen
speakingtohe is online now   Reply With Quote
Old 07-24-2011, 11:36 PM   #8
jackastor
Wizard
jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.jackastor ought to be getting tired of karma fortunes by now.
 
jackastor's Avatar
 
Posts: 1,534
Karma: 2604523
Join Date: Jun 2011
Device: iphone stanza, kobo touch,ASUS TF300,KOBO GLO, Kobo Aura HD, Kobo Mini
over 11,000 books

Walt did I read that correctly 11k+ books?
jackastor is offline   Reply With Quote
Old 07-25-2011, 12:16 AM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,768
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by jackastor View Post
Walt did I read that correctly 11k+ books?
Yes, like most of the other folks in this thread.

This is my working library, my finished library is at 8,444. As I go through the working library I often delete books if they are from the middle of series that I don't have. Others I clean up the metadata, cover and convert the books with my preferences. Then review the book and use Sigil for basic clean-up if needed. Once done I copy the finished book to my finished library.

Once the book is in my finished library I know it is readable without junk cluttering things up.
DoctorOhh is offline   Reply With Quote
Old 07-25-2011, 06:11 AM   #10
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,968
Karma: 802238
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by phil_ga View Post
Searching from the tag browser is anything from 5 secs (which is ok) to 30 seconds - with the program appearing to freeze while it does so.
FWIW: searches from the tag browser are entirely in-memory. They do not touch the database and therefore should not touch the disk. For example, a search for a tag will loop through the library cache (the in-memory copy of the data from the DB), looking at the tags for each book. The compare takes (on my machine) .4 seconds for a search across 20,000 books for a tag that is applied to 24 books. That .4 seconds is roughly constant: it takes the same amount of time if the tag is on only one book. This is expected, because search must look at each book even if it finds nothing.

Searches on user categories are different. They must not only loop though the books, but must also loop through the categories. They can take much longer, depending on the number of items in the category.

Given that you are seeing times that are much larger than I see, I wonder if calibre is thrashing (unable to keep its cache in real memory as opposed to virtual memory). Do you have other programs running that are consuming memory? (You said not, but there is always something running.) Do you have odd virtual memory settings? While running searches, is the hard disk being used? It shouldn't be. If you run the same search twice, is it much faster the second time? If so, that indicates that the cache is being pushed out of physical memory.
chaley is offline   Reply With Quote
Old 07-25-2011, 06:46 AM   #11
phil_ga
Member
phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.
 
Posts: 10
Karma: 280
Join Date: Feb 2011
Device: kindle
I rebuilt my metadata.db file. In the end it took about 10 minutes, but crashed the program three times before it worked (on each freeze I waited up to 1 hour before killing the program). On the fourth attempt it made it thru in about 10 minutes. No other programs were being run (intentionally) in parallel. The new file is now 46Mb instead of 52, so something was definitely crap in the file.
The program remains "sluggish" (ie I probably wasted much of this day). If one searches by author using the categories (which is a major pain to navigate down a 3,221 author tree) the search is subsecond as chaley says. If you search via the search bar (thats why its there huh??) it takes 5 seconds for some authors and 30 or more for others (with a program appearing to freeze in between). The same searches mostly take the same time: eg danielle steel is 5 seconds, yet jon sprunk is more than 30. Its all very unsatisfying. I only search via typing in the search bar. Using categories takes even longer to search - because there are so many navigation steps to go thru to find the actual author (the search itself is fast). The same sluggish behavior also is felt on editing any books properties like changing the genre.

No, I am not specifically running any other program while doing the tasks i report here. The problem is restricted to calibre - I can run a dozen programs at once normally - including transcoding mp3 and videos. Please simply believe me when i say its calibre - not my pc.

I remain with the feeling that its the size of the db file that is the problem...but I cannot be sure.

Can any one answer about the conditions under which opf files are actually saved to disk? I bet never, untill you do the backup manually (about 5 hours for my library).
phil_ga is offline   Reply With Quote
Old 07-25-2011, 06:59 AM   #12
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,768
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by phil_ga View Post
If you search via the search bar (thats why its there huh??) it takes 5 seconds for some authors and 30 or more for others (with a program appearing to freeze in between). The same searches mostly take the same time: eg danielle steel is 5 seconds, yet jon sprunk is more than 30.
I see one possible problem, you may not searching for authors or tags correctly. Try:

authors:"=jon sprunk"
or
tags:"=Mystery"

I think you'll find these searches to be almost instantaneous.

Quote:
Originally Posted by phil_ga View Post
Can any one answer about the conditions under which opf files are actually saved to disk? I bet never, untill you do the backup manually (about 5 hours for my library).
I just updated the metadata for one book then immediately opened the folder and the OPF file had already been updated.

One other thing, You still haven't answered the question, are you running your database off of the network or from a local drive?

Last edited by DoctorOhh; 07-25-2011 at 07:05 AM.
DoctorOhh is offline   Reply With Quote
Old 07-25-2011, 07:20 AM   #13
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,022
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by phil_ga View Post
Can any one answer about the conditions under which opf files are actually saved to disk? I bet never, untill you do the backup manually (about 5 hours for my library).
Any time you change the metadata for a book, the database is updated and the OPF file is also updated. If it was only done when users requested it the object of having it in the first place would be largely defeated. It is meant to allow you to recover from an unpredictable event (database damage) so the OPF files need to be kept as up-to-date as possible.

Compacting the database might regain some space, but is unlikely to improve performance (it may even have the opposite effect). It is perfectly normal for a database to have unused space internally as pages are marked as free when a record is deleted, but the space is not removed from the file. These 'free' pages get re-used when required.

In terms of the performance issue - do you have the "Search as you type" option set under Preferences->Interface->Searching. I could see that severely impacting perceived performance. I much prefer searching to only occur when I press Enter so I always have it switched off. I am not sure what the default is on a new installation.

Last edited by itimpi; 07-25-2011 at 07:23 AM.
itimpi is offline   Reply With Quote
Old 07-25-2011, 07:25 AM   #14
phil_ga
Member
phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.phil_ga has a complete set of Star Wars action figures.
 
Posts: 10
Karma: 280
Join Date: Feb 2011
Device: kindle
Thanks dwanthy. I did answer both your questions in my post above. I do NOT run off networks or flash drives, but the internal HDD.
I said above that searching via the tag browser tree is indeed very fast, but the tree itself takes too long to hunt thru. I dont type authors:"=jon sprunk" into the search bar, but, as i said, if i do it that way it is very fast. I search there with free text. As I also said, search this way is not the only sluggish element of my system - its many other places! The slowness has been increasing steadily since the library passed about 5k.

..but did you check the opf file size? After a text update the file date changes, but after a library backup the file can be 2-5 time larger.
phil_ga is offline   Reply With Quote
Old 07-25-2011, 07:32 AM   #15
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,968
Karma: 802238
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by phil_ga View Post
If you search via the search bar (thats why its there huh??) it takes 5 seconds for some authors and 30 or more for others (with a program appearing to freeze in between). The same searches mostly take the same time: eg danielle steel is 5 seconds, yet jon sprunk is more than 30. Its all very unsatisfying. I only search via typing in the search bar.
Typing naked words on the search bar means "search in every column for that word". Thus, typing "jon sprunk" means search for "jon" in every column then search for "sprunk" in every column . This would explain some slowness, but certainly not all.

You can speed this up by limiting the columns to search when using naked search terms. See Preferences -> Searching. My guess is that not searching the comments field will do wonders for you.
Quote:
Please simply believe me when i say its calibre - not my pc
Sorry, but no can do. I once spent a week (many hours of my time) working with someone making the same claim, only to discover that there was some daemon running that ate the processor. The person thought that it didn't matter.

You might be right (and probably are), but then again possibly not. Clearly there is something different between what you see and what many other people see.
Quote:
I remain with the feeling that its the size of the db file that is the problem...but I cannot be sure.
The size of the DB is related to the amount of stuff in memory. However, as I said, the DB is not used during searching (I wrote this code and can guarantee it), so it is an indirect relationship.
Quote:
Can any one answer about the conditions under which opf files are actually saved to disk? I bet never, until you do the backup manually (about 5 hours for my library).
They are queued to be written whenever metadata changes. They are actually written sometime later, when their point in the queue arrives (I wrote this code as well). That 'sometime' is usually within seconds, but can be much longer if you just changed 10,000 books.
chaley is offline   Reply With Quote
Reply

Tags
database, metadata, metadata.db

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre Database cp Kindle Database mitch13 Library Management 1 05-22-2011 07:33 PM
Charging Issues and Screen Issues srj321 Sony Reader 2 07-11-2010 11:52 PM
Database Issues after removing books mobelby Calibre 6 05-31-2009 01:28 AM
Adding books to a large database Student1 Calibre 31 04-07-2009 06:43 PM
Management issues with large databases Student1 Calibre 16 02-20-2009 12:42 PM


All times are GMT -4. The time now is 07:26 PM.


MobileRead.com is a privately owned, operated and funded community.