Slow Edit metadata window

[theducks]

Originally Posted by z.nina

These are the spec of my HD

Hard Drive
Form Factor: 3.5″ x 1/3H
Capacity: 2 TB
Interface Type: Serial ATA-300
Buffer Size: 64 MB
Features: IntelliSeek, StableTrac, Perpendicular Magnetic Recording (PMR), IntelliPower, NoTouch ramp load technology, GreenPower technology
Compliant Standards: S.M.A.R.T.

Performance
Drive Transfer Rate: 300 MBps (external)
Internal Data Rate: 110 MBps
Spindle Speed: 5400 rpm

Definitely a modern drive
64MB cache : That is 1.5+ times the size of my first Hard Drive

See if you can find a Utility to read out the SMART status. (I know I can do it with a Ubuntu (even on the Live CD)) It should flag detected issues with the drive long before you may be able to notice them otherwise.

[Divingduck]

Only an additional idea and maybe you did something similar.

I had a problem a couple of weeks ago. What happen? I had a custom column with a bad design. Therefore I decide to change this column. I did it the way that I define a new one with a similar display- and column name and add a digit to it to have it different to the old one. I update then the content to the new column, check the results of the database and was happy that everything was working well. I thought…

After this, I delete the old column and rename the new one to the old column name. So fare so good. Next day Friday, I made a Calibre update and after this, I switch to my test library for testing some other things. Some when later I switched back to my main library because I had to add a book there and I was heavily surprised to wait about 3 (!) minutes for adding a book and then additional 3 minutes for updating my metadata. Doing this with my test library, this process takes only a second.

I check my main library again; I saw that I made a stupid thing:
I renamed my new column display name but I forgot to rename the column name itself. Unfortunately the old column was part of a virtual library and every time I made a change in the metadata, Calibre try to update the data and was not able to do that what result in a long activity to update the caching data (I guess). After I changed the column name to the correct name the library runs like charm again. My lesson learned…

[z.nina]

@BetterRed Sorry for the late answer. I've installed Calibre Portable in the USB3 HD and copied the library in there. The problem is the same. I've to wait more than 30 sec for the metadata editor to move to the next book
The problem is the same even if I use a different PC with win 7 64 bit( I don't have a PC with win7 32 bit so I don't know if that maybe a issue).

I've set the environment variable on my main PC (many many thanks for the detailed explanation !!!!) and the problem is solved

[z.nina]

@Divingduck Thanks for the advice but I don't think I've a similar problem.
I've 2 custom column :
1 (yes/no) for the status of the book (read - to read);
2 for the detail of the series.

If I update my metadata I've no problem. I open the edit windows only if I want to "manualy" download the metadata so I can choose wich metadata result I want to save in my library. Some source don't have the series info.

If I change any value of the colums in the main windows I've no problem and no delay

[chaley]

Originally Posted by z.nina

@[I've set the environment variable on my main PC (many many thanks for the detailed explanation !!!!) and the problem is solved

OK, to me that implies the fundamental problem is sqlite3 transaction handling. As I mentioned earlier, on windows a transaction consists of a file creation, some writes, then a file deletion. These are "expensive" operations.

NTFS is a complex journaled file system that operates a bit like a database (see http://www.ntfs.com/ntfs.htm for a lot of info). Creating a file requires several read and write operations. Most of the read operations can come from the disk cache. Which leads me to speculate ...

Glancing through this thread I didn't see how much memory your computer has. A 2TB disc will have an enormous amount of control space (file allocation tables, journals, open file tables, etc) that for performance reasons should always be in the cache. Unfortunately, if the computer doesn't have enough memory, the cache will be too small and performance will suffer. My guess is that with a disk that size, you would want at least 4GB of RAM so your cache can be large enough. More would be better. Do you have that much?

One reason moving the database to a smaller drive would improve performance is that more of that drive's control space could fit in the cache. The fact that you are seeing a performance improvement supports my argument.

[z.nina]

Originally Posted by chaley

OK, to me that implies the fundamental problem is sqlite3 transaction handling. As I mentioned earlier, on windows a transaction consists of a file creation, some writes, then a file deletion. These are "expensive" operations.

NTFS is a complex journaled file system that operates a bit like a database (see http://www.ntfs.com/ntfs.htm for a lot of info). Creating a file requires several read and write operations. Most of the read operations can come from the disk cache. Which leads me to speculate ...

Glancing through this thread I didn't see how much memory your computer has. A 2TB disc will have an enormous amount of control space (file allocation tables, journals, open file tables, etc) that for performance reasons should always be in the cache. Unfortunately, if the computer doesn't have enough memory, the cache will be too small and performance will suffer. My guess is that with a disk that size, you would want at least 4GB of RAM so your cache can be large enough. More would be better. Do you have that much?

One reason moving the database to a smaller drive would improve performance is that more of that drive's control space could fit in the cache. The fact that you are seeing a performance improvement supports my argument.

I've 4 GB of RAM. Mostly I use my pc to read, watch video or play and sometimes I use a 3d graphic software... It's not ultra fast but fast enough. I didn't know a big HD required a lot of RAM. :-) Maybe it's time to upgrade it.

[Katsunami]

Originally Posted by z.nina

I've 4 GB of RAM. Mostly I use my pc to read, watch video or play and sometimes I use a 3d graphic software... It's not ultra fast but fast enough. I didn't know a big HD required a lot of RAM. :-) Maybe it's time to upgrade it.

IMHO, 4GB is the bare minimum nowadays. I've been using Calibre for about fifteen minutes now, and the amount of free (unused) memory in my system has dropped to under 512 MB. All of the rest is in either active use, or used as cache. This system has 8GB of RAM.

If your computer uses DDR3 memory, looking into a memory upgrade is a sensible thing to do; DDR3 is practically free, compared to DDR2. (At least here, in the Netherlands.)

If DDR2 hadn't been so expensive, I'd even have upgraded this almost six year old system to 16 GB.

[Adoby]

Also I suspect that the write cache is disabled by default on external harddrives? Might help to activate it, at the expense of much higher likelihood of corrupt filesystem if the cable is pulled out by mistake.

[theducks]

Originally Posted by z.nina

I've 4 GB of RAM. Mostly I use my pc to read, watch video or play and sometimes I use a 3d graphic software... It's not ultra fast but fast enough. I didn't know a big HD required a lot of RAM. :-) Maybe it's time to upgrade it.

Would a smaller partition help? On the theory 2 smaller is better than a single huge (control space) one.

GPARTED can repartition a working drive. Depending on how full the drive, you may need to do in multiple stages:
1a)Defrag
1b)shrink the partition
1c) create the new partition

2) Move selected (you logic applied) old partition files files to their new home.

repeat 1a, 1b, expand the new partition to use the recovered space

[BetterRed]

Originally Posted by z.nina

If I change any value of the colums in the main windows I've no problem and no delay

@z.nina - Whoa! Does that mean what I think!!! That you can edit the cells in the Book List, with the database in the library folder - without any delays z.nina - can you confirm that ?

My understanding is that editing a single cell in the book list is similar, with respect to disk I/O, as changing a value in Metadata Edit and terminating with OK. Perhaps chaley can confirm that.

And if that's true, then would it not call the disk control memory theory into question - chaley?

BR

[BetterRed]

Originally Posted by z.nina

@BetterRed Sorry for the late answer. I've installed Calibre Portable in the USB3 HD and copied the library in there. The problem is the same. I've to wait more than 30 sec for the metadata editor to move to the next book
The problem is the same even if I use a different PC with win 7 64 bit( I don't have a PC with win7 32 bit so I don't know if that maybe a issue).

I've set the environment variable on my main PC (many many thanks for the detailed explanation !!!!) and the problem is solved

@z.nina - Well we learnt something, as I suggested we might

I'm shocked to 'learn' the lack of disk control space would create delays of 60 seconds to move to the Next Book in Metadata Edit without doing any edits on the current book - when I assume calibre would do no writes. I might have believed 1 or 2 seconds, even 5 but... not a minute or more!

I have a 6GB Windows 7 system with an i5 processor, I have 2x2TB internal SATA2 drives and 2x2TB external drives in a 2 slot USB3 dock attached to a PCIe USB3 adapter. The drives are WD Black Caviar, unpartitioned, included in AV scanning, included in content indexing, have write caching enabled, and the two external drives are also compressed.

I keep my libraries (4,000-35,000 books) on the internal non-system disk, with a copy on one of the USB3 externals.

I don't have any delays of any significance in Metadata Edit (sub-second response), or elsewhere in calibre. Calibre performance is even faster on the external drives - but so is everything else.

I run ~20 programs in my system tray. When I run calibre I'm typically running (in addition to the tray applets), xplorer2, conEMU, firefox, quodlibet, notepad++, utorrent, and at least one or two of, DAMWerks, chrome, excel, word, autocad, outlook, avidemux, mpc, VS2010. I also dive in and out of the ebook-viewer and ebook-editor programs. Calibre's performance is the same at the end of the day as it was at the beginning, I don't close it, or anything else unless there's a reason - eg I change a Tweak.

I run a similar application to calibre to manage a large engineering image collection (+2M images), it keeps all the metadata (much much more per image than any ebook has) in a relational database. I don't have any performance problems with it either.

I'm tempted to remove 2GB to see what difference it does make

Be interesting to know if other users with this 'problem' can overcome it by using the CALIBRE_OVERRIDE_DATABASE_PATH env. variable.

BR

[chaley]

Originally Posted by BetterRed

@z.nina - Whoa! Does that mean what I think!!! That you can edit the cells in the Book List, with the database in the library folder - without any delays z.nina - can you confirm that ?

My understanding is that editing a single cell in the book list is similar, with respect to disk I/O, as changing a value in Metadata Edit and terminating with OK. Perhaps chaley can confirm that.

And if that's true, then would it not call the disk control memory theory into question - chaley?

BR

First, the "disk control memory theory" is just that, a theory. I do know that the larger the volume the more memory required and the more time it takes to process it. What I don't know is where the knee in the memory curve is, of even if there is a knee.

As for differences between editing a single field and the edit metadata window: I expect that there are many. Editing a single field creates a single transaction within which that one field is modified. Edit metadata verifies every metadata item for the book (I think), something I think requires many transactions. It also requires verification of the file structure because the title and author information might have changed. This verification includes checking letter case on case-less file systems (like windows).

In the end, all we seem to know is:

1. Edit metadata is *very slow*
2. Move the DB to C: (?) and edit metadata becomes faster. This implies that the problem is related to the drive.
3. Adding swap space to the drive increased performance by a factor of 2. This implies that something is thrashing, and perhaps that the system's virtual memory size is too small or that some program is wired into real memory.
4. Touching a single field in spreadsheet view is fast. Not sure what this implies other than that a single change using a single transaction is fast enough. This kind of operation certainly uses less of calibre than edit metadata. For example, I am not sure if internal caches are rebuilt with single changes, but I think that they are rebuilt when using edit metadata.

This list (assuming it is correct) doesn't take us to a single possible cause. It does imply that the problem is memory related, but that is only an implication. It could also be related to how the drive is connected, whether there are any other devices on the connection "port", whether error correction is being brought into play (a weak sector in a critical spot), or other difficult-to-ponder possibilities.

[z.nina]

Originally Posted by BetterRed

@z.nina - Whoa! Does that mean what I think!!! That you can edit the cells in the Book List, with the database in the library folder - without any delays z.nina - can you confirm that ?

My understanding is that editing a single cell in the book list is similar, with respect to disk I/O, as changing a value in Metadata Edit and terminating with OK. Perhaps chaley can confirm that.

And if that's true, then would it not call the disk control memory theory into question - chaley?

BR

Yes! I can change them without problem or delay.
Calibre is fast in all the other operations this is what puzzle me!

For example :
- If I donwload the metadata and covers for 20-30 books in a single operation and in the windows that ask me if I want to apply the change I choose "review downloaded metadata" and I chose one by one if I want to accept it or not I've no delay!

- If I select 20-30 books and open "Edit metadata" and change 1 o more value for all the books or use "search and replace" I've no delay.

I don't understand why Calibre slow down only when I open "Edit Metada" windows for a single book even if I don't change anything in it.

[BetterRed]

I removed 2GB of memory, so I now have 4GB

Metadata Edit termination has barely changed, maybe < 1 second to < 2 seconds. To test (in my media library 7000 books) I changed 5 standard columns - including 1 Title correction (INsied to Inside) and 10 custom columns. And I downloaded the metadata for a regular book.

If I edit cells in the book list, the time to Tab to the next cell appears to be a tad slower than it is with 6GB - but it's marginal.

Looking at calibre in Task Manager I don't see it thrashing, the number of read and write operations is modest when I change something. Each field changed in Metadata edit seems to add about 20 writeops to a base of about 100, and 40 readops to a base of about 70. There's 16 writeops for the opf file, when cells are edited this occurs for every cell, with Metadata Edit its once only.

I'm wondering if its "some program is wired into real memory", maybe a service that's always loaded, even in Safe mode.

The gigabyte program that monitors my temps, starts fans etc, thrashes the disks more than calibre does. When I did the 4GB tests my AV scanner was doing its morning run, and the content indexer found itself something to do and wrote 3GB of something.

So... I dunno. Anyway I found the OP a solution, even though I don't really understand why it is so. That's the second one in a few days - giving calibre its own Temp folder was the other one, dunno why it worked either.

BR

[BetterRed]

Originally Posted by z.nina

Yes! I can change them without problem or delay.
Calibre is fast in all the other operations this is what puzzle me!

For example :
- If I donwload the metadata and covers for 20-30 books in a single operation and in the windows that ask me if I want to apply the change I choose "review downloaded metadata" and I chose one by one if I want to accept it or not I've no delay!

- If I select 20-30 books and open "Edit metadata" and change 1 o more value for all the books or use "search and replace" I've no delay.

I don't understand why Calibre slow down only when I open "Edit Metada" windows for a single book even if I don't change anything in it.

@z.nina, as you can see from the above I can't even induce the behaviour you're seeing by ripping out memory.

If you want to observe what calibre's doing with reading and writing you can use Task Manager, here's a How to do that ==>> http://www.watchingthenet.com/identi...n-windows.html

Using what's suggested at that Watching the Net post, it doesn't look like calibre is thrashing my disks - in fact its disk i/o is quite modest. Maybe you'll see something different.

Better put that 2GB back, before I lose it

BR