Slow to write tags to disk?

[Sirtel]

Originally Posted by Karellen

It's not a general and widespread problem. Seems a problem on your system. I am a Windows 10 and now 11 user and haven't experienced the problem you describe.

Me either.

[BetterRed]

Originally Posted by abrogard

Here we are. Safe mode. No Firefox. Still takes forever.
And here is the process explorer output.
I did get the library onto another machine and ran it there.
No trouble at all.
So without doubt it is a Windows problem or, we could say I suppose, a possible windows/calibre conflict. There's not just me experiencing this.
Anyway here is the process explorer output for that session in safe mode.
It'll be lovely if it reveals something to someone somewhere, this is a real hassle.

The process explorer output confirms it's not a 3rd party program sticking its oar in, and the fact that all's well on another system tells us that the library isn't corrupted. Which leaves us with Windows… or hardware. My money's on the latter. If you have a portable HDD, move the library onto it and see what happens.

BR

[abrogard]

Originally Posted by Karellen

It's not a general and widespread problem. Seems a problem on your system. I am a Windows 10 and now 11 user and haven't experienced the problem you describe.

sure. but there's a thread here about it somewhere from another guy. i was there for a while but i was butting in on their conversation so i started my own.

and how many users who never resort to forums we don't know, of course. same with all apps.

[abrogard]

Originally Posted by BetterRed

The process explorer output confirms it's not a 3rd party program sticking its oar in, and the fact that all's well on another system tells us that the library isn't corrupted. Which leaves us with Windows… or hardware. My money's on the latter. If you have a portable HDD, move the library onto it and see what happens.

BR

can you tell how you tell from the explorer output that its not the library? I'd just like to be able to read the explorer output intelligently myself. at the moment its just a list to me.

hardware? so use a portable hdd? like a usb hdd? I've got a number of them. I can try it on one of them.

[BetterRed]

Originally Posted by abrogard

can you tell how you tell from the explorer output that its not the library?

I deduced it wasn't the library from what you posted, that I bolded in my reply:

I did get the library onto another machine and ran it there.
No trouble at all.

Originally Posted by abrogard

I'd just like to be able to read the explorer output intelligently myself. at the moment its just a list to me.

I changed procexp17401303.txt to procexp17401303.csv, and opened it in nirsoft's CSVFileView did some column sorts (name and company)

Apart from a few familiar unattributed processes (registry, interrupts etc) everything else was Microsoft, calibre, or Sysinternals. I also booted my rig into Safe mode, ran calibre and had a look at a csv from process explorer - more or less identical to yours.

Originally Posted by abrogard

hardware? so use a portable hdd? like a usb hdd? I've got a number of them. I can try it on one of them.

Go for it

BR

[abrogard]

Apart from a few familiar unattributed processes (registry, interrupts etc) everything else was Microsoft, calibre, or Sysinternals. I also booted my rig into Safe mode, ran calibre and had a look at a csv from process explorer - more or less identical to yours.

Ah, right. Comparison with a known good. Fair enough. I was just wondering if there was any more to than that. Some known evil files or something.

Thanks for that.

Yep. I'll put it on a usb external - and copy it to the other internal, I have two on that machine. I'll test it all over the place.

[PeterT]

I wonder if using Process Monitor from SysInternals might provide some insight into file activity

Sent from my Pixel 7a using Tapatalk

[sgmoore]

What sort of changes are you doing?

I got hit with an issue where I tagged a book without looking at what I was typing and without realising that my capslock was on. So the book ended up being tagged mYSTERY instead of Mystery.

Calibre then spent a long time updating the opf files for all the other books that had been tagged as Mystery to make them all say mYSTERY

So sometimes updating one book can affect lots of other records.

[theducks]

Originally Posted by sgmoore

What sort of changes are you doing?

I got hit with an issue where I tagged a book without looking at what I was typing and without realising that my capslock was on. So the book ended up being tagged mYSTERY instead of Mystery.

Calibre then spent a long time updating the opf files for all the other books that had been tagged as Mystery to make them all say mYSTERY

So sometimes updating one book can affect lots of other records.

But a case change is just 1 DB write. This is a Relational DB so the Tag table was the min change.
Dirty bit also gets set for all uses (that triggers the backup creation metadata.opf)

[DNSB]

Wouldn't the issue as suggested by sgmoore's example of having caps lock on be a one shot issue. After the change was made and everything was updated, that slowness would disappear unless something was triggering a change on every startup.

[theducks]

Originally Posted by DNSB

Wouldn't the issue as suggested by sgmoore's example of having caps lock on be a one shot issue. After the change was made and everything was updated, that slowness would disappear unless something was triggering a change on every startup.

I would think so.
OP have you looked at the file timestamps for metadata.opf in the book folder?
That might give a clue if these books are stuck in some weird update loop.
Does anyone know where the 'dirty' flag is kept? Maybe that is RO

[PeterT]

I know at times I've seen local hard drives flagged as "removable" by Windows which could possibly change how the IO subsystem handles writes to the device. Does the hard drive show up in the ejectable icon in the system tray?

[BetterRed]

Originally Posted by PeterT

I wonder if using Process Monitor from SysInternals might provide some insight into file activity

Been there, done that, see the log in OP's post #33 -

Originally Posted by sgmoore

What sort of changes are you doing?

I got hit with an issue where I tagged a book without looking at what I was typing and without realising that my capslock was on. So the book ended up being tagged mYSTERY instead of Mystery.

Calibre then spent a long time updating the opf files for all the other books that had been tagged as Mystery to make them all say mYSTERY

So sometimes updating one book can affect lots of other records.

Mass updates to metadata.opf files are done via a separate background process… slowly, can take minutes, hours, days even, and if you close calibre, next time it starts it will carry on updating them regardless.

And it's not just Tags, it's any column e.g. Comments; no casing issues there or triggering of mass metadata backup writes.

Originally Posted by DNSB

Wouldn't the issue as suggested by sgmoore's example of having caps lock on be a one shot issue. After the change was made and everything was updated, that slowness would disappear unless something was triggering a change on every startup.

The list from Process Explorer in Safe Mode, where the problem persists, suggests not… and its almost identical to me doing the same on my system.

Originally Posted by theducks

I would think so.
OP have you looked at the file timestamps for metadata.opf in the book folder?
That might give a clue if these books are stuck in some weird update loop.
Does anyone know where the 'dirty' flag is kept? Maybe that is RO

Snippet from Process Monitor log, only one book gets its metadata backup written, duration is the last column, as I already said it's diddlysquat… 0.002 secs.

10:53:50.60	calibre.exe	12148	WriteFile	D:\NOV 2022 CALIBRE LIBRARY\Otto Carius\Tigers in the Mud (1261)\metadata.opf	SUCCESS	Offset: 0, Length: 1,030, Priority: Normal	0.0001689
10:53:50.60	calibre.exe	12148	WriteFile	D:\NOV 2022 CALIBRE LIBRARY\Otto Carius\Tigers in the Mud (1261)\metadata.opf	SUCCESS	Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal	0.0012767
10:53:53.24	calibre.exe	12148	WriteFile	D:\NOV 2022 CALIBRE LIBRARY\Otto Carius\Tigers in the Mud (1261)\metadata.opf	SUCCESS	Offset: 0, Length: 1,108, Priority: Normal	0.000089
10:53:53.24	calibre.exe	12148	WriteFile	D:\NOV 2022 CALIBRE LIBRARY\Otto Carius\Tigers in the Mud (1261)\metadata.opf	SUCCESS	Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal	0.0007957

The 'dirty' flag is in the database, in a table named metadata_dirtied:

CREATE TABLE metadata_dirtied(id INTEGER PRIMARY KEY,
                             book INTEGER NOT NULL,
                             UNIQUE(book))

BR