5 hours updating metadata after downloading meta data for 300 books on i-3770k

[N13L5]

I'm puzzled,

I used to update a lot of meta data on an extremely crappy netbook on an older version of Calibre in 2010, and it used to apply it without noticeable delay, meaning as soon as I hit the 'next' button, it was applied and showing me the next book instantly.

So I'm extremely surprised the current version seems so slow on an exceedingly fast system, that encodes a 1080p movie in ~20 minutes.

Apparently, the current version of Calibre lets you make your changes for all selected books without applying them, until you reach the end and only then it starts...

This design is a bit precarious, cause if you edited Metadata for 300 books and then your system crashes or electricity goes out, before it completes all of them, you just lost many hours of work. So now my system is grinding away for hours, even though its pretty much idle...

Has anyone else experienced this issue? I'm close to 5 hours in and its at 64%...

I only have the browser running and both CPU and Memory are less than 20% commited.

[N13L5]

The new Metadata window is really nice though and all the new download options are awsome. 98% of the time it gets the correct stuff too...

[Adoby]

I don't think that the computer is the limiting factor at all, more your Internet connection and the response time for the metadata source(s) you have active.

The metadata is only updated in the metadata.db file, and that's not demanding at all.

[theducks]

Originally Posted by Adoby

I don't think that the computer is the limiting factor at all, more your Internet connection and the response time for the metadata source(s) you have active.

The metadata is only updated in the metadata.db file, and that's not demanding at all.

Adoby
Updating metadata of 300 books, will mark those records. At low priority, Calibre will start backing those up (metadata.opf) in the books folder.
You can see the backlog: right-click Calibre Library: Library Maintenance: Library Metadata Backup Status.
I think one of the sources is having troubles (?Goodreads?)

[BetterRed]

@N13L5 The new bulk metadata has 3 stages - gather, review and update - I'm assuming you're referring to the last one.

I've not done that many books in a batch. But I've recently been doing batches of 30-50 books on a friends 7 year old Tosh Portege (dual Core 1.2GHz Centrino, 2GB of RAM) running Vista and Calibre 0.9.3n 32bit. I've not noticed the sort of performance you seem to be getting on those batch sizes, the update phase typically takes < 5 minutes.

And I'm usually running Firefox, Open Office Writer and almost 30 tray applets. I don't know if its relevant but I run calibre jobs at Low Priority (Pref->Behave) and One at Time (Pref Miscellaneous). My perception is that those settings give me better performance.

At a glance I don't think calibre accesses the internet in the update phase, my up/down network indicators show minimal activity, never more than 2.00kb/s and most of the time at 0.00 - assuming there's no other activity (browsers, ftp etc).

Looks to me like some sort of resource contention, try shutting down other applications - what OS are you on?

BR

[N13L5]

Originally Posted by Adoby

I don't think that the computer is the limiting factor at all, more your Internet connection and the response time for the metadata source(s) you have active.

My internet is very fast, but that can't be it anyway, because I'm not waiting on the downloads.

The steps go like this:

1) download metadata and book covers for 300 books. (that took less than 30 minutes, I didn't measure it exactly)

2) go through all the books, pick or correct any of the Metadata.

3) After you hit 'next' after the last book, it automatically goes to apply the metadata. There is no more downloading at this point. This is the process that took almost 8 hours to complete.

[N13L5]

Originally Posted by BetterRed

@N13L5 The new bulk metadata has 3 stages - gather, review and update - I'm assuming you're referring to the last one.

I believe your entire Post is correct on every point.

I was not running anything but browser and skype in the background during that time...

And this computer is fast enough, where I can have 2 browsers with 100 tabs open, plus word, excel, skype, viber, and a few utilities at the same time while playing a demanding computer game like crysis or gw2 or skyrim.

Resource contention is a good idea, I just can't detect anything.

I will follow your advice for changing the Calibre settings and try with 50 books next.

Thanks for your detailed answer

[N13L5]

Originally Posted by theducks

Adoby
Updating metadata of 300 books, will mark those records. At low priority, Calibre will start backing those up (metadata.opf) in the books folder.
You can see the backlog: right-click Calibre Library: Library Maintenance: Library Metadata Backup Status.
I think one of the sources is having troubles (?Goodreads?)

Excellent suggestion, I will check into that too!

thanks!

[kovidgoyal]

Disable your antivirus. There's no way that updating metadata on 300 books can possibly take 8 hours unless you have some external program interfering or your library is ona very slow disk, like a networked disk.

[BetterRed]

Originally Posted by theducks

Adoby
Updating metadata of 300 books, will mark those records. At low priority, Calibre will start backing those up (metadata.opf) in the books folder.
You can see the backlog: right-click Calibre Library: Library Maintenance: Library Metadata Backup Status.
I think one of the sources is having troubles (?Goodreads?)

@theducks - with 300 books the reading of the updates from the database by the backup process maybe contending with the process that is updating the database with the remaining reviewed downloads.

When does the writing of the database changes to the .opf files (backing up) start in this scenario - after the first update is applied or after they all been applied. If former maybe database contention is a possibility.

I'm sure I've seen a post in relation to one of the plugins that suggests limiting the number of books in a batch to - 50ish.

BR

[theducks]

BR I don't know when it starts, but it is a throttled task so as to not impact normal use.

[BetterRed]

Originally Posted by theducks

BR I don't know when it starts, but it is a throttled task so as to not impact normal use.

Aaaah, but is the applying of 300 updates 'normal use' - the background task that writes the new opf files yields if I start doing something, but does it yield if another task or thread is doing something - I dunno either.

We'll have to wait for the Magister Ludi to enlighten us.

BR

[kovidgoyal]

The background task always yields. It sleeps after doing one opf.

[Elwynd02]

Waaaaah! This made me very sad. I think this probably means my reviewing has been lost. (I accidentally closed the 'review metadata' window about halfway through. Tedious to do in the first place, really annoying to have to re-do!)

Would be really good it if could write the changes after a batch of 10 or something.

[BetterRed]

@Elwynd02 - you're conflating when the database (metadata.db) is updated after a bulk metadata download and when the backup metadata (metadata.opf) for each book gets rewritten.

As I've posted in your other thread - I believe the database is updated when the review is complete, that happens lickety-split.

The rewriting of the metadata.opf files is slow in order that it doesn't hog resources, but if you close calibre down before its rewritten everything, then when you next start calibre the rewriting of the metadata.opf files will resume.

The need to rewrite the metadata.opf files can be triggered by many things, some are obvious - such as downloading metadata, others not so obvious - removing a tag will trigger an update to every book that uses the tag.

BR