Causes for library updates to be slow

[Rellwood]

What would cause the "journal-db metadata" file to idk how to put it, but when I am trying to import a list sometimes when the list takes way too long to import and update this file shows up in the library folder and indicates each book being updated (i assume). I noticed that when I am doing a lot of mass updates (like importing lists) that unless I give it a break and let the metadata for the books catch up then the import list ends up being very slow, and that journal-db metadata file shows up.

I just imported a simple list of titles/authors/dates/source with only updates to the source and date. However it took a very long time. I know the metadata was up to date (I checked using preferences), so, my question is, what would cause Calibre to slow down like that when all the metadata in the books is caught up (with the exception of the changes I was making)?

ALso, I know this seems like a plugin question, but it's really about that file that that indicates Calibre is running slow to update.

[ownedbycats]

metadata.db-journal is a rollback journal for sqlite. If something goes wrong mid-change (e.g. power interuption - FAT32 filesystems were known for corrupting after them) it can restore and prevents things from breaking. It deletes itself when done.

Could be a number of things causing slowness. Overly complex composite columns needing to be calculated, a database that hasn't been vacuumed in a while, the drive being stored on a slow (or failing) hard drive.

[capink]

Have a look at this post in the Import List plugin. It outlines the two issues affecting speed and how to deal with them.

There is a tip for people using linux that dramatically increases the speed of importing lists. If you can somehow have ramdisk in Windows —I don't know how because I don't use Windows— you can replicate the steps.

[Rellwood]

Originally Posted by ownedbycats

metadata.db-journal is a rollback journal for sqlite. If something goes wrong mid-change (e.g. power interuption - FAT32 filesystems were known for corrupting after them) it can restore and prevents things from breaking. It deletes itself when done.

Could be a number of things causing slowness. Overly complex composite columns needing to be calculated, a database that hasn't been vacuumed in a while, the drive being stored on a slow (or failing) hard drive.

Ok, well the library has been vaccumed, there are no composite columns outside of identifiers, the hard drive is good. I have a 1 year old computer with Windows 10 and a 1tb SSD and 16G RAM.

Oh, well, it is transient, so I was just wondering.

[ownedbycats]

Originally Posted by capink

Have a look at this post in the Import List plugin. It outlines the two issues affecting speed and how to deal with them.

There is a tip for people using linux that dramatically increases the speed of importing lists. If you can somehow have ramdisk in Windows —I don't know how because I don't use Windows— you can replicate the steps.

There's Dataram's RAMDisk and AMD Radeon RAMDisk. (Note that you don't actually require an AMD Radeon to use the latter.)

Freeware versions are limited in size, but you're only copying metadata.db so you'll only need a fraction of what they offer you.

[Rellwood]

Originally Posted by ownedbycats

There's Dataram's RAMDisk and AMD Radeon RAMDisk. (Note that you don't actually require an AMD Radeon to use the latter.)

Freeware versions are limited in size, but you're only copying metadata.db so you'll only need a fraction of what they offer you.

1. What I read about the plugin performance issue is pretty much what I am dealing with, so I think this is the cause.

2. I have no clue how that ramdisk works, and I have no idea how to import a list from the command line, so I guess I am stuck with the occasional bottlenecking.

[capink]

You will not import the books from the command line. You simply will copy one command to the command line, after which a dialog will popup where you can complete the process. The instructions for this is detailed in the linked post. Note that the path in the first step applies to linux and should be replaced by the path to your ramdisk in Windows, if you somehow managed to setup that.

I cannot offer any guidance on how to make ramdisk in Windows as I don't use it.

Edit: Also make sure you not using the same library you importing to in calibre gui. And that you have the latest version of the Import List plugin.

Edit2: If you cannot setup a ramdisk in Windows and want to just prevent the speed deterioration over time, you skip the step of copying your metadata.db to ramdisk, and simply copy the command and start from there. In that case, you will leave the override db checkbox unchecked, and make sure you tick the checkbox for re-initialize the db (lowering the number to 500 if you are not using ramdisk).