Calibre a bit slow editing in large Collections

Evilwarning · 10-26-2010, 03:29 PM

Hi

I have a small collection of 20k books/articles and while trying to edit the metadata i find that after editing an item it takes between 10-15 seconds until i can select another item and start editing it. Is this normal ? Makes editing a bit slow.

i have tried to create small library and do some editing just for testing and doesn't behave this way, ready to edit more items just after making changes. I have defraged the hard drive although in Mac OS ins't much of a problem but just to test.

Any Ideas ? Or the library is just too large and i should just create several smaller libraries ?

speakingtohe · 10-26-2010, 04:01 PM

Quote:

have a small collection of 20k books/articles and while trying to edit the metadata i find that after editing an item it takes between 10-15 seconds until i can select another item and start editing it. Is this normal ? Makes editing a bit slow.

Normal for me. Still your collection is very small so should not be that normal for you.
My fiction library is over 8GB. (multiple formats)

Quote:

Any Ideas ? Or the library is just too large and i should just create several smaller libraries ?

20k is pretty small IMO so I would stick with one.
I have 3
Fiction
Non Fiction
Cookbooks

Switching libraries can take a lot longer than waiting for the GUI to respond so I just wait.

Starson17 · 10-26-2010, 04:29 PM

Quote:

Originally Posted by speakingtohe

My fiction library is over 8GB. (multiple formats)

20k is pretty small IMO so I would stick with one.

I think he meant 20,000 books, possibly each one having multiple formats, not 20KB in size.

That's 30% larger than mine. I'm not as slow as he is, but I'm a bit slower than I'd like. (OTOH, I'm infinitely faster than before the speed improvements of a few months ago.)

speakingtohe · 10-26-2010, 04:35 PM

@Starson17
Ahh I thought so too but then I reread and this stood out

Quote:

I have a small collection of 20k books/articles

But maybe he was being facetious

Starson17 · 10-26-2010, 04:47 PM

Quote:

Originally Posted by speakingtohe

But maybe he was being facetious

On the internet no one knows when you're ..... being facetious.

speakingtohe · 10-26-2010, 05:44 PM

Quote:

On the internet no one knows when you're ..... being facetious.

Not me anyway

Evilwarning · 10-29-2010, 01:52 PM

Hi

Yes i mean 20000 Books but it uses 60GB of disk space some PDF's are quite large.

Since i load everything in bulk i have to spend quite some time editing the Metadata and there is were i see the slowdown. 10 - 15 seconds per item. I still have to review at least 8000 items. Maybe my Add wasn't as good as it could but but its difficult to find the documents with a nice file name to load.

I thought of multiple Libraries, that would be fine but the search only works on one library at the time, maybe a new feature search multiple libraries.

I think calibre is a great tool just it's taking me a lot of time to edit the metadata and some tweaks would help.

Starson17 · 10-29-2010, 02:18 PM

Quote:

Originally Posted by Evilwarning

I think calibre is a great tool just it's taking me a lot of time to edit the metadata and some tweaks would help.

What tweaks did you have in mind? I put in code changes I could think of that would speed things up. They included:

1) Improved Firstname Lastname <-> Lastname, Firstname reversing logic.
2) Merge records code to allow duplicate records to be merged together when you get duplicates.
3) Prevent Author/Title overwrites during bulk metadata online fetch operations to allow mass updates of other metadata without screwing up a correct author/title.
4) Autosort/Automerge of incoming formats into existing records with fuzzy title matching to minimize duplicate record creation. Drag and drop blocks of books and they are sorted into the existing records or new records are created, as needed.
5)Autosort/Automerge applied during Copy to Library operations to allow the Autosort/Automerge to run on an existing library (not just during the initial Add) by copying all books to a newly created library.

If you can think of other improvements that will speed things up, they will always be considered.

Gwen Morse · 10-30-2010, 02:33 AM

Quote:

Originally Posted by Starson17

What tweaks did you have in mind? I put in code changes I could think of that would speed things up. They included:

5)Autosort/Automerge applied during Copy to Library operations to allow the Autosort/Automerge to run on an existing library (not just during the initial Add) by copying all books to a newly created library.

If you can think of other improvements that will speed things up, they will always be considered.

When Copy to Library is used, does it also update the metadata in the books (the way "copy to disk" or "copy to device" is supposed to do)?

Evilwarning · 10-30-2010, 04:30 AM

Hi

For example would make editing faster if the updates to the database or file renaming would be done in the background allowing the GUI to be always responsive. For example the records select would be locked and we would not be able to update them while the task was running but we could start editing other items, or for example the changes would be queued and we could continue editing.

Another nice feature would the ability to select part of the title and define that hey are the Title + Author + ISBN( For example) . I know that the Add has regular expressions but if the import wasn't that well made (usually user not understanding the regular expressions, :c) ) would be nice to have a way of selecting the text and the system would create the expression and correct the data. I use a tool for MP3 that is called TriTag that has something similar to help the user correct the ID3 Tag from the file name, single or bulk, create the expression on the background and shows the changes before applying.

2 things that would make Calibre not better, cos already does the tasks, but more user friendly

Starson17 · 10-31-2010, 11:15 AM

Quote:

Originally Posted by Gwen Morse

When Copy to Library is used, does it also update the metadata in the books (the way "copy to disk" or "copy to device" is supposed to do)?

I don't know. I didn't work on anything beyond the metadata merging, which happens inside the database. My guess is that it does not change the metadata in the formats, but it's only a guess. That's how I would have done it. Why don't you test and find out?

Quote:

(the way "copy to disk" or "copy to device" is supposed to do)?

Why do you say "supposed to do"? It works for me.

Gwen Morse · 11-03-2010, 09:11 PM

Quote:

Originally Posted by Starson17

I don't know. I didn't work on anything beyond the metadata merging, which happens inside the database. My guess is that it does not change the metadata in the formats, but it's only a guess. That's how I would have done it. Why don't you test and find out?

Why do you say "supposed to do"? It works for me.

I'm not sure how to check the metadata in the books themselves. Ebooks are still somewhat mysterious packages to me

. I can see the metadata in the calibre database, but I don't know how to check if the book metadata is different.

As for saying "supposed to do"...that's just my way I speaking. I'm overly cautious and I don't make declarative statements unless I'm certain of the results. Since I don't really know what happens to the metadata, I don't feel comfortable saying does happen, only what I think is supposed to happen

.

kovidgoyal · 11-03-2010, 10:00 PM

copy to library does not update file metadata and you can check the metadata in a file by opening it with the calibre viewer and click the information button.

Gwen Morse · 11-04-2010, 04:27 AM

Quote:

Originally Posted by kovidgoyal

copy to library does not update file metadata and you can check the metadata in a file by opening it with the calibre viewer and click the information button.

Oh, nifty.

Karma to you for that information.

10-26-2010, 03:29 PM	#1
Evilwarning Member Posts: 13 Karma: 10 Join Date: Oct 2010 Device: iPad	Calibre a bit slow editing in large Collections Hi I have a small collection of 20k books/articles and while trying to edit the metadata i find that after editing an item it takes between 10-15 seconds until i can select another item and start editing it. Is this normal ? Makes editing a bit slow. i have tried to create small library and do some editing just for testing and doesn't behave this way, ready to edit more items just after making changes. I have defraged the hard drive although in Mac OS ins't much of a problem but just to test. Any Ideas ? Or the library is just too large and i should just create several smaller libraries ?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Editing SONY collections on device in calibre	Sparkling	Calibre	2	06-19-2010 11:54 AM
Collections seem to be a bit messed up for me now	speakingtohe	Calibre	6	05-26-2010 01:42 AM
Slow editing	avresbo	Sigil	14	02-03-2010 10:07 PM
Large FB2s slow?	edwlarkey	Astak EZReader	2	10-09-2009 12:29 AM
v0.1.2 very slow when editing on Mac	DanielCoffey	Sigil	13	08-18-2009 05:27 PM

10-29-2010, 01:52 PM	#7
Evilwarning Member Posts: 13 Karma: 10 Join Date: Oct 2010 Device: iPad	Hi Yes i mean 20000 Books but it uses 60GB of disk space some PDF's are quite large. Since i load everything in bulk i have to spend quite some time editing the Metadata and there is were i see the slowdown. 10 - 15 seconds per item. I still have to review at least 8000 items. Maybe my Add wasn't as good as it could but but its difficult to find the documents with a nice file name to load. I thought of multiple Libraries, that would be fine but the search only works on one library at the time, maybe a new feature search multiple libraries. I think calibre is a great tool just it's taking me a lot of time to edit the metadata and some tweaks would help.

10-30-2010, 04:30 AM	#10
Evilwarning Member Posts: 13 Karma: 10 Join Date: Oct 2010 Device: iPad	Hi For example would make editing faster if the updates to the database or file renaming would be done in the background allowing the GUI to be always responsive. For example the records select would be locked and we would not be able to update them while the task was running but we could start editing other items, or for example the changes would be queued and we could continue editing. Another nice feature would the ability to select part of the title and define that hey are the Title + Author + ISBN( For example) . I know that the Add has regular expressions but if the import wasn't that well made (usually user not understanding the regular expressions, :c) ) would be nice to have a way of selecting the text and the system would create the expression and correct the data. I use a tool for MP3 that is called TriTag that has something similar to help the user correct the ID3 Tag from the file name, single or bulk, create the expression on the background and shows the changes before applying. 2 things that would make Calibre not better, cos already does the tasks, but more user friendly

11-03-2010, 10:00 PM	#13
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	copy to library does not update file metadata and you can check the metadata in a file by opening it with the calibre viewer and click the information button.

Advert

Advert