bulk meta and series info

[gabby98]

For some reason lately when I do a bulk meta download books of the same series all get number 1 for the series index, however if I go one by one the numbers are correct.

Any suggestions? I really don't want to do 1800 book by hand, that's what I get for doing a mass cleanup

[Adoby]

You can specify what metadata to download. For new books with very bad metadata I usually set ISBN and download everything else. And then carefully normalize what is downloaded to what already is in the library. For books already in the library I usually only download comments and cover.

The series information that is available for download usually is very unreliable. The reason is that there usually are several different possible series names as well as several different series numbering possible. Order of publishing or internal chronology or suggested reading order by the author or by some fan. Also the spelling of the authors may vary. Minor things like different punctuation and initials for middle names may cause calibre to move a book to a completely new author. Even the title can vary. Junk like "A Novel" or "A NN book" or other things may be added.

It's up to you to keep the metadata normalized.

Download and normalize author, title and series information when the book is first added to the library. And at that time make sure the spelling of the authors is the same as the spelling used in all other books by the same author. And that you use the same series name and numbering. And so on... Don't download that metadata again, because it would most likely ruin your normalization.

It might help if you only download metadata from only one source. But even then things like series information may be bonkers. Junk in, junk out.

[speakingtohe]

I often find that the first download does not get the series info and a second one does. I also find I get a lot of bad metadata at times.

Try bulk downloading the metadata for 10-100 books twice and see if this works better perhaps?

When I am doing a bulk download I download the metadata without covers as it makes it easier for me to check if the title at least matches the cover.

theducks has some good tips on metadata downloading and configuring etc. but I have yet to muster the patience to try them.

Helen

[gabby98]

thank you...i set up my meta download to only pull series info from goodreads since it seems most in line with my need (wants) and it seems to pull the right thing for the most part the first time i put the books in calibre but when i went to do the cleanup it went all wonky. I may revert them back to where they were and as I think it was closer to correct than what I am getting now. don't know what changed but all my new stuff seems to be coming through correctly. computers - who knows why they get grumpy D

[Adoby]

The ISBN plugin is a great help. If the ISBN is set correctly then junk in the title or weird spelling of the authors doesn't matter. The ISBN will decide what is downloaded. So at least the metadata that is downloaded is for the right book...

[theducks]

Originally Posted by gabby98

thank you...i set up my meta download to only pull series info from goodreads since it seems most in line with my need (wants) and it seems to pull the right thing for the most part the first time i put the books in calibre but when i went to do the cleanup it went all wonky. I may revert them back to where they were and as I think it was closer to correct than what I am getting now. don't know what changed but all my new stuff seems to be coming through correctly. computers - who knows why they get grumpy D

Keep a close eye on series names. in ANY download :
Consistent Naming of the series across books is not always the case
Consistent Numbering comes in close (usually messed up by stories published in other publications or spin-off of the series. Calibre only allows 1 series index per book)
Consistent use or placement of Articles is another issue. Series, The Series, The Series Books/Novel

[Katsunami]

Personally, I've never trusted bulk data downloads; not with music, and not with books.

With all of my CD's, I've downloaded the metadata for one album at a time only, checked it for correctness and changed it where necessary, and put the album covers into the songs. Rinse, repeat.

I do the same with books, but obviously, I combine series and authors to edit. I even put the cover in myself using the new Edit Book feature. (Before, I'd use Sigil for that.) I really HATE a double cover, as Calibre will just tack a new cover page on top of the book if you let it insert covers.

I also find a list of the series on the internet first, so I can also quickly put in ISBN's and first publication dates. (And if I'm lucky, descriptions as well.) I don't care if it's the ISBN of the particular ebook, or the ISBN of a paperback or something; I just want an ISBN to refer to SOME version of the book. I also use the first publication date. If the paperback wash published in 1977, and the eBook was published in 2010, I put in 1977.

Yes, it takes more time, but I know that everything will be correct.

You know what? Bulk downloading metadata and then having to fix it often costs more time than putting in the correct stuff yourself immediately, especially if you combine authors and series.

How do you get 1800 books done like that? By doing 10 books a day. After some practice, it won't take you more than 30 minutes to do 10 *unrelated* books. Same series and authors will be faster. After 180 days, you're done.

[gabby98]

Originally Posted by Katsunami

Personally, I've never trusted bulk data downloads; not with music, and not with books.

With all of my CD's, I've downloaded the metadata for one album at a time only, checked it for correctness and changed it where necessary, and put the album covers into the songs. Rinse, repeat.

I do the same with books, but obviously, I combine series and authors to edit. I even put the cover in myself using the new Edit Book feature. (Before, I'd use Sigil for that.) I really HATE a double cover, as Calibre will just tack a new cover page on top of the book if you let it insert covers.

I also find a list of the series on the internet first, so I can also quickly put in ISBN's and first publication dates. (And if I'm lucky, descriptions as well.) I don't care if it's the ISBN of the particular ebook, or the ISBN of a paperback or something; I just want an ISBN to refer to SOME version of the book. I also use the first publication date. If the paperback wash published in 1977, and the eBook was published in 2010, I put in 1977.

Yes, it takes more time, but I know that everything will be correct.

You know what? Bulk downloading metadata and then having to fix it often costs more time than putting in the correct stuff yourself immediately, especially if you combine authors and series.

How do you get 1800 books done like that? By doing 10 books a day. After some practice, it won't take you more than 30 minutes to do 10 *unrelated* books. Same series and authors will be faster. After 180 days, you're done.

Yea I usually work somewhat similar to that, but was starting to see the end of my own self imposed cleanup and was hoping to move it along.and you're right I'd probably end up wanting to double check everything anyway.

I also somehow have a ton of star trek books and you know I'll have fun getting the right series in for those. Happily I put them in a different cleanup library and will probably have to take my time on those bit by bit as well

It now seems that what's causing the problem is the erroneous info that's already causing a problem, so maybe I'll do an author a day or something and just get it over with. All my new books are doing what they are supposed to so that's a good thing

Thanks all for your input and for humoring me :-)

[unboggling]

Lots of ditto's. Prior to metadata download I always clean up authors, series, and title, referring to a bibliographic site like ISFDB or Fantastic Fiction. I download metadata only for cover, identifiers, comments, published, and publisher. Even after extracting ISBN first, bulk metadata download requires more time fixing inevitable wrong metadata for some of the books than it would have taken to do a succession of individual metadata downloads without ISBN in the first place. The latter requires some cleaning up, referring to bibliographic site, but not as much.

Edit:

Originally Posted by gabby98

... so maybe I'll do an author a day or something and just get it over with. All my new books are doing what they are supposed to so that's a good thing...

I do that. One author at a time. Slow but sure. And long-term faster.

[theducks]

Originally Posted by gabby98

Yea I usually work somewhat similar to that, but was starting to see the end of my own self imposed cleanup and was hoping to move it along.and you're right I'd probably end up wanting to double check everything anyway.

I also somehow have a ton of star trek books and you know I'll have fun getting the right series in for those. Happily I put them in a different cleanup library and will probably have to take my time on those bit by bit as well

It now seems that what's causing the problem is the erroneous info that's already causing a problem, so maybe I'll do an author a day or something and just get it over with. All my new books are doing what they are supposed to so that's a good thing

Thanks all for your input and for humoring me :-)

Definitely get the Manage Series Plugin, I can help with the rename. But do use care if you use Fractional series Indexes, it will re sequence those the first time encountered (you must use the Lock feature to lock a specific index)

[Katsunami]

Originally Posted by gabby98

I also somehow have a ton of star trek books and you know I'll have fun getting the right series in for those. Happily I put them in a different cleanup library and will probably have to take my time on those bit by bit as well.

That is what I do. I have a library called "Workspace". After I acquire new books I put them into that library (and the DRM gets removed as well). If it is an Amazon book, I'll extract an EPUB using KindleUnpack, or convert to EPUB if that doesn't work. Then I fix the metadata, export the book (the original one and the EPUB), and stash that copy away as original.

Now I delete the non-EPUB format if there is one.

After that, I clean up the book by running Modify EPUB, do things like fixing covers and CSS where necessary, and after I'm done, I move the book or series to my main library. There, it will have its pages counted, be hyphenated, polished to put everything from the database into the book, and then it'll be converted from EPUB to AZW3 (as I have a Kindle).

In this workflow, I can treat an entire series as one book, except for fixing covers and CSS of course, but those are small fixes most of the time, assuming the book is marked up correctly. It's one reason why I always buy an entire series at once.

[gabby98]

Originally Posted by theducks

Definitely get the Manage Series Plugin, I can help with the rename. But do use care if you use Fractional series Indexes, it will re sequence those the first time encountered (you must use the Lock feature to lock a specific index)

Love that plugin....

Originally Posted by Katsunami

That is what I do. I have a library called "Workspace". After I acquire new books I put them into that library (and the DRM gets removed as well). If it is an Amazon book, I'll extract an EPUB using KindleUnpack, or convert to EPUB if that doesn't work. Then I fix the metadata, export the book (the original one and the EPUB), and stash that copy away as original.

Now I delete the non-EPUB format if there is one.

After that, I clean up the book by running Modify EPUB, do things like fixing covers and CSS where necessary, and after I'm done, I move the book or series to my main library. There, it will have its pages counted, be hyphenated, polished to put everything from the database into the book, and then it'll be converted from EPUB to AZW3 (as I have a Kindle).

In this workflow, I can treat an entire series as one book, except for fixing covers and CSS of course, but those are small fixes most of the time, assuming the book is marked up correctly. It's one reason why I always buy an entire series at once.

It's what I do now but when I first started I kept all the formats and didn't know about the polish plugin so I decided it was time to go back and fix all the old ones and so the fun began lol