Best practice for setting up IDs on side loaded books for Kindle?

[Magic815]

Hi all -

So I'm trying to side load a bunch of EPUB files that will be converted to KFX for my Kindle Paperwhite - and I'd like to have GoodReads integration.

I'm struggling to wrap my head around how to best set up my workflow, to ensure that the ID metadata is set up properly to allow for that GoodReads link/integration.

An example of the steps I've tried and what I'm seeing when I try to side load The Hobbit:

1. I add the epub file, and notice that the IDs field is filled with a non-sensical ID, so I manually remove it using Edit metadata.
2. I run the "Extract ISBN" plugin. This netted me "isbn:9780547951973" in my IDs metadata field, which I confirmed is the right ISBN for this ebook file.
3. With Amazon and Goodreads configured as my metadata sources, I hit 'Download metadata' and one option appears. When selected, it overwrites the above extracted ISBN with "isbn:9780618968633, amazon:0618968636, goodreads:13513718" in the IDs metadata field.
4. Upon hitting OK, it seems to have a valid link to Goodreads in Calibre. (When I use the Goodreads Sync plugin to 'view the linked book' - it takes me to here - and that page references the originally extracted ISBN number.)
5. However my IDs field in that file's metadata still references isbn:9780618968633 - which looks to be the hardcover edition of that book. And the Amazon ID in my metadata also references the hardcover edition.

Is there something I can do to allow for the addition of the amazon and GoodReads IDs, while maintaining the initial extracted ebook ISBN? As opposed to the isbn and amazon ID getting overwritten with the hardcover version?

[theducks]

Originally Posted by Magic815

Hi all -

So I'm trying to side load a bunch of EPUB files that will be converted to KFX for my Kindle Paperwhite - and I'd like to have GoodReads integration.

I'm struggling to wrap my head around how to best set up my workflow, to ensure that the ID metadata is set up properly to allow for that GoodReads link/integration.

An example of the steps I've tried and what I'm seeing when I try to side load The Hobbit:

1. I add the epub file, and notice that the IDs field is filled with a non-sensical ID, so I manually remove it using Edit metadata.
2. I run the "Extract ISBN" plugin. This netted me "isbn:9780547951973" in my IDs metadata field, which I confirmed is the right ISBN for this ebook file.
3. With Amazon and Goodreads configured as my metadata sources, I hit 'Download metadata' and one option appears. When selected, it overwrites the above extracted ISBN with "isbn:9780618968633, amazon:0618968636, goodreads:13513718" in the IDs metadata field.
4. Upon hitting OK, it seems to have a valid link to Goodreads in Calibre. (When I use the Goodreads Sync plugin to 'view the linked book' - it takes me to here - and that page references the originally extracted ISBN number.)
5. However my IDs field in that file's metadata still references isbn:9780618968633 - which looks to be the hardcover edition of that book. And the Amazon ID in my metadata also references the hardcover edition.

Is there something I can do to allow for the addition of the amazon and GoodReads IDs, while maintaining the initial extracted ebook ISBN? As opposed to the isbn and amazon ID getting overwritten with the hardcover version?

You can delete specific types of ID's (Identifiers) in bulk (all of a type) via the Tag Browser. Just right-click the type:...

[jhowell]

Originally Posted by Magic815

So I'm trying to side load a bunch of EPUB files that will be converted to KFX for my Kindle Paperwhite - and I'd like to have GoodReads integration.
...
Is there something I can do to allow for the addition of the amazon and GoodReads IDs, while maintaining the initial extracted ebook ISBN? As opposed to the isbn and amazon ID getting overwritten with the hardcover version?

What you are trying to accomplish? Are you trying to get the GoodReads link to work on your Kindle for sideloaded books?

Kindles do not care about ISBN numbers. Books are identified by a 10 character ASIN beginning with the letter B.

Before converting to KFX the book entry needs to have an identifier named either "mobi-asin", "amazon", or "asin" containing the ASIN of the same book sold by Amazon as a Kindle edition. That will be embedded into the book file produced.

Calibre's Download Metadata function is hit or miss at finding ASINs. You often have to set it manually after locating the book on Amazon's website.

[Magic815]

Originally Posted by jhowell

What you are trying to accomplish? Are you trying to get the GoodReads link to work on your Kindle for sideloaded books?

Kindles do not care about ISBN numbers. Books are identified by a 10 character ASIN beginning with the letter B.

Before converting to KFX the book entry needs to have an identifier named either "mobi-asin", "amazon", or "asin" containing the ASIN of the same book sold by Amazon as a Kindle edition. That will be embedded into the book file produced.

Calibre's Download Metadata function is hit or miss at finding ASINs. You often have to set it manually after locating the book on Amazon's website.

Correct, I'm trying to get Goodreads link working with sideloaded books to my Kindle .. AND .. it would be awesome to have a consistently working 'download metadata' feature, to make quick work of setting up my book metadata when I first bring it into my library.

It sounds like you're saying that the amazon ASIN is the only thing needed in the IDs field to allow for GoodReads integration? That's good to know. However, I've been putting in "isbn:xxxx, amazon:yyyy, goodreads:zzzz" and then the 'download metadata' button has worked pretty much 100% of the time in pulling proper metadata, without overwriting the IDs field. But maybe that's overkill?

I was going to ask this in the KFX output plugin thread, but since you're here:
Is there a reason the IDs field gets overwritten with just the mobi-asin/amazon ID after a KFX export? In my scenario above where I've spent the time to enter all three, could KFX export not just keep it as is, with no overwriting?
Also, is there a reason that series, number rating, and tags can't get embedded in the KFX output?

[jhowell]

Originally Posted by Magic815

Is there a reason the IDs field gets overwritten with just the mobi-asin/amazon ID after a KFX export? In my scenario above where I've spent the time to enter all three, could KFX export not just keep it as is, with no overwriting?

Do you mean that if you export a book in KFX format and then import it back into calibre you only see a mobi-asin identifier? If so, that is normal.

KFX has only one ASIN metadata field. When producing KFX output the plugin checks for a "mobi-asin", "amazon", or "asin" identifier in that order. Whichever is found first becomes the ASIN of the KFX book. When importing back into calibre the ASIN metadata field from KFX format becomes the mobi-asin identifier. This is the same behavior as when importing other Kindle formats.

If you are refering to something else that is happening you will need to explain in more detail.

Originally Posted by Magic815

Also, is there a reason that series, number rating, and tags can't get embedded in the KFX output?

Those fields are not supported in KFX format. See "Updating book metadata" in the first post of the KFX Output plugin thread and "Reading book metadata" in the first post of the KFX Input plugin thread for more information.

[Magic815]

Originally Posted by jhowell

Do you mean that if you export a book in KFX format and then import it back into calibre you only see a mobi-asin identifier? If so, that is normal.

KFX has only one ASIN metadata field. When producing KFX output the plugin checks for a "mobi-asin", "amazon", or "asin" identifier in that order. Whichever is found first becomes the ASIN of the KFX book. When importing back into calibre the ASIN metadata field from KFX format becomes the mobi-asin identifier. This is the same behavior as when importing other Kindle formats.

Got it - that's exactly what I was seeing. It's good to know that's the expected behavior.

Originally Posted by jhowell

Those fields are not supported in KFX format. See "Updating book metadata" in the first post of the KFX Output plugin thread and "Reading book metadata" in the first post of the KFX Input plugin thread for more information.

Yea, the first post in your plugin thread is what made me realize that some of the metadata fields weren't supported. I wasn't sure if that was a permanent limit of the KFX container/format, or just a temporary state of where the plugin stands today (i.e. is it possible those fields could be supported in the future, with plugin updates?)

[jhowell]

Originally Posted by Magic815

Yea, the first post in your plugin thread is what made me realize that some of the metadata fields weren't supported. I wasn't sure if that was a permanent limit of the KFX container/format, or just a temporary state of where the plugin stands today (i.e. is it possible those fields could be supported in the future, with plugin updates?)

KFX format as produced by Amazon has only a limited set of metadata. Kindles do not need anything else.

I am reluctant to define my own new metadata fields for KFX beyond what Amazon uses. I cannot be sure how Kindles would react to that unexpected data. Even if it appeared to work for now, Amazon is always making changes to KFX and something might break later.

[Magic815]

Originally Posted by jhowell

KFX format as produced by Amazon has only a limited set of metadata. Kindles do not need anything else.

I am reluctant to define my own new metadata fields for KFX beyond what Amazon uses. I cannot be sure how Kindles would react to that unexpected data. Even if it appeared to work for now, Amazon is always making changes to KFX and something might break later.

I see, fair enough. So if I have a metadata plugboard that writes the series and series_index fields into the file's title when sending to device - I'm guessing Calibre uses it's dB to complete the plugboard, since the KFX file itself does not contain that info directly?

In my scenario, I'm hosting my library via calibre-web so that I can grab my files remotely, if needed. Initially, I was thinking I would just grab the KFX files, since all the work in generating it was already done on my local Calibre install. But I'm realizing that if I intend to make use of a metadata plugboard like that on the remote machine, I'd have to first bring over an EPUB file with full metadata embedded, right? (Or find some other way to have the complete metadata in that remote Calibre dB?)


Side question (and sorry for hitting you with all these, but you've been super helpful): What happens if I bring an EPUB file with it's metadata embedded (including custom metadata: #pages, #gfog) into a new Calibre library instance on another machine that doesn't have those custom columns set up? Is the #page and #gfog data still safe and maintained? Is there an easy way to have Calibre add-back missing columns in a new instance, based on custom file metadata?

[jhowell]

Originally Posted by Magic815

So if I have a metadata plugboard that writes the series and series_index fields into the file's title when sending to device - I'm guessing Calibre uses it's dB to complete the plugboard, since the KFX file itself does not contain that info directly?

Yes.

Originally Posted by Magic815

In my scenario, I'm hosting my library via calibre-web so that I can grab my files remotely, if needed. Initially, I was thinking I would just grab the KFX files, since all the work in generating it was already done on my local Calibre install. But I'm realizing that if I intend to make use of a metadata plugboard like that on the remote machine, I'd have to first bring over an EPUB file with full metadata embedded, right? (Or find some other way to have the complete metadata in that remote Calibre dB?)

I only use calibre one one local machine. You will need to ask others for the best approaches to sharing data across multiple systems.

Originally Posted by Magic815

Side question (and sorry for hitting you with all these, but you've been super helpful): What happens if I bring an EPUB file with it's metadata embedded (including custom metadata: #pages, #gfog) into a new Calibre library instance on another machine that doesn't have those custom columns set up? Is the #page and #gfog data still safe and maintained? Is there an easy way to have Calibre add-back missing columns in a new instance, based on custom file metadata?

As far as I know the extra metadata will be ignored, but that is not anything I have experience with.

[spiritomb]

My workflow for "convert book to use on my Kindle Paperwhite that can be marked on my Goodreads account" has been:

1. convert to azw3
2. download metadata in calibre
3. if metadata includes a mobi-asin/amazon ID that begins with B0, skip to step 6
4. open the amazon page in browser
5. switch amazon page to kindle format and find the on the page, copy it to clipboard
6. open terminal and use the mobimeta utility from the libmobi C library to manually modify the azw3 file's mobi-asin:

mobimeta -s asin=THECORRECTASIN theactualbookfilename.azw3

Books transferred to my Kindle after using this method will be able to connect to Goodreads the same way as Kindle books purchased directly. The device might replace the cover with the one in the Kindle store, but it won't delete it.

I also find it helpful to run the following, so the converted book will allow me to view it with my preferred fonts on-device:

mobimeta -s 528=false theactualbookfilename.azw3