kepubify 2.0 - A standalone tool to convert ePubs to Kepubs

[libertango]

My first conversion with Kepubify today, on windows. I just dragged and dropped the file on the exe icon, and obtained the kepub. When I open the kepub on my pc, a big part of the text shows as red characters, while some of it is black. The issue looks like random (change of color even midway sentence). When I open the book in my Kobo Clara, the red characters show as grey, so they are not easily readable.
The conversion in Calibre with Kepub Output works fine.

[DNSB]

One question is why Kepubify adds <div class="book-columns"> and <div class="book-inner"> to the start of each file? Looking at several Kobo kepub, they added <div xmlns="http://www.w3.org/1999/xhtml" id="book-columns"> and <div xmlns="http://www.w3.org/1999/xhtml" id="book-inner">.

[geek1011]

Originally Posted by DNSB

One question is why Kepubfy adds <div class="book-columns"> and <div class="book-inner"> to the start of each file? Looking at several Kobo kepub, they added <div xmlns="http://www.w3.org/1999/xhtml" id="book-columns"> and <div xmlns="http://www.w3.org/1999/xhtml" id="book-inner">.

OK, the class vs id seems like a mistake I made, probably when doing the recent refactor of the conversion code. The xmlns is unnecessary there. Thanks for letting me know! I'll release a fix now.

Update: fix released in v2.3.2

[Cyfranek]

Hi,
I've got a problem with some EPUBs. KEPUBs created by Kepubify seems to be partially empty inside. There are some text at the beginning and at the end of file. But core is empty. The same EPUB converted by Calibre works fine.
I've just notice that while converting one Polish newspaper (see example in attachment).

[geek1011]

Originally Posted by Cyfranek

Hi,
I've got a problem with some EPUBs. KEPUBs created by Kepubify seems to be partially empty inside. There are some text at the beginning and at the end of file. But core is empty. The same EPUB converted by Calibre works fine.
I've just notice that while converting one Polish newspaper (see example in attachment).

Can you post the original epub? I think this is problem is due to kepubify's stricter parsing of HTML.

[Cyfranek]

There is an original EPUB:

[geek1011]

Kepubify v2.4.0 has been released

This release mainly affects seriesmeta and the new covergen tool.

Seriesmeta now works even before books are imported (i.e. you don't have to reconnect to update series metadata for new books), and it now supports EPUB3-style series metadata. macOS is now supported as well.

Covergen (re)generates cover images for nickel, with optional stretching to fit a specific aspect ratio (I use 1.5). It works on both EPUBs and KEPUBs. This speeds up browsing the library, and if stretching is used, will also make it more consistent. In addition, covergen is useful when the automatically generated cover images are not satisfactory (too small, white margins, etc).

In addition, the README has been completely redone.

Usage: seriesmeta [OPTIONS] [KOBO_PATH]

Options:
  -h, --help         Show this help message
  -p, --no-persist   Don't ensure metadata is always set (this will cause series metadata to be lost if opening a book after an import but before a reboot)
  -n, --no-replace   Don't replace existing series metadata (you probably don't want this option)
  -u, --uninstall    Uninstall seriesmeta table and hooks (imported series metadata will be left untouched)

Arguments:
  KOBO_PATH is the path to the Kobo eReader. If not specified, seriesmeta will
  try to automatically detect the Kobo.

Usage: covergen [OPTIONS] [KOBO_PATH]

Options:
  -a, --aspect-ratio float   Stretch the covers to fit a specific aspect ratio (for example 1.3, 1.5, 1.6)
  -h, --help                 Show this help message
  -m, --method string        Resize algorithm to use (bilinear, bicubic, lanczos2, lanczos3) (default "lanczos3")
  -r, --regenerate           Re-generate all covers

Arguments:
  KOBO_PATH is the path to the Kobo eReader. If not specified, covergen will try
  to automatically detect the Kobo.

For those interested in the details of how the new seriesmeta works, see here: https://github.com/geek1011/kepubify...a.go#L141-L229.

[rtiangha]

Ok, so let me see if I have the workflow correct:

• I click and drag my epub(s) over Kepubify.
• I copy it over to my Kobo reader.
• I disconnect the device and let it scan.
• I reconnect the device and run seriesmeta.

Is that correct?

Questions:

• If I just want to update the books that don't have the series data set (i.e. it's set to NULL in the database), I'd use the 'n' switch, right? Mainly asking because I have some books converted by the previous version of kepubify on the reader that I haven't updated series data for yet.
• Does seriesmeta populate other fields too like ISBN (if present) and author? I use Calibre to manage my collection but Kepubify to do the conversion (because it's faster on the netbook I now use exclusively to house my collection) but I'm finding that if a book has multiple authors, the Kobo sometimes only picks up one of the authors, or if I delete some of the authors (ex. Calibre pulls down someone who writes a foreword or is an illustrator as an additional author), the Kobo will still pick up the deleted author and sometimes use it as the sole author instead (which means I have no idea where it's pulling it from; that data is supposed to be gone). It'd just be nice to not have to re-edit author data after the fact too.
• Can seriesmeta (and covergen) only target one book (or a few) if I give a path(s), or does it have to scan the entire reader? I do have a really large collection, so not having it scan the entire collection each time I add just a couple of books or need to correct/change the cover for one book because it didn't scan correctly would be nice and save a lot of time. It's why I just edit the database manually (but also to correct the author issue above so it'd need to be done anyway; although it just hit me: maybe the whole author thing is an epub3 issue? I haven't been paying attention to book format).
• Am I interpreting this right: any new books converted with the new version of kepubify no longer need to have seriesmeta run on it anymore? Let me guess: SQL triggers or something like that? If so, that's really cool.

Feature request: Since covergen is doing transformations on the image already, any chance for it to have the option to convert to greyscale too, just to save a bit of space?

Anyways, I love kepubify; thanks for making it!

[geek1011]

Originally Posted by rtiangha

Ok, so let me see if I have the workflow correct:

• I click and drag my epub(s) over Kepubify.
• I copy it over to my Kobo reader.
• I disconnect the device and let it scan.
• I reconnect the device and run seriesmeta.

Is that correct?

Questions:

• If I just want to update the books that don't have the series data set (i.e. it's set to NULL in the database), I'd use the 'n' switch, right?
• Does seriesmeta populate other fields too like ISBN (if present) and author? I use Calibre to manage my collection but Kepubify to do the conversion (because it's faster on the netbook I now use exclusively to house my collection) but I'm finding that if a book has multiple authors, the Kobo sometimes only picks up one of the authors, or if I delete some of the authors (ex. Calibre pulls down someone who writes a foreword or is an illustrator as an additional author), the Kobo will still pick up the deleted author and sometimes use it as the sole author instead (which means I have no idea where it's pulling it from; that data is supposed to be gone). It'd just be nice to not have to re-edit author data after the fact too.
• Can seriesmeta (and covergen) only target one book (or a few) if I give a path(s), or does it have to scan the entire reader? I do have a really large collection, so not having it scan the entire collection each time I add just a couple of books or need to correct/change the cover for one book because it didn't scan correctly would be nice and save a lot of time.

Feature request: Since covergen is doing transormations on the image already, any chance for it to have the option to convert to greyscale too, just to save a bit of space?

Anyways, I love kepubify; thanks for making it!

For the first question, that used to be the case, but now you can run seriesmeta before reconnecting (but note that unless you add the -p or the -n option, it will take precedence over metadata changes made by Calibre). Also, you can set the output directory directly if you want (and you can detect the Kobo patch and wait for it using kobo-find from koboutils). See here for how I do it (this runs in a udev hook for when I plug my Kobo in).

Yes to the question about the -n switch.

And no, seriesmeta doesn't currently support other metadata fields, as Kobo should be able to extract them itself if set properly (IIRC). I can make you a custom build to set them in seriesmeta if you want (it's a simple addition).

Seriesmeta doesn't currently have an option to limit which books are scanned, but that's in the plans for the next version. The same applies to covergen.

Additional filters (like grayscale, dithering, jpg quality, and inversion) are coming in the next version as well.

[rtiangha]

Oh, I think I get it. So I would copy over the kepub as before, but immediately run seriesmeta on the device after, then? That makes sense to me.

No need for a custom build; the Kobo will display ISBN in the book details if it's there, but it doesn't seem to pick it up on my sideloaded books; I just assumed it was like the series data and needed to be updated after the fact. It'd just be a cosmetic nice-to-have, but no need if it means extra work.

[davidfor]

Originally Posted by rtiangha

• Does seriesmeta populate other fields too like ISBN (if present) and author? I use Calibre to manage my collection but Kepubify to do the conversion (because it's faster on the netbook I now use exclusively to house my collection) but I'm finding that if a book has multiple authors, the Kobo sometimes only picks up one of the authors, or if I delete some of the authors (ex. Calibre pulls down someone who writes a foreword or is an illustrator as an additional author), the Kobo will still pick up the deleted author and sometimes use it as the sole author instead (which means I have no idea where it's pulling it from; that data is supposed to be gone). It'd just be nice to not have to re-edit author data after the fact too.

The author problem is probably two things. Firstly, calibre writes multiple authors into the metadata in separate dc:contributor tags. But, the firmware only reads the first tag. Hence, you will only get one author on the device and it will be whatever is in the first tag. The workaround for this is to use a metadata plugboard to set the author during the send-to-device. This changes the metadata during the update to use a single tag with all of the authors. The one I suggest using is:

program: raw_list('authors', ', ')

That will set the author to look like "FN1 LN1, FN2 LN2" which the device will parse as two authors.

The other problem is that calibre does not automatically update the book file with any metadata changes. It only does this when the book is being sent out of the library (Save-to-disk or Send-to-device), during a conversion or when you use the Embed metadata tool. If you are using kepubify to transform a book in the calibre library to kepub, you need make sure the metadata is written to the file first.

[rtiangha]

Originally Posted by davidfor

The author problem is probably two things. Firstly, calibre writes multiple authors into the metadata in separate dc:contributor tags. But, the firmware only reads the first tag. Hence, you will only get one author on the device and it will be whatever is in the first tag. The workaround for this is to use a metadata plugboard to set the author during the send-to-device. This changes the metadata during the update to use a single tag with all of the authors. The one I suggest using is:

program: raw_list('authors', ', ')

That will set the author to look like "FN1 LN1, FN2 LN2" which the device will parse as two authors.

Yeah, I never noticed this until I switched away from using Calibre to transfer books (because on reconnect, your plugin would fill in everything correctly), only because on this netbook, it'll take forever to scan on device connect (again, ridiculously large collection with an even more ridiculously sized database), which is fine if I was uploading a batch of books from time to time, but not if it was a book at a time multiple times a week (orders of magnitude and all that).

The other problem is that calibre does not automatically update the book file with any metadata changes. It only does this when the book is being sent out of the library (Save-to-disk or Send-to-device), during a conversion or when you use the Embed metadata tool. If you are using kepubify to transform a book in the calibre library to kepub, you need make sure the metadata is written to the file first.

But that's what I do; Save to Disk and then drag to kepubify. It does save the book description so I know that part works. It was just the weirdness with the author field and I figured it had to be something like you described above.

Just curious: Can a plugboard like you suggested above be used to alter the author metadata on Save-to-Disk instead to populate that first field properly? I'm using one to append Word and Page count to the description already (I think you might have provided the sample code in another thread way back in the day, so thanks!), but I don't really understand how the plugboard system works and if it's all the same or how to alter multiple fields (for example, is it safe to have two rules that affect any format, Save to Disk that affects different Destination fields? The warning Calibre throws when you try to do so confuses me).

[davidfor]

Originally Posted by rtiangha

But that's what I do; Save to Disk and then drag to kepubify. It does save the book description so I know that part works. It was just the weirdness with the author field and I figured it had to be something like you described above.

Just curious: Can a plugboard like you suggested above be used to alter the author metadata on Save-to-Disk instead to populate that first field properly? I'm using one to append Word and Page count to the description already (I think you might have provided the sample code in another thread way back in the day, so thanks!), but I don't really understand how the plugboard system works and if it's all the same or how to alter multiple fields (for example, is it safe to have two rules that affect any format, Save to Disk that affects different Destination fields? The warning Calibre throws when you try to do so confuses me).

You can have a metadata plugboard for the save-to-disk, just like the device ones. That will work the same. You can copy the details of the existing plugboard to it.

The alternative is to use one plugboard for "any device". That will be used for either save-to-disk or send-to-device. I think this where you are getting the warning. If you have a more specific plugboard, such as one for the KoboTouch driver, you will get an warning when adding the "any device" plugboard. It can still be added, and I'm pretty sure the more specific plugboard will be used. I don't think I have ever tried this.

[konstantinus]

Originally Posted by geek1011

Seriesmeta now works even before books are imported (i.e. you don't have to reconnect to update series metadata for new books)

Series metadata is only visible after rebooting or reconnecting. So it should be?

[geek1011]

Originally Posted by konstantinus

Series metadata is only visible after rebooting or reconnecting. So it should be?

Yes, if the book is being initially imported. The advantage is you don't have to connect multiple times.