Need for a dual mobi metadata tweaker

[KevinH]

Hi,

So what happens if you pass the kindlegen output through this script specifying a valid uuid as the asin (just like calibre generates) and load the result into calibre? Won't calibre then properly upload it to the device and sideload everything?

Perhaps calibre looks for its own creator code or some other calibre generated metatag in the exth but that should be easy to mimic in dualmetafix.py.

Alternatively, submitting a patch to calibre should work, as the same fake asin should be added to both headers when it creates a dual mobi.

Sorry this didn't help.

Take care,

KevinH

[kovidgoyal]

If a mobi file has an EBOK and an asin, amazon's software will *always* try to download the cover from the amazon servers, using that asin. The stupid part is that if the download fails (as it always will, for a fake asin), the software does not fallback to using the cover present in the mobi file. Not just that, this change was deliberately introduced by Amazon at some point. Earlier versions of Amazon's apps did not behave in this brain dead fashion.

calibre gets around this by putting a copy of the cover into the cover download cache on the device. This causes the Kindle software to just use that cached cover if the download fails.

Naturally, this only works with eink Kindles since on android/idevices calibre does not know how to upload covers into the cache for the Kindle app. Hence, this FAQ entry: http://manual.calibre-ebook.com/faq....droid-ipad-etc

[KevinH]

Hi Kovid,
When calibre creates a joint/dual mobi (one with both Mobi 7 and Mobi 8 pieces inside), does calibre remember to set EBOK for metadata 501 and the same asin (uuid) in both the mobi 7 header and the later mobi 8 header in the joint mobi file?

I thought from the earlier discussion in this thread, that it only sets that in the first section (mobi 7 header) and not the mobi 8 header that comes after the BOUNDARY section?

I thought from what AcidWeb wrote that calibre did not make those changes in the second header, which is partially causing the problem from what he wrote.

Thanks,

Kevin

[kovidgoyal]

Yes it does. AcidWeb is talking about *updating* metadata in existing joint mobi files, for which calibre will only update the first header record, i.e. the mobi 6 header record. He is creating his joint files using kindlegen, for the comic support, and then trying to use some tool to change the metadata records in the joint file.

I have no interest in adding support to calibre for updating both metadata records in a joint file, but patches are welcome. However, given that MOBI is a "fragile" format and making this kind of change would require extensive testing on various kindle devices/apps to ensure it doesn't break anything, dont expect to get those patches merged easily.

A better approach, IMO, is to simply add support for the KF8 fixed layout format to calibre's azw3 output plugin, when converting comic files (cbr/cbz).

That will require a bit of reverse engineering again, but it should not be too hard. However, I am currently too busy writing Sigil's replacement to get involved.

[AcidWeb]

I have admit that adding hybrid MOBI editing to Cablire could be QA nightmare.

@kovidgoyal - KF8 fixed layout format is only tip of iceberg. To achieve proper output quality entire KCC would need to be rewritten as Calibre plugin. And we not planning to do that for multiple reasons.

AFAIK Paul Durrant will try optimize KindleUnpack code for splitting MOB8 filled with graphic. We will see how that goes.

Output as AZW3 without ASIN is currently best option I see.
In light of recent discoveries editing both of headers does not give us anything. Yes. It is much faster but output is just not practical. I'm terrible sorry for wasting your time KevinH.

[KevinH]

Hi AcidWeb,
No problems. I am sure Paul will be able to speed things up. The original code was not written for speed so I am sure merging the graphics sections into one block or something along those lines should help as it is the allocation and deallocation of long strings that really hurts python performance. Thus the trick of appending segments of strings to a list and then joining the list once at the end so python will only allocate the space needed for the entire block once and won't be allocating and deallocating blocks as every concatenation is done.

Take care,

KevinH




Good luck

[AcidWeb]

ARISE! :-)

Well. At last I have real PW2 so I could research this topic from scratch without tormenting my testers.

@KevinH - it seems that your script is fully usable. No idea what changed but it simply... work. I will incorporate your code in next KCC release (and drop KindleUnpack).

The process looks like this:
* Source EPUB file is converted with newest Kindlegen using -dont_append_source option.
* We receive hybrid MOBI (M6/KF8 headers) without SOURCE section.
* Ouput file is processed by DualMetaFix. We change fields 113, 501, 504 in both headers.

Now if we upload ready file manually to Kindle - it appear without Personal banner. Kindle is trying to download cover but it fails due fake ASIN and fallback to generic placeholder cover. It don't use low-res cover from file at all.

If we upload ready file with Calibre - fields 113, 501, 504 are untouched. And Calibre upload book cover to Kindle cache. And everything is displayed correctly - we have cover and no Personal banner. But we must remember that Calibre upload cover from own metadata database NOT from MOBI file itself.

If we send that file through Amazon Cloud we receive AZW3 file (KF8 only) - personal document with generic placeholder cover.

/offtopic
It would be nice feature if Calibre could import cover from MOBI. I have some code what can cut covers from MOBI files but it is hacky, hack and currently I don't have time to research Calibre code and implement it properly.

[quiris]

Originally Posted by AcidWeb

If we send that file through Amazon Cloud we receive AZW3 file (KF8 only) - personal document with generic placeholder cover.

As workaround of this incredibly stupid issue* I use my own app ExtractCoverThumbs to automatically extract from existing books on Kindle and sideload it back via USB missing cover thumbnails for books downloaded from MYK cloud: https://www.mobileread.com/forums/sho...7&postcount=15

* Answer from Amazon: As per our technical team, "For Pdoc files sent for conversion via email, we strip the cover image. For Pdocs sideloaded via usb, we do not change the files, as the files are not sent to our server, but directly side loaded into device. For Books purchased in our website, we preserve the cover image. For certain mobi files, we do not reconvert it, but directly send the file received. Hence cover image is not stripped. This is the expected behavior of our system."

[AcidWeb]

Yeah - I seen your code. It work correctly but I prefer approach that just get cover from file without unpacking entire MOBI.

I work on 300MB+ MOBI files and performance is important.

[quiris]

Originally Posted by AcidWeb

Yeah - I seen your code. It work correctly but I prefer approach that just get cover from file without unpacking entire MOBI.

All I need is a simple function to extract cover image from Mobi/AZW3. KevinH, could you point me how to do it? Unfortunately, I don't know binary Mobi format so I used KindleUnpack for that.

[AcidWeb]

I have experimental grade code for that:
https://github.com/AcidWeb/KindleBut...r/File.py#L118

[quiris]

So is it absolutely necessary to process entire resource section to find all images? Are they 'mixed' with "FLIS", "FCIS", "FDST", "DATP" etc?

[AcidWeb]

Considering that CoverOffset is not static and not always first image record is cover - yes.
Also this code not really process them. It is more like building index of resources to interpret CoverOffset number.

[EbokJunkie]

KindleButler fails to load external/custom covers. Tried multiple cover formats, always get Failed to load custom cover.

[AcidWeb]

That entire project is experimental, untested and not finished.

It is not officially released yet.