Kindle Fire periodicals need EXTH[501]=NWPR

[nickredding]

Kindle Fire uses the EXTH[501] field to decide if the newsreader should be used for a document. At present calibre doesn't set that for recipe output at all, so recipe output is viewed by Kindle Fire as a book. To get the (much nicer) newsreader calibre has to set EXTH[501]='NWPR'. This will also cause the document to be placed in the "Newsstand" tab on the home page of the Fire.

There are two places where this should be set--in mobi/writer.py and mobi/writer2/main.py. (The code seems to have been reorganized since I last worked on it so I'm unclear as to whether the tweak new_mobi_writer is still relevant.)

In mobi/writer.py, replace lines 1578-1583 with

        # Write cdetype
        if not self.opts.mobi_periodical:
            data = 'EBOK'
            exth.write(pack('>II', 501, len(data)+8))
            exth.write(data)
            nrecs += 1
        else: # fix for Kindle Fire--use newsreader for this
            data = 'NWPR'
            exth.write(pack('>II', 501, len(data)+8))
            exth.write(data)
            nrecs += 1

In mobi/writer2/main.py, replace lines 537-547 with

        # Write cdetype
        if not self.is_periodical and not self.opts.share_not_sync:
            exth.write(pack(b'>II', 501, 12))
            exth.write(b'EBOK')
            nrecs += 1
        elif self.is_periodical:
            # fix for Kindle Fire--use newsreader for this
            exth.write(pack(b'>II', 501, 12))
            exth.write(b'NWPR')

The Fire newsreader is obviously derived from the Kindle Android newsreader, but it's interesting to note the latter doesn't require the EXTH[501] to be set. It looks like Kindle Fire is using the EXTH[501] to categorize documents, and then decides how to display them based on the category rather than looking at the MOBI type.

[nickredding]

Update: this works as expected when transferring files to the Fire via USB but for some reason emailing the files doesn't have the same result. In that case the files still show up vanilla docs. I will investigate and report back.

[kovidgoyal]

writer.py is now no longer used, unless asked for explicitly by a tweak. By default calibre uses writer2.py. There is another thread on this forum which has some info about news downloads needing to have a .pobi extension to be recognized.

[kovidgoyal]

Also, I have seen values NWPR and MAGZ for that field (corresponding to doctype of 0x101 and 0x103.

[kovidgoyal]

Committed code to set 501 for periodicals.

[nickredding]

Amazon steps on any EXTH[501] value in a document being emailed to a Kindle and replaces it with "PDOC". Therefore, you'll only get the newsreader/newsstand treatment if you copy the files to the Fire manually.

Amazon has rendered the calibre email delivery option useless (unless you are willing to put up with navigating your periodicals as an ebook). I can't think of any way of fixing this short of developing an app that modifies the files after they are delivered, and even that would involve manually invoking the app for each periodical file.

I'm pretty sure Amazon won't be sympathetic since it doesn't affect Amazon content, so I'm not going to bother complaining in their forums (they typically ignore complaints that involve non-Amazon content).

[wallcraft]

Originally Posted by nickredding

I can't think of any way of fixing this short of developing an app that modifies the files after they are delivered, and even that would involve manually invoking the app for each periodical file.

There might be room for an app that edits the EXTH[501] value for files on the device and triggers the local sync that is currently done by connecting and disconnecting USB to a computer. Even folks who did not care about EXTH might like the local sync as an alternative to a reboot. The app developer would have to work out how to do the on-device sync though.

[DiapDealer]

Amazon has rendered the calibre email delivery option useless (unless you are willing to put up with navigating your periodicals as an ebook).

Hasn't emailing a document to a Kindle device always resulted in EXTH[501] being set to PDOC, though? That's why emailing always used to break the different synchronization kludges in the past.

Nevermind: I see what you're saying. In the past, EXTH[501] wasn't used to differentiate between ebook or periodical. So it didn't matter (in that regard) that it was being set to PDOC... the reader app on the Kindle still knew it was a periodical.

[nickredding]

Originally Posted by kovidgoyal

Committed code to set 501 for periodicals.

There's still a problem with the fix. Calibre-generated periodicals all have the MOBI type set to 0x103 which results in EXTH[501]=MAGZ instead of NWPR. This causes the Fire to display the periodical TOC as a linear list of articles without a drop-down menu to jump to a specific section. I suspect the MAGZ classification is intended for periodicals that have only one section. For multi-section periodicals we really need EXTH[501]=NWPR.

I looked at the code and I see

        if oeb.metadata.publication_type:
            x = unicode(oeb.metadata.publication_type[0]).split(':')
            if len(x) > 1:
                pt = x[1].lower()
        self.publication_type = pt

and then the MOBI type is set as

{'newspaper':0x101}.get(self.publication_type, 0x103)

so obviously the oeb.metadata.publication_type isn't being set to 'newspaper' since the type always ends up as 0x103.

I don't see any preferences that would affect oeb.metadata.publication_type, so I suspect there needs to be a change to ensure multi-section periodicals get the type 0x101, or make all periodicals (0x101 and 0x103) have EXTH[501]=NWPR.

[nickredding]

Originally Posted by nickredding

Amazon has rendered the calibre email delivery option useless (unless you are willing to put up with navigating your periodicals as an ebook).

I should have qualified this by saying this only applies to the Fire (and possibly the KTouch which mnay have the same behaviour of using EXTH[501] to decide what reader to use).

[kovidgoyal]

OK, committed code to set it to 0x103 and MAGZ only if the periodical is flat.

[nickredding]

Thanks for the quick fixes. If only Amazon was as responsive!