Replacement of Replacement Character

[Leonatus]

Originally Posted by lui1

So any webpage which claims to be encoded with ISO-8859-1 should be treated as being encoded with Windows-1252.

encoding = 'windows-1252'

That's interesting! Is it also valid for transactions as downloading news in Calibre
and transferring to an e-book-reader?

[Leonatus]

encoding = 'windows-1252'

Alas! Didn't work either. So, it seems impossible to get rid of this replacement character?

[Leonatus]

Sorry, but the problem as such tickles me.
Could it be that there is something wrong with the preprocess_regexps?
I tried to simply replace A by B, using:

preprocess_regexps = [
   (re.compile(r'A', re.DOTALL),
    lambda match: 'B'),

and it didn't work.

[kovidgoyal]

make sure you have the right indentation for preprocess_regexps it should be at the same level as the title for example.

[Leonatus]

It looks like this:

class AdvancedUserRecipe1295262156(BasicNewsRecipe):
    title = u'kath.net'
    __author__ = 'Bobus'
    description = u'Katholische Nachrichten'
    oldest_article = 7
    language = 'de'
    max_articles_per_feed = 100
    no_stylesheets = True
    encoding = 'iso-8859-1'
    preprocess_regexps = [
         (re.compile(r'A', re.DOTALL),
          lambda match: 'B'),

Edit: No, by inserting into the CODE the last two lines went two spaces forward.
Edit': The missing square bracket is, in fact, there (fault at copy/paste).

[kovidgoyal]

post the actual recipe file as a zipped up attachment, too much chance of things chaning with copy paste

[Leonatus]

Here we are!

[kovidgoyal]

The folowing recipe works for me with quotes preserved:

from calibre.web.feeds.news import BasicNewsRecipe


class AdvancedUserRecipe1295262156(BasicNewsRecipe):
    title = u'kath.net'
    __author__ = 'Bobus'
    description = u'Katholische Nachrichten'
    oldest_article = 7
    language = 'de'
    max_articles_per_feed = 100
    no_stylesheets = True
    encoding = 'cp1252'

    feeds = [(u'kath.net', u'https://www.kath.net/2005/xml/index.xml')]

    def print_version(self, url):
        return url + "/print/yes"

    def get_browser(self, *a, **kwargs):
        kwargs['verify_ssl_certificates'] = False
        return BasicNewsRecipe.get_browser(self, *a, **kwargs)

    extra_css = 'td.textb {font-size: medium;}'

[Leonatus]

Nope, no success here. Same appearance as always.

[Leonatus]

Perhaps is it important to know that I reiceive the news via Joel Goguen's "KoboTouch-extended"-plugin in the Kobo-epub format (kepub), as usual with basically all my books.

[kovidgoyal]

look at the downloaded epub file using the calibre viewer first.

[Leonatus]

Yes, that's what I do, and that's why my last post is superfluous, because above I did already tell that the appearance in Calibre's ebook viewer is the same as on my ereader. I apologize!

[Leonatus]

Originally Posted by kovidgoyal

The folowing recipe works for me with quotes preserved:

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe


class AdvancedUserRecipe1295262156(BasicNewsRecipe):
    title = u'kath.net'
    __author__ = 'Bobus'
    description = u'Katholische Nachrichten'
    oldest_article = 7
    language = 'de'
    max_articles_per_feed = 100
    no_stylesheets = True
    encoding = 'cp1252'

    feeds = [(u'kath.net', u'https://www.kath.net/2005/xml/index.xml')]

    def print_version(self, url):
        return url + "/print/yes"

    def get_browser(self, *a, **kwargs):
        kwargs['verify_ssl_certificates'] = False
        return BasicNewsRecipe.get_browser(self, *a, **kwargs)

    extra_css = 'td.textb {font-size: medium;}'

Are you sure? There are various authors writing in various styles, some of them using the "classical" keyboard quotes - which are, indeed, preserved. I tried with three computers, and it's always the same result, even taking in account your proposal: The more "typographical" quotes are transformed into the replacement character, might they be single or double.
I played a lot, but found no resoöution, whereas, downloading another newspaper with similar structure (i. e. using "classical" and "typographic" quotes, everything is allright, the encoding beeing ISO-8859-1. The recipe of it is:

#!/usr/bin/env python2
# vim:fileencoding=utf-8
# License: GPLv3 Copyright: 2016, Kovid Goyal <kovid at kovidgoyal.net>

from __future__ import (unicode_literals, division, absolute_import,
                        print_function)
from calibre.web.feeds.recipes import BasicNewsRecipe


def classes(classes):
    q = frozenset(classes.split(' '))
    return dict(attrs={'class': lambda x: x and frozenset(x.split()).intersection(q)})


class BerlinerZeitung(BasicNewsRecipe):
    title = 'Berliner Zeitung'
    __author__ = 'Kovid Goyal'
    language = 'de'
    description = 'Berliner Zeitung RSS'
    timefmt = ' [%d.%m.%Y]'
    ignore_duplicate_articles = {'title', 'url'}
    remove_empty_feeds = True

    # oldest_article = 7.0
    no_stylesheets = True
    remove_javascript = True
    use_embedded_content = False
    publication_type = 'newspaper'

    keep_only_tags = [
        classes('dm_article_body dm_article_header'),
    ]
    remove_tags = [
        classes('dm_article_share'),
    ]

    feeds = [x.split() for x in [
        'Berlin http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23699382-asYahooFeed.xml',
        'Brandenburg http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23699570-asYahooFeed.xml',
        'Politik http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23699614-asYahooFeed.xml',
        'Wirtschaft http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23699644-asYahooFeed.xml',
        'Sport http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23699874-asYahooFeed.xml',
        'Kultur http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23700020-asYahooFeed.xml',
        'Panorama http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23700178-asYahooFeed.xml',
        'Wissen http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23700222-asYahooFeed.xml',
        'Digital http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23700594-asYahooFeed.xml',
        'Ratgeber http://www.berliner-zeitung.de/blueprint/servlet/xml/berliner-zeitung/23700190-asYahooFeed.xml',
    ]]

[Leonatus]

Eventually, I came to the conclusion that the problem cannot be resolved, as the characters in question are already replaced on the rss-page of the journal. There appear - strange enough - empty quadrats instead of the quotes, for example. Why this happens, and why there are no problems when the rss-feed is subscribed on a computer, this will remain a secret of the deepest depths of the internet, at least for my simple mind.