Recipe for Wirtschaftswoche / Wiwo.de (German Business Weekly)

[hegi]

Hi Divingduck,

thanks for your hints. - Well the debug directory is what I already used for my last post. - The bit with the print statements comes in handy, however, when I try to fill these with the regexps, e.g. like this:

    print '*** c-overline tag    --->:', (re.compile(r'(<span class="c-overline">[^<]*)(</span>)', re.DOTALL|re.IGNORECASE), lambda match: match.group(1) + ': ' + match.group(2))
    print '*** hcf-location-mark --->:', (re.compile(r'(<span class="hcf-location-mark">[^<]*)(</span>)', re.DOTALL|re.IGNORECASE), lambda match: match.group(1) + '. ' + match.group(2))

I don't get any meaningful output. - Or can you make sense from stuff like:

*** c-overline tag    --->: (<_sre.SRE_Pattern object at 0x7fdef18de540>, <function <lambda> at 0x7fdee09ddaa0>)
*** hcf-location-mark --->: (<_sre.SRE_Pattern object at 0x7fdee0dcb5e8>, <function <lambda> at 0x7fdee09ddaa0>)

In theory, I'd be a step further, If I could manage to grab the match.group information for the logfile.

I think I'm really stuck here, and this is quite frustrating.

Thanks a lot in advance.

Hegi.

[Divingduck]

Your welcome.

I had a bit time to take a closer look at the problem.

There are two things I saw.
One is, to remember when a regex will happen. You are using preprocess_regexps. This means this refer to the downloaded HTML as source input. Therefore you can check debug\input\ as your source for the regex to find out how the downloaded HTML file looks for calibre at the moment you are manipulate the file.
Second problem is the class you are looking for include spaces in its name and that do not to work (I think that had never work).

Taking that in account, I would make it slightly different. I don't take care about the complete class string, I look only for the end of the class name for a unique identification:

... c-overline--article"> ... </span> ...

(re.compile(r'(c-overline--article">[^>]*)(</span>)', re.DOTALL|re.IGNORECASE), lambda match: match.group(1) + ': ' + match.group(2))

I attach an updated version of the recipe.

[hegi]

Thanks Divingduck,

... as usual, the problem lies in open sight and once you know the solution, everything seems simple and easy.

I took the freedom to merge my earlier fork from your recipe with your actual version, to come up with an improved version. - Please feel free to review and edit or enhance even further.

My evolutionary changes over the last five years:

• add also regexp to add ". " after hcf-location-mark (the Place where the article is set).
• further css entries for teaser text and other elements
• options for conversion and duplicate articles
• optional settings to reduce size on b/w readers
• played a bit with tags filtering

For Amazone Kindle [4|Paperwhite] these settings work nicely:

    # if you want to reduce size for an b/w or E-ink device, uncomment the following 4 lines:
    compress_news_images  = True
    #compress_news_images_auto_size = 16
    scale_news_images     = (400,300)
    compress_news_images_max_size = 35

Currently one of my former versions ships with calibre OOTB. So, once you are happy with the combined efforts as well, we should ask Kovid to integrate the recipe upstream.

Thanks again and looking forward to your comments.

Hegi.

[Divingduck]

Thanks, you are welcome. It's fine for me.

DD

PS: No need to ask for approval. I like your changes for the recipe.

[hegi]

Hi Divingduck,
I noticed for some time, that for some format of articles, pictures are no longer downloaded with the recipe, while for other articles it still works.

Havn't had a chance yet to dig deeper, but wonder, if you maybe had already a look at it?
Cheers,
Hegi.

[Divingduck]

Hi Hegi,
Hope you are well these day's.

Yes I did, but quite some time ago

[Divingduck]

I made a quick update as I saw a small issue today.

@hegi , forgot to mention, you need to integrate your additional code for your kindle. I still use my old Sony device

Best regards,
DD

[hegi]

Hi Divingduck,
thanks a lot for your quick reply and the new version of the recipe.

According to your comments, this is the from March 2018, when we wrote about this the last time. - However, when doing a quick diff on the versions, there seems to be some changes.

I currently load both versions (mine and this one) to compare the output.
Cut it be, that you ommitted updating the comments (date/version) on your last adaptations?

According to my analysis the most relevant difference between our versions is the following code within my recipe:

# don't duplicate articles from "Schlagzeilen" / "Exklusiv" to other rubrics
ignore_duplicate_articles = {'title', 'url'}

However, the problem observed remains. If there are picture-galleries like in this article (https://www.wiwo.de/unternehmen/auto.../26185402.html), you get in the output only the text of the gallery like this:

1 / 8

Volkswagens neues E-Modell ID.4 feierte Ende September digitale Weltpremiere. Vorbestellt werden konnte er schon, nun soll er in den ersten Wochen des neuen Jahres auch zu den Kunden rollen.

Bild: Volkswagen

2 / 8

Wo der ID.3 in der schrumpfenden Kompaktklasse antritt, startet der ID.4 im Boom-Segment der handlichen Geländewagen. Und während es den einen nur in Europa geben wird, feiern die Niedersachsen den anderen als Weltauto. Kein anderes Auto, so meint man bei VW, wird wichtiger im Kampf gegen Tesla & Co. Kein Wunder also, dass der Konzern reichlich trommelt für den elektrischen Weltbürger in Spe und bereits vor der offiziellen Enthüllung im Spätsommer zu einer ersten Ausfahrt im nur noch dezent getarnten Prototypen auf das sonst so streng geheime Testgelände in Ehra-Lessien bat.

Bild: Volkswagen

Any ideas as how to tackle this issue?

Thanks a lot in advance ...

Hegi.

[hegi]

Hi Divingduck,
... just saw that you send another post on Tuesday, while I was preparing mine. Sorry for the confusion caused (if any)

As it appears, your change fixed the picture-gallery-issue ... well at least almost . Within the galleries, there is some "extra content" - mostly "internal adds". - Have a look here: https://www.wiwo.de/politik/deutschl.../26760374.html. - As a result only the first 5 pics go into the ebook, and the additional text is only for 7 out of 17 in ... I suspect these are tweaks to nag readers to buy premium ...

Due to the changes in the Articles (The Teaser-Text no longer seems to start with a location), this additional code of mine (no longer included in your version) for the css seems deprecated:

   .hcf-location-mark {font-style: italic; font-weight:bold}                                 
   .c-overline {font-size: 1em; text-align: left;font-style: normal; font-weight:bold}

However, I'm a bit surprised, that the .c-overline format does not catch any more (also tried other stuff, like italics and smaller fonts), as the insertion of the colon afterwards in your code still works ...

I won't play with this any longer for now ... well let's say for this year .

Thanks and all the best to you
Hegi.