La Jornada

[pablofunes]

Renewed interest in a recipe for Mexican newspaper La Jornada. My recipe below has two problems:

1. The Photos go to the bottom of each article, why?

2. I would like to include the front cover but I don't know how to. The source provides a front cover PDF, here's my bash script to download it:

Y=`date +%Y` # Year 4 digit
m=`date +%m` # Month 2 digit
d=`date +%d` # day 2 digit
wget -q "http://www.jornada.unam.mx/$Y/$m/$d/portda.pdf" -O - | convert pdf:- png:-

Here's my basic recipe:

__license__   = 'GPL v3'
__copyright__ = '2009, Pablo Funes <pablo at imprentaluz.com>'
'''
La Jornada
'''

# TODO: Pictures should go to the top, not the bottom of each article.  
# TODO: Front cover? 

class AdvancedUserRecipe1262065387(BasicNewsRecipe):
    title          = u'La Jornada'
    oldest_article = 7
    max_articles_per_feed = 100

    feeds          = [
                ('opinion','http://www.jornada.unam.mx/rss/opinion.xml'),
                ('politica','http://www.jornada.unam.mx/rss/politica.xml'),
                ('economia','http://www.jornada.unam.mx/rss/economia.xml'),
                ('mundo','http://www.jornada.unam.mx/rss/mundo.xml'),
                ('estados','http://www.jornada.unam.mx/rss/estados.xml'),
                ('capital','http://www.jornada.unam.mx/rss/capital.xml'),
                ('sociedad','http://www.jornada.unam.mx/rss/sociedad.xml'),
                ('ciencias','http://www.jornada.unam.mx/rss/ciencias.xml'),
                ('cultura','http://www.jornada.unam.mx/rss/cultura.xml'),
                ('gastronomia','http://www.jornada.unam.mx/rss/gastronomia.xml')
,
                ('espectaculos','http://www.jornada.unam.mx/rss/espectaculos.xml
'),
                ('deportes','http://www.jornada.unam.mx/rss/deportes.xml'),
                ('cartones','http://www.jornada.unam.mx/rss/cartones.xml'),

                ]


    keep_only_tags = [
                        dict(name='div', attrs={'class':["sumarios","cabeza","te
xt","foto"]}),
                          ]

[kovidgoyal]

You can use the pdf metadata reader in calibre to convert the PDF to a jpeg for the cover.

Not sure about pictures going to bottom, probably they use javascript in the website to move them to the top. You can do something similar by overriding the postprocess_html method in the recipe

[rogeliodh]

Originally Posted by kovidgoyal

You can use the pdf metadata reader in calibre to convert the PDF to a jpeg for the cover.

Where can I find how to use this pdf metadata reader?

[kovidgoyal]

look at the file calibre/ebooks/metadata/pdf.py in the calibre source code

[pablofunes]

Hi Rogelio, I saw your La Jornada script as well... It doesn't have the pictures-at-the bottom problem but it loses all formatting, perhaps that's too radical. Perhaps not. I'd sent you an email as well. Yo do seem to know more Python than me ;-)

We need to figure out several steps: how to generate the YYYY/MM/DD field, how to download the PDF and how to inject it as the Calibre doc cover.

[rogeliodh]

Hi,

This is the new version (rename it to .recipe), it uses the per section feeds and a cover.jpg available in the site.

Next version:
* Use portada.pdf to get a better resolution cover (using kovidgoyal recommendation)
* Get URL for portada.* in a more reliable way (I'm using ugly code right now)
* Do not lose all the article formatting
* 'Cartones' section!

[rogeliodh]

Originally Posted by kovidgoyal

look at the file calibre/ebooks/metadata/pdf.py in the calibre source code

I didn't get howto use the metadata :-(

But I think that it would be better if calibre itself autoconvert cover files to a reliable format (as jpg). So I implemented a change in BasicNewsRecipe so that recipes could tell calibre that the cover needs conversion to jpg

Attached is the change and the recipe using it. It is my first digging into calibre's source code and I'm a python beginner so any comments are welcome, hoping this change to be included in a future calibre release.

Best regards.
Rogelio

[kovidgoyal]

Using imagemagick to render PDf is not very portable as IIRC it requires ghostscript. Here's a patch to feeds/news.py taht will automatically handle a pdf cover, test it and let me know

=== modified file 'src/calibre/web/feeds/news.py'
--- src/calibre/web/feeds/news.py       2009-12-29 16:04:28 +0000
+++ src/calibre/web/feeds/news.py       2009-12-30 22:47:18 +0000
@@ -823,6 +823,14 @@
             cpath = os.path.join(self.output_dir, 'cover.'+ext)
             with nested(open(cpath, 'wb'), closing(self.browser.open(cu))) as (cfile, r):
                 cfile.write(r.read())
+            if ext.lower() == 'pdf':
+                from calibre.ebook.metadata.pdf import get_metadata
+                stream = open(cpath, 'rb')
+                mi = get_metadata(stream)
+                cpath = None
+                if mi.cover_data and mi.cover_data[1]:
+                    cpath = os.path.join(self.output_dir, 'cover.png')
+                    open(cpath, 'wb').write(mi.cover_data[1])
             self.cover_path = cpath

[rogeliodh]

It works perfectly! Just a little typo:

-                from calibre.ebook.metadata.pdf import get_metadata
+                from calibre.ebooks.metadata.pdf import get_metadata

Please let me know if this will be included in a future version.

Thanks,
Rogelio

[kovidgoyal]

It will be in the next release

[rogeliodh]

I've created a new recipe for La Jornada. You can find it here

[rogeliodh]

It is now included in calibre 0.7.15 :-)