Enthusiast
Posts: 25
Karma: 1896
Join Date: Aug 2011
Device: Kindle 3
Thanks Starson!
Quote:
Originally Posted by
Starson17
Try this:
Code:
for section in soup.findAll('div', attrs={'class':'seccionx'}):
section_title = self.tag_to_string(section.find('a'))
That should find the first <a> tag in the each div of class seccionx.
Tried that, but nothing happens. The thing that puzzles me is that I cannot have ANY "hits" (aka, section detected), no matter what. For instance, I tried with...
Quote:
for section in soup.findAll('li'):
...and again, nothing:
Spoiler :
Code:
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
1% Fetching feeds...
1% Got feeds from index page
Python function terminated unexpectedly
No articles found, aborting (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\ebooks\conversion\cli.py", line 286, in main
File "site-packages\calibre\ebooks\conversion\plumber.py", line 937, in run
File "site-packages\calibre\customize\conversion.py", line 204, in __call__
File "site-packages\calibre\web\feeds\input.py", line 105, in convert
File "site-packages\calibre\web\feeds\news.py", line 737, in download
File "site-packages\calibre\web\feeds\news.py", line 882, in build_index
ValueError: No articles found, aborting
Here's the complete recipe (with your suggestion):
Spoiler :
Code:
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''
pagina12.com.ar
'''
import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import Tag, NavigableString
class Pagina12(BasicNewsRecipe):
title = 'Pagina12'
__author__ = 'Juan De Los Palotes'
description = 'Una solucion europea a los problemas argentinos'
INDEX = 'http://www.pagina12.com.ar/diario/secciones/'
language = 'es'
#remove_tags_before = dict(id='menu_rosario')
#remove_tags_after = dict(id='suples')
#remove_tags = [dict(id=['header', 'printAds', 'pageControls'])]
no_stylesheets = True
preprocess_regexps = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
#def print_version(self, url):
# return url.replace('/archive/', '/print/')
def parse_index(self):
articles = []
numero = 1
soup = self.index_to_soup(self.INDEX)
#ts = soup.find(id='magazineTopStories')
#ds = self.tag_to_string(ts.find('h1')).split(':')[-1]
#self.timefmt = ' [%s]'%ds
cover = soup.find('img', src=True, attrs={'class':'cover'})
if cover is not None:
self.cover_url = cover['src']
feeds = []
seen_titles = set([])
for section in soup.findAll('div', attrs={'class':'seccionx'}):
section_title = self.tag_to_string(section.find('a'))
numero+=1
print (numero)
self.log('Found section:', section_title)
articles = []
for post in section.findAll('div', attrs={'class':lambda x : x and
'post' in x}):
h = post.find('h3')
title = self.tag_to_string(h)
if title in seen_titles:
continue
seen_titles.add(title)
a = post.find('a', href=True)
url = a['href']
if url.startswith('/'):
url = 'http://www.pagina12.com.ar'+url
p = post.find('p', attrs={'class':'dek'})
desc = None
self.log('\tFound article:', title, 'at', url)
if p is not None:
desc = self.tag_to_string(p)
self.log('\t\t', desc)
articles.append({'title':title, 'url':url, 'description':desc,
'date':''})
if articles:
feeds.append((section_title, articles))
return feeds
def postprocess_html(self, soup, first):
for table in soup.findAll('table', align='right'):
img = table.find('img')
if img is not None:
img.extract()
caption = self.tag_to_string(table).strip()
div = Tag(soup, 'div')
div['style'] = 'text-align:center'
div.insert(0, img)
div.insert(1, Tag(soup, 'br'))
if caption:
div.insert(2, NavigableString(caption))
table.replaceWith(div)
return soup
And the HTML for the index page:
Spoiler :
Code:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="es" lang="es">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="-1" />
<meta name="robots" content="noarchive" />
<meta name="description" content="Página/12 Web" />
<meta name="keywords" content="página/12, página12, página, 12, diario, argentina, noticias, política, economía, deportes, cultura, espectáculos, internacional, ciencia, últimas noticias, las12, radar, supleno, m2, newspaper, news, politics, business, sport, culture, entertainment, international, world, science" />
<meta name="copyright" content="2000-2011 Página/12" />
<meta name="_created_by" content="192.168.12.102" />
<meta name="_served_by" content="192.168.12.102" />
<meta name="_date_creation" content="Mon, 29 Aug 2011 21:31:32 -0300" />
<meta name="_revision" content="2285" />
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
<link rel="icon" href="/favicon.ico" type="image/x-icon" />
<script type="text/javascript" src="/commons/js/prototype.js"></script>
<script type="text/javascript" src="/commons/js/pag12.js"></script>
<script type="text/javascript" src="/commons/js/clippings.js"></script>
<link rel="stylesheet" href="/commons/css/base.css" type="text/css" />
<link rel="stylesheet" href="/commons/css/clippings.css" type="text/css" />
<link rel="alternate" type="application/rss+xml" title="RSS" href="/diario/rss/principal.xml" />
<title>Página/12</title>
</head>
<body onload="clippings.load_cookie()">
<div id="contenedor_principal">
<div><script language="JavaScript" src="http://ads.pagina12.com.ar/banner.php?s=102&w=232&h=60&t=_blank&cla=bannerp10%20fright%20right12"></script></div>
<div><script language="JavaScript" src="http://ads.pagina12.com.ar/banner.php?s=103&w=232&h=60&t=_blank&cla=bannerp10%20fright%20right12"></script></div>
<div><script language="JavaScript" src="http://ads.pagina12.com.ar/banner.php?s=110&w=476&h=60&t=_blank&cla=bannerp10%20fright%20right12"></script></div>
<div id="logo">
<a href="/" title="Página/12 Web"><img src="/commons/imgs/logo-home.gif" alt="Página/12"/></a>
<p id="fecha_logo">
<a href="/diario/principal/index-2011-08-28.html" title="Edición anterior"><img src="/commons/imgs/ed_anterior.gif" alt="anterior"></a>
<img src="/commons/imgs/ed_palito.gif" alt="|" />
<img src="/commons/imgs/ed_siguiente_off.gif" alt="siguiente deshabilitado" />
Lunes, 29 de agosto de 2011
</p>
</div>
<div class="clear"></div>
<div id="pre_menu">
<ul>
<li id="menu_usuario_1" class="fff"></li><li id="menu_usuario_2" class="fff"></li><li><a href="/usuarios/anteriores.php" class="curva1" title="Seleccione desde un almanaque la edición que desea"><img src="/commons/imgs/ico-anteriores.gif" /> Ediciones anteriores</a></li><li><a href="/buscador/index.php" title="Búsqueda por sección/suple"><img src="/commons/imgs/ico-buscar.gif" /> Busqueda avanzada</a></li><li><a href="/usuarios/correo.php" title="Escriba desde aquí a las distintas secciones/suples"><img src="/commons/imgs/ico-correo.gif" /> Correo</a></li><li><a href="http://www.kiosco12.com/" title="Venta de productos Página/12"><img style="padding-top:4px" src="/commons/imgs/ico-kiosco.gif" /> Kiosco|12</a></li>
</ul>
</div>
<script type="text/javascript">
//<![CDATA[
user.menu();
//]]>
</script>
<script type="text/javascript">
//<![CDATA[
var active_menu='secciones';
//]]>
</script>
<div id="menu">
<div id="buscar"><form action="/buscador/resultado.php" method="get" id="form_search" name="form_search"><a href="javascript
I'm using NOTEPAD++ for both editing the recipe file, and analizing the HTML code.
08-27-2011, 05:15 PM
Similar Threads