Error in my recipe for le Point

rhuang76 · 02-20-2014, 06:25 AM

Hello, I am a newbie in Calibre.

I tested a script under python 2.7. Then I ported the main function into a calibre recipe. But but it doesn't work.

All the urls cannot be downloaded, But actually, these urls are alive. It shows like:
"Could not fetch link http://www.lepoint.fr/municipales-20...94120_1966.php
Failed to download article: Municipales 2014 - Béziers : Aboud-Ménard, un duel au coude-à-coude from http://www.lepoint.fr/municipales-2014/municipales-2014-beziers-aboud-menard-un-duel-au-coude-a-coude-20-02-2014-1794120_1966.php"

I know I am stupid, but I need some help.

Code:

class AdvancedUserRecipe1392888003(BasicNewsRecipe):
    title          = u'Le Point'
    oldest_article = 3
    max_articles_per_feed = 10
    auto_cleanup = True

    feeds          = [(u'Actualit\xe9', u'http://www.lepoint.fr/rss.xml')]


    def preprocess_html(self, soup):
        html = u'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n <html xmlns="http://www.w3.org/1999/xhtml"> \n  <head> \n <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />'
        html += str(soup.title).decode('utf-8')+u'\n'
        html += u'<body>' 

        article_title_meta = soup.find('meta', attrs={'property': 'og:title'})  
        article_title = article_title_meta['content']
        html += u'<h1> ' + article_title + u'</h1>'

        article_title_img_meta = soup.find('meta', attrs={'property': 'og:image'})
        if article_title_img_meta:
            article_title_img = article_title_img_meta['content']

        html += u'<img src=' + article_title_img + u'>'

        article_body = soup.find('span', attrs={'itemprop': 'articleBody'})
        html += str(article_body).decode('utf-8')
            
        return BeautifulSoup(html)

Krittika Goyal · 02-21-2014, 04:59 AM

Hi I dont know enough about recipes to fully understand your code but the following works:

Code:

class AdvancedUserRecipe1392888003(BasicNewsRecipe):
    title          = u'Le Point'
    oldest_article = 3
    max_articles_per_feed = 10
    auto_cleanup = True

    feeds          = [(u'Actualit', u'http://www.lepoint.fr/rss.xml')]

Do you really need the rest?

rhuang76 · 02-21-2014, 09:22 AM

In fact, that part works. But every article is without the title and image. Because the title and image are located in the <meta> tag within the <head>. I want to modify the tags and put them in the <body>.

kovidgoyal · 02-21-2014, 10:41 AM

Run your recipe using

ebook-convert file.recipe .epub -vv --test

and you will get a traceback showing you what the problem is.

rhuang76 · 02-21-2014, 09:13 PM

Thank you. It runs as follow:

Quote:

D:\Downloads>ebook-convert "Le Point_1007.recipe" .epub -vv --test
Resolved conversion options
calibre version: 1.24.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x02629850
,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.OutputProfile object at 0x02629A
0>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': (2, 2),
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Using custom recipe
1% Fetching feeds...
Non-ASCII text detected. You must set your Console's font to Lucida Console or
onsolas or some other TrueType font to see this text -- 1% Fetching feed Actual
te...
1% Trying to download cover...
1% Generating masthead...
Synthesizing mastheadImage
1% Starting download [4 thread(s)]...
Downloading
Downloading
Fetching http://www.lepoint.fr/c-est-arrive-a...r-1358-ce-debi
e-d-etienne-marcel-empeche-les-emeutiers-d-egorger-le-futur-charles-v-21-02-201
-1630795_494.php
Fetching http://www.lepoint.fr/sport/rugby-to...pays-de-galles
france-a-suivre-en-direct-des-21-heures-sur-le-point-fr-21-02-2014-1794524_26.p
p
Removing unlikely candidate - li#reseauxsociauxHeader - ul#liens
Candid: 7.330 .g - #bandeau link density 0.551 -> 3.289
Candid: 6.165 #bandeau - header link density 0.269 -> 4.508
Candid: 7.810 div - object link density 0.000 -> 7.810
Candid: 1.405 object - object#FlashID link density 0.000 -> 1.405
Candid: 10.460 div - header link density 0.000 -> 10.460
Candid: 2.730 header - .page_article link density 0.032 -> 2.641
Candid: 71.310 .article.zoomable - .page_article link density 0.017 -> 70.094
Candid: 50.655 .page_article - article link density 0.019 -> 49.701
Candid: -17.680 #football_one_right - #football_one link density 0.000 -> -17.6
0
Candid: -18.840 #football_one - #football_general link density 0.000 -> -18.840
Candid: 397.290 #football_comment - .match_euro link density 0.000 -> 397.290
Candid: 213.645 .match_euro - .article.zoomable link density 0.000 -> 213.645
Candid: 9.250 .b - #pied1 link density 0.651 -> 3.229
Candid: 7.125 #pied1 - footer link density 0.490 -> 3.631
Candid: 7.520 .ojd - #pied2 link density 0.000 -> 7.520
Candid: 6.260 #pied2 - #pied1 link density 0.517 -> 3.024
Candid: 10.950 #footer-texte - #footer-cookies link density 0.062 -> 10.276
Candid: -17.025 #footer-cookies - #pied1 link density 0.062 -> -15.977
Top 5 : 397.290 #football_comment - .match_euro
Top 5 : 213.645 .match_euro - .article.zoomable
Top 5 : 70.094 .article.zoomable - .page_article
Top 5 : 49.701 .page_article - article
Top 5 : 10.460 div - header
Could not fetch link http://www.lepoint.fr/sport/rugby-to...s-6-nations-pa
s-de-galles-france-a-suivre-en-direct-des-21-heures-sur-le-point-fr-21-02-2014-
794524_26.php
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 530, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 204, in get_soup
File "<string>", line 19, in preprocess_html
TypeError: 'NoneType' object has no attribute '__getitem__'

http://www.lepoint.fr/sport/rugby-to...alles-france-a
suivre-en-direct-des-21-heures-sur-le-point-fr-21-02-2014-1794524_26.php saved
o
Failed to download article: Tournoi des 6 Nations : le pays de Galles assomme l
France (27-6) from http://www.lepoint.fr/sport/rugby-to...-6-nations-pay
-de-galles-france-a-suivre-en-direct-des-21-heures-sur-le-point-fr-21-02-2014-1
94524_26.php
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 1106, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 1101, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in
this log

17% Article download failed: Tournoi des 6 Nations : le pays de Galles assomme
a France (27-6)
Removing unlikely candidate - li#reseauxsociauxHeader - ul#liens
Candid: 7.330 .g - #bandeau link density 0.551 -> 3.289
Candid: 6.165 #bandeau - header link density 0.269 -> 4.508
Candid: 7.810 div - object link density 0.000 -> 7.810
Candid: 1.405 object - object#FlashID link density 0.000 -> 1.405
Candid: 36.760 #entete_article - header link density 0.000 -> 36.760
Candid: 3.380 header - article link density 0.093 -> 3.066
Candid: 138.000 span - .article.zoomable link density 0.003 -> 137.648
Candid: 99.000 .article.zoomable - article link density 0.015 -> 97.508
Candid: 14.290 .message - #div_commentaire_4439036.com link density 0.000 -> 14
290
Candid: -15.355 #div_commentaire_4439036.com - #liste_commentaires.liste_commen
aires link density 0.000 -> -15.355
Candid: 8.050 .message - #div_commentaire_3260372.com link density 0.000 -> 8
050
Candid: -18.475 #div_commentaire_3260372.com - #liste_commentaires.liste_commen
aires link density 0.000 -> -18.475
Candid: 10.050 .message - #div_commentaire_3260292.com link density 0.000 -> 10
050
Candid: -17.475 #div_commentaire_3260292.com - #liste_commentaires.liste_commen
aires link density 0.000 -> -17.475
Candid: 11.000 .message - #div_commentaire_3259779.com link density 0.000 -> 11
000
Candid: -17.000 #div_commentaire_3259779.com - #liste_commentaires.liste_commen
aires link density 0.000 -> -17.000
Candid: 43.250 .message - #div_commentaire_3258938.com link density 0.000 -> 43
250
Candid: -0.875 #div_commentaire_3258938.com - #liste_commentaires.liste_comment
ires link density 0.000 -> -0.875
Candid: 8.690 .message - #div_commentaire_3257924.com link density 0.000 -> 8
690
Candid: -18.155 #div_commentaire_3257924.com - #liste_commentaires.liste_commen
aires link density 0.000 -> -18.155
Candid: -23.200 form#frm_commentaire - #depot_commentaire.depot_commentaire lin
density 0.308 -> -16.045
Candid: -42.600 #depot_commentaire.depot_commentaire - section#bloc_commentaire
link density 0.264 -> -31.354
Candid: 0.680 li - ul#culture link density 0.919 -> 0.055
Candid: 6.540 ul#culture - .bloc.derniere_depeche link density 0.909 -> 0.597
Candid: -0.310 li - ul#culture link density 0.920 -> -0.025
Candid: -0.310 li - ul#culture link density 0.920 -> -0.025
Candid: -0.370 li - ul#culture link density 0.913 -> -0.032
Candid: -0.320 li - ul#culture link density 0.919 -> -0.026
Candid: 1.710 li - ul#culture link density 0.922 -> 0.133
Candid: 9.760 .t - #abo_newsletter.bloc link density 0.000 -> 9.760
Candid: 7.380 #abo_newsletter.bloc - #droite link density 0.000 -> 7.380
Candid: 7.840 .categorie_cam.float_left - li.n1 link density 0.764 -> 1.853
Candid: -1.580 li.n1 - ol#populaire1.liste link density 0.764 -> -0.373
Candid: 7.710 .categorie_cam.float_left - li.n2.last link density 0.710 -> 2.
36
Candid: -1.645 li.n2.last - ol#populaire1.liste link density 0.710 -> -0.477
Candid: 7.520 .categorie_cam.float_left - li.n1 link density 0.867 -> 1.003
Candid: -1.740 li.n1 - ol#populaire1.liste link density 0.867 -> -0.232
Candid: 7.300 .categorie_cam.float_left - li.n2.last link density 0.682 -> 2.
23
Candid: -1.850 li.n2.last - ol#populaire1.liste link density 0.682 -> -0.589
Candid: 16.420 form#form_855 - .sondage link density 0.214 -> 12.904
Candid: 14.710 .sondage - #droite link density 0.214 -> 11.561
Candid: 7.500 .ligne - #droite link density 0.000 -> 7.500
Candid: 13.825 #droite - .contenu link density 0.525 -> 6.562
Candid: 7.430 .ligne - #droite link density 0.000 -> 7.430
Candid: 7.510 .ligne - #droite link density 0.000 -> 7.510
Candid: 7.430 .ligne - #droite link density 0.000 -> 7.430
Candid: 7.530 .ligne - #droite link density 0.000 -> 7.530
Candid: 7.620 .ligne - #droite link density 0.000 -> 7.620
Candid: 7.630 .ligne - #droite link density 0.000 -> 7.630
Candid: 7.260 .t - #pied1 link density 0.000 -> 7.260
Candid: 8.345 #pied1 - footer link density 0.468 -> 4.442
Candid: 9.430 .b - #pied1 link density 0.625 -> 3.532
Candid: 7.330 .ligne - #lp_dossiers.g link density 0.989 -> 0.079
Candid: 6.165 #lp_dossiers.g - .b link density 0.989 -> 0.066
Candid: 7.520 .ojd - #pied2 link density 0.000 -> 7.520
Candid: 6.260 #pied2 - #pied1 link density 0.517 -> 3.024
Candid: 10.950 #footer-texte - #footer-cookies link density 0.062 -> 10.276
Candid: -17.025 #footer-cookies - #pied1 link density 0.062 -> -15.977
Top 5 : 137.648 span - .article.zoomable
Top 5 : 97.508 .article.zoomable - article
Top 5 : 43.250 .message - #div_commentaire_3258938.com
Top 5 : 36.760 #entete_article - header
Top 5 : 14.290 .message - #div_commentaire_4439036.com
Could not fetch link http://www.lepoint.fr/c-est-arrive-a...ui/22-fevrier-
358-ce-debile-d-etienne-marcel-empeche-les-emeutiers-d-egorger-le-futur-charles
v-21-02-2013-1630795_494.php
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 530, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 204, in get_soup
File "<string>", line 19, in preprocess_html
TypeError: 'NoneType' object has no attribute '__getitem__'

http://www.lepoint.fr/c-est-arrive-a...-debile-d-etie
ne-marcel-empeche-les-emeutiers-d-egorger-le-futur-charles-v-21-02-2013-1630795
494.php saved to
Failed to download article: Non-ASCII text detected. You must set your Console'
font to Lucida Console or Consolas or some other TrueType font to see this tex
-- 22 fevrier 1358. Etienne Marcel empeche les emeutiers d'egorger le futur Ch
rles V, il le paiera cher. from http://www.lepoint.fr/c-est-arrive-aujourd-hui/
2-fevrier-1358-ce-debile-d-etienne-marcel-empeche-les-emeutiers-d-egorger-le-fu
ur-charles-v-21-02-2013-1630795_494.php
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 1106, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 1101, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in
this log

Non-ASCII text detected. You must set your Console's font to Lucida Console or
onsolas or some other TrueType font to see this text -- 34% Article download fa
led: 22 fevrier 1358. Etienne Marcel empeche les emeutiers d'egorger le futur C
arles V, il le paiera cher.
34% Feeds downloaded to C:\Users\ADMINI~1\AppData\Local\Temp\calibre_rz8td k\yjb
r9_plumber\index.html
34% Download finished
Failed to download the following articles:
Tournoi des 6 Nations : le pays de Galles assomme la France (27-6) from Non-ASC
I text detected. You must set your Console's font to Lucida Console or Consolas
or some other TrueType font to see this text -- Actualite
http://www.lepoint.fr/sport/rugby-to...alles-france-a
suivre-en-direct-des-21-heures-sur-le-point-fr-21-02-2014-1794524_26.php
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 1106, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 1101, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in
this log

Non-ASCII text detected. You must set your Console's font to Lucida Console or
onsolas or some other TrueType font to see this text -- 22 fevrier 1358. Etienn
Marcel empeche les emeutiers d'egorger le futur Charles V, il le paiera cher.
rom Non-ASCII text detected. You must set your Console's font to Lucida Console
or Consolas or some other TrueType font to see this text -- Actualite
http://www.lepoint.fr/c-est-arrive-a...-debile-d-etie
ne-marcel-empeche-les-emeutiers-d-egorger-le-futur-charles-v-21-02-2013-1630795
494.php
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 1106, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 1101, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in
this log

Parsing all content...
Parsing feed_0/index.html ...
Initial parse failed, using more forgiving parsers
Parsing feed_0/index.html as HTML
Parsing index.html ...
Forcing index.html into XHTML namespace
Reading TOC from NCX...
34% Running transforms on ebook...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Found 3 items of level: div_1
Found 2 items of level: div_2
Found 2 items of level: p_2
Ignoring level p_2
div_1 left margin stats: Counter()
div_1 right margin stats: Counter()
div_2 left margin stats: Counter()
div_2 right margin stats: Counter()
Cleaning up manifest...
Trimming unused files from manifest...
Creating EPUB Output...
67% Running EPUB Output plugin
Found non-unique filenames, renaming to support broken EPUB readers like FBRead
r, Aldiko and Stanza...
{u'index.html': u'index_u1.html'}
Splitting markup on page breaks and flow limits, if any...
Looking for large trees in feed_0/index.html...
No large trees found
Looking for large trees in index_u1.html...
No large trees found
This EPUB file has no Table of Contents. Creating a default TOC
The cover image has an id != "cover". Renaming to work around bug in Nook Color
EPUB output written to D:\Downloads\Le Point_1007.epub
Output saved to D:\Downloads\Le Point_1007.epub

kovidgoyal · 02-21-2014, 09:14 PM

See line 19 of your recipe.

Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 530, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 204, in get_soup
File "<string>", line 19, in preprocess_html
TypeError: 'NoneType' object has no attribute '__getitem__'

02-21-2014, 04:59 AM	#2
Krittika Goyal Vox calibre Posts: 412 Karma: 1175230 Join Date: Jan 2009 Device: Sony reader prs700, kobo	Hi I dont know enough about recipes to fully understand your code but the following works: Code: class AdvancedUserRecipe1392888003(BasicNewsRecipe): title = u'Le Point' oldest_article = 3 max_articles_per_feed = 10 auto_cleanup = True feeds = [(u'Actualit', u'http://www.lepoint.fr/rss.xml')] Do you really need the rest?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Running recipe error: Python function terminated unexpectedly (Error Code: 1)	jpassarella	Recipes	0	10-05-2012 12:54 PM
Entry point not found error	DaleM	Sigil	8	01-28-2012 07:26 PM
PRS-650 Exclamation Point error opening epub book	TimeTraveler	Sony Reader	3	02-15-2011 10:38 PM
Error: Could not find reasonable point at which to split	MTR	Calibre	7	10-09-2009 11:03 AM
Exclamation point document error	Bob Russell	Sony Reader	6	11-15-2006 06:58 PM

02-21-2014, 09:22 AM	#3
rhuang76 Junior Member Posts: 3 Karma: 10 Join Date: Feb 2010 Device: Kindle 2	In fact, that part works. But every article is without the title and image. Because the title and image are located in the <meta> tag within the <head>. I want to modify the tags and put them in the <body>.

02-21-2014, 10:41 AM	#4
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Run your recipe using ebook-convert file.recipe .epub -vv --test and you will get a traceback showing you what the problem is.

02-21-2014, 09:14 PM	#6
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	See line 19 of your recipe. Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 530, in process_links File "site-packages\calibre\web\fetch\simple.py", line 204, in get_soup File "<string>", line 19, in preprocess_html TypeError: 'NoneType' object has no attribute '__getitem__'

Advert

Advert