Pls. help to add new newspaper

[Kevinchan]

Could anyone hlep me to add a hong kong newspaper "Hong kong economic journal" with following link into calibre recipes?

http://www.hkej.com/template/dailyne...?dnews_id=3434

Thank you very much in advance.

[NotTaken]

Not sure if the login will work, but you can try this (another based on economist by Kovid Goyal).

[NotTaken]

Changed login method to form submit.

[cupcakeg2]

Hi NotTaken,

Thanks for your recipe. I have got the following error when using this:

calibre, version 2.27.0 (win32, isfrozen: True)
Conversion Error: Failed: Fetch news from Hong Kong Economic Journal

Fetch news from Hong Kong Economic Journal
Resolved conversion options
calibre version: 2.27.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x03F65A10>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleDXOutput object at 0x03F65D70>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
unbound method get_browser() must be called with BasicNewsRecipe instance as first argument (got nothing instead) (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 193, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1042, in run
File "site-packages\calibre\customize\conversion.py", line 241, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 116, in convert
File "site-packages\calibre\web\feeds\news.py", line 918, in __init__
File "<string>", line 41, in get_browser
TypeError: unbound method get_browser() must be called with BasicNewsRecipe instance as first argument (got nothing instead)


Many thanks

[cupcakeg2]

It looks ok now. After changing Line 41 to "get_browser(self)"

But now the format needs to be tidied up because it fetches a lot of links to different articles, typical to news websites.

Originally Posted by cupcakeg2

Hi NotTaken,

Thanks for your recipe. I have got the following error when using this:

calibre, version 2.27.0 (win32, isfrozen: True)
Conversion Error: Failed: Fetch news from Hong Kong Economic Journal

Fetch news from Hong Kong Economic Journal
Resolved conversion options
calibre version: 2.27.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x03F65A10>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleDXOutput object at 0x03F65D70>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
unbound method get_browser() must be called with BasicNewsRecipe instance as first argument (got nothing instead) (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 193, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1042, in run
File "site-packages\calibre\customize\conversion.py", line 241, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 116, in convert
File "site-packages\calibre\web\feeds\news.py", line 918, in __init__
File "<string>", line 41, in get_browser
TypeError: unbound method get_browser() must be called with BasicNewsRecipe instance as first argument (got nothing instead)


Many thanks

[cupcakeg2]

I've resolved the issue of removing ads and links by using "auto_cleanup = True
".

The problem is it also removed all the images.

I tried to use various code like
auto_cleanup_keep = '//div[@href="http://static.hkej.com/hkej/images/]'
auto_cleanup_keep = '//div[@id="article-detail-wrapper"]'

But this will include lots of unwanted headers which should be filtered out by auto_cleanup.
Feel free to share your ideas. The news content in Chinese but the content doesn't matter. Thanks.

Source of the news website:
<div id="article-detail-wrapper">

<p id="date">2015年8月15日</p>
<p class="info">

</p>
<h1 id="article-title" class="sub_ex_article-title">港次季GDP增2.8%勝預期<br>財富效應帶動 私人消費大漲 <meta content="港次季GDP增2.8%勝預期<br>財富效應帶動 私人消費大漲" name="title">
</h1>
<div id='article-content'>
<p>雖然本港第二季出口、旅遊服務業仍未見起色，不過本地需求強勁，股市、樓市及人工皆升，財富效應顯著 ，刺激私人消費大漲6%，帶動本港第二季本地生產總值（GDP）錄得2.8%增長，較市場預期為佳。政府指 出，基本因素如勞工市場仍然穩健、入息增加等，料下半年本地消費可望續見增長，將2015年全年GDP增長 預測由原本1%至3%，修訂至2%至3%。</p>
<p>第二季貨物出口表現依然欠佳，按年下跌3.6%。至於服務輸出止跌回升，按年漲1%，主要是金融及商 用服務輸出大增11%，反映季內的跨境金融服務以及集資活動暢旺；訪港旅客人次減以及人均消費下滑，旅遊服 務輸出按年跌1.5%，但已較第一季的4.4%跌幅有所收窄。</p>
<p><strong>全年預測升2%至3%</strong></p>
<p>私人消費開支為季內經濟增長主要動力，按年上升6%，升幅高於第一季的5.3%增長，按照恒生銀行報 告稱，這亦是最近兩年最高的單季升幅。政府經濟顧問陳李藹倫表示，私人消費強勁增長主要是受惠就業以及收入 情況良好，除了勞工市場繼續處於全民就業狀態，第二季住戶入息中位數名義增長高達7.9%，實際增長亦有4 .7%。此外，第二季內股、樓皆升，正面財富效應提供額外的增長動力。</p>
<p>政府將原本第一季的GDP增長由2.1%，修訂至2.4%，第二季錄2.8%增長，上半年合計增長2 .6%。展望下半年，政府預期歐洲、日本經濟仍疲弱，相信出口在短期內難有起色，內部需求仍會是主要增長動 力，但估計下半年較難見到如上半年般高增長。政府並對全年GDP增長預測由原本1%至3%，修訂至2%至3 %。恒生銀行則將本港GDP預測上調0.1個百分點，至2.5%。</p>
<p><strong>零售業展望大幅改善</strong></p>
<p>按照政府在7月初調查顯示，各行業對前景展望大致正面，其中零售業對第三季的展望指數錄-2，較第二季時錄得-18大幅改善，對增聘人手的展望由第二季的-2，升至5。</p>
<p>陳李藹倫指出，零售業很大部分仍是靠本地消費支持，估計商家相信本地消費動力能維持下去，不過主要做 遊客生意的高檔次消費如珠寶鐘錶等仍然具挑戰性。她指出，近日內地改革滙率制度，有利人民幣國際化，中長期 對香港和中國的發展有利。人民幣貶值將有助出口，但對本港旅遊業有負面影響，不過，人民幣過去數日貶值約3 %至4%，幅度很小，而且相信只是一次過的短暫波動。</p>
<p>澳新銀行高級經濟師楊宇霆認為，第三季的本地零售仍有隱憂，因為除了人民幣外，亞洲其他地區包括日本 、台灣、南韓等旅遊熱點的貨幣亦在貶值，令市民在暑假期間外遊意欲大升，留港消費相應減少。該行預測本港全 年GDP增長2.1%，接近港府預測的下限。</p> </div>




<p><a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_dc05eb03e41ae14a8e10830f50134d0b.jpg" title="政府經濟顧問陳李藹倫（中）指出，人幣貶值將有利出口，但對本港旅遊業有負面影響。（黃潤根 攝）" class="thickbox"><img src="http://static.hkej.com/hkej/images/2015/08/15/1120432_dc05eb03e41ae14a8e10830f50134d0b_620.jpg" border="0"></a></p>
<span class='enlargeImg'><a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_dc05eb03e41ae14a8e10830f50134d0b.jpg" title="政府經濟顧問陳李藹倫（中）指出，人幣貶值將有利出口，但對本港旅遊業有負面影響。（黃潤根 攝）" class="thickbox"><img src='/images/2011/enlarge.gif' align='absmiddle' />放大圖片</a> / <a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_dc05eb03e41ae14a8e10830f50134d0b.jpg" target="_blank" title="政府經濟顧問陳李藹倫（中）指出，人幣貶值將有利出口，但對本港旅遊業有負面影響。（黃潤根 攝）">顯示原圖</a></span>

<p><a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_aed578c4b4500172eb7740609e3d238d.jpg" title="私人消費開支是次季經濟增長主要動力。（資料圖片）" class="thickbox"><img src="http://static.hkej.com/hkej/images/2015/08/15/1120432_aed578c4b4500172eb7740609e3d238d_620.jpg" border="0"></a></p>
<span class='enlargeImg'><a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_aed578c4b4500172eb7740609e3d238d.jpg" title="私人消費開支是次季經濟增長主要動力。（資料圖片）" class="thickbox"><img src='/images/2011/enlarge.gif' align='absmiddle' />放大圖片</a> / <a href="http://static.hkej.com/hkej/images/2015/08/15/1120432_aed578c4b4500172eb7740609e3d238d.jpg" target="_blank" title="私人消費開支是次季經濟增長主要動力。（資料圖片）">顯示原圖</a></span>