Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 02-13-2012, 06:15 PM   #1
camiller
Addict
camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.
 
Posts: 280
Karma: 1363918
Join Date: Aug 2011
Device: Kobo Wireless
Trying to make a modified version of the recipe for "The Atlantic"

Ok, I'm a bit of a newb and I'm trying to make a modified version of the recipe for "The Atlantic" to fetch their special edition commemorating the American Civil War located at http://www.theatlantic.com/magazine/toc/2012/02/ and it appears to get all the way to the last article then throws the error below. The only change I made to the built-in recipe was to replace the URL in the INDEX variable to http://www.theatlantic.com/magazine/toc/2012/02/

any thoughts?

Code:
calibre, version 0.8.35
ERROR: Conversion Error: <b>Failed</b>: Fetch news from The Atlantic Special Issue - The Civil War

Fetch news from The Atlantic Special Issue - The Civil War
Resolved conversion options
calibre version: 0.8.35
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'enable_heuristics': False,
 'epub_flatten': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x058F7050>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.KoboReaderOutput object at 0x058F73D0>,
 'page_breaks_before': None,
 'password': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Found section: Part I: Pre-War
	Found article: Where Will It End? at http://www.theatlantic.com/magazine/archive/2012/02/where-will-it-end/8795/
		 In its second issue, The Atlantic urged readers to take a stand against slavery.
	Found article: Nat Turner’s Insurrection at http://www.theatlantic.com/magazine/archive/2012/02/nat-turner-8217-s-insurrection/8791/
		 An account of America's bloodiest slave revolt and its repercussions.
	Found article: A True Story, Word for Word as I Heard It at http://www.theatlantic.com/magazine/archive/2012/02/a-true-story-word-for-word-as-i-heard-it/8792/
		 In his first Atlantic contribution, the author tells the story of a mother’s surprise reunion with her son, a former slave.
	Found article: The Freedman’s Story at http://www.theatlantic.com/magazine/archive/2012/02/the-freedman-8217-s-story/8793/
		 An escaped slave recalls his violent showdown with slave-catchers.
	Found article: Paul Revere’s Ride at http://www.theatlantic.com/magazine/archive/2012/02/paul-revere-rsquo-s-ride/8823/
		 The famous Revolutionary War poem that’s really about slavery
	Found article: John Brown and His Friends at http://www.theatlantic.com/magazine/archive/2012/02/john-brown-and-his-friends/8794/
		 How a coterie of New Englanders—including the author—secretly funded the raid on Harpers Ferry
	Found article: Bardic Symbols at http://www.theatlantic.com/magazine/archive/2012/02/bardic-symbols/8824/
		 The author’s first Atlantic poem
	Found article: The Reign of King Cotton at http://www.theatlantic.com/magazine/archive/2012/02/the-reign-of-king-cotton/8797/
		 In 1861, the grandson of John Quincy Adams argued that slavery could still end without war.
	Found article: Sojourner Truth, The Libyan Sibyl at http://www.theatlantic.com/magazine/archive/2012/02/sojourner-truth-the-libyan-sibyl/8872/
		 Harriet Beecher Stowe describes her encounter with the legendary African American activist.
	Found article: Recollections of Lincoln at http://www.theatlantic.com/magazine/archive/2012/02/recollections-of-lincoln/8796/
		 A journalist who covered the Lincoln-Douglas debates recalls the future president’s bawdy appeal.
	Found article: The Election in November at http://www.theatlantic.com/magazine/archive/2012/02/the-election-in-november/8786/
		 In 1860, The Atlantic endorsed Abraham Lincoln for president.
	Found article: Charleston Under Arms at http://www.theatlantic.com/magazine/archive/2012/02/charleston-under-arms/8785/
		 A Northern journalist records his visit to Charleston during the Fort Sumter standoff.
Found section: Part II: The War
	Found article: Our March to Washington at http://www.theatlantic.com/magazine/archive/2012/02/our-march-to-washington/8788/
		 A dispatch from a Union soldier who was later killed in action
	Found article: Voluntaries at http://www.theatlantic.com/magazine/archive/2012/02/voluntaries/8825/
		 A poem in praise of soldiers who gave up their lives for the Union
	Found article: Bread and the Newspaper at http://www.theatlantic.com/magazine/archive/2012/02/bread-and-the-newspaper/8789/
		 In 1861, an Atlantic editor captured the anxious mood on the home front.
	Found article: The Advantages of Defeat at http://www.theatlantic.com/magazine/archive/2012/02/the-advantages-of-defeat/8817/
		 A scholar argues that the Union debacle at Bull Run was not such a disaster.
	Found article: Chiefly About War Matters, By a Peaceable Man at http://www.theatlantic.com/magazine/archive/2012/02/chiefly-about-war-matters-by-a-peaceable-man/8816/
		 The novelist visits Washington in wartime—and is then censored by The Atlantic.
	Found article: The Cumberland at http://www.theatlantic.com/magazine/archive/2012/02/the-cumberland/8827/
		 A poem commemorating a mighty Union ship done in by the Virginia, a rebel “ironclad”
	Found article: My Hunt After the Captain at http://www.theatlantic.com/magazine/archive/2012/02/my-hunt-after-the-captain/8787/
		 An account of the author’s frantic search for his wounded son, who lived to become a Supreme Court justice
	Found article: Barbara Fritchie at http://www.theatlantic.com/magazine/archive/2012/02/barbara-frietchie/8826/
		 The classic poem mythologizing an old woman who flew her Union flag as the rebels marched past
	Found article: The Man Without a Country at http://www.theatlantic.com/magazine/archive/2012/02/the-man-without-a-country/8790/
		 The famous short story about an Army officer who learns, too late, to love his country
	Found article: American Civilization at http://www.theatlantic.com/magazine/archive/2012/02/american-civilization/8815/
		 An Atlantic founder argues vehemently for the emancipation of the slaves.
	Found article: The President’s Proclamation at http://www.theatlantic.com/magazine/archive/2012/02/the-president-8217-s-proclamation/8814/
		 Seven months after his call to free the slaves, Emerson hails the Emancipation Proclamation.
	Found article: Women, Unite Against Slavery at http://www.theatlantic.com/magazine/archive/2012/02/women-unite-against-slavery/8819/
		 The author of Uncle Tom’s Cabin issues a call to action.
	Found article: The Story of a Year at http://www.theatlantic.com/magazine/archive/2012/02/the-story-of-a-year/8784/
		 One of the earliest pieces published by the author, who was 21 years old at the time
	Found article: The Ladies of New Orleans at http://www.theatlantic.com/magazine/archive/2012/02/the-ladies-of-new-orleans/8821/
		 A Union general is stymied by the ornery women of the South.
	Found article: Leaves From an Officer’s Journal at http://www.theatlantic.com/magazine/archive/2012/02/leaves-from-an-officer-8217-s-journal/8820/
		 The white colonel of the first official black regiment recounts his experience. 
	Found article: Life on the Sea Islands at http://www.theatlantic.com/magazine/archive/2012/02/life-on-the-sea-islands/8818/
		 A young black woman describes her experience teaching freed slaves.
	Found article: The Brothers at http://www.theatlantic.com/magazine/archive/2012/02/the-brothers/8802/
		 Set in a wartime hospital, a short story about a family with a poisonous secret
	Found article: The Words That Remade America at http://www.theatlantic.com/magazine/archive/2012/02/the-words-that-remade-america/8801/
		 The significance of the Gettysburg Address
	Found article: A Rebel’s Recollections at http://www.theatlantic.com/magazine/archive/2012/02/a-rebel-8217-s-recollections/8800/
		 A Confederate soldier from a plantation family provides a Southern perspective.
	Found article: Lee in Battle at http://www.theatlantic.com/magazine/archive/2012/02/lee-in-battle/8798/
		 A Northerner pays tribute to the general’s humility and heroism.
	Found article: Toward Appomattox at http://www.theatlantic.com/magazine/archive/2012/02/toward-appomattox/8822/
		 Reliving the war’s final battles
	Found article: Late Scenes in Richmond at http://www.theatlantic.com/magazine/archive/2012/02/late-scenes-in-richmond/8799/
		 A reporter describes the rebels’ flight from Richmond, and Lincoln’s surprise visit two days later.
Found section: Part III: Post-War
	Found article: The End, and After at http://www.theatlantic.com/magazine/archive/2012/02/the-end-and-after/8803/
		 A Confederate soldier recalls the chaotic days following surrender.
	Found article: Assassination at http://www.theatlantic.com/magazine/archive/2012/02/assassination/8804/
		 Three months after Lincoln’s murder, The Atlantic seeks to make sense of it.
	Found article: Ode to Lincoln at http://www.theatlantic.com/magazine/archive/2012/02/ode-to-lincoln/8828/
		 The magazine’s first editor gives poetic voice to the nation’s grief. 
	Found article: Three Months Among the Reconstructionists at http://www.theatlantic.com/magazine/archive/2012/02/three-months-among-the-reconstructionists/8809/
		 In 1866, a journalist offered a scathing report on post-war life in the South.
	Found article: The Mistress of Sydenham Plantation at http://www.theatlantic.com/magazine/archive/2012/02/the-mistress-of-sydenham-plantation/8808/
		 The famous novelist’s tale of an elderly Southerner, oblivious to what the war had cost her.
	Found article: The Case of George Dedlow at http://www.theatlantic.com/magazine/archive/2012/02/the-case-of-george-dedlow/8807/
		 An absurdist short story about a Union doctor—which many Atlantic readers erroneously believed at the time to be nonfiction.
	Found article: For the Union Dead at http://www.theatlantic.com/magazine/archive/2012/02/for-the-union-dead/8829/
		 The classic 1960 poem pays tribute to the glory of the Civil War era.
	Found article: The Freedmen’s Bureau at http://www.theatlantic.com/magazine/archive/2012/02/the-freedmen-8217-s-bureau/8805/
		 A leading black intellectual surveys the government’s efforts to aid the freed slaves.
	Found article: Reconstruction, and an Appeal to Impartial Suffrage at http://www.theatlantic.com/magazine/archive/2012/02/reconstruction-and-an-appeal-to-impartial-suffrage/8806/
		 A former slave urges Congress to grant black Americans the vote.
	Found article: The Death of Slavery at http://www.theatlantic.com/magazine/archive/2012/02/the-death-of-slavery/8871/
		 A poem hailing the demise of slavery’s “cruel reign”
	Found article: The Result in South Carolina at http://www.theatlantic.com/magazine/archive/2012/02/the-result-in-south-carolina/8812/
		 A Southerner describes mounting racial tensions in the aftermath of Reconstruction.
	Found article: The Awakening of the Negro at http://www.theatlantic.com/magazine/archive/2012/02/the-awakening-of-the-negro/8813/
		 An educator’s controversial argument contends that blacks should advance by making themselves useful to whites.
	Found article: Of the Training of Black Men at http://www.theatlantic.com/magazine/archive/2012/02/of-the-training-of-black-men/8811/
		 Taking issue with Booker T. Washington, the author argues that blacks should attend college.
	Found article: Strivings of the Negro People at http://www.theatlantic.com/magazine/archive/2012/02/strivings-of-the-negro-people/8810/
		 Du Bois gives voice to the aspirations of black Americans in the post-Civil War world.
Found section: Poems
Python function terminated unexpectedly
  'NoneType' object has no attribute 'parent' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 191, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 959, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 105, in convert
  File "site-packages\calibre\web\feeds\news.py", line 856, in download
  File "site-packages\calibre\web\feeds\news.py", line 1000, in build_index
  File "c:\users\chris~1.mil\appdata\local\temp\calibre_0.8.35_tmp_9di6zz\v5wmkb_recipes\recipe0.py", line 77, in parse_index
    pd = soup.find('h2', text='Poetry').parent.parent
AttributeError: 'NoneType' object has no attribute 'parent'
for what it is worth, there does not appear to actually be a "Poems" section on that page like there is in their normal monthly issues. I suspect some non-visible
camiller is offline   Reply With Quote
Old 02-13-2012, 06:19 PM   #2
camiller
Addict
camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.
 
Posts: 280
Karma: 1363918
Join Date: Aug 2011
Device: Kobo Wireless
OK, python is not my normal programming language but a figured out enough to be dangerous. I found that the existing code was specifically looking for a Poems section, then tried to do something with it without first making sure it was there. I just deleted that part of the code and now it works.
camiller is offline   Reply With Quote
 
Enthusiast
Old 02-13-2012, 06:21 PM   #3
camiller
Addict
camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.camiller ought to be getting tired of karma fortunes by now.
 
Posts: 280
Karma: 1363918
Join Date: Aug 2011
Device: Kobo Wireless
Oh, the recipe if anyone is interested:

I only changed the "INDEX =" and removed the special code for the "Poems" section.

Code:
#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''
theatlantic.com
'''
import re

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import Tag, NavigableString

class TheAtlantic(BasicNewsRecipe):

    title      = 'The Atlantic Special Issue - The Civil War'
    __author__ = 'Kovid Goyal and Sujata Raman modified by Chris Miller'
    description = 'Current affairs and politics focussed on the US'
    INDEX = 'http://www.theatlantic.com/magazine/toc/2012/02/'
    language = 'en'

    remove_tags_before = dict(name='div', id='articleHead')
    remove_tags_after  = dict(id='copyright')
    remove_tags        = [dict(id=['header', 'printAds', 'pageControls'])]
    no_stylesheets = True

    preprocess_regexps = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]


    def print_version(self, url):
        return url.replace('/archive/', '/print/')

    def parse_index(self):
        articles = []

        soup = self.index_to_soup(self.INDEX)
        ts = soup.find(id='magazineTopStories')
        ds = self.tag_to_string(ts.find('h1')).split(':')[-1]
        self.timefmt = ' [%s]'%ds

        cover = soup.find('img', src=True, attrs={'class':'cover'})
        if cover is not None:
            self.cover_url = cover['src']

        feeds = []
        seen_titles = set([])
        for section in soup.findAll('div', attrs={'class':'magazineSection'}):
            section_title = self.tag_to_string(section.find('h2'))
            self.log('Found section:', section_title)
            articles = []
            for post in section.findAll('div', attrs={'class':lambda x : x and
                'post' in x}):
                h = post.find(['h3', 'h4'])
                title = self.tag_to_string(h)
                if title in seen_titles:
                    continue
                seen_titles.add(title)
                a = post.find('a', href=True)
                url = a['href']
                if url.startswith('/'):
                    url = 'http://www.theatlantic.com'+url
                p = post.find('p', attrs={'class':'dek'})
                desc = None
                self.log('\tFound article:', title, 'at', url)
                if p is not None:
                    desc = self.tag_to_string(p)
                    self.log('\t\t', desc)
                articles.append({'title':title, 'url':url, 'description':desc,
                    'date':''})
            if articles:
                feeds.append((section_title, articles))

        return feeds

    def postprocess_html(self, soup, first):
        for table in soup.findAll('table', align='right'):
            img = table.find('img')
            if img is not None:
                img.extract()
                caption = self.tag_to_string(table).strip()
                div = Tag(soup, 'div')
                div['style'] = 'text-align:center'
                div.insert(0, img)
                div.insert(1, Tag(soup, 'br'))
                if caption:
                    div.insert(2, NavigableString(caption))
                table.replaceWith(div)

        return soup
camiller is offline   Reply With Quote
Old 02-14-2012, 03:59 PM   #4
satisficer
Junior Member
satisficer began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Jul 2011
Device: Kindle3
Output looks great -- thank you!
satisficer is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New version for "Expansion" recipe (now work correctly) [Spanish] desUBIKado Recipes 0 11-19-2011 03:11 PM
New VErsion for "IL GIORNALE" recipe (now work correctly) gambarini Recipes 0 11-09-2011 12:05 PM
Is "language description" modified when uploading PoP Library Management 4 04-30-2011 06:10 AM
E-book Viewer Updates "Date Modified" on File. Why? spedinfargo Calibre 3 02-28-2011 05:24 PM
Help! the "Make Sony Reader File" under "Options" is different Dr. Drib Sony Reader 6 04-23-2007 02:56 AM


All times are GMT -4. The time now is 12:45 PM.


MobileRead.com is a privately owned, operated and funded community.