Trying to make a modified version of the recipe for "The Atlantic"

[camiller]

Ok, I'm a bit of a newb and I'm trying to make a modified version of the recipe for "The Atlantic" to fetch their special edition commemorating the American Civil War located at http://www.theatlantic.com/magazine/toc/2012/02/ and it appears to get all the way to the last article then throws the error below. The only change I made to the built-in recipe was to replace the URL in the INDEX variable to http://www.theatlantic.com/magazine/toc/2012/02/

any thoughts?

calibre, version 0.8.35
ERROR: Conversion Error: <b>Failed</b>: Fetch news from The Atlantic Special Issue - The Civil War

Fetch news from The Atlantic Special Issue - The Civil War
Resolved conversion options
calibre version: 0.8.35
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'enable_heuristics': False,
 'epub_flatten': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x058F7050>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.KoboReaderOutput object at 0x058F73D0>,
 'page_breaks_before': None,
 'password': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Found section: Part I: Pre-War
	Found article: Where Will It End? at http://www.theatlantic.com/magazine/archive/2012/02/where-will-it-end/8795/
		 In its second issue, The Atlantic urged readers to take a stand against slavery.
	Found article: Nat Turner’s Insurrection at http://www.theatlantic.com/magazine/archive/2012/02/nat-turner-8217-s-insurrection/8791/
		 An account of America's bloodiest slave revolt and its repercussions.
	Found article: A True Story, Word for Word as I Heard It at http://www.theatlantic.com/magazine/archive/2012/02/a-true-story-word-for-word-as-i-heard-it/8792/
		 In his first Atlantic contribution, the author tells the story of a mother’s surprise reunion with her son, a former slave.
	Found article: The Freedman’s Story at http://www.theatlantic.com/magazine/archive/2012/02/the-freedman-8217-s-story/8793/
		 An escaped slave recalls his violent showdown with slave-catchers.
	Found article: Paul Revere’s Ride at http://www.theatlantic.com/magazine/archive/2012/02/paul-revere-rsquo-s-ride/8823/
		 The famous Revolutionary War poem that’s really about slavery
	Found article: John Brown and His Friends at http://www.theatlantic.com/magazine/archive/2012/02/john-brown-and-his-friends/8794/
		 How a coterie of New Englanders—including the author—secretly funded the raid on Harpers Ferry
	Found article: Bardic Symbols at http://www.theatlantic.com/magazine/archive/2012/02/bardic-symbols/8824/
		 The author’s first Atlantic poem
	Found article: The Reign of King Cotton at http://www.theatlantic.com/magazine/archive/2012/02/the-reign-of-king-cotton/8797/
		 In 1861, the grandson of John Quincy Adams argued that slavery could still end without war.
	Found article: Sojourner Truth, The Libyan Sibyl at http://www.theatlantic.com/magazine/archive/2012/02/sojourner-truth-the-libyan-sibyl/8872/
		 Harriet Beecher Stowe describes her encounter with the legendary African American activist.
	Found article: Recollections of Lincoln at http://www.theatlantic.com/magazine/archive/2012/02/recollections-of-lincoln/8796/
		 A journalist who covered the Lincoln-Douglas debates recalls the future president’s bawdy appeal.
	Found article: The Election in November at http://www.theatlantic.com/magazine/archive/2012/02/the-election-in-november/8786/
		 In 1860, The Atlantic endorsed Abraham Lincoln for president.
	Found article: Charleston Under Arms at http://www.theatlantic.com/magazine/archive/2012/02/charleston-under-arms/8785/
		 A Northern journalist records his visit to Charleston during the Fort Sumter standoff.
Found section: Part II: The War
	Found article: Our March to Washington at http://www.theatlantic.com/magazine/archive/2012/02/our-march-to-washington/8788/
		 A dispatch from a Union soldier who was later killed in action
	Found article: Voluntaries at http://www.theatlantic.com/magazine/archive/2012/02/voluntaries/8825/
		 A poem in praise of soldiers who gave up their lives for the Union
	Found article: Bread and the Newspaper at http://www.theatlantic.com/magazine/archive/2012/02/bread-and-the-newspaper/8789/
		 In 1861, an Atlantic editor captured the anxious mood on the home front.
	Found article: The Advantages of Defeat at http://www.theatlantic.com/magazine/archive/2012/02/the-advantages-of-defeat/8817/
		 A scholar argues that the Union debacle at Bull Run was not such a disaster.
	Found article: Chiefly About War Matters, By a Peaceable Man at http://www.theatlantic.com/magazine/archive/2012/02/chiefly-about-war-matters-by-a-peaceable-man/8816/
		 The novelist visits Washington in wartime—and is then censored by The Atlantic.
	Found article: The Cumberland at http://www.theatlantic.com/magazine/archive/2012/02/the-cumberland/8827/
		 A poem commemorating a mighty Union ship done in by the Virginia, a rebel “ironclad”
	Found article: My Hunt After the Captain at http://www.theatlantic.com/magazine/archive/2012/02/my-hunt-after-the-captain/8787/
		 An account of the author’s frantic search for his wounded son, who lived to become a Supreme Court justice
	Found article: Barbara Fritchie at http://www.theatlantic.com/magazine/archive/2012/02/barbara-frietchie/8826/
		 The classic poem mythologizing an old woman who flew her Union flag as the rebels marched past
	Found article: The Man Without a Country at http://www.theatlantic.com/magazine/archive/2012/02/the-man-without-a-country/8790/
		 The famous short story about an Army officer who learns, too late, to love his country
	Found article: American Civilization at http://www.theatlantic.com/magazine/archive/2012/02/american-civilization/8815/
		 An Atlantic founder argues vehemently for the emancipation of the slaves.
	Found article: The President’s Proclamation at http://www.theatlantic.com/magazine/archive/2012/02/the-president-8217-s-proclamation/8814/
		 Seven months after his call to free the slaves, Emerson hails the Emancipation Proclamation.
	Found article: Women, Unite Against Slavery at http://www.theatlantic.com/magazine/archive/2012/02/women-unite-against-slavery/8819/
		 The author of Uncle Tom’s Cabin issues a call to action.
	Found article: The Story of a Year at http://www.theatlantic.com/magazine/archive/2012/02/the-story-of-a-year/8784/
		 One of the earliest pieces published by the author, who was 21 years old at the time
	Found article: The Ladies of New Orleans at http://www.theatlantic.com/magazine/archive/2012/02/the-ladies-of-new-orleans/8821/
		 A Union general is stymied by the ornery women of the South.
	Found article: Leaves From an Officer’s Journal at http://www.theatlantic.com/magazine/archive/2012/02/leaves-from-an-officer-8217-s-journal/8820/
		 The white colonel of the first official black regiment recounts his experience. 
	Found article: Life on the Sea Islands at http://www.theatlantic.com/magazine/archive/2012/02/life-on-the-sea-islands/8818/
		 A young black woman describes her experience teaching freed slaves.
	Found article: The Brothers at http://www.theatlantic.com/magazine/archive/2012/02/the-brothers/8802/
		 Set in a wartime hospital, a short story about a family with a poisonous secret
	Found article: The Words That Remade America at http://www.theatlantic.com/magazine/archive/2012/02/the-words-that-remade-america/8801/
		 The significance of the Gettysburg Address
	Found article: A Rebel’s Recollections at http://www.theatlantic.com/magazine/archive/2012/02/a-rebel-8217-s-recollections/8800/
		 A Confederate soldier from a plantation family provides a Southern perspective.
	Found article: Lee in Battle at http://www.theatlantic.com/magazine/archive/2012/02/lee-in-battle/8798/
		 A Northerner pays tribute to the general’s humility and heroism.
	Found article: Toward Appomattox at http://www.theatlantic.com/magazine/archive/2012/02/toward-appomattox/8822/
		 Reliving the war’s final battles
	Found article: Late Scenes in Richmond at http://www.theatlantic.com/magazine/archive/2012/02/late-scenes-in-richmond/8799/
		 A reporter describes the rebels’ flight from Richmond, and Lincoln’s surprise visit two days later.
Found section: Part III: Post-War
	Found article: The End, and After at http://www.theatlantic.com/magazine/archive/2012/02/the-end-and-after/8803/
		 A Confederate soldier recalls the chaotic days following surrender.
	Found article: Assassination at http://www.theatlantic.com/magazine/archive/2012/02/assassination/8804/
		 Three months after Lincoln’s murder, The Atlantic seeks to make sense of it.
	Found article: Ode to Lincoln at http://www.theatlantic.com/magazine/archive/2012/02/ode-to-lincoln/8828/
		 The magazine’s first editor gives poetic voice to the nation’s grief. 
	Found article: Three Months Among the Reconstructionists at http://www.theatlantic.com/magazine/archive/2012/02/three-months-among-the-reconstructionists/8809/
		 In 1866, a journalist offered a scathing report on post-war life in the South.
	Found article: The Mistress of Sydenham Plantation at http://www.theatlantic.com/magazine/archive/2012/02/the-mistress-of-sydenham-plantation/8808/
		 The famous novelist’s tale of an elderly Southerner, oblivious to what the war had cost her.
	Found article: The Case of George Dedlow at http://www.theatlantic.com/magazine/archive/2012/02/the-case-of-george-dedlow/8807/
		 An absurdist short story about a Union doctor—which many Atlantic readers erroneously believed at the time to be nonfiction.
	Found article: For the Union Dead at http://www.theatlantic.com/magazine/archive/2012/02/for-the-union-dead/8829/
		 The classic 1960 poem pays tribute to the glory of the Civil War era.
	Found article: The Freedmen’s Bureau at http://www.theatlantic.com/magazine/archive/2012/02/the-freedmen-8217-s-bureau/8805/
		 A leading black intellectual surveys the government’s efforts to aid the freed slaves.
	Found article: Reconstruction, and an Appeal to Impartial Suffrage at http://www.theatlantic.com/magazine/archive/2012/02/reconstruction-and-an-appeal-to-impartial-suffrage/8806/
		 A former slave urges Congress to grant black Americans the vote.
	Found article: The Death of Slavery at http://www.theatlantic.com/magazine/archive/2012/02/the-death-of-slavery/8871/
		 A poem hailing the demise of slavery’s “cruel reign”
	Found article: The Result in South Carolina at http://www.theatlantic.com/magazine/archive/2012/02/the-result-in-south-carolina/8812/
		 A Southerner describes mounting racial tensions in the aftermath of Reconstruction.
	Found article: The Awakening of the Negro at http://www.theatlantic.com/magazine/archive/2012/02/the-awakening-of-the-negro/8813/
		 An educator’s controversial argument contends that blacks should advance by making themselves useful to whites.
	Found article: Of the Training of Black Men at http://www.theatlantic.com/magazine/archive/2012/02/of-the-training-of-black-men/8811/
		 Taking issue with Booker T. Washington, the author argues that blacks should attend college.
	Found article: Strivings of the Negro People at http://www.theatlantic.com/magazine/archive/2012/02/strivings-of-the-negro-people/8810/
		 Du Bois gives voice to the aspirations of black Americans in the post-Civil War world.
Found section: Poems
Python function terminated unexpectedly
  'NoneType' object has no attribute 'parent' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 191, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 959, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 105, in convert
  File "site-packages\calibre\web\feeds\news.py", line 856, in download
  File "site-packages\calibre\web\feeds\news.py", line 1000, in build_index
  File "c:\users\chris~1.mil\appdata\local\temp\calibre_0.8.35_tmp_9di6zz\v5wmkb_recipes\recipe0.py", line 77, in parse_index
    pd = soup.find('h2', text='Poetry').parent.parent
AttributeError: 'NoneType' object has no attribute 'parent'

for what it is worth, there does not appear to actually be a "Poems" section on that page like there is in their normal monthly issues. I suspect some non-visible

[camiller]

OK, python is not my normal programming language but a figured out enough to be dangerous. I found that the existing code was specifically looking for a Poems section, then tried to do something with it without first making sure it was there. I just deleted that part of the code and now it works.

[camiller]

Oh, the recipe if anyone is interested:

I only changed the "INDEX =" and removed the special code for the "Poems" section.

#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''
theatlantic.com
'''
import re

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import Tag, NavigableString

class TheAtlantic(BasicNewsRecipe):

    title      = 'The Atlantic Special Issue - The Civil War'
    __author__ = 'Kovid Goyal and Sujata Raman modified by Chris Miller'
    description = 'Current affairs and politics focussed on the US'
    INDEX = 'http://www.theatlantic.com/magazine/toc/2012/02/'
    language = 'en'

    remove_tags_before = dict(name='div', id='articleHead')
    remove_tags_after  = dict(id='copyright')
    remove_tags        = [dict(id=['header', 'printAds', 'pageControls'])]
    no_stylesheets = True

    preprocess_regexps = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]


    def print_version(self, url):
        return url.replace('/archive/', '/print/')

    def parse_index(self):
        articles = []

        soup = self.index_to_soup(self.INDEX)
        ts = soup.find(id='magazineTopStories')
        ds = self.tag_to_string(ts.find('h1')).split(':')[-1]
        self.timefmt = ' [%s]'%ds

        cover = soup.find('img', src=True, attrs={'class':'cover'})
        if cover is not None:
            self.cover_url = cover['src']

        feeds = []
        seen_titles = set([])
        for section in soup.findAll('div', attrs={'class':'magazineSection'}):
            section_title = self.tag_to_string(section.find('h2'))
            self.log('Found section:', section_title)
            articles = []
            for post in section.findAll('div', attrs={'class':lambda x : x and
                'post' in x}):
                h = post.find(['h3', 'h4'])
                title = self.tag_to_string(h)
                if title in seen_titles:
                    continue
                seen_titles.add(title)
                a = post.find('a', href=True)
                url = a['href']
                if url.startswith('/'):
                    url = 'http://www.theatlantic.com'+url
                p = post.find('p', attrs={'class':'dek'})
                desc = None
                self.log('\tFound article:', title, 'at', url)
                if p is not None:
                    desc = self.tag_to_string(p)
                    self.log('\t\t', desc)
                articles.append({'title':title, 'url':url, 'description':desc,
                    'date':''})
            if articles:
                feeds.append((section_title, articles))

        return feeds

    def postprocess_html(self, soup, first):
        for table in soup.findAll('table', align='right'):
            img = table.find('img')
            if img is not None:
                img.extract()
                caption = self.tag_to_string(table).strip()
                div = Tag(soup, 'div')
                div['style'] = 'text-align:center'
                div.insert(0, img)
                div.insert(1, Tag(soup, 'br'))
                if caption:
                    div.insert(2, NavigableString(caption))
                table.replaceWith(div)

        return soup

[satisficer]

Output looks great -- thank you!