TypeError: 'NoneType' object is not subscriptable

[unkn0wn]

File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in _run_code
File "site.py", line 82, in <module>
File "site.py", line 77, in main
File "site.py", line 49, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 137, in convert
File "calibre\web\feeds\news.py", line 1056, in download
File "calibre\web\feeds\news.py", line 1225, in build_index
File "<string>", line 19, in parse_index
TypeError: 'NoneType' object is not subscriptable

the website i'm trying to parse prevents mouse right-click, f12 developer tools and ctrl+s can't save. (if i click f12 before website loads i'm able to use developer tools)

anyhow is this the complexity that's giving this error or is it just my simple recipe (i dont think anything is wrong in recipe).. how do i overcome this?

and the website is old too.

[kovidgoyal]

that error means you are usin ['something'] on a object that is none. Which likely means that some find() call is not returning a tag.

[unkn0wn]

What does this mean

Traceback (most recent call last):
File "calibre\web\feeds\news.py", line 1400, in download_masthead
File "calibre\web\feeds\news.py", line 1390, in _download_masthead
File "mechanize\_mechanize.py", line 257, in open
File "mechanize\_mechanize.py", line 273, in _mech_open
mechanize._mechanize.BrowserStateError: can't fetch relative reference: not viewing any document

the website i'm trying to parse prevents mouse right-click, f12 developer tools and ctrl+s can't save. (if i click f12 before website loads i'm able to use developer tools)

does above have anything to do with this..

[kovidgoyal]

That error means you are trying to opena relative url not an absolute one.

[unkn0wn]

TypeError: expected string or bytes-like object

what does this mean.. when using json

[kovidgoyal]

it means you passed it something that is neither a string nor a sequence of bytes, the two types of data the json module can de-serialize

[unkn0wn]

how to i get data from this json with pre_process_raw_html

might be a big question but give me an outline of sorts

cant upload json here

	{"@context":"https:\/\/schema.org","@type":"NewsArticle","mainEntityOfPage":"https:\/\/www.outlookindia.com\/magazine\/national\/cadres-of-political-parties-the-unsung-heroes-of-democracy-magazine-191473","headline":"Cadres Of Political Parties: The Unsung Heroes Of Democracy","inLanguage":"en","datePublished":"2022-04-15T11:41:52+05:30","dateModified":"2022-04-22T13:13:35+05:30","keywords":["Democracy","Political Parties","Politics","BSP","Dalits","BJP","CPI(M)","DMK","Trinamool Congress (TMC)","Congress","AAP: Aam Aadmi Party"],"articleBody":"Democracy is unthinkable without political parties. And without a committed base of volunteers, workers and cadres, it is hard for any party to survive in the competitive arena of electoral politics. While the burgeoning rese*a*rch on the transactional relationship between politicians and voters has improved our understanding about the role of political workers as middlemen, the research on their role as activist of political parties is conspicuous by its absence.\r\n\r\nEven the knowledge of basic things such as the demographic profile of workers of a par*ty largely relies on proxy measures such as its voting base. We assume that a BJP worker is more likely to be upper caste and urban, and a BSP activist would be a Dalit. Whereas the social base of political parties keeps shrinking and expanding, as we know from the experience of the BJP and the BSP over the past decade.\r\n\r\nALSO READ:  Decoding The Journey Of Rashtriya Swayamsevak Sangh\r\n\r\nSimilarly, we have rarely inquired into the levels of ideological attachment that political workers display. Do they see themselves as ideological messengers of a party, or merely as loyal soldiers of an ambitious politician? Do political intermed*iaries display partisan loyalties, or do they change political preferences at the drop of a hat? We also have very little knowledge of what proportion of such workers see politics as a full-time vocation and main source of earning their living. What do part-time political workers expect in return from their engagement in this field?\r\n\r\nALSO READ: Social Awakening, Individual Character Building: What RSS Expects From Its Cadres\r\n\r\nAs party activists, politician’s agent, ticket aspi*rant, broker, fixer, etc all rolled into one, political workers are the most critical element of an understudied aspect of Indian politics. And this gaping hole is having serious consequences in interpreting the changing nature of Indian democracy.\r\n\r\nAmong other things, They put up posters, collect money, mobilise supporters and convince them to turn out in large numbers at the polling booth. \r\n\r\nEach year, India witnesses the emergence of dozens of new parties, but few manage to enter the Lok Sabha or Vidhan Sabha, and fewer manage to survive beyond two election cycles. Elec*t*i*ons are also becoming increasingly costly, and yet the number of contesting candidates is also growingly rapidly, despite the fact that a vast majority of candidates lose their deposits. \r\n\r\nALSO READ: Unfailing Commitment Of CPI(M) Red Brigades\r\n\r\nIt is not uncommon for such contradictory tend*e*ncies to co-exist. The rapid proliferation of political parties and candidates has given rise to a sch*olarly consensus in defining the nature and character of Indian politics. Yet, empirical realit*ies continue to defy the foundation of this consensus. The picture is much more complex than what the simple depictions and provocative generalisations tend to assume. To be clear, this assertion is not to suggest that the prevailing wisdom is mythical and has no connection with real world.  \r\n\r\nFor example, the scholarly consensus that Ind*ian political parties have very thin penetration on the ground is rooted in the fact that institutional rules governing organisational life of India’s part*ies remain weak. Most of them, except ideologica*lly visible parties such as the BJP, lack physical inf*rastructure such as party offices below distr*ict-level, paid staff, formal relationship with civil society-based affiliates (e.g. trade unions), training mod*ules for cadres, among other things. Only some state-level parties like the CPI(M) in Kerala, DMK in Tamil Nadu and the BSP in Uttar Pradesh used to display similar traits.\r\n\r\nALSO READ: Death Of The Political Cadre: What Impacted Congress Party's Student And Youth Wings\r\n\r\nAmid this backdrop of organisati*o*nal weakness, it is surprising that political parties in India tend to perf*orm some core functions very well. First, they manage to mobilise a huge cache of human resources (workers and volunteers) and financial resources during elections. The spectacle of campaigning in India (though now much muted by the Election Commission’s restrictions) has perhaps no parallel on this planet. And very few would dou*bt the competitive nature of Indian elections. Across vast stretches of India’s political landscape, there are only a handful of constituencies that any party can boast as a long-term stronghold.\r\n\r\nOf course, we all have heard about BJP’s much-*publicised panna pramukhs (activist responsible for one page of the electoral roll) in recent times, but our knowledge of workers affiliated to other parties remains limited to them acting primarily as vote mobilisers. They put posters, mobilise supporters for rallies and procession during the campaign, convince their supporters to turn out in large numbers at the polling booth, collect donat*i*ons for their parties and local candidates, among other things. The data presented in Figure 1 bears testimony to the fact that during campaigns, these workers-as-canvassers manage to reach out to a large number of households.\r\n\r\n\r\nFigure 1: Increased Political Canvassing During Lok Sabha Elections  Centre for Policy Research\r\n\r\n\r\nALSO READ: Shiv Sena In New Avtar: How Far Will The Tamed Beast Go?\r\n\r\nA majority of these workers often engage in such mobilisational activities with the aspiration that one day they will rise up the ranks. The party will reward them for their hard work and may even nominate them as candidates. Lakhs if not milli*ons carry this hope every election cycle, and yet a majority of them continue volunteering at the same position where they started.\r\n\r\nSecond, India’s political parties often get depic*ted as machines that are assembled months bef*ore an election, before getting disassembled. If this is true, then what about the consensus that India is a patronage democracy, in which the main role of cadres is to act as intermediaries between citizens and the State? Or, do we think everyone in the system—voters, political workers, politicians, parties—are just freewheelers who keep shifting loyalties. If this is true, then what explains the stable patterns of electoral competition and resilience of existing power structures?  \r\n\r\nALSO READ: RJD: Time For Reappraisal Of Party Cadres\r\n\r\nIt is true that the capacity of the Indian State to meet the demand of its citizens effectively has con*tinued to remain low; especially in a scenario when a large segment of the population is critica*lly dependent on State services for their well-being. While distributive politics often occurs thr*ough partisan channels and ethnic networks, citizens turn to their local party workers to assist them in navigating government offices, meeting with administrative officials, or simply getting work done. In the process, these activists sometime engage in charging what some may describe as a ‘convenience fee’, while others call it ‘rent-extraction’. Notwithstanding the nomenclature, very few political workers (or brokers) earn eno*ugh to live a comfortable life.\r\n\r\nALSO READ: The Slow And Steady Decline Of BSP\r\n\r\nWhat motivates these political workers to continue volunteering with such little rewards? Dur*ing the course of our research for our book Ide*o*l*ogy and Identity: The Changing Party Systems of India that extensively uses National Election Stu*dy (NES) surveys conducted by Lokniti-CSDS, Pradeep Chhibber and I made some limited obse*r*vations regarding party members. Since 1971, NES surveys have asked respondents not only whi*ch party they voted for, but also whether they identify or feel close to any party, and whether they are registered members of any party. The data indicates that since 1999, the proportion of those who identify with a political party and are a member of one, has been consistently above 30 per cent and 10 per cent respectively. While in com*parative terms, the level of party identificat*ion in India is much lower than in Western demo*cracies, but if one looks at party membership—the most advanced form of political identification—the metric compares favourably with the West.\r\n\r\nALSO READ: AAP: The Anti-corruption Zealots On The March\r\n\r\nSimilarly, consistent with the literature on ideological contestation in the arena of party politics in Western democracies, the party identifiers as well as the party members in India display greater commitment to the party’s ideological worldview than the voters. And contrary to the claims of voters and political elites of changing political preferences, we find robust patterns of ideological com*petition in Indian politics. This is neither to downplay the outsized role charismatic leaders play in driving voters, or the role of money and patronage during elections. In our framework, leaders are heuristic or communicating devices who covey complex ideological and policy platforms to voters in simple language. The party members act as messengers in this framework. And the patronage effects remain marginal in structuring the ele*ction narrative. After all, the dismal rates at which incumbents get re-nominated and re-elected sho*uld serve a reminder to exaggerated claims of pat*ronage politics driving electoral outcomes.\r\n\r\nALSO READ: AAP At The Grassroots And Its Big Strides In Gujarat\r\n\r\nFinally, we noted that the social composit*ion of those who identify with a party and report being a member, has undergone a major transformation. While the bottom half of the soc*ial pyramid—low*er castes, the poor, women, relig*ious min*orities, less educated, etc—*con*ti**nue to rem*ain proporti*o*nally under-represented, their share has risen significantly since the 1970s. In that sense, these two metrics reflects democratisation. In addition, the partisan composition of these two met*rics has closely foll*owed the trajectory of cha*nging party systems in India. The data in Figure 2 on party identification reflects the alte*red posit*ion of the Congress and the BJP in the political setup.\r\n\r\n\r\nFigure 2: Increasing Political Identification with the BJP  Centre for Policy Research\r\n\r\n\r\nALSO READ: Cadres Keep Kashmir's National Conference Relevant\r\n\r\nIn conclusion, these political workers in many ways remain the unsung heroes of any political setup, carrying the burden of democracy on their shoulders. Their engagement in everyday political life carries both; the promises as well as the pitfalls of our democratic project. Needless to say, we must focus our gaze on political parties and their volunteers to understand the changing nat*ure of Indian democracy.\r\n\r\n***\r\n\r\n(This appeared in the print edition as "Unsung Heroes of Democracy")\r\n\r\n(Views expressed are personal)\r\n\r\n\r\nRahul Verma is a Fellow at the Centre for Policy Research, New Delhi\r\n","description":"As activist, agent, ticket aspirant, broker, fixer, etc., all rolled into one, the political worker is the least studied element of Indian politics \r\n","image":{"@type":"ImageObject","url":"https:\/\/imgnew.outlookindia.com\/uploadimage\/library\/16_9\/16_9_5\/IMAGE_1649866541.webp","height":"675","width":"1200"},"author":[{"@type":"Person","name":"Rahul Verma","url":"https:\/\/www.outlookindia.com\/author\/rahul-verma-3540"}],"isPartOf":{"@type":["CreativeWork","Product"],"name":"Cadres Of Political Parties: The Unsung Heroes Of Democracy","productID":"outlookindia.com:basic","description":"As activist, agent, ticket aspirant, broker, fixer, etc., all rolled into one, the political worker is the least studied element of Indian politics \r\n","sku":"https:\/\/subscriptions.outlookindia.com\/"},"publisher":{"id":"https:\/\/www.outlookindia.com\/","@type":"NewsMediaOrganization","name":"Outlook India","url":"https:\/\/www.outlookindia.com\/","logo":{"@type":"ImageObject","url":"https:\/\/www.outlookindia.com\/images\/home_new_v4\/logo_outlook.svg","height":"60","width":"600"}},"isAccessibleForFree":"false"}

so I copy this.. create json.. open it in firefox or chrome.. i see a structure.. i want headline datepublished and articleBody to be in html

do i use raw json and redefine the t \r\n\r as <div> and \n as <p> and such..

def preprocess_raw_html(self, raw, url):
soup = self.index_to_soup(raw)
script = soup.find('script', type="application/ld+json")
data = json.loads(raw(script.contents[0]))

[kovidgoyal]

You basically have to parse the json and convert it to html. There is no set procedure for that since it depends on how th ejson is structured. You can find examples of doing it in various builtin recipes, for instance The Economist

[unkn0wn]

Can we set different output profiles & output formats for scheduled news sources?

Like can we add certain instructions in code to override default settings in the application!

I want some to be in epub while others to be in mobi (which can be auto-mailed to device) and some to be in kfx format (which can be used with more fonts & with info-graphic images that can be zoomed in on.. like economist)

[kovidgoyal]

No, you cant control output formats from recipes.

[unkn0wn]

news sites want Google to index their content so it shows up in search results. So they don't show a paywall to the Google crawler. We benefit from this because the Google crawler will cache a copy of the site every time it crawls it.

All we do is show you that cached, unpaywalled version of the page.

can we replicate this.. use googlebot.. access this cache

like they do here and here

Would open up lots of websites without the need to parse json and stuff.

[kovidgoyal]

No, googlebot always comes from a know set of IPs published by google. So you cant pretend to be googlebot unless the website you are trying to fool is very unsophisticated.

[unkn0wn]

There must be something that tells us that the url has been internalized (resolve_internal_links = True).. like a dot or ^ above it or just change in link color (but wont work on e-ink devices), which will help readers in distinguishing links that would take us out of the application into a browser.. or to an article within.

[unkn0wn]

Why is this failing..

this is simple test code for businessweek.. to check why main one is failing

from calibre.web.feeds.news import BasicNewsRecipe, classes
from calibre import browser

class bb(BasicNewsRecipe):
    title = 'Bloomberg Businessweek test'
    language = 'en'
    __author__ = "unkn0wn"
    no_stylesheets = True
    remove_javascript = True
    remove_attributes = ['style', 'width', 'height']
    
    
    keep_only_tags = [
        dict(name='div', attrs={'class':lambda x: x.startswith('lede-text__')}),
        dict(name='figure', attrs={'class':lambda x: x.startswith('lede-media__')}),
        classes('body-content fence-body')
    ]

    def parse_index(self):
        soup = self.index_to_soup('https://www.bloomberg.com/magazine/businessweek/22_21')
        ans = []
        
        for a in soup.findAll('a', href = lambda x: x.__contains__('/news/features/')):
            url = a['href']
            if url.startswith('/'):
                url = 'https://www.bloomberg.com' + url
            return url
            title = self.tag_to_string(a)
            self.log(title, ' at ', url)
            ans.append({'title': title, 'url': url})
        return [('Articles', ans)] 

    # BB changes the content it delivers based on cookies, so the
    # following ensures that we send no cookies
    def get_browser(self, *args, **kwargs):
        return self

    def clone_browser(self, *args, **kwargs):
        return self.get_browser()

    def open_novisit(self, *args, **kwargs):
        br = browser()
        return br.open_novisit(*args, **kwargs)

    open = open_novisit

for a in soup.findAll('a', href = lambda x: x.__contains__('/news/features/')):

this should return at-least 3/4 links.. and i checked with javascript disabled too..
I tried different ways to fetch links.. I dont think calibre is looking at the same soup page that i'm able to see in my browser. how can we know?