How to get Business Standard to work, when it needs to fetch from a script

[unkn0wn]

the whole content is present in this part of html/script. With article body, headline, author, ,image link, etc all present within this script.

Feeds are plenty available.

<script type="application/ld+json">

        [{
          "@context": "http://schema.org",
          "mainEntityOfPage":{
            "@type":"WebPage",
            "@id":"https://www.business-standard.com/article/finance/banks-told-to-tighten-monitoring-have-ironclad-kyc-to-avoid-new-jamtara-122021601581_1.html"
          },
          "@type": "NewsArticle",
          "url": "https://www.business-standard.com/article/finance/banks-told-to-tighten-monitoring-have-ironclad-kyc-to-avoid-new-jamtara-122021601581_1.html",
	 	  				"articleBody":"
	The alarming increase in online frauds in recent times has prompted the regulator to ask commercial banks to shore up monitoring and make know-your-customer (KYC) credentials ironclad before opening accounts in specific districts of concern across the country.
Bankers said the home ministry may have received inputs on areas from where these fraudsters operate in and sounded off the regulator on the need for reassessing KYC documentation.

	&ldquo;The regulator has highlighted some jurisdictions where phishing is rampant. It said banks need to be watchful, monitor, insist on inflexible KYC verification, and further re-examine credentials belonging to geographical locations of concern,&rdquo; said a senior public sector banker.

	&ldquo;The perils of this nature will only increase with increased digitisation, availability of data, and exponential growth of financial technology (tech),&rdquo; said another senior banker.

	&ldquo;The digital and tech ecosystems are evolving every day, and the industry has been plagued by individuals trying to defraud companies and unsuspecting citizens. While there are many cases of fraudsters taking advantage of people who mistakenly share credentials and OTPs of their own volition, cases of identity theft are on a steady rise. Our risk management and tech teams have been on overdrive, constantly building more robust systems to try and keep such activities at bay,&rdquo; said the spokesperson.",
	                 "articleSection":"Finance",
          "headline": "Banks told to tighten monitoring, have ironclad KYC to avoid new Jamtara",
          "author": "Manojit Saha",
          "datePublished": "2022-02-17T06:05:00+05:30",
          "dateModified": "2022-02-17T09:13:59+05:30",
          "image":{
                "@context": "http://schema.org",
                "@type": "ImageObject",
                "width": "100px",
                "height": "100px",
                "url":"https://bsmedia.business-standard.com/_media/bs/img/article/2021-05/09/full/1620576306-5391.jpg"
              },
          "publisher":{
                "@context": "http://schema.org",
                "@type": "Organization",
                "name": "Business Standard",
                "logo": {
                        "@context": "http://schema.org",
                        "@type": "ImageObject",
                        "width": "100px",
                        "height": "100px",
                      "url": "https://bsmedia.business-standard.com/_media/bs/wap/images/bs_logo_amp.png"
                    },
                "url":"https://www.business-standard.com"
              },				"isAccessibleForFree": "False",
				"hasPart":
				{
				"@type": "WebPageElement",
				"isAccessibleForFree": "False",
				"cssSelector" : ".paywall"
				}
			          }]
   </script>

[kovidgoyal]

You use the json module ot parse thta data and convert it to html in the preprocess_raw_html() function. There are many existing recipes that do this. Just search for json in the builting recipes. For example economist.

[unkn0wn]

I did check out the economist .. twas very complex, i will try again.

Maybe if there's something that's less complex.

[kovidgoyal]

All you need is something like this in preprocess_raw_html

  soup = self.index_to_soup(raw)
  script = soup.find('script', type="application/ld+json")
  data = json.loads(str(script.contents[0]))

then you simply exract the data from json and cnvert it to simple html and return that.