Quote:
news sites want Google to index their content so it shows up in search results. So they don't show a paywall to the Google crawler. We benefit from this because the Google crawler will cache a copy of the site every time it crawls it.
All we do is show you that cached, unpaywalled version of the page.
|
can we replicate this.. use googlebot.. access this cache
like they do
here and
here
Would open up lots of websites without the need to parse json and stuff.