Web scrape to Calibre.

[neo1]

Hi everyone, This post is to promote some code I wrote, and use daily. It's an alternative to Calibre News feature. (The code is Linux only.)

https://codeberg.org/neo1/mkebook

The idea is to use an rss reader to fetch only the articles you want for reading in Calibre.
The program will fetch the article, convert to epub, and add to Calibre, similar the Calibre News button.

Some advantages:
Filter articles for download in rss reader.
Use pandoc to convert to epub. I find the results look better than Calibre ebook-convert.
Impersonates a web browser bypassing bot detection.

Here's an animated gif showing a demo using the newsboat rss reader in the left window.

[neo1]

I migrated the code entirely to Python, so it should be much more portable.

I don't have a way to test this on Windows, so testers wanted. One should be able to scrape to calibre in Windows now, via Guard RSS for example.

[JSWolf]

Do you have a new demo as your old demo is gone?

[neo1]

JSWolf, thanks for pointing out the links stopped working. I did not realize that.
It was due to a mistake on my part. Here are some updated links.

Here is a demo 1 showing regular use.

Sometimes to fetch the web page you have to spin up a proper browser, which is a bit slower. Here is a demo 2 to show what is going on (in server mode). Under normal use the browser is invisible, and everything looks like demo 1, but things fetch slower.

The demos use newsboat as the rss reader to drive the tool, but one can a different rss reader.

[neo1]

Had to replace demo 2, new link: demo 2.

[JSWolf]

Does this work with Edge? Can it be made to work with Firefox?

[neo1]

That's a NO for Firefox, unfortunately.

Initially, Firefox was my target browser/platform, and I had some code
to that effect. I abandoned it due to problems. Furthermore, as I
understand it, if one did get Firefox to work, via the selenium
webdriver engine, it's detectable by the website and it lacks concurrency,
i.e. you can only work on one tab at a time. Whereas the chrome
cdp nodriver engine is undetectable and concurrent, i.e. tabs load in parallel.
Simply put, I could make it work with Firefox, but it would be sub-par so I don't
see the point.

Edge is just a flavor of chrome. If you were to point my program to
the Edge executable instead of chrome in the configuration file,
everything works. I just tested this. Testing I learned the
library I use for moving the mouse around to entertain cloudflare is
hard to build, so now it's optional and turned off by default.

[JSWolf]

IMHO, it would be best to post a version that uses Edge as a lot of people use Windows and have Edge. Not everyone has Chrome. I don't and never will.

[neo1]

JSWolf,

Thanks for the advice. It's a good idea. I put it in as a
fallback, i.e. if it doesn't find Chrome it will look for Edge. I
pushed the changes to the repository. I also realized that things on
Windows don't "just work" as I was expecting them to, so I had to be
more explicit about where to find calibre and such. I'll make a release once
I've used it a little.

I would still point out that
1. I have no way to test this on Windows.
2. If a user wants to use the calibre library server they have to edit
the configuration file because
2a. I don't want it to be used by default
2b. I need to know what port it's running on and
2c. (optionally) the name of the server

[PeterT]

Originally Posted by neo1

JSWolf,



Thanks for the advice. It's a good idea. I put it in as a

fallback, i.e. if it doesn't find Chrome it will look for Edge. I

pushed the changes to the repository. I also realized that things on

Windows don't "just work" as I was expecting them to, so I had to be

more explicit about where to find calibre and such. I'll make a release once

I've used it a little.



I would still point out that

1. I have no way to test this on Windows.

2. If a user wants to use the calibre library server they have to edit

the configuration file because

2a. I don't want it to be used by default

2b. I need to know what port it's running on and

2c. (optionally) the name of the server

You might want to change how you post. I think you're hitting return at the end of every input line leading to a large number of blank lines

[neo1]

JSWolf,

Thanks for the advice. It's a good idea. I put it in as a fallback, i.e. if it doesn't find Chrome it will look for Edge. I pushed the changes to the repository. I also realized that things on Windows don't "just work" as I was expecting them to, so I had to be more explicit about where to find calibre and such. I'll make a release once I've used it a little.

I would still point out that. 1. I have no way to test this on Windows and 2. If a user wants to use the calibre library server they have to edit the configuration file because
2a. I don't want it to be used by default
2b. I need to know what port it's running on

PeterT,
I'm copy pasting from a proper text editor. It was working before. Preview looks fine.
I *think* what might have happened was: this was the first post I did not preview.
See how ugly it looks if I don't add the newlines.

[PeterT]

This post is fine