enhancement proposal: DOWNLOADING NEWS

[Ersatzreifen]

I was going to post this in the development subforum, which is the most logical subforum for making this kind of suggestion, but the 'read this before posting' sticky thread said "This subforum is not the place for....enhancement proposals unless you make clear in your post that you are prepared to develop the enhancements yourself...". It also inconveniently neglected to recommend where to post suggestions, and no subforum description seems to welcome suggestions, so I'm putting it here. If this is the wrong place, then please make an obvious place.

I want to suggest the following changes to News downloading:

Calibre should be used only to configure the scheduling of downloads, but not automatically initiate any downloads at all, either upon calibre bootup or by schedule. It should only optionally allow the user to manually initiate a download.

Instead, all scheduled downloads of News should be done by calibre-server while it's running. This way I won't have to start calibre every damned day just to initiate a news download, and won't have to leave it running to do scheduled downloads - that's the job of server software running 24/7, not client software which is started only on logins. I also won't have to wait and wait for News download jobs to finish every time I start calibre to make some quick adjustments. (I don't like interrupting jobs, but don't like them interrupting me either.)

If there's anyone who, for whatever reason, doesn't want this change, and still wants calibre to do it the current way, then it should be easy enough to include a setting to restore everything to calibre for backward compatibility.

This rearrangement of the way News downloads are done would be a great improvement for News.

[kovidgoyal]

Perhaps it slipped by you that the calibre server is read-only. So it cannot do things like adding new books to the calibre library which is needed for news download. You dont want to run the main calibre program then setup cron jobs to download the news for you. That's what cron is for.

[Ersatzreifen]

Originally Posted by kovidgoyal

Perhaps it slipped by you that the calibre server is read-only. So it cannot do things like adding new books to the calibre library which is needed for news download. You dont want to run the main calibre program then setup cron jobs to download the news for you. That's what cron is for.

Perhaps it slipped by YOU that I haven't done any programming since before 1980. In those days we could rewrite programs to do things like that. There's no reason as far as I can see where calibre has to be read only, that's just a programmer's personal aesthetics. It could work both ways, as does an ftp server. That way people could even do amazing things like upload ebooks which get put into a quarantine directory to be vetted before merging into the library. But then, what do I know? Perhaps compilers have evolved to the point where it's no longer possible to write server software that can upload, download, and fetch files. Or maybe it's paranoia. Or just personal aesthetics.

I thought of using crontab, and I figured that if there were a separate program dedicated to doing just that, crontab would be perfect. But when I looked into /opt/calibre/bin/ I didn't see any separate module to handle only downloading news from a cron job, (which would have been a perfect solution.) Or maybe calibre-download-news (or whatever you call it) "slipped by me". So, if I can just start caliber from a cron job just to grab news downloads only, how do I instruct it to do only that, then terminate to free up memory as soon as it's done?

I checked "calibre --help" and there's no procedure for this; nothing to tell it to ONLY download news, then exit as soon as it's done. There is a "-s, --shutdown-running-calibre" switch, but then I have to guess how long downloads will take and run a second cron job just to terminate the first. Possible, but messy programming!

So tell me, how would you write the cron job so that it only grabs the news then terminates cleanly?

[Divingduck]

Deleted

[PeterT]

Did you try reading the calibre FAQ?

I see this

You can download news and convert it into an ebook with the command:

/opt/calibre/ebook-convert "Title of news source.recipe" outputfile.epub

If you want to generate MOBI, use outputfile.mobi instead and use*--output-profile*kindle.

You can email downloaded news with the command:

/opt/calibre/calibre-smtp

I leave figuring out the exact command line as an exercise for the reader.

[Ersatzreifen]

Originally Posted by PeterT

Did you try reading the calibre FAQ?

Did you read the OP for this thread? The topic concerns cleanly automating the action of scheduled downloads. What you're talking about is a non sequitur, a red herring. It has nothing to do with being able to automate the downloading of news without having to start up calibre. Is that in the FAQ?

[itimpi]

Originally Posted by Ersatzreifen

Did you read the OP for this thread? The topic concerns cleanly automating the action of scheduled downloads. What you're talking about is a non sequitur, a red herring. It has nothing to do with being able to automate the downloading of news without having to start up calibre. Is that in the FAQ?

i would say it does answer the question. It gives the command you need to run and only leaves to you to find out how to set up your OS to run that specific task at a specified time/frequency.

[PeterT]

Originally Posted by Ersatzreifen

Did you read the OP for this thread? The topic concerns cleanly automating the action of scheduled downloads. What you're talking about is a non sequitur, a red herring. It has nothing to do with being able to automate the downloading of news without having to start up calibre. Is that in the FAQ?

Huh.. automate it by running those commands via cron or its equivalent on other systems. Sorry if the concept is to complex for you to gather

[davidfor]

Originally Posted by Ersatzreifen

Did you read the OP for this thread? The topic concerns cleanly automating the action of scheduled downloads. What you're talking about is a non sequitur, a red herring. It has nothing to do with being able to automate the downloading of news without having to start up calibre. Is that in the FAQ?

Did you read PeterT's post and your second post? You said you could use crontab if there was a command to do the news download. That's what PeterT post was about. You will have to do the scheduling yourself, but you can create a crontab job for each recipe or one big one for all recipes.

[Ersatzreifen]

Originally Posted by davidfor

Did you read PeterT's post and your second post? You said you could use crontab if there was a command to do the news download. That's what PeterT post was about. You will have to do the scheduling yourself, but you can create a crontab job for each recipe or one big one for all recipes.

IF there was a command to do the news download. That's a big if.

Peter's post lists two different commands:

/opt/calibre/ebook-convert --help
Convert an ebook from one format to another.
and:
/opt/calibre/calibre-smtp --help
Send mail using the SMTP protocol.
Which one just downloads news? Peter didn't say. The --help feature of each doesn't mention fetching news, so unless you can demonstrate how to use ebook-convert or calibre-smtp just ti initiate a news fetch from the command line, then it looks to me like they weren't designed to do that.

One is for converting one book to another format, the other for sending books by email. NEITHER ONE can be used from the command line to download scheduled news feeds, can they? So no, his non sequitur does not tell me how to initiate a news download from the command line. Using crontab isn't the problem. Using crontab is a fricking piece of cake. So if
ebook-convert or calibre-smtp has an undocumented feature of fetching news from the command line, then pray tell precisely how. I don't see any documentation on it, nor is it intuitive.

[davidfor]

Originally Posted by Ersatzreifen

IF there was a command to do the news download. That's a big if.

Peter's post lists two different commands:

/opt/calibre/ebook-convert --help
Convert an ebook from one format to another.
and:
/opt/calibre/calibre-smtp --help
Send mail using the SMTP protocol.
Which one just downloads news? Peter didn't say. The --help feature of each doesn't mention fetching news, so unless you can demonstrate how to use ebook-convert or calibre-smtp just ti initiate a news fetch from the command line, then it looks to me like they weren't designed to do that.

One is for converting one book to another format, the other for sending books by email. NEITHER ONE can be used from the command line to download scheduled news feeds, can they? So no, his non sequitur does not tell me how to initiate a news download from the command line. Using crontab isn't the problem. Using crontab is a fricking piece of cake. So if
ebook-convert or calibre-smtp has an undocumented feature of fetching news from the command line, then pray tell precisely how. I don't see any documentation on it, nor is it intuitive.

You might want to actually read everything. The FAQ page is at https://manual.calibre-ebook.com/faq...n-linux-server. It explicitly tells you that that command will download the news.

And if you had actually read the output from the command you would have seen it handles the news recipes. (Highlighted in red below to make it clear.)

E:\Download>"C:\Program Files\Calibre2\ebook-convert.exe"
Usage: ebook-convert.exe input_file output_file [options]

Convert an ebook from one format to another.

input_file is the input and output_file is the output. Both must be specified as the first two arguments to the command.

The output ebook format is guessed from the file extension of output_file. output_file can also be of the special format .EXT where EXT is the output file extension. In this case, the name of the output file is derived from the name of the input file. Note that the filenames must not start with a hyphen. Finally, if output_file has no extension, then it is treated as a directory and an "open ebook" (OEB) consisting of HTML files is written to that directory. These files are the files that would normally have been passed to the output plugin.

After specifying the input and output file you can customize the conversion by specifying various options. The available options depend on the input and output file types. To get help on them specify the input and output file and then use the -h option.

For full documentation of the conversion system see
https://manual.calibre-ebook.com/conversion.html

Whenever you pass arguments to ebook-convert.exe that have spaces in them, enclose the arguments in quotation marks. For example "C:\some path with spaces"

Options:
  --version       show program's version number and exit

  -h, --help      show this help message and exit

  --list-recipes  List builtin recipe names. You can create an ebook from a
                  builtin recipe like this: ebook-convert "Recipe Name.recipe"
                  output.epub


Created by Kovid Goyal <kovid@kovidgoyal.net>

And do you really think I would have wasted my time replying if PeterT had been wrong? No, I would have replied to PeterT telling him why he was wrong. Of course, I know both PeterT and itimpi both well enough here to know I'm not likely to find them wrong except when I feel like pedantic.

[Divingduck]

I had delete my previous post because I thought, you had look on this already, but maybe not.

calibre help, fetching news via cmd line:

PHP Code:

Code sample, with adjustment for e-reder and a additional log file:
ebook-convert HighCountryNews_AGe.recipe .epub --output-profile sony -vv --debug-pipeline debug > HighCountryNews_AGe.log 


Depending on what you like to do, you need to do additional steps like adding to library, sending to an location and so on.

[Divingduck]

Oh, now we had this two times. Maybe this will help

[Ersatzreifen]

Originally Posted by Divingduck

I had delete my previous post because I thought, you had look on this already, but maybe not.

calibre help, fetching news via cmd line:

PHP Code:

Code sample, with adjustment for e-reder and a additional log file:
ebook-convert HighCountryNews_AGe.recipe .epub --output-profile sony -vv --debug-pipeline debug > HighCountryNews_AGe.log 


Depending on what you like to do, you need to do additional steps like adding to library, sending to an location and so on.

All that manual configuration and additional steps too? For each item to be downloaded?

Oh. I see.

Okay, I give up. You win. You folks simply won't even try to understand my concern, and why I've been trying to suggest a way to make it easier for users,instead of requiring a poor choice between more manual work or clunky memory-wasting workarounds. If all I had to do was this:
0 5 * * * ebook-convert --fetch-scheduled-news-convert-it-and stick-it-in-my-library-and-use-the-settings-from-calibre's-"Schedule-news-download"-to know-what-to-fetch-and-where-to-put-it-without-me-having-to-pedantically-configure-it-manually-for-each-seperate-news-item-and-still-have-to-manage-it-manually-when-changes-are-needed
....THAT sort of thing would have done what I'm looking for.

But you're right. Why use the extant "Schedule news download" to set up downloads then have an external module actually perform the downloads, when each user can be forced to leave calibre running or spend countless hours digging through documentation, figuring out how to build up a shell script to do what is mostly already automated, and then have to manage it without the extant gui? I foolishly thought that what I'm looking for would make sense and save time and memory, but requiring users to spend precious hours learning how to manually configure downloads of dozens to hundreds of news sources and tinkering with code is much more fun, isn't it?

It's nice to "be able" to micromanage software with such time-consuming configuration options - if you have lots of spare time, but like most people I just don't have enough hours in a day to do everything that needs done. So I want something simpler. I'm funny that way. All I really wanted is something external to calibre, like a module, to take the execution of the actual download out of calibre and allow the user to decide either to allow calibre to initiate the scheduled download as it does now, or to initiate it from a cron job - without users having to either keep calibre running or reinvent the scheduling wheel by having to tediously configure everything from scratch, then still have to "do additional steps like adding to library". And being able to configure it from the existing "Schedule news download" gui makes maintenance easy for users. But I see your point: a separate module for easy automated downloading of news is a silly idea, let's keep it monolithic. Right? Anyone who doesn't like keeping both calibre-server and calibre running 24/7 can just do it the hard way.

Right now this "works" (anything works, it just depends on how you define "works"):
0 5 * * * /opt/calibre/bin/calibre
0 6 * * * /opt/calibre/bin/calibre --shutdown-running-calibre
...so I'll just be forced to keep using it until someone figures out what I'm really asking for. It's a sloppy way to do it, I know, and doesn't guarantee that the jobs will all be done before it's terminated, but I don't have enough time every day to do things the tedious way. All I want is a simple external module I can run to get it done, like this:
0 5 * * * /opt/calibre/bin/fetch-news <----<<< (a module, not a homemade script)
which will read the settings saved by the "Schedule news download" gui ....without users having to build an involved shell script. (...and the more news sources scheduled, the more involved it gets. Right?)

All the code is already built into calibre, trivial for the developers to move, so why can't it be cut out and made in to a separate module so users can easily use it from a cron job? (Rhetorical question.) I had originally suggested moving the code into the calibre-server, but the developers don't want the server being able to initiate the downloads - they want it to remain read-only. So that suggests that a separate module would be better.

In ending this thread, here's my suggestion to the developers:

1. Leave the scheduling gui code in calibre so that the setup is done from the gui as it now is.
2. Add a switch so users can disable calibre from initiating the scheduled downloads so it must then be initiated externally.
   Add a box so users can tell it which library to put the downloaded news in.
   
3. Extract the code that actually does the downloading and modularize it so it can be initiated either from calibre, command line, or cron job - let the users choose.

THAT is what I'm trying to suggest to the developers. Now I'm sorry I mentioned it.

[Divingduck]

Originally Posted by Ersatzreifen

[COLOR=black]
All the code is already built into calibre, trivial for the developers to move, so why can't it be cut out and made in to a separate module so users can easily use it from a cron job? (Rhetorical question.)

.... because not a few users look forward to calibre 3.0 and we all (I at least) hope that Kovid will find enough time to finish this part soon

But serious, calibre had a lot of tiny nice kinks, but we all can find that important special one that isn't there. We can ask for it and Kovid will answer with yes, no or maybe some when in the future. Maybe it will come or not. For the meantime there is mostly a workaround available and we are free to use it or not. If there is a problem with a workaround, we can ask and mostly we will receive a kindly and serious answer/help from experienced members or Kovid. That's all what seems to be possible for your special problem for now.

Apropos meantime, this is maybe also a good workaround these days: