Enhancement Guidance: Auto-update

[uriuri]

Hey I've been using Calibre on and off for a while, and it always seems like there's an update every time I launch it (Kudos to Kovid and the active dev community).

I consider myself to be a capable developer and I would like to explore the possibility of adding auto-updates to Calibre. I haven't started digging through the code yet but I was wondering if this has been attempted in the past, and if anyone has any tips.

Thanks!

[eschwartz]

This has simply not been a priority. See: http://manual.calibre-ebook.com/faq....tomatic-update

The issue has come up several times I can think of, but only from people who want loudly expect it.

If someone else develops the feature, I believe Kovid would accept it. It should be incremental however, as noted in the FAQ.

[jgaiser]

No. No. No. No automatic updates. I want to decide when to update.

And yes, this gets brought up from time to time and the answer is always no. There are scripts available if you feel a need.

[BetterRed]

Originally Posted by uriuri

Hey I've been using Calibre on and off for a while, and it always seems like there's an update every time I launch it (Kudos to Kovid and the active dev community).

I consider myself to be a capable developer and I would like to explore the possibility of adding auto-updates to Calibre. I haven't started digging through the code yet but I was wondering if this has been attempted in the past, and if anyone has any tips.

Thanks!

If you peruse the sticky threads here ==>> Related Tools you might find something to suit you.

but my needs are jgaiser's

BR

[eschwartz]

Originally Posted by jgaiser

No. No. No. No automatic updates. I want to decide when to update.

And yes, this gets brought up from time to time and the answer is always no. There are scripts available if you feel a need.

Seriously?

1. It would be optional.
2. The answer is "I have better things to do with my life" -- that does not preclude accepting a merge request.
3. The previous times you allude to were whiny demands. This is the first time I am aware of that someone approaches from the other end and offers to contribute it.
4. I helped author, and currently maintain, one of those scripts. I think I have the right to say, this way is better (if coded in a way that meets Kovid's approval).

Both of you -- I managed to answer all your concerns without dismissing the OP out of hand.

[eschwartz]

I think the model of Firefox and Google Chrome is an excellent one. If every application out there could do the same, that would be awesome. Granted, most don't have the popularity that makes it worth the developer's time to implement.

If someone wishes to scratch that itch anyway, they should not be rejected out of hand.

[BetterRed]

When I am notified of new releases (mainly via RSS) I download whatever (including Firefox & Chrome) at a time that suits me, over a link that suits me. It goes into an install queue, which I process after taking a system checkpoint at a time that suits me.

Some applications, including Firefox, are first installed into a VM where I check them out for surprises. Been bitten too many times with things not working satisfactorily. I didn't install Firefox 34.0 because it had a bug that affected me, a few days later a fix release was issued, 34.05 I think.

I've been following that basic process for 30+ years for my personal systems and longer for work. It isn't going to change until up pushing up daisies.

BR

[eschwartz]

And that makes you unusual, so surely you should realize that your perspective is not the most accurate guide to the acceptability of such a feature in a more general sense.

Witness: That feature exists in FF/Chrome, even though you don't use it.

Assuming the trend continues, you wouldn't use this either -- but tons of other people would.

[kovidgoyal]

Auto-updates are not something that can be implemented in a pull request. They require a lot of backup infrastructure, at a minimum, the release process has to be modified to maintain binary deltas of the various installers, and the distribution servers have to be modified to host those deltas.

That said if someone is willing to do the coding for the calibre side of the equation, I am willing to help with the rest.

What would be needed, is

1) A way to *reliably* download binary diffs, and use them to update installer images. This means hash checks of the patched installer images at a minimum and the ability to rollback if a patch fails.

2) A way to *atomically* update the calibre installation on launch, when an update is waiting. This is particularly important, a background update must *never* break a working calibre install.

3) A way to notify the user if auto-updating is failing repeatedly, so that the user can perform manual updates.

4) Auto-updates have to be optional, defaulting to off. That is necessary for calibre, since unlike Chrome and Firefox it does not have a large development and QA team to ensure releases are relatively bug free. I do a fairly good job with that on my lonesome ownsome, but I absolutely refuse to responsible for an on-by-default auto update process breaking working software on people's systems.

5) You need to figure out a way to have auto updates work for installations in system folders for which the normal user account does not have permissions, without prompting the user. Most applications get around this by running a root background updater service, which is a particularly ugly (security nightmare) and resource intensive solution, but is probably the only robust way.

Auto-updates are particularly problematic on windows, where the installer uses MSI and files are mandatorily locked by the OS. I have no idea how one would incrementally update an MSI based installation in the background.

So while, to the uninitiated it might seem like enabling auto-updates is relatively easy, doing so *reliably* and *robustly* is a whole other ball game.
I'd estimate it would take over a man month of work -- for only the initial implementation, assuming there were no nasty surprises along the way, and there are almost always nasty surprises. And then will come all the work to make it robust.

[BetterRed]

Originally Posted by eschwartz

And that makes you unusual, so surely you should realize that your perspective is not the most accurate guide to the acceptability of such a feature in a more general sense.

Witness: That feature exists in FF/Chrome, even though you don't use it.

Assuming the trend continues, you wouldn't use this either -- but tons of other people would.

Multi user, metadata maintenance via the web browser, author link like fields on all categories, searching across physical libraries, full text search of book contents, add files of any type to a book via dropping it into book details... are all higher up my wish list - and a good many others I suspect.

There is also the issue that auto updating will probably push up the download traffic by a significant volume. I doubt Kovid's ongoing revenue would increase in proportion to the ongoing expense of the download traffic charges he would have to pay.

BR

@Kovid - please replace "I do a fairly good job with that..." with "I do an awesome job with that...", it will scan more better

[itimpi]

Originally Posted by BetterRed

There is also the issue that auto updating will probably push up the download traffic by a significant volume. I doubt Kovid's ongoing revenue would increase in proportion to the ongoing expense of the download traffic charges he would have to pay.

If an auto-update process can be implemented as deltas then it is likely to significantly reduce traffic I would have thought.

If the default setting is off, then I would not be surprised if those who switch it to on are also likely to b those who download the full (new) release every week.

[eschwartz]

Originally Posted by BetterRed

Multi user, metadata maintenance via the web browser, author link like fields on all categories, searching across physical libraries, full text search of book contents, add files of any type to a book via dropping it into book details... are all higher up my wish list - and a good many others I suspect.

That is an entirely different point.

And I am sure Kovid will not delay release of those *tentative* enhancements... any more than they already are.
Perhaps @uriuri would like to work on those instead, but then again, perhaps not.

Where an item ranks on the average wishlist is an important qualification... to someone who isn't specifically interested in a specific feature. I wouldn't automatically assume that is not the case here.

There is also the issue that auto updating will probably push up the download traffic by a significant volume. I doubt Kovid's ongoing revenue would increase in proportion to the ongoing expense of the download traffic charges he would have to pay.

That is why I recommended looking into incremental patching. As itimpi said, that should result in a net decrease in bandwidth costs... especially if it defaults to off.

@Kovid - please replace "I do a fairly good job with that..." with "I do an awesome job with that...", it will scan more better

Agreed!

[theducks]

Would it be possible to shift the built in News recipes to a system more like the get user PI notice uses?:

saving 5 M of base traffic (the Zip file) on every calibre installation

Even the full available 'list' would only be pulled (refreshes) only if the user has active subscriptions/uses the Get News icon

Individual news-source updates would be near instantly available, instead of needing to wait for the next release cycle

If those of us who do not us any news source, the News related PI traffic would approach zero

[eschwartz]

I believe Kovid has said recipes are automatically updated when run -- I think it first looks at the up-to-date version hosted on the calibre website, before running the job.

I suspect a list of the current ones is needed, so calibre knows what is available without pinging the server first.

[uriuri]

And I am sure Kovid will not delay release of those *tentative* enhancements... any more than they already are.

Perhaps @uriuri would like to work on those instead, but then again, perhaps not.

Is there a list of enhancements or "todos"? I wouldn't mind starting to work on something simpler.