Beta test phase for Calibre updates?

[kovidgoyal]

Strangely enough, the only people that ever request that calibre be released less frequently are the relatively technically proficient users. Most "ordinary" users that I hear from love the fact that calibre is released so frequently. Apparently they feel that someone cares. It seems that we geeks have a slightly distorted picture of what the ordinary user wants. I often get notes from people who send donations and many of those notes mention how appreciative people are of the frequent updates.

My philosophy with updates is to never release an update that could cause data loss without a beta first. Regressions in features that affect limited functionality on the other hand are acceptable, and quickly fixed.

About 60% of downloads (about 80000) of every calibre release are people installing calibre for the first time, given that there are currently 3 million calibre installs, a million of whom have started the calibre GUI at least once in the last 3 months. So I'm not convinced that ordinary users download new releases all the time.

As toddos pointed out, calibre is *vast*. Don't be fooled by the regression lists. It's relatively rare that a regression is very serious or affects a large fraction of users. And in such cases an update is usually released the next day. And that means that about 5k people are affected by the regression.

And I have to agree with chaley. In my experience, betas never have enough coverage to catch all regressions. So I don't see how to run a successful beta program.

Finally, to those people that suggest having updates advertised only every N releases. IMO that would be counter productive. Imagine you are on a release that has regressions, you will now have to wait N weeks instead of 1 week or 1 day for the fix. The point is that for regressions to be found and identified, large uptake is needed, and if update notifications are delayed, the fixes will be delayed and finding the regressions will also be delayed.

[Lady Fitzgerald]

Originally Posted by kovidgoyal

...It seems that we geeks have a slightly distorted picture of what the ordinary user wants...

Wow, Kovid! Most geeks (not an insult here) have difficulty understanding that they do not always understand the needs of us "ordinary" users (in my case, computer idiot). My hat's off to you (wait, I'm not wearing one)!

Originally Posted by kovidgoyal

...So I'm not convinced that ordinary users download new releases all the time...

I'm one of the ordinary users who doesn't always download new releases. First, I like to wait a few days to make sure there aren't any bugs before I download. Second, if the new release doesn't have anything that would be useful to me, what would be the point in downloading? With weekly updates, one doesn't have long to wait for an update that will be of interest. I leave the update notification on since I run calibre 24/7 and the notification pops up only when I reboot. It's nice to know a new one is out and it doesn't take all that long to delete the pop-up (1 second; big deal). At least the pop-up isn't an obnoxious ad.

Originally Posted by kovidgoyal

...In my experience, betas never have enough coverage to catch all regressions. So I don't see how to run a successful beta program...

I so totally agree. I see professional programs that do run betas and still don't "get it right." Typically, their mistakes don't get fixed for at least a year, if then. It's impossible to duplicate all hardware combinations nor anticipate how many boneheaded manuevers users may make or how they will use features. Calibre, on the other hand, has a large user base that will report any problems which are, typically, fixed in a week.

[kovidgoyal]

Originally Posted by Lady Fitzgerald

Wow, Kovid! Most geeks (not an insult here) have difficulty understanding that they do not always understand the needs of us "ordinary" users (in my case, computer idiot). My hat's off to you (wait, I'm not wearing one)!

Over the years if there's one thing that calibre has taught me, it's that other people (whether geeks or not) think about things in ways that I never would have conceived. Some of them are crazy, some of them are awesome, but they're all different

[DoctorOhh]

Originally Posted by kovidgoyal

Finally, to those people that suggest having updates advertised only every N releases. IMO that would be counter productive. Imagine you are on a release that has regressions, you will now have to wait N weeks instead of 1 week or 1 day for the fix.

I'm happy with the way it is now. My only suggestion might be that the default be a 12 or 24 hour delay in notification of a new release and provide a tweak to allow for a 0 hour notification.

This might also help the mirrors to handle the load.

[Agama]

Originally Posted by kovidgoyal

Finally, to those people that suggest having updates advertised only every N releases. IMO that would be counter productive. Imagine you are on a release that has regressions, you will now have to wait N weeks instead of 1 week or 1 day for the fix. The point is that for regressions to be found and identified, large uptake is needed, and if update notifications are delayed, the fixes will be delayed and finding the regressions will also be delayed.

Agreed! Scratch my suggestion, it was not a good idea. I generally download each new calibre straight after release and have not yet had to roll one back due to a significant problem. Thanks for all the hard work.