UKKOSS13: A university project on calibre usability

[ukkoss_project]

Greetings!

We are a usability research team consisting of 5 students from the Computer Science / Information Processing Sciences department at University of Oulu, Finland. All five of us are in the latter half of our studies, working on our Master's degrees. As part of our studies, our task is to research usability of open-source software and usability activities within the OSS community. We've decided to base our research on calibre, as it fits the scope and intentions of our project (software aimed at ordinary people; has an active community; is under active development, project isn’t too small or too big, etc.).

As usability experts, our job is to represent the ordinary users, presenting improvement ideas in terms developers can understand, backed by data collected from expert evaluations and usability tests. The point is not to dumb calibre down but to streamline its functionality. So far we have conducted cognitive walkthroughs, heuristic evaluations and have just started doing usability tests in a lab.

We would love to contribute to the calibre project in any way we possibly can. The first step from our part would be to make a summary of the results of the usability activities we have conducted so far and present it to the community. We are also willing to make an effort in helping bring about our proposed changes to the best of our ability.

We'd like to know how interested overall the calibre community is in usability, and what kind of usability activities might be appreciated. We know that there has been some usability-related discussion in the forums in the past, but haven't been able to deduct if there's some general consensus on the state of calibre's current usability.

Our team has the additional support and usability expertise of university teacher Mikko Rajanen and professor Kari Kuutti. We believe that our usability efforts could benefit the calibre project.

Drop us a line - tell us what you think about this kind of project?

BR, UKKOSS13 team:

Kaisu, Lassi, Jari, Ville, Arto

[kovidgoyal]

Suggestions on improving calibre in any way, including usability are always welcome, as long as you remember that your suggestions may or may not be accepted, depending on the judgement of the calibre developers involved, which usually means me

Some things to keep in mind:

1) calibre's interface is the end result of almost a decade of steady improvements based on constant user feedback. That is not to say that it cannot be improved further, but, if the interface does something in a particular way, there is almost always a reason for it.

2) The specific trumps the general. This means that

a) Specific suggestions are much likelier to be useful rather than general ones. So for example: "Change the wording in dialog X to be Y is useful". "The wording in the dialogs should be made clearer" is not.

b) Many UI "experts" tend to have a bunch of general principles that they advocate with out taking into consideration how those principles apply to the specific situation, or whether there are special circumstances that trump those considerations. I'm not saying *you* will do that, but it is a tendency to watch out for.

3) Any given suggestion has to be implemented by a real human being, who is likely to have many other demands on his time. Keep that in mind when making suggestions.

[Katsunami]

Originally Posted by kovidgoyal

1) calibre's interface...
...
b) Many UI "experts" tend to have a bunch of general principles...

Kovid (and all regular calibre users) already expect where this is going. I imagine half of the suggestions with regard to usability can be thrown out of the window already

[mrmikel]

The calibre UI is also cross platform. This limits the ability to perfect it for each platform and this must be taken into account by any study.

[theducks]

Originally Posted by mrmikel

The calibre UI is also cross platform. This limits the ability to perfect it for each platform and this must be taken into account by any study.

*Plus 1*

There are very few features in Calibre that are not available on all platforms (including their subsets: eg no MTP on XP). Kovid chose to support the mainstream use, rather than support custom streams (that way leads to madness and loss of hair) for various platforms. Have a look near the bottom, here:http://status.calibre-ebook.com/

Lack of cross Platform compatibility (this may be caused by a toolbox limitation) is the most often cited reason for not implementing a feature request. (Patches welcome replies: is not a 'not', just a no interest in coding, response.)

The feature list (and plugin list) has grown by an amazing amount since I started using calibre in 2009.
I don't consider that just doing a skin-change be counted as an improvement.

[DrChiper]

As usability experts, our job is to represent the ordinary users, presenting improvement ideas in terms developers can understand, backed by data collected from expert evaluations and usability tests.

I do not think I really want you to represent me, nor do I believe it's worth a job either to represent me

But please start to provide us with *your* definition of usability, then we can see whether it fits my opinion of usability as YMMV.

Just my 2cts.

[theducks]

Dr C brings to mind another point.

You represent a University User.

Take a look at the MR user Profiles (it will be tedious ) and look at the ages of the Calibre users (when supplied).

Can you represent the over 40 crowd? The over 50 crowd? The over 60 crowd? The over 70 crowd? (Do we have any over 80's? )

[DrChiper]

I don't consider that just doing a skin-change be counted as an improvement.

*Plus 1*

(Mumbling some words resembling "windows 8" ...)

[BetterRed]

@MR Members - Here's list of the papers published by one of the lead researchers, Mikko Rajanen ==>> http://oulu.academia.edu/MikkoRajanen

@UKKOSS Team - It would probably be helpful to many of us if you could provide, brief summaries in simple prose targeted at lay persons - who may not be fluent in English, and whose formal education may not have progressed beyond Junior High - of:-

• your methodologies, e.g. where you recruit your 'users', how you measure usability, what weight you place on the software providing features such as:-
  • more than one way to accomplish a task,
  • usable on wide variety of 'platforms',
  • integration with a wide variety of 3rd party technologies much of which comes from the 'walled garden' side of the street rather than the 'open source' side.
  • extensible by non core developers - plug-ins, templates, recipes, breadboards etc
  • documentation - technical and user guide.
• b) your previous projects - ie what sort of software - phone apps, line of business, personal productivity, vocation specific, recreational etc. Along with any vendor testimonials.

BR

[ukkoss_project]

Hello,

I hope we didn’t get off on the wrong foot here. We’re university students; we’re here to learn, and if we can also provide something of value to the calibre community, that’s great. If not, then so be it - we’ll still learn from this experience.

We’re not here to “take over” or proclaim our superior usability boffin ways; we just want to hone our skills while hopefully providing something real or useful to you, the calibre community. We understand that this whole scenario might seem rather annoying, that is, a group of random students coming out of the blue and starting a big usability discussion. We wouldn’t have chosen calibre as our subject matter if we hadn’t considered it as quality software. The idea of choosing this software in the first place for our project subject came from one of our members who had been a regular user of calibre for some time, and we wanted to see if it could be further improved in some way.

Now, some specifics; we plan to conduct tests with a small sample of people. Relatives, friends, etc. from different backgrounds and age groups who have not previously used calibre but express an interest in reading, and who have at least basic computer skills. We cannot devote a huge amount of time and resources to only just this however, as it would be outside the scope of what we are doing. We have a limited amount of work hours to devote to this project, and once those are filled, we are done. As such, we have to plan the project so that we do not over-extend ourselves on one task and end up not being able to finish the project properly with the remaining time.

In addition to these tests, we will also perform so called expert reviews from the point of view of a computer scientist (student). We will apply our knowledge to these reviews as it has been taught to us during our studies; whether it works in the context of the calibre project remains to be seen. We are shifting our focus on specific matters more than broad generalizations, however, as suggested by you. This may, in fact, prove to be a good way of applying usability work to something with a long legacy like that of calibre.

Regarding the comments on heuristic evaluation, we completely agree that blindly following some checklists will do no good if the software’s special characteristics aren’t taken into account. However, “more than one way to complete tasks”, “cross-platform”, “integration with wide variety of technologies”, “extensibility” and “documentation” aren’t characteristics that would make calibre different from any other software in terms of usability / how “pleasant” the program is to use overall.

Any software is usable to someone who has used the software for a decade and knows all of its ins and outs. This doesn’t mean that a new user has the same experience. Our aim is to diagnose these problem points for newer users and propose possible changes to them, WITHOUT hampering the existing functionalities. We do not want to dumb down calibre!

What you do with our findings is of course up to you. We hope that we can do a good enough of a job that we can at least spark conversation of usability work within the calibre project. But if you deem that there is nothing that usability efforts could provide for you, that’s your prerogative.

TL;DR: We’re here to learn and hopefully provide value to the calibre community; please don’t murder us!

[user_none]

Originally Posted by ukkoss_project

However, “more than one way to complete tasks”, “cross-platform”, “integration with wide variety of technologies”, “extensibility” and “documentation” aren’t characteristics that would make calibre different from any other software in terms of usability / how “pleasant” the program is to use overall.

Cross platform really does make calibre different than other software. That's not to say other software doesn't have to worry about this because there are plenty of cross platform applications out there.

Part of usability is expectation. OS X applications (for example) are expected to act and behave in a certain way. Windows applications are again expected to act and behave in a certain way. These behaviors are not the same.

Saying X, Y, Z is more usable may be true for a Windows user due to their expectations but it can potentially be completely backwards to what an OS X user expects. This needs to be balanced. What ends up happing is a compromise that works and is usable but doesn't make users of either platform happy.

This is a very common usability issue that comes up time and time again. This in particular is one reason why many people are cautious with your proposal. I'm not saying that it's not welcome but there are many threads here which basically go like this:

• new_user: Hi! I have some ideas on how calibre could be more usable.
• community: Great! We'd love to hear them.
• new_user: Change everything and make it work like other OS X apps.
• ....or.....
• new_user: Change everything and make it work like other Windows apps.
• ....or.....
• new_user: Change everything to make it work like this other application.
• community: That can't or won't be done for multiple reasons explained below. These also have little to do with usability but with your expication on how an application should function on your preferred platform.
• new_user: GO F**K yourselves for not doing what I say you should do because calibre is S**T if you don't and this is the only way to make people want to use this piece of garbage.

Discussion about specific identifiable issues and concrete solutions to these problems is welcome. But again take into account that while something may appear usable on OS X it may not be usable to a Windows user due to expectations. Saying, "change calibre to work how I think it should work because I say so", isn't welcome.

[Katsunami]

Maybe this comes across as a stupid remark, but I really believe the following to be true to some extent: calibre's usability or learning curve is basically unimportant.

Why?

1. Someone not willing to study calibre is problably unwilling to maintain a huge library and will just use Adobe Digital Editions, Kindle 4 PC, or maybe even download books directly from the Kindle/Kobo cloud.

2. Someone looking for a program capable of doing what Calibre can do, basically has no choice. You use Calibre, or not.

Calibre can focus on functionality and power instead of userfriendlyness, because if you need or want calibre's capabilities, you must study and use it, whatever GUI it has. If you don't like the way it works: pity for you. There is no competitor in town.

In short: at this point, many people who are using Calibre don't really care about the interface, or dialog layout, and so on. I see many points that could be improved if (as user_none states) a base line would be drawn, such as: "We make it work like in Windows/OSX/Linux/like app X." However, doing this will improve usablility for some, and degrade it for others, so this will not happen.

At this time, Calibre is Calibre. It's different from any other program I've ever used on whatever platform. Does it matter? No, because there is no viable alternative.

[Adoby]

Yes, but that doesn't mean that there isn't room for improvement. I agree that there currently is no real alternative to calibre. It is a killer app, in the sense that it would be very difficult to make a competing app from scratch.

However it would be very possible to make an open source app based on calibre that essentially is calibre lite. Aimed at the very big(?) group of users that find calibre difficult to understand and use.

For instance an app that doesn't have an internal library, but works like most mp3, movie and image media managers. Perhaps store paths to the books, not the books themselves, and update if files move. Use hashes to reconnect moved or renamed books to the database. It would be less efficient, more prone to file corruption and not very suitable for a large library. But I bet that a lot of people would love it.

It could be native to one operating system, and use conventions that are common in that operating system. That would make the app stick out less from other apps and perhaps lower the learning curve. The user base would of course be smaller. Or it could have a common codebase and functions with several different native user interfaces. More native than calibre is.

If it was built using the calibre codebase, then it obviously have to be open source and acknowledge the calibre origin. And also have its own support and so on, separate from calibre. But by building on calibre it could also be easier to develop. Things like conversion and book editing and metadata download, save to disk and device could perhaps be copied and used as is. Perhaps it would be possible to use automatic conversion from python to C or C++ to improve performance?

And of course calibre could benefit from users that grow out of "calibre lite" or even add design elements or other improvements.

Actually, I am surprised that there is no open source "calibre lite". That might even be a nice project for a group of students...

[chaley]

@ukkoss_project: do your project. Assuming that you are academically honest and justify your findings with evidence (and I have every reason to believe you will be and will do), then the findings will be interesting and will be used as we go forward.

I have done a fair amount of calibre development (person-years), and I can say with certainty that the combination of a) functionality I want, b) functionality other people want, c) capabilities of the cross-platform tool kit, d) time I have to spend do development, and e) interest I have in the problem; all play a significant role in what I do. I am certainly willing to add information from your project to the mix, but it would be one factor out of many.

Another issue that you will face in this project: calibre has a *huge* installed base. According to calibre's statistics page (http://status.calibre-ebook.com/) there are 3.5 million current users spread across a lot of countries. A large number of these users will be unhappy if anything changes because then they must spend brain cycles to work out how to do something they knew how to do before. They don't care that it might be easier for some unknown new person. (To give a personal example, I have refused to upgrade MS Word beyond 2003 because I detest the ribbon.) These calibre users can be vocal and sometimes hostile. You will need to be rather thick skinned, ignoring the hostility and concentrating on evidence-based recommendations.

Good luck!
Charles Haley, PhD
(retired) Lecturer in Computing at The Open University, UK

[Katsunami]

Originally Posted by Adoby

Actually, I am surprised that there is no open source "calibre lite". That might even be a nice project for a group of students...

I have already created many "Calibre Lite" installations, just by removing all toolbars and icons and using the menu and context menu, populated with needed items only. Many people can make do with basics, and they never get into the configuration.

It would be possible to supply a configuration setup with Calibre, asking the user in the installation wizard if he wants an "Easy" or "Expert" interface. The former will just hide all advanced items (and possibly even parts of the configuration screen).