The Future of this Forum

[GRiker]

I'm posting this because I'm concerned about the future of this forum, and more generally, the future of calibre.

Brief background: I discovered calibre in April of 2009 after purchasing a Kindle. After learning the basics of version 0.5.9, I started asked questions on this forum, learning from fellow users. When I learned that some features I wanted weren't available, I decided to roll up my sleeves, learn Python, and teach calibre some new tricks. My first code contribution was in July of 2009, and I've been contributing code and occasional forum answers ever since.

My concern: Over time, I have witnessed a slow but certain shift in the basic attitude of new folks joining this forum. When I started spending time here, the general attitude was "Wow, this is free software?" and "Wow, the actual developer will talk to me, usually within the same day?"

Now, I'm seeing a growing number of new posts more along the lines of "I can't figure out how to do X and someone needs to solve it for me." They may have made some half-hearted attempt to search the forum, but decided it would be easier to ask everyone else. The sense of new user entitlement has grown at an alarming rate.

I'm not blaming the newbies. Calibre is a massive piece of software with an enormous range of functionality that evolves every week. As a contributing developer, you might think that I am aware of all the changes. I'm not - I just know about the changes I'm involved with, a small percentage of the overall changes from release to release.

When I see a snarky response to a newby question that boils down to "RTFM", that pains me. To me, it means that we, as a community of developers and users have not done an adequate job of simplifying the learning curve and sharing our knowledge.

In theory, the answer to just about any question about calibre lies in this forum. The knowledge contained within these posts is astounding. I frequently search the forum for answers I remember seeing, and I often save threads I know I'll want to come back to later. (Thread tools|Subscribe to this thread). And if an answer is not here, posing a question in a competent manner usually yields a thoughtful answer.

What I'm proposing: A restructuring of the sticky section, with a single FAQ topic:

New to calibre? Start here

The goal of this new FAQ would be to provide hand-holding to new users with respect to the non-changing elements of calibre, and the most common new user problems.

This topic would be a moderated thread, meaning that some designated individual(s) would have the responsibility of maintaining it. Initially, it would contain links to all the current sticky threads, which would be converted to regular threads, plus links to recurring topics.

This new FAQ would be read-only, unlike the current FAQ sticky which quickly became unwieldy. Requests to add content to the FAQ would be directed to the folks maintaining it.

I don't know how Kovid feels about this, nor the other active developers, but I am certain that if we don't reduce the noise around here, the quality of support we've all enjoyed will rapidly erode.

If you have constructive comments, chime in.

If you would be willing to participate in the moderation of such a FAQ, please identify yourself. You don't have to be technical, but you do need to be literate and willing to invest the time.

G

Proposed forum Structure:

All stickies are read only

Calibre
  Stickies: 
    1. Read this first with instruction on how best to use the forum, as brief as possible

Sub forums:

Recipes: 
  Stickies:
    1. Code snippets (open)
    2. Howto use a custom recipe in calibre
    3. How to write your own recipes

Devices:
    Stickies:
    1. What to do when your device is not detected (Kovid)
    2. Apple device issues (GRiker)
    3. Plugboards (chaley)
 
Catalogs: (GRiker)

Conversion:
   1. Using search and replace
   2. Creating Table of Contents
   3. PDF issues

Library management:
  1. Using templates for adding/exporting books
  2. You cannot control the calibre directory structure
  3. Tips on advanced searching

Plugins
  1. How to use plugins
  2. A list of plugins with a link to the thread for each plugin

Volunteers:

Post in this thread if you are willing to volunteer and I will update this section. I envisage volunteers having moderating permissions on the calibre forum (and subforums)

Piper_
GRiker
kovidgoyal
theducks
Manichean
kiwidude
ldolse
dwanthny

[Manichean]

There was some discussion a few days ago about maybe improving the documentation of Calibre. However, after recently having a quick look at what was actually in there, I find it more than adequate to get a new user started in doing the basic stuff Calibre is capable of. Astonishingly enough, we get more than a few users who seem to haven't even made the effort to look at the documentation, despite there being a quickstart guide available when you first install Calibre. As, by default, Calibre has a button on its toolbar that opens the manual, I don't know how to solve that.
As I've said when the discussion about the documentation, I'm willing to start an effort to (further) improve the documentation of Calibre, however, I won't find time for that until at least march. I couldn't exactly tell how to go about doing that, though- maybe a few HOWTOs might help, but then again, much of the documentation already is a howto.

I like the idea of restructuring the sticky section, however, I don't know how that should be done. I didn't ever have a careful look at the FAQ thread, however, that and stickying the thread Worldwalker made about the directory structure as well as the regex thread would cover about 90% of the questions we get. I'd also be willing to work on improving the stickies on the forum, however, the same modifier as above applies.

[Lady Fitzgerald]

I suspect many of the recent posts putting down calibre have been from the same troll; this has been reported.

Please keep in mind many of the people (including me) who come to this forum do not know diddly about computers and computer programming. I do read the FAQ and other posts here as well as the user manual and I'm frequently overwhelmed by what I read (having ADD doesn't help). What may make sense to you all doesn't necessarily make sense to everyone. When a program has around 3,000,000 downloads, you have to expect there will be a large number if new users who will be asking about what may seem obvious to the power users here (I'm not knocking the power users here; they have frequently saved my bacon!).

[pthwaite]

I find the help files on the Calibre website not that helpful. THey assume a greater knowledge of some issues than a lot of the newer members may have. Certainly if you read through this forum a lot more information is gleaned. So, while I don't use this forum much, I do find it more helpful in resolving issues than the Calbre website on occasions.

I think your "start hear" would be a good idea.
Howard

[Alexander Turcic]

Any suggestions in terms of stickies and also guiding newbies to Calibre are always welcomed. If you have ideas, please share them with the team or contact me directly via PM.

One idea I had (although I am not sure if it's what's the best way to address the recent inflow of new members) is to add a new subcategory to Calibre where specifically people new to Calibre can share their thoughts and ask for help. Like I said, just an idea.

Cheers,
Alex

[beckywc]

I try to use the help files but the expressions are like greek to me. And I am honestly trying to understand. Plus there is so much it can be pretty overwhelming.

But I also understand that those that help us, help us out of their generosity and I'm am always grateful.

[nynaevelan]

I agree, I too do not find the help file/manual very helpful. There is alot of information in there based on alot of assumptions. One of those assumptions being the user is proficient in regular expressions and programming. It also does not fully explain the layout of Calibre or the functions of each area. For example, I was typing a post today and I wanted to give the correct terminology for the icons on the left hand side of the GUI, I looked in the manual and there is no explanation for the fields which I was looking for. It would be helpful if this document was written with the end user in mind, no matter what his level of expertise. Also as to this forum, even when a newbie asks for help, oftentimes she (myself) will be given an answer to go find this, or use regex for that, and once again assumptions are made that the user is proficient in regex or other programming skills. Oh,and the infamous search the internet for the regex documentation and fix it yourself. I do not know why Calibre was created and who the demographic audience was when it was created, but as you can see there are many users who have been advised that this is the best program to use to organize your e-library. Since it is publicly released, it would be nice if the preferred method of assistance is to use the documentation, then the documentation should be accommodating for all users.

Now with all of that stated, I am in no way complaining about the program, the users or the forum posters. I am just stating facts as I have experienced here on the forum.

Nyn

[kiwidude]

The issue I have with the existing Help documentation is that the "search" feature with it isn't particularly useful in my experience. In response you get a group of topic headings (the majority of which inevitable end up being techy Calibre API help which the average user does not care about) and have to click and pray within each that you drill down to something relevant.

I know there are some genuine gold nuggets within Calibre Help, and I don't mean to denigrate those who have contributed so hard to add tutorials etc. We completely understand with a fast moving program the docs will not be as up to date. But if you can't quickly find what you want you will end up coming to the forums (assuming you did look in the Help first at all of course). Out of date screenshots in the opening GUI section don't endear confidence about how frequently the help is maintained and given how fast new features appear in Calibre you are never quite sure if the answer in there is the "best one".

When I first got Calibre I'm sure I asked a couple of the same questions that I have since seen repeatedly made over and over. From memory there was one on author sorts and another for regexes for importing. I was fortunate to get some patient and helpful responses. I had used the search but as the threads at the time covered so many calibre versions and I thought my particular author sort issues were a bug I can understand why some people just want the "latest" version of an answer.

Plus the existing FAQ thread is not an FAQ, it's a train wreck. It is a "I will ask yet another question in here" thread. No-one is going to read through 80 pages of discussion in the hope they learn all they need to know. By all means I think it should be "demoted" from sticky status and your "start here" thread put in it's place.

Calibre is a HUGE program, like Griker I have dabbled as a developer around the code to a much lesser extent but there are massive areas I have not a clue about. As a Calibre user I don't want to have to wade through page after page of forum search results of varying relevance and up-to-dateness. It is hard enough reading through one thread at times, let alone multiple. What would be awesome is to have one "proper" FAQ thread as an authoritative up-to-date version maintained only by those who "know" and can link only to threads of relevance. Read-only to all but admins is imperative.

It isn't an easy task though as is stated for documentation. The quality of the indexing on the FAQ could significantly affect it's success imho. But how many Calibre users actually make it to the forums? And of those, how many read regularly the forum threads? A miniscule % I would suggest. I've been using Calibre maybe 4 months or so ago, yet I still understand very little about the recommended way of doing conversions. Or the wonders of catalogs. Or metadata plugboards. Or sharing as a website. Or news recipes. Or specific issues related to Calibre with a Kindle. I'm sure most users wouldn't have a clue about there being gui plugins. There are so many fundamental things like author sorts, regexes, duplicates, default settings you *will* want to change that I found took weeks at least of trial and error and hours of forum reading to glean.

Is the information here on the forum? Yeah, somewhere at some point in time it probably has been. You may have to pick one of those 80 page threads and hope that what is in the first post has been maintained with all I need to know though

It is an intimidating program after you scratch the surface to discover all it really does. The frustrating thing for me is that I am sure there are ways to be more effective with doing some of the basics - for instance to reduce the time spent in Sigil editing a book which may have been sorted automatically by a Calibre setting. However knowing "where to begin" on some of this stuff particularly when whole new features get added every week can be scary, even for geeks like me just from choosing to not spend every waking hour on trying to understand what Calibre does

[rhadin]

I'm somewhat in the position of Lady Fitzgerald. I'm not computer illiterate and I'm not illiterate in the software that I need to do my daily work (e.g., MS Word), but when it comes to programming and the expressions and detailed understanding, well I might as well be brainless.

I recently asked how to stop Calibre from creating collections on my Sony. There was nothing obvious that I could find anywhere (I did look). Even if my question had been previously answered on MR, I couldn't find the answer by searching. Perhaps I used the wrong search terms. I had no idea that I needed to edit the Sony plugin; this is not obvious to someone like me (truthfully, I had no idea there was a Sony plugin -- I have no idea how Calibre is structured underneath its veneer).

Calibre is a great program; I use few of its features. Largely I use it to upload to my 950 and 505 ebooks that I have bought from ebookstores other than the Sony ebookstore, something I could do with the Sony software but Calibre is easier. Except for moving books from my hard drive to my 950 or 505, I make no use of Calibre's other functions, and I expect that many users of Calibre do the same. We are the technically challenged.

FWIW, I do appreciate the aid that is given me and I express my thanks not only with a thank you post but by giving karma. But sometimes the manual is not enough, especially when it isn't written for dummies like me.

[user_none]

It is very frustrating writing documentation and having it ignored. Especially when, as Manichean noted, many questions can be answered by reading the short quickstart guide that is added to every new library.

I agree that a better FAQ thread would be a good idea.

As for the idea of having to know regexes, too bad. The features that use them require them. There is no way around it. If you don't know how to write a regex for that feature then don't use it. Sorry but many of the features like search and replace cannot be acomplished in any other way that would be simplier and cover as many use cases.

I do agree that some aspects can be made easier for new users. Such as having a repository for GUI plugins instead of a thread. The answer that X can be done with a GUI plugin can easily be taken the wrong way.

I see the quality around calibre support having gone down a bit and I think it's two fold. Current users not taking the time to think how an answer of "This has been answered use the search" is taken by a new user. Many of those new users had enough trouble just trying to register for a forum account and don't really know what to search for. I always try to think how my answer will be taken before posting and I always try to remain positive. It doesn't always happen but that's my fault when it does.

The other part of the issue really is new users feeling entitled. I was told I'm rude in a bug when I informed the poster that the output of FB2 conversion was valid and the issue was with their reader software. I said that if their reader didn't implement the FB2 spec they should look for another reader. The poster didn't like my answer. But frankly I can't fix something that isn't broken or an issue with another app.

[Manichean]

First off, this is nothing aimed at someone in particular, just a general assumption.
My feeling looking at many of the responses around here is that people expect a kind of hand-holding, full-service support desk. You have to remember that not only do the developers create Calibre in their free time voluntarily, but the same goes for the forum regulars voluntarily helping others in their free time. We get basically the same questions over and over and over, with maybe a few thrown in that are genuinely interesting. That is actually one reason I started creating the regex tutorial- so that we could just point people asking about regexes to said tutorial and tell 'em to come back if they still have questions after reading it.

You also have to understand that at least a few of us, as far as the mindset goes, come from the geek/hacker culture. We enjoy solving problems, but only interesting ones- the ones asking about things that are already solved get pointed to the canned solutions.
Also, we only can work on a problem if a user provides details. Often times, one really has to pry and nag before one has all the info needed to solve the problem. Other times, maybe you point people at one of the tutorials only to get the response that they'd already read it- maybe a general guideline as to how to ask for help, specifically what to include in a request, would be a Good Thing to have.
I've actually recently had the situation where I asked a few people having problems with the regex tutorial and I asked them what should be improved, only to get answers that boiled down to "dunno it's just too complicated lol" (I'm exaggerating here, but still...). You have to realize that the process of using things like the regex options or similar stuff is actualy aimed at what I'll just call "power users" for the sake of convenience. There's a limit to how far you can simplify an explanation on a technical matter. If you're not willing to invest the energy to understand that explanation or for whatever reason unable to understand it, maybe it just isn't for you yet. The Calibre manual cannot and should not be a manual to get from zero computer knowledge to knowing Python and regexes and all kinds of funky stuff- it is about how to use Calibre, and if you want to to some of the more advanced stuff, you should be willing to look for other information if you're missing some background you'd need.

[Manichean]

Originally Posted by user_none

I see the quality around calibre support having gone down a bit and I think it's two fold. Current users not taking the time to think how an answer of "This has been answered use the search" is taken by a new user. Many of those new users had enough trouble just trying to register for a forum account and don't really know what to search for. I always try to think how my answer will be taken before posting and I always try to remain positive. It doesn't always happen but that's my fault when it does.

I see your point. I'm actually guilty of that, I believe. Maybe the regulars should feel at least a little like staffing a hand-holding, full-service support desk

[cybmole]

when I arrived, as a noob, but with some programming background, I have no idea what a "regex" was & no great desire to find out - it looked like something I could gloss over ( I was wrong!" )

so a regular expressions for dummys intro sticky would be good. even calling it a regex is too advanced for beginners!

the key stickies should be read only, and I agree that the forum would benefit form some sub-forum structure - another program forum which I use a lot has sections for:
help & tech support
web site & documentation
feature requests
chat...

yet another ( VLCplayer) splits help & troubleshooting by O/s so windows issues, mac issues, ubuncto ( or whatever its called) issues are segregated.

for calibre maybe separate core PC software issues form device ( ereader ) connectivity , and from news subscrtiptions / emailing type issues

that said, many folks will post in the nearest section regardless...

[theducks]

1) I wish the newbies would take time to read the last months (or so) Headers to see if the subject may have already come up b4efore starting a NEW thread.

Calibre evolves fast.
2) could the Mods threads on obsolete versions and or hang a Obsolete Tag on it.

IMHO we have too many stickies already ( I was taught the rule of 6. Only show the user 6 choices or they tend to glance over them0)

Might I suggest more sub-forum. (Some that comes to mind)
Device connection Help
Importing Help
Converting to ... (might be a separate sections for problem high traffic formats)
Plug-In Use and Development

Personally, I prefer the "Cookbook" recipe type user manual.
So you want to:
Start with "C" and get "N" (section 22: Conversions-Comics )

Reference manuals are great for coding Syntax but can fail on the "when and Why" (but usually fail on the : use this one... )

my 2 cents (And I am probably guilty of sounding peevish after seeing 6 threads (headers) in 30 days on nearly the EXACT same question).

[Worldwalker]

I think we're looking at two separate problems here.

First, we've got the people who want to find answers, and are struggling with the documentation, the forums, etc. You'll find some of them posting in this thread, and some of them standing around the forum with glassy eyes and confused expressions. They want to find answers, but either they don't understand where to look, or the answers don't match up to their skill level.

Then, we've got the ones accurately described as entitled. Trolls aside, they certainly exist. I'll admit to a certain level of hostility toward these people because they are entitled and seemingly proud of it. I have a lot of respect for the calibre developers, and have had since the first day I used it. These people show no respect, and instead flounce in here expecting everyone to kowtow to them and run to do as they demand. I don't like people who show contempt where I show respect, especially when that contempt is the result of the world not working exactly the way they, as little princesses, want it to. I'm not nearly as polite to them as I am to someone who asks a legitimate question, and I don't think anyone should be expected to be. When someone does a great deal of work for your benefit, the correct answer is gratitude, not contempt, and civilized people know that. Uncivilized people need not be treated as though they're civilized; it's unfair to the civilized ones, and will make no impression on the uncivilized ones, as it's only what they expect anyway.

A lot of good ideas have been proposed here. I particularly agree with doing something with the FAQ thread, which just plain scares me. We need a clear collection of questions and answers -- and, because the answer to a question about calibre may not be the same today as it was a month ago, that collection needs to be regularly updated to replace old answers when they change (which, with the rate of development going on, they almost certainly will). That does mean we need a thread keeper for that one, and it's an effort-consuming job. We also need to compile a list of questions that such a thread could answer. This is where some of the newer users could be invaluable if they can come up with lists of all the "How do I...?" questions they needed to know when they first started.

One part of the problem is that calibre can do so many things, and its users use it for so many things, that no one person's questions, let alone answers, are sufficient. Some people use it to organize their ebooks. Some use it strictly for conversion. Some mostly need the content server. Some just use it to connect to their e-reader. Some use it for multiple functions, but I doubt that anyone, even Kovid, uses everything calibre can do, because it's gotten to where it can do so many things. So we're going to need multiple people, with multiple approaches, contributing questions and answers.

Another thing we should post, and it should also probably be both stickied and locked, is an explanation of how to ask a question.

Yes, you'd think that would be obvious, but apparently it isn't. You'd think that people would look at other questions and their answers, and post in ways likely to get results, but they don't. As a random example, over in the "promote your book" section, where book announcements generally provide the title of the book, sometimes the genre, maybe the available formats, DRM status, etc., one person posted such an announcement entitled "Hey!" I doubt if that sold a lot of books. We've seen plenty of them here in the calibre forum: they post with a title like "Help me!" (sometimes with far to many exclamation points) and a post that gives next to no information, and it's like pulling teeth to find out what their actual problem is. Some of them, yes, are the entitled jerks who think that we should all roll over and cater to them, not to mention divine their exact needs without them taking the time to tell us, but some of them are just utterly clueless. Microsoft and Eric Raymond have both written excellent explanations of how to ask an effective question. We might wish to write our own version specific to calibre, as one is rather MS-oriented and the other is long enough to need its own FAQ, but I think they'd be good places to start. (since I'm one of the primary complainers here, if we decide this needs to be done, I'll volunteer to write up how to ask better questions)

We should also divide our question and answer collection into categories, such as "getting it running", "using the server", etc. That will help people find their particular answers, and their particular level of answers. We need to have some basic information that would work for my mother as well as answers that would be necessary to people at much higher skill levels.

And this is starting to ramble and get out of hand ... down, you in the back snickering "starting???" ... so I'll end here.