Forum Effectiveness Survey

[gunksta]

I'm curious to see how effective this forum (Absolute Beginner's) is at solving the problems posted here. To do this I would like to look at a minimum of 250 threads and evaluate them on a number of criteria. I started working on this idea and thought I would get some feedback from the community before I get too far in to this. Here is the info I'm thinking about gathering at this point in time:

Thread URL - For QA and to make sure that I only measure/count a thread once.
Date - mm/dd/yy This registers when the thread is first screened.
Time - 24 Hour Time Rounded This registers when the thread is first screened.

Post Category - X-Server, Sound, etc. (I'm still working on the categories).
Posted On Date - Self Explanatory

Solved within 5 Days? - True or false. Was the problem solved within 5 days of first post?
Solved With Help From The Forum? - True / False
Solved With Outside Information - True / False
Solved For Themselves - True / False

Command Line Solution Suggested? - True / False
Command Line Suggestion Necessary? - True / False

I am looking for input from ALL participants of this forum to see if the proposed tool measures the stuff you think is important. What categories of questions should be on the list? (I'd like to keep it to ~6-12 categories) Is there anything else that should be measured?

At this point I am looking for suggestions, ideas, and some volunteers to help with this. The more people who help with this, the faster I'll be able to get a large number of threads assessed (more is always better). If you would like to volunteer, just post to this thread and I'll contact you (PM) when I'm ready to go.

At the risk of sounding like a jerk, I am looking for volunteers who usually answer threads in this forum (Absolute Beginner's) rather than people who ask questions in this forum. I want to make sure that the people who complete the evaluations can accurately assess a thread based on the criteria developed in this discussion. But, I am interested in the comments and ideas of everyone who reads this regardless of your level of experience with Linux.

Disclaimer: The results of this survey will (when completed) be posted to this forum. For transparency's sake I will post the actual dataset to this thread as a .csv and .ods file for others to examine.

In my day job I'm a policy analyst/researcher for a small consulting firm that specializes in human services. I do not work for Canonical or any IT magazine. I'm just a Kubuntu user who likes to answer interesting questions about technology.

[Oldsoldier2003]

I think a you'll find that the odds of getting a problems resolved on the forums are directly related to a couple of factors.

The first is the poster's ability to outline the issue clearly and provide the supporting data when asked. Clearly means without attitude and 'leetspeak'.

Secondly, posting in the correct forum makes a difference as well.

And of course, it also depends on just how tough the question is and if the resident guru's are online.


the easy questions get answered in minutes under most circumstances.

[gunksta]

One quick follow-up comment to my original post. If you would like to help me look at a few threads, I would like to get your feedback on HOW you would best like me to structure the data collection.

Should I send you a blank spreadsheet? Should I put together a simple web-based tool that you can answer. I want to make this comfortable for the widest number of people possible. If people want me to, I can send them a Kexi database or an OpenOffice.org database.

Right now I am leaning towards distributing a blank .ods file to people who want to assess forum threads but using the new forms front-end to Google's spreadsheet application is very tempting.

The only tool that I will NOT consider is using an Access database for data collection. IMHO a research project to assess the effectiveness of this forum should not be sullied by using proprietary software for data collection so please suggest web-based tools (I know these aren't usually open-source) or open source tools. I don't expect this to be a barrier to participation. :-)

[LaRoza]

Thread moved because it wasn't for support.

[cprofitt]

The issue I have with finding answers is that when I search for answers I often find conflicting suggestions which makes it difficult to know if any of them is correct.

[gunksta]

I think a you'll find that the odds of getting a problems resolved on the forums are directly related to a couple of factors.

The first is the poster's ability to outline the issue clearly and provide the supporting data when asked. Clearly means without attitude and 'leetspeak'.

Secondly, posting in the correct forum makes a difference as well.

And of course, it also depends on just how tough the question is and if the resident guru's are online.


the easy questions get answered in minutes under most circumstances.


Anecdotally, I agree with you 100%. I want to try and quantify this idea. When I try to turn friends, co-workers, etc. on to Ubuntu I am often asked about support. I know that the support offered in community forums is often very good (My bias assumes that the Ubuntu Forum is among the best). I think it could be useful and interesting to quantify exactly how effective this forum is and if there are any weaknesses that can be identified. I will confess that this has GEEK written all over it, but I think that's probably OK on a site dedicated to Linux.

[Bubba64]

Your idea is a noble one except that you don't have an independent variable as far as looking at it statistically, you also have no demographic information if you put together a strict dependent and independent set of variables and you measure the demographics with at least 1200 people responding you might have an answer that is representative of your basic set of questions. Unless your going to get credit in a statistical class or in some satisfying manner I wouldn't waste my time. In the end it is a totally subjective experience for each individual person with a set of possibilities which is not quantifiable, in other words to broad of a subject, good luck though we all need a hobby.

[aysiu]

Given the rate at which threads are created, I think 250 threads wouldn't be a large enough sample size to make a meaningful generalization about support quality. I think you'd have to examine an entire week's worth of threads.

From my own anecdotal perspective (having seen literally tens of thousands of threads over the almost three years I've been here), support is excellent (quick, efficient, effective, and friendly) for all easy support questions, good for most intermediate support questions, and lacking to non-existent for difficult/obscure support questions.

Even though I certainly advise users to post in the correct forum, use informative subject titles, and post useful details, I don't think these factors really matter in terms of how much help you get or the quality of help you get. If you look at the unanswered posts (something I and others check quite frequently), you'll see 99% of them are an obscure problem that no one knows the answer to. Very rarely do I see one that is an easy-to-answer unanswered thread that simply had a bad subject title.

[gunksta]

Your idea is a noble one except that you don't have an independent variable as far as looking at it statistically, you also have no demographic information if you put together a strict dependent and independent set of variables and you measure the demographics with at least 1200 people responding you might have an answer that is representative of your basic set of questions. Unless your going to get credit in a statistical class or in some satisfying manner I wouldn't waste my time. In the end it is a totally subjective experience for each individual person with a set of possibilities which is not quantifiable, in other words to broad of a subject, good luck though we all need a hobby.


You are correct, this structure doesn't allow one to gather any meaningful demographic information. I certainly won't be able to measure important things like age, gender, enthinicity, etc. On the other hand, that's not necessary to answer the original question.

How effective is the forum at solving problems?

Effectiveness is the independent variable. You don't need to know demographic details to determine the success or failure of a thread. All I'm really trying to do is measure the percent of help requests that result in a solution.

[gunksta]

Given the rate at which threads are created, I think 250 threads wouldn't be a large enough sample size to make a meaningful generalization about support quality. I think you'd have to examine an entire week's worth of threads.

From my own anecdotal perspective (having seen literally tens of thousands of threads over the almost three years I've been here), support is excellent (quick, efficient, effective, and friendly) for all easy support questions, good for most intermediate support questions, and lacking to non-existent for difficult/obscure support questions.

Even though I certainly advise users to post in the correct forum, use informative subject titles, and post useful details, I don't think these factors really matter in terms of how much help you get or the quality of help you get. If you look at the unanswered posts (something I and others check quite frequently), you'll see 99% of them are an obscure problem that no one knows the answer to. Very rarely do I see one that is an easy-to-answer unanswered thread that simply had a bad subject title.

I think somewhere in the ~250 range is easily enough to measure performance. 250 samples can easily be representative of a large sample, if the sampling is structured correctly. Equally important to the number of samples is when the sampling happens.

For example, the sampling must be distributed over a number of days / times. If every sample was taken at 10:00 PM EST on Saturday, the results are going to be skewed if I take 50 samples or 500. Saturday is not necessarily representative of the forum at all.

That's partly why I'd like to get a couple of random people to help me, so I can make sure that the sampling happens at different times.

I also intend to restrict the sample size to a single forum, rather than attempt to measure the effectiveness of all the forums on the site. I am only interested in looking at the Absolute Beginner's Forum, which will also help make sure that I can accurately answer the original question.