Objective testing

[cyberjar09]

First off, im not entirely sure if this thread belongs here. If it doe snot please let me know so that i may move it.

I am looking for an objective way of testing the difference in usability/productivity levels of people when using Ubuntu versus windows 7.

Are there some tests that maybe i should conduct say, ask people to burn a CD from start to finish or change desktop wallpaper / resolution or maybe downloading and installing a bunch of applications.

im open to ideas here. I could use all the help i can get.

Thanks.

[d3v1150m471c]

you may want to setup a free webpage or a blog and present approximately 10 tasks or that they could complete on both operating systems. i'd also request a small summary from their personal experiences with both machines. definitely ask how long they have been using both systems. i say this because someone who has used ubuntu their whole life may have trouble with windows only for that reason and vice versa. once you get enough submissions then turn it into some useful statistical data. in my opinion the information would probably be very useful to the devs and the users here.

btw i think this is a great idea. if you need any help let me know.

[cyberjar09]

you may want to setup a free webpage or a blog and present approximately 10 tasks or that they could complete on both operating systems. i'd also request a small summary from their personal experiences with both machines. definitely ask how long they have been using both systems. i say this because someone who has used ubuntu their whole life may have trouble with windows only for that reason and vice versa. once you get enough submissions then turn it into some useful statistical data. in my opinion the information would probably be very useful to the devs and the users here.

btw i think this is a great idea. if you need any help let me know.

What kind of tasks do you think i should ask them to attempt?

Also, the problem i would face is that if the users have a bias, they will not be adequate test subjects. Im looking to get the common man to attempt these tasks, not a linux expert. Finding users who have never seen the windows environment is next to impossible in my opinion.
:confused::confused::confused:

[d3v1150m471c]

give me a while and i will come up with a list of things and some important questions to go along with them.

[cyberjar09]

give me a while and i will come up with a list of things and some important questions to go along with them.

wow..! true testament to the support you get with the ubuntu forum ...

thanks a ton.

Here are maybe a few sample tasks that I think could be included :

Download and install *a linux distro* in virtualbox
Burn a Dual Layer DVD using nothign but the tools provided with the OS
Set up File sharing with users on your lan
.....


Questions could be based on each task
How long did the task take?
rate on a scale of 1 to 10 the ease of completion of the task
.....

[cyberjar09]

Please excuse me if i dont reply here for the next 8 hours or so ... im in India so im guessing our time zones are totally reversed. Ill check back in the morning .

[d3v1150m471c]

Here's what I had in mind. You may want to add to it. Let me know if i can be of any more assistance.

Requirements:
Those volunteering for the research must have an installed copy
of both Ubuntu 9.10 (or a higher version) and either Windows
Vista or Windows 7. Neither operating system can be tested from
a virtual machine as this may affect the performance of the
guest operating system.



Background Information:

What version of windows are you testing?

What version of Ubuntu are you testing?

How long have you been using Ubuntu?

How long have you been using Windows?

Which OS did you begin using first?


Objectives (Please provide a short explanation for your experiences in each of the provided tasks):

1. When you installed Ubuntu/Windows, which one did you find easier to
install?

2. Which OS provided the best default programs post installation?

3. Which of the OS's provided the best default web browser based on
functionality, security, and user-friendliness?

4. Which OS did you find better to customize based on how easy it was
to customize, the amount of visual and audible appeal, and the
variety of customization available for the OS.

5. Which OS did you find it easier to add new programs based on
how easy it was to find the software, how easy it was to install.

6. Which OS provided a better networking experience in terms of
security, setting up a LAN (wired and wireless), and functionality.

7. Which OS is a better base for productivity, whether it be programming,
writing, graphical design, etc. ( It can be a combination of
these if you do more than one. Feel free to add your own.)

8. Which OS provided better file and folder management based on copying
and moving files, finding your files, adding and deleting files.

9. Which OS was easier to maintain based on scanning/removing viruses and malware,
defragmenting and cleaning up disk space, and backing up your information.

10. Which OS do you find easier to troubleshoot based on man/info/help pages,
working out issues on your own, and online help like forums and online
documentation.

11. Which OS provided the most stability and performance based on cpu usage,
peripheral hardware compatibility (IE: mouse, tablet, webcam), software
and OS freezing, crashing, locking-up, and hardware compatibility.

[ShakeyJake]

I'd guess that using users that are familiar with each system is the best way. As you've said, finding someone unfamiliar with Windows is next to impossible.

And because Linux is very configurable, that may actually count agaisnt the linux users in this test. Presenting someone with the default system will lead to some people being just as lost as if the were on a different OS. I, for instance, have no idea how to use pitivi, tomboy, or many of the things with Ubuntu as standard because they get removed straight away and replaced.

[d3v1150m471c]

I'd guess that using users that are familiar with each system is the best way.

As far as weeding out a bias that would help. Though allowing everyone with both platforms to participate you'll find more information and participants from various
scenarios. Otherwise, I concur, which is why I suggested a small amount of pertinent
background information be provided.

[koenn]

First off, im not entirely sure if this thread belongs here. If it doe snot please let me know so that i may move it.

I am looking for an objective way of testing the difference in usability/productivity levels of people when using Ubuntu versus windows 7.

Are there some tests that maybe i should conduct say, ask people to burn a CD from start to finish or change desktop wallpaper / resolution or maybe downloading and installing a bunch of applications.

im open to ideas here. I could use all the help i can get.

Thanks.
Since you want your test to be objective,
you'd first have to define what it is you want to test.
You say usability/productivity levels. Define usability. Define productivity. Then find ways to quantify them, and then try to find tests.
Note that usability and productivity are probably not the same (unless you can proof that they are related), so measuring usability does not necessarily tell you something about productivity.

Also, you want to test productivity in what kind of people ? first time users ? people who've had formal training in the software at hand ? People who usually use the other operating system ? people who regularly use both the operating systems ? 6 year old children ?

[cyberjar09]

I'd guess that using users that are familiar with each system is the best way. As you've said, finding someone unfamiliar with Windows is next to impossible.

And because Linux is very configurable, that may actually count agaisnt the linux users in this test. Presenting someone with the default system will lead to some people being just as lost as if the were on a different OS. I, for instance, have no idea how to use pitivi, tomboy, or many of the things with Ubuntu as standard because they get removed straight away and replaced.

Thanks for the suggestion.

You are right, in cases like your's where users' are comfortable with certain software in their operating system and know how to install them and make those their defaults, it wold not really matter... using the 'default' is a subjective thing.

[cyberjar09]

Since you want your test to be objective,
you'd first have to define what it is you want to test.
You say usability/productivity levels. Define usability. Define productivity. Then find ways to quantify them, and then try to find tests.
Note that usability and productivity are probably not the same (unless you can proof that they are related), so measuring usability does not necessarily tell you something about productivity.

Also, you want to test productivity in what kind of people ? first time users ? people who've had formal training in the software at hand ? People who usually use the other operating system ? people who regularly use both the operating systems ? 6 year old children ?

yes you have brought a lot of valid points.

In order to quantify the results I would need something to compare them to. So Probably a survery for 2 test groups, one for users of both ubuntu and windows and another for first time users of both os's ... but there lies the catch.

Where will i find first time windows users?

[cyberjar09]

Also, would be better to have seperate tests for both test groups.?

beginners will get tasks like
create a document
burn a dvd
print a document
connect to chat client
...

advanced users should probably have that and more to do ...


maybe also calculate how long the tasks take and what they had to do to acheive the task.

[koenn]

In order to quantify the results I would need something to compare them to. So Probably a survery for 2 test groups, one for users of both ubuntu and windows and another for first time users of both os's ... but there lies the catch.

Where will i find first time windows users?

simple, find someone who never used windows before. You may have to look in the under 6 or over 65 yo age groups, though.

What's the point of testing 'productivity' in first time users, anyway ?

[cyberjar09]

simple, find someone who never used windows before. You may have to look in the under 6 or over 65 yo age groups, though.

What's the point of testing 'productivity' in first time users, anyway ?


maybe hoping that people find using ubuntu as primary and windows as a secondary is actually beneficial... windows is hard to eliminate completely, with games and design software on it.

[koenn]

with bias like that, it's going to be hard to keep your testing objective.

But what I meant is, if your test is to show something about productivity, a first time user is the wrong choice. A trained or experienced user would be a better test subject. "first time user' is a passing stage.

[d3v1150m471c]

So what's the word?

[cyberjar09]

im looking into it... Will post back with my findings ... hopefully it would be decipherable..