Confused about cadence testing cycles

[kansasnoob]

* elfy still doesn't understand all that cadence stuff 
<balloons> :-( I'm sorry elfy
<elfy> good job I'll be testing xubuntu 
<elfy> :)
<balloons> Well, what are you still missing?
<elfy> oh d'oh just realised I'll be doing arm as well ... lol
* cwayne (~cwayne@pool-98-118-37-160.bstnma.fios.verizon.net) has joined #ubuntu-quality
<elfy> balloons: I suspect what I am missing is the more normal term people would use :)
<balloons> elfy, sure sure
<balloons> basically we're going to have test-a-thons if you will every couple weeks
<balloons> each time we'll pick somethng else to focus on
<balloons> we'll pull the tests from the tracker and make a milestone for them and have people report on them
<elfy> ok - that makes sense :)
<balloons> so for next week I've got to pull the tests and make the milestone and announce i
<elfy> we'll not be doing that
<balloons> it's that simple
<balloons> everyone will have saturday - saturday to test and report
<elfy> well kind of we will - when we have something we need specifically testing then we'll be shouting 
<elfy> right - and then the 'other' week is not cadence testing, just normal dailies I assume 
* elfy likes the idea of that with arm 
<elfy> ty balloons :)
<balloons> the other week is an off week
<balloons> I'll be doing things like maintaining and updating tests, playing with bugs, etc, etc
<elfy> but dailies will be updated daily still?
<balloons> writing new tests
<balloons> yes dailies will update, and no one is stopping your from running them if you wish :-)
<elfy> okey doke
<balloons> our focus is just much wider than the iso's now
<elfy> yea - as it perhaps should be 
<balloons> <3
* alourie_lunch has quit (Remote host closed the connection)
<elfy> so it in effect gives people longer than a day to test - have the whole of a week to do so in 
<elfy> is that correct?
<balloons> yes, exactly
<balloons> flexibility
<balloons> we're spread out and it's so much easier for you to plan to help out at some point in 7 days, rather than 7 hours
<elfy> k - I will post that on the forum, bit of confusion I think 
* balloons hasn't been to the forum
<balloons> I've been a bit busy
<balloons> but I wanted to hop over there today
<elfy> balloons: has missed this then Confused about cadence testing cycles 
<balloons> gotcha..
<balloons> well let's try and clear everything up then
<balloons> I think it should be really understandable after our first (I hope)
<elfy> I'll just post this over there :)

I was a bit confused too - hope this helps a bit, bear in mind when I'm talking of we I mean Xubuntu

As far as I can see - it means you have a week to test in rather than a day.

I'll be testing Lubuntu here, not because I hate Unity, but I maintain over 20 PC's with P4M800 graphics that simply won't run either Compiz or Mutter in an acceptable manner

But I've been iso/upgrade testing since Hardy or Intrepid and I really don't get this change

When I started we'd typically have 3 or 4 days (generally about Monday thru Thursday) to test before each point-release and you could depend on the devs to fix any critical bugs, or at least set a target to have the bug fixed.

And the alphas were typically about 4 weeks apart, while the betas were closer together, and we'd generally even have an RC about a week before final. IMHO things have continued to go downhill ever since they dropped the RC.

Anyway it now sounds to me like no images will be frozen and they want testers to test each test case every day, seven days straight, every other week

Before I retired I worked in manufacturing engineering and I worked closely with the QA team to achieve ISO 9000 certification and I very strongly feel that this newest plan is an excellent example of letting the inmates run the asylum.

Or maybe Canonical wants to move all testing to "internal automated VM" which will mean even less testing on actual hardware ........... I don't know

I hope I'm wrong, maybe this will work out great

[kansasnoob]

OK, this notification clears up a lot of my confusion:

Ok, so we're a bit delayed in starting this because, well, there was a Holiday in North America called Thanksgiving. Forgive me. :-p

So, this week (through Saturday) is our first cadence week. What that means is we will be focusing testing specific packages (or isos) during the week. What you can do to help is to run through a testcase (or a few ) at some point during the next week and report your results. The goal here is NOT to make all the tests green everyday. Instead, we want to test on a deeper level and really look for bugs in the software (or even the testcases ) so we can be confident in the health of the package(s) we are testing. To that end, there is no daily target -- rather we want to test through Saturday (whenever that is in your localtime ). Think quality, not quantity (I know, it's a bad pun.. quality quality...)

What this means practically is during the next week, try to execute some testcases for our week 1 targets (which are libreoffice, unity autopilot tests and the daily raring iso). You can find the wiki page detailing this here:

https://wiki.ubuntu.com/QATeam/Cadence/Raring/Week1

The master schedule can be found here:

https://wiki.ubuntu.com/QATeam/Cadence/Raring

Again, I would suggest finding a day and time that works for you and adding your results. Feel free to use IRC (#ubuntu-quality on freenode) or this mailing list to discuss any bugs you may find. At the end of the week, I'll recap what we did (bugs, tests ran, etc), and we can evaluate the results to see if further followup and focus is needed or not. Now, our testing focus's each week will change in order to target packages that need more QA work, or new packages that recently landed in QA.

Now, this week I will also be posting the next bits in our autopilot series -- intending to teach you about running and writing autopilot testcases. Our cadence testing this cycle will include running both automated and manual tests -- we need both You'll note the autopilot unity testsuite is included this week. I hope our subsequent testing weeks can feature some new automated autopilot tests written by some of you If writing automated tests aren't up your alley, don't worry we need manual testcases too!

Happy Testing everyone,

Nicholas

[ventrical]

I got time to check into that this week. Not sure why Nick did not post that here.

[grahammechanical]

I am trying to get in on this.

Just downloaded the 20121126 ISO image and during the second purple screen I do not get all the usual disk activity as the hardware is being detected. I then get a black screen with the error message "unable to find medium containing a live file system."

As regards those Autopilot tests beware. There are a lot of them. Note what is said by Nick on 20th November on this page:

http://qa.ubuntu.com/

As of this writing, I get 461 tests returned. Feel free to try and run them. Pick one from the list and see what happens. For example,

autopilot run unity.tests.test_dash.DashRevealTests.test_alt_f4_ close_dash

Just make sure you run them in a guest session -- I don't want anyone's default desktop to get hammered by the tests!

If you are feeling adventurous, you can actually run all the unity testcases like this (this will take a LONG TIME!).

These tests are also supposed to be run on 12.10 or 13.04 and not 12.04.

Regards.

[ventrical]

I am trying to get in on this.

Just downloaded the 20121126 ISO image and during the second purple screen I do not get all the usual disk activity as the hardware is being detected. I then get a black screen with the error message "unable to find medium containing a live file system."

As regards those Autopilot tests beware. There are a lot of them. Note what is said by Nick on 20th November on this page:

http://qa.ubuntu.com/



These tests are also supposed to be run on 12.10 or 13.04 and not 12.04.

Regards.

Ahhhhhhhhrrggg.... I just logged on and ran the Unity AutoPilot .. and it just keeps on going .. also on one of my main *installs*! I was just going to start a new thread about 'no heads up" but .. well .. here it is !

So far , so good. Nice work on the Unity auto Pilot test. Behaves like a remote access.

I don't understand the code comming from terminal though .. ssssss...FF..ssss EE FE .. etc... so I will just let the test run it's course. It it foobars my desktop, well.. then thats just the way it is.

Umm.. thanks for the *heads up* Graham.

[kansasnoob]

My "normal" test box won't run Unity since they dropped Metacity/Unity-2D in favor of "llvmpipe+compiz" so I guess my testing is simple

All I need to do is test the installation of the available Lubuntu images and report pass or fail. Of course I'll have to file a bug report to mark a test failed but we'll see how this goes.

I should have some time to test an Lubuntu i386 iso this evening.

[balloons]

Glad you guys are enjoying things! I hope all is clear now -- it's not gloom and doom at all! I hope the autopilot tests are the first of many..

If you run all of the unity autopilot tests, yes it will take a long time. I think I'm going to split up the tests and have people run them a bit differently with some heads-up. This is a learning process on this end too -- need to figure out the best way to run and report (these are a bit different than the manual tests), but we'll figure it out. Enjoy the testing guys!

[ventrical]

This is a learning process on this end too -- need to figure out the best way to run and report (these are a bit different than the manual tests), but we'll figure it out. Enjoy the testing guys!

Whew..!. Good to know that we are not alone !

[Elfy]

Be good to look at how other flavours can adapt these tests to suit their needs.

No unity here

[ventrical]

Be good to look at how other flavours can adapt these tests to suit their needs.

No unity here

There is lots of testing needed here also : http://iso.qa.ubuntu.com/qatracker/m...nes/243/builds

..all flavours.. looks like a gazzilion things to keep us busy

I guess it would have to be up to the devs of any particular flavour whether or not they are going to write an auto-pilot test for those DEs.