Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: So called "cadence testing" is a failure

  1. #1
    Join Date
    Apr 2008
    Beans
    10,346

    So called "cadence testing" is a failure

    Look at the standing bugs for just Lubuntu:

    Lubuntu_2013-03-23 18:21:53.jpg

    And the final Beta dates don't even match between "cadence" and the release schedule:

    https://wiki.ubuntu.com/QATeam/Cadence/Raring

    https://wiki.ubuntu.com/RaringRingtail/ReleaseSchedule

    We've been dealing with the same live installer breakage since mid November:

    https://bugs.launchpad.net/ubuntu/+s...y/+bug/1080701

    And now we have a 'network-manager' bug that effects both Ubuntu and Lubuntu for several days!

    https://bugs.launchpad.net/ubuntu/+s...r/+bug/1159201

    And the source bug is locked as private

    I'm sure the Captain still thinks he's steering the ship but the rudder is totally detached

  2. #2
    Join Date
    Jan 2013
    Beans
    1

    Re: So called "cadence testing" is an utter failure!

    While i don't like the idea of candence testing either, you've went too far.
    first, the bug list you had shows all bugs found in raring reported on the iso-tracker. for all *buntu's
    second, the freeze bug has been found and confirmed (making it no longer a problem caused by bad testing), but is very difficult to fix, if it's "our fault", please, show us how we could have fixed it.
    third, the network manager bug is private because it shows private info of some form.
    third, i'm shocked that you overlooked the areas even worse, like testdrive, the PPC recursive bug, and the lack of testing for netboot and ppc.

  3. #3
    Join Date
    Jan 2006
    Location
    Denmark
    Beans
    545
    Distro
    Ubuntu Development Release

    Re: So called "cadence testing" is a failure

    Quote Originally Posted by kansasnoob View Post
    I'm sure the Captain still thinks he's steering the ship but the rudder is totally detached
    +1
    I couldn't have said it better myself.
    I hope we don't have a shipwreck

  4. #4
    Join Date
    Mar 2006
    Beans
    4,373
    Distro
    Ubuntu Development Release

    Re: So called "cadence testing" is a failure

    When I think of the bugs that have persisted or constantly recurred over the many cycles that I have been doing Ubuntu testing it saddens me that we seem to be beating our heads against a stone wall .
    if it ain't broke you haven't tweaked it enough

  5. #5
    Join Date
    Jun 2010
    Beans
    699

    Re: So called "cadence testing" is a failure

    Not in general but i sometimes do see trouble spots where things change drastically between release. For example bluetooth support

    http://ubuntuforums.org/showthread.php?t=2121960

    I did think fixing bugs and making it work in Ubuntu 12.04 should stick for some time but i was wrong. But this are small things in general i am pleased and most things always work as i expect them to.

  6. #6
    Join Date
    Apr 2006
    Location
    USA
    Beans
    402

    Re: So called "cadence testing" is a failure

    Quote Originally Posted by kansasnoob View Post
    Look at the standing bugs for just Lubuntu:

    Lubuntu_2013-03-23 18:21:53.jpg

    And the final Beta dates don't even match between "cadence" and the release schedule:

    https://wiki.ubuntu.com/QATeam/Cadence/Raring

    https://wiki.ubuntu.com/RaringRingtail/ReleaseSchedule

    We've been dealing with the same live installer breakage since mid November:

    https://bugs.launchpad.net/ubuntu/+s...y/+bug/1080701

    And now we have a 'network-manager' bug that effects both Ubuntu and Lubuntu for several days!

    https://bugs.launchpad.net/ubuntu/+s...r/+bug/1159201

    And the source bug is locked as private

    I'm sure the Captain still thinks he's steering the ship but the rudder is totally detached
    It's a busy busy week as you know, but I'll address a couple of these points. We should discuss this more later when we all have a bit more time (post-cycle?).

    On the dates being incorrect on the calendar, there was an original note of april 4th for the actual release date for the beta, but it wasn't showing up. I think it was a separate line item even. Anyways, I made sure the note shows up now. That said, the cadence calendar is showing the saturday start dates, not actual event dates, whereas the release schedule is showing the exact important dates. The Final Beta release happens the week of March 30th from our cadence perspective, hence it's listed on that weeks' line -- I hope this makes sense. (EDIT: I should note here that the calendar needs to be easy to understand, so please do give more feedback on tweaks that should happen to make it so. We had an integrated calendar (it was part of the release schedule calendar) for testing last cycle, and it was very confusing to follow because so many events and timelines where trying to be convened on one page; I hope this is an improvement, but let's keep iterating!)

    In general we as testers actually like bugs.. Bug counts are funny. A low bug count during the cycle doesn't mean a stable release (it could mean we're not testing!), and a high bug count doesn't necessarily mean the release is horrible (maybe we're testing really well). Now, the caveat with bugs is while we like them, we don't like them in the stable release -- in development though, if we find a large amount of bugs, I would consider our job as successful.

    I feel you on the private and long-standing bugs, but we as a testing community have done our part on those by finding and documenting the bug. For the private bug, given the severity and longevity it would likely be worth asking someone to manually prune sensitive info from the report and open it, or ask the original reporter to merely publicize his info willingly if he is able to do so.

    All that said, what can we do about high bug counts and longstanding bugs or regressions? If we've reported them and documented them well, we've done our part in helping get the bug fixed. From the development side, we need those folks to also address and work the bugs found. I did a post last year about bug shepherding, and it's important. I know many of you practice this;

    you report the bug in detail, making sure it can be reproduced by others
    you follow-up on the bug to make sure it's properly filed and can be seen by those who can fix it
    you help test the fix if necessary

    Beyond that, the actual bug fix is in the hands of the developers. We can and should do everything we can to help make sure the bug is fixed, but the fixes themselves rely on a developer.
    Last edited by balloons; March 25th, 2013 at 07:17 PM. Reason: added calendar note

  7. #7
    Join Date
    May 2009
    Location
    North West England
    Beans
    2,674
    Distro
    Ubuntu Development Release

    Re: So called "cadence testing" is a failure

    Regarding https://bugs.launchpad.net/ubuntu/+s...r/+bug/1159201 This has now been un-privatised (or should that be nationalised?). If you come across bugs marked private, or your own bug suddenly goes private, please do ask on #ubuntu-bugs. If you're not a fan of IRC, then please email the quality mailing list ubuntu-quality@lists.ubuntu.com Yes, bugs that carry over are infuriating, I've been on the receiving end of one that prevented my 3G device working for two cycles before it got finally fixed.

    Don't give up hope! A lot of what has happened in cadence, has in itself been a test.

    Regards,

    Phill.

  8. #8
    Join Date
    Apr 2008
    Beans
    10,346

    Re: So called "cadence testing" is an utter failure!

    Quote Originally Posted by Noskcaj10 View Post
    While i don't like the idea of candence testing either, you've went too far.
    first, the bug list you had shows all bugs found in raring reported on the iso-tracker. for all *buntu's
    second, the freeze bug has been found and confirmed (making it no longer a problem caused by bad testing), but is very difficult to fix, if it's "our fault", please, show us how we could have fixed it.
    third, the network manager bug is private because it shows private info of some form.
    third, i'm shocked that you overlooked the areas even worse, like testdrive, the PPC recursive bug, and the lack of testing for netboot and ppc.
    You're right, it's per testcase rather than per flavor.

  9. #9
    Join Date
    Apr 2008
    Beans
    10,346

    Re: So called "cadence testing" is an utter failure!

    Quote Originally Posted by Noskcaj10 View Post
    While i don't like the idea of candence testing either, you've went too far.
    first, the bug list you had shows all bugs found in raring reported on the iso-tracker. for all *buntu's
    second, the freeze bug has been found and confirmed (making it no longer a problem caused by bad testing), but is very difficult to fix, if it's "our fault", please, show us how we could have fixed it.
    third, the network manager bug is private because it shows private info of some form.
    third, i'm shocked that you overlooked the areas even worse, like testdrive, the PPC recursive bug, and the lack of testing for netboot and ppc.
    I made a suggestion to narrow down the cause a long time ago:

    https://bugs.launchpad.net/ubuntu/+s...01/comments/48


    Super dumb question :^)

    Is there a way to temporarily revert these partman changes on the live image:

    ubiquity (2.13.2) raring; urgency=low

    * Try to copy signed kernel from vmlinuz.efi in preference to
    vmlinuz.efi.signed; vmlinuz.efi is friendlier to archaic 8.3 file name
    restrictions which apply to isolinux.
    * Automatic update of included source packages: partman-auto 105ubuntu1,
    partman-auto-lvm 46ubuntu1, partman-crypto 55ubuntu1, partman-lvm
    82ubuntu1.

    -- Colin Watson <cjwatson@ubuntu.com> Mon, 12 Nov 2012 15:11:08 +0000

    I'm asking because I archive some testing discs and as best I can tell that's about when things broke. I'm thinking a temporary reversion would be acceptable just for testing purposes since we're not doing alphas in Raring.

    Sorry to be a pain in the neck.
    Of course it's now far too late to think about that because we've been building on top of an unstable foundation for 4 months

  10. #10
    Join Date
    Apr 2008
    Beans
    10,346

    Re: So called "cadence testing" is an utter failure!

    Quote Originally Posted by Noskcaj10 View Post
    While i don't like the idea of candence testing either, you've went too far.
    first, the bug list you had shows all bugs found in raring reported on the iso-tracker. for all *buntu's
    second, the freeze bug has been found and confirmed (making it no longer a problem caused by bad testing), but is very difficult to fix, if it's "our fault", please, show us how we could have fixed it.
    third, the network manager bug is private because it shows private info of some form.
    third, i'm shocked that you overlooked the areas even worse, like testdrive, the PPC recursive bug, and the lack of testing for netboot and ppc.
    Eerm, maybe. I've reported bugs with loads of personal info and I was asked whether or not to allow that info to be posted while actually using "apport". But if a bug is reported using "http://bugs.launchpad.net/ubuntu/+source/PACKAGENAME/+filebug?no-redirect" the bug is frequently marked private with no outstanding warning.

    In fact I need to check one against the netboot mini.iso right now

    Edit: One more thing, maybe the bug-bot could be reprogrammed to never mark a bug report as a duplicate of a private report. Maybe it could just add a note to the private report about potential dupes.
    Last edited by kansasnoob; March 29th, 2013 at 04:36 PM.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •