Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: Would you benefit from a "Testing Weather Report"? What should it include?

  1. #1
    Join Date
    Mar 2005
    Beans
    6,040

    Would you benefit from a "Testing Weather Report"? What should it include?

    During the Karmic cycle I'm planning to develop a "Testing Weather Report" web application (inspired by its counterpart developed for the release team), which will display various bits of information on the state of the development branch.

    The information I plan it will aggregate would involve, but is not limited to:

    • Held back packages
    • Critical bugs in core packages (kernel, gcc, glibc, etc.) that may have high impact on a broad range of testers
    • Ongoing library transitions
    • Archive integrity
    • Daily image build status
    • Daily image health check (oversized or not)
    • ISO package diff (how up-to-date are the current ISO images with the archive?)
    • Upgrade situation ( possibly with automatic upgrade test results)
    • Current milestone, release notes / technical overview, next milestone name, how many days are left
    • Freeze status (freezes we are on and upcoming ones)
    • New packages in the archive
    • Bugs targeted for the next milestone
    • Calls for testing of specific packages (possibly not yet pushed to the development branch) by developers
    • Significant updates in components related to release goals
    • Archive mirror status
    • Build daemon status


    It's worth emphasizing that all of this information is already available to testers, but is scattered over various resources, and can be hard to evaluate altogether at a single time point. The goals are to:

    • come up with a list of most relevant metrics and other data that will help testers
    • display the data in a manner in which the most relevant elements can be evaluated at a single glance, on a single page, and the slightly less relevant can still be easy to reach
    • present it all on an attractive web page that will hopefully present a small bit of extra incentive for people to do testing for Ubuntu.


    The feedback I'm hoping to get would be on:

    • whether you'd find this useful at all and why
    • what kind of information you'd want aggregated by it, in addition to what I've listed above
    • what particularly useful ways of displaying the information would be (don't worry, I do have some ideas).


    I'll be holding a session tomorrow on the QA track at UDS-Karmic to lay down the initial plan and get preliminary feedback. Those interested can tune in to room #9 at 14:15 UTC tomorrow to follow up, and participate live via IRC (#ubuntu-devel-summit on Freenode) and Gobby (gobby.ubuntu.com - document name will be "karmic-qa-testing-weather-report"). I'll also keep you posted once I have some mockups and rudimentary code.
    Last edited by 23meg; May 27th, 2009 at 11:00 PM.

  2. #2
    Join Date
    Nov 2005
    Location
    Valencia, Spain
    Beans
    622
    Distro
    Hardy Heron (Ubuntu Development)

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    I definitely would!

  3. #3
    Join Date
    Jun 2008
    Location
    Manchester, UK
    Beans
    425
    Distro
    Ubuntu Development Release

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    Quote Originally Posted by Bou View Post
    I definitely would!
    +1
    Black holes are where God divided by zero

  4. #4
    Join Date
    May 2008
    Beans
    1,020

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    I thought it was some kind of a new weather forecast app that need testing.
    OB-untu. #!crunchbang

  5. #5
    Join Date
    Sep 2007
    Location
    South Dakota
    Beans
    48
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    I would think that having substantial discussions before-hand about how you propose to model this would be useful.

    Might I suggest that each of the metrics you're mentioning be modeled on their own (for trending- up/down) and that an aggregate model also be developed?

    I'm just a stats n00b, but know enough about logarithmic normalization and similar trending techniques that I *might* be able to contribute to something.

    Best
    -C

  6. #6
    Join Date
    Apr 2006
    Location
    New Hampshire, U.S.A.
    Beans
    2,528
    Distro
    Ubuntu Development Release

    Smile Re: Would you benefit from a "Testing Weather Report"? What should it include?

    Good idea!

    Some of the time we spend re-discovering the wheel, chasing bugs that are already defined. Yes, I try to find out what's known to be broken but I find the info scattered.

    I'd rather spend time on what's new/changed/fixed to get some early feedback to whoever may be interested.

    Jerry

  7. #7
    Join Date
    Mar 2005
    Beans
    6,040

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    Quote Originally Posted by cjsteele View Post
    I would think that having substantial discussions before-hand about how you propose to model this would be useful.

    Might I suggest that each of the metrics you're mentioning be modeled on their own (for trending- up/down) and that an aggregate model also be developed?

    I'm just a stats n00b, but know enough about logarithmic normalization and similar trending techniques that I *might* be able to contribute to something.

    Best
    -C
    I feel you're misinterpreting the goal. I'm not really planning this to do any statistical computation with the data, at least at this point; it will merely pull all the data together as they are at a given time point (of viewing the page) and display it all in a convenient and well-designed "dashboard".
    Last edited by 23meg; May 27th, 2009 at 05:04 PM.

  8. #8
    Join Date
    Aug 2008
    Location
    South East Montana
    Beans
    6,153

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    Yes, I believe that would be a great tool.

    One thing that it would do is to help those of us that are new at this to see were we should be looking and some idea what we should be doing.
    Dell 480 XPS 3G ram Quad Core 2.40GHz, Radeon HD 2400 PRO, Audigy1, 3x320G HDD, 320G External, Debian Testing for use, Debian Squeeze for secure use, Debian Sid for FUN

  9. #9
    Join Date
    Jun 2008
    Location
    Manchester, UK
    Beans
    425
    Distro
    Ubuntu Development Release

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    I don't know how hard it would be to set this up, but someone could create an automated twitter feed that tweets updates whenever the information you're aggregating changes.
    Black holes are where God divided by zero

  10. #10
    Join Date
    Nov 2006
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Would you benefit from a "Testing Weather Report"? What should it include?

    Yes, please. I think we would all benefit greatly from having all that testing information gathered in one place.
    Want a quality CD ripping program for Ubuntu? Check out my how-to: Installing Rubyripper on Ubuntu!

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •