Maemo QA process

Hello,

I like to share my thoughts on the QA strategy of Maemo.

I am the owner of an N900 program and promoted my package to
extras-testing ca. 2 months ago. Right now there are only 2 votes on it.
I wonder how many more months it will take until 10 persons vote for it
such that I can promote it to extras (given that the votes are positive).

I already have a bunch of improvements ready since several weeks.
However, I do not dare to upload them because my package will start the
evaluation sequence then again from scratch. I think this is a big
disadvantage for the community, because the tool could already be much
better. Actually, I think in the early lifetime of a SW project there
will probably be a release every few months, which could permanently
block the tool's promotion to extras.

I personally see my program as very useful - well, otherwise I wouldn't
have spent the time for development. I assume that just nobody is aware
of it, since everybody just looks at the available programs in extras.
At least this is what I did to search for useful tools.

>From my perspective, the current QA process should be reconsidered.
Maybe an automatic promotion after 4 weeks would make sense, given its
actual package karma is positive?

Best regards,
Roman

P.S.: You are welcome to evaluate and vote on my package:
http://maemo.org/packages/package_instance/view/fremantle_extras-testing_free_armel/babyphone/0.1-2/

On Wed, Jul 21, 2010 at 3:53 PM, Roman Morawek <lists@morawek.at> wrote:
>
> I am the owner of an N900 program and promoted my package to
> extras-testing ca. 2 months ago. Right now there are only 2 votes on it.
> I wonder how many more months it will take until 10 persons vote for it
> such that I can promote it to extras (given that the votes are positive).
>

Don't feel bad, mine is at 4 months.


> I already have a bunch of improvements ready since several weeks.
> However, I do not dare to upload them because my package will start the
> evaluation sequence then again from scratch. I think this is a big
> disadvantage for the community, because the tool could already be much
> better.
>

Been there, done that, might as well upload it. It's annoying but in the end
the new features will take the same number of votes whether you do it now or
later.


> I personally see my program as very useful - well, otherwise I wouldn't
> have spent the time for development. I assume that just nobody is aware
> of it, since everybody just looks at the available programs in extras.
> At least this is what I did to search for useful tools.
>

Did you post on talk.maemo.org? That will get a few eyeballs on it. Be sure
to post screen shots. Also, get it into the weekly news letter that goes
out. It has a section for applications.


> >From my perspective, the current QA process should be reconsidered.
> Maybe an automatic promotion after 4 weeks would make sense, given its
> actual package karma is positive?
>

I would be OK with this as long as there were a way to have a package
removed in the event that something was found when it did reach a wider
audience. Allow thumbs down on extras but require a bug report and give the
developer a way to respond.

Frank

Hi,

On 21 July 2010 22:53, Roman Morawek <lists@morawek.at> wrote:
> Hello,
>
> I like to share my thoughts on the QA strategy of Maemo.
>
> I am the owner of an N900 program and promoted my package to
> extras-testing ca. 2 months ago. Right now there are only 2 votes on it.
> I wonder how many more months it will take until 10 persons vote for it
> such that I can promote it to extras (given that the votes are positive).

I've promoted a package to extras-testing too and in 3 days I've 0
votes. Other packages are having the same problem and I think that
this is a real fail for Maemo users.
I agree that we must ensure some quality to application distributed to
end users, but this process must be improved as soon as possible
because it doesn't work well.

How to improve it?

1) Automatic promotion to maemo-extras if no one send a negative
feedback within 10 days.
2) Immediatly promoted after 5 positive feedback.

I think we should discuss this as soon as possible.

Regards,

--
Andrea Grandi
email: a.grandi [AT] gmail [DOT] com
website: http://www.andreagrandi.it
PGP Key: http://www.andreagrandi.it/pgp_key.asc

On Wednesday 21 July 2010 22:53:43 Roman Morawek wrote:
>P.S.: You are welcome to evaluate and vote on my package:

Indeed, the best strategy for developers is to lobby a bit for their
application. Open a thread on talk.maemo.org . Blog about it. Write to the
list. I hope the fellow devs won't take this the wrong way, but... this aspect
is actually a feature, not a bug, as it encourages discussion and
interaction/community building, not the fully disconnected 'dump in extras'
and 'download moar apps' approach. Obviously being delayed for months is very
bad, so we're constantly exploring ways to improve the process, but...

> Maybe an automatic promotion after 4 weeks would make sense, given its
> actual package karma is positive?

...the problem with automatic promotion is that then we don't really have any
guarantee that what gets to stable has actually been tested - and the testing
should be at least as much about protecting users from obvious faults as
providing feedback to developers. If we enable automatic promotion, that
actually might discourage testing due to the 'why bother, its going to get to
Extras anyway' factor.

Note that since not long ago we also have super-testers, people with proven
track records, it's enough to get three of their votes if your package is
stuck and you're good to go.

>I personally see my program as very useful - well, otherwise I wouldn't
>have spent the time for development. I assume that just nobody is aware
>of it, since everybody just looks at the available programs in extras.

Even now the problem is not that people are not finding the app, the download
number tells apps in testing have thousands of downloads - it's the feedback
that is missing, IMO for two reasons: one, users not accustomed to give
feedback, and two, the unavailability to give feedback EASILY. This is being
worked on, but progress is slow, sadly. Ideas (but preferably hands) are
welcome on improving the process.

PS. And the part I dont understand: we have 25+ apps that are unlocked and not
being promoted by their respective authors. Orphans ?

Risking saying the obvious: I think if the voting happened on the device
(and not on the web site after you have to find the package, login and
figure out where to vote) we would get many more people voting.

Also, I think it would be really beneficial to make easier for users to
decide to download and install an application from testing and I can't think
of a better way than having something like the "download assistant". Having
a tool to browse the available applications, see comments and average rate
would be a huge incentive for me to download an application... and if it was
easy enough to just go back to the same application and vote I sure would.

I added a message to the "download assistant" thread but never received a
reply. http://talk.maemo.org/showpost.php?p=717824&postcount=17

Maybe someone knows the maintainer or what is behind the scenes and can find
out if it is a viable solution or not.

I would suggest the exact same application but working with extras-testing.
If is so easy that we can wish for more I would like to see a "tab" with my
"recent downloaded/installed" applications so I could easy find and vote on
them after testing.

Felipe

-----Original Message-----
From: maemo-developers-bounces@maemo.org
[mailto:maemo-developers-bounces@maemo.org] On Behalf Of Attila Csipa
Sent: Wednesday, July 21, 2010 5:56 PM
To: maemo-developers@maemo.org
Subject: Re: Maemo QA process

On Wednesday 21 July 2010 22:53:43 Roman Morawek wrote:
>P.S.: You are welcome to evaluate and vote on my package:

Indeed, the best strategy for developers is to lobby a bit for their
application. Open a thread on talk.maemo.org . Blog about it. Write to the
list. I hope the fellow devs won't take this the wrong way, but... this
aspect
is actually a feature, not a bug, as it encourages discussion and
interaction/community building, not the fully disconnected 'dump in extras'
and 'download moar apps' approach. Obviously being delayed for months is
very
bad, so we're constantly exploring ways to improve the process, but...

> Maybe an automatic promotion after 4 weeks would make sense, given its
> actual package karma is positive?

...the problem with automatic promotion is that then we don't really have
any
guarantee that what gets to stable has actually been tested - and the
testing
should be at least as much about protecting users from obvious faults as
providing feedback to developers. If we enable automatic promotion, that
actually might discourage testing due to the 'why bother, its going to get
to
Extras anyway' factor.

Note that since not long ago we also have super-testers, people with proven
track records, it's enough to get three of their votes if your package is
stuck and you're good to go.

>I personally see my program as very useful - well, otherwise I wouldn't
>have spent the time for development. I assume that just nobody is aware
>of it, since everybody just looks at the available programs in extras.

Even now the problem is not that people are not finding the app, the
download
number tells apps in testing have thousands of downloads - it's the feedback

that is missing, IMO for two reasons: one, users not accustomed to give
feedback, and two, the unavailability to give feedback EASILY. This is being

worked on, but progress is slow, sadly. Ideas (but preferably hands) are
welcome on improving the process.

PS. And the part I dont understand: we have 25+ apps that are unlocked and
not
being promoted by their respective authors. Orphans ?

I bet if you developed an app like that, you'd get a lot of positive votes from developers that are frustrated that they can't get their packages voted on. :)

Necessity is the mother of invention, no?

---- Felipe Crochik <felipe@crochik.com> wrote:

=============
Risking saying the obvious: I think if the voting happened on the device
(and not on the web site after you have to find the package, login and
figure out where to vote) we would get many more people voting.

Also, I think it would be really beneficial to make easier for users to
decide to download and install an application from testing and I can't think
of a better way than having something like the "download assistant". Having
a tool to browse the available applications, see comments and average rate
would be a huge incentive for me to download an application... and if it was
easy enough to just go back to the same application and vote I sure would.

I added a message to the "download assistant" thread but never received a
reply. http://talk.maemo.org/showpost.php?p=717824&postcount=17

Maybe someone knows the maintainer or what is behind the scenes and can find
out if it is a viable solution or not.

I would suggest the exact same application but working with extras-testing.
If is so easy that we can wish for more I would like to see a "tab" with my
"recent downloaded/installed" applications so I could easy find and vote on
them after testing.

Felipe

-----Original Message-----
From: maemo-developers-bounces@maemo.org
[mailto:maemo-developers-bounces@maemo.org] On Behalf Of Attila Csipa
Sent: Wednesday, July 21, 2010 5:56 PM
To: maemo-developers@maemo.org
Subject: Re: Maemo QA process

On Wednesday 21 July 2010 22:53:43 Roman Morawek wrote:
>P.S.: You are welcome to evaluate and vote on my package:

Indeed, the best strategy for developers is to lobby a bit for their
application. Open a thread on talk.maemo.org . Blog about it. Write to the
list. I hope the fellow devs won't take this the wrong way, but... this
aspect
is actually a feature, not a bug, as it encourages discussion and
interaction/community building, not the fully disconnected 'dump in extras'
and 'download moar apps' approach. Obviously being delayed for months is
very
bad, so we're constantly exploring ways to improve the process, but...

> Maybe an automatic promotion after 4 weeks would make sense, given its
> actual package karma is positive?

...the problem with automatic promotion is that then we don't really have
any
guarantee that what gets to stable has actually been tested - and the
testing
should be at least as much about protecting users from obvious faults as
providing feedback to developers. If we enable automatic promotion, that
actually might discourage testing due to the 'why bother, its going to get
to
Extras anyway' factor.

Note that since not long ago we also have super-testers, people with proven
track records, it's enough to get three of their votes if your package is
stuck and you're good to go.

>I personally see my program as very useful - well, otherwise I wouldn't
>have spent the time for development. I assume that just nobody is aware
>of it, since everybody just looks at the available programs in extras.

Even now the problem is not that people are not finding the app, the
download
number tells apps in testing have thousands of downloads - it's the feedback

that is missing, IMO for two reasons: one, users not accustomed to give
feedback, and two, the unavailability to give feedback EASILY. This is being

worked on, but progress is slow, sadly. Ideas (but preferably hands) are
welcome on improving the process.

PS. And the part I dont understand: we have 25+ apps that are unlocked and
not
being promoted by their respective authors. Orphans ?

Attila Csipa wrote:
> On Wednesday 21 July 2010 22:53:43 Roman Morawek wrote:
>> P.S.: You are welcome to evaluate and vote on my package:
>
> Indeed, the best strategy for developers is to lobby a bit for their
> application. Open a thread on talk.maemo.org . Blog about it. Write to the
> list. I hope the fellow devs won't take this the wrong way, but... this aspect
> is actually a feature, not a bug, as it encourages discussion and
> interaction/community building, not the fully disconnected 'dump in extras'
> and 'download moar apps' approach. Obviously being delayed for months is very
> bad, so we're constantly exploring ways to improve the process, but...
>
>> Maybe an automatic promotion after 4 weeks would make sense, given its
>> actual package karma is positive?
>
> ...the problem with automatic promotion is that then we don't really have any
> guarantee that what gets to stable has actually been tested - and the testing
> should be at least as much about protecting users from obvious faults as
> providing feedback to developers. If we enable automatic promotion, that
> actually might discourage testing due to the 'why bother, its going to get to
> Extras anyway' factor.
>
> Note that since not long ago we also have super-testers, people with proven
> track records, it's enough to get three of their votes if your package is
> stuck and you're good to go.

Where is the super-testers list?
I've tested a few apps - though I need to get back into the pile of apps.

On Thursday 22 July 2010 02:35:47 Ian Stirling wrote:
> > Note that since not long ago we also have super-testers, people with
> > proven track records, it's enough to get three of their votes if your
> > package is stuck and you're good to go.
>
> Where is the super-testers list?
> I've tested a few apps - though I need to get back into the pile of apps.

We have no separate list for the super-testers, we (ab)use the testing-squad
list[0] for those purposes.

For more deails on the super-testers, see
https://garage.maemo.org/pipermail/testingsquad-list/2010-July/000089.html

[0] https://garage.maemo.org/mail/?group_id=1273


Regards,
Attila

Servus!

2010/7/21 Roman Morawek <lists@morawek.at>:
> [...] Actually, I think in the early lifetime of a SW project there
> will probably be a release every few months, which could permanently
> block the tool's promotion to extras.

If there are enough users / testers, releasing every month is no
problem. I have the same situation with my app (a new release nearly
every month), and with enough testers, I usually get enough votes in
the first ten days (quarantine time), so even if you have, say, a
bi-weekly release cycle, it would still be enough to get new upgrades
into Extras before the next version is released.

If you have a blog, create a "maemo" category/tag and let it be
syndicated on the Planet (planet.maemo.org) and announce your releases
there - it helps getting attention, and with some screenshots will
also describe the app to potential new users/testers (you already have
a very informative web page, which is very good - some apps don't even
have something like that).

I've reviewed your app and posted a comment on the testing page -
please have a look :)

Greetings,
Thomas

On Thu, Jul 22, 2010 at 1:11 PM, Thomas Perl <th.perl@gmail.com> wrote:

> If there are enough users / testers, releasing every month is no
> problem. I have the same situation with my app (a new release nearly
> every month), and with enough testers, I usually get enough votes in

Well, using, say, gPodder (a "bestseller" of sorts that everybody
uses) does not really paint the whole picture.

How about a system where developers could convince other developers to
test drive their application by voting on theirs? I.e. being able to
give a conditional vote that is "realized" only after the receiver
votes up/down on your application?

This would also encourage voting on less popular applications, as
popular apps don't need to care about conditional votes.

Incidentally:

<whoring>Someone please vote for qtdone</whoring>

--
Ville M. Vainio @@ Forum Nokia