Quality Awareness

1. Introduction

This document is a generic test plan that defines a set of test cases for applications targeting the maemo platform. Good quality applications should pass all these test cases. If you see missing test cases in this document, please join "Quality Awareness" project in garage.maemo.org and submit your cases to this document.

2. Test Cases

2.1 Packaging

2.1.1 Installation from application catalog

Precondition:

  • Device without application and with default set of repositories.

Actions:

  • Install application from install file (application catalog) by clicking install file in the browser.

Result:

  • Application installs and runs correctly.
  • All the needed dependences are downloaded automatically.
  • There are no errors in the Application manager log.
  • All icons created during installation are visible.

2.1.2 Application upgrade

Precondition:

  • Device with an older version of the application.

Actions:

  • From the Application manager.
    • Update the repositories.
    • Upgrade application.

Result:

  • The new application version works correctly in the device.
  • There are no errors in the Application manager log.

2.1.3 Application removal

Precondition:

  • Device with the application installed.

Actions:

  • Remove application by using Application manager.

Result:

  • All the files installed by the package (except for modified configuration files) are removed.
  • There are no errors in the Application manager log.
  • Kill application process if it is running
  • Remove tmp files

2.2 Networking

These cases assume that your application uses network.

2.2.1 Offline mode

Precondition:

  • Application is installed and works well in online mode.

Actions:

  • Switch device to offline mode.
  • Start application and try to use its networking functionality.

Result:

  • Application should give indication to user that network is not available.

2.2.2 Offline mode change

Precondition:

  • Application is executing and works well in online mode.
  • Working network available.

Actions:

  • Start application.
  • Switch device to offline mode.
  • Try to use its networking functionality.

Result:

  • Application should give indication to user that network is not available.
  • If application UI blocks until network timeouts (in 1/2 minutes), it first indicates to user that it's working.

2.2.3 Wlan disconnected

Precondition:

  • Application is installed and works well in online mode.
  • Working network available.

Actions:

  • Disconnect network
  • Start application and try to use networking functionality

Result:

  • Device should connect automatially to network and application works correctly.

2.2.4 WLAN and bluetooth connectivity works

Precondition:

  • Application is installed and works well in online mode.
  • Working network available and phone with data and bluetooth connectivity.

Actions:

  • Run network test cases with WLAN connection
  • Run network test cases with phone over bluetooth

Result:

  • Application works with both connections

2.3 Memory card

Run these testcases both memory cards (if applicable).

2.3.1 Memory card full

Precondition:

  • Application installed and works.
  • Fill the memory card.

Actions:

  • Run the application

Result:

  • Application should give correct error message

2.3.2 No memory card

Precondition:

  • Application installed and works.
  • Remove all memory cards.

Actions:

  • Run the application

Result:

  • Application should give correct error message

2.3.3 Memory card removed during application usage

Precondition:

  • Application installed and works
  • Memory card(s) inserted into slot

Actions:

  • Start application
  • Execute mmc functionality (e.g. access, save, delete, rename, backup)
  • Remove mmc from slot

Result:

  • Application should give correct error message

Observation:

  • MMC functionality needs to take enough time (e.g. saving large sized file) so that memory card removing can be tested during its execution.

2.5 Camera

These cases are relevant if the application uses camera.

2.5.1 Camera retracted

Precondition:

  • Application installed and works
  • Camera retracted

Actions:

  • Start application
  • Try to use camera funcionality

Result:

  • Application works and indicates that camera is not available

2.5.2 Camera popped up during application usage

Precondition:

  • Application installed and works
  • Camera retracted

Actions:

  • Start application
  • Pop up camera
  • Try to use camera functionality

Result:

  • Camera functionality works correctly

2.5.3 Simultaneous camera usage

Precondition:

  • Application installed and works
  • Other camera application running (e.g. Internet Call)
  • Camera retracted

Actions:

  • Start application
  • Try to use camera functionality

Result:

  • If application does not allow simultaneous camera usage, give correct

error message

  • If application does allow simultaneous camera usage by either having

its own switching mechanism or switching applications on the task navigator, camera functionality works correctly

2.7 Platform Integration

2.7.1 Hildon UI style

Precondition:

  • Application installed and works

Actions:

  • Test the application so that all UI elements are visible

Result:

  • Menus and toolbars are Hildonized
  • Buttons sizes are usable with fingers
  • Application icons are visible in menus & task bars
  • HW Keys are mapped to relevant features
  • Full screen mode works correctly

2.7.2 Input method integration

Precondition:

  • Application is installed to the device

Actions:

  • Test all the text inputs and HW keys in the application UI

Result:

  • Input method is invoked correctly when each of the text input widgets is tapped (by default normal input method for stylus tap and thumb keyboard for thumb tap)
  • Enter key in the keyboard has logical behaviour
  • HW keys have logical behaviour

2.7.3 Backup

Precondition:

  • Application is installed to the device

Actions:

  • Change all application settings and quit the application
  • Do a backup
  • Flash the latest IT OS release
  • Re-install the application
  • Restore backup

Results:

  • All changed application settings were restored for the selected categories

2.7.4 Help

Result:

  • Help item in application menu exists
  • Help for the application in the device help exists

2.8 Security

2.8.1 No unsecured TCP/IP ports

Precondition:

  • Application isn't installed
  • Device has either ssh or terminal installed from which netstat can be run

Actions:

  • Run "netstat -ta"
  • Install application
  • Run "netstat -ta"
  • Run application
  • Run "netstat -ta"
  • Perform application use-cases and run "netstat -ta" after each use-case

Result:

  • All identified ports should have documented security solution.

2.8.2 Network source verification

Actions:

  • Download code from or upload sensitive data to the network

Result:

  • HTTPS or public key encryption used for the downloaded content

2.8.3 Bluetooth security

Precondition:

  • Application running in the device
  • Ensure that your PC and tablet are not paired.

Actions:

  • For each RFCOMM channel that your application opens, perform: "rfcomm connect <btaddr> <channel>". To find out which channel is used, publish it in SDP database and use "sdptool browse <btaddr>".

Result:

  • Make sure that PIN is asked and user is asked to authorize connection in tablet before connection is established. Your security needs may vary, so authorization might not be necessary.

2.9 Performance

2.9.1 Startup and close time

Actions:

  • Boot the device
  • Start the application (for the first time) and note when user can interact with the application
  • Close application and note when application window disappears
  • Start application again and note when user can interact with the application

It's preferably to repeat these a few times and take the best times.

Result:

  • Mandatory: 
 First startup < 6 secs
 Second startup < 5 secs
 Closing < 2 secs
  • Desired: 
 First startup < 2 secs
 Second startup < 1 sec
 Closing < 0.5 secs

2.9.2 UI responsiveness and feedback

Actions:

  • Go through all application use-cases

Result:

  • All active UI elements react within 0.5 seconds to user actions (change state etc.)
  • If an activity takes more than 3 seconds, there's progress indication

2.9.3 Stress conditions

Actions:

  • In (ssh) console run following command to use all CPU: "while true; do false; done &"
  • Start application
  • Go through the application use-cases

Result:

  • Application works without problems

2.10 Battery usage

Precondition:

  • ssh package installed to the device and logged through ssh to the device

2.10.1 No unnecessary CPU usage (no polling)

Precondition:

  • Tested application is running, but it's not being used

Actions:

  • Start "top" in (SSH) console
  • Perform all application test-cases

Result:

  • After each application test-case ends, all processes stop their CPU usage according to "top"

2.10.2 No activity when device is idle or background

Precondition:

  • "strace" package is installed to the device
  • "No unnecessary CPU usage" test-case is performed and 
  application is left running

Actions:

  • Wait until screen blanks
  • Strace the application process and if it's threaded, all of it's threads: "strace -p <PID1> -p <PID2> ...?"
  • Restart the application and repeat above steps

Result:

  • Application has no activity when screen is blanked.

Note: threaded applications wake up at 8 sec intervals until device LinuxThreads? thread library is replaced with NPTL.

2.11 Reliability

2.11.1 Memory leaks

Precondition:

  • Application compiles and works in x86 Scratcbox
  • Read Maemo Debugging Guide
  • Install Valgrind to Scratchbox

Actions:

  • Go through application use-cases under Valgrind/Memcheck

Result:

  • Valgrind doesn't report (definitive) memory leaks for the application

2.12 Error handling

2.12.1 Full Flash memory

Actions:

  • Fill internal device flash as "user" user, e.g. by copying a file in File manager
  • Start the tested application
  • Go through application use-cases
  • Exit application

Result:

  • Application starts
  • Application use-cases which save data to Flash, give correct error message when the data save fails (this can mean that application shows an error at startup and exits after user OKs it)
  • Application exits gracefully

Note: Part of the Flash is reserved for root, so applications cannot rely on just checking whether Flash has free space.

2.12.2 Resource usage

Precondition: -application running

Action: -Start browser and browse to slashdot.org

Result: -Our application should not affect to browsing experience

2.13 Scalability

2.13.1 Different amounts of input data

This concerns applications which can load external data, either from the filesystem or network.

Precondition:

  • Start "top" in (SSH) console

Actions:

  • Load data (files) with sizes of 1KB, 10KB, 100KB, 1MB, 10MB

With compressed formats, the data size can refer to uncompressed data size in memory.

Result:

  • Loading of all files succeeds, at least when no other applications are running or swap is used
  • Mandatory: The time application spends parsing/loading the data does not increase exponentially nor does the application memory usage increase exponentially
  • Desired: With large data files, time and memory usage is less than linear in relation to the data size (application doesn't read all of the data, scales images on the run etc)

2.13.2 Different screen sizes

Actions:

  • Start application
  • Switch application window between full screen and normal mode

Result:

  • Both modes work

2.14 Documentation

Results:

  • Release note
  • Change log
  • Copyrights in place
  • Licensing in place
  • Possibility to generate API docs from source if application provides APIs

2.15 Policy conformance

Results:

  • Copyright & License defined
  • Source code available if license require it
  • Maemo trademark followed

Improve this page