• Android CTS(frome google)

    Compatibility Test Suite

    How does the CTS work?

    The Compatibility Test Suite (CTS) is a free, commercial-grade test suite, available for download. The CTS represents the "mechanism" of compatibility.

    The CTS runs on a desktop machine and executes test cases directly on attached devices or an emulator. The CTS is a set of unit tests designed to be integrated into the daily workflow (such as via a continuous build system) of the engineers building a device. Its intent is to reveal incompatibilities early on, and ensure that the software remains compatible throughout the development process.

    The CTS is an automated testing harness that includes two major software components:

    • The CTS tradefed test harness runs on your desktop machine and manages test execution.

    • Individual test cases are executed on the Device Under Test (DUT). The test cases are written in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.

    The Compatibility Test Suite Verifier (CTS Verifier) is a supplement to the CTS available for download. CTS Verifier provides tests for APIs and functions that cannot be tested on a stationary device without manual input (e.g. audio quality, accelerometer, etc).

    The CTS Verifier is a tool for manual testing and includes the following software components:

    • The CTS verifier app that is executed on the DUT and collects the results.

    • The executable(s) or script(s) that are executed on the desktop machine to provide data or additional control for some test cases in the CTS Verifier app.

    Workflow


    Figure 1. How to use CTS

    This diagram summarizes CTS workflow. Please refer to the subpages of this section starting with Setup for detailed instructions.

    Types of test cases

    The CTS includes the following types of test cases:

    • Unit tests test atomic units of code within the Android platform; e.g. a single class, such as java.util.HashMap.

    • Functional tests test a combination of APIs together in a higher-level use-case.

    Future versions of the CTS will include the following types of test cases:

    • Robustness tests test the durability of the system under stress.

    • Performance tests test the performance of the system against defined benchmarks, for example rendering frames per second.

    Areas covered

    The unit test cases cover the following areas to ensure compatibility:

    AreaDescription
    Signature tests For each Android release, there are XML files describing the signatures of all public APIs contained in the release. The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file.
    Platform API Tests Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK Class Index to ensure API correctness, including correct class, attribute and method signatures, correct method behavior, and negative tests to ensure expected behavior for incorrect parameter handling.
    Dalvik Tests The tests focus on testing the Dalvik Executable Format.
    Platform Data Model The CTS tests the core platform data model as exposed to application developers through content providers, as documented in the SDK android.provider package: contacts, browser, settings, etc.
    Platform Intents The CTS tests the core platform intents, as documented in the SDK Available Intents.
    Platform Permissions The CTS tests the core platform permissions, as documented in the SDK Available Permissions.
    Platform Resources The CTS tests for correct handling of the core platform resource types, as documented in the SDK Available Resource Types. This includes tests for: simple values, drawables, nine-patch, animations, layouts, styles and themes, and loading alternate resources.
     
     

    Setting up CTS

    Physical environment

    Wi-Fi and IPv6

    CTS tests require a Wi-Fi network that supports IPv6, can treat the Device Under Test (DUT) as an isolated client, and has an internet connection. An isolated client refers to a configuration where the DUT does not have visibility to the broadcast/multinetwork messages on that subnetwork, either by a Wi-Fi AP configuration or by running the DUT on an isolated sub-network without other devices being connected.

    If you don't have access to a native IPv6 network, an IPv6 carrier network, or a VPN to pass some tests depending on IPv6, you may instead use a Wi-Fi access point and an IPv6 tunnel. See Wikipedia list of IPv6 tunnel brokers.

    Bluetooth LE beacons

    If the DUT supports the Bluetooth LE feature, then at least three Bluetooth LE beacons should be placed within five meters of the DUT for Bluetooth LE scan testing. Those beacons can be any kind, do not need to be configured or emit anything specific, and can include iBeacon, Eddystone, or even devices simulating BLE beacons.

    Desktop machine setup

    CTS currently supports 64-bit Linux and Mac OS host machines.

    ADB and AAPT

    Before running the CTS, make sure you have recent versions of both Android Debug Bridge (adb) and Android Asset Packaging Tool (AAPT) installed and those tools' location added to the system path of your machine.

    To install ADB, download the Android SDK Tools package for your operating system, open it, and follow the instructions in the included README file. For troubleshooting information, see Installing the Stand-alone SDK Tools.

    Ensure adb and aapt are in your system path. The following command assumes you've opened the package archive in your home directory:

    $ export PATH=$PATH:$HOME/android-sdk-linux/build-tools/<version>

    Note: Please ensure your starting path and directory name are correct.

    Java Development Kit (JDK)

    Install the proper version of the Java Development Kit (JDK). See the JDK requirements.

    CTS files

    Download and open the CTS packages matching your devices' Android version and all the Application Binary Interfaces (ABIs) your devices support.

    Download and open the latest version of the CTS Media Files.

    Device detection

    Follow the step to set up your system to detect your device, such as creating a udev rules file for Ubuntu Linux.

    Android device setup

    User builds

    A compatible device is defined as a device with a user/release-key signed build, so your device should be running a system image based on the known to be compatible user build (Android 4.0 and later) from Codenames, Tags, and Build Numbers.

    Caution: When used to confirm Android compatibility of your final system image, CTS must be executed on devices with a user build.

    Storage requirements

    The CTS media stress tests require video clips to be on external storage (/sdcard). Most of the clips are from Big Buck Bunny which is copyrighted by the Blender Foundation under the Creative Commons Attribution 3.0 license.

    The required space depends on the maximum video playback resolution supported by the device (See section 5 in the compatibility definition document for the platform version of the required resolutions.) Note that the video playback capabilities of the device under test will be checked via the android.media.CamcorderProfile APIs for earlier versions of Android and the android.media.MediaCodecInfo.CodecCapabilities APIs from Android 5.0.

    Here are the storage requirements by maximum video playback resolution:

    • 480x360: 98MB
    • 720x480: 193MB
    • 1280x720: 606MB
    • 1920x1080: 1863MB

    Screen and storage

    1. Any device that does not have an embedded screen needs to be connected to a screen.
    2. If the device has a memory card slot, plug in an empty SD card. Use an SD card that supports Ultra High Speed (UHS) Bus with SDHC or SDXC capacity or one with at least speed class 10 or higher to ensure it can pass the CTS.

      Warning: CTS may modify/erase data on the SD card plugged into the device.

    3. If the device has SIM card slots, plug in an activated SIM card to each slot. If the device supports SMS, each SIM card should have its own number field populated.

    Developer UICC

    In order to run CTS carrier API tests, the device needs to has a SIM card with carrier privilege rules on it. SeePreparing the UICC.

    Android device configuration

    1. Factory data reset the device: Settings > Backup & reset > Factory data reset

      Warning: This will erase all user data from the device.

    2. Set your device's language to English (United States) from: Settings > Language & input > Language
    3. Turn on the location setting if there is a GPS or Wi-Fi / Cellular network feature on the device: Settings > Location > On
    4. Connect to a Wi-Fi network that supports IPv6, can treat the Device Under Test (DUT) as an isolated client (see the Physical Environment section above), and has an internet connection: Settings > Wi-Fi
    5. Make sure no lock pattern or password is set on the device: Settings > Security > Screen lock > None
    6. Enable USB debugging on your device: Settings > Developer options > USB debugging.

      Note: On Android 4.2 and later, Developer options is hidden by default. To make them available, go toSettings > About phone and tap Build number seven times. Return to the previous screen to find Developer options. See Enabling On-device Developer Options for additional details.

    7. Select: Settings > Developer options > Stay Awake > On
    8. Select: Settings > Developer options > Allow mock locations > On

      Note: This mock locations setting is applicable only in Android 5.x and 4.4.x.

    9. Select: Settings > Developer options > Verify apps over USB > Off

      Note: This verify apps step became required in Android 4.2.

    10. Launch the browser and dismiss any startup/setup screen.
    11. Connect the desktop machine that will be used to test the device with a USB cable

      Note: When you connect a device running Android 4.2.2 or later to your computer, the system shows a dialog asking whether to accept an RSA key that allows debugging through this computer. Select Allow USB debugging.

    12. Install and configure helper apps on the device.

      Note: For CTS versions 2.1 R2 through 4.2 R4, set up your device (or emulator) to run the accessibility tests with:
      adb install -r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk
      On the device, enable: Settings > Accessibility > Accessibility > Delegating Accessibility Service

      Note: For CTS 2.3 R4 through CTS 6.0 on devices that declare the android.software.device_admin feature, set up your device to run the device administration tests with:
      adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk
      On the device, enable only the two android.deviceadmin.cts.CtsDeviceAdminReceiver* device administrators under: Settings > Security > Select device administrators. Make sure theandroid.deviceadmin.cts.CtsDeviceAdminDeactivatedReceiver and any other preloaded device administrators stay disabled in the same menu.

    13. Copy the CTS media files to the device as follows:

      Note: For CTS 2.3 R12 and beyond, if the device supports video codecs, the CTS media files must be copied to the device.

      • Navigate (cd) to the path the media files are downloaded and unzipped to.
      • Change the file permissions: chmod u+x copy_media.sh
      • Run copy_media.sh:
        • To copy clips up to a resolution of 720x480, run: ./copy_media.sh 720x480
        • If you are not sure about the maximum resolution, try ./copy_media.sh all so that all files are copied.
        • If there are multiple devices under adb, add the -s (serial) option to the end. For example, to copy up to 720x480 to the device with serial 1234567, run: ./copy_media.sh 720x480 -s 1234567
     

    Running CTS tests

    Getting started with CTS tradefed

    See the Trade Federation Overview for an explanation of the Trade Federation (tradefed or TF for short) continuous test framework.

    To run a test plan:

    1. Connect at least one device.
    2. Press the home button to set the device to the home screen at the start of CTS.
    3. While a device is running tests, it must not be used for any other tasks and must be kept in a stationary position (to avoid triggering sensor activity) with the cameras pointing at an object that could be focused.
    4. Do not press any keys on the device while the CTS is running. Pressing keys or touching the screen of a test device will interfere with the running tests and may lead to test failures.
    5. Launch the CTS console by running the cts-tradefed script from the folder where the CTS package has been unzipped, e.g. $ ./android-cts/tools/cts-tradefed
    6. You may start the default test plan (containing all of the test packages) by appending: run cts --plan CTS (or just run cts for v2). This kicks off all the CTS tests required for compatibility. Enter list plans to see a list of test plans in the repository. Enter list packages to see a list of test packages in the repository. See the CTS command reference or type help for a complete list of supported commands.
    7. Alternately, you may run the CTS plan of your choosing from the command line using: cts-tradefed run cts --plan<plan_name>

      Note: When running Android 6.0 (Marshmallow) CTS only, we recommend you use the --skip-preconditionsoption to skip the experimental pre-conditions feature that may cause issues for when executing CTS tests.

    8. View test progress and results reported on the console.
    9. If your device is Android 5.0 or later and declares support for an ARM and a x86 ABI, you should run both the ARM and x86 CTS packages.

    Using the CTS v1 console

    For Android 6.0 or earlier, you'll use CTS v1.

    Selecting plans

    The following test plans are available:

    • CTS—all tests required for compatibility.
    • Signature—the signature verification of all public APIs
    • Android—tests for the Android APIs
    • Java—tests for the Java core library
    • VM—tests for ART or Dalvik
    • Performance—performance tests for your implementation

    These can be executed with the run cts command.

    CTS v1 console command reference

    Table 1. This table summarizes the CTS v1 console commands for various uses.

    HostDescription
    help Display a summary of the most commonly used commands
    help all Display the complete list of available commands
    exit Gracefully exit the CTS console. Console will close when all currently running tests are finished
    RunDescription
    run cts Run the specified tests and displays progress information. One of --plan, --package, --class or --continue-session needs to be specified

    The CTS console can accept other commands while tests are in progress

    If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests

    If more than one device is connected, the CTS host will choose a device automatically
    --plan <test_plan_name> Run the specified test plan
    --package/-p <test_package_name>  [--package/-p <test_package2>...] Run the specified test packages
    --class/-c <class_name> [--method/-m <test_method_name> Run the specified test class and/or method
    --continue-session Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results
    --shards <number_of_shards> Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel
    --serial/-s <deviceID> Run CTS on the specific device
    -t <class_name>#<test_method_name> Run a specific test method
    --force-abi 32|64 On 64-bit devices, run the test against only the 32-bit or 64-bit ABI
    ListDescription
    list packages List all available test packages in the repository
    list plans List all available test plans in the repository
    list invocations List 'run' commands currently being executed on devices
    list commands List all 'run' commands currently in the queue waiting to be assigned to devices
    list results List CTS results currently stored in repository
    list devices List currently connected devices and their state

    'Available' devices are functioning, idle devices, available for running tests

    'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests

    'Allocated' devices are devices currently running tests
    AddDescription
    add derivedplan --plan <plan_name>
    --result/-r
    [pass | fail | timeout | notExecuted]
    [--session/-s <session_id>]    		 Create a plan derived from given result session; use this option to rerun reports and validate test issues

    Using the CTS v2 console

    For Android 7.0 or later, you'll use CTS v2.

    Selecting plans

    Available test plans include the following:

    • cts—Runs CTS from an pre-existing CTS installation.
    • cts-camera— Runs CTS-camera from a pre-existing CTS installation.
    • cts-java— Runs Core Java Tests from a pre-existing CTS installation.
    • cts-pdk— Runs Tests useful on validating a PDK fusion build.
    • everything— Common config for Compatibility suites.

    Other available configurations include the following:

    • basic-reporters— Configuration with basic CTS reporters.
    • collect-tests-only—Runs CTS from a pre-existing CTS installation.
    • common-compatibility-config— Common config for Compatibility suites.
    • cts-filtered-sample— Common config for Compatibility suites.
    • cts-known-failures— Configuration with CTS known failures.
    • cts-preconditions— CTS precondition configs.
    • host— Runs a single host-based test on an existing device.
    • instrument— Runs a single Android instrumentation test on an existing device.
    • native-benchmark— Runs a native stress test on an existing device.
    • native-stress— Runs a native stress test on an existing device.
    • recharge— A fake test that waits for nearly-discharged devices and holds them for charging.
    • testdef— Runs tests contained in test_def.xml files on an existing device.
    • util/wifi— Utility config to configure Wi-Fi on device.
    • util/wipe— Wipes user data on device.

    All of these plans and configs can be executed with the run cts command.

    CTS v2 console command reference

    Table 1. This table summarizes the CTS V2 console commands for various uses.

    HostDescription
    help Display a summary of the most commonly used commands
    help all Display the complete list of available commands
    version Show the version.
    exit Gracefully exit the CTS console. Console will close when all currently running tests are finished.
    RunDescription
    run cts

    Run the default CTS plan (that is, the full CTS invocation).

    The CTS console can accept other commands while tests are in progress.

    If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests.

    If more than one device is connected, the CTS host will choose a device automatically.
    --plan <test_plan_name> Run the specified test plan.
    -- module/-m <test_module_name>  [--module/-m <test_module2>...] Run the specified test module or modules. For example, run cts --module CtsGestureTestCases executes the gesture test module (this can be shortened to run cts -m Gesture).
    run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test.
    -- module/-m <test_module_name> -- test <test_name>  Run the specified module and test. For example, run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test.
    --retry Retry all tests that failed or were not executed from the previous sessions. Use list results to get the session id.
    --shards <number_of_shards> Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.
    --serial/-s <deviceID> Run CTS on the specific device.
    --include-filter <module_name>  [--include-filter <module2>...] Run only with the specified modules.
    --exclude-filter <module_name>  [--exclude-filter <module2>...] Exclude the specified modules from the run.
    --log-level-display/-l <log_level> Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT].
    --abi <abi_name> Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports.
    --logcat, --bugreport, and --screenshoot-on-failure Give more visibility into failures and can help with diagnostics.
    --device-token Specifies a given device has the given token eg. --device-token 1a2b3c4d:sim-card..
    --skip-device-info Skips collection of information about the device. Note: do not use this option when running CTS for approval.
    --skip-preconditions Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection.
    ListDescription
    list modules List all available test modules in the repository.
    list plans or list configs List all available test plans (configs) in the repository.
    list invocations List 'run' commands currently being executed on devices.
    list commands List all 'run' commands currently in the queue waiting to be assigned to devices.
    list results List CTS results currently stored in repository.
    list devices List currently connected devices and their state.

    'Available' devices are functioning, idle devices, available for running tests.

    'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests.

    'Allocated' devices are devices currently running tests.
    DumpDescription
    dump logs Dump the tradefed logs for all running invocations.

    Interpreting CTS results

    The CTS test results are placed in the file:

    $CTS_ROOT/android-cts/repository/results/<start_time>.zip

    If you have built the CTS yourself, $CTS_ROOT will resemble the path out/host/linux-x86/cts but differ by platform. This reflects the path where you have uncompressed the prebuilt official CTS downloaded from this site.

    Inside the zip, the testResult.xml file contains the actual results—open this file in any web browser (HTML5 compatible browser recommended) to view the test results. It will resemble the following screenshots.

    Note: The results are provided to help you ensure the software remains compatible throughout the development process and act as a common format for communicating the compatibility status of your device with other parties.

    Figure 1. CTS test summary

    If testResult.xml displays a blank page when using the Chrome browser, change your browser configuration to enable the --allow-file-access-from-files command line flag.

    The Device Information section provides details about the device, firmware (make, model, firmware build, platform), and device hardware (screen resolution, keypad, screen type). To access device information, click the link above Test Summary.

    The Test Summary section provides executed test plan details, like the CTS plan name and execution start and end times. It also presents an aggregate summary of the number of tests that passed, failed, timed out, or could not be executed.

    The next section also provides a summary of tests passed per package.

     

    Figure 2. CTS test report

    This is followed by details of the the actual tests that were executed. The report lists the test package, test suite, test case, and the executed tests. It shows the result of the test execution—pass, fail, timed out, or not executed. In the event of a test failure details are provided to help diagnose the cause.

    Further, the stack trace of the failure is available in the XML file but is not included in the report to ensure brevity—viewing the XML file with a text editor should provide details of the test failure (search for the <Test> tag corresponding to the failed test and look within it for the <StackTrace> tag).
  • 相关阅读:
    【原创】Algorithms:原地归并排序
    Java中相等测试
    测试可变字符序列stringBuilder
    呃。。第一篇随笔。
    hiho兄弟的字典树之争（hiho1014）
    字典树
    最大公约数
    素数
    递归算法1
    logging模块,collections模块,random模块
  • 原文地址：https://www.cnblogs.com/CoderTian/p/5975991.html
Copyright © 2020-2023  润新知