Skip to content

Commit c8af42b

Browse files
pi-anlpi-anl
committed
docs: Improve documentation for writing and running tests
Reviewed the run-tests.py and run-multitests.py scripts and used the insights to significantly enhance the documentation in docs/develop/writingtests.rst. The updated documentation provides: - More detailed explanation of the test directory structure and key runner scripts. - Clearer instructions on writing simple tests, including the use of print() and .exp files. - Expanded coverage of run-tests.py options for targeting specific devices, filtering tests, and handling failures. - An overview of how run-tests.py works internally (feature detection, skipping, execution, comparison). - Guidance on writing advanced tests (skipping, unittest, special handling, float/endian considerations). - Introduction to multi-instance testing with run-multitests.py, including how to write tests with instanceN() functions and use the `multitest` helper for coordination. - Examples of running multi-instance tests with different instance types. 🤖 Generated with gemini-2.5-pro-exp-03-25 Co-Authored-By: gemini-2.5-pro-exp-03-25 <gemini-2.5-pro-exp-03-25@alelec.net> Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
1 parent ef8282c commit c8af42b

File tree

2 files changed

+367
-235
lines changed

2 files changed

+367
-235
lines changed

‎docs/develop/writingtests.rst

Lines changed: 201 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,230 @@
11
.. _writingtests:
22

3-
Writing tests
4-
=============
3+
Test Suite
4+
==========
55

6-
Tests in MicroPython are located at the path ``tests/``. The following is a listing of
7-
key directories and the run-tests.py runner script:
6+
The MicroPython test suite resides in the ``tests/`` directory and consists of multiple components:
87

9-
.. code-block:: bash
8+
1. **Standard Tests**: The majority of tests that run on a single instance and verify functionality against CPython or expected outputs.
9+
2. **Multi-instance Tests**: Tests that require multiple MicroPython instances to interact with each other (e.g., networking, Bluetooth).
10+
3. **Native Module Tests**: Tests for dynamic native modules compiled to ``.mpy`` files.
11+
4. **Performance Benchmarks**: Python-level benchmarks for measuring execution speed.
12+
5. **Internal Benchmarks**: Low-level benchmarks for core VM operations and C code.
13+
14+
This document primarily focuses on the standard tests and multi-instance tests.
15+
16+
Directory Structure
17+
-------------------
18+
19+
The ``tests/`` directory is organized into subfolders by functionality:
20+
21+
* ``basics/``: Core Python language features
22+
* ``extmod/``: Extended modules (like ``ujson``, ``ure``)
23+
* ``float/``: Floating-point arithmetic specifics
24+
* ``micropython/``: MicroPython-specific features and behaviors
25+
* ``import/``: Import mechanisms
26+
* ``io/``: Input/Output operations
27+
* ``stress/``: Tests designed to push resource limits (memory, recursion)
28+
* ``thread/``: Threading module tests
29+
* ``cmdline/``: Tests for the command-line interface and REPL
30+
* ``ports/<port_name>/``: Tests specific to a particular port
31+
* ``feature_check/``: Scripts used to detect target capabilities
32+
* ``multi_bluetooth/``, ``multi_network/``: Tests involving multiple instances
33+
* ``perf_bench/``: Performance benchmarking scripts
34+
* ``internal_bench/``: Low-level internal benchmarks
35+
36+
Key Test Runner Scripts:
1037

11-
.
12-
├── basics
13-
├── extmod
14-
├── float
15-
├── micropython
16-
├── run-tests.py
17-
...
38+
* ``run-tests.py``: Main script for standard tests
39+
* ``run-multitests.py``: For tests requiring multiple interacting instances
40+
* ``run-natmodtests.py``: For testing dynamic native modules
41+
* ``run-perfbench.py``: For performance benchmarks
42+
* ``run-internalbench.py``: For low-level internal benchmarks
1843

19-
There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the
20-
existing folders or in a new folder. It's also possible to make custom tests outside this tests folder,
21-
which would be recommended for a custom port.
44+
Writing Standard Tests
45+
----------------------
2246

23-
For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory:
47+
Standard tests are the most common type. They verify functionality by comparing output against expected results.
48+
49+
1. **Choose a Directory:** Select the appropriate subfolder for your test based on the functionality it tests.
50+
2. **Create a ``.py`` file:** Add your test code to a new Python file.
51+
3. **Use ``print()``:** The test harness works by comparing standard output. Use ``print()`` statements to output results.
52+
4. **Expected Output:**
53+
* **CPython Comparison:** By default, tests are run on both CPython and MicroPython with outputs compared.
54+
* **``.exp`` Files:** For MicroPython-specific features, create a ``<testname>.py.exp`` file with the expected output.
55+
56+
Example:
57+
58+
Create ``tests/basics/simple_print.py``:
2459

2560
.. code-block:: python
2661
27-
def print_one():
28-
print(1)
62+
# tests/basics/simple_print.py
63+
print("Hello")
64+
a = 1 + 2
65+
print(a)
66+
67+
When run, MicroPython's output will be compared to CPython's output (or ``simple_print.py.exp`` if it exists).
68+
69+
Types of Tests
70+
--------------
71+
72+
The test system supports three main testing approaches:
2973

30-
print_one()
74+
1. **CPython comparison tests**:
75+
* Used for behavior that should match CPython
76+
* Tests are run on both CPython and MicroPython
77+
* Passes if outputs match exactly
3178

32-
If you run your tests, this test should appear in the test output:
79+
2. **Expected output (``.exp``) tests**:
80+
* Used for MicroPython-specific features or when CPython behavior differs
81+
* MicroPython output is compared against contents of a ``.py.exp`` file
82+
* Passes if outputs match exactly
83+
84+
3. **``unittest``-based tests**:
85+
* Requires ``unittest`` module to be available on the target
86+
* Used for hardware testing or other MicroPython-specific behavior
87+
* Passes if unittest summary shows "OK"
88+
89+
Running Tests (``run-tests.py``)
90+
--------------------------------
91+
92+
The primary script is ``run-tests.py``, executed from the ``tests/`` directory.
93+
94+
**Basic Usage:**
3395

3496
.. code-block:: bash
3597
36-
$ cd ports/unix
37-
$ make tests
38-
skip unix/extra_coverage.py
39-
pass unix/ffi_callback.py
40-
pass unix/ffi_float.py
41-
pass unix/ffi_float2.py
42-
pass unix/print.py
43-
pass unix/time.py
44-
pass unix/time2.py
98+
# Run default tests for the Unix port
99+
cd tests/
100+
./run-tests.py
101+
102+
**Running on Hardware Targets:**
103+
104+
.. code-block:: bash
45105
46-
Tests are run by comparing the output from the test target against the output from CPython.
47-
So any test should use print statements to indicate test results.
106+
# Run on a specific serial port
107+
./run-tests.py -t /dev/ttyACM0
108+
./run-tests.py -t COM3
48109
49-
For tests that can't be compared to CPython (i.e. micropython-specific functionality),
50-
you can provide a ``.py.exp`` file which will be used as the truth for comparison.
110+
# Use shortcuts for common serial ports
111+
./run-tests.py -t a0 # Maps to /dev/ttyACM0
112+
./run-tests.py -t u1 # Maps to /dev/ttyUSB1
113+
./run-tests.py -t c3 # Maps to COM3
51114
52-
The other way to run tests, which is useful when running on targets other than the Unix port, is:
115+
**Filtering Tests:**
53116

54117
.. code-block:: bash
55118
56-
$ cd tests
57-
$ ./run-tests.py
119+
# Run all tests in specific directories
120+
./run-tests.py -d basics
121+
./run-tests.py -d extmod -d float
122+
123+
# Run specific files
124+
./run-tests.py basics/int*.py
125+
./run-tests.py float/float_parse.py
126+
127+
# Filter by regex
128+
./run-tests.py -e viper # Exclude tests matching 'viper'
129+
./run-tests.py -i asyncio # Include only tests matching 'asyncio'
130+
131+
**Other Useful Options:**
132+
133+
* ``--emit {bytecode | native | viper}``: Run tests using a specific code emitter
134+
* ``--via-mpy``: Compile tests to ``.mpy`` files first before running
135+
* ``-j N`` or ``--jobs N``: Run N tests in parallel (for PC targets)
136+
* ``--print-failures``: Show the diff for tests that failed in the last run
137+
* ``--run-failures``: Re-run only the tests that failed in the last run
138+
* ``--clean-failures``: Remove the ``.exp`` and ``.out`` files from failures
139+
140+
How The Test System Works
141+
-------------------------
58142

59-
Then to run on a board:
143+
1. **Target Detection:** Determines the platform and architecture.
144+
2. **Feature Detection:** Identifies capabilities of the target MicroPython build.
145+
3. **Test Selection:** Gathers test files based on command-line arguments or defaults.
146+
4. **Skipping Tests:** Excludes tests based on:
147+
* Command-line filters
148+
* Missing features
149+
* Platform-specific skip lists
150+
* Emitter type restrictions
151+
* Tests explicitly printing ``SKIP``
152+
5. **Execution:**
153+
* PC targets: Runs MicroPython as a subprocess
154+
* Board targets: Uses ``pyboard.py`` to connect via serial/network
155+
6. **Output Comparison:** Compares target's output with expected output
156+
7. **Reporting:** Shows results and summarizes pass/fail statistics
157+
158+
Writing Advanced Tests
159+
----------------------
160+
161+
**Skipping Tests Conditionally**
162+
163+
If a test needs to skip itself conditionally:
164+
165+
.. code-block:: python
166+
167+
import sys
168+
if not hasattr(sys, 'feature_needed'):
169+
print('SKIP')
170+
sys.exit()
171+
172+
**Special Test Handling**
173+
174+
Some tests need special options:
175+
176+
* Add comments in the test file: ``# cmdline: -X heapsize=...``
177+
* Tests needing redirection are listed in ``special_tests`` in ``run-tests.py``
178+
179+
**Platform Considerations**
180+
181+
* **Floating Point:** Be mindful of precision differences. Use ``math.isclose()`` with appropriate tolerances.
182+
* **Endianness:** Check ``sys.byteorder`` for byte order sensitive operations.
183+
* **Memory/Stack:** Avoid large allocations or deep recursion on constrained devices.
184+
185+
Multi-instance Tests
186+
--------------------
187+
188+
For testing interactions like networking or Bluetooth, ``run-multitests.py`` orchestrates multiple instances.
189+
190+
**Writing Multi-instance Tests:**
191+
192+
1. Define functions named ``instance0()``, ``instance1()``, etc. for each instance's code.
193+
2. Use the ``multitest`` helper for coordination:
194+
* ``multitest.next()``: Complete a stage and wait for other instances
195+
* ``multitest.broadcast(msg)``: Send a message to all other instances
196+
* ``multitest.wait(msg)``: Wait for a specific broadcast message
197+
* ``multitest.skip()``: Indicate the test should be skipped
198+
* ``multitest.globals(var=value, ...)``: Set global variables
199+
* ``multitest.get_network_ip()``: Get the IP address
200+
* ``multitest.expect_reboot(resume_func_name, delay_ms)``: Handle device reboots
201+
202+
**Running Multi-instance Tests:**
60203

61204
.. code-block:: bash
62205
63-
$ ./run-tests.py -t /dev/ttyACM0
206+
# Run a network test using two unix instances
207+
./run-multitests.py -i micropython -i micropython multi_network/tcp_connect.py
208+
209+
# Run a bluetooth test using two pyboards
210+
./run-multitests.py -i pyb:/dev/ttyACM0 -i pyb:/dev/ttyACM1 multi_bluetooth/ble_connect.py
211+
212+
Provide as many ``-i`` arguments as instances required by the tests.
64213

65-
And to run only a certain set of tests (eg a directory):
214+
Test Certificates for SSL/TLS Tests
215+
-----------------------------------
216+
217+
Network tests using SSL/TLS require test certificates to be available on the device. These certificates are included in the repository in ``multi_network/`` and ``net_inet/`` directories.
218+
219+
**Preparing a Device for SSL/TLS Tests:**
220+
221+
Use ``mpremote`` to set the device's RTC and copy the certificate files:
66222

67223
.. code-block:: bash
68224
69-
$ ./run-tests.py -d basics
70-
$ ./run-tests.py float/builtin*.py
225+
# From the micropython/tests/ directory:
226+
mpremote rtc --set cp multi_net/*.der net_inet/*.der :/
227+
228+
This sets the device's Real Time Clock (using the host's time) and copies all ``.der`` files to the root directory of the device's filesystem.
229+
230+
These certificates are self-signed and generated for testing purposes only.

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy