Skip to content

Commit 9fc2353

Browse files
fkdosilovicfkdosilovic
committed
Update contributing guide. Update tests to use default client fixtures.
1 parent fa0c502 commit 9fc2353

File tree

4 files changed

+92
-29
lines changed

4 files changed

+92
-29
lines changed

‎docs/source/contributors_guide/contributing.rst

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -47,24 +47,47 @@ You'll see a different output since the documentation is build with
4747
Testing
4848
-------
4949

50-
If you implemented a feature or fixed a bug, please add tests for it.
50+
.. warning::
51+
If you are implementing features or fixing bugs, you are expected to have
52+
all of the three API keys (ATD, Sulu, and RapidAPI) setup and set in you
53+
environment variables - ``JUDGE0_SULU_API_KEY``, ``JUDGE0_RAPID_API_KEY``,
54+
and ``JUDGE0_ATD_API_KEY``.
5155

52-
Unfortunately, at the moment you cannot run full test suite because it requires
53-
access to API keys for all implemented API hubs (ATD, Sulu, and RapidAPI) and
54-
a private Judge0 instance. To partially address this situation, you can run and
55-
test your implemented feature and tests locally and use the GitHub CI pipeline
56-
to run the full test suite.
56+
Every bug fix or new feature should have tests for it. The tests are located in
57+
the ``tests`` directory and are written using `pytest <https://docs.pytest.org/en/stable/>`_.
5758

58-
To run the tests locally, you can use the following command:
59+
While working with the tests, you should use the following fixtures:
5960

60-
.. code-block:: console
61+
* ``default_ce_client`` - a client, chosen based on the environment variables set, that uses the CE flavor of the client.
62+
* ``default_extra_ce_client`` - a client, chosen based on the environment variables set, that uses the Extra CE flavor of the client.
6163

62-
$ pytest -svv tests -k '<test_name>'
64+
The ``default_ce_client`` and ``default_extra_ce_client`` are fixtures that
65+
return a client based on the environment variables set. This enables you to
66+
run the full test suite locally, but also to run the tests on the CI pipeline
67+
without changing the tests.
6368

64-
To make the test compatible with the CI pipeline, you should use one of the
65-
client fixtures:
69+
You can use the fixtures in your tests like this:
6670

6771
.. code-block:: python
6872
6973
def test_my_test(request):
70-
client = request.getfixturevalue("judge0_ce_client") # or judge0_extra_ce_client
74+
client = request.getfixturevalue("default_ce_client") # or default_extra_ce_client
75+
76+
To run the tests locally, you can use the following command:
77+
78+
.. code-block:: console
79+
80+
$ pytest -svv tests -k '<test_name>'
81+
82+
This will enable you to run a single test, without incurring the cost of
83+
running the full test suite. If you want to run the full test suite, you can
84+
use the following command:
85+
86+
.. code-block:: console
87+
88+
$ pytest -svv tests
89+
90+
or you can create a draft PR and let the CI pipeline run the tests for you.
91+
The CI pipeline will run the tests on every PR, using a private instance
92+
of Judge0, so you can be sure that your changes are not breaking the existing
93+
functionality.

‎tests/conftest.py

Lines changed: 50 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,23 +13,31 @@ def judge0_ce_client():
1313
api_key = os.getenv("JUDGE0_TEST_API_KEY")
1414
api_key_header = os.getenv("JUDGE0_TEST_API_KEY_HEADER")
1515
endpoint = os.getenv("JUDGE0_TEST_CE_ENDPOINT")
16-
client = clients.Client(
17-
endpoint=endpoint,
18-
auth_headers={api_key_header: api_key},
19-
)
20-
return client
16+
17+
if api_key is None or api_key_header is None or endpoint is None:
18+
return None
19+
else:
20+
client = clients.Client(
21+
endpoint=endpoint,
22+
auth_headers={api_key_header: api_key},
23+
)
24+
return client
2125

2226

2327
@pytest.fixture(scope="session")
2428
def judge0_extra_ce_client():
2529
api_key = os.getenv("JUDGE0_TEST_API_KEY")
2630
api_key_header = os.getenv("JUDGE0_TEST_API_KEY_HEADER")
2731
endpoint = os.getenv("JUDGE0_TEST_EXTRA_CE_ENDPOINT")
28-
client = clients.Client(
29-
endpoint=endpoint,
30-
auth_headers={api_key_header: api_key},
31-
)
32-
return client
32+
33+
if api_key is None or api_key_header is None or endpoint is None:
34+
return None
35+
else:
36+
client = clients.Client(
37+
endpoint=endpoint,
38+
auth_headers={api_key_header: api_key},
39+
)
40+
return client
3341

3442

3543
@pytest.fixture(scope="session")
@@ -63,12 +71,44 @@ def rapid_extra_ce_client():
6371
@pytest.fixture(scope="session")
6472
def sulu_ce_client():
6573
api_key = os.getenv("JUDGE0_SULU_API_KEY")
74+
if api_key is None:
75+
pytest.fail(
76+
"Sulu API key is not available for testing. Make sure to have "
77+
"JUDGE0_SULU_API_KEY in your environment variables."
78+
)
79+
6680
client = clients.SuluJudge0CE(api_key)
6781
return client
6882

6983

7084
@pytest.fixture(scope="session")
7185
def sulu_extra_ce_client():
7286
api_key = os.getenv("JUDGE0_SULU_API_KEY")
87+
if api_key is None:
88+
pytest.fail(
89+
"Sulu API key is not available for testing. Make sure to have "
90+
"JUDGE0_SULU_API_KEY in your environment variables."
91+
)
92+
7393
client = clients.SuluJudge0ExtraCE(api_key)
7494
return client
95+
96+
97+
@pytest.fixture(scope="session")
98+
def default_ce_client(judge0_ce_client, sulu_ce_client):
99+
if judge0_ce_client is not None:
100+
return judge0_ce_client
101+
if sulu_ce_client is not None:
102+
return sulu_ce_client
103+
104+
pytest.fail("No default CE client available for testing.")
105+
106+
107+
@pytest.fixture(scope="session")
108+
def default_extra_ce_client(judge0_extra_ce_client, sulu_extra_ce_client):
109+
if judge0_extra_ce_client is not None:
110+
return judge0_extra_ce_client
111+
if sulu_extra_ce_client is not None:
112+
return sulu_extra_ce_client
113+
114+
pytest.fail("No default Extra CE client available for testing.")

‎tests/test_api_test_cases.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,7 @@ def test_single_test_case_in_iterable(self, test_cases, stdin, expected_output):
207207
def test_test_cases_from_run(
208208
source_code_or_submissions, test_cases, expected_status, request
209209
):
210-
client = request.getfixturevalue("judge0_ce_client")
210+
client = request.getfixturevalue("default_ce_client")
211211

212212
if isinstance(source_code_or_submissions, str):
213213
submissions = judge0.run(
@@ -254,7 +254,7 @@ def test_test_cases_from_run(
254254
],
255255
)
256256
def test_no_test_cases(submissions, expected_status, request):
257-
client = request.getfixturevalue("judge0_ce_client")
257+
client = request.getfixturevalue("default_ce_client")
258258

259259
submissions = judge0.run(
260260
client=client,
@@ -269,7 +269,7 @@ def test_no_test_cases(submissions, expected_status, request):
269269

270270
@pytest.mark.parametrize("n_submissions", [42, 84])
271271
def test_batched_test_cases(n_submissions, request):
272-
client = request.getfixturevalue("judge0_ce_client")
272+
client = request.getfixturevalue("default_ce_client")
273273
submissions = [
274274
Submission(source_code=f"print({i})", expected_output=f"{i}")
275275
for i in range(n_submissions)

‎tests/test_submission.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ def test_from_json():
5252

5353

5454
def test_status_before_and_after_submission(request):
55-
client = request.getfixturevalue("judge0_ce_client")
55+
client = request.getfixturevalue("default_ce_client")
5656
submission = Submission(source_code='print("Hello World!")')
5757

5858
assert submission.status is None
@@ -65,7 +65,7 @@ def test_status_before_and_after_submission(request):
6565

6666

6767
def test_is_done(request):
68-
client = request.getfixturevalue("judge0_ce_client")
68+
client = request.getfixturevalue("default_ce_client")
6969
submission = Submission(source_code='print("Hello World!")')
7070

7171
assert submission.status is None
@@ -77,7 +77,7 @@ def test_is_done(request):
7777

7878

7979
def test_language_before_and_after_execution(request):
80-
client = request.getfixturevalue("judge0_ce_client")
80+
client = request.getfixturevalue("default_ce_client")
8181
code = """\
8282
public class Main {
8383
public static void main(String[] args) {
@@ -97,7 +97,7 @@ def test_language_before_and_after_execution(request):
9797

9898

9999
def test_language_executable(request):
100-
client = request.getfixturevalue("judge0_ce_client")
100+
client = request.getfixturevalue("default_ce_client")
101101
code = b64decode(
102102
"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAAABAAAAAAABAAAAAAAAAAEAQAAAAAAAAAAAAAEAAOAABAEAABAADAAEAAAAFAAAAABAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAJQAAAAAAAAAlAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADHAjVANsAG+GABAAInHDwUx/41HPA8FAGhlbGxvLCB3b3JsZAoALnNoc3RydGFiAC50ZXh0AC5yb2RhdGEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAEAAAAGAAAAAAAAAAAAQAAAAAAAABAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABEAAAABAAAAAgAAAAAAAAAYAEAAAAAAABgQAAAAAAAADQAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAABAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAlEAAAAAAAABkAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAA" # noqa: E501
103103
)

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy