Add contest export script #3039
Conversation
…via CLI Fetching the event feed from the API via `internalApiRequest` wasn't working since it returns a StreamedResponse. Fix this by only using an output buffer in `internalApiRequest` and only calling `ob_flush` from `getEventFeedAction` if the top-level output buffer allows it. Also return contents from `internalApiRequest` without JSON decoding it, since the event feed endpoint is NDJSON, *not* JSON. Explicitly decode the JSON at the other places calling internalApiRequest`.
This will be used by the contest export script for downloading the event feed (NDJSON) and other files. Also improve error reporting when an internal API call fails.
Also initialize `domjudge_api_url = None` and simply call the API via CLI if `domjudge_api_url` is unset.
Capture all function parameters and pass these as is to the function itself again, just now with the global `ca_check = False`.
This follows the specifications correctly, but beware that specifying a base URL that doesn't end with a `/` leads to strange results, see #2378 Replace `exit(1)` by an exception so this can be caught also if this code runs asynchronously in a thread or separate process.
This is needed for binary content that otherwise gets mangled when fetched through the CLI. Also force all but the `endpoint` and `method` parameters to `do_api_request` to be named parameters to prevent mistakes.
This gives more flexibility for extension in the future and leaves the default usage without arguments unchanged.
f55ed07 to
d13e643
Compare
| recurse_find_files(data[0], store_path, default_name) | ||
| else: | ||
| for i, item in enumerate(data): | ||
| recurse_find_files(item, store_path, f"{default_name}.{i}") |
There was a problem hiding this comment.
Initially we pass '' in default_name, so this becomes .{i}? (same with .{key} below)
There was a problem hiding this comment.
Hmm, I think you're right. The idea is that this builds up a default file name while recursively descending into the object's structure. So at the base level you always find the object itself as a dict, and that means that at line 94 default_name is set to the property key, but as you say with a prefix .. So for example for multiple JPEG files in the array under contest.logo, their default names would become .logo.0.jpg, .logo.1.jpg. So that first . must indeed be stripped.
|
|
||
| if ext == '.json': | ||
| data = json.loads(data) | ||
| store_path = name |
There was a problem hiding this comment.
Why use store_path here? Can't we just use name below?
There was a problem hiding this comment.
Yeah, I think so. I guess this is a left-over of previous refactoring. It's a bit annoying that there's too many similar concepts floating around: name of the endpoint, endpoint path in the URL, path to store the downloaded endpoint data to on disk, and filename on disk.
|
|
||
| parser = ArgumentParser(description='Import a contest archive to DOMjudge via the API.') | ||
| parser.add_argument('-d', '--dir', help="directory containing the contest archive, defaults to current directory") | ||
| parser.add_argument('-u', '--url', help="DOMjudge API URL to use, if not specified use the CLI interface") |
There was a problem hiding this comment.
This might mean some of our automation scripts need to change. I do like it though.
There was a problem hiding this comment.
I hadn't checked in ansible, but note that I added the dir argument and the default behaviour is as before. Did we ever explicitly pass the API base URL instead of using the CLI directly?
There was a problem hiding this comment.
I also see I forgot to change this in configure-contest too.
There was a problem hiding this comment.
I'm not sure actually. But if we do it's an easy change.
Co-authored-by: Nicky Gerritsen <nickygerritsen@me.com>
Closes #2726