gh-131531: Add android.py package command
#131532
Conversation
There was a problem hiding this comment.
The mechanics of what is here generally looks good; it's the higher-level design that gives me pause.
In particular, I'm unclear on the motivation behind moving the compilation process - and especially the build target - out of the cross-build folder.
From a purely practical perspective, the build folder can be re-used between emscripten, iOS and Android; so it make some sense to keep it in a common location. This set of changes would mean needing to keep an independent set of builds for Android.
More generally, it's breaking the expectation that a build can be performed "out-of-tree" in a (mostly arbitrary) location.
If the issue is ensuring that the prefix directory is in a known location for testing purposes, would it not be better to keep the Android folder as a source directory, and "package" the distribution artefact using a temporary folder that is a clone of the parts of the Android source folder that are known to be required, plus a copy of the prefix directory in the expected location?
That would also allow the distribution artefact to include files that aren't in the Android folder - for example, license files and the like that may be required for distribution purposes, but won't be duplicated in the Android folder itself because they're common across distribution artefacts on all platforms.
That approach would also be consistent with how the iOS testbed works. The iOS testbed has 2 modes: "clone" and "run". The iOS folder contains the source project; during a test pass, "clone" is used to copy the testbed to a new location, and is provided a path to a compiled iOS Framework as part of that clone process. The test is then run on the cloned version, not the original source location.
The iOS testbed doesn't manage the initial build, or building the final release artefact, but it could - and probably should as part of any move to make an official iOS artefact). However, the general approach of using the iOS/Android folder purely as a source, rather than a working directory, should work for both platforms, and for both testing and distribution packaging. It also allows the distribution process to be "opt in" on what should be distributed, rather than excluding specific artefacts from a complete build (which could easily lead to new files being inadvertently included in a distribution artefact because we forget to exclude them).
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
… rather than extracting it from a Gradle release
…oth the source layout and the packaged layout
OK, those are all good points. I've moved the build directories back to |
|
!buildbot android |
|
🤖 New build scheduled with the buildbot fleet by @mhsmith for commit 150137f 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131532%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
|
I have made the requested changes; please review again. |
|
Thanks for making the requested changes! @freakboy3742: please review the changes made to this pull request. |
There was a problem hiding this comment.
One question inline about a possible simplification of the directory structure.
The only other question I have is about how this will interact with the next piece of work that you've flagged (running a third-party test suite).
The current state of the code involves handling the "running in the source tree" case as a special case, and that identification seems to be based on the name of the parent directory, and the existence of a pyconfig.h.in file. The existence of pyconfig.h.in file will be a reasonably robust test, but it won't be infallible; I'm wondering it that potential flaw could be avoided entirely by running the normal test suite against the directory structure "as packaged", rather than making the inSourcetree special case?
That might be part of what you've got in mind for the next piece of work, especially given that it may require some sort of copy/clone/unpack process for the packaged artefact. I'm happy defer that work for later if that's what you've got in mind; I just wanted to make sure I understood the scope for the next piece.
Android/android.py
Outdated
| os.chdir(subdir("build", clean=context.clean)) | ||
| if context.clean: | ||
| clean("build") | ||
| os.chdir(subdir("build", "build", create=True)) |
There was a problem hiding this comment.
Is there a purpose to this extra level of build here? It makes at least some sense in the android folder, because of the prefix, build and dist subfolders; but even there, I'm not sure I see why the dist and prefix folders couldn't be mounted in the `cross-build/*-linux-android - my read of the OOT build handling is that it's designed for this sort of "output artefact" storage, without the need for another level of indirection.
I'm mostly being aware that this results in a cross-build/build/build/build folder which... seems like too many builds :-)
There was a problem hiding this comment.
I'm not sure I see why the
distandprefixfolders couldn't be mounted in the `cross-build/*-linux-android
That's a good idea. Done.
For comparison, the other scripts that use the cross-build directory are:
Tools/wasm/wasi.py– Builds in the top-level triplet directory; doesn't use anyprefixdirectory.Tools/wasm/emscripten/__main__.py– Usesbuildandprefixsubdirectories, though it doesn't useprefixso heavily as Android does. Maybe it could be migrated to the Android layout next time the script is updated.
There was a problem hiding this comment.
Agreed there's an opportunity to make the emscripten build consistent.
I've also got some informal build scripts for iOS that I really should formalize into some utility methods... but FWIW, they all use the triplet directory.
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
That was my reason for originally keeping everything in the But even then, there was still a need to detect whether we were in a source tree, because of the code in build.gradle.kts which copies the standard library from its source directory ( However, after this PR is merged I am planning to update the buildmaster script so it will:
I haven't worked out exactly how to deal with third-party test suites, but I'll leave that to a separate PR. |
|
!buildbot android |
|
🤖 New build scheduled with the buildbot fleet by @mhsmith for commit c537d97 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131532%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
|
I have made the requested changes; please review again. |
|
Thanks for making the requested changes! @freakboy3742: please review the changes made to this pull request. |
There was a problem hiding this comment.
Filling in details of a conversation @mhsmith had in a private channel - there may be some additional cleanups possible, but if they are, those will likely fall out of the next piece of work to make the testbed suitable for use in testing third-party projects.
|
Thanks @mhsmith for the PR, and @freakboy3742 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13. |
Adds a `package` entry point to the `android.py` build script to support creating an Android distribution artefact. (cherry picked from commit fe5c4c5) Co-authored-by: Malcolm Smith <smith@chaquo.com>
|
GH-131960 is a backport of this pull request to the 3.13 branch. |
|
|
The Fedora test failure has to be a false positive - this doesn't change any code that Fedora would be running. |
|
Adds a `package` entry point to the `android.py` build script to support creating an Android distribution artefact.
This PR makes two changes:
buildandprefixdirectories fromcpython/cross-buildintocpython/Android.android.py packagecommand which creates a tarball from a subset of theAndroiddirectory, including all the files useful either for embedding Python in an app, or building third-party packages against it.The resulting package is about 30 MB. Here's an example artifact (temporary URL).
The package includes a copy of the testbed and the android.py script, which can be used to run the CPython test suite outside of the context of the source tree. A future PR will add the ability to run test suites of third-party packages.
For Python 3.13 and older, we can host these packages in the Chaquopy Maven Central repository for the use of cibuildwheel and Chaquopy itself. From 3.14 onwards, they will hopefully be hosted on python.org.
📚 Documentation preview 📚: https://cpython-previews--131532.org.readthedocs.build/