How about following the same pattern that kubernetes adopts, i.e. turning the generated code into a real python submodule by manually crafting our own setup.py file? Something like this: https://github.com/kubernetes-client/python/blob/master/setup.py. This design allows us to control package dependency and all the pypi metadata. It also makes it easy for us to inject other helper submodules into the same package namespace, for example, kubernetes client maintains their own config module to customize how users can configure the generated python client. To implement the same pattern, we would need to change the python code gen script to generate the client using name client not airflow_client. Then we can name the outer package airflow_client and import the generated client using airflow_client.client.

@houqp I think what you are suggesting makes sense. Keeping the structure as it was done in k8s python client.

@houqp done

@msumit @kaxil @ryw @ashb @potiuk @mik-laj I think this client in a good state where it's ready to be published into PyPI. What's the process for doing that? Who should be responsible for creating the PyPI account to own the package?

@msumit @kaxil @ryw @ashb @potiuk @mik-laj I think this client in a good state where it's ready to be published into PyPI. What's the process for doing that? Who should be responsible for creating the PyPI account to own the package?

We will need to follow a similar process as https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md

where we create a Release candidate that PMC and community members can vote on and after the vote passes we publish it to somewhere in https://dist.apache.org/repos/dist/release/airflow/ -- maybe https://dist.apache.org/repos/dist/release/airflow/python-api or something.

Sounds good, looks like the release process itself needs to be finalized and discussed and approved by the community first. I can see us having different release processes for different languages because in Golang, release a package is just done with a single git tag push without needing to publish anything to a centralized registry.

@msumit Since you are working on the Python client actively, do you want to drive the Airflow client release discussion in the mailing list? If not, I can write something up to kick off the discussion.

Just one comment to that - we MUST make source package release signed and check summed in downloads.apache.org and released via official voting and +3 PMC votes via email, no matter what language it is in. This is a non-negotiable part of release process of the Apache Software Foundation. Anything else (including tag, PYPi releases and docker images) are just 'convenience' way of publishing the packages but they are not considered as official releases.

This is one of the most important policies of ASF and it has serious legal consequences and as PMC members we have to make sure this is happening in this way.

See those two docs: https://infra.apache.org/release-publishing.html

And

https://infra.apache.org/release-distribution.html

IMHO - this is THE most important policy for us to follow and we should make sure we are following it.

This is good info to keep in mind, thanks @potiuk for the pointers.

Sounds good, looks like the release process itself needs to be finalized and discussed and approved by the community first. I can see us having different release processes for different languages because in Golang, release a package is just done with a single git tag push without needing to publish anything to a centralized registry.

@msumit Since you are working on the Python client actively, do you want to drive the Airflow client release discussion in the mailing list? If not, I can write something up to kick off the discussion.

Yeah irrelevant of whether it needs to separately released to PyPI or for golang -- just needs pushing a tag, it is important thet they are signed and voted on.