Skip to content

feat: adds time_zone to external config and load job #2229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 9, 2025

Conversation

chalmerlowe
Copy link
Collaborator

@chalmerlowe chalmerlowe commented Jul 2, 2025

This commit introduces new configuration options for BigQuery load jobs and external table definitions, aligning with recent updates to the underlying protos.

New options added:

time_zone: Time zone used when parsing timestamp values that do not have specific time zone information. (Applies to LoadJobConfig, LoadJob, and ExternalConfig)

Changes include:

Added corresponding properties (getters/setters) to LoadJobConfig, LoadJob, and ExternalConfig.
Updated docstrings and type hints for all new attributes.
Updated unit tests to cover the new options, ensuring they are correctly handled during object initialization, serialization to API representation, and deserialization from API responses.

@chalmerlowe chalmerlowe requested review from a team as code owners July 2, 2025 20:25
@chalmerlowe chalmerlowe requested a review from Neenu1995 July 2, 2025 20:25
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery API. labels Jul 2, 2025
Copy link
Contributor

@Linchin Linchin Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to testing to_api_repr(), I think we also need to test against the class property itself, and also from_api_repr().

Copy link
Collaborator Author

@chalmerlowe chalmerlowe Jul 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the original author of this test file (specifically test_external_config.py) designed it to perform the tests you are discussing, even if it is not immediately apparent and would argue the current state is satisfactory:

  • the new attribute (time_zone) is tested in test_to_api_repr_base()
  • the new attribute is tested in test_from_api_repr_base()
  • during the above tests, we also explicitly test the getter and setter for the new attribute

If we decide that this pattern/philosophy is not satisfactory, then it seems that we will need to completely rewrite this file from top to bottom.

See below for a walkthrough:

TestExternalConfig.test_to_api_repr_base()

  • We create an ExternalConfig object (ec) from the ExternalConfig class
  • We then set each class attribute individually (ec.autodetect=True, ec.time_zone=self.TIME_ZONE, etc)
    • by default, this tests the setter for each attribute covered by BASE_RESOURCE including time_zone
  • We use ec.to_api_repr() to create got_resource. This tests the ability to use to_api_repr()
  • We compare got_resource to exp_resource (which includes all the attributes we just set) ensuring that to_api_repr() performed as expected

TestExternalConfig.test_from_api_repr_base()

  • Also relies on BASE_RESOURCE to facilitate testing
  • we use from_api_repr() to create an ExternalConfig object (ec) based on BASE_RESOURCE #L43
  • we use _verify_base() to ensure that the ec object matches the expected values from BASE_RESOURCE #L44 & #L129
    • by default, this tests the getter for each attribute covered by BASE_RESOURCE including time_zone
    • it is intended to check every single item in BASE_RESOURCE (as long as it and BASE_RESOURCE are in sync)
  • we then update the api_repr with more elements (schema and fields, etc) #L51
  • we use _verify_base() again to ensure that the expected values from BASE_RESOURCE are present along with the additional elements #L67

If we look at the remainder of the file, we see this pattern repeated several times. Looking at:

  • test_from_api_repr_csv
  • test_to_api_repr_csv
  • test_from_api_repr_bigtable
  • test_to_api_repr_bigtable

we see this philosophy and pattern repeated almost exactly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding adding a to_api_repr and from_api_repr pair, there may be some merit to creating a single test that includes all the attributes as part of a BASE_RESOURCE much as we see in test_external_config.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to https://github.com/googleapis/python-bigquery/pull/2229/files#r2191183829, I think in general we need to test against to/from_api_repr and the property itself. The same goes for LoadJobConfig and LoadJob. Could you add these changes? Otherwise the PR looks good in general.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pattern/philosophy for test_load_config.py is different than the approach used in test_external_config.py

Here, the author chose to test each attribute in more of a stand alone fashion using a trifecta of tests:

  • x_missing()
  • x_hit()
  • x_setter()

But:
the author did not test to_api_repr() or from_api_repr() directly (I can add such a pair of tests).

Examples:

  • def test_write_disposition_missing()
  • def test_write_disposition_hit()
  • def test_write_disposition_setter()
  • def test_time_zone_missing()
  • def test_time_zone_hit()
  • def test_time_zone_setter()

missing

Confirms that when the target class is created, the attribute is not populated

self.assertIsNone(config.write_disposition)`

hit

Sets (behind the scenes using ._properties) the value of the attribute
Confirms that the getter functions correctly and that the expected/result values match

config._properties["load"]["writeDisposition"] = write_disposition
self.assertEqual(config.write_disposition, write_disposition)

setter

Sets the value of the attribute directly using the setter function
Confirms that the setter functions correctly by examining the _properties object directly

config.write_disposition = write_disposition
self.assertEqual(config._properties["load"]["writeDisposition"], write_disposition)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test_to_api_repr() and a test_from_api_repr() test because the original file did not include them.

@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Jul 9, 2025
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Jul 9, 2025
@chalmerlowe chalmerlowe added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 9, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 9, 2025
@chalmerlowe chalmerlowe added the automerge Merge the pull request once unit tests and other checks pass. label Jul 9, 2025
@Linchin Linchin self-requested a review July 9, 2025 22:10
Linchin
Linchin previously approved these changes Jul 9, 2025
@Linchin Linchin self-requested a review July 9, 2025 22:56
@gcf-merge-on-green gcf-merge-on-green bot merged commit b2300d0 into main Jul 9, 2025
28 checks passed
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jul 9, 2025
@gcf-merge-on-green gcf-merge-on-green bot deleted the feat-374142081-add-time_zone_options branch July 9, 2025 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy