ci: Per-repo configuration for manually trigger tasks

anarazel · macdice · nbyavuz · anarazel · commit 49cba82bec1f · 2025-08-14T11:54:03.000-04:00
We do not want to trigger some tasks by default, to avoid using too many compute credits. These tasks have to be manually triggered to be run. But e.g. for cfbot we do have sufficient resources, so we always want to start those tasks. With this commit, an individual repository can be configured to trigger them automatically using an environment variable defined under "Repository Settings", for example: REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd" This will enable cfbot to turn them on by default when running tests for the Commitfest app. Backpatch this back to PG 15, even though PG 15 does not have any manually triggered task. Keeping the CI infrastructure the same seems advantageous. Author: Andres Freund <andres@anarazel.de> Co-authored-by: Thomas Munro <thomas.munro@gmail.com> Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com> Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com> Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de Backpatch-through: 16
diff --git a/.cirrus.star b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,32 +18,74 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
         output += config_from(repo_config_url)
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # Some tasks are manually triggered by default because they might use too
+    # many resources for users of free Cirrus credits, but they can be
+    # triggered automatically by naming them in an environment variable e.g.
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="task_name other_task" under "Repository
+    # Settings" on Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
@@ -239,7 +239,6 @@ task:
 
 task:
   depends_on: SanityCheck
-  trigger_type: manual
 
   env:
     # Below are experimentally derived to be a decent choice.
@@ -257,6 +256,8 @@ task:
 
   matrix:
     - name: NetBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_NETBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
       env:
         OS_NAME: netbsd
@@ -274,6 +275,8 @@ task:
       <<: *netbsd_task_template
 
     - name: OpenBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_OPENBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
       env:
         OS_NAME: openbsd
@@ -754,13 +757,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)

Original file line number	Diff line number	Diff line change
`@@ -10,12 +10,20 @@`
`10`	`10`	`#`
`11`	`11`	`# 1) the contents of this file`
`12`	`12`	`#`
`13`		`-# 2) if defined, the contents of the file referenced by the, repository`
	`13`	`+# 2) computed environment variables`
	`14`	`+#`
	`15`	`+# Used to enable/disable tasks based on the execution environment. See`
	`16`	`+# .cirrus.star: compute_environment_vars()`
	`17`	`+#`
	`18`	`+# 3) if defined, the contents of the file referenced by the, repository`
`14`	`19`	`# level, REPO_CI_CONFIG_GIT_URL variable (see`
`15`	`20`	`# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted`
`16`	`21`	`# format)`
`17`	`22`	`#`
`18`		`-# 3) .cirrus.tasks.yml`
	`23`	`+# This allows running tasks in a different execution environment than the`
	`24`	`+# default, e.g. to have sufficient resources for cfbot.`
	`25`	`+#`
	`26`	`+# 4) .cirrus.tasks.yml`
`19`	`27`	`#`
`20`	`28`	`# This composition is done by .cirrus.star`
`21`	`29`