Overlay: report telemetry #2975

cklin · 2025-07-17T17:45:11Z

This PR adds telemetry fields for overlay analysis.

analysis_is_overlay
analysis_builds_overlay_base_database
overlay_base_database_download_size_bytes
overlay_base_database_download_duration_ms

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

Copilot

Pull Request Overview

This PR adds telemetry tracking for overlay database analysis operations. The changes enable the CodeQL Action to collect and report statistics about overlay-base database downloads and analysis modes.

Modified the download function to return telemetry statistics instead of a boolean
Added new telemetry fields to track download size, duration, and analysis modes
Expanded test coverage for the updated download function behavior

Reviewed Changes

Copilot reviewed 8 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/overlay-database-utils.ts	Updated download function to return telemetry stats and measure database size/download time
src/overlay-database-utils.test.ts	Added comprehensive test cases for the modified download function
src/init-action.ts	Updated to capture and pass overlay database stats to telemetry reporting
src/analyze.ts	Added telemetry flags to track overlay vs overlay-base analysis modes
lib/*	Generated JavaScript files reflecting the TypeScript changes

src/overlay-database-utils.ts

mbg

This generally looks good, assuming all the necessary backend work is also done to support this. Thanks also for including some tests.

There are no blocking issues. I flagged up a couple points that might be good to deal with before merging this though, as well as some other trivial points.

mbg · 2025-07-18T10:30:48Z

src/analyze.ts

+  } else if (
+    config.augmentationProperties.overlayDatabaseMode ===
+    OverlayDatabaseMode.OverlayBase
+  ) {
+    statusReport.analysis_builds_overlay_base_database = true;


Minor: perhaps this would be slightly nicer as, similar to what we already have for analysis_is_diff_informed:

Suggested change

} else if (

config.augmentationProperties.overlayDatabaseMode ===

OverlayDatabaseMode.OverlayBase

) {

statusReport.analysis_builds_overlay_base_database = true;

}

statusReport.analysis_builds_overlay_base_database =

config.augmentationProperties.overlayDatabaseMode ===

OverlayDatabaseMode.OverlayBase;

Thanks—I implemented the suggestion.

mbg · 2025-07-18T10:35:03Z

src/overlay-database-utils.ts

@@ -235,41 +235,47 @@ export async function uploadOverlayBaseDatabaseToCache(
  return true;
 }

+export interface OverlayBaseDatabaseDownloadStats {
+  databaseSizeBytes: number;
+  databaseDownloadDurationMillis: number;


Minor: elsewhere we use Ms as suffix rather than Millis. Might be nicer to use that for consistency.

Thanks—I implemented the suggestion.

mbg · 2025-07-18T10:38:09Z

src/overlay-database-utils.ts

 */
 export async function downloadOverlayBaseDatabaseFromCache(
  codeql: CodeQL,
  config: Config,
  logger: Logger,
-): Promise<boolean> {
+): Promise<undefined | OverlayBaseDatabaseDownloadStats> {


Minor: Conventionally the undefined seems to come last. There's nothing wrong with this I think, but it's just a bit unusual. I see we have this order in a couple of other places in the codebase though.

A bigger question here is probably why the return type is used to indicate failure at all. Wouldn't it be better to have this as Promise<OverlayBaseDatabaseDownloadStats> and reject if the download was either not performed or failed?

I moved undefined to the last.

The question of using a rejected promise instead of promise of undefined is interesting. There are three cases:

We are not supposed to download, so we didn't

We are supposed to download, but something went wrong

We are supposed to download, and the download completed successfully

The caller does not care about the distinction between 1 and 2, and the function returns undefined for both of these cases. We could use a rejected promise for both 1 and 2, and it would work, but the control flow would be a bit strange—it is like throwing an exception (and then catch it gracefully) to skip a feature that is disabled by flag.

An alternative is return undefined for 1 and a rejected promise for 2. That avoids the aforementioned awkwardness, at the cost of making the caller more complicated, having to check for both undefined and add a catch block and do the same thing in both cases.

I think the current implementation is better than these two alternatives, but I can be persuaded otherwise.

You could use a rejected promise with different values for 1 and 2 respectively that the caller can then check for (these can be arbitrary values). It is indeed a bit odd to handle such cases with catch, although you could have something as simple as .catch(console.debug) (or similar) after the call to downloadOverlayBaseDatabaseFromCache if there are appropriate messages given to throw.

Alternatively, the feature check could be moved out from downloadOverlayBaseDatabaseFromCache to the caller.

I don't feel strongly about this though and am be happy for it to stay as-is.

In that case, let's keep the current code.

mbg · 2025-07-18T10:46:06Z

src/overlay-database-utils.ts

+    logger.warning(
+      "Filesystem error while accessing downloaded overlay-base database",
+    );


tryGetFolderBytes will log a warning (since you don't set quiet=true) before returning undefined, so this will always lead to two warnings for the same problem, which I believe will show up as separate workflow annotations.

I'd suggest either removing this warning or setting quiet=true in the tryGetFolderBytes call.

I think both messages are worth preserving because they tell the user different things.

The warning from tryGetFolderBytes explains what went wrong. This is important if the user were to fix the underlying technical problem.

The warning here in downloadOverlayBaseDatabaseFromCache explains the consequence of the failure (that the downloaded overlay-base database cannot be accessed) so that the user can assess the severity and impact of the problem.

That said, I agree that they do not both have to be warnings. I downgraded this message to info.

Perhaps it would be reasonable for tryGetFolderBytes to return the error rather than undefined and then leave it up to the caller to decide what to do with that (e.g. include it in a combined log message with the consequence + underlying problem).

Not something you need to act on if you're happy to downgrade this to an info-level message.

Thanks. I already downgraded the log from warning and info, and I am happy with that change.

mbg · 2025-07-18T10:47:50Z

src/overlay-database-utils.ts

+    // unable to determine the size of the database. Rather, it is that we
+    // encountered a filesystem error while accessing the database, which
+    // indicates that an overlay analysis will likely fail.
+    logger.warning("Downloaded overlay-base database is invalid");


Similarly here: it would be better to merge this into the previous .warning call if you want both messages. Otherwise, maybe remove the previous one or make it a .debug level message?

I think this extra warning call is an oversight on my part, and it should not be here. I have removed it.

Thanks for catching this!

cklin marked this pull request as ready for review July 17, 2025 18:13

Copilot AI review requested due to automatic review settings July 17, 2025 18:13

cklin requested a review from a team as a code owner July 17, 2025 18:13

Copilot AI reviewed Jul 17, 2025

View reviewed changes

src/overlay-database-utils.ts Outdated Show resolved Hide resolved

cklin force-pushed the cklin/overlay-telemetry branch from 76a1b2d to 2b3bae6 Compare July 17, 2025 18:21

cklin requested a review from mbg July 17, 2025 19:03

mbg reviewed Jul 18, 2025

View reviewed changes

cklin added 4 commits July 18, 2025 07:18

Overlay: report overlay analysis mode

19075c4

Overlay: report overlay-base database stats

e37b293

Add downloadOverlayBaseDatabaseFromCache tests

c3bbcab

build: refresh js files

39b0524

cklin force-pushed the cklin/overlay-telemetry branch from 2b3bae6 to 39b0524 Compare July 18, 2025 15:02

Overlay: report telemetry #2975

Are you sure you want to change the base?

Overlay: report telemetry #2975

Conversation

cklin commented Jul 17, 2025

Merge / deployment checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

mbg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.