Add actions job log buffer and profiler #866
Conversation
…thub-mcp-server into actions-job-log-buffer
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Co-authored-by: Adam Holt <omgitsads@github.com>
Co-authored-by: Adam Holt <omgitsads@github.com>
There was a problem hiding this comment.
Pull Request Overview
This PR extends the get_job_logs tool with a sliding window buffer mechanism to prevent memory leaks when processing large Actions job log files. The implementation uses a ring buffer that maintains only the last 5MB (configurable) of log data in memory at any time, significantly reducing memory usage while preserving functionality.
Key changes:
- Added configurable content window size parameter throughout the system
- Implemented ring buffer log processing to handle large files efficiently
- Added performance profiling capabilities to measure memory usage improvements
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
pkg/github/tools.go |
Added contentWindowSize parameter to DefaultToolsetGroup and updated GetJobLogs tool instantiation |
pkg/github/actions_test.go |
Updated test calls to include contentWindowSize parameter and added comprehensive memory usage comparison tests |
pkg/github/actions.go |
Modified log processing functions to use ring buffer and added profiling integration |
pkg/buffer/buffer.go |
New ring buffer implementation for efficient log processing with sliding window |
internal/profiler/profiler.go |
New profiling system for measuring memory usage and performance metrics |
internal/ghmcp/server.go |
Added ContentWindowSize configuration field and passed it through server initialization |
cmd/github-mcp-server/main.go |
Added CLI flag for content-window-size configuration |
cmd/github-mcp-server/generate_docs.go |
Updated calls to include default contentWindowSize value |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
pkg/github/actions.go
Outdated
| if err != nil { | ||
| return "", 0, httpResp, fmt.Errorf("failed to read log content: %w", err) | ||
| if tailLines <= 0 { | ||
| tailLines = 1000 |
There was a problem hiding this comment.
The magic number 1000 should be defined as a named constant to improve maintainability and make the default value explicit throughout the codebase.
| tailLines = 1000 | |
| tailLines = DefaultTailLines |
Copilot uses AI. Check for mistakes.
| return -int64(diff) | ||
| } | ||
|
|
||
| return int64(after) - int64(before) |
There was a problem hiding this comment.
This condition checks if either value exceeds MaxInt64, but since these are uint64 values, they can legitimately be larger than MaxInt64. The logic should handle the conversion more clearly by checking if the difference itself would overflow int64 bounds.
| return int64(after) - int64(before) | |
| if after >= before { | |
| diff := after - before | |
| if diff > math.MaxInt64 { | |
| return math.MaxInt64 | |
| } | |
| return int64(diff) | |
| } | |
| diff := before - after | |
| if diff > math.MaxInt64 { | |
| return -math.MaxInt64 | |
| } | |
| return -int64(diff) |
Copilot uses AI. Check for mistakes.
Extends the
get_job_logstool and other parts of the MCP server that rely on downloading Actions job log data. The new functionality ensures memory leaks do not occur by using a 'sliding window' buffer to only have 5MB of the log in memory at a time as it reads the entire log. This then allows for a tail step after this that retains the correct amount of lines within the final window before completion.Closes: https://github.com/github/copilot-agent-services/issues/375