Skip to content

feat: use proto streams to increase maximum module files payload #18268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from

Conversation

Emyrk
Copy link
Member

@Emyrk Emyrk commented Jun 6, 2025

Increase maximum module files payload using protobuf streams

Problem

Coder has a 4MB protobuf message size limit that prevents templates with large Terraform modules from being processed. When dynamic parameters require module files from ~/.terraform/modules that exceed this limit, template creation fails.

Solution

This PR implements protobuf streaming to handle large module files by:

  1. Streaming large payloads: When module files exceed the 4MB limit, they're streamed in chunks using a new UploadFile RPC method
  2. Database storage: Streamed files are stored in the database and referenced by hash for deduplication
  3. Backward compatibility: Small module files continue using the existing direct payload method

Key Changes

  • New protobuf streams: Added UploadFile RPC and DataUpload/ChunkPiece message types
  • Smart fallback: Automatically detects oversized payloads and switches to streaming
  • File deduplication: Module files are deduplicated per template version using content hashes
  • Testing: Added test coverage for the streaming upload functionality.

Architecture

  • Coderd ↔️ Provisionerd: Uses new streaming RPC when payload exceeds limits.

    // UploadFile streams files to be inserted into the database.
    // The file upload_type should be used to determine how to handle the file.
    rpc UploadFile(stream UploadFileRequest) returns (Empty);

  • Provisionerd ↔️ Provisioner: Extends existing stream with new message types for in-memory communication

This change enables templates with arbitrarily large Terraform modules while maintaining performance through deduplication and preserving the existing fast path for smaller payloads.

TODO:

  • Bump protobuf versions
  • Implement backwards compatible?
    • Do not upload large files to old coderds
  • Verify forward compatible
  • Add e2e test with a large module

Things that I wish I could change

  • We have 2 proto files, and they have overlapping types. Ideally DataUpload and Chunk would be defined in 1 place and have their types reusable in both contexts 😢.

@Emyrk Emyrk force-pushed the stevenmasley/4mb branch from 4807dbc to 124d366 Compare June 6, 2025 14:19
@Emyrk Emyrk changed the title Stevenmasley/4mb feat: increase maximum module files payload using proto streams Jun 6, 2025
@Emyrk Emyrk changed the title feat: increase maximum module files payload using proto streams feat: use proto streams to increase maximum module files payload Jun 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy