feat: Stronger incident-level ticketing integration (#4981) #4995
Conversation
…eephq#4981) Implements Phase 1 of incident-level ticketing integration: Backend Infrastructure: - Add FormFieldSchema and IncidentFormSchema Pydantic models with full validation - Create incident_form_schema database table (one schema per tenant) - Add API endpoints: GET/POST/DELETE /incidents/form-schema - Support 7 field types: text, textarea, select, checkbox, radio, number, date - Comprehensive field validation and error handling This enables tenants to define custom fields during incident creation that become incident enrichments, which workflows can then use for automatic ticket creation in systems like Jira. Next: Frontend form rendering and Create Incident modal integration
…#4981) - Add DynamicIncidentForm component that renders custom fields based on tenant schema - Integrate dynamic form into CreateOrUpdateIncidentForm modal - Support 7 field types: text, textarea, select, radio, checkbox, number, date - Add comprehensive field validation with error display - Use two-step incident creation: create incident then enrich with custom fields - Handle loading states and graceful fallback when no schema configured
…eephq#4981) - Add comprehensive Jira ticket creation workflow example - Add incident enrichment workflow showing custom field usage - Add detailed setup guide with API examples and best practices - Document all 7 supported field types with JSON schemas - Include workflow integration patterns and troubleshooting guide
- Create TicketLink component with url and id props - Extract ticket ID from URL path as fallback display text - Add Ticket column to incidents table showing clickable links - Use ticket_url for link and ticket_id for display text from enrichments - Render empty element when no valid URL or ID available - Update documentation with ticket link display section - Update workflow examples to use standard ticket_id/ticket_url enrichments
- Add join to AlertEnrichment table for incident enrichments using incident ID - Modify query to include enrichment data in select statement - Process enrichment data from query results and populate incident._enrichments - Use table aliases to distinguish between incident and alert enrichments - Update field mapping configuration to support both enrichment sources
- Fix form schema fields storage as proper JSON by changing database model type from List[FormFieldSchema] to List[dict]
- Fix Pydantic validator execution order by moving default_value field after options field
- Fix frontend authentication by replacing raw fetch() with authenticated useApi() hook
- Add granular permissions for form schema operations (read/write/delete:incidents-form-schema)
- Fix route ordering to prevent form-schema endpoints being matched by /{incident_id} route
- Add proper session dependency injection for form schema endpoints
- Add field conversion logic to handle mixed FormFieldSchema objects and dicts in responses
- Add detailed ticketing integration guide covering special enrichment fields (ticket_url, ticket_id) - Document incident form schema system for extensible incident creation forms - Include API endpoints, permissions, and security considerations - Provide complete workflow examples for Jira, ServiceNow, and multi-system integrations - Add form schema design best practices and troubleshooting guidance - Update incidents navigation to include ticketing integration documentation - Add ticketing integration reference to incidents overview
|
@ns-rboyd is attempting to deploy a commit to the KeepHQ Team on Vercel. A member of the Team first needs to authorize it. |
🚨 BugBot couldn't runPull requests from forked repositories are not yet supported (requestId: serverGenReqId_2cdf6140-dc64-4b55-bfa5-d4c8886ff1d5). |
|
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
PR Summary
Implements comprehensive incident-level ticketing integration with dynamic form schemas and automated ticket linking capabilities.
- Added new
DynamicIncidentFormandTicketLinkcomponents in/keep-uifor tenant-specific form fields and clickable ticket links - Created
incident_form_schematable and models with concerning PRIMARY KEY constraint on tenant_id, limiting to one schema per tenant - Introduced automated ticket creation workflow in
examples/workflows/jira_incident_ticketing.ymlwith enrichment-based Jira integration - Security concern: Field validation and XSS prevention should be reviewed in
DynamicFormField.tsx - API design issue: Incident form schema endpoints in
keep/api/routes/incidents.pyneed rate limiting protection
18 files reviewed, 11 comments
Edit PR Review Bot Settings | Greptile
| Expected Resolution: {{ incident.enrichments.resolution_date }} | ||
|
|
||
| Description: | ||
| {{ incident.enrichments.description or incident.user_summary }} |
There was a problem hiding this comment.
logic: incident.user_summary is not a standard field. Should this be incident.description?
keep-ui/features/incidents/create-or-update-incident/ui/create-or-update-incident-form.tsx
Show resolved
Hide resolved
| ### Delete Schema | ||
| ```bash | ||
| curl -X DELETE "${KEEP_API_URL}/incidents/form-schema" |
There was a problem hiding this comment.
style: DELETE endpoint missing authentication headers in example. Add security headers to all API examples.
|
|
||
| from datetime import datetime | ||
| from enum import Enum | ||
| from typing import Any, List, Optional, Union |
There was a problem hiding this comment.
style: Union is imported but never used in the code
| from typing import Any, List, Optional, Union | |
| from typing import Any, List, Optional |
keep/api/routes/incidents.py
Outdated
| # Convert fields to dicts for the response to avoid Pydantic validation issues | ||
| fields_as_dicts = [] | ||
| for field in schema.fields: | ||
| if isinstance(field, FormFieldSchema): | ||
| fields_as_dicts.append(field.dict()) | ||
| elif isinstance(field, dict): | ||
| fields_as_dicts.append(field) | ||
| else: | ||
| # Handle case where it might be some other type | ||
| try: | ||
| if hasattr(field, 'dict') and callable(getattr(field, 'dict')): | ||
| fields_as_dicts.append(field.dict()) | ||
| elif hasattr(field, '__dict__'): | ||
| fields_as_dicts.append(field.__dict__) | ||
| else: | ||
| # Skip invalid fields rather than crashing | ||
| logger.warning(f"Skipping unsupported field type: {type(field)}") | ||
| continue | ||
| except Exception as e: | ||
| logger.error(f"Failed to convert field to dict: {e}") | ||
| continue |
There was a problem hiding this comment.
style: Field conversion logic duplicated 3 times in this file. Consider extracting to a reusable helper function to reduce code duplication
| class IncidentFormSchema(SQLModel, table=True): | ||
| """Single form schema per tenant for incident creation""" | ||
| __tablename__ = "incident_form_schema" | ||
|
|
||
| tenant_id: str = Field(primary_key=True, foreign_key="tenant.id") | ||
| name: str = Field(sa_column=Column(TEXT)) | ||
| description: Optional[str] = Field(sa_column=Column(TEXT)) | ||
| fields: List[FormFieldSchema] = Field(sa_column=Column(JSON)) | ||
| created_by: str = Field(sa_column=Column(TEXT)) | ||
| created_at: datetime = Field(default_factory=datetime.utcnow) | ||
| updated_at: datetime = Field(default_factory=datetime.utcnow) | ||
| is_active: bool = Field(default=True) |
There was a problem hiding this comment.
logic: No declared imports for SQLModel, Column, TEXT, datetime. This will cause runtime errors.
| class IncidentFormSchema(SQLModel, table=True): | |
| """Single form schema per tenant for incident creation""" | |
| __tablename__ = "incident_form_schema" | |
| tenant_id: str = Field(primary_key=True, foreign_key="tenant.id") | |
| name: str = Field(sa_column=Column(TEXT)) | |
| description: Optional[str] = Field(sa_column=Column(TEXT)) | |
| fields: List[FormFieldSchema] = Field(sa_column=Column(JSON)) | |
| created_by: str = Field(sa_column=Column(TEXT)) | |
| created_at: datetime = Field(default_factory=datetime.utcnow) | |
| updated_at: datetime = Field(default_factory=datetime.utcnow) | |
| is_active: bool = Field(default=True) | |
| from sqlmodel import SQLModel, Field, Column, TEXT, JSON | |
| from datetime import datetime | |
| class IncidentFormSchema(SQLModel, table=True): | |
| """Single form schema per tenant for incident creation""" | |
| __tablename__ = "incident_form_schema" | |
| tenant_id: str = Field(primary_key=True, foreign_key="tenant.id") | |
| name: str = Field(sa_column=Column(TEXT)) | |
| description: Optional[str] = Field(sa_column=Column(TEXT)) | |
| fields: List[FormFieldSchema] = Field(sa_column=Column(JSON)) | |
| created_by: str = Field(sa_column=Column(TEXT)) | |
| created_at: datetime = Field(default_factory=datetime.utcnow) | |
| updated_at: datetime = Field(default_factory=datetime.utcnow) | |
| is_active: bool = Field(default=True) |
| ["tenant_id"], | ||
| ["tenant.id"], | ||
| ), | ||
| sa.PrimaryKeyConstraint("tenant_id"), |
There was a problem hiding this comment.
logic: Using tenant_id as the sole primary key prevents multiple form schemas per tenant. Consider adding a composite key with name or using a separate id field.
| - type: incident | ||
| filters: | ||
| - key: "jira_project" | ||
| value: ".*" # Any value indicates form was filled |
There was a problem hiding this comment.
logic: Loose regex pattern .* matches empty strings. Use ^.+$ to ensure non-empty values
| type: "text" | "textarea" | "select" | "checkbox" | "radio" | "number" | "date"; | ||
| description?: string; | ||
| required: boolean; | ||
| default_value?: any; |
There was a problem hiding this comment.
style: default_value should be typed more strictly than 'any' to prevent runtime type errors
- Add UUID primary key to incident_form_schema table - Update database model to allow multiple schemas with unique (tenant_id, name) constraint - Modify API endpoints to handle schema CRUD operations by ID - Update migration to reflect new table structure - Add schema_id to response models for proper identification
- Remove unreachable code in DynamicIncidentForm.tsx - Add schema ID field to TypeScript interfaces - Update deleteSchema to accept schemaId parameter - Handle array response from form-schema endpoint
- Convert component to forwardRef pattern for proper method exposure - Add useImperativeHandle to expose getFieldErrors method - Fix issue where method was defined after return statement - Add proper TypeScript types for ref interface
There was a problem hiding this comment.
PR Summary
Enhances incident ticketing integration with tenant-specific form schemas and UI improvements.
- Added tenant isolation in
keep/api/core/incidents.pywith proper enrichment data joins for secure multi-tenant ticket handling - Implemented custom field validation and type safety in
DynamicIncidentForm.tsxto prevent invalid enrichment data - Added schema management hooks in
useIncidentFormSchema.tsfor proper form data caching and tenant-level CRUD operations - Added comprehensive error handling in
incident_form_schema.pywith proper validation rules for field types and constraints - Added new 'ticketing-integration.mdx' documentation with detailed integration guides and workflow examples
18 files reviewed, 3 comments
Edit PR Review Bot Settings | Greptile
keep/api/core/incidents.py
Outdated
| from sqlalchemy import literal_column | ||
| built_query_result = __build_base_incident_query( | ||
| tenant_id=tenant_id, | ||
| cel=cel, | ||
| select_args=[Incident], | ||
| select_args=[Incident, literal_column("incident_enrichment.enrichments").label("incident_enrichments")], | ||
| ) |
There was a problem hiding this comment.
style: Import statement added within function body. Move from sqlalchemy import literal_column to top level imports.
| from sqlalchemy import literal_column | |
| built_query_result = __build_base_incident_query( | |
| tenant_id=tenant_id, | |
| cel=cel, | |
| select_args=[Incident], | |
| select_args=[Incident, literal_column("incident_enrichment.enrichments").label("incident_enrichments")], | |
| ) | |
| from sqlalchemy import literal_column | |
| built_query_result = __build_base_incident_query( | |
| tenant_id=tenant_id, | |
| cel=cel, | |
| select_args=[Incident, literal_column("incident_enrichment.enrichments").label("incident_enrichments")], | |
| ) |
|
@ns-rboyd any news with this PR? :) |
|
@shahargl working through some playwright tests, hopefully close. |
|
@ns-rboyd cool 👑 note that you have some conflicts so it may be good point to merge from main. |
- Remove unused UUIDType import from incidents.py - Remove unused literal_column import from incidents.py - Remove unused FormFieldSchema import from incident_form_schema.py - Remove unused Union import from incident_form_schema.py - Remove unused IntegrityError import from incidents.py
- Replace server_default with default_factory pattern - Follows established SQLModel datetime field conventions - Fixes SQLAlchemy CompileError with NullType()
- Replace TEXT columns with AutoString in migration - Remove custom TEXT column definitions from model - Follows pattern used by other models with unique constraints - Fixes MySQL error: BLOB/TEXT column 'name' used in key specification
|
@ns-rboyd any news? happy to help if you need |
- Use Tremor DatePicker component for date fields - Add incident_date field to E2E test schema - Update Playwright tests to interact with date picker - Add verification for date enrichments in incident details - All E2E tests passing with date field functionality
- Remove obsolete E2E test files that were causing CI failures - Remove test runner scripts that are no longer needed - Remove docker-compose-test.yml - Remove all playwright test artifacts (HTML, PNG, console.txt files) - Keep only test_incident_form_schema_complete.py as the active E2E test
- Add delete_incident() function to clean up test incidents via API - Update test_create_incident_with_dynamic_fields to capture incident ID and delete it - Update test_required_field_validation to capture incident ID if created - Ensure tests clean up both incidents and form schemas in finally blocks - Remove .swp file from repository - Remove debug PNG files (debug_1_incidents_page.png, debug_2_after_create_click.png)
- Add delete_incident() function to clean up test incidents via API - Update test_create_incident_with_dynamic_fields to capture incident ID and delete it - Update test_required_field_validation to capture incident ID if created - Ensure tests clean up both incidents and form schemas in finally blocks
- Remove .swp file from repository - Add .swp to .gitignore to prevent future commits
…-ticketing-integration
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4995 +/- ##
==========================================
- Coverage 46.18% 45.95% -0.23%
==========================================
Files 173 175 +2
Lines 17970 18236 +266
==========================================
+ Hits 8299 8380 +81
- Misses 9671 9856 +185 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@talboren I need to take a closer look, but it seems I clobbered existing e2e tests in https://github.com/keephq/keep/actions/runs/15913491870/job/44886355288?pr=4995 .. must be my changes to model or migration? Have to step away now but if you get a chance to take a look that would be great. After that, still need to satisfy cov, docs, and workflow tests. |
- Fix inconsistent alias reference in field mapping configuration - Changed 'incident_enrichment.enrichments' to 'incidentenrichment.enrichments' to match the SQLAlchemy alias name - Fixes facet query errors in E2E tests for incident filtering
|
@ns-rboyd your PR is not up to date with main, can I update it? |
happy to help with anything if I can, just ping me over Slack if needed |
Signed-off-by: Rob Boyd <rboyd@netskope.com>
…ltiple provider snippets
- Add required 'name' field to both workflow definitions - Fix incident trigger format to use 'events' instead of 'filters' - Convert string conditions to proper array format with assert type - Add required provider config references - Ensure workflows properly demonstrate incident-level ticketing features: - Use form schema enrichments for ticket creation - Enrich incidents with ticket_id and ticket_url for link display - Show conditional ticket creation based on enrichments - Demonstrate dynamic project and priority mapping
…nt workflow The workflow validation schema expects conditional execution to use the 'if' field directly on steps, not the 'condition' array format. This aligns with the pattern used in other example workflows.
Auto-generated documentation now includes references to the new incident ticketing workflow examples that demonstrate the incident-level ticketing integration features.
|
@skynetigor this ^ failing unit test also flakey I think |
There was a problem hiding this comment.
Bug: Undefined Alias and Incorrect Field Mapping
The incident_enrichment alias is used in the SQL query join but is not defined within the function scope, causing a NameError. Additionally, the field mapping configuration contains two errors: it incorrectly references JSON(incidentenrichment.enrichments).* (missing an underscore) and uses JSON(alert_enrichment.enrichments).* which should be JSON(alert_enrichment.c.enrichments).* for proper SQLAlchemy table alias column access.
keep/api/core/incidents.py#L137-L260
keep/keep/api/core/incidents.py
Lines 137 to 260 in 70dd33b
Bug: Type Mismatch Causes Data Loss
The IncidentFormSchema.fields attribute has a type mismatch between the database (List[dict]) and API Pydantic DTOs (List[FormFieldSchema]). This forces duplicated, error-prone conversion logic across multiple endpoints that silently skips unsupported field types with a warning, leading to potential data loss and making the API fragile.
keep/api/routes/incidents.py#L402-L499
keep/keep/api/routes/incidents.py
Lines 402 to 499 in 70dd33b
Was this report helpful? Give feedback by reacting with 👍 or 👎
|
On the #5136 merge (great @DolevNe !) I think we now need to make use of getTicketViewUrl or findLinkedTicket to render the ticket links on the main /incidents page table. Should there be a single function that returns the first (ticket_id, ticket_url) for any provider? Anything else to consider that would conflict? I think besides the links in the Ticket column on the /incidents page this PR is 90% about dynamic form schema for the incident creation form, where form fields <-> enrichments and these enrichments can be used during workflow run (on incident create) to create the ticket with any ticketing providers. Just for some more context: one of the main drivers here is that on our existing home-grown event console, our teams used a dropdown list to select a target Jira project for the ticket to be created under. So we wanted to recreate that pattern here, but the rest was low enough hanging fruit and potentially pretty useful to other folks. Let me know any other suggestions/ideas |
Closes #4981
📑 Description
Based on the user's custom enrichments, ticketing providers can easily now create tickets using the incident's enrichments, enabling use-cases such as Jira creating the ticket in a particular Jira Project (which the incident creator has chosen from a dropdown list). The workflow can add the
ticket_urlandticket_idas the final action so that the ticket is linked from the Incidents table.Some docs at https://github.com/ns-rboyd/keep/blob/4981-incident-level-ticketing-integration/docs/incidents/ticketing-integration.mdx
ℹ Additional Information
Hyperlink Ticket column:
Incident form schemas:
