SynopticAI — Collaborative AI Agents for Fault Detection

Concept

SynopticAI is an API-first SaaS that brings collaborative AI reasoning to the SRE/DevOps incident response workflow. Instead of a single model making a diagnosis from noisy telemetry, SynopticAI deploys a pool of specialized agents — each an expert in one signal domain — and a CriticAgent that arbitrates their hypotheses into a single, high-confidence root-cause report.

This mirrors how elite SRE teams actually work: a logs specialist, a metrics specialist, and a senior engineer who synthesizes both views. SynopticAI encodes that pattern into an asynchronous, scalable microservice.

---

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│  Client (CI/CD pipeline, Datadog webhook, Prometheus AlertManager)  │
└────────────────────────────┬────────────────────────────────────────┘
                             │ POST /api/v1/ingest  [Bearer JWT]
                             ▼
┌────────────────────────────────────────────────────────────────────┐
│  TelemetryIngestionController                                      │
│  · Validates JWT via JwtAuthenticationFilter                       │
│  · Returns HTTP 202 immediately (fire-and-forget)                  │
│  · Calls AnalysisOrchestrator.process() → @Async virtual thread   │
└───────────────────────────┬────────────────────────────────────────┘
                            │ virtual thread (Executors.newVirtualThreadPerTaskExecutor)
                            ▼
┌───────────────────────────────────────────────────────────────────┐
│  AnalysisOrchestrator                                             │
│                                                                   │
│  ┌───────────────┐   virtual   ┌────────────────┐                │
│  │   LogAgent    │◄──thread────│ TelemetryEvent │                │
│  │  hypotheses   │             │    batch       │                │
│  └───────┬───────┘             └───────┬────────┘                │
│          │                            │  virtual thread           │
│  ┌───────▼───────┐                   ▼                           │
│  │  MetricAgent  │            ┌──────────────┐                   │
│  │  hypotheses   │            │  MetricAgent │                   │
│  └───────┬───────┘            └──────────────┘                   │
│          │                                                        │
│          └──────────────────┐                                     │
│                             ▼                                     │
│                    ┌────────────────┐                             │
│                    │  CriticAgent   │  synthesizes root cause     │
│                    └───────┬────────┘                             │
│                            │ IncidentReport                       │
│              ┌─────────────┴──────────────┐                      │
│              ▼                            ▼                       │
│     PostgreSQL (JPA)          AlertNotificationService            │
│     IncidentReportEntity      · Slack webhook (virtual thread)    │
│                               · PagerDuty Events API v2           │
└───────────────────────────────────────────────────────────────────┘
                            │
                            ▼ GET /api/v1/incidents/{id}  [Bearer JWT]
              ┌─────────────────────────────┐
              │  IncidentReportController   │
              │  Returns IncidentReport JSON│
              └─────────────────────────────┘

Key Design Decisions

Decision	Rationale
Virtual Threads for ingestion	Java 25 virtual threads allow thousands of concurrent telemetry batches with near-zero thread overhead — critical for high-throughput SRE environments.
Parallel agent execution	LogAgent and MetricAgent run on separate virtual threads within the orchestrator, halving latency versus sequential execution.
CriticAgent arbitration	Reduces false positives by requiring corroboration across agents before escalating to CRITICAL status.
Jackson 3 Records	DTOs are immutable Java Records; Jackson 3 (tools.jackson.*) handles native record deserialization.
JWT stateless auth	No session state — each API call is independently authenticated, enabling horizontal scaling.
Text column for hypotheses JSON	Avoids complex relational mapping while preserving full fidelity of the agent output for future ML training data.

---

Domain Model

TelemetryEvent (Record)
  ├── source:    String          // "prometheus" | "datadog" | "otel-collector"
  ├── type:      String          // "metric" | "log" | "trace"
  ├── payload:   Map<String,Object>  // raw telemetry key-value pairs
  └── timestamp: Instant

FaultHypothesis (Record)
  ├── agentId:     String        // "log-agent" | "metric-agent"
  ├── description: String        // human-readable fault narrative
  ├── confidence:  double        // 0.0 (uncertain) → 1.0 (certain)
  └── evidence:    List<String>  // supporting data points

IncidentReport (Record)
  ├── id:          UUID
  ├── rootCause:   String        // synthesized narrative from CriticAgent
  ├── hypotheses:  List<FaultHypothesis>
  ├── confidence:  double        // aggregated confidence
  ├── timestamp:   Instant
  └── status:      String        // HEALTHY | WATCHLIST | WARNING | CRITICAL

---

API Endpoints

All endpoints (except Actuator health) require:

Authorization: Bearer <HS256-signed-JWT>

POST /api/v1/ingest

Ingest a batch of telemetry events for asynchronous analysis.

Request Body:

[
  {
    "source": "prometheus",
    "type": "metric",
    "payload": {
      "metric_name": "cpu_usage_percent",
      "value": 95.2,
      "host": "api-pod-3"
    },
    "timestamp": "2026-04-08T10:00:00Z"
  },
  {
    "source": "loki",
    "type": "log",
    "payload": {
      "level": "ERROR",
      "message": "NullPointerException in OrderService.processPayment()",
      "service": "order-service"
    }
  }
]

Response: 202 Accepted

{
  "status": "accepted",
  "correlationId": "a3f7b2c1-...",
  "eventsReceived": 2,
  "message": "Telemetry batch accepted. Collaborative analysis started."
}

---

GET /api/v1/incidents

Returns all persisted incident reports.

Response: 200 OK

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "rootCause": "[PRIMARY — metric-agent, confidence=85%] CPU utilization exceeded 90%... | Corroborated by: log-agent (72%)",
    "hypotheses": [...],
    "confidence": 0.785,
    "timestamp": "2026-04-08T10:00:03Z",
    "status": "CRITICAL"
  }
]

---

GET /api/v1/incidents/{id}

Retrieves a specific incident report by UUID.

Response: %%INLINE5%% (or %%INLINE6%%)

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "rootCause": "[PRIMARY — metric-agent, confidence=85%] CPU utilization exceeded 90% threshold...",
  "hypotheses": [
    {
      "agentId": "metric-agent",
      "description": "CPU utilization exceeded 90.0% threshold on 3 event(s)...",
      "confidence": 0.85,
      "evidence": ["cpu=95.2% on api-pod-3", "cpu=91.0% on api-pod-7"]
    },
    {
      "agentId": "log-agent",
      "description": "Detected 4 error-level log entries across 2 source(s)...",
      "confidence": 0.72,
      "evidence": ["[order-service] ERROR: NullPointerException in OrderService"]
    }
  ],
  "confidence": 0.785,
  "timestamp": "2026-04-08T10:00:03Z",
  "status": "CRITICAL"
}

---

GET /actuator/health

Public health check — no authentication required.

---

Incident Status Levels

Status	Meaning	Alert Sent?
HEALTHY	No anomalous signals detected	No
WATCHLIST	Low-confidence signals present; monitoring advised	No
WARNING	Significant anomalies detected (confidence 50–74%)	Yes
CRITICAL	High-confidence root cause identified (≥75%)	Yes

---

Supported Telemetry Payload Formats

Metric Event

{
  "source": "prometheus",
  "type": "metric",
  "payload": {
    "metric_name": "cpu_usage_percent",  // must contain: "cpu" | "memory" | "error_rate"
    "value": 95.2                         // numeric threshold comparison
  }
}

Log Event

{
  "source": "fluentd",
  "type": "log",
  "payload": {
    "level": "ERROR",          // ERROR or FATAL triggers analysis
    "message": "..."           // "exception" or "error" in message also triggers
  }
}

---

Security

JWT tokens must be HS256-signed with the secret configured via:

synoptic:
  security:
    jwt-secret: your-production-secret-here

Generating a test token (online tool):

1. Header: { "alg": "HS256", "typ": "JWT" }
2. Payload: { "sub": "api-client-1", "exp": 9999999999 }
3. Secret: synoptic-ai-default-dev-secret-2026! (default dev secret)

---

Outbound Alerting Configuration

synoptic:
  alerts:
    slack:
      webhook-url: https://hooks.slack.com/services/T00000/B00000/xxxxxxxx
    pagerduty:
      integration-key: your-pagerduty-routing-key

Alert channels are silently skipped if not configured (safe default for local dev).

---

How to Run

Prerequisites

• Java 25+
• PostgreSQL 15+ running locally (or via Docker)
• Maven 3.9+

1. Start PostgreSQL

docker run -d \
  --name synoptic-db \
  -e POSTGRES_DB=synopticai \
  -e POSTGRES_USER=synoptic \
  -e POSTGRES_PASSWORD=synoptic \
  -p 5432:5432 \
  postgres:16

2. Configure application.yml

Add to src/main/resources/application.yml:

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/synopticai
    username: synoptic
    password: synoptic
  jpa:
    hibernate:
      ddl-auto: create-drop   # use 'validate' in production
    show-sql: false

synoptic:
  security:
    jwt-secret: change-me-in-production
  alerts:
    slack:
      webhook-url: ""          # optional
    pagerduty:
      integration-key: ""      # optional

3. Build and Run

cd solutions/2026-04-08-synoptic-ai-api
mvn clean package -DskipTests
java -jar target/synoptic-ai-api-0.0.1-SNAPSHOT.jar

4. Test the API

# Ingest mixed telemetry
curl -X POST http://localhost:8080/api/v1/ingest \
  -H "Authorization: Bearer <your-jwt>" \
  -H "Content-Type: application/json" \
  -d '[
    {"source":"prometheus","type":"metric","payload":{"metric_name":"cpu_usage_percent","value":96.0}},
    {"source":"loki","type":"log","payload":{"level":"ERROR","message":"Connection refused to payment-service"}}
  ]'

# List all incident reports
curl http://localhost:8080/api/v1/incidents \
  -H "Authorization: Bearer <your-jwt>"

# Get specific report
curl http://localhost:8080/api/v1/incidents/<uuid> \
  -H "Authorization: Bearer <your-jwt>"

# Health check (no auth required)
curl http://localhost:8080/actuator/health

---

Business Analysis

Problem & Market Opportunity

Modern cloud-native architectures (Kubernetes + microservices) generate terabytes of telemetry daily. SRE teams spend an estimated 41% of their time on toil, with incident triage being the largest contributor. The global AIOps market is projected to reach $40B by 2026.

Current tools (Datadog, New Relic, PagerDuty) excel at collecting telemetry but are weak at explaining incidents — they surface anomalies but leave root-cause correlation to humans.

Competitive Moat

SynopticAI's collaborative critic pattern is novel:

Approach	Current Tools	SynopticAI
Alert generation	Rule-based thresholds	Agent-generated hypotheses
Root cause	Manual correlation	CriticAgent synthesis
False positive rate	High (alert fatigue)	Reduced via confidence filtering
Explainability	Low	Full hypothesis chain

Monetization

Tier	Target	Price
Starter	Small teams, <10K events/day	$99/month
Growth	Mid-size SRE teams, <1M events/day	$999/month
Enterprise	Full integration + SLA + custom agents	Custom

Go-to-Market

1. Developer-led growth — open-source LogAgent + MetricAgent; sell CriticAgent + persistence + alerting as SaaS.
2. Integration-first — Datadog/Prometheus webhook compatibility means zero friction adoption.
3. Land-and-expand — Start with one service's telemetry; expand to full fleet.

---

References

• Collaborative AI Agents Survey — arxiv.org/list/cs.AI
• OpenTelemetry Specification
• Prometheus Remote Write
• PagerDuty Events API v2
• Slack Incoming Webhooks
• Spring Boot 4 Virtual Threads
• Jackson 3 Migration Guide