Management APIOverview

Management API

The Management API is a Go REST + WebSocket service that powers the Datafly Signal management interface. It provides programmatic access to all platform configuration: pipelines, integrations (catalog, templates, revisions), transformations, users, and audit logs.

Base URL

All endpoints are prefixed with:

https://{cluster}/api/v1/admin

In local development, the Management API runs on port 8084:

http://localhost:8084/v1/admin

Authentication

All requests must include a valid JWT access token in the Authorization header:

Authorization: Bearer {access_token}

Obtain tokens via the authentication endpoints. Access tokens expire after 15 minutes; use the refresh endpoint to obtain new ones.

Content Type

All request and response bodies use JSON:

Content-Type: application/json

Rate Limiting

The API enforces a rate limit of 100 requests per minute per authenticated user. When the limit is exceeded, the API returns a 429 Too Many Requests response with a Retry-After header indicating when the client can retry.

{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Rate limit exceeded. Retry after 32 seconds."
  }
}

Pagination

List endpoints use cursor-based pagination. Pass a cursor and limit as query parameters:

GET /v1/admin/sources?cursor=eyJpZCI6MTAwfQ&limit=50
ParameterTypeDefaultDescription
cursorstring—Opaque cursor from a previous response’s next_cursor field
limitinteger50Number of items per page (max 100)

Paginated responses include a next_cursor field. When next_cursor is null, there are no more results.

{
  "data": [ ... ],
  "next_cursor": "eyJpZCI6MTUwfQ",
  "has_more": true
}

Error Format

All error responses follow a consistent format:

{
  "error": "not_found",
  "message": "pipeline not found",
  "request_id": "8b0c3e4a-5f1d-4c2e-9a7b-2d6e1f0c9a31"
}
FieldTypeDescription
errorstringMachine-readable error code
messagestringHuman-readable description. It never contains internal detail such as stack traces or database errors.
request_idstringCorrelation ID, also sent as the X-Request-ID response header. Quote it to support: it identifies the request in the server logs.

HTTP Status Codes

StatusMeaning
200Success
201Resource created
204Success, no content (delete operations)
400Bad request (invalid JSON, missing fields)
401Unauthorized (missing or invalid token)
403Forbidden (insufficient permissions)
404Resource not found
409Conflict (duplicate resource)
422Unprocessable entity (validation error)
429Rate limited
500Internal server error

Sections

SectionDescription
AuthenticationLogin, token refresh, JWT format
PipelinesManage data collection pipelines, pipeline keys, JS collector builds
Integration CatalogBrowse and install from the integration library
Integration TemplatesManage org-level integration templates and immutable revisions
Pipeline IntegrationsAssign integrations to pipelines at specific revisions
TransformationsDefine and test event transformations
Data LayerOrg-wide data layer configuration
Users & RBACUser management and role-based access
Audit LogsQuery the audit trail

The Management API also exposes a WebSocket endpoint at ws://{cluster}/v1/admin/debug/live for real-time event debugging. Connect with a valid JWT to stream events as they flow through the pipeline.