CIP-137: Ceramic API

Simple Summary

An API for reading and writing events on Ceramic event streams.

Abstract

This CIP introduces a OpenAPI specification that can be used to subscribe to a number of streams inside of an interest range, and write new events to these streams. The API exposes two main ways of retrieving events. A simple method that uses the nodes local ordering of events to retrieve events by simply keeping track of a single vector client side. More advanced users can use a simplified version of the Recon protocol to retrieve events in their logical order.

Motivation

In order to enable services and databases other than ComposeDB to be built on top of Ceramic, an API to read and write directly on event streams is needed. This also enables developers to get direct access to the event streams underlaying ComposeDB.

Specification

This specification introduces a OpenAPI spec for interacting directly with event streams on a node.

GET /ceramic/subscribe/{streamid}

This endpoint is called to subscribe to new events in a given interest range. This endpoint uses the node’s local ordering of events within the given range, constructed in the order in which the node received the events. If you want the global ordering of events you can use the recon endpoint.

The range is normally defined as ceramic://*?<sort-key>=<sort-value>, where sort-key describes a field in the header of the InitEvent of a stream, and sort-value is the value of this field. For example to subscribe to all MIDs of a particular Model you can use ceramic://*?model=kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr.

This endpoint will respond as soon as the node knows about new events, given the offset, or when duration amount of seconds has passed. If the node already knows about a lot of new events, only limit amount of events will be returned.

The first time a client makes a request for a given interest range, the node creates a log file where eventId bytes are written as they are observed from the network (or locally). This log file is then used to allow the node to easily find events given an offset from the client.

Example:

/ceramic/subscribe/*?model=kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr&ctrlRangeStart=0x0000000000000000&ctrlRangeEnd=0xffffffffffffffff&offset=123&limit=5&duration=1000

Parameters:

• streamid - the stream to subscribe to (Normally `*` is used to subscribe to sets of streams)

• {sort-key} - the *sort-key* and *sort-value* to subscribe to (e.g. `model` and `kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr` respectively)
• ctrlRangeStart - hex encoded string describing the start of the range of controllers to sync (defaults to `0x0000000000000000`)
• ctrlRangeEnd - hex encoded string describing the end of the range of controllers to sync (defaults to `0xffffffffffffffff`)
• offset - the number of eventId bytes already consumed
• limit - the max number of events to return
• duration - the amount of time, in seconds, to wait for a response

Returns:

There are two different content types supported as return value, as specified by the Accept http header.

JSON:

Content-Type: application/json

This content type returns a json representation of the events received. Note that this is just a representation of the event content, full verifiability is not retained.

• events <array
  • eventId - the id of the event
  • id - CID of the InitEvent for this stream
  • prev <array> - CIDs of previous hash linked events
  • header
  • data
  • timestamp - the unixtime this event was timestamped (if it has been)
• offset - the number of eventId bytes already consumed

CAR file:

Content-Type: application/vnd.ipld.car

This content types returns the underlaying IPLD data structure of the event streams encoded as a CAR file. This data can be used to trustlessly verify the integrity of the event stream.

The root object of the CAR file is a SubscriptionResult which contains an array of EvenIds the new offset. The CAR file also includes all Event objects, corresponding to the eventids in the subscription result.

It’s worth noting that an EventId always includes the CID of its Event.

// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent

// https://cips.ceramic.network/CIPs/cip-124#eventids
type EventId Bytes // sort-data + Event CID

type SubscriptionResult {
  events [EventId]
	offset Integer
}

GET /ceramic/recon/{key*}

Interact with the node using the Recon protocol directly. This allows you to have greater control over the data you consume, but you have to be able to run the Recon algorithm client side.

Example:

/ceramic/recon/kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr&duration=1000

Parameters:

• key* <Array> - a comma separated list of eventids and ahashes (multicodec encoded), first and last key must be eventid, every even key must be an ahash
• duration - the amount of time, in seconds, to wait for a response

Returns:

Content-Type: application/json

• key* <array> - a list of eventids and ahashes (multicodec encoded), first and last key must be eventid, every even key must be an ahash

GET /ceramic/events/{eventid*}

Retrieve events given an array of eventids.

Example:

/ceramic/events/kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr

Returns:

There are two different content types supported as return value, as specified by the Accept http header.

JSON:

Content-Type: application/json

This content type returns a json representation of the events requested. Note that this is just a representation of the event content, full verifiability is not retained.

• events <array
  • eventId - the id of the event
  • id - CID of the InitEvent for this stream
  • prev <array> - CIDs of previous hash linked events
  • header
  • data
  • timestamp - the unixtime this event was timestamped (if it has been)

CAR file:

Content-Type: application/vnd.ipld.car

This content types returns the underlaying IPLD data structure of the event streams encoded as a CAR file. This data can be used to trustlessly verify the integrity of the event stream.

The root object of the CAR file is EventsResult which is a convenience type containing an array of CIDs corresponding to the eventsids requested. The CAR file also includes all Event objects, including all IPLD blocks for these event, but no data from the previous events, e.g. DataEvents include signature envelope, event, and potentially detached payload, while TimeEvents include their entire merkle tree witness.

// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent

type Events [&Event]

POST /ceramic/events

Adds a new event to the node.

Request body:

The request body must contain the complete event data.

CAR:

Content-Type: application/vnd.ipld.car

A CAR file where the root is an Event as described below.

// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent

JSON:

Content-Type: application/json

A possible convenience method for adding an event by only submitting a JWT.

Only possible once we’ve migrated to use Varsig and can create events as plain signed JWTs.

{
  event: "<jwt>"
}

Returns:

JSON:

• eventid - the eventId of this event

GET /stats/active-ranges

A status endpoint that returns all ranges that this node is actively monitoring.

Returns:

• ranges <array
  • reconRange <Array> - a Recon message, *`[eventid or ahash]`*
  • offset - the total number of eventId bytes observed in this range

Rationale

The choice of using OpenAPI enables ease of integration and content types to more easily send and respond using binary formats such as CAR files. It’s also a common pattern to have service workers serve http requests opening up for future browser native implementations of the Ceramic protocol.

There are two main ways of polling for new events. Using recon and a more simple poll. This enables advanced developers to get full control with recon, while a simple poll API based on a single event counter is much easier to use for most developers. The disadvantage of the latter is that events will be received in the order that the node received them, not in the global order of the network.

Offset using EventId bytes

One question that might arise is why the poll api (e.g. /ceramic/subscribe/) couldn’t also return events in the global order of the network. The reason for this is that the node might not receive the events in this order from the network, and we want to be able to tell clients about events as they come in. Therefore we store a log of eventId bytes in the order they were observed (and validated) given a particular interest range. This allows clients to have a simple incrementing byte offset counter, while the node doesn’t need to keep client specific state.

Backwards Compatibility

The Ceramic API provides a new way of interacting with event streams. The main backwards compatibility consideration is that the ComposeDB implementation in js-ceramic would need to be refactored, but this should not imply any major breaking changes.

Implementation

A partial implementation exists in rust-ceramic. It currently implements the following OpenAPI schema.

Security Considerations

Currently this API is not designed to be exposed to the public internet. Instead it’s intended for internal use, e.g. consumed by a ComposeDB node that in turn has a more strict access controlled API open to the internet.

Copyright

Copyright and related rights waived via CC0.

Citation

Please cite this document as:

Danny Browning, Nathaniel Cook, Aaron Goldman, Joel Thorstensson, Spencer T Brody, "CIP-137: Ceramic API [DRAFT]," Ceramic Improvement Proposals, no. 137, May 2023. [Online serial]. Available: https://github.com/ceramicnetwork/CIPs/blob/main/CIPs/cip-137.md