FOREST: An Interacting Object Network

FOREST is one of the layers making up The Object Network.

Part of FOREST is the definition of a would-be application/forest+json Media Type. This page describes that.

Functional Observer

Functional Observer describes a linked object graph structure - the Object Network. The Objects are essentially JSON structures: lists, maps, strings, etc. Here's an example in JSON:

{ "b": ["c", "x", 123, "y"], "d": true }

Single elements and single-element lists are not distinguished, and string numbers and booleans aren't distinguished from native numbers and booleans. Also, null elements disappear.

All Object Network objects have UID object references which will usually contain a GUID, a UUID or some other random-looking string after a "uid-" prefix. Links are just UID or URL strings as values after a property tag and can appear anywhere in an object. Links are transparent: so any object linked to can be merged in-line instead:

{ "b": "c", "f": "uid-222" }

where uid-222 is:

{ "a": "x", "b": [ "e", 123 ] }

is the same as:

{ "b": "c", "f": { "a": "x", "b": [ "e", 123 ] } }

Objects (and sub-objects) can link to "further details" using More:

{ "b": "c", "More": "uid-222" }

where uid-222 is:

{ "a": "x", "b": [ "e", 123 ] }

is the same as:

{ "a": "x", "b": [ "c", "e", 123 ] }

The elements at 'More:' are merged in set-wise: duplicates are removed.

Links to list objects are also seen as the same as inline lists. For example:

{ .. "email": "uid-111", .. }

where uid-111 is:

{ "is": "list", "list": [ "a@b.com", "c@d.com" ] }

is the same as:

{ .. "email": [ "a@b.com", "c@d.com" ] .. }

This allows lists to be grown outside of their containing object, or reduced back in again. An object can switch at any time between these forms: single-to-list-to-single, link-to-inline-to-link.

There are no nulls or empty lists in the Object Network, or rather, null and empty list are equivalent to non-existent. So:

{ "a": null, "b": [ null, "c" ] }

is the same as:

{ "b": "c" }

These objects interact by mutual observation. The Functional Observer interaction model is simply:

Set an object's state as a Function of its current state plus the state of other objects it Observes through links.

This observation occurs through either pull or push of linked object state.

Such a programming model is declarative in nature, and thus very expressive, as well as being naturally concurrent.

FOREST

FOREST is a distributed object architecture. FOREST is an acronym for Functional Observer REST: it maps directly to RESTful distribution over HTTP, using GET for pull and POST for push of object state, in both directions between interacting peer servers. POST is effectively an idempotent GET response or cache update. Even though HTTP is client-server and doesn't have notification, FOREST implements a symmetric peer-to-peer protocol over HTTP's GET and POST. Each server can act as client; GET is used to pull data, POST to notify updates to that data. PUT, DELETE, etc aren't used. HEAD may be used.

Objects are published into a global interacting object network by turning their UIDs into URLs. A link can be a full URL starting "http://", or just a UID starting "uid-". UIDs can be converted to full URLs by wrapping them appropriately with prefix and postfix - e.g. uid-0fe6-7d5d can be wrapped to give: http://object.network/events/uid-0fe6-7d5d.json. Object Network URLs are considered opaque to clients. The use of the .json extension is optional but advised for human readability, editing and saving of files.

The only response codes in FOREST are 200 for successful GET or POST, 204 for successful POST without response or long-poll heartbeat, 303 for redirect, 304 for Etag match, 401 for auth challenge, 403 for unsubscribing from updates, 404 object not found on that URL, 405 if POST not allowed, 50X server failures.

The only headers used to manage the cache are Etag and max-age. Etag is an incrementing Version number. Thus If-None-Match is used for conditional GET.

Here's a simple FOREST GET example:

GET /events/uid-f330a3-4011e.json HTTP/1.1
Host: object.network
If-None-Match: "32"

HTTP/1.1 200 OK
Cache-Control: max-age=1800
ETag: "33"
Content-Type: application/forest+json

{ "is": [ "attendable", "event" ],
  "title": "The Object Network Conference",
  "content": "This year is expected to be the best yet!",
  "start": "30 Sep 2016 09:00:00 GMT",
  "end":   " 1 Oct 2016 23:00:00 GMT",
  "location": "http://bletchleypark.org/address.json",
  "attendees": "uid-0fe6-7d5d"
}

POST

In FOREST, POST is defined as a "pushed GET response". It therefore is idempotent - you can re-POST the same thing as often as you like.

The POSTed content body, being equivalent to a cache-filling GET response, can have HTTP headers set for Content-Location, Etag and Cache-Control: max-age. Cache-Control is somewhat ill-specified in this case, so may be replaced with Expires. If there is any reason not to send these headers in the POST, they can be carried in the body, as the "URL", "Version" and "Max-Age" properties.

POST normally returns 204 No Content. It is also an "open channel" for return notifications of new state, which otherwise would have been POSTed back. In this case, the response can either be 200 OK with the content body and its Content-Location if it differs from the original POST target, or 303 with a Location, or 303 with content body and both Location and Content-Location.

The POST response, and GET responses, can return a Cache-Notify header giving the URL of "the cache". This can be used to POST to it directly, allowing the state update to be propagated to all interested parties, to save an object being POSTed directly to multiple targets on the same host. Similarly, if the Cache-Notify header is set in a request header, then updates to the target will be POSTed back there.

The Cache-Notify URL can also be used in an asymmetric long-polling setup, where a response is returned on a GET on that server Cache-Notify URL whenever any update of interest occurs for the client Cache-Notify given on that GET's headers. A POST can be used in the same way to long-poll, combining notifications in both directions. If there was nothing to POST in the first instance, an empty POST body can be sent, to set up the long poll channel.

It is possible for a server or client that isn't visible and cannot publish its own URLs to work asymmetrically by POSTing the basic UIDs of its own data instead of their wrapped URL.

Here's a simple FOREST POST example:

POST /events/uid-f330a3-4011e.json HTTP/1.1
Host: object.network
Content-Location: http://fred.info/o/uid-f330a3-4011e.json
Cache-Control: max-age=1800
ETag: "1"
Content-Type: application/forest+json

{ "is": "rsvp",
  "event": "http://object.network/events/uid-f330a3-4011e.json",
  "person": "http://..",
  "attending": "yes",
}

HTTP/1.1 200 OK
Cache-Control: max-age=1800
ETag: "34"
Content-Type: application/forest+json

{ "is": [ "attendable", "event" ],
  "title": "The Object Network Conference",
  "content": "This year is expected to be the best yet!",
  "start": "30 Sep 2016 09:00:00 GMT",
  "end":   " 1 Oct 2016 23:00:00 GMT",
  "location": "http://bletchleypark.org/address.json",
  "attendees": [ "uid-0fe6-7d5d", "http://fred.info/o/uid-f330a3-4011e.json" ]
}

The RSVP request is a spontaneous notification - a "pushed observation" - to the target Event object. The Event will then link to this notified object if it is interested in it. It can then poll for changes, or wait for further POST updates. If not interested, or stops being interested, it would return 403.

The POST response is optional, and is sent on the assumption that the source, the RSVP, is interested in the resulting update - that the RSVP object is observing this Event, which it probably is. This could be made explicit through the use of Cache-Notify.

Coffee Shop Example

There is a tradition in the distributed systems or integration community to use the example of ordering coffee to demonstrate your asynchronous system integration or distribution model.

It started with Gregor Hohpe's article about asynchronous messaging, then was continued by Jim Webber as "RESTbucks" to discuss his REST interpretation in an open forum and subsequently used to describe a refined version of the approach in the book by Jim and Ian Robinson.

Here's the FOREST version. It's based on the Foreign Exchange ordering example I used in my FOREST paper, which in turn was actually a coffee ordering scenario in its first drafts.

GET /baristas HTTP/1.1
Host: coffee-shop.com

HTTP/1.1 200 OK
Etag: "313"
Cache-Control: max-age=10
Content-Type: application/json

{ "is": [ "coffee", "shop", "vendor" ],
  "products": [
    { "title": "Espresso",   "More": "/prod/esp1" },
    { "title": "Cappuccino", "More": "/prod/cap2" }
  ],
  "trackers": [
    "http://coffee-shop.com/track110",
    "http://coffee-shop.com/track109"
  ]
}

Here we have an entry point URL on the coffee-shop.com server which returns a "vendor" object listing products and currently outstanding orders being tracked. Let's have a look at that Espresso:

GET /prod/esp1 HTTP/1.1
Host: coffee-shop.com

HTTP/1.1 200 OK
Etag: "12"
Cache-Control: max-age=3600
Content-Type: application/json

{ "is": [ "coffee", "product" ],
  "title": "Espresso",
  "description": "A shot of pure coffee",
  "options": {
    "shots":     { "input": "textfield", "label": "Shots", "range": [ 1, "..", 5 ] }
    "ristretto": { "input": "checkbox",  "label": "Ristretto" }
  }
  "price-text": "£1.50 single, £2.50 double"
}

So now we know all about it, and it allows certain options - "shots" and "ristretto" - that we can present on a user interface. Lets give our order, now:

POST /baristas HTTP/1.1
Host: coffee-shop.com
Content-Type: application/json

{ "URL": "http://my-orders.com/ordr321",
  "Version": 1,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 1, "ristretto": false } ],
  "vendor": "http://coffee-shop.com/baristas"
}

This is hosted on my-orders.com. We're proactively notifying the vendor, who doesn't know about this object yet, that here's something it may be interested in. If so, it'll add the order's URL to itself and take up observing it. In this case, it's not interested in the order, but creates a delegate - a tracker object - that is observing the order. Since the vendor can guess that the order is observing it, it could return itself showing the link to the new tracker:

HTTP/1.1 200 OK
Etag: "314"
Cache-Control: max-age=10
Content-Type: application/json

{ "is": [ "coffee", "shop", "vendor" ],
  "products": [
    { "title": "Espresso",   "More": "/prod/esp1" },
    { "title": "Cappuccino", "More": "/prod/cap2" }
  ],
  "trackers": [
    "http://coffee-shop.com/track111",
    "http://coffee-shop.com/track110",
    "http://coffee-shop.com/track109"
  ]
}

But that's pretty useless to the order. Better to be redirected to the new tracker:

HTTP/1.1 303 See Other
Location: http://coffee-shop.com/track111

Or better still to be returned the whole tracker - it notifies itself, again pro-actively, to the order:

HTTP/1.1 200 OK
Content-Location: http://coffee-shop.com/track111
Etag: "1"
Cache-Control: max-age=10
Content-Type: application/json

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 1, "ristretto": false } ],
  "price": "£1.50",
  "status": "waiting"
}

Or do both:

HTTP/1.1 303 See Other
Location: http://coffee-shop.com/track111
Content-Location: http://coffee-shop.com/track111
Etag: "1"
Cache-Control: max-age=10
Content-Type: application/json

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 1, "ristretto": false } ],
  "price": "£1.50",
  "status": "waiting"
}

Or return nothing (204 No Content), but POST the tracker to the order when it's good and ready:

HTTP/1.1 204 No Content

:

POST /ordr321 HTTP/1.1
Host: my-orders.com
Content-Location: http://coffee-shop.com/track111
Etag: "1"
Cache-Control: max-age=10
Content-Type: application/json

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 1, "ristretto": false } ],
  "price": "£1.50",
  "status": "waiting"
}

HTTP/1.1 204 No Content

Or the same but with those HTTP headers in the body:

HTTP/1.1 204 No Content

:

POST /ordr321 HTTP/1.1
Host: my-orders.com
Content-Type: application/json

{ "URL": "http://coffee-shop.com/track111",
  "Version": 1,
  "Max-Age": 10,
  "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 1, "ristretto": false } ],
  "price": "£1.50",
  "status": "waiting"
}

HTTP/1.1 204 No Content

The Cache-Control and Content-Type headers will not be shown from now on, for brevity. We'll also use the in-body version of URL, Version and Max-Age.

Now the updates to the order are notified at the tracker directly - let's add a shot:

POST /track111 HTTP/1.1
Host: coffee-shop.com

{ "URL": "http://my-orders.com/ordr321",
  "Version": 2,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "vendor": "http://coffee-shop.com/baristas",
  "tracker": "http://coffee-shop.com/track111"
}

HTTP/1.1 200 OK
Content-Location: http://coffee-shop.com/track111
Etag: "2"

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£2.50",
  "status": "waiting"
}

The Content-Location is actually not necessary here. Let's check if we missed our order being fulfilled:

GET /track111 HTTP/1.1
Host: coffee-shop.com
If-None-Match: "2"

HTTP/1.1 304 Not Modified
Etag: "2"

Nope. Time passes. The order gets completed:

POST /ordr321 HTTP/1.1
Host: my-orders.com

{ "URL": "http://coffee-shop.com/track111",
  "Version": 3,
  "Max-Age": 10,
  "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£2.50",
  "status": "filled"
}

HTTP/1.1 204 No Content

But we have a race condition - the order has been updated, but it's too late to change, now:

POST /track111 HTTP/1.1
Host: coffee-shop.com

{ "URL": "http://my-orders.com/ordr321",
  "Version": 3,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": [ { "product": "/prod/cap2", "quantity": 1, "shots": 2 } ],
  "vendor": "http://coffee-shop.com/baristas",
  "tracker": "http://coffee-shop.com/track111"
}

HTTP/1.1 200 OK
Content-Location: http://coffee-shop.com/track111
Etag: "4"

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£2.50",
  "status": [ "filled" , "not-as-ordered" ]
}

We have a tantrum:

POST /track111 HTTP/1.1
Host: coffee-shop.com

{ "URL": "http://my-orders.com/ordr321",
  "Version": 4,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": "cancelled",
  "vendor": "http://coffee-shop.com/baristas",
  "tracker": "http://coffee-shop.com/track111"
}

HTTP/1.1 200 OK
Content-Location: http://coffee-shop.com/track111
Etag: "5"

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£0.00",
  "status": "cancelled"
}

Then, feeling thirsty, decide to take it anyway:

POST /track111 HTTP/1.1
Host: coffee-shop.com

{ "URL": "http://my-orders.com/ordr321",
  "Version": 5,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "vendor": "http://coffee-shop.com/baristas",
  "tracker": "http://coffee-shop.com/track111"
}

HTTP/1.1 200 OK
Content-Location: http://coffee-shop.com/track111
Etag: "6"

{ "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£2.50",
  "status": "filled"
}

Now it's time to pay up, so create a payment object and add a link to it on the order:

POST /track111 HTTP/1.1
Host: coffee-shop.com

{ "URL": "http://my-orders.com/ordr321",
  "Version": 6,
  "Max-Age": 10,
  "is": [ "coffee", "order" ],
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "vendor": "http://coffee-shop.com/baristas",
  "tracker": "http://coffee-shop.com/track111",
  "payment": "http://my-orders.com/paym432"
}

HTTP/1.1 204 No Content

The tracker observes the payment:

GET /paym432 HTTP/1.1
Host: my-orders.com

HTTP/1.1 200 OK
Etag: "1"

{ "is": "payment",
  "invoice": "http://coffee-shop.com/track111",
  "order": "http://my-orders.com/ordr321",
  "amount": "£2.50",
  "account": { .. }
}

And adds it, takes the payment and sets the status to "paid":

POST /ordr321 HTTP/1.1
Host: my-orders.com

{ "URL": "http://coffee-shop.com/track111",
  "Version": 7,
  "Max-Age": 10,
  "is": [ "coffee", "order-tracker" ],
  "order": "http://my-orders.com/ordr321",
  "products": [ { "product": "/prod/esp1", "quantity": 1, "shots": 2, "ristretto": false } ],
  "price": "£2.50",
  "status": "paid",
  "payment": "http://my-orders.com/paym432"
}

HTTP/1.1 204 No Content

Duncan Cragg, 2014

Contact me and/or subscribe to my blog and/or follow me on Twitter.

[E225]