Application architecture predictions
I'm still trying to keep up with web / app technology, and every now and again something comes up and I think: "Ooh, that seems like what I was trying to do with Project Ion" (my old experimental project, mostly dead). I figured it was worth to document these so I can see how many ideas in it eventually make it into general use, and what to work on next. So, to summarize the main ideas:
1) Reactive data-binding UI
Status: Many popular libraries (react, angular, knockout, polymer, ...)
This was the biggest divergence at the time, but is now pretty much everyone. It's always a pain updating you view once your model changes, making sure all the constraints still hold, so a declarative UI that binds things tends to be a clear improvement. The idea has been around for a while now, and it's nice to see that its popularity for application development has increased.
2) Observable live data model
Status: Implemented in Firebase, share.js
Not surprisingly, this idea (as with many) came from Wave - rather having static data which you fetch from a server via RPC or REST, you have 'live' data which is kept up-to-date via a websocket to the server, and you can process via change events. This type of model is idea for when you want your data to update within the application automatically whenever some data it has loaded changes. If you add OT, things like offline apps also become easier, which is nice. This leads to...
3) JSON references
Implemented in Falcor references
A small thing, but this is really needed once you start doing object binding. Say you have an object you need to access in two places (e.g. a user's name and image), you need to bind them to the same reference. In code you can pass the same object to both, but in declarative form (e.g. templates), you can't, so it's simpler to pass a reference instead (rather than denormalize the data to both, which also breaks binding). Some things already do this a bit with REST results too (using URLs as references), but few places have it formalized.
4) Reactive data (view-)model projections
Status: Not implemented?
Once you have a live server<->model binding and a live viewmodel<->view binding, the next obvious step seems to be model<->viewmodel bindings. Sadly, nothing seems to do this currently? What makes it even stranger is that (a) viewmodel<->view bindings are already essentially this, just one half is DOM rather than JSON, and (b) Object<->Object mappings are really, really useful. They make up a huge proportion of what written code does, there's just not an easy way to do it declaratively...
The closest equivalent in the past is XSLT, which perhaps explains why people haven't done likewise for JSON. I'm assuming it'll become popular eventually though, once some techniques from template binding can be borrowed and made efficient. Once a static version is available, a live will probably follow too.
Note: I think of this map as a projection, where project ion gets its name.
5) DOM-like event system
Status: Implemented in Treevent
Given all this data binding logic, you need a way to see when stuff changes in order to update the bindings. There are some ways to obtain this, but what you really need is heirarchical binding - e.g. if your template binds to {{param1.field1}}, you need to bind further down the tree from its parameters. DOM solved this issue a long time ago with event bubbling (very useful!), unfortunately it's a bit trickier in JS (to do it efficiently, and with arrays), hopefully this explains why I ended up writing Treevent.
6) Portable logic
Status: Implemented(ish) in apps, but no standard way?
This idea is easy to state, but hard to implement: if your server has data S, and the client has the data C, and there's a projection P where C = P(S), you should be able to decide to run P on the client or server. Some apps can do this for some things - server-side rendering in JS apps is a great example of this, where S = data, C = HTML, and P is your template. Good apps can run P both on the server or on the client, whichever is more efficient at the time. Some apps even do this at the model side (e.g. spreadsheets choosing to evaluating formulae on the client or server). It's hard to formalize, although hopefully it'll come as a side-effect of using more projections.
Side note: this also requires code sharing between client and server. React does this via running JS everywhere, while Inbox does it via transpilation, which seems to be more efficient.
7) Object = Reduce(initial state, operation list)
Status: Implemented in Redux
This is another idea borrowed from Wave and other functional paradigms - once you define operations to transform your data, it's useful to formalize your object's state as a reduction of all of the transforms applied to some initial state. As with data binding UIs, this makes it much simpler to verify the correctness of your data, but also enables nice things like time travel, by picking different prefixes of the operation list. One thing to note is that some databases work this way too - e.g. transaction logs or Redis's AOF utilize this. This leads to...
8) Storage = operation list plus versioned snapshots
Status: Implemented, but thrown away?
As mentioned, a number of storage systems already utilize transaction logs, but generally these are cleaned up over time. If client-side apps migrate to using transaction logs (plus reductions) to represent state, it seems to make sense to do likewise on the server. One other benefit is the reduction of effort on changes: I was always surprised at Google to see how much data was loaded, sometimes even sent across a network!, then thrown away, when toggling a single boolean.
The other requirement here is versioned snapshots: essentially, reductions of the operation list prefixes so save recalculating it in the future. This leads to...
9) Timestamped versioning
Status: Implemented(ish) in Google's Spanner
Mentioning distributed timestamps for storage tends to make people shudder at the thought, but some progress is being made here - either with Lamport clocks or, more recently, with Spanner's TrueTime. The benefits are then clear: you can do a consistent read at some point in the past, essentially the 'time travel' mentioned in #7 becomes actual time travel!
10) JSON replacing HTML for page data
Status: Not implemented
One of the more extreme ideas, so it seemed good to leave this for #10. Web pages used to be that: pages, i.e. documents that were mostly text and read like they were laid out on pages of paper. This is reflected in both the HTML and CSS that were used - very text-heavy, with a vertical-based layout system. With flex-box, CSS has been updated to be much more app-friendly, however HTML has stayed mostly the same, and instead now includes a lot of divs... component frameworks like React and Polymer are solving that a bit, but it's still the case where most people are projecting JS objects into HTML substrates...so why not just use JS directly? Maybe this will come with #4's templating system.
[Edit]
Thanks to discussion on my G+ post, I was reminded of one area I'd intended to include but had forgotten. So, adding it back in...
11) Slices / Segments / ... = blocks of fields.
Status: Implemented on servers, but not formalized in clients?
The idea here is that, for each object loaded from the server (as in #2), it may have a large collection of fields. You probably don't use all of them to display your data, so you can do like GraphQL or the Google API 'fields' parameter. In theory that means you just load the data you need, which is nice, however there are a few problems: (a) requests have overhead - you don't want to request a single field (e.g. user name) and follow it with a second request for another single field (e.g. user photo) separately. (b) caching is hard - after asking for fields A & B, and caching them locally, if I later ask for A & C, do I have to fetch A again? Or B as well, even, if it's related to A? (c) At the lowest persistence layer, fields are probably grouped and loaded together already, so it may be much easier to return them both.
This is where Slices / Segments etc. come in - by grouping fields by likely co-access (manually configured due to expected usage, although you might be able to learn them), you may load a bit more than you need, but it can make later interactions faster, and makes caching much simpler. As mentioned, this is already done at a persistence layer, so my guess is it'll come in handy in clients as well.
There's a lot of smaller things to go into but this post is pretty long already, so just to list a few: projections as command-line apps, frontend apps working in the same way as clients (i.e. data-bound projections), syncing client cache keys on the server, treating projections and schemas as models (hence, live), and binding to URL path and query params just to name a few.
Anyway, hopefully that's a snapshot of what I've been keeping my eye out for. I once joked when leaving Google that I'd think about returning when a lot of the pains of writing apps were removed, perhaps this post can act as a checklist :) The good news is that many are implemented, I just need 4 and maybe 8 (10 is optional), but mainly for all these to be available in one setup, they are mostly separate for now. Thankfully, I'm interning at Facebook over summer, so maybe I'll get a chance to help add a few of them to React!
1) Reactive data-binding UI
Status: Many popular libraries (react, angular, knockout, polymer, ...)
This was the biggest divergence at the time, but is now pretty much everyone. It's always a pain updating you view once your model changes, making sure all the constraints still hold, so a declarative UI that binds things tends to be a clear improvement. The idea has been around for a while now, and it's nice to see that its popularity for application development has increased.
2) Observable live data model
Status: Implemented in Firebase, share.js
Not surprisingly, this idea (as with many) came from Wave - rather having static data which you fetch from a server via RPC or REST, you have 'live' data which is kept up-to-date via a websocket to the server, and you can process via change events. This type of model is idea for when you want your data to update within the application automatically whenever some data it has loaded changes. If you add OT, things like offline apps also become easier, which is nice. This leads to...
3) JSON references
Implemented in Falcor references
A small thing, but this is really needed once you start doing object binding. Say you have an object you need to access in two places (e.g. a user's name and image), you need to bind them to the same reference. In code you can pass the same object to both, but in declarative form (e.g. templates), you can't, so it's simpler to pass a reference instead (rather than denormalize the data to both, which also breaks binding). Some things already do this a bit with REST results too (using URLs as references), but few places have it formalized.
4) Reactive data (view-)model projections
Status: Not implemented?
Once you have a live server<->model binding and a live viewmodel<->view binding, the next obvious step seems to be model<->viewmodel bindings. Sadly, nothing seems to do this currently? What makes it even stranger is that (a) viewmodel<->view bindings are already essentially this, just one half is DOM rather than JSON, and (b) Object<->Object mappings are really, really useful. They make up a huge proportion of what written code does, there's just not an easy way to do it declaratively...
The closest equivalent in the past is XSLT, which perhaps explains why people haven't done likewise for JSON. I'm assuming it'll become popular eventually though, once some techniques from template binding can be borrowed and made efficient. Once a static version is available, a live will probably follow too.
Note: I think of this map as a projection, where project ion gets its name.
5) DOM-like event system
Status: Implemented in Treevent
Given all this data binding logic, you need a way to see when stuff changes in order to update the bindings. There are some ways to obtain this, but what you really need is heirarchical binding - e.g. if your template binds to {{param1.field1}}, you need to bind further down the tree from its parameters. DOM solved this issue a long time ago with event bubbling (very useful!), unfortunately it's a bit trickier in JS (to do it efficiently, and with arrays), hopefully this explains why I ended up writing Treevent.
6) Portable logic
Status: Implemented(ish) in apps, but no standard way?
This idea is easy to state, but hard to implement: if your server has data S, and the client has the data C, and there's a projection P where C = P(S), you should be able to decide to run P on the client or server. Some apps can do this for some things - server-side rendering in JS apps is a great example of this, where S = data, C = HTML, and P is your template. Good apps can run P both on the server or on the client, whichever is more efficient at the time. Some apps even do this at the model side (e.g. spreadsheets choosing to evaluating formulae on the client or server). It's hard to formalize, although hopefully it'll come as a side-effect of using more projections.
Side note: this also requires code sharing between client and server. React does this via running JS everywhere, while Inbox does it via transpilation, which seems to be more efficient.
7) Object = Reduce(initial state, operation list)
Status: Implemented in Redux
This is another idea borrowed from Wave and other functional paradigms - once you define operations to transform your data, it's useful to formalize your object's state as a reduction of all of the transforms applied to some initial state. As with data binding UIs, this makes it much simpler to verify the correctness of your data, but also enables nice things like time travel, by picking different prefixes of the operation list. One thing to note is that some databases work this way too - e.g. transaction logs or Redis's AOF utilize this. This leads to...
8) Storage = operation list plus versioned snapshots
Status: Implemented, but thrown away?
As mentioned, a number of storage systems already utilize transaction logs, but generally these are cleaned up over time. If client-side apps migrate to using transaction logs (plus reductions) to represent state, it seems to make sense to do likewise on the server. One other benefit is the reduction of effort on changes: I was always surprised at Google to see how much data was loaded, sometimes even sent across a network!, then thrown away, when toggling a single boolean.
The other requirement here is versioned snapshots: essentially, reductions of the operation list prefixes so save recalculating it in the future. This leads to...
9) Timestamped versioning
Status: Implemented(ish) in Google's Spanner
Mentioning distributed timestamps for storage tends to make people shudder at the thought, but some progress is being made here - either with Lamport clocks or, more recently, with Spanner's TrueTime. The benefits are then clear: you can do a consistent read at some point in the past, essentially the 'time travel' mentioned in #7 becomes actual time travel!
10) JSON replacing HTML for page data
Status: Not implemented
One of the more extreme ideas, so it seemed good to leave this for #10. Web pages used to be that: pages, i.e. documents that were mostly text and read like they were laid out on pages of paper. This is reflected in both the HTML and CSS that were used - very text-heavy, with a vertical-based layout system. With flex-box, CSS has been updated to be much more app-friendly, however HTML has stayed mostly the same, and instead now includes a lot of divs... component frameworks like React and Polymer are solving that a bit, but it's still the case where most people are projecting JS objects into HTML substrates...so why not just use JS directly? Maybe this will come with #4's templating system.
[Edit]
Thanks to discussion on my G+ post, I was reminded of one area I'd intended to include but had forgotten. So, adding it back in...
11) Slices / Segments / ... = blocks of fields.
Status: Implemented on servers, but not formalized in clients?
The idea here is that, for each object loaded from the server (as in #2), it may have a large collection of fields. You probably don't use all of them to display your data, so you can do like GraphQL or the Google API 'fields' parameter. In theory that means you just load the data you need, which is nice, however there are a few problems: (a) requests have overhead - you don't want to request a single field (e.g. user name) and follow it with a second request for another single field (e.g. user photo) separately. (b) caching is hard - after asking for fields A & B, and caching them locally, if I later ask for A & C, do I have to fetch A again? Or B as well, even, if it's related to A? (c) At the lowest persistence layer, fields are probably grouped and loaded together already, so it may be much easier to return them both.
This is where Slices / Segments etc. come in - by grouping fields by likely co-access (manually configured due to expected usage, although you might be able to learn them), you may load a bit more than you need, but it can make later interactions faster, and makes caching much simpler. As mentioned, this is already done at a persistence layer, so my guess is it'll come in handy in clients as well.
There's a lot of smaller things to go into but this post is pretty long already, so just to list a few: projections as command-line apps, frontend apps working in the same way as clients (i.e. data-bound projections), syncing client cache keys on the server, treating projections and schemas as models (hence, live), and binding to URL path and query params just to name a few.
Anyway, hopefully that's a snapshot of what I've been keeping my eye out for. I once joked when leaving Google that I'd think about returning when a lot of the pains of writing apps were removed, perhaps this post can act as a checklist :) The good news is that many are implemented, I just need 4 and maybe 8 (10 is optional), but mainly for all these to be available in one setup, they are mostly separate for now. Thankfully, I'm interning at Facebook over summer, so maybe I'll get a chance to help add a few of them to React!
Comments
Post a Comment