Schema evolution

Evolution is a natural part of a systems lifecycle; requirements change, reality sets in and bugs are fixed. As a result, migrating data from one version to another is a normal part of running a system in production.

One approach to schema evolution is running some kind of batch job which upgrades the old types to the new in-place. This approach is not without its risks: if done without discipline, there is the chance of data loss and other unhappy happenstances. It also goes against the philosophy of data immutability. Another downside of this approach to migration is that due to the behavior of stateful actors, an all or nothing migration may in certain circumstances, require downtime.

An alternative approach which fits in with the idea of event sourcing and immutable data is lazy upgrades between schema versions

For example let us imagine we have versions S1, S2 and S3 of a schema S. Messages m1 and m2 were persisted with S1, while m3 was saved with S2. We've made a change and are forced to use S3 of the schema. When replaying messages, all we need to do is define two functions: S1=> S2 and S2=> S3. We apply S2=> S3 to m3 to upgrade it to latest version of the S. For m1,2 we map S1=> S2 then map from S2=> S3 to complete the upgrade. Being able to support this strategy was the primary motivation for introducing decoders and encoders.

Persistent Actors and JSON

Nact uses JSON for persistence and message passing. Naïvely serializing your objects may work for a while as you are prototyping your system, however using a stable representation for your data models should very much be a concern when engineering a robust production grade system.

The spawnPersistent function includes a number of optional arguments, namely encoder, snapshotEncoder, decoder and snapshotDecoder. These functions map deserialized JSON-safe objects to Javascript objects and map Javascript objects to JSON safe representations respectively. The decoders are well suited to performing schema evolution, while the encoders are useful for adding versioning information and creating a more stable persistent representation.

Cracking the Code

The DaVinci_Decode example below demonstrates how to deal with an enthusiastic but naïve coworker who a) thought that ROT₁₃ was a good encryption scheme and b) applied it everywhere.

In the example, version zero of the message protocol is ROT13 encoded and needs to be unscrambled before it is processed by the actor. Version one is encoded in plain text.

/* Rot13 code */
let a = 'a'.charCodeAt(0);

let toPositionInAlphabet = (c) => c.charCodeAt(0) - a;

let fromPositionInAlphabet = (c) => String.fromCharCode(c + a);

let rot13 = str => [...str].map(chr => fromPositionInAlphabet((toPositionInAlphabet(chr) + 13) % 26)).join('');

let decoder = (json) => {
  if (msg.version == 0) {
    return rot13(msg.text);
  } else {
    return msg.text;
  }
};

let encoder = (msg) => ({ version: 1, text: msg.text });

let system = start(/* Specify a concrete persistence engine here */);

let actor =
  spawnPersistent(
    system,    
    async (state = [], msg, ctx) => {
      console.log(msg);      
      if (! ctx.recovering) {
        await ctx.persist(msg);
      }
      return [msg, ...state];
    },
    'da-vinci-code',
    'da-vinci-code-actor',
    {
      decoder,    
      encoder,
      system      
    }    
  );