Introducing harmonized.js

Today we are anouncing harmonized.js, our first open source project at Hyphe.

Harmonized will be a client side data model and state manager as well as a local storage and a service, that keeps everything in sync with a server. It works best with react and react-native.
We will be using Mobx for the state management part and indexedDb and SQLite to store everything locally. Harmonized can keep everything synchronized via http(s) REST-calls and/or sockets.

HarmonizedCoreConceptMin

We already have a base concept and did some prototyping and we are confident, that harmonized will be powerful, flexible but relatively easy to use. Mobx is a really awesome library to archive this and we can't point out enough, how useful and easy to understand this state manager is.

You can find harmonized on GitHub and you are welcome to contribute if you are interested.

Requirements

We want harmonized to be able to handle your complete app state. This includes client only data as well as data which is synced via the network. As we want to use it in react as well as react-native, we need support for multiple local storage technologies.

These are our core requirements:

  • Support for different local storages such as SQLite and indexedDb. We also want an option to not use local storage at all.
  • Support for different transport layers such as http(s) and web sockets. Like the local storage we want the usage of a transporter to be optional.
  • auto updates of all state dependencies (especially views) when things change
  • offline usage - We want the app to be usable (at least as much as possible) even if your device has no internet connection.

This leads to a lot of use cases harmonize covers:

with transporter without transporter
with local storage Your app data you want to use offline as well as kept in sync. Most common case. Persistent client only data. This can be things like settings for notifications.
without local storage For things you don't need locally all the time, such as user settings. Session data like a click history.

Core concept

As we want to use Mobx as our state manager, we had a look at their best practices for stores. We liked the structure an went with it. Harmonized will contain four major parts:

  • Store: every route/domain will be stored in a store. The store contains everything the route contains and keeps everything together. You can create and delete items as well as trigger different routines such as refetching everything from the transporter.
  • Item: every entry of a store is a harmonized item. This item contains most of the magic as every item decides for itself, what it should do next. If something needs to get synchronized, the item will do it automatically.
  • LocalStorage: this is the interface to the local storage. As we want to have multiple local storages supported all these storages need to follow the same interface.
  • Transporter: same as LocalStorage, but it's the interface for the network transporter.

HarmonizedCoreConcept

By doing so, we can move the local storage and transporter implementations out of harmonized and let the user decide which one to use. This should keep harmonized at its core very small and allows lots of supported platforms/setups in the future.

No local storage is still a local storage

... and no transporter is still a transporter.

We decided, that you will always need a transporter and a local storage, even if you don't even store something locally. There will be none-implementations for both interfaces which will be basically fakes as they aren't doing anything but keeping the store and the item happy.

This results in much easier store and item implementations as they don't need to care anymore wether a local store is used or not.

Offline capability and its consequences

Using a local storage as cache/buffer for your network isn't that difficult, if you just store and read data.
The biggest challenge lies in the Item relations. Normally you just make your api calls and you connect your data based on these calls.

Example

If you have authors and articles, you need to create an author before he can work on articles.

POST /authors {name: 'some name'} => returns id 123  
POST /articles {title: 'title', authorId: 123}  
PUT /articles/456/author {id: 123}  

You can't do that if you have no internet as you would never receive the id 123 which is used later on. So in order to get this working, we need the client to create and manage it's own representation of this relations.
In fact, we need three ways for each relation to be handled:

  • inside the app state itself
  • through the local storage
  • via the api

Lets have a look on the representation of the item in all these states:

Inside the app state

Inside the app state we can use references to build our relations. This makes it very comfortable to use:

const newAuthor = authors.create({  
  name: 'some name',
});
const newArticle = articles.create({  
  title: 'title', 
  author: newAuthor,
});
console.log(newArticle.author.name); // => 'some name'  

Note that this is (and will be) synchronous!

Through the local storage

We could just save the complete app state on every change. This way, restoring the state on app start would be very easy. The downside is, that you would need to store a lot of data on every little change which is very inefficient.

We decided against such an approach and are going with local identifiers for each item. This makes the implementation a bit more difficult, but now you are able to store only the items that are changing instead of the complete state. We also decided to tag all local only properties with _, so the above would kind of look like the following inside the local storage:

// authors:
[{
  _id: 'uniqueLocalAuthorId',
  name: 'some name',
}];
// articles
[{
  _id: 'uniqueLocalArticleId',
  title: 'title', 
  _authorId: 'uniqueLocalAuthorId',
}];

via the api

The transporter representation looks much like the local storage one. The main difference is, that we now have to use the identifiers the api provides us.

// authors:
[{
  id: 'uniqueServerAuthorId',
  name: 'some name',
}];
// articles
[{
  id: 'uniqueServerArticleId',
  title: 'title', 
  authorId: 'uniqueServerAuthorId',
}];

As we see, every relation contains three different parts. The direct reference inside our app state, the local _id and the transporter id. The nice thing about this is, that we only need one of them to build our complete app state. So if there is an item stored locally but not synced on app start, we can rebuild our state through the local _ids. If there are new items coming through the network, we use the server ids.

Things are actually a bit more complicated if you have computed primary keys, but that would be to much ground to cover right here.

To summarize:

  • each relation has three representations - the state reference, the local relation id(s) and the transporter/api id(s).
  • our item needs to be able to export/import these three representations so that it can be used by the app, the local storage and the transporter.
  • The transporter is always right. In case of doubt we let the api decide and go with the data we get from the transporter.

Roadmap

Implementing harmonized will not be an easy task especially because we want offline usage. Therefor the Item and Store must be solid and well thought through and we will start with that.

These are the steps we want to take:

  • First version of the item and store.
  • Transporter & local storage interfaces and none implementation.
  • Http(s) transporter - harmonized should be usable by a bunch of apps from this point on.
  • SQLite local storage - we want to have the persistence on the mobile devices (and therefore react native) first.
  • IndexedDb local storage.
  • (after some time using harmonized) a bunch of helper methods to simplify the usage.

The first release versions may be a bit difficult to use, but we think it's more important to do the basic stuff right. Adding sugar and some useful setup functions can be done easily if the rest works well.

Thats it for the introduction. If you are interested and have questions and or ideas, feel free to leave a comment or contact us directly.

comments powered by Disqus