Behind the Product: CloudStore (Part 1)
The story behind the innovative graph database technology that runs Visor (Part 1 of 3)
Visor recently completed a sizable build: a collaborative, spreadsheet-like tool for PMs and engineering teams that integrates with Jira. We finished this in a short 45-day window with three engineers and one designer. Our speed to market with this new product was partly due to our in-house datastore framework, CloudStore.
Customers have asked us about our stack, and CloudStore typically becomes a conversation point. They are excited to know how the product they’re using is built, and they’re especially interested that CloudStore is home-grown. To share more broadly, we decided to break it down in a series of posts describing why we built it, how it works, and what advantages and tradeoffs we realized. I also hope that for our customers, it will provide a greater sense of appreciation and understanding about the product they’re using.
In this first post, I’ll share why we built the framework. In the next post, I’ll share more about what CloudStore does. In the final post, I’ll discuss the advantages and tradeoffs.
The unique technical challenge
When the project began in 2016, our company was building a Chrome extension. That was a core differentiator for us: our application would be accessed on top of any web page in the browser. One requirement we set for ourselves to provide a great user experience was that a user should be able to switch between tabs in the browser without losing state in our application.
Before committing to this, we considered less challenging alternatives. Many Chrome extensions open a pop out window on the side of the browser that doesn’t change as users navigate, so there’s no need to synchronize state across tabs. This didn’t appeal to us as a great user experience, because our target customers seemed to be working with limited screen real estate on laptops. We also wanted to provide overlay functionality in each page that required our UX to live inside each tab, effectively ruling out the pop out window.
Once we were committed to this task, I researched how to do this quickly. All of the existing technology I read about used network connections to maintain synchronization across tabs, because they were meant to synchronize web apps.
For example, using something like Firebase would require running the Firebase client in every tab in the browser, and each synchronization operation would require a network call for every tab. Our target customers sometimes had numerous windows, each with dozens of tabs. If something was updated and the user had 50 tabs open (it may seem ridiculous, but we have the data…), that means the update would occur through 50 different network requests.
We knew this would add an unacceptable amount of overhead to our Chrome extension, so we ruled out Firebase and other ways of synchronizing state that scaled in network resource requirements as the number of tabs increased.
Having not found anything we could use, we considered ways to build our own solution that synchronized state using message passing between our tabs. But I was concerned about the complexity this might introduce to everywhere we wanted to read or write the state, especially if reading or writing became asynchronous.
Building a new solution
Around this time, I was digging into a stack of JavaScript books about ECMAScript 2015 (ES6). A new capability introduced in the ES6 update called proxies caught my attention. With proxies, ordinary objects could behave in custom ways, encapsulating advanced logic in routine get and set operations. This is a concept called metaprogramming. We didn’t have to worry about browser compatibility at the time, because our extension targeted just Chrome.
I wondered if this new technology could allow me to convert ordinary objects into synchronized objects by intercepting the ‘set’ calls using proxies and sending those updates as messages to the other tabs. If that worked, these objects could just be treated like normal JavaScript objects. The syntax for reading or writing to these objects wouldn’t be any more complicated.
I built a proof of concept the next morning, and it worked pretty well. It started out as something like this:
We implemented this system, and it solved our needs.
Over time, our requirements for these synchronized state objects grew. We added capability to persist these objects on the server. Then we made it possible to reference the objects from one another, turning the datastore into a graph (with normalized data). We added type schemes to maintain object and graph integrity. We replicated certain objects to Elasticsearch so we could run fast text search.
As we rolled out new features to our core product, some would require a framework addition first. For example, when we added full text search, 90% of the code needed went into the framework. Other times, we could just use the framework as is, reaping the benefits.
We realized that by following this pattern, we were solving really hard problems once in a way that we could benefit from repeatedly later. This is exactly what, as I later read, Asana used as a reason to move forward with Luna:
“there were all kinds of extremely difficult programming tasks that we were doing over and over again for every feature we wanted to write”
In other words… abstraction.
With CloudStore, we aimed to make persistence and synchronization trivial. Could we synchronize state without needing an API layer that with front-end code for the request, server code for the endpoint, and database code for the table? And could this all be done without introducing complicated syntax?
Here is the normal way of doing things:
And here is what we sought to achieve, something terse and obvious:
While this was an exciting project (and probably a naive one for such a technically inexperienced team to undertake) we didn’t feel we had the choice to do anything other than build this ourselves. We couldn’t use off-the-shelf technologies, since it would require even more work to adapt to running in a Chrome extension with multiple tabs.
While we’ve decided to move away from the Chrome extension platform, CloudStore applies very well to a regular web app that needs to synchronize state with a server and other clients. When CloudStore is paired with a reactive UI framework like Vue, we can write little or no server code to add most new product features that change persisted data. We can create collaborative applications that synchronize in milliseconds without explicitly having to write new code to synchronize it. We successfully turned persistence and synchronization into a really easy problem that requires almost no extra thinking.
At the start, we didn’t intend to build a framework. We’re a product company, not a framework company. We just needed to solve a challenge core to our product that didn’t seem to be solved well in the open source community or on the market.
In the next post in this series, I’ll share more about what CloudStore is and how it works.