Introduction to local-first applications

"Imagine you're working on a new software project, and you've completed the preliminary research into the problem this project is trying to solve, and now you're ready to start designing a solution.

At this point in every software development project, there's a question that
you should ask yourself, before moving forward, but if you're like most software engineers today, you might skip this question, because you think you already know the answer.

So, what's the question I'm referring to here?

Well, here it is: "What architecture should we use for this application?"

Ok, there are lots of ways you could answer this question, so to clarify, let me make this a multi-choice question.

So, "What architecture should we use for this application? Should it be a client/server application or a local application?

Now, as I said a moment ago, I bet most of you don't explicitly ask this question anymore, because you almost don't need to, right?

I mean, if you understand the problem you're trying to solve, then this question is easily answered, and the answer is almost always client/server, right?

Well, this used to be true, but I would argue that this question is no longer easy to answer.

What I mean is, you probably need to start explicitly asking this question again, and the reason why I say this, is because the answers to this multi-choice question have changed recently.

There aren't just two answers to this question anymore, now there's a third possible answer, and I bet most developers, right now, aren't even aware of this third option.

So, what's the third option?

Well, there's a new way of architecting applications which is called local-first.

So, what's local-first?

Well, you can think of it as a sort of hybrid between local applications, and client/server applications. I'm calling it a hybrid because it borrows many of the best features from both of these other kinds of applications.

You can think of local-first as a new best-of-both-worlds paradigm.

Now, the first thing you need to know about local-first applications is that they don't store their application data on remote servers, they actually store it on the local computing device, but this kind of sounds like a local application, right?

So how is a local-first application, different from a local application?

Well, I think the best way to differentiate local-first applications, is to juxtapose the features of local-first applications, with local and client/server applications.

So, I'm going to create a feature matrix, that compares these three types of applications.

So, which of these types of applications is the fastest and most responsive to user interactions?

Well, Client/Server applications can be reasonably fast, depending on how close the client is to the server, but the fastest and most responsive applications are local and local-first applications, because both of these types of applications have all the data they need, right there on the local computing device, so they never have to query servers over relatively slow network connections.

If you need multi device support, in other words, you need the same application to run on things like desktops, laptops, tablets, smartphones and so on, well, Local applications don't support multi-devices, and as you probably know, client/server applications do support it, but what you might not know is that local-first also offers excellent support for multi-device applications.

Ok, you're probably curious how local-first is able to support multiple devices, stick with me and I'll explain how it's possible in a moment.

If your application needs to support multiple users, then local applications won't work for you, client/server will work for you, and what might surprise you is that local-first applications support multiple users as well.

Ok, you're probably curious how local-first applications are able to support multiple users, right? Well, multi-user support is possible because of a new kind of data type that local-first applications use, but we'll talk more about these new data types in a moment.

If your application needs to support collaboration, then obviously local isn't going to work for you, client/server will work for you, and again, local-first will support it.

If your application needs near real-time updates than you could use either local-first or client/server, and of course, local won't work here.

Now, most seasoned developers have a good idea how to support real-time collaboration in a client/server architecture, it would probably include servers connected to clients, using WebSockets, to pass updates between the client and the server in real-time, but how do you do real-time updates with local-first? Well, again it has to do with the special datatypes that get used in local-first, which I'll talk more about a bit later.

If your application has fairly coarse-grained authorization requirements, then local-first will probably work for you, however, if you've got complicated, granular authorization rules, then a client/server architecture is likely to be your best option, at least right now.

Currently, fine-grained access control in local-first applications isn't a solved problem, and there will likely be limits on how granular you can even get, with local-first, because of the underlying technology that enables local-first, but more on this in a moment.

If you need things like ACID and server-side validation, then local architectures won't work for you, and you'll need to look to client/server solutions.

If your application needs to work well in poor network conditions, or if the application needs to work when there's no network connection at all, then local architectures are the best option because they have all the data they need to work colocated with the application, and local-first works particularly well, because it's able to seamlessly transition between a connected state, too disconnected and back to connected again.

If you want your applications to work forever, in other words, long after the company that created it, is gone, then local architectures are probably the right choice. This is because local architectures don't rely on remote servers to function, so local and local-first software can continue to function, long after its authors are gone.

If you want to create the most secure software, that also offers the best privacy, then local architectures are the best option. This is because with local architectures, you've got a less tempting target to hackers, versus servers, and with local-first, you can offer end-to-end encryption, so even if a hacker gained access to user data on a peer server, the data is of little value because it's encrypted.

If your application needs to offer its users total ownership and control of their data, then local architectures are the best choice because all the user's data is colocated with the application and the local data is considered to be a source of truth, and of course, client/server offers much less agency and control.

I bet this matrix is surprising to a lot of developers, what I mean is, the architecture choices you now face, aren't as cut and dry as they used to be. In other words, you can't just blindly pick a client/server architecture anymore, because that may not be the best choice for your problems and users.

I mean in terms of features, local-first is pretty compelling when compared to client-server.

Ok, so how does this new option called local-first work? I mean, if you look at this feature set for local-first, it probably seems kind of hard to pull off, right?

So, how is local-first even possible?

Well, there's a breakthrough technology that's coming out of academia called Conflict-free replicated data types or CRDTs, which enables local-first applications.

So, what are CRDTs?

Well, they are a collection of special datatypes that are similar to the datatypes you're already familiar with, such as: Lists, and Maps and strings or Text.

Now what makes these CRDTs special is that they are essentially sharable datatypes.

So for example, if you've got a CRDT list on one computer or node, you can replicate that list to other nodes, and then, each node can modify the list directly, and a synchronization process will replicate and merge all the updates, from all the nodes to the other nodes, and these updates are guaranteed to merge, without conflict, thus putting each node into a consistent state.

Now, one of the nice things about CRDTs is that they enable offline mode, so for example, Node A could go offline, then it could edit its list, changing three to four, and then when the node goes back online, the changes it made while offline, can be automatically synchronized to the other nodes.

Now, I'm not going to lie to you, CRDTs are complicated and challenging to get right, but what you should know is, they're based on Math, and in particular, Order theory and join semilattices, and there are mathematical proofs that show the correctness of these algorithms. So, this should give you confidence in the technology, but there are tradeoffs you need to be aware of.

So, what are some of these tradeoffs?

One of the big tradeoffs is CRDTs use more memory than their conventional counterparts, and the reason they use more memory is because they need to store extra metadata, to facilitate conflict-free merging. But, recently developed techniques from academia have figured out how to minimize and optimize the metadata, to the point where CRDTs are now a viable technology for many scenarios.

Another tradeoff is the consistency model for CRDTs. CRDTs offer high availability, and strong Eventual consistency, the key word there being Eventual, consistency.

Now, using CRDTs by themselves is one way of creating local-first applications, but there's a lot of other interesting work combining CRDTs with databases like SQLite.

The idea here being, you'd use SQLite locally, as your source of truth, and you'd perform reads, inserts, updates and deletes, as you normally would, but then behind the scenes, an SQLite extension, would synchronize all local changes to other instances of SQLite and potentially other database, via CRDTs.

Using SQLite and CRDTs together is a very promising solution for building local-first applications, and it's something we're working on here at Mycelial, and we plan on releasing it as an open-source library.

So in summary, there's a new application architecture that's emerged, called local-first, and it offers a compelling alternative to client/server architectures, and you should probably start asking yourself if local-first is the right solution for your problems and your users.

If you're interested in CRDTs used in conjunction with SQLite, I'd encourage you to join our mailing list, so you can stay in the loop as we release our open-source libraries.

Additionally, if you liked this video, please click the like button and consider subscribing to our channel."

‍