clientful

How to create real-time collaborative offline-first apps without traditional backend services or centralized databases Instead of: Client + Backend + DB + Cache + Files storage + Realtime Layer + Kubernetes + Servers We will need only: Client + Files storage + Realtime Layer

From the Author

 

Hello, dear reader. My name is David Shekunts, and I am a Tech Lead with 12 years of experience.

clientful is not an established term. I use it here because it best describes the class of solutions I discuss in this article.

This approach is already proving viable and unlocks not only technical advantages, but also meaningful business advantages when building products.

In this work, I want to share my experience, principles, and practical techniques for building full-fledged SaaS-like applications without a backend.

i. Introduction

1. All business logic and computation must happen on the client

2. All database work must happen on the client side

3. The system must support multiple devices for the same user

4. The system must be multi-user

5. The system must also be able to store files: photos, videos, audio, PDFs, etc.

6. The system must be secure

1. File Storage – external file storage*: S3-like, Google Drive-like, or IPFS-like services for persisting and transferring data and files between devices and users. At the same time, users can use their own storage without requiring us to provide our own backend.

2. WebSocket-like Broker for routing messages between clients. A clientful implementation can use almost any service that lets clients connect to a room and broadcast messages to all connected users.

ii. Advantages

a. Business

1. Zero infrastructure cost – for some businesses, the cost of storing user data and processing user requests is a barrier that prevents them from surviving long enough to reach payback. In clientful, all compute resources and storage systems are shifted to the user side, which means you do not have to pay for servers, applications, databases, caches, and similar infrastructure at all.

2. Integration speed – the whole system is a set of a client application, for example HTML/CSS/JS, S3-like storage, and a WebSocket server. This makes it possible to integrate such products into third-party systems very quickly.

3. Own Your Data – data can be stored entirely on the user side and can be fully encrypted, which can become a critical advantage over competitors.

4. 99.9999% uptime – the application either exists on the user’s devices and in large clouds such as Google Drive, or it is connected to your S3 server. As a result, the system has the lowest possible probability of failure.

5. Maximally interactive UI — the user never waits for a “server response”. Everything changes locally, so the application feels extremely fast. As soon as external changes arrive, they are immediately reflected in the UI.

6. The application always remains with the user – as long as the user has a copy of your HTML/CSS/JS, the application remains available and works. You can sell the product with a guarantee that the user will be able to use it from that moment onward.

7. Anti Vendor-lock – all data will be represented as JSON files organized in folders, meaning that if a user urgently needs to migrate (for example, due to distrust of the platform, or new country regulations), they can migrate freely between the file system, Google Drive-like providers, S3-like providers, and similar storage backends.

b. Technical

1. No vendor lock-in — there is no dependency on server providers at all.

2. Infinite scalability by design — every new user brings their own resources, so load does not grow.

3. Development speed – no staging or dev servers, no deployment of ten applications, no Kubernetes, and not even Docker. You need nothing except to build a client-side web or native application that only requires a CDN.

4. E2E testing out of the box – the entire system is literally just the client, so writing a client test is effectively writing an E2E test that covers the whole infrastructure.

5. LLM-optimized – the whole application is literally client-side code, which means the LLM only needs the application context to understand the entire system.

6. Operational stability – how often has a technology failed simply because the backend application, server, or database went down? In this case, we have offline-first behavior.

7. Offline-first out of the box — the application works without an internet connection, and the data is always available.

8. Non-internet connection – devices can communicate over any protocol, for example LAN, BLE, I2C, and similar options.

9. Option for centralization – in practice, if your project needs a central “database” that stores all or part of the data, you can connect your own central S3-like or Google Drive-like storage and publish part of the users’ content there.

10. Real-time-native – as soon as any change reaches the client, the interface immediately reflects it.

c. Security

1. Smaller attack surface — no server means no server-side attack point for hackers.

2. Privacy by default — data does not leave the user’s device, so you do not have to think about GDPR exposure and data leaks in the same way.

3. Theoretically unbreakable system – encryption strategies are possible where keys are created and always remain on users’ devices. They literally do not exist in any centralized place, and even if someone downloads the data from file storage, they cannot decrypt it back into readable form.

iii. Implementation

1. Data storage algorithm

2. Storage

3. Communication protocol

4. Auth / Identity

5. Encryption

a. Data Storage Algorithm

1. Multiple devices – when a user opens the application on another device, for example on both a computer and a phone, they must be able to work with the same data.

2. Multi-user collaboration – users must be able to work collaboratively on the same data, both separately while seeing saved changes and in real time.

1. Pure P2P – every node is equal and can make decisions fully independently.

2. Host-based P2P – at a specific point in time, or for specific operations, one of the nodes becomes the main node and processes requests from other nodes.

1. Pure P2P
1. CRDT — data structures and algorithms that allow multiple nodes to independently change the same state and then merge those changes without conflicts so that all replicas converge to the same result.
2. CouchDB — an eventual-consistency document database and replication protocol: nodes exchange versioned documents, while conflicts are stored as alternative versions and require a resolution strategy. In other words, merge is not “magical” as in CRDT; it is defined by the application’s model or logic.
3. Blockchain — a distributed ledger where nodes agree on a single transaction order through a consensus mechanism such as PoW or PoS. It is “democratic” in the sense that there is no trusted center, but that “democracy” comes at the cost of expensive consensus.
4. Git-like log – users, either manually or automatically according to application logic, create branches for changes and then merge them into each other when exchanging messages.

2. Host-based P2P
1. OT — an approach to collaborative editing where changes are represented as operations, such as insert/delete/replace, and competing operations are transformed relative to each other so that all participants converge to the same result.
2. PAXOS / Raft — consensus algorithms for log replication in a cluster: a group of nodes elects a leader, and the leader accepts commands, writes them to a log, and replicates them to a quorum.

1. Yjs — one of the most widely used CRDT libraries

2. Automerge – another CRDT implementation, but it stores data as an immutable graph, similar to a Git tree

3. OrbitDB – a database based on an “operation log” over CRDT, with a built-in libp2p communication protocol over IPFS and extended operations over CRDT structures

4. PouchDB – a client-side implementation of the CouchDB protocol

5. Gun.js — a distributed graph database with P2P synchronization

6. Hypercore – a distributed append-only immutable log, with a large ecosystem around it

7. SQLite + CRDT – libraries that combine CRDT and SQLite
1. CR-Sqlite
2. Sqlite-sync

In practice, only CRDT really makes sense

1. OT – most often still assumes a single node that collects the sequence of operations.

2. CouchDB – the protocol was designed around the idea of an eventual-consistency database where clients can connect to central nodes and transform part of the data. Attempts to make it fully P2P do exist today, but they are still highly experimental.

3. PAXOS / Raft – these are possible, but they only have an advantage over CRDT when (a) you need strict transactionality and (b) you can trust one of the participants. The implementation complexity is so high that PAXOS / RAFT only makes sense when there is no other viable option.

4. Blockchain – it is practically similar to PAXOS / Raft, but requires trust not in one node, but in a majority of nodes. It has the same high technological complexity, so its use should require an extremely strong reason.

5. Git-like log – an interesting model, but it assumes either user-side conflict resolution or simply does not fit many business cases.

Bonus

1. SQLite – even without built-in CRDT support, you can store CRDT content directly in a SQLite database. This can be useful if, for example, you want to index part of that content.

2. PGlite – similar to SQLite, but literally PostgreSQL compiled to WASM, and not only working, but also supporting extensions.

3. DuckDB – an embedded analytical database that can even load files by URL, for example directly from S3, which makes it an ideal candidate for use in clientful.

4. LanceDB – similar to DuckDB, but vector-oriented and designed out of the box for storage in external file stores.

b. Storage

1. Remote
1. Google Drive-like – an ideal candidate, because most clients already have this kind of storage. That means file storage can be fully moved to the user side.
2. S3-like – any MinIO, rustfs, or cloud S3 is well suited for this task.
3. CDN-like – using any CDN provider to upload and download files.
4. IPFS / Filecoin — decentralized crypto storage.
5. Nostr-like — a decentralized protocol for storing and exchanging data without a server.
6. Torrent – again, why not? It is literally a protocol for P2P data transfer and storage.
7. Git-like – using Git repositories for file persistence and transfer? Why not?

2. Local
1. Browser Local Storage / IndexedDB / OPFS — native browser storage, ideal for client-side-first applications.
2. File System – literally the local file system.

c. Communication

1. WebRTC – enables P2P connections between clients

2. WebSocket – persistent two-way communication over TCP

3. QUIC – a newer UDP-based protocol with TCP-like guarantees and TLS security

Holepunch

The Peer to Peer Company. Holepunch has 613 repositories available. Follow their code on GitHub.

https://github.com/holepunchto

d. Auth / Identity

1. Google Drive-like OAuth
• We use an OAuth provider as the identity source: Google, Microsoft, and so on.
• Tokens are needed not only for login, but also for access to the user’s File Storage, such as Drive, Dropbox, etc. In other words, authorization for file read/write operations is “built in” through the provider.
• Pros: minimal UX friction, a clear access model, and many users already have such accounts.
• Cons: dependency on the provider, API limits/quotas, and potential complexity around shared folders and access between users.

2. Sign-in with Ethereum (SIWE)
• Identity means ownership of the wallet’s private key; login means signing a challenge message with a nonce.
• Authorization between users can be built through on-chain or off-chain ACLs: for example, “who can read/write” is determined by a list of addresses and roles.
• Pros: portable identity, no central authority, convenient for web3/community products.
• Cons: UX, seed phrases, wallets, chain-specific behavior, and greater difficulty for a mass audience; an external data delivery channel such as storage or a broker is still often needed.

3. Local-first keys (device/user keys)
• Identity is built around locally generated keys, for example Ed25519/X25519:
• device key + user key and/or workspace keys.
• Public keys are distributed through File Storage, a registry, or invites; private keys never leave the devices.
• Authorization equals signatures and capability tokens, for example “this key can write to this folder/document”.
• Pros: maximally aligned with the clientful/privacy-by-default idea, and E2E encryption is possible.
• Cons: access recovery, key rotation, onboarding new devices, and access revocation.

4. AT Protocol (Bluesky stack)
• Identity equals a DID, Decentralized Identifier, bound to a handle/domain; account portability is provided by the protocol.
• AT can be used as an “identity + directory + transport” layer, while application data is stored in File Storage with permissions bound to DIDs.
• Pros: decentralized identity, built-in replication/federation mechanisms, and potential convenience for social-like products.
• Cons: the stack is relatively specific; you will need to carefully separate what lives in AT from what lives in your file storage / CRDT data.

e. (coming soon) Encryption

f. Backend-like

iv. Challenges

1. Dependency on the CRDT Implementation

2. Collaboration Only

• Use private-key signatures and public-key verification for every operation, and mark operations as untrusted if verification fails

• Use a Merkle Tree to recompute entity states while taking signatures into account

• Use blockchain-like structures

• Or assign one master client that can decide whether a given action is legal

3. Inconsistent States

1. A group of people is working on a document, and one of them boards a plane

2. While offline, that person deletes the document.

3. The others continue working on that document.

4. When the offline user comes back online, their state synchronizes; since nobody “cancelled the deletion”, the document is actually “deleted”, even though colleagues were working on it

• When “deleting”, first move the document to “trash” instead of deleting it; when coming back online, if work was done on the document, restore it automatically

• Or prohibit deleting documents while offline

• Or create a “deletion” task, but do not apply it until the user comes online and sees colleagues’ changes

4. “If X, then Y is allowed/forbidden” does not work

1. Client A creates a post

2. Client B receives this post

3. Client A creates a comment on it

4. Client B sees the comment

5. Client A deletes the post and, along with it, all comments, in this case one comment

6. Client A tries to create a new comment on the post, but is prevented from doing so because the post no longer exists

7. Client B has not yet received information about the deleted post, so they calmly create a second comment

8. Clients A and B receive each other’s changes, and the result is that the post is deleted while the second comment still exists

1. Aftermath Compensation – after state updates, check whether there are comments without posts and delete them

2. Preemptive Compensation – when creating a comment, in a local transaction, “recreate the post if it suddenly does not exist”. In other words, you send all clients both your comment and the post it belongs to; therefore, even if it was deleted on Client A, it will be created there again.

3. Conflict management – detect this conflict and show users a message about it; for example, do not fully delete the post until the deletion has synchronized across most nodes

5. Classic Problems of Offline Applications

• Data migrations – you must remember that the client already has data locally. You cannot simply “delete a key and replace it with a new one”; at some point, you will have to migrate data from the old format to the new one

• If state breaks, you cannot open a central database and inspect what happened. You will need mechanisms for remotely checking what happened to the client’s state

• Different versions on different devices

v. Bottom Line

vi. What Comes Next

1. I will explain and show how libraries for clientful are built

2. Real examples of problems in clientful applications, the technologies used, and their solutions

3. Analyses of existing projects

4. Studying transactionality in such systems

5. How clientful fits into the current ecosystem of AI pipelines and agents

6. Structures that extend CRDT capabilities, such as Merkle Tree, Merge Tree, etc.

7. Operation authorization

vii. Useful Links

a. Clientful-format Technologies

1. GitHubGitHub - alexanderop/awesome-local-first: Useful Links for Everything related to LocalFirst
   

   GitHub - alexanderop/awesome-local-first: Useful Links for Everything related to LocalFirst

   Useful Links for Everything related to LocalFirst. Contribute to alexanderop/awesome-local-first development by creating an account on GitHub.

   GitHub

2. www.localfirst.fm

3. Pears_p2pPears | Unleash the Power of P2P
   

   Pears | Unleash the Power of P2P

   Empowering Developers, Disrupting the Norm!

   Pears_p2p

b. Transactions in P2P Systems

1. YouTubeCRDT Research Meetup // AntidoteDB - Nuno Preguiça
   

   CRDT Research Meetup // AntidoteDB - Nuno Preguiça

   Originally recorded during the Lisbon Hack Week from May 21-25, 2018, this talk provides an overview of AntidoteDB and other CRDT-related projects on-going at NOVA LINCS. AntidoteDB (antidotedb.eu) is an open-source geo-replicated database that provides high availability while ensuring strong semantics for applications. To this end, AntidoteDB features CRDTs for providing data convergence, highly available transactions for providing atomic operations over multiple objects, causal consistency for enforcing the causal relations among updates, and mechanisms for enforcing global invariants without compromising availability. AntidoteDB is being developed in the context of EU H2020 LightKone project.

   YouTube

2. YouTubeAnnette Bieniusa - AntidoteDB: highly available, transactional database - Code BEAM Lite Munich 2018
   

   Annette Bieniusa - AntidoteDB: highly available, transactional database - Code BEAM Lite Munich 2018

   This video was recorded at Code BEAM Lite Munich 18: https://codesync.global/conferences/cbl-munich-2018/ Get involved in Code BEAM Lite conference: https://codesync.global/conferences/code-beam-lite-italy/ --- TALK TITLE AntidoteDB: a planet scale, highly available, transactional database by Annette Bieniusa Cloud-scale applications must be highly available and offer low latency responses while serving an ever-growing number of users concurrently. To achieve these goals, these apps rely on distributed cloud data stores that provide availability even under network partition. It takes significant effort and expertise from programmers to understand the intricate semantics and develop correct applications on top of such highly available databases. In my talk, I will show how AntidoteDB supports a principled approach to build highly-available, yet correct applications by combining conflict-free replicated data types (CRDTs), highly-available transactions and stronger synchronisation mechanisms. --- TALK OBJECTIVES You will learn how conflict-free replicated datatypes (CRDTs), transactions, and causal consistency interact to keep application data safe. Further, you will see how to identify patterns in your application that have specific synchronisation requirements und learn how to employ them with AntidoteDB. --- THEMES Consistency CRDTs AntidoteDB --- Code BEAM Lite LDN was brought to you by Code Sync. CODE SYNC Website: www.codesync.global Twitter: www.twitter.com/CodeMeshIO Facebook: https://www.facebook.com/CodeSyncGlobal LinkedIn: https://www.linkedin.com/company/code-sync/ Mail: info at codesync.global

   YouTube

c. CRDT

1. ElectricSQLIntroducing Rich-CRDTs | ElectricSQL
   

   Introducing Rich-CRDTs | ElectricSQL

   Rich-CRDTs are conflict-free data types extended to provide additional database guarantees.

   ElectricSQL

2. EDB Postgres Distributed (PGD) v4.3.8 - Conflicts
   

   EDB Postgres Distributed (PGD) v4.3.8 - Conflicts

   BDR is an active/active or multi-master DBMS. If used asynchronously, writes to the same or related rows from multiple different nodes can result in data conflicts when using standard data types.