Get excited
A reactive database framework
Build powerful React and React Native apps that scale from hundreds to tens of thousands of records and remain fast ⚡️
WatermelonDB | |
---|---|
⚡️ | Launch your app instantly no matter how much data you have |
📈 | Highly scalable from hundreds to tens of thousands of records |
😎 | Lazy loaded. Only load data when you need it |
🔄 | Offline-first. Sync with your own backend |
📱 | Multiplatform. iOS, Android, and the web |
⚛️ | Works with React. Easily plug data into components |
⏱ | Fast. And getting faster with every release! |
✅ | Proven. Powers Nozbe Teams since 2017 (and many others) |
✨ | Reactive. (Optional) RxJS API |
🔗 | Relational. Built on rock-solid SQLite foundation |
⚠️ | Static typing with Flow or TypeScript |
Why Watermelon?
WatermelonDB is a new way of dealing with user data in React Native and React web apps.
It's optimized for building complex applications in React Native, and the number one goal is real-world performance. In simple words, your app must launch fast.
For simple apps, using Redux or MobX with a persistence adapter is the easiest way to go. But when you start scaling to thousands or tens of thousands of database records, your app will now be slow to launch (especially on slower Android devices). Loading a full database into JavaScript is expensive!
Watermelon fixes it by being lazy. Nothing is loaded until it's requested. And since all querying is performed directly on the rock-solid SQLite database on a separate native thread, most queries resolve in an instant.
But unlike using SQLite directly, Watermelon is fully observable. So whenever you change a record, all UI that depends on it will automatically re-render. For example, completing a task in a to-do app will re-render the task component, the list (to reorder), and all relevant task counters. Learn more.
![]() | ![]() |
---|---|
📺 Next-generation React databases |
Usage
Quick (over-simplified) example: an app with posts and comments.
First, you define Models:
class Post extends Model {
@field('name') name
@field('body') body
@children('comments') comments
}
class Comment extends Model {
@field('body') body
@field('author') author
}
Then, you connect components to the data:
const Comment = ({ comment }) => (
<View style={styles.commentBox}>
<Text>{comment.body} — by {comment.author}</Text>
</View>
)
// This is how you make your app reactive! ✨
const enhance = withObservables(['comment'], ({ comment }) => ({
comment,
}))
const EnhancedComment = enhance(Comment)
And now you can render the whole Post:
const Post = ({ post, comments }) => (
<View>
<Text>{post.name}</Text>
<Text>Comments:</Text>
{comments.map(comment =>
<Comment key={comment.id} comment={comment} />
)}
</View>
)
const enhance = withObservables(['post'], ({ post }) => ({
post,
comments: post.comments
}))
The result is fully reactive! Whenever a post or comment is added, changed, or removed, the right components will automatically re-render on screen. Doesn't matter if a change occurred in a totally different part of the app, it all just works out of the box!
➡️ Learn more: see full documentation
Who uses WatermelonDB







Does your company or app use 🍉? Open a pull request and add your logo/icon with link here!
Contributing

WatermelonDB is an open-source project and it needs your help to thrive!
If there's a missing feature, a bug, or other improvement you'd like, we encourage you to contribute! Feel free to open an issue to get some guidance and see Contributing guide for details about project setup, testing, etc.
If you're just getting started, see good first issues that are easy to contribute to. If you make a non-trivial contribution, email me, and I'll send you a nice 🍉 sticker!
If you make or are considering making an app using WatermelonDB, please let us know!
Author and license
WatermelonDB was created by @Nozbe. Main author and maintainer is Radek Pietruszewski.
Contributors: @mobily, @kokusGr, @rozPierog, @rkrajewski, @domeknn, @Tereszkiewicz and more.
WatermelonDB is available under the MIT license. See the LICENSE file for more info.
Demo
See how WatermelonDB performs at large scales in the demo app.
Online demo

Check out WatermelonDB demo online
Note that where Watermelon really shines is in React Native apps — see instructions below ⬇️
Running React Native demo
To compile the WatermelonDB demo on your own machine:
- Install React Native toolkit if you haven't already
- Download this project
git clone https://github.com/Nozbe/WatermelonDB.git cd WatermelonDB/examples/native yarn
- Run the React Native packager:
yarn dev
- Run the app on iOS or Android:
yarn start:ios # or: yarn start:android
⚠️ Note that for accurate measurement of performance, you need to compile the demo app in Release mode and run it on a real device, not the simulator.
⚠️ If iOS app doesn't compile, try running it from Xcode instead of the terminal first
⚠️ You might want to git checkout
the latest stable tag if the demo app doesn't work
Running web demo
To compile the WatermelonDB demo on your own machine:
- Download this project
git clone https://github.com/Nozbe/WatermelonDB.git cd WatermelonDB/examples/web yarn
- Run the server:
yarn dev
- Webpack will point you to the right URL to open in the browser
You can also use Now to deploy the demo app (requires a Zeit account):
now
⚠️ You might want to git checkout
the latest stable tag if the demo app doesn't work
Learn to use Watermelon
Learn the basics of how to use WatermelonDB
Installation
First, add Watermelon to your project:
yarn add @nozbe/watermelondb
yarn add @nozbe/with-observables
or alternatively if you prefer npm:
npm install @nozbe/watermelondb
npm install @nozbe/with-observables
React Native setup
-
Install the Babel plugin for decorators if you haven't already:
yarn add --dev @babel/plugin-proposal-decorators
or
npm install -D @babel/plugin-proposal-decorators
-
Add ES6 decorators support to your
.babelrc
file:{ "presets": ["module:metro-react-native-babel-preset"], "plugins": [ ["@babel/plugin-proposal-decorators", { "legacy": true }] ] }
-
Set up your iOS or Android project — see instructions below
iOS (React Native)
-
Set up Babel config in your project
See instructions above ⬆️
-
Add Swift support to your Xcode project:
- Open
ios/YourAppName.xcodeproj
in Xcode - Right-click on Your App Name in the Project Navigator on the left, and click New File…
- Create a single empty
Swift
file to the project (make sure that Your App Name target is selected when adding), and when Xcode asks, press Create Bridging Header and do not removeSwift
file then.
- Open
-
Link WatermelonDB's native library with the Xcode project:
You can link WatermelonDB manually or using CocoaPods:
-
Manually
- Open your project in Xcode, right click on Libraries in the Project Navigator on the left and click Add Files to "Your Project Name". Look under
node_modules/@nozbe/watermelondb/native/ios
and selectWatermelonDB.xcodeproj
- Go to Project settings (top item in the Project navigator on the left), select your app name under Targets → Build Phases → Link Binary With Libraries, and add
libWatermelonDB.a
For more information about linking libraries manually, see React Native documentation.
- Open your project in Xcode, right click on Libraries in the Project Navigator on the left and click Add Files to "Your Project Name". Look under
-
Link WatermelonDB's native library with the Xcode project -- using CocoaPods:
-
Add this to your CocoaPods (might not be needed if you're using autolinking):
pod 'WatermelonDB', :path => '../node_modules/@nozbe/watermelondb'
-
Unfortunately, the build will fail due to an issue with React Native's Pods, so you need to modify this line:
# Before: pod 'React-jsi', :path => '../node_modules/react-native/ReactCommon/jsi' # Change to: pod 'React-jsi', :path => '../node_modules/react-native/ReactCommon/jsi', :modular_headers => true
-
Note that Xcode 9.4 and a deployment target of at least iOS 9.0 is required (although Xcode 11.5+ and iOS 12.0+ are recommended).
-
Android (React Native)
-
Set up Babel config in your project
See instructions above ⬆️
-
In
android/settings.gradle
, add:include ':watermelondb' project(':watermelondb').projectDir = new File(rootProject.projectDir, '../node_modules/@nozbe/watermelondb/native/android')
-
In
android/app/build.gradle
, add:apply plugin: "com.android.application" apply plugin: 'kotlin-android' // ⬅️ This! // ... dependencies { // ... implementation project(':watermelondb') // ⬅️ This! }
-
In
android/build.gradle
, add Kotlin support to the project:buildscript { ext.kotlin_version = '1.3.21' // ... dependencies { // ... classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version" } }
-
And finally, in
android/app/src/main/java/{YOUR_APP_PACKAGE}/MainApplication.java
, add:// ... import com.nozbe.watermelondb.WatermelonDBPackage; // ⬅️ This! // ... @Override protected List<ReactPackage> getPackages() { return Arrays.<ReactPackage>asList( new MainReactPackage(), new WatermelonDBPackage() // ⬅️ Here! ); }
-
Troubleshooting. If you get this error:
Can't find variable: Symbol
You might need a polyfill for ES6 Symbol:
yarn add es6-symbol
And in your
index.js
:import 'es6-symbol/implement'
Alternatively, we also recommend
jsc-android
, with which you don't need this polyfill, and it also makes your app faster.
NodeJS setup
-
Install better-sqlite3 peer dependency
yarn add --dev better-sqlite3
or
npm install -D better-sqlite3
Web setup
This guide assumes you use Webpack as your bundler.
- If you haven't already, install Babel plugins for decorators, static class properties, and async/await to get the most out of Watermelon. This assumes you use Babel 7 and already support ES6 syntax.
oryarn add --dev @babel/plugin-proposal-decorators yarn add --dev @babel/plugin-proposal-class-properties yarn add --dev @babel/plugin-transform-runtime
npm install -D @babel/plugin-proposal-decorators npm install -D @babel/plugin-proposal-class-properties npm install -D @babel/plugin-transform-runtime
- Add ES7 support to your
.babelrc
file:{ "plugins": [ ["@babel/plugin-proposal-decorators", { "legacy": true }], ["@babel/plugin-proposal-class-properties", { "loose": true }], [ "@babel/plugin-transform-runtime", { "helpers": true, "regenerator": true } ] ] }
If you want to use Web Worker for WatermelonDB (this has pros and cons, we recommend you start without Web Workers, and evaluate later if it makes sense for your app to use them):
-
Install worker-loader Webpack plugin to add support for Web Workers to your app:
yarn add --dev worker-loader
or
npm install -D worker-loader
-
And add this to Webpack configuration:
// webpack.config.js { module: { rules: [ // ⬇️ Add this: { test: /\.worker\.js$/, use: { loader: 'worker-loader' } } ] }, // ... output: { // ... globalObject: 'this', // ⬅️ And this } }
Set up Database
Create model/schema.js
in your project:
import { appSchema, tableSchema } from '@nozbe/watermelondb'
export default appSchema({
version: 1,
tables: [
// tableSchemas go here...
]
})
You'll need it for the next step. Now, in your index.js
:
import { Database } from '@nozbe/watermelondb'
import SQLiteAdapter from '@nozbe/watermelondb/adapters/sqlite'
import schema from './model/schema'
// import Post from './model/Post' // ⬅️ You'll import your Models here
// First, create the adapter to the underlying database:
const adapter = new SQLiteAdapter({
schema,
// dbName: 'myapp', // optional database name or file system path
// migrations, // optional migrations
synchronous: true, // synchronous mode only works on iOS. improves performance and reduces glitches in most cases, but also has some downsides - test with and without it
// experimentalUseJSI: true, // experimental JSI mode, use only if you're brave
})
// Then, make a Watermelon database from it!
const database = new Database({
adapter,
modelClasses: [
// Post, // ⬅️ You'll add Models to Watermelon here
],
actionsEnabled: true,
})
The above will work on React Native (iOS/Android) and NodeJS. For the web, instead of SQLiteAdapter
use LokiJSAdapter
:
import LokiJSAdapter from '@nozbe/watermelondb/adapters/lokijs'
const adapter = new LokiJSAdapter({
schema,
// migrations, // optional migrations
useWebWorker: false, // recommended for new projects. tends to improve performance and reduce glitches in most cases, but also has downsides - test with and without it
useIncrementalIndexedDB: true, // recommended for new projects. improves performance (but incompatible with early Watermelon databases)
// dbName: 'myapp', // optional db name
// It's recommended you implement this method:
// onIndexedDBVersionChange: () => {
// // database was deleted in another browser tab (user logged out), so we must make sure we delete
// // it in this tab as well
// if (checkIfUserIsLoggedIn()) {
// window.location.reload()
// }
// },
// Optional:
// onQuotaExceededError: (error) => { /* do something when user runs out of disk space */ },
})
// The rest is the same!
Next steps
➡️ After Watermelon is installed, define your app's schema
Schema
When using WatermelonDB, you're dealing with Models and Collections. However, underneath Watermelon sits an underlying database (SQLite or LokiJS) which speaks a different language: tables and columns. Together, those are called a database schema and we must define it first.
Defining a Schema
Say you want Models Post
, Comment
in your app. For each of those Models, you define a table. And for every field of a Model (e.g. name of the blog post, author of the comment) you define a column. For example:
// model/schema.js
import { appSchema, tableSchema } from '@nozbe/watermelondb'
export const mySchema = appSchema({
version: 1,
tables: [
tableSchema({
name: 'posts',
columns: [
{ name: 'title', type: 'string' },
{ name: 'subtitle', type: 'string', isOptional: true },
{ name: 'body', type: 'string' },
{ name: 'is_pinned', type: 'boolean' },
]
}),
tableSchema({
name: 'comments',
columns: [
{ name: 'body', type: 'string' },
{ name: 'post_id', type: 'string', isIndexed: true },
]
}),
]
})
Note: It is database convention to use plural and snake_case names for table names. Column names are also snake_case. So Post
become posts
and createdAt
becomes created_at
.
Column types
Columns have one of three types: string
, number
, or boolean
.
Fields of those types will default to ''
, 0
, or false
respectively, if you create a record with a missing field.
To allow fields to be null
, mark the column as isOptional: true
.
Naming conventions
To add a relation to a table (e.g. Post
where a Comment
was published, or author of a comment), add a string column ending with _id
:
{ name: 'post_id', type: 'string' },
{ name: 'author_id', type: 'string' },
Boolean columns should have names starting with is_
:
{ name: 'is_pinned', type: 'boolean' }
Date fields should be number
(dates are stored as Unix timestamps) and have names ending with _at
:
{ name: 'last_seen_at', type: 'number', isOptional: true }
Special columns
All tables automatically have a string column id
to uniquely identify records. (Also two special columns for sync purposes). You can add special created_at
/ updated_at
columns to enable automatic create/update tracking.
Modifying Schema
Whenever you change the Schema, you must increment the version number. During development, this will cause the database to clear completely on next launch.
To seamlessly change the schema (without deleting data), use Migrations.
⚠️ Always use Migrations if you already shipped your app.
Indexing
To enable database indexing, add isIndexed: true
to a column.
Indexing makes querying by a column faster, at the slight expense of create/update speed and database size.
For example, you will often want to query all comments belonging to a post (that is, query comments by its post_id
column), and so you should mark the post_id
column as indexed.
However, if you rarely query all comments by its author, indexing author_id
is probably not worth it.
In general, most _id
fields are indexed. Sometimes, boolean
fields are worth indexing if you often use it for queries. However, you should almost never index date (_at
) columns or string
columns.
Advanced
Unsafe SQL schema
If you want to modify the SQL used to set up the SQLite database, you can pass unsafeSql
parameter
to tableSchema
and appSchema
. This parameter is a function that receives SQL generated by Watermelon,
and you can return whatever you want - so you can append, prepend, replace parts of SQL, or return
your own SQL altogether. When passed to tableSchema
, it receives SQL generated for just that table,
and when to appSchema
- the entire schema SQL.
Note that SQL generated by WatermelonDB is not considered to be a stable API, so be careful about your transforms as they can break at any time.
appSchema({
...
tables: [
tableSchema({
name: 'tasks',
columns: [...],
unsafeSql: sql => sql.replace(/create table [^)]+\)/, '$& without rowid'),
}),
],
unsafeSql: sql => `create blabla;${sql}`,
})
Next steps
➡️ After you define your schema, go ahead and define your Models
Defining Models
A Model class represents a type of thing in your app. For example, Post
, Comment
, User
.
Before defining a Model, you first need to define its schema.
Create a Model
Let's define the Post
model:
// model/Post.js
import { Model } from '@nozbe/watermelondb'
export default class Post extends Model {
static table = 'posts'
}
Mark the table name for this Model — the same you defined in the schema.
Now add the new Model to Database
:
// index.js
import Post from 'model/Post'
const database = new Database({
// ...
modelClasses: [Post],
})
Associations
Your models almost surely relate to one another. A Post
has many Comment
s. And every Comment
belongs to a Post
. (Every relation is double-sided). Define those associations like so:
class Post extends Model {
static table = 'posts'
static associations = {
comments: { type: 'has_many', foreignKey: 'post_id' },
}
}
class Comment extends Model {
static table = 'comments'
static associations = {
posts: { type: 'belongs_to', key: 'post_id' },
}
}
On the "child" side (comments
) you define a belongs_to
association, and pass a column name (key) that points to the parent (post_id
is the ID of the post the comment belongs to).
On the "parent" side (posts
) you define an equivalent has_many
association and pass the same column name (here named foreignKey
).
Add fields
Next, define the Model's fields (properties). Those correspond to table columns defined earlier in the schema.
import { field } from '@nozbe/watermelondb/decorators'
class Post extends Model {
static table = 'posts'
static associations = {
comments: { type: 'has_many', foreignKey: 'post_id' },
}
@field('title') title
@field('body') body
@field('is_pinned') isPinned
}
Fields are defined using ES6 decorators. Pass column name you defined in Schema as the argument to @field
.
Field types. Fields are guaranteed to be the same type (string/number/boolean) as the column type defined in Schema. If column is marked isOptional: true
, fields may also be null.
Note: Why do I have to type the field/column name twice? The database convention is to use snake_case
for names, and the JavaScript convention is to use camelCase. So for any multi-word name, the two differ. Also, for resiliency, we believe it's better to be explicit, because over time, you might want to refactor how you name your JavaScript field names, but column names must stay the same for backward compatibility.
Date fields
For date fields, use @date
instead of @field
. This will return a JavaScript Date
object (instead of Unix timestamp integer).
import { date } from '@nozbe/watermelondb/decorators'
class Post extends Model {
// ...
@date('last_event_at') lastEventAt
}
Relation fields
To-one relation
To point to a related record, e.g. Post
a Comment
belongs to, or author (User
) of a Comment
, use @relation
:
import { relation } from '@nozbe/watermelondb/decorators'
class Comment extends Model {
// ...
@relation('posts', 'post_id') post
@relation('users', 'author_id') author
}
➡️ Learn more: Relation API
Children (To-Many relation)
To point to a list of records that belong to this Model, e.g. all Comment
s that belong to a Post
, you can define a simple Query
using @children
:
import { children } from '@nozbe/watermelondb/decorators'
class Post extends Model {
static table = 'posts'
static associations = {
comments: { type: 'has_many', foreignKey: 'post_id' },
}
@children('comments') comments
}
Pass the table name of the related records as an argument to @children
. The resulting property will be a Query
you can fetch, observe, or count.
Note: You must define a has_many
association in static associations
for this to work
➡️ Learn more: Queries
Advanced
Actions
Define actions to simplify creating and updating records.
➡️ Learn more: Actions
Queries
In addition to @children
, you can define custom Queries or extend existing ones.
➡️ Learn more: Queries
Advanced fields
You can also use these decorators:
@text
trims whitespace from user-input text@json
for complex serialized data@readonly
to make the field read-only@nochange
to disallow changes to the field after the first creation
➡️ Learn more: Advanced fields
Next steps
➡️ After you define some Models, learn the Create / Read / Update / Delete API
Create, Read, Update, Delete
When you have your Schema and Models defined, learn how to manipulate them!
Collections
The Collection
object is how you find, query, and create new records of a given type.
Get a collection
const postsCollection = database.collections.get('posts')
// Shortcut syntax:
const postsCollection = database.get('posts')
Pass the table name as the argument.
Find a record (by ID)
const post = await postsCollection.find('abcdef')
find()
returns a Promise. If the record cannot be found, the Promise will be rejected.
Query records
Find a list of records matching given conditions using .query()
:
const allPosts = await postsCollection.query().fetch()
const starredPosts = await postsCollection.query(Q.where('is_starred', true)).fetch()
➡️ Learn more: Queries
Modifying the database
To create, update, or delete records, use the respective operations wrapped in an Action:
await database.action(async () => {
const post = await postsCollection.find('abcdef')
await post.update( /* update the post */ )
await post.markAsDeleted()
})
➡️ Learn more: Actions
Create a new record
await database.action(async () => {
const newPost = await postsCollection.create(post => {
post.title = 'New post'
post.body = 'Lorem ipsum...'
})
})
.create()
takes a "builder function". In the example above, the builder will get a Post
object as an argument. Use this object to set values for fields you defined.
Note: Always await
the Promise returned by create
before you access the created record.
Note: You can only use field setters in create()
or update()
builder functions.
Update a record
await database.action(async () => {
await somePost.update(post => {
post.title = 'Updated title'
})
})
Like creating, updating takes a builder function, where you can use field setters.
Note: Always await
the Promise returned by update
before you access the modified record.
Delete a record
There are two ways of deleting records: syncable (mark as deleted), and permanent.
If you only use Watermelon as a local database, destroy records permanently, if you synchronize, mark as deleted instead.
await database.action(async () => {
await somePost.markAsDeleted() // syncable
await somePost.destroyPermanently() // permanent
})
Note: Don't access, update, or observe records after they're destroyed.
Advanced
Model.observe()
- usually you only use this when connecting records to components, but you can manually observe a record outside of React components. The returned RxJSObservable
will emit the record immediately upon subscription, and then every time the record is updated. If the record is deleted, the Observable will complete.Query.observe()
,Relation.observe()
— analagous to the above, but for Queries and RelationsQuery.observeWithColumns()
- used for sorted listsCollection.findAndObserve(id)
— same as using.find(id)
and then callingrecord.observe()
Model.prepareUpdate()
,Collection.prepareCreate
,Database.batch
— used for batch updatesDatabase.unsafeResetDatabase()
destroys the whole database - be sure to see this comment before using it- To override the
record.id
during the creation, e.g. to sync with a remote database, you can do it byrecord._raw
property. Be aware that theid
must be of typestring
.
await postsCollection.create(post => {
post._raw.id = serverId
})
Next steps
➡️ Now that you can create and update records, connect them to React components
Connecting to Components
After you define some Models, it's time to connect Watermelon to your app's interface. We're using React in this guide.
Install withObservables
The recommended way to use Watermelon with React is with withObservables
HOC (higher-order component). It doesn't come pre-packaged with Watermelon, but you can install it with:
yarn add @nozbe/with-observables
Note: If you're not familiar with higher-order components, read React documentation, check out recompose
… or just read the examples below to see it in practice!
Reactive components
Here's a very simple React component rendering a Comment
record:
const Comment = ({ comment }) => (
<div>
<p>{comment.body}</p>
</div>
)
Now we can fetch a comment: const comment = await commentsCollection.find(id)
and then render it: <Comment comment={comment} />
. The only problem is that this is not reactive. If the Comment is updated or deleted, the component will not re-render to reflect the changes. (Unless an update is forced manually or the parent component re-renders).
Let's enhance the component to make it observe the Comment
automatically:
const enhance = withObservables(['comment'], ({ comment }) => ({
comment // shortcut syntax for `comment: comment.observe()`
}))
const EnhancedComment = enhance(Comment)
Now, if we render <EnhancedComment comment={comment} />
, it will update every time the comment changes.
Reactive lists
Let's render the whole Post
with comments:
import withObservables from '@nozbe/with-observables'
const Post = ({ post, comments }) => (
<article>
<h1>{post.name}</h1>
<p>{post.body}</p>
<h2>Comments</h2>
{comments.map(comment =>
<EnhancedComment key={comment.id} comment={comment} />
)}
</article>
)
const enhance = withObservables(['post'], ({ post }) => ({
post,
comments: post.comments, // Shortcut syntax for `post.comments.observe()`
}))
const EnhancedPost = enhance(Post)
Notice a couple of things:
-
We're starting with a simple non-reactive
Post
component -
Like before, we enhance it by observing the
Post
. If the post name or body changes, it will re-render. -
To access comments, we fetch them from the database and observe using
post.comments.observe()
and inject a new propcomments
. (post.comments
is a Query created using@children
).Note that we can skip
.observe()
and just passpost.comments
for convenience —withObservables
will call observe for us -
By observing the Query, the
<Post>
component will re-render if a comment is created or deleted -
However, observing the comments Query will not re-render
<Post>
if a comment is updated — we render the<EnhancedComment>
so that it observes the comment and re-renders if necessary.
Reactive relations
The <Comment>
component we made previously only renders the body of the comment but doesn't say who posted it.
Assume the Comment
model has a @relation('users', 'author_id') author
field. Let's render it:
const Comment = ({ comment, author }) => (
<div>
<p>{comment.body} — by {author.name}</p>
</div>
)
const enhance = withObservables(['comment'], ({ comment }) => ({
comment,
author: comment.author, // shortcut syntax for `comment.author.observe()`
}))
const EnhancedComment = enhance(Comment)
comment.author
is a Relation object, and we can call .observe()
on it to fetch the User
and then observe changes to it. If author's name changes, the component will re-render.
Note again that we can also pass Relation
objects directly for convenience, skipping .observe()
Reactive counters
Let's make a <PostExcerpt>
component to display on a list of Posts, with only a brief summary of the contents and only the number of comments it has:
const PostExcerpt = ({ post, commentCount }) => (
<div>
<h1>{post.name}</h1>
<p>{getExcerpt(post.body)}</p>
<span>{commentCount} comments</span>
</div>
)
const enhance = withObservables(['post'], ({ post }) => ({
post: post.observe(),
commentCount: post.comments.observeCount()
}))
const EnhancedPostExcerpt = enhance(PostExcerpt)
This is very similar to normal <Post>
. We take the Query
for post's comments, but instead of observing the list of comments, we call observeCount()
. This is far more efficient. And as always, if a new comment is posted, or one is deleted, the component will re-render with the updated count.
Hey, what about React Hooks?
We get it — HOCs are so 2017, and Hooks are the future! And we agree.
Instead of using withObservables
HOC you can use an alternative open-source Hook for Rx Observables. But be warned that they are probably not as optimized for performance and WatermelonDB use as withObservables
.
If you'd like to see official useObservables
Hook - please contribute ❤️
Understanding withObservables
Let's unpack this:
withObservables(['post'], ({ post }) => ({
post: post.observe(),
commentCount: post.comments.observeCount()
}))
- Starting from the second argument,
({ post })
are the input props for the component. Here, we receivepost
prop with aPost
object. - These:
are the enhanced props we inject. The keys are props' names, and values are({ post: post.observe(), commentCount: post.comments.observeCount() })
Observable
objects. Here, we override thepost
prop with an observable version, and create a newcommentCount
prop. - The first argument:
['post']
is a list of props that trigger observation restart. So if a differentpost
is passed, that new post will be observed. If you pass[]
, the rendered Post will not change. You can pass multiple prop names if any of them should cause observation to re-start. - Rule of thumb: If you want to use a prop in the second arg function, pass its name in the first arg array
Advanced
- findAndObserve. If you have, say, a post ID from your Router (URL in the browser), you can use:
withObservables(['postId'], ({ postId, database }) => ({ post: database.collections.get('posts').findAndObserve(postId) }))
- RxJS transformations. The values returned by
Model.observe()
,Query.observe()
,Relation.observe()
are RxJS Observables. You can use standard transforms like mapping, filtering, throttling, startWith to change when and how the component is re-rendered. - Custom Observables.
withObservables
is a general-purpose HOC for Observables, not just Watermelon. You can create new props from anyObservable
.
Advanced: observing sorted lists
If you have a list that's dynamically sorted (e.g. sort comments by number of likes), use Query.observeWithColumns
to ensure the list is re-rendered when its order changes:
// This is a function that sorts an array of comments according to its `likes` field
// I'm using `ramda` functions for this example, but you can do sorting however you like
const sortComments = sortWith([
descend(prop('likes'))
])
const CommentList = ({ comments }) => (
<div>
{sortComments(comments).map(comment =>
<EnhancedComment key={comment.id} comment={comment} />
)}
</div>
)
const enhance = withObservables(['post'], ({ post }) => ({
comments: post.comments.observeWithColumns(['likes'])
}))
const EnhancedCommentList = enhance(CommentList)
If you inject post.comments.observe()
into the component, the list will not re-render to change its order, only if comments are added or removed. Instead, use query.observeWithColumns()
with an array of column names you use for sorting to re-render whenever a record on the list has any of those fields changed.
Advanced: observing 2nd level relations
If you have 2nd level relations, like author's Contact
info, and want to connect it to a component as well, you cannot simply use post.author.contact.observe()
in withComponents
. Before accessing and observing the Contact
relation, you need to resolve the author
itself. Here is the simplest way to do it:
const enhancePostAndAuthor = withObservables(['post'], ({post}) => ({
post,
author: post.author,
}));
const enhanceAuthorContact = withObservables(['author'], ({author}) => ({
contact: author.contact,
}));
const EnhancedPost = enhancePostAndAuthor(enhanceAuthorContact(PostComponent));
If you are familiar with rxjs
, another way to achieve the same result is using switchMap
operator:
import { switchMap } from 'rxjs/operators'
const enhancePost = withObservables(['post'], ({post}) => ({
post: post,
author: post.author,
contact: post.author.observe().pipe(switchMap(author => author.contact.observe()))
}));
const EnhancedPost = enhancePost(PostComponent);
Now PostComponent
will have Post
, Author
and Contact
props.
Note: If you have an optional relation between Post
and Author
, the enhanceAuthorContact
might receive null
as author
prop. For this case, as you must always return an observable for the contact
prop, you can use rxjs
of
function to create a default or empty Contact
prop:
import { of as of$ } from 'rxjs';
const enhanceAuthorContact = withObservables(['author'], ({author}) => ({
contact: author ? author.contact.observe() : of$(null)
}));
With the switchMap
approach, you can obtain the same result by doing:
contact: post.autor.observe().pipe(switchMap(author => author ? autor.contact : of$(null)))
Database Provider
To prevent prop drilling you can utilise the Database Provider and the withDatabase
Higher-Order Component.
import DatabaseProvider from '@nozbe/watermelondb/DatabaseProvider'
// ...
const database = new Database({
adapter,
modelClasses: [Blog, Post, Comment],
actionsEnabled: true,
})
render(
<DatabaseProvider database={database}>
<Root />
</DatabaseProvider>, document.getElementById('application')
)
To consume the database in your components you just wrap your component like so:
import { withDatabase } from '@nozbe/watermelondb/DatabaseProvider'
// ...
export default withDatabase(withObservables([], ({ database }) => ({
blogs: database.collections.get('blogs').query().observe(),
}))(BlogList))
The database prop in the withObservables
Higher-Order Component is provided by the database provider.
useDatabase
You can also consume Database
object using React Hooks syntax:
import { useDatabase } from '@nozbe/watermelondb/hooks'
const Component = () => {
const database = useDatabase()
}
Next steps
➡️ Next, learn more about custom Queries
Query API
Querying is how you find records that match certain conditions, for example:
- Find all comments that belong to a certain post
- Find all verified comments made by John
- Count all verified comments made by John or Lucy published under posts made in the last two weeks
Because queries are executed on the database, and not in JavaScript, they're really fast. It's also how Watermelon can be fast even at large scales, because even with tens of thousands of records total, you rarely need to load more than a few dozen records at app launch.
Defining Queries
@children
The simplest query is made using @children
. This defines a Query
for all comments that belong to a Post
:
class Post extends Model {
// ...
@children('comments') comments
}
➡️ Learn more: Defining Models
Extended Query
To narrow down a Query
(add extra conditions to an existing Query), use .extend()
:
import { children, lazy } from '@nozbe/watermelondb/decorators'
class Post extends Model {
// ...
@children('comments') comments
@lazy verifiedComments = this.comments.extend(Q.where('is_verified', true))
@lazy verifiedAwesomeComments = this.verifiedComments.extend(Q.where('is_awesome', true))
}
Note: Use the @lazy
when extending or defining new Queries for performance
Custom Queries
You can query any table using this.collections.get(tableName).query(conditions)
. Here, post.comments
will query all users that made a comment under post
.
class Post extends Model {
// ...
@lazy commenters = this.collections.get('users').query(
Q.on('comments', 'post_id', this.id)
)
}
Executing Queries
Most of the time, you connect Queries to Components by using observe
or observeCount
:
withObservables(['post'], ({ post }) => ({
post: post.observe(),
comments: post.comments.observe(),
verifiedCommentCount: post.verifiedComments.observeCount(),
}))
Fetch
To simply get the current list or current count, use fetch
/ fetchCount
. You might need it in Actions.
const comments = await post.comments.fetch()
const verifiedCommentCount = await post.verifiedComments.fetchCount()
// Shortcut syntax:
const comments = await post.comments
const verifiedCommentCount = await post.verifiedComments.count
Query conditions
import { Q } from '@nozbe/watermelondb'
// ...
commentCollection.query(
Q.where('is_verified', true)
)
This will query all comments that are verified (all comments with one condition: the is_verified
column of a comment must be true
).
When making conditions, you refer to column names of a table (i.e. is_verified
, not isVerified
). This is because queries are executed directly on the underlying database.
The second argument is the value we want to query for. Note that the passed argument must be the same type as the column (string
, number
, or boolean
; null
is allowed only if the column is marked as isOptional: true
in the schema).
Empty query
const allComments = await commentCollection.query().fetch()
A Query with no conditions will find all records in the collection.
Note: Don't do this unless necessary. It's generally more efficient to only query the exact records you need.
Multiple conditions
commentCollection.query(
Q.where('is_verified', true),
Q.where('is_awesome', true)
)
This queries all comments that are both verified and awesome.
Conditions with other operators
Query | JavaScript equivalent |
---|---|
Q.where('is_verified', true) | is_verified === true (shortcut syntax) |
Q.where('is_verified', Q.eq(true)) | is_verified === true |
Q.where('archived_at', Q.notEq(null)) | archived_at !== null |
Q.where('likes', Q.gt(0)) | likes > 0 |
Q.where('likes', Q.weakGt(0)) | likes > 0 (slightly different semantics — see "null behavior" for details) |
Q.where('likes', Q.gte(100)) | likes >= 100 |
Q.where('dislikes', Q.lt(100)) | dislikes < 100 |
Q.where('dislikes', Q.lte(100)) | dislikes <= 100 |
Q.where('likes', Q.between(10, 100)) | likes >= 10 && likes <= 100 |
Q.where('status', Q.oneOf(['published', 'draft'])) | status === 'published' \|\| status === 'draft' |
Q.where('status', Q.notIn(['archived', 'deleted'])) | status !== 'archived' && status !== 'deleted' |
Q.where('status', Q.like('%bl_sh%')) | /.*bl.sh.*/i (See note below!) |
Q.where('status', Q.notLike('%bl_sh%')) | /^((!?.*bl.sh.*).)*$/i (Inverse regex match) (See note below!) |
Note: It's NOT SAFE to use Q.like
and Q.notLike
with user input directly, because special characters like %
or _
are not escaped. Always sanitize user input like so:
Q.like(`%${Q.sanitizeLikeString(userInput)}%`)
Q.notLike(`%${Q.sanitizeLikeString(userInput)}%`)
You can use Q.like
for search-related tasks. For example, to find all users whose username start with "jas" (case-insensitive) you can write
usersCollection.query(
Q.where("username", Q.like(`${Q.sanitizeLikeString("jas")}%`)
)
where "jas"
can be changed dynamically with user input.
AND/OR nesting
You can nest multiple conditions using Q.and
and Q.or
:
commentCollection.query(
Q.where('archived_at', Q.notEq(null)),
Q.or(
Q.where('is_verified', true),
Q.and(
Q.where('likes', Q.gt(10)),
Q.where('dislikes', Q.lt(5))
)
)
)
This is equivalent to archivedAt !== null && (isVerified || (likes > 10 && dislikes < 5))
.
Conditions on related tables ("JOIN queries")
For example: query all comments under posts published by John:
// Shortcut syntax:
commentCollection.query(
Q.on('posts', 'author_id', john.id),
)
// Full syntax:
commentCollection.query(
Q.on('posts', Q.where('author_id', john.id)),
)
Normally you set conditions on the table you're querying. Here we're querying comments, but we have a condition on the post the comment belongs to.
The first argument for Q.on
is the table name you're making a condition on. The other two arguments are same as for Q.where
.
Note: The two tables must be associated before you can use Q.on
.
Multiple conditions on a related table
For example: query all comments under posts that are written by John and are either published or belong to draftBlog
commentCollection.query(
Q.on('posts', [
Q.where('author_id', john.id)
Q.or(
Q.where('published', true),
Q.where('blog_id', draftBlog.id),
)
]),
)
Instead of an array of conditions, you can also pass Q.and
, Q.or
, Q.where
, or Q.on
as the second argument to Q.on
.
Nesting Q.on
within AND/OR
If you want to place Q.on
nested within Q.and
and Q.or
, you must explicitly define all tables you're joining on. (NOTE: The Q.experimentalJoinTables
API is subject to change)
tasksCollection.query(
Q.experimentalJoinTables(['projects']),
Q.or(
Q.where('is_followed', true),
Q.on('projects', 'is_followed', true),
),
)
Deep Q.on
s
You can also nest Q.on
within Q.on
, e.g. to make a condition on a grandparent. You must explicitly define the tables you're joining on. (NOTE: The Q.experimentalNestedJoin
API is subject to change). Multiple levels of nesting are allowed.
// this queries tasks that are inside projects that are inside teams where team.foo == 'bar'
tasksCollection.query(
Q.experimentalNestedJoin('projects', 'teams'),
Q.on('projects', Q.on('teams', 'foo', 'bar')),
)
Advanced Queries
Advanced observing
Call query.observeWithColumns(['foo', 'bar'])
to create an Observable that emits a value not only when the list of matching records changes (new records/deleted records), but also when any of the matched records changes its foo
or bar
column. Use this for observing sorted lists
Count throttling
By default, calling query.observeCount()
returns an Observable that is throttled to emit at most once every 250ms. You can disable throttling using query.observeCount(false)
.
Column comparisons
This queries comments that have more likes than dislikes. Note that we're comparing likes
column to another column instead of a value.
commentCollection.query(
Q.where('likes', Q.gt(Q.column('dislikes')))
)
sortBy, take, skip
When using SQLite adapter, you can use these experimental clauses to sort the result of the query and to limit the number of results
commentCollection.query(
Q.experimentalSortBy('likes', Q.asc), // sorts ascending by `likes`
Q.experimentalSkip(100),
Q.experimentalTake(100),
)
NOTE: This does not currently work on web/LokiJS (please contribute!), and causes query observation to fall back to a less efficient method. We recommend using sortBy only when you absolutely need to limit queries, otherwise, it may be better to sort in JavaScript.
Security
Remember that Queries are a sensitive subject, security-wise. Never trust user input and pass it directly into queries. In particular:
- Never pass into queries values you don't know for sure are the right type (e.g. value passed to
Q.eq()
should be a string, number, boolean, or null -- but not an Object. If the value comes from JSON, you must validate it before passing it!) - Never pass column names (without whitelisting) from user input
- Values passed to
oneOf
,notIn
should be arrays of simple types - be careful they don't contain objects - Do not use
Q.like
/Q.notLike
withoutQ.sanitizeLikeString
- Do not use
unsafe raw queries
without knowing what you're doing and sanitizing all user input
Unsafe raw queries
If this Query syntax is not enough for you, and you need to get your hands dirty on a raw SQL or Loki query, you need rawQueries.
Please don't use this if you don't know what you're doing. The method name is called unsafe
for a reason.
SQL queries
For now, only record SQL queries are available. If you need other SQL queries or LokiJS raw queries, please contribute!
const records = commentCollection.unsafeFetchRecordsWithSQL('select * from comments where ...')
You need to be sure to properly sanitize user values to avoid SQL injection, and filter out deleted records using where _status is not 'deleted'
clause
SQL/Loki expressions
You can also include smaller bits of SQL and Loki expressions so that you can still use as much of Watermelon query builder as possible:
// SQL example:
postsCollection.query(
Q.where('is_published', true),
Q.unsafeSqlExpr('tasks.num1 not between 1 and 5'),
)
// LokiJS example:
postsCollection.query(
Q.where('is_published', true),
Q.unsafeLokiExpr({ text1: { $contains: 'hey' } })
)
For SQL, be sure to prefix column names with table name when joining with other tables.
Multi-table column comparisons and Q.unsafeLokiFilter
Example: we want to query comments posted more than 14 days after the post it belongs to was published.
There's sadly no built-in syntax for this, but can be worked around using unsafe expressions like so:
// SQL example:
commentsCollection.query(
Q.on('posts', 'published_at', Q.notEq(null)),
Q.unsafeSqlExpr(`comments.createad_at > posts.published_at + ${14 * 24 * 3600 * 1000}`)
)
// LokiJS example:
commentsCollection.query(
Q.on('posts', 'published_at', Q.notEq(null)),
Q.unsafeLokiFilter((record, loki) => {
const post = loki.getCollection('posts').by('id', record.post_id)
return post && record.created_at > post.published_at + 14 * 24 * 3600 * 1000
}),
)
For LokiJS, remember that record
is an unsanitized object and must not be mutated. Q.unsafeLokiFilter
only works when using LokiJSAdapter
with useWebWorkers: false
. There can only be one Q.unsafeLokiFilter
clause per query.
null
behavior
There are some gotchas you should be aware of. The Q.gt
, gte
, lt
, lte
, oneOf
, notIn
, like
operators match the semantics of SQLite in terms of how they treat null
. Those are different from JavaScript.
Rule of thumb: No null comparisons are allowed.
For example, if you query comments
for Q.where('likes', Q.lt(10))
, a comment with 8 likes and 0 likes will be included, but a comment with null
likes will not! In Watermelon queries, null
is not less than any number. That's why you should avoid making table columns optional unless you actually need it.
Similarly, if you query with a column comparison, like Q.where('likes', Q.gt(Q.column('dislikes')))
, only comments where both likes
and dislikes
are not null will be compared. A comment with 5 likes and null
dislikes will NOT be included. 5 is not greater than null
here.
Q.oneOf
operator: It is not allowed to pass null
as an argument to Q.oneOf
. Instead of Q.oneOf([null, 'published', 'draft'])
you need to explicitly allow null
as a value like so:
postsCollection.query(
Q.or(
Q.where('status', Q.oneOf(['published', 'draft'])),
Q.where('status', null)
)
)
Q.notIn
operator: If you query, say, posts with Q.where('status', Q.notIn(['published', 'draft']))
, it will match posts with a status different than published
or draft
, however, it will NOT match posts with status == null
. If you want to include such posts, query for that explicitly like with the example above.
Q.weakGt
operator: This is weakly typed version of Q.gt
— one that allows null comparisons. So if you query comments
with Q.where('likes', Q.weakGt(Q.column('dislikes')))
, it WILL match comments with 5 likes and null
dislikes. (For weakGt
, unlike standard operators, any number is greater than null
).
Contributing improvements to Watermelon query language
Here are files that are relevant. This list may look daunting, but adding new matchers is actually quite simple and multiple first-time contributors made these improvements (including like, sort, take, skip). The implementation is just split into multiple files (and their test files), but when you look at them, it'll be easy to add matchers by analogy.
We recommend starting from writing tests first to check expected behavior, then implement the actual behavior.
src/QueryDescription/test.js
- Test clause builder (Q.myThing
) output and test that it rejects bad/unsafe parameterssrc/QueryDescription/index.js
- Add clause builder and type definitionsrc/__tests__/databaseTests.js
- Add test ("join" if it requires conditions on related tables; "match" otherwise) that checks that the new clause matches expected records. From this, tests running against SQLite, LokiJS, and Matcher are generated. (If one of those is not supported, addskip{Loki,Sql,Count,Matcher}: true
to your test)src/adapters/sqlite/encodeQuery/test.js
- Test that your query generates SQL you expect. (If your clause is Loki-only, test that error is thrown)src/adapters/sqlite/encodeQuery/index.js
- Generate SQLsrc/adapters/lokijs/worker/encodeQuery/test.js
- Test that your query generates the Loki query you expect (If your clause is SQLite-only, test that an error is thrown)src/adapters/lokijs/worker/encodeQuery/index.js
- Generate Loki querysrc/adapters/lokijs/worker/{performJoins/*.js,executeQuery.js}
- May be relevant for some Loki queries, but most likely you don't need to look here.src/observation/encodeMatcher/
- If your query can be checked against a record in JavaScript (e.g. you're adding new "by regex" matcher), implement this behavior here (index.js
,operators.js
). This is used for efficient "simple observation". You don't need to write tests -databaseTests
are used automatically. If you can't or won't implement encodeMatcher for your query, add a check tocanEncode.js
so that it returnsfalse
for your query (Less efficient "reloading observation" will be used then). Add your query totest.js
's "unencodable queries" then.
Next steps
➡️ Now that you've mastered Queries, make more Relations
Relations
A Relation
object represents one record pointing to another — such as the author (User
) of a Comment
, or the Post
the comment belongs to.
Defining Relations
There's two steps to defining a relation:
-
A table column for the related record's ID
tableSchema({ name: 'comments', columns: [ // ... { name: 'author_id', type: 'string' }, ] }),
-
A
@relation
field defined on aModel
class:import { relation } from '@nozbe/watermelondb/decorators' class Comment extends Model { // ... @relation('users', 'author_id') author }
The first argument is the table name of the related record, and the second is the column name with an ID for the related record.
Relation API
In the example above, comment.author
returns a Relation
object.
Remember, WatermelonDB is a lazily-loaded database, so you don't get the related
User
record immediately, only when you explicitly fetch it
Observing
Most of the time, you connect Relations to Components by using observe()
(the same as with Queries):
withObservables(['comment'], ({ comment }) => ({
comment: comment.observe(),
author: comment.author.observe(),
}))
The component will now have an author
prop containing a User
, and will re-render both when the user changes (e.g. comment's author changes its name), but also when a new author is assigned to the comment (if that was possible).
Fetching
To simply get the related record, use fetch
. You might need it in Actions
const author = await comment.author.fetch()
// Shortcut syntax:
const author = await comment.author
Note: If the relation column (in this example, author_id
) is marked as isOptional: true
, fetch()
might return null
.
ID
If you only need the ID of a related record (e.g. to use in an URL or for the key=
React prop), use id
.
const authorId = comment.author.id
Assigning
Use set()
to assign a new record to the relation
await commentsCollection.create(comment => {
comment.author.set(someUser)
// ...
})
Note: you can only do this in the .create()
or .update()
block.
You can also use set id
if you only have the ID for the record to assign
await comment.update(() => {
comment.author.id = userId
})
Advanced relations
immutableRelation
If you have a relation that cannot change (for example, a comment can't change its author), you can use @immutableRelation
for extra protection and performance:
import { immutableRelation } from '@nozbe/watermelondb/decorators'
class Comment extends Model {
// ...
@immutableRelation('posts', 'post_id') post
@immutableRelation('users', 'author_id') author
}
Many-To-Many Relation
If for instance, our app Post
s can be authored by many User
s and a user can author many Post
s. We would create such a relation following these steps:-
- Create a pivot schema and model that both the
User
model andPost
model has association to; sayPostAuthor
- Create has_many association on both
User
andPost
pointing toPostAuthor
Model - Create belongs_to association on
PostAuthor
pointing to bothUser
andPost
- Retrieve all
Posts
for a user by defining a query that uses the pivotPostAuthor
to infer thePost
s that were authored by the User.
import {lazy } from '@nozbe/watermelondb/decorators'
class Post extends Model {
static table = 'posts'
static associations = {
post_authors: { type: 'has_many', foreignKey: 'post_id' },
}
@lazy
authors = this.collections
.get('users')
.query(Q.on('post_authors', 'post_id', this.id));
}
import { field } from '@nozbe/watermelondb/decorators'
class PostAuthor extends Model {
static table = 'post_authors'
static associations = {
posts: { type: 'belongs_to', key: 'post_id' },
users: { type: 'belongs_to', key: 'user_id' },
}
@field('post_id') postId
@field('user_id') userId
}
import {lazy } from '@nozbe/watermelondb/decorators'
class User extends Model {
static table = 'users'
static associations = {
post_authors: { type: 'has_many', foreignKey: 'user_id' },
}
@lazy
posts = this.collections
.get('posts')
.query(Q.on('post_authors', 'user_id', this.id));
}
withObservables(['post'], ({ post }) => ({
authors: post.authors.observe(),
}))
Next steps
➡️ Now the last step of this guide: define custom Actions
Actions
Although you can .create()
and .update()
records anywhere in your app, we recommend defining explicit Actions to encapsulate all ways to make changes.
Defining explicit Actions
An Action is a function that can modify the database (create, update, and delete records).
To define it, just add a method to a Model
class marked with the @action
decorator
import { action } from '@nozbe/watermelondb/decorators'
class Post extends Model {
// ...
@action async addComment(body, author) {
return await this.collections.get('comments').create(comment => {
comment.post.set(this)
comment.author.set(author)
comment.body = body
})
}
}
Note:
- Always mark actions as
async
and remember toawait
on.create()
and.update()
- You can use
this.collections
to accessDatabase.collections
Another example: updater action on Comment
:
class Comment extends Model {
// ...
@field('is_spam') isSpam
@action async markAsSpam() {
await this.update(comment => {
comment.isSpam = true
})
}
}
Now we can create a comment and immediately mark it as spam:
const comment = await post.addComment('Lorem ipsum', someUser)
await comment.markAsSpam()
Batch updates
Whenever you make more than one change (create, delete or update records) in an action, you should batch them.
It means that the app doesn't have to go back and forth with the database (sending one command, waiting for the response, then sending another), but instead sends multiple commands in one big batch. This is faster, safer, and can avoid subtle bugs in your app
Take an action that changes a Post
into spam:
class Post extends Model {
// ...
@action async createSpam() {
await this.update(post => {
post.title = `7 ways to lose weight`
})
await this.collections.get('comments').create(comment => {
comment.post.set(this)
comment.body = "Don't forget to comment, like, and subscribe!"
})
}
}
Let's modify it to use batching:
class Post extends Model {
// ...
@action async createSpam() {
await this.batch(
this.prepareUpdate(post => {
post.title = `7 ways to lose weight`
}),
this.collections.get('comments').prepareCreate(comment => {
comment.post.set(this)
comment.body = "Don't forget to comment, like, and subscribe!"
})
)
}
}
Note:
- Call
await this.batch
in the Action (outside of actions, you can also call.batch()
on theDatabase
object) - Pass the list of prepared operations as arguments:
- Instead of calling
await record.update()
, passrecord.prepareUpdate()
— note lack ofawait
- Instead of
await collection.create()
, usecollection.prepareCreate()
- Instead of
await record.markAsDeleted()
, userecord.prepareMarkAsDeleted()
- Instead of
await record.destroyPermanently()
, userecord.prepareDestroyPermanently()
- You can pass falsy values (null, undefined, false) to batch — they will simply be ignored.
- You can also pass a single array argument instead of a list of arguments
- Instead of calling
- Otherwise, the API is the same!
Calling Actions from Actions
If you try to call an Action from another Action, you'll notice that it won't work. This is because while Action is running, no other Action can run simultaneously. To override this behavior, wrap the Action call in this.subAction
:
class Comment extends Model {
// ...
@action async appendToPost() {
const post = await this.post.fetch()
// `appendToBody` is an `@action` on `Post`, so we call subAction to allow it
await this.subAction(() => post.appendToBody(this.body))
}
}
Delete action
When you delete, say, a Post
, you generally want all Comment
s that belong to it to be deleted as well.
To do this, override markAsDeleted()
(or destroyPermanently()
if you don't sync) to explicitly delete all children as well.
class Post extends Model {
static table = 'posts'
static associations = {
comments: { type: 'has_many', foreignKey: 'post_id' },
}
@children('comments') comments
async markAsDeleted() {
await this.comments.destroyAllPermanently()
await super.markAsDeleted()
}
}
Then to actually delete the post:
database.action(async () => {
await post.markAsDeleted()
})
Note:
- Use
Query.destroyAllPermanently()
on all dependent@children
you want to delete - Remember to call
super.markAsDeleted
— at the end of the method!
Inline actions
If you want to call a number of write operations outside of a Model action, do it like so:
const newPost = await database.action(async action => {
// Note: function passed to `database.action()` MUST be asynchronous
const posts = database.collections.get('posts')
const post = await posts.create( /* configure Post here */ )
// Note: to call an action from an inline action, call `action.subAction`:
await action.subAction(() => post.markAsPromoted())
// Note: Value returned from the wrapped function will be returned to `database.action` caller
return post
})
Advanced: Why actions are necessary?
WatermelonDB is highly asynchronous, which is a BIG challange in terms of achieving consistent data. Read this only if you are curious:
Consider a function
markCommentsAsSpam
that fetches a list of comments on a post, and then marks them all as spam. The two operations (fetching, and then updating) are asynchronous, and some other operation that modifies the database could run in between. And it could just happen to be a function that adds a new comment on this post. Even though the function completes successfully, it wasn't actually successful at its job.This example is trivial. But others may be far more dangerous. If a function fetches a record to perform an update on, this very record could be deleted midway through, making the action fail (and potentially causing the app to crash, if not handled properly). Or a function could have invariants determining whether the user is allowed to perform an action, that would be invalidated during action's execution. Or, in a collaborative app where access permissions are represented by another object, parallel execution of different actions could cause those access relations to be left in an inconsistent state.
The worst part is that analyzing all possible interactions for dangers is very hard, and having sync that runs automatically makes them very likely.
Solution? Group together related reads and writes together in an Action, enforce that writes MUST occur in an Action, and only allow one Action to run at the time. This way, it's guaranteed that in an action, you're looking at a consistent view of the world. On the other hand, most reads are safe to perform without grouping them. If you suspect they're not, you can also wrap them in an Action.
Next steps
➡️ Now that you've mastered all basics of Watermelon, go create some powerful apps — or keep reading advanced guides
Advanced guides
Advanced guides for using WatermelonDB
Migrations
Schema migrations is the mechanism by which you can add new tables and columns to the database in a backward-compatible way.
Without migrations, if a user of your app upgrades from one version to another, their local database will be cleared at launch, and they will lose all their data.
⚠️ Always use migrations!
Migrations setup
-
Add a new file for migrations:
// app/model/migrations.js import { schemaMigrations } from '@nozbe/watermelondb/Schema/migrations' export default schemaMigrations({ migrations: [ // We'll add migration definitions here later ], })
-
Hook up migrations to the Database adapter setup:
// index.js import migrations from 'model/migrations' const adapter = new SQLiteAdapter({ schema: mySchema, migrations, })
Migrations workflow
When you make schema changes when you use migrations, be sure to do this in this specific order, to minimize the likelihood of making an error.
Step 1: Add a new migration
First, define the migration - that is, define the change that occurs between two versions of schema (such as adding a new table, or a new table column).
Don't change the schema file yet!
// app/model/migrations.js
import { schemaMigrations, createTable } from '@nozbe/watermelondb/Schema/migrations'
export default schemaMigrations({
migrations: [
{
// ⚠️ Set this to a number one larger than the current schema version
toVersion: 2,
steps: [
// See "Migrations API" for more details
createTable({
name: 'comments',
columns: [
{ name: 'post_id', type: 'string', isIndexed: true },
{ name: 'body', type: 'string' },
],
}),
],
},
],
})
Refresh your simulator/browser. You should see this error:
Migrations can't be newer than schema. Schema is version 1 and migrations cover range from 1 to 2
If so, good, move to the next step!
But you might also see an error like "Missing table name in schema", which means you made an error in defining migrations. See "Migrations API" below for details.
Step 2: Make matching changes in schema
Now it's time to make the actual changes to the schema file — add the same tables or columns as in your migration definition
⚠️ Please double and triple check that your changes to schema match exactly the change you defined in the migration. Otherwise you risk that the app will work when the user migrates, but will fail if it's a fresh install — or vice versa.
⚠️ Don't change the schema version yet
// model/schema.js
export default appSchema({
version: 1,
tables: [
// This is our new table!
tableSchema({
name: 'comments',
columns: [
{ name: 'post_id', type: 'string', isIndexed: true },
{ name: 'body', type: 'string' },
],
}),
// ...
]
})
Refresh the simulator. You should again see the same "Migrations can't be newer than schema" error. If you see a different error, you made a syntax error.
Step 3: Bump schema version
Now that we made matching changes in the schema (source of truth about tables and columns) and migrations (the change in tables and columns), it's time to commit the change by bumping the version:
// model/schema.js
export default appSchema({
version: 2,
tables: [
// ...
]
})
If you refresh again, your app should show up without issues — but now you can use the new tables/columns
Step 4: Test your migrations
Before shipping a new version of the app, please check that your database changes are all compatible:
- Migrations test: Install the previous version of your app, then update to the version you're about to ship, and make sure it still works
- Fresh schema install test: Remove the app, and then install the new version of the app, and make sure it works
Why is this order important
It's simply because React Native simulator (and often React web projects) are configured to automatically refresh when you save a file. You don't want to database to accidentally migrate (upgrade) with changes that have a mistake, or changes you haven't yet completed making. By making migrations first, and bumping version last, you can double check you haven't made a mistake.
Migrations API
Each migration must migrate to a version one above the previous migration, and have multiple steps (such as adding a new table, or new columns). Larger example:
schemaMigrations({
migrations: [
{
toVersion: 3,
steps: [
createTable({
name: 'comments',
columns: [
{ name: 'post_id', type: 'string', isIndexed: true },
{ name: 'body', type: 'string' },
],
}),
addColumns({
table: 'posts',
columns: [
{ name: 'subtitle', type: 'string', isOptional: true },
{ name: 'is_pinned', type: 'boolean' },
],
}),
],
},
{
toVersion: 2,
steps: [
// ...
],
},
],
})
Migration steps:
createTable({ name: 'table_name', columns: [ ... ] })
- same API astableSchema()
addColumns({ table: 'table_name', columns: [ ... ] })
- you can add one or multiple columns to an existing table. The columns table has the same format as in schema definitions- Other types of migrations (e.g. deleting or renaming tables and columns) are not yet implemented. See
migrations/index.js
. Please contribute!
Database reseting and other edge cases
- When you're not using migrations, the database will reset (delete all its contents) whenever you change the schema version.
- If the migration fails, the database will fail to initialize, and will roll back to previous version. This is unlikely, but could happen if you, for example, create a migration that tries to create the same table twice. The reason why the database will fail instead of reset is to avoid losing user data (also it's less confusing in development). You can notice the problem, fix the migration, and ship it again without data loss.
- When database in the running app has newer database version than the schema version defined in code, the database will reset (clear its contents). This is useful in development
- If there's no available migrations path (e.g. user has app with database version 4, but oldest migration is from version 10 to 11), the database will reset.
Rolling back changes
There's no automatic "rollback" feature in Watermelon. If you make a mistake in migrations during development, roll back in this order:
- Comment out any changes made to schema.js
- Comment out any changes made to migrations.js
- Decrement schema version number (bring back the original number)
After refreshing app, the database should reset to previous state. Now you can correct your mistake and apply changes again (please do it in order described in "Migrations workflow").
Unsafe SQL migrations
Similar to Schema, you can add unsafeSql
parameter to every migration step to modify or replace SQL generated by WatermelonDB to perform the migration. There is also an unsafeExecuteSql('some sql;')
step you can use to append extra SQL. Those are ignored with LokiJSAdapter and for the purposes of migration syncs.
Synchronization
WatermelonDB has been designed from scratch to be able to seamlessly synchronize with a remote database (and, therefore, keep multiple copies of data synced with each other).
Note that Watermelon is only a local database — you need to bring your own backend. What Watermelon provides are:
- Synchronization primitives — information about which records were created, updated, or deleted locally since the last sync — and which columns exactly were modified. You can build your own custom sync engine using those primitives
- Built-in sync adapter — You can use the sync engine Watermelon provides out of the box, and you only need to provide two API endpoints on your backend that conform to Watermelon sync protocol
Using synchronize()
in your app
To synchronize, you need to pass two functions, pullChanges
and pushChanges
that talk to your backend and are compatible with Watermelon Sync Protocol. The frontend code will look something like this:
import { synchronize } from '@nozbe/watermelondb/sync'
async function mySync() {
await synchronize({
database,
pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => {
const response = await fetch(`https://my.backend/sync`, {
body: JSON.stringify({ lastPulledAt, schemaVersion, migration })
})
if (!response.ok) {
throw new Error(await response.text())
}
const { changes, timestamp } = await response.json()
return { changes, timestamp }
},
pushChanges: async ({ changes, lastPulledAt }) => {
const response = await fetch(`https://my.backend/sync?last_pulled_at=${lastPulledAt}`, {
method: 'POST',
body: JSON.stringify(changes)
})
if (!response.ok) {
throw new Error(await response.text())
}
},
migrationsEnabledAtVersion: 1,
})
}
Troubleshooting
⚠️ Note about a React Native / UglifyES bug. When you import Watermelon Sync, your app might fail to compile in release mode. To fix this, configure Metro bundler to use Terser instead of UglifyES. Run:
yarn add metro-minify-terser
Then, update metro.config.js
:
module.exports = {
// ...
transformer: {
// ...
minifierPath: 'metro-minify-terser',
},
}
You might also need to switch to Terser in Webpack if you use Watermelon for web.
Implementing pullChanges()
Watermelon will call this function to ask for changes that happened on the server since the last pull.
Arguments:
lastPulledAt
is a timestamp for the last time client pulled changes from server (ornull
if first sync)schemaVersion
is the current schema version of the local databasemigration
is an object representing schema changes since last sync (ornull
if up to date or not supported)
This function should fetch from the server the list of ALL changes in all collections since lastPulledAt
.
- You MUST pass an async function or return a Promise that eventually resolves or rejects
- You MUST pass
lastPulledAt
,schemaVersion
, andmigration
to an endpoint that conforms to Watermelon Sync Protocol - You MUST return a promise resolving to an object of this shape (your backend SHOULD return this shape already):
{ changes: { ... }, // valid changes object timestamp: 100000, // integer with *server's* current time }
- You MUST NOT store the object returned in
pullChanges()
. If you need to do any processing on it, do it before returning the object. Watermelon treats this object as "consumable" and can mutate it (for performance reasons)
Implementing pushChanges()
Watermelon will call this function with a list of changes that happened locally since the last push so you can post it to your backend.
Arguments passed:
{
changes: { ... }, // valid changes object
lastPulledAt: 10000, // the timestamp of the last successful pull (timestamp returned in pullChanges)
}
- You MUST pass
changes
andlastPulledAt
to a push sync endpoint conforming to Watermelon Sync Protocol - You MUST pass an async function or return a Promise from
pushChanges()
pushChanges()
MUST resolve after and only after the backend confirms it successfully received local changespushChanges()
MUST reject if backend failed to apply local changes- You MUST NOT resolve sync prematurely or in case of backend failure
- You MUST NOT mutate or store arguments passed to
pushChanges()
. If you need to do any processing on it, do it before returning the object. Watermelon treats this object as "consumable" and can mutate it (for performance reasons)
General information and tips
- You MUST NOT connect to backend endpoints you don't control using
synchronize()
. WatermelonDB assumes pullChanges/pushChanges are friendly and correct and does not guarantee secure behavior if data returned is malformed. - You SHOULD NOT call
synchronize()
while synchronization is already in progress (it will safely abort) - You MUST NOT reset local database while synchronization is in progress (push to server will be safely aborted, but consistency of the local database may be compromised)
- You SHOULD wrap
synchronize()
in a "retry once" block - if sync fails, try again once. This will resolve push failures due to server-side conflicts by pulling once again before pushing. - You can use
database.withChangesForTables
to detect when local changes occured to call sync. If you do this, you should debounce (or throttle) this signal to avoid callingsynchronize()
too often.
Adopting Migration Syncs
For Watermelon Sync to maintain consistency after migrations, you must support Migration Syncs (introduced in WatermelonDB v0.17). This allows Watermelon to request from backend the tables and columns it needs to have all the data.
- For new apps, pass
{migrationsEnabledAtVersion: 1}
tosynchronize()
(or the first schema version that shipped / the oldest schema version from which it's possible to migrate to the current version) - To enable migration syncs, the database MUST be configured with migrations spec (even if it's empty)
- For existing apps, set
migrationsEnabledAtVersion
to the current schema version before making any schema changes. In other words, this version should be the last schema version BEFORE the first migration that should support migration syncs. - Note that for apps that shipped before WatermelonDB v0.17, it's not possible to determine what was the last schema version at which the sync happened.
migrationsEnabledAtVersion
is used as a placeholder in this case. It's not possible to guarantee that all necessary tables and columns will be requested. (If user logged in when schema version was lower thanmigrationsEnabledAtVersion
, tables or columns were later added, and new records in those tables/changes in those columns occured on the server before user updated to an app version that has them, those records won't sync). To work around this, you may specifymigrationsEnabledAtVersion
to be the oldest schema version from which it's possible to migrate to the current version. However, this means that users, after updating to an app version that supports Migration Syncs, will request from the server all the records in new tables. This may be unacceptably inefficient. - WatermelonDB >=0.17 will note the schema version at which the user logged in, even if migrations are not enabled, so it's possible for app to request from backend changes from schema version lower than
migrationsEnabledAtVersion
- You MUST NOT delete old migrations, otherwise it's possible that the app is permanently unable to sync.
Adding logging to your sync
You can add basic sync logs to the sync process by passing an empty object to synchronize()
. Sync will then mutate the object, populating it with diagnostic information (start/finish time, resolved conflicts, and more):
const log = {}
await synchronize({
database,
log,
...
})
console.log(log.startedAt)
console.log(log.finishedAt)
⚠️ Remember to act responsibly with logs, since they might contain your user's private information. Don't display, save, or send the log unless you censor the log. Example logger and censor code you can use.
Additional synchronize()
flags
_unsafeBatchPerCollection: boolean
- if true, changes will be saved to the database in multiple batches. This is unsafe and breaks transactionality, however may be required for very large syncs due to memory issuessendCreatedAsUpdated: boolean
- if your backend can't differentiate between created and updated records, set this totrue
to supress warnings. Sync will still work well, however error reporting, and some edge cases will not be handled as well.conflictResolver: (TableName, local: DirtyRaw, remote: DirtyRaw, resolved: DirtyRaw) => DirtyRaw
- can be passed to customize how records are updated when they change during sync. Seesrc/sync/index.js
for details.
Implementing your Sync backend
Understanding changes
objects
Synchronized changes (received by the app in pullChanges
and sent to the backend in pushChanges
) are represented as an object with raw records. Those only use raw table and column names, and raw values (strings/numbers/booleans) — the same as in Schema.
Deleted objects are always only represented by their IDs.
Example:
{
projects: {
created: [
{ id: 'aaaa', name: 'Foo', is_favorite: true },
{ id: 'bbbb', name: 'Bar', is_favorite: false },
],
updated: [
{ id: 'ccc', name: 'Baz', is_favorite: true },
],
deleted: ['ddd'],
},
tasks: {
created: [],
updated: [
{ id: 'tttt', name: 'Buy eggs' },
],
deleted: [],
},
...
}
Valid changes objects MUST conform to this shape:
Changes = {
[table_name: string]: {
created: RawRecord[],
updated: RawRecord[],
deleted: string[],
}
}
Implementing pull endpoint
Expected parameters:
{
lastPulledAt: Timestamp,
schemaVersion: int,
migration: null | { from: int, tables: string[], columns: { table: string, columns: string[] }[] }
}
Expected response:
{ changes: Changes, timestamp: Timestamp }
- The pull endpoint SHOULD take parameters and return a response matching the shape specified above.
This shape MAY be different if negotiated with the frontend (however, frontend-side
pullChanges()
MUST conform to this) - The pull endpoint MUST return all record changes in all collections since
lastPulledAt
, specifically:- all records that were created on the server since
lastPulledAt
- all records that were updated on the server since
lastPulledAt
- IDs of all records that were deleted on the server since
lastPulledAt
- all records that were created on the server since
- If
lastPulledAt
is null or 0, you MUST return all accessible records (first sync) - The timestamp returned by the server MUST be a value that, if passed again to
pullChanges()
aslastPulledAt
, will return all changes that happened since this moment. - The pull endpoint MUST provide a consistent view of changes since
lastPulledAt
- You should perform all queries synchronously or in a write lock to ensure that returned changes are consistent
- You should also mark the current server time synchronously with the queries
- This is to ensure that no changes are made to the database while you're fetching changes (otherwise some records would never be returned in a pull query)
- If it's absolutely not possible to do so, and you have to query each collection separately, be sure to return a
lastPulledAt
timestamp marked BEFORE querying starts. You still risk inconsistent responses (that may break app's consistency assumptions), but the next pull will fetch whatever changes occured during previous pull. - An alternative solution is to check for the newest change before and after all queries are made, and if there's been a change during the pull, return an error code, or retry.
- If
migration
is not null, you MUST include records needed to get a consistent view after a local database migration- Specifically, you MUST include all records in tables that were added to the local database between the last user sync and
schemaVersion
- For all columns that were added to the local app database between the last sync and
schemaVersion
, you MUST include all records for which the added column has a value other than the default value (0
,''
,false
, ornull
depending on column type and nullability) - You can determine what schema changes were made to the local app in two ways:
- You can compare
migration.from
(local schema version at the time of the last sync) andschemaVersion
(current local schema version). This requires you to negotiate with the frontend what schema changes are made at which schema versions, but gives you more control - Or you can ignore
migration.from
and only look atmigration.tables
(which indicates which tables were added to the local database since the last sync) andmigration.columns
(which indicates which columns were added to the local database to which tables since last sync). - If you use
migration.tables
andmigration.columns
, you MUST whitelist values a client can request. Take care not to leak any internal fields to the client.
- You can compare
- Specifically, you MUST include all records in tables that were added to the local database between the last user sync and
- Returned raw records MUST match your app's Schema
- Returned raw records MUST NOT not contain special
_status
,_changed
fields. - Returned raw records MAY contain fields (columns) that are not yet present in the local app (at
schemaVersion
-- but added in a later version). They will be safely ignored. - Returned raw records MUST NOT contain arbitrary column names, as they may be unsafe (e.g.
__proto__
orconstructor
). You should whitelist acceptable column names. - Returned record IDs MUST only contain safe characters
- Default WatermelonDB IDs conform to
/^[a-zA-Z0-9]{16}$/
_-.
are also allowed if you override default ID generator, but'"\/$
are unsafe
- Default WatermelonDB IDs conform to
- Changes SHOULD NOT contain collections that are not yet present in the local app (at
schemaVersion
). They will, however, be safely ignored.- NOTE: This is true for WatermelonDB v0.17 and above. If you support clients using earlier versions, you MUST NOT return collections not known by them.
- Changes MUST NOT contain collections with arbitrary names, as they may be unsafe. You should whitelist acceptable collection names.
Implementing push endpoint
- The push endpoint MUST apply local changes (passed as a
changes
object) to the database. Specifically:- create new records as specified by the changes object
- update existing records as specified by the changes object
- delete records by the specified IDs
- If the
changes
object contains a new record with an ID that already exists, you MUST update it, and MUST NOT return an error code.- (This happens if previous push succeeded on the backend, but not on frontend)
- If the
changes
object contains an update to a record that does not exist, then:- If you can determine that this record no longer exists because it was deleted, you SHOULD return an error code (to force frontend to pull the information about this deleted ID)
- Otherwise, you MUST create it, and MUST NOT return an error code. (This scenario should not happen, but in case of frontend or backend bugs, it would keep sync from ever succeeding.)
- If the
changes
object contains a record to delete that doesn't exist, you MUST ignore it and MUST NOT return an error code- (This may happen if previous push succeeded on the backend, but not on frontend, or if another user deleted this record in between user's pull and push calls)
- If the
changes
object contains a record that has been modified on the server afterlastPulledAt
, you MUST abort push and return an error code- This scenario means that there's a conflict, and record was updated remotely between user's pull and push calls. Returning an error forces frontend to call pull endpoint again to resolve the conflict
- If application of all local changes succeeds, the endpoint MUST return a success status code.
- The push endpoint MUST be fully transactional. If there is an error, all local changes MUST be reverted on the server, and en error code MUST be returned.
- You MUST ignore
_status
and_changed
fields contained in records inchanges
object - You SHOULD validate data passed to the endpoint. In particular, collection and column names ought to be whitelisted, as well as ID format — and of course any application-specific invariants, such as permissions to access and modify records
- You SHOULD sanitize record fields passed to the endpoint. If there's something slightly wrong with the contents (but not shape) of the data (e.g.
user.role
should beowner
,admin
, ormember
, but user sent empty string orabcdef
), you SHOULD NOT send an error code. Instead, prefer to "fix" errors (sanitize to correct format).- Rationale: Synchronization should be reliable, and should not fail other than transiently, or for serious programming errors. Otherwise, the user will have a permanently unsyncable app, and may have to log out/delete it and lose unsynced data. You don't want a bug 5 versions ago to create a persistently failing sync.
- You SHOULD delete all descendants of deleted records
- Frontend should ask the push endpoint to do so as well, but if it's buggy, you may end up with permanent orphans
Tips on implementing server-side changes tracking
If you're wondering how to actually implement consistent pulling of all changes since the last pull, or how to detect that a record being pushed by the user changed after lastPulledAt
, here's what we recommend:
- Add a
last_modified
field to all your server database tables, and bump it toNOW()
every time you create or update a record. - This way, when you want to get all changes since
lastPulledAt
, you query records whoselast_modified > lastPulledAt
. - The timestamp should be at least millisecond resolution, and you should add (for extra safety) a MySQL/PostgreSQL procedure that will ensure
last_modified
uniqueness and monotonicity- Specificaly, check that there is no record with a
last_modified
equal to or greater thanNOW()
, and if there is, increment the new timestamp by 1 (or however much you need to ensure it's the greatest number) - An example of this for PostgreSQL can be found in Kinto
- This protects against weird edge cases - such as records being lost due to server clock time changes (NTP time sync, leap seconds, etc.)
- Specificaly, check that there is no record with a
- Of course, remember to ignore
last_modified
from the user if you do it this way. - An alternative to using timestamps is to use an auto-incrementing counter sequence, but you must ensure that this sequence is consistent across all collections. You also leak to users the amount of traffic to your sync server (number of changes in the sequence)
- To distinguish between
created
andupdated
records, you can also store server-sideserver_created_at
timestamp (if it's greater thanlast_pulled_at
supplied to sync, then record is to becreated
on client, if less than — client already has it and it is to beupdated
on client). Note that this timestamp must be consistent with last_modified — and you must not use client-createdcreated_at
field, since you can never trust local timestamps.- Alternatively, you can send all non-deleted records as all
updated
and Watermelon will do the right thing in 99% of cases (you will be slightly less protected against weird edge cases — treatment of locally deleted records is different). If you do this, passsendCreatedAsUpdated: true
tosynchronize()
to supress warnings about records to be updated not existing locally.
- Alternatively, you can send all non-deleted records as all
- You do need to implement a mechanism to track when records were deleted on the server, otherwise you wouldn't know to push them
- One possible implementation is to not fully delete records, but mark them as DELETED=true
- Or, you can have a
deleted_xxx
table with just the record ID and timestamp (consistent with last_modified) - Or, you can treat it the same way as "revoked permissions"
- If you have a collaborative app with any sort of permissions, you also need to track granting and revoking of permissions the same way as changes to records
- If permission to access records has been granted, the pull endpoint must add those records to
created
- If permission to access records has been revoked, the pull endpoint must add those records to
deleted
- Remember to also return all descendants of a record in those cases
- If permission to access records has been granted, the pull endpoint must add those records to
Local vs Remote IDs
WatermelonDB has been designed with the assumption that there is no difference between Local IDs (IDs of records and their relations in a WatermelonDB database) and Remote IDs (IDs on the backend server). So a local app can create new records, generating their IDs, and the backend server will use this ID as the true ID. This greatly simplifies synchronization, as you don't have to replace local with remote IDs on the record and all records that point to it.
We highly recommend that you adopt this practice.
Some people are skeptical about this approach due to conflicts, since backend can guarantee unique IDs, and the local app can't. However, in practice, a standard Watermelon ID has 8,000,000,000,000,000,000,000,000 possible combinations. That's enough entropy to make conflicts extremely unlikely. At Nozbe, we've done it this way at scale for more than a decade, and not once did we encounter a genuine ID conflict or had other issues due to this approach.
Using the birthday problem, we can calculate that for 36^16 possible IDs, if your system grows to a billion records, the probability of a single conflict is 6e-8. At 100B records, the probability grows to 0.06%. But if you grow to that many records, you're probably a very rich company and can start worrying about things like this then.
If you absolutely can't adopt this practice, there's a number of production apps using WatermelonDB that keep local and remote IDs separate — however, more work is required this way. Search Issues to find discussions about this topic — and consider contributing to WatermelonDB to make managing separate local IDs easier for everyone!
Existing backend implementations for WatermelonDB
Note that those are not maintained by WatermelonDB, and we make no endorsements about quality of these projects:
- How to Build WatermelonDB Sync Backend in Elixir
- Firemelon
- Did you make one? Please contribute a link!
Current Sync limitations
- If a record being pushed changes between pull and push, push will just fail. It would be better if it failed with a list of conflicts, so that
synchronize()
can automatically respond. Alternatively, sync could only send changed fields and server could automatically always just apply those changed fields to the server version (since that's what per-column client-wins resolver will do anyway) - During next sync pull, changes we've just pushed will be pulled again, which is unnecessary. It would be better if server, during push, also pulled local changes since
lastPulledAt
and responded with NEW timestamp to be treated aslastPulledAt
. - It shouldn't be necessary to push the whole updated record — just changed fields + ID should be enough
Note: That might conflict with "If client wants to update a record that doesn’t exist, create it"
You don't like these limitations? Good, neither do we! Please contribute - we'll give you guidance.
Contributing
- If you implement Watermelon sync but found this guide confusing, please contribute improvements!
- Please help out with solving the current limitations!
- If you write server-side code made to be compatible with Watermelon, especially for popular platforms (Node, Ruby on Rails, Kinto, etc.) - please open source it and let us know! This would dramatically simplify implementing sync for people
- If you find Watermelon sync bugs, please report the issue! And if possible, write regression tests to make sure it never happens again
Sync primitives and implementing your own sync entirely from scratch
See: Sync implementation details
Create/Update tracking
You can add per-table support for create/update tracking. When you do this, the Model will have information about when it was created, and when it was last updated.
When to use this
Use create tracking:
- When you display to the user when a thing (e.g. a Post, Comment, Task) was created
- If you sort created items chronologically (Note that Record IDs are random strings, not auto-incrementing integers, so you need create tracking to sort chronologically)
Use update tracking:
- When you display to the user when a thing (e.g. a Post) was modified
Note: you don't have to enable both create and update tracking. You can do either, both, or none.
How to do this
Step 1: Add to the schema:
tableSchema({
name: 'posts',
columns: [
// other columns
{ name: 'created_at', type: 'number' },
{ name: 'updated_at', type: 'number' },
]
}),
Step 2: Add this to the Model definition:
import { date, readonly } from '@nozbe/watermelondb/decorators'
class Post extends Model {
// ...
@readonly @date('created_at') createdAt
@readonly @date('updated_at') updatedAt
}
Again, you can add just created_at
column and field if you don't need update tracking.
How this behaves
If you have the magic createdAt
field defined on the Model, the current timestamp will be set when you first call collection.create()
or collection.prepareCreate()
. It will never be modified again.
If the magic updatedAt
field is also defined, then after creation, model.updatedAt
will have the same value as model.createdAt
. Then every time you call model.update()
or model.prepareUpdate()
, updatedAt
will be changed to the current timestamp.
Advanced Fields
@text
You can use @text
instead of @field
to enable user text sanitization. When setting a new value on a @text
field, excess whitespace will be trimmed from both ends from the string.
import { text } from '@nozbe/watermelondb/decorators'
class Post extends Model {
// ...
@text('body') body
}
@json
If you have a lot of metadata about a record (say, an object with many keys, or an array of values), you can use a @json
field to contain that information in a single string column (serialized to JSON) instead of adding multiple columns or a relation to another table.
⚠️ This is an advanced feature that comes with downsides — make sure you really need it
First, add a string column to the schema:
tableSchema({
name: 'comments',
columns: [
{ name: 'reactions', type: 'string' }, // You can add isOptional: true, if appropriate
],
})
Then in the Model definition:
import { json } from '@nozbe/watermelondb/decorators'
class Comment extends Model {
// ...
@json('reactions', sanitizeReactions) reactions
}
Now you can set complex JSON values to a field:
comment.update(() => {
comment.reactions = ['up', 'down', 'down']
})
As the second argument, pass a sanitizer function. This is a function that receives whatever JSON.parse()
returns for the serialized JSON, and returns whatever type you expect in your app. In other words, it turns raw, dirty, untrusted data (that might be missing, or malformed by a bug in previous version of the app) into trusted format.
The sanitizer might also receive null
if the column is nullable, or undefined
if the field doesn't contain valid JSON.
For example, if you need the field to be an array of strings, you can ensure it like so:
const sanitizeReactions = rawReactions => {
return Array.isArray(rawReactions) ? rawReactions.map(String) : []
}
If you don't want to sanitize JSON, pass an identity function:
const sanitizeReactions = json => json
The sanitizer function takes an optional second argument, which is a reference to the model. This is useful is your sanitization logic depends on the other fields in the model.
Warning about JSON fields:
JSON fields go against relational, lazy nature of Watermelon, because you can't query or count by the contents of JSON fields. If you need or might need in the future to query records by some piece of data, don't use JSON.
Only use JSON fields when you need the flexibility of complex freeform data, or the speed of having metadata without querying another table, and you are sure that you won't need to query by those metadata.
@nochange
For extra protection, you can mark fields as @nochange
to ensure they can't be modified. Always put @nochange
before @field
/ @date
/ @text
import { field, nochange } from '@nozbe/watermelondb/decorators'
class User extends Model {
// ...
@nochange @field('is_owner') isOwner
}
user.isOwner
can only be set in the collection.create()
block, but will throw an error if you try to set a new value in user.update()
block.
@readonly
Similar to @nochange
, you can use the @readonly
decorator to ensure a field cannot be set at all. Use this for create/update tracking, but it might also be useful if you use Watermelon with a Sync engine and a field can only be set by the server.
Custom observable fields
You're in advanced RxJS territory now! You have been warned.
Say, you have a Post model that has many Comments. And a Post is considered to be "popular" if it has more than 10 comments.
You can add a "popular" badge to a Post component in two ways.
One is to simply observe how many comments there are in the component:
const enhance = withObservables(['post'], ({ post }) => ({
post: post.observe(),
commentCount: post.comments.observeCount()
}))
And in the render
method, if props.commentCount > 10
, show the badge.
Another way is to define an observable property on the Model layer, like so:
import { distinctUntilChanged, map as map$ } from 'rxjs/operators'
import { lazy } from '@nozbe/watermelondb/decorators'
class Post extends Model {
@lazy isPopular = this.comments.observeCount().pipe(
map$(comments => comments > 10),
distinctUntilChanged()
)
}
And then you can directly connect this to the component:
const enhance = withObservables(['post'], ({ post }) => ({
isPopular: post.isPopular,
}))
props.isPopular
will reflect whether or not the Post is popular. Note that this is fully observable, i.e. if the number of comments rises above/falls below the popularity threshold, the component will re-render. Let's break it down:
this.comments.observeCount()
- take the Observable number of commentsmap$(comments => comments > 10)
- transform this into an Observable of boolean (popular or not)distinctUntilChanged()
- this is so that if the comment count changes, but the popularity doesn't (it's still below/above 10), components won't be unnecessarily re-rendered@lazy
- also for performance (we only define this Observable once, so we can re-use it for free)
Let's make this example more complicated. Say the post is always popular if it's marked as starred. So if post.isStarred
, then we don't have to do unnecessary work of fetching comment count:
import { of as of$ } from 'rxjs/observable/of'
import { distinctUntilChanged, map as map$ } from 'rxjs/operators'
import { lazy } from '@nozbe/watermelondb/decorators'
class Post extends Model {
@lazy isPopular = this.observe().pipe(
distinctUntilKeyChanged('isStarred'),
switchMap(post =>
post.isStarred ?
of$(true) :
this.comments.observeCount().pipe(map$(comments => comments > 10))
),
distinctUntilChanged(),
)
}
this.observe()
- if the Post changes, it might change its popularity status, so we observe itthis.comments.observeCount().pipe(map$(comments => comments > 10))
- this part is the same, but we only observe it if the post is starredswitchMap(post => post.isStarred ? of$(true) : ...)
- if the post is starred, we just return an Observable that emitstrue
and never changes.distinctUntilKeyChanged('isStarred')
- for performance, so that we don't re-subscribe to comment count Observable if the post changes (only if theisStarred
field changes)distinctUntilChanged()
- again, don't emit new values, if popularity doesn't change
Watermelon ❤️ Flow
Watermelon was developed with Flow in mind.
If you're a Flow user yourself (and we highly recommend it!), here's some things you need to keep in mind:
Setup
Add this to your .flowconfig
file so that Flow can see Watermelon's types.
[options]
module.name_mapper='^@nozbe/watermelondb\(.*\)$' -> '<PROJECT_ROOT>/node_modules/@nozbe/watermelondb/src\1'
Note that this won't work if you put the entire node_modules/
folder under the [ignore]
section. In that case, change it to only ignore the specific node modules that throw errors in your app, so that Flow can scan Watermelon files.
Tables and columns
Table and column names are opaque types in Flow.
So if you try to use simple strings, like so:
class Comment extends Model {
static table = 'comments'
@text('body') body
}
You'll get errors, because you're passing 'comments'
(a string
) where TableName<Comment>
is expected, and 'body'
(again, a string
) where ColumnName
is expected.
When using Watermelon with Flow, you must pre-define all your table and column names in one place, then only use those symbols (and not strings) in all other places.
We recommend defining symbols like this:
// File: model/schema.js
// @flow
import { tableName, columnName, type TableName, appSchema, tableSchema } from '@nozbe/watermelondb'
import type Comment from './Comment.js'
export const Tables = {
comments: (tableName('comments'): TableName<Comment>),
// ...
}
export const Columns = {
comments: {
body: columnName('body'),
// ...
}
}
export const appSchema = appSchema({
version: 1,
tables: [
tableSchema({
name: Tables.comments,
columns: [
{ name: Columns.comments.body, type: 'string' },
],
}),
// ...
]
})
And then using them like so:
// File: model/Comment.js
// @flow
import { Model } from '@nozbe/watermelondb'
import { text } from '@nozbe/watermelondb/decorators'
import { Tables, Columns } from './schema.js'
const Column = Columns.comments
export default class Comment extends Model {
static table = Tables.comments
@text(Column.body) body: string
}
But isn't that a lot of boilerplate?
Yes, it looks more boilerplate'y than the non-Flow examples, however:
- you're protected from typos — strings are defined once
- easier refactoring — you only change column name in one place
- no orphan columns or tables — no way to accidentally refer to a column or table that was removed from the schema
TableName
is typed with the model class it refers to, which allows Flow to find other mistakes in your code
In general, we find that untyped string constants lead to bugs, and defining typed constants is a good practice.
associations
When using Flow, you define model associations like this:
import { Model, associations } from '@nozbe/watermelondb'
import { Tables, Columns } from './schema.js'
const Column = Columns.posts
class Post extends Model {
static table = Tables.posts
static associations = associations(
[Tables.comments, { type: 'has_many', foreignKey: Columns.comments.postId }],
[Tables.users, { type: 'belongs_to', key: Column.authorId }],
)
}
Common types
Many types are tagged with the model class the type refers to:
TableName<Post> // a table name referring to posts
Collection<Post> // the Collection for posts
Relation<Comment> // a relation that can fetch a Comment
Relation<?Comment> // a relation that can fetch a Comment or `null`
Query<Comment> // a query that can fetch many Comments
Always mark the type of model fields. Remember to include ?
if the underlying table column is optional. Flow can't check if model fields match the schema or if they match the decorator's signature.
@text(Column.body) body: string
@date(Column.createdAt) createdAt: Date
@date(Column.archivedAt) archivedAt: ?Date
If you need to refer to an ID of a record, always use the RecordId
type alias, not string
(they're the same, but the former is self-documenting).
If you ever access the record's raw data (DON'T do that unless you really know what you're doing), use DirtyRaw
to refer to raw data from external sources (database, server), and RawRecord
after it was passed through sanitizedRaw
.
Local storage
WatermelonDB has a simple key/value store, similar to localStorage:
// setting a value
await database.adapter.setLocal("user_id", "abcdef")
// retrieving a value
const userId = await database.adapter.getLocal("user_id") // string or null if no value for this key
// removing a value
await database.adapter.removeLocal("user_id")
When to use it. For things like the ID of the logged-in user, or the route to the last-viewed screen in the app. You should generally avoid it and stick to standard Watermelon records.
This is a low-level API. You can't do things like observe changes of a value over time. If you need that, just use standard WatermelonDB records. Also, you can only store strings. You can build your own abstraction that (de)serializes those values to/from JSON.
What to be aware of. DO NOT let the local storage key be a user-supplied value. Only allow predefined/whitelisted keys.
Why not use localStorage/AsyncStorage? Because this way, you have only one source of truth — one database that, say, stores the logged-in user ID and the information about all users. So there's a lower risk that the two sets of values get out of sync.
Various Pro Tips
Database viewer
Android - you can use the new database inspector which comes with the Android Studio Beta. https://medium.com/androiddevelopers/database-inspector-9e91aa265316 . You can also use Facebook Flipper with this plugin: https://github.com/panz3r/react-native-flipper-databases#readme
iOS - check open database path in iOS System Log (via Console for plugged-in device, or Xcode logs), then open it via sqlite3
in the console, or an external tool like https://sqlitebrowser.org
Prepopulating database on native
There's no built-in support for this. One way is to generate a SQLite DB (you can use the the Node SQLite support in 0.19.0-2 pre-release or extract it from an ios/android app), bundle it with the app, and then use a bit of code to check if the DB you're expecting it available, and if not, making a copy of the default DB — before you attempt loading DB from JS side. See this thread: https://github.com/Nozbe/WatermelonDB/issues/774#issuecomment-667981361
Performance
Performance tips — TODO
Dig deeper into WatermelonDB
Details about how Watermelon works, how to hack and contribute
📺 Digging deeper into WatermelonDB — more architectural info about caching, observation, and sync
Architecture
Base objects
Database
is the root object of Watermelon. It owns:
- a
DatabaseAdapter
- a map of
Collection
s
DatabaseAdapter
connects Watermelon's reactive world to low-level imperative world of databases. See Adapters.
Collection
manages all records of a given kind:
- it has a cache of records already fetched from the database (
RecordCache
) - it has the public API to
find
,query
andcreate
existing records - it implements fetch/update/delete operations on records
Model
is an instance of a collection record. A model class describes a kind of a record. Model
is the base class for your concrete models (e.g. Post
, Comment
, Task
):
- it describes the specific instance -
id
+ all custom fields and actions - it has public API to
update
,markAsDeleted
anddestroyPermanently
- implements record-level observation
observe()
- static fields describe base information about a model (
table
,associations
) - See Defining models
As a general rule, Model
manages the state of a specific instance, and Collection
of the entire collection of records. So for example, model.markAsDeleted()
changes the local state of called record, but then delegates to its collection to notify collection observers and actually remove from the database
Query
is a helper object that gives us a nice API to perform queries (query.observe()
, query.fetchCount()
):
- created via
collection.query()
- encapsulates a
QueryDescription
structure which actually describes the query conditions - fetch/observe methods actually delegate to
Collection
to perform database operations - caches
Observable
s created byobserve/observeCount
methods so they can be reused and shared
Helper functions
Watermelon's objects and classes are meant to be as minimal as possible — only manage their own state and be an API for your app. Most logic should be stateless, and implemented as pure functions:
QueryDescription
is a structure (object) describing the query, built using Q.*
helper functions
encodeMatcher()
, simpleObserver()
, reloadingObserver()
, fieldObserver()
implement query observation logic (See Observation.)
Model decorators transform simple class properties into Watermelon-aware record fields.
Much of Adapters' logic is implemented as pure functions too. See Adapters.
Database adapters
The idea for the Watermelon architecture is to be database-agnostic. Watermelon
is a cross-platform high-level layer for dealing with data, but can be plugged in to any underlying database, depending on platform needs.
Think of it this way:
- Collection/Model/Query is the reactive layer
DatabaseAdapter
is the imperative layer
The adapter merely performs simple CRUD (create/read/update/delete) operations.
DatabaseAdapter
is a Flow interface. Watermelon comes with two concrete implementations:
React Native
SQLiteAdapter
is an adapter for React Native, based on SQLite:
- Queries are converted to SQL on app thread using
adapters/sqlite/encodeQuery
- Communication happens over
NativeModules
with a native-side bridge - Native database handling happens on a separate thread
DatabaseBridge
is the React Native bridge stubDatabaseDriver
implements Watermelon-specific logic (caching, etc.)Database
is a simple SQLite abstraction layer (over FMDB on iOS and built-insqlite.SQLiteDatabase
on Android)
Web
LokiJSAdapter
is an adapter for the web, based around LokiJS:
- Why LokiJS? WebSQL would be a perfect fit for Watermelon, but sadly is a dead API, so we must use IndexedDB, but it's too low-level. LokiJS implements a fast querying API on top of IndexedDB.
LokiJSAdapter
delegates everything to a separate thread overWorkerBridge
WorkerBridge
spins up a worker thread runningLokiWorker
LokiWorker
maintains a queue of operations and executes them onLokiExecutor
LokiExecutor
actually implements the Adapter operationsencodeQuery
translatesQueryDescription
objects to Loki query objectsexecuteQuery
implements join queries (Q.on
), which Loki does not support
Writing your own adapter
If you want to write a new adapter, please contact @radex for more information.
⚠️ TODO: This section needs more concrete tips
Sync implementation details
If you're looking for a guide to implement Watermelon Sync in your app, see Synchronization.
If you want to contribute to Watermelon Sync, or implement your own synchronization engine from scratch, read this.
Implementing your own sync from scratch
For basic details about how changes tracking works, see: 📺 Digging deeper into WatermelonDB
Why you might want to implement a custom sync engine? If you have an existing remote server architecture that's difficult to adapt to Watermelon sync protocol, or you specifically want a different architecture (e.g. single HTTP request -- server resolves conflicts). Be warned, however, that implementing sync that works reliably is a hard problem, so we recommend sticking to Watermelon Sync and tweaking it as needed.
The rest of this document contains details about how Watermelon Sync works - you can use that as a blueprint for your own work.
If possible, please use sync implementation helpers from sync/*.js
to keep your custom sync implementation have as much commonality as possible with the standard implementation. This is good both for you and for the rest of WatermelonDB community, as we get to share improvements and bug fixes. If the helpers are almost what you need, but not quite, please send pull requests with improvements!
Watermelon Sync -- Details
General design
- master/replica - server is the source of truth, client has a full copy and syncs back to server (no peer-to-peer syncs)
- two phase sync: first pull remote changes to local app, then push local changes to server
- client resolves conflicts
- content-based, not time-based conflict resolution
- conflicts are resolved using per-column client-wins strategy: in conflict, server version is taken except for any column that was changed locally since last sync.
- local app tracks its changes using a _status (synced/created/updated/deleted) field and _changes field (which specifies columns changed since last sync)
- server only tracks timestamps (or version numbers) of every record, not specific changes
- sync is performed for the entire database at once, not per-collection
- eventual consistency (client and server are consistent at the moment of successful pull if no local changes need to be pushed)
- non-blocking: local database writes (but not reads) are only momentarily locked when writing data but user can safely make new changes throughout the process
Sync procedure
- Pull phase
- get
lastPulledAt
timestamp locally (null if first sync) - call
pullChanges
function, passinglastPulledAt
- server responds with all changes (create/update/delete) that occured since
lastPulledAt
- server serves us with its current timestamp
- server responds with all changes (create/update/delete) that occured since
- IN ACTION (lock local writes):
- ensure no concurrent syncs
- apply remote changes locally
- insert new records
- if already exists (error), update
- if locally marked as deleted (error), un-delete and update
- update records
- if synced, just replace contents with server version
- if locally updated, we have a conflict!
- take remote version, apply local fields that have been changed locally since last sync (per-column client wins strategy)
- record stays marked as updated, because local changes still need to be pushed
- if locally marked as deleted, ignore (deletion will be pushed later)
- if doesn't exist locally (error), create
- destroy records
- if alredy deleted, ignore
- if locally changed, destroy anyway
- ignore children (server ought to schedule children to be destroyed)
- insert new records
- if successful, save server's timestamp as new
lastPulledAt
- Push phase
- Fetch local changes
- Find all locally changed records (created/updated record + deleted IDs) for all collections
- Strip _status, _changed
- Call
pushChanges
function, passing local changes object, and the newlastPulledAt
timestamp- Server applies local changes to database, and sends OK
- If one of the pushed records has changed on the server since
lastPulledAt
, push is aborted, all changes reverted, and server responds with an error
- IN ACTION (lock local writes):
- markLocalChangesAsSynced:
- take local changes fetched in previous step, and:
- permanently destroy records marked as deleted
- mark created/updated records as synced and reset their _changed field
- note: do not mark record as synced if it changed locally since
fetch local changes
step (user could have made new changes that need syncing)
- markLocalChangesAsSynced:
Notes
- This procedure is designed such that if sync fails at any moment, and even leaves local app in
inconsistent (not fully synced) state, we should still achieve consistency with the next sync:
- applyRemoteChanges is designed such that if all changes are applied, but
lastPulledAt
doesn't get saved — so during next pull server will serve us the same changes, second applyRemoteChanges will arrive at the same result - local changes before "fetch local changes" step don't matter at all - user can do anything
- local changes between "fetch local changes" and "mark local changes as synced" will be ignored (won't be marked as synced) - will be pushed during next sync
- if changes don't get marked as synced, and are pushed again, server should apply them the same way
- remote changes between pull and push phase will be locally ignored (will be pulled next sync) unless there's a per-record conflict (then push fails, but next sync resolves both pull and push)
- applyRemoteChanges is designed such that if all changes are applied, but
Migration Syncs
Schema versioning and migrations complicate sync, because a client might not be able to sync some tables and columns, but after upgrade to the newest version, it should be able to get consistent sync. To be able to do that, we need to know what's the schema version at which the last sync occured. Unfortunately, Watermelon Sync didn't track that from the first version, so backwards-compat is required.
synchronize({ migrationsEnabledAtVersion: XXX })
. . . .
LPA = last pulled at
MEA = migrationsEnabledAtVersion, schema version at which future migration support was introduced
LS = last synced schema version (may be null due to backwards compat)
CV = current schema version
LPA MEA LS CV migration set LS=CV? comment
null X X 10 null YES first sync. regardless of whether the app
is migration sync aware, we can note LS=CV
to fetch all migrations once available
100 null X X null NO indicates app is not migration sync aware so
we're not setting LS to allow future migration sync
100 X 10 10 null NO up to date, no migration
100 9 9 10 {9-10} YES correct migration sync
100 9 null 10 {9-10} YES fallback migration. might not contain all
necessary migrations, since we can't know for sure
that user logged in at then-current-version==MEA
100 9 11 10 ERROR NO LS > CV indicates programmer error
100 11 X 10 ERROR NO MEA > CV indicates programmer error
Reference
This design has been informed by:
- 10 years of experience building synchronization at Nozbe
- Kinto & Kinto.js
- https://github.com/Kinto/kinto.js/blob/master/src/collection.js
- https://kintojs.readthedocs.io/en/latest/api/#fetching-and-publishing-changes
- Histo - https://github.com/mirkokiefer/syncing-thesis
Dig deeper into WatermelonDB
Details about how Watermelon works, how to hack and contribute
📺 Digging deeper into WatermelonDB — more architectural info about caching, observation, and sync
WatermelonDB Roadmap
From today to 1.0
WatermelonDB is currently in active development at Nozbe for use in advanced projects. It's mostly feature-complete. However, there are a few features left before we can call it 1.0.
v0.xxx
- Full transactionality (atomicity) support ???
- Field sanitizers
- Optimized tree deleting
- API improvements
v1.0
Everything above plus having at least one non-trivial app using WatermelonDB in production to verify its concepts
Beyond 1.0
- Replace
withObservables
HOC and Prefetching with a solution based on React 17 Suspense feature - Query templates
Contributing guidelines
Before you send a pull request
-
Did you add or changed some functionality?
Add (or modify) tests!
-
Check if the automated tests pass
yarn ci:check
-
Format the files you changed
yarn prettier
-
Mark your changes in CHANGELOG
Put a one-line description of your change under Added/Changed section. See Keep a Changelog.
Running Watermelon in development
Download source and dependencies
git clone https://github.com/Nozbe/WatermelonDB.git
cd WatermelonDB
yarn
Developing Watermelon alongside your app
To work on Watermelon code in the sandbox of your app:
yarn dev
This will create a dev/
folder in Watermelon and observe changes to source files (only JavaScript files) and recompile them as needed.
Then in your app:
cd node_modules/@nozbe
rm -fr watermelondb
ln -s path-to-watermelondb/dev watermelondb
This will work in Webpack but not in Metro (React Native). Metro doesn't follow symlinks. Instead, you can compile WatermelonDB directly to your project:
DEV_PATH="/path/to/your/app/node_modules/@nozbe/watermelondb" yarn dev
Running tests
This runs Jest, ESLint and Flow:
yarn ci:check
You can also run them separately:
yarn test
yarn eslint
yarn flow
Editing files
We recommend VS Code with ESLint, Flow, and Prettier (with prettier-eslint enabled) plugins for best development experience. (To see lint/type issues inline + have automatic reformatting of code)
Editing native code
In native/ios
and native/android
you'll find the native bridge code for React Native.
It's recommended to use the latest stable version of Xcode / Android Studio to work on that code.
Integration tests
If you change native bridge code or adapter/sqlite
code, it's recommended to run integration tests that run the entire Watermelon code with SQLite and React Native in the loop:
yarn test:ios
yarn test:android
Running tests manualy
- For iOS open the
native/iosTest/WatermelonTester.xcworkspace
project and hit Cmd+U. - For Android open
native/androidTest
in AndroidStudio navigate toapp/src/androidTest/java/com.nozbe.watermelonTest/BridgeTest
and click green arrow nearclass BridgeTest
Native linting
Make sure the native code you're editing conforms to Watermelon standards:
yarn swiftlint
yarn ktlint
Native code troubleshooting
- If
test:ios
fails in terminal:
- Run tests in Xcode first before running from terminal
- Make sure you have the right version of Xcode CLI tools set in Preferences -> Locations
- Make sure you're on the most recent stable version of Xcode / Android Studio
- Remove native caches:
- Xcode:
~/Library/Developer/Xcode/DerivedData
: - Android:
.gradle
andbuild
folders innative/android
andnative/androidTest
node_modules
(because of React Native precompiled third party libraries)
Changelog
All notable changes to this project will be documented in this file.
Contributors: Please add your changes to CHANGELOG-Unreleased.md
0.20 - 2020-10-05
New features
- [Sync] Conflict resolution can now be customized. See docs for more details
- [Android] Autolinking is now supported
- [LokiJS] Adapter autosave option is now configurable
Changes
- Interal RxJS imports have been refactor such that rxjs-compat should never be used now
- [Performance] Tweak Babel config to produce smaller code
- [Performance] LokiJS-based apps will now take up to 30% less time to load the database (id and unique indicies are generated lazily)
Fixes
- [iOS] Fixed crash on database reset in apps linked against iOS 14 SDK
- [LokiJS] Fix
Q.like
being broken for multi-line strings on web - Fixed warn "import cycle" from DialogProvider (#786) by @gmonte.
- Fixed cache date as instance of Date (#828) by @djorkaeffalexandre.
0.19 - 2020-08-17
New features
- [iOS] Added CocoaPods support - @leninlin
- [NodeJS] Introducing a new SQLite Adapter based integration to NodeJS. This requires a peer dependency on better-sqlite3 and should work with the same configuration as iOS/Android - @sidferreira
- [Android]
exerimentalUseJSI
option has been enabled on Android. However, it requires some app-specific setup which is not yet documented - stay tuned for upcoming releases - [Schema] [Migrations] You can now pass
unsafeSql
parameters to schema builder and migration steps to modify SQL generated to set up the database or perform migrations. There's also newunsafeExecuteSql
migration step. Please use this only if you know what you're doing — you shouldn't need this in 99% of cases. See Schema and Migrations docs for more details - [LokiJS] [Performance] Added experimental
onIndexedDBFetchStart
andindexedDBSerializer
options toLokiJSAdapter
. These can be used to improve app launch time. Seesrc/adapters/lokijs/index.js
for more details.
Changes
- [Performance] findAndObserve is now able to emit a value synchronously. By extension, this makes Relations put into withObservables able to render the child component in one shot. Avoiding the extra unnecessary render cycles avoids a lot of DOM and React commit-phase work, which can speed up loading some views by 30%
- [Performance] LokiJS is now faster (refactored encodeQuery, skipped unnecessary clone operations)
0.18 - 2020-06-30
Another WatermelonDB release after just a week? Yup! And it's jam-packed full of features!
New features
-
[Query]
Q.on
queries are now far more flexible. Previously, they could only be placed at the top level of a query. See Docs for more details. Now, you can:-
Pass multiple conditions on the related query, like so:
collection.query( Q.on('projects', [ Q.where('foo', 'bar'), Q.where('bar', 'baz'), ]) )
-
You can place
Q.on
deeper inside the query (nested insideQ.and()
,Q.or()
). However, you must explicitly list all tables you're joining on at the beginning of a query, using:Q.experimentalJoinTables(['join_table1', 'join_table2'])
. -
You can nest
Q.on
conditions insideQ.on
, e.g. to make a condition on a grandchild. To do so, it's required to passQ.experimentalNestedJoin('parent_table', 'grandparent_table')
at the beginning of a query
-
-
[Query]
Q.unsafeSqlExpr()
andQ.unsafeLokiExpr()
are introduced to allow adding bits of queries that are not supported by the WatermelonDB query language without having to useunsafeFetchRecordsWithSQL()
. See docs for more details -
[Query]
Q.unsafeLokiFilter((rawRecord, loki) => boolean)
can now be used as an escape hatch to make queries with LokiJSAdapter that are not otherwise possible (e.g. multi-table column comparisons). See docs for more details
Changes
- [Performance] [LokiJS] Improved performance of queries containing query comparisons on LokiJSAdapter
- [Docs] Added Contributing guide for Query language improvements
- [Deprecation]
Query.hasJoins
is deprecated - [DX] Queries with bad associations now show more helpful error message
- [Query] Counting queries that contain
Q.experimentalTake
/Q.experimentalSkip
is currently broken - previously it would return incorrect results, but now it will throw an error to avoid confusion. Please contribute to fix the root cause!
Fixes
- [Typescript] Fixed types of Relation
Internal
QueryDescription
structure has been changed.
0.17.1 - 2020-06-24
- Fixed broken iOS build - @mlecoq
0.17 - 2020-06-22
New features
-
[Sync] Introducing Migration Syncs - this allows fully consistent synchronization when migrating between schema versions. Previously, there was no mechanism to incrementally fetch all remote changes in new tables and columns after a migration - so local copy was likely inconsistent, requiring a re-login. After adopting migration syncs, Watermelon Sync will request from backend all missing information. See Sync docs for more details.
-
[iOS] Introducing a new native SQLite database integration, rewritten from scratch in C++, based on React Native's JSI (JavaScript Interface). It is to be considered experimental, however we intend to make it the default (and eventually, the only) implementation. In a later release, Android version will be introduced.
The new adapter is up to 3x faster than the previously fastest `synchronous: true` option, however this speedup is only achieved with some unpublished React Native patches. To try out JSI, add `experimentalUseJSI: true` to `SQLiteAdapter` constructor.
-
[Query] Added
Q.experimentalSortBy(sortColumn, sortOrder)
,Q.experimentalTake(count)
,Q.experimentalSkip(count)
methods (only availble with SQLiteAdapter) - @Kenneth-KT -
Database.batch()
can now be called with a single array of models -
[DX]
Database.get(tableName)
is now a shortcut forDatabase.collections.get(tableName)
-
[DX] Query is now thenable - you can now use
await query
andawait query.count
instead ofawait query.fetch()
andawait query.fetchCount()
-
[DX] Relation is now thenable - you can now use
await relation
instead ofawait relation.fetch()
-
[DX] Exposed
collection.db
andmodel.db
as shortcuts to get to their Database object
Changes
- [Hardening] Column and table names starting with
__
, Object property names (e.g.constructor
), and some reserved keywords are now forbidden - [DX] [Hardening] QueryDescription builder methods do tighter type checks, catching more bugs, and preventing users from unwisely passing unsanitized user data into Query builder methods
- [DX] [Hardening] Adapters check early if table names are valid
- [DX] Collection.find reports an error more quickly if an obviously invalid ID is passed
- [DX] Intializing Database with invalid model classes will now show a helpful error
- [DX] DatabaseProvider shows a more helpful error if used improperly
- [Sync] Sync no longer fails if pullChanges returns collections that don't exist on the frontend - shows a warning instead. This is to make building backwards-compatible backends less error-prone
- [Sync] [Docs] Sync documentation has been rewritten, and is now closer in detail to a formal specification
- [Hardening] database.collections.get() better validates passed value
- [Hardening] Prevents unsafe strings from being passed as column name/table name arguments in QueryDescription
Fixes
- [Sync] Fixed
RangeError: Maximum call stack size exceeded
when syncing large amounts of data - @leninlin - [iOS] Fixed a bug that could cause a database operation to fail with an (6) SQLITE_LOCKED error
- [iOS] Fixed 'jsi/jsi.h' file not found when building at the consumer level. Added path
$(SRCROOT)/../../../../../ios/Pods/Headers/Public/React-jsi
to Header Search Paths (issue #691) - @victorbutler - [Native] SQLite keywords used as table or column names no longer crash
- Fixed potential issues when subscribing to database, collection, model, queries passing a subscriber function with the same identity more than once
Internal
- Fixed broken adapter tests
0.15.1, 0.16.1-fix, 0.16.2 - 2020-06-03
This is a security patch for a vulnerability that could cause maliciously crafted record IDs to cause all or some of user's data to be deleted. More information available via GitHub security advisory
0.16.1 - 2020-05-18
Changes
Database.unsafeResetDatabase()
is now less unsafe — more application bugs are being caught
Fixes
- [iOS] Fix build in apps using Flipper
- [Typescript] Added type definition for
setGenerator
. - [Typescript] Fixed types of decorators.
- [Typescript] Add Tests to test Types.
- Fixed typo in learn-to-use docs.
- [Typescript] Fixed types of changes.
Internal
- [SQLite] Infrastruture for a future JSI adapter has been added
0.16 - 2020-03-06
⚠️ Breaking
experimentalUseIncrementalIndexedDB
has been renamed touseIncrementalIndexedDB
Low breakage risk
- [adapters] Adapter API has changed from returning Promise to taking callbacks as the last argument. This won't affect you unless you call on adapter methods directly.
database.adapter
returns a newDatabaseAdapterCompat
which has the same shape as old adapter API. You can usedatabase.adapter.underlyingAdapter
to get backSQLiteAdapter
/LokiJSAdapter
- [Collection]
Collection.fetchQuery
andCollection.fetchCount
are removed. Please useQuery.fetch()
andQuery.fetchCount()
.
New features
- [SQLiteAdapter] [iOS] Add new
synchronous
option to adapter:new SQLiteAdapter({ ..., synchronous: true })
. When enabled, database operations will block JavaScript thread. Adapter actions will resolve in the next microtask, which simplifies building flicker-free interfaces. Adapter will fall back to async operation when synchronous adapter is not available (e.g. when doing remote debugging) - [LokiJS] Added new
onQuotaExceededError?: (error: Error) => void
option toLokiJSAdapter
constructor. This is called when underlying IndexedDB encountered a quota exceeded error (ran out of allotted disk space for app) This means that app can't save more data or that it will fall back to using in-memory database only Note that this only works whenuseWebWorker: false
Changes
- [Performance] Watermelon internals have been rewritten not to rely on Promises and allow some fetch/observe calls to resolve synchronously. Do not rely on this -- external API is still based on Rx and Promises and may resolve either asynchronously or synchronously depending on capabilities. This is meant as a internal performance optimization only for the time being.
- [LokiJS] [Performance] Improved worker queue implementation for performance
- [observation] Refactored observer implementations for performance
Fixes
- Fixed a possible cause for "Record ID xxx#yyy was sent over the bridge, but it's not cached" error
- [LokiJS] Fixed an issue preventing database from saving when using
experimentalUseIncrementalIndexedDB
- Fixed a potential issue when using
database.unsafeResetDatabase()
- [iOS] Fixed issue with clearing database under experimental synchronous mode
New features (Experimental)
- [Model] Added experimental
model.experimentalSubscribe((isDeleted) => { ... })
method as a vanilla JS alternative to Rx basedmodel.observe()
. Unlike the latter, it does not notify the subscriber immediately upon subscription. - [Collection] Added internal
collection.experimentalSubscribe((changeSet) => { ... })
method as a vanilla JS alternative to Rx basedcollection.changes
(you probably shouldn't be using this API anyway) - [Database] Added experimental
database.experimentalSubscribe(['table1', 'table2'], () => { ... })
method as a vanilla JS alternative to Rx-baseddatabase.withChangesForTables()
. Unlike the latter,experimentalSubscribe
notifies the subscriber only once after a batch that makes a change in multiple collections subscribed to. It also doesn't notify the subscriber immediately upon subscription, and doesn't send details about the changes, only a signal. - Added
experimentalDisableObserveCountThrottling()
to@nozbe/watermelondb/observation/observeCount
that globally disables count observation throttling. We think that throttling on WatermelonDB level is not a good feature and will be removed in a future release - and will be better implemented on app level if necessary - [Query] Added experimental
query.experimentalSubscribe(records => { ... })
,query.experimentalSubscribeWithColumns(['col1', 'col2'], records => { ... })
, andquery.experimentalSubscribeToCount(count => { ... })
methods
0.15 - 2019-11-08
Highlights
This is a massive new update to WatermelonDB! 🍉
-
Up to 23x faster sync. You heard that right. We've made big improvements to performance. In our tests, with a massive sync (first login, 45MB of data / 65K records) we got a speed up of:
- 5.7s -> 1.2s on web (5x)
- 142s -> 6s on iOS (23x)
Expect more improvements in the coming releases!
-
Improved LokiJS adapter. Option to disable web workers, important Safari 13 fix, better performance, and now works in Private Modes. We recommend adding
useWebWorker: false, experimentalUseIncrementalIndexedDB: true
options to theLokiJSAdapter
constructor to take advantage of the improvements, but please read further changelog to understand the implications of this. -
Raw SQL queries now available on iOS and Android thanks to the community
-
Improved TypeScript support — thanks to the community
⚠️ Breaking
- Deprecated
bool
schema column type is removed -- please change toboolean
- Experimental
experimentalSetOnlyMarkAsChangedIfDiffers(false)
API is now removed
New featuers
-
[Collection] Add
Collection.unsafeFetchRecordsWithSQL()
method. You can use it to fetch record using raw SQL queries on iOS and Android. Please be careful to avoid SQL injection and other pitfalls of raw queries -
[LokiJS] Introduces new
new LokiJSAdapter({ ..., experimentalUseIncrementalIndexedDB: true })
option. When enabled, database will be saved to browser's IndexedDB using a new adapter that only saves the changed records, instead of the entire database.This works around a serious bug in Safari 13 (https://bugs.webkit.org/show_bug.cgi?id=202137) that causes large databases to quickly balloon to gigabytes of temporary trash
This also improves performance of incremental saves, although initial page load or very, very large saves might be slightly slower.
This is intended to become the new default option, but it's not backwards compatible (if enabled, old database will be lost). You're welcome to contribute an automatic migration code.
Note that this option is still experimental, and might change in breaking ways at any time.
-
[LokiJS] Introduces new
new LokiJSAdapter({ ..., useWebWorker: false })
option. Before, web workers were always used withLokiJSAdapter
. Although web workers may have some performance benefits, disabling them may lead to lower memory consumption, lower latency, and easier debugging. YMMV. -
[LokiJS] Added
onIndexedDBVersionChange
option toLokiJSAdapter
. This is a callback that's called when internal IDB version changed (most likely the database was deleted in another browser tab). Pass a callback to force log out in this copy of the app as well. Note that this only works when using incrementalIDB and not using web workers -
[Model] Add
Model._dangerouslySetRawWithoutMarkingColumnChange()
method. You probably shouldn't use it, but if you know what you're doing and want to live-update records from server without marking record as updated, this is useful -
[Collection] Add
Collection.prepareCreateFromDirtyRaw()
-
@json decorator sanitizer functions take an optional second argument, with a reference to the model
Fixes
- Pinned required
rambdax
version to 2.15.0 to avoid console logging bug. In a future release we will switch to our own fork oframbdax
to avoid future breakages like this.
Improvements
- [Performance] Make large batches a lot faster (1.3s shaved off on a 65K insert sample)
- [Performance] [iOS] Make large batch inserts an order of magnitude faster
- [Performance] [iOS] Make encoding very large queries (with thousands of parameters) 20x faster
- [Performance] [LokiJS] Make batch inserts faster (1.5s shaved off on a 65K insert sample)
- [Performance] [LokiJS] Various performance improvements
- [Performance] [Sync] Make Sync faster
- [Performance] Make observation faster
- [Performance] [Android] Make batches faster
- Fix app glitches and performance issues caused by race conditions in
Query.observeWithColumns()
- [LokiJS] Persistence adapter will now be automatically selected based on availability. By default, IndexedDB is used. But now, if unavailable (e.g. in private mode), ephemeral memory adapter will be used.
- Disabled console logs regarding new observations (it never actually counted all observations) and time to query/count/batch (the measures were wildly inaccurate because of asynchronicity - actual times are much lower)
- [withObservables] Improved performance and debuggability (update withObservables package separately)
- Improved debuggability of Watermelon -- shortened Rx stacks and added function names to aid in understanding call stacks and profiles
- [adapters] The adapters interface has changed.
query()
andcount()
methods now receive aSerializedQuery
, andbatch()
now takesTableName<any>
andRawRecord
orRecordId
instead ofModel
. - [Typescript] Typing improvements
- Added 3 missing properties
collections
,database
andasModel
in Model type definition. - Removed optional flag on
actionsEnabled
in the Database constructor options since its mandatory since 0.13.0. - fixed several further typing issues in Model, Relation and lazy decorator
- Added 3 missing properties
- Changed how async functions are transpiled in the library. This could break on really old Android phones but shouldn't matter if you use latest version of React Native. Please report an issue if you see a problem.
- Avoid
database
prop drilling in the web demo
0.14.1 - 2019-08-31
Hotfix for rambdax crash
- [Schema] Handle invalid table schema argument in appSchema
- [withObservables] Added TypeScript support (changelog)
- [Electron] avoid
Uncaught ReferenceError: global is not defined
in electron runtime (#453) - [rambdax] Replaces
contains
withincludes
due tocontains
deprecation https://github.com/selfrefactor/rambda/commit/1dc1368f81e9f398664c9d95c2efbc48b5cdff9b#diff-04c6e90faac2675aa89e2176d2eec7d8R2209
0.14.0 - 2019-08-02
New features
- [Query] Added support for
notLike
queries 🎉 - [Actions] You can now batch delete record with all descendants using experimental functions
experimentalMarkAsDeleted
orexperimentalDestroyPermanently
0.13.0 - 2019-07-18
⚠️ Breaking
-
[Database] It is now mandatory to pass
actionsEnabled:
option to Database constructor. It is recommended that you enable this option:const database = new Database({ adapter: ..., modelClasses: [...], actionsEnabled: true })
See
docs/Actions.md
for more details about Actions. You can also passfalse
to maintain backward compatibility, but this option will be removed in a later version -
[Adapters]
migrationsExperimental
prop ofSQLiteAdapter
andLokiJSAdapter
has been renamed tomigrations
.
New features
- [Actions] You can now batch deletes by using
prepareMarkAsDeleted
orprepareDestroyPermanently
- [Sync] Performance:
synchronize()
no longer calls yourpushChanges()
function if there are no local changes to push. This is meant to save unnecessary network bandwidth. ⚠️ Note that this could be a breaking change if you rely on it always being called - [Sync] When setting new values to fields on a record, the field (and record) will no longer be
marked as changed if the field's value is the same. This is meant to improve performance and avoid
unnecessary code in the app. ⚠️ Note that this could be a breaking change if you rely on the old
behavior. For now you can import
experimentalSetOnlyMarkAsChangedIfDiffers
from@nozbe/watermelondb/Model/index
and call if with(false)
to bring the old behavior back, but this will be removed in the later version -- create a new issue explaining why you need this - [Sync] Small perf improvements
Improvements
- [Typescript] Improved types for SQLite and LokiJS adapters, migrations, models, the database and the logger.
0.12.3 - 2019-05-06
Changes
- [Database] You can now update the random id schema by importing
import { setGenerator } from '@nozbe/watermelondb/utils/common/randomId'
and then callingsetGenerator(newGenenerator)
. This allows WatermelonDB to create specific IDs for example if your backend uses UUIDs. - [Typescript] Type improvements to SQLiteAdapter and Database
- [Tests] remove cleanup for react-hooks-testing-library@0.5.0 compatibility
0.12.2 - 2019-04-19
Fixes
- [TypeScript] 'Cannot use 'in' operator to search for 'initializer'; decorator fix
Changes
- [Database] You can now pass falsy values to
Database.batch(...)
(false, null, undefined). This is useful in keeping code clean when doing operations conditionally. (Also works withmodel.batch(...)
) - [Decorators]. You can now use
@action
on methods of any object that has adatabase: Database
property, and@field @children @date @relation @immutableRelation @json @text @nochange
decorators on any object with aasModel: Model
property. - [Sync] Adds a temporary/experimental
_unsafeBatchPerCollection: true
flag tosynchronize()
. This causes server changes to be committed to database in multiple batches, and not one. This is NOT preferred for reliability and performance reasons, but it works around a memory issue that might cause your app to crash on very large syncs (>20,000 records). Use this only if necessary. Note that this option might be removed at any time if a better solution is found.
0.12.1 - 2019-04-01
⚠️ Hotfix
-
[iOS] Fix runtime crash when built with Xcode 10.2 (Swift 5 runtime).
⚠️ Note: You need to upgrade to React Native 0.59.3 for this to work. If you can't upgrade React Native yet, either stick to Xcode 10.1 or manually apply this patch: https://github.com/Nozbe/WatermelonDB/pull/302/commits/aa4e08ad0fa55f434da2a94407c51fc5ff18e506
Changes
- [Sync] Adds basic sync logging capability to Sync. Pass an empty object to
synchronize()
to populate it with diagnostic information:
See Sync documentation for more details.const log = {} await synchronize({ database, log, ...}) console.log(log.startedAt)
0.12.0 - 2019-03-18
Added
- [Hooks] new
useDatabase
hook for consuming the Database Context:import { useDatabase } from '@nozbe/watermelondb/hooks'; const Component = () => { const database = useDatabase(); }
- [TypeScript] added
.d.ts
files. Please note: TypeScript definitions are currently incomplete and should be used as a guide only. PRs for improvements would be greatly appreciated!
Performance
- Improved UI performance by consolidating multiple observation emissions into a single per-collection batch emission when doing batch changes
0.11.0 - 2019-03-12
Breaking
- ⚠️ Potentially BREAKING fix: a
@date
field now returns a Jan 1, 1970 date instead ofnull
if the field's raw value is0
. This is considered a bug fix, since it's unexpected to receive anull
from a getter of a field whose column schema doesn't sayisOptional: true
. However, if you relied on this behavior, this might be a breaking change. - ⚠️ BREAKING:
Database.unsafeResetDatabase()
now requires that you run it inside an Action
Bug fixes
- [Sync] Fixed an issue where synchronization would continue running despite
unsafeResetDatabase
being called - [Android] fix compile error for kotlin 1.3+
Other changes
- Actions are now aborted when
unsafeResetDatabase()
is called, making reseting database a little bit safer - Updated demo dependencies
- LokiJS is now a dependency of WatermelonDB (although it's only required for use on the web)
- [Android] removed unused test class
- [Android] updated ktlint to
0.30.0
0.10.1 - 2019-02-12
Changes
- [Android] Changed
compile
toimplementation
in Library Gradle file- ⚠️ might break build if you are using Android Gradle Plugin <3.X
- Updated
peerDependency
react-native
to0.57.0
- [Sync] Added
hasUnsyncedChanges()
helper method - [Sync] Improved documentation for backends that can't distinguish between
created
andupdated
records - [Sync] Improved diagnostics / protection against edge cases
- [iOS] Add missing
header search path
to support ejected expo project. - [Android] Fix crash on android < 5.0
- [iOS]
SQLiteAdapter
'sdbName
path now allows you to pass an absolute path to a file, instead of a name - [Web] Add adaptive layout for demo example with smooth scrolling for iOS
0.10.0 - 2019-01-18
Breaking
- BREAKING: Table column
last_modified
is no longer automatically added to all database tables. If you don't use this column (e.g. in your custom sync code), you don't have to do anything. If you do, manually add this column to all table definitions in your Schema:
Don't bump schema version or write a migration for this.{ name: 'last_modified', type: 'number', isOptional: true }
New
-
Actions API.
This was actually released in 0.8.0 but is now documented in CRUD.md and Actions.md. With Actions enabled, all create/update/delete/batch calls must be wrapped in an Action.
To use Actions, call
await database.action(async () => { /* perform writes here */ }
, and in Model instance methods, you can just decorate the whole method with@action
.This is necessary for Watermelon Sync, and also to enable greater safety and consistency.
To enable actions, add
actionsEnabled: true
tonew Database({ ... })
. In a future release this will be enabled by default, and later, made mandatory.See documentation for more details.
-
Watermelon Sync Adapter (Experimental)
Added
synchronize()
function that allows you to easily add full synchronization capabilities to your Watermelon app. You only need to provide two fetch calls to your remote server that conforms to Watermelon synchronization protocol, and all the client-side processing (applying remote changes, resolving conflicts, finding local changes, and marking them as synced) is done by Watermelon.See documentation for more details.
-
Support caching for non-global IDs at Native level
0.9.0 - 2018-11-23
New
- Added
Q.like
- you can now make queries similar to SQLLIKE
0.8.0 - 2018-11-16
New
- Added
DatabaseProvider
andwithDatabase
Higher-Order Component to reduce prop drilling - Added experimental Actions API. This will be documented in a future release.
Fixes
- Fixes crash on older Android React Native targets without
jsc-android
installed
0.7.0 - 2018-10-31
Deprecations
- [Schema] Column type 'bool' is deprecated — change to 'boolean'
New
- Added support for Schema Migrations. See documentation for more details.
- Added fundaments for integration of Danger with Jest
Changes
- Fixed "dependency cycle" warning
- [SQLite] Fixed rare cases where database could be left in an unusable state (added missing transaction)
- [Flow] Fixes
oneOf()
typing and some other variance errors - [React Native] App should launch a little faster, because schema is only compiled on demand now
- Fixed typos in README.md
- Updated Flow to 0.85
0.6.2 - 2018-10-04
Deprecations
- The
@nozbe/watermelondb/babel/cjs
/@nozbe/watermelondb/babel/esm
Babel plugin that ships with Watermelon is deprecated and no longer necessary. Delete it from your Babel config as it will be removed in a future update
Refactoring
- Removed dependency on
async
(Web Worker should be ~30KB smaller) - Refactored
Collection
andsimpleObserver
for getting changes in an array and also adds CollectionChangeTypes for differentiation between different changes - Updated dependencies
- Simplified build system by using relative imports
- Simplified build package by outputting CJS-only files
0.6.1 - 2018-09-20
Added
- Added iOS and Android integration tests and lint checks to TravisCI
Changed
- Changed Flow setup for apps using Watermelon - see docs/Advanced/Flow.md
- Improved documentation, and demo code
- Updated dependencies
Fixed
- Add quotes to all names in sql queries to allow keywords as table or column names
- Fixed running model tests in apps with Watermelon in the loop
- Fixed Flow when using Watermelon in apps
0.6.0 - 2018-09-05
Initial release of WatermelonDB