Indexing

Each list can contain multiple indexes. These indexes are used to efficiently sort, filter, and search items in the list.

Indexes use JSON pointer syntax to point to a specific field or sub-section of an item's data. This is a "best effort" operation, meaning that if the field does not exist in a given item, the index will skip that item.

As such, indexes are most effective when the field being indexed is present in all items - this works especially well with list schemas because they enforce a consistent structure across all items in the list.

Let's walk through the process of creating an index, then using it for various purposes to see what it looks like in practice.

Creating an index

To create an index, you need to make a POST request to the /lists/{listId}/indexes endpoint.

Here we've created a basic index inside the sample-list list which points to the /name field in each item's data.

If we have 3 items in this list with the following data:

Item #1

Item #2

Item #3

Then we will have indexed the following values:

Once an index has been built, the API keeps its values up to date whenever items are created, updated or deleted.

Index builds

An index's values are built in the background. When you create an index, or change its pointer, the response comes back straight away with "buildStatus": "building", and jsonpad works through every item in the list. For most lists that takes a second or two; a list with hundreds of thousands of items can take a few minutes.

While an index is being built, requests that depend on its values are refused, so you never get results from a half-built index:

• filtering items by the index,
• ordering items by the index,
• looking up an item by alias (looking it up by id still works),
• searching a list while one of its searchable indexes is building,
• realtime subscriptions that name items by alias.

These requests get a 409 Conflict response with the INDEX_BUILDING error code, and a Retry-After header saying how many seconds to wait before trying again:

Everything else keeps working while an index is built. Items can still be created, updated and deleted, and their values for the index are kept up to date as they change.

To wait until an index is ready, fetch it until its buildStatus is ready, or use waitForIndex in the JS SDK:

Only a change to an index's pointer rebuilds it. Renaming it, changing its valueType, or turning sorting, filtering or searching on and off take effect immediately. Turning alias on also takes effect immediately, but it's refused if more than one item already has the same value, or if an item's value is too long to be an alias.

Failed builds

If an index can't be built, its buildStatus becomes failed, and requests that depend on it are refused with the INDEX_BUILD_FAILED error code until it's built again.

The most common cause is an alias index where more than one item has the same value at the index's pointer, since alias values have to be unique, or where an item's value is too long. A failed build is never retried automatically, since it would only fail again until the data is fixed. Fix the data, then rebuild the index to start a new build. The dashboard shows why a build failed, and has a button to rebuild the index both on the index's page and in the background jobs menu.

Alias indexes

The index we created above is an "alias" index (note that we set "alias": true), which means that it provides a way to reference each item by a unique value instead of the item's id.

This is similar to how we can set a pathName on lists. The main difference is that alias indexes allow us to dynamically generate the alias based on the item's data.

Here are two ways of fetching an item:

Fetching the item by id:

Fetching the item by alias:

Long values

An index stores up to 1024 characters of each item's value, or 2048 bytes of UTF-8, whichever comes first, so characters that take more bytes (like Chinese text or emoji) reach the limit sooner. An object or array is stored as JSON, and counts towards the limit the same way.

A longer value is still saved in full in the item's data, but the index only holds its start. Filtering, sorting and searching use what the index holds, so they work on the start of a long value: a filter only matches text that appears before the limit, and two values that only differ after it sort as equal.

Alias values have to be stored in full, because an alias has to be unique and has to find the item by its whole value. A write that would give an item an alias value over the limit is refused (for example with ITEM_UNABLE_TO_CREATE), with a message naming the index. An alias index can't be built while an item has a value over the limit (its build fails), and turning alias on for an existing index is refused, with the ids of the items whose values are too long.

Sorting lists

If an index is enabled for sorting, we can use the index pathName in the ?order parameter to sort items in the list. Items might be sorted differently depending on the value type of the index; numbers will be sorted numerically, while strings will be sorted alphabetically.

Here's an example:

Filtering items

If an index is enabled for filtering, we can use the index pathName as a parameter when fetching items. The parameter value will be used to filter items.

If the index value type is 'string', items will be filtered based on partial string match. For example, using the index we created above, filtering by ?name=al would match items with name equal to "Alice", "Mallory", "Gonzales", "Khalil", and so on.

Index value types 'number' or 'date' will be matched exactly, however we can prefix the filter value in various ways to change how this works:

Here's an example:

Searching lists

If an index is enabled for searching, indexed values will be used to identify matching items when searching a list.

Searching uses Okapi BM25 scoring to rank items based on how well they match the search query. Search terms must be at least 3 characters long and are case-insensitive.

Here's an example of searching a list using the index we created above:

Guard indexes

A guard index works in the opposite direction to the others: instead of making a value easier to reach, it hides it. When an index has "guard": true, the value at its pointer is removed from item data in every response made using an API token.

The value can still be written, so a form in a public page can collect an email address or a phone number that nobody reading the list can see. This is what makes it safe to put a jsonpad token in a browser.

Here's an index that guards an email address:

An item created with both fields is stored intact, but reads back without the guarded one:

A guarded value reads back exactly like a value that was never there, so a guard doesn't advertise which items have something worth looking for. Asking for the guarded pointer directly (/lists/sample-list/items/7bfb299b-dec2-47e3-ac78-c3d562b44150/data/email) returns null, the same as any other missing field.

Reading guarded values

Guards only apply to token auth. Your own data is always fully visible in the jsonpad dashboard.

An identity can read the guarded values in the items it owns by adding ?includeGuarded=true to the request alongside its credentials. Every endpoint that returns item data accepts this parameter, and it has no effect on items the identity doesn't own, or on requests made without an identity.

Where guards apply

A guarded value is removed everywhere item data leaves the API, not just when fetching an item:

• every endpoint that returns items or item data, including the ones that return a JSON Path or JSON Pointer slice of it
• the item returned by a write, when ?includeData=true
• event log snapshots, and older item versions fetched with ?version=
• realtime messages - realtime clients authenticate with a token and can't use an identity, so guarded values never reach them
• the existing items used as examples by the generative API

A JSON Patch can read as well as write, so test, copy and move operations that read from a guarded pointer (or from anything containing one) are refused.

Restrictions

A guard index can't also be an alias index, or be used for sorting, filtering or searching. Each of those would expose the value it's supposed to hide: an alias appears in item URLs, a searchable value can be recovered by searching for candidates and watching what matches, and a filterable or sortable value can be narrowed down a request at a time. Creating or updating an index that combines them is refused.