DEV Community: Feranmi Okafor

Introducing our new monthly release schedule

Feranmi Okafor — Fri, 12 Jan 2024 15:20:25 +0000

Introduction

Over the course of the last 18 months, SurrealDB product releases have been feature-driven, rolling out updates as and when they were ready. This approach, while flexible, often led to unpredictable release dates which could cause challenges for users of the database, and engineers who develop against our embedded Rust SDK.

In our pursuit of improvement and efficiency, we are excited to announce a significant shift in our product release schedule. Taking inspiration from the Rust programming language’s release train model, we are transitioning to a more structured, monthly release cycle. This process will ensure that we release often, with predictable updates and product patches, allowing developers and organisations to build on top of SurrealDB, with predictable timelines for software improvements and fixes.

With this new release cycle, on the second Tuesday of each month, a new beta version of SurrealDB will be released, allowing users to test and update their software locally and in continuous integration environments, before the stable release of that version. On the second Tuesday of the following month, the beta release will be promoted to a stable release version, at which time a new beta version will be released.

Release channels

With this new approach to releases there are now three release channels for the SurrealDB database server, and SurrealDB Rust client.

Nightly (along with a surrealdb-nightly Rust crate)
Beta (along with a surrealdb-beta Rust crate)
Stable (available with the surrealdb Rust crate) Most developers and organisations will primarily use the stable release channel, but those who want to try out future or new experimental features may use the beta or nightly channels. Every night a new nightly version of the crate, and of the database server are released, both as a downloadable binary, and as an embeddable library. On the second Tuesday of each month, a new beta release is made consisting of the new features and improvements available in the nightly releases. Over the course of the next month, a number of further beta releases will be made as features are stabilised. On the second Tuesday of the following month, the beta release becomes stable, ensuring users can upgrade to the latest stable version. Here’s an example of how the development and release process works. Each night, a new nightly version of the SurrealDB server and library are produced. Every day is a release day, and these releases are created by our release infrastructure automatically. So as time passes, our releases look like this, once a night:

nightly: * - - * - - *

On the second Tuesday of every month, we build and deploy a new release. The beta branch of the SurrealDB repository branches off from the main branch from which the nightly releases are made. Now, there are two releases:

nightly: * - - * - - *
                     |
beta:                *

Some SurrealDB users will actively test against the beta release in continuous integration environments to help discover any possible regressions. In addition, the beta release can be used to develop applications locally, to ensure that future functionality can be used within projects. In the meantime, there’s still a nightly release every night:

nightly: * - - * - - * - - * - - *
                     |
beta:                *

Let’s say a regression is found. Good thing we had some time to test the beta release before the regression snuck into a stable release! The fix is applied to the main branch, so that the nightly release is fixed, and then the fix is backported to the beta branch, and a new release of beta is produced:

nightly: * - - * - - * - - * - - * - - *
                     |
beta:                * - - - - - - - - *

On the second Tuesday of the following month, after the beta was created, it’s time for a stable release. The stable branch is now produced from the beta branch:

nightly: * - - * - - * - - * - - * - - * - * - *
                     |
beta:                * - - - - - - - - *
                                       |
stable:                                *

We now have a new stable release of SurrealDB! Which is great. In addition however, we also need a new beta of the next version of SurrealDB. So after stable branches off of beta, the next version of beta branches off of nightly again:

nightly: * - - * - - * - - * - - * - - * - * - *
                     |                         |
beta:                * - - - - - - - - *       *
                                       |
stable:                                *

In a similar manner to Rust's release train model which occurs every 6 weeks, the SurrealDB release train will guarantee a timely and reliable release progression. This is called the 'train model' because every month, a new release 'leaves the station', but still has to take a journey through the beta channel before it arrives as a stable release.

Why the Change?

The unpredictability of our current process sometimes led to prolonged periods without updates, followed by a flood of changes. This irregularity can be disorienting and inconvenient for developers and organisations building on top of SurrealDB. Monthly releases strike a balance between the rapid iteration of continuous delivery and the stability of longer release cycles. They allow us to incorporate user feedback more swiftly and maintain a consistent development pace. Regular releases will provide our users with a predictable schedule, allowing them to plan for and adapt to new changes more effectively. This model enables us to iterate quickly and incorporate user feedback into subsequent releases, significantly improving the overall quality of the SurrealDB platform.

Summary

Our goal is to make our product more reliable and user-friendly. We expect this change to significantly enhance our ability to meet and exceed user expectations. We believe this new monthly release schedule will bring a new level of efficiency and predictability to our product development. We're excited about this change and confident it will bring about positive outcomes for both our team and our users.

We would love to hear your thoughts on this new approach. Please feel free to provide feedback, subscribe to our updates, or join our community forum to engage in further discussion.

Introducing Nightly and Beta Rust Crates

Feranmi Okafor — Wed, 10 Jan 2024 11:43:56 +0000

We are pleased to announce two additional Rust crates, surrealdb-nightly and surrealdb-beta. These crates are designed to complement the surrealdb crate. Starting with v1.1.0 we plan to publish a stable release on the second Tuesday of every month. These stable releases are only published to the stable surrealdb crate.

If you use SurrealDB in production, this is the ideal crate for you. You can add it to your project using cargo add or by simply adding a line like this one to your dependencies section:

surrealdb = "1"

Nightly Crate

If you prefer living on the bleeding edge, this crate is perfect for you. It follows our development branch very closely. It’s automatically published every night but only when there are new changes to the development branch. It makes it easy to test new features as an embedded library or as a client when running a nightly binary on the server.

When using nightly or beta crates, we recommend adding a line similar to this one to your dependencies section in Cargo.toml:

surrealdb = { version = "1", package = "surrealdb-nightly" }

This makes it easy to switch between the three crates by simply changing the package name. You could start your development using surrealdb-nightly if the nightly version has a feature you want that hasn’t stabilised yet, later switch to surrealdb-beta once it reaches beta and finally switch to surrealdb once it’s stable. All this by simply changing the package name. This way you won’t have to update your feature flag names if you have features that activate SurrealDB features. For example memory = ["surrealdb/kv-mem"].

Beta Crate

Every month we will freeze features on the development branch by creating a new branch which will only receive bug fixes from that point onwards. This gives us a month to polish our releases before they hit the stable crate. Updates in that branch are pushed to the surrealdb-beta crate. New features in this crate are not guaranteed to reach the stable crate. Such features may be reverted to give them more time to develop if we think they are not ready to stabilise yet.

Conclusion

The nightly and beta crates are great for testing new features and bug fixes. However, these are, by definition, pre-releases. Use them responsibly 😄.

Live Queries in Rust

Feranmi Okafor — Wed, 10 Jan 2024 11:30:14 +0000

SurrealDB comes with a LIVE SELECT statement that allows you to listen for creations, updates and deletions to specific records you are interested in or entire tables. While you could already take advantage of this powerful feature with our JavaScript SDK or WebSockets, the Rust SDK added an API for it in v1.1.0.

The Rust API for live queries builds on top of the already existing select method by simply adding a live method which converts the select query into a live select one. It works seamlessly with our current API, so you can use it with single records, a range of records, or entire tables. Unlike the normal select method which returns either a single result or a vector of results, it returns a stream of notifications. This works for the WebSocket engine and the local ones (the key value stores you can embed in your app). The only engine not yet supported is the HTTP one.

In this article, we show some examples of running live queries via the Rust SDK. We will skip imports for brevity but your IDE and/or the Rust compiler should give you the correct suggestions. Please refer to this example in our repo for a full, working example.

Getting Started

// Connect to a WebSocket endpoint
// We are using the WebSocket engine for this example but you can also
// use a local engine embeded in your app.
let db = Surreal::new::<Ws>("localhost:8000").await?;

// Signin as a namespace, database, or root user
db.signin(Root {
    username: "root",
    password: "root",
})
.await?;

// Select a specific namespace / database
db.use_ns("namespace").use_db("database").await?;

Once you have signed into your server and selected the namespace and database to use, you are ready to start sending queries to your database.

Listening for Changes on a Table

// Listen to all updates on the person table
let mut stream = db.select("person").live().await?;

This query listens to all changes made on the person table and returns a stream of notification results. Since the Rust SDK automatically deserialises responses on your behalf, you will either need to give the compiler a way to infer the response type of the notifications, or manually tell it what the response type should be. There are multiple ways to do this. The best one for you will depend on how you structure the rest of your code. In this article, we are going to process responses from a function. This allows the compiler to infer the type from the function parameter.

Let’s define a handle function that simply prints out the notification results:

// Handle the result of the live query notification
fn handle(result: Result<Notification<Person>>) {
    match result {
        Ok(notification) => println!("{notification:?}"),
        Err(error) => eprintln!("{error}"),
    }
}

Person, in this case, is what you expect the records coming from the database to deserialise into. This means that it must implement serde::Deserialize. notification.action will tell you whether the action that triggered this notification was a creation, update or deletion. [notification.data](http://notification.data) will contain your deserialised object, Person in this particular case.

With this out of the way, we can now listen to changes being made on the table and process them in real time.

// The returned stream implements `futures::Stream` so we can
// use it with `futures::StreamExt`
while let Some(result) = stream.next().await {
    handle(result);
}

If you do not care about strong types for your particular use case or you simply want to try out the feature, you can use surrealdb::opt::Resource as your select parameter. This makes the compiler infer that the response type will be surrealdb::sql::Value so you won’t need to define it in that case.

// Listen to all updates on the person table
let mut stream = db.select(Resource::from("person")).live().await?;

// The returned stream implements `futures::Stream` so we can
// use it with `futures::StreamExt`
while let Some(notification) = stream.next().await {
    println!("{notification:?}");
}

That’s all you need when using Resource. It doesn’t return a result because Values are not deserialised first and if a notification itself can’t be deserialised from a remote server, it can’t be routed anyway. surrealdb::sql::Value is the internal type SurrealDB uses so it’s returned as is. This is not unique to live queries or new to this release. Resources can be used with any CRUD method.

Listening for Changes on a Range of Records

Listening on a range of records is simply a matter of using the range method on a select query before calling live on it.

let mut stream = db.select(Resource::from("person"))
    // This is a range on the `id` of the person
    .range("jane".."john")
    .live()
    .await?;

while let Some(notification) = stream.next().await {
    println!("{notification:?}");
}

Again, we are using Resource as a parameter here for brevity but the first approach works too. the only difference with this code is the range method. This supports the full range syntax that Rust supports; start..end, start.., .., start..=end, ..end and ..=end. You can also specify the bounds manually using the Bound enum directly, (start, end), for extra flexibility.

Listening to Changes on a Single Record

As you may have guessed already, listening to changes on a particular Person record is not much different from what we have done so far. In this case, we simply need to listen on a record ID instead of a table.

// Listen only to changes happening to John's data
let mut stream = db.select(Resource::from(("person", "john")).live().await?;

while let Some(notification) = stream.next().await {
    println!("{notification:?}");
}

Conclusion

If you need to listen to changes happening to your data in real-time, live queries are a powerful feature that enables you to do this. They integrate seamlessly with the Rust SDK allowing you to take advantage of the other SurrealDB features like authentication and permissions. The SDK also manages creating and closing the query for you automatically. As you may have noticed, we did not need to manually close the stream after using it. It’s automatically closed when the stream is dropped. Do try out the Live Queries in the Rust SDK & share your experience in our community!

Release v1.1.0

Feranmi Okafor — Tue, 09 Jan 2024 17:50:29 +0000

Hello developers!

After months of hard work, we're excited to roll out v1.1.0 .

This release is packed with features and improvements that we believe will make your development experience smoother and more efficient. From additional configuration flags, performance improvements, memory leak fixes, bug fixes, and new features, there's a lot to explore. We've also improved the internal memory allocators and released machine learning model computation powered by SurrealML. This release incorporates a number of contributions and feature suggestions from the community. For full release information, view the release notes.

Let's dive right into it and see what's new:

Run machine learning models in SurrealQL

SurrealML is our first step towards bringing Machine Learning to the Surreal ecosystem. With the introduction of our own file format with metadata and versioning included within the file, machine learning models can be greatly simplified, ensuring reproducibility and consistency in machine learning pipelines.

Using the surreal command line tool, we can import our SurrealML models directly into SurrealDB, effortlessly. SurrealDB automatically reads and understands the model requirements, immediately setting up a custom in-built function which can be used to infer results from the model itself. Inference works with either raw data inputs, for advanced usage, or with the field-name key bindings, packaged into the SurrealML file format itself. Developers no longer have to use external platforms or systems to run model predictions against data residing in their database. Instead the model logic can sit directly within the SurrealQL language, extending the power of custom functions.

As with live queries, the real power comes when we combine this functionality with the other powerful features within SurrealQL. Model inference can now be used seamlessly within EVENTs, LIVE QUERIES, and custom functions, both as a traditional database backend, or connected to directly from the browser.

CREATE house_listing SET squarefoot_col = 500.0, num_floors_col = 1.0;
CREATE house_listing SET squarefoot_col = 1000.0, num_floors_col = 2.0;
CREATE house_listing SET squarefoot_col = 1500.0, num_floors_col = 3.0;

SELECT * FROM (
    SELECT 
        *, 
        ml::houses::price-prediction<1.0.0>({ 
            squarefoot: squarefoot_col, 
            num_floors: num_floors_col 
        }) AS price_prediction 
    FROM house_listing
)
WHERE price_prediction > 177206.21875;

For more information regarding SurrealML, view the documentation and read the blog post.

Use live queries in the Rust SDK

This release introduces a new Live Query API to the Rust SDK, for powerful realtime updates in your Rust applications. By simply specifying .live() at the end of a select() query, you can receive a stream of changes, as they occur whtin a local or remote database.

// Select the namespace/database to use
db.use_ns("namespace").use_db("database").await?;

// Listen to all updates on a table
let mut stream = db.select("person").live().await?;

// Listen to updates on a range of records
let mut stream = db.select("person").range("jane".."john").live().await?;

// Listen to updates on a specific record
let mut stream = db.select(("person", "h5wxrf2ewk8xjxosxtyc")).live().await?;

// The returned stream implements `futures::Stream` so we can
// use it with `futures::StreamExt`, for example.
while let Some(result) = stream.next().await {
    handle(result);
}

// Handle the result of the live query notification
fn handle(result: Result<Notification<Person>>) {
    match result {
        Ok(notification) => println!("{notification:?}"),
        Err(error) => eprintln!("{error}"),
    }
}

Improved string parsing functionality

Previously, any string within SurrealQL was parsed as a Record ID, Datetime, or UUID if they had the same format as one of those types. In this release, string prefixes allow you to decide what value a string holds, forcing it to be parsed as a raw string, or forcing it to be parsed as a specific type.

-- Interpeted as a record ID, because of the structure with the semicolon:
RETURN "5:20";
-- Forcefully parsed as just a string
RETURN s"5:20";

-- This will be a record ID.
RETURN r"person:john";
-- This fails, as it's not a valid record ID.
RETURN r"I am not a record ID";

-- Example for a date and a UUID.
RETURN d"2023-11-28T11:41:20.262Z";
RETURN u"8c54161f-d4fe-4a74-9409-ed1e137040c1";

We're excited to see what users will build with the new functionality and performance improvements introduced in this release. For questions, find us on GitHub and Discord.