Data Synchronization Between Microservices

With microservice architectures becoming the standard nowadays, it is a common need to extract data from your database to synchronize other downstream services.

Consider a social network where you have a service that stores new posts in a MongoDB collection. You may want to notify other services when a post is created, perhaps so that you can save it on Redis for fast retrieval, or add it to Elastic for full text search queries.

k3s is a lightweight Kubernetes distribution suitable for IoT and edge computing environments. One component k3s leverages is KINE , which is a shim enabling the replacement of etcd with alternate storage backends originally targeting relational databases.

In April 2022, the v0.9.0 release of KINE introduced native support for NATS as a backend. In June 2022, the k3s v1.23.7+k3s1 release included this KINE version making it possible for k3s deployments to connect to an existing NATS system.

The NATS maintainers are very proud to announce the 2.9.16 release ! 🥳

This release brings another round of stability and performance improvements, leveraging our new quality engineering practices described in the 2.9.15 announcement post , and iterating directly with users having high-scale and latency-sensitive workloads.

Given the positive feedback on the previous announcement post focusing on the higher-level impact of the changes, going forward all releases will have an announcement post!

In this post we will cover a few key areas:

Normally, the NATS maintainers do not publish a dedicated announcement for a patch release, but the 2.9.15 release of the NATS server has several significant stability and performance improvements worth highlighting, all driven by our new quality engineering process! 🥳

What does a patch release mean to NATS? And what are these new quality engineering processes?

Release management

At the beginning of 2023, the NATS maintainers drafted an end-to-end “release management” document which includes the policies and procedures used for each NATS server release, everything from initial scoping to communication of the release itself.

WunderGraph

At WunderGraph, we’re building an infra-less cloud for developers, meaning that we free developers from dealing with infrastructure. We believe a developer doesn’t need to be proficient in deploying and maintaining infrastructure to build a global SaaS.

WunderGraph is here to make our life easier (We use WunderGraph to build WunderGraph Cloud ). It allows you to integrate data sources of different types like REST, GraphQL, and Databases into a unified representation (the Virtual Graph) to make the work with your API dependencies a pleasant experience.

For those relatively new to NATS, it may come as a surprise that the NATS server has native support for WebSockets !

What this means in practice, is that the NATS server binary is able to support direct WebSocket connections from the browser (or other clients) using it as a transport layer for the NATS protocol rather than raw TCP packets.

In addition to server-side support, there is an official nats.ws client library which builds upon the nats.deno client library and swaps out the transport layer for WebSockets.

One feature, released in the v2.9.0 NATS server , that flew under the radar was the new DiscardNewPerSubject option on a stream. This blog post will describe this new feature as well as give a practical example of how it can be used to provide exactly-once message publication quality of service (QoS) through infinite deduplication that goes beyond the existing time-based deduplication feature of JetStream as well as many other streaming systems, such as Kafka.

Matrix Dendrite

Matrix is a federated protocol for real-time communications built on top of distributed data structures and Dendrite is an implementation of a Matrix homeserver built following a microservice architecture. It was originally intended to help us to overcome some load issues that were present in earlier monolithic implementations at the time. As a result, flexibility, performance, and scalability have been three critical areas of focus in our development.

In the Beginning with Kafka

When we first started the Dendrite project, we were using Kafka as a means to distribute events and asynchronous tasks between microservice components. In addition to being able to scale up and down as needed, we also needed durability in the message queue to ensure that if parts of the service were restarted, no tasks or events were lost. Given our need for deployment flexibility, it’s equally important for all of the microservices to run under a single process as it is to be able to split them out and run them under separate processes - or even on different machines. We wanted to be able to target embedded and small platforms, as well as the datacentre.

The NATS core team and contributors are proud to announce the first release of the 2.9.x series! 🚀

Thank you to the 48(!) people who contributed to this release through GitHub issues, discussions, and pull requests, as well as the hundreds of people in the Slack community who asked questions, presented use cases, and tested early versions of the 2.9 release. ❤️

This release is a milestone for users leveraging JetStream with many improvements and a handful of new features.

Benchmarks for throughput and speed have always been a desired data point for choosing messaging infrastructures but can be controversial due to the varying circumstances of the hardware, environment, and software version in use.

Published benchmarks are often deemed not credible as the results can be skewed to improve performance and therefore become unreliable for critical infrastructure decision making. With this in mind, we always recommend performing benchmarks using an environment as close to your production use as possible.