Luxant Solutions

NATS Leaving CNCF: Some History and Thoughts

Mon, 28 Apr 2025 17:45:46 GMT

Updated May 1, 2025:

Synadia and CNCF have come to an agreement: https://www.prnewswire.com/news-releases/cncf-and-synadia-align-on-securing-the-future-of-the-natsio-project-302444407.html

I’m glad things have settled down on the legal front. This leaves the NATS project and maintainers in an interesting position. There was a vote to leave, and the issue with CNCF concerning maintainership as related to graduation has not been resolved yet. With these updates, I sincerely hope a path to graduation can be found that will preserve the ethos and heart of NATS.

Updated April, 28, 2025: Removed comment about Synadia compensation.

There’s been a lot of heated discussion around NATS leaving CNCF and there are two sides to this coin. I’d like to share a few thoughts and history since I’ve been a maintainer with varying degrees of engagement since 2015.

CNCF and NATS

First, I want to say CNCF is an excellent organization for many projects! Projects like Kubernetes, Open Telemetry, Prometheus have thrived under CNCF. They have many contributors from large corporations and are solid projects mature in their lifecycle. These types of projects are a great fit for CNCF.

I’d also like to acknowledge that CNCF did help the NATS project in ways, and as a maintainer I’m grateful for that. The exciting conferences were a blast, we were able to speak and advocate for NATS, and we had a booth at most conferences. NATS was allowed keynote speaker time, we had some basic marketing assistance and reports. It was good stuff, and I have great memories of the conferences.

While CNCF is great for many projects, they aren’t for all. Behind the scenes things weren’t great for the NATS project.

Graduation Efforts

I was part of the effort to get the NATS project to a graduated status in CNCF. We were excited and proud of NATS - millions of downloads of the server, and we’d built a thriving community, so we proposed NATS for graduation in CNCF. From 2018 through 2020, at the behest of the CNCF TOC (Technical Oversight Committee), we changed our processes and added external maintainers to help in order to graduate. We had met the written qualifications of the graduation criteria. Despite this, the bar kept moving. The CNCF TOC felt that we had too many maintainers from Synadia on the server. This can be seen in the graduation PR and in comments made in the CNCF TOC meetings. It seemed like the graduation process became qualitative, and we were being held to a higher standard than previous projects. We had noticed another project with maintainers from primarily one organization had graduated with ease, which didn’t sit right.

Graduation stalled. After some time, it seemed pretty clear that NATS wasn’t a fit for the types of projects CNCF wanted. At this point we internally started discussing leaving CNCF as we felt NATS couldn’t graduate in a form that would keep the NATS culture of moving fast and allow NATS to evolve at the pace the community wanted and expected. Drastically changing the project wasn’t what we signed up for or expected. We’d discussed another home, and even kicked around the idea of creating a foundation of our own. This was 2020, so leaving now isn’t a surprise to many of the NATS maintainers.

Ultimately this resulted in a 100% “yea” vote to leave from the maintainers that participated in the voting process. While CNCF is wonderful for many projects, it was very clear that CNCF was not the right fit for NATS.

NATS and Synadia

When I was product manager of NATS (across Apcera and Synadia), we spent hundreds of thousands of engineering hours developing NATS, primarily paid by Synadia. We had some external contributions, primarily around clients and integrations, but core NATS remained primarily funded by Synadia with dedicated NATS engineers. There was a certain level of expertise required and it took months to truly understand the server. We would have certainly loved external maintainers, and invited some of our users who had the resources (namely large corporations) to donate developer hours. Some of the feedback we received was that the server was too complicated and companies preferred to fund development through commercial relationships. I can understand this, and it’s not difficult to connect the dots from this to Synadia forking with a BUSL license.

Synadia had to defend the NATS trademark, and I recall CNCF offered to compensate only a fraction of what we paid. It was a lot for a startup to cover, and made me wonder how committed CNCF was to NATS.

Leaving CNCF

Given the history, and the prospect of having NATS archived because it doesn't fit the CNCF mold, I understand and support the decision to leave CNCF.

To support their decision to prevent NATS from leaving, CNCF used the argument that NATS has a breadth of contributions across companies, even citing “ over 700 other organizations” . CNCF dismissed this same argument made from the NATS maintainers, preventing graduation . This was wild to read; I’m still shaking my head.

I do also understand CNCF’s frustration with NATS leaving. It’s unprecedented and could open the door for other projects to leave. Unfortunately Synadia was painted in a bad light when they will still offer free software to the NATS community (albeit via a different process). The maintainers of NATS and NATS’ creator, Derek Collison, strongly believe in open source principals.

Licensing

I don’t begrudge companies from using a BUSL license alongside OSS contributions. Sure, there’s a knee-jerk reaction - I’m a big open source advocate and to be honest, it stings. However, from the other point of view, an OSS project of NATS magnitude is a tremendous contribution from a startup. Large enterprises who use free projects should support them in some form or another. But they often do not. I’d rather see NATS live on than be archived by CNCF. We have to remind ourselves - as much as we like free open source software, we’re not entitled to it. It’s a gift.

Wrapping Up

We can’t tell exactly what the future of NATS will look like but I’m hoping for the best. Given that Synadia has funded nearly all of the development, allowing them to further commercialize the technology will help ensure the future of the NATS project. I understand NATS will still be OSS, and they’ll still provide open and free software with new features for companies under a certain revenue level, developers, and hobbyists.

All in all, this is a change for NATS and CNCF, but I understand it.

NATS Graduation PR: https://github.com/cncf/toc/pull/168
CNCF's Post about NATS leaving: https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-integrity-of-open-source-cncfs-commitment-to-the-community
Synadia's Response: https://www.synadia.com/blog/synadia-response-to-cncf

Squandering Money: Fear-Driven Decisions in Streaming

Mon, 27 Jan 2025 17:34:11 GMT

Most messaging and streaming technologies have the ability to persist data to replay later or allow for stateless application design (persistence) and to guarantee a quality of service - ensuring message delivery (durability). In most messaging technologies this involves storing data on the file system.

Some systems, like Kafka, require at least some level of persistent storage and can be tuned via data retention configuration and replication factor. In other systems, like MQTT, a per message QoS dictates durability from none at all to exactly once delivery where replication options vary by vendor. NATS.io enables data durability and persistence when the JetStream subsystem is used, allowing for both persistence/durability and at-most-once delivery at the same time.

Regardless of your messaging system choice, configuration around data storage and replication has implications. Many organizations struggle with finding the best choices for their business and make the mistake of defaulting to what is perceived as the safe bet - a data persistence strategy where data is replicated far beyond what is required. This comes with a cost that should not be ignored.

Defaulting to Data Persistence

Should organizations default to maximizing data persistence and minimizing risk?

No! I’ve seen teams default to producer and consumer acknowledgements and choose to replicate every message, all the time, in five places. When asked why, the answer is simply “we cannot miss a message,” without knowing business SLAs (Service Level Agreements) or the consequences of a SLA violation.

This stance of replication-to-the-max is certainly understandable, especially in large enterprises where there’s a disconnect between business requirements and technical requirements. When there’s a missing or delayed transaction, who takes the blame? What is the cost of temporarily lost data or a lost transaction? Is the transaction truly lost or can it be retried? Fear and uncertainty around these questions result in overprovisioned systems that are expensive and negatively impact performance and scalability. While you can reduce the probability of downtime (or even data loss) to something tenable, you’ll never get to zero. Ever.

Back to the question, I challenge you to never default to persistence . In fact, go the other way, and work up from the minimum to what you need and will let you sleep at night. Default to doing what is the best fit for your business in terms of cost and meeting SLAs through balancing throughput (units of work per second), latency (how fast a unit of work can be completed), and availability (the probability a unit of work will complete). Consider the notion that persistence is not even necessary in some data flows and retries/errors can be ignored or handled at the application layer.

Think of the many resilient applications built atop HTTP while still adhering to stringent business SLAs. It is of course cumbersome and pushes a lot of logic to the application layer, but the point is you don’t always need to use persistence provided by your distributed communication technology. Durability and persistence is a tremendous convenience which comes at the cost of some increased operational complexity, additional hardware/network resources, or additional SaaS cost (if you’re using a service).

Determining Persistence and Durability Needs

There are a number of factors to consider around data persistence and durability. Note that these come from business requirements - which should drive technical decisions.

Some questions to think about include:

What is the business SLA for a transaction or unit of work?
What is your risk tolerance? Businesses should look at what the consequences are for not meeting SLAs in both the tangible and intangible. Is a lost/delayed transaction or some downtime a rare and minor inconvenience? How much potential revenue will you lose per hour of downtime? Will you incur hefty financial penalties? Or worse, will you lose a customer?
How important is total cost of ownership? Is spending significantly more money for a few hundredths of a percentage of uptime actually worth it? In some sectors, like Fintech, this is a justified resounding “yes!”, but for other lines of business it may be difficult to rationalize the additional expenditure.
Is the transaction retryable outside the messaging system?
Which applications require what quality of service? Most distributed messaging systems have flexibility in selecting durability options so you can mix and match based on the business needs behind groups of applications. Optimization here will reduce complexity and improve performance while meeting your SLAs.

Statistics over Intuition

It’s easy and safe just to decide you want to cover your bases and go all out, then request the additional budget for more machines, clusters, storage, etc. However, let’s run through a quick scenario.

Assuming you require a 99.9% uptime guarantee for your business and are deciding between a server cluster of three nodes versus five nodes with quorum based replication. We’ll start with a cloud provider instance level guarantee of 99.5% uptime per instance (messaging server). For the sake of argument, let’s pessimistically subtract 0.5 % for software/network errors to have a 99.0% uptime for software and hardware resources. This results in a downtime of several hours per month by instance… but remember, we’re clustered.

With three nodes in a quorum based system and a replication factor of three (meaning at least two nodes must be available for business continuity), using binary probability calculations with each node having a 99.0% percent uptime, you’ll have a cumulative probability of P(X >= 2) of 99.97% uptime.

With five nodes, you’ll need at least three nodes available to continue business, resulting in a cumulative probability of P(X >= 3) of 99.999% uptime.

In this example, either choice meets the business SLA requirements. If you choose the three node deployment and you’ve saved yourself provisioning two extra nodes and potentially have increased performance with the lower replication factor - lean and mean. This could reduce TCO in terms of resources by 40% at the expense of potentially losing 0.029% uptime. In large multi-clustered deployments, this can represent hundreds of thousands of dollars per year.

If you measure your uptime and continue to meet the business SLAs, then you’ve justified the cost savings and become a hero to those that budget. Bonuses, anyone?

Buffers, Not Databases

Since we’re talking about risk and persistence, let's go the other direction and discuss a relatively risky usage pattern of messaging persistence that ends up being expensive as well. One should not use a messaging system in lieu of a database/datastore or other source of truth. There are blogs and articles about using your messaging system to replace a database, and honestly some good arguments can be made on paper. Some vendors offer database-like interfaces to access data (e.g. KSQL).

In practice I suggest you do not rely on your messaging system as a database or long term datastore. There’s no problem using it as such, but the point is not to rely on it. Messaging systems are highly complex and less reliable than time tested database technology which have been production tested for decades. Messaging technology may get close someday, but it is not there quite yet. Just think about the distributed nature of all the moving parts and the unstructured way data can be handled in applications and this makes sense. Bugs are a fact of life, data corruption is possible, and occasionally user implementation mistakes such as inadvertent poison pill messages will take down an application ecosystem (where the best option to recover is usually to wipe out and replace data in the messaging system).

Instead, think of your messaging system’s persistent features as a buffer - albeit a potentially very large and long lived buffer - that enables you to meet your SLAs and provide some short/medium term guarantees. With this mindset you’ll implement the ability to repopulate data in your messaging system from a source of truth like a database in case there’s a catastrophic incident.

This also opens up other opportunities to shed costs, such as shutting a cluster down at night and increasing performance and saving resources by keeping data for a shorter period of time, potentially on high speed local SSDs.

Wrapping it Up

The decision to persist data in a messaging system is more nuanced than it seems at first glance. Don’t let fear default you into overprovisioning, but instead calculate what uptime you need from your messaging system then design and provision appropriately, and of course measure and test.

Business groups and technical groups should collaborate on the SLAs required from the messaging system and track them. If you’re meeting your agreed upon uptime SLAs you can continue to justify decisions to reduce cost.

If you have questions around designing your distributed system, certainly reach out at colin@luxantsolutions.com . We’re happy to help.

How to Tank a Great POC

Wed, 13 Sep 2023 19:01:02 GMT

Here are a few ways to totally tank a POC for edge or cloud, with some tips to succeed too.

Vendor selection and Proof of Concepts (POCs) are not things your teams look forward to, especially with distributed technologies behind cloud and edge computing. While learning a new technology is exciting, executing a POC is like going to the dentist: It needs to be done, be done right, and no-one really enjoys going through the experience.

The goal of selecting a vendor through a POC is to determine the best technology to use to enable your business so a holistic approach is ideal. Choosing the best technology will increase efficiency, reduce costs, accelerate time to value, and hold up under your needs of today and of tomorrow to create a competitive advantage for your business. Choosing the wrong technology will likely result in lost revenue, opportunity cost, and may hurt your business (and reputation).

The most effective vendor selection processes and POCs require a mix of understanding the true needs of a business, software engineering, and devops. A technology can be wildly successful in performance and feature match, yet ultimately not best for your business goals. Or inexpensive but hard to use and unreliable.

Here are a few of the many ways one can tank vendor selection and a POC:

Don’t expand the POC beyond the technology team. Business analysts and internal customers do not need to be in the weeds or even provide technical requirements, but garnering as much input as you can to inform high level POC requirements will pay off.
Don’t think about the future, only think about what's needed today. Instead you’ll want to plan for what your business will look like in three years or even five years from now, as best you can and be optimistic! You don’t necessarily want to be doing another POC next year, and I’ve seen this happen a few times in my career.
Don’t consider the total cost of ownership. In reality costs come in many forms: capital, additional physical resources, devops time, support, and engineering hours. Consider ramp up time for those who will be maintaining the system to estimate time to value. There are multiple facets as to how technology incurs costs on your business.
Disregard vendor advice, especially around architecture and required features. This happens frequently. For example an evaluator will say: “I need every piece of data ever transmitted replicated in five places, always.” This isn’t realistic and robust systems have been built upon much, much less stringent requirements. The vendor wants you to succeed and will optimize to meet your goals which should work in your favor. Listen to them.
Don’t perform any business or project due-diligence. A great technology without support or growth is not ideal for your business. You don’t want to invest your future in a product with inadequate support, a dubious future, or an OSS project without a healthy and active maintainer base. Let’s avoid another POC next year.

How to best approach vendor selection and POCs takes more effort but will pay off.

Gather all stakeholders and get the high level business requirements with future projections . Along with general requirements these include SLAs, data sovereignty requirements, and compliance needs, if applicable (PCI, SOX, GDPR, etc). Understanding compliance penalties can really help prioritize features. This can uncover new items missing today, and it doesn’t hurt to get a wishlist started. Additional features, data, and insights could be low hanging fruit for a new technology you’re evaluating and could give your business a competitive advantage.
Garner technical requirements from the business requirements to identify general needs, throughput, latency, maximum data sizes, authN and authZ requirements (informed by compliance). SLAs will dictate uptime and data availability which in turn determines HA configuration, cluster sizes and deployment topology.
Find appropriate technologies and perform due diligence. You’ll want to get a sense of the health of the company behind the technology, and if you are using open source, determine the health of the project including growth and adoption. Ask for references of customers operating at the scale you plan to. Vendors hate them, but backdoor references are extremely valuable.
Simple testing of throughput, latency, and identifying feature gaps can weed out some vendors. Speak with the vendor if you’re not happy with the results; they’ll fight to stay in the POC and could have alternatives that work out nicely or guide your usage of the technology. Emulate your target production environment, especially if you’re dealing with network or CPU limitations at the edge.
Build out the technical POC . Create a few examples of data flows emulating real time systems and build these into the POC to perform benchmarks. Don’t forget about observability, especially in edge and IoT - identifying and predicting trends can avoid costly outages and inform your business with valuable metrics. If applicable, be sure to look at auditing features for compliance.
Now that you have a POC running, estimate how much this will cost. Along with cloud and edge spend, consider support and human costs. Certainly project costs as you scale up to your five year goal. Note that many cloud providers start out economical, but costs may grow exponentially as you scale - so you’ll want to do the math (and they often won't make the calculations easy). An aside - the pricing model of the big CPs has been great for NATS adoption at the edge!
Take inventory of your team and efforts to build/convert your applications. For example, If your team works in C and Go, but there are only java/.NET SDKs available, then that could inform your decision. Check out integrations with your existing technology stack. Also weigh the input of those who will be working with the technology. You don’t necessarily want to select something everyone hates to work with because it’s a bit cheaper. Like it or not, that’ll affect time to value.

A good POC will inform the next steps in putting a technology to task and look at a vendor and technology holistically. Hopefully this will help you in your journey to find the right technology that will propel your business forward through efficiency, cost reduction, and getting that competitive edge.

If you need guidance in selecting and proving out a distributed communications strategy for your business, Luxant Solutions can help. Email me at colin@luxantsolutions.com .

Launching Luxant Solutions

Tue, 04 Apr 2023 15:51:19 GMT

Luxant Solutions is open for business!

Open for Business

Many companies have a need around expertise with distributed systems, and choosing the right technologies and best patterns, identifying bottlenecks and vulnerabilities, and learning how to scale and best integrate with their business processes. Unlike 25 years ago when high speed messaging was primarily leveraged by trading floors, cloud computing has become ubiquitous and the market is headed very quickly to the edge, and nearly every business today is forced to deal with similar problems, except often at a much larger scale. I created Luxant Solutions to help companies solve those problems.

The Dream

Years ago I was flying to New York City as a TIBCO employee to consult with a fintech trading/clearing corporation and struck up a conversation with an independent marketing consultant. He spoke of taking a month off in the summer to vacation and taking December off to ski. I couldn’t believe it! That planted the seed… Someday I wanted the freedom to choose engagements, take time off, or lean in when excited about a new technology.

The Journey

Over the next 25 years, my career wandered through consulting, engineering, software architecture, principal engineer roles, and ultimately into product management to reach an executive level. Product was fun - I loved the creative side, and due to the nature of a startup which involves wearing many hats was still able to help NATS users and Synadia customers with their architecture and design. That was one of the things I most enjoyed and often looked forward to.

I’ve been truly lucky in my career to work on cutting edge communication technologies. I’ve helped build six distributed communications technologies: PLATINUM’s PEC, Talarian’s SmartSockets, TIBCO’s RV, EMS, FTL, and PGM and most recently NATS.io at Apcera and Synadia - most if not all of these are still in use today! I’ve helped integrate many systems over the years with these technologies.

I’d always wanted to hang out a shingle of my own, have more control over my work/life balance, and be able to spend more time with my family while still experiencing the reward of leaving a customer successful and delighted.

Today

It’s interesting. As noted earlier, many companies and enterprises are struggling with distributed communications and computing, and encountering similar problems to what we were solving 20 years ago, just at scale. In fact, in some ways things are easier (Kubernetes deployments), yet at the same time much more complicated (debugging the deep tech stack of Kubernetes). We have massive deployments extending to the edge and devices. We’re no longer dealing as much around shaving off microseconds (although that’s still fun and has its place) as we are handling resiliency at a truly massive scale. We can do so much more with the technologies we have today and are always pushing the boundaries from making edge deployments in stores and oil rigs, and making connected cars autonomous via built-in store, and going forward all the way to forming arbitrary mesh networks in space with satellites. Exciting times!

At Synadia we used to joke that being good at distributed system design wasn’t necessarily rocket science, but rather just having a lot of scars. If you need some expertise in building out your system, microservices, or distributed applications, reach out and hopefully avoid some scars of your own.

No matter the scale of the deployment, what I care most about is leaving a customer with efficient business processes backed by a smoothly running and predictable distributed system. Let me know if Luxant can help.