Meet DBOS: A Database Alternative to Kubernetes
Turing Award laureate Dr. Mike Stonebraker just keeps on inventing databases. Forty years ago, it was the first relational system, Ingress, and thirty years ago, it was PostgreSQL, More recently, he co-created an in-memory transactional database system, VoltDB.
Now, he’s back with a database system designed to replace the entire cloud native computing stack, DBOS (Database Operating System).
Linux is too old, and Kubernetes is too complicated, the start-up behind this work has proclaimed. Now a database has been designed to replace all of them.
To make this vision happen, DBOS, Inc., has raised $8.5 million in seed funding led by Engine Ventures, along with Construct Capital, Sinewave, and GutBrain Ventures.
The project was founded by Dr. Stonebraker, along with Apache Spark creator (and Databricks co-founder and CTO, Matei Zaharia), and a joint team of MIT and Stanford computer scientists.
DBOS runs operating system services on top of a high-performance distributed database. All state, logs, and other system data are stored in SQL-accessible tables.
The result is a scalable, fault-tolerant, and cyber-resilient serverless compute cloud for cloud native applications, the creators claim.
With an OS running on top of a distributed database, you get fault tolerance, multinode scaling, and state management. Observability and security gets easier. Gone are containers and orchestration layers.
“You write less code because the OS is doing more for you,” Stonebraker said.
Today, distributed systems on largely built on an operating system (Linux) designed to run on single server. This results in an incredible number of different variable states to manage, across the infrastructure stack (app data, authentication systems, messaging, cluster management).
This fragmented nature of course necessitates an untold amount of observation tools, and security tools, as all the states gives malicious hackers a fertile ground to work with.
What could handle a million states with ease? A database, of course.
In the DBOS design, a high-performance distributed OLTP would implement a suite of OS services. It would run on a minimal OS kernel, with support for memory management, device drivers, interrupt handlers, and basic tasks of byte management.
One Database to Rule Them All
This is not the first time this idea has been raised: As early as 2001, we recall Larry Ellison arguing that middleware was a “nitwit idea,” that everything should be managed by a database itself.
The central idea behind this DBOS project comes from a simple idea: Keeping track of the operating system state should be a database problem, Dr. Stonebraker said.
The idea came about from a talk from Zaharia. He noted that the Databricks cloud service for Apache Spark routinely managed a million subtasks at once. All the state and scheduling information was being tracked in a PostgreSQL database, the sluggish performance of which had frustrated the admin team at Databricks.
The database bottleneck could be solved easily enough. In fact, that’s what VoltDB was all about, concurrent ACID-compliant transactional processing that could be spread across multiple servers.
Back in the day, Dr. Stonebraker was an early user of Unix on a PDP 11/40 with 48k of main memory, and 25MB of disk memory. Then, all the states were kept by Unix itself. Of course, a million states is a jump of six orders of magnitude compared to the PDP. But “the amount of the amount of state that the operating system has to keep track of is basically proportional to resources,” Dr. Stonebraker said.
DBOS itself was tested on the Massachusetts Institute of Technology’s SuperCloud, with over 32,000 processors, terabytes of main memory and lots more terabytes of secondary storage.
At the bottom of the stack is a distributed transactional database system, with a file system, scheduling engine and messaging system all built on top.
They researchers discussed the stack at the Very Large Databases Conference 2023, detailing the work in a set of papers, covering ACID transactions and system replay.
Dr. Stonebraker labeled Linux as “leaky,” meaning there are many ways in which security vulnerabilities can be introduced. Plus, building an OS on top of a database would offer the ability to roll back to a state before a vulnerability has been exploited (Think of it as an Apple Time Machine but for servers).
The centralized database will also help in debugging, Dr. Stonebraker said. Breaking applications into microservices makes them very hard to debug or even to get the unwarranted behavior to show up on each test (these are known as “Heisenbugs“)
“We run all your microservices transactionally. So parallel microservices are sorted out by our concurrency control system and, basically, there are either no Heisenbugs or they’re much, much harder to run across,” Dr. Stonebraker said.
Originally, the system was mocked up on VoltDB, but he backers wanted to go with an open source key-value system instead, so they went with FoundatiolDB as the base. (Dr. Stonebraker admitted that any PostgreSQL wire-compatible distributed OLTP system would work, such as, say CockroachDB).
DBOS Cloud: A Distributed Database for Transactional Support
The first commercial service built around is DBOS Cloud, a transactional Functions as a Service (FaaS) platform, available for developers in this initial launch.
DBOS, which runs on Amazon Web Services‘ Firecracker, is initially available for developers to experience via DBOS Cloud, which launched today.
DBOS Cloud is a transactional serverless application platform powered by the DBOS operating system, and it can be used to build and run serverless functions, workflows, and applications. Think of it as AWS Lambda but with transactional support.
DBOS makes fault-tolerant TypeScript code vastly easier to create and less expensive to run (DBOS).
The service provides the following benefits:
- Support for stateful functions and workflows
- Built-in fault tolerance with guaranteed once-and-only-once execution
- Time-travel debugging
- SQL-accessible observability data
- Enablement of cyberattack self-detection and self-recovery
A GitHub repository includes some of the tools the company has developed, including a TypeScript framework for interacting with DBOS, and the “time-traveler debugger” for VSCode.
DBOS Cloud keeps a complete audit trail of code and data processing and stores it in encrypted SQL tables. The DBOS Cloud Time Travel Debugger allows that data to be replayed and examined to troubleshoot issues, ensure regulatory compliance, or look for fraud, etc. (DBoss)
