Neki solves the scaling limits of single-instance Postgres without sacrificing compatibility or forcing application-level sharding. It provides a managed path to horizontal scaling with online operations, making it easier to handle massive workloads while keeping standard Postgres tools.
Neki is now available in platform preview.
Neki is built from lessons we’ve learned over eight years of running some of the largest sharded MySQL clusters in the world. Thousands of production workloads with millions of queries per second for companies where even a few seconds of downtime is a very public event. We know what it means to power the world’s biggest tier 0 workloads.
When we released PlanetScale Postgres a year and a half ago, we knew we needed to do more. In that time we’ve onboarded several thousands of customers on PlanetScale, some of them rivaling the size of our largest MySQL customers. Time and time again, we watched teams approach the ceiling of a single machine with Postgres. Metal bought them time, but with customers hitting the upper limit of what a single machine is capable of, we found there was no good option to hand them. Enter Neki.
Neki is sharded Postgres from PlanetScale. It lets you scale a Postgres database across many machines while keeping real Postgres on every shard.
Your application connects to a Neki router over the standard Postgres wire protocol, so your existing drivers, ORMs, and connection string keep working. Each shard is a full Postgres cluster with one primary and at least two replicas across 3 availability zones. There is no custom storage engine, so extensions, SQL support, and performance behave the way Postgres does.
You choose the shard key and control how tables are grouped and distributed through a JSON data topology. Schema changes, version upgrades, failovers, imports, and resharding all run as built-in fully online workflows. You also get the PlanetScale features you already rely on, including Insights, schema recommendations, branching, and MCP.
You don't have to shard on day one. Run Neki as a single primary with replicas, and when you outgrow one machine, resharding is a workflow you run against the cluster you already have.
You already know the problems that come with fast-growing Postgres databases: tables too large to vacuum or index without affecting traffic, backups taking hours, connection limits, maintenance windows for schema changes, transaction wraparound and so much more.
You can move to a bigger instance, but eventually you run out of big enough machines, and the problems don’t scale linearly as you add more cores and IOPS.
The existing answers each ask you to give something up. Application-level sharding pushes routing into your code. Postgres-”compatible” distributed databases hide the shard key from you, take away your extensions, and add complexity and latency which becomes difficult to handle and debug.
So we built Neki with a few principles, the biggest one being: stick to Postgres, don’t work around it, fake it, or turn away from it.
We architected Neki from first principles for Postgres, with real Postgres on every shard. There are four moving parts.
Your application first connects to a Neki router. The router speaks the Postgres wire protocol so your existing drivers and ORMs keep working with a single connection string. A router has a full Postgres query parser, a distributed query planner, query buffering and more. It parses your query, builds a plan that decides which shards should run it, sends the work out, and combines the results back into one stream. Routers can scale vertically and horizontally, so no single router becomes the bottleneck.
Every shard in Neki is real Postgres with 1 primary and at least 2 replicas, spread across availability zones. There is no modified storage engine. Extensions, SQL support, and performance behave the way Postgres behaves, because it is Postgres.
Shards are organized into shard groups, so different tables or workloads can live on different sets of shards. Each shard uses a configuration profile that defines its instance size, replica count, storage, Postgres parameters, and extensions, so you can size each group for its own traffic.
Sidecars run alongside every Postgres instance. This is the piece that makes Neki's connection handling meaningfully better than just sticking PgBouncer in front of a database. Because Neki controls both ends of the connection, the router side and the Postgres side, it can size pools to what each instance can actually serve instead of estimating from outside the process.
The control plane tracks the health of every node, runs planned switchovers and unplanned failovers, and coordinates the workflows that reshard data, apply schema changes, and perform version upgrades.
Tying it together is the data topology, a JSON configuration that maps your logical tables onto physical shards. You define shard indexes, which specify the column Neki routes on and how that value gets hashed, and shard groups, which control how many shards a set of tables spreads across and which shards those are. Routers cache the topology and consult it on every plan.
Everything you would normally schedule a maintenance window for runs as a built-in workflow in Neki. Workflows provision new target nodes, catch them up with replication, switch traffic with a __neki metafunction, and retire the old nodes. All through the same psql connection your application uses.
This online operations model covers schema changes, version upgrades, planned and unplanned failovers, imports, and resharding.
Neki also includes all of the features you’ve come to rely on with PlanetScale: Insights, schema recommendations, branching, MCP, and more.
You can also run Neki unsharded, as a single primary with replicas. You get the improved connection pooling, online DDL, zero downtime upgrades, and health monitoring before you need to shard. When you do, resharding is a workflow you run against the cluster you already have.
We wanted to get Neki into your hands as soon as possible. You should not run production workloads on Neki during the platform preview. The product is still changing, and some of those changes will be breaking.
If you have any feedback, questions, or face any issues during the platform preview, please let us know. Fill out a support ticket or join our Discord
Sign in to PlanetScale, opt in to the platform preview, and create a Neki cluster. Read the Neki docs for more information about Neki's architecture, how to shard, and more.
If you have a large Postgres cluster and are curious whether Neki is a good fit, get in touch. We would love to do a private demo for your team, dig into your schema and query patterns, and give you real suggestions on how to shard.
Continue reading on the original blog to support the author
Read full articleUnderstanding sharded query lifecycles is essential for engineers scaling relational databases. It reveals the trade-offs in query routing and aggregation necessary for high-performance distributed systems.
Neki enables Postgres to scale horizontally by decoupling connection management and query routing from the core engine. It allows developers to use standard drivers while transparently handling sharding, complex distributed queries, and high connection counts.
Large tables in Postgres create systemic risks, from vacuum bloat to transaction ID wraparound. Understanding how to mitigate these through partitioning or sharding is critical for maintaining high-availability systems as data scales beyond the limits of vertical hardware.
Understanding the evolution of database sharding helps engineers choose between application-level logic, proxy-based routing, or extension-driven distribution when scaling relational workloads beyond a single node's capacity.
