The history of Postgres sharding

Over the past 20 years, many companies have achieved such a scale that their database could no longer support the workload assigned to it. The solution is almost universally to shard the database, but only when absolutely necessary due to the added complexity.

MySQL was a much more popular choice for relational databases in the 2010s and because of this, the tooling around things like online schema changes and sharding progressed much quicker. Tools like gh-ost, Vitess, and more were built around this ecosystem.

Postgres, on the other hand, had one-offs, then Citus, then proxy-like solutions. Now, in 2026, Neki builds on the lessons learned from the incredible engineers that have built not only Vitess, but all the other sharding solutions that came before it.

To fully understand Neki, and why we built it, it's important to have an understanding of how we got here.

It started in a video game

One of the more prominent theories is that the term "Shard" originated, of all places, from a video game. Ultima Online is an MMORPG created in 1997. The developers quickly realized they needed to host players on multiple servers to sustain growth.

To support this, they invented lore that would make the idea of multiple servers make sense in the fictional universe. After digging through heaps of lore, "Shards" seemed like the best fit.

At the end of Ultima I, an evil wizard is defeated who had tried to trap the world into a crystal. When the wizard is defeated, his crystal breaks into "shards", each holding a copy of the world within it.

And thus, the term "sharding" was used to denote parallel, independently evolving servers. This term slowly made its way through the industry, narrowing its meaning to database sharding in particular.

Why MySQL got there first

The LAMP stack was all the rage in the early 2010s. This gave the sharding story for MySQL a head start, as most of the successful companies had started in an era where MySQL was the popular choice.

Facebook was one of the first companies to truly hit a global scale, serving over 100m active users in 2008 on roughly 1,800 MySQL nodes. Initially, sharding was done at the application level. Each user was assigned an ID that could be used to determine which shard their data lives on.

Eventually, as Facebook grew, their need for a specialized layer on top of MySQL became apparent. Facebook created TAO to accommodate the scale and access patterns that fit their needs.

YouTube, similarly, started as a single primary node. When the load grew too high, they added replicas. When the primary could no longer handle write pressure, they began the monumental task of sharding.

Originally, their solution was also baked into the application layer itself. YouTube's backend would determine which database server a user request should go to. Around 2010, they pulled out the routing layer into a standalone project, and thus Vitess was born.

Many companies that began scaling their LAMP-based stacks slowly consolidated on Vitess as their sharding solution.

It took Postgres much longer to get a similar solution. Many companies that would later hit Postgres scaling limits built in-house solutions. Most of these were never separated into standalone software, often due to how fine tuned they were for each company's needs.

Build it yourself, then live in it

Early Postgres sharding was frequently built as custom, in-house solutions. If a single Postgres node couldn't handle your scale, start spinning up more nodes, create a router, and a team of engineers to manage it.

For companies successful enough to need sharding, dedicating a team to solving database scaling was not an impossible ask. If the database can't scale up, the application will start failing, and growth stops.

Skype and PL/Proxy

In 2007 Skype announced PL/Proxy to the Postgres mailing list. The announced version was Skype's second iteration (v2) of PL/Proxy that ran as an extension on a Postgres Proxy database.

PL/Proxy was one of the earliest sharding solutions, allowing operators to route queries to different shards by defining SQL functions in the proxy database. The downside to this approach was that every operation that would result in a sharded query would need a defined function to route to the proper shard, defined by the user.

For instance, if you wanted to shard a users table, you would need to create PL/Proxy functions to insert, update, and read. For each new query pattern, you would need to create a new function and call it from the application layer.

Although this drastically improved scalability by removing the need for the application to keep track of each downstream shard, the application and database operators would need to work in tandem to ensure every query pattern is defined in the proxy and routed correctly.

Instagram and logical shards

Instagram is another early pioneer of Postgres sharding. In 2012, Instagram's ~30 million users could no longer fit onto a single EC2 instance running Postgres.

The solution for Instagram was to do logical sharding at the application layer. The application would map thousands of logical schemas down to a few physical shards. Any given physical database could have multiple logical schemas (collections of tables) within it.

When a given schema grew too large, it could be migrated to a new physical database, and the application would re-map that logical schema to the new database address. This was an elegant solution for their use case, since it required low overhead to manage. This system helped Instagram scale for years.

Instagram had debated going with PL/Proxy, but decided to hand-roll their own solution to keep with a more minimalistic design, specifically not needing to deal with the SQL function overhead, and for reduced network latency. Their application would talk directly to each shard without a proxy.

The downside to this approach, however, is that cross shard queries must be managed in the application layer, and moving data between databases requires careful coordination between the app's mapping of schemas and physical shards. Sharding was also limited to their custom integer ID approach, so every table that needed sharding would also need an ID column.

Then someone put it in Postgres

Citus was the first open-source sharding solution for Postgres that was designed for wider adoption. Designed originally in 2011, Citus started as a fork of Postgres.

Later on in 2016, Citus was refactored into a pure extension of Postgres, rather than a fork. Citus shards data by defining nodes as either coordinators or workers.

Every node is a standalone Postgres instance running the Citus extension. The coordinator node holds the routing map, routes queries as needed, and keeps track of all downstream worker nodes. Worker nodes store the sharded data, and execute the queries passed on to them from the coordinator.

Citus improves on earlier Postgres sharding solutions by keeping the sharding metadata and routing logic within one central node. This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck. Although some queries in Citus 11+ can go through worker nodes, your application either needs to send all requests to the coordinator node, or keep track of multiple connection strings to each worker it wants to distribute load onto.

Another downside is that shard management and backups are a semi-manual process. Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus. Managing backups is also external to Citus, so operators still need to build the proper infrastructure to accommodate disaster recovery and healing nodes as needed.

Other similar attempts have been made to bring sharding to Postgres such as PgDog and Aurora Limitless. PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially.

PgDog operates as a pure proxy layer in front of Postgres clusters. Although PgDog supports sharding, managing schema changes and resharding operations is still a very involved and manual task. Managing backups and disaster recovery of the Postgres nodes is also up to the operator.

Aurora operates in a similar fashion. Routers front Postgres nodes backed by EBS, with support for sharding.

Hide the shard key

Other solutions to scaling Postgres resulted in not using real Postgres at all, opting for a Postgres-compatible approach. Google published the Spanner research paper in 2012 while using it internally, later releasing it as a public service in 2017.

Spanner's approach hides sharding from the user and operator altogether, opting for automatic distribution of data across nodes. Instead of defining shard keys or indexes, Spanner splits and moves tables automatically as table size grows.

This is done by storing the data as key-value pairs. These key-value pairs are grouped into "splits" and replicated across the storage layer, using the Paxos consensus algorithm to maintain consistency.

CockroachDB and Yugabyte are similar distributed Postgres-compatible database solutions. Note that they are Postgres-compatible, not actual Postgres solutions. Inevitably this results in lower fine-tune control over the cluster compared to the large amount of settings and flags available for actual Postgres clusters.

All of these solutions share the same pitfalls. Performance can be disjoint from expected results since the underlying data access layer is not truly Postgres.

Extension support also becomes limited, along with some Postgres native SQL features and settings.

Data placement also becomes hard to predict. Related rows can easily drift across multiple storage locations causing drastically increased latency.

Explicit sharding makes it easy for engineers to understand and plan around cross-shard latency penalties. Automatic sharding blurs these lines, making it impossible to determine when or how a table will shard, and how drastically performance will drop because of it.

Building the best solution

The amount of engineering work that went into many of these systems cannot be overstated. As we have seen however, that does not make them perfect. Neki builds on the lessons and foundations from all of its predecessors.

Neki's goal is to raise the bar for scaling Postgres.

While Spanner and CockroachDB shard data automatically, Neki requires explicit sharding. Developers have complete control over exactly how data is distributed across clusters.

Neki also uses true Postgres clusters under the hood. This means support for Postgres extensions, stronger SQL compatibility, and predictable performance.

Compared to PL/Proxy or Citus, sharding is defined by a routing file called the data topology, allowing database operators to define and evolve sharding schemes over time. Since each Neki router only holds a cache of the topology, routers can be horizontally scaled with ease.

One connection string can go to any router, letting your application treat 1,000 shards as one unified database. No single router becomes the bottleneck.

Each Neki shard has its own primary and replicas. If a primary dies, Neki can bring the cluster back to a healthy state. Routers sit in front, connecting all the shards together.

Neki also allows each group of shards to be independently sizeable and scalable, making it possible to precisely control resources for every part of your cluster.

Unlike PgDog or Citus, Neki handles backups, restores, node health, Insights, connection pooling, schema changes, and much more. By being more than just an extension or pure proxy, Neki can manage every part of the database life cycle.

Sharding used to be one of the scariest things a growing team could face. Neki aims to change that, making it easier than ever for teams to reach truly planet-scale with as little friction as possible, the familiarity of real Postgres, and incredible performance. Sign up today for early access at neki.dev.