The 14-Year-Old Ghost in the Machine: How Tailscale and SQLite Solved a Decade-Old WAL Race Condition

In the world of distributed systems, there is no monster more terrifying than silent data corruption. When a system crashes, you have a stack trace. When a system hangs, you have a profile. But when a database silently writes the wrong bits to disk while reporting success, you are standing on a foundation of sand.

For the engineering team at Tailscale, this nightmare became a reality over a six-month period. It wasn’t a catastrophic, site-wide outage, but rather a slow, rhythmic pulse of failure: 19 instances of database corruption across their control plane shards. Each instance was a “Heisenbug”—nearly impossible to reproduce in staging and vanishing upon simple inspection.

The culprit turned out to be a race condition buried deep within the Write-Ahead Logging (WAL) mechanism of SQLite. It was a bug that had survived undetected for 14 years, dating back to the very introduction of WAL mode in 2010. This is the story of how a modern mesh networking company used custom filesystem shims and forensic I/O tracing to exorcise a ghost that had been haunting one of the world’s most used software libraries for over a decade.

Architecture: SQLite at the Heart of a Mesh Network

Tailscale is often thought of as a “VPN,” but technically it is a coordinated mesh network built on WireGuard. At its center lies the “control plane”—the coordination server that tells every node in your network how to find every other node.

To handle the massive scale of millions of connected devices, Tailscale uses a sharded architecture. Instead of one giant database, the control plane is split into many independent shards. Each shard is managed by a single-writer Go process. This design choice was intentional: by ensuring only one process writes to a specific database at a time, they eliminated a massive class of concurrency bugs common in distributed environments.

For the storage engine, Tailscale chose SQLite. The decision was rooted in the “boring technology” philosophy. SQLite is arguably the most tested piece of software on the planet. It is predictable, requires zero administration, and, when used in WAL mode, offers excellent performance for concurrent reads and writes.

The Backup Strategy and Manual Checkpointing

Because the control plane is critical infrastructure, Tailscale cannot afford to lose data. Their backup strategy involves streaming database snapshots to AWS S3. To ensure these backups are consistent, the system performs “manual checkpointing.”

In SQLite, a checkpoint is the process of moving transactions from the temporary WAL file back into the main database file. While SQLite can handle this automatically, Tailscale’s architecture required explicit control over this process to synchronize backups with S3 uploads. As we would later discover, it was this specific interaction between high-frequency manual checkpointing and heavy write loads that provided the perfect environment for a 14-year-old bug to finally emerge.

The Mechanics of SQLite WAL and Checkpointing

To understand the bug, we must first look under the hood of SQLite’s Write-Ahead Logging (WAL) and how it handles concurrency.

Before WAL was introduced in SQLite 3.7.0 (2010), SQLite used a “rollback journal.” To write data, it had to lock the entire database, preventing anyone else from reading. WAL changed the game by allowing multiple readers to coexist with a single writer.

How WAL Works

When you write to a WAL-enabled database, the changes aren’t written to the .db file immediately. Instead, they are appended to a separate .wal file.

1. The WAL-index (.shm file): To keep track of where the latest version of a page lives (is it in the main DB or the WAL?), SQLite uses a shared-memory index. This file allows processes to quickly locate the most recent data without scanning the entire WAL.
2. Snapshot Isolation: When a reader starts a transaction, it records the current end of the WAL. It sees a “snapshot” of the database at that moment. Even if a writer adds more data to the WAL, the reader remains isolated in its point-in-time view.
3. The Checkpoint: Eventually, the WAL file grows too large. A “checkpoint” operation takes the pages from the WAL and writes them back into the main .db file. Once all pages are migrated, the WAL can be “reset” or truncated, and the cycle begins anew.

The Checkpoint States

SQLite’s checkpointing is not an all-or-nothing affair. It uses a concept called nBackfill, which tracks how many pages in the WAL have been successfully copied to the main database. A checkpoint can be “passive” (copying what it can without blocking readers) or “restart” (resetting the WAL once all readers have finished with the old data).

The bug lived in the logic that determines when it is safe to reset the WAL and start writing new transactions at the beginning of the file.

The Investigation: Hunting a Heisenbug

The first sign of trouble was a failed integrity check. Tailscale runs PRAGMA integrity_check; on its S3 backups as a post-processing step. Occasionally, a backup would fail this check, reporting “Page X is stored in an unconventional way” or “Database disk image is malformed.”

The challenge was the frequency. 19 corruptions over six months across hundreds of shards is a needle in a haystack. The team couldn’t trigger it on demand. They tried “torture tests”—running thousands of concurrent transactions and checkpoints—but the database remained rock-solid.

Developing tstmpvfs

When standard debugging fails, you have to change the environment. The Tailscale team decided to build a custom SQLite Virtual File System (VFS) shim called tstmpvfs.

In SQLite, the VFS is the layer that sits between the database engine and the operating system’s filesystem. By writing a shim, the engineers could intercept every single Read, Write, and Lock call the database made.

“If we couldn’t reproduce the bug, we would record it when it happened in the wild.”

The tstmpvfs shim was deployed to production. It acted like a flight data recorder. When an integrity check eventually failed, the engineers didn’t just have a corrupt file; they had a complete log of every I/O operation and every POSIX lock acquired leading up to the corruption.

Analyzing the Patterns

By analyzing the logs from the corrupted instances, a pattern emerged. Every single corruption event involved a specific sequence:

1. A large write transaction was in progress.
2. A checkpoint operation completed and reset the WAL.
3. The writer continued, but its view of the WAL-index was now inconsistent with the actual state of the disk.

The “Heisenbug” was finally pinned down to a race condition between the sqlite3_checkpoint and the Pager (the module responsible for reading/writing pages).

The Breakthrough: The WAL-Reset Race Condition

The bug was located in how SQLite handles the “WAL-reset.” When a checkpoint finishes copying all pages from the WAL to the main DB, and there are no active readers using the WAL, the next writer is supposed to start writing from the beginning of the WAL file (offset 0). This is an optimization to keep the WAL file from growing indefinitely.

The 14-Year-Old Flaw

The flaw existed in the interaction between a writer and a checkpointer. Here is the simplified sequence of the race:

1. Writer A starts a transaction. It checks the WAL-index and sees that the WAL has, say, 100 pages.
2. Checkpointer B runs. It copies all 100 pages to the main DB. Since there are no active readers, it marks the WAL as “reset.”
3. Writer A continues its transaction. It wants to write a new page. Because it started before the reset, it still thinks it is appending to a WAL that is 100 pages long.
4. However, the internal state of the WAL-index header has been updated by the checkpointer.
5. Due to a logic error in the SQLite source code, the writer would occasionally use a stale “salt” value (a pair of integers used to validate WAL frames).

The writer would write a frame to the WAL with an inconsistent salt. Later, when the database tried to read that page, the salt wouldn’t match. SQLite would assume the WAL frame was part of an old, deleted transaction and ignore it, or worse, it would read half-written data. This resulted in the “silent” corruption: the write was successful, but the data was effectively unreadable or pointed to the wrong location.

This bug had been present since WAL was first committed to the SQLite repository in 2010. It remained hidden because it requires a very specific interleaving of manual checkpointing and high-concurrency writes—scenarios that are common in high-scale cloud infrastructure but rare in the embedded/mobile environments where SQLite is most prevalent.

Resolution and Upstreaming the Fix

Once the Tailscale team had a reproducible trace and a theory, they didn’t just patch their local version. They reached out to the SQLite core team, including its creator, Richard Hipp.

The collaboration was a masterclass in open-source maintenance. The SQLite team confirmed the race condition. It was a subtle flaw in the walIndexPage() function and the handling of the wal-index header.

The Fix: SQLite 3.45.1

The fix was officially released in SQLite 3.45.1. It involved hardening the WAL-reset logic to ensure that any writer currently in the middle of a transaction would correctly detect if a checkpoint had reset the WAL underneath it. If a reset is detected, the writer must now effectively “roll back” its internal index state to match the new reality of the disk.

Comparison of SQLite Versions Post-Discovery:

For more on how even the most robust CI/CD pipelines can miss edge cases in SQLite, see our analysis on SQLite pipeline vulnerabilities.

Lessons Learned: Hardening Database Infrastructure

The Tailscale investigation provides a roadmap for other teams managing critical data.

1. Treat “Rock-Solid” as a Probability, Not a Guarantee

SQLite is perhaps the most reliable database in existence, yet it still had a 14-year-old bug. Never assume your dependencies are perfect. If you are seeing “impossible” errors, believe your logs over the library’s reputation.

2. The Power of VFS Shims

If the Tailscale team had only looked at the corrupted database files, they might never have found the bug. By building tstmpvfs, they were able to observe the behavior of the system in real-time. If you are working with complex state machines (databases, filesystems, network protocols), invest in building observability shims.

3. Automated Integrity Checks

The only reason Tailscale caught this was their rigorous backup validation.

Rule of thumb: A backup that hasn’t been verified with PRAGMA integrity_check; (or its equivalent) is not a backup; it’s a liability.

4. Contribute Upstream

It would have been easy for Tailscale to simply switch to a different database or apply a hacky workaround. By working with Richard Hipp to fix the root cause, they improved the reliability of billions of devices—from iPhones to space probes—that rely on SQLite.

Conclusion: The Future of SQLite Reliability

The resolution of the WAL-reset bug marks a significant milestone in the history of SQLite. It serves as a reminder that as we push “boring” technologies into new environments—like sharded cloud control planes—we will inevitably find the edge cases that were invisible in simpler times.

Tailscale’s commitment to infrastructure stability has resulted in a more resilient SQLite for everyone. Today, the Tailscale control plane runs on the fixed versions of SQLite, and the “ghost” has been successfully exorcised. For the rest of the industry, the lesson is clear: robustness is not a destination you reach, but a continuous process of observation, skepticism, and contribution.

As we move toward more complex distributed systems, the “boring technology” philosophy remains more relevant than ever. But as this 14-year-old bug shows, even the most boring technology requires an adventurous engineering team to keep it running smoothly.