OpenAI announced the Pando project, within which a new backend has been developed, compatible with the Git protocol and tools, and has already been implemented to serve the company’s internal monolithic repository. The need to create your own repository for Git is explained by the exponential growth in the number of parallel push and fetch operations due to the massive use of AI agents at OpenAI, to which classical approaches to scaling Git, such as Three-phase_commit_protocol and write-ahead logging (WAL). The implementation of Pando reduced the median latency of push operations by almost 80% and significantly increased the reliability of merge operations. The OpenAI repository deployed on Pando processes about a petabyte of Git data per day with peak loads of up to 57 TB of data per hour.
The key difference between the Pando architecture and the standard Git repository was the refusal to ensure global consistency of branches and links at the level of the entire repository at any given time in favor of causal consistency for individual branches. The new approach made it possible to process push requests in parallel and update different branches independently without locking. For metadata, a separate distributed database is used, which stores pointers to branches (refs) and data about which commits and objects are located inside the saved pack files.
To reduce the load on the CPU and storage, instead of generating a unique pack file for each client, during fetch operations pack files are stored as immutable objects in a shared blob storage and distributed to other clients directly from the cache. To ensure that pack files can be reused, when performing a push operation, the server shows only the trunk to the client, which ensures predictable dependencies in the pack file. To transfer missing data to clients, background packaging is used, implemented in a similar way to LSM trees (many small packages are combined into larger and better compressed ones).
The Pando backend is written in Rust and brought ready for implementation in 10 weeks. Most of the code was generated and tested by AI agents using formal verification using the TLA+ (Temporal Logic of Actions + Data) and Lean