--- description: "Scality AI Inference Factory runs open-weight models on-prem with KV cache on ADI over RDMA: 166 ms warm TTFT, 14x faster than recompute." title: Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute image: https://www.storagereview.com/wp-content/uploads/2026/10/Storagereview-scality-AI-Inference-Factory-diagram-e1791449818719.jpeg --- [![Storage Review]()](https://storagereview.com/) --- [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [![StorageReview.com](https://www.storagereview.com/wp-content/uploads/2026/07/Storage-Review-2x.png)](https://www.storagereview.com) ≡ Menu - [Home](https://www.storagereview.com/) - [Storage Reviews](https://www.storagereview.com/review) - [End User Reviews](https://www.storagereview.com/consumer) - [Enterprise Reviews](https://www.storagereview.com/enterprise) - [Ubiquiti Reviews](https://www.storagereview.com/best/ubiquiti-reviews) - [SR Merch](https://store.storagereview.com) - [Leaderboards](https://www.storagereview.com/best) - [Best Storage Arrays](https://www.storagereview.com/best/storage-arrays) - [Best Enterprise SSDs](https://www.storagereview.com/best/enterprise-ssds) - [Best Servers](https://www.storagereview.com/best/servers) - [Best Enterprise GPUs](https://www.storagereview.com/best/enterprise-gpus) - [Best Desktops for Local AI](https://www.storagereview.com/best/desktops-local-ai) - [Best Laptops for Local AI](https://www.storagereview.com/best/laptops-local-ai) - [Best Local LLM Tools](https://www.storagereview.com/best/local-llm-tools) - [Agentic AI Hardware](https://www.storagereview.com/best/agentic-ai-hardware) - [Best SSDs & Hard Drives](https://www.storagereview.com/best_drives) - [Best Portable SSDs](https://www.storagereview.com/best/portable-ssds) - [Best Business Laptops](https://www.storagereview.com/best/business-laptops) - [Best Mobile Workstations](https://www.storagereview.com/best/mobile-workstations) - [Best Desktop Workstations](https://www.storagereview.com/best/desktop-workstations) - [Best Mini PCs](https://www.storagereview.com/best/mini-pcs) - [Best Laptop Battery Life](https://www.storagereview.com/best/laptop-battery-life) - [Storage Reference Guide](https://www.storagereview.com/storage-reference-guide) - [About SR](https://www.storagereview.com/about-storagereview) - [StorageReview.com Sweepstakes Rules and Regulations](https://www.storagereview.com/storagereview-com-sweepstakes-rules-and-regulations) Search [Home](https://www.storagereview.com/) » [News](https://www.storagereview.com/news) » Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute # Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute by Harold Fritts on October 8, 2026 [AI](https://www.storagereview.com/enterprise/ai)  ◇  [Enterprise](https://www.storagereview.com/enterprise) Scality today launched Scality AI Inference Factory, an open-code software stack for running open-weight models on infrastructure the customer owns. Scality validates, ships, and maintains the whole stack, which places its [AI Data Infrastructure (ADI)](https://www.storagereview.com/news/scality-introduces-adi-platform-for-ai-driven-sovereign-data-infrastructure) object storage under the inference layer as a shared [key-value (KV) cache](https://www.storagereview.com/review/the-token-efficient-path-for-long-context-inference-kv-cache-offload-to-flash), and it is aimed at enterprises, government agencies, and neo-cloud providers that want an alternative to per-token cloud inference. ![Scality AI Inference Factory architecture diagram showing client harnesses and agent frameworks above one endpoint, with control, compute, network, and storage planes, separate prefill and decode GPU pools, and the AI Connector on the storage plane](https://www.storagereview.com/wp-content/uploads/2026/10/Storagereview-scality-AI-Inference-Factory-diagram-e1791449818719.jpeg) Scality frames the launch around cost and control. Per-token pricing grows with every workflow that proves useful, providers can change model versions and quantization on their own schedule, and regulated organizations have to send prompts, documents, and code to infrastructure they do not run. The stack targets coding assistants, specialized chatbots, and agentic applications, and Scality says tools such as Claude Code, Codex, and OpenCode keep working once developers point them at the local endpoint. ## Disaggregated Serving Architecture and KV Caching Over RDMA The stack combines validated open-weight models, a serving layer built on vLLM and Dynamo that splits prefill from decode so each GPU pool scales on its own, and a control plane that authenticates and meters requests, routes them to GPUs holding the relevant context, and schedules work against SLA targets. Underneath sits ADI, which stores models, enterprise data, and inference state, connected to the GPUs through the Scality AI Connector. ADI’s main job is holding the KV cache outside GPU high-bandwidth memory. The prefill pool computes the cache for a new prompt and writes it to ADI, and the decode pool reads it back, so any decode GPU can pick up any returning session. vLLM breaks the cache into fixed-size named blocks that ADI stores as objects, and on that path the AI Connector replaces the S3 control plane with a stripped-down one that uses single-pass authentication and drops XML and multipart framing. On a read, the GPU reserves memory and tells ADI which object to place there. The storage server moves the object from TLC NVMe into its RAM over PCIe, and the network card writes it directly into GPU memory over RDMA, so the data bypasses the CPU, the kernel, and the software stack. Scality says that path reaches 97% of network line rate between GPUs and storage. Scality CTO Giorgio Regni described what that changes for serving: “The KV cache on Scality ADI is fast enough to sit in the serving path. Restoring a context from ADI is the same order of magnitude as GPU memory, and 14 times faster than recomputing it, with the GPUs staying busy the whole time. Storage is no longer the reason to keep the KV cache inside the GPU server.” ## Benchmarks and Internal Performance Metrics Scality ran its tests in a deliberately small lab: one eight-GPU server, five storage servers a few generations old, and four 100Gb/s RDMA links, which made the network the bottleneck. Scality says the product itself ran on a 400Gb/s fabric, and it did not name the GPU in the test server. Warm time to first token on a 14K-token context restored from ADI measured 166 ms, 83 ms behind serving the same context from HBM. Recomputing that context took 2.3 seconds, so retrieval was 14 times faster. On GLM-5.2 with a 438,764-token context, recompute took 529 seconds and the read-back took 7.3 seconds, a 72x gap, and that single session produced 189GB of KV cache, about 430KB per token, more than the HBM left over with the model loaded across all eight GPUs. In a second run with one vLLM instance per GPU, ADI held 8TB of KV cache, more than 80 times the memory of one GPU and about ten times that of the whole server. First-token latency stayed between 355 and 373 ms at every cache size from 69GB to 8TB, and inter-token latency did not move because the fetch completes before the first token. Scality says that capacity supports up to 1,000 concurrent sessions resumable without recompute. The same path loads model weights, and Gemma-3 27B’s 54.86GB of weights loaded into one GPU in 1.9 seconds, which Scality puts at almost 10x faster than the GPU server’s local drives because the read spreads across many drives in the storage tier. That’s about 28.9GB/s, roughly 58% of the four 100Gb/s links combined, and Scality says a GPU can swap models up to a thousand times a day depending on load. Scality also says it sent its fixes upstream. vLLM’s KV offload connector was fetching about five times the cache blocks each restore needed, and Scality corrected the connector without touching the vLLM core. It also extended the open-source ModelExpress loader to pull weights straight from object storage, and it says the shipping stack contains no Scality forks of any component. ## Hardware Compatibility and Tiering Scality AI Inference Factory deploys on standard servers from Dell, HPE, Lenovo, and Supermicro. ADI runs at multi-petabyte scale on NVMe, with RDMA access to HDD for bulk KV cache capacity, and a single namespace automatically tiers data across TLC flash, HDD, and optional tape, so model weights, enterprise data, and cold cache can each sit on the media that matches how often they are read. Validated model families include Mistral, Gemma, gpt-oss, Qwen, Kimi, GLM, and DeepSeek, and the stack is compatible with harnesses and agent frameworks including OpenCode, Hermes, Goose, LangGraph, and Pydantic AI. Every component ships as open code, so customers can inspect how inference state is stored and moved and submit changes for Scality’s engineers to review. Scality AI Inference Factory is available now as a software license or as a fully managed service. Scality is demonstrating the stack and its reference architecture at Scality Day in Paris today. ### [Scality AI Inference Factory](https://www.scality.com/ai-inference-factory) **Engage with StorageReview** [Newsletter](https://www.storagereview.com/storage_newsletter) |  [YouTube](https://www.youtube.com/user/StorageReview "Opens in a new window") | Podcast  [iTunes](https://podcasts.apple.com/gb/podcast/storagereview-com-storage-reviews/id1060681115 "Opens in a new window")/ [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0 "Opens in a new window") |  [Instagram](https://www.instagram.com/storagereview/ "Opens in a new window") |  [Twitter](https://twitter.com/storagereview "Opens in a new window") |  [TikTok](https://www.tiktok.com/@storagereview? "Opens in a new window") |  [RSS Feed](https://www.storagereview.com/rss.xml) ![]() ### Harold Fritts I have been in the tech industry since IBM created Selectric. My background, though, is writing. So I decided to get out of the pre-sales biz and return to my roots, doing a bit of writing but still being involved in technology. Previous post: [KIOXIA LD4 Brings QLC to E1.L for 1U Servers, Sampling at 30.72TB With a Path to 122.88TB](https://www.storagereview.com/news/kioxia-ld4-qlc-e1-l-ssd-1u-servers-30-72tb-122-88tb) Next post: [CTERA Forward Deployed Engineering Embeds Engineers With Customers to Put File-Data AI Into Production](https://www.storagereview.com/news/ctera-forward-deployed-engineering-embeds-engineers-file-data-ai-production) Trusted Vendors Products and solutions from our affiliate partners: - [ ![Ubiquiti Logo](https://www.storagereview.com/wp-content/uploads/2024/11/2-1.webp) Ubiquiti ](https://store.ui.com/us/en?a_aid=StorageReview "Opens in a new window") - [ ![Newegg Logo](https://www.storagereview.com/wp-content/uploads/2025/11/newegg.webp) Newegg ](https://click.linksynergy.com/fs-bin/click?id=g5terNMDhz0&offerid=1207190.18&subid=0&type=4 "Opens in a new window") - [ ![Amazon Logo](https://www.storagereview.com/wp-content/uploads/2025/11/amazon.webp) Amazon ](https://amzn.to/4fLJUEW "Opens in a new window") Newsletter Subscribe to the StorageReview newsletter to stay up to date on the latest news and reviews. We promise no spam! 1 Leave this field empty if you’re human: ## Advertisement Content Categories [Facebook](https://www.facebook.com/@storagereview) [X](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) Copyright © 1998-2025 Flying Pig Ventures, LLC Cincinnati, Ohio. All rights reserved. Manage your privacy To provide the best experiences, we and our partners use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us and our partners to process personal data such as browsing behavior or unique IDs on this site and show (non-) personalized ads. Not consenting or withdrawing consent, may adversely affect certain features and functions. Click below to consent to the above or make granular choices. Your choices will be applied to this site only. You can change your settings at any time, including withdrawing your consent, by using the toggles on the Cookie Policy, or by clicking on the manage consent button at the bottom of the screen. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage your privacy To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage consent Manage consent ```json {"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma","url":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma","name":"Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute - StorageReview.com","isPartOf":{"@id":"https:\/\/www.storagereview.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma#primaryimage"},"image":{"@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma#primaryimage"},"thumbnailUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2026\/10\/Storagereview-scality-AI-Inference-Factory-diagram-e1791449818719.jpeg","datePublished":"2026-10-08T16:32:44+00:00","description":"Scality AI Inference Factory runs open-weight models on-prem with KV cache on ADI over RDMA: 166 ms warm TTFT, 14x faster than recompute.","breadcrumb":{"@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma#primaryimage","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2026\/10\/Storagereview-scality-AI-Inference-Factory-diagram-e1791449818719.jpeg","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2026\/10\/Storagereview-scality-AI-Inference-Factory-diagram-e1791449818719.jpeg","width":1500,"height":753,"caption":"Scality AI Inference Factory architecture diagram showing client harnesses and agent frameworks above one endpoint, with control, compute, network, and storage planes, separate prefill and decode GPU pools, and the AI Connector on the storage plane"},{"@type":"BreadcrumbList","@id":"https:\/\/www.storagereview.com\/news\/scality-ai-inference-factory-kv-cache-object-storage-rdma#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.storagereview.com\/"},{"@type":"ListItem","position":2,"name":"News","item":"https:\/\/www.storagereview.com\/news"},{"@type":"ListItem","position":3,"name":"Scality AI Inference Factory Serves KV Cache From Object Storage Over RDMA, 14x Faster Than Recompute"}]},{"@type":"WebSite","@id":"https:\/\/www.storagereview.com\/#website","url":"https:\/\/www.storagereview.com\/","name":"StorageReview.com","description":"StorageReview.com is a leading provider of news and reviews throughout the entire IT stack - from the datacenter to the edge, and all points in between.","publisher":{"@id":"https:\/\/www.storagereview.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.storagereview.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.storagereview.com\/#organization","name":"StorageReview.com","url":"https:\/\/www.storagereview.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","width":344,"height":61,"caption":"StorageReview.com"},"image":{"@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/storagereview","http:\/\/youtube.com\/user\/storagereview"]}]} ```