iFANN
    Search iFANN...
    Log in
    Home
    News
    Videos
    Books
    Photos
    GIFs
    Explore
    Polls
    Awards
    iFAMOUS
    Wiki
    Anime
    Rooms
    Notifications
    Messages
    Bookmarks
    Profile
    WikiAwardsiFAMOUSRankingsIndustriesCreator RewardsUser RewardsTermsPrivacyCommunity GuidelinesTakedown / DMCAHelpDevelopers

    © 2026 iFANN

    Home
    Search
    Messages
    Alerts
    Profile

    Post

    Nate
    Nate@nate_512
    🏢Nvidia💭AI💭Tech

    NaceAI Drex v1.5 caching benchmarks

    A 51-page technical report just landed from NaceAI, and the headline number in it is genuinely absurd: a fresh question asked against a cached context of 114k to 117k tokens came back in 21.6 milliseconds. The same request without caching? 1.78 seconds. That's not a small win, that's a different sport. The trick is pointer models. Instead of re-reading the whole document every time you ask something, the system encodes the context one time, holds onto it, and then pulls from that stored version for every follow-up question. Each question still gets probabilities over whatever options you feed it, so you're not losing the actual decision-making part. And the caching advantage compounds. Run 16 questions over 32k tokens and shared prefill gives you an 11, 12x speedup compared to encoding everything over and over. Measurements here were taken model-side on a single NVIDIA B200 GPU, with API gateway latency excluded, so this is raw model throughput we're talking about. The honest tradeoff is accuracy versus efficiency. Max v1 brings higher accuracy to the table but re-encodes context for every single query. Pointer models go the other direction, reusing what's already been processed to keep costs down on long documents. Which one you pick depends on what your workload actually looks like. Full report is 51 pages if you want the deep dive, and free access to Drex is being offered too.

    1w

    14 Likes0 Dislikes2 Reposts0 Comments
    ?

    Comments

    No comments yet. Be the first!

    Post

    Nate
    Nate@nate_512
    🏢Nvidia💭AI💭Tech

    NaceAI Drex v1.5 caching benchmarks

    A 51-page technical report just landed from NaceAI, and the headline number in it is genuinely absurd: a fresh question asked against a cached context of 114k to 117k tokens came back in 21.6 milliseconds. The same request without caching? 1.78 seconds. That's not a small win, that's a different sport. The trick is pointer models. Instead of re-reading the whole document every time you ask something, the system encodes the context one time, holds onto it, and then pulls from that stored version for every follow-up question. Each question still gets probabilities over whatever options you feed it, so you're not losing the actual decision-making part. And the caching advantage compounds. Run 16 questions over 32k tokens and shared prefill gives you an 11, 12x speedup compared to encoding everything over and over. Measurements here were taken model-side on a single NVIDIA B200 GPU, with API gateway latency excluded, so this is raw model throughput we're talking about. The honest tradeoff is accuracy versus efficiency. Max v1 brings higher accuracy to the table but re-encodes context for every single query. Pointer models go the other direction, reusing what's already been processed to keep costs down on long documents. Which one you pick depends on what your workload actually looks like. Full report is 51 pages if you want the deep dive, and free access to Drex is being offered too.

    1w

    14 Likes0 Dislikes2 Reposts0 Comments
    ?

    Comments

    No comments yet. Be the first!