Skip to content

WorthPosting

  • About
  • Resource Links

Tag: vLLM

Cat Links AI News

Continuous Batching: The Scheduling Model Behind Every LLM Serving Stack

Posted on November 10, 2026November 11, 2026

Ask an inference server to answer one question and you get one GPU doing something odd: running a matrix multiplication

Continue readingContinuous Batching: The Scheduling Model Behind Every LLM Serving Stack

Cat Links AI News

Constrained Decoding: How LLMs Get Forced Into Valid JSON

Posted on November 1, 2026

Ask a language model for JSON and most of the time it complies. Every so often it wraps the object

Continue readingConstrained Decoding: How LLMs Get Forced Into Valid JSON

Cat Links AI News

Speculative Decoding: The Math, the Drafters, and When the Bet Pays Off

Posted on October 27, 2026

Every token an LLM emits costs a full forward pass. The GPU reads hundreds of gigabytes of weights, computes one

Continue readingSpeculative Decoding: The Math, the Drafters, and When the Bet Pays Off

Cat Links AI News

The KV Cache: Why LLM Serving Is a Memory Problem

Posted on October 19, 2026

Every LLM serving stack lives or dies by one data structure most developers never see: the KV cache. It is

Continue readingThe KV Cache: Why LLM Serving Is a Memory Problem

Cat Links AI News

Temperature, Top-p, Top-k, and Min-p: How the LLM Sampler Actually Works

Posted on October 9, 2026

Every LLM application has two knobs that get blamed for everything: temperature and top-p. When outputs are boring, someone raises

Continue readingTemperature, Top-p, Top-k, and Min-p: How the LLM Sampler Actually Works

Cat Links AI News

4-Bit by Default: Choosing Between GPTQ, AWQ, GGUF, and MXFP4 in 2026

Posted on October 5, 2026October 6, 2026

Four-bit weights used to be a hobbyist compromise — the thing you did to squeeze a 70B model onto a

Continue reading4-Bit by Default: Choosing Between GPTQ, AWQ, GGUF, and MXFP4 in 2026

Cat Links AI News

DeepSeek-V4.1-Flash: Why the Most Interesting AI Paper This Month Is About Storage

Posted on September 20, 2026

Serving a large language model to thousands of concurrent users is, underneath all the marketing, a memory management problem. Every

Continue readingDeepSeek-V4.1-Flash: Why the Most Interesting AI Paper This Month Is About Storage

Cat Links AI News

Speculative Decoding Explained: How EAGLE-3 Makes LLMs 2-3x Faster Without Changing Outputs

Posted on September 17, 2026September 17, 2026

Autoregressive decoding is the reason large language models feel slow: every token is generated by a full forward pass, and

Continue readingSpeculative Decoding Explained: How EAGLE-3 Makes LLMs 2-3x Faster Without Changing Outputs

Cat Links AI News

PagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving

Posted on September 13, 2026September 14, 2026

When an LLM serves a 2,000-token response to a prompt of 10,000 tokens, it does something that looks absurd from

Continue readingPagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving

Cat Links AI News

LLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins

Posted on September 10, 2026September 11, 2026

Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB —

Continue readingLLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins

Posts navigation

Older posts
  • Home
  • About
Copyright © 2026 WorthPosting | Signify by WEN Themes
Scroll Up