Guangba Yu.
中文 EN
Ideas in practice

Tech Notes

Technical ideas, implementations, and lessons learned.

Notes are prepared with AI assistance; the original research and views belong to their respective authors.

Filters

    Latest note45 NOTES

    2026-10-22 | Flint on vLLM: lowering single-GPU long-context decode latency

    • Inference Serving

    Recursive Flint’s local/global attention, NVFP4 loading and small-batch kernels, separating model-forward latency from end-to-end speed.

    Read note ↗

    The archive

    October 202645 NOTES
    Oct 22, 20262026-10-22 | Flint on vLLM: lowering single-GPU long-context decode latency
    • Inference Serving
    Oct 22, 20262026-10-22 | Modal observability: connect multi-cloud incidents into one investigation
    • Observability & Diagnosis
    • Systems Reliability
    Oct 22, 20262026-10-22 | EAGLE-4: adapt the drafting budget to hardware and load
    • Inference Serving
    Oct 21, 20262026-10-21 | Humming: Unifying Quantized Kernels Without Erasing Their Differences
    • Inference Serving
    Oct 21, 20262026-10-21 | Autoscaling Recovery: Measuring the Wait with MTSS
    • Observability & Diagnosis
    • Systems Reliability
    Oct 21, 20262026-10-21 | vLLM TorchTPU: Reusing the Serving Control Plane
    • Inference Serving
    • Systems Reliability
    Oct 21, 20262026-10-21 | TorchTPU: Compilation and Communication Behind Native PyTorch APIs
    • Inference Serving
    • Systems Reliability
    Oct 21, 20262026-10-21 | Thin Orchestrator: Moving Domain Reasoning Out of the Router
    • Agent Reliability
    • Coding Agents
    • Multi-Agent Systems
    Oct 21, 20262026-10-21 | Prometheus: Making Large-Cluster Failures Recoverable
    • Observability & Diagnosis
    • Systems Reliability
    Oct 21, 20262026-10-21 | LocalPay: Making Payment Failures Repeatable Locally
    • Observability & Diagnosis
    • Systems Reliability
    Oct 20, 20262026-10-20 | LMCache 0.5.6: Trust and Lifecycle Boundaries in KV Cache Services
    • Inference Serving
    • Systems Reliability
    Oct 20, 20262026-10-20 | Unattended Agents: Seven Failure Scenarios to Validate Before Release
    • Agent Reliability
    • Observability & Diagnosis
    • Systems Reliability
    Oct 20, 20262026-10-20 | Agent Skills: Does Following Instructions Actually Help?
    • Agent Reliability
    • Coding Agents
    • Self-Improvement
    Oct 20, 20262026-10-20 | TorchNeuron: Native PyTorch Integration and Its Compatibility Boundaries
    • Observability & Diagnosis
    • Inference Serving
    • Systems Reliability
    Oct 20, 20262026-10-20 | MiniMax M3.1 Flash: Evaluating Low Prices Across Complete Agent Tasks
    • Agent Reliability
    • Coding Agents
    • Inference Serving
    Oct 19, 20262026-10-19 | Jev Diagnosis: Evidence Defines the Limits of Inexpensive Judgments
    • Agent Reliability
    • Observability & Diagnosis
    • Systems Reliability
    Oct 19, 20262026-10-19 | Helion and CompileIQ: Adding Compiler Controls to Kernel Tuning
    • Inference Serving
    Oct 19, 20262026-10-19 | Elastic EP: Restricting Scaling Pauses to Topology Handover
    • Inference Serving
    • Systems Reliability
    Oct 19, 20262026-10-19 | Cloudflare Review Agents: Checking Finding Quality After Cutting Costs
    • Agent Reliability
    • Coding Agents
    • Inference Serving
    Oct 19, 20262026-10-19 | From Microservices to Agents: Making Decision Quality a Reliability Concern
    • Agent Reliability
    • Multi-Agent Systems
    • Observability & Diagnosis
    • Systems Reliability
    Oct 17, 20262026-10-17 | DeepSeek Search/Fetch: Connecting Web Evidence to Agents
    • Agent Reliability
    • Inference Serving
    Oct 16, 20262026-10-16 | Zephon: Preserving Training Batches When GPU Counts Change
    • Systems Reliability
    Oct 16, 20262026-10-16 | LMCache MP Coordinator: Finding Caches Across Servers
    • Inference Serving
    • Systems Reliability
    Oct 16, 20262026-10-16 | Kimi K3 on MI355X: From State Reuse to Kernel Scheduling
    • Inference Serving
    • Systems Reliability
    Oct 16, 20262026-10-16 | Specialized Kimi K3 Decode: Speed and Validation Boundaries
    • Inference Serving
    • Systems Reliability
    Oct 15, 20262026-10-15 | Scout: Post-Training for Evidence-Based SRE Investigations
    • Agent Reliability
    • Observability & Diagnosis
    Oct 14, 20262026-10-14 | Bitwise Training–Inference Consistency: Keep Low Precision, Fix the Computation Path
    • Self-Improvement
    • Inference Serving
    Oct 14, 20262026-10-14 | NIXL Push: Move KV-Transfer Setup Off the First-Token Path
    • Inference Serving
    Oct 14, 20262026-10-14 | Claude Agent Observability: Bring Execution Into Existing OTel Systems
    • Agent Reliability
    • Observability & Diagnosis
    Oct 14, 20262026-10-14 | Online Verification Budgets Without a Confidence Head
    • Inference Serving
    Oct 13, 20262026-10-13 | Scion: How coding agents change source-control scaling
    • Coding Agents
    • Systems Reliability
    Oct 12, 20262026-10-12 | Building an AI SRE: The Engineering Beyond an Answer
    • Agent Reliability
    • Observability & Diagnosis
    Oct 12, 20262026-10-12 | Bidirectional KV Transfer in vLLM: Returning Multi-Turn Context to Prefill
    • Inference Serving
    Oct 9, 20262026-10-09 | vLLM and Vera Rubin: Turning Memory Locality into Serving Throughput
    • Inference Serving
    Oct 8, 20262026-10-08 | Dynamo: Scheduling Agent Inference and KV Cache by Session
    • Observability & Diagnosis
    • Inference Serving
    Oct 8, 20262026-10-08 | Google AQuA: Finding Quality Issues in Production Traces
    • Agent Reliability
    • Observability & Diagnosis
    Oct 8, 20262026-10-08 | Anyscale: Caching, Routing, and the Cost of Coding Agents
    • Coding Agents
    • Inference Serving
    Oct 7, 20262026-10-07 | DeepSeek-V4.1-Flash: Replay and Kernel Optimizations Behind Fivefold Throughput
    • Inference Serving
    Oct 7, 20262026-10-07 | PAI: What Agent Monitoring Needs from Telemetry
    • Agent Reliability
    • Observability & Diagnosis
    Oct 6, 20262026-10-06 | FBTriton: Organizing Embedding Training Around Sparse Access Patterns
    • Systems Reliability
    Oct 3, 20262026-10-03 | GPU Cluster Readiness: Validate the Workload, Not Only the Components
    • Observability & Diagnosis
    • Systems Reliability
    Oct 2, 20262026-10-02 | Helion Linear Backend: Make Algorithm Selection Tunable
    • Inference Serving
    Oct 1, 20262026-10-01 | Incident Replay: Can the Agent See Enough Evidence?
    • Agent Reliability
    • Observability & Diagnosis
    Oct 1, 20262026-10-01 | TLX Jagged Flash Attention: Keeping Blackwell Busy on Variable-Length Sequences
    • Inference Serving
    Oct 1, 20262026-10-01 | KV Cache and LMCache: From Generation to Reusable Multi-Turn State
    • Inference Serving