All projects
Infrastructure
Internal system
GPU servers for LLMs
8× NVIDIA B200 and 8× RTX PRO 6000 Blackwell: language models, image and video generation, speech
- My role
- DevOps / MLOps
- Period
- 2026
I brought into production and run two GPU servers behind the AI features of our products. The HGX node with 8× NVIDIA B200 serves large language and multimodal models on SGLang behind an LLM gateway with keys and budgets. The 8× RTX PRO 6000 Blackwell node (768 GB of VRAM, 1 TB of RAM) runs vLLM, ComfyUI for image, video and lip-sync generation, speech recognition and synthesis. Configs and runbooks live in git.
What I did
- Model evaluation (GLM, Qwen, DeepSeek, Kimi, MiniMax) and benchmarks
- An LLM gateway with strict routing, virtual keys and budgets
- Incident runbooks: driver reloads, CUDA graph crashes, a warranty GPU tray replacement
- OS installs through the BMC and virtual media — no physical access needed
- These models power the ticketing AI assistant, the Zamon translations and the EnglishKids helper