Project context and disclosure boundary
The customer needed private LLM inference inside its own data center, keeping models, traffic, and operational logs within the enterprise network. The site had no dependency on public internet access, so model weights, Python packages, system installers, and container images had to be prepared and transferred offline.
To protect confidentiality, this case does not disclose the customer name, domain, personnel, server addresses, or internal security configuration. Only the anonymized architecture, delivery method, and test conclusions are presented.
Delivery challenges
Sixteen H200 GPUs were split across two inference nodes, requiring IB/RoCE validation and careful control of cross-node overhead.
Weights, packages, and images had to be prepared externally, transferred by offline media, and backed by integrity evidence.
Available memory could not host every target model simultaneously, so the design required one model online at a time, on-demand switching, and memory reclamation.
The deliverable needed more than a running model: unified APIs, process supervision, recovery on boot, and repeatable test tooling were required.
Production architecture
- vLLM runs directly on the host to avoid unnecessary inference overhead.
- Ray provides distributed scheduling across the two servers for GLM-5.2-FP8.
- LiteLLM exposes a consistent OpenAI-compatible endpoint and isolates upstream clients from model switching.
- Hermes Agent uses centralized deployment and separate user workspaces for controlled access.
Offline delivery and operations
Hardware and fabric validation
Validated GPUs, drivers, PCIe topology, IB/RoCE connectivity, and inter-node bandwidth.
Weight integrity
Calculated SHA256 checksums before and after transfer to verify offline migration integrity.
Service supervision
Used systemd for standardized start, stop, boot recovery, and automatic restart of Ray and vLLM.
Delivery evidence
Produced architecture, startup configuration, functional test evidence, and on-site operating instructions.
Functional verification
All six executed functional tests passed with no failures. An authorized fault-injection test remained disabled and is therefore excluded from this conclusion.
- The model-list endpoint returned the intended model.
- Non-streaming responses and completion states were correct.
- SSE events were complete, ordered, and terminated normally.
- Multi-turn context was retained without cross-session interference.
- Four concurrent streams completed without request-ID confusion.
These results represent functional verification, not final performance acceptance. Throughput, long-context behavior, concurrency tuning, and customer UAT remain subject to the formal acceptance report.
Planning a multi-node private LLM cluster?
We size the model, context, concurrency, network, and security boundaries before defining on-site delivery and acceptance criteria.