Building a Two-Node DGX Spark Cluster at Home

I already had useful local AI hardware before buying the Sparks. My desktop has an RTX 5090 and is very good for models that fit comfortably on a single desktop GPU. I also have an RTX 3090 system handling supporting inference workloads such as embeddings, reranking, speech-to-text and text-to-speech.

What I did not have was a good way to experiment with some of the much larger open models and long-context serving recipes I had become interested in. That was the reason for buying two Sparks rather than treating one as a standalone appliance.

Each system has an NVIDIA GB10 with 128 GB of unified memory. Together they gave me a compact platform for experimenting with model-serving recipes that can split work across two nodes.

I was also interested in the systems themselves. I wanted to learn how the distributed side actually worked: the direct fabric, RoCE, NCCL, tensor parallelism, model placement, KV cache behavior, speculative decoding — and what happens when a recipe that looks good on paper meets real hardware.

This project is still active. The initial cluster build is finished, but most of the interesting work now is testing and comparing model recipes on top of it.

The hardware

The cluster consists of:

• 2× ASUS Ascent GX10 / NVIDIA DGX Spark-class systems
• NVIDIA GB10 in each node
• 128 GB unified memory per node
• approximately 121 GiB usable memory per node in my early system checks
• 1 TB internal SSD per node
• node names: spark1 and spark2
• direct ConnectX-7 / RoCE connection between the nodes

The systems were purchased as a two-pack with the direct interconnect cable.

My ordinary homelab network was only 2.5 GbE when I built this. That matters because I did not want tensor-parallel traffic crossing the same 2.5 GbE network used by the rest of the house.

The two Sparks therefore have two conceptually different network paths:

1. Normal LAN connectivity for management, APIs, file access and the rest of the homelab.
2. A dedicated direct ConnectX-7/RoCE path for high-bandwidth communication between spark1 and spark2.

Initial software environment

The early cluster configuration I documented was:

• DGX OS 7.5.0
• Ubuntu 24.04.4
• kernel 6.17.0-1031-nvidia
• NVIDIA driver 580.173.02
• CUDA 13.0-era stack

These describe my systems during the initial build. They are not a claim about what NVIDIA ships in general.

Bringing up the cluster

I deliberately tried to validate the system layer by layer instead of jumping straight into a giant model. The rough order was:

1. Bring up both systems and finish the initial DGX OS setup.
2. Verify the GPU, memory, storage, driver and CUDA environment.
3. Name the machines spark1 and spark2.
4. Establish SSH communication needed by distributed launch scripts.
5. Bring up the direct ConnectX-7 link.
6. Configure the RoCE/NCCL path.
7. Test the fabric independently of any particular model.
8. Confirm NCCL was actually using the intended interfaces.
9. Only then start qualifying distributed model-serving recipes.

That separation became important later.

When a distributed model fails, there are a lot of possible causes. It might be the model, runtime, container, launcher, SSH configuration, NCCL settings or the actual network. Having a known-good fabric removes several of those variables.

Direct fabric results

These are measurements from my own two systems, taken over the direct ConnectX-7/RoCE path between spark1 and spark2 — not generic NVLink figures.

The direct Spark-to-Spark fabric test reported approximately 185.2 Gb/s of bandwidth and around 1.8 microseconds of latency. NCCL all_gather measured approximately 22.84 GB/s, equivalent to roughly 182.7 Gb/s, and NCCL logs showed communication using NET/IB over the RoCE interfaces.

Those results mattered because they demonstrated that distributed model traffic was actually using the high-speed direct path instead of falling back to my normal 2.5 GbE LAN.

Early problems

The setup was not completely smooth.

One of the early distributed-launch preflight checks failed because SSH authentication between the nodes was not set up the way the launcher expected. I also ran into an NCCL/OpenMPI test path that failed with undefined C++ symbols. That initially made the networking work more confusing than it needed to be.

The useful lesson was that one broken benchmark binary is not proof that the fabric itself is broken. Once I tested the underlying path separately and looked at what NCCL was actually selecting, I had much better evidence about what was working.

From a working fabric to a working model

One of the first substantial deployments I qualified on the pair was DeepSeek-V4-Flash-0731. The deployment used:

• both GB10 nodes
• tensor parallel size 2
• the direct RoCE fabric
• approximately 155 GiB of model/checkpoint data
• 1,048,576-token configured maximum model length
• an FP4 / NVFP4 expert execution path
• nvfp4_ds_mla KV-cache path
• DSpark K3 probabilistic speculative decoding
• an OpenAI-compatible serving endpoint

Getting the model onto both machines was part of the process too. One recorded synchronization from spark1 to spark2 transferred 166,898,692,570 bytes in 5 minutes 28.464 seconds, at approximately 485.09 MB/s.

That number is file-transfer/staging performance. It is not inference throughput.

Selected DeepSeek measurements

These are a few representative measurements from the dual-Spark DeepSeek-V4-Flash-0731 deployment. The detailed benchmark work will become separate Lab Entries.

Code workloads of roughly 8K tokens showed a median decode speed of 45.73 tok/s. On a cold roughly 131K-token prompt, prefill ran at 1,435.24 tok/s, time to first token was 91.53 seconds, and decode ran at 42.37 tok/s.

1,435.24 tok/s is prefill throughput. 1

An early target of roughly 70+ tok/s came from community reports I was trying to reproduce. That is a target, not a result I am claiming for my own machines.

What the cluster has been used for

After the initial bring-up, I started treating the Sparks as an experimentation platform rather than a single-model appliance.

Model families and recipes I have worked with on the pair include:

• DeepSeek-V4-Flash-0731
• GLM-5.3-Flash
• newer GLM-5.3-Flash recipe candidates
• Qwen 3.8 Flash Next

The cluster has given me a place to test:

• multi-node tensor parallelism
• RoCE and NCCL behavior
• very large model serving
• long-context inference
• different quantization approaches
• quantized KV caches
• speculative decoding
• prompt/prefill performance
• generation/decode performance
• time to first token
• concurrent serving
• model qualification and regression testing
• OpenAI-compatible endpoints
• routing models into the rest of my self-hosted AI stack

Not every recipe worked well. Some were slower than expected. Some crashed. Some behaved well at short context and poorly at long context. Some loaded successfully but were not useful enough to replace an older configuration.

That became one of the more important parts of the project. A model loading is not the same thing as a model being qualified.

Keeping working recipes around

One practice I adopted fairly quickly was to stop replacing a known-good deployment just because a newer recipe appeared.

For example, when testing newer GLM-5.3-Flash configurations, I kept the existing working deployment available and built the candidate beside it. That takes more storage and a little more organization, but it makes testing much safer.

A candidate can fail its startup test, crash during a long prompt or simply perform worse without taking away the configuration I already know works. This has become part of the way I operate the cluster rather than something specific to one model.

How the Sparks fit into the rest of my local AI setup

The Sparks do not replace my RTX 5090 or RTX 3090. They solve a different problem.

The RTX 5090 workstation is still a very useful machine for development and fast single-node inference when the model fits well there. The RTX 3090 system handles supporting inference roles including embeddings, reranking, speech-to-text and text-to-speech.

The dual Sparks are where I go when I want:

• substantially more memory
• a model intended for two-node execution
• large context
• distributed inference experiments
• model-serving recipes involving tensor parallelism and high-speed inter-node communication

I prefer this split to trying to force one GPU or machine to do every AI job in the homelab.

What worked

The decisions that have held up well so far:

• separating distributed inference traffic from my normal 2.5 GbE LAN
• validating the ConnectX-7/RoCE path before debugging model runtimes
• checking that NCCL really uses the intended interfaces
• keeping stable and candidate model recipes separate
• exposing models through OpenAI-compatible APIs
• treating prefill, decode, TTFT and concurrency as different measurements
• using the Sparks for workloads that benefit from their memory and topology instead of assuming they must replace faster desktop hardware

What didn't

A few things have been useful specifically because they failed:

• SSH/preflight trouble during the first distributed-launch setup
• the broken NCCL/OpenMPI test path
• assuming community benchmark numbers would reproduce automatically
• treating a successful model load as evidence that the whole recipe worked
• later model/runtime candidates that crashed or behaved badly under certain workloads
• model configurations that looked promising but were not obviously better once tested

None of this makes the project a disaster story — or a spotless success story either. The failures are useful because they changed how I test new deployments.

What I learned

The network deserves its own test. 1

Keep the known-good version. 1

"Running" is a very low bar. 1

Tokens per second needs context. 1

Community results are targets, not my benchmark results. 1

And finally, the heterogeneous setup makes more sense for my use than trying to crown one machine the winner. The 5090, 3090 and Sparks each ended up with jobs they are good at.

Current state

The cluster is now useful enough that most of my work has shifted from “Can I make these two machines talk to each other?” to “Which model recipe is actually worth keeping?”

The direct fabric is working, distributed serving is established, and the pair is integrated into the rest of my local AI environment. I still keep known-good deployments while testing replacements. That is a little slower than constantly swapping to the newest recipe, but a failed experiment no longer takes the useful configuration with it.

The project remains in progress. The detailed model deployments, context-window tests, benchmark comparisons, speculative-decoding experiments and failures will be documented as separate Lab Entries rather than making this page enormous.