Choosing a Host for an AI Assistant on a $1,399 Budget

I have an AI assistant that runs 24/7, and it needs a dedicated host.

My budget was $1,399, though the cheaper entry-level options deserved a fair look too. Five routes made the shortlist: a $799 Mac Mini M4, a $249 Jetson Orin Nano 8GB, a $1,399 Mac Mini M4 Pro, a Mini PC with a discrete NVIDIA GPU, and an NVIDIA Jetson AGX Orin 64GB.

This is the decision process, including why I eventually took a completely different route.

Five routes

Before comparing machines near the full budget, it helps to look at two entry-level options. They cost far less than $1,399 and may already be enough when the workload is a small local model rather than a general-purpose AI server.

Base Mac Mini M4 (16GB unified memory)

$799 from Apple. The base configuration has 16GB of unified memory, a 10-core CPU, and a 10-core GPU.

It is a strong all-around machine at this price:

16GB of unified memory is enough for small models. Quantized 7B–13B models can run smoothly. Unified memory means the GPU can read from the same memory pool as the CPU instead of relying on a separate, much smaller VRAM pool.
Power efficiency is excellent. The whole machine draws under 20W and is even quieter than the Jetson.
It works out of the box. With macOS, Homebrew, and MLX, llama.cpp can be running within half an hour. MLX is Apple's framework for machine learning on Apple silicon, so it takes advantage of the hardware without a CUDA setup.

It also inherits the same hard limits as the M4 Pro version:

There is no CUDA. The mainstream AI toolchain still assumes NVIDIA hardware often enough that compatibility becomes a recurring problem.
16GB is a hard ceiling. A 34B+ model is basically out of reach, and the memory cannot be upgraded later.
The GPU is small. A 10-core GPU has limited inference throughput. It is suitable for one person's local assistant, but much less convincing as a service host.

If the job is to run a 7B model quietly at low power for one user, the $799 Mac Mini is reasonable. If model size matters, the ceiling arrives quickly, and no later GPU or memory upgrade can move it.

Jetson Orin Nano 8GB

~$249. This is the entry-level board in NVIDIA's embedded AI platform.

8GB unified memory plus CUDA. It can run a quantized 7B model, and native CUDA gives it far better toolchain compatibility than a Mac.
1,024 CUDA cores. That is modest GPU capacity, but enough for small-model inference.
Very low power draw (7–15W). Electricity cost is almost irrelevant for a machine that runs 24/7.
The lowest price. It puts a real CUDA device on the desk for under $250.

The constraints are just as clear:

8GB is painfully tight. The operating system takes part of it, leaving perhaps 5–6GB for the model and runtime. A 13B model is already a squeeze; anything larger will not fit.
The CPU is very weak. Its 6-core ARM A78AE delivers roughly 1/3 of the single-core performance of a desktop x86 processor. Compilation and data processing feel slow even when the GPU side is adequate.
The old ARM ecosystem problem remains. Many packages have no prebuilt ARM64 release, so compiling dependencies and working around packaging gaps becomes routine.

My assistant originally ran on an Orin Nano. It does work. The gap between "works" and "pleasant to operate" is substantial once the assistant starts doing compilation, automation, and data processing alongside inference.

Mac Mini M4 Pro (24GB unified memory)

About $1,399. Apple's unified-memory design is friendly to large-language-model inference: the GPU cores can access all 24GB directly, without moving model data across a PCIe link. MLX optimization is also improving quickly.

The problems are hard to ignore:

There is no CUDA. Most of the AI ecosystem is built first around NVIDIA. PyTorch, vLLM, and llama.cpp all have mature CUDA paths; on a Mac, a tool may be unsupported or require a separate backend and extra adaptation.
macOS adds ecosystem friction. Many tools assume Linux first. Docker can run Linux containers on a Mac, but virtualization adds overhead and another compatibility layer.
There is only 24GB. Unified memory is flexible, but it cannot invent capacity. A quantized 70B model would be very tight.

This is a good daily development computer. As a dedicated inference server, the ecosystem mismatch is a serious drawback, especially when the goal is to try tools written with Linux and CUDA as their default environment.

Mini PC + RTX 4060/4070

In the $1,200–$1,500 range, I could build an i5 or Ryzen 5 machine with 32–64GB of system memory and an RTX 4060 Ti 16GB.

The advantages are refreshingly direct:

x86 + Linux + CUDA has the best compatibility. Almost every AI project expects this combination, so installation instructions tend to work without translation.
It is upgradeable. System memory and the GPU can both be replaced instead of becoming permanent purchase-time decisions.
The server ecosystem is complete. Docker, systemd, and SSH behave exactly as standard Linux documentation says they should.

The drawbacks are equally direct:

VRAM becomes the bottleneck. An RTX 4060 has only 8GB, and even the 4060 Ti has only 16GB. Larger models need CPU offload, where part of the model sits in ordinary system memory and crosses the PCIe bus during inference. Performance falls off a cliff.
Power and size are higher. This machine is much larger than a Jetson and consumes much more electricity.
VRAM and system memory are separate. The large system-memory number cannot be allocated as flexibly as a unified-memory pool. A machine with 64GB of RAM still has a 16GB GPU limit for fast inference.

This is the most pragmatic route. Within a $1,400 budget, though, the shortage of VRAM remains the pain point that no amount of ordinary RAM fixes.

Jetson AGX Orin 64GB

$1,399, the flagship of NVIDIA's embedded AI platform.

Its central appeal is 64GB of unified memory with CUDA support:

64GB of unified memory can hold quantized 34B and even 70B models.
Native CUDA means llama.cpp and PyTorch both run.
Power draw is very low (15–60W), which makes 24/7 operation easy to justify.
The machine is small and quiet.

That combination is unusual. Consumer NVIDIA cards offer CUDA and speed but attach it to a much smaller fixed pool of VRAM. The AGX Orin offers a large shared pool while keeping the software interface that AI projects expect.

ARM still creates plenty of friction:

Software compatibility. JetPack is based on Ubuntu, but many packages do not publish ARM64 binaries. Building them yourself is common rather than exceptional.
CPU performance is limited. ARM A78 single-core performance trails x86 by a visible margin, so non-GPU tasks such as compilation and data processing are slow.
Community resources are sparse. When something fails, the answer on Stack Overflow is probably written for x86 and may assume packages or binaries that do not exist on the Jetson.

Jetson is designed as an edge AI platform. It can serve as a personal server, but the job comes with a steady supply of unexpected ARM-specific tinkering.

Comparison

	Mac Mini M4	Mac Mini M4 Pro	Mini PC + 4060 Ti	Orin Nano 8GB	AGX Orin 64GB
Price	$799	~$1,399	~$1,300–1,500	~$249	$1,399
GPU memory	16GB (unified)	24GB (unified)	16GB (discrete)	8GB (unified)	64GB (unified)
GPU cores	10-core Apple GPU	18-core Apple GPU	4,352 CUDA	1,024 CUDA	2,048 CUDA
CUDA	❌	❌	✅	✅	✅
CPU performance	High	High	High	Low	Medium
Power draw	Very low	Low	High	Very low	Very low
Software compatibility	macOS limits	macOS limits	Best	ARM limits	ARM limits
Upgradeable	❌	❌	✅	❌	❌
Largest practical model	~13B	~20B	~13B (VRAM)	~7B	~70B

These five options span $249 to $1,500, and no machine wins every column. The choice changes depending on whether the binding constraint is price, model capacity, software compatibility, power, or the amount of time available for ARM debugging.

$249 entry point: The Orin Nano 8GB is the cheapest CUDA device. It runs 7B models and suits learning or lightweight deployment.
$799 balanced option: The Mac Mini M4 has the strongest overall package, with the lack of CUDA as its hard limitation.
$1,400 with inference first: The Jetson AGX Orin 64GB has the largest unified-memory pool and can run a 70B model.
$1,400 with ecosystem first: A Mini PC with a discrete GPU is the least troublesome and has the best software compatibility.
$1,400 in Apple's ecosystem: The Mac Mini M4 Pro fits someone already committed to macOS and MLX.

What actually happened

I bought none of them, although I did use the Orin Nano for a while.

I already had an unused PC: Ryzen 5 5600 + 64GB DDR4 + RTX 4090 24GB. It had been a gaming machine and was now collecting dust. Reusing it changed the comparison because its purchase price for this project was zero.

The RTX 4090 overwhelms every option above on raw compute:

	RTX 4090 host	Jetson AGX Orin 64GB
GPU cores	16,384 CUDA cores	2,048 CUDA cores
GPU memory	24GB GDDR6X	64GB (unified)
FP16 performance	83 TFLOPS	~5.3 TFLOPS
System memory	64GB DDR4	64GB (unified)
CPU	Ryzen 5 5600 (6C12T)	ARM A78 (12C)
Price	Already owned	$1,399

The GPU compute difference is roughly 16x. In actual inference tasks, the 4090 is 5–10x faster in most cases. The smaller real-world multiple makes sense because inference is constrained by more than the advertised arithmetic rate: model size, memory movement, runtime overhead, and CPU work all participate.

The 4090 still has only 24GB of VRAM, so the AGX Orin retains one genuine advantage: it can fit models that the 4090 cannot keep entirely in GPU memory. For the models and services I actually planned to run, speed and x86 compatibility mattered more than fitting the largest possible quantized model.

The saved $1,399 can pay the electricity bill for several years.

Later benchmark results

After installing the system and bringing the services online, I ran a set of practical benchmarks:

CPU multi-threading (sysbench): about 6.8x faster than the Jetson Orin Nano
Sequential disk reads: about 15.4 GB/s, a 10x improvement
Video encoding (NVENC H.264, 1080p): 253 fps, about 2.5x faster
AI inference: the same speech-recognition model processes data about 5x faster

A necessary note on the comparison: the baseline here is the Jetson Orin Nano 8GB at about $250, not the AGX Orin 64GB. The gap against the AGX Orin would be smaller, although the difference in CUDA-core count is still 8x. These measurements describe the migration I actually made; they should not be read as a direct benchmark of the $1,399 AGX Orin.

The spread between the numbers also matters. A 16x FP16 specification gap does not turn every workload into a 16x speedup. The measured results range from about 2.5x for NVENC video encoding to about 6.8x for CPU multi-threading, with AI inference at about 5x. A spec-sheet peak is useful for describing the hardware, but it is a poor promise for a whole application.

Conclusion

If you have a $1,400 budget and are seriously choosing a host for AI workloads, I would work through the decision in this order:

Check for idle hardware first. An old gaming PC with a discrete GPU may be far more capable than expected, and its effective purchase price is already paid.
Treat VRAM as the primary bottleneck. It determines how large a model can stay on the GPU, which often matters more than the amount of ordinary system RAM.
Value CUDA compatibility. Unless there is a specific reason to choose Mac or ARM, x86 + NVIDIA is the path with the least software friction.
Count the ecosystem cost of Jetson. Its unified memory is tempting, but time spent compiling and debugging ARM dependencies is also a cost.

There is no standard hardware answer. The result depends on what you already own, what has to run, and how much platform-specific tinkering you are willing to absorb.

For me, the answer happened to be a gaming PC already gathering dust in the corner.