Gemma 4 meets NVIDIA GPUs
Open source AI models are moving from the cloud onto our own hardware, and that is great news for PC enthusiasts and power users. Google’s latest Gemma 4 models are designed to run efficiently on local devices, and NVIDIA has stepped in to optimize them for its GPUs, from GeForce RTX gaming PCs to DGX Spark personal AI supercomputers and Jetson edge modules.
The Gemma 4 family includes several compact models that focus on performance and efficiency. NVIDIA and Google have worked together so these models can make full use of Tensor Cores and the CUDA software stack, delivering low latency and higher throughput on RTX graphics cards and NVIDIA workstations.
Benchmarks using Q4 K M quantization on an NVIDIA GeForce RTX 5090 desktop show that Gemma 4 can generate tokens quickly with small batch sizes and long prompts. This makes the models a strong fit for real time AI assistants, coding help and on device agents that respond instantly without needing constant cloud access.
Gemma 4 model lineup and what they do
The Gemma 4 family covers several sizes aimed at different use cases and hardware levels.
- E2B and E4B: Ultra efficient models built for low latency at the edge. They can run fully offline on many devices, including Jetson Orin Nano modules, making them ideal for small form factor systems and embedded projects.
- 26B and 31B: Larger models tuned for stronger reasoning and developer workflows. These are designed for high performance RTX GPUs, workstations and DGX Spark systems, where more VRAM and compute are available.
Across these sizes, Gemma 4 supports a wide set of capabilities that are increasingly important for local AI workloads:
- Reasoning for complex problem solving and more advanced decision making.
- Coding for generating code, refactoring and debugging in developer toolchains.
- Agents and function calling for building structured tool use and workflow automation.
- Multimodal support for vision, video and audio, including object recognition, document and video intelligence, and automated speech recognition.
- Interleaved multimodal input so text and images can be mixed freely inside a single prompt.
- Multilingual support with pretrained coverage across more than thirty five languages and exposure to over one hundred forty languages during training.
The idea is to give users open models that are small enough to run locally yet capable enough to handle real workloads, from personal assistants to development tools.
Local agents on RTX PCs and DGX Spark
One of the biggest trends behind these optimizations is local agentic AI. Instead of sending everything to cloud models, users can run AI agents directly on their RTX desktop or workstation that understand personal files, apps and workflows.
OpenClaw is an example of this direction. It enables always on AI assistants that live on your PC or DGX Spark system. The latest Gemma 4 models are compatible with OpenClaw, so you can assemble agents that:
- Index and search local documents and notes.
- Automate repetitive desktop tasks or development workflows.
- Combine local context with AI reasoning without exposing private data to external servers.
NVIDIA has published guides for running OpenClaw on RTX GPUs and DGX Spark, as well as a dedicated DGX Spark OpenClaw playbook to help users get started quickly.
How to run Gemma 4 on your GPU
NVIDIA has worked with several popular tools so users can deploy Gemma 4 locally with minimal setup.
- Ollama: A simple way to download and run Gemma 4 models on your PC. You pull the model and then interact with it through a local interface, with all inference happening on your RTX GPU.
- llama.cpp with GGUF checkpoints: For users who prefer a more bare bones setup, you can install llama.cpp and pair it with Gemma 4 GGUF checkpoints from Hugging Face. This gives you a light, highly optimized C plus plus runtime that works well for local inference.
- Unsloth Studio: Offers day one support for Gemma 4 with optimized and quantized variants for fine tuning and deployment. If you want to adapt a Gemma 4 model to your own data or workflows, Unsloth lets you do this efficiently on your local GPU.
By running Gemma 4 on NVIDIA GPUs you benefit from Tensor Cores that accelerate transformer workloads and from the mature CUDA ecosystem that already supports leading frameworks and inference engines. The same stack scales from Jetson Orin Nano at the edge up through RTX gaming PCs to full DGX Spark setups, which means you do not need to completely rework your deployment when you move between systems.
More RTX AI PC updates
Gemma 4 is part of a wider wave of AI improvements around RTX PCs. NVIDIA has been highlighting more open models for local agents, including Nemotron 3 Nano 4B and Nemotron 3 Super 120B, as well as optimizations for Qwen 3.5 and Mistral Small 4.
NVIDIA also introduced NemoClaw, an open source stack that tunes and secures OpenClaw experiences on NVIDIA hardware. It is aimed at making local model execution safer and more performant across devices.
On the desktop side, Accomplish FREE is a no cost AI agent for PCs that comes with built in open weight models and uses NVIDIA GPUs for local inference. A hybrid router can balance workloads between your RTX hardware and the cloud, giving you speed and privacy while keeping configuration simple and avoiding the need for API keys.
Together these updates show how quickly local AI is becoming a real workload for gaming rigs, creator PCs and workstations. If you are already running a GeForce RTX card, you now have access to a growing ecosystem of tools and models like Gemma 4 that turn your PC into a powerful on device AI platform.
Original article and image: https://blogs.nvidia.com/blog/rtx-ai-garage-open-models-google-gemma-4/
