Architecture
NuFi is an NPUOps platform that runs on top of a Kubernetes cluster. Core workloads are managed through Kubernetes resources and NuFi CRDs.
NuFi System Composition
Main Components
| Component | Role |
|---|---|
| Dashboard | Web UI — the frontend users interact with directly |
| API Server | Handles REST API, creates/manages Custom Resources, directly creates Lab workloads (Kubeflow Notebook / File Manager Pod), and manages Optimizer execution flows |
| NuFi Controller (K8s Operator) | Watches NuFi CRDs and provisions K8s resources such as Serving Pods and various Jobs |
| nufi-proxy | Reverse proxy deployed alongside each Serving (separate Deployment/Service). Client requests pass through a VirtualService to nufi-proxy, which then forwards them to the inference server. Handles the Async Queue, Transformer pre/post-processing chaining, and metrics collection. Load balancing is handled by the Istio DestinationRule that NuFi Controller creates from NpuDeploy settings, and temperature-based traffic shutoff by the Temperature Sidecar. |
| nufi-notebook-servers | Jupyter / VS Code / LlamaFactory container images for Labs (development environments) |
| nufi-file-manager | File manager for uploading, downloading, and managing files in a Volume |
Custom Resources (CRDs) and Outputs
The CRDs managed by NuFi Controller and the workloads each produces:
| CRD | Output Workload |
|---|---|
| NpuDeploy | Serving (Inference Server + nufi-proxy + Transformer + Temperature Sidecar + Service + VirtualService) |
| ModelImport | Model Import Job (Hugging Face / MLflow → Registry) |
| NpuPortingPipeline | NPU Compile Job (per-device compile) |
| DatasetImportRun | Dataset Import Job (Upload / Hugging Face Import) |
| DatasetProfileRun | Dataset Profile Job (Parquet profile generation) |
| EvaluationRun | Evaluation Job (lm-eval, etc.) |
Lab (Notebook Server, File Manager) is created directly by the API Server, not by the NuFi Controller. Notebook is provisioned through the Kubeflow Notebook CR, and File Manager is provisioned as a Pod created directly by the API Server.
Compile Option Optimization and Evaluation Tools
The API Server hosts tools for validating and comparing NPU compile candidates.
- Optimizer: Generates compile option candidates per model Artifact and manages trial execution. Each trial records compile results and evaluation metrics together so operators can choose the NPU artifact that best matches their operating goals.
Traffic & Scaling
Supporting components for the Serving runtime.
- Auto-scaling (KEDA): Scales serving workloads driven by NpuDeploy.
- Async Queue: Async request buffering for nufi-proxy.
Infrastructure dependencies:
- Istio / VirtualService: Used for external traffic routing for Serving and Labs. A VirtualService is created per Serving to route inference requests through the endpoint URL.
Request Flow
Model Deployment Flow
When a user creates a Serving in the Dashboard, the NuFi Controller automatically provisions the K8s resources.
Inference Request Flow
Inference requests to a deployed service pass through nufi-proxy on their way to the inference server.
Deployment Structure
NuFi is installed into a Kubernetes cluster primarily through Helm charts. In airgapped environments, required images and charts are preloaded, then deployed with the same Helm-based flow.
| Feature | Description |
|---|---|
| Helm-based deployment | Deploy NuFi applications and required infrastructure configuration through Helm values |
| Kubernetes operating model | Main components are managed as K8s resources. Operations, deployment, and observability follow Kubernetes conventions, so you can use existing K8s operational tools (kubectl, Helm, Prometheus, etc.) as is. |
Supported Devices
| Vendor | Device | Supported Features |
|---|---|---|
| NVIDIA | CUDA-capable GPU | Lab, Serving |
| FuriosaAI | RNGD | Lab, Serving |
Supported devices are managed through the built-in catalog shipped with NuFi releases.