Automation framework for analysis, tuning, and migration of Apache Spark workloads from CPU to GPU.
NVIDIA Aether
NVIDIA Aether is an automation framework for analyzing, tuning, and migrating Apache Spark workloads from CPU to GPU using cuDF for Apache Spark. It provides ML-powered qualification, iterative performance tuning, cost-aware cluster optimization, and validation — accessible through a CLI, REST API, and web dashboard.
Installation
bash aether_v26.04.0/install-26.4.0.sh \
-w aether_v26.04.0/nvidia_aether-26.4.0-py3-none-any.whl \
-p <platform>
Requirements: Python 3.10–3.12, Java 8/11/17. GPU jobs require cuDF for Apache Spark on cluster nodes.
After installation, activate the environment and initialize:
source venv-aether/bin/activate
aether config init
aether config set core.selected_platform <platform>
Features
Services
| Service / Command | Purpose |
|---|---|
| Config | Manage configuration settings, platform selection, and environment options |
| Run | End-to-end workflow: qualify, provision cluster, submit GPU job, tune, and validate |
| Qualify | Analyze CPU Spark event logs to assess GPU acceleration potential and generate recommended configurations using pre-trained ML models |
| Profile | Analyze GPU job event logs to identify bottlenecks and generate optimized configurations, optionally using pre-trained ML models |
| Tune | Iteratively submit and profile GPU Spark jobs to optimize configurations for performance and cost |
| Submit | Submit Spark jobs with GPU-optimized configurations |
| Validate | Compare CPU and GPU event-log metrics to detect mismatches |
| Cluster | Provision, manage, and scale GPU clusters on supported platforms |
| Report | Generate performance and cost reports; launch the Aether Report Dashboard |
| Diagnose | Analyze paired CPU/GPU event logs with plan-level SQL matching and performance comparison |
| Train | Train custom qualification models from validated CPU/GPU job pairs |
Services are accessible via the CLI, with select services also available through the REST API.
REST API
Aether includes a REST API server for integration with orchestration systems, automated pipelines, and the Smart Router.
aether rest run
API documentation is available at http://localhost:8000/docs. Enable with aether config set rest.enable_docs true.
| Endpoint | Method | Description |
|---|---|---|
/api/v2/qualify | POST | Qualify a Spark job for GPU acceleration |
/api/v2/profile | POST | Profile a GPU Spark job |
/api/v2/process_completed_job | POST | Asynchronously process a completed job |
/api/v2/route_job | GET | Get a routing recommendation (CPU vs GPU, config selection) |
/api/v2/job_recommendations | GET | Get recommended Spark configurations for a job |
/api/v2/job_migration_readiness | GET | Get migration readiness status for a job |
/api/v2/cluster | GET/POST/DELETE | Cluster lifecycle management |
/healthz | GET | Health check |
Smart Router
The Aether Smart Router is a gRPC proxy for Apache Spark Connect that enables CPU/GPU routing with session management and configuration injection. It sits between Spark clients and Spark Connect servers.
- Dynamic Routing — Proxies each Spark Connect request to the appropriate CPU or GPU server based on job context and migration readiness.
- Configuration Injection — Applies Aether-recommended Spark configurations to each session.
- Shadow Run — After a CPU session completes, the Smart Router can re-execute the recorded session on a GPU server with the Sandbox Plugin in the background.
java -Dconfig.file=/path/to/smart-router.yaml \
-cp grpc-proxy.jar:aether-connect.jar \
com.nvidia.smartrouter.GrpcGateway
The Smart Router ships as two JARs bundled with the NGC resource. Requires Java 17+ and a running Aether REST API server.
Sandbox Plugin
The Aether Sandbox Plugin is a Spark plugin that isolates workload execution by redirecting supported Hive table operations to a sandbox environment, allowing GPU testing without modifying production tables. It operates in two modes: staging (creates sandbox table references) and workload (executes against the sandbox).
Enable sandbox via Aether configuration (platform.<platform>.cluster.enable_sandbox), through the Smart Router (shadowRun.requiresSandbox), or directly via Spark plugin configuration. JARs for Spark 3.3 and Spark 4.0 are bundled inside the Aether wheel.
Aether Report Dashboard
Launch the web-based analytics dashboard to visualize job performance, cost savings, and migration progress:
aether report ui
Job History Database
Aether persists job runs, recommendations, and routing decisions in a database. SQLite is used by default for single-user deployments. For shared or production REST API deployments, PostgreSQL is supported via the database.job_history_db_type configuration.
Use aether migrate apply after version upgrades to apply schema changes.
Aether Assistant
Aether Assistant is an AI-assisted tool for converting Spark UDFs to GPU-accelerated implementations. It includes agent skills for test generation, UDF conversion (to cuDF or Spark SQL), and GPU benchmarking. The assistant wheel and skills are included in the NGC resource download.
bash install-26.4.0.sh -t assistant -w aether_assistant-26.4.0-py3-none-any.whl
What's Included
| Artifact | Description |
|---|---|
nvidia_aether wheel | Aether CLI, REST API, Report Dashboard, bundled JARs (Sandbox Plugin, Smart Router), pre-trained models |
| Offline wheelhouses | Python 3.10, 3.11, and 3.12 dependency bundles for air-gapped installation |
| Install script | Automated setup with Python virtual environment and platform-specific dependencies |
| Aether Assistant | AI-assisted UDF migration tool (separate wheel) |
Supported Platforms
Aether integrates with the following Spark platforms.
| Platform | Capabilities |
|---|---|
| On-premises | Cluster management, job submission, event log analysis |
| Google Cloud Dataproc | Cluster management, job submission, event log analysis, cost estimation |
| Amazon EMR | Cluster management, job submission, event log analysis, cost estimation |
| Google Kubernetes Engine | Event log analysis |
| Databricks (AWS) | Event log analysis |
Documentation
| Resource | Access |
|---|---|
| CLI help | aether <service> --help |
| REST API reference | http://localhost:8000/docs |
| Report Dashboard | aether report ui |
| Configuration options | aether config list |
License
This project requires, and may download and install, additional third-party open source software projects. Review the license terms of these open source projects before use.