Triton and Ray Serve are both powerful platforms for deploying machine learning models, but they solve slightly different problems and excel in different scenarios. The most surprising truth is that they aren’t direct competitors in most use cases; they often complement each other.
Let’s see Triton in action. Imagine you have a trained PyTorch model for image classification. You’ve saved it as model.pt.
# model.py
import torch
import torch.nn as nn
class SimpleCNN(nn.Module):
def __init__(self):
super(SimpleCNN, self).__init__()
self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)
self.relu = nn.ReLU()
self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
self.fc = nn.Linear(16 * 16 * 16, 10) # Assuming input image size 32x32
def forward(self, x):
x = self.conv1(x)
x = self.relu(x)
x = self.maxpool(x)
x = x.view(x.size(0), -1)
x = self.fc(x)
return x
model = SimpleCNN()
# Load pre-trained weights if available
# model.load_state_dict(torch.load('model_weights.pth'))
model.eval()
To deploy this with Triton, you’d first need to convert it to a format Triton understands, typically ONNX.
# Install onnxruntime and PyTorch
pip install onnx onnxruntime
# Convert PyTorch to ONNX
python -c "
import torch
from model import SimpleCNN
dummy_input = torch.randn(1, 3, 32, 32) # Batch size 1, 3 channels, 32x32 image
model = SimpleCNN()
torch.onnx.export(model, dummy_input, 'model.onnx', verbose=True, input_names=['input'], output_names=['output'])
"
Now, you’ll set up a Triton model repository. This is a directory structure that Triton uses to discover and load models.
triton_model_repo/
└── my_cnn_model/
├── config.pbtxt
└── 1/
└── model.onnx
The config.pbtxt file tells Triton about your model.
name: "my_cnn_model"
platform: "onnxruntime_onnx"
input [
{
name: "input"
data_type: TYPE_FP32
dims: [ 3, 32, 32 ]
}
]
output [
{
name: "output"
data_type: TYPE_FP32
dims: [ 10 ]
}
]
You then run the Triton inference server, pointing it to this repository.
triton_docker_image="nvcr.io/nvidia/tritonserver:23.10-py3" # Or your preferred version
docker run --gpus all -d -p 8000:8000 -p 8001:8001 -p 8002:8002 \
-v /path/to/triton_model_repo:/models \
$triton_docker_image \
tritonserver --model-repository=/models
Triton’s core strength is its ability to serve multiple models and frameworks (TensorFlow, PyTorch, ONNX, TensorRT) from a single server with high throughput and low latency. It handles batching, model versioning, and concurrent model execution efficiently.
Ray Serve, on the other hand, is built on top of Ray, a distributed computing framework. It’s designed for building complex, end-to-end ML applications that might involve multiple model deployments, pre/post-processing steps, and Python-native workflows.
Here’s how you might deploy the same PyTorch model using Ray Serve. You’d typically define a Serve application.
# ray_serve_app.py
import ray
from ray import serve
from ray.serve.handle import DeploymentHandle
import torch
from model import SimpleCNN # Assuming model.py is in the same directory
@serve.deployment(num_replicas=2)
class PyTorchModel:
def __init__(self):
self.model = SimpleCNN()
# Load weights if you have them
# self.model.load_state_dict(torch.load('model_weights.pth'))
self.model.eval()
async def __call__(self, request):
# Assuming the request contains image data that can be converted to a tensor
# For simplicity, let's assume raw image bytes or a numpy array
image_data = await request.json() # Example: request body is JSON with image data
# Convert image_data to tensor (this is a placeholder)
input_tensor = torch.tensor(image_data['image']).float().unsqueeze(0) # Add batch dim
with torch.no_grad():
predictions = self.model(input_tensor)
return {"predictions": predictions.tolist()}
# Option 1: Deploy directly
# ray.init(address="auto") # Connect to an existing Ray cluster or start one
# serve.start()
# model_deployment = PyTorchModel.bind()
# serve.run(model_deployment)
# Option 2: Using a dashboard/controller (more common for complex apps)
# You'd typically run `serve run ray_serve_app:model_deployment`
# after defining model_deployment as above and initializing ray/serve.
# Example of a more complex Ray Serve app with multiple deployments:
@serve.deployment
class Preprocessor:
def __init__(self, target_shape=(32, 32)):
self.target_shape = target_shape
async def __call__(self, request):
image_bytes = await request.body()
# Process image_bytes to a tensor (e.g., using OpenCV, Pillow)
# This is a placeholder:
processed_image = torch.randn(3, self.target_shape[0], self.target_shape[1])
return processed_image
@serve.deployment(num_replicas=2)
class PyTorchModel:
def __init__(self):
self.model = SimpleCNN()
self.model.eval()
async def __call__(self, processed_image_tensor):
with torch.no_grad():
predictions = self.model(processed_image_tensor.unsqueeze(0)) # Add batch dim
return {"predictions": predictions.tolist()}
preprocessor = Preprocessor.bind()
pytorch_model = PyTorchModel.bind()
# Chain them together: request goes to preprocessor, then to model
# This requires a routing mechanism or explicit call
# For simplicity, let's show a direct call after deployment:
# ray.init(address="auto")
# serve.start()
# chained_app = preprocessor.then(pytorch_model)
# serve.run(chained_app)
# To run this:
# 1. Save the code as ray_serve_app.py
# 2. Start Ray: ray start --head
# 3. Deploy: serve run ray_serve_app:PyTorchModel
# 4. Send a request (example using http client):
# curl -X POST -H "Content-Type: application/json" -d '{"image": [ ... ]}' http://localhost:8000/predict
Ray Serve’s strength lies in its Python-native integration, flexibility for complex pipelines, and seamless scaling across a Ray cluster. You can easily define Python functions or classes as deployments, orchestrate them into complex workflows, and leverage Ray’s distributed primitives.
The key differentiator is the scope. Triton is optimized for high-performance inference serving of individual models. It’s a specialized tool for that job. Ray Serve is a more general application deployment platform that includes model serving as a component. You’d choose Triton when your primary bottleneck is raw inference throughput and latency for a set of models, and you want a dedicated, highly optimized serving layer. You’d choose Ray Serve when you need to build a more complex ML application that involves multiple Python services, custom pre/post-processing logic, or integration with other distributed Python workloads managed by Ray.
One thing most people don’t realize is that you can actually run Triton within a Ray Serve deployment. This allows you to leverage Triton’s inference optimization while still managing the deployment and orchestration through Ray Serve’s Pythonic interface. You could have a Ray Serve deployment that acts as a proxy, forwarding requests to a Triton server (either managed by Ray or running separately). This hybrid approach combines the best of both worlds: Triton’s raw inference power and Ray Serve’s application orchestration capabilities.
Ultimately, the choice often comes down to whether you need a dedicated, high-performance inference server (Triton) or a flexible platform for building and deploying distributed Python ML applications, which can include model serving (Ray Serve).