Triton’s TensorFlow SavedModel backend doesn’t just load your SavedModel; it can actually improve its performance and flexibility.
Let’s see Triton serve a simple TensorFlow model.
First, we need a SavedModel. Here’s a quick way to create one using TensorFlow:
import tensorflow as tf
import numpy as np
# Define a simple model
input_tensor = tf.keras.layers.Input(shape=(10,), name="input_layer")
dense_layer = tf.keras.layers.Dense(5, activation='relu')(input_tensor)
output_tensor = tf.keras.layers.Dense(1, activation='linear', name="output_layer")(dense_layer)
model = tf.keras.Model(inputs=input_tensor, outputs=output_tensor)
# Save the model
export_path = "./tf_saved_model"
tf.saved_model.save(model, export_path)
Now, let’s configure Triton to serve this model. We’ll create a config.pbtxt file in a model_repository directory.
name: "tf_saved_model"
platform: "tensorflow_savedmodel"
max_batch_size: 16
input [
{
name: "input_layer"
data_type: TYPE_FP32
dims: [ 10 ]
}
]
output [
{
name: "output_layer"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
The platform: "tensorflow_savedmodel" line is crucial. It tells Triton to use the specialized backend for TensorFlow. max_batch_size is set to 16, meaning Triton can dynamically batch up to 16 incoming requests for this model. The input and output definitions must precisely match the tensor names and shapes within your SavedModel. Triton infers these from the MetaGraphDef within the SavedModel, but explicitly defining them here is good practice and sometimes necessary for complex models.
With the model saved and the configuration in place, you can start Triton with this repository. Assuming Triton is installed and your model_repository is in the current directory, you’d run:
triton_server --model-repository $(pwd)/model_repository
Once Triton is running, you can send requests. Here’s a Python client example using the tritonclient library:
import numpy as np
import tritonclient.http as httpclient
# Initialize client
url = "localhost:8000"
client = httpclient.InferenceServerClient(url=url)
# Prepare input data
input_data = np.random.rand(1, 10).astype(np.float32)
inputs = [
httpclient.InferInput("input_layer", input_data.shape, "FP32")
]
inputs[0].set_data_from_numpy(input_data)
# Send inference request
results = client.infer("tf_saved_model", inputs)
output_data = results.as_numpy("output_layer")
print("Input shape:", input_data.shape)
print("Output shape:", output_data.shape)
print("Output:", output_data)
This client sends a single inference request with a batch size of 1. Triton receives this, and if other requests arrive quickly, it will group them up to the max_batch_size of 16 before sending them to the TensorFlow runtime. This dynamic batching is a core feature for improving throughput by keeping the GPU busy.
The TensorFlow SavedModel backend goes beyond simple inference. It can automatically handle model optimization passes like graph fusion and constant folding by leveraging TensorFlow’s own optimization capabilities. When Triton loads a SavedModel, it analyzes the graph and applies these optimizations before serving. This means you often get a more performant model without any changes to your original TensorFlow code.
A key aspect of deploying with Triton is understanding how it maps model inputs and outputs to the actual tensors in your SavedModel. While the config.pbtxt specifies names like "input_layer" and "output_layer", Triton’s TensorFlow backend looks for these exact names within the SignatureDef of your SavedModel. If your SavedModel has multiple SignatureDefs, Triton defaults to the one marked as the default signature. You can also explicitly specify which signature to use in the config.pbtxt using the saved_model_config field, which is particularly useful if your model has multiple distinct prediction endpoints.
name: "tf_saved_model_specific_sig"
platform: "tensorflow_savedmodel"
max_batch_size: 16
input [
{
name: "input_layer"
data_type: TYPE_FP32
dims: [ 10 ]
}
]
output [
{
name: "output_layer"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
saved_model_config {
signature_name: "serving_default" # Or whatever your signature is named
}
This allows Triton to precisely target the intended inference function within your SavedModel, avoiding ambiguity and ensuring the correct graph is executed. It also enables Triton to manage different versions of models or different inference tasks within the same SavedModel artifact by selecting the appropriate signature.
When you have multiple TensorFlow models or even multiple versions of the same model, you can organize them within Triton’s model repository. Each model should have its own subdirectory (e.g., model_repository/my_tf_model/1/model.savedmodel and model_repository/my_tf_model_v2/1/model.savedmodel). Triton automatically discovers and loads models from these directories. For versioning, placing models in numbered subdirectories (e.g., /1/, /2/) under the model’s root directory allows Triton to manage different versions, and you can configure which version(s) are active.
The true power of the SavedModel backend lies in its integration with Triton’s inference graph capabilities. You can chain multiple TensorFlow SavedModels together, or even combine them with models from other backends (like PyTorch or ONNX), to create complex processing pipelines. For instance, a preprocessing TensorFlow model could be followed by an inference TensorFlow model, all orchestrated by Triton without requiring explicit client-side orchestration for each step. This allows for highly efficient, low-latency end-to-end inference pipelines deployed as a single service.
The next step after optimizing your TensorFlow SavedModels for Triton is to explore the custom operations your model might rely on.