The most surprising thing about W&B sweeps is that they don’t actually run your training jobs; they orchestrate them.
Let’s see this in action. Imagine you have a train.py script that takes hyperparameters as command-line arguments:
# train.py
import argparse
import wandb
import random
parser = argparse.ArgumentParser()
parser.add_argument("--learning_rate", type=float, default=0.01)
parser.add_argument("--batch_size", type=int, default=32)
parser.add_argument("--optimizer", type=str, default="adam")
args = parser.parse_args()
wandb.init(project="my-sweep-project", config=args)
# Simulate training with these hyperparameters
loss = 1.0 / (args.learning_rate * args.batch_size) + random.random() * 0.1
accuracy = args.learning_rate * 100 - args.batch_size + random.random() * 5
print(f"Learning Rate: {args.learning_rate}, Batch Size: {args.batch_size}, Optimizer: {args.optimizer}")
print(f"Loss: {loss:.4f}, Accuracy: {accuracy:.4f}")
wandb.log({"loss": loss, "accuracy": accuracy})
wandb.finish()
To start a sweep, you define a configuration file, typically in YAML, that describes the search space and the strategy.
# sweep.yaml
program: train.py
method: random
metric:
name: accuracy
goal: maximize
parameters:
learning_rate:
min: 0.001
max: 0.1
batch_size:
values: [16, 32, 64]
optimizer:
values: ["adam", "sgd", "rmsprop"]
Then, you use the W&B CLI to initialize the sweep:
wandb sweep sweep.yaml
This command outputs a sweep-id (e.g., abcdef12). This ID is the key. It tells W&B’s servers how to manage this particular hyperparameter search.
Now, to actually run the jobs, you launch multiple instances of your training script, but with a special command that tells them to listen to the sweep. You’ll need to provide the sweep-id and a project name.
wandb agent <your-username>/my-sweep-project/<sweep-id>
For example, if your username is myuser and your sweep ID is abcdef12:
wandb agent myuser/my-sweep-project/abcdef12
When you run this wandb agent command, it connects to W&B’s backend. The backend, knowing about your sweep.yaml configuration, assigns a set of hyperparameters to this agent. The agent then takes those hyperparameters, constructs the command to run your train.py script with them, and executes it.
For instance, an agent might run:
python train.py --learning_rate 0.05 --batch_size 32 --optimizer adam
After train.py finishes and logs its metrics to W&B, the agent reports back to the W&B backend, saying "I’m done, here are my results." The backend then assigns the next set of hyperparameters to this agent. This continues until the sweep is complete (e.g., a set number of runs are finished, or a performance goal is met).
The wandb agent process is essentially a worker. It doesn’t know about the sweep strategy (random, grid, bayesian); it just asks for the next set of hyperparameters and runs the script. The intelligence for what hyperparameters to try next resides entirely in the W&B backend, driven by the sweep.yaml configuration.
This separation is powerful. You can launch dozens or hundreds of wandb agent processes on different machines, or even on your local machine in separate terminals, and they will all coordinate through the W&B backend to perform the hyperparameter search without interfering with each other. W&B handles the assignment of hyperparameter combinations and the collection of results.
The method in sweep.yaml defines the search strategy. random is simple but effective. grid exhaustively tries all combinations. bayes uses a probabilistic model to intelligently choose the next hyperparameters based on past results, aiming to converge faster to good performance. The metric section tells W&B what to optimize (goal: maximize or goal: minimize) and what metric name to look for in your logs.
The most common pitfall is forgetting that the wandb agent command needs to be running in parallel with your training script. The wandb sweep command only registers the sweep; it doesn’t run anything. The wandb agent command is what actually launches and monitors your training jobs according to the sweep’s instructions.
The next concept you’ll explore is how to efficiently manage and analyze the results of these sweeps, often involving W&B’s reporting and visualization tools to identify the best-performing hyperparameter configurations.