Kryo serialization isn’t just about making Spark faster; it’s about making your data smaller in a way that significantly impacts network traffic and disk I/O.
Let’s watch Kryo in action. Imagine you have a Spark Streaming job processing some JSON data. Without Kryo, Spark defaults to Java serialization. This can be verbose, especially for complex objects.
// Example Spark Streaming setup (simplified)
val sparkConf = new SparkConf()
.setAppName("KryoStreamingExample")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrationRequired", "true") // Crucial for efficiency
val ssc = new StreamingContext(sparkConf, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999)
// Assume lines are JSON strings, parsed into case classes
case class Event(id: Int, name: String, timestamp: Long)
val events = lines.map(parseJsonToEvent) // parseJsonToEvent is a function that deserializes JSON to Event
// Registering custom classes is key for Kryo
val kryoRegistrator = new KryoRegistrator {
override def register(kryo: Kryo): Unit = {
kryo.register(classOf[Event])
// Register other custom classes here
}
}
sparkConf.set("spark.kryo.registrator", kryoRegistrator.getClass.getName)
events.print() // Or any other transformation/action
ssc.start()
ssc.awaitTermination()
// Example JSON input to localhost:9999:
// {"id": 1, "name": "user_login", "timestamp": 1678886400000}
// {"id": 2, "name": "page_view", "timestamp": 1678886401000}
When Spark processes RDDs (or DStreams in this case), it needs to shuffle data between executors. This involves serializing and deserializing objects. Java serialization, while robust, often includes class names and other metadata that bloat the serialized output. Kryo, on the other hand, is designed for speed and compactness. It uses a more efficient binary format and can infer types, especially when you register your custom classes.
The core problem Kryo solves in Spark Streaming is the overhead of shuffling and intermediate data storage. When executors need to exchange data (e.g., for reduceByKey, join, or even just checkpointing intermediate RDDs), serialization becomes a bottleneck. By reducing the size of the serialized data, Kryo directly cuts down on:
- Network Bandwidth: Less data to transfer between nodes means faster shuffles and less strain on your network.
- Disk I/O: If Spark spills data to disk (e.g., due to memory pressure), smaller serialized objects mean less data written and read from disk.
- Memory Usage: While not always the primary benefit, smaller serialized objects can sometimes lead to more efficient memory utilization within the JVM.
The key to unlocking Kryo’s full potential is registration. When you tell Kryo about your custom classes beforehand, it can assign them shorter integer IDs. This avoids embedding the full class name in the serialized output for every instance of that class.
// In your Spark configuration (spark-defaults.conf or programmatically)
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrationRequired=true
spark.kryo.registrator=com.example.MyKryoRegistrator
And the MyKryoRegistrator:
import com.esotericsoftware.kryo.Kryo
import org.apache.spark.serializer.KryoRegistrator
class MyKryoRegistrator extends KryoRegistrator {
override def register(kryo: Kryo): Unit = {
kryo.register(classOf[com.example.MyCustomObject1])
kryo.register(classOf[com.example.MyCustomObject2])
// ... register all your custom classes
}
}
The spark.kryo.registrationRequired=true setting is vital. If you forget to register a class and this is set to true, Spark will throw an error. This forces you to be explicit and ensures you’re getting the most out of Kryo. Without it, Kryo will fall back to writing the full class name, negating much of its efficiency gains.
The one thing most people overlook is how Kryo handles null values and primitive types. While Kryo is excellent for custom objects, its overhead for very simple, small data types like a single Int or String might be comparable to or even slightly more than Java serialization in isolation. The real win comes when you have collections of complex objects, where the repeated serialization of class metadata in Java serialization becomes a major drag. Kryo’s efficiency shines when serializing many instances of the same registered complex types.
The next hurdle you’ll likely encounter is understanding how Kryo interacts with external libraries and their classes, especially when those libraries aren’t explicitly registered.