Spark Streaming Checkpointing: HDFS and S3 Setup (2026)

Spark Streaming’s checkpointing mechanism is actually a distributed file system’s best friend, not just a Spark feature.

Let’s see it in action. Imagine a simple Spark Streaming application that reads from Kafka and writes to a database.

val spark = SparkSession.builder()
  .appName("SparkStreamingCheckpointExample")
  .master("local[*]") // For local testing, use yarn or spark:// for cluster
  .getOrCreate()

val ssc = new StreamingContext(spark.sparkContext, Seconds(5))

// Enable checkpointing
ssc.checkpoint("/user/spark/checkpoints") // Or "s3a://my-bucket/spark-checkpoints"

// Kafka stream setup
val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "localhost:9092",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "spark-streaming-group"
)
val topics = Set[String]("my-topic")

val stream = KafkaUtils.createDirectStream[String, String](
  ssc,
  LocationStrategies.PreferConsistent,
  ConsumerStrategies.Subscribe[String, String](topics, kafkaParams)
)

// Process the stream
stream.foreachRDD { rdd =>
  if (!rdd.isEmpty()) {
    val count = rdd.count()
    println(s"Received $count records.")
    // Simulate writing to a database
    rdd.foreachPartition { partition =>
      // Replace with actual database connection and write logic
      partition.foreach { record =>
        println(s"Processing record: ${record.key()} -> ${record.value()}")
      }
    }
  }
}

ssc.start()
ssc.awaitTermination()

This setup initializes a Spark Streaming context, points it to a Kafka topic, and for each incoming RDD, it prints the record count and then iterates through each record for simulated processing. The critical part here is ssc.checkpoint("/user/spark/checkpoints"). This tells Spark to periodically save its operational state to HDFS (or S3 if configured).

What problem does this solve? Without checkpointing, if your Spark Streaming application restarts (due to a failure, upgrade, or manual stop/start), it loses all context about what it has processed. It would start from the very beginning of the Kafka topic, reprocessing all data. This is unacceptable for most production scenarios where exactly-once or at-least-once processing semantics are required. Checkpointing allows Spark Streaming to recover its state, including the offsets of the last processed records from Kafka, and resume processing from where it left off.

Internally, Spark Streaming checkpointing involves saving two main pieces of information:

1. Metadata: This includes the configuration of the DStream graph itself. When Spark restarts, it can reconstruct the entire processing pipeline based on this metadata.
2. State: For stateful operations like updateStateByKey or mapWithState, Spark saves the accumulated state. This is crucial for maintaining the correct context across restarts. For operations like createDirectStream from Kafka, the "state" is primarily the set of offsets that have been processed.

The checkpoint directory you specify is where these files are stored. For HDFS, it’s a standard HDFS path like /user/spark/checkpoints. For S3, you’d use an S3a path like s3a://my-bucket/spark-checkpoints. Spark writes metadata files and, for stateful operations, state files into this directory. When the application starts, Spark checks this directory. If it finds valid checkpoint data, it loads it and resumes execution. If the directory is empty or corrupted, it starts fresh.

The checkpointInterval (defaults to the batch interval) determines how often Spark writes this state. For Kafka direct streams, Spark writes the latest consumed offsets to HDFS/S3 at this interval. This ensures that even if the driver crashes between batches, the next driver will know exactly which offsets to pick up from Kafka, preventing duplicate processing or missed messages.

The most surprising thing about Spark Streaming checkpointing is that it’s not just about recovering your application’s state; it’s also about enabling the underlying sources to provide a reliable recovery point. For Kafka, the direct stream relies on Spark writing offsets to the checkpoint location. Spark then uses these saved offsets to tell Kafka which messages have been successfully processed. If you were using a different source that didn’t integrate with Spark’s checkpointing for offset management, you’d have to build that offset management logic yourself, which is significantly more complex.

When you use createDirectStream with Kafka, Spark manages the offsets internally and saves them to the checkpoint directory. This is how it achieves fault tolerance for Kafka sources. If you’re using createRDD or other non-direct methods, Spark doesn’t manage offsets for you, and checkpointing won’t magically give you fault tolerance for those sources.

The next hurdle you’ll likely face is understanding how to manage and clean up these checkpoint directories, especially in long-running applications, to avoid disk space exhaustion or performance degradation.