Instagram’s photo upload, feed, and discovery systems are a marvel of distributed engineering, but the real magic isn’t in the sheer volume of data, it’s in how they make a billion users feel like they’re the only ones using the app.
Let’s see it in action. Imagine you’re Sarah, a photographer in New York, and you just took a stunning shot of a sunset.
-
Upload: Sarah taps the "+" button, selects her photo, adds a caption: "Golden hour over the city #NYC #SunsetLover," and hits "Share."
- Her phone compresses the image (say, from 8MB to 500KB) and uploads it to Instagram’s S3-compatible object storage.
- A lightweight service, let’s call it
media_uploader, receives the upload, generates a unique ID (e.g.,post_1234567890), and stores metadata (user ID, caption, timestamp) in a Cassandra database. - The
media_uploaderthen triggers a background job to create multiple image sizes (thumbnail, medium, large) and store those too. This happens asynchronously so Sarah’s upload feels instant.
-
Feed Generation (The Fan-out): This is where it gets interesting. Sarah’s followers (let’s say Alice, Bob, and Charlie) need to see her new post in their feeds.
- Instagram uses a "fan-out" approach. When Sarah posts, a service called
fanout_workerreads her follower list from a Redis cache (populated from a primary user graph database like Neo4j). - For each follower, it inserts
post_1234567890into a Redis sorted set associated with that follower’s user ID. The score for the sorted set is the post’s timestamp. So Alice’s feed might look like this in Redis:{ post_1234567890: timestamp_A, post_9876543210: timestamp_B, ... }. - This fan-out happens at write time. It’s expensive, but it makes read time (when Alice opens her app) incredibly fast.
- Instagram uses a "fan-out" approach. When Sarah posts, a service called
-
Feed Display (The Fan-in): Alice opens Instagram.
- Her app makes a request to the
feed_service. - The
feed_servicequeries Alice’s Redis sorted set for the latest 50 post IDs (e.g.,post_1234567890,post_9876543210, etc.). - It then fetches the full post details (media URLs, captions, author info) for these IDs from a highly optimized cache (e.g., Memcached or another Cassandra cluster specifically for posts).
- The
feed_servicealso injects ads and suggested posts (more on discovery later). - Finally, it sends the fully composed feed back to Alice’s app.
- Her app makes a request to the
Discovery is where Instagram surprises you with content you didn’t even know you wanted.
The "Explore" tab isn’t just random. It uses a sophisticated recommendation engine. When you like Sarah’s sunset photo, that signal is fed into a machine learning model. This model analyzes your past behavior (photos you like, accounts you follow, stories you watch, even how long you spend looking at a post) and compares it to the behavior of millions of other users.
It identifies users with similar tastes and then looks at what they are interacting with. If many people who like Sarah’s sunset photo also like posts about vintage cars, the system might start showing you more vintage car content. This happens through a complex pipeline involving data processing (Spark, Hadoop), model training (TensorFlow, PyTorch), and real-time serving. The system pre-computes recommendations for users and stores them in a key-value store, ready to be served quickly when you hit the Explore tab.
The system uses a combination of collaborative filtering (finding users with similar tastes) and content-based filtering (recommending items similar to what you’ve liked). It’s constantly learning and adapting.
A key optimization for feed generation is handling users with millions of followers (celebrities, brands). A pure fan-out for them would be astronomically expensive. Instead, Instagram employs a hybrid approach. For "hot" users, the fan-out is done asynchronously and in batches. When a celebrity posts, their post ID is added to a queue. Dedicated workers then fan it out to a subset of their followers or generate personalized feed updates for their most active fans. For less active followers, the celebrity’s post might be "injected" into their feed at read time, rather than being pre-computed. This balances the cost of fan-out with the need for timely delivery.
The next challenge is understanding the nuances of real-time content moderation and scaling the video upload and streaming infrastructure.