The most surprising thing about Pastebin’s URL shortening and object storage is how much it relies on a few fundamental, almost ancient, computer science concepts to achieve massive scale and low cost.
Let’s see it in action. Imagine you’ve just pasted some code. Your browser sends a POST request to the Pastebin API.
POST /api/1.0/pastes HTTP/1.1
Host: pastebin.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 123
content=print('hello%20world')&title=hello_world&format=python
The backend receives this. It hashes the content to get a unique identifier (let’s call it content_hash). If this content_hash already exists, it means this exact paste has been uploaded before. Pastebin just returns the existing short URL. If it’s new, it stores the content associated with content_hash in object storage and generates a unique, short, random string for the URL (e.g., aBc123). This short string is then mapped to the content_hash in a database. The final URL is https://pastebin.com/aBc123.
The core problem Pastebin solves is storing arbitrary amounts of text data efficiently and serving it quickly via short, memorable URLs. It needs to handle millions of pastes, many of which are duplicates, and serve them with low latency.
Internally, the system works like this:
- Content Deduplication: When a new paste is uploaded, its content is hashed (e.g., using SHA-256). This hash acts as a unique fingerprint for the content.
- Object Storage: If the hash doesn’t exist in their lookup system, the actual content is stored in an object storage service. Object storage is ideal for this because it’s highly scalable, durable, and cost-effective for storing large, immutable binary data (text files are just binary data). Think of services like Amazon S3, Google Cloud Storage, or even self-hosted MinIO. The object itself is typically keyed by its hash.
- Metadata Database: A database (often a key-value store like Redis or DynamoDB for speed, or a more traditional SQL database) stores the mapping between the short URL’s unique identifier (the random string like
aBc123) and thecontent_hash. This database is optimized for fast lookups of thecontent_hashgiven the short URL identifier. - URL Generation: A small, random string (e.g., 6-8 characters) is generated for each new unique paste. This string is stored in the metadata database, linked to the
content_hash. This is the part that becomes the public URL.
This architecture allows Pastebin to:
- Save Storage: By hashing content, identical pastes are only stored once. If 10,000 people paste the same
print("hello world")snippet, only one copy of that string is ever saved to object storage. - Fast Retrieval: When a user requests
https://pastebin.com/aBc123, the system looks upaBc123in the metadata database to find thecontent_hash. It then uses thecontent_hashto retrieve the actual content directly from object storage. This is typically very fast. - Scalability: Object storage and key-value databases are designed to scale horizontally to handle massive amounts of data and requests.
The "shortening" part isn’t about compressing the URL itself, but about generating a short identifier that maps to the content. The random string is a compact key. The real "object" being stored is the paste’s content, and object storage is the perfect fit for that.
The magic of deduplication happens at the content hash level. If you upload a 10MB file and then upload the exact same 10MB file, Pastebin checks the hash. If it’s seen before, it just returns the existing URL. If it’s new, it stores the 10MB file in object storage. The random URL string is just a pointer to that stored object, identified by its hash.
The specific levers you control are the length of the random identifier string and the choice of hashing algorithm for content. Longer random strings reduce the chance of collisions (where two different pastes get the same short URL identifier) but don’t affect storage or retrieval speed directly. The hashing algorithm affects how unique the fingerprint is and the performance of the hashing operation itself.
A common optimization is to pre-generate a pool of short URL identifiers. When a new unique paste needs a URL, it picks one from the pool. This avoids the computation of generating a random string on the fly for every single new paste and ensures a consistent rate of URL creation.
The next challenge is handling rate limiting and abuse, especially for anonymous users.