Nearby Friends
A code-dependent life form.
In my last post I wrote about the proximity service, the system behind "restaurants near me".
https://writer.mrmehta.in/how-nearby-actually-works
I finished that thinking I had location problems figured out. Then Chapter 2 of System Design Interview (II) opened with a feature that looks almost the same: show me which of my friends are nearby.
A restaurant stays where it is. My friends do not. Once every point on the map moves every few seconds, the whole design flips from "search a static index" to "push a stream of updates to the right people". That shift is what this post is about.
What we are building
Think of the old "Nearby Friends" feature in Facebook. If you opt in and share your location, the app shows friends who are close to you.
Nearby list: friends within 5 miles (configurable), each with a distance and a "last updated" time.
Live updates: the list refreshes every few seconds.
Inactive friends drop off: if a friend has not sent a location for 10 minutes, they disappear from the list.
Location history: stored separately, useful for things like machine learning later.
Non-functional needs are low latency, reliable overall (losing the odd update is fine) and eventually consistent. A few seconds of delay between replicas does not hurt anyone.
The numbers that change everything
1 billion users, 10% use the feature, so 100 million daily users.
About 10% of them are online at once, so 10 million concurrent users.
Each one sends a location every 30 seconds. People walk at 3 to 4 miles an hour, so 30 seconds is plenty.
10 million ÷ 30 gives roughly 334,000 location updates per second.
Compare that with 5,000 searches a second in the proximity service. And it gets worse. With 400 friends each and about 10% of them online and nearby, every update has to reach about 40 people. That is around 14 million messages to forward every second.
How it differs from the proximity service
| Proximity service | Nearby Friends | |
|---|---|---|
| What moves | Nothing, businesses are static | Every user, every 30 seconds |
| Traffic shape | Read heavy, about 5,000 QPS | Write heavy, about 334,000 updates/s |
| Who starts it | Client asks (pull) | Server pushes updates |
| Core tools | Geohash index, cache | WebSocket, Redis Pub/Sub |
| Freshness | Next day is fine | A few seconds |
The part that surprised me: there is no geospatial index at all in the main design. I only care about my friends, and I already know who they are. So instead of searching the map, the system just calculates the distance to each friend when their location changes.
The high-level design
In theory, every phone could keep a direct connection to every nearby friend. That falls apart quickly on mobile, with patchy networks and limited battery. So there is a shared backend with three jobs: receive every location update, work out which friends should get it, and skip anyone who is too far away.
REST API servers: stateless, handle the boring parts like adding friends and updating profiles.
WebSocket servers: stateful. Each online user keeps one long-lived connection here, and updates are pushed down it.
Redis location cache: the latest location per active user, with a TTL. When the TTL runs out, that user counts as inactive.
Location history DB: a write-heavy store like Cassandra, sharded by user ID. Not on the hot path.
Redis Pub/Sub: the routing layer. Every user gets their own channel, and their friends' connection handlers subscribe to it.
Following one location update
This is the flow the chapter keeps coming back to, and once it clicked the rest made sense. Say user 1 has three friends online: users 2, 3 and 4.
The clever bit is where the distance check lives. Each friend's connection handler already keeps that friend's latest location in memory. When a message arrives on the channel, the handler compares the two points and decides whether to push or drop. No extra lookups.
What happens when I open the app
When a client connects, its WebSocket handler does a short setup:
Save my location in the cache and in the handler's memory.
Load my friend list from the user database.
Fetch all my friends' locations from the cache in one batch. Inactive friends have expired, so they simply are not there.
Send back the friends within 5 miles.
Subscribe to every friend's channel, active or not.
Publish my own location to my channel.
Step 5 caught my eye. Subscribing to inactive friends sounds wasteful, but an idle Redis channel costs a little memory and zero CPU. It saves the system from tracking who just came online. Trading some memory for a much simpler design is a nice lesson on its own.
Scaling it
WebSocket servers
Auto-scaling is fine, but these servers hold live connections. Before removing one, mark it as "draining" at the load balancer so no new connections land there, then wait for the existing ones to close. The same care applies when deploying new code.
Location cache and user database
10 million locations at about 100 bytes each fits on a single Redis server. 334,000 writes a second does not. Since each user's location is independent, sharding by user ID spreads the load cleanly. A standby replica per shard covers failures. The user and friendship data is also sharded by user ID.
Redis Pub/Sub: the real bottleneck
This is where the chapter's maths got interesting for me.
Memory: 100 million channels, about 100 subscribers each, about 20 bytes per subscriber. Roughly 200 GB, so two big servers.
CPU: 14 million pushes a second. Assuming a very conservative 100,000 pushes per server, that is about 140 servers.
So we need a distributed Pub/Sub cluster, and the question becomes: which server holds which channel?
Channels are sharded by the publisher's user ID on a hash ring. The ring itself is stored in a service discovery tool like etcd or ZooKeeper, and each WebSocket server keeps a local copy.
One thing I had not thought about: the messages are not stored, but the subscriber lists are state. If a channel moves to another server, every subscriber has to resubscribe. So the book treats this cluster like a storage cluster:
Over-provision it so you rarely need to resize.
If you must resize, do it at the quietest time of day and expect a wave of resubscriptions.
Replacing one dead server is low risk, since only its channels move.
The edge cases
Adding or removing a friend: the app fires a callback, and the WebSocket server subscribes to or unsubscribes from that friend's channel.
Users with thousands of friends: friend counts are capped (Facebook caps at 5,000), and those subscribers are spread across many servers, so no single machine takes the hit.
Showing random nearby people: this brings geohash back from the last chapter. Create a channel per geohash cell and subscribe to your own cell plus its 8 neighbours.
An alternative to Redis Pub/Sub: Erlang. Each user becomes a lightweight process of about 300 bytes, and millions fit on one server. The authors actually prefer it, but good Erlang engineers are hard to hire.
Taking away
Moving data changes the whole design. Static points mean indexing and caching. Moving points mean streaming and fan-out.
Sometimes you do not need a spatial index. If the set of people you care about is small and known, just calculate the distance.
Do the maths on CPU, not just memory. Pub/Sub looked like a two-server problem until the push rate was worked out.
"Stateless" messages can still sit on stateful servers. Subscriber lists are why the Pub/Sub cluster needs careful scaling.
Cheap idle resources simplify things. Subscribing to inactive friends removes a whole class of bookkeeping.
If you ever need help or just want to chat, DM me on Twitter / X or LinkedIn.
