The probe
The FYF is not a follow-graph feed — it’s a recommendation feed. The candidate pool is “every video ever uploaded” filtered by ML, not “videos from accounts you follow.” Same fan-out and ranking architecture as Twitter Timeline, but the candidate generation layer is completely different.
Step 1 — Clarify
- Follow-graph feed or pure recommendation? (FYF = recommendation — no follow required)
- Short video only (≤60s) or mixed? This drives storage and encoding decisions - Real-time engagement signals (likes, shares, completion rate) must feed back into ranking within minutes — how fresh?
- DAU? (TikTok: ~1B DAU globally)
- SLO: next video must load in < 500ms — the “infinite scroll” UX depends on it
Step 2 — Estimate
- 1B DAU × 40 videos watched/day = 40B video plays/day = ~463K plays/sec - Each user needs a pre-fetched queue of 10–20 recommended videos to enable instant scroll
- Candidate generation per user: rank top 500 from a pool of billions → return top 20 - Engagement events (play, like, share, skip): ~5 events per video = 200B events/day = 2.3M events/sec feeding the ranking model
Step 3 — API Design
GET /v1/feed/foryou
Query: cursor, limit (default 10), device_type
Response: { videos: [{video_id, cdn_url, creator, caption, music}], next_cursor, prefetch_next_10: [urls] }
POST /v1/engagement
Body: { video_id, event_type: “play”|”like”|”share”|”skip”|”complete”, watch_duration_ms }
The prefetch_next_10 in the response is the key design signal — the client pre-fetches the next 10 videos while the user watches the current one, achieving the illusion of instant load.
Step 4 — Data Model
- videos table: video_id, creator_id, s3_key, duration_ms, transcript, embedding (vector), tags[], upload_at
- engagement_events (Kafka → time-series store): user_id, video_id, event_type, watch_pct, timestamp
- user_interest_profile: user_id, topic_vectors[], creator_affinity[], updated_at — updated continuously from engagement stream
- video_recommendation_cache (Redis): user_id → [ordered list of 50 pre-ranked video_ids, TTL=10min]
