How SoundCloud scaled its Architecture using BFF and Value-Added Services?

The SoundCloud’s journey from monolith to BFF and Microservices. (4 minutes)

Jul 28, 2024

Intro

Initially, SoundCloud operated through a single monolithic system that managed all interactions across its web client and mobile apps. This system handled numerous requests and services, growing increasingly complex with each new feature. The shared API was a common factor across all platforms, but as SoundCloud expanded, this approach soon reached its limitations, setting the stage for a strategic transformation.

Transition to Microservices and BFF

Recognizing the need for scalability, both operationally and organizationally, SoundCloud embarked on a transition to a microservices architecture. This shift mean breaking down the monolith into smaller, more manageable services, each responsible for specific feature.

A pivotal aspect of this transition was the introduction of Backend for Frontend (BFF) architecture. BFF allowed each team to develop APIs tailored to the specific needs of different frontends - whether web, iOS, Android, or Desktop - enhancing the autonomy and effectiveness of teams.

Challenges with BFF and Introduction of VAS

Despite the initial success, SoundCloud faces challenges with the BFF approach, notably the duplication of business and authorization logic across multiple BFFs. This complexity led to maintenance challenges and synchronization issues.

To address these problems, SoundCloud introduced Value-Added Services (VAS).

The different service layers at SoundCloud that make up this new architecture. Source: https://developers.soundcloud.com/blog/.

VAS operates on a few core principles from Domain-Driven Design (DDD) like domains, entities, value objects and aggregates.

VAS centralized the logic related to entity handling, reducing duplication and improving system coherence. VAS managed entities like Tracks, User, Comments, etc. and their life cycles, returning aggregates that included all necessary metadata, yet without synthesizing it.

This design significantly streamlined operations.

Before and After the introduced VAS Architecture.

Example of the Playlists VAS and its usage.

VAS and its impact on System Design

The implementation of VAS changed how SoundCloud handled data. It offered a centralized approach where entities could be managed through distinct commands and queries, following the Command Query Responsibility Segregation (CQRS) pattern.

This separation allowed different aspects of the system to evolve independently, improving maintainability and scalability. For instance, operations that altered the entity’s state were handled separately from those that retrieved data, ensuring efficient processing and consistency.

Tracks VAS - querying different parts of the Aggregate from several services.

Liking/Disliking a track in the Track VAS using commands.

Evolution to Domain Gateways

As VAS matured, the need to manage entities across different business domains became apparent. SoundCloud introduced the Domain Gateway, a tailored VAS that served to specific business needs. Each gateway was maintained by a different team, providing customized views and operations on entities like Tracks, which could be used differently by Consumers and Creators. This approach minimized cross-team dependencies and maximized domain-specific enhancements.

Track Domain Gateway for managing the Track Entity in the Creator and Consumer business domains.

Conclusion

SoundCloud’s journey from a monolithic architecture to a sophisticated system utilizing BFF and VAS illustrates a strategic approach to scalability and system efficiency. By continuously adapting and refining its architecture, SoundCloud supported its growth while maintaining a robust and flexibly service system.

🌟 TL;DR

Transitioned from monolithic to microservices architecture to improve scalability and maintainability.
Implemented Backend for Frontends (BFF) pattern to optimize APIs for different clients, enhancing team autonomy and operational scalability.
Faced issues with duplicated business logic and increased complexity across multiple BFFs.
Introduced Value-Added Services (VAS) to centralize core functionalities and reduce duplication, simplifying the architecture.
Developed Domain Gateways to manage and centralize services for specific business domains, further improving scalability and system coherence.