Requirements
Scalable and reliable storage: The system must scale horizontally to manage the growing data volume from all applications, ensuring reliability and uninterrupted service delivery according to the agreed SLA.
Performance: Each application will need the storage service to maintain high performance, with low latency and high throughput, even as demand fluctuates.
SLA management: The infrastructure team must define, monitor, and enforce strict SLAs that dictate the performance and availability standards of the storage service.
Cost efficiency: With the potential for extensive resource utilization, the platform must optimize for cost efficiency without sacrificing quality or performance.
Ease of use: Simplified access and interaction with the storage platform are crucial. Developers from various teams should find the system intuitive, with straightforward processes for provisioning, accessing, and managing data.
Data segregation and access control: The platform must ensure strict data segregation for security and compliance. Access controls must be robust and granular to prevent unauthorized access to sensitive information from different applications.
Additional considerations
Data governance: As the central repository for various applications, the storage platform must adhere to data governance policies, ensuring data integrity, quality, and regulatory compliance.
Backup and recovery: A robust backup and disaster recovery strategy is essential, providing guarantees against data loss and enabling quick restoration of services in case of an outage.
Customization and extensibility: Similar to what we saw in the SaaS application use case, the platform should offer customization options that cater to specific application needs, including support for various data types and structures.
Monitoring and optimization: Continuous monitoring for operational health and performance optimization is necessary to maintain the platform’s efficiency and to preemptively address potential issues.
Operational data store
An operational data store, often described as a data-as-a-service (DaaS) model, centralizes data storage and consolidates data from myriad sources, providing a single point of access for different applications. This type of architecture is critical for applications that provide a comprehensive view of data from different domains, such as a “customer 360” application, which amalgamates customer information from CRM, order management, support systems, and more.
Architecture
IDG
This architecture is typically composed of three main components: data sources, a central operational data store (ODS), and data consumers. Data from CRM, ERP, SCM, and other systems is consolidated into the ODS using extract, transform, and load (ETL) processes or change data capture (CDC) methods, where it becomes accessible for queries and analytics by various data consumer applications.
Requirements
Data integration and quality: Effective ETL/CDC processes are essential for integrating data from disparate sources while ensuring its quality and consistency.
Consolidation and transformation: The central data store must efficiently consolidate and transform data, ensuring it’s in the right format and structure for consumption by various applications.
Low-latency access: Applications such as real-time dashboards require immediate access to data, necessitating a low-latency system that can quickly process and serve data requests.
Robust query performance: With multiple consumers accessing the platform — often with complex queries — the system needs to maintain high-performance levels without imposing bottlenecks.
Data security and privacy: The centralized nature of the platform means it must have stringent security measures and privacy controls to protect sensitive data and comply with regulations.
Scalable and reliable infrastructure: As the central hub for organizational data, the infrastructure must be scalable to handle growing data volumes and resilient to ensure constant availability.
Additional considerations
Data governance: There should be clear policies and procedures in place to manage the data life cycle, ensuring accountability and regulatory compliance.
Advanced analytics: The platform should be capable of supporting advanced analytics and business intelligence (BI) applications, providing valuable insights across the organization.
Customizable access patterns: Different applications may require different access patterns; hence, the platform should be flexible to accommodate these variations.
Monitoring and alerts: The system should include comprehensive monitoring capabilities to detect and respond to issues promptly, ensuring system health and data integrity.
Multi-tenant application design patterns
Use caseTenant typeTenant countRequirementsChallengesSaaS applicationExternal tenantThousands to hundreds of thousandsManage a huge number of customer/user data with scalability, multi-tenant isolation and protection, SLA compliance, and agility1. Huge tenant count 2. Tenant Isolation 3. Total cost 4. Schema changes 5. Availability 6. Scalability of large number of (usually large) tenantsCentralized storage platformInternal tenantTens to hundredManage a massive number of database instances in a safe and cost-efficient way. This is database consolidation. 1. Scalability for large services 2. Reliability for critical services 3. Cost efficiency for large numbers of small/non-critical services 4. Maintenance cost of the database platformOperational data storeMultiple tenants share the same data set. Must isolate write/ingestion and read workloads and read workloads from different tenants 1. Scalability 2. Flexibility in querying data 3. Ingestion speed and the impact to read 4. Handling complex queries against a large data set 5. Isolation between the services1. Scalability 2. Flexibility in querying data 3. Ingestion speed and the impact to read 4. Handling complex queries against a large data set 5. Isolation between the services
Multi-tenant data management design patterns
Now that we’ve outlined the main use cases for multi-tenancy, we can explore architectural designs that meet different needs.