ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDB
•Download as PPTX, PDF•
0 likes•101 views
Join ScyllaDB’s CEO, Dor Laor, as he introduces the revolutionary tablet architecture that makes one of the fastest databases fully elastic. Dor will also detail the significant advancements in ScyllaDB Cloud’s security and elasticity features as well as the speed boost that ScyllaDB Enterprise 2024.1 received.
Recommended
More Related Content
Similar to ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDB
Similar to ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDB (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDB
- 1. ScyllaDB Leaps Forward Dor Laor, Co-founder & CEO of ScyllaDB
- 2. Dor Laor ■ Hello world, Core router, KVM, OSv, ScyllaDB ■ Phd in Snowboard, aspiring MTB rider ■ Let’s shard!
- 3. 3 Learnings from the KVM Hypervisor Dev Period 1.Layer == overhead 2.Locking == Evil 3.Simplicity == True 4.1 million op/s == Expectations 5.Off-by == one || 0x10
- 4. ScyllaDB is Proud to Serve!
- 5. You’ll hear from many of our customers
- 6. Why ScyllaDB? Best High Availability in the industry Best Disaster Recovery in the industry Best scalability in the industry Best Price/Performance in the industry Auto-tune - out of the box performance Compatible with Cassandra & DynamoDB The power of Cassandra at the speed of Redis with the usability of DynamoDB No Lock-in Open Source Software
- 7. Agenda Part 1 ■ Arch overview ■ New 2024 results & benchmarks ■ ScyllaDB Cloud brief Part 2 ■ ScyllaDB 6.0 ■ Tablets ■ What’s coming
- 8. Shard Per Core Architecture Shards Threads
- 9. ScyllaDB Architecture Homogeneous nodes Ring Architecture
- 10. ScyllaDB Architecture Homogeneous nodes Ring Architecture?@$@#
- 11. Can we linearly scale up?
- 13. ■ Time between failures TBF ■ Ease of maintenance ■ No noisy neighbours ■ No virtualization, container overhead ■ No other moving parts ■ Scale up before out! Small vs Large Machines
- 14. Linear Scale Ingestion 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 15. Linear Scale Ingestion 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 16. Linear Scale Ingestion 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 17. Linear Scale Ingestion 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 18. Linear Scale Ingestion 2X 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 19. Linear Scale Ingestion 2X 2X 2X 2X 2X 3 x 4-vcpu VMs 1B keys 3 x 8-vcpu VMs 2B keys 3 x 16-vcpu VMs 4B keys 3 x 32-vcpu VMs 8B keys 3 x 64-vcpu VMs 16B keys 3 x 128-vcpu VMs 32B keys Credit: Felipe Cardeneti Mendes
- 20. Linear Scale Ingestion (2023 vs 2018) Constant Time Ingestion 2X 2X 2X 2X 2X
- 21. Ingestion: 17.9k OPS per shard * 16
- 23. “Nodes must be small, in case they fail” You can scale OPS but can you scale failures?
- 24. 2X 2X 2X 2X 2X “Nodes must be small, in case they fail” No they don’t! {Replace, Add, remove} Node at constant time
- 25. 2X 2X 2X 2X 2X “Nodes must be small, in case they fail” No they don’t! {Replace, Add, remove} Node at constant time
- 26. 2X 2X 2X 2X 2X “Nodes must be small, in case they fail” No they don’t! {Replace, Add, remove} Node at constant time
- 27. What’s new in 2024.1?
- 28. 2024.1 vs 2023.1 vs OSS 5.4
- 30. Benchmark: ScyllaDB vs MongoDB
- 31. Benchmark: ScyllaDB vs MongoDB ■ Fair ■ 3rd party ■ SaaS - zero config ■ YCSB - standard ■ 132 wins in 133 workloads ■ 10x-20x better performance ■ 10x-20x better latency ■ MongoDB doesn’t scale!
- 32. Scale Benchmark: ScyllaDB vs MongoDB
- 34. ScyllaDB Cloud - 65% of Customers ■ Terraform providers, launch, scale, manage ■ Multiple networking modes ■ Encryption at rest, BYOK ■ Certifications: SoC2, ISO, PCI ■ Coming: Azure
- 35. Welcome ScyllaDB Our journey from eventual to immediate consistency
- 36. Scylla today is awesome but ■ Topology changes are allowed one-at-a-time ■ Rely on 30+ second timeouts for consistency ■ Node failed/down block scale ■ Streaming time is a function of the schema ■ Additional complex operations: Cleanup, repair
- 37. ScyllaDB 6.0 ■ Consistent schema changes (raft, >= 5.2) ■ Consistent topology changes (raft, 6.0) ■ Tablets
- 38. ScyllaDB 6.0 Value Elasticity ■ Faster bootstrap ■ Concurrent node operations ■ Immediate request serving Simplicity ■ Transparent cleanups ■ Semi transparent repairs ■ Auto gc-grace period ■ Parallel maintenance operations Speed ■ Streaming sstables - 30x faster ■ Load balancing of tablets Consistency TCO ■ Shrink free space ■ Reduce static, over provisioned deployments
- 39. Behind the Scene
- 40. Raft- Consistent metadata Protocol for state machine replication Total order broadcast of state change commands X = 0 X += 1 CAS(X, 0, 1)
- 41. node A bootstrap bootstrap Linearizable Token Metadata node B node C system.token_metadata Read barrier Read barrier
- 42. ■ Fencing - each write is signed with topology version ■ If there is a version mismatch, the write doesn’t go through Changes in the Data Plane Replica Coordinator Topology coordinator
- 43. Consistent Metadata Journey RAFT Safe schema changes Safe topology changes Dynamic partitioning Consistent tables Tablets 5.0 5.2 5.2+ 6.0
- 44. 6.0 Tablets FTW
- 45. Standard tables {R1, R2, R3} R1 R2 R3 key1 replication metadata: (per keyspace)
- 46. Standard tables {R1, R2, R3} R1 R2 R3 key1 key2 Sharding function generates good load distribution between CPUs
- 47. RAFT group No. 299238 RAFT group No. 299236 RAFT group No. 299237 RAFT tables key1 key2 tablet tablet replica tablet replica
- 48. RAFT tables key1 key2 Good load distribution requires lots of RAFT groups.
- 49. Tablets - balancing Table starts with a few tablets. Small tables end there Not fragmented into tiny pieces like with tokens
- 50. Tablets - balancing When tablet becomes too heavy (disk, CPU, …) it is split
- 51. Tablets - balancing When tablet becomes too heavy (disk, CPU, …) it is split
- 52. Tablets - balancing The load balancer can decide to move tablets
- 53. Tablets - balancing Depends on fault-tolerant, reliable, and fast topology changes.
- 54. Tablets Resharding is cheap. SStables split at tablet boundary. Reassign tablets to shards (logical operation).
- 55. Tablets Cleanup after topology change is cheap. Just delete SStables.
- 56. Tablet Scheduler Scheduler globally controls movement, maintenance operation on a per tablet basis repair migration tablet 0 tablet 1 schedule schedule repair Backup
- 57. Tablet Scheduler Goals: ■ Maximize throughput (saturate) ■ Keep migrations short (don’t overload) Rules: ■ migrations-in <= 2 per shard ■ migrations-out <= 4 per shard
- 58. Tablets - Repair - Fire & forget ■ Tablet based ■ Continuous, transparent controlled by the load balancer ■ Auto GC grace period
- 59. Tablets - Streaming (Enterprise) Send files over RPC. No per schema, per row processing. 30x faster, saturate links. Scylla Enterprise only Sstables files Sstables files Sstables files
- 60. Post 6.0
- 61. Since we have ■ 30x faster streaming ■ Parallel operations ■ Small unit size - based on tablet/shard, not hardware ■ Negligent performance impact ■ Incremental serving as you add machines No reliance on cluster size, instance size or instance type Tablets =~ Serverless
- 62. Serverless Time ie3n’s i4i Time Capacity Required Time On-demand Base Typeless Sizeless limitless
- 63. What’s Cooking?
- 64. ■ Full transactional consistency with Raft ■ HDD and dense standard nodes (d3en) ■ S3 backend ■ Incremental repair ■ Point in time backup/restore ■ Tiered storage What’s in the Oven Eventual Consistency
- 65. Thank You! Keep Innovating! IoT Crypto eCommerce Telco Feature Store Streaming Fintech Cyber Social network Graph Storage Layer Recommendation & Personalization Engine Fraud & Threat Detection AI/ML Analytics Customer Experience