Tmp
Systems
TODO
Topics [8/10] Storage engine Distributed KV store Distributed block storage Distributed lock Distributed batch processing Distributed stream processing Containerization/virtualization Container orchestration Ranking/recommender system Service Discovery Service Mesh Counter Transient cache
Algorithms Strong consistency Paxos Basic paxos Hard to impl for multi roles (client, proposer, acceptor, learner) Slow (2 RPCs) Liveness Multi paxos Only leader can propose Cheap paxos Fast paxos Raft - simplified multi-paxos Leader election Log replication Safety Term Leader, follower, (candidate) Election timeout and heartbeat timeout Zab - used in ZooKeeper also similar to multi-paxos Epoch Heartbeat: follower => leader Eventual consistency Gossip protocol (Cassandra)
Systems ZooKeeper Etcd
Architecture Lambda architecture Microservices/SOA
Papers Bigtable/HBase/Dynamo MapReduce GFS/HDFS Chubby Spanner Kafka Spark Paxos made simple Scaling Memcache at Facebook Megastore A Gossip-Style Failure Detection Service In Search of an Understandable Consensus Algorithm (Extended Version)
Resources/reading list donnemartin/system-design-primer Grokking the System Design Interview Youtube best system design list Distributed Systems in One Lesson by Tim Berglund 九章系统设计 Gaurav Sen DDIA 大数据日知录 Database Internals 一致性算法（Paxos、Raft、ZAB） http://thesecretlivesofdata.com/raft/ NewSQL大神黄东旭：从0到1，如何设计一个分布式数据库 https://blog.pramp.com/how-to-succeed-in-a-system-design-interview-27b35de0df26 https://jack-vanlightly.com/blog/2017/12/3/rabbitmq-vs-kafka-series-introduction
Questions Rate limiter
Structured Data Storage/KV Store Single Node Storage Engine LSM Tree + SSTable + MemTable (1) LevelDB RocksDB Distributed ZippyDB Bigtable/HBase/Cassandra Redis Dynamo Block Storage Search Engine MLHub Container Service Infra (SF) Data Infra Web Foundation Experiment Platform Routing LB Service Mesh Service Discovery
FB Docs: Core Data Wiki Counter Index Persistence TransientCache RTI Data tools
Systems: Leveldb -> rocksdb -> zippydb Shardmanager / datashuttle zeus/zookeeper Smc Multi-paxos (paxos-simple.pdf) Async Tier Logging infra Laser Live Videos (streaming) Manifold Everstore WarmStorage Gluster HDFS Predictor X Training platform
This video describes how to prepare for system design interview. Below is all the content described in the video. A - Ask good questions B - Don't use buzzwords C - Clear and organized thinking D - Drive discussions with 80-20 rule
Things to consider Features API Availability Latency Scalability Durability Class Diagram Security and Privacy Cost-effective
Concepts to know Vertical vs horizontal scaling CAP theorem ACID vs BASE Partitioning/Sharding Consistent Hashing Optimistic vs pessimistic locking Strong vs eventual consistency RelationalDB vs NoSQL Types of NoSQL Key value Wide column Document-based Graph-based Caching Data center/racks/hosts CPU/memory/Hard drives/Network bandwidth Random vs sequential read/writes to disk HTTP vs http2 vs WebSocket TCP/IP model ipv4 vs ipv6 TCP vs UDP DNS lookup Http & TLS Public key infrastructure and certificate authority(CA) Symmetric vs asymmetric encryption Load Balancer CDNs & Edges Bloom filters and Count-Min sketch Paxos Leader election Design patterns and Object-oriented design Virtual machines and containers Pub-sub architecture MapReduce Multithreading, locks, synchronization, CAS(compare and set)
Tools Cassandra MongoDB/Couchbase Mysql Memcached Redis Zookeeper Kafka NGINX HAProxy Solr, Elastic search Amazon S3 Docker, Kubernetes, Mesos Hadoop/Spark and HDFS
References https://docs.datastax.com/en/cassandr... http://cloudurable.com/blog/kafka-arc... https://zookeeper.apache.org/doc/trun... http://www.allthingsdistributed.com/f... https://research.google.com/archive/b... https://en.wikipedia.org/wiki/CAP_the... https://en.wikipedia.org/wiki/Consist... https://www.mongodb.com/mongodb-archi... https://en.wikipedia.org/wiki/HTTP/2 https://en.wikipedia.org/wiki/Transpo...