《CCTC2016-才云科技邓德源-谷歌容器集群管理系统实践.doc》由会员分享，可在线阅读，更多相关《CCTC2016-才云科技邓德源-谷歌容器集群管理系统实践.doc（34页珍藏版）》请在三一文库上搜索。
1、CCTC2016-才云科技邓德源-谷歌容器集群管理系统实践谷歌容器集群管理系统实践 邓德源 才云科技 Engineering in Google ? Development ? 20% time side project ? work on any project you want ? dashboard for posting jobs ? you can do more than you think! ? Be SRE for 2 or 3 months (20% more salary!) ? SRE (Site Reliability Engineering) ? 50% time f
2、or development ? to automate routine tasks ? scale nicely ? SRE consists of two categories: 55% SDE, and 45% Ops ? Say no to development team ? error quota, e.g. 99.95% slo monthly means 4hrs impact ? both SRE and Development team mange the risk ? Ability to change code base ? Postmortem ? Not to bl
3、ame, but to find problems and fix it Container ? Reason ? primary goal: save money - VM is heavy ? high-density and performance ? fast to start ? Start Container Journey ? 2004 - ? ? Everything runs in container ? use container for decade ? >2B container a week Limited Isolation chroot cgroups lx
4、c lmctfy 2004 2006 2008 2013 user namespace 2014 2007 namespaces? 2015 OCI Container Image from Redhat Limited Isolation chroot cgroups lxc lmctfy 2004 2006 2008 2013 user namespace 2014 2007 namespaces? 2015 OCI ? cgroup ? resource isolation (cpu, memory, blockio, etc.) Container Limited Isolation
5、chroot cgroups lxc lmctfy 2004 2006 2008 2013 user namespace 2014 2007 namespaces? 2015 OCI ? cgroup v1 -> v2 ? multiple hierarchy ? subcontainer and asymmetric isolation ? v2: unified hierarchy Image from Redhat sub1 sub2 sub3 subcontainer CPU Mem Net container1 container2 container3 asymmetric
6、isolation cg1 cg10 cg11 cg100 cg101 . Container Limited Isolation chroot cgroups lxc lmctfy 2004 2006 2008 2013 user namespace 2014 2007 namespaces? view of the operating environment ? Mount, UTS, IPC, PID, Network, User ? Support added fair recently ? Primarily Linux cgroups ? namespace is done at
7、user-space policies Container ? lxc? lmctfy? ? internal version of Imctfy exists long before lxc ? lxc: no strong abstraction ? namespace abstraction + raw cgroup ? lxc: no programmable API ? must be built to work with other tools ? lmctfy: more abstraction and enhancement ? e.g. subcontainer and as
8、ymmetric isolation ? e.g. quality of service ? OCI ? Container: runtime + image ? Based on Docker Limited Isolation chroot cgroups lxc lmctfy 2004 2006 2008 2013 user namespace 2014 2007 namespaces? 2015 OCI Container? ? Container management ? clustering is the hard part ? hundreds of engineerings b
9、uilding cluster management system Borg: concepts ? Configuration ? Borg configuration language ? Most hated language ? Job and Task ? Job: unit of deployment, including resources requirement, number of tasks, etc ? Task: unit of running entity ? Resource container: the container to run task Job Task
10、1 Task2 Borg: concepts ? AllocSet and Alloc ? co-scheduling: e.g. logsaver and application ? share resources for all tasks ? persist data when tasks exit Job Task1 Task2 host bind mount /cg11 /cg12 Borg: concepts ? Scheduling ? priority ? resources: resource has priority ? quota: quota has priority
11、? packages: vs docker image ? machine constraints ? %ports% (BNS, chubby) ? options: ? repeated failures ? quorum requirement ? etc ? Runtime ? application class ? resource estimation: ? decrease resource reservation (gradual decay function) borg config file borgcfg 1 RPC/Protocol Buffer Primary Bor
12、gMaster scheduler server task1 pkg1 2 Paxos 4 pkg2 resource container Borglet 5 6 task_N resource container Borglet 7 polling installing package Borg Cell 8 run + args + resource 9 3 AuthN/Z Borg: Request Lifecycle link shard Borg: The Good, The Bad, The Ugly ? 高稳定性、高自动化、高智能 ? 极其复杂的配置文件 最流行语言排行榜？ Go
13、, Python, Java, C+, Javascript, 最令人畏惧语言排行榜？ Python, Borgcfg, Borgmon, Shell, ? Borgmaster becomes borg monster ? People step on each others foot Omega: 早年寄予厚望 ? From Monolithic to Shared State ? Persistent Storage as the ground truth ? Look to the Future! ? Collaboration across the globe ? A bottom
14、up driven project Omega: 中年四面楚歌 ? Engineering mayhem ? Testing cell ? API rewrites, all Borg callers have to change (along with behavioral change)! ? Borgmaster is hard to rewrite ?向现实妥协： ? 项目进度 ? 与Borg的关系 Omega: 晚年戛然而止 ? 工程与理论的差距 ? 精确度 vs 吞吐量 ? 预测的困难 ? Borg的进化 ? 性能 ? 软件工程 ? 终究下架 ? Big shuffle Cloud
15、: The Ursquake ? 经济上的巨大回报 ? 技术上的巨大领先 ? 产品化上的差异 ? PaaS与IaaS的矛盾 ? GAE vs GCE ? Managed VM Kubernetes ? 此处为何地? ? Google Mountain View ? Google Pittsburgh ? Kubernetes birth place ? kubernetes乳名? Kubernetes yaml config file kubectl 1 HTTP Single Master scheduler API server con1 2 etcd 4 Kubelet Kubernet
16、es Cluster 3 AuthN/Z controller manager con2 infra 5 6 7 installing package watch post con1 Kubelet con2 infra ? 一个Cluster里有一个或多个Borg cell ? Borg容器之外的其他任务 ? Borg自身谁来管理？ ? 机器层面谁来管理？ ? 网络层面谁来管理？ ? 安全谁来控制？ ? 镜像如何管理？ Cluster management Borg Hareware Ops流程工具 自动系统安装工具 在线硬盘修复系统 生产系统维护管理系统 节点医生 网络医生 耗竭系统 集群
17、工作流管理系统 日历系统 Capacity安全把控系统 分布式存储 集群数据库 用户和组管理系统 SDN网络 自动节能系统 负载均衡 资源分配扩容系统 集群监测系统 安全扫描系统 分布式构建系统 分布式测试系统 调试系统和工具 调试系统和工具 发布管理工具 The entire family And . The Robot ? Disk Erase ? Decommission ? Inventory Management An example workflow: Pre-borg 集群工作流管理系统 自动系统安装工具 集群数据库 节点管理agent Borg driver Chubby dr
18、iver 节点医生 节点医生agent An example workflow: with borg Borg(let/master) 集群工作流管理系统 (状态机) Borg driver 节点医生agent 节点医生 集群数据库 Capacity安全把控系统 自动系统安装工具 在线硬盘修复系统 HWops 节点管理agent 耗竭系统 迁移工具 An example workflow: more on the dev side Borg 本地代码库 分布式构建系统 分布式测试系统 云端代码库 代码检查平台 BorgCLI 资源分配系统 监测系统 配置管理系统 发布工具 镜像管理工具 Thanks!