The commoditization of infrastructure is one of the most significant developments over the last couple of decades. The growth of web-scale companies like Google, Facebook, and Twitter (which collect, analyze, and extract information from a large volume of data) has influenced this commoditization of the infrastructure.
Looking back at the last couple of decades, enterprises have realized that the problems faced and solved by web-scale companies end up being problems in the enterprise after a short gestation time. Enterprises start seeing similar issues in scaling, management, and analysis of infrastructure, processes, and data. At multiple layers of the application stack, the enterprises adopt web-scale strategies to solve similar problems (Figure 1).
At the infrastructure layer, web-scale companies have focused on scale-out systems where compute, storage, and networking components blend into units of infrastructure that can be quickly replicated and grown without worrying about the complex organization of each of the components. Hyperconverged systems are an example of this trend.
At the data management layer, concepts like BigTable and MapReduce morphed from internal tools of web-scale companies to concepts and then to open source software and ecosystems. Every enterprise has a big data project using ideas and tools similar to the big data solutions practiced in web-scale companies.
However, at the infrastructure management layer, the adoption of web-scale tools and processes has been the slowest. Interestingly, even from the web-scale companies, the infrastructure management tools are the last ones being released and discussed. For example, MapReduce, BigTable, Spanner, Cassandra, DynamoDB, etc., were all discussed by Google, Facebook, and Amazon first. Google and Facebook also discussed how they build servers out of commodity hardware and promoted open compute. The infrastructure management tools such as Omega, Borg, Andromeda, and Tupperware were usually the last ones that were talked about publicly.
The reason for this is mainly because the infrastructure management was typically where more people were involved than software. Web-scale companies geared the rest of the stack to be able to use infrastructure automation because it is not possible to scale infrastructure with people anymore.
Hyperconverged systems were easier to adopt because enterprises are already used to heterogeneous hardware, and having some parts of the infrastructure populated with hyperconverged systems was cost effective and not a significant overhead to the IT team. One type of hardware was being replaced by another type of hardware.
The data management layer is mostly the domain of application developers and data analysts who are already very software-centric. Going from building solutions using a combination of database warehousing and ETL to big data solutions was still in the software engineers’ comfort zone. One type of software was being replaced by another type of software.
However, infrastructure management was always based on people or system administrators doing most of the heavy lifting. IT teams ran network operation centers where eyeballs were glued to the screen to watch graphs moving left to right and up and down. Web-scale companies and even public cloud companies have shown that managing infrastructure with software achieves significant economies of scale and reduces operations overhead. Adopting concepts like DevOps, infrastructure as code, API-driven infrastructure, Openstack, containers, etc., significantly lowers operational costs and the need for expertise.
Every enterprise from auto manufacturers to cosmetics is becoming a software house to build better products and analyze customer behavior. Developers are adopting cloud native application development, container technologies, and elastic applications at a rapid pace. To be competitive at these efforts, IT has to be an enabler of faster processes rather than being the bottleneck. Larger enterprises feel the pain of this first but it trickles down to every IT team sooner or later.
Enterprises and their CIOs have to expand their IT teams to deliver services in a more agile and scalable manner. This can only happen by adopting strategies where the infrastructure management layer uses software to manage the resources available. This includes not only hardware and provisioning but also operations. Software-driven operations are the key to moving beyond a people-driven infrastructure. It is not easy to go from a people-managed infrastructure to a software-managed infrastructure, but the economies of scale that can be achieved by doing so can be transformational to an enterprise.