Kubernetes container orchestration platform

GKE is predicated on Google's Kubernetes container orchestration platform. Kubernetes version 1.1, released Nov. 24, four months after 1.0 made its debut, was the primary on the market to autoscale pods with horizontal pod autoscaling, a feature highly sought by users to justify many use cases for GKE.
"We use the autoscaling quite bit for all kinds of projects," said Tim Kelton, co-founder and head of cloud architecture for Descartes Labs Inc., a machine learning startup based in Los Alamos , N.M., which processes petabytes of satellite data. 

Autoscaling pods are available handy running an outsized batch job, Kelton explained. At times, his company processes a petabyte of knowledge , which needs scaling up to 30,000 cores.virtualization technology within the first release of Kubernetes -- which was incorporated soon after by GKE -- "that wasn't a part of the core feature set," he said.

GKE doesn't support vertical container scaling or node autoscaling, but these features are coming soon, consistent with David Aronchick, senior product manager for GKE, who also leads product management for Kubernetes.
also leads product management for Kubernetes.

Spinning up an additional instance means you've got extra capacity to run additional tasks, but it doesn't suggest that any new tasks are going to be spun up.
Chris Moyer
vice president of technology with ACI Information Group
Amazon's EC2 Container Service (ECS), meanwhile, consists of services, tasks and instances.cloud technology Services are groups of tasks that structure an application, while instances are the Elastic Compute Cloud VMs that underpin the containers -- very similar to nodes in GKE.

Amazon ECS' autoscaling capabilities are the inverse of how it works with GKE: Services are often autoscaled using Amazon CloudWatch and Amazon Web Services (AWS) Lambda, and instances are often autoscaled supported CloudWatch metrics also , but tasks -- the rough logical equivalent of pods -- can't be autoscaled.

While all the kinds of autoscaling are important, Amazon users want task autoscaling added to ECS.

"Spinning up an additional instance means you've got extra capacity to run additional tasks, but it doesn't suggest that any new tasks are going to be spun up," said Chris Moyer, vice chairman of technology with ACI Information Group, an internet content aggregator based in ny , and a TechTarget contributor. information technology education"If you're only autoscaling your instances, it isn't really doing anything to assist you handle extra load -- you've got to truly spin up extra tasks to scale out."

Redundancy across zones

In the development of ECS, Amazon prioritized the power to natively span availability zones (AZs) within the same cluster for redundancy over task autoscaling supported customer demand. When the ECS service scheduler launches new tasks, it also attempts to balance them across the AZs during a cluster automatically.

"That's important, because one AZ is allowed to fail, so if both tasks were within the same AZ, that would easily take down your service," Moyer said.

Google can span multiple zones in GKE through the command-line interface (CLI), consistent with Google's Aronchick.

"It's very easy -- two or three commands," he said. 

However, this touches on GKE customers' biggest list item: improvements to the online UI, including scaling clusters across zones.

"The UI needs plenty of labor ," said Dale Hopkins, chief architect at Vendasta Technologies in Saskatoon, Sask., which builds sales and marketing software for media companies. The UI currently allows for cluster creation and tiny more, Hopkins said. "And it's non-intuitive how you scale the cluster."


ECS was built as an extensible platform, designed to be dropped into a customer's existing workflow, mainly to handle cluster state on users' behalf. a part of this integration into existing workflows accommodates tools that customers already use, like Apache Mesos for advanced scheduling. Amazon also boasts an in depth network of Container Partners that contribute features, like monitoring, continuous integration and security, to Amazon ECS.

Google, meanwhile, has built a coalition of cloud containers partners that allow Kubernetes to be deployed across multiple cloud providers -- also a CLI feature today, consistent with Aronchick. Google led the creation of the Cloud Native Computing Foundation when Kubernetes 1.0 was released last summer. Foundation members include enterprise cloud services companies, like IBM and Red Hat, also as end-users Box, eBay and Twitter.

"[With] Kubernetes, I can actually go deploy on Amazon, I could deploy on Azure, I could deploy on IBM, I could deploy on premises on my very own physical hardware," Descartes' Kelton said. "That's very attractive, since we've options."

Google also has an open source project, with many committers and thousands of commits a month, allowing Kubernetes to quickly add new features, like horizontal pod autoscaling.

"Google is that the origin of Kubernetes, and Google's done a very good job enlarging that community," said Jay Lyman, analyst with 451 Research. 

The rich get richer

Still, integration with established and familiar secondary Amazon services makes Amazon ECS particularly appealing for brand spanking new customers.

One New York-based company that consults with large enterprises thereon projects plans to use ECS in two new projects, consistent with its founder, John D'Esposito. "The main advantages that drove us to use ECS [included] seamless integration with existing, proven infrastructure services, like [Elastic Load Balancing, Virtual Private Cloud, Identity and Access Management, and Elastic Block Store]."

GKE and Compute Engine pricing can also be attractive to customers. additionally to charging in 10-minute increments for underlying VM resources, GKE includes the Kubernetes master for free of charge -- something that particularly appeals to Vendasta's Hopkins.

"I don't pay a premium for Kubernetes until i buy into huge numbers of machines -- GKE offers me the Kubernetes master for free of charge for the primary set of machines," he said.

Both Hopkins and Kelton already used Google cloud services, including Google App Engine, before Kubernetes and Container Engine were introduced. Thus, data gravity also plays a task during which cloud containers service they prefer to deploy.

"Most of our data sets are within the petabyte scale, so you cannot just move them or copy them, you've got to truly move the compute next to the info ," Kelton said. Most of that data currently lives within the Google Cloud Platform, though Descartes does work with partners in AWS.

Microsoft Azure Container Service waits within the wings

While Google and AWS are at the forefront of the cloud containers battle thus far , Amazon's closest competitor remains Microsoft Azure, which has its own Linux-based cloud containers service in limited preview, also as a replacement version of Windows Server due out this year which will support Windows-based containers.The majority of our clients ... are either in Azure or in Amazon," said Chris Riley, a founding partner at HKM Consulting LLC, in Rochester, Mass. "[Microsoft] possesses some interesting tools that they are developing. If we were to seem at a secondary one, it might probably be Azure before Google."As with many Microsoft products, simplicity and simple use are the planning priorities, consistent with Kristian Nese, CTO of Lumagate, a Microsoft Azure systems integrator in Norway.

"When we're deploying the Azure Container Service today, it's 100 lines of code," Nese said. "Once you've got deployed the Azure Container Service, you really have 23 resources deployed ... if you'd do that manually, it might presumably end in many thousands of lines of code."

The Azure Container Service also has autoscaling within the works within the sort of a separate service also in preview, called VM Scale Sets.

Azure also will have the advantage of offering established and familiar tools to manage containers, like Azure Resource Manager, Nese added.

Is big data too big for cloud-based analytics?As cloud-based analytics are exposed to ever-increasing volumes of knowledge , the pressure is on to deal the onslaught. Consider it a tug-of-war with the necessity for near-limitless scalability pitted against what proportion of that collected data actually ever gets scrutinized.When it involves capturing data for cloud-based analytics, "big" doesn't even compared to being an adequate characharacterization. In October 2015, IDC reiterated previous research noting that the quantity of knowledge created annually is predicted to grow from 4.4 zettabytes in 2013 to a whopping 44 zettabytes (44 trillion gigabytes) in 2018 worldwide -- a rate of growth of an astonishing 40% per annum .

Without a doubt, businesses are handling huge amounts of knowledge . At Weather Underground, the Weather Channel affiliate whose digital assets are within the process of being acquired by IBM, weather readings within the us are collected from quite 180,000 weather stations every quarter-hour , generating in more than 100 gigabytes of knowledge per day.

Media giant Time Warner Cable (TWC) tracks every navigational move and button click that its quite 15 million customers make when using the company's mobile apps and website. A customer of the cloud-based Adobe Analytics service, TWC rid itself of handling torrents of incoming data and storage overhead by completely offloading to Adobe's analytics as a service (AaaS).

Despite TWC's large size, as long because the Adobe service has bigger customers, no cause for concern exists, consistent with Jeff Henshaw, TWC's senior director of business intelligence. "For TWC, scalability isn't an element ," he said. "Our data sizing doesn't approach what another organizations have in situ . Until they hit the ceiling, we've no worries about data limitations, performance or network latency."

Jeff Morris, vice chairman of strategy at analytics services provider GoodData, agreed that scalability should not be a priority . "As long as you point your data pipeline to me, we will scale into many terabytes," he said.
t would appear logical that within the world of cloud-based analytics, more data is best . That's not always the case. Given the rocket-like velocities and colossal volumes of incoming data of varied formats, capturing, normalizing, and storing all bit are often an upscale -- and sometimes unnecessary -- endeavor.

"There are some sorts of data that get more valuable over time," said Mike O'Rourke, IBM's vice chairman of business analytics. For vineyards, historical data on grape growth, weather, climate, harvest and other factors could also be valuable when analyzed over multiple decades. But, O'Rourke noted, after a period of years, storing daily high and coldness readings could also be sufficient compared with the expense of keeping all readings taken quarter-hour apart.

Similarly, within the retail industry, eventual aggregation of detailed minute-by-minute sales data into daily, weekly and monthly roll-ups could also be sufficient for analytic purposes as time passes. "It totally is sensible to aggregate data [as it ages]," O'Rourke said. "That's need to be a neighborhood of the general plan." Though the value per terabyte of storage continues to say no , unless data is aggregated because it ages, actual expenditures will still rise thanks to continuous amassing of latest data, he said.

While cloud data-storage providers work to expand storage capacities, the good irony is that the miniscule percentage of collected data that's ever accessed for analytics purposes.

John Bates, group product manager at Adobe Analytics, estimated current data-access rates at but 2%. "Users tend to focus in on the info that's most-directly associated with how they account for fulfillment or key performance indicators," he said.

O'Rourke's estimate is even lower. "In terms of the info [IBM is] pulling for patrons and therefore the things they're watching ... it's definitely but 1%."

The good news is that those low data-access rates are likely to climb as analytics algorithms evolve to the where they will discover trends in data that users were never looking within the first place. "As we still see advancements in analytics, machine learning and AI , I see our ability to scale the quantity of data being analyzed and leveraged growing much larger," Bates said.