Helping media companies create consumer streaming formats with Transcoder API

Media and entertainment companies across the world are in the midst of a transformational shift to direct-to-consumer (D2C) streaming experiences. With audiences sheltering in place in 2020, this shift has been accelerated as audiences have adopted new streaming services more readily in 2020 than ever before. With more content being streamed in higher resolutions and operations becoming more distributed, media companies today require a cost efficient, high speed, and scalable way to process and deliver video content to an ever increasing number of end devices and platforms.Google Cloud is committed to building industry products and solutions that help media companies simplify operations and distribute their content to the widest possible audience. We’re building a set of video-focused APIs that empower developers to easily build and deploy flexible high quality video experiences. Today we’re announcing the first of these products, the preview availability of the Transcoder API. The Transcoder API is an easy-to-use API for creating consumer streaming formats, including MPEG-4 (MP4), Dynamic Adaptive Streaming over HTTP (DASH, also known as MPEG-DASH), and HTTP Live Streaming (HLS). Many D2C streaming platforms use a multi-codec strategy, and the Transcoder API supports popular codecs, including H.264, VP9, and HEVC. This strategy allows providers to offer a better, high definition experience to more viewers. The API also supports full partitioning for fast encoding of large video files, meaning that entire hours-long movies can be prepared in minutes.Developers can get started quickly by submitting transcoding jobs through REST API, transcoding files in Google Cloud Storage, and using Google Cloud CDN or third party CDNs to effectively distribute content to audiences across the globe. To learn more about the Transcoder API, please visit the documentation and pricing pages.Related ArticleHelping media companies navigate the new streaming normalAs media and entertainment companies evolve their future plans as a result of COVID-19, they should keep new audience behaviors top of mi…Read Article
Quelle: Google Cloud Platform

A developer’s guide to Google Kubernetes Engine, or GKE

When people think about whether or not to deploy on a container management platform like Kubernetes, the decision often comes down to its operational benefits: better resource efficiency, higher scalability, advanced resiliency, security, etc. But Kubernetes is also beneficial to the software development side of the house. Whether it’s improved portability of your code, or better productivity, Kubernetes is a win for developers, not just operators.For one thing, as we argued in Re-architecting to cloud native: an evolutionary approach to increasing developer productivity at scale, Kubernetes makes it easier to adopt modern cloud-native software development patterns like microservices, which can give you:Increased developer productivity, even as you increase your team sizes. Faster time-to-market – Add new features and fix defects more quickly. Higher availability – Increase the uptime of your software, reduce the rate of deployment failures, and reduce time-to-restore in the event of incidents. Improved security – Reduce the attack surface area of your applications, and make it easier to detect and respond rapidly to attacks and newly discovered vulnerabilities.Better scalability – Cloud-native platforms and applications make it easy to scale horizontally where necessary—and to scale down too. Reduced costs – A streamlined software delivery process reduces the costs of delivering new features, and effective use of cloud platforms substantially reduces the operating costs of your services. Google of course invented Kubernetes, which Google Cloud offers as the fully managed service, Google Kubernetes Engine (GKE). But did you know that Google Cloud also offers a full complement of developer tools that are tightly integrated with GKE? Today, in honor of KubeCon, we’re revisiting a few blogs that will show you how to develop apps destined for GKE, how to deploy them safely and efficiently, and how to monitor and debug them once they’re in production. Developing for GKE: It all starts with youEven the most enterprise-y applications get their start in life on a developer’s laptop. The same goes for applications running on GKE. To make that possible, there’s a variety of tools you can use to integrate your local development environment with GKE. Developers are known for tricking out their laptops with lots of compute resources. Using Minikube, you can take advantage of GPUs, for example. There are also local development tools to help you containerize Java apps: Jib, and Skaffold. Jib helps to containerize your Java apps without having to install Docker, run a Docker daemon, or even write a Dockerfile, and is available as a plugin for Maven or Gradle. Then, you can use Skaffold to deploy those containerized Java apps to a Kubernetes cluster when it detects a change. Skaffold can even inject a new version of a file into a running container! Read about this in depth at Livin’ la vida local: Easier Kubernetes development from your laptop.Another popular tool among GKE developers is Cloud Code, which provides plugins for the popular Visual Studio and IntelliJ integrated development environments (IDEs) to simplify developing for GKE. For example, we recently updated Cloud Code to have much more robust support for Kubernetes YAML and Custom Resource Definitions (CRDs). Read more at Cloud Code makes YAML easy for hundreds of popular Kubernetes CRDs. Have a quick and dirty development task to do? Check out Cloud Shell Editor, which launches a full-featured, but self-contained, container development environment in your browser. Read more at New Cloud Shell Editor: Get your first cloud-native app running in minutes.Related ArticleLivin’ la vida local: Easier Kubernetes development from your laptopRunning applications in containers on top of Kubernetes is all the rage. However, the brave new world of containers isn’t always kind to …Read ArticleGet in the (pipe)lineEventually, you’ll be ready to push the apps you developed on your laptop to production. Along the way, you’ll probably want to make sure that the code has been properly tested, and that it passes requisite security and compliance tests. Google Cloud offers a variety of tools to help you push that code through that pipeline. Setting up an automated deployment pipeline to GKE doesn’t have to be hard. In Create deployment pipelines for your GKE workloads in a few clicks, learn how to use Cloud Build to create a pipeline from scratch, including selecting your source, build configuration, and Kubernetes YAML files. But before you do, make sure that the image that you’re deploying is secure. Binary Authorization provides a policy enforcement chokepoint to ensure only signed and authorized images are deployed in your environment. You can read more about it in Deploy only what you trust: introducing Binary Authorization for GKE.Even better, Artifact Registry has built-in vulnerability scanning. Once enabled, all container images built using Cloud Build are automatically scanned for OS package vulnerabilities as they’re pushed to Artifact Registry. Read more at Turbocharge your software supply chain with Artifact Registry.Monitor, Debug, Repeat: Remote development for GKE appsNow that your app is in production on a GKE cluster, your work is done, right? Wrong. For developers, getting an app to production is still just the beginning of the software lifecycle. Chances are, you have ideas about how to improve your app, and you’ll definitely want to monitor it for signs of trouble. GKE is tightly integrated with several monitoring, debugging, and performance management tools that can help ensure the health of your GKE app—so you can make them even better!When there’s a problem in your production environment, one of the first places you’ll want to look is your logs. You can do that with Cloud Logging and Cloud Monitoring, both enabled by default when you create a GKE cluster. To learn more about how to use Cloud Logging for GKE logs, use cases and best practices, check out Using logging for your apps running on Kubernetes Engine.Once you’ve found the culprit, find out how you can use Cloud Logging and Cloud Monitoring to debug your applications.We’re developers tooAs long-standing leaders of the open source community, including the Cloud Native Computing Foundation (CNCF) and Open Container Initiative (OCI), we’re always thinking about how industry developments impact your day-to-day as a GKE developer. For example, Docker’s recent announcements about new limits on pull requests prompted us to write this post on how to manage these restrictions in a GKE environment. In addition to making GKE the most scalable and robust container management platform, we’re deeply committed to making it the easiest to use and develop on. New to Kubernetes and GKE? Learn more with this free, hands-on training. And if you’re participating in KubeCon this week, be sure to stop by our (virtual) booth to meet an expert.Related ArticleYour guide to Kubernetes best practicesOur entire Kubernetes best practices blog series in one location.Read Article
Quelle: Google Cloud Platform

Four ways to generate value from your APIs

It’s been 20 years since Jerry Maguire hit movie theaters in 1996, and yet one line from the movie still resonates like no other. The constant banter between the agent, Jerry Maguire (Tom Cruise) and budding professional football player Rod Tidwell (Cuba Gooding Jr.) is fun, if not bombastic, with one central theme that Tidwell not only enthusiastically expresses but also insists Maguire express as well:“Show me the money.”Tidwell’s passion for football was far outweighed by his passion to become rich; indeed, one of the movie’s themes is that only once he got past playing for the money could he achieve the level of play required to drive real value to his team, his team’s owners, and ultimately himself.What does this have to do with API management? Digital product owners tend to follow a similar path: the knee-jerk reaction is that the easiest way to drive value from APIs is to charge for them. While this may be the easiest thing to do, more often than not, API value is best extracted from other indirect means. By creating a tri-partite value exchange—a proposition that satisfies end users, partner developers, and the company publishing the APIs— a great amount of untapped value may be mined. And just like the football player, the agent, and the team, putting some heart into the game can make all three of them winners.How to start deriving value from APIsBehold: here are the four best practices to derive value from APIs:1. Extend channel reachSuppose your application is great but targeted toward a specific set of users. What if there’s a set of users—perhaps even an entire channel—that cannot use it? Perhaps your application doesn’t integrate well with other corporate systems, or perhaps it isn’t available in certain markets, doesn’t accept certain currencies, or can’t support certain business models (such as pre-paid or post-paid). Creating an API product—that is, an API designed for developer consumption and productivity, not just integration between systems—is the single best way to make your application flexible enough that the functionality can be adopted into channels that aren’t being addressed by your current go-to-market approach. An excellent example of this is the Walgreens Photo Prints API. As photos have moved from digital cameras to mobile phones, a cadre of third-party photo applications has cropped up…yes, pun intended. These applications took great pictures and featured wonderful effects but offered no easy way to print the photos. By leveraging the API and the connection it facilitated to Walgreens photo printing facilities in stores nationwide, these apps can now use Walgreens stores as a venue for photo printing. This has enabled customers to quickly get prints of their favorite photos, helped developers to build richer apps, and let Walgreens photo services go well beyond the store, embedding a presence in a multitude of apps and handsets they wouldn’t have addressed without a productized API. Walgreens has turned their developer ecosystem into channel partners and now offers them much more than just photo printing services.In this model, the API product is offered for free, as there is an obvious value proposition to all three parties—API publisher, developer, and user. (Learn more about how Walgreens uses Apigee.) 2. Consider brand awareness and promotionYour application is lost in a sea of hundreds of other similar apps in an app store. What now? One way of driving awareness is to extend your branch reach/footprint via an API, then reward users or developers for sign-ups and usage, in order to proliferate the application to new surface areas, experiences, and form factors. Streaming services, for example, generally have an incentive to make their streaming players easy to integrate across a wide variety of devices, form factors, and digital experiences. This can in turn create an incentive among device-makers and app-makers to integrate the service, creating a potentially exponential increase in the value proposition for the end consumer and the developers integrating the streaming player API. And when this variety of viewing options for the user meets quality content from the service, the result can be a self-reinforcing cycle of more subscribers and increased reach across more consumer touchpoints. Developers, device-makers, and the service publishing the API create ways to make money, and end users get a steadily-improving and flexible service.Similarly, Brazilian retailer Magalu (formerly Magazine Luiza) leveraged APIs to achieve, as CTO Andre Fatala put it in a 2019 blog post, a “newfound ease and speed of spinning up new services and customer experiences and adjusting existing ones,” which let “everyone … work in small teams of five or six people that take care of segments of an application, whether it’s online checkout, physical store checkout, or order management.” The approach means Magalu “work[s] much more like a software company than a retail company now,” he said.With these new agility, the company has expanded its e-commerce strategy to third-party sellers and created a digital marketplace that lets merchants easily join the ecosystem via Magalu’s API platform. Whereas the company’s old legacy sales and distribution systems only supported 50,000 SKUs, the marketplace supports thousands of sellers and millions of SKUs, significantly expanding the brand’s reach.3. Enable customization to create new value propositionsIn a bid to create an ecosystem around its banking products, ABN Amro partnered with telecommunications company KPN, smart homes expert 50five, and Energizer—and begun leveraging its first-party payments app Tikkie, the Olisto IoT device triggering platform, and the Google Nest API—to create an entirely new value proposition around smart home solutions. Thanks to this collaboration, when the Nest Protect smart smoke alarm runs low on batteries, it can order replacements automatically, performing the payment through Tikkie and triggering the batteries to be delivered directly to the owner’s home. This capability significantly reduces risk of a smoke alarm not working due to battery failure—and with Google currently rolling out new Nest initiatives to replace the program under which ABN Amro’s solution was created, we’re excited to see what novel and convenient user value propositions our APIs enable in the future. These API partnerships let ABN Amro position their offering as a flexible platform, able to generate network effects to aggregate demand. As new value propositions are created in these many ecosystems, ABN Amro will already be integrated and available to meet the needs of the ecosystem. The company’s flexible, customizable platform will offer the path of least resistance for future similar API partnerships.In this model, the API product is offered for free, as there is an obvious value proposition to all three parties—the API publisher, the bank, is a more desirable destination to keep money (and do business with) because of its flexibility; the developer drives value and differentiation to their product by alleviating a significant consumer pain point; and the end user benefits from having a functioning smoke detector.4. Enable access to rare and valuable competenciesThere are situations in which the best way to generate value from an API is to charge for it. If the API product substitution risk is low—if your value proposition is rare, valuable, and is not being competed against by indirect competitors—there is an opportunity to generate direct value from the API by charging for access. A great example of this is telecommunications infrastructure APIs, such as telematics APIs for in-vehicle connectivity. Though there are other telecommunications providers likely competing, the threat of other non-telecom providers offering in-vehicle applications and connectivity—i.e., the product substitution risk—is very low. As a result, telecoms can charge for those API products, as they are rare and valuable. Reducing the importance of API value generation to simple revenue generation or cost savings frequently misses the key sources of untapped value available when targeting developers with an API program. This ability to inspire developers—not unlike Gooding’s Rod Tidwell character rediscovering the drive that inspires people to watch football—is the bedrock of the internet economy, the “demand aggregation” model that enables ecosystem effects and significant value generation. When considering an API productization approach, make sure to have all of these in mind as you map your path to success. To learn more, check out our ebook on API productization.Related ArticleGoogle named a leader in the 2020 Gartner Magic Quadrant for Full Life Cycle API ManagementFor the fifth year in a row, Google Cloud (Apigee) has been named a leader in Gartner’s Full Life Cycle API Management Magic Quadrant.Read Article
Quelle: Google Cloud Platform

The 10 most popular sessions from Google Cloud Next ‘20: OnAir

Google Cloud Next ‘20: OnAir looked a little different this year. Instead of a three-day conference, we launched a nine-week digital event series that brought together our global cloud community to discuss and collaborate on the most significant cloud technology challenges facing companies today. We may have gone virtual, but some things stayed the same—all of our sessions (over 200 to be exact) are now available on YouTube to watch and learn from. That’s a lot of content to sift through, so here’s a quick breakdown of our top 10 sessions from Next OnAir: 1. Supercharge Productivity With No-Code Apps Using AppSheetThe road to building new applications in the digital age comes with a tough choice for business and technology leaders: buy or build. Packaged software is often too rigid to meet unique requirements, and building custom apps takes up too much time and resources. But is there a third option? In this session, Santiago Uribe Montoya, Google Senior Product Manager, and Richard Glass, Director of Information Technology at KLB Construction, discuss how AppSheet makes it possible to automate processes while leveraging existing Google Workspace data to build mobile and desktop apps—without coding. 2. How PwC Migrated 275,000+ Users to Google WorkspaceEver wondered what it’s like to migrate to Google Workspace (formerly G Suite)? What about migrating over 275,000 users at a 150-year-old professional services firm operating in over 158 countries? PwC Global Change Director Adrienne Schutte, along with Google Technical Account Manager Regina Houston, shares the challenges, key lessons, management strategies, and long-term impact of PwC’s journey with Google Workspace.3. The Future of Meetings in Google Workspace: Vision and RoadmapThe new normal is here—and so is the new work normal. In this session, Smita Hashim, Director of Product Management for Meeting Solutions and Time Management, and Greg Funk, Group Product Manager, share Google Workspace’s vision for the future of meetings as teams navigate an increasingly video-first world. You’ll also get a sneak peek into how Google Workspace is transforming the lifecycle of a meeting and reimagining teamwork so that people can stay connected no matter where they are working from.4. Do it live! Fitbit’s Zero-Downtime Migration to GCPMoving a monolith without downtime is impossible, right? Think again. In this session, Fitbit’s Principal Software Engineer Sean Michael-Lewis explains how Fitbit migrated its production operations from managed hosting to Google Cloud Platform without impacting its real users. You’ll learn what made Fitbit’s migration challenge unique, how they created a user-centric migration plan, the technology and processes they used, and the key takeaways that have provided a foundation for their new multi-region architecture. 5. What’s New in BigQuery, Google Cloud’s Modern Data WarehouseData is at the heart of many business transformations today. Organizations want to make real-time decisions and future predictions that keep them competitive, but traditional data warehousing wasn’t designed to scale fast or process emerging data processing patterns. In this session, Sudhir Hasbe, Google Cloud Director of Product for Data Analytics, and Tino Tereshko, Google Product Manager, talk about how Google BigQuery addresses the needs of data-driven enterprises and share demos of the latest feature innovations.6. Communication in Google Workspace: The Future of Gmail, Chat, Meet, and MoreHow is communication changing as work goes remote and becomes more flexible? See how new improvements to Gmail, Chat, and Meet are making it easier for modern workers to communicate and collaborate anywhere they are working from—on the web or from an Android or iOS device. In this session, Tom Holman, Google Senior Product Manager, and Dave Loxton, Google Product Manager, share the latest updates, what’s up next, and why Google is more excited than ever about the future of these products in Google Workspace.7. Building Data Lakes on Google CloudTraditional approaches to building data lakes often land organizations with data swamps.   In this session, Google Product Manager Nitin Motgi discusses how Google Cloud makes it easy for enterprises to create and maintain data lakes, allowing customers to aggregate their data and analyze it using cloud-native and open source tools. He’ll also share the most common use cases for how companies use data lakes on Google Cloud.8. Data Catalog for Data Discovery and Metadata ManagementWouldn’t it be great to be able to easily search through your enterprise data assets with the same search technology that powers Gmail and Drive? Many enterprises struggle with data discovery and metadata management across disparate systems and silos. Shekhar Bapat, Google Product Manager, discusses how Data Catalog helps accelerate time to insight by providing discoverability, context, and governance for all of your data assets. He is joined by Shruti Thaker, Head of Alt Vendor Data & Alpha Capture at Blackrock, who shares how Data Catalog helped BlackRock create an effective metadata solution for its data assets. 9. Analytics in a Multi-Cloud World with BigQuery OmniWhile data is a critical component of decision making across organizations, for many, this data is scattered across multiple public clouds. So, how do you help analysts and data scientists handle data from all the tools, systems, and silos? Meet BigQuery Omni, a flexible, fully-managed, multi-cloud analytics solution that lets you analyze data across public clouds without ever leaving the familiar BigQuery user interface. In this session, Google Product Manager, Emily Rapp, shows you how to break down data silos right in your environment and run analytics in a multi-cloud world.10. Master Security and Compliance in the Public CloudUnlocking the promise of the public cloud often brings security and compliance challenges—especially if you’re a leading market-infrastructure provider under the highest level of supervisory scrutiny in Europe. In this session, Christian Tüffers, Senior Cloud Architect at Deutsche Boerse Group, and Grace Mollison, Google Head Cloud Solution Architect, discuss ways customers can work with Google Cloud to create security blueprints to help them deploy workloads that meet regulatory and compliance requirements. Do you want to watch more sessions? You can browse the full session roster from Google Cloud Next ‘20: OnAir here.Related ArticleGoogle Cloud Next ‘20: OnAir—Accelerating digital transformation in the cloudGoogle Cloud CEO Thomas Kurian shares his vision for the future of cloud as we kick off Google Cloud Next ‘20: OnAir.Read Article
Quelle: Google Cloud Platform

How BigQuery helped Theta Labs and NASA bring science and hope to streaming

Editor’s note: We’re hearing today from Theta Labs, a leading decentralized video streaming platform that is powered by users and decentralized on a new blockchain. With their peer-to-peer bandwidth sharing distributed ledger technology, Theta Labs has been able to revolutionize the livestream experience. By adopting Google Cloud, Theta Labs has been able to scalably meet a growing active user base on their blockchain platform which in turn, has helped them expand their strategic partnership with NASA, including hosting the latest SpaceX rocket launch.When we established Theta Labs back in 2016, the goal was to set up a streaming video service with an emphasis on rendering popular PC video games like League of Legends, CS:GO and Dota2 into immersive 360° virtual reality experiences. And yet, thanks to our unique approaches to streaming, video rendering, and patented blockchain video technology, we’ve grown into something so much bigger that we’ve even caught the attention of NASA. All of this was possible thanks to Google Cloud and their databases and analytics products such as BigQuery, Dataflow, Pub/Sub and Firestore. Reaching the heights of video streamingBack when we first launched Sliver.tv—now Theta.tv—we decided to differentiate ourselves from the competition by creating unique live streaming video experiences, especially for streamers and viewers in regions with little or no access to high-speed internet. Our blockchain-based peer-to-peer video delivery technology lets users share their bandwidth with others, letting our streamers reach audiences they never could before.It was this ability to reach more unique and remote viewers and to give larger audiences the opportunity to discover new things that caught NASA’s attention. NASA saw the potential in our service to spread interest in science and technology to an audience of mostly younger viewers. They gave us the privilege of becoming one of only four or five video services with direct access to NASA’s source video feed, and we recently collaborated to premiere NASA’s August Women’s Equality Day  broadcast.The biggest highlight of this partnership so far was the opportunity to livestream the SpaceX launch. In a year where we all needed a bit more hope, being able to bring the live launch of a space shuttle to a wider audience was an amazing experience, inspiring so many to reach for the stars.Video of the stars starts with the cloudTo facilitate an event as large as a space launch with so many viewers takes a powerful infrastructure. To do all of this with our unique peer-to-peer blockchain system that rewards viewers and streamers for sharing bandwidth, we needed Google Cloud’s reliable, scalable, and stable infrastructure. With the strength of Google Cloud and their help on creating auto-scaling DevOps solutions, we were able to reach more viewers than ever without hitting the VM caps that previously caused issues with latency and customer experience. Previously, we’d faced challenges like infrastructure scaling limitations to meet application demands, high costs, and too much of our time wasted on managing and maintaining solutions.Google Cloud offers us better scalability, so we’re no longer capped by the number of active streamers we could have on our platform. Google Cloud gave us:Performance, and flexibility of implementationBreadth of capabilities and supportAbility to ingest streaming data for real-time insightsRelationship & communication with the Google account teamExpansive feature optionsPrice point compared to offered features/servicesOur partnership with Google Cloud has also let us reach viewers in regions that normally would have trouble accessing streaming video. Edge computing allows most of the computation work to be done near the source, improving response times and bandwidth usage—a perfect synergy leveraging Google and Theta Network’s core strengths. And with Google Cloud’s over 1600 nodes, we are able to get closer to our users than ever before.Running analytics on our skyrocketing dataBeyond the video streaming, Google Cloud’s enterprise data warehouse BigQuery gave us the capacity to do the typically difficult—if not impossible—task of sorting real-time data from the blockchain system. We have built a real-time pipeline for the viewership data using Dataflow, Pub/Sub and BigQuery. A Dataflow job continuously pulls the data from a Pub/Sub topic and ingests into BigQuery. We’ve seen Pub/Sub quickly ingest roughly 12,000 to 14,000 blocks of data containing 60,000 to 200,000 transactions daily into BigQuery for real-time analysis. We also used Pub/Sub and Dataflow to create the listener/subscriber for the topic our ETL pipeline publishes, then ingest that into BigQuery tables. By running fast queries in BigQuery, we were able to uncover findings such as: How many people watched and shared a certain video stream in the past hour;How many donations were made to a streamer in total;Which livestream has the highest donation to viewer ratio;What was the most impactful moment during a livestream.Prior to BigQuery, finding this information required writing customized scripts to analyze the blockchain raw data and the analysis used to take hours or even days of engineering time. Now we can gain such insights in a little as a few seconds, and effectively in real time. Now we can gather information to let streamers, advertisers, and partners know when more viewers were online and engaged. This means that NASA and other content creators could better find and reach their audiences. Results that let us grow and scale to the moon and beyondGoogle Cloud helps us better to forecast how many concurrent users we need to support during livestream events and predict multi-variable reputation scores for our network of thousands of edge and guardian nodes to identify and address bad actors and under-performing nodes. Today, our BigQuery environment has 45GB of data, which contains almost 7.5 million blocks and 57 million transactions and counting. We migrated to Google Cloud in less than six months, and saw the return on investment almost immediately. We’re able to bring top-notch connectivity, scalability and security capabilities for our branded content partners like NASA, enterprise validator partners including Google and community members that run Theta edge and guardian nodes, and we’re reducing costs over time.All of this is just the beginning of the ways we’re looking to spread more entertainment, science, and hope during these dark times. And thanks to Google Cloud’s strength and scalability, we’ll be able to keep growing, reaching even more audiences and partners.Learn more about Theta Labs here.
Quelle: Google Cloud Platform

Rate Limiting by the Numbers

As a critical part of Docker’s transition into sustainability, we’ve been gradually rolling out limits on docker pulls to the heaviest users of Docker Hub. As we near the end of the implementation of the rate limits, we thought we’d share some of the facts and figures behind our effort. Our goal is to ensure that Docker becomes sustainable for the long term, while continuing to offer developers 100% free tools to build, share, and run their applications.

We announced this plan in August with an effective date of November 1. We also shared that “roughly 30% of all downloads on Hub come from only 1% of our anonymous users,” illustrated in this chart:

This shows the dramatic impact that a very small percentage of anonymous, free users have on all of Docker Hub. That excessive usage by just 1%–2% of our users results not only in an unsustainable model for Docker but also slows performance for the other 98%–99% of the 11.3 million developers, CI services, and other platforms using Docker Hub every month. Those developers rely upon us to save and share their own container images, as well as to pull images from Docker Verified Publishers and our own trusted library of Docker Official Images, amounting to more than 13.6 billion pulls per month.

Based on our goal of ensuring the vast majority of developers can remain productive, we designed limits of 100 or 200 pulls in a 6-hour window for anonymous and authenticated free users, respectively. In the context of a developer’s daily workflow, 100 pulls in 6 hours amounts to a docker pull every 3.6 minutes on average, for 6 consecutive hours. We considered this more than adequate for an individual developer, while other use cases involving high pull rates such as CI/CD pipelines or production container platforms can decrease their usage or subscribe to a paid plan.

Over the course of a month, a single anonymous developer can (with the help of automation) make up to 12,000 docker pulls. By authenticating, that number increases to 24,000 docker pulls for free. As Docker container images vary in size from a few MB to well above 1 GB, focusing on pulls rather than size provides predictability to developers. They can pull images as they’re building applications, without worrying about their size but rather about their value.

Based on these limits, we expected only 1.5% of daily unique IP addresses to be included — roughly 40,000 IPs in total, out of more than 2 million IPs that pull from Docker Hub every day. The other 98.5% of IPs using Docker Hub can carry on unaffected — or more likely, receive improved performance as the heaviest users decreased.

As November 1st approached, we created a rollout plan that provided additional advance notice and decreased impact — even to developers we haven’t been able to reach through our emails or blog posts. We’ve put a few things in place to ease the transition for anyone affected:

Providing a grace period after November 1 prior to full enforcement for all usage, so only a small fraction of the heaviest users were limited early in our rollout;Progressive rollout of enforcement across the affected population, to provide for additional opportunities for communications to reach Docker developers, and to minimize any inadvertent impact; andTemporary time windows of full enforcement, to raise awareness of unknown reliance upon Docker Hub and to reach developers without Docker IDs who we could not otherwise.

On Wednesday, November 18, we expect to complete our progressive rollout to the new limits of 100 pulls and 200 pulls per 6-hour window for anonymous and authenticated free users, respectively. At that point, anyone who has not yet encountered the limits can reasonably conclude that their current usage of docker pulls is in that 98.5% of unaffected Docker Hub users.

As we’ve progressed down this path toward creating a sustainable Docker, we’ve heard multiple times from developers that the temporary full-enforcement windows were valuable. They surfaced unknown reliance upon Docker Hub, as well as areas where our paying customers had not yet authenticated their usage. We’ve also worked with customers to identify problems that were unknowingly causing some of the massive downloads, like runaway processes downloading once every 3 seconds. Alongside this, we’ve created additional paid offerings to support large enterprises, ISVs, and service providers with needs like IP whitelisting or namespace whitelisting.

We greatly appreciate the trust placed in Docker by the entire software community, and we look forward to helping you continue to build the great applications of the future!
The post Rate Limiting by the Numbers appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/