Splunk wants to make machine data accessible, usable and valuable to everyone. With over 14,000 customers in 110 countries, providing the best software for visualizing machine data involves hours and hours of testing against multiple supported platforms and various configurations. For Mike Dickey, Sr. Director in charge of engineering infrastructure at Splunk, the challenge was that 13 different engineering teams in California and Shanghai had contributed to test infrastructure sprawl, with hundreds of different projects and plans that were all being managed manually.
At DockerCon Europe, Mike and Harish Jayakumar, Docker Solutions Engineer, shared how Splunk leveraged Docker Enterprise Edition (Docker EE) to dramatically improve build and deployment times on their test infrastructure, converge on a unified Continuous Integration (CI) workflow, and how they’ve now grown to 600 bare-metal servers deploying tens of thousands of Docker containers per day.
You can watch the entire session here:
Hitting the Limits of Manual Test Configurations
As Splunk has grown, so has their customers’ use of their software. Many Splunk customers now process petabytes of data, and that has forced Splunk to scale their testing to match. That means more infrastructure needs to be reserved in the shared test environment for these large-scale tests. Besides running out of data center capacity, reserving test infrastructure was being managed manually through a Wiki page – a process with obvious limitations.
At the time Mike was leading the Performance Engineering team, and they had started working with Docker containers. Seeing near-bare metal performance for containerized applications, Splunk began to test Docker in smaller proof-of-concept projects and saw that it could be effective for performance testing. They saw the ability to leverage Docker as the foundation for a unified test and CI platform.
Building a App Development Platform with Docker EE
Splunk chose Docker EE to power their test and CI platform for a few key reasons:
Windows and Linux support: Splunk software runs on both Linux and Windows and so they wanted a single solution that could support both Linux and Windows
Role-Based Access Control: As the environment is a shared resource between multiple teams, Splunk needed a way to integrate with Active Directory and assign resources by roles.
Consistent Dev Experience: With most developers already using Docker on their desktops, Splunk wanted to maintain a consistent experience with support for Docker APIs and the use of Compose files.
Vendor to Partner With: Given the scale of this project, Splunk wanted to work with a vendor who would be their partner. A bonus was that our offices were only a few blocks apart.
Results and What’s Next
Today, Docker EE powers Splunk’s CI and test platforms. As part of the CI solution, Splunk is leveraging Docker to create an agentless Jenkins workflow where each build stage is replaced by a container. This delivers a more consistent and scalable experience (2000 concurrent jobs today vs. 200 per master with standard agents) that is much more efficient as well. For performance testing, teams can reserve an entire host to get accurate performance results. These can be dynamically provisioned for different configurations in minutes instead of days.
At Splunk, the Docker EE environment has grown from 150 servers to now 600 servers, starting with one team of developers to now 385 unique developers who deploy between 10,000 and 20,000 containers a day. In addition to the fast deployment times, Splunk is seeing more efficient use of the hardware than before, averaging 75% utilization of the available capacity. With the platform in place, the developers at Splunk have a simple and fast way to provision and execute tests. As a result, Splunk has seen an increase in testing frequency, which is helping to improve product quality.
Check out how @Splunk used #Docker EE to scale and deploy 10,000+ containers per dayClick To Tweet
To learn more about Docker EE, check out the following resources:
Learn more about Docker EE
Try Docker EE for yourself
Contact Sales for more information
The post Using Docker to Scale Operational Intelligence at Splunk appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
Published by