Vodafone India looks to IBM Hybrid Cloud to improve engagement

A new, five-year agreement between IBM and Vodafone India brings together the telecommunications company&;s cloud-ready applications and enterprise data on a hybrid cloud platform. The goal? Faster IT operations and speedier time to market, which will help the company better connect and engage with customers. In a joint statement, Vishant Vora, Vodafone India&8217;s Director of [&;]
The post Vodafone India looks to IBM Hybrid Cloud to improve engagement appeared first on Thoughts On Cloud.
Quelle: Thoughts on Cloud

OpenStack Days Silicon Valley 2016 (The Unlocked Infrastructure Conference) Day 2

The post OpenStack Days Silicon Valley 2016 (The Unlocked Infrastructure Conference) Day 2 appeared first on Mirantis | The Pure Play OpenStack Company.
The second day of  OpenStack Days Silicon Valley continued with conversations about containers and the processes of managing OpenStack. If you missed the event and the live stream, no worries; here are the highlights.
Christian Carrasco &; When OpenStack Fails. (Hint: It’s not the Technology)
Christian, a cloud advisor at Tapjoy, started off the day by sharing what Tapjoy has learned from working with OpenStack. Tapjoy is an SaaS player that from early on set its sights on OpenStack. The company has grown to be the leading player in the mobile app monetization space, with more than two million daily engagements, 270,000 active apps, and 500 million users.
However, interestingly, most of the lessons Christian has learned while working at Tapjoy have less to do with the technology or maturity underpinning OpenStack or the many components necessary to its deployment. Instead, they revolve around the people, process, and organizational choices that are necessary for your OpenStack cloud to succeed.
Christian urged the audience to stop focusing on building a better buggy, and to instead focus on making a better cloud—the next generation of cloud. Christian argued that before we can hyper-converge the cloud, we need interoperability standards, arguing that there were many industries that couldn’t have existed without standards, such as the internet, the automobile industry, and healthcare.
Luke Kanies &8211; DevOps: Myths vs. Realities
Next up was Luke Kanies, the founder and CEO of Puppet. Luke spoke to the audience about the myths that surround DevOps in the enterprise, and argued that we need to leave behind the old way of delivering software to adopt the new world of DevOps practices.
Luke made it clear just why: top performing DevOps teams deploy 200 times more often and recover from failure 24 times faster, he said.
Luke argued that the fears companies have about adopting DevOps practices are due to two beliefs. First, that certain practices just won’t work for an organization, due to factors such as legacy environments, traditional enterprises, or hierarchical organizations. The second belief, he said, is that DevOps practices are simply unworkable when enterprises are subject to a host of external regulatory and compliance requirements.
Luke said that most organizations (97-98 percent) that had fears about introducing DevOps practices had legacy issues, but he argued that ignoring those legacy issues undermined their work.
Luke ended his talk by discussing how to overcome misconceptions by dispelling the most common myths. He said that adopting DevOps practices didn’t have to be all or nothing, they could be simpler than it appeared, and that often the largest returns come from unexpected areas. Ultimately, he argued, you have a choice—do you want to start using DevOps practices, or would you prefer for your competitors to beat you to it?
James Staten &8211; Hybrid Cloud is About the Apps, Not the Infrastructure
James Staten, Microsoft’s Chief Strategist for the Cloud and Enterprise division, was next up on stage to talk about building and deploying true enterprise cloud apps.
James said the key to this is understanding how to blend your environments, as leading enterprise examples of are not exclusively private or exclusively public cloud deployments, but are instead a mixture of both plus multiple public clouds. He said that even Microsoft runs on a hybrid cloud.
James argued that the hybrid cloud is here to stay, and not just because of the legacy code that can’t move anywhere (let alone to the cloud). He pointed to statistics that showed 74 percent of enterprises believe a hybrid cloud will enable business growth, and 82 percent have a hybrid cloud strategy (up from 74 percent a year ago).
He said that organizations used to be worried about application integration, security, and data sovereignty when considering moving apps to a public cloud. However, now organizations say they don’t use public clouds because of needing compute on premises, the Internet of Things, optimization of economics, and wanting to leverage the right resources in the right places.
James ended his session by outlining new hybrid models with many elements, including local resources, public clouds, and SAAS apps and microservices. He said that hybrid isn’t just about location, but the programming languages, devices, and operating systems. He said that the apps we are building need to have compute capability everywhere, because a hybrid cloud is about the apps you are designing.
Alex Williams, Frederic Lardinois, Craig Matsumoto, Mitch Wagner &8211; Open Source and the News Media
The first panel discussion of the day was about Open Source and the News Media, with four technology journalists: Alex Williams (founder of The New Stack), Frederic Lardinois (writer for TechCrunch), Craig Matsumoto (Managing Editor at SDxCentral, and Mitch Wagner (Editor, Enterprise Cloud, for Light Reading).
Key takeaway: People are often confused by messages coming from open source projects and companies that build products and services using them, and acronyms, clever names, and not-for-profit foundations had further contributed to this confusion. Alex said that some people would say that cloud service providers are the greatest threat to open source.
In addition, the panelists discussed the difficulties they had with tracking and learning all of the players and their interests in the Open Source movement.
Kim Bannerman (Director, Advocacy & Community &8211; Office of the CTO, Blue Box), Kenneth Hui (Senior Technical Marketing Manager, Rackspace), Patrick Reilly (Founder and Former CEO of Kismatic) — All Open Source Problems Solved in This Session
Next, Kim Bannerman, Kenneth Hui, and Patrick Reilly, a group of open source veterans, discussed critiques that are common for open source projects and looked at how to address them.
Patrick pointed out that OpenStack really is a community, and to have OpenStack work better you really need to participate. He argued that if you have a complaint, you should follow up and work to fix those issues.
Interestingly, the panel discussed the idea that often criticisms about open source projects, such as their governance, roadmap, and focus, are often just the downsides of advantages open source provides: transparency, inclusiveness, and agility.
Jonathan Donaldson (VP & GM, Software Defined Infrastructure at Intel) — The Future of OpenStack Clouds
Following the two panel discussions, Jonathan Donaldson of Intel and Craig McLuckie from Google talked to us about their collaboration and the future of OpenStack clouds.
Craig said that Google wants to be an enterprise software company, using OpenStack, because the market is too big to ignore. However, Craig said that Google is pretty behind.
During the talk, Jonathan discussed Intel’s Cloud for All initiative. It began last year and Intel began heavily investing in the OpenStack platform in an effort to improve OpenStack for the enterprise and to speed up its rate of adoption around the world. He said that Intel cares so much about a cloud for all because fostering innovation leads to use cases and creates value.
This has led to Intel and the broader community making OpenStack production-ready for enterprise workloads. He said that this has led to new features and significantly lower barriers for businesses that want to deploy private and hybrid clouds.
Randy Bias (VP of Technology, EMC), Sean Roberts (Director Technical Program Management, Walmart Labs), Mike Yang (GM of Quanta Cloud Technology) &8211; The State of OpenStack on Commodity Hardware
The first discussion of the final session was about OpenStack and commodity hardware. In the early days of OpenStack, open cloud software with “open” or commodity hardware was seen as a perfect match.
One question the panel discussed was whether BOMs that mix commodity and proprietary components were the norm, or whether pre-integrated and fully commodity BOMs with components from one manufacturer were more popular.
Randy pointed out that the bottom line is that open or commodity hardware is not free, as it still takes skill to deploy. He argued that while it eventually will be easy to deploy open hardware, it’s not there yet.
Adrian Cockcroft (Battery Ventures Technology Fellow), Boris Renski (Mirantis Co-Founder and CMO) &8211; Infrastructure Software is Dead… Or is it?
Next up Boris Renski from Mirantis and Adrian Cockcroft, a Battery Ventures Technology Fellow, conversed about Boris&; premise that Infrastructure software is dead.
As the two discussed the cloud revolution, Boris argued that the cloud revolution isn’t just about software, but also the delivery model, and that the delivery model for enterprise on-premises software has changed radically. Adrian agreed, adding that traditional hardware and software procurement cycles have collapsed with the cloud.
The two finished their talk by discussing the future of OpenStack. They said its future will not be in making the most “enterprise ready” software, but in building models for delivering customer outcomes that move the needle. Adrian said that he believed that unless you had very specialized or very large scale workloads, there is no competitive advantage to having your own data center.
Michael Miller (President of Strategy, Alliances and Marketing, SUSE) &8211; OpenStack Past, Present and Future
To wrap-up the conference, Michael Miller from SUSE discussed OpenStack’s journey from inception to the present and shared some thoughts on what to expect next, discussing just how quickly enterprise IT is now adopting OpenStack, despite initial apprehensions.
The Last Word
From hallway conversations, to expert commentary, to the swarms of people who were visiting sponsor booths, the OpenStack Days Silicon Valley conference was a great success in getting people talking not about whether OpenStack was a success—that part&8217;s a given—but why. Users were talking about where OpenStack fits in, how it&8217;s still important for enterprise workloads, and how to most efficiently leverage new technologies such as containers.
So here&8217;s our question to you: what do you think we&8217;ll be talking about next year?
 
The post OpenStack Days Silicon Valley 2016 (The Unlocked Infrastructure Conference) Day 2 appeared first on Mirantis | The Pure Play OpenStack Company.
Quelle: Mirantis

Docker Hub Hits 5 Billion Pulls

Last week, the total number of image pulls from the Docker Hub Repository Service reached 5 billion. That’s an increase of 150% since just February. It’s pretty amazing for a three year old project. Docker Hub has become a part of the daily life of developers because it

Is a central, reliable, and secure service to host your own repositories and get access to high quality images
The repository service also serves Docker Cloud repos and in fact with your existing content and subscriptions

That means the over 650,000 registered users are pulling images over 13000 times a minute. That’s almost twice as fast as in February when we had 7000 pulls a minute.
Equally interesting, the total number of pulls of official images exceeded 1 billion at the same time. The Docker Official Repositories are a curated set of image repositories that are promoted on Docker Hub. Official images are scanned by Docker Security Scanning, making them the most secure base images you can use. In fact, you can see how secure each of them are by using Security Scanning. And you can use the same scanning service on your own private repositories during the free trial period.
If you haven’t gotten in on the fun yet, it’s really easy to get started. Create your Docker ID today and get one free private repo.

You can create a repository right there

Or you can just push from the command line
$ docker login
Login with your Docker ID to push and pull images from Docker. If you don&;t have a Docker ID, head over to https://cloud.docker.com to create one.
Username (manomarks): manomarks
Password:
Login Succeeded
$ docker build -t manomarks/visualizer .
$ docker push manomarks/visualizer
And we’re building even more functionality with the new private beta of the Docker Store, which will provide a scalable self-service system for ISVs to publish and distribute trusted and enterprise-ready content. Head over to store.docker.com to give it a look.

We&8217;re happy to announced that @Docker Hub had reached 5 billion PullsClick To Tweet

Quelle: https://blog.docker.com/feed/

DAY 1- OPENSTACK DAYS SILICON VALLEY 2016

The post DAY 1- OPENSTACK DAYS SILICON VALLEY 2016 appeared first on Mirantis | The Pure Play OpenStack Company.
THE UNLOCKED INFRASTRUCTURE CONFERENCE
By Catherine Kim

This year&;s OpenStack Days Silicon Valley, held once again at the Computer History Museum, carried a theme of industry maturity; we&8217;ve gone, as Mirantis CEO and co-Founder Alex Freedland said in his introductory remarks, from wondering if OpenStack was going to catch on to wondering where containers fit into the landscape to talking about production environments of both OpenStack and containers.

Here&8217;s a look at what you missed.
OpenStack: What Next?
OpenStack Foundation Executive Director Jonathan Bryce started the day off talking about the future of OpenStack. He&8217;s been traveling the globe visiting user groups and OpenStack Days events, watching as the technology takes hold in different parts of the world, but his predictions were less about what OpenStack could do and more about what people &; and other projects &8212; could do with it.

Standard frameworks, he said, provided the opportunity for large numbers of developers to create entirely new categories. For example, before the LAMP stack (Linux, Apache, MySQL and PHP) the web was largely made up of static pages, not the dynamic applications we have now. Android and iOS provided common frameworks that enable developers to release millions of apps a year, supplanting purpose-built machinery with a simple smartphone.

To make that happen, though, the community had to do two things: collaborate and scale. Just as the components of LAMP worked together, OpenStack needed to collaborate with other projects, such as Kubernetes, to reach its potential.

As for scaling, Jonathan pointed out that historically, OpenStack has been difficult to set up. It’s important to make success easier to duplicate. While there are incredible success stories out there, with some users using thousands of nodes, those users originally had to go through a lot of iterations and errors. For future developments, Jonathan felt it was important to share information about errors made, so that others can learn from those mistakes, making OpenStack easier to use.

To that end, the OpenStack foundation is continuing to produce content to help with specific needs, such as explaining the business benefits to a manager to more complex topics such as security. He also talked about the need to raise the size of the talent pool, and about the ability for students to take the Certified OpenStack Administrator exam (or others like it) to prove their capabilities in the market.
User talks
One thing that was refreshing about OpenStack Days Silicon Valley was the number of user-given talks. On day one we heard from Walmart, SAP, and AT&T, all of which have significantly transformed their organizations through the use of OpenStack.

OpenStack, Sean Roberts explained, enabled Walmart to make applications that can heal themselves, with failure scenarios that have rules about how they can recover from those failures. In particular, WalmartLabs, the online end of the company, had been making great strides with OpenStack, and in particular with a devops tool called OneOps. The tool makes it possible for them to manage their large number of nodes easily, and he suggested that it might do even better as an independent project under OpenStack.

Markus Riedinger talked about SAP and how it had introduced OpenStack. After making 23 acquisitions in a small period of time, the company was faced with a diverse infrastructure that didn&8217;t lend itself to collaboration. In the last few years it has begun to move towards cloud based work and in 2013 it started to move towards using OpenStack. Now the company has a container-based OpenStack structure based on Puppet, providing a clean separation of control and data, and a fully automatic system with embedded analytics and pre-manufactured PODs for capacity extension.  Their approach means that 1-2 people can take a data center from commissioned bare metal to an operational, scalable Kubernetes cluster running a fully configured OpenStack platform in less than a day.

Greg Stiegler discussed AT&T’s cloud journey, and Open Source and OpenStack at AT&T. He said that the rapid advancements in mobile data services have resulted in numerous benefits, and in turn this has exploded network traffic, with traffic expected to grow 10 times by 2020. To facilitate this growth, AT&T needed a platform, with a goal of remaining as close to trunk as possible to reduce technical debt. The result is the AT&T Integrated Cloud. Sarobh Saxena spoke about it at the OpenStack Summit in Austin earlier this year, but new today was the notion that the community effort should have a unified roadmap leader, with a strategy around containers that needs to be fully developed, and a rock solid core tent.

Greg finished up by saying that while AT&T doesn’t expect perfection, it does believe that OpenStack needs to be continually developed and strengthened. The company is grateful for what the community has always provided, and AT&T has provided an AT&T community team. Greg felt that the moral of his story was that by working together, community collaboration brings solutions at a faster rate, while weeding out mistakes through the experiences of others.
What venture capitalists think about open source
Well that got your attention, didn&8217;t it?  It got the audience&8217;s attention too, as Martin Casado, a General Partner from Adreessen Horowitz, started the talk by saying that current prevailing wisdom is that infrastructure is dead. Why? Partly because people don’t understand what the cloud is, and partly because they think that if the cloud is free, then they think “What else is there to invest in?” Having looked into it he thinks that view is dead wrong, and even believes that newcomers now have an unfair advantage.

Martin  (who in a former life was the creator of the &;software defined&; movement through the co-founding of SDN maker Nicira) said that for this talk, something is “software defined” if you can implement it in software and distribute it in software. For example, in the consumer space, the GPS devices have largely been replaced by software applications like Waze, which can be distributed to millions of phones, which themselves can run diverse apps to replace may functionalities that used to be &8220;wrapped in sheet metal&8221;.

He argued that infrastructure is following the same pattern. It used to be that the only common interface was internet or IP, but that we have seen a maturation of software that allows you to insert core infrastructure as software. Martin said that right now is one of these few times where there’s a market sufficient for building a company with a product that consists entirely of software.  (You still, however, need a sales team, sorry.)

The crux of the matter, though, is that the old model for Open Source has changed. The old model for Open Source companies was being a support company, however, now many companies will use Open Source to access customers and get credibility, but the actual commercial offering they have is a service. Companies such as Github (which didn&8217;t even invent Git) doing this have been enormously successful.
And now a word from our sponsors&;
The morning included several very short &8220;sponsor moments&8221;; two of which included very short tech talks.

The third was Michael Miller of Suse, who was joined onstage by Boris Renski from Mirantis. Together they announced that Mirantis and Suse would be collaborating with each other to provide support for SLES as both hosts and guests in Mirantis OpenStack, which already supports Ubuntu and Oracle Linux.

“At this point, there is only one conspicuous partner missing from this equation,” Renski said. Not to worry, he continued. SUSE has an expanded support offering, so in addition to supporting SUSE hosts, through the new partnership, Mirantis/SUSE customers with CentOS and RHEL hosts can also get support. “Mirantis  is now a one-stop shop for supporting OpenStack.”

Meanwhile,  Sujal Das, SVP of Marketing for Netronome, discussed networking and security and the many industry reports that highlight the importance of zero-trust defense security, with each VM and application needing to be trusted. OpenStack enables centralised control and automation in these types of deployments, but there are some challenges when using OVS and connection tracking, which affect VMs and the efficiency of the server. Ideally, you would like line red performance, but Netronome did some tests that show you do not get that performance with zero-trust security and OpenStack. Netronome is working on enhancements and adaptations to assist with this.

Finally, Evan Mouzakitis of Data Dog gave a great explanation of how you can look at events that happen when you are using OpenStack more closely to see not only what happened, but why. Evan explained that OpenStack uses RabbitMQ by default for message passing, and that once you can listen to that, you can know a lot more about what’s happening under the hood, and a lot more about the events that are occurring. (Hint: go to http://dtdg.co/nova-listen.)
Containers, containers, containers
Of course, the main thrust was OpenStack and containers, and there was no shortage of content along those lines.
Craig McLuckie of Google and Brandon Philips of CoreOS sat down with Sumeet Singh of AppFormix to talk about the future of OpenStack, namely the integration of OpenStack and Kubernetes. Sumeet started this discussion swiftly, asking Craig and Brandon “If we have Kubernetes, why do we need OpenStack?”

Craig said that enterprise needs hybrids of technologies, and that there is a lot of alignment between the two technologies, so  both can be useful for enterprises. Brandon also said that there’s a large incumbent of virtual machine users and they aren’t going to go away.

There’s a lot of integration work, but also a lot of other work to do as a community. Some is the next level of abstraction &; one of those things is rallying together to help software vendors to have a set of common standards for describing packages. Craig also believed that there’s a good opportunity to think about brokering of services and lifecycle management.

Craig also mentioned that he felt that we need to start thinking about how to bring the OpenStack and Cloud Native Computing foundations together and how to create working groups that span the two foundation’s boundaries.

In terms of using the two together, Craig said that from his experience he found that enterprises usually ask what it looks like to use the two. As people start to understand the different capabilities they shift towards it, but it’s very new and so it’s quite speculative right now.

Finally, Florian Leibert of Mesosphere, Andrew Randall of Tigera, Ken Robertson of Apcera, and Amir Levy of Gigaspaces sat down with Jesse Proudman of IBM to discuss &8220;The Next Container Standard&8221;.

Jesse started off the discussion by talking about how rapidly OpenStack has developed, and how in two short years containers have penetrated the marketplace. He questioned why that might be.

Some of the participants suggested that a big reason for their uptake is that containers drive adoption and help with inefficiencies, so customers can easily see how dynamic this field is in providing for their requirements.

A number of participants felt that containers are another wonderful tool in getting the job done and they’ll see more innovations down the road. Florian pointed out that containers were around before Docker, but what docker has done is that it has allowed individuals to use containers on their own websites. Containers are just a part of an evolution.

As far as Cloud Foundry vs Mesos or Kubernetes, most of the participants agreed that standard orchestration has allowed us to take a step higher in the model and that an understanding of the underlying tools can be used together &8212; as long as you use the right models. Amir argued that there is no need to take one specific technology’s corner, and that there will always be new technologies around the corner, but whatever we see today will be different tomorrow.

Of course there&8217;s the question of whether these technologies are complementary or competitive. Florian argued that it came down to religion, and that over time companies will often evolve to be very similar to one another. But if it is a religious decision, then who was making that decision?

The panel agreed that it is often the developers themselves who make decisions, but that eventually companies will choose to deliberately use multiple platforms or they will make a decision to use just one.

Finally, Jesse asked the panel about how the wishes of companies for a strong ROI affects OpenStack, leading to a discussion about the importance of really strong use cases, and showing customers how OpenStack can improve speed or flexibility.
Coming up
So now we head into day 2 of the conference, where it&8217;s all about thought leadership, community, and user stories. Look for commentary from users such as Tapjoy and thought leadership from voices such as James Staten from Microsoft, Luke Kanies of Puppet, and Adrian Cockroft of Battery Ventures.

 

 The post DAY 1- OPENSTACK DAYS SILICON VALLEY 2016 appeared first on Mirantis | The Pure Play OpenStack Company.
Quelle: Mirantis

Docker Online Meetup # 41: Docker Captains Share their Tips and Tricks for Built In Docker Orchestration

It’s been nearly two weeks since Docker released Docker 1.12 as generally available for production environments, introducing a number of new features and concepts to the Docker project. Our  team has already started to dig in and share their learnings with the community via blog posts, talks and peer-to-peer help. Docker Captains are technology experts who have been awarded the distinction of being a Docker Captain in part because of their passion for sharing their Docker knowledge with others. So, we’ve invited three of our Docker Captains to speak at the next Docker Online Meetup on August 31st and share their tips and tricks for using Docker 1.12.

Ajeet Singh Raina is currently working as Technical Lead Engineer in the Enterprise Solution Group at Dell India R&D and has solid understanding of a diverse range of IT infrastructure, system management, system integration engineering and quality assurance. Ajeet has a great passion for upcoming trends and technologies. He loves contributing towards Open source space through writing and blogging @ http://www.collabnix.com.
Ajeet has shared a number of fantastic articles on Service Discovery, including Demonstrating Docker 1.12 Service Discovery with Docker Compose and How Service Discovery works under Docker Engine 1.12. In the meetup, Ajeet will quickly share his key takeaways and the best ways to use Docker 1.12 Service Discovery.

Viktor Farcic is a Senior Consultant at CloudBees. He coded using a plethora of languages starting and can often be found speaking about Docker around the world. His big passions are Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).  He wrote The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices and the Test-Driven Java Development books. His random thoughts and tutorials can be found on his blog TechnologyConversations.com.
If you haven’t had the chance to check out Viktor’s Docker Swarm Introduction and Integrating Proxy With Docker Swarm (Tour Around Docker 1.12 Series), you are going to want to make some time to dive in. Viktor covers the basics of how Swarm in Docker v1.12 works and then deep-dives into some of the more complicated aspects. In the meetup, Viktor will share best practices for setting a Swarm cluster and integrating it with HAProxy.

For 20 years Bret Fisher has designed, built, and operated distributed systems from 4 to 4000. He currently focuses on DevOps style activities in the public cloud and enterprise. Bret works on creating immutable infrastructures, automation, containers, CI/CD, cloud monitoring, and an occasional JavaScript developer. He spends his free time in Virginia&;s local, thriving tech scene helping lead local Code for America and Docker Meetup groups. He basically spend his days helping people. Bret lives at the beach. Dogs over Cats.
Bret frequents Docker forums to lend a helping hand when others get stuck, so he knows a thing or two about how to optimize getting started with 1.12. Bret will share his favorite Docker 1.12 command options and aliases that will make your life easier including cli aliases for quick container management; the shortest path to secure production-ready swarm; how to use cli filters for easier management of larger swarms; and docker remote cli security setup.
Register now to attend the meetup and learn tips and tricks for using Docker 1.12!

Join @Docker Captains @ajeetsraina @BretFisher @vfarcic to learn about @docker 1.12 built in&;Click To Tweet

Quelle: https://blog.docker.com/feed/

Mirantis and SUSE: Creating a One-Stop Shop for OpenStack Support on Major Linux Distros

The post Mirantis and SUSE: Creating a One-Stop Shop for OpenStack Support on Major Linux Distros appeared first on Mirantis | The Pure Play OpenStack Company.
This week, at OpenStack Silicon Valley, Mirantis and SUSE announced a partnership which will make Mirantis a &;one-stop shop&; for Mirantis OpenStack supported on all major Linux distributions. Mirantis does not ship a Linux distribution, but rather works with Linux distribution vendors on support of underlying Linux operating systems.This  partnership positions SUSE as Mirantis strategic Enterprise Linux partner providing Mirantis OpenStack customers with enterprise grade SLA’s for SUSE Linux Enterprise Server (SLES), Red Hat Enterprise Linux (RHEL), and CentOS. For ongoing support of Mirantis OpenStack running on Ubuntu, Mirantis and Canonical have had a collaborative relationship for several years, jointly supporting large customers like AT&T.
As a pure-play OpenStack provider, committed to freedom of choice, Mirantis will leverage this partnership to provide more flexibility to customers, reducing lock-in. While &8220;one-stop shop&8221; used to mean that a single technology vendor offered all the components of the solution, services, and support, Mirantis one-stop shop is about being a single source of support, services, and expertise to help customers in their cloud transformation journey using a wide range of certified best of breed technology selections. This approach is hugely valuable to large customers who may be broadly committed to a Linux distribution, but don&;t want to be locked into that choice, or limited in choosing other best-of-breed data center technologies to work with OpenStack..
Mirantis and SUSE will begin engineering collaboration upstream to fine tune Mirantis OpenStack on SUSE enterprise Linux leading to a certified/supported solution for customers. Additional upstream and downstream engineering/support collaboration will accelerate Mirantis taking on front line L1 and L2 support for the entire solution while SUSE provides L3 support for SLES, RHEL and CentOS.
Being free to run OpenStack on a preferred Linux distro is a big deal for enterprises — touching on every aspect of reliability, security, performance, usability, interoperability, and cost. In the past, such freedom has been hard to come by in the OpenStack space, because supporting production OpenStack on multiple spins requires both broad and specialized expertise. In some cases, vendors such as Red Hat have touted the value of Linux and OpenStack being &;co-engineered,&8217; effectively promoting lock-in. Mirantis has historically taken the opposite approach: as a pure-play OpenStack provider, we think of OpenStack as an application that should run on any host OS (or in containers, as our recent announcement about Kubernetes makes clear). This new partnership will help us deliver that kind of freedom of choice and reassurance to OpenStack customers in the real world.
The post Mirantis and SUSE: Creating a One-Stop Shop for OpenStack Support on Major Linux Distros appeared first on Mirantis | The Pure Play OpenStack Company.
Quelle: Mirantis

Your Docker Agenda in August

From webinars to workshops, to conference talks, check out our list of events that are coming up in August!

North America | South America | Europe | Oceania | Asia | Africa | Official Docker Training Courses
 

Check out the @Docker agenda for August! Tons of awesome meetups, webinars & conferences!Click To Tweet

Official Docker Training Courses
View the full schedule of instructor led training courses here! Description of courses are below.

Docker Datacenter Training Series
Introduction to Docker
Docker Administration and Operations
Advanced Docker Operations
Managing Container Services with Universal Control Plane
Deploying Docker Datacenter
User Management and Troubleshooting UCP

North America
 
Aug 3rd: Docker Meetup at Docker HQ &; San Francisco, CA
Come and join us at Docker HQ on Wednesday for our 47th meetup! Ben Bonnefoy , a member of the Docker technical staff, will give an insight into Docker for Mac and Docker for Windows and then Nishant Totla , a software engineer in the core open source team, will give some updates on Docker .12. This will be followed by a talk by Neil Gehani , a Sr. Product Manager at HPE, on in-cluster testing. It will be a fun evening of learning, exchanging ideas and networking with pizza, beer and plenty of Docker stickers for everyone.
RSVP
Aug 3rd: Docker Meetup at Meltmedia &8211; Tempe, AZ
This meetup will focus on Docker for AWS, specifically running distributed apps from localhost to AWS.
RSVP
Aug 4th: Docker Meetup at Rackspace &8211; Austin, TX
A discussion about Docker Tips and Tricks.
RSVP
Aug 9th: Docker Meetup at CA Technologies &8211; Denver, CO
A talk about moving from SaaS to On-Premise with Docker, in particular how Docker made it possible to deploy a SaaS web application into firewalled networks and a journey of orchestrating together micro-service architecture from raw bash script to Replicated.
RSVP
Aug 11th: Docker Meetup at Full Sail Campus &8211; Orlando, FL
Docker Ecosystem and Use Case talks, followed by networking.
RSVP
Aug 11th: Docker Meetup at Braintree &8211; Chicago, IL
Ken Sipe will take the group through a look at the anatomy of a container including control groups (cgroups) and namespaces. Then there will be a discussion about Java&;s memory management and GC characteristics and how JRE characteristics change based on core count.
RSVP
Aug 16th: Docker Meetup at AEEC Innovation Lab &8211; Alexandria, VA
Docker Captain, Phil Estes, will present.
RSVP
Aug 16th: Docker Meetup at Datastax &8211; Santa Clara, CA
Databases, Image Management, In-cluster and Chaos Testing talks by Baruch Sadogursky, Ben Bromhead and Neil Gehani.
RSVP
Aug 16th: Docker Meetup at Impact Hub &8211; Santa Barbara, CA
This meetup will be about leveraging Docker + Compose for a real world dev environment. James Brown from Invoca will discuss how the move to Docker has benefited their development process.
RSVP
Aug 18th: Docker Meetup at CirrusMio &8211; Lexington, KY
Come and learn how others are using Docker! There will be two demos/talks scheduled for this meetup. The first will be about using Jenkins to build containers and the second will be about Docker in production.
RSVP
Aug 18th: Docker Meetup in Minneapolis &8211; Minneapolis, MN
The Container Summit City Series comes to Minneapolis on August 18th to continue the conversation surrounding containers in production! Bryan Cantrell, CTO of Joyent, will be joined in speaking by other expert users from companies that have been running containers in production for years and have experience with what solution stacks work best and what pitfalls to avoid.
RSVP
Aug 22nd: Docker Meetup at Issuetrak &8211; Virginia Beach , VA
Bret Fisher will tell all about DockerCon 2016 and what&8217;s in store for Docker in 1.12.
Aug 22nd &8211; 24th: LinuxCon/ ContainerCon &8211; Toronto, CA
There’s plenty of us at LinuxCon/ ContainerCon this year! Come see us at Booth to meet the Docker speakers and pick up your swag.
Aug 23rd: Docker and NATS Cloud Native Meetup During LinuxCon &8211; Toronto, Canada
The Docker Toronto meetup group and the Toronto NATS Cloud Native and IoT meetup group are joining forces to bring you a mega-meetup during LinuxCon! Riyaz Faizullabhoy from Docker will present on &;The Update Framework&8217; and , Diogo Monteiro will discuss implementing microservices with NATS. Raffi Der Haroutiounian will give an overview of NATS, Docker and Microservices.
Aug 23rd: Docker Meetup at the Iron Yard &8211; Houston, TX
Join us for our next meetup event!
RSVP
Aug 24th: Docker Meetup at CodeGuard &8211; Atlanta, GA
Talk by Eldon Stegall entitled, &8216;Abusing The Bridge: Booting a baremetal cluster from a docker container.&8217;
RSVP
Aug 28th &8211; 31h: VMworld 16 US &8211; Las Vegas, CA
Docker returns to VMworld this year and in Las Vegas! We’re launching our newest and biggest booth yet, so be sure to catch us at Booth . Yes, there will be swag given away.
Aug 31st: Docker Meetup in Salt Lake City &8211; Salt Lake City , UT
Come for a tutorial on new Docker 1.12 features and a review of DockerCon 2016 by Ryan Walls.
RSVP

South America
 
Aug 4th: Docker Meetup at Globant &8211; Córdoba, Argentina
Come for a talk on Docker for AWS. Talks by Florencia Caro, Ruben Dopazo, Carlos Santiago Moreno y Luis Barrueco.
RSVP
Aug 6th: Docker Meetup at Universidad Interamericana de Panamá &8211; Panamá, Panama
An introduction to Docker and Docker Cluster.
RSVP
Aug 9th: Docker Meetup at VivaReal&8211; Sao Paulo, Brazil
RSVP
Aug 13th: Docker Meetup at Microsoft Peru &8211; Lima, Peru
Join for a DockerCon recap.
RSVP
Aug 20th: Docker Meetup at Auditório-Unijorge Campus Comércio &8211; Salvador, Brazil
This is the beginning of the Docker Tour: the Docker Salvador meetup group&8217;s initiate to spread Docker technology among IT students in Salvador. This event will have two lectures for beginners where they can install the tool and learn Docker at ease in a friendly environment.
RSVP
Aug 23rd: Docker Meetup at Auditório Tecnopuc &8211; Porto Alegre, Brazil
A meetup to discuss PHP and Docker.
RSVP

Europe
 
Aug 3rd: Docker HandsOn &8211; Meet-Repeat C#+1 &8211; Hamburg, Germany
Aug 4th: Docker Meetup at SkyScanner Glasgow &8211; Glasgow, United Kingdom
What&8217;s new in Docker Land (@rawkode and @GJTempleton). Guy & I will be walking you through all the latest developments in Docker Land, including Docker Engine 1.12, Docker Compose 1.8, and Docker for Mac and Windows. Also well as these Docker updates, we&8217;ll be providing a quick review of DockerCon 2016 and highlighting some of the best talks for you to watch in your own time.
RSVP
Aug 8th: Docker Talk at Golang Conference &8211; Golang, UK
Speaking Docker Captain Tiffany Jernigan
Aug 9th: IOT RpiCar si ASP.NET Core + Docker &8211; Bucharest, Romania
Aug 10th:  Docker Meetup at KWORKS &8211; Istanbul, Turkey
Dockerizing a Complex Application Stack [w/Istanbul DevOps]
Aug 24th: Docker Meetup at Pipedrive &8211; Tallinn, Estonia
Let&8217;s share and discuss our experience with Docker ecosystem. More details of the content coming up!
RSVP
Aug 24th: Docker Meetup at Elastx &8211; Stockholm, Sweden
Continuously Deploying Containers To Docker Swarm Cluster. Speaker: Viktor Farcic (Docker Captain), & Senior Consultant, CloudBees. Abstract: Many of us have already experimented with Docker &8211; for example, running one of the pre-built images from Docker Hub. It is possible that your team might have recognized the benefits that Docker, in conjunction with experimentation, provides in building microservices and the advantages the technology could bring to development, testing, integration, and, ultimately, production.
RSVP
Aug 25th: Day of Containers &8211; Stockholm &8211; Stockholm, Sweden
Andrey Devyatkin & Viktor Farcic (Docker Captain) will give a talk &;Docker 101.&; If you are new to docker, this session is for you! In this sessions you will learn all the basics of docker and its main components. We will go through the the concept of containers, writing your own docker files, connecting data volumes, and basic orchestration with compose and swarm. Bring your laptops!
Aug 28th: Docker Meetup at Praqma &8211; Copenhagen, Denmark
Continuously Deploying Containers To Docker Swarm Cluster. Speaker: Viktor Farcic, Docker Captain & Senior Consultant, CloudBees. Abstract: Many of us have already experimented with Docker &8211; for example, running one of the pre-built images from Docker Hub. It is possible that your team might have recognized the benefits that Docker, in conjunction with experimentation, provides in building microservices and the advantages the technology could bring to development, testing, integration, and, ultimately, production.
RSVP
Aug 28th: Docker Talk at Agile Peterborough &8211; Peterborough, UK
Speaker Docker Captain Alex Ellis
Aug 28th: Docker Pre- Conference Meetup &8211; Praqma, Copenhagen
Speaker Docker Captain Viktor Farcic
Aug 29th: Docker Meetup at Praqma &8211; Copenhagen, Denmark
Laura Frank (Docker Captain) &8211; &8220;Stop being lazy and test your software.&8221; Testing software is necessary, no matter the size or status of your company. Introducing Docker to your development workflow can help you write and run your testing frameworks more efficiently, so that you can always deliver your best product to your customers and there are no excuses for not writing tests anymore. Jan Krag &8211; &8220;Docker 101.&8221; If you are new to docker, this session is for you! In this sessions you will learn all the basics of docker and its main components.
Viktor Farcic (Docker Captain)

Aug 31st: Docker Meetup at INCUBA &8211; Aarhus, Denmark
Rohde & Schwarz will give a talk about how they use Docker for development and test. HLTV.org will give a talk about how they use Docker to easily deploy microservices as part of their web platform.
RSVP
Aug 31st &8211; Sep 2: Software Circus &8211; Amsterdam, Netherlands
In Amsterdam for Software Circus? So is Docker! Speaking from Docker Ben Firshman

Asia
 
Aug 20th: Docker Meetup at Red Hat India Pvt. Ltd &8211; Bangalore, India
Docker for AWS and Azure &8211; Neependra Khare (Docker Captain), CloudYuga. Service Discovery and Load Balancing with Docker Swarm &8211; Ajeeth S. Raina (Docker Captain), Dell. Docker Application Bundle Overview &8211; Thomas Chacko. Logging as a service using Docker &8211; Manoj Goyal, Cisco. SDN-Like App Delivery Controller using Docker Swarm &8211; Prasad Rao, Avi Networks.
RSVP

Oceania 
Aug 1st: Docker Meetup in Auckland &8211; Auckland, New Zealand
Learn about all the new Docker features and offerings announced at DockerCon16 in Seattle!
RSVP
Aug 8th: Docker Meetup at Commbank &8211; Sydney, Australia
The Big Debate: AWS v Azure vs Google Cloud vs EMC Hybrid Cloud. One of the questions will help bring to light each platform&8217;s integration with the Docker ecosystem.
RSVP

Africa
Aug 6th: Docker Meetup at LakeHub &8211; Kisumu, Kenya
Please join us to learn about all the exciting announcements from DockerCon! Talk 1: What&8217;s New in Docker 1.12, by William Ondenge. In this presentation, William will describe Docker 1.12 new features and help you get your hands on the latest builds of Docker to try them on your own.
RSVP
// <![CDATA[
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s); js.id=id;js.async=true;js.src="https://a248.e.akamai.net/secure.meetupstatic.com/s/script/2012676015776998360572/api/mu.btns.js?id=65gk05ie6n07ijoq3eq5vchs6f";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","mu-bootjs");
// ]]>
Quelle: https://blog.docker.com/feed/

In cloud’s ‘second wave,’ hybrid is the innovator’s choice

In the cloud business, there&;s plenty of &;tech talk&; about APIs, containers, object storage and any number of other IT topics. I don&8217;t discount its value, but my view of cloud is a little different because my job begins and ends with IBM clients&8217; success in adopting cloud, nothing more or less. As a result, I [&;]
The post In cloud&8217;s ‘second wave,’ hybrid is the innovator’s choice appeared first on Thoughts On Cloud.
Quelle: Thoughts on Cloud

Unexpected continuous location tracking/energy change in android?

So this is really weird, but I have found what seems to be unexpected continuous location tracking that is causing noticeable battery drain on Android 6.0. Right now, it&;s looking like a change in an automatically updated component, so it is probably due to a closed source service or app. So this is in the style of the work from Vern Paxson&8217;s group on characterizing the observed behavior of third party software.
Has anybody else running Android 6.0 noticed a particularly large increase in power drain, with the GPS icon displayed continuously? I will be running additional tests in the coming days, but wanted to report the unusual behavior and see if other researchers have noticed it as well, or want to investigate it while it lasts.
Background

I&8217;ve been doing power profiling of power drain under various regimes as part of understanding the power/accuracy tradeoffs for my travel pattern tracking project. So I basically install apps with different data collection regimes on multiple test phones of the same make, model and OS version, and carry all of them around for comparison.

From last Thu/Fri/Sat, it looks like the power drain behavior on android has changed dramatically. In particular, it looks like some system component has GPS location turned on continuously, and is draining the battery quite dramatically. See details below.
This is a Nexus 6 running a stock android kernel (v 6.0.1, patch level: March 1, 2016), with no non-OEM apps installed other than mine, and with google maps location history turned off, so this must be due to unexpected background access by either the OS or some stock google app. And since I didn&8217;t update the OS, my guess is that it is a closed source component such as google play services or google maps that is automatically updated/patched.
Details
Here are the graphs for power drain on Sat v/s Tue v/s Thu v/s Fri. I think that the change happened sometime during the day on Thursday, because I know that the GPS icon was off on phones 2 and 4 on Thursday morning and was displayed on Thursday night. It was gone again when I rebooted on Thu, but came back again sometime on Friday. Has been on ever since then, even after rebooting.

Battery levels when tracking was off on the same phone (note the higher drain on Thu and the big change on Fri + Sat)

 Before we compare levels across phones, we need to understand the data collection regimes for each of them.

table#{
border:1px solid !important;
border:none !important;
border-collapse: collapse !important;white-space: pre;}
tablew3t945 td {
border:none !important;
border-top:1px solid !important;
border-bottom:1px solid !important;
}
tablew3t945 td {border-color:bbbbbb !important;}

Phone 1Phone 2Phone 3Phone 4

Sattracking offtracking offtracking offtracking off

Tuehigh, 1 secmed, 1 sechigh, 15-30 secmed, 15-30 sec

Thuhigh, 1 secmed, 1 sechigh, 30 sectracking off

Fri + Sathigh, 1 secmed, 1 sechigh, 30 sectracking off

Next Tuehigh, 1 secmed, 1 sechigh, 30 sectracking off

 

Battery levels across phones (note the abrupt phase change that happens on Fri+Sat, and how the change is staggered across phones, consistent with an automatically updated component)

It is clear that on Tuesday, phones 2 and 3 are fairly close to each other, and both are very different from phone 1. This is consistent with intuition and results before Thursday as well.
On Thursday, the difference between phone 1 and phone 2 is much less pronounced, and the difference between phone 2 and phone 3 is also much larger. On Friday and this Tuesday, there is essentially no difference between high and medium accuracy at the fast sampling rate (phone 1 and phone 2), and no difference between slow sampling and no tracking (phone 3 and phone 4).
I also note that the GPS icon is constantly turned on, even on the phone where the tracking is stopped.

Of course, this could be a bug in my code, but:

I didn&8217;t really change the code between Tue and Thu, and
I don&8217;t get the notifications about activity changes on the phone where it is turned off, and
my app does not show up in the location or battery drain screens

Next steps
In the next few days, I plan to poke around at this some more to see if I can figure out what&8217;s going on. Some thoughts are:

uninstall my app. This is very annoying because then I have to record the battery level manually, but I can suck it up for a day.
uninstall potential culprits &; google play services, maps, ??? It turns out that most of these are system services that cannot be uninstalled, but I can try disabling them.
&;&8211;> your suggestion here <&8212;&8212;&8212; If you have any thoughts on things to try, let me know!

We can do this together
This is complicated because we are trying to treat the phone like a natural phenomenon that we cannot control but can try to understand through observation. I&8217;d love to hear from other members of the community so that we can figure out whether google is really continuously tracking us without letting us know, and killing our battery while doing so.

Quelle: Amplab Berkeley

Technical Preview of Apache Spark 2.0: Easier, Faster, and Smarter

This is a guest blog post originally published on the Databricks blog.
 
For the past few months, we have been busy working on the next major release of the big data open source software we love: Apache Spark 2.0. Since Spark 1.0 came out two years ago, we have heard praises and complaints. Spark 2.0 builds on what we have learned in the past two years, doubling down on what users love and improving on what users lament. While this blog summarizes the three major thrusts and themes—easier, faster, and smarter—that comprise Spark 2.0, the themes highlighted here deserve deep-dive discussions that we will follow up with in-depth blogs in the next few weeks.
Prior to the general release, a technical preview of Apache Spark 2.0 is available on Databricks. This preview package is built using the upstream branch-2.0. Using the preview package is as simple as selecting the “2.0 (branch preview)” version when launching a cluster.
Whereas the final Apache Spark 2.0 release is still a few weeks away, this technical preview is intended to provide early access to the features in Spark 2.0 based on the upstream codebase. This way, you can satisfy your curiosity to try the shiny new toy, while we get feedback and bug reports early before the final release.
Now, let’s take a look at the new developments.

Easier: SQL and Streamlined APIs
One thing we are proud of in Spark is creating APIs that are simple, intuitive, and expressive. Spark 2.0 continues this tradition, with focus on two areas: (1) standard SQL support and (2) unifying DataFrame/Dataset API.
On the SQL side, we have significantly expanded the SQL capabilities of Spark, with the introduction of a new ANSI SQL parser and support for subqueries. Spark 2.0 can run all the 99 TPC-DS queries, which require many of the SQL:2003 features. Because SQL has been one of the primary interfaces Spark applications use, this extended SQL capabilities drastically reduce the porting effort of legacy applications over to Spark.
On the programming API side, we have streamlined the APIs:

Unifying DataFrames and Datasets in Scala/Java: Starting in Spark 2.0, DataFrame is just a type alias for Dataset of Row. Both the typed methods (e.g. map, filter, groupByKey) and the untyped methods (e.g. select, groupBy) are available on the Dataset class. Also, this new combined Dataset interface is the abstraction used for Structured Streaming. Since compile-time type-safety in Python and R is not a language feature, the concept of Dataset does not apply to these languages’ APIs. Instead, DataFrame remains the primary programing abstraction, which is analogous to the single-node data frame notion in these languages. Get a peek from a Dataset API notebook.
SparkSession: a new entry point that replaces the old SQLContext and HiveContext. For users of the DataFrame API, a common source of confusion for Spark is which “context” to use. Now you can use SparkSession, which subsumes both, as a single entry point, as demonstrated in this notebook. Note that the old SQLContext and HiveContext are still kept for backward compatibility.
Simpler, more performant Accumulator API: We have designed a new Accumulator API that has a simpler type hierarchy and support specialization for primitive types. The old Accumulator API has been deprecated but retained for backward compatibility
DataFrame-based Machine Learning API emerges as the primary ML API: With Spark 2.0, the spark.ml package, with its “pipeline” APIs, will emerge as the primary machine learning API. While the original spark.mllib package is preserved, future development will focus on the DataFrame-based API.
Machine learning pipeline persistence: Users can now save and load machine learning pipelines and models across all programming languages supported by Spark.
Distributed algorithms in R: Added support for Generalized Linear Models (GLM), Naive Bayes, Survival Regression, and K-Means in R.

Faster: Spark as a Compiler
According to our 2015 Spark Survey, 91% of users consider performance as the most important aspect of Spark. As a result, performance optimizations have always been a focus in our Spark development. Before we started planning for Spark 2.0, we asked ourselves a question: Spark is already pretty fast, but can we push the boundary and make Spark 10X faster?
This question led us to fundamentally rethink the way we build Spark’s physical execution layer. When you look into a modern data engine (e.g. Spark or other MPP databases), majority of the CPU cycles are spent in useless work, such as making virtual function calls or reading/writing intermediate data to CPU cache or memory. Optimizing performance by reducing the amount of CPU cycles wasted in these useless work has been a long time focus of modern compilers.
Spark 2.0 ships with the second generation Tungsten engine. This engine builds upon ideas from modern compilers and MPP databases and applies them to data processing. The main idea is to emit optimized bytecode at runtime that collapses the entire query into a single function, eliminating virtual function calls and leveraging CPU registers for intermediate data. We call this technique “whole-stage code generation.”
To give you a teaser, we have measured the amount of time (in nanoseconds) it would take to process a row on one core for some of the operators in Spark 1.6 vs. Spark 2.0, and the table below is a comparison that demonstrates the power of the new Tungsten engine. Spark 1.6 includes expression code generation technique that is also in use in some state-of-the-art commercial databases today. As you can see, many of the core operators are becoming an order of magnitude faster with whole-stage code generation.
You can see the power of whole-stage code generation in action in this notebook, in which we perform aggregations and joins on 1 billion records on a single machine.

cost per row (single thread)

primitive
Spark 1.6
Spark 2.0

filter
15ns
1.1ns

sum w/o group
14ns
0.9ns

sum w/ group
79ns
10.7ns

hash join
115ns
4.0ns

sort (8-bit entropy)
620ns
5.3ns

sort (64-bit entropy)
620ns
40ns

sort-merge join
750ns
700ns

How does this new engine work on end-to-end queries? We did some preliminary analysis using TPC-DS queries to compare Spark 1.6 and Spark 2.0:

Beyond whole-stage code generation to improve performance, a lot of work has also gone into improving the Catalyst optimizer for general query optimizations such as nullability propagation, as well as a new vectorized Parquet decoder that has improved Parquet scan throughput by 3X.
Smarter: Structured Streaming
Spark Streaming has long led the big data space as one of the first attempts at unifying batch and streaming computation. As a first streaming API called DStream and introduced in Spark 0.7, it offered developers with several powerful properties: exactly-once semantics, fault-tolerance at scale, and high throughput.
However, after working with hundreds of real-world deployments of Spark Streaming, we found that applications that need to make decisions in real-time often require more than just a streaming engine. They require deep integration of the batch stack and the streaming stack, integration with external storage systems, as well as the ability to cope with changes in business logic. As a result, enterprises want more than just a streaming engine; instead they need a full stack that enables them to develop end-to-end “continuous applications.”
One school of thought is to treat everything like a stream; that is, adopt a single programming model integrating both batch and streaming data.
A number of problems exist with this single model. First, operating on data as it arrives in can be very difficult and restrictive. Second, varying data distribution, changing business logic, and delayed data—all add unique challenges. And third, most existing systems, such as MySQL or Amazon S3, do not behave like a stream and many algorithms (including most off-the-shelf machine learning) do not work in a streaming setting.
Spark 2.0&;s Structured Streaming APIs is a novel way to approach streaming. It stems from the realization that the simplest way to compute answers on streams of data is to not having to reason about the fact that it is a stream. This realization came from our experience with programmers who already know how to program static data sets (aka batch) using Spark’s powerful DataFrame/Dataset API. The vision of Structured Streaming is to utilize the Catalyst optimizer to discover when it is possible to transparently turn a static program into an incremental execution that works on dynamic, infinite data (aka a stream). When viewed through this structured lens of data—as discrete table or an infinite table—you simplify streaming.
As the first step towards realizing this vision, Spark 2.0 ships with an initial version of the Structured Streaming API, a (surprisingly small!) extension to the DataFrame/Dataset API. This unification should make adoption easy for existing Spark users, allowing them to leverage their knowledge of Spark batch API to answer new questions in real-time. Key features here will include support for event-time based processing, out-of-order/delayed data, sessionization and tight integration with non-streaming data sources and sinks.
Streaming is clearly a pretty broad topic, so stay tuned to this blog for more details on Structured Streaming in Spark 2.0, including details on what is possible in this release and what is on the roadmap for the near future.
Conclusion
Spark users initially came to Spark for its ease-of-use and performance. Spark 2.0 doubles down on these while extending it to support an even wider range of workloads. We hope you will enjoy the work we have put it in, and look forward to your feedback.
Of course, until the upstream Apache Spark 2.0 release is finalized, we do not recommend fully migrating any production workload onto this preview package. This technical preview version is now available on Databricks. You can sign up for an account here.
Read More
If you missed our webinar for Spark 2.0: Easier, Faster, and Smarter, you can register and watch the recordings and download slides and attached notebooks.
You can also import the following notebooks and try on Databricks Community Edition with Spark 2.0 Technical Preview.

SparkSession: A new entry point
Datasets: A more streamlined API
Performance of whole-stage code generation
Machine learning pipeline persistence

Quelle: Amplab Berkeley