Monat: Juni 2019

Compute and stream IoT insights with data-driven applications

There is a lot more data in the world than can possibly be captured with even the most robust, cutting-edge technology. Edge computing and the Internet of Things (IoT) are just two examples of technologies increasing the volume of useful data. There is so much data being created that the current telecom infrastructure will struggle to transport it and even the cloud may become strained to store it. Despite the advent of 5G in telecom, and the rapid growth of cloud storage, data growth will continue to outpace the capacities of both infrastructures. One solution is to build stateful, data-driven applications with technology from SWIM.AI.

The Azure platform offers a wealth of services for partners to enhance, extend, and build industry solutions. Here we describe how one Microsoft partner uses Azure to solve a unique problem.

Shared awareness and communications

The increase in volume has other consequences, especially when IoT devices must be aware of each other and communicate shared information. Peer-to-peer (P2P) communications between IoT assets can overwhelm a network and impair performance. Smart grids are an example of how sensors or electric meters are networked across a distribution grid to improve the overall reliability and cost of delivering electricity. Using meters to determine the locality of issues can help improve service to a residence, neighborhood, municipality, sector, or region. The notion of shared awareness extends to vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. As networked AI spreads to more cars and devices, so do the benefits of knowing the performance or status of other assets. Other use cases include:

Traffic lights that react to the flow of vehicles across a neighborhood.
Process manufacturing equipment that can determine the impact from previous process steps.
Upstream oil/gas equipment performance that reacts to downstream oil/gas sensor validation.

Problem: Excess data means data loss

When dealing with large volumes of data, enterprises often struggle to determine which data to retain, how much to retain, and for how long they must retain it. By default, they may not retain any of it. Or, they may sub-sample data and retain an incomplete data set. That lost data may potentially contain high value insights. For example, consider traffic information that could be used for efficient vehicle routing, commuter safety, insurance analysis, and government infrastructure reviews. The city of Las Vegas maintains over 1,100 traffic light intersections that can generate more than 45TB of data every day. As stated before, IoT data will challenge our ability to transport and store data at these volumes.

Data may also become excessive when it’s aggregated. For example, telecom and network equipment typically create snapshots of data and send it every 15 minutes. By normalizing this data into a summary over time, you lose granularity. This means the nature or pattern of data over time along with any unique, intuitive events would be missed. The same applies to any equipment capturing fixed-time, window summary data. The loss of data is detrimental to networks where devices share data, either for awareness or communication. The problem is also compounded, as only snapshots are captured and aggregated for an entire network of thousands or millions of devices. Real-time is the goal.

Real-time is the goal

Near real-time is the current standard for stateless application architectures, but “near” real-time is not fast enough anymore. Real-time processing or processing within milliseconds is the new standard for V2V or V2I communications and requires a much more performant architecture. Swim does this by leveraging stateful API’s. With stateful connections, it’s possible to have a rapid response between peers in a network. Speed has enormous effects on efficiency and reliability and it’s essential for systems where safety is paramount such as preventing crashes. Autonomous systems will rely on real-time performance for safety purposes.

An intelligent edge data strategy

SWIM.AI delivers a solution for building scalable streaming applications. According to their site Meet Swim:

“Instead of configuring a separate message broker, app server and database, Swim provides for its own persistence, messaging, scheduling, clustering, replication, introspection, and security. Because everything is integrated, Swim seamlessly scales across edge, cloud, and client, for a fraction of the infrastructure and development cost of traditional cloud application architectures.”

The figure below shows an abstract view of how Swim can simplify IoT architectures:

Harvest data in mid-stream

SWIM.AI uses the lightweight Swim platform, only generating a 2MB footprint to compute and stream IoT insights, building what they call “data-driven applications.” These applications sit in the data stream and generate unique, intelligent web agents for each data source it sees. These intelligent web agents then process the raw data as it streams, only publishing state changes from the data stream. This streamed data can be used by other web agents or stored in a data lake, such as Azure.

Swim uses the “needle in a haystack” metaphor to explain this unique advantage. Swim allows you to apply a metal detector while harvesting the grain to find the needle, without having to bail, transport, or store the grain before searching for the needle. The advantage is in continuously processing data, where intelligent web agents can learn over time or be influenced by domain experts that set thresholds.

Because of the stateful architecture of Swim, only the minimum data necessary is transmitted over the network. Furthermore, application services need not wait for the cloud to establish application context. This results in extremely low latencies, as the stateful connections don’t incur the latency cost of reading and writing to a database or updating based on poll requests.

On SWIM.AI’s website, a Smart City application shows the real-time status of lights and traffic across a hundred intersections with thousands of sensors. The client using the app could be a connected or an autonomous car approaching the intersection. It could be a handheld device next to the intersection, or a browser a thousand miles away in the contiguous US. The latency to real-time is 75-150ms, less than the blink of an eye across the internet.

Benefits

The immediate benefit is saving costs for transporting and storing data.
Through Swim’s technology, you can retain the granularity. For example, take the case of 10 seconds of TB per day generated from every 1000 traffic light intersections. Winnow that data down to 100 seconds of GB per day. But the harvested dataset fully describes the original raw dataset.
Create efficient networked apps for various data sources. For example, achieve peer-to-peer awareness and communications between assets such as vehicles, devices, sensors, and other data sources across the internet.
Achieve ultra-low latencies in the 75-150 millisecond range. This is the key to creating apps that depend on data for awareness and communications.

Azure services used in the solution

The demonstration of DataFabric from SWIM.AI relies on core Azure services for security, provisioning, management, and storage. DataFabric also uses the Common Data Model to simplify sharing information with other systems, such as Power BI or PowerApps, in Azure. Azure technology enables the customer’s analytics to be integrated with events and native ML and cognitive services.

DataFabric is based on the Microsoft IoT reference architecture and uses the following core components:

IoT Hub: Provides a central point in the cloud to manage devices and their data.
IoT Edge Field gateway: An on-premises solution for delivering cloud intelligence.
Azure Event Hubs: Ingests millions of events per second.
Azure Blob: Efficient storage that includes options for hot, warm and archived data.
Azure Data Lake storage: A highly scalable and cost-effective data lake solution for big data analytics.
Azure Streaming Analytics: For transforming data into actionable insights and predictions in near real-time.

Next steps

To learn more about other industry solutions, go to the Azure for Manufacturing page.

To find out more about this solution, go to DataFabric for Azure IoT and select Get it now.
Quelle: Azure

Azure.Source – Volume 86

News and updates

Microsoft hosts HL7 FHIR DevDays

One of the largest gatherings of healthcare IT developers will come together on the Microsoft campus June 10-12 for HL7 FHIR DevDays, with the goal of advancing the open standard for interoperable health data, called HL7® FHIR® (Fast Healthcare Interoperability Resources, pronounced “fire”). Microsoft is thrilled to host this important conference, and engage with the developer community on everything from identifying immediate use cases to finding ways for all of us to hack together in ways that help advance the FHIR specification.

Announcing self-serve experience for Azure Event Hubs Clusters

For businesses today, data is indispensable. Innovative ideas in manufacturing, health care, transportation, and financial industries are often the result of capturing and correlating data from multiple sources. Now more than ever, the ability to reliably ingest and respond to large volumes of data in real time is the key to gaining competitive advantage for consumer and commercial businesses alike. To meet these big data challenges, Azure Event Hubs offers a fully managed and massively scalable distributed streaming platform designed for a plethora of use cases from telemetry processing to fraud detection.

A look at Azure's automated machine learning capabilities

The automated machine learning capability in Azure Machine Learning service allows data scientists, analysts, and developers to build machine learning models with high scalability, efficiency, and productivity all while sustaining model quality. With the announcement of automated machine learning in Azure Machine Learning service as generally available last December, we have started the journey to simplify artificial intelligence (AI). We are furthering our investment for accelerating productivity with a new release that includes exciting capabilities and features in the areas of model quality, improved model transparency, the latest integrations, ONNX support, a code-free user interface, time series forecasting, and product integrations.

Technical content

Securing the hybrid cloud with Azure Security Center and Azure Sentinel

Infrastructure security is top of mind for organizations managing workloads on-premises, in the cloud, or hybrid. Keeping on top of an ever-changing security landscape presents a major challenge. Fortunately, the power and scale of the public cloud has unlocked powerful new capabilities for helping security operations stay ahead of the changing threat landscape. Microsoft has developed a number of popular cloud based security technologies that continue to evolve as we gather input from customers. This post breaks down a few key Azure security capabilities and explain how they work together to provide layers of protection.

Customize your automatic update settings for Azure Virtual Machine disaster recovery

In today’s cloud-driven world, employees are only allowed access to data that is absolutely necessary for them to effectively perform their job. The ability to hence control access but still be able to perform job duties aligning to the infrastructure administrator profile is becoming more relevant and frequently requested by customers. When we released the automatic update of agents used in disaster recovery (DR) of Azure Virtual Machines (VMs), the most frequent feedback we received was related to access control. The request we heard from you was to allow customers to provide an existing automation account, approved and created by a person who is entrusted with the right access in the subscription. You asked, and we listened!

Azure Stack IaaS – part nine

Before we built Azure Stack, our program manager team called a lot of customers who were struggling to create a private cloud out of their virtualization infrastructure. We were surprised to learn that the few that managed to overcome the technical and political challenges of getting one set up had trouble getting their business units and developers to use it. It turns out they created what we now call a snowflake cloud, a cloud unique to just their organization. This is one of the main problems we were looking to solve with Azure Stack. A local cloud that has not only automated deployment and operations, but also is consistent with Azure so that developers and business units can tap into the ecosystem. In this blog we cover the different ways you can tap into the Azure ecosystem to get the most value out of IaaS.

What is the difference between Azure Application Gateway, Load Balancer, Front Door and Firewall?

Last week at a conference in Toronto, an attendee came to the Microsoft booth and asked something that has been asked many times in the past. So, this blog post covers all of it here for everyone’s benefit. What are the differences between Azure Firewall, Azure Application Gateway, Azure Load Balancer, Network Security Groups, Azure Traffic Manager, and Azure Front Door? This blog offers a high-level consolidation of what they each do.

Azure shows

Five tools for building APIs with GraphQL | Five Things

Burke and Chris are back and this week they're bringing you five tools for building API's with GraphQL. True story, they shot this at the end of about a twelve hour day and you can see the pain in Burke's eyes. It's not GraphQL he doesn't like, it's filming for six straight hours. Also, Chris picks whistles over bells (because of course he does) and Burke fights to stay awake for four minutes.

Microservices and more in .NET Core 3.0 | On .NET

Enabling developers to build resilient microservices is an important goal for .NET Core 3.0. In this episode, Shayne Boyer is joined by Glenn Condron and Ryan Nowak from the ASP.NET team who discuss some of the exciting work that's happening in the microservice space for .NET Core 3.0.

Interknowlogy mixes Azure IoT and mixed reality | The Internet of Things Show

When mixed reality meets the Internet of Things through Azure Digital Twins, a new way of accessing data materializes. See how Interknowlogy mixes Azure IoT and Mixed Reality to deliver not only stunning experiences but also accrued efficiency and productivity to workforce.

Bring DevOps to your open-source projects: Top three tips for maintainers | The Open Source Show

Baruch Sadogurksy, Head of Developer Relations at JFrog, and Aaron Schlesinger, Cloud Advocate at Microsoft and Project Athens Maintainer, talk about the art of DevOps for Open Source. Balancing contributor needs with the core DevOps principles of people, process, and tools. You'll learn how to future-proof your projects, avoid the dreaded Bus Factor, and get Aaron and Baruch's advice for evaluating and selecting tools, soliciting contributor input and voting, documenting processes, and so much more.

Episode 282 – Azure Front Door Service | The Azure Podcast

Cynthia talks with Sharad Agrawal on what Azure Front Door Service is, how to choose between Azure Front Door Service, CDN, Azure Traffic Manager and App Gateway, and how to get started.

HTML5 audio not supported

Atley Hunter on the Business of App Development | Azure DevOps Podcast

In this episode, Jeffrey and Atley are discussing the business of app development. Atley describes some of the first apps he’s ever developed, some of the most successful and popular apps he’s ever created, how he’s gone about creating these apps, and gives his tips for other developers in the space.

Industries and partners

Empowering clinicians with mobile health data: Right information, right place, right time

Improving patient outcomes and reducing healthcare costs depends on healthcare providers such as doctors, nurses, and specialized clinician ability to access a wide range of data at the point of patient care in the form of health records, lab results, and protocols. Tactuum, a Microsoft partner, provides the Quris solution that empowers clinicians with access to the right information, the right place, at the right time, enabling them to do their jobs efficiently and with less room for error.

Building a better asset and risk management platform with elastic Azure services

Elasticity means services can expand and contract on demand. This means Azure customers who are on a pay-as-you-go plan will reap the most benefit out of Azure services. Their service is always available, but the cost is kept to a minimum. Together with elasticity, Azure lets modern enterprises migrate and evolve more easily. For financial service providers, the modular approach lets customers benefit from best-of-breed analytics in three key areas. Read the post to learn what they are.

Symantec’s zero-downtime migration to Azure Cosmos DB

How do you migrate live, mission-critical data for a flagship product that must manage billions of requests with low latency and no downtime? The consumer business unit at Symantec faced this exact challenge when deciding to shift from their costly and complex self-managed database infrastructure, to a geographically dispersed and low latency managed database solution on Azure. The Symantec team shared their business requirements and decision to adopt Azure Cosmos DB in a recent case study.
Quelle: Azure