Monat: September 2017

Query across resources

We’re excited to introduce cross-resources querying – the ability to query not only the current workspace or application, but analyze data from other resources as well, in a single query.

Until now, queries were limited to the scope of a single Application Insights app, or a single Log Analytics workspace. Today, we support querying across multiple apps or across multiple workspaces, providing a true system-wide view on your data.

Querying across Application Insights apps

Refer to an external application by using an app identifier:

union app('mmsportal-prod').requests, app('fabrikamapp').requests, requests
| summarize count() by bin(timestamp, 1h)

The above example queries records of the requests table in 3 separate apps: mmsportal-prod, fabrikamprod and my current app (which doesn’t require a name, I refer directly to the table). It then counts the total number of records, regardless of the application that holds each record.

Querying across Log Analytics workspaces

Refer to an external workspace by using a workspace identifier:

union Update, workspace("contosoretail-it").Update
| where TimeGenerated >= ago(1h)
| where UpdateState == "Needed"
| summarize dcount(Computer) by Classification

The above example queries the Update table both in my current workspace, and in another workspace named contosoretail-it. It then counts distinct records of needed updates by their classification, regardless of the workspace that holds each record.

Identifying resources

Identifying applications and workspace can be done in several ways:

Resource name – this is a human-readable name of the app or workspace. We sometimes refer to this as the “component name”.
workspace("contosoretail").Update | count
Note: Since workspace and app names are not unique across subscriptions or resource groups, this identifier can be ambiguous if the user has access to multiple components with the same name. In such cases the query will fail on ambiguity.
Qualified Name – this is the “full name” of the app or workspace, composed of thei subscription name, resource group and component name, in this format: <subscriptionName>/<resourceGroup>/<componentName>.
app('AI-Prototype/Fabrikam/fabrikamprod').requests | count
Note: Since Azure subscription names are not unique, this identifier might be ambiguous.
App or workspace ID – this is a GUID, the unique, immutable, public identifier of the app or workspace:
workspace("b438b4f6-912a-46d5-9cb1-b44069212ab4").Update | count

Azure Resource ID – the Azure-defined identity of the app or workspace.

For apps, the format is: /subscriptions/<subscriptionId>/resourcegroups/<resourceGroup>/providers/microsoft.insights/components/<componentName>.
For example:
app("/subscriptions/7293b69-db12-44fc-9a66-9c2005c3051d/resourcegroups/Fabrikam/providers/microsoft.insights/components/fabrikamapp").requests | count

For workspaces, the format is: /subscriptions/<subscriptionId>/resourcegroups/<resourceGroup>/providers/microsoft.OperationalInsights/workspaces/<componentName>.
For example:
workspace("/subscriptions/e427267-5645-4c4e-9c67-3b84b59a6982/resourcegroups/ContosoAzureHQ/providers/Microsoft.OperationalInsights/workspaces/contosoretail").Event | count

Favorite resources

A cool new feature is “Favorites” –  in the Analytics portal, the Scehma pane now has a list of your favorite resources, intended to provide quick access to the resources you query the most. To add an item to the list, you can either hover over the currently active resource and click the star icon, or select the "Edit" button and search for the relevant resource

In the context of cross-resource querying, you’ll notice that Intellisense suggests identifiers based on your favorites. Regardless of suggestions, you can in fact refer to any resource you have access to.

Next steps

Learn more on how to analyze your data:

· Query language doc site

· Getting started with queries
Quelle: Azure

Get started with Monitoring in Azure

We’re happy to announce the public preview of a new overview landing page in Azure Monitor. This landing page is designed to help you understand the monitoring capabilities offered by Azure, and to make it easier to discover, configure, and on-board Azure’s platform and premium monitoring capabilities.

The new Overview helps users that are new to Azure get started by on-boarding Azure alerts, Log Analytics, and Application Insights. It also provides a view to Azure’s always-on platform monitoring, starting with Activity Log error counts and an Azure Service Health summary that helps you catch any failure points in your environment.

As you on-board to richer capabilities, the overview gives you a starting point for navigation, and shows notable issues from different services to let you know if you should take a look at them. You might use the overview for a quick daily review of environment health, or to see what else needs attention after you receive an alert. Today, the overview can be scoped to a single subscription at a time, so the view shows you the health and configuration for the part of your environment you choose. We’ll be expanding out the scope of visibility in future releases.

If any of your Azure resources are logging events with error-level severity, the Activity Log Error count shows you this key indicator, and you can then click through to the dedicated Activity Log page to investigate each event. Similarly, Azure Service Health, which provides personalized information about any issues in Azure that are impacting your services, gives you an always-on view to service issues, planned maintenance events, and health advisories.

We recommend three core services to get more visibility to your Azure resources. Configuring Azure Alerts is a great way to get notified of any unexpected performance degradations or unexpected activity on your resources. If you don’t have Log Analytics set up for your subscription, we’ll guide you to get started so you can unlock deep insights on your data.

Finally, the Azure Monitor overview gives you a new high-level view to your Application Insights monitoring, showing you KPIs for load, latency, failures, and availability. In addition to the alerting and highly customizable workflows you can set up inside Application Insights, the Overview provides a quick view to your application health to see which ones are worth checking on. The Application Insights table is optimized for server-side application monitoring across ASP.NET web apps, Java, and Node.js applications.

We’ll be continuing to expand this page to cover more of Azure’s monitoring capabilities, and to make it as easy as possible to discover and navigate to the monitoring capabilities that are right for your environment. If you have any feedback, please reach us on User Voice.
Quelle: Azure

Azure SQL Database VNET Service Endpoints now in public preview

We are excited to announce that Azure SQL Database and Azure SQL Data Warehouse VNET Service Endpoints are now in public preview in the following regions: West Central US, West US2, and East US1.

This feature allows you to isolate connectivity to your SQLDB to only a given Subnet or set of Subnets within your VNET(s). Even though the connectivity will be on Azure SQL Databases public endpoint, the traffic will stay within the Azure backbone network. This direct route will be preferred over any forced-tunneling route to take Internet traffic back to on-premises. We also provide for separation of roles with the ability to provision VNET Service Endpoints either on the Network Admin, the Database Admin, splitting the roles between these two, or the ability to create a new entity with the help of custom RBAC roles. The following diagram gives more information on the architecture:

Limitations

Each SQL Server can have up to 128 Virtual Network based ACLs
Applies only to ARM VNETs

This does not extend to on-premises via Expressroute, Site-to-Site (S2S) VPN, or Peered VNets.

Considerations

At the time of this preview, Network Security Groups (NSGs) should be opened to the Internet to allow Azure SQL Database traffic. In future, NSGs could be opened to only IP ranges for the PaaS services. IP tags for Azure SQL Database are on the roadmap for CY17.

With VNET Service Endpoints, source IP addresses of resources in your VNet's subnet will switch from using public IPV4 addresses to VNet's private addresses, for traffic to Azure SQL Database. Any existing open TCP connections to your databases service may be closed during this switch. Please make sure no critical tasks are run when Service Endpoints is turned on or off.

If traffic to Azure SQL Database is to be inspected by a network virtual appliance (NVA), it is recommended that VNET Service Endpoints is turned on for the NVA subnet, instead of the subnet where the Azure SQL Database is originating from in the given VNET.

When Service Endpoints is turned on, a Subnet it is sequentially applied to all VMs in that Subnet. The call commits only when Service Endpoints is successfully applied to all VMs. You will be able to ACL given VNET/Subnet your Server only after Service Endpoints from the VNET/Subnet is successfully applied. So there can be potential downtime after the Service Endpoints call is issued until when you ACL the Server.

To learn more check out VNet Service Endpoints and rules for Azure SQL Database.
Quelle: Azure

General availability of HDInsight Interactive Query – blazing fast queries on hyper-scale data

It’s 2017, and big data challenges are as real as they get. Our customers have petabytes of data living in elastic and scalable commodity storage systems such as Azure Data Lake Store and Azure Blob storage.

One of the central questions today is finding insights from data in these storage systems in an interactive manner, at a fraction of the cost. 

Interactive Query leverages [Hive on LLAP] in Apache Hive 2.1, brings the interactivity to your complex data warehouse style queries on large datasets stored on commodity cloud storage.

Today, we announce the general availability of the Interactive Query cluster type in Azure HDInsight (formerly known as Interactive Hive). With this offering, we are bringing the following benefits to our customers:

Fast Data warehouse style SQL queries on petabyte-scale data

Intelligent caching and optimizations in Interactive Query produces blazing-fast query results on remote Cloud storage, such as Azure Blob and Azure Data Lake Store.

Interactive Query enables data analysts to query data interactively in the same storage where data is prepared, eliminating the need for moving data from storage to another analytical engine for data warehousing needs. With zero data migration, you gain faster insights, operational resiliency, and reduced efforts, as well as simplified architecture.

Modern scalable query concurrency architecture

With the introduction of much improved fine-grain resource management and preemption, Interactive Query [Hive on LLAP] makes it better for concurrent users. In addition, HDInsight supports creating multiple clusters on shared Azure storage, and Hive metastore helps in achieving a high degree of concurrency

Rich connectivity with the most popular authoring tools

Interactive Query enables end-users to consume data from rich business intelligence tools, such as PowerBI, Tableau, Excel, Hive View 2.0, Beeline, Hive CLI, and Visual Studio, as well as built-in Zeppelin notebook.

Today, we are happy to announce the preview of Interactive Query tools for Visual Studio code. Rich connectivity options eliminate user learning curves so that they are more productive sooner.

Leverage your existing investments in HDInsight by sharing the data and Hive metastore

If you already run your Batch and ETL workloads in HDInsight, leveraging Interactive Query cluster for fast querying is straightforward. Customers can attach an Interactive Query cluster to existing metastore and data storage, and start querying the data right away.

Achieve low latency with SSD caching without the cost of SSDs

Interactive Query SSD Cache enables you to combine RAM and SSD into a giant pool of memory with all of the other benefits the LLAP cache brings. By using the LLAP SSD cache, a typical daemon can cache four times more data, letting you process larger datasets or support more users. In HDInsight, cluster nodes have built-in SSD at no extra cost.

Say no to data format conversion in order to get faster results

Fast analytics on Hadoop have always come with one big catch: they require up-front conversion to a columnar format like ORCFile, Parquet or Avro, which is time-consuming, complex and limits your agility. With Interactive Query Dynamic Text Cache, which converts CSV or JSON data into optimized in-memory format on-the-fly, caching is dynamic, so the queries determine what data is cached. After text data is cached, analytics run just as fast as if you had converted it to specific file formats.

Enterprise Grade Security and Monitoring (preview)

Interactive Query is built on top of highly secure Azure & HDInsight Platform. With features such as, Domain-joined HDInsight clusters, you can create an interactive query cluster joined to an Active Directory domain, and configure a list of employees from the enterprise who can authenticate through Azure Active Directory to log on to HDInsight cluster.

You can monitor Interactive Query clusters with built-in tools such as Grafana and Ambari, as well as the integration we have built with Azure Log Analytics to monitor all of your resources with a single pane of glass.

Additional resources

Get started with HDInsight Interactive Query Cluster in Azure
Learn more about Azure HDInsight
Use Hive on HDInsight
Open Source component guide on HDInsight
HDInsight release notes
Ask HDInsight questions on Msdn forums
Ask HDInsight questions on stackoverflow

Summary

This week at Ignite, we are pleased to announce general availability of Azure HDInsight Interactive Query. Backed by our enterprise-grade SLA, HDInsight Interactive Query brings sub-second speed to data warehouse style SQL queries to the hyper-scale data stored in commodity cloud storage.
Quelle: Azure

Announcing Preview of Azure Storage Firewalls and Virtual Networks

Enterprises are faced with the daunting challenge of engaging their employees, customers, and partners in today's hostile and dangerous online environement. They are increasingly turning to layered security approaches that help them stay secure in the face of potential failures of process or technology.

To help our customers address this challenge, the Azure Storage team is excited to announce the preview of Firewalls and Virtual Networks for Azure Storage. With this preview, the authorization capabilities of Azure Storage are expanded to allow administrators to also control which networks have access to their data.

By leveraging the recently announced Virtual Network Service Endpoints, administrators can now create network rules that allow traffic only from selected Virtual Networks and subnets, creating a secure network boundary for their data. On-premises networks and other trusted internet traffic can also be granted access using network rules based in IP address ranges. These rules can be managed using the Azure portal, PowerShell, CLIv2, and Azure Resource Manager templates. 

 

This preview is available now for new and existing Storage accounts in the following Azure regions:

East US
West US
West US 2
West Central US
Australia East
Australia Southeast

To learn more and get started with Firewalls and Virtual Networks for Azure Storage, check out Configure Azure Storage Firewalls and Virtual Networks​.  

Thanks!
Quelle: Azure

Announcing Virtual Network integration for Azure Storage and Azure SQL

Today, we are glad to announce the public preview of Virtual Network (VNet) Service Endpoints for Azure Storage and Azure SQL. 

For many of our customers moving their business-critical data to the cloud, data breaches remain a top concern. Various Azure services that store or process the business data have Internet-reachable IP addresses. Leaked credentials or malicious insiders with administrative privileges gaining access to the data, from anywhere in the world, is an increasing concern to our customers.

To protect against these threats, private connectivity to Azure services is becoming essential to moving more critical workloads to the cloud. Most customers want to limit access to their critical resources to only their private environments, i.e. their Azure Virtual Networks and on-premises.

While some of the Azure services can be directly deployed into VNets, many others still remain public. With VNet service endpoints, we are expanding Virtual Network support to more multi-tenant Azure services.

Service endpoints extend your VNet private address space and identity to the Azure services, over a direct connection. This allows you to secure your critical service resources to only your virtual networks, providing private connectivity to these resources and fully removing Internet access.

Configuring service endpoints is very simple with a single click on a subnet in your VNet. Direct route to the services is auto-configured for you. There are no NAT or gateway devices required to set up the endpoints. You also no longer need reserved, public IP addresses in your VNets to secure Azure resources through IP firewall. Service endpoints makes it easy to configure and maintain network security for your critical resources.

Step1: Set up service endpoints once on your Virtual Network. Network administrators can turn this setting independently, allowing for separation of duties.

Step 2: Secure your new or existing Azure service resources to the VNet, with a simple click. Set up once for the Storage account or SQL server and automatically applies to any access to child resources. Data administrators can set up independently (optional).

Service endpoints is available in preview for below services and regions:

Azure Storage: WestUS, EastUS, WestCentralUS, WestUS2, AustraliaEast, and AustraliaSouthEast

Azure SQL: EastUS, WestCentralUS, WestUS2

We will be expanding the feature to more regions soon.

We are very excited to bring enhanced network security for your Azure service resources. This is only a beginning for our roadmap for tightening security for Azure services. We will expand the service endpoints to more Azure services. In addition to service endpoints, we are also very committed to giving you private connectivity to your Azure resources, from your firewalls and on-premises. Service tags is yet another investment in this direction, for your Network Security Groups (NSGs) to selectively open access only to Azure services from your VNets. Service tags is also available in preview now. More enhancements to follow!

Next Steps

To start using VNet Service Endpoints, refer to the documentation.

For full list of network security capabilities in Azure, see “Azure Network security”.

We need your help in improving the features and broadening network security for Azure services. Share your feedback on StackOverflow with the tag “vnet-azure-services”. You can also email us directly at vnetserviceintegration@microsoft.com

For those of you at Microsoft Ignite, we will be covering the feature in more detail in our session on security:  “Network security for applications in Azure”! See you all there!
Quelle: Azure