Announcing new Stackdriver Logging features and expanded free logs limits
By Mary Koes and Deepak Tiwari, Product Managers
When we announced the general availability of Google Stackdriver, our integrated monitoring, logging and diagnostics suite for applications running on cloud, we heard lots of enthusiasm from our user community as well as some insightful feedback:
Analysis – Logs based metrics are great, but you’d like to be able to extract labels and values from logs, too.
Exports – Love being able to easily export logs, but it’s hard to manage them across dozens or hundreds of projects.
Controls – Aggregating all logs in a single location and exporting them various places is fantastic, but you want control over which logs go into Stackdriver Logging.
Pricing – You want room to grow with Stackdriver without worrying too much about the cost of logging all that data.
We heard you, which is why today we’re announcing a variety of new updates to Stackdriver, as well as updated pricing to give you the flexibility to scale and grow.
Here’s a little more on what’s new.
Easier analysis with logs-based metrics
Stackdriver was created with the belief that bringing together multiple signals from logs, metrics, traces and errors can provide greater insight than any single signal. Logs-based metrics are a great example. That’s why the new and improved logs-based metrics are:
Faster – We’ve decreased the time from when a log entry arrives until it’s reflected in a logs-based metric from five minutes to under a minute.
Easier to manage – Now you can extract user-defined labels from text in the logs. Instead of creating a new logs based metric for each possible value, you can use a field in the log entry as a label.
More powerful – Extract values from logs and turn them into distribution metrics. This allows you to efficiently represent many data points at each point in time. Stackdriver Monitoring can then visualize these metrics as a heat map or by percentile.
The example above shows a heat map produced from a distribution metric extracted from a text field in log entries.
Tony Li, Site Reliability Engineer at the New York Times, explains how they use the new user defined labels applied to proxies help them improve reliability and performance from logs.
“With LBMs [Logs based metrics], we can monitor errors that occur across multiple proxies and visualize the frequency based on when they occur to determine regressions or misconfigurations.”
The faster pipeline applies to all logs-based metrics, including the already generally available count-based metrics. Distribution metrics and user labels are now available in beta.
Manage logs across your organization with aggregated exports
Stackdriver Logging gives you the ability to export logs to GCS, PubSub or BigQuery using log sinks. We heard your feedback that managing exports across hundreds or thousands of projects in an organization can sometimes be tedious and error prone. For example, if a security administrator in an organization wanted to export all audit logs to a central project in BigQuery, she would have to set up a log sink at every project and validate that the sink was in place for each new project.
With aggregated exports, administrators of an organization or folder can set up sinks once to be inherited by all the child projects and subfolders. This makes it possible for the security administrator to export all audit logs in her organization to BigQuery with a single command:
gcloud beta logging sinks create my-bq-sink
bigquery.googleapis.com/projects/my-project/datasets/my_dataset
–log-filter=’logName= “logs/cloudaudit.googleapis.com%2Factivity”‘
–organization=1234 –include-children
Aggregated exports help ensure that logs in future projects will be exported correctly. Since the sink is set at the organization or folder level, it also prevents an individual project owner from turning off a sink.
Control your Stackdriver Logging pipeline with exclusion filters
All logs sent to the Logging API, whether sent by you or by Google Cloud services, have always gone into Stackdriver Logging where they’re searchable in the Logs Viewer. But we heard feedback that users wanted more control over which logs get ingested into Stackdriver Logging, and we listened. To address this, exclusion filters are now in beta. Exclusion filters allow you to reduce costs, improve the signal to noise ratio by reducing chatty logs and manage compliance by blocking logs from a source or matching a pattern from being available in Stackdriver Logging.
The new Resource Usage page provides visibility into which resources are sending logs and which are excluded from Stackdriver Logging.
This makes it easy to exclude some or all future logs from a specific resource. In the example above, we’re excluding 99% of successful load balancer logs.
We know the choice and freedom to choose any solution is important, which is why all GCP logs are available to you irrespective of the logging exclusion filters, to export to BigQuery, Google Cloud Storage or any third party tool via PubSub. Furthermore, Stackdriver will not charge for this export, although BigQuery, GCS and PubSub charges will apply.
Starting Dec 1, Stackdriver Logging offers 50GB of logs per project per month for free
You told us you wanted room to grow with Stackdriver without worrying about the cost of logging all that data, which is why on December 1 we’re increasing the free logs allocation to an industry-leading 50GB per project per month. This increase aims to bring the power of Stackdriver Logging search, storage, analysis and alerting capabilities to all our customers.
Want to keep logs beyond the free 50GB/month allocation? You can sign up for the Stackdriver Premium Tier or the logs overage in the Basic Tier. After Dec 1, any additional logs will be charged at a flat rate of $0.50/GB.
Audit logs, still free and now available for 13 months
We’re also exempting admin activity audit logs from the limits and overage. They’ll be available in Stackdriver in full without any charges. You’ll now be able to keep them for 13 months instead of 30 days.
Continuing the conversation
We hope this brings the power of Stackdriver Logging search, storage, analysis and alerting capabilities to all our customers. We have many more exciting new features planned, including a time range selector coming in September to make it easier to get visibility into the timespan of search results. We’re always looking for more feedback and suggestions on how to improve Stackdriver Logging. Please keep sending us your requests and feedback.
Interested in more information on these new features?
Discover new ways to extract value from your logs with the improved logs-based metrics.
Learn how to control log sinks across your organization using aggregated exports.
Read more on controlling which logs you bring into your Stackdriver Logging account using exclusion filters.
Find the details on our pricing model on the Stackdriver pricing page.
Learn our best practices on managing logs and controlling costs.
View your current logs usage on the Logging Resource Usage page.
Estimate the charges at your current usage rate with the Stackdriver Billing Calculator.
Quelle: Google Cloud Platform
