da Agency

Scalable Python on BigQuery using Dask and GPUs

BigQuery is Google Cloud’s fully managed serverless data platform that supports querying using ANSI SQL. BigQuery also has a data lake storage engine that unifies SQL queries with other open source processing frameworks such as Apache Spark, Tensorflow, and Dask. BigQuery storage provides an API layer for OSS engines to process data. This API enables mixing and matching programming in languages like Python with structured SQL in the same data platform. This post provides an introduction to using BigQuery with one popular distributed Python framework, Dask, an open source library that makes it easy to scale Python tools to BigQuery sized datasets. We will also show you how to extend Dask with RAPIDS, a suite of open-source libraries and APIs to execute GPU-accelerated pipelines directly on BigQuery storage.Integrating Dask and RAPIDS with BigQuery storage A core component of BigQuery architecture is the separation of compute and storage. BigQuery storage can be directly accessed over a highly performant Storage Read API which enables users to consume data in multiple streams and provides both column projections and filtering at the storage level. Coiled, a Google Cloud Partner that provides enterprise-grade Dask in your GCP account, developed an open-source Dask-BigQuery connector (GitHub) that enables Dask processing to take advantage of the Storage Read API and governed access to BigQuery data. RAPIDSis an open sourced library spawned from NVIDIA that uses Dask to distribute data and computation over multiple NVIDIA GPUs. The distributed computation can be done on a single machine or in a multi-node cluster. Dask integrates with both RAPIDS cuDF, XGBoost, and RAPIDS cuML for GPU-accelerated data analytics and machine learning.To start using Dask using BigQuery data, you can install the dask-bigquery connector from any Python IDE. You simply install `dask-bigquery` with `pip` or `conda`, authenticate with Google Cloud, and then use the few lines of python code as shown below to pull data from a BigQuery table.code_block[StructValue([(u’code’, u’import dask_bigqueryrnrnddf = dask_bigquery.read_gbq(rn project_id=”your_project_id”,rn dataset_id=”your_dataset”,rn table_id=”your_table”,rn)rnddf.head()’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e6b8a655150>)])]Achieving Python scalability on BigQuery with Dataproc While Dask and the BQ connector can essentially be installed anywhere that Python can be run and scale to the number of cores available in that machine, the real power of scaling comes in when you can use an entire cluster of virtual machines. An easy way to do this on Google Cloud is by using Dataproc. Using the initialization actions outlined in this GitHub repo, getting setup with Dask and RAPIDS on a Dataproc cluster with NVIDIA GPUs is fairly straightforward.Let’s walk through an example using the NYC taxi dataset. As a first step, let’s create a RAPIDS accelerated Dask yarn cluster object on Dataproc by running the following code:code_block[StructValue([(u’code’, u’from dask.distributed import Clientrnfrom dask_yarn import YarnClusterrnrncluster = YarnCluster(worker_class=”dask_cuda.CUDAWorker”, rn worker_gpus=1, worker_vcores=4, worker_memory=’24GB’, rn worker_env={“CONDA_PREFIX”:”/opt/conda/default/”})rncluster.scale(4)’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e6ba01f7510>)])]Now that we have a Dask client, we can use it to read the NYC Taxi dataset in a BigQuery table through the Dask BigQuery connector:code_block[StructValue([(u’code’, u’d_df = dask_bigquery.read_gbq(rn project_id=”k80-exploration”,rn dataset_id=”spark_rapids”,rn table_id=”nyc_taxi_0″,rn)’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e6ba01f7150>)])]Next, let’s use RAPIDS Dask cuDF libraries to accelerate the preprocessing with GPUs.code_block[StructValue([(u’code’, u”taxi_df = dask_cudf.from_dask_dataframe(d_df)rntaxi_df = clean(taxi_df, remap, must_haves)rntaxi_df = taxi_df.query(‘ and ‘.join(query_frags))”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e6ba345dc90>)])]Finally, we can use a feature of the Dask dataframe to split into two datasets — one for training and one for testing. These datasets can also be converted to XGBoost Dmatrix and sent into XGBoost for training on GPU.code_block[StructValue([(u’code’, u”xgb_clasf = xgb.dask.train(client, rn params,rn dmatrix_train, rn num_boost_round=2000,rn evals=[(dmatrix_train, ‘train’), (dmatrix_test,’test’)]rn )”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e6b8aa98b90>)])]The complete notebook can be accessed at this GitHub link. Currently, Dask-BigQuery connector doesn’t support native write back to BigQuery, user need work around that through cloud storage, with Dask or Dask Rapids, write back to GCS first with `to_parquet(“gs://temp_path/”)`, then having BigQuery load from GCS with: `bigquery.Client.load_table_from_uri(“gs://temp_path/”)`.What’s nextIn this blog, we introduced a few key components to allow BigQuery users to scale their favorite Python libraries through Dask to process large datasets. With the broad portfolio of NVIDIA GPUs embedded across Google Cloud data analytics services like BigQuery and Dataproc and the availability of GPU-accelerated software like RAPIDS, developers can significantly accelerate their analytics and machine learning workflows. Acknowledgements: Benjamin Zaitlen, Software Engineer Manager, NVIDIA; Jill Milton, Senior Partnership Manager, NVIDIA, Coiled Developer Team.Related ArticleLearn how BI Engine enhances BigQuery query performanceThis blog explains how BI Engine enhances BigQuery query performance, different modes in BI engine and its monitoring.Read Article
Quelle: Google Cloud Platform

da Agency

Google Cloud Data Heroes Series: Meet Tomi, a data engineer based in Germany and creator of the ‘Not So BigQuery Newsletter’

Google Cloud Data Heroes is a series where we share stories of the everyday heroes who use our data tools to do incredible things. Like any good superhero tale, we explore our Google Cloud Data Heroes’ origin stories, how they moved from data chaos to a data-driven environment, what projects and challenges they are overcoming now, and how they give back to the community. In this month’s edition, we’re pleased to introduce Tomi! Tomi grew up in Croatia, and is now residing in Berlin, Germany, where he currently works as a freelance Google Cloud data engineer. In this role, he regularly uses BigQuery. Tomi’s familiarity with BigQuery and his passion for Google Cloud led him to creating the weekly newsletter Not So BigQuery, where he discusses the latest data-related information from the GCP world.  Additionally, he also works for one of the largest automotive manufacturers in Germany as an analyst. When not in front of the keyboard, Tomi enjoys walking with his dog and his girlfriend, going to bakeries, or spending a night watching television.When were you introduced to the cloud, tech, or data field? What made you pursue this in your career? I always struggled with the question ‘what do you want to do in your life?. I attended school at Zagreb University of Applied Science for my information technology studies degree, but I was still unsure if I should become a developer, data engineer or something completely different.A couple of years into working as a junior IT Consultant, I stumbled upon a job advertisement looking for a Data Analyst/Scientist. Back then, finding out that you can get paid to just work with data all day sounded mind-blowing to me. A dream job.I immediately applied for the role and started learning about the skills needed. This is also where I gained my first experience with the Cloud as I signed up for a Google Cloud Platform free trial in February 2018. On the platform, there was a blog post describing how to run Jupyter notebooks in the Cloud. It interested me, and I went ahead and created my very first Compute Engine instance in Google Cloud Platform.I didn’t get the job I initially applied for, but this was the trigger for me that set things in motion and got me to where I am now.What courses, studies, degrees, or certifications were instrumental to your progression and success in the field? In your opinion, what data skills or competencies should data practitioners be focusing on acquiring to be successful in 2022 and why? Looking back at my university days, I really enjoyed the course about databases, which was partially because I had a really great teacher, but also because this was the first time I got to do something which catered to my then still-unknown data-nerdy side.In 2019, I got my Google Cloud Certified Associate Cloud Engineer Certification which was a challenging and rewarding entry-level certification for Google Cloud. I would recommend considering getting one of these as a way of focusing one’s learning.One major change I’ve observed since working in the data field is the ongoing transition from on-prem to cloud and serverless. I remember a story from my early consulting days working in an IT operations team, when there was a major incident caused by an on-prem server outage. At some point one frustrated colleague said something like, ‘why do we even have to have servers? Why can’t it just *run* somehow?’ What sounded like a bit of a silly question back then turned out to be quite ‘visionary’ with all the serverless and cloud-based tech we have today.What drew you to Google Cloud? Tell us about that process, what you’re most proud of in this area, and why you give back to the community? There is this great newsletter on Google Cloud Platform called GCP Weekly, run by a data community member named Zdenko Hrček that I really like. However, since the GCP ecosystem is growing at a rapid pace there are sometimes just too many news and blogs in a single week. I really struggled to catch up with all the new product updates and tutorials. That’s when I had the idea: ‘what if there would be a shorter newsletter with only news about BigQuery and other data-related tools’? Fast forward to today, my Not So BigQuery newsletter has more than 220 subscribers.I was also inspired by the awesome content created by Priyanka Vergadia, Staff Developer Advocate at Google Cloud, such as her Sketchnotes series. I created the GCP Data Wiki, which is a public Notion page with cards for every database/storage service in GCP with useful details such as links to official docs, Sketchnotes and more.What are 1-2 of your favorite projects you’ve done with Google Cloud’s data products? One of my first projects built with Google Cloud products was an automated data pipeline to get track data from the official Spotify API. I was looking for a data project to add to my portfolio and found out that Spotify lets you query their huge library via a REST API. This later evolved into a fully-serverless pipeline running on Google Cloud Functions and BigQuery. I also wrote a blog post about the whole thing, which got 310 claps on Medium.Additionally, the Not So BigQuery newsletter I created is actually powered by a tool I built using Google Sheets and Firebase (Functions). I have a Google Sheet where I pull in the news feed sections from sources such as the Google Cloud Blog and Medium. Using the built-in Sheets formulas such as IMPORTFEED and FILTER, I built a keyword-based article curation algorithm pre-selecting the articles to include in the next issue of the newsletter. Then my tool called crssnt (pronounced as the french pastry) takes the data from the Google Sheet and displays it in the newsletter. If you are curious how the Google Sheet looks like, you can check it out here.What are your favorite Google Cloud Platform data products within the data analytics, databases, and/or AI/ML categories? What use case(s) do you most focus on in your work? What stands out about GCP’s offerings?My favorite is BigQuery but I’m also a huge fan of Firestore. BigQuery is my tool of choice for pretty much all of my data warehouse needs (for both personal and client projects). What really stood out to me for me is the ease of use when it comes to setting up new databases from scratch and getting first results in the form of e.g. a Data Studio dashboard built on top of a BigQuery table. Similarly, I always go back to Firestore whenever I have an idea about some new front-end project since it’s super easy to get started and gives me a lot of flexibility.From similar non-Google products, I used Snowflake a while ago but didn’t find the user interface nearly as intuitive and user-friendly as BigQuery.What’s next for you in life? It’s going to be mostly ‘more of the same’ for me: as a data nerd, there is always something new to discover and learn. My overall message to readers would be to try to not worry too much about fitting into predefined career paths, job titles and so on, and just do your thing. There is always more than one way of doing things and reaching your goals. Want to join the Data Engineer Community?Register for the Data Engineer Spotlight on July 20th, where attendees have the chance to learn from four technical how-to sessions and hear from Google Cloud Experts on the latest product innovations that can help you manage your growing data. Begin your own Data Hero journeyReady to embark on your Google Cloud data adventure? Begin your own hero’s journey with GCP’s recommended learning path where you can achieve badges and certifications along the way. Join the Cloud Innovators program today to stay up to date on more data practitioner tips, tricks, and events.If you think you have a good Data Hero story worth sharing, please let us know! We’d love to feature you in our series as well.Related ArticleGoogle Cloud Data Heroes Series: Meet Francisco, the Ecuadorian American founder of Direcly, a Google Cloud PartnerIn the Data Heroes series we share stories of people who use data analytics tools to do incredible things. In this month’s edition, Meet …Read Article
Quelle: Google Cloud Platform

da Agency

Using Google Kubernetes Engine’s GPU sharing to search for neutrinos

Editor’s note: Today we hear from the San Diego Supercomputer Center (SDSC) and University of Wisconsin-Madison about how GPU sharing in Google Kubernetes Engines is helping them detect neutrinos at the South Pole with the gigaton-scale IceCube Neutrino Observatory.IceCube Neutrino Observatory is a detector at the South Pole designed to search for nearly massless subatomic particles called neutrinos. These high-energy astronomical messengers provide information to probe events like exploding stars, gamma-ray bursts, and cataclysmic phenomena involving black holes and neutron stars. Scientific computer simulations are run on the sensory data that IceCube collects on neutrinos to pinpoint the direction of detected cosmic events and improve their resolution.The most computationally intensive part of the IceCube simulation workflow is the photon propagation code, a.k.a. ray-tracing, and that code can greatly benefit from running on GPUs. The application is high throughput in nature, with each photon simulation being independent of the others. Apart from the core data acquisition system at the South Pole, most of IceCube’s compute needs are served by an aggregation of compute resources from various research institutions all over the world, most of which use the Open Science Grid (OSG) infrastructure as their unifying glue. GPU resources are relatively scarce in the scientific resource provider community. In 2021, OSG had only 6M GPU hours vs 1800M CPU core hours in its infrastructure. The ability to expand the available resource pool with cloud resources is thus highly desirable.The SDSC team recently extended the OSG infrastructure to effectively use Kubernetes-managed resources to support IceCube compute workloads on the Pacific Research Platform (PRP). The service manages dynamic provisioning in a completely autonomous fashion by implementing horizontal pilot pod autoscaling based on the queue depth of the IceCube batch system. Unlike on-premises systems, Google Cloud offers the benefits of elasticity (on-demand scaling) and cost efficiency (only pay for what gets used). We needed a flexible platform that can avail these benefits to our community. We found Google Kubernetes Engine (GKE) to be a great match for our needs due to its support for auto-provisioning, auto-scaling, dynamic scheduling, orchestrated maintenance, job API and fault tolerance, as well as support for co-mingling of various machine types (e.g. CPU + GPU and on-demand + Spot) in the same cluster and up to 15,000 nodes per cluster.While IceCube’s ray-tracing simulation greatly benefits from computing on the GKE GPUs, it still relies on CPU compute for feeding the data to the GPU portion of the code. And GPUs have been getting faster at a much higher rate than CPUs have! With the advent of the NVIDIA V100 and A100 GPUs, the IceCube code is now CPU-bound in many configurations. By sharing a large GPU between multiple IceCube applications, the IceCube ray-tracing simulation again becomes GPU-bound, and therefore we get significantly more simulation results from the same hardware. GKE has native support for both simple GPU time-sharing and the more advanced A100 Multi-Instance GPU (MIG) partitioning, making it incredibly easy for IceCube — and OSG at large — to use.To leverage the elasticity of the Google Cloud, we fully relied on GKE horizontal node auto-scaling for provisioning and de-provisioning GKE compute resources. Whenever there were worker pods that could not be started, the auto-scaler provisioned more GKE nodes, up to a set maximum. Whenever a GKE node was unused, the auto-scaler de-provisioned it to save costs.Performance resultsUsing Google Cloud GPU resources was very simple through GKE. We used the same setup we were already using on the on-prem PRP Kubernetes cluster, simply pointing our setup to the new cluster.After the initial setup, IceCube was able to efficiently use Google Cloud resources, without any manual intervention by the supporting SDSC team beyond setting the auto-scaling limits. This was a very welcome change from other cloud activities the SDSC team has performed on behalf of IceCube and others, that required active management of provisioned resources.AutoscalingThe GKE auto-scaling for autonomous provisioning and de-provisioning of cloud resources worked as advertised, closely matching the demand from IceCube users, as seen in Fig. 1. We were particularly impressed by GKE’s performance in conjunction with GPU sharing; the test run shown used seven A100 MIG partitions per GPU.Fig. 1: Monitoring snapshot of the unconstrained GKE auto-scaling test run.GPU sharingBoth full-GPU and shared-GPU Kubernetes nodes with A100, V100 and T4 GPUs were provisioned, but IceCube jobs did not differentiate between them, since all provisioned resources met the jobs’ minimum requirements.We assumed that GPU sharing benefits would vary based on the CPU-to-GPU ratio of the chosen workflow, so during this exercise we picked one workflow from each extreme. IceCube users can choose to speed up the GPU-based ray-tracing compute of some problems by, roughly speaking, increasing the size of the target for the photons by some factor. For example, setting oversize=1 gives the most precise simulation, and oversize=4 gives the fastest. Faster compute (of course) results in a higher CPU-to-GPU ratio. The fastest oversize=4 workload benefitted the most from GPU sharing. As can be seen from Fig. 2, IceCube oversize=4 jobs cannot make good use of anything faster than a NVIDIA T4. Indeed, even for the low-end T4 GPU, sharing increases the job throughput by about 40%! For the A100 GPU, GPU sharing gets us a 4.5x throughput increase, which is truly transformational. Note that MIG and “plain” GPU sharing provide comparable throughput improvements, but MIG comes with much stronger isolation guarantees, which would be very valuable in a multi-user setup.Fig. 2: Number of IceCube oversize=4 jobs per hour, grouped by GPU setup.The more demanding oversize=1 workload makes much better use of the GPUs, so we observe no job throughput improvement for the older T4 and V100 GPUs. The A100 GPU, however, is still too powerful to be used as a whole, and GPU sharing gives us almost a 2x throughput improvement here, as illustrated in Fig. 3.Fig. 3: Number of IceCube oversize=1 jobs per day, grouped by GPU setup.GPU sharing of course increases the wallclock time needed by any single job to run to completion. This is however not a limiting factor for IceCube, since the main objective is to produce the output of thousands of independent jobs, and the expected timeline is measured in days, not minutes. Job throughput and cost effectiveness are therefore much more important than compute latency.Finally, we would like to stress that most of the used resources were provisioned on top of Spot VMs, making them significantly cheaper than their on-demand equivalents. GKE gracefully handled any preemption, making this mode of operation very cost effective.Lessons learnedGKE with GPU sharing has proven to be very simple to use, given that our workloads were already Kubernetes-ready. From a user point of view, there were virtually no differences from the on-prem Kubernetes cluster they were accustomed to.The benefits of GPU sharing obviously depend on the chosen workloads, but at least for IceCube it seems to be a necessary feature for the latest GPUs, i.e. the NVIDIA A100. Additionally, a significant fraction of IceCube jobs can benefit from GPU sharing even for lower-end T4 GPUs.When choosing the GPU-sharing methodology, we definitely prefer MIG partitioning. While less flexible than time-shared GPU sharing, MIG’s strong isolation properties make management of multi-workload setups much more predictable. That said, “plain” GPU sharing was still more than acceptable, and was especially welcome on GPUs that lack MIG support.In summary, the GKE shared-GPU experience was very positive. The observed benefits of GPU sharing in Kubernetes were an eye-opener and we plan to make use of it whenever possible.Want to learn more about sharing GPUs on GKE? Check out this user guide.Related ArticleTurbocharge workloads with new multi-instance NVIDIA GPUs on GKEYou can now partition a single NVIDIA A100 GPU into up to seven instances and allocate each instance to a single Google Kubernetes Engine…Read Article
Quelle: Google Cloud Platform

da Agency

Deploying high-throughput workloads on GKE Autopilot with the Scale-Out compute class

GKE Autopilot is a full-featured, fully managed Kubernetes platform that combines the full power of the Kubernetes API with a hands-off approach to cluster management and operations. Since launching Autopilot last year, we’ve continued to innovate, adding capabilities to meet the demands of your workloads. We’re excited to introduce the concept of compute classes in Autopilot, together with the Scale-Out compute class, which offers high performance x86 and Arm compute, now available in Preview.Autopilot compute classes are a curated set of hardware configurations on which you can deploy your workloads. In this initial release, we are introducing the Scale-Out compute class, which is designed for workloads that are optimized for a single-thread-per-core and scale horizontally. The Scale-Out compute class currently supports two hardware architectures — x86 and Arm — allowing you to choose whichever one offers the best price-performance for your specific workload. The Scale-Out compute class joins our original, general-purpose compute option and is designed for running workloads that benefit from the fastest CPU platforms available on Google Cloud, and with greater cost-efficiency for applications that have high CPU utilization.We also heard from you that some workloads would benefit from higher-performance compute. To serve this need, x86 workloads running on the Scale-Out compute class are currently served by 3rd Gen AMD EPYCTM processors, with Simultaneous Multithreading (SMT) disabled, achieving the highest per-core benchmark among x86 platforms in Google Cloud.And for the first time, Autopilot supports Arm workloads. Currently utilizing the new Tau T2A VMs running on Ampere® Altra® Arm-based processors, the Scale-Out compute class gives your Arm workloads price-performance benefits combined with a thriving, open, end-to-end platform independent ecosystem. Autopilot Arm Pods are currently available in us-central, europe-west4, and asia-southeast1.Deploying Arm workloads using the Scale-Out compute classTo deploy your Pods on a specific compute class and CPU, simply add a Kubernetes nodeSelector or node affinity rule with the following labels in your deployment specification:cloud.google.com/COMPUTE-CLASSkubernetes.io/ARCHTo run an Arm workload on Autopilot, you need a cluster running version 1.24.1-gke.1400 or later and in one of the supported regions. You can create a new cluster at this version, or upgrade an existing one. To create a new Arm-supported cluster on the CLI, use the following:code_block[StructValue([(u’code’, u’CLUSTER_NAME=autopilot-armrnREGION=us-central1rnVERSION=1.24.1-gke.1400rngcloud container clusters create-auto $CLUSTER_NAME \rn –release-channel “rapid” –region $REGION \rn –cluster-version $VERSION’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2ad76ca790>)])]For example, the following Deployment specification will deploy the official Nginx image on the Arm architecture:code_block[StructValue([(u’code’, u’apiVersion: apps/v1rnkind: Deploymentrnmetadata:rn name: nginx-arm64rnspec:rn selector:rn matchLabels:rn app: nginxrn template:rn metadata:rn labels:rn app: nginxrn spec:rn nodeSelector:rn cloud.google.com/compute-class: Scale-Outrn kubernetes.io/arch: arm64rn containers:rn – name: nginxrn image: nginx:latest’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2ad7dcb7d0>)])]Deploying x86 workloads on the Scale-Out compute classThe Scale-out compute class also supports the x86 architecture by simply adding a selector for the `Scale-Out` compute class. You can either explicitly set the architecture with kubernetes.io/arch: amd64 or omit that label from the selector, as x86 is the default.To run an x86 Scale-Out workload on Autopilot, you need a cluster running version 1.24.1-gke.1400 or later and in one of the supported regions. The same CLI command from the example above will get you an x86 Scale-Out-capable GKE Autopilot cluster.code_block[StructValue([(u’code’, u’apiVersion: apps/v1rnkind: Deploymentrnmetadata:rn name: nginx-arm64rnspec:rn selector:rn matchLabels:rn app: nginxrn template:rn metadata:rn labels:rn app: nginxrn spec:rn nodeSelector:rn cloud.google.com/compute-class: Scale-Outrn containers:rn – name: nginxrn image: nginx:latest’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2ad7b83090>)])]Deploying Spot Pods using the Scale-Out compute classYou can also combine compute classes with Spot Pods by adding the label cloud.google.com/gke-spot: “true”to the nodeSelector:code_block[StructValue([(u’code’, u’apiVersion: apps/v1rnkind: Deploymentrnmetadata:rn name: nginx-arm64rnspec:rn selector:rn matchLabels:rn app: nginxrn template:rn metadata:rn labels:rn app: nginxrn spec:rn nodeSelector:rn cloud.google.com/gke-spot: “true”rn cloud.google.com/compute-class: Scale-Outrn kubernetes.io/arch: arm64rn containers:rn – name: nginxrn image: nginx:latest’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2ad7b83050>)])]Spot Pods are supported for both the x86 and Arm architectures when using the Scale-Out compute class.Try the Scale-Out compute class on GKE Autopilot today!To help you get started, check out our guides on creating an Autopilot cluster, getting started with compute classes, building images for Arm workloads, and deploying Arm workloads on GKE Autopilot.Related ArticleRun your Arm workloads on Google Kubernetes Engine with Tau T2A VMsWith Google Kubernetes Engine’s (GKE) support for the new Tau VM T2A, you can run your containerized workloads on the Arm architecture.Read Article
Quelle: Google Cloud Platform

da Agency

How to overcome 5 common SecOps challenges

Editor’s note: This blog was originally published by Siemplify on April 12, 2022.The success of the modern security operations center, despite the infusion of automation, machine learning, and artificial intelligence, remains heavily dependent on people. This is largely due to the vast amounts of data a security operations center must ingest—a product of an ever-expanding attack surface and the borderless enterprise brought on by the rapid rise of cloud adoption. All those alerts coming in mean proactive and reactive human decision making remains critical.Perhaps it should come as no surprise that the information security analyst now ranks as No. 1 in U.S. News’ 100 Best Jobs Rankings, “determined by identifying careers with the largest projected number and percentage of openings through 2030, according to the U.S. Bureau of Labor Statistics.” Security, and specifically detection and response, is not only a business imperative—it is arguably the top worry on the minds of CEOs.However, the security analyst is also one of the most likely professionals to want to leave their jobs, according to a newly released “Voice of the SOC Analyst” study conducted by Tines. What gives? Turnover woes are attributable to several key SecOps challenges that never seem to budge.1)  Alert fatigue and false positives: Have you ever received so much spam or junk mail that you end up ignoring your new messages entirely, leading you to miss an important one? The same can happen for alerts. Too much noise is unsustainable and can lead to the real threats being missed, especially as perimeters expand and cloud adoption increases.2) Disparate tools: Already in the company of too many point-detection tools, security operations professionals are saying hello to a few more in the era of remote work and increased cloud demands. The latest count is north of 75 security tools that need to be managed by the average enterprise.3) Manual processes: Use case procedures that result in inconsistent, unrepeatable processes can bottleneck response times and frustrate SecOps teams. Not everything in the SOC needs to—or should be—automated, but much can be, which then frees up analysts and engineers to concentrate on higher-order tasks and be able to more easily train new employees.4) Talent shortage: Death, taxes, and the cybersecurity skills shortage. As sure as the sun will rise tomorrow, so will the need for skilled individuals to wage the cybersecurity fight. But what happens when not enough talent is filling the seats? Teams must compensate to fill the gap.5) Lack of visibility: Security operations metrics are critical for improving productivity and attracting executive buy-in and support, but SecOps success can be difficult to track, as reports can require a significant amount of work to pull together.The caveat of course is that it would be rare to find a SecOps team working without the above challenges. As such, some of the immediate steps you can take to push back against these constraints focus on people-powered processes and technologies to remedy the issues.According to a recent paper co-authored by Google Cloud and Deloitte: Humans are—and will be—needed to both perform final triage on the most obtuse security signals (similar to conventional SOC Level 3+) and to conduct a form of threat hunting (i.e. looking for what didn’t trigger that alert). Machines will be needed to deliver better data to humans, both in a more organized form (stories made of alerts) and in improved quality detections using rules and algorithms— all while covering more emerging IT environments.Both humans and machines will need to work together on mixed manual and automated workflows.So, what does this ultimately mean you must do to improve your security operations? Here are five practical suggestions:Detect Threats More EfficientlyEfficiencies within the SOC can be realized from a SIEM solution that automatically detects threats in real-time and at scale. The right platform will support massive data ingestion and storage, relieve traditional cost and scaling limitations, and broaden the lens for anomaly and machine learning/AI-based detection. With data stored and analyzed in one place, security teams can investigate and detect threats more effectively.Respond to Threats AutomaticallySOAR can be a game-changer in terms of caseload reduction and faster (and smarter, especially when integrated with threat intelligence) response times. But before rushing headfirst into automation, you should consider your processes, review outcomes you are trying to achieve (such as reduced MTTD)–and then decide exactly what you want to automate (which can be a lot with SOAR). Once clear processes are determined where automation can contribute, SOC personnel are freed up to be more creative in their work.Prioritize LogsMany teams lack a strategy for collecting, analyzing and prioritizing logs, despite the fact that these sources of insight often hold the clues of an ongoing attack. To help, here are two cheat sheets featuring essential logs to monitor.Outsource What You Can’t Do YourselfProcess improvements may help you compensate for perceived personnel shortages (for example, perhaps fixing a misconfigured monitoring tool will reduce alert noise). Of course, many organizations need additional human hands to help them perform tasks like round-the-clock monitoring and more specialized functions like threat hunting. Here is where a managed security services provider or managed detection provider can be helpful. Be realistic about your budget, however, as you may be able to introduce some solutions in-house. Institute Career ModelsLack of management support is cited as the fourth-biggest obstacle to a fully functioning SOC model, according to the 2022 SANS Security Operations Center Survey. To overcome this, SecOps leaders must help improve workflow processes, protect innovation, keep teams absorbed in inspiring and impactful work versus mundane tasks, remain flexible with staff, and endorse training and career development. Because at the end of the day, the SOC is still distinctly human–and that is who will be the difference maker between success and failure.Related ArticleRaising the bar in Security Operations: Google Acquires SiemplifyGoogle has acquired Siemplify, a leading security orchestration, automation and response (SOAR) provider. Siemplify will join Google Clou…Read Article
Quelle: Google Cloud Platform

da Agency

Migrate and modernize with Azure to power innovation across the entire digital estate

Cloud adoption increased significantly during COVID-19 and continues for many companies. However, an enormous migration and modernization opportunity remains as organizations continue their digital transformation. In fact, 72 percent of organizations reported their industry’s pace of digital transformation accelerated because of COVID-19, according to a survey sponsored by Microsoft in The Economist1. And we don’t expect that to slow down anytime soon.

We are hearing key themes from our customers that reinforce this, including:

Cloud has become the catalyst for innovation. Customers are moving beyond operational efficiency to create new products and offerings, leveraging the unique capabilities of cloud to differentiate themselves.
Customers are not just looking for technology, they’re looking for a trusted partner. They need an expert to help them navigate these tough issues as they move toward hybrid, multi-cloud, and edge environments, facing new complexities and opportunities.
Security, data privacy, and compliance are top of mind for customers in every industry as cyberattacks continue to rise2.

These areas are opportunities for us to strengthen our partnerships and help our mutual customers realize their greatest potential. This is why we are focusing on three key areas of growth for Azure at Microsoft Inspire:

Innovation with new cloud-native experiences.
Modernization of app and data estates.
Migration and modernization of infrastructure and mission-critical workloads.

This week, we are announcing new Microsoft Azure capabilities to help partners increase return on investments, grow leads, and shorten sales cycles. This includes updates to our Azure Migration and Modernization Program (AMMP) and announcing the ISV Success Program.

Read on for more details on our latest developments and each of the opportunity areas to see how we can deliver the greatest value to our customers. Our Azure keynote session is also a great resource to learn more.

Key investments to help partners achieve success

Azure Migration and Modernization Program

We now have more than 500 partners enrolled in the Azure Migration and Modernization Program (AMMP) across apps, data, and infrastructure. AMMP is our hero program to help simplify and accelerate migration and modernization, with the right mix of incentives, best practice guidance, tools, and expert help. AMMP has powered deeper go-to-market connections with partners—and our goal is to ensure every migration and modernization opportunity has a partner attached to it. Partners are at the center of how we execute with customers.

We’re making substantive investments and updates to AMMP to help drive scale and velocity of migrations:

Up to 2.5 times larger incentives for Windows Server and SQL Server migrations. 
Empowerment for Microsoft sales organizations to locally allocate incentives for their areas, providing more opportunity for partners to engage and feed into local plans.
Updated best practice guidance with the Cloud Adoption Framework for Azure and Azure Well Architected Framework (WAF).
New modernization capabilities in Azure Migrate, including the option for ISVs to integrate their own IP.

Now more than ever, we need partners’ help to scale our customers’ migration and modernization journeys. Sign up or nominate customers for the AMMP.

ISV Success Program

As the cloud becomes the fabric of every business, across every industry, customers need more complete solutions to support their growth and innovation. This creates tremendous opportunity as the demand for software as a service (SaaS) and "anything as a service solutions continue to increase. To help unlock opportunities for software providers we are excited to announce new benefits with the ISV Success Program to help ISVs innovate rapidly, build well architected applications, publish them to our commercial marketplace, and grow their sales. Currently in preview, and broadly available in fall 2022, the program is intended to be the pathway to ISV success in the Microsoft Cloud Partner Program.

Software providers can take advantage of this new program to build across the Microsoft Cloud and get access to cloud sandboxes, developer tooling, technical and business resources, and a dedicated community.

Innovate with new cloud-native experiences

Organizations across industries are looking to deliver highly personalized experiences to their end customers. Cloud-native applications can help meet these needs. For example, Microsoft Azure has helped retailers like Walgreens gain immediate access to rich transactional data and insights, enabling faster decisions and better customer experiences. According to IDC3, 750 million new logical applications will be built by 2025. This is why so many ISVs and enterprises are turning to Azure for their cloud-native applications. Partners can help customers build cloud apps using scalable containerized architectures combined with globally scalable databases infused with intelligence through AI.

Modernize application and data estates

Modernization of applications and data represents a huge partner opportunity where one project will lead to the next. In fact, 59 percent of organizations see modernizing apps to the cloud as a top initiative4. According to Microsoft estimates, the opportunity for data exceeds $42 billion today, and will grow to $85 billion by fiscal year 2025. Our goal is to help partners close deals to modernize the application and data estates even faster. We’ve made this even more seamless with our newly released Microsoft Intelligent Data Platform. This end-to-end ecosystem integrates databases, analytics, and governance across the customer estate—enabling organizations to adapt in real-time, add layers of intelligence to their applications, unlock fast and predictive insights, and govern their data—wherever it resides.

No matter where our customers are in their modernization journey, Azure offers flexibility between control over managing infrastructure and the level of productivity desired. Partner advisory services and technical expertise is extremely valuable in this estate-level opportunity.

Migrate and modernize infrastructure and mission-critical workloads

Our customers often face time-sensitive decisions with datacenter contract renewals or software end-of-support. As a result, many customers are looking to shift large parts of their IT spend to the cloud with infrastructure as a service (IaaS) seeing the biggest increase. 

This year also brings timely migration opportunities for Windows Server and SQL Server. It’s a great time for partners to advise customers using SQL Server 2012 and Windows Server 2012/2012 R2 about End of Support (EOS) timelines and help them take action to stay secure. It’s also an opportunity to talk with customers about their cloud migration and modernization plans. We offer the best value at every stage of cloud migration. To share just two examples, it’s up to 80 percent less expensive to run Windows Server VMs and Azure SQL Managed Instance on Azure than it is with our main competitor.  And it’s not just about costs—we offer unique capabilities like Azure Automanage to simplify VM management and the broadest SQL Server compatibility to ease the move.

Read more about the top workloads with the largest migration opportunity as a key growth and revenue driver this coming year with announcement details, including:

Azure Confidential Computing capabilities, now generally available, allow partners to transition to Azure workloads that handle sensitive data with additional levels of protection.
Azure Center for SAP solutions, now in preview, is an end-to-end solution to deploy and manage SAP workloads on Azure, enabling customers and partners to create and run an SAP system as a unified workload, and providing a more seamless foundation for innovation on the Microsoft Cloud.
A new Azure Arc Boost Program, in partnership with Intel, will drive the deployment of Azure Arc and Azure Stack HCI in customer hybrid environments through our Systems Integrator partner ecosystem.
Our most recent release of Azure Stack HCI, in preview, delivers more customer value and partner opportunity with new features by providing increased value on investment, shortened time to value, and improved support experience for Azure Stack HCI.

Get started at Microsoft Inspire

With so much opportunity ahead, where should you get started? Be sure to tune into the keynote and the Azure sessions linked below to hear from partners about how they’re already growing their business through Microsoft Cloud and Azure. Partners can bring tailored industry expertise and solutions to complement the innovation that Azure delivers. Together with our amazing partner community, we are creating opportunities with the most trusted cloud to empower customers to transform today, tomorrow, and build for the future.

Azure sessions

Power innovation across the digital estate
Grow revenue by accelerating customer adoption of Azure infrastructure
New SAP on Azure solutions and GTM offers to accelerate your business
Winning the data estate with Microsoft Intelligent Data Platform
Drive digital and application innovation with Microsoft Azure
Addressing sovereign requirements with Microsoft Cloud
SQL Server 2022: the most Azure-connected SQL Server release ever
Move your Azure hybrid business forward with Azure Arc
Enhance your customers' network security and drive business growth on
Latest Azure Confidential Computing innovations – generally available
How to make money migrating with Azure VMware Solution
Onboard as a HPC partner
Azure migration and modernization – Tools & Programs
Bring AI to Every App, Process and employee with Azure AI
Building industry sustainability solutions together with our partners
Innovate with cloud-scale apps, data, and AI
Winning the toughest analytics workloads on Azure
Advancing enterprise Linux application modernization on Azure
New opportunities to grow your practice with Azure Virtual Desktop

Sources: 

1The transformation imperative: Digital drivers in the covid-19 pandemic, The Economist

2Microsoft Digital Defense Report

3750 Million New Logical Applications: More Background, IDC

4Flexera Releases 2021 State of the Cloud Report Press Release, flexera.com

 
Quelle: Azure

da Agency

New SCIM Capabilities for Docker Business

Managing users across hundreds of applications and systems can be a painful process. And it only gets more challenging the larger your organization gets. To make it easier, we introduced Single Sign-On (SSO) earlier this year so you could securely manage Docker users through your standard identity provider (IdP).
Today, we’re excited to announce enhancements to the way you manage users with the addition of System for Cross-Domain Identity Management (SCIM) capabilities. By integrating Docker with your IdP via SCIM, you can automate the provisioning and deprovisioning of user seats. In fact, whatever user changes you make in your IdP will automatically be updated in Docker, eliminating the need to manually add or remove users as they come and go from your organization.
The best part? SCIM is now available with a Docker Business subscription!

What is System for Cross-Domain Identity Management (SCIM)?
SCIM is a provisioning system that allows customers to manage Docker users within their IdP. When SCIM is enabled, you no longer need to update both your organization’s IdP and Docker with any user changes like adding/removing users of profile updates. Your IdP can be the single source of truth. Whatever updates are made there will automatically be reflected in the Members tab on Docker Hub. We recommend enabling SCIM after you verify your domain and set up the SSO connection between your IdP and Docker (SSO enforcement won’t be necessary).
For more information on SSO and SCIM, check out our docs page.
Check out SSO and SCIM in action!
View our webinar on demand. We walk through the advanced management and security tools included in your Docker Business subscription — including a demo of SSO and SCIM — and answer some questions along the way.
SSO and SCIM are available to organizations with a Docker Business subscription.
Click here to learn more about how Docker Business supercharges developer productivity and collaboration without compromising on security and compliance.
Quelle: https://blog.docker.com/feed/