da Agency

Introducing Cloud Build private pools: Secure CI/CD for private networks

A recent survey found that developers spend 39% of their time managing the DevOps infrastructure that powers their continuous integration (CI) and continuous delivery (CD) pipelines. Unreliable availability, manual provisioning, limited scaling, breaking upgrades, long queue times, and high fixed costs all slow down development and take valuable time and focus away from DevOps teams. And while cloud-based CI/CD solutions can solve many of these friction points, they largely only work with cloud-hosted resources. That’s why we’re excited to announce that starting today, you can take advantage of serverless build environments within your own private network, with new Cloud Build private pools. Launched in 2018, Cloud Build has helped thousands of customers modernize their CI/CD workloads to run on fully managed, secure, pay-as-you-go ‘workers’ with no infrastructure to manage. Cloud Build offers on-demand auto-scaling capabilities, active build minute billing, all with no infrastructure to manage. The new private pools feature augments Cloud Build with secure, fully managed CI/CD and DevOps workflow automation that uses network peering to connect into your private networks. Private pools also unlocks a host of new customization options such as new machine types, higher maximum concurrency, regional builds, and network configuration options.With Cloud Build private pools, you get the benefits of a cloud-hosted, fully managed CI/CD product while meeting enterprise security and compliance requirements—even for highly regulated industries like finance, healthcare, retail, and others. For instance, you can trigger fully managed DevOps workflows from source-code repositories hosted in private networks, including Github Enterprise.With private pools, Cloud Build now supports:VPC PeeringVPC-SCStatic IP rangesNo public IPsOrg policy enforcementCross-project buildsBuild from private source repositories with first class integrations, including Github EnterpriseRegionalization in 15 regions across the US, EU, Asia, Australia, and South AmericaHundreds of concurrent builds per pool15 machine typesAnd while designed primarily for private networking use cases, private pools work just as well with resources in Google Cloud, if you’re interested in trying out new features like higher concurrency or additional machine types.Same Cloud Build, new build environmentPrivate pools introduces a new build environment for executing your builds with Cloud Build while maintaining a consistent product and API experience. All the same great features of Cloud Build are available with private pools, including fully managed workers, pay-as-you-go pricing, Cloud Console UI, source repo integrations, IAM permissions, Secret Manager and Pub/Sub integrations, and native support for Google Cloud runtimes like Google Kubernetes Engine (GKE), Cloud Run, Cloud Functions, App Engine, and Firebase.Running builds on a private pool is as easy as creating the pool and setting it as your build environment in your cloudbuild.yaml config file. Private networking is optionally configured via Service Networking by peering your private pool to your customer-managed VPC and supports both peered and shared VPCs.Running your first build is easy:We’re excited to share private pools with you, so you can enjoy the secure, fully managed Cloud Build developer automation platform from your private network. The private pools feature is generally available today, and we look forward to introducing per-trigger service accounts and approval gates soon. To get started, try the quickstart or read the overview documentation for more details.Want to learn more about Cloud Build, and how to use it to improve the security of your software supply chain? On July 29 event Building trust in your software supply chain explores this topic in depth. Click here to register for the live event or to watch it on demand.Related ArticleDevOps on Google Cloud: tools to speed up software development velocityGoogle Cloud’s application development and continuous integration/continuous delivery (CI/CD) tools help ForgeRock developers stay produc…Read Article
Quelle: Google Cloud Platform

da Agency

BigQuery Admin reference guide: Query processing

BigQuery is capable of some truly impressive feats, be it scanning billions of rows based on a regular expression, joining large tables, or completing complex ETL tasks with just a SQL query.  One advantage of BigQuery (and SQL in general), is it’s declarative nature. Your SQL indicates your requirements, but the system is responsible for figuring out how to satisfy that request.However, this approach also has its flaws – namely the problem of understanding intent. SQL represents a conversation between the author and the recipient (BigQuery, in this case). And factors such as fluency and translation can drastically affect how faithfully an author can encode their intent, and how effectively BigQuery can convert the query into a response.In this week’s BigQuery Admin Reference Guide post, we’ll be providing a more in depth view of query processing. Our hope is that this information will help developers integrating with BigQuery, practitioners looking to optimize queries, and administrators seeking guidance to understand how reservations and slots impact query performance. A refresher on architectureBefore we go into an overview of query processing. Let’s revisit BigQuery’s architecture. Last week, we spoke about BigQuery’s native storage on the left hand side. Today, we’ll be focusing on Dremel, BigQuery’s query engine. Note that today we’re talking about BigQuery’s standard execution engine, however BI Engine represents another query execution engine available for fast, in-memory analysis.As you can see from the diagram, dremel is made up of a cluster of workers.  Each one of these workers executes a part of a task independently and in parallel. BigQuery uses a distributed memory shuffle tier to store intermediate data produced from workers at various stages of execution. The shuffle leverages some fairly interesting Google technologies, such as our very fast petabit network technology, and RAM wherever possible. Each shuffled row can be consumed by workers as soon as it’s created by the producers.This makes it possible to execute distributed operations in a pipeline. Additionally, if a worker has partially written some of its output and then terminated (for example, the underlying hardware suffered a power event), that unit of work can simply be re-queued and sent to another worker.  A failure of a single worker in a stage doesn’t mean all the workers need to re-run. When a query is complete, the results are written out to persistent storage and returned to the user. This also enables us to serve up cached results the next time that query executes. Overview of query processingNow that you understand the query processing architecture, we’ll run through query execution at a high level, to see how each step comes together.  First Steps: API request managementBigQuery supports an asynchronous API for executing queries: callers can insert a query job request, and then poll it until complete -as we discussed a few weeks ago.  BigQuery supports a REST-based protocol for this, which accepts queries encoded via JSON.To proceed, there’s some level of API processing that must occur.  Some of things that must be done are authenticating and authorizing the request, plus building and tracking associated metadata such as the SQL statement, cloud project, and/or query parameters.Decoding the query text: Lexing and parsingLexing and parsing is a common task for programming languages, and SQL is no different.  Lexing refers to the process of scanning an array of bytes (the raw SQL statement) and converting that into a series of tokens. Parsing is the process of consuming those tokens to build up a syntactical representation of the query that can be validated and understood by BigQuery’s software architecture.If you’re super interested in this, we recommend checking out the ZetaSQL project, which includes the open source reference implementation of the SQL engine used by BigQuery and other GCP projects.Referencing resources: Catalog resolutionSQL commonly contains references to entities retained by the BigQuery system – such as tables, views, stored procedures and functions. For BigQuery to process these references, it must resolve them into something more comprehensible.  This stage helps the query processing system answer questions like:Is this a valid identifier?  What does it reference?Is this entity a managed table, or a logical view?What’s the SQL definition for this logical view?What columns and data types are present in this table?How do I read the data present in the table?  Is there a set of URIs I should consume?Resolutions are often interleaved through the parsing and planning phases of query execution.Building a blueprint: Query planningAs a more fully-formed picture of the request is exposed via parsing and resolution, a query plan begins to emerge.  Many techniques exist to refactor and improve a query plan to make it faster and more efficient.  Algebraization, for example, converts the parse tree into a form that makes it possible to refactor and simplify subqueries.  Other techniques can be used to optimize things further,  moving tasks like pruning data closer to data reads (reducing the overall work of the system).Another element is adapting it to run as a set of distributed execution tasks.  Like we mentioned in the beginning of this post, BigQuery leverages large pools of query computation nodes, or workers. So, it must coordinate how different stages of the query plan share data through reading and writing from storage, and how to stage temporary data within the shuffle system.Doing the work: Query executionQuery execution is simply the process of working through the query stages in the execution graph, towards completion.  A query stage may have a single unit of work, or it may be represented by many thousands of units of work, like when a query stage reads all the data in a large table composed of many separate columnar input files.Query management: scheduling and dynamic planningBesides the workers that perform the work of the query plan itself, additional workers monitor and direct the overall progress of work throughout the system. Scheduling is concerned with how aggressively work is queued, executed and completed.However, an interesting property of the BigQuery query engine is that it has dynamic planning capabilities.  A query plan often contains ambiguities, and as a query progresses it may need further adjustment to ensure success.  Repartitioning data as it flows through the system is one example of a plan adaptation that may be added, as it helps ensure that data is properly balanced and sized for subsequent stages to consume.Finishing up: finalizing resultsAs a query completes, it often yields output artifacts in the form of results, or changes to tables within the system.  Finalizing results includes the work to commit these changes back to the storage layer. BigQuery also needs to communicate back to you, the user, that the system is done processing the query. The metadata around the query is updated to note the work is done, or the error stream is attached to indicate where things went wrong.Understanding query executionArmed with our new understanding of the life of a query, we can dive more deeply into query plans.  First, let’s look at a simple plan. Here, we are running a query against a public BigQuery dataset to count the total number of citi bike trips that began at stations with “Broadway” in the name. SELECT COUNT(*)FROM `bigquery-public-data.new_york.citibike_trips`WHERE start_station_name LIKE “%Broadway%”Now let’s consider what is happening behind the scenes when BigQuery processes this query. First, a set of workers access the distributed storage to read the table, filter the data, and generate partial counts. Next, these workers send their counts to the shuffle.The second stage reads from those shuffle records as its input, and sums them together. It then writes the output file into a single file, which becomes accessible as the result of the query.You can clearly see that the workers don’t communicate directly with one another at all; they communicate through reading and writing data. After running the query in the BigQuery console, you can see the execution details and gather information about the query plan (note that the execution details shown below may be slightly different than what you see in the console since this data changes). Note that you can also get execution details from the information_schema tables or the Jobs API. For example, by running:SELECT*FROM`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECTWHEREjob_id = “bquxjob_49c5bc47_17ad3d7778f”Interpreting the query statisticsQuery statistics include information about how many work units existed in a stage, as well as how many were completed.  For example, inspecting the result of the information schema query used earlier we can get the following:Input and outputUsing the parallel_inputs field, we can see how finely divided the input is. In the case of a table read, it indicates how many distinct file blocks were in the input.  In the case of a stage that reads from shuffle, the number of inputs tells us how many distinct data buckets are present.  Each of these represent a distinct unit of work that can be scheduled independently. So, in our case, there are 57 different columnar file blocks in the table. In this representation, we can also see the query scanned more than 33 million rows while processing the table.  The second stage read 57 rows, as the shuffle system contained one row for each input from the first stage. It’s also perfectly valid for a stage to finish with only a subset of the inputs processed.  Cases where this happens tend to be execution stages where not all the inputs need to be processed to satisfy what output is needed; a common example of this might be a query stage that consumes part of the input and uses a LIMIT clause to restrict the output to some smaller number of rows.It is also worth exploring the notion of parallelism.  Having 57 inputs for a stage doesn’t mean the stage won’t start until there’s 57 workers (slots) available.  It means that there’s a queue of work with 57 elements to work through.  You can process that queue with a single worker, in which case you’ve essentially got a serial execution.  If you have multiple workers, you can process it faster as they’re working independently to process the units.  However, more than 57 slots doesn’t do anything for you; the work cannot be more finely distributed.Aside from reading from native distributed storage, and from shuffle, it’s also possible for BigQuery to perform data reads and writes from external sources, such as Cloud Storage (as we discussed in our earlier post). In such cases the notion of parallel access still applies, but it’s typically less performant. Slot utilizationBigQuery communicates resource usage through a computational unit known as a slot. It’s simplest to think of it as similar to a virtual CPU and it’s a measure that represents the number of workers available / used. When we talk about slots, we’re talking about overall computational throughput, or rate of change.  For example, a single slot gives you the ability to make 1 slot-second of compute progress per second.  As we just mentioned, having fewer workers – or less slots – doesn’t mean that a job won’t run. It simply means that it may run slower. In the query statistics, we can see the amount of slot_ms (slot-milliseconds) consumed. If we divide this number by the amount of milliseconds it took for the query stage to execute, we can calculate how many fully saturated slots this stage represents. SELECT job_stages.name, job_stages.slot_ms/(job_stages.end_ms – job_stages.start_ms) as full_slotsFROM`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT, UNNEST(job_stages) as job_stagesWHEREjob_id = “bquxjob_49c5bc47_17ad3d7778f”This information is helpful, as it gives us a view of how many slots are being used on average across different workloads or projects – which can be helpful for sizing reservations (more on that soon). If you see areas where there is a higher number of parallel inputs compared to fully saturated slots, that may represent a query that will run faster if it had access to more slots. Time spent in phasesWe can also see the average and maximum time each of the workers spent in the wait, read, compute and write phase for each stage of the query execution:Wait Phase: the engine is waiting for either workers to become available or for a previous stage to start writing results that it can begin consuming. A lot of time spent in the wait phase may indicate that more slots would result in faster processing time. Read Phase: the slot is reading data either from distributed storage or from shuffle.  A lot of time spent here indicates that there might be an opportunity to limit the amount of data consumed by the query (by limiting the result set or filtering data). Compute Phase: where the actual processing takes place, such as evaluating SQL functions or expressions. A well-tuned query typically spends most of its time in the compute phase. Some ways to try and reduce time spent in the compute phase are to leverage approximation functions or investigate costly string manipulations like complex regexes. Write phase: where data is written, either to the next stage, shuffle, or final output returned to the user. A lot of time spent here indicates that there might be an opportunity to limit the results of the stage (by limiting the result set or filtering data)If you notice that the maximum time spent in each phase is much greater than the average time, there may be an uneven distribution of data coming out of the previous stage. One way to try and reduce data skew is by filtering early in the query.Large shufflesWhile many query patterns use reasonable volumes of shuffle, large queries may exhaust available shuffle resources. Particularly, if you see that a query stage is heavily attributing its time spent to writing out to shuffle, take a look at the shuffle statistics.  The shuffleOutputBytesSpilled tells us if the shuffle was forced to leverage disk resources beyond in-memory resources. SELECT job_stages.name, job_stages.shuffle_output_bytes_spilledFROM`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT, UNNEST(job_stages) as job_stagesWHEREjob_id = “bquxjob_49c5bc47_17ad3d7778f”Note that a disk-based write takes longer than an in-memory write. To prevent this from happening, you’ll want to filter or limit the data so that less information is passed to the shuffle. Tune in next weekNext week, we’ll be digging into more advanced queries and talking through tactical query optimization techniques so make sure to tune in! You can keep an eye out for more in this series by following Leigha on LinkedIn and Twitter.Related ArticleBigQuery Admin reference guide: Storage internalsLearn how BigQuery stores your data for optimal analysis, and what levers you can pull to further improve performance.Read Article
Quelle: Google Cloud Platform

da Agency

Why retailers should run in our trusted cloud

Whether they were ready for it or not, the COVID-19 pandemic transformed many retailers into digital businesses. Retailers made huge investments into commerce technologies, customer experience tools, sales and fulfillment technology, and improving digital experiences to continue providing their goods and services to their customers. Now, more than a year into the COVID-19 pandemic, digital retail is the new normal. In fact, many retailers are planning on expanding their digital investments. However, as their digital footprint expands, so do their threats and security concerns. As a digital-focused retailer, your website is the most visible part of your attack surface. Your website is where your customers search for goods or services, make payments, or learn more about your brand. However, your website does not operate in isolation. There is an underlying infrastructure as well as the services that run on top of it that need protection from a wide array of attacks that may seek to compromise your data, internal employees, business, and customers. During this week’s Google Cloud Retail Summit, we’ve shared why Google Cloud is built to be the most trusted cloudfor retailers. From providing you with control over your data as you move from your own data centers to the public cloud to giving you built-in technology to protect your applications all the way to your end users, Google Cloud helps you safely migrate to and operate in our Trusted Cloud.Trusted Cloud Gives You Control, Transparency, and SovereigntyAccess Transparency: We offer the ability to monitor and approve access to your data or configurations by Google Cloud support and engineering based on specific justifications and context, so you have visibility and control over insider access. Certificate Authority Service (CAS): CAS is a highly scalable and available service that simplifies and automates the management and deployment of private CAs while meeting the needs of modern developers and applications. With CAS, you can offload to the cloud time-consuming tasks associated with operating a private CA, like hardware provisioning, infrastructure security, software deployment, high-availability configuration, disaster recovery, backups, and more, allowing you to stand up a private CA in minutes, rather than the months it might normally take to deploy.Confidential Computing: We already encrypt data at-rest and in-transit, but customer data needs to be decrypted for processing. Confidential Computing is a breakthrough technology which encrypts data in-use—while it’s being processed. Confidential VMs take this technology to the next level by offering memory encryption so that you can further isolate your workloads in the cloud. With the beta launch of Confidential VMs, we’re the first major cloud provider to offer this level of security and isolation while giving you a simple, easy-to-use option for your newly built and “lift and shift” applications.Cloud Key Management: We allow you to configure the locations where your data is stored, where your encryption keys are stored, and where your data can be accessed from. We give you the ability to manage your own encryption keys, even storing them outside Google’s infrastructure. Using our External Key Management service, you have the ability to deny any request by Google to access encryption keys necessary to decrypt customer data at rest for any reason.Trusted cloud Helps You Prevent, Detect, and Respond to ThreatsBeyondCorp Enterprise is Google’s comprehensive zero trust product offering. Google has over a decade of experience managing and securing cloud applications at a global scale, and this offering was developed based on learnings from our experience managing our own enterprise, feedback from customers and partners, as well as informed by leading engineering and security research. We understand that most customers host resources across different cloud providers. With this in mind, BeyondCorp Enterprise was purpose-built as a multicloud solution, enabling customers to securely access resources hosted not only on Google Cloud or on-premises, but also across other clouds such as Azure and Amazon Web Services (AWS).Cloud Armor: We’re simplifying how you can use Cloud Armor to help protect your websites and applications from exploit attempts, as well as Distributed Denial of Service (DDoS) attacks. With Cloud Armor Managed Protection Plus, you will get access to DDoS and WAF services, curated rule sets, and other services for a predictable monthly price.Chronicle: Chronicle is a threat detection solution that identifies threats, including ransomware, at unparalleled speed and scale. Google Cloud Threat Intelligence for Chronicle surfaces highly actionable threats based on Google’s collective insight and research into Internet-based threats. Threat Intel for Chronicle allows you to focus on real threats in the environment and accelerate your response time.Google Workspace Security: Used by more than five million organizations worldwide, from large banks and retailers with hundreds of thousands of people to fast-growing startups, Google Workspace and Google Workspace for Education include the collaboration and productivity tools found here. Google Workspace and Google Workspace for Education are designed to help teams work together securely in new, more efficient ways, no matter where members are located or what device they use. For instance, Gmail scans over 300 billion attachments for malware every week and prevents more than 99.9% of spam, phishing, and malware from reaching users. We’re committed to protecting against security 1 threats of all kinds, innovating new security tools for users and admins, and providing our customers with a secure cloud service.Identity & Access Management IAM: Identity and Access Management (IAM) lets administrators authorize who can take action on specific resources, giving you full control and visibility to manage Google Cloud resources centrally. For enterprises with complex organizational structures, hundreds of workgroups, and many projects, IAM provides a unified view into security policy across your entire organization, with built-in auditing to ease compliance processes.reCAPTCHA Enterprise: reCAPTCHA has over a decade of experience defending the internet and data for its network of more than 5 million sites. reCAPTCHA Enterprise builds on this technology with capabilities, such as two-factor authentication and mobile application support, designed specifically for enterprise security concerns. With reCAPTCHA Enterprise, you can defend your website against common web-based attacks like credential stuffing, account takeovers, and scraping and help prevent costly exploits from malicious human and automated actors. And, just like reCAPTCHA v3, reCAPTCHA Enterprise will never interrupt your users with a challenge, so you can run it on all webpages where your customers interact with your services.Security Command Center: With Security Command Center (SCC), our native posture management platform, you can prevent and detect abuse of your cloud resources, centralize security findings from Google Cloud services and partner products, and detect common misconfigurations, all in one easy-to-use platform. We have Premium tier for Security Command Center to provide even more tools to protect your cloud resources. It adds new capabilities that let you spot threats using Google intelligence for events in Google Cloud Platform (GCP) logs and containers, surface large sets of misconfigurations, perform automated compliance scanning and reporting. These features help you understand your risks on Google Cloud, verify that you’ve configured your resources properly and safely, and document it for anyone who asks.VirusTotal:VirusTotal inspects items with over 70 antivirus scanners and URL/domain blocklisting services, in addition to a myriad of tools to extract signals from the studied content. Any user can select a file from their computer using their browser and send it to VirusTotal.Web Risk API: With Web Risk, you can quickly identify known bad sites, warn users before they click infected links, and prevent users from posting links to known infected pages from your site. Web Risk includes data on more than a million unsafe URLs and stays up to date by examining billions of URLs each day.Trusted cloud Plays an Active Role in Our Shared Fate Our trusted cloud provides a shared-fate model for risk management. We stand with retailers from day one, helping them implement best practices for safely migrating to and operating in our trusted cloud.We hope you enjoy the sessions we’ve created for you with Google Cloud Retail Summit and that they help you understand the ways our trusted cloud can help secure retailers all over the world.Related ArticleHow to grow brands in times of rapid changeCatch up on all the best content from Google Cloud’s 2021 Retail & Consumer Packaged Goods summitRead Article
Quelle: Google Cloud Platform

da Agency

Getting the most out of Cloud IDS for advanced network threat detection

Google Cloud IDS, now available in preview, delivers cloud-native, managed, network-based threat detection, built with Palo Alto Networks’ industry-leading threat detection technologies to provide high levels of security efficacy. Cloud IDS can help customers gain deep insight into network-based threats and support industry-specific compliance goals that call for the use of an intrusion detection system. In this blog, we’re diving deeper into how Cloud IDS works to detect network-based threats and how you can get the most out of a Cloud IDS implementation. Getting the most out of Cloud IDSThe implementation of an Intrusion Detection System (IDS) into virtual cloud networks has been a requirement for many cloud customers as a key measure to help keep their networks safe.  The best practices and design strategies to integrate such a system have changed and matured with new technologies in cloud networking. Overcoming issues such as network bottlenecks and the inability to inspect intra-VPC (east/west) traffic has historically troubled network and security teams. Google Cloud IDS is deployed “out-of-path”, abating both of these concerns. Cloud IDS uses Packet Mirroring to copy and forward network traffic, and Private Service Access (PSA) to connect to a set of cloud-native IDS instances which exist in a Google managed project. This allows a Cloud IDS to be seamlessly integrated into an existing Google Cloud Platform (GCP) network without needing to change the VPC design.In order to provide visibility into threats and intrusions detected by IDS instances, Cloud IDS feeds Threat Logs and Security Alerts into Cloud Logging and the Cloud IDS user-interface in the customer project. This is all done under the hood, making it simple to deploy and manage Cloud IDS. Here are some finer points to know and consider when deploying Cloud IDS:Everything starts by creating a Cloud IDS Endpoint — a collector of connection flows — which, behind the scenes, deploys three Palo Alto VM-Series firewall virtual machines (VMs), which live in a Google Cloud managed project.During the IDS Endpoint creation process, the zone and VPC being analyzed must be specified. A specific Cloud IDS instance is capable of inspecting traffic within a region of a VPC.Updates from Palo Alto Networks are picked up weekly by Cloud IDS and pushed to all existing IDS endpointsDuring Endpoint creation, you’ll choose a minimum alert severity level from critical (least verbose) to informational (most verbose).  To feed traffic to the IDS, you will create and attach a Packet Mirroring Policy to the Endpoint.When creating the Packet Mirroring Policy to attach to your Cloud IDS, there are three options to select the mirrored sources: subnets, tags, individual instances.Subnets: Subnets are useful when every instance in a subnet needs to be analyzed. A policy can contain up to 5 subnets.Tags: Tags are useful when groups of instances in one or multiple subnets need to be analyzed. A policy can contain up to 5 tags.Individual instances: Use individual instances only when very select instances need to be analyzed. 50 instances are allowed per policy.Now that you are more familiar with some of the features and steps to create a Cloud IDS, let’s get into some key points that can help to get the most out of your deployment.Use the right mirrored sources and filters for Packet Mirroring for better control of inspected trafficInside a Packet Mirroring Policy, there is an option to filter traffic. It is important to understand that an IP based filter assumes that the address range specified is the remote subnet. In other words, for ingress traffic, the filter address range would be the source network and for egress traffic it would be the destination network. Do not use IP based filters in your packet mirroring policies if the remote network is unknown, such as general, Internet-based traffic. If you do choose to use filters, be aware that you might prevent the Packet Mirroring Policy from sending traffic to the IDS leaving more chances for false negatives. Also, if you choose to use filters, make sure to keep in mind the filter order for Packet Mirroring. A misconfigured filter strategy can mirror traffic away from your Cloud IDS. Lastly, always capture bidirectional traffic in the Traffic Direction filter option.There are some use cases, however, where filters may be quite useful. For example, consider the case where you want to have different alert severities for trusted and untrusted remote networks. In this case, you could create two IDS Endpoints with the same mirrored sources but different filters and “Minimum Alert Severity”. This configuration would push the more trusted remote network traffic to the IDS Endpoint with a more moderate Alert Severity Level and general Internet traffic to the IDS Endpoint with more verbose alerting.In this example, traffic sourced from trusted network 10.2.0.0/16 would be analyzed by ids-endpoint2 and alert on “Medium” level (and above) severity. However, traffic sourced from the untrusted Internet would be mirrored to ids-endpoint1 and alert on “Informational” (and above) level threats. Note that “Internet IP” in the following screenshot will actually show the source address as seen by the mirrored VM.Attach multiple Packet Mirroring Policies to the same IDS EndpointCloud IDS offers flexibility when attaching Packet Mirroring Policies. Multiple Packet Mirroring Policies can be attached to the same IDS Endpoint. For example, a Packet Mirroring Policy that mirrors traffic for “app1” tagged instances and a second policy that captures traffic for “app2” tagged instances can both be attached to “ids-endpoint-1”.  An alternative is a single policy that captures traffic for both network tags. Because a policy can only have up to 5 tags today, when you need to add a 6th tag to the policy, you would have to attach a 2nd policy to the IDS Endpoint.Once a policy is attached, it can be edited as with any other Packet Mirroring Policy. To remove the policy from the endpoint, simply delete the policy; it can always be recreated. There is currently no “detach” option.  Use Shared VPCs and a single Cloud IDS for multiple projectsIf your organization has a centralized networking and security team supporting various projects, and multiple projects require IDS coverage, consider using a Shared VPC. By using a Shared VPC, a single Cloud IDS can support multiple projects as these projects share network resources, including Cloud IDS. The IDS Endpoint must be created in the Host project where the shared VPC actually exists. Cloud IDS in a Shared VPC supports the same three types of mirrored sources as in a conventional VPC, including individual instances that exist in the service projects.Use HTTPS load balancers to offload SSLCloud IDS inspects not only the IP header of the packet, but also the payload. In the case where the payload is encrypted, such as with HTTPS/TLS/SSL traffic, consider placing the application behind a L7 Internal Load Balancer (ILB) or HTTP(S) External Load Balancer. By using these types of load balancers, SSL decryption can be addressed at a layer above the mirrored instances and Cloud IDS would see SSL decrypted traffic and thus be able to inspect and detect intrusions and threats. To learn more about encryption from Google Front Ends to the load balancer backends, see this document.Integrate with SIEM/SOAR solutions for additional analytics and actionLogs from Cloud IDS can be sent from Cloud Logging to 3rd-party tools (e.g., Security Information and Event Management (SIEM) systems and Security Orchestration Automation and Response (SOAR) systems) for further analysis and responsive mitigating action, as defined by your security operation teams. Third-party SIEM and SOAR systems can be configured to run playbooks that will automatically block an attacker’s IP address based on the information ingested from Cloud IDS. Whether using automation or manual efforts, be careful when blocking the recorded source IPs for targets behind proxy based External or Internal Load Balancers. Proxy based load balancers will translate the true source address with a proxy address and denying a perceived attacking address may result in also blocking legitimate traffic.  Consider using Cloud Armor for this level of security. The Web Application Firewall (WAF) and Layer 4 based access control features of Cloud Armor occur before the Cloud Load Balancer’s source NAT, making for a great combination in a security suite.Adding Cloud IDS to your security toolboxBeing cloud-native and Google Cloud integrated, Cloud IDS is simple to deploy, provides high performance and can be up and ready in just a few clicks. Adding Cloud IDS to your existing VPC is easy and requires little to no network redesign because it is placed “out-of-path”. It can be fully deployed, running and alerting quickly. It may also help satisfy your compliance requirements that mandate the use of an intrusion detection system. To help you get started with Cloud IDS, watch this video, and to sign up for access to the preview, visit our webpage.Related ArticleExtending our Trusted Cloud: Introducing Cloud IDS for Network-based Threat DetectionCloud IDS (Intrusion Detection System) helps detect malware, spyware, and command-and-control attacksRead Article
Quelle: Google Cloud Platform

da Agency

What’s new with Google Cloud’s infrastructure – Q2 edition

What a difference a quarter makes. In the span of just three months, Google Cloud teams deliver a raft of new, innovative capabilities for compute, networking, storage, serverless and containers. In this quarterly bulletin, we’re highlighting the key updates that dropped in Q2 2021, for each of the product areas that make up our Infrastructure as a Service (IaaS) capabilities.Which of our Q2 updates were the most exciting? Depends on who you ask. We each had our favorites from our respective teams, as did our colleagues in compute, networking and storage. Below, you can find a longer list of our top updates.ComputeWe made a bunch of great additions to our compute portfolio this quarter, but at the top of the list is the launch of Tau VMs, a new family of virtual machines optimized for scale-out applications and that deliver the best price-performance among leading cloud vendors. The first instances in the Tau family are based on the AMD Milan processors and deliver leading price-performance without compromising x86 compatibility. We are currently registering customers for Preview with availability planned for late Q3.Here’s some other compute news of note: OS Configuration Management (Preview). Simplify compliance across large VM fleets with a new version of OS Configuration Manager.ML-BasedPredictive Autoscaling for Managed Instance Groups. Improve response times for applications with long initialization times and whose workloads vary predictably with daily or weekly cycles.We made several updates to Google Cloud VMware Engine in Q2. We are continuing to deliver innovative features making it easier to run your VMware workloads in the cloud, including autoscaling, Mumbai expansion, HIPAA Compliance, and more.Migrate for Compute Engine V5.0 (GA) – The first offering of Migrate for Compute Engine as a Google Cloud managed service, making it simple and easier to migrate your VMs to Google Cloud.Two new free white papers available for download. The first covers a simple framework — up, out, or both — for getting your cloud migration going. The second covers strategies on how to put your company on a path to successful cloud migration.Fantastic story on how ServiceNow and Google Cloud are partnered with Sabre to optimize and improve their cloud migrations and operations. Insightful story on how PayPal leverages Google Cloud to flawlessly manage surges in financial transactionsNetworkingOn the networking front, we announced Network Connectivity Center, which “provides a single management experience so customers can easily connect and manage on-prem and cloud networks such as VPNs, dedicated and partner interconnects, and SD-WANs,” said Wendy Cartee, Director of Outbound Product Management for Networking. We added high-bandwidth networking options with 50/75/100 Gbpsfor N2 and C2 VM families for high performance computing. And in the security space, we launched Firewall Insights and Cloud Armor Managed Protection Plus, providing increased firewall metrics and ML-powered DDoS protection.Here’s some other networking news to explore: Network Connectivity Center expanded its reach with new partners, Fortinet, Palo Alto Networks, Versa Networks and VMware, allowing enterprises to embrace the power of automation and simplify their networking deployments even further.High bandwidth networking with 100, 75, and 50 Gbps configurations for General Purpose N2 and Compute Optimized C2 Compute Engine VM families: You can now take advantage of these high-throughput VMs for tightly-coupled high performance computing (HPC), network appliances, financial risk modeling and simulation, and scale-out analytics. Firewall Insights provides metrics reports and insight reports on firewall rules to ensure they are being used appropriately and as intended. This report contains information on firewall usage and the impact of various firewall rules on your VPC network. Cloud Armor Managed Protection Plus is a managed application protection service that bundles advanced DDoS protection capabilities, WAF capabilities, ML-based Adaptive Protection, efficient pricing, bill protection and access to Google’s DDoS response support into a subscription.GKE Gateway Controller is Google’s implementation of the Gateway API defined by the K8s community and manages internal and external HTTP/S load balancing for a GKE cluster or a fleet of GKE clusters. Network Intelligence Center now has Dynamic Reachability within the Connectivity Tests module generally available. You can get VM level granularity for loss and latency measurements for network troubleshooting, Storage“For storage, the top-three most exciting things that happened in Q2 were around education, openness and innovation,” said Brian Schwarz, Director of Product Management for Storage. For education, we published a best practices blog for saving money with our Cloud Storage object store offering. We posted simple cheat sheets on block storage and transfer options and rounded it out with a peek inside our infrastructure (turns out we are really good at storage!). Later in Q2 we reaffirmed our commitment to an open ecosystem by announcing some enhancements with NetApp, and ended the quarter on a nice innovation note showcasing our new transfer appliance. Check out our other noteworthy Q2 storage news:Cloud Storage Assured Workloads are GA, your path to running compliant workloads on Google Cloud.Our latest Transfer Appliance is now available in 40TB and 300TB capacities, making it even easier for customers with limited connectivity or bandwidth constraints to transfer data into Google Cloud.We’re excited to report that CMEK support is now available for composite objects in Cloud Storage, adding to the security options available to our customers.For customers supporting performance-critical applications, our new Extreme Persistent Disk tier is GA. Achieve higher maximum IOPS and throughput, and provision IOPS and capacity separately so you can configure your storage to meet your exact needs.FIlestore Basic Backups is now GA. Filestore Backups serves customers employing a disaster recovery and long-term data retention strategy, where having copies in a separate storage system or a different geographical region is a requirement. Read our documentation to learn more.Our Storage Transfer Service has a number of new features, including support for hourly transfers and source/destination paths. Containers and serverlessFinally, in the container and serverless space, it was great to see some extra attention paid to cost optimization. Check out our recent blog post about it. Committed use discounts are a great deal for customers, and we added these for both GKE Autopilot and Cloud Run. We also delivered multi-instance GPUs that let you partition a given GPU across multiple containers. Instead of wasting a whole GPU if you only needed a fraction for a given workload, now you can efficiently distribute it across containers. Not only is Google Cloud the most innovative cloud, but we’re also the most cost-optimized option.Here’s a recap of other major container and serverless news:For GKE, we previewed and generally released key security and networking functionality including GKE Gateway controller, Seccomp, Dataplane v2, Networking Policy Logging, Container-native Cloud DNS, internal load balancer subsetting and Muti-cluster services. Anthos 1.8 is generally available. For vSphere clusters, you can see previews of cluster autoscaling, auto-sizing for user cluster control plan nodes, Windows container support, and admin cluster backup. Meanwhile, Workload Identity, an improved vSphere CSI driver, and more cluster authentication options are now all generally available.For bare metal clusters, there’s a new edge-based profile (2vCPU, 4GB RAM clusters), new audit log options, new networking capabilities, and Workload Identity is GA.Anthos Service Mesh now offers a Google-managed control plane. Move the Istio control plane to a service that we scale and secure on your behalf. Get the value of Istio, without the need to manage it.Migrate for Anthos 1.7.x and 1.8.0 added new discovery and assessment tooling, more control over VM migration plans, and new runtime support for GKE Autopilot and Cloud Run.Speaking of Cloud Run, we introduced a handful of powerful security features including Identity-aware Proxy support for Cloud Run, restriction of ingress for Cloud Run, Secrets Manager integration, Binary Authorization support, and customer-manage encryption keys.Google Cloud’s infrastructure is the launchpad for you to accelerate digital business models, achieve faster time to service, and integrate best-in-class tools for data-powered innovations. Whether you’re just getting started on your infrastructure modernization journey, or you’re looking to try the latest features and tools to advance your infrastructure, we have resources for you. In between quarterly updates on our blog, you can stay up to date on the latest product news and releases by subscribing to our newsletter, visiting our release notes page, or talking to one of our sales experts.Related ArticleRegistration is open for Google Cloud Next: October 12–14Register now for Google Cloud Next on October 12–14, 2021Read Article
Quelle: Google Cloud Platform

da Agency

Engineering Update: BuildKit 0.9 and Docker Buildx 0.6 Releases

On July 16th we released BuildKit 0.9.0, Docker Buildx 0.6.0, Dockerfile 1.3.0 and Dockerfile 1.3.0-labs. These releases come with bug fixes, feature-parity improvements, refactoring and also new features.

Dockerfile new features

Installation

There is no installation needed: BuildKit supports loading frontends dynamically from container images. Images for Dockerfile frontends are available at docker/dockerfile repository.

To use the external frontend, the first line of your Dockerfile needs to be pointing to the specific image you want to use:

# syntax=docker/dockerfile:1.3

More info: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md

RUN command now supports –network flag

RUN command supports –network flag for requesting a specific type of network conditions (host, none, default). This allows the user to disable all networking during a docker build to ensure no network is used, or opt-in to explicitly using the hosts network, which requires a specific flag to be set before this works.

–network=host requires allowing network.host entitlement. This feature was previously only available on labs channel:

# syntax=docker/dockerfile:1.3
FROM python:3.6
ADD mypackage.tgz wheelz/
RUN –network=none pipe install –find-links wheels mypackage

RUN –mount flag variable expansion

Values for RUN –mount flag now support variable expansion, except for the from field:

# syntax=docker/dockerfile:1.3
FROM golang
…
ARG GO_CACHE_DIR=/root/.cache/go-build
RUN –mount=type=cache,target=$GO_CACHE_DIR go build …

Here-document syntax

RUN and COPY commands now support Here-document syntax allowing writing multiline inline scripts and files (labs channel) without lots of && and characters:

Before:

# syntax=docker/dockerfile:1.3
FROM debian
RUN apt-get update
&& apt-get install -y vim

With new Here-document syntax:

# syntax=docker/dockerfile:1.3-labs
FROM debian
RUN <<eot bash
apt-get update
apt-get install -y vim
eot

In COPY commands source parameters can be replaced with here-doc indicators. Regular here-doc variable expansion and tab stripping rules apply:

# syntax=docker/dockerfile:1.3-labs
FROM alpine
ARG FOO=bar
COPY <<-eot /app/foo
hello ${FOO}
eot

More info on BuildKit repository.

OpenTracing providers replaced with support for OpenTelemetry

OpenTelemetry has superseded OpenTracing. The API is quite similar but additional collector endpoints should allow forwarding traces from client or container in the future.

JAEGER_TRACE env is supported like before. OTEL collector is also supported both via grpc and HTTP protos.

This is also the first time cross-process tracing from Buildx is available:

# create jaeger container
$ docker run -d –name jaeger
-p 6831:6831/udp -p 16686:16686
jaegertracing/all-in-one

# create builder with JAEGER_TRACE env
$ docker buildx create
–name builder
–driver docker-container
–driver-opt network=host
–driver-opt env.JAEGER_TRACE=localhost:6831
–use

# open Jaeger UI at http://localhost:16686/ and see results as shown below

Resource limiting

Users can now limit the parallelism of the BuildKit solver, Buildkitd config now allows max-parallelism for limiting the parallelism of the BuildKit solver, which is particularly useful for low-powered machines.

Here is an example if you want to do this with GitHub Actions:

We are also now limiting TCP connections to 4 per registry with an additional connection not used for layer pulls and pushes. This limitation will be able to manage TCP connection per host to avoid your build being stuck while pulling images. The additional connection is used for metadata requests (image config retrieval) to enhance the overall build time.

GitHub Actions cache backend (experimental)

We have released a new experimental GitHub cache backend to speed up Docker builds in GitHub actions.

This solves a previously inefficient method which reuses the build cache by using official actions/cache action together with local cache exporter/importer, which is inefficient as the cache needs to be saved/loaded every time and tracking does not happen per blob.

To start using this experimental cache, use our example on Build Push Action repository.

Git improvements

Default branch name is now detected correctly from remote when using the Git context.

Support for subdirectories when building from Git source is now released:

$ docker buildx build git://github.com/repo:path/to/subapp

SSH agent is automatically forwarded when building from SSH Git URL:

$ docker buildx build git@github.com:myrepo.git

New platforms and QEMU update

Risc-V (buildkitd, buildctl, buildx, frontends)Windows ARM64 (buildctl, buildx)MacOS ARM64 (buildctl)

Also embedded QEMU emulators have been updated to v6.0.0 and provide emulation for additional platforms.

–metadata-file for buildctl and buildx

New –metadata-file flag has been added to build and bake command for buildx that allows saving build result metadata in JSON format:

$ docker buildx build –output type=docker –metadata-file ./metadata.json .

{
“containerimage.config.digest”: “sha256:d8b8b4f781520aeafedb5a88ff50fbb625cfebad87e392794f1e26a724a2f22a”,
“containerimage.digest”: “sha256:868f04872380274dcf8528222e84dc66702394de80889e51c87a14126ea9ff6a”
}

Buildctl also allows –metadata-file flag to output build metadata.

Docker buildx image

Buildx binaries can now be accessed through buildx-bin Docker image to build able to use buildx inside a Docker image in your CI for example. Here is how to use buildx inside a Dockerfile:

FROM docker
COPY –from=docker/buildx-bin:latest /buildx /usr/libexec/docker/cli-plugins/docker-buildx
RUN docker buildx version

Other changes

For the complete list of changes, see the official release notes:

https://github.com/moby/buildkit/releases/tag/v0.9.0https://github.com/moby/buildkit/releases/tag/dockerfile%2F1.3.0https://github.com/moby/buildkit/releases/tag/dockerfile%2F1.3.0-labshttps://github.com/docker/buildx/releases/tag/v0.6.0

The post Engineering Update: BuildKit 0.9 and Docker Buildx 0.6 Releases appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/