Using Generative AI to Create Runnable Markdown

This ongoing GenAI Docker Labs series will explore the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing things as open source so you can play, explore, and hack with us, too.

Generative AI (GenAI) is changing how we interact with tools. Today, we might experience this predominantly through the use of new AI-powered chat assistants, but there are other opportunities for generative AI to improve the life of a developer.

When developers start working on a new project, they need to get up to speed on the tools used in that project. A common practice is to document these practices in a project README.md and to version that documentation along with the project. 

Can we use generative AI to generate this content? We want this content to represent best practices for how tools should be used in general but, more importantly, how tools should be used in this particular project.

We can think of this as a kind of conversation between developers, agents representing tools used by a project, and the project itself. Let’s look at this for the Docker tool itself.

Generating Markdown in VSCode

For this project, we have written a VSCode extension that adds one new command called “Generate a runbook for this project.” Figure 1 shows it in action:

Figure 1: VSCode extension to generate a runbook.

This approach combines prompts written by tool experts with knowledge about the project itself. This combined context improves the LLM’s ability to generate documentation (Figure 2).

Figure 2: This approach combines expert prompts with knowledge about the project itself.

Although we’re illustrating this idea on a tool that we know very well (Docker!), the idea of generating content in this manner is quite generic. The prompts we used for getting started with the Docker build, run, and compose are available from GitHub. There is certainly an art to writing these prompts, but we think that tool experts have the right knowledge to create prompts of this kind, especially if AI assistants can then help them make their work easier to consume.

There is also an essential point here. If we think of the project as a database from which we can retrieve context, then we’re effectively giving an LLM the ability to retrieve facts about the project. This allows our prompts to depend on local context. For a Docker-specific example, we might want to prompt the AI to not talk about compose if the project has no compose.yaml files. 

“I am not using Docker Compose in this project.”

That turns out to be a transformative user prompt if it’s true. This is what we’d normally learn through a conversation. However, there are certain project details that are always useful. This is why having our assistants right there in the local project can be so helpful.

Runnable Markdown

Although Markdown files are mainly for reading, they often contain runnable things. LLMs converse with us in text that often contains code blocks that represent actual runnable commands. And, in VSCode, developers use the embedded terminal to run commands against the currently open project. Let’s short-circuit this interaction and make commands runnable directly from these Markdown runbooks.

In the current extension, we’ve added a code action to every code block that contains a shell command so that users can launch that command in the embedded terminal. During our exploration of this functionality, we have found that treating the Markdown file as a kind of REPL (read-eval-print-loop) can help to refine the output from the LLM and improve the final content. Figure 3 what this looks like in action:

Figure 3: Adding code to allow users to launch the command in the embedded terminal.

Markdown extends your editor

In the long run, nobody is going to navigate to a Markdown file in order to run a command. However, we can treat these Markdown files as scripts that create commands for the developer’s edit session. We can even let developers bind them to keystrokes (e.g., type ,b to run the build code block from your project runbook).

In the end, this is just the AI Assistant talking to itself. The Assistant recommends a command. We find the command useful. We turn it into a shortcut. The Assistant remembers this shortcut because it’s in our runbook, and then makes it available whenever we’re developing this project.

Figure 4: The Assistant in action.

Figure 4 shows a real feedback loop between the Assistant, the generated content, and the developer that is actually running these commands. 

As developers, we tend to vote with our keyboards. If this command is useful, let’s make it really easy to run! And if it’s useful for me, it might be useful for other members of my team, too.

The GitHub repository and install instructions are ready for you to try today.

For more, see this demo: VSCode Walkthrough of Runnable Markdown from GenAI.

Subscribe to Docker Navigator to stay current on the latest Docker news.

Learn more

Subscribe to the Docker Newsletter.

Read Docker, Putting the AI in Containers.

Read the AI Trends Report 2024: AI’s Growing Role in Software Development.

Quelle: https://blog.docker.com/feed/

ReadMeAI: An AI-powered README Generator for Developers

This post was written in collaboration with Docker AI/ML Hackathon participants Gitanshu Sankhla and Vijay Barma.

In this AI/ML Hackathon post, we’ll share another interesting winning project from last year’s Docker AI/ML Hackathon. This time, we will dive into ReadMeAI, one of the honorable mention winners. 

For many developers, planning and writing code is the most enjoyable part of the process. It’s where creativity meets logic, and lines of code transform into solutions. Although some developers find writing documentation equally fulfilling, crafting clear and concise code instructions isn’t for everyone.

Imagine you’re a developer working on a complex project with a team. You just pushed your final commit with a sign of relief, but the clock is ticking on your deadline. You know that clear documentation is crucial. Your teammates need to understand your code’s intricacies for smooth integration, but writing all that documentation feels like another project entirely, stealing your precious time from bug fixes and testing. That’s where ReadMeAI, an AI-powered README generator fits in. 

What makes ReadMeAI unique?

The following demo, which was submitted to the AI/ML Hackathon, provides an overview of ReadMeAI (Figure 1).

Figure 1: Demo of the ReadMeAI as submitted to the AI/ML Hackathon.

The ReadMeAI tool allows users to upload a code file and describe their project. The tool generates Markdown code, which can be edited in real-time using a code editor, and the changes are previewed instantly.

The user interface of ReadmeAI is designed to be clean and modern, making the application easy to use for all users.

Benefits of ReadMeAI include:

Effortless documentation: Upload your code, provide a brief description, and let ReadMeAI generate a comprehensive markdown file for your README seamlessly.

Seamless collaboration: ReadMeAI promotes well-structured READMEs with essential sections, making it easier for your team to understand and contribute to the codebase, fostering smoother collaboration.

Increased efficiency: Stop wasting time on boilerplate documentation. ReadMeAI automates the initial draft of your README, freeing up valuable developer time for coding, testing, and other crucial project tasks.

Use cases include:

API documentation kick-off: ReadMeAI provides a solid foundation for your API documentation. It generates an initial draft outlining API endpoints, parameters, and expected responses. This jumpstarts your process and lets you focus on the specifics of your API’s functionality.

Rapid prototyping and documentation: During rapid prototyping, functionality often takes priority over documentation. ReadMeAI bridges this gap. It quickly generates a basic README with core information, allowing developers to have documentation in place while focusing on building the prototype.

Open source project kick-off: ReadMeAI can jumpstart the documentation process for your open source project. Simply provide your codebase and a brief description, and ReadMeAI generates a well-structured README file with essential sections like installation instructions, usage examples, and contribution guidelines. This saves you time and ensures consistent documentation across your projects.

Focus on what you do best — coding. Let ReadMeAI handle the rest.

How does it work?

ReadMeAI converts code and description into a good-looking README file. Users can upload code files and describe their code in a few words, and ReadMeAI will generate Markdown code for your README. You will get a built-in editor to format your README according to your needs, and then you can download your README in Markdown and HTML format. 

Figure 2 shows an overview of the ReadMeAI architecture.

Figure 2: Architecture of the ReadMeAI tool displaying frontend and backend.

Technical stack

The ReadMeAI tech stack includes:

Node.js: A server-side runtime that handles server-side logic and interactions.

Express: A popular Node.js framework that handles routing, middleware, and request handling.

Google PaLM API: Google’s Pathways Language Model (PaLM) is a 540-billion parameter transformer-based large language model. It is used in the ReadMeAI project to generate a Markdown README based on the uploaded code and user description.

Embedded JavaScript (EJS): A templating engine that allows you to render and add dynamic content to the HTML on the server side.

Cascading Style Sheets (CSS): Add styling to the generated Markdown content.

JavaScript: Add interactivity to the front end, handle client-side logic, and communicate with the server side.

AI integration and markdown generation

The AI integration is handled by the controllers/app.js file (as shown below), specifically in the postApp function. The uploaded code and user description are passed to the AI integration, which uses the Google Palm API to generate a Markdown README. 

The Markdown generator is implemented in the postApp function. The AI-generated content is converted into Markdown format using the showdown library.

const fs = require('fs');
const path = require('path');

const showdown = require('showdown');
const multer = require('multer');
const zip = require('express-zip');

const palmApi = require('../api/fetchPalm');

// showdown converter
const converter = new showdown.Converter();
converter.setFlavor('github');

// getting template
let template;
fs.readFile('./data/template.txt', 'utf8', (err, data) => {
if (err) {
console.error(err)
return
}
template = data;
});

// getting '/'
exports.getApp = (req, res)=>{
res.render('home', {
pageTitle: 'ReadMeAI – Home'
})
}

exports.getUpload = (req, res)=>{
res.render('index', {
pageTitle: 'ReadMeAI – Upload'
})
}

// controller to sent generate readme from incoming data
exports.postApp = (req, res)=>{
let html, dt;
const code = req.file.filename;
const description = req.body.description;

try {
dt = fs.readFileSync(`uploads/${code}`, 'utf8');
} catch (err) {
console.error("read error",err);
}

palmApi.getData(template, dt, description)
.then(data => {
html = converter.makeHtml(data);
res.render('editor', {
pageTitle: 'ReadMeAI – Editor',
html: html,
md: data
});
//deleting files from upload folder
fs.unlink(`uploads/${code}`, (err) => {
if (err) {
console.error(err);
return;
}
console.log('File deleted successfully');
});

}).catch(err => console.log('error occured',err));

}

exports.postDownload = (req, res) => {
const html = req.body.html;
const md = req.body.markdown;

const mdFilePath = path.join(__dirname, '../downloads/readme.md');
const htmlFilePath = path.join(__dirname, '../downloads/readme.html');

fs.writeFile(mdFilePath, md, (err) => {
if (err) console.error(err);
else console.log('Created md file successfully');
});

fs.writeFile(htmlFilePath, html, (err) => {
if (err) console.error(err);
else console.log('Created html file successfully');
});

res.zip([
{ path: mdFilePath, name: 'readme.md' },
{ path: htmlFilePath, name: 'readme.html' }
]);
}

The controller functions (gettApp, getUpload, postApp, postDownload) handle the incoming requests and interact with the AI integration, markdown generator, and views. After generating the Markdown content, the controllers pass the generated content to the appropriate views.

These controller functions are then exported and used in the routes defined in the routes/app.js file.

Views 

The views are defined in the views/ directory. The editor.ejs file is an Embedded JavaScript (EJS) file that is responsible for rendering the editor view. It is used to generate HTML markup that is sent to the client.

<%- include('includes/head.ejs') %>
<!– google fonts –>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@24,400,0,0" />
<!– stylesheets –>
<link rel="stylesheet" href="/css/edistyles.css">
<link rel="stylesheet" href="/css/output.css">

</head>
<body>
<header class="header-nav">
<h1 class="logo">ReadMeAI</h1>
<div class="light-container">
<div class="phone">
<span class="material-symbols-outlined" id="rotate-item">
phone_iphone</span>
</div>
<div class="tubelight">
<div class="bulb"></div>
</div>
</div>
</header>
<main class="main">
<div class="mobile-container">
<p>Sorry but the editor is disable on mobile device's, but it's best experienced on a PC or Tablet </p>
…..
<button class="btn-containers" id="recompile">
<span class="material-symbols-outlined">bolt</span>
</button>
</header>
<textarea name="textarea" id="textarea" class="sub-container output-container container-markdown" ><%= md %></textarea>
</div>
…..
<!– showdown cdn –>
<script src="https://cdnjs.cloudflare.com/ajax/libs/showdown/2.1.0/showdown.min.js" integrity="sha512-LhccdVNGe2QMEfI3x4DVV3ckMRe36TfydKss6mJpdHjNFiV07dFpS2xzeZedptKZrwxfICJpez09iNioiSZ3hA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<!– ionicons cdn –>
<script type="module" src="https://unpkg.com/ionicons@7.1.0/dist/ionicons/ionicons.esm.js"></script>
<script nomodule src="https://unpkg.com/ionicons@7.1.0/dist/ionicons/ionicons.js"></script>

<script src="/scripts/edi-script.js"></script>
<script src="/scripts/tubelightBtn.js"></script>
</body>

Rendering the view

The controllers render the appropriate views with the generated content or serve API responses. The editor.ejs view is rendered with the generated Markdown content (html: html, md: data).

exports.postApp = (req, res) => {
//…
// Generate Markdown content
//…

res.render('editor', {
pageTitle: 'ReadMeAI – Editor',
html: html,
md: data
});
};

When the postApp function is called, the palmApi.getData function is used to fetch data from the Palm API based on the template, the incoming Markdown content, and the provided description. Once the data is fetched, the converter.makeHtml function is used to convert the Markdown content to HTML.

The res.render function is then used to render the editor view with the generated HTML content and Markdown content. The editor.ejs view should have the necessary code to display the HTML content and Markdown content in the desired format.

This approach allows for the dynamic generation of README content based on the incoming Markdown content and the provided template. The generated HTML content then gets rendered into the web page for the user to view.

Sending the response 

The rendered view is sent as a response to the client using the res.render function. This function is used to render a view. This process ensures that the generated Markdown content is dynamically rendered into a web page using the provided template, and the web page is then sent as a response to the client.

Getting started

To get started, ensure that you have installed the latest version of Docker Desktop.

Clone the repository

Open a terminal window and run the following command to clone the sample application:

git clone https://github.com/Gitax18/ReadMeAI

You should now have the following files in your ReadMeAI directory:

ReadMeAI
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── README.md
├── api
│ └── fetchPalm.js
├── controllers
│ └── app.js
├── data
│ ├── output.md
│ └── template.txt
├── downloads
│ ├── readme.html
│ └── readme.md
├── package-lock.json
├── package.json
├── public
│ ├── css
│ │ ├── edistyles.css
│ │ ├── home.css
│ │ ├── index.css
│ │ └── output.css
│ ├── images
│ │ ├── PaLM_API_Graphics-02.width-1200.format-webp.webp
│ │ ├── logos
│ │ │ ├── dh.png
│ │ │ ├── dp.png
│ │ │ └── gh.png
│ │ ├── pre.png
│ │ └── vscode.jpg
│ └── scripts
│ ├── edi-script.js
│ ├── home.js
│ ├── index.js
│ └── tubelightBtn.js
├── routes
│ └── app.js
├── server.js
├── uploads
│ ├── 1699377702064#Gradient.js
│ └── important.md
└── views
├── 404.ejs
├── editor.ejs
├── home.ejs
├── includes
│ └── head.ejs
└── index.ejs

14 directories, 35 files

Understanding the project directory structure

Here’s an overview of the project directory structure and the purpose of each folder and file:

api/: Contains code to connect to third-party APIs, such as Google PaLM 2.

controllers/: Includes all the business logic for handling POST/GET requests.

views/: Contains files for rendering on the client side.

data/: Holds the ‘template’ for the output and ‘output.md’ for the generated markdown.

public/: Contains client-side CSS and scripts.

routes/: Manages routes and calls the respective controller functions for each route.

uploads/: Temporarily stores files received from the client side, which are deleted once the session ends.

server.js: The main Express server file, executed when starting the server.

Dockerfile: Contains the script to containerize the project.

Building the app

Run the following command to build the application.

docker build -t readmeai .

Run the app:

docker run -d -p 3333:3333 readmeai

You will see log output similar to the following:

> readme-ai-generator@1.0.0 start
> node server.js

server is listening at http://localhost:3333

Figure 3: Docker dashboard listing the running ReadMeAI container.

Alternatively, you can pull and run the ReadMeAI Docker image directly from Docker Hub (Figure 3) using the following command:

docker run -it -p 3333:3333 gitax18/readmeai

You should be able to access the application at http://localhost:3333 (Figure 4).

Figure 4: The landing page of the ReadMeAI tool.

Select Explore and upload your source code file by selecting Click to upload file (Figure 5).

Figure 5: The Main UI page that allows users to upload their project file.

Once you finish describing your project, select Generate (Figure 6).

Figure 6: Uploading the project file and creating a brief description of the code/project.

ReadMeAI utilizes Google’s Generative Language API to create draft README files based on user-provided templates, code snippets, and descriptions (Figure 7).

Figure 7: Initial output from ReadMeAI. The built-in editor makes minor changes simple.

What’s next?

ReadMeAI was inspired by a common problem faced by developers: the time-consuming and often incomplete task of writing project documentation. ReadMeAI was developed to streamline the process, allowing developers to focus more on coding and less on documentation. The platform transforms code and brief descriptions into comprehensive, visually appealing README files with ease.

We are inspired by the ingenuity of ReadMeAI, particularly in solving a fundamental issue in the developer community. 

Looking ahead, the creators plan to enhance ReadMeAI with features like GitHub integration, custom templates, and improved AI models such as Llama. By adopting newer technologies and architectures, they plan to make ReadMeAI even more powerful and efficient.

Join us in this journey to improve ReadMeAI making it an indispensable tool for developers worldwide.

Learn more

Subscribe to the Docker Newsletter.

Get the latest release of Docker Desktop.

Vote on what’s next! Check out our public roadmap.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Quelle: https://blog.docker.com/feed/

Understanding the Docker USER Instruction

In the world of containerization, security and proper user management are crucial aspects that can significantly affect the stability and security of your applications. The USER instruction in a Dockerfile is a fundamental tool that determines which user will execute commands both during the image build process and when running the container. By default, if no USER is specified, Docker will run commands as the root user, which can pose significant security risks. 

In this blog post, we will delve into the best practices and common pitfalls associated with the USER instruction. Additionally, we will provide a hands-on demo to illustrate the importance of these practices. Understanding and correctly implementing the USER instruction is vital for maintaining secure and efficient Docker environments. Let’s explore how to manage user permissions effectively, ensuring that your Docker containers run securely and as intended.

Docker Desktop 

The commands and examples provided are intended for use with Docker Desktop, which includes Docker Engine as an integrated component. Running these commands on Docker Community Edition (standalone Docker Engine) is possible, but your output may not match that shown in this post. The blog post How to Check Your Docker Installation: Docker Desktop vs. Docker Engine explains the differences and how to determine what you are using.

UID/GID: A refresher

Before we discuss best practices, let’s review UID/GID concepts and why they are important when using Docker. This relationship factors heavily into the security aspects of these best practices.

Linux and other Unix-like operating systems use a numeric identifier to identify each discrete user called a UID (user ID). Groups are identified by a GID (group ID), which is another numeric identifier. These numeric identifiers are mapped to the text strings used for username and groupname, but the numeric identifiers are used by the system internally.

The operating system uses these identifiers to manage permissions and access to system resources, files, and directories. A file or directory has ownership settings including a UID and a GID, which determine which user and group have access rights to it. Users can be members of multiple groups, which can complicate permissions management but offers flexible access control.

In Docker, these concepts of UID and GID are preserved within containers. When a Docker container is run, it can be configured to run as a specific user with a designated UID and GID. Additionally, when mounting volumes, Docker respects the UID and GID of the files and directories on the host machine, which can affect how files are accessed or modified from within the container. This adherence to Unix-like UID/GID management helps maintain consistent security and access controls across both the host and containerized environments. 

Groups

Unlike USER, there is no GROUP directive in the Dockerfile instructions. To set up a group, you specify the groupname (GID) after the username (UID). For example, to run a command as the automation user in the ci group, you would write USER automation:ci in your Dockerfile.

If you do not specify a GID, the list of groups that the user account is configured as part of is used. However, if you do specify a GID, only that GID will be used. 

Current user

Because Docker Desktop uses a virtual machine (VM), the UID/GID of your user account on the host (Linux, Mac, Windows HyperV/WSL2) will almost certainly not have a match inside the Docker VM.

You can always check your UID/GID by using the id command. For example, on my desktop, I am UID 503 with a primary GID of 20:

$ id
uid=503(jschmidt) gid=20(staff) groups=20(staff),<–SNIP–>

Best practices

Use a non-root user to limit root access

As noted above, by default Docker containers will run as UID 0, or root. This means that if the Docker container is compromised, the attacker will have host-level root access to all the resources allocated to the container. By using a non-root user, even if the attacker manages to break out of the application running in the container, they will have limited permissions if the container is running as a non-root user. 

Remember, if you don’t set a USER in your Dockerfile, the user will default to root. Always explicitly set a user, even if it’s just to make it clear who the container will run as.

Specify user by UID and GID

Usernames and groupnames can easily be changed, and different Linux distributions can assign different default values to system users and groups. By using a UID/GID you can ensure that the user is consistently identified, even if the container’s /etc/passwd file changes or is different across distributions. For example:

USER 1001:1001

Create a specific user for the application

If your application requires specific permissions, consider creating a dedicated user for your application in the Dockerfile. This can be done using the RUN command to add the user. 

Note that when we are creating a user and then switching to that user within our Dockerfile, we do not need to use the UID/GID because they are being set within the context of the image via the useradd command. Similarly, you can add a user to a group (and create a group if necessary) via the RUN command.

Ensure that the user you set has the necessary privileges to run the commands in the container. For instance, a non-root user might not have the necessary permissions to bind to ports below 1024. For example:

RUN useradd -ms /bin/bash myuser
USER myuser

Switch back to root for privileged operations

If you need to perform privileged operations in the Dockerfile after setting a non-root user, you can switch to the root user and then switch back to the non-root user once those operations are complete. This approach adheres to the principle of least privilege; only tasks that require administrator privileges are run as an administrator. Note that it is not recommended to use sudo for privilege elevation in a Dockerfile. For example:

USER root
RUN apt-get update && apt-get install -y some-package
USER myuser

Combine USER with WORKDIR

As noted above, the UID/GID used within a container applies both within the container and with the host system. This leads to two common problems:

Switching to a non-root user and not having permissions to read or write to the directories you wish to use (for example, trying to create a directory under / or trying to write in /root.

Mounting a directory from the host system and switching to a user who does not have permission to read/write to the directory or files in the mount.

USER root
RUN mkdir /app&&chown ubuntu
USER ubuntu
WORKDIR /app

Example

The following example shows you how the UID and GID behave in different scenarios depending on how you write your Dockerfile. Both examples provide output that shows the UID/GID of the running Docker container. If you are following along, you need to have a running Docker Desktop installation and a basic familiarity with the docker command.

Standard Dockerfile

Most people take this approach when they first begin using Docker; they go with the defaults and do not specify a USER.

# Use the official Ubuntu image as the base
FROM ubuntu:20.04

# Print the UID and GID
CMD sh -c "echo 'Inside Container:' && echo 'User: $(whoami) UID: $(id -u) GID: $(id -g)'"

Dockerfile with USER

This example shows how to create a user with a RUN command inside a Dockerfile and then switch to that USER.

# Use the official Ubuntu image as the base
FROM ubuntu:20.04

# Create a custom user with UID 1234 and GID 1234
RUN groupadd -g 1234 customgroup &&
useradd -m -u 1234 -g customgroup customuser

# Switch to the custom user
USER customuser

# Set the workdir
WORKDIR /home/customuser

# Print the UID and GID
CMD sh -c "echo 'Inside Container:' && echo 'User: $(whoami) UID: $(id -u) GID: $(id -g)'"

Build the two images with:

$ docker build -t default-user-image -f Dockerfile1 .
$ docker build -t custom-user-image -f Dockerfile2 .

Default Docker image

Let’s run our first image, the one that does not provide a USER command. As you can see, the UID and GID are 0/0, so the superuser is root. There are two things at work here. First, we are not defining a UID/GID in the Dockerfile so Docker defaults to the superuser. But how does it become a superuser if my account is not a superuser account? This is because the Docker Engine runs with root permissions, so containers that are built to run as root inherit the permissions from the Docker Engine.

$ docker run –rm default-user-image
Inside Container:
User: root UID: 0 GID: 0
Custom User Docker Image

Custom Docker image

Let’s try to fix this — we really don’t want Docker containers running as root. So, in this version, we explicitly set the UID and GID for the user and group. Running this container, we see that our user is set appropriately.

$ docker run –rm custom-user-image
Inside Container:
User: customuser UID: 1234 GID: 1234

Enforcing best practices

Enforcing best practices in any environment can be challenging, and the best practices outlined in this post are no exception. Docker understands that organizations are continually balancing security and compliance against innovation and agility and is continually working on ways to help with that effort. Our Enhanced Container Isolation (ECI) offering, part of our Hardened Docker Desktop, was designed to address the problematic aspects of having containers running as root.

Enhanced Container Isolation mechanisms, such as user namespaces, help segregate and manage privileges more effectively. User namespaces isolate security-related identifiers and attributes, such as user IDs and group IDs, so that a root user inside a container does not map to the root user outside the container. This feature significantly reduces the risk of privileged escalations by ensuring that even if an attacker compromises the container, the potential damage and access scope remain confined to the containerized environment, dramatically enhancing overall security.

Additionally, Docker Scout can be leveraged on the user desktop to enforce policies not only around CVEs, but around best practices — for example, by ensuring that images run as a non-root user and contain mandatory LABELs.

Staying secure

Through this demonstration, we’ve seen the practical implications and benefits of configuring Docker containers to run as a non-root user, which is crucial for enhancing security by minimizing potential attack surfaces. As demonstrated, Docker inherently runs containers with root privileges unless specified otherwise. This default behavior can lead to significant security risks, particularly if a container becomes compromised, granting attackers potentially wide-ranging access to the host or Docker Engine.

Use custom user and group IDs

The use of custom user and group IDs showcases a more secure practice. By explicitly setting UID and GID, we limit the permissions and capabilities of the process running inside the Docker container, reducing the risks associated with privileged user access. The UID/GID defined inside the Docker container does not need to correspond to any actual user on the host system, which provides additional isolation and security.

User namespaces

Although this post extensively covers the USER instruction in Docker, another approach to secure Docker environments involves the use of namespaces, particularly user namespaces. User namespaces isolate security-related identifiers and attributes, such as user IDs and group IDs, between the host and the containers. 

With user namespaces enabled, Docker can map the user and group IDs inside a container to non-privileged IDs on the host system. This mapping ensures that even if a container’s processes break out and gain root privileges within the Docker container, they do not have root privileges on the host machine. This additional layer of security helps to prevent the escalation of privileges and mitigate potential damage, making it an essential consideration for those looking to bolster their Docker security framework further. Docker’s ECI offering leverages user namespaces as part of its security framework.

Conclusion

When deploying containers, especially in development environments or on Docker Desktop, consider the aspects of container configuration and isolation outlined in this post. Implementing the enhanced security features available in Docker Business, such as Hardened Docker Desktop with Enhanced Container Isolation, can further mitigate risks and ensure a secure, robust operational environment for your applications.

Learn more

Read the Dockerfile reference guide.

Get the latest release of Docker Desktop.

Explore Docker Guides.

New to Docker? Get started.

Subscribe to the Docker Newsletter.

Quelle: https://blog.docker.com/feed/

How to Measure DevSecOps Success: Key Metrics Explained

DevSecOps involves the integration of security throughout the entire software development and delivery lifecycle, representing a cultural shift where security is a collective responsibility for everyone building software. By embedding security at every stage, organizations can identify and resolve security issues earlier in the development process rather than during or after deployment.

Organizations adopting DevSecOps often ask, “Are we making progress?” To answer this, it’s crucial to implement metrics that provide clear insights into how an organization’s security posture evolves over time. Such metrics allow teams to track progress, pinpoint areas for improvement, and make informed decisions to drive continuous improvement in their security practices. By measuring the changing patterns in key indicators, organizations can better understand the impact of DevSecOps and make data-driven adjustments to strengthen their security efforts. 

Organizations commonly have many DevSecOps metrics that they can draw from. In this blog post, we explore two foundational metrics for assessing DevSecOps success. 

Key DevSecOps metrics

1. Number of security vulnerabilities over time

Vulnerability analysis is a foundational practice for any organization embarking on a software security journey. This metric tracks the volume of security vulnerabilities identified in a system or software project over time. It helps organizations spot trends in vulnerability detection and remediation, signaling how promptly security gaps are being remediated or mitigated. It can also be an indicator of the effectiveness of an org’s vulnerability management initiatives and their adoption, both of which are crucial to reducing the risk of cyberattacks and data breaches.

2. Compliance with security policies

Many industries are subject to cybersecurity frameworks and regulations that require organizations to maintain specific security standards. Policies provide a way for organizations to codify the rules for producing and using software artifacts. By tracking policy compliance over time, organizations can verify consistent adherence to established security requirements and best practices, promoting a unified approach to software development.

The above metrics are a good starting point for most organizations looking to measure their transformation from DevSecOps activities. The next step — once these metrics are implemented — is to invest in an observability system that enables relevant stakeholders, such as security engineering, to easily consume the data. 

DevSecOps insights with Docker Scout

Organizations interested in evaluating their container images against these metrics can get started in a few simple steps with Docker Scout. The Docker Scout web interface provides a comprehensive dashboard for CISOs, security teams, and software developers, offering an overview of vulnerability trends and policy compliance status (Figure 1). The web interface is a one-stop shop where users can drill down into specific images for deeper investigations and customize out-of-the-box policies to meet their specific needs.

Figure 1: Docker Scout dashboard.

Furthermore, the Docker Scout metrics exporter is a powerful addition to the Docker Scout ecosystem to bring vulnerability and policy compliance metrics into existing monitoring systems. This HTTP endpoint enables users to configure Prometheus-compatible tools to scrape Docker Scout data, allowing organizations to integrate with popular observability tools like Grafana and Datadog to achieve centralized security observability. 

Figures 2 and 3 show two sample Grafana dashboards illustrating the vulnerability trends and policy compliance insights that Docker Scout can provide.

Figure 2: Grafana Dashboard — Policy compliance.

Figure 2 displays a dashboard that illustrates the compliance posture for each policy configured within a Docker Scout organization. This visualization shows the proportion of images in a stream that complies with the defined policies. At the top of the dashboard, you can see the current compliance rate for each policy, while the bottom section shows compliance trends over the past 30 days.

Figure 3 shows a second Grafana dashboard illustrating the number of vulnerabilities by severity over time within a given stream. In this example, you can see notable spikes across all vulnerabilities, indicating the need for deeper investigation and prioritizing remediation.

Figure 3: Grafana Dashboard — Vulnerabilities by severity trends.

Conclusion

Docker Scout metrics exporter is designed to help security engineers improve containerized application security posture in an operationally efficient way. To get started, follow the instructions in the documentation. The instructions will get you up and running with the current public release of metrics exporter. 

Our product team is always open to feedback on social channels such as X and Slack and is looking for ways to evolve the product to align with our customers’ use cases.

Learn more

Visit the Docker Scout product page.

Looking to get up and running? Use our Docker Scout quickstart guide.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Subscribe to the Docker Newsletter.

Quelle: https://blog.docker.com/feed/

New Beta Feature: Deep Dive into GitHub Actions Docker Builds with Docker Desktop

We’re excited to announce the beta release of a new feature for inspecting GitHub Actions builds directly in Docker Desktop 4.31. 

Centralized CI/CD environments, such as GitHub Actions, are popular and useful, giving teams a single location to build, test, and verify their deployments. However, remote processes, such as builds in GitHub Actions, often lack visibility into what’s happening in your Docker builds. This means developers often need additional builds and steps to locate the root cause of issues, making diagnosing and resolving build issues challenging. 

To help, we’re introducing enhancements in GitHub Actions Summary and Docker Desktop to provide a deeper understanding of your Docker builds in GitHub Actions.

Get a high-level view with Docker Build Summary in GitHub Actions 

We now provide Docker Build Summary, a GitHub Actions Summary that displays reports and aggregates build information. The Docker Build Summary offers additional details about your builds in GitHub Actions, including a high-level summary of performance metrics, such as build duration, and cache utilization (Figure 1). Users of docker/build-push-action and docker/bake-action will automatically receive Docker Build Summaries. 

Key benefits

Identify build failures: Immediate access to error details eliminates the need to sift through logs.

Performance metrics: See exactly how long the Docker Build stage took and assess if it met expectations.

Cache utilization: View the percentage of the build that used the cache to identify performance impacts.

Configuration details: Access information on build inputs to understand what ran during build time.

Figure 1: Animated view of Docker Build Summary in GitHub Actions, showing Build details, including Build status, error message, metrics, Build inputs, and more.

If further investigation is needed, we package your build results in a .dockerbuild archive file. This file can be imported to the Build View in Docker Desktop, providing comprehensive build details, including timings, dependencies, logs, and traces.

Import and inspect GitHub Actions Builds in Docker Desktop

Initially announced last year, the Build View in Docker Desktop now supports importing the dockerbuild archive from GitHub Actions, providing greater insight into your Docker builds. 

In Docker Desktop, navigate to the Builds View tab and use the new Import Builds button. Select the .dockerbuild file you downloaded to access all the details about your remote build as if you ran it locally (Figure 2). 

Figure 2: Animated view of Docker Desktop, showing steps to navigate to the Builds View tab and use the new Import Builds button.

You can view in-depth information about your build execution, including error lines in your Dockerfile, build timings, cache utilization, and OpenTelemetry traces. This comprehensive view helps diagnose complex builds efficiently.

For example, you can see the stack trace right next to the Dockerfile command that is causing the issues, which is useful for understanding the exact step and attributes that caused the error (Figure 3).

Figure 3: Inspecting a build error in Builds View.

You can even see the commit and source information for the build and easily locate who made the change for more help in resolving the issue, along with other useful info you need for diagnosing even the most complicated builds (Figure 4). 

Figure 4: Animated view of Docker Desktop showing info for inspecting an imported build, such as source details, build timing, dependencies, configuration, and more.

Enhance team collaboration

We aim to enhance team collaboration, allowing you to share and work together on Docker builds and optimizing the build experience for your team. These .dockerbuild archives are self-contained and don’t expire, making them perfect for team collaboration. Share the .dockerbuild file via Slack or email or attach it to GitHub issues or Jira tickets to preserve context for when your team investigates.

Get started

To start using Docker Build Summary and the .dockerbuild archive in Docker Desktop, update your Docker Build GitHub Actions configuration to:

uses: docker/build-push-action@v6

uses: docker/bake-action@v5

Then, update to Docker Desktop 4.31 to inspect build archives from GitHub Actions. Learn more in the documentation.

We are incredibly excited about these new features, which will help you and your team diagnose and resolve build issues quickly. Please try them out and let us know what you think!

Learn more

Subscribe to the Docker Newsletter.

Get the latest release of Docker Desktop.

Vote on what’s next! Check out our public roadmap.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Quelle: https://blog.docker.com/feed/

LXC vs. Docker: Which One Should You Use?

In today’s evolving software development landscape, containerization technology has emerged as a key tool for developers aiming to enhance efficiency and ensure consistency across environments. Among the various container technologies available today, Linux Containers (LXC) and Docker are two of the most popular choices. Understanding the differences between these technologies is crucial for developers to select the right tool that aligns with their specific project needs. 

This blog post delves into the LXC vs. Docker virtual environments, exploring their functionalities and helping you make an informed decision.

What is LXC?

Linux Containers, or LXC, is an advanced virtualization technology that utilizes key features of the Linux kernel to create lightweight and efficient isolated environments for running multiple applications on a single host system. This technology uses Linux kernel features, such as cgroups (control groups) and namespaces, to manage system resources and provide process isolation.

LXC began as an open source project to provide a virtualization method that operates at the operating system level, using the Linux kernel’s inherent capabilities. The project emerged in the late 2000s — with significant contributions from IBM, among others — and quickly became part of the mainstream Linux kernel. This integration allowed LXC to benefit from native support and optimizations, facilitating its adoption and ongoing development.

LXC has played a pivotal role in the evolution of container technologies. It laid the groundwork for future innovations in containerization, including the development of Docker, which initially relied on LXC as its default execution environment before transitioning to its own container runtime, libcontainer (now part of runc).

Key features of LXC

Resource management with cgroups: LXC manages resource allocation using cgroups, ensuring that each container has access to the resources it needs without impacting others, promoting efficient and stable performance.

Isolation with namespaces: Namespaces ensure that containers are kept isolated from each other, preventing processes in one container from interfering with those in another. This feature enhances security and system reliability.

Benefits of LXC

Lightweight nature: Unlike traditional virtual machines that require separate operating system (OS) instances, LXC containers share the host system’s kernel, making them more resource-efficient and faster to start.

Proximity to the operating system: Thanks to its integration with the Linux kernel, LXC provides functionality similar to that of virtual machines but with a fraction of the resource demand.

Efficient use of system resources: LXC maximizes resource utilization and scalability by enabling multiple containers to run on a single host without the overhead of multiple OS instances.

LXC is especially beneficial for users who need granular control over their environments and applications that require near-native performance. As an open source project, LXC continues to evolve, shaped by a community of developers committed to enhancing its capabilities and integration with the Linux kernel. LXC remains a powerful tool for developers looking for efficient, scalable, and secure containerization solutions.

What are Docker containers?

Docker offers a comprehensive platform and suite of tools that has revolutionized how applications are developed, shipped, and run. It is built upon the concept of containerization, simplifying it to such an extent that it has become synonymous with containers.

Docker, which launched in 2013, initially utilized LXC to provide an easier way to create, deploy, and run applications using containers. Docker’s introduction marked a significant shift in virtualization technology, offering a lighter, faster, and more agile way of handling applications than traditional virtual machines. Docker quickly evolved from using LXC as its default execution environment by developing its own container runtime, libcontainer, which now powers Docker containers.

This move enabled Docker to provide a standardized unit of software deployment, encapsulating applications and their dependencies in containers that could run anywhere, from a developer’s local laptop to a production server in the cloud.

Docker’s ecosystem

Docker DesktopDocker EngineDocker ScoutDocker HubDocker Build CloudKnown for its user-friendly interface, Docker Desktop simplifies tasks in building, running, and managing containers.The core runtime component of Docker shipped in Docker Desktop provides a lightweight and secure environment for running containerized applications.Docker Scout delivers real-time actionable insights, making it simple to secure and manage the software supply chain end-to-end.The world’s largest and most widely used image repository, Docker Hub serves as the go-to container registry for developers to share and manage containerized applications securely.Docker Build Cloud is a premium service that enhances the image-building process in enterprise environments.

These tools collectively form a solution stack that addresses the entire lifecycle of container management, from development to deployment. 

How Docker enhances LXC

Although Docker started with LXC, it added significant value by layering tools and services that enhance user experience and management capabilities. Docker Desktop abstracts much of the complexity of managing containers through user-friendly interfaces and commands, making containerization accessible to a broader range of developers.

Docker containers are lightweight, portable, and self-sufficient units that contain everything needed to run an application. They ensure consistency across multiple development and deployment environments.

Key benefits of Docker containers

Portability: Containers can be moved effortlessly between environments, from development to testing to production, without needing changes, thanks to Docker’s ability to ensure consistency across platforms.

Ease of use: Docker simplifies container management with intuitive commands like docker run, significantly lowering the learning curve for new users.

Vast ecosystem: Docker’s extensive library of container images available on Docker Hub and a wide array of management tools support rapid application development and deployment.

The evolution of Docker from a product that simplified the use of LXC to a comprehensive ecosystem that defines modern containerization practices showcases its transformative impact on the technology landscape. Docker made containers mainstream and established a global community of developers and organizations that continue to innovate on its platform.

Understanding use cases for LXC and Docker

Understanding their strengths and typical use cases is crucial when deciding between LXC and Docker. Both technologies serve the purpose of containerization but cater to different operational needs and user profiles.

LXC use cases

Efficient access to hardware resources: LXC’s close interaction with the host OS allows it to achieve near-native performance, which is beneficial for applications that require intensive computational power or direct hardware access. This can include data-heavy applications in fields like data analysis or video processing where performance is critical.

Virtual Desktop Infrastructure (VDI): LXC is well-suited for VDI setups because it can run full operating systems with a smaller footprint than traditional VMs. This makes LXC ideal for businesses deploying and managing virtual desktops efficiently.

LXC is not typically used for application development but for scenarios requiring full OS functionality or direct hardware integration. Its ability to provide isolated and secure environments with minimal overhead makes it suitable for infrastructure virtualization where traditional VMs might be too resource-intensive.

Docker use cases

Docker excels in environments where deployment speed and configuration simplicity are paramount, making it an ideal choice for modern software development. Key use cases where Docker demonstrates its strengths include:

Streamlined deployment: Docker packages applications into containers along with all their dependencies, ensuring consistent operation across any environment, from development through to production. This eliminates common deployment issues and enhances reliability.

Microservices architecture: Docker supports the development, deployment, and scaling of microservices independently, enhancing application agility and system resilience. Its integration with Kubernetes further streamlines the orchestration of complex containerized applications, managing their deployment and scaling efficiently.

CI/CD pipelines: Docker containers facilitate continuous integration and deployment, allowing developers to automate the testing and deployment processes. This approach reduces manual intervention and accelerates release cycles.

Extensive image repository and configuration management: Docker Hub offers a vast repository of pre-configured Docker images, simplifying application setup. Docker’s configuration management capabilities ensure consistent container setups, easing maintenance and updates.

Docker’s utility in supporting rapid development cycles and complex architectures makes it a valuable tool for developers aiming to improve efficiency and operational consistency in their projects.

Docker vs. LXC: Detailed comparison chart

Feature/AspectDockerLXCCore functionalityApplication containerization; runs apps in isolated containers.OS-level virtualization; runs multiple Linux systems on a host from a single OS.User interfaceHigh-level commands and graphical interface options for simpler management.Lower-level, command-line focused with finer granular control over settings.Ease of useUser-friendly for developers with minimal Linux/container knowledge.Requires more in-depth knowledge of Linux systems and configurations.Setup complexitySimplified setup with pre-built packages and extensive documentation.More complex setup requiring detailed OS configuration knowledge.PerformanceLightweight, with minimal overhead; suitable for microservices.Close to native performance, suitable for intensive computational tasks.SecurityStrong isolation with Docker Engine, support for namespaces, and cgroups.Uses Linux kernel security features, including AppArmor and SELinux profiles.ScalabilityHighly scalable, ideal for applications needing quick scaling.Less scalable compared to Docker; best used for more static, controlled environments.Application use casesIdeal for CI/CD pipelines, microservices, and any container-based applications.Best for running full Linux distributions, VDI, or applications needing direct hardware access.Resource efficiencyHighly efficient in resource usage due to shared OS components.More resource-intensive than Docker but less so than traditional VMs.Community and ecosystemLarge community with a vast ecosystem of tools and resources.Smaller community focused mainly on system administrators and advanced users.Typical deploymentCommon in development environments, cloud platforms, and serverless computing.Used in environments requiring stable, long-term deployments without frequent changes.

Although Docker and LXC are both powerful options for building containers, they serve different purposes and are suitable for different skill levels. Docker is designed for developers who want to quickly and efficiently build and deploy applications in various environments with minimal setup. On the other hand, LXC is more suitable for users who need a lightweight alternative to virtual machines and want more control over the operating system and hardware.

Conclusion

Choosing between Linux Containers vs. Docker depends on your project’s specific needs and operational environment. 

LXC is ideal for scenarios requiring full operating system functionality or extensive hardware interaction, making it suitable for projects needing deep system control or stable, long-term deployments. 

Docker is optimized for developers seeking to enhance application development and deployment efficiency, particularly in dynamic environments that demand rapid scaling and frequent updates. 

Each platform offers unique benefits tailored to different technical requirements and use cases, ensuring the selection aligns with your project goals and infrastructure demands.

Try Docker containers using Docker Desktop today.

Learn more

Get started by exploring Docker containers.

Download the latest version of Docker Desktop. 

Learn more about Linux containers. 

Visit Docker Resources to explore more materials.

Subscribe to the Docker newsletter. 

Quelle: https://blog.docker.com/feed/

Docker Launches 2024 State of Application Development Report

Docker launched its 2024 State of Application Development Report, providing a deep-focus snapshot of today’s rapidly evolving world of software development. Based on a wide-ranging survey of more than 1,300 respondents, the report shares a broad array of findings about respondents’ work, including what tools they use, their processes and frustrations, opinions about industry trends, participation in developer communities, Docker usage, and more.

What emerges is an illuminating picture of the current state of application development, along with insights into key trends such as the expanding roles of cloud and artificial intelligence/machine learning (AI/ML) in software development, the continued rise of microservices, and attitudes toward the shift-left approach to security.

Reflecting the changing state of the industry, the 2024 report drills down into three main areas: 

The state of application development today 

AI’s expanding role in application development

Security in application development

The online survey is a key vector through which Docker product managers, engineers, and designers gather insights from users to continuously develop and improve the company’s suite of tools.

“The findings in this report demonstrate how Docker continuously seeks to address market needs so that we can better empower development teams not just to compete, but to thrive and innovate with the right processes and tools for their workflows,” said Nahid Samsami, Vice President of Developer Experience at Docker.

Read on for key findings from this year’s report and download the full report for more details.

The rise of cloud for software development

A key insight was the growing popularity of developing software in the cloud. When asked about their main development environment, almost 64% of respondents cited their laptop or desktop. But the real story is that more than 36% cited non-local environments, such as ephemeral environments, personal remote dev environments or clusters, and remote development tools such as GitHub Codespaces, Gitpod, and Coder.

These findings appear to underscore the growing popularity of developing software in the cloud — a trend fanned by benefits such as increased efficiency, shorter build times, reduced time to market, and faster innovation. 

Why this apparent increase in reliance on cloud during development? It seems likely that the growing size of applications is a factor, along with the increasing number of dependencies and overall growth in complexity — all of which would render an all-local environment difficult, if not impossible, to maintain in parity with production.

AI/ML goes mainstream in app development

Another key finding was the growing penetration of AI/ML into the software development field. Most respondents (64%) reported already using AI for work — for tasks such as code writing, documentation, and research.

This trend is notably driven by junior/mid-level developers and DevOps/platform engineers, who expressed a higher dependency on AI compared with senior developers surveyed. In terms of AI tools, respondents most often used ChatGPT (46%), GitHub Copilot (30%), and Gemini (formerly Bard) (19%).

This year’s survey showed a growing interest in ML engineering and data science within the Docker community. When asked if they were working on ML in any capacity, almost half of respondents (46%) replied in the affirmative. Within that group, 54% said that they trained and deployed ML models in one or more projects, 43% worked on ML infrastructure, and 39% leveraged pre-trained ML models.

Where developers get stuck

A primary goal of the survey was to gain insights into how Docker can improve the app development experience. When we asked where their team gets stuck in the development process, respondents cited multiple stages, including planning (31%), estimation (24%), and designing (22%). Planning was also one of the most-selected areas in which respondents desired better tools (28% of respondents). 

These findings demonstrate that respondents hit sticking points in project-level tasks before development. However, there are areas identified for improvement within the development process itself, as 20% of respondents reported getting stuck during debugging/troubleshooting or testing phases. Testing was also one of the top areas in which respondents wanted better tools (28%).

Microservices, security, and open source

Other notable trends include the continued rise of microservices, frustration with the shift-left approach to security, and interest in open source. Underscoring the growing popularity of microservices, nearly three times more respondents (29%) said they were transitioning from monolithic to microservices than were moving in the other direction, from microservices to monolithic (11%).

The shift-left approach to security appears to be a source of frustration for developers and an area where more effective tools could make a difference. Security-related tasks topped the list of those deemed difficult/very difficult, with 34% of respondents selecting one of these options. Regarding the need for better tools in the development process, 25% of respondents selected security/vulnerability remediation tools (behind testing, planning, and monitoring/logging/maintenance).

Open source software is important to developer ecosystems and communities, with 59% of respondents saying they contributed to open source in the past year, compared with 41% saying they did not. Of the 41% who did not contribute, a large majority (72%) expressed interest in contributing to open source, while less than 25% did not. 

Get the full report

The 2024 Docker State of Application Development Report is based on an online, 20-minute survey conducted by Docker’s User Research Team in the fall of 2023. Survey respondents ranged from home hobbyists to professionals at companies with more than 5,000 employees. The findings are based on 885 completed responses from the roughly 1,300 respondents surveyed. 

The survey was developed to inform Docker’s product strategy. Given the fascinating information we discovered, we wanted to share the findings with the community. This was the second annual Docker State of Application Development survey; the third will take place in the fall of 2024. 

Download the full report now. 

Learn more

Read the AI Trends Report 2024: AI’s Growing Role in Software Development.

Subscribe to the Docker Newsletter.

Get the latest release of Docker Desktop.

Vote on what’s next! Check out our public roadmap.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Docker’s User Research Team — Olga Diachkova, Julia Wilson, and Rebecca Floyd — conducted this survey, analyzed the results, and provided insights.

For a complete methodology, contact uxresearch@docker.com.
Quelle: https://blog.docker.com/feed/

Docker Desktop 4.31: Air-Gapped Containers, Accelerated Builds, and Beta Releases of Docker Desktop on Windows on Arm, Compose File Viewer, and GitHub Actions

In this post:

Air-gapped containers: Ensuring security and compliance

Accelerating Builds in Docker Desktop with Docker Build Cloud

Docker Desktop on Windows on Arm (Beta)

Compose File Viewer (Beta)

Enhanced CI visibility with GitHub Actions in Docker Desktop (Beta)

Docker Desktop’s latest release continues to empower development teams of every size, providing a secure hybrid development launchpad that supports productively building, sharing, and running innovative applications anywhere. 

Highlights from the Docker Desktop 4.31 release include: 

Air-gapped containers help secure developer environments and apps to ensure peace of mind. 

Accelerating Builds in Docker Desktop with Docker Build Cloud helps developers build rapidly to increase productivity and ROI.

Docker Desktop on Windows on Arm (WoA) Beta continues our commitment to supporting the Microsoft Developer ecosystem by leveraging the newest and most advanced development environments.

Compose File Viewer (Beta) see your Compose configuration with contextual docs.

Enhanced CI visibility with GitHub Actions in Docker Desktop (Beta) that streamline accessing detailed GitHub Actions build summaries, including performance metrics and error reports, directly within the Docker Desktop UI.

Air-gapped containers: Ensuring security and compliance

For our business users, we introduce support for air-gapped containers. This feature allows admins to configure Docker Desktop to restrict containers from accessing the external network (internet) while enabling access to the internal network (private network). Docker Desktop can apply a custom set of proxy rules to network traffic from containers. The proxy can be configured to allow network connections, reject network connections, and tunnel through an HTTP or SOCKS proxy (Figure 1). This enhances security by allowing admins to choose which outgoing TCP ports the policy applies to and whether to forward a single HTTP or SOCKS proxy, or to implement policy per destination via a PAC file.

Figure 1: Assuming enforced sign-in and Settings Management are enabled, add the new proxy configuration to the admin-settings.json file.

This functionality enables you to scale securely and is especially crucial for organizations with strict security requirements. Learn more about air-gapped containers on our Docker Docs.  

Accelerating Builds in Docker Desktop with Docker Build Cloud 

Did you know that in your Core Docker Subscription (Personal, Pro, Teams, Business) you have an included allocation of Docker Build Cloud minutes? Yes! This allocation of cloud compute time and shared cache lets you speed up your build times when you’re working with multi-container apps or large repos. 

For organizations, your build minutes are shared across your team, so anyone allocated Docker Build Cloud minutes with their Docker Desktop Teams or Business subscription can leverage available minutes and purchase additional minutes if necessary. Docker Build Cloud works for both developers building locally and in CI/CD.With Docker Desktop, you can use these minutes to accelerate your time to push and gain access to the Docker Build Cloud dashboard (build.docker.com)  where you can view build history, manage users, and view your usage stats. 

And now, from build.docker.com, you can quickly and easily create your team’s cloud builder using a one-click setup that connects your cloud builder to Docker Desktop. At the same time, you can choose to configure the Build Cloud builder as the default builder in Docker Desktop in about 30 seconds — check the Set the default builder radio button during the Connect via Docker Desktop setup (Figure 2).

Figure 2: Setting the default builder in Docker Desktop.

Docker Desktop on Windows on Arm

At Microsoft Build, we were thrilled to announce that Docker Desktop is available on Windows on Arm (WoA) as a beta release. This version will be available behind authentication and is aimed at users with Arm-based Windows devices. This feature ensures that developers using these devices can take full advantage of Docker’s capabilities. 

To learn more about leveraging WoA to accelerate your development practices, watch the Microsoft Build Session Introducing the Next Generation of Windows on Arm with Ivette Carreras and Jamshed Damkewala. You can also learn about the other better-together opportunities between Microsoft and Docker by visiting our Microsoft Build Docker Page and reading our event highlights blog post. 

Compose File Viewer (Beta)

With Compose File Viewer (Beta), developers can now see their Docker Compose configuration file in Docker Desktop, with relevant docs linked. This makes it easier to understand your Compose YAML at a glance, with proper syntax highlighting. 

Check out this new File Viewer through the View Configuration option in the Compose command line or by viewing a Compose stack in the Containers tab, then clicking the View Configuration button.

Introducing enhanced CI visibility with GitHub Actions in Docker Desktop

We’re happy to announce the beta release of our new feature for inspecting GitHub Actions builds directly in Docker Desktop. This enhancement provides in-depth summaries of Docker builds, including performance metrics, cache utilization, and detailed error reports. You can download build results as a .dockerbuild archive and inspect them locally using Docker Desktop 4.31. Now you can access all the details about your CI build as if you had reproduced them locally. 

Figure 3: Docker Desktop 4.31 Builds tab supporting one-click importing of builds from GitHub Actions.

Not familiar with the Builds View in Docker Desktop? It’s a feature we introduced last year to give you greater insight into your local Docker builds. Now, with the import functionality, you can explore the details of your remote builds from GitHub Actions just as thoroughly in a fraction of the time. This new capability aims to improve CI/CD efficiency and collaboration by offering greater visibility into your builds. Update to Docker Desktop 4.31 and configure your GitHub Actions with docker/build-push-action@v5  or docker/bake-action@v4 to get started.

Conclusion 

With this latest release, we’re doubling down on our mission to support Docker Desktop users with the ability to accelerate innovation, enable security at scale, and enhance productivity. 

Stay tuned for additional details and upcoming releases. Thank you for being part of our community as we continuously strive to empower development teams. 

Learn more

Authenticate and update to receive your subscription level’s newest Docker Desktop features.

New to Docker? Create an account. 

Visit our Microsoft Build Docker Page to learn about our partnership in supporting Microsoft developers.

Learn how Docker Build Cloud in Docker Desktop can accelerate builds.

Secure Your Supply Chain with Docker Scout in Docker Desktop.

Learn more about air-gapped containers.

Subscribe to the Docker Newsletter.

Quelle: https://blog.docker.com/feed/

10 Years Since Kubernetes Launched at DockerCon

It is not often you can reflect back and pinpoint a moment where an entire industry changed, less often to pinpoint that moment and know you were there to see it first hand.

On June 10th, 2014, day 2 of the first ever DockerCon, 16:04 seconds into his keynote speech, Google VP of Infrastructure Eric Brewer announced that Google was releasing the open source solution they built for orchestrating containers: Kubernetes. This was one of those moments. The announcement of Kubernetes began a tectonic shift in how the internet runs at scale, so many of the most important applications in the world today would not be possible without Docker and Kubernetes.

You can watch the announcement on YouTube.

We didn’t know how much Kubernetes would change things at that time. In fact, in those two days, Apache Mesos, Red Hat’s GearD, Docker Libswarm, and Facebook’s Tupperware were all also launched. This triggered what later became known by some as “the Container Orchestration War.” Fast forward three years and the community had consolidated on Kubernetes for the orchestration layer and Docker (powered by containerd) for the container format, distribution protocol, and runtime. In 2017,  Docker integrated Kubernetes in its desktop and server products, and this helped cement Kubernetes leadership.

Why was it so impactful? Kubernetes landed at just the right time and solved just the right problems. The number of containers and server nodes in production was increasing exponentially every day. The role of DevOps put a lot of burden on the engineer. They needed solutions that could help manage applications at unprecedented scale. Containers and their orchestration engines were, and continue to be, the lifeblood of modern application deployments because they are the only real way to solve this need.

We, the Docker team and community, consider ourselves incredibly fortunate to have played a role in this history. To look back and say we had a part in what has been built from that one moment is humbling.

… and the potential of what is yet to come is beyond exciting! Especially knowing that our impact continues today as a keystone to modern application development. Docker enables app development teams to rapidly deliver applications, secure their software supply chains, and do so without compromising the visibility and controls required by the business.

Happy 10th birthday Kubernetes! Congratulations to all who were and continue to be involved in creating this tremendous gift to the software industry.

Learn more

Join the conversation on LinkedIn.

Build Kubernetes-ready applications on your desktop with Docker.

Get the latest Docker news.

Quelle: https://blog.docker.com/feed/

Develop Kubernetes Operators in Java without Breaking a Sweat

Developing Kubernetes operators in Java is not yet the norm. So far, Go has been the language of choice here, not least because of its excellent support for writing corresponding tests. 

One challenge in developing Java-based projects has been the lack of easy automated integration testing that interacts with a Kubernetes API server. However, thanks to the open source library Kindcontainer, based on the widely used Testcontainers integration test library, this gap can be bridged, enabling easier development of Java-based Kubernetes projects. 

In this article, we’ll show how to use Testcontainers to test custom Kubernetes controllers and operators implemented in Java.

Kubernetes in Docker

Testcontainers allows starting arbitrary infrastructure components and processes running in Docker containers from tests running within a Java virtual machine (JVM). The framework takes care of binding the lifecycle and cleanup of Docker containers to the test execution. Even if the JVM is terminated abruptly during debugging, for example, it ensures that the started Docker containers are also stopped and removed. In addition to a generic class for any Docker image, Testcontainers offers specialized implementations in the form of subclasses — for components with sophisticated configuration options, for example. 

These specialized implementations can also be provided by third-party libraries. The open source project Kindcontainer is one such third-party library that provides specialized implementations for various Kubernetes containers based on Testcontainers:

ApiServerContainer

K3sContainer

KindContainer

Although ApiServerContainer focuses on providing only a small part of the Kubernetes control plane, namely the Kubernetes API server, K3sContainer and KindContainer launch complete single-node Kubernetes clusters in Docker containers. 

This allows for a trade-off depending on the requirements of the respective tests: If only interaction with the API server is necessary for testing, then the significantly faster-starting ApiServerContainer is usually sufficient. However, if testing complex interactions with other components of the Kubernetes control plane or even other operators is in the scope, then the two “larger” implementations provide the necessary tools for that — albeit at the expense of startup time. For perspective, depending on the hardware configuration, startup times can reach a minute or more.

A first example

To illustrate how straightforward testing against a Kubernetes container can be, let’s look at an example using JUnit 5:

@Testcontainers
public class SomeApiServerTest {
@Container
public ApiServerContainer<?> K8S = new ApiServerContainer<>();

@Test
public void verify_no_node_is_present() {
Config kubeconfig = Config.fromKubeconfig(K8S.getKubeconfig());
try (KubernetesClient client = new KubernetesClientBuilder().withConfig(kubeconfig).build()) {
// Verify that ApiServerContainer has no nodes
assertTrue(client.nodes().list().getItems().isEmpty());
}
}
}

Thanks to the @Testcontainers JUnit 5 extension, lifecycle management of the ApiServerContainer is easily handled by marking the container that should be managed with the @Container annotation. Once the container is started, a YAML document containing the necessary details to establish a connection with the API server can be retrieved via the getKubeconfig() method. 

This YAML document represents the standard way of presenting connection information in the Kubernetes world. The fabric8 Kubernetes client used in the example can be configured using Config.fromKubeconfig(). Any other Kubernetes client library will offer similar interfaces. Kindcontainer does not impose any specific requirements in this regard.

All three container implementations rely on a common API. Therefore, if it becomes clear at a later stage of development that one of the heavier implementations is necessary for a test, you can simply switch to it without any further code changes — the already implemented test code can remain unchanged.

Customizing your Testcontainers

In many situations, after the Kubernetes container has started, a lot of preparatory work needs to be done before the actual test case can begin. For an operator, for example, the API server must first be made aware of a Custom Resource Definition (CRD), or another controller must be installed via a Helm chart. What may sound complicated at first is made simple by Kindcontainer along with intuitively usable Fluent APIs for the command-line tools kubectl and helm.

The following listing shows how a CRD is first applied from the test’s classpath using kubectl, followed by the installation of a Helm chart:

@Testcontainers
public class FluentApiTest {
@Container
public static final K3sContainer<?> K3S = new K3sContainer<>()
.withKubectl(kubectl -> {
kubectl.apply.fileFromClasspath(“manifests/mycrd.yaml”).run();
})
.withHelm3(helm -> {
helm.repo.add.run(“repo”, “https://repo.example.com”);
helm.repo.update.run();
helm.install.run(“release”, “repo/chart”);
);
// Tests go here
}

Kindcontainer ensures that all commands are executed before the first test starts. If there are dependencies between the commands, they can be easily resolved; Kindcontainer guarantees that they are executed in the order they are specified.

The Fluent API is translated into calls to the respective command-line tools. These are executed in separate containers, which are automatically started with the necessary connection details and connected to the Kubernetes container via the Docker internal network. This approach avoids dependencies on the Kubernetes image and version conflicts regarding the available tooling within it.

Selecting your Kubernetes version

If nothing else is specified by the developer, Kindcontainer starts the latest supported Kubernetes version by default. However, this approach is generally discouraged, so the best practice would require you to explicitly specify one of the supported versions when creating the container, as shown:

@Testcontainers
public class SpecificVersionTest {
@Container
KindContainer<?> container=new KindContainer<>(KindContainerVersion.VERSION_1_24_1);
// Tests go here
}

Each of the three container implementations has its own Enum, through which one of the supported Kubernetes versions can be selected. The test suite of the Kindcontainer project itself ensures — with the help of an elaborate matrix-based integration test setup — that the full feature set can be easily utilized for each of these versions. This elaborate testing process is necessary because the Kubernetes ecosystem evolves rapidly, and different initialization steps need to be performed depending on the Kubernetes version.

Generally, the project places great emphasis on supporting all currently maintained Kubernetes major versions, which are released every 4 months. Older Kubernetes versions are marked as @Deprecated and eventually removed when supporting them in Kindcontainer becomes too burdensome. However, this should only happen at a time when using the respective Kubernetes version is no longer recommended.

Bring your own Docker registry

Accessing Docker images from public sources is often not straightforward, especially in corporate environments that rely on an internal Docker registry with manual or automated auditing. Kindcontainer allows developers to specify their own coordinates for the Docker images used for this purpose. However, because Kindcontainer still needs to know which Kubernetes version is being used due to potentially different initialization steps, these custom coordinates are appended to the respective Enum value:

@Testcontainers
public class CustomKubernetesImageTest {
@Container
KindContainer<?> container=new KindContainer<>(KindContainerVersion.VERSION_1_24_1.withImage(
“my-registry/kind:1.24.1”));
// Tests go here
}

In addition to the Kubernetes images themselves, Kindcontainer also uses several other Docker images. As already explained, command-line tools such as kubectl and helm are executed in their own containers. Appropriately, the Docker images required for these tools are configurable as well. Fortunately, no version-dependent code paths are needed for their execution. 

Therefore, the configuration shown in the following is simpler than in the case of the Kubernetes image:

@Testcontainers
public class CustomFluentApiImageTest {
@Container
KindContainer<?> container=new KindContainer<>()
.withKubectlImage(DockerImageName.parse(“my-registry/kubectl:1.21.9-debian-10-r10”))
.withHelm3Image(DockerImageName.parse(“my-registry/helm:3.7.2”));
// Tests go here
}

The coordinates of the images for all other containers started can also be easily chosen manually. However, it is always the developer’s responsibility to ensure the use of the same or at least compatible images. For this purpose, a complete list of the Docker images used and their versions can be found in the documentation of Kindcontainer on GitHub.

Admission controller webhooks

For the test scenarios shown so far, the communication direction is clear: A Kubernetes client running in the JVM accesses the locally or remotely running Kubernetes container over the network to communicate with the API server running inside it. Docker makes this standard case incredibly straightforward: A port is opened on the Docker container for the API server, making it accessible. 

Kindcontainer automatically performs the necessary configuration steps for this process and provides suitable connection information as Kubeconfig for the respective network configuration.

However, admission controller webhooks present a technically more challenging testing scenario. For these, the API server must be able to communicate with external webhooks via HTTPS when processing manifests. In our case, these webhooks typically run in the JVM where the test logic is executed. However, they may not be easily accessible from the Docker container.

To facilitate testing of these webhooks independently of the network setup, yet still make it simple, Kindcontainer employs a trick. In addition to the Kubernetes container itself, two more containers are started. An SSH server provides the ability to establish a tunnel from the test JVM into the Kubernetes container and set up reverse port forwarding, allowing the API server to communicate back to the JVM. 

Because Kubernetes requires TLS-secured communication with webhooks, an Nginx container is also started to handle TLS termination for the webhooks. Kindcontainer manages the administration of the required certificate material for this. 

The entire setup of processes, containers, and their network communication is illustrated in Figure 1.

Figure 1: Network setup for testing webhooks.

Fortunately, Kindcontainer hides this complexity behind an easy-to-use API:

@Testcontainers
public class WebhookTest {
@Container
ApiServerContainer<?> container=new ApiServerContainer<>().withAdmissionController(admission -> {
admission.mutating()
.withNewWebhook("mutating.example.com")
.atPort(webhookPort) // Local port of webhook
.withNewRule()
.withApiGroups("")
.withApiVersions("v1")
.withOperations("CREATE", "UPDATE")
.withResources("configmaps")
.withScope("Namespaced")
.endRule()
.endWebhook()
.build();
});

// Tests go here
}

The developer only needs to provide the port of the locally running webhook along with some necessary information for setting up in Kubernetes. Kindcontainer then automatically handles the configuration of SSH tunneling, TLS termination, and Kubernetes.

Consider Java

Starting from the simple example of a minimal JUnit test, we have shown how to test custom Kubernetes controllers and operators implemented in Java. We have explained how to use familiar command-line tools from the ecosystem with the help of Fluent APIs and how to easily execute integration tests even in restricted network environments. Finally, we have shown how even the technically challenging use case of testing admission controller webhooks can be implemented simply and conveniently with Kindcontainer. 

Thanks to these new testing possibilities, we hope more developers will consider Java as the language of choice for their Kubernetes-related projects in the future.

Learn more

Visit the Testcontainers website.

Get started with Testcontainers Cloud by creating a free account.

Get the latest release of Docker Desktop.

Vote on what’s next! Check out our public roadmap.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Subscribe to the Docker Newsletter.

Quelle: https://blog.docker.com/feed/