ChainGPT: Exploring open source projects with LLM agents

TLDR

We designed a LLM agent, ChainGPT, to accelerate exploring and understanding open source projects. Our agent can browse source code and execute commands within GitHub according to user instructions. 

Introduction

Wolfi is the first Linux (un)distro designed for securing the software supply chain in the cloud-native era. Combining the minimalism of Alpine and the versatility of GNU’s toolchain, Wolfi enables developers to easily create container images with low-to-zero known CVEs. Currently, Wolfi is small, with a few thousand essential, popular packages. To grow this number Chainguard Labs is seeking ways to automate our discovery and exploration of new packages for Wolfi.

A description of creating Wolfi packages might include adjectives like “hard,” ”tedious,” and “manual.” Late last year, Chainguard therefore investigated a few moonshot ideas for automating the process and found that large language models (LLM) show some potential. Specifically, our latest GPT-4-powered application successfully built near-accurate package files for a handful of simple go packages. A solution for packaging more complex projects spanning several languages and software standards is likely still a ways off. Today, we wanted to share this exploratory research and release the research tool we developed: ChainGPT. 

ChainGPT is a GPT-4-powered chatbot that can be connected to your GitHub repository of choice. We empowered ChainGPT with the same skills we used in our packaging research, allowing the application to service many useful requests. In this post we will describe the skills granted to ChainGPT and the agent framework we leverage to orchestrate the usage of these skills to interact with open source projects.

ChainGPT’s skills

Skill #1 — Directory list: 

ChainGPT has the ability to list directories, giving it knowledge of the repository’s structure. Useful for tasks such as:

• “What languages does the project use?”

• “How is the source code organized?”

• “Where is the documentation for XYZ feature?”

Skill #2 — File read: 

ChainGPT has the ability to read files (albeit slowly for large ones). Useful for tasks such as:

• “Describe the project using the README.”

• “How do I build the project from source?”

• “How does the Makefile work?”

Skill #3 — Shell code execution (scary): 

We gave ChainGPT the powerful ability to execute shell code in an isolated Wolfi container. Useful for tasks such as:

• “Run the test cases.”

• “Test script XYZ.”

• “Build the project from source”.

You might already be able to tell how we leveraged these skills in this research. To build a package, we asked our application to explore the repository with directory listing, use file read to read documentation and code about the build procedure, test build commands with shell execution, and finally return a Melange YAML file formatted according to our few-shot prompt. This task is merely a subset of how these skills can be used. Let’s dive into how ChainGPT works.

The LLM behind the curtain

ChainGPT is a LangChain application built on top of OpenAI’s GPT-4 API. There are many categories of LLM applications including chatbots and retrieval augmented generation (RAG).

ChainGPT falls under the LLM agent category (with a sprinkle of chatbot). Unlike the pure chatbots and RAG architectures, LLM agents use the underlying model to form a plan for completing the task. This plan may include calling tools provided to the model whose outputs may adjust the plan.

ChainGPT defines four tools to GPT-4 using OpenAI’s Functions API, one tool for directory listing, one for file read, and two for shell code execution. When you supply a prompt to ChainGPT, a request is sent to GPT-4 along with definitions for each tool. The LLM replies with its plan and the tools it wants to call. These tools are executed locally and the results are returned to GPT-4. This process repeats until GPT-4 reports that the task is complete. The figure below visualizes the exchange between ChainGPT and GPT-4 when the user asks about the Grype project.

We aim for ChainGPT to serve as a platform to further explore LLMs and assist in securing open-source software; automating package creation is where we started and saw the greatest need, but our work won’t stop there as we will continue to leverage cutting-edge technologies that will increase productivity and speed up product innovation for Chainguard Images. Check out the repository and clone our ChainGPT tool here. Join us in welcoming the age of LLMs to open-source software security!

If you're interested in learning more about how Chainguard can help with your AI/ML toolchains, look no further than our Chainguard AI Images, which aim to harden this burgeoning aspect of the software supply chain for developers and data scientists.