🛠️ DevOps Support Agent with Human in the Loop

Notebook by Amna Mubashar

This notebook demonstrates how a Haystack Agent can interactively ask for user input when it's uncertain about the next step. We'll build a Human-in-the-Loop tool that the Agent can call dynamically. When the Agent encounters ambiguity or incomplete information, it will ask for more input from the human to continue solving the task.

For this purpose, we will create a DevOps Support Agent.

CI/CD pipelines occasionally fail for reasons that can be hard to diagnose including manually—broken tests, mis-configured environment variables, flaky integrations, etc.

The support Agent is created using Haystack Tools and Agent and will perform the following tasks:

• Check for any failed CI workflows
• Fetch build logs and status
• Lookup and analyze git commits
• Suggest next troubleshooting steps
• Escalate to humans via Human-in-Loop tool

Install the required dependencies

Next, we configure our variables. We need to set the OPENAI_API_KEY for OpenAIChatGenerator and a GITHUB_TOKEN with appropriate permissions to access repository information, CI logs, and commit history. Make sure your GitHub token has sufficient access rights to the repository you want to analyze.

Define Tools

We start by defining the tools which will be used by our Agent.

git_list_commits_tool: Fetches and formats the most recent commits on a given GitHub repository and branch.
Useful for quickly surfacing the latest changes when diagnosing CI failures or code regressions.

git_diff_tool: Retrieves the patch (diff) between two commits or branches in a GitHub repository.
Enables side-by-side inspection of code changes that may have triggered test failures.

ci_status_tool: Checks the most recent GitHub Actions workflow runs for failures, downloads their logs, and extracts the first failed test name and error message. Automates root-cause identification by surfacing the exact test and error that caused a pipeline to fail.

shell_tool: Executes a local shell command with a configurable timeout, capturing stdout or returning detailed error output. Great for grepping, filtering, or tailing CI log files before passing only the relevant snippets to the LLM.

human_in_loop_tool: Prompts the user with a clarifying question via input() when the Agent encounters ambiguity or needs additional information. Ensures the Agent only interrupts for human feedback when strictly necessary.

Create and Configure the Agent

Create a Haystack Agent instance and configure its behavior with a system prompt and tools.

Agent Introspection with Streaming

Haystack supports streaming of ToolCalls and ToolResults during Agent execution. By passing a streaming_callback to the Agent, you can observe the internal reasoning process in real time.

This allows you to:

• See which tools the Agent decides to use.
• Inspect the arguments passed to each tool.
• View the results returned from each tool before the final answer is generated.

This is especially useful for debugging and transparency in complex multi-step reasoning workflows of Agent.

Note: You can pass any streaming_callback function. The print_streaming_chunk utility function in Haystack is configured to stream tool calls and results by default.