Lesson 4: Call transcript summarizer

In this lesson, we're going to write a complex prompt for a common customer use-case: summarizing. Specifically, we'll summarize long customer service call transcripts. Our goal is to summarize customer service calls for customer support metrics. We want summaries of complete customer service calls to evaluate the efficacy of our customer support team. This means we'll exclude calls that have connection issues, language barriers, and other issues that hinder effective summarization.

Let's imagine we work for Acme Corporation, a company that sells smart home devices. The company handles hundreds of customer service calls daily and needs a way to quickly turn these conversations into useful, structured data.

Some important considerations include:

• Calls can be short and sweet or long and complicated.
• Customers might be calling about anything from a simple Wi-Fi connection issue to a complex system malfunction.
• We need our summaries in a specific format so they're easy to analyze later.
• We have to be careful not to include any personal customer information in our summaries.

To help us out, we'll follow the best practices we described previously:

• Use a system prompt to set the stage.
• Structure the prompt for optimal performance.
• Give clear instructions and define your desired output.
• Use XML tags to organize information.
• Handle special cases and edge scenarios.
• Provide examples to guide the model.

Understanding the data

Now that we understand our task, let's take a look at the data we'll be working with. In this lesson, we'll use a variety of simulated customer service call transcripts from Acme Corporation's smart home device support team. These transcripts will help us create a robust prompt that can handle different scenarios.

Let's examine some of the types of call transcripts we might encounter:

A short and simple transcript:

A medium-length transcript with an eventual resolution:

A longer call with no resolution:

These examples showcase the variety of calls and considerations we need to handle:

• Calls have wildly different lengths.
• Calls feature various support issues (simple fixes, device malfunctions, complex problems).
• Some calls end with a resolution and others remain unresolved cases.
• Some calls require follow-up.

As we build our prompt, we'll need to ensure it can effectively summarize all these types of calls, extracting the key information and presenting it in a consistent, structured format. In the next section, we'll start building our prompt, step by step, to handle this diverse range of call transcripts.

A simple version of the prompt

Now that we understand our task and the kind of data we're working with, let's start building our prompt. We'll begin with a basic version and gradually refine it to handle the complexities of our call summarization task.

Let's begin with this very simple prompt that outlines the basic task:

This basic prompt gives Claude a general idea of what we want, but it has several limitations:

• It doesn't specify the desired output format, which could lead to inconsistent summaries.
• It doesn't provide guidance on how to handle different scenarios (like unresolved issues or insufficient information).
• It doesn't set any constraints on length or content, potentially resulting in overly long or detailed summaries.
• It doesn't instruct Claude to omit personal information, which could lead to privacy issues.

With that said, let's test it out to get a sense of how it performs:

As you can see, while Claude does provide a summary, it's not in a format that would be easy to analyze systematically. The summary might be too long or too short, and it might not consistently cover all the points we're interested in.

In the next steps, we'll start adding more structure and guidance to our prompt to address these limitations. We'll see how each addition improves the quality and consistency of Claude's summaries.

Remember, prompt engineering is an iterative process. We start simple and gradually refine our prompt.

Adding a system prompt

The easiest place to start is with a system prompt that sets the overall context and role for Claude, helping to guide its behavior throughout the interaction.

Let's start with this system prompt:

Structuring our main prompt

Next, we're going to start writing the main prompt. We'll rely on some of these prompting tips:

• Put long documents (our transcripts) at the top.
• Add detailed instructions and output format requirements.
• Introduce XML tags for structuring the prompt and output.
• Give Claude space "to think out loud".

Because this prompt may get quite long, we'll write individual pieces in isolation and then combine them together.

The input data

When working with large language models like Claude, it's crucial to put long documents, like our call transcripts, at the beginning of the prompt. This ensures that Claude has all the necessary context before receiving specific instructions. We should also use XML tags to identify the transcript in the prompt:

Instructions and output format

Before we go any further, let's think clearly about what a good structured output format might look like. To make our life easier when parsing the results, it's often easiest to ask Claude for a JSON response. What should a good JSON look like in this case?

At a minimum, our JSON output should include the following:

• A status as to whether Claude had enough information to generate a summary. We'll come back to this. For now, we'll assume that all summaries have a status of "COMPLETE" meaning that Claude could generate a summary.
• A summary of the customer issue
• If the call requires additional follow up
• Details on any follow up actions, if required (call the customer back, etc.)
• How the issue was resolved
• A list of ambiguities or vague points in the conversation

Here's a proposed sample JSON structure:

{
  "summary": {
    "customerIssue": "Brief description of the main problem or reason for the call",
    "resolution": "How the issue was addressed or resolved, if applicable",
    "followUpRequired": true/false,
    "followUpDetails": "Description of any necessary follow-up actions, or null if none required"
  },
  "status": "COMPLETE",
  "ambiguities": ["List of any unclear or vague points in the conversation, or an empty array if none"]
}

Let's create a new piece of our prompt that includes specific instructions, including:

• Create a summary focusing on the main issue, resolution, and any follow-up actions required.
• Generate a JSON output following our specific, standardized format.
• Omit specific customer information in the summaries.
• Keep each piece of the summary short.

Here's an attempt at providing the output instructions, including our specific output JSON format:

Using XML tags and giving Claude room to think

Next, we'll employ two more prompting strategies: giving Claude room to think and using XML tags.

• We'll ask Claude to start by outputting <thinking> tags that contain its analysis.
• Then, we'll ask Claude to output its JSON output inside of <json>.

Here's the final piece of our first draft prompt:

By asking Claude to put its analysis within <thinking> tags, we're prompting it to break down its thought process before formulating the final JSON output. This encourages a more thorough and structured approach to analyzing the transcript. The <thinking> section allows us (and potentially other reviewers or systems) to see Claude's reasoning process. This transparency can be crucial for debugging and quality assurance purposes.

By separating the analysis (<thinking>) from the structured output (<json>), we create a clear distinction between Claude's interpretation of the transcript and its formatted summary. This can be helpful in cases where we might want to review the analysis separately from the JSON output, but also by isolating the JSON content inside of <json> tags, we make it easy to parse the final response and capture the JSON we want to work with.

Testing our updated prompt

Here's the complete version of the prompt, constructed by combining the individual prompt pieces we've written so far:

Here's a function we can use to test our prompt:

Let's test out the prompt using some of the call transcripts we previously defined:

Those responses all look great! Let's try another call transcript that has a bit of ambiguity to it to see if the JSON result includes those ambiguities:

Great! Everything seems to be working as intended

Edge cases

So far, all of the call transcripts we've tried have been relatively straightforward customer service calls. In the real world, we would expect to also encounter transcripts that perhaps we don't want to summarize, including:

• Calls with connection issues
• Calls with language barriers
• Calls with garbled transcripts
• Calls with irrational or upset customers

Remember, our goal is to summarize these calls to help gauge the effectiveness of the customer service we offer. If we include these edge-case calls in the summaries, we'll likely get skewed results.

Let's see what happens with some of these edge cases with our current prompt. Below we've defined some new call transcripts:

Let's run these edge-case transcripts through our prompt and see what sort of results we get:

Unfortunately, we're getting full summaries for these edge-case transcripts. Here are some problematic parts of the responses:

"customerIssue": "Customer spoke in a language the agent did not understand (Spanish)."

"customerIssue": "Unclear due to garbled voice from the customer"

"customerIssue": "The customer dialed the wrong number for technical support"

Remember that our goal is to summarize our customer service calls to get some insight into how effective our customer service team is. These edge-case transcripts are resulting in complete summaries that will cause problems when analyzing all the summaries. We'll need to decide on a strategy for handling these calls.

Further prompt improvements

As we previously saw, our prompt is currently generating full summaries for edge-case transcripts. We want to change this behavior. We have a couple of options for how we handle these edge-cases:

• Flag them in some way to indicate they are not summarizable, allowing for later human-review.
• Categorize them separately (e.g., "technical difficulty,"  "language barrier,"  etc.).

For simplicity's sake, we'll opt to flag these edge-case calls by asking the model to output JSON that looks like this:

{
  "status": "INSUFFICIENT_DATA"
}

In order to make this work, we'll need to update our prompt in the following ways:

• Add instructions explaining the desired "INSUFFICIENT_DATA" output
• Add examples to show summarizable and non-summarizable transcripts along with their corresponding JSON outputs.

Updating our instructions

Let's write a new part of the instructions portion of the prompt to explain when the model should output our "INSUFFICIENT_DATA" JSON.

Adding examples

As we discussed previously in this course, it's almost always a good idea to add examples to a prompt. In this specific use case, examples will help Claude generally understand the types of summaries we want for both summarizable and non-summarizable call transcripts.

Here's a set of examples we could include in our prompt:

Note that the examples cover three different situations:

• A complete interaction that does not require follow up
• A complete interaction that does require follow up and contains ambiguities
• A non-summarizable interaction that contains insufficient data

When providing examples to Claude, it's important to cover a variety of input/output pairs.

Our final prompt

Let's combine our initial prompt with the additions we made in the previous section:

• the instructions on handling calls with insufficient data
• the set of example inputs and outputs

This is the new complete prompt:

The above prompt is quite long, but here is the general structure:

• The system prompt sets the context, role, and tone for the model.
• The main prompt includes the following:
  • the call transcript
  • a set of instructions containing:
    • general instructions
    • guidelines
    • output format requirements
    • details on handling edge-case calls
    • examples
  • details on the XML tags to use in the output

Here's a summary to help visualize the flow of the prompt:

Analyze the following customer service call transcript and generate a JSON summary of the interaction:

<transcript>
[INSERT CALL TRANSCRIPT HERE]
</transcript>

<instructions>
- General instructions and guidelines
- Output JSON format description
- Insufficient data (edge-case) criteria
<examples>
varied example inputs and outputs
</examples>
</instructions>

Before generating the JSON, please analyze the transcript in <thinking> tags. 
Include your identification of the main issue, resolution, follow-up requirements, and any ambiguities. 
Then, provide your JSON output in <json> tags.

Let's test the final prompt with a new function. Note that this function extracts the JSON summary content inside the <json> tags:

Let's test it out with a bunch of our existing call variables:

Let's try our call transcript that should result in a summary with a non-empty ambiguities array:

Now let's try some of our edge case prompts that we do not want summarized:

Great! We're getting the exact outputs we want! Let's try pushing it even further:

Excellent, the prompt is handling all of our edge cases!

Wrap up

In this lesson, we walked through the process of developing a complex prompt for summarizing customer service call transcripts. Let's recap the prompting techniques we employed:

• System Prompt: We used a system prompt to set the overall context and role for Claude.
• Structured Input: We placed the call transcript at the beginning of the prompt using XML tags.
• Clear Instructions: We provided detailed guidelines on what to focus on and how to structure the output.
• Output Formatting: We specified a JSON structure for the summary, ensuring consistent and easily parseable results.
• Handling Edge Cases: We added criteria for identifying calls with insufficient data.
• Examples: We included diverse examples to illustrate desired outputs for different scenarios.
• Thinking Aloud: We asked Claude to show its analysis in tags before providing the final JSON output.

By employing these techniques, we created a robust prompt capable of generating structured summaries for a wide range of customer service call transcripts, while appropriately handling edge cases. This approach can be adapted to many other complex prompting scenarios beyond call summarization.

Important Note: While we've developed a sophisticated prompt that appears to handle our test cases well, it's crucial to understand that this prompt is not yet production-ready. What we've created is a promising starting point, but it requires extensive testing and evaluation before it can be reliably used in a real-world setting. Our current eye-ball test evaluation has been based on a small set of examples. This is not representative of the diverse and often unpredictable nature of real customer service calls. To ensure the prompt's effectiveness and reliability, we need to implement a comprehensive evaluation process that includes quantitative metrics. Robust, data-driven evaluations are the key to bridging the gap between a promising prototype and a reliable, production-grade solution.