Step 2 Transcript Writer
Notebook 2: Transcript Writer
This notebook uses the Llama-3.1-70B-Instruct model to take the cleaned up text from previous notebook and convert it into a podcast transcript
SYSTEM_PROMPT is used for setting the model context or profile for working on a task. Here we prompt it to be a great podcast transcript writer to assist with our task
Experimentation with the SYSTEM_PROMPT below is encouraged, this worked best for the few examples the flow was tested with:
For those of the readers that want to flex their money, please feel free to try using the 405B model here.
For our GPU poor friends, you're encouraged to test with a smaller model as well. 8B should work well out of the box for this example:
Import the necessary framework
Read in the file generated from earlier.
The encoding details are to avoid issues with generic PDF(s) that might be ingested
Since we have defined the System role earlier, we can now pass the entire file as INPUT_PROMPT to the model and have it use that to generate the podcast
Hugging Face has a great pipeline() method which makes our life easy for generating text from LLMs.
We will set the temperature to 1 to encourage creativity and max_new_tokens to 8126
Loading checkpoint shards: 0%| | 0/30 [00:00<?, ?it/s]
Setting `pad_token_id` to `eos_token_id`:None for open-end generation. Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
This is awesome, we can now save and verify the output generated from the model before moving to the next notebook
SPEAKER 1: Welcome to this week's episode of AI Insights, where we explore the latest developments in the field of artificial intelligence. Today, we're going to dive into the fascinating world of knowledge distillation, a methodology that transfers advanced capabilities from leading proprietary Large Language Models, or LLMs, to their open-source counterparts. Joining me on this journey is my co-host, who's new to the topic, and I'll be guiding them through the ins and outs of knowledge distillation. So, let's get started! SPEAKER 2: Sounds exciting! I've heard of knowledge distillation, but I'm not entirely sure what it's all about. Can you give me a brief overview? SPEAKER 1: Of course! Knowledge distillation is a technique that enables the transfer of knowledge from a large, complex model, like GPT-4 or Gemini, to a smaller, more efficient model, like LLaMA or Mistral. This process allows the smaller model to learn from the teacher model's output, enabling it to acquire similar capabilities. SPEAKER 2: That sounds like a great way to make AI more accessible. But how does it actually work? SPEAKER 1: Ah, that's a great question! The distillation process involves several stages, including knowledge elicitation, knowledge storage, knowledge inference, and knowledge application. The teacher model shares its knowledge with the student model, which then learns to emulate the teacher's output behavior. SPEAKER 2: Hmm, I see. So, it's like a teacher-student relationship, where the teacher model guides the student model to learn from its output. SPEAKER 1: Exactly! And this process can be formulated as a loss function, where the student model learns to minimize the discrepancy between its output and the teacher model's output. SPEAKER 2: Right. That makes sense. But what about the different approaches to knowledge distillation? I've heard of supervised fine-tuning, divergence and similarity, reinforcement learning, and rank optimization. SPEAKER 1: Ah, yes! Those are all valid approaches to knowledge distillation. Supervised fine-tuning involves training the student model on a smaller dataset, while divergence and similarity focus on aligning the hidden states or features of the student model with those of the teacher model. Reinforcement learning and rank optimization are more advanced methods that involve feedback from the teacher model to train the student model. SPEAKER 2: Wow, that's a lot to take in. Can you give me some examples of how these approaches are used in real-world applications? SPEAKER 1: Of course! For instance, the Vicuna model uses supervised fine-tuning to distill knowledge from the teacher model, while the UltraChat model employs a combination of knowledge distillation and reinforcement learning to create a powerful chat model. SPEAKER 2: That's fascinating! I can see how knowledge distillation can be applied to various domains, like natural language processing, computer vision, and even multimodal tasks. SPEAKER 1: Exactly! Knowledge distillation has far-reaching implications for AI research and applications. It enables the transfer of knowledge across different models, architectures, and domains, making it a powerful tool for building more efficient and effective AI systems. SPEAKER 2: I'm starting to see the bigger picture now. Knowledge distillation is not just a technique; it's a way to democratize access to advanced AI capabilities and foster innovation across a broader spectrum of applications and users. SPEAKER 1: That's right! And as we continue to explore the frontiers of AI, knowledge distillation will play an increasingly important role in shaping the future of artificial intelligence. SPEAKER 2: Well, I'm excited to learn more about knowledge distillation and its applications. Thanks for guiding me through this journey, and I'm looking forward to our next episode! SPEAKER 1: Thank you for joining me on this episode of AI Insights! If you want to learn more about knowledge distillation and its applications, be sure to check out our resources section, where we've curated a list of papers, articles, and tutorials to help you get started.
Let's save the output as pickle file and continue further to Notebook 3
Next Notebook: Transcript Re-writer
We now have a working transcript but we can try making it more dramatic and natural. In the next notebook, we will use Llama-3.1-8B-Instruct model to do so.