SEC Filing Data Extraction for GraphRAG

This notebook processes XBRL files, extracts specific sections, and saves them in JSON format. This is particularly useful for creating a GraphRAG (Graph-based Retrieval-Augmented Generation) system for financial documents. By extracting structured data from these filings, we can enhance the knowledge graph with detailed financial information, which can be used for various analytical and generative tasks.

Import Necessary Libraries

We begin by importing the necessary libraries. os and json are used for file operations and data handling, while BeautifulSoup from the bs4 library is used for parsing HTML/XBRL content. Parsing is crucial for extracting structured information from the filings.

Define Functions for Data Extraction

Extract Sections

This function extracts specific sections from the filing text. For a GraphRAG, it's important to have well-defined sections as they represent different nodes or entities in the graph. Each section corresponds to a key aspect of the financial document, such as business overview, risk factors, etc.

Extract Section

This helper function extracts a section of text between two headings. It's essential for isolating the content of interest, which can then be used to populate nodes in the knowledge graph.

Extract CIK

The Central Index Key (CIK) is a unique identifier for companies in the SEC's EDGAR database. Extracting this allows us to link the filing to the correct entity in our knowledge graph.

Extract Fiscal Year

Extracting the fiscal year helps in organizing the data temporally within the knowledge graph, allowing for time-based queries and analyses.

Extract Fiscal Quarter

Similar to fiscal year, the fiscal quarter provides finer granularity for temporal data organization in the graph.

Process Tickers

This function processes each ticker, extracts relevant data, and saves it in JSON format. The structured JSON output is suitable for ingestion into a knowledge graph, where each section can be linked to other related data points.

Example Usage

Set up the paths for the ticker file, input directory, and output directory, and then call the process_tickers function to start processing. This step is crucial for preparing the data for integration into a GraphRAG system.