File index
The file index stores files in a local folder and index them for retrieval. This file index provides the following infrastructure to support the indexing:
- SQL table Source: store the list of files that are indexed by the system
- Vector store: contain the embedding of segments of the files
- Document store: contain the text of segments of the files. Each text stored in this document store is associated with a vector in the vector store.
- SQL table Index: store the relationship between (1) the source and the docstore, and (2) the source and the vector store.
The indexing and retrieval pipelines are encouraged to use the above software infrastructure.
Indexing pipeline¶
The ktem has default indexing pipeline: ktem.index.file.pipelines.IndexDocumentPipeline
.
This default pipeline works as follow:
- Input: list of file paths
- Output: list of nodes that are indexed into database
- Process:
- Read files into texts. Different file types has different ways to read texts.
- Split text files into smaller segments
- Run each segments into embeddings.
- Store the embeddings into vector store. Store the texts of each segment into docstore. Store the list of files in Source. Store the linking between Sources and docstore + vectorstore in Index table.
You can customize this default pipeline if your indexing process is close to the default pipeline. You can create your own indexing pipeline if there are too much different logic.
Customize the default pipeline¶
The default pipeline provides the contact points in flowsettings.py
.
FILE_INDEX_PIPELINE_FILE_EXTRACTORS
. Supply overriding file extractor, based on file extension. Example:{".pdf": "path.to.PDFReader", ".xlsx": "path.to.ExcelReader"}
FILE_INDEX_PIPELINE_SPLITTER_CHUNK_SIZE
. The expected number of characters of each text segment. Example: 1024.FILE_INDEX_PIPELINE_SPLITTER_CHUNK_OVERLAP
. The expected number of characters that consecutive text segments should overlap with each other. Example: 256.
Create your own indexing pipeline¶
Your indexing pipeline will subclass BaseFileIndexIndexing
.
You should define the following methods:
run(self, file_paths)
: run the indexing given the pipelineget_pipeline(cls, user_settings, index_settings)
: return the fully-initialized pipeline, ready to be used by ktem.user_settings
: is a dictionary contains user settings (e.g.{"pdf_mode": True, "num_retrieval": 5}
). You can declare these settings in theget_user_settings
classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to yourget_pipeline
method.index_settings
: is a dictionary. Currently it's empty for File Index.
get_user_settings
: to declare user settings, return a dictionary.
By subclassing BaseFileIndexIndexing
, You will have access to the following resources:
self._Source
: the source tableself._Index
: the index tableself._VS
: the vector storeself._DS
: the docstore
Once you have prepared your pipeline, register it in flowsettings.py
: FILE_INDEX_PIPELINE = "<python.path.to.your.pipeline>"
.
Retrieval pipeline¶
The ktem has default retrieval pipeline:
ktem.index.file.pipelines.DocumentRetrievalPipeline
. This pipeline works as
follow:
- Input: user text query & optionally a list of source file ids
- Output: the output segments that match the user text query
- Process:
- If a list of source file ids is given, get the list of vector ids that associate with those file ids.
- Embed the user text query.
- Query the vector store. Provide a list of vector ids to limit query scope if the user restrict.
- Return the matched text segments
Create your own retrieval pipeline¶
Your retrieval pipeline will subclass BaseFileIndexRetriever
. The retriever
has the same database, vectorstore and docstore accesses like the indexing
pipeline.
You should define the following methods:
run(self, query, file_ids)
: retrieve relevant documents relating to the query. Iffile_ids
is given, you should restrict your search within thesefile_ids
.get_pipeline(cls, user_settings, index_settings, selected)
: return the fully-initialized pipeline, ready to be used by ktem.user_settings
: is a dictionary contains user settings (e.g.{"pdf_mode": True, "num_retrieval": 5}
). You can declare these settings in theget_user_settings
classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to yourget_pipeline
method.index_settings
: is a dictionary. Currently it's empty for File Index.selected
: a list of file ids selected by user. If user doesn't select anything, this variable will be None.
get_user_settings
: to declare user settings, return a dictionary.
Once you build the retrieval pipeline class, you can register it in
flowsettings.py
: FILE_INDEXING_RETRIEVER_PIPELIENS = ["path.to.retrieval.pipelie"]
. Because there can be
multiple parallel pipelines within an index, this variable takes a list of
string rather than a string.
Software infrastructure¶
Infra | Access | Schema | Ref |
---|---|---|---|
SQL table Source | self._Source | - id (int): id of the source (auto) - name (str): the name of the file - path (str): the path of the file - size (int): the file size in bytes - note (dict): allow extra optional information about the file - date_created (datetime): the time the file is created (auto) |
This is SQLALchemy ORM class. Can consult |
SQL table Index | self._Index | - id (int): id of the index entry (auto) - source_id (int): the id of a file in the Source table - target_id: the id of the segment in docstore or vector store - relation_type (str): if the link is "document" or "vector" |
This is SQLAlchemy ORM class |
Vector store | self._VS | - self._VS.add: add the list of embeddings to the vector store (optionally associate metadata and ids) - self._VS.delete: delete vector entries based on ids - self._VS.query: get embeddings based on embeddings. |
kotaemon > storages > vectorstores > BaseVectorStore |
Doc store | self._DS | - self._DS.add: add the segments to document stores - self._DS.get: get the segments based on id - self._DS.get_all: get all segments - self._DS.delete: delete segments based on id |
kotaemon > storages > docstores > base > BaseDocumentStore |