Context is Everything: Developing Effective Context Models

Introduction

‍

Context really is everything. For humans it’s hard when you walk into the middle of a conversation and are asked to provide impromptu input. You don’t have visibility into the arc of the conversation and you may not be prepared to answer a complex question that requires supporting information that you haven’t prepared. No matter how capable you are, there is a good chance that you’ll struggle to immediately synthesize the best (and undoubtedly wittiest) reply.

The same challenge exists for Large Language Models (LLMs) that are being asked to provide high quality replies to potentially complex user requests. No matter how capable the LLM may be, two things are needed to ensure that the LLM can provide an expert and accurate reply:

Visibility into the conversation in progress - This “thread-based” context organizes relevant messages within the current conversation between a group of people and with the LLM that is being asked to participate in the conversation.

Access to relevant knowledge - If the user has asked about world history and that associated information is in the model’s training set then the LLM will likely provide a solid response. But what about cases where an expert response requires access to proprietary knowledge that is protected within your organization’s walls. That data needs to be organized for rapid retrieval and then made available to the LLM to reply to the user’s question. This “relevance-based” retrieval strategy pulls the most relevant information in your organization and fills a pile of relevant information that the LLM can use to prepare an expert response.

Solving these challenges is the role of Retrieval-Augmented Generation (RAG) solutions that leverage repositories of natural language embeddings to enhance the contextual relevance of responses generated by Large Language Models (LLMs). In essence, the RAG solution retrieves embeddings relevant to a user's request to provide a contextual mapping of supporting information that the LLM can consume to synthesize a high quality response.

Unfortunately, common practices such as thread-based context determination that are used for building the data set of relevant supporting information can introduce inefficiencies and inaccuracies by introducing data that has limited relevance to a user's request.

Foundation4.ai provides LLMs with a superior context mapping that ensures the highest quality responses using our hybrid context model that builds a contextual map of highly relevant information by combining the best of thread-based and relevance-driven approaches while also incorporating weightings driven by annotation metadata available within the Foundation4 pipeline datasets.

To get an understanding of why we're focused on developing superior contextual mappings, it is helpful to get an understanding of the limitations of thread-based contextual definition and how those shortcomings can be addressed.

Limitations of Thread-Based Contextual Models

Building context definitions based on threads may seem intuitive, as threads often group related discussions and provide a clear structure. However, this approach has significant limitations when it comes to capturing the dynamic and multifaceted nature of real-world conversations. Threads can introduce noise, omit critical out-of-thread information, and struggle to adapt to shifting topics. Understanding the following challenges is essential for developing a more effective and nuanced method for defining context in RAG systems.

1. Topic Shifts Within Threads:

Human communication often shifts topics within the same conversational thread, leading to contextual drift.

Example: A discussion starts with a focus on one technical issue but shifts to another unrelated concern. Relying on the thread as a contextual unit retrieves embeddings related to both topics, diluting relevance.

2. Missed Out-of-Thread Relevance:

Embeddings relevant to a query might exist outside the thread but remain unconsidered due to thread constraints.

Example: A discussion about troubleshooting network protocols may omit relevant embeddings from a separate thread on firmware updates that directly address the issue.

3. Noise Amplification:

Threads can contain unrelated or tangential content, increasing noise in the retrieved embeddings.

This noise can obscure the most relevant data, leading to suboptimal model responses.

Benefits of Relevance-Driven Contextual Models

Relevance-driven contextual models represent a major step forward in improving the precision and quality of RAG systems. By prioritizing embeddings based on their semantic closeness to the query, Foundation4.ai delivers tighter and more accurate contextual mappings. The relevance-driven approaches, helps to address limitations of thread-based systems that were discussed earlier while scaling effectively across diverse data domains.

1. Increased Precision:

By selecting embeddings based on relevance, the retrieved context has tighter vector dispersion.

Example: A query about "optimizing database queries" retrieves embeddings specifically about indexing and performance tuning, ignoring tangential discussions.

The LLM generates responses informed by focused, high-quality embeddings and generally can deliver more specific answers to users.

2. Scalability Across Domains:

This model adapts seamlessly to diverse datasets, including highly dynamic or unstructured data.

Relevance-driven retrieval uses embeddings and similarity scoring, which are agnostic to dataset type. This ensures that whether the data is structured (e.g., technical documentation) or unstructured (e.g., forum posts), embeddings consistently represent the semantic meaning.

Example: For technical documentation, relevance models can precisely segment and retrieve knowledge specific to user queries. In user-generated forums, embeddings capture conversational intent and nuanced topics, enabling extraction of relevant responses.

Dynamic and Unstructured Data Handling: Embeddings naturally encode semantic relationships, making them effective for diverse data formats. For example, a technical manual on "network protocols" and a forum thread on "Wi-Fi troubleshooting" can both yield embeddings that align with technical accuracy and user-experience insights.

Scalability Factors: Variations in data format require normalization and preprocessing, but the adaptability of relevance-driven approaches ensures consistent performance across domains.

The Foundation4 Approach: Hybrid Context Models

Foundation4's hybrid context model combines thread-based and relevance-driven approaches, leveraging the strengths of each to provide superior contextual definitions.

Core Principles

1. Thread-Based Strengths:

Threads offer broad contextual information that may capture background or indirectly relevant data.

Example: A thread on "database migration challenges" might include insights on tool compatibility that are useful for a query about "data consistency issues during migration."

2. Relevance-Driven Precision:

Relevance-driven models focus on retrieving the most semantically aligned embeddings with the query.

Example: For a query about "optimizing server performance," relevance-driven retrieval highlights specific configurations and best practices while avoiding tangential content.

3. Combined Ranking Mechanisms:

Hybrid models dynamically rank thread-level embeddings against individual relevance-driven embeddings while also incorporating annotations stored with each embedding to prioritize those that maximize query relevance.

Example: A query about "secure API design" might retrieve embeddings from a specific thread on best practices while also pulling in relevant data from threads discussing encryption standards that have been annotated as validated.

4. Contextual Synthesis:

Hybrid models synthesize data from both approaches, balancing specificity with background context.

Example: A synthesized context might provide targeted API design principles while addressing compliance with specific industry regulations.

Implementation Strategy

Regardless of your RAG solution of choice (although we think you should be using Foundation4) you should be following the core implementation strategies outlined here to provide high quality results:

1. Metadata and Annotation Considerations:

Annotate data with relevant metadata to enhance retrieval hen loading data to the pipeline. Metadata might include timestamps, source identifiers, and topic tags to ensure embeddings retain critical context and should be retained through all pipeline processing stages

Example: A dataset of forum posts should include metadata identifying the discussion thread, user role (e.g., expert or novice), and creation date to aid contextual filtering

2. Weighted Scoring Systems:

Assign weights to embeddings based on metadata and direct query alignment. Use dynamic weighting to adjust relevance scores as queries evolve.

Example: Metadata such as recency or expert-authored content can increase the weight of embeddings that align more closely with the user query.

3. Dynamic Query Expansion:

Automatically expand queries to include synonyms, related terms, or adjacent concepts. Utilize domain-specific ontologies or pre-trained language models to identify expansions.

Example: A query on "server optimization" might expand to include related terms like "load balancing," "caching strategies," and "scalability techniques."

4. Hierarchical Embedding Structures:

Organize embeddings into a hierarchy based on granularity, such as sentence, paragraph, and document levels. Implement retrieval pipelines that prioritize finer-grained embeddings while considering higher-level structures for background context.

Example: For a technical query, sentence-level embeddings directly addressing the query are retrieved first, while document-level embeddings provide supplementary insights.

5. Feedback Loops for Refinement:

Incorporate user feedback to iteratively refine retrieval models. Feedback can include explicit ratings, selections, or implicit signals such as query reformulations.

Example: You can consider including a source bibliography with responses and enable users to upvote/downvote inclusion of certain embeddings over others, adjust weighting mechanisms to prioritize similar content in future queries.

Conclusion

Thread-based contextual models, while intuitive, fall short in delivering optimal relevance for RAG solutions. Relevance-driven models address these shortcomings but may overlook valuable broader context. Hybrid context models effectively balance the precision of relevance-driven retrieval with the breadth of thread-based approaches, offering a superior framework for defining context. By combining these strategies, organizations can enhance response accuracy, reduce noise, and better support diverse datasets.

‍

We’d love to have the opportunity to talk with you about how Foundation4.ai produces tighter vector dispersions, more relevant context maps and better outcomes for your users.

‍