Reframing LLM Chat with Data: Introducing LLM-Assisted Data Recipes by Matthew Harris

Instead of fine-tuning an LLM as a first approach, try prompt architecting instead

It falls in line with a direction the company is taking beyond just training and fine-tuning LLMs and toward special-purpose, extremely use case-specific data and leaving more general-purpose data out. Explicit feedback is information users provide in response to a request by our product; implicit feedback is information we learn from user interactions without needing users to deliberately provide feedback. Coding assistants and Midjourney are examples of implicit feedback while thumbs up and thumb downs are explicit feedback. If we design our UX well, like coding assistants and Midjourney, we can collect plenty of implicit feedback to improve our product and models.

Generative AI (GenAI) is revolutionizing industries and reformulating how organizations engage with customers, design products and streamline operations. However, many companies are grappling with how to get ready to adopt LLMs. This is the 6th article in a series on using large language models (LLMs) in practice. Previous articles explored how to leverage pre-trained LLMs via prompt engineering and fine-tuning. While these approaches can handle the overwhelming majority of LLM use cases, it may make sense to build an LLM from scratch in some situations.

Instead of fine-tuning an LLM as a first approach, try prompt architecting instead

Next, we create an assistant class property that maps to our newly created Assistant. Sarvam AI also uses NVIDIA NIM microservices, NVIDIA Riva for conversational AI, NVIDIA TensorRT-LLM software and NVIDIA Triton Inference Server to optimize and deploy conversational AI agents with sub-second latency. It’s important to anticipate that data could easily become the bottleneck of your project, as it takes the most time to organize. Community created roadmaps, articles, resources and journeys for developers to help you choose your path and grow in your career. There is no guarantee that the LLM will not hallucinate or swerve offtrack. Nonetheless, these response accuracy checks strive to nip anomalous output in the bud.

But he adds most organizations won’t create their own LLM and maybe not even their own version of an LLM. Be prepared to revisit decisions about building or buying as the technology evolves, Lamarre warns. “The question comes down to, ‘How much can I competitively differentiate if I build versus if I buy,’ and I think that boundary is going to change over time,” he says. Apparently credible internal data can be wrong or just out of date, too, she cautioned.

While leaders are figuring out the answer to this question, many are taking it on faith when their employees say they’re making better use of their time. She said the first version of the LLM will be trained on 24,000 hours of audio, while the second will need 500,000 hours. Moses Daudu, a senior AI engineer at Awarri, told Rest of World that text token parameters will run into billions. “[We are] targeting 10 billion tokens for the pre-training, and for the fine tuning we’re targeting 600,000 instruction samples for the first version,” he said. In April, Awarri launched LangEasy, a platform that allows anyone with a smartphone to help train the model through voice and text inputs.

Open the iGot notebook and install the required libraries

They have generally found them to be astounding in terms of their ability to express complex ideas in articulate language. However, most users realize that these systems are primarily trained on internet-based information and can’t respond to prompts or questions regarding proprietary content or knowledge. Enhancing large language models (LLMs) with knowledge beyond their training data is an important area of interest, especially for enterprise applications.

He brought in technical co-founders Kevin Kastberg and Arvid Winterfeldt, both with backgrounds in Physics and the former in AI research. Stockholm-based legal tech startup Qura, founded in 2023, has raised a seed round of €2.1M led by Cherry Ventures and GP Sophia Bendz who will join the board. The funding round, led by Cherry Ventures and followed by senior Swedish lawyers and other angels, will be used to improve the platform further and collect more legal data.

By extracting and clustering these keywords, I aim to uncover underlying connections between the titles, offering a versatile strategy for structuring the dataset. In this blog, I intend to explore the efficacy of combining traditional NLP and machine learning techniques with the versatility of LLMs. This exploration includes integrating ChatGPT App simple keyword extraction using KeyBERT, sentence embeddings with BERT, and employing UMAP for dimensionality reduction coupled with HDBSCAN for clustering. All these are used in conjunction with Zephyr-7B-Beta, a highly performant LLM. The findings are uploaded into a knowledge graph for enhanced analysis and discovery.

Developer Tools 2.0

Having clustered the keywords, we are now ready to employ GenAI once more to enhance and refine our findings. At this step, we will use a LLM to analyze each cluster, summarize the keywords and keyphrases while assigning a brief label to the cluster. If you want to skip the data processing steps, you may use the cs dataset, available in the Github repository. It’s important to note that each step in this process offers the flexibility to experiment with alternative methods, algorithms, or models. While the Large Language Models (LLMs) are useful and skilled tools, relying entirely on their output is not always advisable as they often require verification and grounding. However, merging traditional NLP methods with the capabilities of generative AI typically yields satisfactory results.

DevOps is not fundamentally about reproducible workflows or shifting left or empowering two pizza teams—and it’s definitely not about writing YAML files. Taken together, this means models are likely to be the least durable component in the system.

Leaders generally customize models through fine-tuning instead of building models from scratch.

I’m grateful to Maarten Grootendorst for highlighting this aspect, as can be seen here. The threshold parameter determines the minimum similarity required for documents to be grouped into the same community. A higher value will group nearly identical documents, while a lower value will cluster documents covering similar topics. Start with creating a TextGeneration pipeline wrapper for the LLM and instantiate KeyBERT.

It’s beneficial for companies to clarify data ownership in their provider contracts before investing. The middleware layer facilitates seamless interaction between the operating system and various applications. It supports a wide range of programming languages, including Python, .NET and Java, which enables compatibility and smooth communication across different platforms. Unlike content safety or PII defects which have a lot of attention and thus seldom occur, factual inconsistencies are stubbornly persistent and more challenging to detect. They’re more common and occur at a baseline rate of 5 – 10%, and from what we’ve learned from LLM providers, it can be challenging to get it below 2%, even on simple tasks such as summarization. One particularly powerful application of LLM-as-Judge is checking a new prompting strategy against regression.

He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies operationalize Large Language Models (LLMs) to accelerate their AI product journey. One way to get quality annotations is to integrate Human-in-the-Loop (HITL) into the user experience (UX). By allowing users to provide feedback and corrections easily, we can improve the immediate output and collect valuable data to improve our models. Taken together, a carefully crafted workflow using a smaller model can often match, or even surpass, the output quality of a single large model, while being faster and cheaper.

Investigate prompt techniques like chain-of-thought or few-shot to make it higher quality. Don’t let your tooling hold you back on experimentation; if it is, rebuild it, or buy something to make it better. This aligns with a recent a16z report showing that many companies are moving faster with internal LLM applications compared to external ones. By experimenting with AI for internal productivity, organizations can start capturing value while learning how to manage risk in a more controlled environment.

For example, when asked to extract specific attributes or metadata from a document, an LLM may confidently return values even when those values don’t actually exist. Alternatively, the model may respond in a language other than English because we provided non-English documents in the context. Create unit tests (i.e., assertions) consisting of samples of inputs and outputs from production, with expectations for building llm from scratch outputs based on at least three criteria. While three criteria might seem arbitrary, it’s a practical number to start with; fewer might indicate that your task isn’t sufficiently defined or is too open-ended, like a general-purpose chatbot. These unit tests, or assertions, should be triggered by any changes to the pipeline, whether it’s editing a prompt, adding new context via RAG, or other modifications.

Or the data could include hidden repetitions that provide minimum or no value to the training process, and not represent the domain or task entirely, which may cause the resulting AI model to overfit. Before delving into the world of foundational models and LLMs, take a step back and note the problem you are looking to solve. Once you identify this, it’s important to determine which natural language tasks you need.

As is the case with many innovative solutions, there is not a one-size-fits-all approach. Weighing your options regarding the model that is right for your business is the first step when starting your company’s AI journey. For business leaders, training an LLM from scratch could sound daunting, but if you have data available and a domain-specific “business problem” that a generic LLM will not solve, it will be worth the investment in the long run. Business leaders have been under pressure to find the best way to incorporate generative AI into their strategies to yield the best results for their organization and stakeholders.

This write-up has an example of an assertion-based test for an actual use case. It is fairly well established that LLMs are pretty good at generating code. Not yet perfect, for sure, but a lot of the world right now is using tools like GitHub Copilot for software development. It is becoming a common pattern in LLM applications to have them generate and execute code as part of solving tasks.

Collaborating with an AI provider is a viable option for businesses implementing LLMs. You can foun additiona information about ai customer service and artificial intelligence and NLP. These providers offer expertise and resources to build and deploy tailored language models. The advantage of partnering with an AI provider is gaining access to their expertise and support. They have deep machine learning and natural language processing knowledge, guiding businesses effectively. They offer insights, recommend models, and provide support throughout development and deployment. Consider that collaborating with an AI provider may involve additional costs.

The company develops AI-based data management and analysis tools for enterprises, with services ranging from text analytics to measuring online sentiment on social media platforms and websites. For both types, this stage involves evaluating the trained model’s performance on a different, previously unseen data set to assess how it handles new data. This is measured through standard machine learning metrics — such as accuracy, precision and F1 score — and applying cross-validation and other techniques to improve the model’s ability to generalize to new data. While the above are the hellos-world of LLM applications, none of them make sense for virtually any product company to build themselves.

We’ve been communicating this to our customers and partners for months now. Nearest Neighbor Search with naive embeddings yields very noisy results and you’re likely better off starting with a keyword-based approach. This is typically quantified via ranking metrics such as Mean Reciprocal Rank (MRR) or Normalized Discounted Cumulative Gain (NDCG). MRR evaluates how well a system places the first relevant result in a ranked list while NDCG considers the relevance of all the results and their positions. They measure how good the system is at ranking relevant documents higher and irrelevant documents lower. For example, if we’re retrieving user summaries to generate movie review summaries, we’ll want to rank reviews for the specific movie higher while excluding reviews for other movies.

“Nigeria has that human capacity to build out the model, and potentially sustain it. But I think that the infrastructure is really the biggest roadblock to that,” she said. The tool is just in its first version, and I plan to evolve it into a more user-friendly solution.

So our team selected a BERT-based model for fine-tuning in cybersecurity two years ago. Finally, we process user queries through the router and provide ChatGPT an appropriate response. Here, we fetch relevant baggage policy information based on the user’s query by searching through the ChromaDB collection.

A little-known AI startup is behind Nigeria’s first government-backed LLM

These are general problems for many businesses with a large gap between promising demo and dependable component—the customary domain of software companies. Investing valuable R&D resources on general problems being tackled en masse by the current Y Combinator batch is a waste. Already, we have interactive arenas for neutral, crowd-sourced evaluation of chat and coding models—an outer loop of collective, iterative improvement.

Input-output pairs from production are the “real things, real places” (genchi genbutsu) of LLM applications, and they cannot be substituted. Recent research highlighted that developers’ perceptions of what constitutes “good” and “bad” outputs shift as they interact with more data (i.e., criteria drift). While developers can come up with some criteria upfront for evaluating LLM outputs, these predefined criteria are often incomplete. For instance, during the course of development, we might update the prompt to increase the probability of good responses and decrease the probability of bad ones.

Graph RAG systems can turn the user’s query into a chain that contains information from different nodes from a graph database. Implicit fact queries introduce additional challenges, including the need for coordinating multiple context retrievals and effectively integrating reasoning and retrieval capabilities. The most common approach for addressing these queries is using basic RAG, where the LLM retrieves relevant information from a knowledge base and uses it to generate a response. The best-known way to incorporate domain- and customer-specific knowledge into LLMs is to use retrieval-augmented generation (RAG). The tools and patterns we’ve laid out here are likely the starting point, not the end state, for integrating LLMs. We’ll update this as major changes take place (e.g., a shift toward model training) and release new reference architectures where it makes sense.

AI For Everyone is a fantastic course designed for those who want to understand the basics of artificial intelligence without diving into complex math or coding. This course is perfect for anyone looking to grasp how AI can be integrated into various fields, including business and technology. Unfortunately, as anyone who has worked on shipping real-world software knows, there’s a world of difference between a demo that works in a controlled setting and a product that operates reliably at scale. By centering humans and asking how an LLM can support their workflow, this leads to significantly different product and design decisions. Ultimately, it will drive you to build different products than competitors who try to rapidly offshore all responsibility to LLMs—better, more useful, and less risky products. It didn’t shorten the feedback gap between models and their inferences and interactions in production.

• The effectiveness of the process is highly reliant on the choice of the LLM and issues are minimal with a highly performant LLM.
• In part 1 of this essay, we introduced the tactical nuts and bolts of working with LLMs.
• Fine-tuning an existing LLM is simpler, but still technically challenging and resource-intensive.
• Also, the concept of LLMs As Tool Makers (LATM) is established (see for example Cai et al, 2023).

First, even with a context window of 10M tokens, we’d still need a way to select information to feed into the model. Second, beyond the narrow needle-in-a-haystack eval, we’ve yet to see convincing data that models can effectively reason over such a large context. Thus, without good retrieval (and ranking), we risk overwhelming the model with distractors, or may even fill the context window with completely irrelevant information. Imagine we’re building a RAG system to generate SQL queries from natural language. But, what if we include column descriptions and some representative values?

These prompts often give good results but fall short of accuracy levels required for production deployments. But since they are so new—and behave so differently from normal computing resources—it’s not always obvious how to use them. As a financial data company, Bloomberg’s data analysts have collected and maintained financial language documents over the span of forty years. The team pulled from this extensive archive of financial data to create a comprehensive 363 billion token dataset consisting of English financial documents.

Thus, if we have to migrate prompts across models, expect it to take more time than simply swapping the API endpoint. Don’t assume that plugging in the same prompt will lead to similar or better results. Also, having reliable, automated evals helps with measuring task performance before and after migration, and reduces the effort needed for manual verification. Despite their impressive zero-shot capabilities and often delightful outputs, their failure modes can be highly unpredictable.

The lack of specialization means that the system can’t prioritize recent information, parse domain-specific formats, or understand the nuances of specific tasks. As a result, users are left with a shallow, unreliable experience that doesn’t meet their needs. Don’t point your shears at the same yaks that OpenAI or other model providers will need to shave if they want to provide viable enterprise software. If you’re going to fine-tune, you’d better be really confident that you’re set up to do it again and again as base models improve—see the “The model isn’t the product” and “Build LLMOps” below.

Given how prevalent the embedding-based RAG demo is, it’s easy to forget or overlook the decades of research and solutions in information retrieval. As a result, we’ve split our single prompt into multiple prompts that are each simple, focused, and easy to understand. And by breaking them up, we can now iterate and eval each prompt individually.