The AI Ladder

There are four ways to adapt AI to work for your business. I call this the AI ladder. Most organizations will use the top two steps of this ladder: retrieval augmented generation (RAG)and prompt engineering. And, many will find the best effort/ reward ratio with RAG.

Foundation models: Trained on vast amounts of data like the pile, foundation models learn the basic constructs of a language including grammar (i.e., syntax), meaning (i.e., semantics), and the contextual usage of language. You are unlikely to build one unless you’re Bloomberg.

Fine-Tuned Models: These are built on foundation models like Llama or GPT. They tailor a foundation model for a domain, task type, or industry by putting the foundation model through further training using additional data. For example, you can fine-tune a foundation model using a
medical system's after-visit summary data set to generate new summaries using the medical jargon.

Retrieval Augmented Generation (RAG): A key problem with the first two approaches is that they don’t have access to the latest information; they rely on the data provided during the last training run. RAG solves this by combining the generative capabilities of these models with real-time data retrieval. For example, the RAG approach will query the latest sales data from your CRM system to accurately answer questions like "What is our biggest opportunity?".

Prompt Engineering: This approach guides a model’s behavior by providing instructions along with the task or the question. For example, through prompt engineering, you can ask a model to identify it by a specific name, take a tone like professional or friendly, and provide the output asa bullet list or a long-winded write-up. This is the fastest and easiest way to change a model’s behavior.

There are four ways to adapt AI to work for your business. I call this the AI ladder. Most organizations will use the top two steps of this ladder: retrieval augmented generation (RAG)and prompt engineering. And, many will find the best effort/ reward ratio with RAG.

The Seven Challenges

Now to the challenges. Here are the seven challenges you will need to solve when implementing AI.

1. Data Freshness: Answering questions and executing tasks using the latest information is critical for accuracy. Imagine an HR policy that has been updated over time — four versions of this policy state that employees are free to work from home and just the last version states that employees must be in office at least three times a week. In this scenario, for a model, the statistically correct answer is the one that appears more often (i.e., the old policy appears ¾ times). However, the factually correct answer is the one in the last version. Data freshness and the use of the correct data source get quite challenging at scale.

2. Security & Access Control: when building foundation models, fine-tuned models, or RAG systems, the AI gets to see a lot of your enterprise data. Data that is typically subjected to role-based access controls. Making sure information is not leaking in violation of the access controls is a challenge. For example, you don’t want a draft proposal for an M&A from corporate development to leak to the whole organization through an AI chat.

3. Guardrails: With power comes responsibility and also scandals. The solution is establishing clear guidelines and boundaries for AI through guardrails. However, what is appropriate is a subjective question, which means you will need the flexibility to update these guardrails and how they are enforced based on the context.

4. Monitoring & Analytics: accuracy, performance, and impact are key areas to monitor when deploying an AI system. When it comes to generative AI systems with non-deterministic outputs, identifying repeatable ways to monitor these KPIs is often a challenge.

5. Sources & Citations: Credibility in AI outputs often depends on the reliability of the data sources and the ability to trace back to these sources. This is especially important in enterprise AI implementations where there are many different possible sources for an answer and getting it from a stale source is likely to produce a wholly inaccurate answer.

6. Sourcing Talent: AI tools and technologies are rapidly evolving, which means it’s extremely difficult for IT leaders to identify the roles, job descriptions, and resource requirements needed for AI projects. There is often much confusion about the differences between ML engineers, ML
ops engineers, AI engineers, and Python developers with Langchain experience, and whom to use when.

7. ROI: Just like with any other technology investment, making sure you have a clear path to ROI is the key to long-term success. Some use cases show a lot of promise for ROI while many
other use cases may result in vanity. Identifying the right order of implementation for these use cases is probably the biggest AI challenge of them all.