New data.world report finds a technique for making LLMs 3x more accurate in answering business questions
data.world’s benchmark report reveals 3x uptick in LLM response accuracy over SQL databases, when those answers were backed by a Knowledge Graph.
AUSTIN, Texas, Dec. 11, 2023 (GLOBE NEWSWIRE) -- data.world, the leader in AI-ready data cataloging, recently released its benchmark report on LLM (Large Language Model) response accuracy, which ignites data professionals’ optimism about using AI for business.
2024 will be a defining year for organizational AI-readiness. But the rush to be at the forefront of the AI wave is coupled with looming concerns. Can teams really trust AI? Initial evaluations have found that LLMs will surface false information backed by fabricated citations as fact – also known as “hallucinations.” This phenomenon led McKinsey to cite “inaccuracy” as the top risk associated with generative AI.
data.world’s benchmark report reveals 3x uptick in LLM response accuracy over SQL databases, when those answers were backed by a Knowledge Graph. Complex business questions around metrics, KPIs, and strategic planning have a much higher chance of accurate answers via the Knowledge Graph.
To date, it has not been understood to what extent LLMs can accurately answer complex business questions over SQL databases. Additionally, it has not been understood to what extent Knowledge Graphs improve the accuracy (and explainability) of LLMs to answer these questions. This report transforms that understanding.
“The main conclusion from our research is that investing in the Knowledge Graph provides much higher accuracy for LLM-powered question-answering systems on SQL databases. And ultimately, to succeed in this AI world, enterprises must treat the business context and semantics as a first-class citizen,” says Dr. Juan Sequeda, Head of the data.world AI Lab.
On the increased effectiveness of this LLM response strategy, data.world CEO Brett Hurt says: “The productivity lift of LLMs is absolutely obvious now as a result of all of the user experimentation and rapid development of them in 2023. The big challenge now is how corporations will leverage these new LLM-driven productivity boosts in a way where the results are accurate, explainable, and governed. Our benchmark report is the first of its kind and shows the power of LLMs + Knowledge Graphs serving as the ‘corporate brain’ to communicate back and forth with the LLM. The Knowledge Graph serves as a form of corporate memory and correcting the sometimes childish mistakes the LLM makes on such basic knowledge.”
dbt Labs validates the study
The results of the report were validated by members of dbt Labs’ developer experience team.
The results were promising. From the subset of eight addressable questions, they saw an 83% accuracy rate for natural language questions being answered via AI. This includes a number of questions that were correctly answered in 100% of attempts.
Jason Ganz of dbt Labs wrote: “On November 14th, Juan Sequeda and the data.world team dropped a bombshell paper that validates the intuition held by many of us - layering structured Semantic Knowledge on top of your data leads to much stronger ability to correctly answer ad-hoc questions about your organizational data with Large Language Models.”
Businesses can increase adoption of AI to drive efficiency
Overall, data.world’s benchmark report shows that organizations who fear the high cost of LLM inaccuracies in a business context can be optimistic. Knowledge Graphs should be part of their technical strategy for ensuring LLM response accuracy.
By achieving over 70% accuracy with the simplest queries - thanks to the integration with the Knowledge Graph - the industry is nearing a practical tool for generating initial responses to on-the-fly questions in real-world scenarios.
In the past year, macroeconomic shifts have propelled efficiency to the top of the agenda for organizational leaders and investors. Reports like this highlight AI's role in efficient budgeting and quicker ROI. Whether business leaders are looking to increase revenue, drive faster time-to-market with new features, or gather customer feedback with ease, there’s justification for AI to become its own budget line item.
The cost of ignoring AI is high, both for organizations and their leaders. But approaching AI-adoption the right way is critical. This report stands at the crossroads of technical know-how and domain expertise. It has forged a unified comprehension of the world today, and provided insight into the direction in which it’s going.
Download data.world’s full benchmark report here.
More information
- Learn more about building the foundation for scalable AI
- Listen to the research team in conversation on the details of this report and what it means in a business context
About data.world
data.world is the data catalog platform built for your AI future. Its cloud-native SaaS (software-as-a-service) platform combines a consumer-grade user experience with a powerful Knowledge Graph to deliver enhanced data discovery, agile data governance, and actionable insights. data.world is a Certified B Corporation and public benefit corporation and home to the world’s largest collaborative open data community with more than two million members, including ninety percent of the Fortune 500. Our company has 76 patents and has been named one of Austin’s Best Places to Work seven years in a row. Follow us on LinkedIn, Twitter, and Facebook, or join us.
Contact
[email protected]
