Comparing DWPI abstracts to generative AI summaries: You asked, and we answered
Reliable insights require quality monitoring and a high level of accuracy.
New generative artificial intelligence (AI) and large language model (LLM) technologies offer new possibilities for patent researchers. They can create workflow efficiencies for those who want to capture quick insights from the ever-expanding volume of global patent publications.
As generative AI becomes more widely adopted, some patent searchers are exploring LLMs, such as ChatGPT, to summarize patent publications. Our community asked us how Derwent World Patents Index™ (DWPI™) abstracts compare to those that can be created with generative AI or LLMs. As transparency is a core value at Clarivate™, we answer when asked.
Patent summaries created by generative AI or LLMs differ from those made by the DWPI team of 900+ subject matter experts in three key ways: accuracy, consistency and scale.
Accuracy
The key to obtaining an accurate patent summary using an LLM depends highly on the prompt and technical comprehension of the prompt writer. These models generate text via prediction. Unfortunately, this means hallucinations and inaccurate material may appear within otherwise sensible-looking content. Moreover, LLMs are not designed specifically for patent abstraction. There are no built-in quality checks to ensure the created abstract accurately summarizes the patent’s claims.
In contrast, DWPI abstracts are written by our team of subject matter experts and undergo systematic reviews and quality checks, along with statistical sampling, to ensure they are more accurate.
How DWPI achieves 98.5% accuracy
Of the 45,000 abstracts we create each week, a statistically significant sample is selected randomly for a quality check, with a certain number coming from various priority jurisdictions, technology classes and editorial teams. Abstracts selected for sampling are then evaluated for 13 types of potential errors.
The current defect rate observed is less than 1.5%. For example, for 100 DWPI records, we would check for up to 1,300 potential errors. With the current defect rate, we would expect to find fewer than 20 errors out of this potential 1,300.
We continue to narrow the gap by implementing corrective action to address the errors observed. By the end of 2024, we expect DWPI abstracts to achieve 99%+ accuracy.
Our customers repeatedly tell us that accuracy is the most valuable component of DWPI. Ensuring that the abstract precisely describes an invention’s claims is necessary, as this content supports stakeholders’ decisions about novelty, patentability and clearance.
Consistency
When asking an LLM tool such as ChatGPT to summarize a patent, the same prompt could yield different outcomes when run later due to the continuously augmenting nature of underlying models.
This variability can occur because a given model is frequently updated with new data and improvements, which can change how it interprets and responds to prompts. This unpredictability can be hugely problematic for users who require reliable and repeatable results. For example, different team members may get different answers to the same question, which may require additional time to investigate and resolve the differences.
To address this issue, users may attempt to save responses or abstracts created using generative AI for future reference, which becomes difficult to manage at scale or across teams.
DWPI uses editorial processes to identify an invention’s novelty, use and advantage consistently and accurately. Our expert team and commitment to quality through statistical sampling promotes consistent accuracy for all stakeholders. As a result, we’ve created a stable repository of over 62m+ indexed invention abstracts. DWPI is also updated as new patents are published.
Scale
Generative AI tools that deliver abstracts one at a time based on user input have another crucial disadvantage relative to DWPI. Whereas generative AI abstracts are typically created one at a time or in batches “on command,” DWPI provides an always available, fully searchable database of 62 m+ invention abstracts written in standardized English.
Because of this, DWPI is an extremely useful tool for Boolean keyword patent searches. A patent search performed with DWPI will often yield more relevant results than a patent search performed against the original (or translated) patent full text.
With descriptive titles and structured data fields for novelty, use, and advantage, DWPI also makes it easy to review results quickly without waiting for an abstract to be generated. This helps stakeholders save time reviewing patents without having to wait for generative AI to summarize them.
Learn more about DWPI across the years
How generative AI can help patent researchers
Undoubtedly, generative AI can be a valuable tool for patent researchers. These models can help the user explore questions that would not be covered by a DWPI structured field — such as, “What are some ways this invention could be used that aren’t claimed by the patent?” — alongside the capability of providing an entry-level summary of a collection of documents.
For these types of questions, generative AI can be useful for idea generation or helping the patent researcher craft their analysis. However, the outputs of generative AI should be considered a starting point that always warrants further analysis and verification using other sources.
At Clarivate, we’re developing solutions that incorporate generative AI (and other AI technologies) to support many of these use cases, prioritizing the problems facing our customers that can be reliably served with AI technology.
Still, when it comes to Boolean patent searching and rapid review of search results, today there is no comparable alternative to the DWPI repository of 62m+ invention abstracts created by subject matter experts.
Want to learn more? Contact us and one of our DWPI specialists will reply within 24 hours.
Related insights
The latest news, technologies, and resources from our team.
