I recently published an expanded benchmark of AI patent infringement detection tools in which I compared the performance of 6 vendors (Patlytics, PioneerIP, PatentWatch, Techson, ClaimHit, and IP Copilot) across 51 patents, examining how the tools differed in the number of leads generated, how aggressive or conservative the tools were in their ratings, how the vendors’ results correlated with each other, and how the results correlated with human analysis.
As I went to publication, one of the vendors, ClaimHit, indicated that they had just upgraded their tool from version V2 to a substantially enhanced version, ClaimHit V3. The founder, Bikramjit Singh, wasn’t particularly pleased about the timing of my article but I chose to proceed with publication, agreeing to subsequently retest the ClaimHit tool and discuss the enhancements with Bikram in an effort to learn specifically about how he evolved the ClaimHit tool. This article presents the retest results as well as the discussion with Bikram.
ClaimHit retest results
ClaimHit V3 returned 2,195 company-product reads against the portfolio, versus 293 for V2, a 7.5× increase. In the benchmark’s “gain/sensitivity” ranking, ClaimHit moves from the lowest-volume vendor (293, versus Patlytics 893, PioneerIP 786, IP Copilot 515, Techson 2,040, PatentWatch 7,980) to the second most prolific, just ahead of Techson. Given PatentWatch appeared to overproduce leads it seems fair to say that ClaimHit V3 produced the largest number of potentially viable leads after revision from V2.
The company-level footprint expanded roughly tenfold, from 152 distinct companies to 1,533. The tool still reports essentially one representative product per company on a given patent (only 2.8% of company-patent pairs carry more than one product), so the growth is across new companies rather than depth within product lines.
In terms of correlation with other vendors, 5.7× more of ClaimHit’s output is confirmed by at least one other vendor. The corroboration rate nevertheless falls from 33.8% to 25.5%, because the added volume is disproportionately vendor-unique: 74.5% of V3 reads are singletons no other vendor flags.
Before looking further at ClaimHit V3 let's step back and look at singleton reads across all six vendors, partly because the sheer number of singletons was discussed in the first article, and partly because it is tempting to treat a low singleton rate as a sign that a tool is doing a better job, which is an assumption worth testing.
ClaimHit V2 produced the least number of leads and also had the lowest percentage of singletons, which is less of a compliment than it sounds, as only 373 combinations in the entire study were flagged by two or more vendors, meaning PatentWatch could not have come in below about 95% no matter how good its reads were. Read down the table and you will see it is very nearly sorted by the number of reads each tool generated. Against our EoU work the rate fares no better, as Techson graded best of the six on the patents we verified, picking up 60% of the reads we confirmed as high likelihood while carrying a 93.2% singleton rate, and PatentWatch, with an even higher rate, found 6.7%. The rate simply cannot distinguish a tool that finds what the others missed from one that is generating noise.
What does survive is the lead-level version of the idea: reads that two or more tools flagged were rated high likelihood about half the time in our EoU work, versus 38% for reads only one tool found, so given two leads and the time to chase one, take the corroborated one. On that measure ClaimHit V3 did well, taking its corroborated count from 99 to 560 while its singleton rate rose only modestly, from 66.2% to 74.5%, and every other vendor’s singleton rate fell as a result, not because their results changed but because V3 corroborated reads no other tool had found.
As I suggested in the initial article, agreement between the vendors ranges from low to lousy, to use a non-statistical term. I’ve generally refrained from naming the specific tests, but the measure here is Krippendorff’s alpha, a reliability index where 1.0 is perfect agreement, 0 is what you’d get by pairing scores at random, and roughly 0.8 is the threshold for drawing firm conclusions. Applied to score strength on the combinations that two or more vendors flagged, the original benchmark came in at 0.183, indicating that the tools may occasionally land on the same target, but their confidence scores in it agree barely better than chance. Recomputing with ClaimHit V3 in place of V2 gives 0.022. I’d caution against reading much into that specific drop, since substituting V3 roughly doubles the corroborated set it’s computed over. The more stable observation is that the other five vendors alone reach only 0.252, and adding either version of ClaimHit pulls the panel down – V3 more so – because it scores most of what it finds as High, including targets its counterparts rated low. Where V3 did help is coverage, as it corroborated far more of the other tools’ reads, and every other vendor’s singleton rate fell as a result. But on strength-of-read agreement, it was near chance before and remains there. You still need to run multiple tools and compare results, or have humans rerun tools under expert guidance to get convergence and sort through the leads. No single tool, including ClaimHit V3, will provide a definitive list of leads.
An additional piece of data regarding the ClaimHit V3 performance relates to what it detected with regards to patent P6, which was a patent whose claims were developed with two particular products in mind. As such, we had definitive knowledge of the top two company-product leads. The table below shows the results from the benchmark with an additional column for ClaimHit V3.
The results are somewhat telling: ClaimHit V2 found neither of the top leads, but zeroed right in on one of them in V3. It also removed a lead, product BL2 from company BL, that we believed was not solid and had rated as low. Based on our scoring this resulted in the conversion of two red boxes to green, indicating a significant upgrade in performance, at least for this patent.
The ClaimHit story
As I mentioned I decided to use this update as an opportunity to learn more about this vendor and how their tool operates. Before readers start with accusations of favoritism let me clarify that I have no affiliations with any of the vendors, or any AI companies for that matter. I simply have a lot of respect for the hard-working entrepreneurs building these tools using the incredibly powerful yet quirky AI foundation models. So don’t confuse a little vendor love for favoritism or biased reporting.
Back to the ClaimHit story. ClaimHit’s founder, Bikramjit Singh (Bikram), is a telecommunications engineer and registered Indian Patent Agent who started in the patent field almost 20 years ago and has been the managing director at Scintillation Research and Analytics Services since 2010. He started ClaimHit about 2 years ago, and with the help of software developers developed the initial tool which is now on its third iteration.
ClaimHit is not selling licenses but is rather a pay-as-you-go platform in which customers can either purchase credits which they apply to portfolio analyses performed by themselves or can receive proposals for batch processing by ClaimHit at a set price . The company has not yet raised money but is contemplating doing so next year, largely to support marketing efforts.
With respect to the upgrade in the ClaimHit product Bikram indicated that with V2 they realized that the LLMs have “discovery issues” which they address in V3 by utilizing 6 distinct “channels” for identifying company/product combinations. Those channels are specifically:
Web search for indexed products
Semantic search, which matches on meaning rather than words, so a product surfaces even when it doesn’t use the patent’s vocabulary
Citation based search
Product library search for products not specifically indexed but which can be identified by crawling through product libraries
Competitor search, based on not only competitors of the patent holder (if relevant) but competitors of already identified company leads
Use of multiple LLMs and anti-hallucination agents to remove irrelevant items and for verification (present in V2)
For V3 channels 1-5 were added; V2 was already employing multiple LLMs. The retest shows a marked change in discovery, which Bikram attributes to these channels.
What does this tell us about the market for these tools?
The bottom line here is that ClaimHit was able to advance their tool significantly through the addition of channels 1-5 which sharply increased the number of company-product combinations they were able to surface (e.g. greatly enhanced discovery). Accuracy appears to have increased on the one patent where we had definitive knowledge, but nothing in this retest suggests that a single pass on any tool, ClaimHit V3 or any other platform, will provide a final curated list of company-product leads that human analysts can focus on.
As per the ClaimHit website you can run the tool yourself or send patent lists in to receive a proposal to obtain a “Hit Matrix.” The pricing to run the tool on a modestly sized portfolio such as the 51 patent set in this study seemed quite reasonable, and using ClaimHit to either verify the results of another tool or as a primary tool would seem entirely feasible. As I mentioned in my prior article, I have not done a detailed pricing comparison of these tools and am reluctant to do so because their business models and scope of services vary substantially.
Reasonable pricing aside, the question that remains for these tools is when they will begin to converge, or when the platforms will support fully iterative analysis that obviates the need to use multiple tools. I think the latter is more likely than the former: as costs drop (inevitable with all the competition in this market) I believe the game will change to tools that enable the analyst to guide the AI searches by supporting inspection of condensed claim charts or Evidence of Use (EoU) graphical mappings (as opposed to the verbose output currently being billed as “claim charts”) to refine the search. In other words, instead of having to run multiple tools and compile the results like we’ve done in our benchmarks, it should be possible to vary the searches, either manually or through automated adjustment of the “channels” as Bikram calls them. I envision products in which skilled patent analysts can rapidly assess graphic EoUs or condensed claim charts for high leads, rate them, and rerun the tool. Presumably the tool could learn what a “good” lead looks like for a particular patent as well as examining the false positives for that patent in more detail to refine the search.
Admittedly, some of the current tools like the Patlytics platform already allow the user to point to specific companies or products in the infringement search and support iterative searches. I have not focused on testing the tools in that manner, although it may be an area of future testing. It’s also unclear how platforms like the Techson Limestone tool could be used in iterative searches, since they generally receive a list of patents and provide a report on a single-pass basis. However, if the pricing on these services and tools starts dropping, I’m sure the vendors that stay in the game will be the ones that find ways to let their users re-run the tools to refine the output, rather than having them run off to other tools to verify the results.