Prediction of patent grant and interpreting the key determinants: an application of interpretable machine learning approach

Abstract

Patents are valuable intellectual property only when granted by the governments, and failing to receive an official grant means disclosing valuable technologies and information, which otherwise would be kept as commercial secrets. Yet, a typical patent application process takes years to complete and the outcome is uncertain. This study implements machine learning models to predict patent examination outcomes based on early information disclosed at patent publication and interpret the mechanism of how these models make predictions, highlighting the key determinants to patent grant and delineating the relationships between the patent features and the examination outcome. The predictive models that integrate patent-level variables with textual information accomplish the best prediction performances with a 0.854 ROC-AUC score and 77% accuracy rate. A number of interpretable machine learning methods are applied. The permutation-based feature importance metric identifies key determinants such as applicants’ prior experience, page length, backward citation, claim counts, number of patent family, etc. SHAP (SHapley Additive exPlanations), a local interpretability method, describes the marginal contributions to the model prediction of key predictors using two actual patent examples. Our study provides several valuable findings with important theoretical insights and practical applications. Specifically, we show that patent-level information can serve as a predictor of examination outcomes and the relationships between the predictors and outcome variables are complex. Knowledge accumulation and technology complexity positively affect the likelihood of patent grants, albeit with a curvilinear relationship. At lower levels, both factors significantly increase the chance of a grant, but beyond a certain threshold, the marginal effect becomes less pronounced. Additionally, prior experience, patent family size, and engagement with the patent agency have a monotonic and positive relationship with the grant likelihood, whereas the impact of patent scope on patent grants remains uncertain. While a narrower and more specific patent claim is associated with a higher grant rate, the number of claims increases it. Moreover, technology range, inventor team size, and examination duration have little effect on the patent grant results. From a practical standpoint, the accurate prediction of patent grants has significant potential applications. For instance, it could help firms better prioritize resources on the patent applications of high grant potentials to secure the final grant, as failure means a waste of R &D effort and disclosure of technology without IP protection. Additionally, patent examiners could utilize our predictive results as prior knowledge to enhance their judgment and accelerate the examination process.

Appendix

Table of literature review

Table 5 Summary of influencing factors to patent grant

Table 6 Literature review summary

Feature selection

While machine learning models are good at handling large dimensions of inputs variables, there is a consequence of using non-informative predictors in that they could add uncertainty to the predictions and reduce the overall effectiveness of the model (Kuhn & Johnson, 2013). Hence, the logic behind feature selection is that removing non-informative features could improve model prediction performance. We measure the informativeness using the difference between variable distributions of the granted and rejected groups. The intuition is that, the more different distributions of the focal variable between the granted and rejected are, the more informative that variable is. Thus, we calculate the differences between the pair for each feature and we believe the difference is proportional to the usefulness of features.

Fig. 6

The different between distribution of the granted and th rejected

The distribution distance are illustrated in Fig. 6. The blue density distribution indicates the granted sample while the yellow distribution indicates the rejected sample. The extent to which the two distributions differ indicates is the evidence of informativeness of that focal predictor.

We apply two well-established approaches to measure the distributional difference.

Kolmogorov–Smirnov test (KS test)

KS test is a form of minimum distance estimation used to compare two datasets. The KS statistic quantifies a distance between the empirical distribution functions of two groups of samples. The two-sample KS test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. The KS test is based on the maximum distance between these two curves produced by two empirical cumulative distribution functions. The KS test statistic is defined as:

$$\begin{aligned} D_n = \max _{1 \le x \le N} (F_a(Y_x) = F_b(Y_x)) \end{aligned}$$

(3)

where F is a theoretical cumulative distribution of the distribution being tested which must be a continuous distribution. The statistic evaluates the greatest separation between the cumulative distribution $F_a$ and $F_b$. When $F_a \ne F_b$, larger values of D are expected.

Jensen–Shannon divergence

JS divergence is based on Kullback–Leibler (KL) Divergence divergence, a method of measuring statistical distance sometimes referred to as relative entropy. KL calculates a score that measures the divergence of one probability distribution from another. It can be used as a distance metric that quantifies the difference between two probability distributions. If $F_a$ and $F_b$ represent the probability distribution of a discrete random variable, the KL divergence is calculated as a summation:

$$\begin{aligned} KL(F_a || F_b) = \sum F_a(x)log \Big (\frac{F_a(x)}{F_b(x)} \Big ) \end{aligned}$$

(4)

The intuition for the KL divergence score is that when the probability for an event is large, but the probability for the same event is small, there is a large divergence. When the probability from is small and the probability from is large, there is also a large divergence, but not as large as the first case.

JS divergence extends KL divergence to calculate a symmetrical score and distance measure of one probability distribution from another. The JS divergence can be calculated as follows:

$$\begin{aligned} JS(F_a || F_b) = \frac{KL(F_a ||F_M)}{2} + \frac{KL(F_b||F_M)}{2} \end{aligned}$$

(5)

where $F_M$ is calculated as:

$$\begin{aligned} F_M = \frac{F_a + F_b}{2} \end{aligned}$$

(6)

Based on both information from the KS and JS statistic measurement of the distributional difference between the granted and rejected, we select the relatively more informative features as predictors of our machine learning models.

The permutation feature importance

Permutation feature importance measures the increase in the prediction error of the model after the features’ values are permuted. The idea is intuitive—if shuffling a feature value increases the model error more, then the model relied on this feature for the prediction to a greater degree. A feature is hence “unimportant” if shuffling its values leaves the model error unchanged because, as we can interpret, the model ignored the feature for the prediction. This method was initially introduced Breiman (2001) for random forest, and extended to a model-agnostic version by Fisher et al. (2019). The permutation feature importance algorithm we use was based on Fisher et al. (2019):

Input: trained model ${\hat{f}}$, feature matrix X, target vector y, error measure $L(y, {\hat{f}})$.

• 1. Estimate the original model error $e_{orig} = L(y, {\hat{f}}(X))$(e.g. R-square or mean squared error)

• 2. For each feature $j \in \{1,\ldots , p\}$ do:

  • Generate feature matrix $X_{perm}$ by permuting feature j in the data X. This breaks the association between feature j and true outcome y.

  • Estimate error $e_{perm} = L(Y, {\hat{f}}(X_{perm}))$ based on the predictions of the permuted data.

  • Calculate permutation feature importance as quotient $FI_j = \frac{e_{perm}}{e_{orig}}$ or difference $FI_j = e_{perm} - e_{orig}$

• 3. Sort features by descending FI

In practice, we use the $vi\_permute$ command in the R package vip to generate feature importance for XGBoost, random forest, SVM, LASSO, and Logit.

The partial dependence plots

The SHAP dependence plot shows a fitted curve for a scatter plot of the SHAP values and the actual feature values, illustrating the relationship between the two, whether it is a linear, monotonic, or more complex relationship.

Fig. 7

The partial dependence plots of major variables

The more traditional way to present the interrelation as a visualization tool is the Partial Dependence Plot (PDP) proposed by Friedman (2001), which is able to illustrate how each variable affects the chance of patent grant. The difference between the SHAP dependence plot and PDP is that first, the y-axis of the SHAP dependence plot is the SHAP value (or marginal contribution) while the y-axis of PDP is the average prediction. Thus, the y-axis in the SHAP dependence plot is centered on zero, while it is centered on 0.46 (the average grant ratio) for PDP. Second, SHAP dependence is able to present more information, such as the distribution of individual samples. Yet, PDP is a more well-accept and applied tool for model interpretation. We apply PDP for the robustness of our results.

Given the output function f(x) of machine learning algorithm, the partial dependence of f on variable $X_s$ is defined as (let c be the complement set of s )

$$\begin{aligned} f^s(x_s) = E_{X_c}[f(x_s, X_c)] = \int f(x_s, x_c)dP(x_c) \end{aligned}$$

(7)

where the PDP $f^s$ is the expectation of f over the marginal distribution of all variables except for $x_s$. In practice, PDP is estimated by taking average over the training data with fixed $X_s$

$$\begin{aligned} {\bar{f}}^s(x_s) = \frac{1}{n} \sum ^n _i = 1 f(x_s, X^i_c)) \end{aligned}$$

(8)

PDP plots indicate direct causal effect of a variable s on the outcome variable if the back-door condition is satisfied (Zhao and Hastie, 2021). We admit that some of the variables are likely endogenous or descendants of other variables in the terminology of the directed acyclic graph (DAG). For instance, the variable duration, the time duration between application and publication, can be affected by other confounding variables. Though may not be explained in a causal manner, nonetheless many of the high correlation nature are worth pointing out.

Figure 7 illustrates 12 PDP plots for the major predictors. To keep the plots comparable, we limit the y-axis ranging from 0.4 to 0.6 and the x-axis according to the distribution of each predictor. Note that from Panel (a) to (f), the PDP generates similar relational patterns between the predictors and output variable as ones from Fig. 5. Backward citation, patent length, number of claims, length of the first claim, grant rate of the previous year, and patent family all show up a similar pattern of positive correlation with the outcome variable. For backward citation, patent length, and the number of claims, there exists a threshold effect. Once beyond 4 for citing, 20 for pages, and 10 for $right\_no$, the marginal benefits of these variables stay positive and constant. While $nchar\_right$, $app\_rate2010$, and nfamily all show up a monotonic increasing relationship with the outcome variable. Hence, PDP confirms the relationships of the predictors and outcome variables as suggested and discussed in our main paper.

Patent length or pages increase the likelihood of a patent grant. There are notable data outliers with one or two pages, and these are patents with an above-average grant rate. It is also present in Panel (b) of Fig. 5 that there are four blues dots on the upper left corner. There are possible data anomalies but it does not interfere the model interpretation.

Besides the six major variables, we also explore other less important yet noteworthy predictors. The agent success rate in the previous year, $agent\_rate2010$, increases the likelihood of patent grant, suggesting relying on a high-quality patent agent could help the examination outcome. Agent application load, $agent_freq$, initially increases but eventually decreases the patent grant rate. Application frequency in the same year, $app\_freq$, could lower the outcome variable because too many applications can be distracting. Panel (i) shows that the duration between application and publication has a complex relationship with the outcome variable. Panel (k) shows that Inventor numbers have little effect on the likelihood of patent grant. There is a slight increase for a higher number of inventors. Panel (l) shows the length of the abstract, $nchar\_ab$, which increases the patent grant because it may indicate the technology complexity similar to patent length.

It is also interesting to note, Panel (m) and (o) show forward citation and the number of IPC assigned have little predicting power of whether a patent gets granted or not.