Meta Analysis

BERT
• BERT (Bidirectional Encoder Representations from Transformers)
• It is a Natural Language Processing Model proposed by researchers at Google Research in 2018.
• It define the object is similar or not based on their features by creating vector.
• It can look at the context of the statement and generate the meaningful number representation
for a given word
• Can also generate embedding for entire sentence.

• Sci BERT :- BERT model trained on scientific text.

2
Multinomial Naive Bayes
➢ Multinomial Naive Bayes (MNB) is a popular machine learning algorithm for text classification
problems in Natural Language Processing (NLP).
➢ MNB works on the principle of Bayes theorem and assumes that the features are conditionally
independent given the class variable.
➢ Formula:-
P(A|B) = P(A) * P(B|A)/P(B)
Where we are calculating the probability of class A when predictor B is already provided.
P(B) = prior probability of B
P(A) = prior probability of class A
P(B|A) = occurrence of predictor B given class A probability

3
Project Outline
1.Data Preprocessing:
➢ Load dataset (abstracts labeled as "Included" or "Excluded").
➢ Handle missing value.
➢ Tokenize the abstracts using the SciBERT tokenizer. This will convert sentences into token
IDs which the model can understand.
➢ Split the data into train and test sets.
2. Model Creation:
➢ Load the pretrained SciBERT model.
➢ Add a classification layer on top. SciBERT outputs embeddings for each token in the input. We can
take the embedding of the [CLS] token (a special token indicating the start of a sequence in BERT-
based models) as the representation of the entire sequence and use it for classification
4
Project Outline
3. Model Creation:
➢ Load the pretrained SciBERT model.
➢ Add a classification layer on top. SciBERT outputs embeddings for each token in the input. We can
take the embedding of the [CLS] token (a special token indicating the start of a sequence in BERT-
based models) as the representation of the entire sequence and use it for classification
4.Model Training :
➢ Train the model on training data while validating on the validation set.
➢ Monitor for overfitting and use techniques like dropout or early stopping if needed.

5
Project Outline

5. Model Evaluation:
➢ After training, evaluate the model's performance on a separate test set (if available).
➢ Use metrics like accuracy, F1-score, precision, recall, and ROC-AUC to understand model
performance.

6
 Accuracy:
• Definition: It's the fraction of predictions our model got right.
• Formula: Accuracy=Number of correct predictions / Total number of predictions
• When to use: Accuracy is a suitable measure when the class distribution is roughly equal.
However, it might not be the best metric when there's a class imbalance.
Precision:
• Definition: Of all the instances predicted as positive, how many were actually positive.
• Formula: Precision=True Positives (TP) / (True Positives (TP) + False Positives(FP))
• When to use: Precision is crucial when the cost of a false positive is high. For example, in
email filtering, wouldn't want an important email (a real positive) to be mistakenly classified
as spam (a false positive).

7
 Recall (or Sensitivity or True Positive Rate):
• Definition: Of all the actual positive instances, how many were predicted as positive.
• Formula: Recall=True Positives (TP) / True Positives (TP) + False Negatives (FN)
• When to use: Recall is vital when the cost of a false negative is high. For instance, in
disease diagnosis, you wouldn't want a sick patient (a real positive) to be mistakenly
classified as healthy (a false negative).
F1-Score:
• Definition: The harmonic mean of precision and recall, which provides a balance
between the two. It's especially useful in situations where one measure is more
important than the other.
• Formula: F1-Score= (2*TP) / (2*TP + FN + FP)
• When to use: It's useful when there's an uneven class distribution and the model's
performance on the smaller class is more critical.
8
MetaAnalysis

▪ A meta-analysis is a statistical analysis that combines the results

of multiple scientific studies.
▪ Researchers use meta-analyses when there are multiple studies
on the same topic.
▪ Meta-analyses are usually conducted using either ‘Fixed-effects’
or ‘Random-effects’ models depending on the type of data,
research question, and assumptions about the variability in effect
sizes across studies.

2
Fixed-effects and random-effects models

➢ Effect models in meta analysis are statistical methods that combine the results of different studies to
estimate a common or average effect size. There are two main types of effect models: fixed-effect and
random-effects.
➢ Fixed-effects and random-effects models are the most commonly employed statistical models for
meta-analysis
➢ The decision to use one statistical model or another is complex and often subjective; however, there
are criteria that can guide decisions about which model to use.
➢ The default model for meta-analysis in reviews should be the random-effects model

5
Fixed-effect model
• Under the fixed-effect model we assume that there is one true effect size that underlies all the studies
in the analysis, and that all differences in observed effects are due to sampling error
• Results apply only to studies included in meta-analysis.
• It is also known as common-effect model.
• fixed-effect model is estimating a common mean
• The fixed-effects model is the appropriate model when the number of studies is small. It was
suggested that the fixed-effects model should be used when the number of studies included in a
meta-analysis is less than five
• If there is statistical heterogeneity among the effect sizes, then the fixed-effects model is not
appropriate
• Under the fixed-effect model the weights are based solely on the within-study variances

6
Some examples of fixed effect models are
➢ A meta-analysis of five studies that used the same drug, dose, researchers, and
recruitment criteria to measure the effect of the drug on blood pressure. The fixed effect
model would estimate the average effect of the drug on blood pressure for this
population1
➢ A meta-analysis of four studies that measured the mean score on a science aptitude
test for freshmen at a specific college. The fixed effect model would estimate the
common mean score for this college2
➢ Ameta-analysis of three studies that compared the effectiveness of two teaching
methods on student achievement in math. The fixed effect model would estimate the
common difference in achievement between the two methods for this subject

7
When should we consider using a fixed
effects model ?

• The studies or units of analysis are functionally identical : If the studies included in analysis are
essentially the same in all important aspects, a fixed effects model may be appropriate

• There is homogeneity among the studies : If there is no significant variation or heterogeneity among
the effect sizes of the studies included in analysis, a fixed effects model can be used.

• The goal is to estimate a common effect size : If our research question is about estimating the
common effect size for a specific population, rather than generalizing to a larger population, a fixed
effects model would be suitable.

8
Random-effects model
• Under the random-effects model the goal is not to estimate one true effect, but to estimate the mean
of a distribution of effects.
• Results apply beyond included studies.
• The random-effects model should be considered when it cannot be assumed that true homogeneity
exists.
• The random-effects model is estimating the grand mean.
• A random-effects model assumes each study estimates a different underlying true effect.
• The default model for meta-analysis in reviews should be the random-effects model.
• Random-effects models are appropriate when the number of studies is large enough.
• It is also known as Practical-effect model.

9
Some examples of random effect models are:
➢ A meta-analysis of studies that measured the effect of a drug on blood pressure in
different countries. The random effect model would estimate the mean and variance of
the drug effect across all possible countries, and allow for heterogeneity among the
studies3.
➢ A study of students’ test scores in different schools and classrooms. The random effect
model would estimate the mean and variance of the test scores across all possible
schools and classrooms, and account for the nested structure of the data4.
➢ Astudy of plant growth in different plots and treatments. The random effect model
would estimate the mean and variance of the plant growth across all possible plots and
treatments, and capture the random variation due to environmental factors.

10
When should we consider using a random
effects model ?
• The studies or units of analysis are a random sample from a larger population : If you are interested
in generalizing your findings beyond the specific studies included in your analysis, a random effects
model may be appropriate. This model assumes that the studies are a random sample from a larger
population of studies, and that there is variation in the effect sizes across this population.

• There is heterogeneity among the studies: If the studies included in your analysis are not functionally
identical, and there is significant variation or heterogeneity among their effect sizes, a random effects
model can account for this heterogeneity.

• The data are hierarchical or clustered: If your data have a nested or hierarchical structure (for
example, students nested within classrooms nested within schools), a random effects model (also
known as a multilevel or hierarchical linear model) can be considered.

11
ComparisonBetweenFixedandRandomEffect Model

Fixed Effect Random Effects

▪ Single true effect size that underlies all ▪ True effect size can vary from study to study.
studies.
▪ The variance is due to sampling error and
▪ The variance is due to sampling error only. between-study heterogeneity.
▪ Studies are weighted more heavily if they ▪ Studies are weighted more evenly, regardless
have larger sample sizes. of sample size.
▪ Confidence intervals are narrower than in ▪ Confidence intervals are wider than in fixed
random effects meta-analysis. effects meta-analysis.
▪ Suitable when there is little or no ▪ Appropriate when there is likely more
heterogeneity among the included studies. heterogeneity among the included studies.

5
Heterogeneity

▪ Heterogeneity in meta-analysis refers to the variation in study

outcomes between studies.
▪ The decision to use a random-effects or fixed-effects model is usually
based on the degree of heterogeneity between the individual studies.
▪ If the estimates of the outcome of interest are approximately the same
across all studies the between-study-heterogeneity is said to be low and
it is therefore appropriate to use a fixed-effect model.
▪ If the individual studies show a wide variation in the outcome estimates
(heterogeneity) then a random-effects model is appropriate.

6
Effect Size
• Effect sizes are the raw data in meta-analysis studies because they are standardized and easy to
compare
• Effect size tells you how meaningful the relationship between variables or the difference between
groups is. It indicates the practical significance of a research outcome.
• A large effect size means that a research finding has practical significance, while a small effect size
indicates limited practical applications.
• For instance, if we have data on the height of men and women and we notice that, on average,
men are taller than women, the difference between the height of men and the height of women is
known as the effect size. The greater the effect size, the greater the height difference between
men and women will be.
• In Meta-analyis, effect size is concerned with different studies and then combines all the studies
into single analysis

2
Cochran’s QTest

▪ Heterogeneity is usually assessed by using a Cochran’s chi-squared test or Cochran's Q test and P-
values of less than 0.1 (and not the usual 0.05) indicate significant heterogeneity.
▪ Cochran’s Q test is the traditional test for heterogeneity in meta- analyses. Based on a chi-square
distribution, it generates a probability that, when large, indicates larger variation across studies rather
than within subjects within a study.
▪ Cochran’s Q test underlay two hypothesis–
▪ Null hypothesis (H0): the treatments are equally effective.
▪ Alternative hypothesis (Ha): there is a difference in effectiveness between treatments.

7
Cochran’s QTest

▪ The Cochran's Q test statistic is

▪ Where
▪ k is the number of treatments
▪ X• j is the column total for the jth treatment
▪ b is the number of blocks
▪ Xi • is the row total for the ith block
▪ N is the grand total
8
How do you know if an effect size is small or large?

Effect sizes can be categorized into small, medium, or large according to Cohen’s criteria.

Cohen’s criteria for small, medium, and large effects differ based on the effect size measurement used.

Cohen’s d can take on any number between 0 and infinity, In general, the greater the
Cohen’s d, the larger the effect size.

4
Thankyou