Scribd
Upload
162 views19 pages

A Biterm Topic Model For Short Texts Slide

This document proposes a Biterm Topic Model (BTM) to address challenges in modeling topics for short texts. BTM directly models word co-occurrence patterns through "biterms" (word pairs that co-occur in documents) rather than word counts. This exploits rich global word co-occurrence information to overcome data sparsity issues in short texts. Experiments on tweet, question, and newsgroup collections show BTM produces more coherent topics and better represents document content than conventional topic models. The authors suggest future work to improve inferring topic proportions for short documents and applying BTM in real-world applications.

Uploaded by

son070719969
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
162 views19 pages

A Biterm Topic Model For Short Texts Slide

This document proposes a Biterm Topic Model (BTM) to address challenges in modeling topics for short texts. BTM directly models word co-occurrence patterns through "biterms" (word pairs that co-occur in documents) rather than word counts. This exploits rich global word co-occurrence information to overcome data sparsity issues in short texts. Experiments on tweet, question, and newsgroup collections show BTM produces more coherent topics and better represents document content than conventional topic models. The authors suggest future work to improve inferring topic proportions for short documents and applying BTM in real-world applications.

Uploaded by

son070719969
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

A Biterm Topic Model for

Short Texts
Xiaohui Yan, Jiafeng Guo, Yanyan Lan, Xueqi Cheng
Institute of Computing Technology,
Chinese Academy of Sciences

1
Short Texts Are Prevalent on Today's Web

2
Background
Understanding the topics of short texts is
important in many tasks
content characterizing
content recomendation
user interest profiling
emerging topic detecting
semantic analysis
...

3
Topic Models

From Blei

Model the generation of documents with latent topic structure


a topic ~ a probability distribution over words
a document ~ a mixture of topics
a word ~ a sample drawn from one topic

Previous studies mainly focus on normal texts


4
Problem on Short Texts: Data Sparsity
Word counts are not discriminative
normal doc: topical words occur frequently
short doc: most words only occur once

Not enougth contexts to identify the senses of


ambiguous words
normal doc: rich context, many topic-related words
short doc: scarce context, few topic-related words

5
Previous Approaches
LDA with document aggregation
e.g. aggregating the tweets published by the same user
heuristic, not general

Mixture of unigrams
each document has only one topic
too strict assumption, result in peaked posteriors P(z|d)

Sparse topic models


each dcoument maintains a sparse distribution over topics, e.g.
Focused Topic Models
too complex, easy to overfitting

6
Key Ideas
Topics are basically groups of correlated words and
the correlation is revealed by word co-occurrence
patterns in documents
why not directly model the word co-occurrences for topic
learning?

Topic models on short texts suffer from the problem of


severe sparse patterns in short documents
why not use the rich global word co-occurrence patterns
for better revealing topics?

7
Biterm Topic Model (BTM)
BTM models the generation of word co-occurrences in a
corpus
A biterm is an unordered word pair co-occurring in the same
short context (document)
Training data includes all the biterms in the corpus

Generative description

8
Biterm Topic Model (BTM)

Model the generation of biterms with latent topic structure


a topic ~ a probability distribution over words
a corpus ~ a mixture of topics
a biterm ~ two i.i.d sample drawn from one topic 9
Inferring Topics in a Document
Assumption
the topic proportions of a document equals to the expectation
of the topic proportions of biterms in it

where

10
Parameters Inference
Gibbs Sampling 700
LDA
600
BTM
sample topic for each biterm

Time cost (s/iteration)


500
400
300
200
100
0
50 100 150 200 250
Topic number K

parameters estimate 14000

12000 LDA
BTM
10000

Memory(M)
8000

6000

4000

2000

0
50 100 150 200 250
Topic Number K

11
Experiments: Datasets

Tweets2011 Question 20Newsgroup


(short text) (short text) (normal text)
#documents 4,230,578 189,080 18,828

#words 98,857 26,565 42,697

#users 2,039,877 - -

#categories - 35 20
avg doc length 5.21 3.94 97.20

12
Experiments: Tweets2011 Collection
Topic quality
evaluation metric: average coherence score
(Mimno'11) on the top T words
A larger coherence score means the topics are more coherent

D. Mimno, H. Wallach, E. Talley, M. Leenders, and A. McCallum. Optimizing semantic oherence 13


in topic models. EMNLP 2011
Experiments: Tweets2011 Collection

The colored words are irrelevant judged by human

14
Experiments: Tweets2011 Collection
Quality of topical representation of documents (i.e.
P(z|d))
select 50 most frequent and meanful hashtag as class labels
organize documents with the same hashtag into a cluster
measure: H score
smaller value indicates better agreement with human labeled classes

15
Experiments: Question Collection
Evaluated by document classification (linear SVM)

16
Experiments: 20Newsgroup Collection
(Normal Texts)
Biterm extraction
two words co-occur within a context window with range no larger
than a threshold r

clustering result

17
Summary
A practical but not well-studied problem
topic modeling on short texts
conventional topic models suffer from the severe data sparsity

A novel way: Biterm Topic Model


model word co-occurrence to uncover topics
fully exploit the rich global word co-occurrens
effective on short texts (and normal texts)

Furture works
better way to infer topic proportation for short text documents
explore BTM in real-world applications
18
Thank You

19

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy