Graph Neural Networks: Foundations, Frontiers, and Applications First Edition Lingfei Wu
Graph Neural Networks: Foundations, Frontiers, and Applications First Edition Lingfei Wu
com
https://ebookmeta.com/product/graph-neural-networks-
foundations-frontiers-and-applications-first-edition-
lingfei-wu/
OR CLICK HERE
DOWLOAD NOW
https://ebookmeta.com/product/graph-neural-networks-in-action-meap-
version-4-keita-broadwater/
ebookmeta.com
https://ebookmeta.com/product/artificial-neural-networks-with-java-
tools-for-building-neural-network-applications-2nd-edition-igor-
livshin/
ebookmeta.com
https://ebookmeta.com/product/ise-ebook-online-access-for-essentials-
of-investments-12th-edition-bodie/
ebookmeta.com
https://ebookmeta.com/product/commando-club-curve-15-1st-edition-ja-
lafrance-2/
ebookmeta.com
https://ebookmeta.com/product/juno-loves-legs-karl-geary/
ebookmeta.com
https://ebookmeta.com/product/the-one-that-knows-no-fear-1st-edition-
steve-stred/
ebookmeta.com
https://ebookmeta.com/product/black-haze-violence-sacrifice-and-
manhood-in-black-greek-letter-fraternities-second-edition-ricky-l-
jones/
ebookmeta.com
Will There Be Tiers in Heaven Disability and the
Resurrection of the Body Nicola Santamaria
https://ebookmeta.com/product/will-there-be-tiers-in-heaven-
disability-and-the-resurrection-of-the-body-nicola-santamaria/
ebookmeta.com
Lingfei Wu · Peng Cui
Jian Pei · Liang Zhao Eds.
Graph Neural
Networks
Foundations,
Frontiers,
and Applications
Graph Neural Networks: Foundations,
Frontiers, and Applications
Lingfei Wu • Peng Cui • Jian Pei • Liang Zhao
Editors
123
Editors
Lingfei Wu Peng Cui
JD Silicon Valley Research Center Tsinghua University
Mountain View, CA, USA Beijing, China
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore
Pte Ltd. 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore
Foreword
“The first comprehensive book covering the full spectrum of a young, fast-growing
research field, graph neural networks (GNNs), written by authoritative authors!”
Jiawei Han (Michael Aiken Chair Professor at University of Illinois at Urbana-
Champaign, ACM Fellow and IEEE Fellow)
“This book presents a comprehensive and timely survey on graph representation
learning. Edited and contributed by the best group of experts in this area, this book
is a must-read for students, researchers and pratictioners who want to learn anything
about Graph Neural Networks.”
Heung-Yeung ”Harry” Shum (Former Executive Vice President for Technology
and Research at Microsoft Research, ACM Fellow, IEEE Fellow, FREng)
“As the new frontier of deep learning, Graph Neural Networks offer great potential
to combine probabilistic learning and symbolic reasoning, and bridge knowledge-
driven and data-driven paradigms, nurturing the development of third-generation
AI. This book provides a comprehensive and insightful introduction to GNN, rang-
ing from foundations to frontiers, from algorithms to applications. It is a valuable
resource for any scientist, engineer and student who wants to get into this exciting
field.”
Bo Zhang (Member of Chinese Academy of Science, Professor at Tsinghua Uni-
versity)
“Graph Neural Networks are one of the hottest areas of machine learning and this
book is a wonderful in-depth resource covering a broad range of topics and applica-
tions of graph representation learning.”
Jure Leskovec (Associate Professor at Stanford University, and investigator at
Chan Zuckerberg Biohub).
“Graph Neural Networks are an emerging machine learning model that is already
taking the scientific and industrial world by storm. The time is perfect to get in on the
action – and this book is a great resource for newcomers and seasoned practitioners
v
vi Foreword
alike! Its chapters are very carefully written by many of the thought leaders at the
forefront of the area.”
Petar Veličković (Senior Research Scientist, DeepMind)
Preface
The field of graph neural networks (GNNs) has seen rapid and incredible strides over
the recent years. Graph neural networks, also known as deep learning on graphs,
graph representation learning, or geometric deep learning, have become one of the
fastest-growing research topics in machine learning, especially deep learning. This
wave of research at the intersection of graph theory and deep learning has also influ-
enced other fields of science, including recommendation systems, computer vision,
natural language processing, inductive logic programming, program synthesis, soft-
ware mining, automated planning, cybersecurity, and intelligent transportation.
Although graph neural networks have achieved remarkable attention, it still faces
many challenges when applying them into other domains, from the theoretical un-
derstanding of methods to the scalability and interpretability in a real system, and
from the soundness of the methodology to the empirical performance in an applica-
tion. However, as the field rapidly grows, it has been extremely challenging to gain
a global perspective of the developments of GNNs. Therefore, we feel the urgency
to bridge the above gap and have a comprehensive book on this fast-growing yet
challenging topic, which can benefit a broad audience including advanced under-
graduate and graduate students, postdoctoral researchers, lecturers, and industrial
practitioners.
This book is intended to cover a broad range of topics in graph neural networks,
from the foundations to the frontiers, and from the methodologies to the applica-
tions. Our book is dedicated to introducing the fundamental concepts and algorithms
of GNNs, new research frontiers of GNNs, and broad and emerging applications
with GNNs.
The website and further resources of this book can be found at: https://
graph-neural-networks.github.io/. The website provides online preprints
and lecture slides of all the chapters. It also provides pointers to useful material and
resources that are publicly available and relevant to graph neural networks.
vii
viii Preface
To the Instructors
The book can be used for a one-semester graduate course for graduate students.
Though it is mainly written for students with a background in computer science,
students with a basic understanding of probability, statistics, graph theory, linear
algebra, and machine learning techniques such as deep learning will find it easily
accessible. Some chapters can be skipped or assigned as homework assignments for
reviewing purposes if students have knowledge of a chapter. For example, if students
have taken a deep learning course, they can skip Chapter 1. The instructors can also
choose to combine Chapters 1, 2, and 3 together as a background introduction course
at the very beginning.
When the course focuses more on the foundation and theories of graph neural net-
works, the instructor can choose to focus more on Chapters 4-8 while using Chapters
19-27 to showcase the applications, motivations, and limitations. Please refer to the
Editors’ Notes at the end of each chapter on how Chapters 4-8 and Chapters 19-27
are correlated. When the course focuses more on the research frontiers, Chapters
9-18 can be the pivot to organize the course. For example, an instructor can make
it an advanced graduate course where the students are asked to search and present
the most recent research papers in each different research frontier. They can also
be asked to establish their course projects based on the applications described in
Chapters 19-27 as well as the materials provided on our website.
To the Readers
This book was designed to cover a wide range of topics in the field of graph neu-
ral network field, including background, theoretical foundations, methodologies, re-
search frontiers, and applications. Therefore, it can be treated as a comprehensive
handbook for a wide variety of readers such as students, researchers, and profession-
als. You should have some knowledge of the concepts and terminology associated
with statistics, machine learning, and graph theory. Some backgrounds of the basics
have been provided and referenced in the first eight chapters. You should better also
have knowledge of deep learning and some programming experience for easily ac-
cessing the most of chapters of this book. In particular, you should be able to read
pseudocode and understand graph structures.
The book is well modularized and each chapter can be learned in a standalone
manner based on the individual interests and needs. For those readers who want
to have a solid understanding of various techniques and theories of graph neural
networks, you can start from Chapters 4-9. For those who further want to perform
in-depth research and advance related fields, please read those chapters of interest
among Chapters 9-18, which provide comprehensive knowledge in the most recent
research issues, open problems, and research frontiers. For those who want to ap-
ply graph neural networks to benefit specific domains, or aim at finding interesting
applications to validate specific graph neural networks techniques, please refer to
Chapters 19-27.
Acknowledgements
Graph machine learning has attracted many gifted researchers to make their seminal
contributions over the last few years. We are very fortunate to discuss the chal-
lenges and opportunities, and often work with many of them on a rich variety of
research topics in this exciting field. We are deeply indebted to these collaborators
and colleagues from JD.COM, IBM Research, Tsinghua University, Simon Fraser
University, Emory University, and elsewhere, who encouraged us to create such a
book comprehensively covering various topics of Graph Neural Networks in order
to educate the interested beginners and foster the advancement of the field for both
academic researchers and industrial practitioners.
This book would not have been possible without the contributions of many peo-
ple. We would like to give many thanks to the people who offered feedback on
checking the consistency of the math notations of the entire book as well as ref-
erence editing of this book. They are people from Emory University: Ling Chen,
Xiaojie Guo, and Shiyu Wang, as well as people from Tsinghua University: Yue He,
Ziwei Zhang, and Haoxin Liu. We would like to give our special thanks to Dr. Xiao-
jie Guo, who generously offered her help in providing numerous valuable feedback
on many chapters.
We also want to thank those who allowed us to reproduce images, figures, or data
from their publications.
Finally, we would like to thank our families for their love, patience and support
during this very unusual time when we are writing and editing this book.
ix
Editor Biography
xi
xii Editor Biography
Miltiadis Allamanis
Microsoft Research, Cambridge, UK
Yu Chen
Facebook AI, Menlo Park, CA, USA
Yunfei Chu
Alibaba Group, Hangzhou, China
Peng Cui
Tsinghua University, Beijing, China
Tyler Derr
Vanderbilt University, Nashville, TN, USA
Keyu Duan
Texas A&M University, College Station, TX, USA
Qizhang Feng
Texas A&M University, College Station, TX, USA
Stephan Günnemann
Technical University of Munich, München, Germany
Xiaojie Guo
JD.COM Silicon Valley Research Center, Mountain View, CA, USA
Yu Hou
Weill Cornell Medicine, New York City, New York, USA
Xia Hu
Texas A&M University, College Station, TX, USA
Junzhou Huang
University of Texas at Arlington, Arlington, TA, United States
Shouling Ji
xv
xvi List of Contributors
Title: QRM-Interplanetary
Language: English
Illustrated by Kolliker
The punched tape from Operator No. 7's machine slid along the line.
It entered a coupling machine and was stripped from the tape and
repunched upon a tape that was operating at better than a thousand
words per minute. Operator 7's tape then left the machine to be
rolled into a file roll and placed in the vaults below the city. It was of
no use save as a reference from now on.
The coupling machine worked furiously. It accepted the tapes from
seventy operators as fast as they could write them. It selected the
messages as they entered the machine, placing a mechanical
preference upon whichever message happened to be ahead of the
others on the moving tapes. The master tape moved continuously at
eleven hundred words per minute, taking teletype messages from
everywhere in the Northern Hemisphere of Venus to any of the other
planets in the Solar System. It was a busy machine, for even at
eleven hundred words per minute, it often got hours behind.
The synchronous-keyed signal from the coupling machine left the
operating room and went to the transmission room. It was amplified
and hurled out of the city to a small, squat building at the outskirts of
Northern Landing.
Here it was impressed upon a carrier wave and flung at the sky.
But not alone. Not unguarded. The upper half of the building carried
a monstrous parabolic reflector, mounted on gimbals. The signal was
focused into a beam. The beam was made of two components. The
center component was a circularly polarized, ultra-high frequency
wave of five centimeter waves, modulated with the keying signals of
the teletype coupling machine. The outer component was a radially
polarized wave of one centimeter waves. A radio frequency armor.
It was hurled at the sky, a concentric wave, out of a reflector, by a
thousand kilowatt transmitter. The wave seared against the Venusian
Heaviside Layer. The outer component bored at the ionization. It
chewed and it bit. It fought and it struggled. It destroyed ionization by
electronically shorting the ionization. And, as is the case with strife, it
lost heavily in the encounter. The beam was resisted fiercely.
Infiltrations of ionization tore at the central component, stripping and
trying to beat it down.
But man triumphed over nature! The megawatt of energy that came
in a tight beam from the building at Northern Landing emerged from
the Heaviside Layer as a weak, piffling signal. It wavered and it
crackled. It wanted desperately to lie down and sleep. Its directional
qualities were impaired, and it wabbled badly. It arrived at the relay
station tired and worn.
One million watts of ultra-high frequency energy at the start, it was
measurable in microvolts when it reached the relay station, only five
hundred miles above the city of Northern Landing.
The signal, as weak and as wabbly as it was, was taken in by eager
receptors. It was amplified. It was dehashed, de-staticked, and
deloused. And once again, one hundred decibels stronger and
infinitely cleaner, the signal was hurled out on a tight beam from a
gigantic parabolic reflector.
Across sixty-seven million miles of space went the signal. Across the
orbit of Venus it went in a vast chord. It arrived at the Venus
Equilateral Station with less trouble than the original transmission
through the Heaviside Layer. The signal was amplified and
demodulated. It went into a decoupler machine where the messages
were sorted mechanically and sent, each to the proper channel, into
other coupler machines. Beams from Venus Equilateral were
directed at Mars and at Terra.
The Terra beam ended at Luna. Here it again was placed in the two-
component beam and from Luna it punched down at Terra's Layer. It
emerged into the atmosphere of Terra, as weak and as tired as it had
been when it had come out of the Venus Layer. It entered a station in
the Bahamas, was stripped of the interference, and put upon the
land beams. It entered decoupling machines that sorted the
messages as to destination. These various beams spread out across
the face of Terra, the one carrying Korvus' message finally coming
into a station at Ten Mile Road and Woodward. From this station at
the outskirts of Detroit, it went upon land wires downtown to the
International Hotel.
The teletype machine in the office of the hotel began to click rapidly.
The message to Wilneda was arriving.
And fifty-five minutes after the operator told Korvus that less than an
hour would ensue, Wilneda was saying, humorously, "So, Korvus
was drunk again last night—"
There was some more silence. The communicator was turned off at
the other end, which made Channing fume. He would have preferred
to hear the interchange of words between his secretary and the
newcomer. Then, instead of having the man announced, the door
opened abruptly and the stranger entered. He came to the point
immediately.
"You're Don Channing? Acting Director of Interplanetary
Communications?"
"I am."
"Then I have some news for you, Dr. Channing. I have been
appointed Director by the Interplanetary Communications
Commission. You are to resume your position as Electronics
Engineer."
"Oh?" said Channing. His face fell. "I sort of believed that I would be
offered that position."
"There was a discussion of that procedure. However, the
Commission decided that a man of more commercial training would
better fill the position. The Communications Division has been
operating at too small a profit. They felt that a man of commercial
experience could cut expenses and so on to good effect. You
understand their reasoning, of course," said Burbank.
"Not exactly."
"Well, it is like this. They know that a scientist is not usually a man to
consider the cost of experimentation. They build thousand-ton
cyclotrons to convert a penny's worth of lead into one and one-tenth
cents' worth of lead and gold. And they use three hundred dollars'
worth of power and a million-dollar machine to do it with.
"They feel that a man with training like that will not know the real
meaning of the phrase, 'cutting expenses.' A new broom sweeps
clean, Dr. Channing. There must be many places where a man of
commercial experience can cut expenses. I, as Director, shall do so."
"I wish you luck," said Channing.
"Then there is no hard feeling?"
"I can't say that. It is probably not your fault. I cannot feel against
you, but I do feel sort of let down at the decision of the Commission.
I have had experience in this job."
"The Commission may appoint you to follow me. If your work shows
a grasp of commercial operations, I shall so recommend."
"Thanks," said Channing dryly. "May I buy you a drink?"
"I never drink. And I do not believe in it. If it were mine to say, I'd
prohibit liquor from the premises. Venus Equilateral would be better
off without it."
Don Channing snapped the communicator. "Miss Westland, will you
come in?"
She entered, puzzlement on her face.
"This is Mr. Burbank. His position places him in control of this office.
You will, in the future, report to him directly. The report on the
operations, engineering projects, and so on that I was to send in to
the Commission this morning will, therefore, be placed in Mr.
Burbank's hands as soon as possible."
"Yes, Dr. Channing." Her eyes held a twinkle, but there was concern
and sympathy in them, too. "Shall I get them immediately?"
"They are ready?"
"I was about to put them on the tape when you called."
"Then give them to Mr. Burbank." Channing turned to Burbank. "Miss
Westland will hand you the reports I mentioned. They are complete
and precise. A perusal of them will put you in grasp of the situation
here at Venus Equilateral better than will an all-afternoon
conference. I'll have Miss Westland haul my junk out of here. You
may consider this as your office, it having been used by Walters.
And, in the meantime, I've got to check up on some experiments
down on the ninth level." Channing paused, "You'll excuse me?"
"Yes, if Miss Westland knows where to find you."
"She will. I'll inform her of my whereabouts."
"I may want to consult you after I read the reports."
"That will be all right. The autocall can find me anywhere on Venus
Equilateral, if I'm not at the place Miss Westland calls."