On-device AI and its thermal

implications

Nader Nikfar, Sr. Dir. of Technology

Qualcomm Technologies, Inc. HotChips, Aug. 2024

Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
1. What is on-device AI?

Agenda
2. Benefits & importance
3. Use-case evolution
4. Thermal Implications
5. Potential solutions
6. Summary
What is
on-device AI?

3
AI is transforming …

4
Intelligence is moving towards edge devices
On-device and AI go hand in hand

• Running machine learning on devices

like smartphones, laptops, and cars,
instead of cloud.
• On-device AI utilizes on-chip processors
such as NPU, CPU, and GPU.
• Current on-device AI can support
inferences on multiple platforms.
• Cloud is currently necessary for pooling
of big data and training AI inference
algorithms, and it complements on-
device processing. Devices, machines, and things are becoming more intelligent

SOC consist of several processors integrated into the same die

5
Al apps enabled by on-device Generative AI

6
 Benefits
&
Importance

7
Advantages of on-device inference

8
• Inference running entirely in the
cloud will have issues for real-time
applications that are latency-
sensitive and mission-critical like
autonomous driving. Automotive
• Such applications cannot afford
Gen AI can be used for ADAS/AD
the roundtrip time or rely on critical
to help improve drive policy by
functions to operate when in predicting the trajectory and
variable wireless coverage. behavior of various agents
Use-Case
Evolution

10
 2015 2016-2022 2023 2023+

Use Case
Audio/ Camera
Speech Video
LLM-powered assistants

Audio/ Multi-modal gen

Speech Stable Diffusion/ControNet AI models

Micro Tile Multi-Modal AI

Hardware Scalar Vector Tensor Scalar Vector
Inferencing
Tensor Scalar Vector

Transformer Support
Transformer Support

Models Simple CNN Transformer / LSTM/ 10B LLMs / LVMs/LMMs 10B++LLMs/LVMs/LMMs

RNN/CNN
12
Thermal
Implications

13
 Devices

Thermal implications:
• Workload dependent
• Device/platform thermal solution & its required boundary conditions
• Hardware configuration
14
 10 billion parameter mark
On device Cloud

As model complexities grow, potential constraints in processing Almost no constraints in processing

power, memory, and power. power, memory, and power.

HI

Active MID
Cooling
Load
LOW
OFF
 Do we have a thermal problem now?

LLM model with billions of parameters run within thermal limits on mobile and laptops
16
Potential
Solutions

17
Improved performance & power consumption
Most impactful approach for thermal

18
Sustained NPU Performance
One hour of continuous NPU usage

>5X
higher performance
at lower thermals
Intel Core Ultra 7 155H

19
Thermal Mitigation

• If necessary, mitigation policies similar to what has been utilized for CPUs will be applicable.
• In some platforms and depending on the workload, latency in completing the task may not be
as time-sensitive.
• Mobile devices can accommodate many useful Gen-AI workloads despite thermal
constraints.

20
Hybrid AI
Distributes and coordinates AI workloads among cloud and edge devices where and when appropriate

Energy-synergy:
- If task is determined too complex for
device, it will be offloaded to cloud avoiding
processing on-device.
- If task is fit for on-device, no inference in
the cloud, hence total energy consumption is
lower
• Eliminates round trip communication to the cloud +
inference on-device consumes much less power +
offloads datacenter energy consumption to meet their
environmental and sustainability goals.

21
 Packaging Innovations

Chiplet Package Integration

and/or

2.5D 3D
Active Die

   Package Technology Integration → → →

More-than-Moore advances will contribute to improved performance/watt

22
Continued AI Research
Research is in our DNA

23
Summary

• On-device AI provides low latency, high performance, lower power

consumption, and reliability.
• Power and thermal efficiency are essential for on-device AI.
• On-device AI still evolving; hard to predict how models will evolve
in complexity to accurately predict thermal bottlenecks across
mobile platforms.
• Current on-device AI can support inferences on multiple platforms.
• As AI model complexities grow, tight power and thermal constraints
within mobile devices will drive innovation.
• Continued AI research and engineering developments will lead to
further efficiency optimization (Lower power, more inference). As
such:
• Microarchitecture improvements
• Technology node advances
• Powerful GenAI models become more efficient
• Hybrid-AI

24
Thank you
Nothing in these materials is an offer to sell any of the components References in this presentation to “Qualcomm” may mean Qualcomm Incorporated,
or devices referenced herein. Qualcomm Technologies, Inc., and/or other subsidiaries or business units within
the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes our
© Qualcomm Technologies, Inc. and/or its affiliated
licensing business, QTL, and the vast majority of our patent portfolio. Qualcomm
Follow us on: companies. All Rights Reserved.
Technologies, Inc., a subsidiary of Qualcomm Incorporated, operates, along with its
Qualcomm is a trademark or registered trademark of Qualcomm Incorporated. subsidiaries, substantially all of our engineering, research and development functions, and
For more information, visit us at: Other products and brand names may be trademarks
substantially all of our products and services businesses, including our QCT semiconductor
business.
or registered trademarks of their respective owners.
qualcomm.com & qualcomm.com/blog Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc.
and/or its subsidiaries. Qualcomm patented technologies are licensed by Qualcomm
Incorporated.