Scribd
Upload
54 views2 pages

Enable Vectorization in Hive

Vectorization allows Hive to process batches of rows together rather than one at a time to improve performance. It can be enabled by setting hive.vectorized.execution.enabled to true. Hive will log whether a query was vectorized. Vectorization currently supports single table read-only queries with selection, filtering, and grouping operators on many data types for ORC files.

Uploaded by

Pranoy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
54 views2 pages

Enable Vectorization in Hive

Vectorization allows Hive to process batches of rows together rather than one at a time to improve performance. It can be enabled by setting hive.vectorized.execution.enabled to true. Hive will log whether a query was vectorized. Vectorization currently supports single table read-only queries with selection, filtering, and grouping operators on many data types for ORC files.

Uploaded by

Pranoy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 2

Query Vectorization in Hive

Vectorization allows Hive to process a batch of rows together instead of processing one row at a time.
Each batch is usually an array of primitive types. Operations are performed on the entire column
vector, which improves the instruction pipelines and cache usage. 
Enable Vectorization in Hive
To enable vectorization, set this configuration parameter:
hive.vectorized.execution.enabled=true
When vectorization is enabled, Hive examines the query and the data to determine whether
vectorization can be supported. If it cannot be supported, Hive will execute the query with
vectorization turned off.
Log Information about Vectorized Execution of Queries
The Hive client will log, at the info level, whether a query's execution is being vectorized. More
detailed logs are printed at the debuglevel.
The client logs can also be configured to show up on the console.
Supported Functionality
The current implementation supports only single table read-only queries. DDL queries or DML queries
are not supported.
The supported operators are selection, filter and group by.
Partitioned tables are supported.
These data types are supported:
 tinyint
 smallint
 int
 bigint
 date
 boolean
 float
 double
 timestamp
 string
 char
 varchar
 binary
These expressions are supported:
 Comparison: >, >=, <, <=, =, !=
 Arithmetic: plus, minus, multiply, divide, modulo
 Logical: AND, OR

Vectorization pg. 1
 Aggregates: sum, avg, count, min, max

Only the ORC file format is supported in the current implementation.

The Hive query execution engine currently processes one row at a time. A single row of data goes through all
the operators before the next row can be processed. This mode of processing is very inefficient in terms of
CPU usage. Research has demonstrated that this yields very low instructions per cycle. Also currently Hive
heavily relies on lazy deserialization and data columns go through a layer of object inspectors that identify
column type, deserialize data and determine appropriate expression routines in the inner loop. These layers of
virtual method call further slowdown the processing.
This work will add support for vectored query execution to Hive, where, instead of individual rows, batches of
about a thousand rows at a time are processed. Each column in the batch is represented as a vector of a
primitive data type. The inner loop of execution scans these vectors very fast, avoiding method calls,
deserialization, unnecessary if-then-else, etc. This substantially reduces CPU time used, and gives excellent
instructions per cycle (i.e. improved processor pipeline utilization).

Vectorization pg. 2

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy