SAP BODS 4.

1
INTRODUCTION TO DATA
SERVICES
WHAT IS DATA SERVICES?

3
DATA SERVICES
-INTEGRATE HETEROGENEOUS DATA

4
DATA QUALITY MANAGEMENT

5
DATA SERVICES TECHNICAL
ARCHITECTURE
-ARCHITECTURE
BODS ARCHITECTURE

7
DATA SERVICES PHYSICAL ARCHITECTURE

8
DEVELOPING IN DATA
SERVICES
-DEVELOPMENT PROCESS
DATA SERVICES DESIGNER

10
BODS DESIGNER INTERFACE

 Key Areas of Designer windows

 Toolbar
 Local Object Library

 Project Area

 Tool palette
 Workspace

11
PRE-BUILT INTERFACES

12
DATASTORES
DATASTORES

14
TYPES OF DATASTORES

15
DATASTORES CONTINUOUS…

16
CONNECTING TO DATASTORE
SOURCE/TARGET

17
CONNECTING TO FILES WITH FILE FORMAT

18
Objects
 All "entities" you define, edit, or work with in Designer are called objects.
The local object library shows objects such as source and target metadata,
system functions, projects, and jobs.

The software has two types of objects: Reusable and single-use.

 Single-use objects:
Some objects are defined only within the context of a single job or data
flow, for example scripts and specific transform definitions.
 Reusable objects
A reusable object has a single definition; all calls to the object refer to that
definition. If you change the definition of the object in one place, you are
changing the object in all other places in which it appears. Example: A data
flow,

19
OBJECT HIERARCHY IN BODS

20
 BODS PLATFORM
TRANSFORMS

21
QUERY TRANSFORM
ACTIONS PERFORMED BY
QUERY TRANSFORM
Query transform is used very regularly in data flows in
conjunction with other transforms.

Join data from multiple sources

Map columns from input to output schemas
Perform transformations and functions on the data
Perform data nesting and un nesting
Add new columns, nested schemas, and function results to the
output schema
Assign primary keys to output columns
A query transform is similar to a SQL SELECT statement.
23
CONTINUES…

Data inputs
A query has data inputs, which are data sets from one or more sources with
rows flagged as any operation code.

Data Outputs
A query has data outputs which are data sets based on the conditions that
you specify, and data outputs use the schema specified in the output schema
area.

Query Editor
The Query Editor is a window that has three areas: where you specify the
following for the Query transform:
• Schema In
• Schema Out
• Option tabs

24
CONTINUES…

Query transform input schema

The input schema area displays all schemas input to the Query transform
as a hierarchical tree. Each input schema can contain zero or more of the
following elements:
• Columns
• Nested schemas

Query transform output schema

The output schema area displays the schema output from the Query transform
as a hierarchical tree. The output schema can contain one or more of the
following elements:
• Columns
• Nested schemas
• Functions

25
EXAMPLE FOR QUERY
TRANSFORM
Ex: Select the employee’s department info from 2 tables with
inner join.

26
QUERY TRANSFORM
CONTINUES….

12/07/23
27
QUERY TRANSFORM
CONTINUES….

28
QUERY TRANSFORM CONTINUES….

29
End of Query Transform
CASE TRANSFORM
CASE TRANSFORM CONTINUES…

 It’s a Upstream Transform

 The Case transform simplifies branch logic in data flows by
consolidating case or decision making logic in one transform.
 Define the Label and Expression

 Expression means the condition to route the rows depends

upon the Business logic.
 Label refers to identification of Expression

 Usage of Case transform in very frequent in Data Quality

implementation
 NO Change in Source and target schemas in terms of no. of
columns

31
 …
CASE TRANSFORM CONTINUES

EX: THE BELOW EXAMPLE DESCRIBES ON HOW TO ROUTE THE RECORDS

AS PER THE LANGUAGE

32
MERGE TRANSFORM
MERGE TRANSFORM
CONTINUES…

 It’s an upstream/Downstream transform

 Combines incoming data sets, producing a single
output data set with the same schema as the input data
sets
 A data set from two or more sources with rows
flagged as any operation code.

34
Merge Transform continues…

 All sources must have the same schema, including:

 The same number of columns
 The same column names
 The same data types of columns
 The sequence of columns is same in input schemas
 If the input data set contains hierarchical data, the names and
data types must match at every level of the hierarchy.

35
Ex: Records coming from 2 heterogeneous systems( File,
Database) and performing union of records to get customers
and suppliers by City

36
VALIDATION TRANSFORM
38
39
40
 ROW GENERATION
TRANSFORM

41
CONTINUES….

 It’s an Generation /UpStreamTransform

 Generates a column filled with integer values starting at
zero and incrementing by one to the end value you
specify.
 It doesn’t require any data input unlike as other
transforms.
 Need to provide/pass Values for options(Row number
starts at, Row count).For added flexibility, you can enter
a global variable or substitution parameter

42
ROW GENERATION TRANSFORM
CONTINUES….

Ex: This will describes on how to generate Item Number for a

specific Plant using Row Generation Transform and lookup the
Item Description of respective Item Number .
(Dataflow :DF_PItem_Des )

43
 SQL TRANSFORM

44
SQL TRANSFORM
CONTINUES…

 It’s an Streamline /Upstream Transform

 This transform to perform standard SQL operations
when other built-in transforms cannot perform or
complex to Perform the Business Use case in SAP
BODS
 The SQL transform supports a single SELECT statement
only.
 Data Inputs are None. This transform does not allow an
input data set.
 With SQL Transform, Data Services can push down
expressions so that they are performed by the underlying
database manager.
45
SQL TRANSFORM
CONTINUES…

Ex: This will describes on how to use the SQL Transform to

avoid the complex ETL code with other built-in transforms for
better performance.(Full Push Down)

46
EXTRACTION OF
NESTED DATA
EXTRACTION OF NESTED DATA
CONTINUES…..

What is Nested Data?

 Nested data means representing the Hierarchical

relationships with Schema and Sub-Schema Format.
 Ex: XML documents and SAP ERP Idocs

 The software maps nested data to a separate schema

implicitly related to a single row and column of the
parent schema. This mechanism is called Nested
Relational Data Modelling (NRDM)

48
EXTRACTION OF NESTED DATA
CONTINUES…..

Representing Nested Data?

 Sales Orders: The line items in a sales order are related
to a single header and are represented using a nested
schema. Each row of the sales order data set contains a
nested line item schema as shown below….

49
EXTRACTION OF NESTED DATA
CONTINUES…..

Ex: Sales Orders:Represention of Multiple

Nested Schemas

50
 XML FILE EXTRACTION IN SAP
BODS

51
XML FILE EXTRACTION
CONTINUES…..

 Data in XML file is defined in nested format

 Data in XML file is stored in 2 files
-Header file( .DTD/.XSD)
-Data file( .XML)
To extract an XML data, BODS provides 2 formats.
1) DTD file format for .DTD file
2) XML Schema format for .XSD file
Different ways to extract the data from DTD format or XML
schema are as follows:
-Query Transform
-XML_Pipeline Transform

52
XML FILE EXTRACTION
CONTINUES…..

DTD file format

53
XML FILE EXTRACTION
CONTINUES…..

 Once you add the DTD file format to the workspace double click on the file
format to specify the XML file path to retrieve the data

54
XML FILE EXTRACTION
CONTINUES…..

XSD file format

55
XML FILE EXTRACTION
CONTINUES…..

 Once you add the XSD Schema format to the workspace double click on
the file format to specify the XML file path to retrieve the data

56
XML FILE EXTRACTION
CONTINUES…..

XML EXTRACTION WITH QUERY

TRANSFORM

 Unnesting Nested Data

 Procedure:
 Create the output that you want to unnest in the output
schema of a query
 Select Unnest option on out schema's that we want to Unnest
 The Output of the Query is as shown in Next Slide….

57
XML FILE EXTRACTION
CONTINUES…..

58
 XML_PIPELINE TRANSFORM

59
XML FILE EXTRACTION
CONTINUES…..

 XML Pipeline is a readymade logic in the form of

transform which helps to extract part of XML data
dynamically.
 XML Pipeline transform supports only XML (nested
structure) input dataset.
 XML Pipeline transform converts Nested structure into
Flat structure.
 No multiple inputs are allowed with XML pipeline.

60
XML FILE EXTRACTION
CONTINUES…..

Limitation:
XML pipeline allows only level (part) of extraction, i.e.
from one level, only a set of columns of one node are
allowed to extract
Note:
Multiple XML Pipeline transforms are used to extract
data from multiple nodes present in same level.
Output Schema maintenance of XML Pipeline transform
is under user control. But there is no flexibility to add new
fields in the output schema.

61
CONTINUES…..

Ex: The Example Describes the structure of input XML

source and is defined as input to XML pipeline transform

62
 DATA GENERATION
TRANSFORM

63
 PIVOT TRANSFORM

64
PIVOT TRANSFORM CONTINUES…..

 Pivot Transformation is used to Convert Column Values

of table\file into Row Values.
 Creates a new row for each value in a column that you
identify as a pivot column.
 The Pivot transform allows you to change how the
relationship between rows is displayed. For each value in
each pivot column, Data Services produces a row in the
output data set. You can create pivot sets to specify more
than one pivot column.
 A data set with rows flagged as NORMAL

65
PIVOT TRANSFORM CONTINUES…..

Ex: Below Example Describes how to rearrange the Data

with Pivot Transform to calculate Expense Summaries

Pre-requisite Parameters:

66
PIVOT TRANSFORM CONTINUES…..

Output Data:

SAPBODS: Dataflow

67
 REVERSE_PIVOT TRANSFORM

68
CONTINUES…..

 Reverse Pivot Transformation is used to Convert Rows

Values of table\file in to Column Values
 It creates a new column for every value that is specified
as a pivoted column
 Creates one row of data from several existing rows by
creating new columns for each unique value in a pivot
axis column and each selected pivot column
 Output: A data set with rows flagged as NORMAL.

69
CONTINUES…..

Ex: How to rearrange all of the contact information for a

particular employee is in the same row with Reverse_pivot
Input Data:

Pre-requisite Parameters:

70
CONTINUES…..

Output Data:

SAPBODS: Dataflow

71
 DATA SERVICES SCRIPTING
LANGUAGE

72
SAP BODS SCRIPTING LANGUAGE CONTINUES…..

 Scripting Language is to write scripts and custom

functions to address complex logical expressions.
 Expressions are a combination of constants, operators,
functions, and variables that evaluate to a value of a
given data type.
 Expressions can be used inside script statements or
added to data flow objects
 Jobs and work flows can use scripts to define detailed
steps in the flow of logic
 A script can run functions and assign values to variables,
which can then be passed to other steps in the flow.

73
SAP BODS SCRIPTING LANGUAGE CONTINUES…..

 Statements in a script object or custom function must

end with a semicolon (;)
 Comment lines must start with a # character.

 Use single quotes for string constants.

 Uses the backslash (\) as the escape character.

 For example:

74
SAP BODS SCRIPTING LANGUAGE CONTINUES…..

 Variable Names must be preceded by a dollar sign ($).

 Local Variables used in a script or expression must be
defined in the Job or Workflow context that calls the
script, using the Variables and Parameters window.
 The square brackets ([]) indicate that the value of the
expression should be substituted.
 The curly braces ({}) indicate that the value of the
expression should be quoted with single quotation
marks.

75
 SCRIPT OBJECT

76
SCRIPT OBJECT CONTINUES…..

 Scripts are single-use objects used to call functions and

assign values to variables in a work flow.
 Usage of Script object is recommended at the start and
end of the Mapping logic
 Script Object facilitates the BODS to send email
notification to Business Users in case of failure or
Success
 Script object is flexible to call the VBScript or Windows
Script to send the reports to end user.
 Example of Scripts in Next Slide……

77
SCRIPT OBJECT CONTINUES…..

Drop and Re-Create the Table

Total Records in a Table

Delete data from table

78
 CONDITIONAL OBJECT

79
CONDITIONAL OBJECT CONTINUES…..

 Conditional is single-use object, used to implement

if/then/else logic in a work flow.
 Unlike to Case transform, the conditional operate at the
work flow level
 Conditional blocks:

80
CONDITIONAL OBJECT CONTINUES…..

The Below Figure Describes on how to split the branch logic

depends on TRUE or FALSE Statement

81
Ex: Describes on how to use the conditional object to
execute the DF if file exists in share folder other wise send
a notification to DL

82
 WHILE LOOP

83
WHILE LOOP CONTINUES…..

 The while loop is a single-use object that you can use in

a work flow.
 Condition: In the While box enter a Boolean expression
that the job evaluates.
 The expression must evaluate to TRUE or FALSE.

 You can use constants, functions, variables, parameters,

and standard operators to construct the expression.

84
WHILE LOOP CONTINUES…..

The Below Figure Describes on how while loop repeats a

sequence of steps as long as a condition is true

85
WHILE LOOP CONTINUES…..

Ex: Describes on how to use the while object to load the

files one by one recurrsively

86
 DATA PROFILING

87
 DATA ASSESSMENT
 With operational systems frequently changing, data
quality control becomes critical in your extract,
transform and load (ETL) jobs.

How to determine and improve the quality and structure of

your source data:
 Use the Data Profiler

 Use the View Data feature

 Use the Design-Time Data Viewer feature

 Use the Validation transform

 Use the auditing data flow feature

 Use data quality transforms

88
DATA PROFILER CONTINUES…..

The Data Profiler executes on a profiler server to provide the

following data profiler information

 Column analysis
-Basic Profiling: Min/Max/Average value, Min/Max string
length.
-Detailed Profiling: distinct count/Percent, Median, median
string length, pattern count, and pattern percent.
 Relationship analysis:

Data mismatches between any two columns for which you

define a relationship

89
DATA PROFILER CONTINUES…..

BASIC PROFILING
Analysis of data beyond viewing

Frequency distribution
Distinct values
Null values
Min/Max values

Can drill down to view

specific records

90
DATA PROFILER CONTINUES…..

DETAIL PROFILING
Analysis of data beyond viewing
Frequency distribution
Distinct values
Data Patterns
(e.g. Xxx Xxxx99, 99-Xxx)

Can drill down to view

specific records

91
DATA PROFILER CONTINUES…..

RELATIONSHIP PROFILING
Comparison of values between data
sets to determine fit

Shows % of non-matching
values among
Table -Table
Flat file -Flat file
Table -Flat file

Can drill down to view

actual records

92
 OPERATION CODES IN
TRANSFORMS

93
OPERATION CODES CONTINUES…..

SAP BODS Maintains Operation codes which describe the

status of each row in each data set in/Out of objects in
Dataflow

94
MOST FREQUENTLY USED
TRANSFORMS TO MANIPULATE
OPERATION CODES

 Table_Comparision

 Map_Operation

95
 TABLE_COMPARISION

96
 Table_Comparision : Parameters to pass

 Compare Columns (Mandatory)

 Comparison method(Mandatory)

 Detect deleted row(s) from comparison table

 Generated Key Column

 Input primary key column(s) (Mandatory)

 Input contains duplicate keys

 Table name (Mandatory)

 Filter

97
TABLE_COMPARISION CONTINUES…..

 Compares two data sets and produces the difference between

them as a data set with rows flagged as INSERT, UPDATE, or
DELETE.
 The Table_Comparision transform allows you to detect and
forward changes that have occurred since the last time a target
was updated.
 Table_Comparision- NORMAL becomes INSERT

98
TABLE_COMPARISION CONTINUES…..

Table_Comparision- NORMAL becomes UPDATE

99
TABLE_COMPARISION CONTINUES…..

Table_Comparision- NORMAL becomes DELETE

100
 MAP_OPERATION

101
MAP_OPERATION CONTINUES…..

 Modifies data based on mapping expressions and current

operation codes. The operation codes can be converted
between data manipulation operations.
 Map_Operation can convert the Operation Codes from

 UPDATE to INSERT or

 DELETE to DISCARD

 NORMAL to INSERT etc.,

102
MAP_OPERATION CONTINUES…..

 A data set with rows flagged as specified by the mapping

operations.

 Rows in the input data set can contain any of the

following operation codes:
 NORMAL
 INSERT
 DELETE
 UPDATE

 Any of these operation codes can be mapped to:

 NORMAL
 INSERT
 DELETE
 UPDATE
103
 CHANGE DATA CAPTURE

104
CHANGE DATA CAPTURE -TYPES

 Source based CDC

 Map_CDC Operation

 Target based CDC

 Table Comparison
 Map Operation
 History Preserving Transform
 Key Generation Transform

105
 TARGET BASED CDC
 Slowly Changing Dimension-Types
 SCD Type 0
 SCD Type 1
 SCD Type 2

 SCD Type 0
 Only Current with no History & CDC
 SCD Type 1
 History +Current but no CDC
 SCD Type 2
 History+Current +CDC 106
SCD IMPLEMENTATION IN SAP BODS
 SCD Type 0
 Query Transform

 SCD Type 1
 Table_Comparision and Map_Operation

 SCD Type 2
 Table_Comparision,History_Preserving,Key_Generation

107
 VARIABLES AND PARAMETERS
 Variables Types
 Global Variables
 Usage at Job level
 Local Variables
 Usage at Workflow/Dataflow and Custom Functions
 Parameters
 Usage at Custom Functions
 Substitution Parameters
 Usage at Local Repository level

108
 DATA EXTRACTION FROM SAP
ECC

109
 SAP AND BODS INTEGRATION

 Steps Involved:

 Make sure if SAP ECC/BW and SAP BODS are in the same
network or if in shared Network
 Deploy the SAPBODS Transports on to SAP ECC in
function Group /BODS/BODS
 Create the Program ID in SAP ECC for RFC Connection
 Register the Program ID in SAP BODS Console
 Create the SAP Applications type of Data store in SAPBODS
to import the metadata from SAP ECC

110
 OBJECTS IN SAP ECC

 Objects can be accessed from SAP ECC

 Extractors
 Hierarchies

 Tables

 Idocs

 BAPI functions

111
OPTIONS IN SAP APPLICATION DATA STORE

 ABAP Execution Option:

 Generate and Execute
 Execute Preloaded

 Data Transfer Method:

 DirectDownload
 Shared Directory
 RFC
 FTP
 Custom Transfer

112
WAYS TO EXTRACT THE DATA FROM
SAPECC /BW
 Normal Dataflow
 Forsmall set of records we will use this Dataflow
 RFC Enabled Functions are responsible for extraction and
Loading of Data
 CPU-Memory consumption would be high

 ABAP Dataflow
 For larger set of records we will use ABAP Dataflow
 ABAP Code Fired Against SAP to get Extract and Load the
data
 Memory consumption would be less when compared to
Normal dataflow 113
Dataflow between SAPECC and SAP BODS

114
 DATA EXTRACTION FROM SAP
BW

115
 OBJECTS IN SAPBW

 Objects can be accessed from SAP BW as a Source

 Open Hub Tables
 Tables

 Idocs

 BAPI functions

116
OPTIONS IN SAP BW AS A SOURCE DATA STORE

 ABAP Execution Option:

 Generate and Execute
 Execute Preloaded

 Data Transfer Method:

 DirectDownload
 Shared Directory
 RFC
 FTP
 Custom Transfer

117
Dataflow between SAPBW and SAP BODS

118
DATA SERVICES OPEN HUB READER RUN-
TIME ARCHITECTURE

119
 PROCESS FLOW STEPS:

 1. The Data Services job executes function

sap_openhub_processchain_execute which starts the
following process:
 a. Sends a web service request to the Open Hub RFC Server
to start the process chain.
 b. The RFC Server starts the Process Chain in BW system.
 c. The Process Chain contains a Data Transfer Process (DTP)
that extracts data from the Info Provider and loads the Open
Hub Destination table.
 d. When the DTP completes, the Open Hub Service notifies
the RFC Server.
 e.
120
  The RFC Server notifies the Job Server that the Open Hub
table is
 loaded and returns the read request ID of the Open Hub
Destination.
 2. The Data Services data flow reads the data from the
Open Hub table using read request ID.
 3. The Data Services job contains a second script that
executes the function sap_openhub_set_read_status
which starts the following process:
 a. Sends a web service request to the Open Hub RFC
Server to set the read status.
 b. RFC Server sets the read status in the Open Hub BW
system which
 deletes the data from the Open Hub Destination table.

121
 DATA QUALITY MANAGEMENT

122
 TYPES OF DATA QUALITY
MANAGEMENT IN SAP BODS

 Address Cleanse
 Data Cleanse

 Geocoder

 Matching

123
ABOUT CLEANING PROCESS

124
EXAMPLE – ADDRESS CLEANSE

125
 HOW THE GEOCODER WORKS?

 The Geocoder transform uses geographic

coordinates expressed as latitude and longitude,
addresses, and point-of-interest (POI) data.
Using the transform, you can append addresses,
latitude and longitude, census data (US only),
and other information to your data.

126
 SAP HANA AND SAP BODS
INTEGRATION

127
 WHAT IS SAP HANA?

SAP HANA is that technology which allows

the processing of huge quantities of real
time data in the ‘main memory’ of a server to
provide near time results from transactions..

128
SAP HANA-TECHNICAL INNOVATIONS

129
DATA PROVISIONING- SAP HANA

130
APPLICATION LAYER-RESPONSE TIME

131
SAP BODS AND SAP HANA
INTEGRATION

132
IMPORT SAP METADATA INTO HANA VIA
DATA SERVICES

133
PROCESS FLOW DIAGRAM : SAP HANA
& SAP BO DATA SERVICES

134
 HANA STUDIO -FEATURES
 HANA Studio- Features:
 Modeling
 Information Models
 Information models are used to create multiple views of
transactional data that can be used for analytical purposes.
 Attribute View, Analytic View, Calculation View ,
Calculation View enhanced with Analytical View
 Information Models

 Import/Export

135
SAP HANA-REPORTING

136