© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Customizing generative AI
applications for your business
using your own data
Maira Ladeira Tanke
(she/her)
Sr. Generative AI Data Scientist
AWS

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

01 Approaches to customizing generative AI solutions

02 Amazon Bedrock overview

03 Retrieval-augmented generation using Amazon Bedrock

04 Fine-tuning and continued pretraining using Amazon Bedrock

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why customize?

Customize for Adapt to Enhance Improve

specific business domain-specific performance context awareness
needs language for specific tasks in responses

E.g., healthcare – Understand E.g., finance – Teach financial and E.g., customer service – Improve E.g., legal services – Better
medical terminology and provide accounting terms to provide good ability to understand and respond understand case facts and law to
accurate responses related to a analysis for earnings reports to a customer’s inquires and provide useful insights for
patient’s health complaints attorneys

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customizing foundation models to understand
your use case

FOUNDATION
MODEL

CUSTOMIZATION YOUR DATA

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Augment models without changing pretrained
model weights with knowledge bases

FOUNDATION
MODEL

KNOWLEDGE BASE VECTOR

DATABASES
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Connect your model with dynamic data sources
using agents

FOUNDATION
MODEL

AGENTS DATABASES,
APIS
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adapt models for your use case with fine-tuning

FOUNDATION
MODEL

FINE-TUNING LABELED
DATA
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Update your models through continued
pretraining

FOUNDATION
MODEL

CONTINUED UNLABELED
PRETRAINING DATA
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaches to customizing models with your data

Augment model with

Augment model with Maximizing model
relatively static data Maximizing accuracy
PURPOSE dynamic data without accuracy for your
without changing for specific tasks
changing weights domain
weights

Contextual
DATA Contextual information Small number Large number
information
NEED based on user’s query of labeled examples of unlabeled datasets
based on user’s query

Medium – once new Infrequent – only Infrequent – only

DATA Frequent – every time a
data arrives necessary when task necessary if domain
UPDATE database changes
(e.g., new product) changes context changes

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customize
External data
Consolidated or
sources
historical info
or up-to-date info Task
information

Relatively static Dynamic

information information Complex Simple
(e.g., docs, FAQs) (e.g., DBs, APIs) or specific or generic
Real time Simple
required? task?

Augment with Augment with Prompt

Knowledge Bases Customize
agents & tools engineering

Amazon Bedrock Amazon Bedrock Amazon Bedrock Amazon Bedrock

Knowledge Bases Agents Custom Models FMs

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Bedrock

Amazon Bedrock Choose Use as is or Send Receive

an FM customize prompt response
Build generative AI
applications using Use the playground Fine-tune FMs as Use the Bedrock API Receive model
foundation models (FMs) to experiment with needed. Bedrock will to send your prompts response in your
through a serverless FMs and select the automatically deploy to the model application
API service one that suits your the FM for inference
needs

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Amazon
Broad choice of models

Jurassic-2 Ultra Titan Text Embeddings Claude 2 Command + Embed Llama 2 Stable Diffusion XL1.0
Jurassic-2 Mid Titan Multimodal Embeddings Claude 2.1 Cohere Command Light Llama 2 13B
Titan Text Lite Claude Instant Cohere Embed English Llama 2 70B
Titan Text Express Cohere Embed Multilingual
Titan Image Generator

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Retrieval-augmented
generation (RAG)

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is retrieval-augmented generation?

Augmentation Generation
Retrieval
Fetches the relevant Adding the retrieved Response from the
content from the relevant context to the foundation model based
external knowledge base user prompt, which goes on the augmented
or data sources based on as an input to the prompt
a user query foundation model

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RAG use cases

Improved content Contextual chatbots and Personalized search Real-time data

quality question answering summarization

E.g., helps in reducing E.g., enhances chatbot capabilities E.g., searches based on a user’s E.g., retrieves and summarizes
hallucinations and connecting by integrating with real-time data previous search history and transactional data from databases
with recent knowledge, including persona or API calls
enterprise data

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are embeddings?
• Numerical representation of
text (vectors) that captures
semantics and relationships
New York 0.027 -0.011 … -0.023
between words.
Paris 0.025 -0.009 … -0.025
• Embedding models capture
features and nuances of the EMBEDDING
MODEL
text. Animal -0.011 0.021 … 0.013

Horse -0.009 0.019 … 0.015

• Rich embeddings can be used
to compare text similarity. Human text Vector embeddings
• Multilingual text embeddings
can identify meaning in
different languages.

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why are embeddings important for RAG?

Powers text retrieval based Used to augment prompts High-accuracy embeddings

on semantic meaning. with more accurate lead to improved context
context from vector stores and higher-quality LLM-
using the retrieval- generated responses to a
augmented generation. user query.

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titan text embeddings model

Amazon Titan Text Embeddings Highlights

V2.0

Translates text inputs (words, phrases) into numerical • Titan Text Embeddings offers fast, cost-
representations (embeddings). Comparing effective, high-performance, accurate
embeddings produces more relevant and contextual embeddings in 25 languages.
responses than word matching.
• Optimized for text retrieval tasks, semantic
similarity, and clustering.

• Applications of this model include semantic

Max tokens: 8,000 search and personalization.
Output vectors: 1,536
Language: Multilingual (25 languages)

Model ID: amazon.titan-embed-g1-text-02

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cohere embeddings model

Cohere Embed – English Cohere Embed – Multilingual

V3.0 V3.0

Embed is Cohere's text representation, or embeddings, Embed is Cohere's text representation, or embeddings,
model. This version supports English only. model. This version supports multiple languages.

Supported use cases: Semantic search, retrieval- Supported use cases: Semantic search, retrieval-
augmented generation (RAG), classification, augmented generation (RAG), classification,
clustering. clustering.

Output vectors: 1,024 Output vectors: 1,024

Language: English Language: Multilingual (100+ supported languages)

Model ID: cohere.embed-english-v3 Model ID: cohere.embed-multilingual-v3

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How RAG works

User input

Text User
Prompt
augmentation
Large language
model
Response
generation
workflow Embeddings
model
Context

Embeddings 0.89 -0.02 -0.53 0.95 0.17 -0.38

Data Semantic
ingestion search

workflow Vector store Embeddings model Document store New data

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
However, when it comes to implementing RAG,
there are challenges…

Managing multiple Creating vector Incremental updates

data sources embeddings for large to vector store
volumes of data

Coding effort Scaling retrieval Orchestration

mechanism

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowledge bases for Amazon Bedrock
NATIVE SUPPORT FOR RETRIEVAL-AUGMENTED GENERATION (RAG)

Securely connect FMs Fully managed RAG Built-in session Automatic citations
to data sources for workflow, including context management with retrievals to
RAG to deliver more ingestion, retrieval, for multi-turn improve transparency
relevant responses and augmentation conversations

USER QUERY AUGMENTED PROMPT MODEL ANSWER

1 4 5 AI21 Labs—Jurassic2 6
A M A Z O N
B E D R O C K

Amazon—Titan Text

2 3
Anthropic—Claude

KNOWLEDGE Meta—Llama2
BASES FOR
AMAZON BEDROCK
Cohere—Command

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data ingestion workflow
KNOWLEDGE BASES FOR AMAZON BEDROCK

Fully
managed
data
ingestion
Data source Embeddings
workflow New data Document chunks
model
Vector store

• Choose your data source Choose your chunking Choose your Choose your vector store
strategy embedding model • Amazon OpenSearch
(Amazon S3)
• Fixed chunks • Amazon Titan
Serverless
• Support for incremental
• No chunking • Cohere Embed
• Amazon Aurora
updates
• Default (200 tokens)
• Pinecone
• Multiple data file formats
• Redis
supported
• MongoDB
(coming soon)

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Retrieval and generate

User input

User Prompt Large language

Response
augmentation model

Text Embeddings
generation model Context

workflow
0.89 -0.02 -0.53 0.95 0.17 -0.38

Embedding
Semantic
search

Knowledge bases for Amazon

Bedrock
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RetrieveAndGenerate API
KNOWLEDGE BASES FOR AMAZON BEDROCK

User Input RetrieveAndGenerate

API

User Response
Fully User query Generated
managed response
RAG

Generate query Retrieve similar Augment query with Generate response

embeddings documents from retrieved documents from LLM
knowledge bases

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customize RAG workflows using Retrieve API
KNOWLEDGE BASES FOR AMAZON BEDROCK

User input

User Prompt Large Response

augmentation language
model

Customized
RAG Retrieve API
workflow
User Retrieved
Context
query documents

Generate query Retrieve similar documents

embeddings from knowledge bases

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vector databases supported by Amazon Bedrock

COMING SOON

Vector Engine for Amazon Redis Enterprise Pinecone MongoDB

Amazon Aurora Cloud
OpenSearch
Serverless

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agents

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agents for Amazon Bedrock
ENABLE GENERATIVE AI APPLICATIONS TO EXECUTE MULTISTEP TASKS USING COMPANY SYSTEMS AND
DATA SOURCES

1 2 3 4

SELECT YOUR PROVIDE BASIC SELECT RELEVANT SPECIFY AVAILABLE

FOUNDATION MODEL INSTRUCTIONS DATA SOURCES ACTIONS

| Breaks down and orchestrates tasks |

| Securely accesses and retrieves company data for RAG |
| Takes action by invoking API calls on your behalf |
| Chain-of-thought trace and ability to modify agent prompts |
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefits

Automates Simplifies building Provides secure Lets you choose Provides fully
orchestration of and deploying access to enterprise implementation managed
multistep tasks AI assistants data and APIs languages infrastructure

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agents build on existing enterprise resources
HR knowledge base Existing resources

HR policy docs

Vacation actions
get-Vacation-Balance

HR time-off
Vacation Vacation
agent microservice database
Leave of absence actions

Leave of absence (LOA)

database

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Knowledge bases integration with agents

Search

Knowledge bases
Query

Retrieval

Agent

Query + Retrieval
Large language
model
Response generation

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fine-tuning and continued
pretraining

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Bedrock custom models
CREATE CUSTOM MODELS USING THE CONSOLE OR API’S

• Maximize accuracy of FMs by providing

labeled or raw unlabeled data

• Once deployed, custom models are invoked

the same way as base models
(playground or API)

• Customizations now supported for Amazon

Titan and some third-party FMs

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customizing Amazon Titan models

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Fine-tune additional models
in Amazon Bedrock
COMING SOON

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fine-tuning and continued pretraining
Domain
adaptation
(e.g., extend
knowledge)
Fine-tuning
Continued
• Instruction training dataset is available? pretraining
• Specific style, behavior required? Continued +
Pretraining Fine-tuning

Continued pretraining
• Raw dataset (e.g., PDFs)
• Additional knowledge through domain adaptation
Fine-tuning

Task specialization
(e.g., behavior, style)
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Datasets for fine-tuning and continued pretraining
Instruction dataset Raw data
(e.g., question-answer) (e.g. PDFs)
Dataset

Fine-tuning Continued
pretraining

Amazon Bedrock Amazon Bedrock

Custom Models Custom Models

{”prompt": "<prompt text>", ”completion": "<expected generated text>"} {"input": "<raw text>"}
{”prompt": "<prompt text>", ”completion": "<expected generated text>"} {"input": "<raw text>"}
{”prompt": "<prompt text>", ”completion": "<expected generated text>"} {"input": "<raw text>"}

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Components of a model customization job

Inputs Outputs Storage Inferencing

Base FM Metrics and Custom models Playground

logs stored securely
by Amazon
Hyper Bedrock API
parameters Output
model
Input
data

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customization architecture overview
Amazon Bedrock service account Model deployment account
(AWS owned and operated)
via the console, SDKs, and API
All incoming network traffic

Training orchestration

Amazon Bedrock
Base model S3
bucket Custom
Provisioned Job
Runtime inference capacity
API compute
endpoint

Fine-tuned model
S3 bucket

Customer account

Identity and access,

Virtual private cloud
monitoring and logging

Training data
AWS Amazon AWS S3 bucket
CloudTrail CloudWatch IAM
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security and privacy
You are always in control of your data

✓ Data not used to improve models, and not shared with model providers

✓ Customer data remain in Region

✓ Support for AWS PrivateLink and VPC configurations

✓ Integration with AWS IAM

✓ API monitoring in AWS CloudTrail, logging and metrics in Amazon CloudWatch

✓ Custom models encrypted and stored with service or customer managed keys
(CMK) – Only you have access to your models

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary

Customizing an FM is Amazon Bedrock You can combine

a cost-effective way to supports multiple multiple customization
improve model ways to customize techniques for better
accuracy FMs performance

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
Maira Ladeira Tanke
mttanke@amazon.com

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.