Managing Context Information at

Large Scale: Introduction

Fermín Galán Márquez - fermin.galanmarquez@telefonica.com

(Reference Orion Context Broker version: 3.12.0)

Introduction

• Context Management in FIWARE

• Orion Context Broker
• Creating and pulling data
• Pushing data and notifications
• Batch operations

2
 Being “Smart” requires first being “Aware”
• Implementing a Smart Application requires gathering and managing
context information, referring to values of attributes characterizing
relevant entities
• There are many application domains where management of context
info is relevant: smart home, smart agrifood, smart industry, smart
logistics, smart cities

Application

standard API

Context Information
Users
• Name-Surname
Boiler
• Manufacturer • Birthday
• Preferences Flowerpot
• Last revision
• Location • Humidity
• Product id
• Watering plan
• temperature

3
Being “Smart” requires first being “Aware”
• Implementing a Smart Application requires gathering and managing
context information, referring to values of attributes characterizing
relevant entities
• There are many application domains where management of context
info is relevant: smart home, smart agrifood, smart industry, smart
logistics, smart cities

Application

standard API

Context Information
Truck
Ship • Driver Container
• Company • Location • Dimension
• Speed • … • Temperature
• Location • …
• …

4
 Being “Smart” requires first being “Aware”
• Implementing a Smart Application requires gathering and managing
context information, referring to values of attributes characterizing
relevant entities
• There are many application domains where management of context
info is relevant: smart home, smart agrifood, smart industry, smart
logistics, smart cities
Application

standard API

Context Information
Citizen
• Name-Surname
Bus • Birthday
• Location • Preferences Shop
• No. passengers • Location • Location
• Driver • Business name
• Licence plate • Franchise
• offerings

5
Context Information Management in Smart Cities

• Systems dealing with management of city services or third-party

apps (subject to access control policies) can consume and
produce context info
• Overall city governance can rely on context information available
(real-time and historic) to monitor KPIs and run BigData analysis

City Governance System Third-party Apps

Bus Context Information Shop

• Location • Location
Citizen
• No. passengers • Name-Surname • Business name
• Driver • Birthday • Franchise
• Licence plate • Preferences • offerings
• Location

City Services City Services

6
Different sources of context need to be handle

• Context information may come from many sources:

– Existing systems
– Users, through mobile apps
– Sensor networks
• Source of info for a given entity may vary over time

What’s the current temperature in

place “X”?
standard API

Place = “X”, temperature =

30º

It’s too hot!

A sensor in a The Public Bus Transport

pedestrian street A person from his smartphone Management system

7
Different sources of context need to be handle

• Context information may come from many sources:

– Existing systems
– Users, through mobile apps
– Sensor networks
• Source of info for a given entity may vary over time

What’s the current traffic in Notify me the changes of

street “X”? traffic in street “X”
Standard API

Street = “X”, traffic = high

The Public Bus Transport

Management system Citizen’s car app or
A sensor in a pedestrian street smartphone

8
 A non-intrusive approach is required

• Capable to integrate with existing or future systems without impact

in their architectures, but bringing a common context information
hub
• Info about attributes of one entity may come from different systems,
which work either as Context Producers or Context Providers

Application/Service

Standard API

attribute “location” attribute “driver”

System A System B

Context Producer Context Provider

9
 Context Management in FIWARE

• The FIWARE Context Broker GE implements the NGSI API: a

simple yet powerful standard API for managing Context
information complying with the requirements of a smart
enabled system
• The FIWARE NGSI API is Restful: any web/backend
programmer gets quickly used to it
Application/Service

NGSI API

Context Broker
Users
Boiler
• Name-Surname
• Manufacturer
• Birthday Flowerpot
• Last revision
• Preferences • Humidity
• Product id
• Location • Watering plan
• temperature

10
 FIWARE NGSI: “The SNMP for IoT”

• Capturing data from, or Acting upon, IoT devices becomes as

easy as to read/change the value of attributes linked to
context entities using a Context Broker
PUT
GET /v2/entities/lamp1/attrs/presenceSensor /v2/entities/lamp1/attrs/status/value
“light on”

NGSI API NGSI API

Context Broker

Issuing a get operation on the Setting up the value of attribute

“presenceSensor” attribute “status” to “light on” triggers
enables the application to get execution of a function in the IoT
info about presence of people device that switches the lamp on
near the lamp

11
 Connecting to the Internet of Things

• Capturing data from, or Acting upon, IoT devices becomes as

easy as to read/change the value of attributes linked to
context entities using a Context Broker
PUT
GET /v2/entities/lamp1/attrs/humidity /v2/entities/lamp1/attrs/status/value
“watering”

NGSI API NGSI API

Context Broker

Issuing a get operation on the Setting up the value of attribute

“humidity” attribute enables the “status” to “watering” triggers
application to find out whether execution of a function in the IoT
the plant has to be watered device that waters the plant

12
 Orion Context Broker

• Main functions:
– Context management
– Context availability management (advanced topic) (*)
• HTTP and REST-based
– JSON payload support
• Context in NGSI is based in an entity-attribute model:

Entity Attributes
• Name
• EntityId has
• Type
• EntityType
1 n • Value

(*) With some limitations in current Orion version, see

13
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md#registration-implementatio
n-differences
 Orion Context Broker in a nutshell

Orion Context Broker Context

Consumers

Context subscriptions query

Producers
1026

update
notify

update

1026
notify
update

DB

14
Orion Context Broker – check health

GET <cb_host>:1026/version

{
"orion" : {
"version" : "3.12.0",
"uptime" : "0 d, 0 h, 0 m, 5 s",
"git_hash" : "...",
"compile_time" : "nodate",
"compiled_by" : "fermin",
"compiled_in" : "centollo",
"release_date" : "nodate",
"machine" : "x86_64",
"doc" : "https://fiware-orion.readthedocs.org/en/3.11.0/",
"libversions": …
}
}
15
Orion Context Broker Basic Operations

Entities
• GET /v2/entities
• Retrieve all entities
• POST /v2/entities
• Creates an entity
• GET /v2/entities/{entityID}
• Retrieves an entity
• [PUT|PATCH|POST] /v2/entities/{entityID}
• Updates an entity (different “flavors”)
• DELETE /v2/entities/{entityID}
• Deletes an entity

16
Orion Context Broker Basic Operations

Attributes
• GET /v2/entities/{entityID}/attrs/{attrName}
• Retrieves an attribute’s data
• PUT /v2/entities/{entityID}/attrs/{attrName}
• Updates an attribute’s data
• DELETE /v2/entities/{entityID}/attrs/{attrName}
• Deletes an attribute
• GET /v2/entities/{entityID}/attrs/{attrName}/value
• Retrieves an attribute’s value
• PUT /v2/entities/{entityID}/attrs/{attrName}/value
• Updates an attribute’s value

17
 Context Broker operations: create & pull data

• Context Producers publish data/context elements by invoking the update

operations on a Context Broker.

• Context Consumers can retrieve data/context elements by invoking the query

operations on a Context Broker

update query

Context Producer Context Broker Context Consumer

18
Quick Usage Example: Car Create

POST <cb_host>:1026/v2/entities
Content-Type: application/json
...
You can add "?options=upsert" to
{
the URL. In that case, the entity is
"id": "Car1",
"type": "Car", updated if already exists.
"speed": {
"type": "Float",
"value": 98
}
}

201 Created

19
Quick Usage Example: Car Speed Update (1)

PUT <cb_host>:1026/v2/entities/Car1/attrs/speed
Content-Type: application/json
...

{
"type": "Float",
"value": 110
}

In the case of id ambiguity, you can use

"?type=Car" to specify entity type

204 No Content
…

20
Quick Usage Example: Car Speed Query (1)

GET <cb_host>:1026/v2/entities/Car1/attrs/speed

200 OK
Content-Type: application/json
...

{
"type": "Float",
"value": 110,
"metadata": {}
}

You can get all the attributes of the entity using the
entity URL:
GET/v2/entities/Car1/attrs

21
Quick Usage Example: Car Speed Update (2)

PUT <cb_host>:1026/v2/entities/Car1/attrs/speed/value
Content-Type: text/plain
...

115

204 No Content
…

22
Quick Usage Example: Car Speed Query (2)

GET <cb_host>:1026/v2/entities/Car1/attrs/speed/value
Accept: text/plain

200 OK
Content-Type: text/plain
...

115.000000

23
Quick Usage Example: Room Create (1)

POST <cb_host>:1026/v2/entities
Content-Type: application/json
...

{
"id": "Room1",
"type": "Room",
"temperature": {
"type": "Float",
"value": 24
},
"pressure": {
"type": "Integer",
"value": 718
}
}
201 Created
...

24
 Quick Usage Example: Room Update (1)

PATCH <cb_host>:1026/v2/entities/Room1/attrs
Content-Type: application/json
...

{
"temperature“: {
"type": "Float",
"value": 25
},
"pressure": {
"type": "Integer",
"value": 720
}
}

204 No Content
…

25
Quick Usage Example: Room Query (1)
GET <cb_host>:1026/v2/entities/Room1/attrs

200 OK
Content-Type: application/json
...

{
"pressure": {
"type": "Integer",
"value": 720,
"metadata": {}
},
"temperature": {
"type": "Float",
"value": 25,
"metadata": {}
}
}

26
Quick Usage Example: Room Query (2)
GET <cb_host>:1026/v2/entities/Room1/attrs?options=keyValues

200 OK
Content-Type: application/json
...

{
"pressure": 720,
"temperature": 25
}

27
Quick Usage Example: Room Create (2)

POST <cb_host>:1026/v2/entities
Content-Type: application/json
...

{
"id": "Room2",
"type": "Room",
"temperature": {
"type": "Float",
"value": 29
},
"pressure": {
"type": "Integer",
"value": 730
}
}
201 Created
...

28
Quick Usage Example: Filters (1)
GET <cb_host>:1026/v2/entities?options=keyValues&q=temperature>27

200 OK
Content-Type: application/json
...

[
{
"id": "Room2",
"pressure": 730,
"temperature": 29,
"type": "Room"
}
]

29
Quick Usage Example: Filters (2)
GET <cb_host>:1026/v2/entities?options=keyValues&q=pressure==715..725

200 OK
The full description of the Simple Content-Type: application/json
Query Language for filtering can be ...
found in the NGSIv2 Specification
document [
{
"id": "Room1",
"pressure": 720,
"temperature": 25,
"type": "Room"
}
]

30
Context Broker operations: push data

• Context Consumers can subscribe to receive context information that satisfy

certain conditions using the subscribe operation. Such subscriptions may
have an expiration time.

• The Context Broker notifies updates on context information to subscribed

Context Consumers by invoking the notify operation they export

Application

subId = subscribeContext (consumer, expr, expiration)

notify (subId, data/context)

Context Consumer
Context Broker

31
Quick Usage Example: Subscription
POST <cb_host>:1026/v2/subscriptions
Content-Type: application/json
…

{
"subject": {
"entities": [
{
"id": "Room1",
"type": "Room"
}
],
"condition": {
"attrs": [ "temperature" ]
}
},
"notification": {
"http": {
"url": "http://<host>:<port>/publish"
}, 201 Created
"attrs": [ "temperature" ] Location: /v2/subscriptions/51c0ac9ed714fb3b37d7d5a8
}, ...
"expires": "2026-04-05T14:00:00.000Z"
}

32
Quick Usage Example: Notification

25

19

33
Quick Usage Example: Notification

POST /publish HTTP/1.1

Content-type: application/json; charset=utf-8
Ngsiv2-AttrsFormat: normalized
…

{
"subscriptionId": "574d720dbef222abb860534a",
"data": [
{
"id": "Room1",
"type": "Room",
"temperature": {
"type": "Float",
"value": 19,
"metadata": {}
}
}
]
}

34
List existing subscriptions
GET <cb_host>:1026/v2/subscriptions 200 OK
Content-Type: application/json
…
[{
"id": " 51c0ac9ed714fb3b37d7d5a8 ",
"expires": "2026-04-05T14:00:00.000Z",
"status": "active",
"subject": {
"entities": [{
"id": "Room1",
"type": "Room"
}],
The full description of the "condition": {
subscription object (including all its "attrs": ["temperature"]
fields) can be found in the NGSIv2 }
},
Specification "notification": {
"timesSent": 3,
"lastNotification": "2016-05-31T11:19:32.000Z",
"lastSuccess": "2016-05-31T11:19:32.000Z",
"attrs": ["temperature"],
"attrsFormat": "normalized",
"http": {
"url": "http://localhost:1028/publish"
}
}
35
}]
 Orion Context Broker batch operations

• Batch query and batch update

• They are equivalent in functionality to previously described RESTful

operations

• All them use POST as verb and the /v2/op URL prefix, including
operation parameters in the JSON payload

• They implement extra functionality that cannot be achieved with

RESTful operations, e.g. to create several entities with the same
operation

• They are not a substitute but a complement to RESTful operations

36
Batch Operation Example: Create Several Rooms

POST <cb_host>:1026/v2/op/update …
Conten-Type: application/json {
... "type": "Room",
"id": "Room4",
{ "temperature": {
"actionType": "append", "value": 31.8,
"entities": [ "type": "Float"
{ },
"type": "Room", "pressure": {
"id": "Room3", "value": 712,
"temperature": { "type": "Integer"
"value": 21.2, }
"type": "Float" }
}, ]
"pressure": { }
"value": 722,
"type": "Integer"
201 Created
}
...
},
…

37
How to get Orion? (Docker containers)

• Assuming docker is installed in your system

• Documentation in
https://github.com/telefonicaid/fiware-orion/tree/develop/docker
• Quick guide
git clone https://github.com/telefonicaid/fiware-orion.git
cd fiware-orion/docker
sudo docker-compose up
• That’s all!
– curl localhost:1026/version
• Installing Orion docker in less than 1 minute
– https://www.youtube.com/watch?v=6taR7e20H9U

38
Would you like to play with this?

• Have a look to the FIWARE Reference Tutorial

application
– git clone
https://github.com/Fiware/tutorials.TourGuide-App.git
– cd tutorials.TourGuide-App/
– docker-compose up orion
– curl localhost:1026/version
• Self-explanatory README.md at root directory
• Open a Postman session and rock and roll
– Postman collection:
https://github.com/Fiware/tutorials.TourGuide-App/blob/
develop/contrib/CampusParty2016.postman_collection

39
NGSI Go

The NGSI Go is a command-line interface supporting FIWARE

NGSIv2 API and Orion Admin API, which simplifies syntax. It's a
powerful tool and easy to use. It has different commands for Orion as
shown:
– NGSI commands to manage NGSI Entity, subscription, registration
– Admin commands for FIWARE Orion
– Print version
– Copy or remove entities at once
– Create template of subscription or registration
– Notification receiver
– Integrated access token management for Keystone (Thinking Cities
platform), Keyrock and so on.

• Source code: https://github.com/lets-fiware/ngsi-go

• How to install: https://github.com/lets-fiware/ngsi-go#install
• Full documentation: https://ngsi-go.letsfiware.jp/

*NGSI Go is a third-party open-source software made by FIWARE community.

40
Would you like to know more?

• References
– This presentation, available at
https://github.com/telefonicaid/fiware-orion#introducto
ry-presentations
– NGSIv2 Specification
• https://github.com/telefonicaid/fiware-orion/blob/master/doc/man
uals/orion-api.md
– Documentation at ReadTheDocs
• https://fiware-orion.readthedocs.io
– Orion support through StackOverflow
• Ask your questions using the “fiware-orion” tag
• Look for existing questions at
http://stackoverflow.com/questions/tagged/fiware-orion

41
Managing Context Information at
Large Scale: Advanced Topics

Fermín Galán Márquez - fermin.galanmarquez@telefonica.com

(Reference Orion Context Broker version: 3.12.0)

 Orion functionality

Creating & pulling data Subscriptions & Notifications

Pushing data Batch operations

Pagination Type browsing

Geo-location Metadata
Update operators Custom
DateTime notifications
support Notification
Query filters status
MQTT
Special notifications
attribute/metadata Transient entities
Unrestricted text attributes Multitenancy
Flow control Attribute/metadata
Subscription based in alteration filtering
type Oneshot subscriptions
Covered
subscriptions Subscription auto-disabling
Compound attribute/metadata values
Subscription
Registrations & context improvements
43
providers
 • Pagination
Advanced Features • Metadata
• Compound attribute/metadata values
• Type browsing
• Geo-location
• Update operators
• Query filters
• DateTime support
• Transient entities
• Unrestricted text attributes
• Flow control
• Custom notifications
• Notification status
• Oneshot subscriptions
• Subscription auto-disabling
• Subscriptions based in alteration type
• Covered subscriptions
• MQTT notifications
• Subscription improvements
• Attribute/metadata filtering and special
attribute/metadata
• Registrations & context providers
• Multitenancy
• Service paths
• CORS
• Notifying services in private networks

44
Pagination

• Pagination helps clients organize query and

discovery requests with a large number of
responses.
• Three URI parameters:
– limit
• Number of elements per page (default: 20, max: 1000)
– offset
• Number of elements to skip (default: 0)
– count (option)
• Returns total elements (default: not return)

45
Pagination

• Example, querying the first 100 entries:

– GET <orion_host>:1026/v2/entities?limit=100&options=count

• The first 100 elements are returned, along with the

following header in the response:
– Fiware-Total-Count: 322

• Now we now there are 322 entities, we can keep querying

the broker for them:
– GET <orion_host>:1026/v2/entities?offset=100&limit=100
– GET <orion_host>:1026/v2/entities?offset=200&limit=100
– GET <orion_host>:1026/v2/entities?offset=300&limit=100

46
Pagination

• By default, results are ordered by entity creation date

• This behavior can be overridden using orderBy URI parameter
– A comma-separated list of attributes (including system/builtin
attributes), id (for entity ID) and type (for entity type). Results are
ordered by the first field. On ties, the results are ordered by the
second field and so on. A "!" before the field name means that the
order is reversed.
• Example: get the first 20 entities ordered by temp in ascending
order, then humidity in descending order
GET <orion_host>:1026/v2/entities?limit=20&offset=0&orderBy=temp,!humidity
• dateCreated and dateModified can be used to ordering by
entity creation and modification date, respectively

47
Metadata

• Users may attach metadata to attributes

• Reserved metadata: dateCreated, dateModified, previousValue, actionType,
ignoreType
• By default, metadata are not removed on updates, except if
overrideMetadata option is used
• Examples:
… …
"temperature": { "temperature": {
"type": "Float", "type": "Float",
"value": 26.5, "value": 26.5,
"metadata": { "metadata": {
"accuracy": { "average": {
"type": "Float", "type": "Float",
"value": 0.9 "value": 22.4
} }
} }
} }
… …

48
Complete NGSI Model

Entity Attributes Metadata

has • Name has • Name
• EntityId
• Type • Type
• EntityType
1 n • Value 1 n • Value

49
Compound Attribute/Metadata Values

• Attributes and metadata can have a structured

value. Vectors and key-value maps are
supported.
• It maps directly to JSON's objects and arrays.

50
Compound Attribute/Metadata Values

• Example: we have {
"type": "Car",
a car whose four "id": "Car1",
wheels' pressure "tirePressure": {
we want to "type": "kPa",
"value": {
represent as a "frontRight": 120,
compound "frontLeft": 110,
attribute for a car "backRight": 115,
"backLeft": 130
entity. We would }
create the car }
entity like this: }

51
Type Browsing

• GET /v2/types
• Retrieve a list of all entity types currently in Orion,
including their corresponding attributes and entities
count
• GET /v2/types/{typeID}
• Retrieve attributes and entities count associated to an
entity type

PRO TIP

GET /v2/contextTypes?options=values
Retrieves just a list of all entity types without any extra info

52
Geo-location

POST <cb_host>:1026/v2/entities
{
"type": "City",
• Entities can have an attribute "id": "Madrid",
that specifies its location "position": {
"type": "geo:json",
– value: a JSON object in "value": {"type": "Point", "coordinates":
GeoJSON format [ -3.691944, 40.418889] }
}
– type: geo:json }
• Example: create an entity called
null is also supported as value,
Madrid meaning "no location". This may be
…and create a couple more towns: useful in some cases.
• Leganés
• Alcobendas

53
Geo-location – Circle

54
Geo-location – Max distance

iu s
rad
k m
3 .5
1 GET <cb_host>:1026/v2/entities?
type=City&
georel=near;maxDistance:13500&
geometry=point&
coords=40.418889,-3.691944

55
Geo-location – Min distance

iu s
rad
k m
3 .5
1
GET <cb_host>:1026/v2/entities?
type=City&
georel=near;minDistance:13500&
geometry=point&
coords=40.418889,-3.691944

56
More geo-relationships

• Apart from near, the following georel can be

used
– georel=coveredBy
– georel=intersects
– georel=equals
– georel=disjoint
• See NGSIv2 Specification for a detailed
description

57
Update operators

• Regular attribute update uses particular values (eg. 12, "on",

etc).
• This can be inefficient and cause race conditions in some cases
– Eg. several context producers updating a common counter attribute

CP A CP B
Sum 2 count = 10 Sum 3
GET /v2/entity/E/attrs/count
GET /v2/entity/E/attrs/count
Calculation 10
at CP: 10
Calculation
10 +2 = 12 at CP:
PUT /v2/entity/E/attrs/count (12)
count = 12 10 +3 = 13

PUT /v2/entity/E/attrs/count (13)

Final value is 13 but it count = 13

should be 15!
58
Update operators

• Update operators can be used in attribute updates

• Eg. "increase the value of attribute by 2"
…
"count": {
"type": "Number",
"value": { "$inc": 2 }
}
…
• Full list of current operators:
– $inc: increase by a given value
– $mul: multiply by a given value
– $max, $min: max/min comparing provided and existing value
– $push, $addToSet: add elements to array
– $pull, $pullAll: remove elements from array
– $set, $unset: add/remove sub-keys in object

59
Update operators

• Simpler work for context producers and avoiding race

conditions
– No need of calculation in context producers

CP A CP B
Sum 2 PUT /v2/entity/E/attrs/count count = 10 Sum 3
{$inc: 2}

count = 12

PUT /v2/entity/E/attrs/count
{$inc: 3}

Final result is correct count = 15

60
Query filters

• For the GET /v2/entities operation

• By entity type GET <cb_host>:1026/v2/entities?type=Room

• By entity id list GET <cb_host>:1026/v2/entities?id=Room1,Room2

• By entity id pattern (regex) GET <cb_host>:1026/v2/entities?idPattern=^Room[2-5]

• By entity type pattern (regex)

GET <cb_host>:1026/v2/entities?typePattern=T[ABC]

• By geographical location
– Described in detail in previous slides

• Filters can be used simultaneously (i.e. like AND condition)

61
Query filters

attribute
name
• By attribute value (q)

GET <cb_host>:1026/v2/entities?q=temperature>25

attribute sub-key (for compound attribute values only)

GET <cb_host>:1026/v2/entities?q=tirePressure.frontRight >130

• By metadata value (mq) attribute

name metadata name

metadata sub-key (for compound

GET <cb_host>:1026/v2/entities?mq=temperature.avg>25 metadata values only)

GET <cb_host>:1026/v2/entities?mq=tirePressure.accuracy.frontRight >90

• See full details about q and mq query language in NGSIv2 specification

62
 POST <cb_host>:1026/v2/subscriptions
…
Query filters {
"subject": {
"entities": [
{
"id": “Car5",
"type": "Car"
• Filters can be also used in },
subscriptions {
"idPattern": "^Room[2-5]",
– id "type": "Room"
– type },
{
– id pattern "id": "D37",
– type pattern "typePattern": "Type[ABC]"
},
– attribute values ],
– metadata value "condition": {
"attrs": [ "temperature" ],
– geographical location "expression": {
"q": "temperature>40",
"mq": "humidity.avg==80..90",
"georel": "near;maxDistance:100000",
"geometry": "point",
"coords": "40.418889,-3.691944"
}
}
},
…
63 }
Datetime support

• Orion implements date support

– Based on ISO ISO8601 format with milliseconds
precision, including partial representations and
timezones
• See
https://github.com/telefonicaid/fiware-orion/blob/master/do
c/manuals/orion-api.md#datetime-support
for syntax details
• null is also supported meaning "no date"
– Use reserved attribute type DateTime to express a date
– Date-based filters are supported

64
Datetime support
POST /v2/entities
…
{
Example: create entity John,
"id": "John", with birthDate attribute using
"birthDate": { type DateTime
"type": "DateTime",
"value": "1979-10-14T07:21:24.238Z"
}
}

• Attribute value arithmetic filters can be used with dates as if they

were numbers
GET /v2/entities?q=birthDate<1985-01-01T00:00:00

• Entity dateModified and dateCreated special attributes, to get

entity creation and last modification timestamps
– They are shown in query responses using
attrs=dateModified,dateCreated
• Entity dateModified and dateCreated special metadata, to get
attribute creation and last modification timestamps
– They are shown in query responses using
metadata=dateModified,dateCreated

65
Transient entities
POST /v2/entities
…
{
Example: create entity
"id": "Warning1", Warning1, to expire at noon
"dateExpires": { on July 30th, 2018
"type": "DateTime",
"value": “2018-07-30T12:00:00Z"
}
}

• Orion will automatically delete the entity, once dateExpires time

is reached
– In fact transient entities may remain in the database up to 60 seconds (or a bit
more, if the database load is high) after expiration date
• BE CAREFUL! Entities removed this way cannot be recovered
• As dateModified and dateCreated (previously described),
dateExpires is not retrieved by default
– It is shown in query responses using attrs=dateExpires
• dateExpires can be used as any othe DatetTime attribute in
filter, e.g.:
GET /v2/entities?q=dateExpires<2018-08-01T00:00:00

66
Unrestricted text attributes

• By default, Orion limits the set of allowed chars in order to

avoid injection attacks
– List of forbidden chars:
https://github.com/telefonicaid/fiware-orion/blob/master/
doc/manuals/orion-api.md#general-syntax-restrictions
• You can avoid this check in attribute values using the
special attribute type TextUnrestricted

…
"description": {
"type": "TextUnrestricted",
"value": "Hey! I'm using <forbidden chars>"
}
…
• Use it at your own risk!

67
Flow control

• By default, Orion updates responses are decoupled from the

sending of the notifications triggered by these updates
• This may cause that if the client sending updates is much faster
than the receiver processing notifications, saturation will happen

Client Receiver

receiver is not
client sending so fast
updates too fast

…
at some point in time, Orion
… notification queue will saturate
and new notifications will be
discarded

68
Flow control

• Orion can implement a flow control mechanism, based in the -

notifFlowControl CLI option and flowControl option in update
requests
• Using flow control, Orion doesn’t response updates immediately. It
introduces a delay based on occupation size of the notification
queue (the more the occupation, the more the delay)

Client Receiver
?options=flowControl

delays
introduced
by Orion
…

…
69
Flow control

• More information on flow control

– Functionality description:
https://fiware-orion.readthedocs.io/en/master/admin/perf
_tuning/index.html#updates-flow-control-mechanism
– Detailed example/tutorial:
https://github.com/telefonicaid/fiware-orion/blob/master/
test/flowControlTest/README.md

70
Custom notifications

• Apart from the standard formats defined in the

previous slides NGSIv2 allows to re-define all the
notification aspects
• httpCustom is used instead of http, with the
following subfields
– URL query parameters
– HTTP method
– HTTP headers
– Payload (not necessarily JSON!)
• A simple macro substitution language based on ${..}
syntax can be used to “fill the gaps” with entity data (id,
type or attribute values) and extra information (service,
servicePath or authToken)
– Exception: this cannot be used in HTTP method field

71
 Custom notifications
Text based payload

PUT /v2/entities/DC_S1-D41/attrs/temp/value?type=Room
…
23.4

…
update "httpCustom": {
"url": "http://foo.com/entity/${id}",
"headers": {
"Content-Type": "text/plain"
},
"method": "PUT",
"qs": {
"type": "${type}"
},
notification "payload": "The temperature is ${temp} degrees"
}
…
PUT http://foo.com/entity/DC_S1-D41?type=Room
Content-Type: text/plain Custom notification configuration
Content-Length: 31

The temperature is 23.4 degrees

72
 Custom notifications
JSON based payload

PUT /v2/entities/DC_S1-D41/attrs?type=Room
…
{ "temp": {"value": 23.4, "type": "Number"},
"status": {"value": "on", "type": "Text"} }

…
update "httpCustom": {
"url": "http://foo.com/entity/${id}",
"method": "PUT",
"qs": {
"type": "${type}"
},
"json": {
"t": "${temp}",
notification "s": "${status}"
}
}
PUT http://foo.com/entity/DC_S1-D41?type=Room …
Content-Type: application/json
Content-Length: 19 Custom notification configuration
{ "t": 23.4, "s": "on"}

73
 Custom notifications
NGSI patching in payload

PUT /v2/entities/DC_S1-D41/attrs?type=Room
…
{ "A": {"value": "foo", "type": "Text"} }

update …
"httpCustom": {
"url": "http://foo.com/entity/${id}",
"method": "PUT",
"ngsi": {
"id": "stamp:${id}"
notification "A:map": { "value": "${A}", "type": "Text"}
"oldId": { "value": "${id}", "type": "Text"}
}
PUT http://foo.com/entity/DC_S1-D41 }
Content-Type: application/json …
Content-Length: 140
Custom notification configuration
{
"id": "stamp:DC_S1-D41",
Notification uses NGSIv2
"A": { "value": "foo", "type": "Text" },
"A:map": { "value": "foo", "type": "Text" },
compliant format (as in
"oldId": { "value": "DC_S1-D41", "type": "Text" } regular notifications)!
} 74
Notification status

• Detailed information in the notifications

element
– timesSent: total number of notifications
200 OK
attempts (both successful and failed) Content-Type: application/json
– failsCounter: number of consecutive …
notifications fails [{
• failsCounter > 0 means subscription is in "id": " 51c0ac9ed714fb3b37d7d5a8 ",
failing state "status": "active",
– lastSuccess: last time that notification was "subject": { … },
successfully sent "notification": {
– lastFailure: last time that notification was tried "timesSent": 3,
and failed "failsCounter": 1,
– lastNotification: last time the notification was "lastNotification": "2016-05-31T11:19:32.000Z",
sent (either success or failure) "lastSuccess": "2016-05-31T10:07:32.000Z",
• Corollary: lastNotification value is the same "lastFailure": "2016-05-31T11:19:32.000Z",
than either lastFailure or lastSuccess
"lastSuccessCode": 200,
– lastFailureReason: cause of last failure (text) "lastFailureReason": "Timeout was reached",
– lastSuccessCode: the HTTP code (number) …
returned by receiving endpoint last time a
}
successful notification was sent
}]
• lastSuccessCode and
lastFailureReason fields only in HTTP
notifications (not in MQTT ones)

75
 Notification diagnosis workflow
"status" field
value?
active/oneshot
expired inactive
YES
failsCounter > 0?
NO

"lastSuccessCode“
field value?

4xx,
2xx 5xx

Subscription is expired Subscription is inactive Notifications are being Notifications are Some problem is
so notifications are not so notifications are not sent and the receiving being sent but the precluding
being sent. Update being sent. Update endpoint confirms receiving endpoint notifications to be
subscription to make it subscription to correct reception. is reporting HTTP sent. In HTTP
permanent or extend activate it. error. Check subscriptions, the
expiration time receiving endpoint "lastFailureReason"
(e.g. logs, etc.) field value should
provide additional
information on it.

Only for HTTP notifications

76
Oneshot subscription
200 OK
Content-Type: application/json
• A variant of the active …
status, so when the [{
"id": "51c0ac..",
subscription is triggered "status": "oneshot",
one time (i.e. a …
}
notification is sent) it }]
automatically steps to
inactive status PATCH /v2/subscriptions/51c0ac..
Subscription is {
• An inactive subscription triggered "status": "oneshot"
can step to oneshot, }
starting again the process 200 OK
Content-Type: application/json
…
[{
"id": "51c0ac..",
"status": "inactive",
…
}
}]

77
Subscription auto-disabling

• A maxFailsLimit can be specified for subscriptions

so when the number of consecutive notifications
overpasses that value, the subscription
automatically passes to inactive state POST /v2/subscriptions

failsCounter > maxFailsLimit => status := inactive {

"subject": { … },
"notification": {
• failsCounter is reset to 0 whenever a successful "maxFailsLimit": 3,
notification is sent …
}
• This allow to protect Orion against failing }
notification endpoints what would consume
notification resources (pool workers, etc.)
• When a subscription is auto-disabled, a trace about
it is printed in logs:

time=... | lvl=WARN | ... | msg= Subscription 51c0ac9ed714fb3b37d7d5a8

automatically disabled due to failsCounter (4) overpasses maxFailsLimit (3)

78
 Subscriptions based in alteration type

• By default, subscriptions are triggered

when entities are created or changed (i.e.
POST /v2/subscriptions
update with a value different from the
previous value) {
• This can be changed with alterationTypes "subject": {
field (within condition) to specify a list of …
"condition": {
the following (OR condition):
"alterationTypes":
– entityUpdate: entity update (no matter if is ["entityCreate","entityDelete"]
actually changed or not) }
– entityChange: entity actual change },
– entityCreate: entity is created "notification": {
…
– entityDelete: entity is deleted "attrs": [ "alterationType", "*" ]
• The alteration type which triggered the }
subscription can be included in }
notifications by adding the special
alterationType attribute (which possible
values are the same tokens above)

79
 Covered subscriptions

• By default, attributes that doesn't exist in the entity are not

notified
• This can be changed with covered field (within
notification) set to true to specify that non existing
attributes has to be notified
– As they don't exist, they use null as value
• We use the term "covered" in the sense the notification
"covers" completely all the attributes in the attrs field within
notification
• This feature can be useful for those notification endpoints
that are not flexible enough for a variable set of attributes
and needs always the same set of incoming attributes in
every received notification

80
Covered subscriptions NOTIF1:
…
SUB 1: SUB 2 (covered) {
"notification": { "notification": { …
... ... "temp": {
"attrs": [ "attrs": [ "type": "Float",
"temp", "value": 19,
"temp",
"metadata": {}
"hum" "hum
}
] ],
}
} "covered": true
}
NOTIF2 (covered)
…
{
…
"temp": {
POST /v2/entities/E/attrs/temp
"type": "Number",
{ ...
"value": 19,
"value": 19, "metadata": {}
"type": "Number" },
} "hum": {
"type": "None",
"value": null,
"metadata": {}
}
81 }
MQTT notifications

• Using mqtt instead of http in the subscription payload

• Taking advantage of MQTT features (shared subscriptions, QoS, retain, etc.)
• MQTT authentication based in user/password is supported
• Efficient connection management (removing idle connections to MQTT brokers if
they are not in use, base din -mqttMaxAge)
• Also supporting custom notifications (mqttCustom)

POST <cb_host>:1026/v2/subscriptions
…
{
"subject": {
"entities": [
{ "idPattern": ".*" } MQTT Broker
],
},
"notification": { Context Broker
"mqtt": {
"url": "mqtt://<mqtt_broker>:1883",
"topic": "/notif", …
"qos": 1
} MQTT broker subscribers
}
}
82
Additional subscription improvements

• By default updates not actually changing attribute value

don’t trigger notification but this behavior can be overridden
by the forcedUpdate URI param option, eg.:
– PATCH /v2/entities/E/attrs?options=forcedUpdate
– PUT /v2/entities/E/attrs/A?options=forcedUpdate
– POST /v2/op/update?options=forcedUpdate
• onlyChangedAttributes (within notification) can be set to
true in order to notify only attributes that change (in
combination with the ones specified in attrs or exceptAttrs)
• notifyOnMetadataChange (within condition) can be set to
false so metadata is not considered part of the value of the
attribute in the context of notification (so if the value doesn't
change but the metadata changes notification is not
triggered)
• Per-subscription notification timeout can be specified for
HTTP subscriptions in timeout field (within
http/httpCustom within notification)
83
Attributes filtering and special attributes

• By default all attribute are included in query

responses or notifications
• The attrs field (as parameter in GET operations
and as notification sub-field in subscriptions)
can be used to specify a filtering list
• The attrs field can be also used to explicitly
include some special attributes (not included by
default)
– dateCreated, dateModified, dateExpires: described
in previous slide
• The “*” can be used as an alias of “all the
(regular) attributes”

84
Attributes filtering and special attributes

• Examples
– Include only attributes temp and lum
• In queries: GET /v2/entities?attrs=temp,lum
• In subscriptions: "attrs": [ "temp", "lum" ]

– Include dateCreated and not any other attribute

• In queries: GET /v2/entities?attrs=dateCreated
• In subscriptions: "attrs": [ "dateCreated" ]

– Include dateModified and all the other (regular)

attributes
• In queries: GET /v2/entities?attrs=dateModified,*
• In subscriptions: "attrs": [ "dateModified", "*" ]

– Include all attributes (same effect that not using attrs,

not very interesting)
• In queries: GET /v2/entities?attrs=*
• In subscriptions: "attrs": [ "*" ]

85
Metadata filtering and special metadata

• By default all attribute metadata are included in query responses

and notifications
• The metadata field (as parameter in GET operations and as
notification sub-field in subscriptions) can be used to specify a
filtering list
• The metadata field can be also used to explicitly include some
special metadata (not included by default)
– dateCreated, dateModified: described in previous slide
– actionType: which value is the action type corresponding to the
update triggering the notification: “update”, “append” or “delete” (*)
– previousValue: which provides the value of the attribute previous to
processing the update that triggers the notification
• The “*” can be used as an alias of “all the (regular) metadata”

(*) actionType “delete” not yet supported by Orion in 3.12.0.

86
Metadata filtering and special metadata

• Examples
– Include only metadata MD1 and MD2
• In queries: GET /v2/entities?metadata=MD1,MD2
• In subscriptions: "metadata": [ "MD1", "MD2" ]

– Include previousValue and not any other metadata

• In queries: GET /v2/entities?metadata=previousValue
• In subscriptions: "attrs": [ "previousValue" ]

– Include actionType and all the other (regular) metadata

• In queries: GET /v2/entities?metadata=actionType,*
• In subscriptions: "attrs": [ "actionType", "*" ]

– Include all metadata (same effect that not using

metadata, not very interesting)
• In queries: GET /v2/entities?metadata=*
• In subscriptions: "metadata": [ "*" ]

87
Registration & Context Providers

• Uncached queries and updates

Application

1. register(provider= )

5. data 4. data

2. query 3. query
Context ContextBroker ContextProvider
Consumer
db

88
 Registration & Context Providers

POST <cb_host>:1026/v2/registrations
…
{
"description": "Registration for Car1",
"dataProvided": {
"entities": [
{
"id": "Car1",
"type": "Car"
}
],
"attrs": [
"speed" 201 Created
] Location: /v2/registrations/5a82be3d093af1b94ac0f730
}, …
"provider": {
"http": {
"url": "http://contextprovider.com/Cars"
},
"legacyForwarding": true
}
}

(*) With some limitations in current Orion version, see

89
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md#registration-implementatio
n-differences
Registration & Context Providers

?options=skipForwarding can be used to avoid

query forwarding, in which case query is evaluated using
exclusively Context Broker local context information

GET <cb_host>:1026/v2/entities/Car1/attrs

200 OK
Content-Type: application/json data
...

{ query
"type": "Float", ContextBroker ContextProvider
"value": 110,
"metadata": {} db
}

90
Multitenancy

Context
• Simple multitenant model based on Broker
logical database separation.
• It eases tenant-based authorization
provided by other components. Tenant1
• Just use an additional HTTP header
called "Fiware-Service", whose value
is the tenant name. Example: Tenant2
Fiware-Service: Tenant1

91
Service Paths

• A service path is a hierarchical scope assigned to an entity

at creation time (with POST /v2/entities).

92
Service Paths

• In order to use a service path we put in a new HTTP header called "Fiware-
ServicePath". For example:
Fiware-ServicePath: /Madrid/Gardens/ParqueNorte/Parterre1
• Special servicePath attribute can be used to retrieve entity service
path along any other attribute
– E.g. GET /v2/entities?attrs=servicePath,*
• Properties:
– A query on a service path will look only into the specified node
– Use "ParentNode/#" to include all child nodes
– Queries without Fiware-ServicePath resolve to "/#"
– Entities will fall in the "/" node by default
ParqueSur ParqueNorte ParqueOeste

Parterre1 Parterre2
93
Service Paths

• Properties (continued):
– You can OR a query using a comma (,) operator in
A B
the header
• For example, to query all street lights that are either in
ParqueSur or in ParqueOeste you would use:
ServicePath: Madrid/Gardens/ParqueSur,
A or B
Madrid/Gardens/ParqueOeste
• You can OR up to 10 different scopes.
– Maximum scope levels: 10
• Scope1/Scope2/.../Scope10
ParqueNorte
– You can have the same element IDs in different
scopes (be careful with this!)
– You can't change scope once the element is created light1
– One entity can belong to only one scope Parterre1

– It works not only with queries, but also with

subscriptions/notifications light1
– It works not only in NGSI10, but also with
registrations/discoveries (NGSI9)

94
Cross-Origin Resource Sharing (CORS)

• Useful for programming clients that run entirely in

browser without backend
– Full support (including pre-flight requests) in NGSIv2
– Partial support (only for GET requests) in NGSIv1
• CLI parameters related with CORS
– -corsOrigin
– -corsMaxAge
• See details
– https://fiware-orion.readthedocs.io/en/master/user/cors/inde
x.html
Notifying services in private networks

notify Private Network

Context Consumer
Context Broker

Private Network

no Context Consumer
Context Broker ti fy

fy
ti
no
./ngrok http 8080
http://9762d35a.ngrok.io

ngrok.io

96
Would you like to know more?

• References
– This presentation, available at
https://github.com/telefonicaid/fiware-orion#introducto
ry-presentations
– NGSIv2 Specification
• https://github.com/telefonicaid/fiware-orion/blob/master/doc/man
uals/orion-api.md
– Documentation at ReadTheDocs
• https://fiware-orion.readthedocs.io
– Orion support through StackOverflow
• Ask your questions using the “fiware-orion” tag
• Look for existing questions at
http://stackoverflow.com/questions/tagged/fiware-orion

97
Thank you!

http://fiware.org
Follow @FIWARE on Twitter
 Integration with existing systems

• Context adapters will be developed to interface with existing systems (e.g.,

municipal services management systems in a smart city) acting as Context
Providers, Context Producers, or both

• Some attributes from a given entity may be linked to a Context Provider

while other attributes may be linked to Context Producers
Application

queryContext (e1,
attr1, attr2)

queryContext (e1, updateContext (e1,

attr1) attr2)

System A System B
Context Broker

Context Provider Context Consumer

99
Integration with sensor networks

• The backend IoT Device Management GE enables creation and

configuration of NGSI IoT Agents that connect to sensor networks

• Each NGSI IoT Agent can behave as Context Consumers or Context

Providers, or both

OMA NGSI API (northbound interface)

FIWARE Context Broker

create/monitor IoT
Agent
Manager
IoT IoT IoT FIWARE Backend
Agent-1 Agent-2 Agent-n IoT
Device Management
(southbound interfaces)

ETSI M2M MQTT IETF

CoAP
100
Context Management in FIWARE

• Federation of infrastructures (private/public regions)

Cloud • Automated GE deployment

• Complete Context Management Platform

Data • Integration of Data and Media Content

• Easy plug&play of devices using multiple protocols

IoT • Automated Measurements/Action Context updates

• Visualization of data (operation dashboards)

Apps • Publication of data sets/services

• Easy support of UIs with advanced web-based 3D and AR

Web UI capabilities
• Visual representation of context information.

• Advanced networking capabilities (SDN) and Middleware

I2ND • Interface to robots

• Security Monitoring
Security • Built-in Identity/Access/Privacy Management

101
FI-WARE Context/Data Management Platform

Applications
Processing/Analysis
OMA NGSI-9/10 Algorithms

Gathered data is
injected for
processing/analysis

Data generated either by CEP

or BigData is published

Gathered data injected

for CEP-like processing BigData
(COSMOS)
Distributed Direct
Context bigdata
Sources Complex Event Processing injection
Processed data is
(PROTON) injected for
processing/analysi
Programming of s
rules
Context/Data Management Platform

102
Special update action types

• Used by /v2/op/update (batch operation)

• Conventional actionTypes
– append: to append (or update if the attribute already
exists)
– update: to update
– delete: to delete
• Special actionTypes
– appendStrict: strict append (returns error if some of
the attributes to add already exists)
– replace: delete all the entity attributes, next append
the ones in the update request

103