Copyright © 2018, Oracle and/or its affiliates. All rights reserved.

|
 Presented with

MySQL DevAPI
Building Modern Apps with Node.js

Filip Janiszewski Speaker, Software Developer

Johannes Schlüter, Engineering Manager
MySQL Connectors Team
July 42, 2018

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
On This Call Today

Filip Janiszewski, MySQL Connector/PHP

Software Developer
• filip.janiszewski@oracle.com

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
NoSQL
• Most users/companies feel compelled to try NoSQL at some point in time
• Either they believe a RDBMS does not scale or because devs don’t like SQL
or something else
• Only later to find out about all its limitations (usually because there was not
enough time to learn a whole new tool in the first place)
• The result is usually one of the following:
– Use some RDBMS with JSON support (smaller companies/users)
– Implement some kind of NoSQL interface on top of a reliable storage engine (e.g. Uber
with Schemaless and Facebook with MyRocks/RocksDB)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
MySQL DocStore Components

Client App
MySQL

Router

DevAPI Doc Store Plugin

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

MySQL DocStore Components - DevAPI
The X DevAPI is a high-level database API enabling easy development of
CRUD-based applications powered by a clustered MySQL setup. While
focusing on ease of use in CRUD NoSQL document operations it also gives
expert-level control and access to advanced features including raw SQL.

Available in most popular languages

Provides a powerful way to query documents and relational tables
Allows for direct execution of SQL
Allows for working with InnoDB clusters via the router
Developers guide found at https://dev.mysql.com/doc/x-devapi-userguide/en/

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

MySQL DocStore Components – Document Store
MySQL Document Store exists as a plugin for MySQL 8. This plugin is enabled
by default and provides a full document storage layer on your existing MySQL 8
server.

• Uses a different set of ports instead of the normal MySQL 3306

• Uses Google protobuf messages at the protocol layer
• Accepts DevAPI requests from clients and processes them using the core MySQL engine
• C++, Java, PHP, Python, Node.JS, .NET ...
• https://dev.mysql.com/doc/refman/8.0/en/document-store.html

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

MySQL Document-store

• “An X-Over between a relational database and document database”

• Common API over multiple languages and environments
– C++, Java, .Net, Node.js, Python and PHP
– New MySQL Shell
– Easy CRUD operations relational tables and document database styled collections

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
X Protocol

• A new protocol implemented by the X Plugin

• Based on Google Protocol Buffers (flexibility for future extensions)
• Better understanding of content
– CRUD expression trees
– Expectations to attach conditions on pipelined statements
– Bound parameters
• Security baked in
– SSL/TLS by default
– No information leaked to unauthenticated users (e.g. MySQL Server version)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
X-Protocol

Mysqlx.Crud.Find {
collection { name: "collection_name", schema: "test" }
data_model: DOCUMENT
criteria {
type: OPERATOR
operator {
name: "=="
param {
type: IDENT,
identifier { name: "_id" }
}
param {
type: LITERAL,
literal {
type: V_STRING,
v_string: { value: "some_string" }
}
}
}
}
}

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Documents and Collections

• Documents in the Docstore are represented as JSON

– “JavaScript Object Notation”
– Cross-platform serialization format (common for web services)
– Standardized as ECMA-404 (http://json.org)
– Well supported by MySQL
• Multiple documents are stored inside a Collection
– Technically, an InnoDB table
– One regular column of type JSON
– Multiple virtual columns on top for indexes

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
X DevAPI
A common API for Document and SQL

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Features
• Fluent API with flexible parameters
– query builder based on common expression language
– raw SQL interface is also available
• Secure by default (SSL/TLS and SHA256 authentication)
• Data-consistency with transactions, savepoints and row-locking
• Faster (in SQL mode) than alternative community drivers

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Fluent API
• Code becomes more readable, maintainable (and even testable)
• Operations encapsulated in specialized and semantic methods
• Nice scaffolding for refactor tasks
• First-class support for text-editor (or IDE) hints and auto-completion
• Smaller SQL injection surface area
• Common standard between different programming environments

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Flexible Parameters
• Most public API methods work with:
– Multiple individual parameters
– A single array of parameters
– An object with named properties (where it applies)

$sakila = ["name" => "Sakila", "age" => 17, "job" => "Singer"];
$coll->add($sakila)->execute();
$coll->add(["name" => "Sakila", "age" => 18, "job" => "Student"])->execute();
$coll->add('{"name": "Filip", "age": 24, "job": "Plumber"}')→execute();

$res = $coll->find('job like :job and age = :age')->fields(['age', 'job'])->groupBy('age', 'job');

$data = $res->bind(['job' => 'Programmer', 'age' => 10])->sort('age desc')->limit(4)->execute();

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Expression Language
• Subset of SQL (easier to learn, yet still powerful)
• Expressive and human-readable
• Common standard between all the official connector implementations
(making it portable)

// PHP // Java
$coll->find('name like \‘foo\’ AND age > 42') collection.find("name = 'foo' AND age > 42")
->fields(['name', 'age']) .fields("name", "age")
->groupBy('name', 'age'); .groupBy("name", "age")
->sort('name ASC',’age DESC’) .sort("name ASC", "age DESC")
->limit(4) .limit(4)
->offset(2) .offset(2)
->execute(); .execute()

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
SSL Modes
Secure by default

• SSL/TLS is enabled by default, for TCP connections, when creating a session

• Local UNIX socket connections disable it (not needed) to be faster
• Additional server certificate validation
• Options can be overridden on-demand

mysql_xdevapi\getSession({ ssl: false })

mysql_xdevapi\getSession('mysqlx://root@localhost?ssl-mode=DISABLED')
mysql_xdevapi\getSession({ ssl: true, sslOptions: { ca: '/path/to/ca.pem' } })
mysql_xdevapi\getSession('mysqlx://root@localhost?ssl-ca=(/path/to/ca.pem)')
mysql_xdevapi\getSession({ ssl: true, sslOptions: { ca: '/path/to/ca.pem', crl: '/path/to/crl.pem' } })
mysql_xdevapi\getSession('mysqlx://root@localhost?ssl-ca=(/path/to/ca.pem)&ssl-crl=(/path/to/crl.pem)')

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Authentication Mechanisms
Secure by default (as well)

• SHA-1 and SHA-2 password hashing

• Supports authentication mechanisms for the following server plugins:
– mysql_native_password
– sha256_password
– caching_sha2_password

mysql_xdevapi\getSession({ user: 'root', auth: 'MYSQL41' })

mysql_xdevapi\getSession('mysqlx://root@localhost?auth=MYSQL41')
mysql_xdevapi\getSession({ user: 'root', auth: 'PLAIN' })
mysql_xdevapi\getSession('mysqlx://root@localhost?auth=PLAIN')
mysql_xdevapi\getSession({ user: 'root', auth: 'SHA256_MEMORY' })
mysql_xdevapi\getSession('mysqlx://root@localhost?auth=SHA256_MEMORY')

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Transactions and Savepoints
Session-level atomic operations

• Create, commit or rollback transactions in the scope of a session

• Create, release or rollback to intermediate savepoints in those transactions

$session→startTransaction();

$coll->add( '{"test1":1, "test2":2}' )→execute();

$sp1 = $session->setSavepoint( 'mysavepoint1' );

$coll->add( '{"test3":3, "test4":4}' )->execute();

$session->rollbackTo( 'mysavepoint1' ); //rollback to mysavepoint1

$session→rollback(); //rollback the entire transaction

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Row Locking
Document-level isolation

• Queries reading a document can determine the isolation level in the

presence of concurrent transactions
• Two types of locks:
– Exclusive locks
– Shared locks
• Three different operation modes:
– DEFAULT
– NOWAIT
– SKIP_LOCKED

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Row Locking
Usage
Type/Modes DEFAULT NOWAIT SKIP_LOCKED

Exclusive Lock Transactions on different sessions Reads in one session fail if there is Reads in one session see inconsistent
are synchronized and reads see an ongoing transaction on a data when there is an ongoing
consistent data. different session. transaction on a different session.

Shared Lock Reads in one session wait until all Same as above. Same as above.
ongoing transactions on different
sessions are commited.

$collection->find('name = :name')->bind([‘name' => 'foo'])->lockExclusive()

$collection->find('name = :name')->bind([‘name' => 'foo'])->lockExclusive(MYSQLX_LOCK_SKIP_DEFAULT) // same as the last
$collection->find('name = :name')->bind([‘name' => 'foo'])->lockExclusive(MYSQLX_LOCK_SKIP_NOWAIT)
$collection->find('name = :name')->bind([‘name' => 'foo'])->lockExclusive(MYSQLX_LOCK_SKIP_LOCKED)

$collection->find('name = :name')->bind([‘name' => 'foo'])->lockShared()

$collection->find('name = :name')->bind([‘name' => 'foo'])->lockShared(MYSQLX_LOCK_SKIP_DEFAULT) // same as the last
$collection->find('name = :name')->bind([‘name' => 'foo'])->lockShared(MYSQLX_LOCK_SKIP_NOWAIT)
$collection->find('name = :name')->bind([‘name' => 'foo'])->lockShared(MYSQLX_LOCK_SKIP_SKIP_LOCKED)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Raw SQL

• For ETL, structured analytics and/or reporting, SQL is always an option

• Allows to tap into features not yet part of the X DevAPI, such as:
– DDL operations for relational tables
– Managing key constraints
– Using JOINs

// create a table
$session->sql('CREATE TABLE foo (bar VARCHAR(3))')->execute()
// add a unique constraint
$session->sql('ALTER TABLE foo ADD COLUMN bar VARCHAR(3) GENERATED ALWAYS AS doc->>"$.bar" VIRTUAL UNIQUE KEY NOT NULL')->execute()
// execute a JOIN query
$session->sql('SELECT DISTINCT t1.bar FROM foo t1 JOIN baz t2 ON t1.bar = t2.qux WHERE t1.qux = t2.quux')->execute()

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Setting up and running the PHP Extension
for MySQLX

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Extension setup

• Verify that PHP has the package installed

• PHP >= 7.1.0 required

fjaniszewski ~ : pecl remote-info mysql_xdevapi

Package details:
================
Latest 8.0.12
Installed - no -
Package mysql_xdevapi
License PHP
Category Database
Summary MySQL X DevAPI for PHP

.
.

fjaniszewski ~ : pecl install mysql_xdevapi

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Extension setup

• Verify that the extension is properly installed

fjaniszewski ~ : php -m
[PHP Modules]
Core
.
.
mysql_xdevapi
.
.
mysqli
Mysqlnd
.
.
[Zend Modules]

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Extension setup

• It is possible to load the extension dynamically without actually installing it

fjaniszewski ~ : php -dextension=mysql_xdevapi.so ./script.php

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Configure MySQL

• For >= 8.0.12 there’s not too much to do, the plugin is installed and can be
verified with:

mysql> show plugins;

+----------------------------+----------+--------------------+---------+---------+
| Name | Status | Type | Library | License |
+----------------------------+----------+--------------------+---------+---------+
.
.
| mysqlx | ACTIVE | DAEMON | NULL | GPL |
.
.
+----------------------------+----------+--------------------+---------+---------+
43 rows in set (0.00 sec)

mysql> install plugin mysqlx soname 'mysqlx.so'; //IF NOT INSTALLED

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Examples

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
How to create a collection

<?php

$session = mysql_xdevapi\getSession('mysqlx://fjanisze:pass@localhost:33060/?ssl-mode=disabled');

$session->createSchema('testx');
$schema = $session->getSchema('testx');

$schema->createCollection('test_collection');
$coll = $schema->getCollection('test_collection');

?>

mysql> describe test_collection;

+-------+-------------------+------+-----+-----------+----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------------+------+-----+-----------+----------------------------+
| doc | json | YES | | NULL | |
| _id | varbinary(32) | NO | PRI | NULL | STORED GENERATED |
+-------+-------------------+------+-----+----------+-----------------------------+
2 rows in set (0.00 sec)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Adding Documents

.
.

$coll->add('{"name": "Marco", "age": 19, "job": "Programmatore"}')->execute();

$coll->add('{"name": "Lonardo", "age": 59, "job": "Paninaro"}')->execute();
$coll->add('{"name": "Riccardo", "age": 27, "job": "Cantante"}')->execute();
$coll→add('{"_id": "ID1", "name": "Carlotta", "age": 23, "job": "Programmatrice"}')->execute();

.
.

mysql> select * from test_collection;

+-----------------------------------------------------------------------------------------------------------------------------------------------+------------------------------
-------------------+
| doc | _id |
+-----------------------------------------------------------------------------------------------------------------------------------------------+------------------------------
-------------------+
| {"_id": "00005b555b140000000000000001", "age": 19, "job": "Programmatore", "name": "Marco"} | 00005b555b140000000000000001 |
| {"_id": "00005b555b140000000000000002", "age": 59, "job": "Paninaro", "name": "Lonardo"} | 00005b555b140000000000000002 |
| {"_id": "00005b555b140000000000000003", "age": 27, "job": "Cantante", "name": "Riccardo"} | 00005b555b140000000000000003 |
| {"_id": "ID1", "age": 23, "job": "Programmatrice", "name": "Carlotta"} | ID1 |
+-----------------------------------------------------------------------------------------------------------------------------------------------+------------------------------
--------------------+
4 rows in set (0.00 sec)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Adding documents

• Different data types can be provided as input for the ‘add’ operation, those
data types are implicitly converted using json_encode()

.
.
$sakila = ["name" => "Sakila", "age" => 17, "job" => "Singer"];

$coll->add($sakila)->execute();

$coll->add(["name" => "Sakila", "age" => 18, "job" => "Student"])→execute();

.
.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
CollectionAdd

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Modify a document

.
.
$coll->modify("job in ('Programmatore','Programmatrice')")->arrayAppend('job','Tassinaro')->set('Overworked','Yes')->execute();
$coll->modify("age > 25 and age < 60")->patch('{"Hobby" : ["Swimming","Dancing"]}')->execute();
.
.

mysql> select * from test_collection;

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
| doc | _id
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
| {"_id": "00005b555b14000000000000000b", "age": 19, "job": ["Programmatore", "Tassinaro"], "name": "Marco", "Overworked": "Yes"} | 00005b555b140000000000000001 |
| {"_id": "00005b555b140000000000000002", "age": 59, "job": "Paninaro", "name": "Lonardo","Hobby": ["Swimming", "Dancing"]} | 00005b555b140000000000000002 |
| {"_id": "00005b555b140000000000000003", "age": 27, "job": "Cantante", "name": "Riccardo","Hobby": ["Swimming", "Dancing"]} | 00005b555b140000000000000003 |
| {"_id": "ID1", "age": 23, "job": ["Programmatrice", "Tassinaro"], "name": "Carlotta", "Overworked": "Yes"} | ID1
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
4 rows in set (0.00 sec)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Modify a document

.
$coll->modify("job in ('Programmatore','Programmatrice')")->arrayAppend('job','Tassinaro')->set('Overworked','Yes')->execute();
$coll->modify("age > 25 and age < 60")->patch('{"Hobby" : ["Swimming","Dancing"]}')→execute();
.
$coll->modify("_id like 'ID1'")->patch('{"Overworked": null}')→execute();
$coll->modify("_id like 'ID1'")->unset('Overworked')->execute();
.

mysql> select * from test_collection;

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
| doc | _id
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
| {"_id": "00005b555b14000000000000000b", "age": 19, "job": ["Programmatore", "Tassinaro"], "name": "Marco", "Overworked": "Yes"} | 00005b555b140000000000000001 |
| {"_id": "00005b555b140000000000000002", "age": 59, "job": "Paninaro", "name": "Lonardo","Hobby": ["Swimming", "Dancing"]} | 00005b555b140000000000000002 |
| {"_id": "00005b555b140000000000000003", "age": 27, "job": "Cantante", "name": "Riccardo","Hobby": ["Swimming", "Dancing"]} | 00005b555b140000000000000003 |
| {"_id": "ID1", "age": 23, "job": ["Programmatrice", "Tassinaro"], "name": "Carlotta"} | ID1
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----
----------------------------------------------+
4 rows in set (0.00 sec)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
CollectionModify

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Search for a document

.
$age = 30;
$res = $coll->find('age > :age_param')->bind([ 'age_param' => $age ])->execute();

$data = $res->fetchAll();
.

array(1) {
[0]=>
array(4) {
["_id"]=>
string(28) "00005b555b14000000000000001e"
["age"]=>
int(59)
["job"]=>
string(8) "Paninaro"
["name"]=>
string(7) "Lonardo"
}
}

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
CollectionFind

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Removing documents

.
$coll->remove('age > :age_from and age < :age_to')->bind(['age_from' => 20, 'age_to' => 50])->limit(7)->execute();

$coll->remove('true')->sort('age desc')->limit(2)->execute();

$coll->remove('job in (\'Barista\', \'Programmatore\', \'Ballerino\', \'Programmatrice\')')->limit(5)->sort(['age desc', 'name asc'])-

>execute();
.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
CollectionRemove

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |