Merge remote-tracking branch 'upstream/master' into basvs-merge-upstream

basvs · basvs · commit ded5c5ef46e9 · 2017-08-20T13:51:47.000+02:00
diff --git a/docs/library/ubinascii.rst b/docs/library/ubinascii.rst
@@ -29,8 +29,12 @@ Functions
 
 .. function:: a2b_base64(data)
 
-   Convert Base64-encoded data to binary representation. Returns bytes string.
+   Decode base64-encoded data, ignoring invalid characters in the input.
+   Conforms to `RFC 2045 s.6.8 <https://tools.ietf.org/html/rfc2045#section-6.8>`_.
+   Returns a bytes object.
 
 .. function:: b2a_base64(data)
 
-   Encode binary data in Base64 format. Returns string.
+   Encode binary data in base64 format, as in `RFC 3548
+   <https://tools.ietf.org/html/rfc3548.html>`_. Returns the encoded data
+   followed by a newline character, as a bytes object.
diff --git a/docs/library/usocket.rst b/docs/library/usocket.rst
@@ -9,12 +9,6 @@
 
 This module provides access to the BSD socket interface.
 
-.. admonition:: Difference to CPython
-   :class: attention
-
-   CPython used to have a ``socket.error`` exception which is now deprecated,
-   and is an alias of `OSError`. In MicroPython, use `OSError` directly.
-
 .. admonition:: Difference to CPython
    :class: attention
 
@@ -27,11 +21,47 @@ This module provides access to the BSD socket interface.
 Socket address format(s)
 ------------------------
 
-The functions below which expect a network address, accept it in the format of
-*(ipv4_address, port)*, where *ipv4_address* is a string with dot-notation numeric
-IPv4 address, e.g. ``"8.8.8.8"``, and port is integer port number in the range
-1-65535. Note the domain names are not accepted as *ipv4_address*, they should be
-resolved first using `usocket.getaddrinfo()`.
+The native socket address format of the ``usocket`` module is an opaque data type
+returned by `getaddrinfo` function, which must be used to resolve textual address
+(including numeric addresses)::
+
+    sockaddr = usocket.getaddrinfo('www.micropython.org', 80)[0][-1]
+    # You must use getaddrinfo() even for numeric addresses
+    sockaddr = usocket.getaddrinfo('127.0.0.1', 80)[0][-1]
+    # Now you can use that address
+    sock.connect(addr)
+
+Using `getaddrinfo` is the most efficient (both in terms of memory and processing
+power) and portable way to work with addresses.
+
+However, ``socket`` module (note the difference with native MicroPython
+``usocket`` module described here) provides CPython-compatible way to specify
+addresses using tuples, as described below. Note that depending on a
+`MicroPython port`, ``socket`` module can be builtin or need to be
+installed from `micropython-lib` (as in the case of `MicroPython Unix port`),
+and some ports still accept only numeric addresses in the tuple format,
+and require to use `getaddrinfo` function to resolve domain names.
+
+Summing up:
+
+* Always use `getaddrinfo` when writing portable applications.
+* Tuple addresses described below can be used as a shortcut for
+  quick hacks and interactive use, if your port supports them.
+
+Tuple address format for ``socket`` module:
+
+* IPv4: *(ipv4_address, port)*, where *ipv4_address* is a string with
+  dot-notation numeric IPv4 address, e.g. ``"8.8.8.8"``, and *port* is and
+  integer port number in the range 1-65535. Note the domain names are not
+  accepted as *ipv4_address*, they should be resolved first using
+  `usocket.getaddrinfo()`.
+* IPv6: *(ipv6_address, port, flowinfo, scopeid)*, where *ipv6_address*
+  is a string with colon-notation numeric IPv6 address, e.g. ``"2001:db8::1"``,
+  and *port* is an integer port number in the range 1-65535. *flowinfo*
+  must be 0. *scopeid* is the interface scope identifier for link-local
+  addresses. Note the domain names are not accepted as *ipv6_address*,
+  they should be resolved first using `usocket.getaddrinfo()`. Availability
+  of IPv6 support depends on a `MicroPython port`.
 
 Functions
 ---------
@@ -250,3 +280,13 @@ Methods
    the length of *buf*.
 
    Return value: number of bytes written.
+
+.. exception:: socket.error
+
+   MicroPython does NOT have this exception.
+
+   .. admonition:: Difference to CPython
+        :class: attention
+
+        CPython used to have a ``socket.error`` exception which is now deprecated,
+        and is an alias of `OSError`. In MicroPython, use `OSError` directly.
diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst
@@ -54,10 +54,10 @@ Glossary
         separate project
         `micropython-lib <https://github.com/micropython/micropython-lib>`_
         which provides implementations for many modules from CPython's
-        standard library. However, large subset of these modules required
+        standard library. However, large subset of these modules require
         POSIX-like environment (Linux, MacOS, Windows may be partially
         supported), and thus would work or make sense only with MicroPython
-        Unix port. Some subset of modules however usable for baremetal ports
+        Unix port. Some subset of modules is however usable for baremetal ports
         too.
 
         Unlike monolithic :term:`CPython` stdlib, micropython-lib modules
@@ -68,7 +68,13 @@ Glossary
         MicroPython supports different :term:`boards <board>`, RTOSes,
         and OSes, and can be relatively easily adapted to new systems.
         MicroPython with support for a particular system is called a
-        "port" to that system.
+        "port" to that system. Different ports may have widely different
+        functionality. This documentation is intended to be a reference
+        of the generic APIs available across different ports ("MicroPython
+        core"). Note that some ports may still omit some APIs described
+        here (e.g. due to resource constraints). Any such differences,
+        and port-specific extensions beyond MicroPython core functionality,
+        would be described in the separate port-specific documentation.
 
     MicroPython Unix port
         Unix port is one of the major :term:`MicroPython ports <MicroPython port>`.
diff --git a/extmod/modubinascii.c b/extmod/modubinascii.c
@@ -105,54 +105,64 @@ mp_obj_t mod_binascii_unhexlify(mp_obj_t data) {
 }
 MP_DEFINE_CONST_FUN_OBJ_1(mod_binascii_unhexlify_obj, mod_binascii_unhexlify);
 
+// If ch is a character in the base64 alphabet, and is not a pad character, then
+// the corresponding integer between 0 and 63, inclusively, is returned.
+// Otherwise, -1 is returned.
+static int mod_binascii_sextet(byte ch) {
+    if (ch >= 'A' && ch <= 'Z') {
+        return ch - 'A';
+    } else if (ch >= 'a' && ch <= 'z') {
+        return ch - 'a' + 26;
+    } else if (ch >= '0' && ch <= '9') {
+        return ch - '0' + 52;
+    } else if (ch == '+') {
+        return 62;
+    } else if (ch == '/') {
+        return 63;
+    } else {
+        return -1;
+    }
+}
+
 mp_obj_t mod_binascii_a2b_base64(mp_obj_t data) {
     mp_buffer_info_t bufinfo;
     mp_get_buffer_raise(data, &bufinfo, MP_BUFFER_READ);
-    if (bufinfo.len % 4 != 0) {
-        mp_raise_ValueError("incorrect padding");
-    }
+    byte *in = bufinfo.buf;
 
     vstr_t vstr;
-    byte *in = bufinfo.buf;
-    if (bufinfo.len == 0) {
-        vstr_init_len(&vstr, 0);
-    }
-    else {
-        vstr_init_len(&vstr, ((bufinfo.len / 4) * 3) - ((in[bufinfo.len-1] == '=') ? ((in[bufinfo.len-2] == '=') ? 2 : 1 ) : 0));
-    }
-    byte *out = (byte*)vstr.buf;
-    for (mp_uint_t i = bufinfo.len; i; i -= 4) {
-        char hold[4];
-        for (int j = 4; j--;) {
-            if (in[j] >= 'A' && in[j] <= 'Z') {
-                hold[j] = in[j] - 'A';
-            } else if (in[j] >= 'a' && in[j] <= 'z') {
-                hold[j] = in[j] - 'a' + 26;
-            } else if (in[j] >= '0' && in[j] <= '9') {
-                hold[j] = in[j] - '0' + 52;
-            } else if (in[j] == '+') {
-                hold[j] = 62;
-            } else if (in[j] == '/') {
-                hold[j] = 63;
-            } else if (in[j] == '=') {
-                if (j < 2 || i > 4) {
-                    mp_raise_ValueError("incorrect padding");
-                }
-                hold[j] = 64;
-            } else {
-                mp_raise_ValueError("invalid character");
+    vstr_init(&vstr, (bufinfo.len / 4) * 3 + 1); // Potentially over-allocate
+    byte *out = (byte *)vstr.buf;
+
+    uint shift = 0;
+    int nbits = 0; // Number of meaningful bits in shift
+    bool hadpad = false; // Had a pad character since last valid character
+    for (size_t i = 0; i < bufinfo.len; i++) {
+        if (in[i] == '=') {
+            if ((nbits == 2) || ((nbits == 4) && hadpad)) {
+                nbits = 0;
+                break;
             }
+            hadpad = true;
         }
-        in += 4;
 
-        *out++ = (hold[0]) << 2 | (hold[1]) >> 4;
-        if (hold[2] != 64) {
-            *out++ = (hold[1] & 0x0F) << 4 | hold[2] >> 2;
-            if (hold[3] != 64) {
-                *out++ = (hold[2] & 0x03) << 6 | hold[3];
-            }
+        int sextet = mod_binascii_sextet(in[i]);
+        if (sextet == -1) {
+            continue;
+        }
+        hadpad = false;
+        shift = (shift << 6) | sextet;
+        nbits += 6;
+
+        if (nbits >= 8) {
+            nbits -= 8;
+            out[vstr.len++] = (shift >> nbits) & 0xFF;
         }
     }
+
+    if (nbits) {
+        mp_raise_ValueError("incorrect padding");
+    }
+
     return mp_obj_new_str_from_vstr(&mp_type_bytes, &vstr);
 }
 MP_DEFINE_CONST_FUN_OBJ_1(mod_binascii_a2b_base64_obj, mod_binascii_a2b_base64);
diff --git a/extmod/modussl_mbedtls.c b/extmod/modussl_mbedtls.c
@@ -38,6 +38,7 @@
 #include "py/nlr.h"
 #include "py/runtime.h"
 #include "py/stream.h"
+#include "py/obj.h"
 
 // mbedtls_time_t
 #include "mbedtls/platform.h"
@@ -268,6 +269,16 @@ STATIC mp_obj_ssl_socket_t *socket_new(mp_obj_t sock, struct ssl_args *args) {
     return o;
 }
 
+STATIC mp_obj_t mod_ssl_getpeercert(mp_obj_t o_in, mp_obj_t binary_form) {
+    mp_obj_ssl_socket_t *o = MP_OBJ_TO_PTR(o_in);
+    if (!mp_obj_is_true(binary_form)) {
+        mp_raise_NotImplementedError(NULL);
+    }
+    const mbedtls_x509_crt* peer_cert = mbedtls_ssl_get_peer_cert(&o->ssl);
+    return mp_obj_new_bytes(peer_cert->raw.p, peer_cert->raw.len);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_2(mod_ssl_getpeercert_obj, mod_ssl_getpeercert);
+
 STATIC void socket_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
     (void)kind;
     mp_obj_ssl_socket_t *self = MP_OBJ_TO_PTR(self_in);
@@ -345,6 +356,7 @@ STATIC const mp_rom_map_elem_t ussl_socket_locals_dict_table[] = {
     { MP_ROM_QSTR(MP_QSTR_write), MP_ROM_PTR(&mp_stream_write_obj) },
     { MP_ROM_QSTR(MP_QSTR_setblocking), MP_ROM_PTR(&socket_setblocking_obj) },
     { MP_ROM_QSTR(MP_QSTR_close), MP_ROM_PTR(&socket_close_obj) },
+    { MP_ROM_QSTR(MP_QSTR_getpeercert), MP_ROM_PTR(&mod_ssl_getpeercert_obj) },
 };
 
 STATIC MP_DEFINE_CONST_DICT(ussl_socket_locals_dict, ussl_socket_locals_dict_table);
diff --git a/py/binary.h b/py/binary.h
@@ -29,8 +29,9 @@
 #include "py/obj.h"
 
 // Use special typecode to differentiate repr() of bytearray vs array.array('B')
-// (underlyingly they're same).
-#define BYTEARRAY_TYPECODE 0
+// (underlyingly they're same).  Can't use 0 here because that's used to detect
+// type-specification errors due to end-of-string.
+#define BYTEARRAY_TYPECODE 1
 
 size_t mp_binary_get_size(char struct_type, char val_type, mp_uint_t *palign);
 mp_obj_t mp_binary_get_val_array(char typecode, void *p, mp_uint_t index);
diff --git a/tests/basics/struct2.py b/tests/basics/struct2.py
@@ -40,3 +40,30 @@
     struct.calcsize('0z')
 except:
     print('Exception')
+
+# check that a count without a type specifier raises an exception
+
+try:
+    struct.calcsize('1')
+except:
+    print('Exception')
+
+try:
+    struct.pack('1')
+except:
+    print('Exception')
+
+try:
+    struct.pack_into('1', bytearray(4), 0, 'xx')
+except:
+    print('Exception')
+
+try:
+    struct.unpack('1', 'xx')
+except:
+    print('Exception')
+
+try:
+    struct.unpack_from('1', 'xx')
+except:
+    print('Exception')
diff --git a/tests/extmod/ubinascii_a2b_base64.py b/tests/extmod/ubinascii_a2b_base64.py
@@ -21,6 +21,13 @@
 print(binascii.a2b_base64(b'f4D+')) # convert '+'
 print(binascii.a2b_base64(b'MTIzNEFCQ0RhYmNk'))
 
+# Ignore invalid characters and pad sequences
+print(binascii.a2b_base64(b'Zm9v\n'))
+print(binascii.a2b_base64(b'Zm\x009v\n'))
+print(binascii.a2b_base64(b'Zm9v=='))
+print(binascii.a2b_base64(b'Zm9v==='))
+print(binascii.a2b_base64(b'Zm9v===YmFy'))
+
 try:
     print(binascii.a2b_base64(b'abc'))
 except ValueError:
diff --git a/tests/net_hosted/ssl_getpeercert.py b/tests/net_hosted/ssl_getpeercert.py
@@ -0,0 +1,21 @@
+# test ssl.getpeercert() method
+
+try:
+    import usocket as socket
+    import ussl as ssl
+except:
+    import socket
+    import ssl
+
+
+def test(peer_addr):
+    s = socket.socket()
+    s.connect(peer_addr)
+    s = ssl.wrap_socket(s)
+    cert = s.getpeercert(True)
+    print(type(cert), len(cert) > 100)
+    s.close()
+
+
+if __name__ == "__main__":
+    test(socket.getaddrinfo('micropython.org', 443)[0][-1])
diff --git a/tests/net_hosted/ssl_getpeercert.py.exp b/tests/net_hosted/ssl_getpeercert.py.exp
@@ -0,0 +1 @@
+<class 'bytes'> True
diff --git a/tools/mpy-tool.py b/tools/mpy-tool.py
@@ -239,7 +239,7 @@ def _unpack_qstr(self, ip):
     def dump(self):
         # dump children first
         for rc in self.raw_codes:
-            rc.freeze()
+            rc.freeze('')
         # TODO
 
     def freeze(self, parent_name):
diff --git a/zephyr/modusocket.c b/zephyr/modusocket.c
@@ -88,16 +88,16 @@ STATIC mp_obj_t format_inet_addr(struct sockaddr *addr, mp_obj_t port) {
     // We employ the fact that port and address offsets are the same for IPv4 & IPv6
     struct sockaddr_in6 *sockaddr_in6 = (struct sockaddr_in6*)addr;
     char buf[40];
-    net_addr_ntop(addr->family, &sockaddr_in6->sin6_addr, buf, sizeof(buf));
-    mp_obj_tuple_t *tuple = mp_obj_new_tuple(addr->family == AF_INET ? 2 : 4, NULL);
+    net_addr_ntop(addr->sa_family, &sockaddr_in6->sin6_addr, buf, sizeof(buf));
+    mp_obj_tuple_t *tuple = mp_obj_new_tuple(addr->sa_family == AF_INET ? 2 : 4, NULL);
 
     tuple->items[0] = mp_obj_new_str(buf, strlen(buf), false);
     // We employ the fact that port offset is the same for IPv4 & IPv6
     // not filled in
     //tuple->items[1] = mp_obj_new_int(ntohs(((struct sockaddr_in*)addr)->sin_port));
     tuple->items[1] = port;
 
-    if (addr->family == AF_INET6) {
+    if (addr->sa_family == AF_INET6) {
         tuple->items[2] = MP_OBJ_NEW_SMALL_INT(0); // flow_info
         tuple->items[3] = MP_OBJ_NEW_SMALL_INT(sockaddr_in6->sin6_scope_id);
     }
diff --git a/zephyr/modzephyr.c b/zephyr/modzephyr.c
diff --git a/zephyr/prj_96b_carbon.conf b/zephyr/prj_96b_carbon.conf
diff --git a/zephyr/prj_base.conf b/zephyr/prj_base.conf
