Move BearSSL from STACK_PROXY to a real, thunked 2nd stack (#5168)

* Update to BearSSL 0.6+ release, add AES_CCM modes

Pull in latest BearSSL head (0.6 + minor additions) release and add AES_CCM
modes to the encryption options.

* Enable the aes_ccm initialization in client/server

* Initial attempt

* Working code with second stack thunking

* Remove #ifdefs in .S file, not needed.

* Clean up thunks and remove separate stack flag

* Fix PIO assembler errors

* Remove #ifdef code changes, ensure same code as PC

Remove "#ifdef ESP8266;...;#else;...;#endif" brackets in BearSSL to
ensure the host-tested code is the same as the ESP8266-run code.

* Move to latest BearSSL w/EC progmem savings

* Merge with master

* Add br_thunk_* calls to do ref counting, painting

Add reference counting br_thunk_add/del_ref() to replace stack handling code
in the class.

Add in stack painting and max usage calculation.

* Add in postmortem stack dump hooks

When a crash occurs while in the second stack, dump the BSSL stack and
then also the stack that it was called from (either cont or sys).

* Update stack dump to match decoder expectations

* Move thunk to code core for linkiage

The thunk code needs to be visible to the core routines, so move it to the
cores/esp8266 directory.  Probably need to refactor the stack setup and the
bearssl portion to avoid dependency on bearssl libs in cores/esp8266

* Add 2nd stack dump utility routine

* Refactor once more, update stack size, add stress

Make stack_thunks generic, remove bearssl include inside of cores/esp8266.

Allocate the stack on a WiFiServerSecure object creation to avoid
fragmentation since we will need to allocate the stack to do any
connected work, anyway.

A stress test is now included which checks the total BearSSL second
stack usage for a variety of TLS handshake and certificate options
from badssl.org.

* Update to latest to-thunks branch

* Add BearSSL device test using stack stress

Run a series of SSL connection and transmission tests that stress
BearSSL and its stack usage to the device tests.

Modify device tests to include a possible SPIFFS generation and
upload when a make_spiffs.py file is present in a test directory.

* Use bearssl/master branch, not /to-thunks branch

Update to use the merged master branch of bearssl.  Should have no code
changes.

tests/README.md

@@ -114,6 +114,7 @@ Some tests need to connect to WiFi AP or to the PC running the tests. In the tes
 
 Environment variables can also be used to pass some information from the test code to the host side helper. To do that, test code can set an environment variable using `setenv` C function. Then the `teardown` host side helper can obtain the value of that variable using `request_env` function defined in `mock_decorators`.
 
+A SPIFFS filesystem may be generated on the host and uploade before a test by including a file called `make_spiffs.py` in the individual test directory.
 
 ### Building and running the tests
 

tests/device/Makefile

@@ -5,6 +5,7 @@ ESP8266_CORE_PATH ?= ../..
 BUILD_DIR ?= $(PWD)/.build
 HARDWARE_DIR ?= $(PWD)/.hardware
 ESPTOOL ?= $(ESP8266_CORE_PATH)/tools/esptool/esptool
+MKSPIFFS ?= $(ESP8266_CORE_PATH)/tools/mkspiffs/mkspiffs
 UPLOAD_PORT ?= $(shell ls /dev/tty* | grep -m 1 -i USB)
 UPLOAD_BAUD ?= 921600
 UPLOAD_BOARD ?= nodemcu
@@ -55,6 +56,16 @@ ifneq ("$(NO_BUILD)","1")
 endif
 ifneq ("$(NO_UPLOAD)","1")
 	@test -n "$(UPLOAD_PORT)" || (echo "Failed to detect upload port, please export UPLOAD_PORT manually" && exit 1)
+	@test -e $(dir $@)/make_spiffs.py && (echo "Generating and uploading SPIFFS" && \
+		(cd $(dir $@) && python ./make_spiffs.py) && \
+		$(MKSPIFFS) --create $(dir $@)data/ --size 0xFB000 \
+			--block 8192 --page 256 $(LOCAL_BUILD_DIR)/spiffs.img && \
+		$(ESPTOOL) $(UPLOAD_VERBOSE_FLAG) \
+			-cp $(UPLOAD_PORT) \
+			-cb $(UPLOAD_BAUD) \
+			-cd $(UPLOAD_BOARD) \
+			-ca 0x300000 \
+			-cf $(LOCAL_BUILD_DIR)/spiffs.img ) || (echo "No SPIFFS to upload")
 	@echo Uploading binary
 	$(SILENT)$(ESPTOOL) $(UPLOAD_VERBOSE_FLAG)  \
 		-cp $(UPLOAD_PORT) \

tests/device/libraries/BSTest/runner.py

@@ -132,7 +132,7 @@ class BSTestRunner(object):
 
     def run_test(self, index):
         self.sp.sendline('{}'.format(index))
-        timeout = 10
+        timeout = 20 # 10
         while timeout > 0:
             res = self.sp.expect(['>>>>>bs_test_start', EOF, TIMEOUT])
             if res == 0:

tests/device/test_BearSSL/make_spiffs.py

@@ -0,0 +1,66 @@
+#!/usr/bin/python
+
+# This script pulls the list of Mozilla trusted certificate authorities
+# from the web at the "mozurl" below, parses the file to grab the PEM
+# for each cert, and then generates DER files in a new ./data directory
+# Upload these to a SPIFFS filesystem and use the CertManager to parse
+# and use them for your outgoing SSL connections.
+#
+# Script by Earle F. Philhower, III.  Released to the public domain.
+
+import csv
+import os
+from subprocess import Popen, PIPE, call
+import urllib2
+try:
+    # for Python 2.x
+    from StringIO import StringIO
+except ImportError:
+    # for Python 3.x
+    from io import StringIO
+
+# Mozilla's URL for the CSV file with included PEM certs
+mozurl = "https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReportPEMCSV"
+
+# Load the manes[] and pems[] array from the URL
+names = []
+pems = []
+response = urllib2.urlopen(mozurl)
+csvData = response.read()
+csvReader = csv.reader(StringIO(csvData))
+for row in csvReader:
+    names.append(row[0]+":"+row[1]+":"+row[2])
+    pems.append(row[28])
+del names[0] # Remove headers
+del pems[0] # Remove headers
+
+# Try and make ./data, skip if present
+try:
+    os.mkdir("data")
+except:
+    pass
+
+derFiles = []
+idx = 0
+# Process the text PEM using openssl into DER files
+for i in range(0, len(pems)):
+    certName = "data/ca_%03d.der" % (idx);
+    thisPem = pems[i].replace("'", "")
+    print names[i] + " -> " + certName
+    ssl = Popen(['openssl','x509','-inform','PEM','-outform','DER','-out', certName], shell = False, stdin = PIPE)
+    pipe = ssl.stdin
+    pipe.write(thisPem)
+    pipe.close()
+    ssl.wait()
+    if os.path.exists(certName):
+        derFiles.append(certName)
+        idx = idx + 1
+
+if os.path.exists("data/certs.ar"):
+    os.unlink("data/certs.ar");
+
+arCmd = ['ar', 'q', 'data/certs.ar'] + derFiles;
+call( arCmd )
+
+for der in derFiles:
+    os.unlink(der)

tests/device/test_BearSSL/test_BearSSL.ino

@@ -0,0 +1,220 @@
+// Stress test the BearSSL connection options to determine
+// maximum memory use for different SSL connections and
+// SPIFFS certstore usage.  Before running you need to run
+// certs-from-mozilla.py and upload the generated SPIFFS file.
+//
+// For more info on CertStores, see the BearSSL_CertStore example
+//
+// November 2018 by Earle F. Philhower, III
+// Released to the public domain
+
+#include <Arduino.h>
+#include <BSTest.h>
+#include <ESP8266WiFi.h>
+#include <CertStoreBearSSL.h>
+#include <FS.h>
+#include <time.h>
+#include <StackThunk.h>
+
+extern "C" {
+#include "user_interface.h"
+}
+
+BS_ENV_DECLARE();
+
+void setClock();
+
+// A single, global CertStore which can be used by all
+// connections.  Needs to stay live the entire time any of
+// the WiFiClientBearSSLs are present.
+BearSSL::CertStore certStore;
+
+// NOTE: The CertStoreFile virtual class may migrate to a templated
+// model in a future release. Expect some changes to the interface,
+// no matter what, as the SD and SPIFFS filesystem get unified.
+
+class SPIFFSCertStoreFile : public BearSSL::CertStoreFile {
+  public:
+    SPIFFSCertStoreFile(const char *name) {
+      _name = name;
+    };
+    virtual ~SPIFFSCertStoreFile() override {};
+
+    // The main API
+    virtual bool open(bool write = false) override {
+      _file = SPIFFS.open(_name, write ? "w" : "r");
+      return _file;
+    }
+    virtual bool seek(size_t absolute_pos) override {
+      return _file.seek(absolute_pos, SeekSet);
+    }
+    virtual ssize_t read(void *dest, size_t bytes) override {
+      return _file.readBytes((char*)dest, bytes);
+    }
+    virtual ssize_t write(void *dest, size_t bytes) override {
+      return _file.write((uint8_t*)dest, bytes);
+    }
+    virtual void close() override {
+      _file.close();
+    }
+
+  private:
+    File _file;
+    const char *_name;
+};
+
+SPIFFSCertStoreFile certs_idx("/certs.idx"); // Generated by the ESP8266
+SPIFFSCertStoreFile certs_ar("/certs.ar"); // Uploaded by the user
+
+
+void setup()
+{
+    Serial.begin(115200);
+    Serial.setDebugOutput(true);
+    WiFi.persistent(false);
+    WiFi.mode(WIFI_STA);
+    WiFi.begin(getenv("STA_SSID"), getenv("STA_PASS"));
+    while (WiFi.status() != WL_CONNECTED) {
+        delay(500);
+    }
+    setClock();
+    SPIFFS.begin();
+    int numCerts = certStore.initCertStore(&certs_idx, &certs_ar);
+    Serial.printf("Number of CA certs read: %d\n", numCerts);
+    if (numCerts == 0) {
+        Serial.printf("No certs found. Did you run certs-from-mozill.py and upload the SPIFFS directory before running?\n");
+        REQUIRE(1==0);
+    }
+    BS_RUN(Serial);
+}
+
+static void stopAll()
+{
+    WiFiClient::stopAll();
+}
+
+static void disconnectWiFI()
+{
+    wifi_station_disconnect();
+}
+
+
+// Set time via NTP, as required for x.509 validation
+void setClock() {
+  configTime(3 * 3600, 0, "pool.ntp.org", "time.nist.gov");
+
+  Serial.print("Waiting for NTP time sync: ");
+  time_t now = time(nullptr);
+  while (now < 8 * 3600 * 2) {
+    delay(500);
+    Serial.print(".");
+    now = time(nullptr);
+  }
+  Serial.println("");
+  struct tm timeinfo;
+  gmtime_r(&now, &timeinfo);
+  Serial.print("Current time: ");
+  Serial.print(asctime(&timeinfo));
+}
+
+// Try and connect using a WiFiClientBearSSL to specified host:port and dump URL
+void fetchURL(BearSSL::WiFiClientSecure *client, const char *host, const uint16_t port, const char *path) {
+  if (!path) {
+    path = "/";
+  }
+
+  Serial.printf("Trying: %s:443...", host);
+  client->connect(host, port);
+  if (!client->connected()) {
+    Serial.printf("*** Can't connect. ***\n-------\n");
+    return;
+  }
+  Serial.printf("Connected!\n-------\n");
+  client->write("GET ");
+  client->write(path);
+  client->write(" HTTP/1.0\r\nHost: ");
+  client->write(host);
+  client->write("\r\nUser-Agent: ESP8266\r\n");
+  client->write("\r\n");
+  uint32_t to = millis() + 5000;
+  if (client->connected()) {
+    do {
+      char tmp[32];
+      memset(tmp, 0, 32);
+      int rlen = client->read((uint8_t*)tmp, sizeof(tmp) - 1);
+      yield();
+      if (rlen < 0) {
+        break;
+      }
+      // Only print out first line up to \r, then abort connection
+      char *nl = strchr(tmp, '\r');
+      if (nl) {
+        *nl = 0;
+        Serial.print(tmp);
+        break;
+      }
+      Serial.print(tmp);
+    } while (millis() < to);
+  }
+  client->stop();
+  Serial.printf("\n-------\n");
+}
+
+
+int run(const char *str)
+{
+    BearSSL::WiFiClientSecure *bear = new BearSSL::WiFiClientSecure();
+    // Integrate the cert store with this connection
+    bear->setCertStore(&certStore);
+
+    char buff[100];
+    uint32_t maxUsage = 0;
+    stack_thunk_repaint();
+    sprintf(buff, "%s.badssl.com", str);
+    Serial.printf("%s: ", buff);
+    fetchURL(bear, buff, 443, "/");
+    Serial.printf("Stack: %d\n", stack_thunk_get_max_usage());
+    maxUsage = std::max(maxUsage, stack_thunk_get_max_usage());
+    delete bear;
+
+    printf("\n\n\nMAX THUNK STACK USAGE: %d\n", maxUsage);
+    return maxUsage;
+}
+
+#define TC(x) TEST_CASE("BearSSL - Maximum stack usage < 5600 bytes @ "x".badssl.org", "[bearssl]") { REQUIRE(run(x) < 5600); }
+
+TC("expired")
+TC("wrong.host")
+TC("self-signed")
+TC("untrusted-root")
+TC("revoked")
+TC("pinning-test")
+TC("no-common-name")
+TC("no-subject")
+TC("incomplete-chain")
+TC("sha1-intermediate")
+TC("sha256")
+TC("sha384")
+TC("sha512")
+TC("1000-sans")
+// TC("10000-sans") // Runs for >10 seconds, so causes false failure.  Covered by the 1000 SAN anyway
+TC("ecc256")
+TC("ecc384")
+TC("rsa2048")
+TC("rsa4096")
+TC("extended-validation")
+TC("dh480")
+TC("dh512")
+TC("dh1024")
+TC("dh2048")
+TC("dh-small-subgroup")
+TC("dh-composite")
+TC("static-rsa")
+TC("tls-v1-0")
+TC("tls-v1-1")
+TC("tls-v1-2")
+TC("invalid-expected-sct")
+
+void loop() {
+}
+