Undo the BearSSL RODATA->PROGMEM changes because there are some bad
performance regressions in EC server operations which can result in
timeouts and WDTs.
Keep the shrunked bearssl.a library as that is orthogonal to the PROGMEM
changes.
Rewrite all the integer math operations with const input parameters to
use PROGMEM properly (pgm_read_xx or memcpy_P), and move all the EC
order and generators and SHA OIDs to PROGMEM.
This frees around 1.2KB of heap for any SSL applications.
Also delete unneeded objects from the bearssl.a library to shrink the
GIT repo size.
Move additional constants to flash and use _P/pgm_read routines to
access them. Minimal runtime impact, but remove variables from RODATA
and gives addition 484 bytes of heap to SSL applications.
Fixes#6005
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
ref: d-a-v/esp82xx-nonos-linklayer#31
origin: #5902me-no-dev/ESPAsyncTCP#108
Following the links above is instructive.
To summarize:
* currently and from a long time lwIP tcp client connections always uses the same tcp source port number right after boot
* this port number is increased everytime a new one is needed (= new tcp client connection)
(to be noted, linux has the same increasing behavior)
* when connecting to the same server (right after boot), the triplet (esp-ip-address, source port, destination port) are the same, and may hit remote server list of sockets in time-wait-state (previous connection unproperly closed from the same esp). Consequently the new connection fails when it happens.
* this is happening only when debugging (esp reboots often, in less time than time-wait expiration), so the nasty effect is amplified especially when bugs are being chased
* efforts had been done when espressif's lwIP implementation wasn't open source, with WiFiClient::setLocalPortStart() #632 but it must be explicitely called with a different random number at every reboot. Efficient but not ideal.
This PR uses espressif firmware's r_rand() everytime a new local source port is needed. A different source port number is now showed by tcpdump right after boot. Source port range and duplication is verified everytime in lwIP's src/core/tcp.c:tcp_new_port(). It is implemented as a local patch for upstream lwIP so it is valid not only with WiFiClient but also with @me-no-dev's Async libraries (they don't use WiFiClient).
WiFiClient::setLocalPortStart() is still usable with the same effects as before.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
There is a bug in the BearSSL PEM decoder when Windows EOLs (\r\n) are
passed in. Avoid the issue by silenly discarding \rs as they are read
from the PEM source in the C code, to keep my sanity by avoiding reworking
the pseudo-Forth parser code.
Fixes#5591
Scripts, makefiles, and users who do no changes will not have exceptions
enabled during builds. This should avoid the sketch inflation issue for
users who are space constrained, while allowing users who care about
exceptions to enable them through the IDE.
* lwip2: better handling of ipv4_addr/t type + 3 sntp servers
* bump lwip2 version
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* local CI runner: select build type
* new ipv4_addr/t definition makes things easier for IPAddress
* update local CI runner
* lwip2 changes
* lwip2: port esp-ping and espconn
Remove the -fno-jump-tables since the new toolchain places these tables
in ROM now. Rebuild using the toolchain. Saves 1-3KB of flash and
has 0 RAM impact plus may make certain bits marginally faster by using
a LUT instead of a if-else-else chain.
The complete toolchain, including mkspiffs, esptool, C, C++, newlib,
and others (BearSSL excluded) is now built and uploaded with a single
command to ensure repeatability and minimize manual mistakes. All
OSes and architectures are built at a time.
Update to 2.5.0-2 throught the chain.
* Make exceptions a configurable menu
Add a menu, Exceptions, which allows exceptions to be disabled for ROM
sensitive scripts. Default is enabled.
* Update to latest JSON builder
* Move to PROGMEM aware libc, allow PSTR in printf()
A Newlib (libc) patch is in progress to move the _P functions from inside
Arduino into first-class citizens in libc. This Arduino patch cleans up
code that's been migrated there. Binaries for the new libs are included
because it seems they're part of the Arduino git tree, and should be
replaced with @igrr built ones when/if the Newlib changes are accepted.
Notable changes/additions for Arduino:
Allow for use of PROGMEM based format and parameter strings in all
*printf functions. No need for copying PSTR()s into RAM before printing
them out (transparently saves heap space when using _P functions) and
makes it easier to print out constant strings for applications.
Add "%S" (capital-S) format that I've been told, but cannot verify,
is used in Arduino to specify a PROGMEM string parameter in printfs,
as an alias for "%s" since plain "%s" can now handle PROGMEM.
Optimized the memcpy_P, strnlen_P, and strncpy_P functions to use 32-bit
direct reads whenver possible (source and dest alignment mediated), but
there is still room for improvement in others.
Finally, move several constant arrays from RODATA into PROGMEM and
update their accessors. Among these are the ctype array, ~260 bytes,
mprec* arrays, ~300 bytes, and strings/daycounts in the time
formatting functions, ~200 bytes. All told, sketches will see from
300 to 800 additional RAM heap free on startup (depending on their
use of these routines).
* Fix merge error in #ifdef/#endif
* Fix host test using the newlib generic pgmspace.h
Host tests now use the sys/pgmspace.h for compiles instead of the
ESP8266-specific version.
* Update with rebuilt libraries using latest newlib
* Include binaries built directly from @igrr repo
Rebuild the binaries using a git clone of
https://github.com/igrr/newlib-xtensa
Build commands for posterity:
````
rm -rf ./xtensa-lx106-elf/
./configure --prefix=<DIR>/esp8266/tools/sdk/libc --with-newlib \
--enable-multilib --disable-newlib-io-c99-formats \
--disable-newlib-supplied-syscalls \
--enable-newlib-nano-formatted-io --enable-newlib-reent-small \
--enable-target-optspace \
--program-transform-name="s&^&xtensa-lx106-elf-&" \
--disable-option-checking --with-target-subdir=xtensa-lx106-elf \
--target=xtensa-lx106-elf
rm -f etc/config.cache
CROSS_CFLAGS="-fno-omit-frame-pointer -DSIGNAL_PROVIDED -DABORT_PROVIDED"\
" -DMALLOC_PROVIDED" \
PATH=<DIR>/esp8266/tools/xtensa-lx106-elf/bin/:$PATH \
make all install
````
* Fix merge define conflict in c_types.h
* Fix strlen_P misaligned source error
Include fix from newlib-xtensa/fix-strlen branch cleaning up misaligned
access on a non-aligned source string.
* Fix strlen_P and strcpy_P edge cases
Ran the included test suite on ESP8266 tstring.c with the following defines:
#define MAX_1 50
#define memcmp memcmp_P
#define memcpy memcpy_P
#define memmem memmem_P
#define memchr memchr_P
#define strcat strcat_P
#define strncat strncat_P
#define strcpy strcpy_P
#define strlen strlen_P
#define strnlen strnlen_P
#define strcmp strcmp_P
#define strncmp strncmp_P
Uncovered edge case and return value problems in the optimized versions of
the strnlen_P and strncpy_P functions. Corrected.
* Fix memcpy_P return value
memcpy-1.c test suite showed error in return value of memcpy_P. Correct it.
* Fix strnlen_P/strlen_P off-by-4 error
Random crashes, often on String constructors using a PSTR, would occur due
to the accelerated strnlen_P going past the end of the string. Would make
debug builds fail, too (ESP.getVersionString() failure).
Fix to fall through to normal copy on a word that's got a 0 byte anywhere
in it.
* Add device tests for libc functional verification
Add test suite used to debug libc optimized _P functions to the device
tests.
* Rebuild from igrr's repo (same source as prior)
Rebuild .a from igrr's repo at 347260af117b4177389e69fd4d04169b11d87a97
* WIP - add exceptions
* Fix exception to have 0-terminator
* Move some exception constants to TEXT from RODATA
* Remove throw stubs
* Move more exception stuff to ROM
* Enable exceptions in platform.io
* Remove atexit, is duplicated in rebuilt lib
Need to look at the quick-toolchain options, there seems to be a definition
for atexit defined there (libgcc?) that needs to be excised. For now,
remove our local do-nothing copy.
* Update libgcc to remove soft-fp functions
The esp-quick-toolchain generated libgcc.a needed to have the soft-FP routines
that are in ROM removed from it. Remove them in the new esp-quick-toolchain
and update.
* Fix merge typos in Makefile
* Add unhandled exception handler to postmortem
* Return our atexit() handler
* Latest stdc++, minimize exception emercengy area
* Remove atexit from newlib
atexit was defined in newlib strongly, but we also define a noop atexit in core.
Since we never exit, use the core's noop and delete the atexit from libc.a
Updated in esp-quick-toolchain as well.
* Move __FUNCTION__ static strings to PROGMEM
__FUNCTION__ is unlikely to be a timing sensitive variable, so move it to
PROGMEM and not RODATA (RAM) using linker magic.
asserts() now should take no RAM for any strings.
* Clean up linker file, update to latest stdc++
* Update to latest stdc++ which doesn't call strerror
* Update to GCC5.1 exception emergency allocator
Using GCC 5.1's emergency memory allocator for exceptions, much less
space is required in programs which do not use exceptions and when
space is allocated it is managed more efficiently.
* Initial try with new compiler toolchain
* Include newlib built from esp-quick-toolchain
* Update JSON with all new esp-quick-toolchain builds
* Use 64bit Windows compiler on 64bit Windows
* Dump std::exception.what() when possible
When doing the panic on unhandled exceptions, try and grab the
.what() pointer and dump it as part of the termination info.
Makes it easy to see mem errors (std::bad_alloc) or std::runtime_error
strings.
* Use scripted install from esp-quick-toolchain
Makes sure proper libraries and includes are present by using a
scripted installation from esp-quick-install instead of a manual
one.
* Update eqk to remove atexit, fix packaging diff
* Update to BearSSL 0.6+ release, add AES_CCM modes
Pull in latest BearSSL head (0.6 + minor additions) release and add AES_CCM
modes to the encryption options.
* Enable the aes_ccm initialization in client/server
* Initial attempt
* Working code with second stack thunking
* Remove #ifdefs in .S file, not needed.
* Clean up thunks and remove separate stack flag
* Fix PIO assembler errors
* Remove #ifdef code changes, ensure same code as PC
Remove "#ifdef ESP8266;...;#else;...;#endif" brackets in BearSSL to
ensure the host-tested code is the same as the ESP8266-run code.
* Move to latest BearSSL w/EC progmem savings
* Merge with master
* Add br_thunk_* calls to do ref counting, painting
Add reference counting br_thunk_add/del_ref() to replace stack handling code
in the class.
Add in stack painting and max usage calculation.
* Add in postmortem stack dump hooks
When a crash occurs while in the second stack, dump the BSSL stack and
then also the stack that it was called from (either cont or sys).
* Update stack dump to match decoder expectations
* Move thunk to code core for linkiage
The thunk code needs to be visible to the core routines, so move it to the
cores/esp8266 directory. Probably need to refactor the stack setup and the
bearssl portion to avoid dependency on bearssl libs in cores/esp8266
* Add 2nd stack dump utility routine
* Refactor once more, update stack size, add stress
Make stack_thunks generic, remove bearssl include inside of cores/esp8266.
Allocate the stack on a WiFiServerSecure object creation to avoid
fragmentation since we will need to allocate the stack to do any
connected work, anyway.
A stress test is now included which checks the total BearSSL second
stack usage for a variety of TLS handshake and certificate options
from badssl.org.
* Update to latest to-thunks branch
* Add BearSSL device test using stack stress
Run a series of SSL connection and transmission tests that stress
BearSSL and its stack usage to the device tests.
Modify device tests to include a possible SPIFFS generation and
upload when a make_spiffs.py file is present in a test directory.
* Use bearssl/master branch, not /to-thunks branch
Update to use the merged master branch of bearssl. Should have no code
changes.
Apply patches developed by Sze Yiu Chau <schau@purdue.edu> which
correct a vulnerability in X509 parsing.
See CVE-2018-16150 and CVE-2018-16149 for more info.
CA certification validation was broken by a change put in during warning
cleanup a long time ago. This binary now includes the 1-line correction
and HTTPSRequestCACert now works again (before was failing
because the key usages in certs were not properly read).
* update to lwIP-2.1.0rc1: partial SACK support
fix#4176
* hash fix
* get some flash back due to mistake in conf (fragmentation & reassembly was incorrectly enabled)
(ahah I scared you)
* add missing include files
* update to lwip-2.1.0(release) + remove unused lwIP's include files
* lwIP release 2.1.0, SACK is now default, bigger, no-SACK is selectable
* fix ldscript
* pio
* rename 'sack' option to 'feat'ure option, + IP fragmentation/reassembly
* merge, fix pio
* change internal/hidden string
* pio: more lwip2 configuration: + without sack for no change in flash footprint
* fwupdate
* fw update to latest version:
WPA working, WEP+Open disabled by default. Need API change.
* helpers to follow sdk updates
* remove compare scripts - made a separate PR for them
* add wep api, restore original espressif comment (wep enabled does not prevent wpa)
* libmain was not up to date
* experimental: DTIM setting in WiFi.setSleepMode(WIFI_LIGHT/MODEM_SLEEP, DTIM-value)
with new getter: .getListenInterval() / .isSleepLevelMax()
* fixes
* fix debug message
* when not using listenInterval, set wifi sleep level to min
* update documentation
* update doc
Pull in latest BearSSL head (0.6 + minor additions) release and add AES_CCM
modes to the encryption options. Enable the aes_ccm initialization in client/server
The EC mul20 and square20 code was identical in two different files,
but because these copies were static, we ended up with an extra 6k of
duplicated code. Updated BearSSL to make them shared, saving 6KB.
* interactive example: update with option for using DHCP again after using static IP
* lwip2: avoid crash when IP address is set to 0(any) by dhcp not getting its lease renewal in due time
* lwip2: automatically remove oldest PCBs in time-wait state, limit their number
thanks to @me-no-dev 07f4d4c241 (diff-f8258e71e25fb9985ca3799e3d8b88ecR399)
* faq: update about tcpCleanup()
* lwip2: add a macro HAS_PHY_CAPTURE=1 indicating capture facility is available
BearSSL (https://www.bearssl.org) is a TLS(SSL) library written by
Thomas Pornin that is optimized for lower-memory embedded systems
like the ESP8266. It supports a wide variety of modern ciphers and
is unique in that it doesn't perform any memory allocations during
operation (which is the unfortunate bane of the current axTLS).
BearSSL is also absolutely focused on security and by default performs
all its security checks on x.509 certificates during the connection
phase (but if you want to be insecure and dangerous, that's possible
too).
While it does support unidirectional SSL buffers, like axTLS,
as implemented the ESP8266 wrappers only support bidirectional
buffers. These bidirectional buffers avoid deadlocks in protocols
which don't have well separated receive and transmit periods.
This patch adds several classes which allow connecting to TLS servers
using this library in almost the same way as axTLS:
BearSSL::WiFiClientSecure - WiFiClient that supports TLS
BearSSL::WiFiServerSecure - WiFiServer supporting TLS and client certs
It also introduces objects for PEM/DER encoded keys and certificates:
BearSSLX509List - x.509 Certificate (list) for general use
BearSSLPrivateKey - RSA or EC private key
BearSSLPublicKey - RSA or EC public key (i.e. from a public website)
Finally, it adds a Certificate Authority store object which lets
BearSSL access a set of trusted CA certificates on SPIFFS to allow it
to verify the identity of any remote site on the Internet, without
requiring RAM except for the single matching certificate.
CertStoreSPIFFSBearSSL - Certificate store utility
Client certificates are supported for the BearSSL::WiFiClientSecure, and
what's more the BearSSL::WiFiServerSecure can also *require* remote clients
to have a trusted certificate signed by a specific CA (or yourself with
self-signing CAs).
Maximum Fragment Length Negotiation probing and usage are supported, but
be aware that most sites on the Internet don't support it yet. When
available, you can reduce the memory footprint of the SSL client or server
dramatically (i.e. down to 2-8KB vs. the ~22KB required for a full 16K
receive fragment and 512b send fragment). You can also manually set a
smaller fragment size and guarantee at your protocol level all data will
fit within it.
Examples are included to show the usage of these new features.
axTLS has been moved to its own namespace, "axtls". A default "using"
clause allows existing apps to run using axTLS without any changes.
The BearSSL::WiFi{client,server}Secure implements the axTLS
client/server API which lets many end user applications take advantage
of BearSSL with few or no changes.
The BearSSL static library used presently is stored at
https://github.com/earlephilhower/bearssl-esp8266 and can be built
using the standard ESP8266 toolchain.
- Fixes WiFi not being able to reconnect after SDK update (#4398)
- Fixes increased current in light sleep mode
- Fixes return value documentation for wifi_{get,set}_country
