* Clean up minor warnings from LGTM.com
LGTM (Semmie) is a tool, bought by GitHub last year, that conducts basic
linting tasks on code and HTML.
Clean up the warnings identified in the latest report:
https://lgtm.com/projects/g/esp8266/Arduino/?mode=list
No functionality should change, however this may fix some issues with
the perl utilities not exiting properly on a Ctrl-C from the command
line.
* Back out HTML changes and rerun boards.txt.py
Run valgrind on host mock example runs to catch more bugs in CI. These
tests would have caught the problem in #7464 before users did.
Add a list of some randomly picked examples to run, and add an option to
run the loop exactly once in the host mock routine, so the test will
actually exit under valgrind.
In the HTTPS example we were using a fingerprint which changes almost
daily as the github.com certificates are regenerated. Replace this with
a trust anchor based on the ultimate root CA that github.com uses to
sign their certificates. Assuming they don't change CAs, this
certificate should be good until 2030+
Fixes#7489
* remove lwip-v1.4 specific code
* ditto
* ditto
* fix ip4_addr definition
* CI: change debug builds to use IPv6, remove regular IPv6 builds
* ditto
* split pio CI in four (because they last twice the time of the other builds)
* remove option from pio
* remove lwIP-1.4 from doc
* restore pio CI splitting
* fix CI debug6 script
* ditto
* Deprecate SPIFFS, move examples to LittleFS
SPIFFS has been a great filesystem, but it has significant problems in
many cases (and it's also pretty slow). Development seems to have
slowed/stopped on the upstream version, and we're not able to provide
support or fix the known issues with it as-is.
Deprecate SPIFFS variable.
Update all examples to use LittleFS instead of SPIFFS.
Also, minor cleanup on very old examples which has obsolete delays
waiting for the Serial port to come up, or which were stuck at 9600 baud
because of their ancient AVR heritage.
Fixes#7095
* Remove leftover debug code
* Clean up comments in some examples
* Update documentation on SPIFFS deprecation
* Fix host tests to avoid deprecation warnings
* Fix cut-n-paste error
* Restore SpeedTest.ino, adjust to allow custom FSes
Co-authored-by: Develo <deveyes@gmail.com>
* Make DNS resolution order selectable during runtime or compile time (only in dual stack mode).
* Extend IPv6 example to show usage of new hostByName function with selectable resolving order
* Fix function definition of fqdn_rt in IPv6.ino.
* Fix function call.
* Fix missing bracket...
* Only run if built with dual stack support
* Make DNS resolution order selectable during runtime or compile time (only in dual stack mode).
* Extend IPv6 example to show usage of new hostByName function with selectable resolving order
* Fix function definition of fqdn_rt in IPv6 example.
* Implement enum class for resolve type
* Fix example IPv6.ino
* Fix typedef in ESP8266WiFiGeneric.h function call
* Change enum class definition to not depend on lwip/dns.h
* Move err_t err definition outside switch. Fix typecast.
* Always define DNSResolveType as pio test otherwise failes even if the enum class isn't used.
Co-authored-by: altelch <heiko.krupp@rhrk.uni-kl.de>
Co-authored-by: david gauchard <gauchard@laas.fr>
Fixes#6811 which found an issue where connecting to scripts.google.com
would *occasionally* cause a crash. On inspection, it was found that up
to 5828 bytes of stack were used once in a while, so expand the stack to
5900 bytes to cover this case plus a little extra.
* Move all scripts and documentation to Python3
Python 2 EOL is Jan 1, 2020. Migrate scripts to run under Python 3.
Under Windows, we're already running Python 3.7, by dumb luck. The
oddness is that the Windows standalone executable for Python 3 is called
"python" whereas under UNIX-like OSes it's called "python3" with
"python" always referring to the Python 2 executable. The ZIP needs to
be updated to include a Python3.exe (copy of Python.exe) so that we can
use the same command lines under Linux and Windows, and to preserve my
sanity.
Fixes#6376
* Add new Windows ZIP with python3.exe file
* Sort options in boards.txt generation for repeatability
The order of the board opts dict changes depending on the Python version
and machine, so sort the options before printing them to get a stable
ordering.
* Re-add Python2 compatibility tweaks
Most scripts can run as Python 2 or Python 3 with minimal changes, so
re-add (and fix, as necessary) compatibility tweaks to the scripts.
__This is a breaking change, but the header and example did warn
everyone that this API was in flux due to the incompatible SD and SPIFFS
File implementations.__
BearSSL CertStores now simply need a filesystem and the names of the
data (generated on-chip) and archive (uploaded by user) files on it.
No more need to roll your own virtual CertStoreFile class.
Update the library, examples, and device test.
do interleave informations on addresses within reception pbuf chain:
before: (data-pbuf) -> (data-pbuf) -> (data-pbuf) -> ... in the receiving order
now: (address+port-info-pbuf -> data-pbuf) -> (address_port-info-pbuf -> data-pbuf) -> ...
address/port informations are updated along with data exposed to user
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
* Minor bug fixes in Maximum Fragment Length Negotation example, mainly giving background processes some time in fetch()
* Minor layout changes to pass travis tests
* Use PolledTimeout for timeout
* polledTimeout: add option to use CPU count instead of millis()
* use more "using" alias
* more c++/clear code, using typename (thanks @devyte)
* rename class name to include unit, introduce timeMax() and check it with assert()
* remove useless defines
* improve api readability, add micro-second unit
* update example
* mock: emulate getCycleCount, add/fix polledTimeout CI test
* + nano-seconds, assert -> message, comments, host test
* allow 0 for timeout (enables immediate timeout, fix division by 0)
* typo, set member instead of local variable
* unify error message
* slight change on checkExpired() allows "never expired"
also removed printed message, add YieldAndDelay, simplify calculations
* remove traces of debug.h/cpp in this PR
* include missing <limits> header
* back to original expired test, introduce boolean _neverExpires, fix reset(), getTimeout() is invalid
* fix expiredOneShot with _timeout==0 check
* reenable getTimeout()
* expose checkExpired with unit conversion
* fix timing comments, move critical code to iram
* add member ::neverExpires and use it where relevant
* improve clarity
* remove exposed checkExpired(), adapt LEAmDNS with equivalent
* add API ::resetToNeverExpires(), use it in LEAmDNS
* remove offending constness from ::flagged() LEAmDNS (due do API fix in PolledTimeout)
* simplify "Fast" base classes
* minor variable rename
* Fix examples
* compliance with good c++ manners
* minor changes for consistency
* add missing const
* expired() and bool() moved to iram
* constexpr compensation computing
* add/update comments
* move neverExpires and alwaysExpired
* weak hook early_setup() #2111#2133#2136
* rename to early_init (more "c" vs early_setup which is more "c++arduino")
* example
* improve earlyWiFi example, slightly change AddrList interface, move WiFi sketches into WiFi examples
* fix CI
* fix local CI runner
* fix local CI runner
* rename early_init() to preinit()
* + static ESP8266WiFiClass::preinit_wifi_off()
* update early disable wifi example
* example update
* IPv6 example update
* Update ESP8266WiFiGeneric.h
camelCase for static method name
* Update ESP8266WiFiGeneric.cpp
camelCase for static method name
* Update EarlyDisableWiFi.ino
Expand comment, fix static method name
* Update core_esp8266_main.cpp
Expanded comment.
* Update core_esp8266_main.cpp
Expanded comment
* Update EarlyDisableWiFi.ino
Expanded comment
* WIP compile examples on host with 'make examples'
* WIP bufferize tcp input
* WIP Makefile
* WIP network to rework, tcp/udp to factorize, udp addresses broken
* minor changes to the core
* WIP basic udp working
* WIP mdns
* WIP mcast receiving, not sending
* WIP mdns OK
* beta version
* SSL + doc
* update travis host test command
* licenses
* typo
* doc: arduino builder is not around: declare functions before calling them
* fix with latest SSL PR, compile in 32 bits mode
* fix make clean
* make -m32 optional
* 32bits compiler ability tester
* WIP
* WIP (fix 1 vtable error, still another one to hunt with using spiffs)
* example astyle
* fix os_printf_plus
* load / save mock spiffs
* fix style
* fix using spiffs/mock
* don't mess ram
* update doc
* remove leftover
* optimization -Os except for CI, rename ARCH32 to FORCE32
* revert useless cast (not even compiled)
* remove unused function
* use proper type for pointer arithmetics
* makefile: sketch object and cpp file moved to bin/ directories
easier to clean, and IDE don't like them
* changes for review
* make use of %zd
* less verbose makefile by default (option)
* update readme
BearSSLX509List, BearSSLSession, BearSSLPublicKey, and BearSSLPrivateKey
were all in the global namespace and not in the BearSSL:: one, due to an
oversight when they were originally created. Move them to the proper
namespace with the following mapping:
BearSSLX509List => BearSSL::X509List
BearSSLSession => BearSSL::Session
BearSSLPublicKey => BearSSL::PublicKey
BearSSLPrivateKey => BearSSL::PrivateKey
Allow the unused stack to be reset to the check value at any time in
the application, allowing for delta-stack calculations to be done.
Add ESP.resetFreeContStack() class method for general use.
Add in some dumping in the BearSSL_Validation example to show the
usage for those that care.
Add a method allowing a user to send in a character string for the
fingerprint, like axTLS supported.
Implements part of PR #4833 from @d-a-v with changes requested in
discussion.
SSL Sessions enable most of the SSL handshake to be skipped when both
client and server agree to use them. Add a BearSSLSession class and
an optional setting to the SSL client to enable this.
Note that SSL sessions are unrelated to HTTP sessions. They are
ephemeral and only relate to the SSL parameters, not anything at
the HTTP protocol level.
* Allow cipher specification for BearSSL
BearSSL has many more ciphers than axTLS, but they are more compute intensive
and slower. Add an option to use only the same, limited security, axTLS ciphers
as well as allow users to specify any suite of ciphers they want using standard
BearSSL formats.
Fixes#5110
* Rename methods to avoid axtls references.
* Allow std::vector to set a list of allowed ciphers
For C++ afficionados, allow std::vectors to be passed in to the setCipher()
routine.
The BearSSL object will now keep a copy of any set ciphers and free on object
destruction. These custom lists should normally only be 1-4 entries long, so it
is not expected to be a memory hog having this extra copy.
As part of the "clear connection configuration for reused objects"
patch, a ::stop would reset the self-signed, trust anchors, etc.
WiFiClient, unfortunately, calls ::stop as part of the connection
process, so all of these settings were lost.
Now only clear the connection settings on ::stop if we've already
been connected.
Also update the github public key which changed yet again.
Fixes#5086
According to the GCC man page, __section__ attributes should only be used
for global variables. However, the PROGMEM and ICACHE_RODATA macros use
this variable decorator even for local variables. Most of the time it works,
but when a static or inlined function tries to use a PROGMEM/PSTR/etc.
variable the compiler can throw an error like:
error: XXX causes a section type conflict with YYY
Change the PROGMEM macro to emit a section name that is unique (a combo
of the file, line, and counter variables to ensure uniqueness). The
standard linker script will place them properly in .IROM without
any changes.
Fixes#5036 and others.
Fixes#4882 and updates GitHub certificate fingerprint to the current one
in BearSSL_Validation example.
When setting a authentication mode or stopping, clear all others out in case
the object is being re-used.
Add in a yield during the SSL handshake to allow a graceful timeout and not
a WDT error when the remote server hiccups. Thanks to @Jeroen88 for
finding and testing this.
Several users have been worried that they need to generate the IDX file,
but don't know how. The CertStore code actually writes this file on object
creation, and the user never needs to explicitly generate or upload it.
Add a comment to the example explicitly noting this.
Due to popular demand, remove the hardcoded dependency on SPIFFS
or SD from the CertStore by factoring out the file interface into
a new class (CertStoreFile) that the user will need to implement
as a thin wrapper around either a SPIFFS.file or a SD.file
Combine the downloaded certificates into a UNIX "ar" archive
and parse that on-the-fly to allow easy inspection and creation
of the Cert Store database.
Examples updated with a new certificate downloader that creates
the certs.ar archive and with a single sample that can be built
for either SPIFFS or SD with a #define. Users can copy the
implementation of the CertStoreFile they need to their own code
as it is self-contained.
Also move the CertStore to the BearSSL namespace and remove the
suffix and separate SPIFFS/SD sources.
Remove the "deep+" change from the CI build as well (no special
options needed on any PIO or makefile build).
We'll revisit the filesystem wrapper for 2.5.0, hopefully having a
unified template for both filesystem usage at a global level. For
current users, be aware the interface may change (simplify!) in
release 2.5.0.
Fixes#4740
BearSSL (https://www.bearssl.org) is a TLS(SSL) library written by
Thomas Pornin that is optimized for lower-memory embedded systems
like the ESP8266. It supports a wide variety of modern ciphers and
is unique in that it doesn't perform any memory allocations during
operation (which is the unfortunate bane of the current axTLS).
BearSSL is also absolutely focused on security and by default performs
all its security checks on x.509 certificates during the connection
phase (but if you want to be insecure and dangerous, that's possible
too).
While it does support unidirectional SSL buffers, like axTLS,
as implemented the ESP8266 wrappers only support bidirectional
buffers. These bidirectional buffers avoid deadlocks in protocols
which don't have well separated receive and transmit periods.
This patch adds several classes which allow connecting to TLS servers
using this library in almost the same way as axTLS:
BearSSL::WiFiClientSecure - WiFiClient that supports TLS
BearSSL::WiFiServerSecure - WiFiServer supporting TLS and client certs
It also introduces objects for PEM/DER encoded keys and certificates:
BearSSLX509List - x.509 Certificate (list) for general use
BearSSLPrivateKey - RSA or EC private key
BearSSLPublicKey - RSA or EC public key (i.e. from a public website)
Finally, it adds a Certificate Authority store object which lets
BearSSL access a set of trusted CA certificates on SPIFFS to allow it
to verify the identity of any remote site on the Internet, without
requiring RAM except for the single matching certificate.
CertStoreSPIFFSBearSSL - Certificate store utility
Client certificates are supported for the BearSSL::WiFiClientSecure, and
what's more the BearSSL::WiFiServerSecure can also *require* remote clients
to have a trusted certificate signed by a specific CA (or yourself with
self-signing CAs).
Maximum Fragment Length Negotiation probing and usage are supported, but
be aware that most sites on the Internet don't support it yet. When
available, you can reduce the memory footprint of the SSL client or server
dramatically (i.e. down to 2-8KB vs. the ~22KB required for a full 16K
receive fragment and 512b send fragment). You can also manually set a
smaller fragment size and guarantee at your protocol level all data will
fit within it.
Examples are included to show the usage of these new features.
axTLS has been moved to its own namespace, "axtls". A default "using"
clause allows existing apps to run using axTLS without any changes.
The BearSSL::WiFi{client,server}Secure implements the axTLS
client/server API which lets many end user applications take advantage
of BearSSL with few or no changes.
The BearSSL static library used presently is stored at
https://github.com/earlephilhower/bearssl-esp8266 and can be built
using the standard ESP8266 toolchain.
Use platform.local.txt to add -Werror to GCC for the build of all
code. Any warnings on a submitted patch will cause an error.
Several examples and libraries had warnings/errors (missing returns
on functions, types, etc.). Clean those up with this commit as well.
The certificate fingerprint included with the HTTPSRequest example seems
to be for an expired api.github.com certificate. Replace with the current
one to avoid reporting "certificate mismatch" errors when running.
