* help in makefile
* fix some device tests, http_server is pending
* fix webserver test, one test is disabled due to general python2->3 failure
* remove debug strings
* minimize diff
* set reset method back to the default one on generic board
* fix vcc range check from datasheet
vcc is read as 2.9V here, datasheet says 2.5-3.6, old low limit was 3v
* tell python to decode string
* Fix Python3 errors for device tests
The Python3 migration didn't include fixes for local scripts in the
device test tree.
Fatal build and run Python errors fixed.
The last update to xunitmerge is ~5 years ago, so it looks to be
unsupported now.
Use a local copy of the two components to allow patching to work with
Python3.
The serial test seems to send garbage chars (non-ASCII/etc.), so use a
codepage 437 which supports all 255 chars.
Fixes: #6660
* Run tests at 160MHz (req'd for some SSL connections)
* Fix debuglevel options for builder
* Fix Python3 interpreter path in xunitmerge
* Remove virtualenv on "make clean"
* Add appropriate attribution, license to xunitmerge
Add like to the original sources with the author's license to the
copied/fixed xunitmerge files.
* Move all scripts and documentation to Python3
Python 2 EOL is Jan 1, 2020. Migrate scripts to run under Python 3.
Under Windows, we're already running Python 3.7, by dumb luck. The
oddness is that the Windows standalone executable for Python 3 is called
"python" whereas under UNIX-like OSes it's called "python3" with
"python" always referring to the Python 2 executable. The ZIP needs to
be updated to include a Python3.exe (copy of Python.exe) so that we can
use the same command lines under Linux and Windows, and to preserve my
sanity.
Fixes#6376
* Add new Windows ZIP with python3.exe file
* Sort options in boards.txt generation for repeatability
The order of the board opts dict changes depending on the Python version
and machine, so sort the options before printing them to get a stable
ordering.
* Re-add Python2 compatibility tweaks
Most scripts can run as Python 2 or Python 3 with minimal changes, so
re-add (and fix, as necessary) compatibility tweaks to the scripts.
* Fix device test environment variables
Device tests were not connecting properly to WiFi because the
environment variables were not set when WiFi.connect was called.
This would result in tests sometimes working *if* the prior sketch run
on the ESP saved WiFi connection information and auto-connect was
enabled. But, in most cases, the tests would simply never connect to
any WiFi and fail.
getenv() works only after BS_RUN is called (because BS_RUN handles the
actual parsing of environment variables sent from the host).
Add a "pretest" function to all tests which is called by the host test
controller only after all environment variables are set. Move all
WiFi/etc. operations that were in each separate test's setup() into it.
So the order of operations for tests now is:
ESP: setup()
-> Set serial baud
-> Call BS_RUN()
HOST: Send environment
Send "do pretest"
ESP: pretest()
-> Set Wifi using env. ariables, etc. return "true" on success
HOST: Send "run test 1"
ESP: Run 1st test, return result
HOST: Send "run test 2"
ESP: Run 2nd test, return result
<and so forth>
If nothing is needed to be set up, just return true from the pretest
function.
All tests now run and at least connect to WiFi. There still seem to be
some actual test errors, but not because of the WiFi/environment
variables anymore.
* Remove unneeded debug prints
* Silence esptool.py output when not in V=1 mode
Esptool-ck.exe had an option to be silent, but esptool.py doesn't so the
output is very chatty and makes looking a the run logs hard (60 lines
of esptool.py output, 3 lines of actual test reports).
Redirect esptool.py STDOUT to /dev/null unless V=1 to clear this up.
* Speed up builds massively by removing old JSON
arduino-builder checks the build.options.json file and then goes off and
pegs my CPU at 100% for over a minute on each test compile checking if
files have been modified.
Simply deleting any pre-existing options.json file causes this step to
be skipped and a quick, clean recompile is done in siginificantly less
time.
* Enable compile warnings, fix any that show up
Enable all GCC warnings when building the tests and fix any that came up
(mostly signed/unsigned, unused, and deprecated ones).
* Fix UMM_MALLOC printf crash, umm_test
Printf can now handle PROGMEM addresses, so simplify and correct the
debug printouts in umm_info and elsewhere.
__This is a breaking change, but the header and example did warn
everyone that this API was in flux due to the incompatible SD and SPIFFS
File implementations.__
BearSSL CertStores now simply need a filesystem and the names of the
data (generated on-chip) and archive (uploaded by user) files on it.
No more need to roll your own virtual CertStoreFile class.
Update the library, examples, and device test.
* optionally allow redirects on http OTA updates
* Refactored HTTPClient::begin(url...) & setURL functions, now only beginInternal parses URL, sets ports
Added HTTPRedirect example.
* fix indentation for style check
* add space after while for style check
* don't use deprecated begin method in redirect example
* moved redirect handling code to HTTPClient.
only GET and HEAD requests are currently handled automatically
Redirects that fail to be automatically handled return the redirect code as before
* added support for POST/303 redirect
added device redirect tests
* add missing getLocation() implementation
* if the new location is only a path then only update the URI
* uart fixes and BW improvements
* uart: read_char straightly use hw buffer
* +attributes for functions called by ISR
* uart: BW improvements
read_char straightly use hw buffer (+ ~10%bw)
read by block (+ ~190%bw) (instead of generic Stream::readBytes)
attributes for functions called by ISR
remove overrun message
remove some ISR flags which were not honoured
* fix merge
* fix buffer overflow
* serial stress test sketch
* astyle
* serial stress example: interactive keyboard, stop reading, overrun
* serial device test: bandwidth & overrun
* update + HardwareSerial::hasError()
* interactive overrun in example
* astyle
* Test using @plerup's SoftwareSerial as submodule (tag 3.4.1)
* update upstream ref (fix warning)
* host mock uart/read(buf,size)
* reset style changes in submodules before style diff
* update build_boards_manager_package.sh for submodules
* trigger CI (removing space)
* cannot reproduce locally the CI issue, setting bash -x option to get live trace
* remove previously added (in this PR) 'set -e' in package builder (passes local tests, not real CI)
script-comment new recipe.hooks.core.prebuild.3 (along with already commented .1 and .2)
moved CI package test to be first on the test list
remove 'set -x', wish me luck
* Move to PROGMEM aware libc, allow PSTR in printf()
A Newlib (libc) patch is in progress to move the _P functions from inside
Arduino into first-class citizens in libc. This Arduino patch cleans up
code that's been migrated there. Binaries for the new libs are included
because it seems they're part of the Arduino git tree, and should be
replaced with @igrr built ones when/if the Newlib changes are accepted.
Notable changes/additions for Arduino:
Allow for use of PROGMEM based format and parameter strings in all
*printf functions. No need for copying PSTR()s into RAM before printing
them out (transparently saves heap space when using _P functions) and
makes it easier to print out constant strings for applications.
Add "%S" (capital-S) format that I've been told, but cannot verify,
is used in Arduino to specify a PROGMEM string parameter in printfs,
as an alias for "%s" since plain "%s" can now handle PROGMEM.
Optimized the memcpy_P, strnlen_P, and strncpy_P functions to use 32-bit
direct reads whenver possible (source and dest alignment mediated), but
there is still room for improvement in others.
Finally, move several constant arrays from RODATA into PROGMEM and
update their accessors. Among these are the ctype array, ~260 bytes,
mprec* arrays, ~300 bytes, and strings/daycounts in the time
formatting functions, ~200 bytes. All told, sketches will see from
300 to 800 additional RAM heap free on startup (depending on their
use of these routines).
* Fix merge error in #ifdef/#endif
* Fix host test using the newlib generic pgmspace.h
Host tests now use the sys/pgmspace.h for compiles instead of the
ESP8266-specific version.
* Update with rebuilt libraries using latest newlib
* Include binaries built directly from @igrr repo
Rebuild the binaries using a git clone of
https://github.com/igrr/newlib-xtensa
Build commands for posterity:
````
rm -rf ./xtensa-lx106-elf/
./configure --prefix=<DIR>/esp8266/tools/sdk/libc --with-newlib \
--enable-multilib --disable-newlib-io-c99-formats \
--disable-newlib-supplied-syscalls \
--enable-newlib-nano-formatted-io --enable-newlib-reent-small \
--enable-target-optspace \
--program-transform-name="s&^&xtensa-lx106-elf-&" \
--disable-option-checking --with-target-subdir=xtensa-lx106-elf \
--target=xtensa-lx106-elf
rm -f etc/config.cache
CROSS_CFLAGS="-fno-omit-frame-pointer -DSIGNAL_PROVIDED -DABORT_PROVIDED"\
" -DMALLOC_PROVIDED" \
PATH=<DIR>/esp8266/tools/xtensa-lx106-elf/bin/:$PATH \
make all install
````
* Fix merge define conflict in c_types.h
* Fix strlen_P misaligned source error
Include fix from newlib-xtensa/fix-strlen branch cleaning up misaligned
access on a non-aligned source string.
* Fix strlen_P and strcpy_P edge cases
Ran the included test suite on ESP8266 tstring.c with the following defines:
#define MAX_1 50
#define memcmp memcmp_P
#define memcpy memcpy_P
#define memmem memmem_P
#define memchr memchr_P
#define strcat strcat_P
#define strncat strncat_P
#define strcpy strcpy_P
#define strlen strlen_P
#define strnlen strnlen_P
#define strcmp strcmp_P
#define strncmp strncmp_P
Uncovered edge case and return value problems in the optimized versions of
the strnlen_P and strncpy_P functions. Corrected.
* Fix memcpy_P return value
memcpy-1.c test suite showed error in return value of memcpy_P. Correct it.
* Fix strnlen_P/strlen_P off-by-4 error
Random crashes, often on String constructors using a PSTR, would occur due
to the accelerated strnlen_P going past the end of the string. Would make
debug builds fail, too (ESP.getVersionString() failure).
Fix to fall through to normal copy on a word that's got a 0 byte anywhere
in it.
* Add device tests for libc functional verification
Add test suite used to debug libc optimized _P functions to the device
tests.
* Rebuild from igrr's repo (same source as prior)
Rebuild .a from igrr's repo at 347260af117b4177389e69fd4d04169b11d87a97
* WIP - add exceptions
* Fix exception to have 0-terminator
* Move some exception constants to TEXT from RODATA
* Remove throw stubs
* Move more exception stuff to ROM
* Enable exceptions in platform.io
* Remove atexit, is duplicated in rebuilt lib
Need to look at the quick-toolchain options, there seems to be a definition
for atexit defined there (libgcc?) that needs to be excised. For now,
remove our local do-nothing copy.
* Update libgcc to remove soft-fp functions
The esp-quick-toolchain generated libgcc.a needed to have the soft-FP routines
that are in ROM removed from it. Remove them in the new esp-quick-toolchain
and update.
* Fix merge typos in Makefile
* Add unhandled exception handler to postmortem
* Return our atexit() handler
* Latest stdc++, minimize exception emercengy area
* Remove atexit from newlib
atexit was defined in newlib strongly, but we also define a noop atexit in core.
Since we never exit, use the core's noop and delete the atexit from libc.a
Updated in esp-quick-toolchain as well.
* Move __FUNCTION__ static strings to PROGMEM
__FUNCTION__ is unlikely to be a timing sensitive variable, so move it to
PROGMEM and not RODATA (RAM) using linker magic.
asserts() now should take no RAM for any strings.
* Clean up linker file, update to latest stdc++
* Update to latest stdc++ which doesn't call strerror
* Update to GCC5.1 exception emergency allocator
Using GCC 5.1's emergency memory allocator for exceptions, much less
space is required in programs which do not use exceptions and when
space is allocated it is managed more efficiently.
* Initial try with new compiler toolchain
* Include newlib built from esp-quick-toolchain
* Update JSON with all new esp-quick-toolchain builds
* Use 64bit Windows compiler on 64bit Windows
* Dump std::exception.what() when possible
When doing the panic on unhandled exceptions, try and grab the
.what() pointer and dump it as part of the termination info.
Makes it easy to see mem errors (std::bad_alloc) or std::runtime_error
strings.
* Use scripted install from esp-quick-toolchain
Makes sure proper libraries and includes are present by using a
scripted installation from esp-quick-install instead of a manual
one.
* Update eqk to remove atexit, fix packaging diff
* Update to BearSSL 0.6+ release, add AES_CCM modes
Pull in latest BearSSL head (0.6 + minor additions) release and add AES_CCM
modes to the encryption options.
* Enable the aes_ccm initialization in client/server
* Initial attempt
* Working code with second stack thunking
* Remove #ifdefs in .S file, not needed.
* Clean up thunks and remove separate stack flag
* Fix PIO assembler errors
* Remove #ifdef code changes, ensure same code as PC
Remove "#ifdef ESP8266;...;#else;...;#endif" brackets in BearSSL to
ensure the host-tested code is the same as the ESP8266-run code.
* Move to latest BearSSL w/EC progmem savings
* Merge with master
* Add br_thunk_* calls to do ref counting, painting
Add reference counting br_thunk_add/del_ref() to replace stack handling code
in the class.
Add in stack painting and max usage calculation.
* Add in postmortem stack dump hooks
When a crash occurs while in the second stack, dump the BSSL stack and
then also the stack that it was called from (either cont or sys).
* Update stack dump to match decoder expectations
* Move thunk to code core for linkiage
The thunk code needs to be visible to the core routines, so move it to the
cores/esp8266 directory. Probably need to refactor the stack setup and the
bearssl portion to avoid dependency on bearssl libs in cores/esp8266
* Add 2nd stack dump utility routine
* Refactor once more, update stack size, add stress
Make stack_thunks generic, remove bearssl include inside of cores/esp8266.
Allocate the stack on a WiFiServerSecure object creation to avoid
fragmentation since we will need to allocate the stack to do any
connected work, anyway.
A stress test is now included which checks the total BearSSL second
stack usage for a variety of TLS handshake and certificate options
from badssl.org.
* Update to latest to-thunks branch
* Add BearSSL device test using stack stress
Run a series of SSL connection and transmission tests that stress
BearSSL and its stack usage to the device tests.
Modify device tests to include a possible SPIFFS generation and
upload when a make_spiffs.py file is present in a test directory.
* Use bearssl/master branch, not /to-thunks branch
Update to use the merged master branch of bearssl. Should have no code
changes.
* update HTTPClient API usage
skip the second POST as end() has different semantics and nulls the client pointer
use bearssl in ssl tests
add delay in python side when shutting down http web server so MacOS does not complain about address already in use
* fix crash if GET/POST was called after end() without a new begin()
update double POST test to insure no crash if POST called after end()
test now are for both AxTLS and BearSSL
* fix small comment typo
boards generator updates:
* simplified mapping description: only flash and spiffs sizes are needed
* some ldscripts are renamed due to autogenerated names
* +2M/0, +2M/512K spiffs, +4M0
* reduce length of hidden strings in boards.txt (#5100, https://github.com/arduino/arduino-builder/issues/284)
* give more details in ldscripts (address, size, +rf-cal, +sdk-wifi-settings)
* add stubs for more exception throw calls
Fixes https://github.com/esp8266/Arduino/issues/3358
* libc: make putc_r implementation weak
newlib provides its own implementation of _putc_r, which will call
_write_r (possibly after buffering). Make our implementation weak to
allow using the one from newlib.
Fixes https://github.com/esp8266/Arduino/issues/4630
* libc: fix incorrect return value of _write_r call
Should return number of bytes written, actually returned zero. This
resulted in std::cout going into failed state after the first write.
* tests: add test for output to std::cout
This allows requesting environment variable values set in the C++ test
code, from python mock code. Use case is cross-checking test
results against values expected by the host side part of the test.
Previously device tests included information such as access point SSID/password at compile time. This made it difficult to compile test binaries once and then send them to multiple test runners for execution.
This change adds a command to the test library to set environment variable on the target device: “setenv key value”. C library setenv/getenv facility is used to store variables.
Test runner, tests, and makefile are updated to use this functionality.
* device test for #4516 (WiFiClient leaking)
* simply count the number of connections that do not change heap
* remove unused heapLost
* + python server (to not depend on gateway's opened tcp port)
* remove old&bad define
* fix another old address
* tests/device/Makefile: use bourne shell compatible "." instead ot "source" (ubuntu's dash does not understand it)
* tests/device: make BEGINTIMEOUT a new error return value instead of a test-process-breaking exception
Reported in https://github.com/esp8266/Arduino/issues/4078.
WiFiClient::stopAll, called from a WiFi disconnected event handler,
could be called while WiFiClient::connect was in progress. This issue
was initially fixed in #4194, by testing `this` pointer for being
non-null in ClientContext::connect.
This change delegates deletion of ClientContext to WiFiClient
destructor. WiFiClient::stop only calls ClientContext::stop, which
closes/aborts the connection.
* Make HTTP server test data easier to examine
* Add HTTP server parameter tests containing & and =
* Fix URL parameter decoding in web server
The parameters string needs to be first split on & and =, and URL
decoding on parts done after that. Otherwise URL encoded & and = within
parameter names and values cause incorrect splitting.
* Use newlib libc library
This change adds libcmin.a, which is created from newlib libc by selectively removing some of the object files (mostly related to heap management).
The list of files is available in tools/sdk/lib/make_libcmin.sh. Files which are not needed are commented out.
This change adds support for various functions which were missing, like sscanf, strftime, etc.
* Fix some of the time functions
* Redirect stdout to serial
* Implement __putc_r
* Switch to custom newlib build
Built from https://github.com/igrr/newlib-xtensa using:
./configure --with-newlib --enable-multilib --disable-newlib-io-c99-formats --enable-newlib-supplied-syscalls --enable-target-optspace --program-transform-name="s&^&xtensa-lx106-elf-&" --disable-option-checking --with-target-subdir=xtensa-lx106-elf --target=xtensa-lx106-elf --enable-newlib-nano-formatted-io --enable-newlib-reent-small --prefix=path-to-arduino-core/tools/sdk/libc
CROSS_CFLAGS="-DMALLOC_PROVIDED -DSIGNAL_PROVIDED -DABORT_PROVIDED" make
make install
* Update tests
