* Fix Updater potential overflow, add host tests
Fixes#4674
The Updater class could, when exactly 4K bytes were in the buffer but
not yet written to flash, allow overwriting data written to it beyond
the passed-in size parameter.
Fix per @jason-but's suggestion, and add a host test (plus minor changes
to Updater code to support host testing).
* Add missed mock file
* Remove most testing ifdefs fro updater
Per @mcspr's suggestion, we can pass in fake link symbols allowing
Updater to take the address of `_FS_start`/etc. even when building on
the host for testing.
There is still a single remaining wifi_set_power_mode ifdef'd and a
duplication of the digitalWrite/pinMode for testing vs. host building.
Co-authored-by: Develo <deveyes@gmail.com>
After verifying that they really were spurious, clean up the warnings
that gcc -wextra reports, except for LeaMDNS.
Upgrade GCC to gcc-7 for host builds
* Convert ESP8266WebServer* into templatized model
Supercedes #4912
Refactor the three versions of ESP8266WebServer and *WebServerSecure to a
single templated class. Use "using" to enable old, non-templated names to b
used (so no user changes required to compile or run).
Fixes#4908 and clean up the code base a lot.
Basic tests run (the ones in the example code).
No code changes are required in userland except for setting the SSL
certificates which now use a cleaner "getServer()" accessor and lets the
app use the native BearSSL calls on the WiFiClientSecure object.
@devyte should be proud, it removes virtuals and even has template specialization...
* Fix HTTPUpdate templates and examples
* Fix HTTPUpdateServer library build
Need to remove dot-a linkage since there are no .cpp files in the
directory anymore due to templates.
* Provide backward-compat names for updt template
Allow existing code to use the same well known names for
HTTPUpdateSecure.
* Remove ClientType from all templates, auto-infer
Remove the ClientType template parameter from all objects. Simplifies
the code and makes it more foolproof.
Add a "using" in each server to define the type of connection returned
by all servers, which is then used in the above templates automatically.
* Can safely include FS.h now that SD/SPIFFS unified
* Move the templates/objects to their own namespaces
* Fix merge issues with untemplated methods
* Address review comments
* Fix mock test, remove warnings inside test dir
Make the simple mock test CI job pass and clean up
any spurious warnings in the test directory.
There still are warnings in the libraries and core, but they
should be addressed in a separate PR.
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
W/o this change, file::write('a'); tries to use the template and fails
since char is a basic type.
The reason it is needed is due to pre 2.5.x behavior, File::write(char)
silently was cast to File::write(uint8_t). With the template write,
though, this is not performed.
* Add Print::write tests and add'l overrides
Ensure that print::write does something sane and doesn't cause a compile
time error about templates when used for Files.
Test using SPIFFS file since Print is an abstract type.
Force GCC to run with -funsigned-char during host tests to make
the PC match the default behaviour used by the xtensa GCC port.
As noted in #6010. Thanks @MichaelBrunn3r
* Add a FAT filesystem for SD cards to Arduino FS
Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.
The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.
This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.
This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.
By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.
Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.
Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).
Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other. Add a simple
bool FS::isDirectory() and bool FS::isFile() method. SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.
Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories. SPIFFS directory operations will, of course, fail
and return false.
Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.
Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls. This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.
The only major visible change for users is that long filenames now are
fully supported and work without any code changes. If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.
While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.
Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.
The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.
* Add a FSConfig and SDFSConfig param to FS.begin()
Allows for configuration values to be passed into a filesystem via the
begin method. By default, a FS will receive a nullptr and should so
whatever is appropriate.
The base FSConfig class has one parameter, _autoFormat, set by the
default constructor to true.
For SPIFFS, you can now disable auto formatting on mount failure by
passing in a FSConfig(false) object.
For SDFS a SDFSConfig parameter can be passed into config specifying the
chip select and SPI configuration. If nothing is passed in, the begin
will fail since there are no safe default values here.
* Add FS::setConfig to set FS-specific options
Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which
takes a FS-specific configuration object and copies any special settings
on a per-FS basis. The call is only valid on unmounted filesystems, and
checks the type of object passed in matches the FS being configured.
Updates the docs and tests to utilize this new configuration method.
* Add ::truncate to File interface
Fixes#3846
* Use polledTimeout for formatting yields, cleanup
Use the new polledTimeout class to ensure a yield every 5ms while
formatting.
Add in default case handling and some debug messages when invalid inputs
specified.
* Make setConfig take const& ref, cleaner code
setConfig now can take a parameter defined directly in the call by using
a const &ref to it, leading to one less line of code to write and
cleaner reading of the code.
Also clean up SDFS implementation pointer definition.
* emulation on host: Add full UART driver emulation.
This PR replaces the high level Serial mock with a more complete
UART driver. This way the HardwareSerial works without any
modifications. Additionally the driver supports UART0 and UART1
at the same time (UART0 is directed to stdout and UART1 writes
to stderr). RX is implemented by switching the terminal into raw
non-blocking mode and injecting each key-press directly into the
FIFO of UART0. A new command line switch -c was added to ignore
CTRL-C and send it via serial as well. The decumentation was
updated accordingly.
Reading and setting of GPIOs does only write to stderr, when compiled
with D=1. But this is subject to be replaced with a proper GPIO emu
anyway. Reading from GPIO0 now returns 1 instead of 0 because this is
most likely a low active input.
* Fixed unused variable.
* Remove unused functions, as long as there are no debug macros using them.
* Move user_interface.cc from MOCK_CPP_FILES_EMU to MOCK_CPP_FILES.
* Move optimistic_yield() from user_interface.cpp to Arduino.cpp
* Remove atexit() weak declaration (fixes segfault when calling atexit).
* Improve resetting of terminal after program exit, convert \r to \r\n.
* Show error, when STDOUT is not a TTY, minor code cleanung.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
This patch splits ARDUINO_LIBS into two variables. The second variable is called OPT_ARDUINO_LIBS and can be overridden by the caller as it uses the ?= assignment operator. Additionally it unifies and simplifies collecting the files in common/ by using the addprefix macro. All changes should be 100% backwards compatible.
* Move to PROGMEM aware libc, allow PSTR in printf()
A Newlib (libc) patch is in progress to move the _P functions from inside
Arduino into first-class citizens in libc. This Arduino patch cleans up
code that's been migrated there. Binaries for the new libs are included
because it seems they're part of the Arduino git tree, and should be
replaced with @igrr built ones when/if the Newlib changes are accepted.
Notable changes/additions for Arduino:
Allow for use of PROGMEM based format and parameter strings in all
*printf functions. No need for copying PSTR()s into RAM before printing
them out (transparently saves heap space when using _P functions) and
makes it easier to print out constant strings for applications.
Add "%S" (capital-S) format that I've been told, but cannot verify,
is used in Arduino to specify a PROGMEM string parameter in printfs,
as an alias for "%s" since plain "%s" can now handle PROGMEM.
Optimized the memcpy_P, strnlen_P, and strncpy_P functions to use 32-bit
direct reads whenver possible (source and dest alignment mediated), but
there is still room for improvement in others.
Finally, move several constant arrays from RODATA into PROGMEM and
update their accessors. Among these are the ctype array, ~260 bytes,
mprec* arrays, ~300 bytes, and strings/daycounts in the time
formatting functions, ~200 bytes. All told, sketches will see from
300 to 800 additional RAM heap free on startup (depending on their
use of these routines).
* Fix merge error in #ifdef/#endif
* Fix host test using the newlib generic pgmspace.h
Host tests now use the sys/pgmspace.h for compiles instead of the
ESP8266-specific version.
* Update with rebuilt libraries using latest newlib
* Include binaries built directly from @igrr repo
Rebuild the binaries using a git clone of
https://github.com/igrr/newlib-xtensa
Build commands for posterity:
````
rm -rf ./xtensa-lx106-elf/
./configure --prefix=<DIR>/esp8266/tools/sdk/libc --with-newlib \
--enable-multilib --disable-newlib-io-c99-formats \
--disable-newlib-supplied-syscalls \
--enable-newlib-nano-formatted-io --enable-newlib-reent-small \
--enable-target-optspace \
--program-transform-name="s&^&xtensa-lx106-elf-&" \
--disable-option-checking --with-target-subdir=xtensa-lx106-elf \
--target=xtensa-lx106-elf
rm -f etc/config.cache
CROSS_CFLAGS="-fno-omit-frame-pointer -DSIGNAL_PROVIDED -DABORT_PROVIDED"\
" -DMALLOC_PROVIDED" \
PATH=<DIR>/esp8266/tools/xtensa-lx106-elf/bin/:$PATH \
make all install
````
* Fix merge define conflict in c_types.h
* Fix strlen_P misaligned source error
Include fix from newlib-xtensa/fix-strlen branch cleaning up misaligned
access on a non-aligned source string.
* Fix strlen_P and strcpy_P edge cases
Ran the included test suite on ESP8266 tstring.c with the following defines:
#define MAX_1 50
#define memcmp memcmp_P
#define memcpy memcpy_P
#define memmem memmem_P
#define memchr memchr_P
#define strcat strcat_P
#define strncat strncat_P
#define strcpy strcpy_P
#define strlen strlen_P
#define strnlen strnlen_P
#define strcmp strcmp_P
#define strncmp strncmp_P
Uncovered edge case and return value problems in the optimized versions of
the strnlen_P and strncpy_P functions. Corrected.
* Fix memcpy_P return value
memcpy-1.c test suite showed error in return value of memcpy_P. Correct it.
* Fix strnlen_P/strlen_P off-by-4 error
Random crashes, often on String constructors using a PSTR, would occur due
to the accelerated strnlen_P going past the end of the string. Would make
debug builds fail, too (ESP.getVersionString() failure).
Fix to fall through to normal copy on a word that's got a 0 byte anywhere
in it.
* Add device tests for libc functional verification
Add test suite used to debug libc optimized _P functions to the device
tests.
* Rebuild from igrr's repo (same source as prior)
Rebuild .a from igrr's repo at 347260af117b4177389e69fd4d04169b11d87a97
* WIP - add exceptions
* Fix exception to have 0-terminator
* Move some exception constants to TEXT from RODATA
* Remove throw stubs
* Move more exception stuff to ROM
* Enable exceptions in platform.io
* Remove atexit, is duplicated in rebuilt lib
Need to look at the quick-toolchain options, there seems to be a definition
for atexit defined there (libgcc?) that needs to be excised. For now,
remove our local do-nothing copy.
* Update libgcc to remove soft-fp functions
The esp-quick-toolchain generated libgcc.a needed to have the soft-FP routines
that are in ROM removed from it. Remove them in the new esp-quick-toolchain
and update.
* Fix merge typos in Makefile
* Add unhandled exception handler to postmortem
* Return our atexit() handler
* Latest stdc++, minimize exception emercengy area
* Remove atexit from newlib
atexit was defined in newlib strongly, but we also define a noop atexit in core.
Since we never exit, use the core's noop and delete the atexit from libc.a
Updated in esp-quick-toolchain as well.
* Move __FUNCTION__ static strings to PROGMEM
__FUNCTION__ is unlikely to be a timing sensitive variable, so move it to
PROGMEM and not RODATA (RAM) using linker magic.
asserts() now should take no RAM for any strings.
* Clean up linker file, update to latest stdc++
* Update to latest stdc++ which doesn't call strerror
* Update to GCC5.1 exception emergency allocator
Using GCC 5.1's emergency memory allocator for exceptions, much less
space is required in programs which do not use exceptions and when
space is allocated it is managed more efficiently.
* Initial try with new compiler toolchain
* Include newlib built from esp-quick-toolchain
* Update JSON with all new esp-quick-toolchain builds
* Use 64bit Windows compiler on 64bit Windows
* Dump std::exception.what() when possible
When doing the panic on unhandled exceptions, try and grab the
.what() pointer and dump it as part of the termination info.
Makes it easy to see mem errors (std::bad_alloc) or std::runtime_error
strings.
* Use scripted install from esp-quick-toolchain
Makes sure proper libraries and includes are present by using a
scripted installation from esp-quick-install instead of a manual
one.
* Update eqk to remove atexit, fix packaging diff
* PolledTimeout Class for wrapping millis() loops
* Add yield policies, improve reset, add host tests
* Fix copyright, comments
* adjust host tests for better time precision
* add fuzzyness to timing tests for CI jitter
* add blink example with polledTimeout
* improve namespace and type naming, add copyright, comments
* fix astyle
* WIP compile examples on host with 'make examples'
* WIP bufferize tcp input
* WIP Makefile
* WIP network to rework, tcp/udp to factorize, udp addresses broken
* minor changes to the core
* WIP basic udp working
* WIP mdns
* WIP mcast receiving, not sending
* WIP mdns OK
* beta version
* SSL + doc
* update travis host test command
* licenses
* typo
* doc: arduino builder is not around: declare functions before calling them
* fix with latest SSL PR, compile in 32 bits mode
* fix make clean
* make -m32 optional
* 32bits compiler ability tester
* WIP
* WIP (fix 1 vtable error, still another one to hunt with using spiffs)
* example astyle
* fix os_printf_plus
* load / save mock spiffs
* fix style
* fix using spiffs/mock
* don't mess ram
* update doc
* remove leftover
* optimization -Os except for CI, rename ARCH32 to FORCE32
* revert useless cast (not even compiled)
* remove unused function
* use proper type for pointer arithmetics
* makefile: sketch object and cpp file moved to bin/ directories
easier to clean, and IDE don't like them
* changes for review
* make use of %zd
* less verbose makefile by default (option)
* update readme
* Add valgrind and string tests to host_tests
Valgrind identified an error that was causing intermittent failures in
the host tests, and can check for subtle memory access and allocation bugs.
Add it to the standard host_test suite, and any errors will cause test
failure.
Also start adding string tests, since two undefined behaviors have been
found so far just by inspection.
* Add additional String tests
Looks like a possible bug in the concatenation operator, need to verify
expected behavior.
* Remove verbose from valgrind run
No need to be so chatty on the test. Errors were a little hard to spot.
Go to normal verbosity.
* Add lcov and more string tests
LCOV and genhtml can generate nice HTML coverage charts hilighting test
coverage issues in a much simpler way than gcov text format. Generate these
automatically from gcov output.
Add additional string creation and comparison tests.
* Move String coverage to >50%
Additional string tests and checks
* 66% test coverage in String
* Add allocation-unit-sized strings test
Ensure that strings that are right on the edge of the allocation
size are handled properly and account for trailing 0.
