Run valgrind on host mock example runs to catch more bugs in CI. These
tests would have caught the problem in #7464 before users did.
Add a list of some randomly picked examples to run, and add an option to
run the loop exactly once in the host mock routine, so the test will
actually exit under valgrind.
* Deprecate SPIFFS, move examples to LittleFS
SPIFFS has been a great filesystem, but it has significant problems in
many cases (and it's also pretty slow). Development seems to have
slowed/stopped on the upstream version, and we're not able to provide
support or fix the known issues with it as-is.
Deprecate SPIFFS variable.
Update all examples to use LittleFS instead of SPIFFS.
Also, minor cleanup on very old examples which has obsolete delays
waiting for the Serial port to come up, or which were stuck at 9600 baud
because of their ancient AVR heritage.
Fixes#7095
* Remove leftover debug code
* Clean up comments in some examples
* Update documentation on SPIFFS deprecation
* Fix host tests to avoid deprecation warnings
* Fix cut-n-paste error
* Restore SpeedTest.ino, adjust to allow custom FSes
Co-authored-by: Develo <deveyes@gmail.com>
* At least the F_CPU define in host mock.h is needed by host Arduino.h - need to include Arduino.h further down in mock.h for this to work.
* Geting the include order right
* Prepare for runtime CPU clock rate selection
* Fix compile for not defined F_CPU
If defined F_CPU, make getCpuFreqMHz() a constexpr
* Use defines for register CPU2X instead of hex value
* Fix build for host - getCpuFreqMHz there was also in conflict with getCycleCount, using F_CPU:
tests/host/common/mock.h:#define F_CPU 80000000 (!)
* Asymmetrical includes and defines on host
* Support restart switch from 160MHz to 80MHz, e.g for OTA. Fixes#579
* wifi: ARP gratuitous API for wifi station mode
* fix with lwip1
* update comment
* update API to allow changing interval on the fly
* update API
* remove debug lines
* mock lwIP's etharp_request()
* unsigned interval
* use scheduled ticker
* ticker: +attach_ms_scheduled_accurate
ticker: +comment
gratuitous: use attach_ms_scheduled_accurate
* move to experimental namespace
* fix for lwIP-v1.4
* attempt to make pio happy
* use directly ETSTimer instead of Ticker
* upstream lwIP is now downloaded by a makefile, not subsubmoduled
* lwip2: upstream lwIP not sub-sub-modules anymore
lwip2: Allow IPv4 and IPv6 DNS and SNTP server configured via DHCP to co-exist (patch against upstream)
* lwip2: enable tcp-listen-with-backlog feature
* lwip2 submodule update:
- enable more efficient chksum algorithm thanks to Richard Allen
- enable tcp listener with backlog
* more comments, fix backlog management, fix API
* move default value definition in .cpp
because one must not believe it can be redefined before including WiFiServer.h
* improved backlog handling, it is no more a breaking change
* optimistic_yield is a yield, not a delay, and avoids yielding if last yield has occurred recently enough.
* Suppress an unused argument warning.
Co-authored-by: Earle F. Philhower, III <earlephilhower@yahoo.com>
* Fix Updater potential overflow, add host tests
Fixes#4674
The Updater class could, when exactly 4K bytes were in the buffer but
not yet written to flash, allow overwriting data written to it beyond
the passed-in size parameter.
Fix per @jason-but's suggestion, and add a host test (plus minor changes
to Updater code to support host testing).
* Add missed mock file
* Remove most testing ifdefs fro updater
Per @mcspr's suggestion, we can pass in fake link symbols allowing
Updater to take the address of `_FS_start`/etc. even when building on
the host for testing.
There is still a single remaining wifi_set_power_mode ifdef'd and a
duplication of the digitalWrite/pinMode for testing vs. host building.
Co-authored-by: Develo <deveyes@gmail.com>
* At least the F_CPU define in host mock.h is needed by host Arduino.h - need to include Arduino.h further down in mock.h for this to work.
* Geting the include order right
* Simple i/f to turn on inverted logic on UART0.
* Refactor invert from HardwareSerial to uart
* Final refactoring of invert bits into config bitmap.
* Overload instead of default arg for subclassing.
* Prevent unwanted effects if setting invert on other than UART0 - only that has these flags defined and documented.
* mDNS debug option + AP address is used by default when STA is also present
* mDNS: store network interface, checking it is up
* igmp: force on selected interface (avoid crash *sometimes*)
* fix for all lwip2 ipv4 ipv6 & lwip1
* mdns: IPAddress is not needed to reference associated interface
* mdns: debug: fix print warnings
* emulation: add ets_strncpy
* emulation: truly emulate AddrList (remove fake one)
After verifying that they really were spurious, clean up the warnings
that gcc -wextra reports, except for LeaMDNS.
Upgrade GCC to gcc-7 for host builds
* Convert ESP8266WebServer* into templatized model
Supercedes #4912
Refactor the three versions of ESP8266WebServer and *WebServerSecure to a
single templated class. Use "using" to enable old, non-templated names to b
used (so no user changes required to compile or run).
Fixes#4908 and clean up the code base a lot.
Basic tests run (the ones in the example code).
No code changes are required in userland except for setting the SSL
certificates which now use a cleaner "getServer()" accessor and lets the
app use the native BearSSL calls on the WiFiClientSecure object.
@devyte should be proud, it removes virtuals and even has template specialization...
* Fix HTTPUpdate templates and examples
* Fix HTTPUpdateServer library build
Need to remove dot-a linkage since there are no .cpp files in the
directory anymore due to templates.
* Provide backward-compat names for updt template
Allow existing code to use the same well known names for
HTTPUpdateSecure.
* Remove ClientType from all templates, auto-infer
Remove the ClientType template parameter from all objects. Simplifies
the code and makes it more foolproof.
Add a "using" in each server to define the type of connection returned
by all servers, which is then used in the above templates automatically.
* Can safely include FS.h now that SD/SPIFFS unified
* Move the templates/objects to their own namespaces
* Fix merge issues with untemplated methods
* Address review comments
* Fix mock test, remove warnings inside test dir
Make the simple mock test CI job pass and clean up
any spurious warnings in the test directory.
There still are warnings in the libraries and core, but they
should be addressed in a separate PR.
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
* polledTimeout: add option to use CPU count instead of millis()
* use more "using" alias
* more c++/clear code, using typename (thanks @devyte)
* rename class name to include unit, introduce timeMax() and check it with assert()
* remove useless defines
* improve api readability, add micro-second unit
* update example
* mock: emulate getCycleCount, add/fix polledTimeout CI test
* + nano-seconds, assert -> message, comments, host test
* allow 0 for timeout (enables immediate timeout, fix division by 0)
* typo, set member instead of local variable
* unify error message
* slight change on checkExpired() allows "never expired"
also removed printed message, add YieldAndDelay, simplify calculations
* remove traces of debug.h/cpp in this PR
* include missing <limits> header
* back to original expired test, introduce boolean _neverExpires, fix reset(), getTimeout() is invalid
* fix expiredOneShot with _timeout==0 check
* reenable getTimeout()
* expose checkExpired with unit conversion
* fix timing comments, move critical code to iram
* add member ::neverExpires and use it where relevant
* improve clarity
* remove exposed checkExpired(), adapt LEAmDNS with equivalent
* add API ::resetToNeverExpires(), use it in LEAmDNS
* remove offending constness from ::flagged() LEAmDNS (due do API fix in PolledTimeout)
* simplify "Fast" base classes
* minor variable rename
* Fix examples
* compliance with good c++ manners
* minor changes for consistency
* add missing const
* expired() and bool() moved to iram
* constexpr compensation computing
* add/update comments
* move neverExpires and alwaysExpired
* Add a FAT filesystem for SD cards to Arduino FS
Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.
The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.
This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.
This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.
By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.
Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.
Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).
Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other. Add a simple
bool FS::isDirectory() and bool FS::isFile() method. SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.
Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories. SPIFFS directory operations will, of course, fail
and return false.
Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.
Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls. This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.
The only major visible change for users is that long filenames now are
fully supported and work without any code changes. If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.
While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.
Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.
The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.
* Add a FSConfig and SDFSConfig param to FS.begin()
Allows for configuration values to be passed into a filesystem via the
begin method. By default, a FS will receive a nullptr and should so
whatever is appropriate.
The base FSConfig class has one parameter, _autoFormat, set by the
default constructor to true.
For SPIFFS, you can now disable auto formatting on mount failure by
passing in a FSConfig(false) object.
For SDFS a SDFSConfig parameter can be passed into config specifying the
chip select and SPI configuration. If nothing is passed in, the begin
will fail since there are no safe default values here.
* Add FS::setConfig to set FS-specific options
Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which
takes a FS-specific configuration object and copies any special settings
on a per-FS basis. The call is only valid on unmounted filesystems, and
checks the type of object passed in matches the FS being configured.
Updates the docs and tests to utilize this new configuration method.
* Add ::truncate to File interface
Fixes#3846
* Use polledTimeout for formatting yields, cleanup
Use the new polledTimeout class to ensure a yield every 5ms while
formatting.
Add in default case handling and some debug messages when invalid inputs
specified.
* Make setConfig take const& ref, cleaner code
setConfig now can take a parameter defined directly in the call by using
a const &ref to it, leading to one less line of code to write and
cleaner reading of the code.
Also clean up SDFS implementation pointer definition.
emulation with socket: fix nasty delay
properly check all return values (read() returning 0 indicates closed peer)
ClientContext::state() triggers read() to check for closed peer
uart emulation on tty: add cmdline -b debug option to disable raw tty
hide annoying MOCK messages, add cmdline -v option to show them
uart emulation on tty: check more return values
emulation on host: tcp/udp port shifting: new cmdline -s(port Shift) option
GCC reports the following warning about ignoring the return value of
write() (to the console/UART in this instance).
common/MockUART.cpp: In function ‘void uart_do_write_char(int, char)’:
common/MockUART.cpp:67:8: warning: ignoring return value of ‘ssize_t write(int, const void*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
write(uart_nr + 1, &c, 1);
~~~~~^~~~~~~~~~~~~~~~~~~~
Add a simple warning printout to STDERR which may not be the same as the
UART stream and could succeed.
* emulation on host: Add full UART driver emulation.
This PR replaces the high level Serial mock with a more complete
UART driver. This way the HardwareSerial works without any
modifications. Additionally the driver supports UART0 and UART1
at the same time (UART0 is directed to stdout and UART1 writes
to stderr). RX is implemented by switching the terminal into raw
non-blocking mode and injecting each key-press directly into the
FIFO of UART0. A new command line switch -c was added to ignore
CTRL-C and send it via serial as well. The decumentation was
updated accordingly.
Reading and setting of GPIOs does only write to stderr, when compiled
with D=1. But this is subject to be replaced with a proper GPIO emu
anyway. Reading from GPIO0 now returns 1 instead of 0 because this is
most likely a low active input.
* Fixed unused variable.
* Remove unused functions, as long as there are no debug macros using them.
* Move user_interface.cc from MOCK_CPP_FILES_EMU to MOCK_CPP_FILES.
* Move optimistic_yield() from user_interface.cpp to Arduino.cpp
* Remove atexit() weak declaration (fixes segfault when calling atexit).
* Improve resetting of terminal after program exit, convert \r to \r\n.
* Show error, when STDOUT is not a TTY, minor code cleanung.
* Remove broken ltoa/ultoa, call itoa/utoa
Use the newlib integer-to-ASCII non-POSIX calls instead of rolling
our own. Should be safe as sizeof(long) == sizeof(int).
The custom functions behaved differently from itoa when passed in
negative values in non-base-10.
Add host tests for negative non-base-10 int/longs
* uart fixes and BW improvements
* uart: read_char straightly use hw buffer
* +attributes for functions called by ISR
* uart: BW improvements
read_char straightly use hw buffer (+ ~10%bw)
read by block (+ ~190%bw) (instead of generic Stream::readBytes)
attributes for functions called by ISR
remove overrun message
remove some ISR flags which were not honoured
* fix merge
* fix buffer overflow
* serial stress test sketch
* astyle
* serial stress example: interactive keyboard, stop reading, overrun
* serial device test: bandwidth & overrun
* update + HardwareSerial::hasError()
* interactive overrun in example
* astyle
* Test using @plerup's SoftwareSerial as submodule (tag 3.4.1)
* update upstream ref (fix warning)
* host mock uart/read(buf,size)
* reset style changes in submodules before style diff
* update build_boards_manager_package.sh for submodules
* trigger CI (removing space)
* cannot reproduce locally the CI issue, setting bash -x option to get live trace
* remove previously added (in this PR) 'set -e' in package builder (passes local tests, not real CI)
script-comment new recipe.hooks.core.prebuild.3 (along with already commented .1 and .2)
moved CI package test to be first on the test list
remove 'set -x', wish me luck
