Supercedes #6027
Make SSO more generic by keeping track of its length explicitly,
allowing for embedded \0s to exist in the String (just like the non-SSO
ones).
Use memmove/memcpy_P when we know the length of a string to save CPU
time.
Add tests to inject \0s in a String to ensure it is still working as
designed.
__This is a breaking change, but the header and example did warn
everyone that this API was in flux due to the incompatible SD and SPIFFS
File implementations.__
BearSSL CertStores now simply need a filesystem and the names of the
data (generated on-chip) and archive (uploaded by user) files on it.
No more need to roll your own virtual CertStoreFile class.
Update the library, examples, and device test.
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
W/o this change, file::write('a'); tries to use the template and fails
since char is a basic type.
The reason it is needed is due to pre 2.5.x behavior, File::write(char)
silently was cast to File::write(uint8_t). With the template write,
though, this is not performed.
* Add Print::write tests and add'l overrides
Ensure that print::write does something sane and doesn't cause a compile
time error about templates when used for Files.
Test using SPIFFS file since Print is an abstract type.
Force GCC to run with -funsigned-char during host tests to make
the PC match the default behaviour used by the xtensa GCC port.
As noted in #6010. Thanks @MichaelBrunn3r
Some users didn't know they had to update the submodules before running tests on host (see esp8266/Arduino/issues/5910). Added a notice to prevent wasting time of users new to git
* Unaligned access support for pgm_read_word/dword
* Fix pgm_read_ptr_aligned() per #5735
* Allow users to use aligned-only via a #define
Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.
Default is still to allow unaligned accesses.
* Fix String.replace overlapping strcpy
Fixes#5949
Adds a test from the issue above and fixes the problem valgrind found.
Additional pathological memcpy->memmove fixes
* polledTimeout: add option to use CPU count instead of millis()
* use more "using" alias
* more c++/clear code, using typename (thanks @devyte)
* rename class name to include unit, introduce timeMax() and check it with assert()
* remove useless defines
* improve api readability, add micro-second unit
* update example
* mock: emulate getCycleCount, add/fix polledTimeout CI test
* + nano-seconds, assert -> message, comments, host test
* allow 0 for timeout (enables immediate timeout, fix division by 0)
* typo, set member instead of local variable
* unify error message
* slight change on checkExpired() allows "never expired"
also removed printed message, add YieldAndDelay, simplify calculations
* remove traces of debug.h/cpp in this PR
* include missing <limits> header
* back to original expired test, introduce boolean _neverExpires, fix reset(), getTimeout() is invalid
* fix expiredOneShot with _timeout==0 check
* reenable getTimeout()
* expose checkExpired with unit conversion
* fix timing comments, move critical code to iram
* add member ::neverExpires and use it where relevant
* improve clarity
* remove exposed checkExpired(), adapt LEAmDNS with equivalent
* add API ::resetToNeverExpires(), use it in LEAmDNS
* remove offending constness from ::flagged() LEAmDNS (due do API fix in PolledTimeout)
* simplify "Fast" base classes
* minor variable rename
* Fix examples
* compliance with good c++ manners
* minor changes for consistency
* add missing const
* expired() and bool() moved to iram
* constexpr compensation computing
* add/update comments
* move neverExpires and alwaysExpired
* Fix String::replace()
Fixes#5883 and supercedes #5890
The replace() function was using len() while in the middle of buffer
operations. In SSO mode len() is not stored separately and is a call to
strlen(), which may not be legal if you're in the middle of overwriting
the SSO buffer, as was the case in ::replace when the replacement string
was longer than the find string. This caused potential garbage at the
end of the string when accessed. Instead, just cache the length in a
local while doing the operation.
Add in test cases from #5890 as well as some new ones that fail on the
unmodified core.
* Fix stack smashing error on 64b
When pointers are 8 bytes long, the size of a String is larger than 16
chars. Increase the allocated array we're using in the test to avoid a
"stack smashing" error.
* Manually call destructor in test
Just for clarity, manually call the destructor for the Strings() that
are "placement new'd" in the String tests. It is a no-op for the
existing test, since thanks to SSO there are no memory allocations, but
will help in case someone adds tests later which include longer strings.
* optionally allow redirects on http OTA updates
* Refactored HTTPClient::begin(url...) & setURL functions, now only beginInternal parses URL, sets ports
Added HTTPRedirect example.
* fix indentation for style check
* add space after while for style check
* don't use deprecated begin method in redirect example
* moved redirect handling code to HTTPClient.
only GET and HEAD requests are currently handled automatically
Redirects that fail to be automatically handled return the redirect code as before
* added support for POST/303 redirect
added device redirect tests
* add missing getLocation() implementation
* if the new location is only a path then only update the URI
* Reimplement SD.h write methods exactly in File
Replace the individual override with the existing SD.h File's
implementation for all methods of File::write.
Fixes#5846
* Add add'l tests
* Fix Print and File incompatible writes w/casts
Print and File have ambiguous resolutions for single-argument
write(0)s.
Fix by adding explicit methods. A write of any integer will not be a
const char* write (i.e. won't write a string) but will instead just
write the integer truncated to 8 bits (as makes sense).
* Use 256byte chunks in ::write template
Reduce stack requirements for templated writes to 256bytes, matching the
size uses in WiFiClient/etc. (from 512bytes). Reduces the chance of
stack overflow.
* Move write(int) methods up to Print.h
Remove some technical debt by moving the ::write(int/short/long) methods
out of FS and HardwareSerial and up into Print.h.
* Add a FAT filesystem for SD cards to Arduino FS
Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.
The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.
This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.
This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.
By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.
Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.
Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).
Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other. Add a simple
bool FS::isDirectory() and bool FS::isFile() method. SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.
Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories. SPIFFS directory operations will, of course, fail
and return false.
Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.
Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls. This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.
The only major visible change for users is that long filenames now are
fully supported and work without any code changes. If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.
While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.
Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.
The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.
* Add a FSConfig and SDFSConfig param to FS.begin()
Allows for configuration values to be passed into a filesystem via the
begin method. By default, a FS will receive a nullptr and should so
whatever is appropriate.
The base FSConfig class has one parameter, _autoFormat, set by the
default constructor to true.
For SPIFFS, you can now disable auto formatting on mount failure by
passing in a FSConfig(false) object.
For SDFS a SDFSConfig parameter can be passed into config specifying the
chip select and SPI configuration. If nothing is passed in, the begin
will fail since there are no safe default values here.
* Add FS::setConfig to set FS-specific options
Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which
takes a FS-specific configuration object and copies any special settings
on a per-FS basis. The call is only valid on unmounted filesystems, and
checks the type of object passed in matches the FS being configured.
Updates the docs and tests to utilize this new configuration method.
* Add ::truncate to File interface
Fixes#3846
* Use polledTimeout for formatting yields, cleanup
Use the new polledTimeout class to ensure a yield every 5ms while
formatting.
Add in default case handling and some debug messages when invalid inputs
specified.
* Make setConfig take const& ref, cleaner code
setConfig now can take a parameter defined directly in the call by using
a const &ref to it, leading to one less line of code to write and
cleaner reading of the code.
Also clean up SDFS implementation pointer definition.
emulation with socket: fix nasty delay
properly check all return values (read() returning 0 indicates closed peer)
ClientContext::state() triggers read() to check for closed peer
uart emulation on tty: add cmdline -b debug option to disable raw tty
hide annoying MOCK messages, add cmdline -v option to show them
uart emulation on tty: check more return values
emulation on host: tcp/udp port shifting: new cmdline -s(port Shift) option
GCC reports the following warning about ignoring the return value of
write() (to the console/UART in this instance).
common/MockUART.cpp: In function ‘void uart_do_write_char(int, char)’:
common/MockUART.cpp:67:8: warning: ignoring return value of ‘ssize_t write(int, const void*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
write(uart_nr + 1, &c, 1);
~~~~~^~~~~~~~~~~~~~~~~~~~
Add a simple warning printout to STDERR which may not be the same as the
UART stream and could succeed.
* emulation on host: Add full UART driver emulation.
This PR replaces the high level Serial mock with a more complete
UART driver. This way the HardwareSerial works without any
modifications. Additionally the driver supports UART0 and UART1
at the same time (UART0 is directed to stdout and UART1 writes
to stderr). RX is implemented by switching the terminal into raw
non-blocking mode and injecting each key-press directly into the
FIFO of UART0. A new command line switch -c was added to ignore
CTRL-C and send it via serial as well. The decumentation was
updated accordingly.
Reading and setting of GPIOs does only write to stderr, when compiled
with D=1. But this is subject to be replaced with a proper GPIO emu
anyway. Reading from GPIO0 now returns 1 instead of 0 because this is
most likely a low active input.
* Fixed unused variable.
* Remove unused functions, as long as there are no debug macros using them.
* Move user_interface.cc from MOCK_CPP_FILES_EMU to MOCK_CPP_FILES.
* Move optimistic_yield() from user_interface.cpp to Arduino.cpp
* Remove atexit() weak declaration (fixes segfault when calling atexit).
* Improve resetting of terminal after program exit, convert \r to \r\n.
* Show error, when STDOUT is not a TTY, minor code cleanung.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
This patch splits ARDUINO_LIBS into two variables. The second variable is called OPT_ARDUINO_LIBS and can be overridden by the caller as it uses the ?= assignment operator. Additionally it unifies and simplifies collecting the files in common/ by using the addprefix macro. All changes should be 100% backwards compatible.
Reduce String memory overhead from 24 bytes to 16 bytes by limiting the
maximum string length to <64Kbytes (which is larger than heap so no
effective problem).
Add Small String Optimization, SSO, which instead of allocating pointers
to small strings on the heap will store the string in place of the
pointer in the class. This should reduce memory fragmentation as
Save up to 12 chars (11 + \0) in String itself by using the terminating
\0 in the inline string as a flag to identify if this is a SSO or a heap
string.
Add a host test that verifies that no memory is allocated until a
full 11 characters are assigned to a string, as well as checking all
intermediate values.
No user code changes should be required to work with this optimization.
* Remove broken ltoa/ultoa, call itoa/utoa
Use the newlib integer-to-ASCII non-POSIX calls instead of rolling
our own. Should be safe as sizeof(long) == sizeof(int).
The custom functions behaved differently from itoa when passed in
negative values in non-base-10.
Add host tests for negative non-base-10 int/longs
