Supercedes #6027
Make SSO more generic by keeping track of its length explicitly,
allowing for embedded \0s to exist in the String (just like the non-SSO
ones).
Use memmove/memcpy_P when we know the length of a string to save CPU
time.
Add tests to inject \0s in a String to ensure it is still working as
designed.
Fixes#6082
Add an info64() call which returns used and total sizes as 64 bit
quantities. A default wrapper that just copies the 32-bit values is
included for LittleFS/SPIFFS which can't hit those capacities.
On return from a BSSL call, check that the last element of the stack is
still untouched. If it is modified, print an error and abort().
Will catch problems like #6143 many times with an informative error
message instead of corrupting the heap and having a random crash
sometime later.
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
Fixes#6081
The SD rewrite blanked out some of the internal FAT info.. Restore the
function calls and return proper values.
Because size() is used in many printf()s, we can't just change its
return type to uint64. Instead, when size is > size-max warn.
Add SD.size64 which can be used by new apps who care about >4GB cards.
Prints a warning if debugging enabled
W/o this change, file::write('a'); tries to use the template and fails
since char is a basic type.
The reason it is needed is due to pre 2.5.x behavior, File::write(char)
silently was cast to File::write(uint8_t). With the template write,
though, this is not performed.
* Add Print::write tests and add'l overrides
Ensure that print::write does something sane and doesn't cause a compile
time error about templates when used for Files.
Test using SPIFFS file since Print is an abstract type.
* Update time.cpp
Migrate configTime() to use sntp_set_timezone_in_seconds() to correctly allow timezone spec in seconds without rounding
* Add sntp-lwip2.h for timezone function
With SSO implementation in String, StreamString::write generates wrong strings under some circumstances.
Reason is that String::len() returns strlen(sso_buf) if SSO=true but with newly written data (in StreamString::write) the null-termination missing at the time len() is called.
Furthermore, len() is called twice which is inefficient if SSO=true.
* Git ignore
* - StreamString fix
* Remove changes to gitignore
* Fix missing space for 0-terminator lost in conversion
* Fix String.replace overlapping strcpy
Fixes#5949
Adds a test from the issue above and fixes the problem valgrind found.
Additional pathological memcpy->memmove fixes
* polledTimeout: add option to use CPU count instead of millis()
* use more "using" alias
* more c++/clear code, using typename (thanks @devyte)
* rename class name to include unit, introduce timeMax() and check it with assert()
* remove useless defines
* improve api readability, add micro-second unit
* update example
* mock: emulate getCycleCount, add/fix polledTimeout CI test
* + nano-seconds, assert -> message, comments, host test
* allow 0 for timeout (enables immediate timeout, fix division by 0)
* typo, set member instead of local variable
* unify error message
* slight change on checkExpired() allows "never expired"
also removed printed message, add YieldAndDelay, simplify calculations
* remove traces of debug.h/cpp in this PR
* include missing <limits> header
* back to original expired test, introduce boolean _neverExpires, fix reset(), getTimeout() is invalid
* fix expiredOneShot with _timeout==0 check
* reenable getTimeout()
* expose checkExpired with unit conversion
* fix timing comments, move critical code to iram
* add member ::neverExpires and use it where relevant
* improve clarity
* remove exposed checkExpired(), adapt LEAmDNS with equivalent
* add API ::resetToNeverExpires(), use it in LEAmDNS
* remove offending constness from ::flagged() LEAmDNS (due do API fix in PolledTimeout)
* simplify "Fast" base classes
* minor variable rename
* Fix examples
* compliance with good c++ manners
* minor changes for consistency
* add missing const
* expired() and bool() moved to iram
* constexpr compensation computing
* add/update comments
* move neverExpires and alwaysExpired
* Fix String::replace()
Fixes#5883 and supercedes #5890
The replace() function was using len() while in the middle of buffer
operations. In SSO mode len() is not stored separately and is a call to
strlen(), which may not be legal if you're in the middle of overwriting
the SSO buffer, as was the case in ::replace when the replacement string
was longer than the find string. This caused potential garbage at the
end of the string when accessed. Instead, just cache the length in a
local while doing the operation.
Add in test cases from #5890 as well as some new ones that fail on the
unmodified core.
* Fix stack smashing error on 64b
When pointers are 8 bytes long, the size of a String is larger than 16
chars. Increase the allocated array we're using in the test to avoid a
"stack smashing" error.
* Manually call destructor in test
Just for clarity, manually call the destructor for the Strings() that
are "placement new'd" in the String tests. It is a no-op for the
existing test, since thanks to SSO there are no memory allocations, but
will help in case someone adds tests later which include longer strings.
* fix DEBUG macros
All fmt strings in flash
fix#5658
This also allows to avoid warnings and easy mistakes with (no brace):
if (something)
DEBUGV("blah");
* use newlib unaligned-compatible printf for DEBUGV
* remove useless putprintf since ::printf already uses ets_putc
* Reimplement SD.h write methods exactly in File
Replace the individual override with the existing SD.h File's
implementation for all methods of File::write.
Fixes#5846
* Add add'l tests
* Fix Print and File incompatible writes w/casts
Print and File have ambiguous resolutions for single-argument
write(0)s.
Fix by adding explicit methods. A write of any integer will not be a
const char* write (i.e. won't write a string) but will instead just
write the integer truncated to 8 bits (as makes sense).
* Use 256byte chunks in ::write template
Reduce stack requirements for templated writes to 256bytes, matching the
size uses in WiFiClient/etc. (from 512bytes). Reduces the chance of
stack overflow.
* Move write(int) methods up to Print.h
Remove some technical debt by moving the ::write(int/short/long) methods
out of FS and HardwareSerial and up into Print.h.
* make (static) ESP8266WebServer::responseCodeToString visible and usable
* esp8266:coreVersionNumeric(): add a define and comment-example to check on its usability
* Add a FAT filesystem for SD cards to Arduino FS
Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.
The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.
This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.
This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.
By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.
Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.
Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).
Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other. Add a simple
bool FS::isDirectory() and bool FS::isFile() method. SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.
Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories. SPIFFS directory operations will, of course, fail
and return false.
Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.
Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls. This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.
The only major visible change for users is that long filenames now are
fully supported and work without any code changes. If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.
While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.
Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.
The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.
* Add a FSConfig and SDFSConfig param to FS.begin()
Allows for configuration values to be passed into a filesystem via the
begin method. By default, a FS will receive a nullptr and should so
whatever is appropriate.
The base FSConfig class has one parameter, _autoFormat, set by the
default constructor to true.
For SPIFFS, you can now disable auto formatting on mount failure by
passing in a FSConfig(false) object.
For SDFS a SDFSConfig parameter can be passed into config specifying the
chip select and SPI configuration. If nothing is passed in, the begin
will fail since there are no safe default values here.
* Add FS::setConfig to set FS-specific options
Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which
takes a FS-specific configuration object and copies any special settings
on a per-FS basis. The call is only valid on unmounted filesystems, and
checks the type of object passed in matches the FS being configured.
Updates the docs and tests to utilize this new configuration method.
* Add ::truncate to File interface
Fixes#3846
* Use polledTimeout for formatting yields, cleanup
Use the new polledTimeout class to ensure a yield every 5ms while
formatting.
Add in default case handling and some debug messages when invalid inputs
specified.
* Make setConfig take const& ref, cleaner code
setConfig now can take a parameter defined directly in the call by using
a const &ref to it, leading to one less line of code to write and
cleaner reading of the code.
Also clean up SDFS implementation pointer definition.
The check if the end address of the SPIFFS partition is smaller or equal to the start of the partition breaks sketches that are linked with zero size SPIFFS linker scripts, but create their own SPIFFS partition later at arbitrary flash locations. But this check can safely be removed, because the check if the the SPIFFS partition is of size 0 is done a few lines further down anyway.
* Implement sntp_get|set_timezone_in_seconds()
* Fix typo
* Fix typo
* Get lwip1.4 to compile
This is just a workaround to get lwip1.4 to compile. It doesn't implement sntp_set_timezone_in_seconds() correctly, but rather does pretty much the same as sntp_set_timezone().
* Fix repaintable stack calculation
Fixes#5794 as found by @mattbradford83
* Overwrite last word of stack as well
Under-by-one error would not reset the absolute end of the stack, adjust
comparison to fix.
* Add progress callback to Updater class.
This is a backport of the same functionality in the ESP32 core.
* Add progress callback to Updater class.
* Add typedef for callback function.
* Fixed initializing order.
* Added missing include (functional).
Reduce String memory overhead from 24 bytes to 16 bytes by limiting the
maximum string length to <64Kbytes (which is larger than heap so no
effective problem).
Add Small String Optimization, SSO, which instead of allocating pointers
to small strings on the heap will store the string in place of the
pointer in the class. This should reduce memory fragmentation as
Save up to 12 chars (11 + \0) in String itself by using the terminating
\0 in the inline string as a flag to identify if this is a SSO or a heap
string.
Add a host test that verifies that no memory is allocated until a
full 11 characters are assigned to a string, as well as checking all
intermediate values.
No user code changes should be required to work with this optimization.
Per 'udp_set_multicast_netif_addr()' signature and comments in lwIP sources:
An IPv4 address designating a specific interface must be used.
When an IPv6 address is given, the matching IPv4 in the same interface must be selected.
fix e3bc3c226b (r32235572)
* added new boards
* corrected block size error
* added board to last commit working well
* override of readString for File class
Stream::readString requires a timeout of 1 seconds + multiple memory resize
* correct indent
return if size 0 before reserve
* correct indent
* good indent
* stricter test for end of string
* same implementation than Stream by replacing timeRead by read
* reading file by block of 256
* make sure there is an end of string
* fixed bug for file size multiple of 256
string::concate(char*) needs a string terminator to work
* Add full gdb support with uart/Serial integration
* Fix GDB merge errors
* Update to unpatched GDB protocol specification
It appears that Espressif patched the open source xtensa GDB port in
order to build their old GDB executable and their old gdbstub (basically
removing any register in a generic xtensa and only leaving those
present in the chip they synthesized). Their GDBStub also assumed this
behavior.
Unpatched upstream GNU GDB now expects all the registers in
xtensa-config.c to be sent/read on a 'g' command. Change the GDB stub
to send "xxxxxxxx"s (legal per the spec) for unimplemented registers.
This makes the 'g' response much longer, but it's results are cached
and in an interactive debugger it isn't noticeable.
* Fix .iram.literal to come before .iram.text for GDB
* Move functions to flash, call using wrappers
All functions which are not interrupt or exception called are now in
flash. A small IRAM wrapper enables flash when processing main GDB ops
by calling Cache_Read_Enable_New() and then jumping to the main flash
code. This seems to work for catching exceptions, data and code breaks,
and Ctrl-C.
The UART ISR handler and exception handler register-saving bits of
code in ASM are still in IRAM.
GDB IRAM usage is now about 670 bytes.
* Remove LWIP2 builder commit
* Add documentation and gdbstub_init header
Add some simple GDB documentation to the main tree showing a worked
example.
Adds the definition of `void gdbstub_init()` to <GDBStub.h>
* Clean up GDB include and library dir
Replace GDBstub.h with the version in the internal/ directory, and
adjust stub code accordingly. This way, only one copy of a file called
"GDBstub.h" will exist.
Update the gdbcommands and replace the obsolete ESPRESSIF readme with
@kylefleming's version since we're mainly doing serial, not TCP,
connected debugging.
Bump the library rev. number since this is a pretty big functionality
change.
Minor documentation tweak.
* Undo much of UART refactoring, set fifo IRQ to 16
Remove the refactoring of pin control and other little things not directly
related to GDB processing. Should greatly reduce the diff size in uart.c.
Should also remove any register value changes (intended or otherwise)
introduced in the original PR from @kylefleming.
Set the FIFO interrupt to 16 chars when in GDB mode, matching the latest
UART configuration for highest speed.
* Add architecture comments, cleanup uart.c code
Comments added to UART.c trying to explain (as best as I understand it)
the changes done to support GDB and how they interact with standard
operation.
Fix the uart_uninit to stop the ISR and then free appropriately.
Fix uart_isr_handle_data (GDB's shim for sending chars to the 8266 app)
to do the exact same thing as the standard UART handler including set
the overflow properly and either discard or overwrite in that case.
Fix serial reception when GDB enabled by enabling the user recv ISR.
Remove commented attributes from gdbstub, leftover from the move to
flash.
General logic cleanup per comments in the PR.
* Also set the UART flags for HW error in GDB
Ensure we also check the UART flags and set the uart status
appropriately when in GDB mode.
* Remove broken ltoa/ultoa, call itoa/utoa
Use the newlib integer-to-ASCII non-POSIX calls instead of rolling
our own. Should be safe as sizeof(long) == sizeof(int).
The custom functions behaved differently from itoa when passed in
negative values in non-base-10.
Add host tests for negative non-base-10 int/longs
