As found by @mhightower83, umm_malloc was placed in flash during the
.c->.cpp conversion because of a missed linker change.
Adjust the link script to the new name .cpp
Move additional constants to flash and use _P/pgm_read routines to
access them. Minimal runtime impact, but remove variables from RODATA
and gives addition 484 bytes of heap to SSL applications.
Fixes#6005
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
The interrupt vectors in IRAM are omitted when there is a PROVIDE
statement in the linker control files when using the PIO method of
-Wl,-T<linkfile>.
Drop the PROVIDES (they're in RAM anyway and not ROM related), and
add the required "-u"s to the PIO build script.
Should have no impact on the Arduino side.
Fixes#6087
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
* Unaligned access support for pgm_read_word/dword
* Fix pgm_read_ptr_aligned() per #5735
* Allow users to use aligned-only via a #define
Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.
Default is still to allow unaligned accesses.
* Split IRAM into 2 linker sections to move std::fcn
Callbacks need to be placed in IRAM when being called from an IRQ (like
the SPISlave callbacks).
This can be done by hacking the std::functional header and making every
single specialization of the template into an IRAM section, which would
take a lot of space for no benefit in the majority of cases.
The alternate is to specify the single instantiation types/operators
required, but the problem is the flash segment matcher would match them
before the IRAM section was begun, and rules for them would just not be
applied.
Get around this by splitting the IRAM section definition into .text and
.text1. This is linker syntactic sugar and does not actually change the
on-chip layout. But it does allow us to put the exception vectors at
the required absolute addresses and add single functions to IRAM.
* Add .text1 segment to space used calculation in IDE
* All functional callers are now placed in IRAM
* Write out the .text1 segment to the BIN
The extra segment name needs to be placed into the output binary as
well, or else only the init code gets stored and none of the real app is
present. This leads to an infinite boot loop.
This change adds in the segment to the generated image.
In some cases the printf implememtation would call an internal puts()
implementation which did not use pgm_read_byte() to access the format
string. In many operating modes this would work, but in interrupts or
when flash was disabled you'd get crashes.
Updated newlib to use pgm_read_byte in that one spot and recompiled.
ref: d-a-v/esp82xx-nonos-linklayer#31
origin: #5902me-no-dev/ESPAsyncTCP#108
Following the links above is instructive.
To summarize:
* currently and from a long time lwIP tcp client connections always uses the same tcp source port number right after boot
* this port number is increased everytime a new one is needed (= new tcp client connection)
(to be noted, linux has the same increasing behavior)
* when connecting to the same server (right after boot), the triplet (esp-ip-address, source port, destination port) are the same, and may hit remote server list of sockets in time-wait-state (previous connection unproperly closed from the same esp). Consequently the new connection fails when it happens.
* this is happening only when debugging (esp reboots often, in less time than time-wait expiration), so the nasty effect is amplified especially when bugs are being chased
* efforts had been done when espressif's lwIP implementation wasn't open source, with WiFiClient::setLocalPortStart() #632 but it must be explicitely called with a different random number at every reboot. Efficient but not ideal.
This PR uses espressif firmware's r_rand() everytime a new local source port is needed. A different source port number is now showed by tcpdump right after boot. Source port range and duplication is verified everytime in lwIP's src/core/tcp.c:tcp_new_port(). It is implemented as a local patch for upstream lwIP so it is valid not only with WiFiClient but also with @me-no-dev's Async libraries (they don't use WiFiClient).
WiFiClient::setLocalPortStart() is still usable with the same effects as before.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
* dynamic WiFi.hostname("newname")
* WiFi.hostname() back to String return type
* no silent hostname fix but proceed with debug message and returning false
* Add full gdb support with uart/Serial integration
* Fix GDB merge errors
* Update to unpatched GDB protocol specification
It appears that Espressif patched the open source xtensa GDB port in
order to build their old GDB executable and their old gdbstub (basically
removing any register in a generic xtensa and only leaving those
present in the chip they synthesized). Their GDBStub also assumed this
behavior.
Unpatched upstream GNU GDB now expects all the registers in
xtensa-config.c to be sent/read on a 'g' command. Change the GDB stub
to send "xxxxxxxx"s (legal per the spec) for unimplemented registers.
This makes the 'g' response much longer, but it's results are cached
and in an interactive debugger it isn't noticeable.
* Fix .iram.literal to come before .iram.text for GDB
* Move functions to flash, call using wrappers
All functions which are not interrupt or exception called are now in
flash. A small IRAM wrapper enables flash when processing main GDB ops
by calling Cache_Read_Enable_New() and then jumping to the main flash
code. This seems to work for catching exceptions, data and code breaks,
and Ctrl-C.
The UART ISR handler and exception handler register-saving bits of
code in ASM are still in IRAM.
GDB IRAM usage is now about 670 bytes.
* Remove LWIP2 builder commit
* Add documentation and gdbstub_init header
Add some simple GDB documentation to the main tree showing a worked
example.
Adds the definition of `void gdbstub_init()` to <GDBStub.h>
* Clean up GDB include and library dir
Replace GDBstub.h with the version in the internal/ directory, and
adjust stub code accordingly. This way, only one copy of a file called
"GDBstub.h" will exist.
Update the gdbcommands and replace the obsolete ESPRESSIF readme with
@kylefleming's version since we're mainly doing serial, not TCP,
connected debugging.
Bump the library rev. number since this is a pretty big functionality
change.
Minor documentation tweak.
* Undo much of UART refactoring, set fifo IRQ to 16
Remove the refactoring of pin control and other little things not directly
related to GDB processing. Should greatly reduce the diff size in uart.c.
Should also remove any register value changes (intended or otherwise)
introduced in the original PR from @kylefleming.
Set the FIFO interrupt to 16 chars when in GDB mode, matching the latest
UART configuration for highest speed.
* Add architecture comments, cleanup uart.c code
Comments added to UART.c trying to explain (as best as I understand it)
the changes done to support GDB and how they interact with standard
operation.
Fix the uart_uninit to stop the ISR and then free appropriately.
Fix uart_isr_handle_data (GDB's shim for sending chars to the 8266 app)
to do the exact same thing as the standard UART handler including set
the overflow properly and either discard or overwrite in that case.
Fix serial reception when GDB enabled by enabling the user recv ISR.
Remove commented attributes from gdbstub, leftover from the move to
flash.
General logic cleanup per comments in the PR.
* Also set the UART flags for HW error in GDB
Ensure we also check the UART flags and set the uart status
appropriately when in GDB mode.
There is a bug in the BearSSL PEM decoder when Windows EOLs (\r\n) are
passed in. Avoid the issue by silenly discarding \rs as they are read
from the PEM source in the C code, to keep my sanity by avoiding reworking
the pseudo-Forth parser code.
Fixes#5591
PR #5538 made exceptions disabled by default and changed some file names
which didn't get updated in the linker file, resulting in exceptions ending up
back in IRAM.
Scripts, makefiles, and users who do no changes will not have exceptions
enabled during builds. This should avoid the sketch inflation issue for
users who are space constrained, while allowing users who care about
exceptions to enable them through the IDE.
The new non-exception libstdc++ was not referenced in the linker script,
allowing it to end up in IRAM when not needed. Add the line to match
and move it into IROM where it belongs.
* lwip2: better handling of ipv4_addr/t type + 3 sntp servers
* bump lwip2 version
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* local CI runner: select build type
* new ipv4_addr/t definition makes things easier for IPAddress
* update local CI runner
* lwip2 changes
* lwip2: port esp-ping and espconn
Remove the -fno-jump-tables since the new toolchain places these tables
in ROM now. Rebuild using the toolchain. Saves 1-3KB of flash and
has 0 RAM impact plus may make certain bits marginally faster by using
a LUT instead of a if-else-else chain.
The complete toolchain, including mkspiffs, esptool, C, C++, newlib,
and others (BearSSL excluded) is now built and uploaded with a single
command to ensure repeatability and minimize manual mistakes. All
OSes and architectures are built at a time.
Update to 2.5.0-2 throught the chain.
* Make exceptions a configurable menu
Add a menu, Exceptions, which allows exceptions to be disabled for ROM
sensitive scripts. Default is enabled.
* Update to latest JSON builder
* Fix the template.json with latest core patches
* Add 64-bit %ll printf format support
Adds support for %lld, %llx, etc. 64-bit integer printing, useful
for logging timestamps and other things.
Fixes#5430
* Remove unwanted updated JSON
* Add %z and %x to printf backend
%z is a C99 format used for size_t and was not included in any printf.
On the 8266 it's a no-op as size_t==int, so ignore it and things just
work.
%x lowercase support added back in (wasn't present in nano-printf).
* Update to toolchain built newlib, fix link error
Previous commit was a hand build and copy, this one used the full
toolchain and should not include atexit().
Looks like the pgm_read_(32bit) defines were not used in the main core, and
they contained syntax errors when invoked due to some bad bracket/parens.
Fix the macros
* Move to PROGMEM aware libc, allow PSTR in printf()
A Newlib (libc) patch is in progress to move the _P functions from inside
Arduino into first-class citizens in libc. This Arduino patch cleans up
code that's been migrated there. Binaries for the new libs are included
because it seems they're part of the Arduino git tree, and should be
replaced with @igrr built ones when/if the Newlib changes are accepted.
Notable changes/additions for Arduino:
Allow for use of PROGMEM based format and parameter strings in all
*printf functions. No need for copying PSTR()s into RAM before printing
them out (transparently saves heap space when using _P functions) and
makes it easier to print out constant strings for applications.
Add "%S" (capital-S) format that I've been told, but cannot verify,
is used in Arduino to specify a PROGMEM string parameter in printfs,
as an alias for "%s" since plain "%s" can now handle PROGMEM.
Optimized the memcpy_P, strnlen_P, and strncpy_P functions to use 32-bit
direct reads whenver possible (source and dest alignment mediated), but
there is still room for improvement in others.
Finally, move several constant arrays from RODATA into PROGMEM and
update their accessors. Among these are the ctype array, ~260 bytes,
mprec* arrays, ~300 bytes, and strings/daycounts in the time
formatting functions, ~200 bytes. All told, sketches will see from
300 to 800 additional RAM heap free on startup (depending on their
use of these routines).
* Fix merge error in #ifdef/#endif
* Fix host test using the newlib generic pgmspace.h
Host tests now use the sys/pgmspace.h for compiles instead of the
ESP8266-specific version.
* Update with rebuilt libraries using latest newlib
* Include binaries built directly from @igrr repo
Rebuild the binaries using a git clone of
https://github.com/igrr/newlib-xtensa
Build commands for posterity:
````
rm -rf ./xtensa-lx106-elf/
./configure --prefix=<DIR>/esp8266/tools/sdk/libc --with-newlib \
--enable-multilib --disable-newlib-io-c99-formats \
--disable-newlib-supplied-syscalls \
--enable-newlib-nano-formatted-io --enable-newlib-reent-small \
--enable-target-optspace \
--program-transform-name="s&^&xtensa-lx106-elf-&" \
--disable-option-checking --with-target-subdir=xtensa-lx106-elf \
--target=xtensa-lx106-elf
rm -f etc/config.cache
CROSS_CFLAGS="-fno-omit-frame-pointer -DSIGNAL_PROVIDED -DABORT_PROVIDED"\
" -DMALLOC_PROVIDED" \
PATH=<DIR>/esp8266/tools/xtensa-lx106-elf/bin/:$PATH \
make all install
````
* Fix merge define conflict in c_types.h
* Fix strlen_P misaligned source error
Include fix from newlib-xtensa/fix-strlen branch cleaning up misaligned
access on a non-aligned source string.
* Fix strlen_P and strcpy_P edge cases
Ran the included test suite on ESP8266 tstring.c with the following defines:
#define MAX_1 50
#define memcmp memcmp_P
#define memcpy memcpy_P
#define memmem memmem_P
#define memchr memchr_P
#define strcat strcat_P
#define strncat strncat_P
#define strcpy strcpy_P
#define strlen strlen_P
#define strnlen strnlen_P
#define strcmp strcmp_P
#define strncmp strncmp_P
Uncovered edge case and return value problems in the optimized versions of
the strnlen_P and strncpy_P functions. Corrected.
* Fix memcpy_P return value
memcpy-1.c test suite showed error in return value of memcpy_P. Correct it.
* Fix strnlen_P/strlen_P off-by-4 error
Random crashes, often on String constructors using a PSTR, would occur due
to the accelerated strnlen_P going past the end of the string. Would make
debug builds fail, too (ESP.getVersionString() failure).
Fix to fall through to normal copy on a word that's got a 0 byte anywhere
in it.
* Add device tests for libc functional verification
Add test suite used to debug libc optimized _P functions to the device
tests.
* Rebuild from igrr's repo (same source as prior)
Rebuild .a from igrr's repo at 347260af117b4177389e69fd4d04169b11d87a97
* WIP - add exceptions
* Fix exception to have 0-terminator
* Move some exception constants to TEXT from RODATA
* Remove throw stubs
* Move more exception stuff to ROM
* Enable exceptions in platform.io
* Remove atexit, is duplicated in rebuilt lib
Need to look at the quick-toolchain options, there seems to be a definition
for atexit defined there (libgcc?) that needs to be excised. For now,
remove our local do-nothing copy.
* Update libgcc to remove soft-fp functions
The esp-quick-toolchain generated libgcc.a needed to have the soft-FP routines
that are in ROM removed from it. Remove them in the new esp-quick-toolchain
and update.
* Fix merge typos in Makefile
* Add unhandled exception handler to postmortem
* Return our atexit() handler
* Latest stdc++, minimize exception emercengy area
* Remove atexit from newlib
atexit was defined in newlib strongly, but we also define a noop atexit in core.
Since we never exit, use the core's noop and delete the atexit from libc.a
Updated in esp-quick-toolchain as well.
* Move __FUNCTION__ static strings to PROGMEM
__FUNCTION__ is unlikely to be a timing sensitive variable, so move it to
PROGMEM and not RODATA (RAM) using linker magic.
asserts() now should take no RAM for any strings.
* Clean up linker file, update to latest stdc++
* Update to latest stdc++ which doesn't call strerror
* Update to GCC5.1 exception emergency allocator
Using GCC 5.1's emergency memory allocator for exceptions, much less
space is required in programs which do not use exceptions and when
space is allocated it is managed more efficiently.
* Initial try with new compiler toolchain
* Include newlib built from esp-quick-toolchain
* Update JSON with all new esp-quick-toolchain builds
* Use 64bit Windows compiler on 64bit Windows
* Dump std::exception.what() when possible
When doing the panic on unhandled exceptions, try and grab the
.what() pointer and dump it as part of the termination info.
Makes it easy to see mem errors (std::bad_alloc) or std::runtime_error
strings.
* Use scripted install from esp-quick-toolchain
Makes sure proper libraries and includes are present by using a
scripted installation from esp-quick-install instead of a manual
one.
* Update eqk to remove atexit, fix packaging diff
* WIP compile examples on host with 'make examples'
* WIP bufferize tcp input
* WIP Makefile
* WIP network to rework, tcp/udp to factorize, udp addresses broken
* minor changes to the core
* WIP basic udp working
* WIP mdns
* WIP mcast receiving, not sending
* WIP mdns OK
* beta version
* SSL + doc
* update travis host test command
* licenses
* typo
* doc: arduino builder is not around: declare functions before calling them
* fix with latest SSL PR, compile in 32 bits mode
* fix make clean
* make -m32 optional
* 32bits compiler ability tester
* WIP
* WIP (fix 1 vtable error, still another one to hunt with using spiffs)
* example astyle
* fix os_printf_plus
* load / save mock spiffs
* fix style
* fix using spiffs/mock
* don't mess ram
* update doc
* remove leftover
* optimization -Os except for CI, rename ARCH32 to FORCE32
* revert useless cast (not even compiled)
* remove unused function
* use proper type for pointer arithmetics
* makefile: sketch object and cpp file moved to bin/ directories
easier to clean, and IDE don't like them
* changes for review
* make use of %zd
* less verbose makefile by default (option)
* update readme
