* Make SPIFFS be an integer number of blocks
boards.txt.py simply calculated the end and start using flash sizes, but
in cases where an 8K page was used (>512KB SPIFFS), this could leave a
4K half-block left at the end of SPIFFS.
mkspiffs and the SPIFFS code uses integer division to calculate the
maximum block number, so it worked fine in practice and the code simply
ignored the extra, fractional block.
Now actually take block size into account when calculating the end of
SPIFFS, ensuring no fractional blocks are passed in. Does not result in
data loss on pre-existing SPIFFS filesystems.
* Fix the 1m512 case and clean up code
Ensure that no SPIFFS_block in the LD files is modified from the
original to endure correct backwards compatibility
* Factor out common if, clean code
* Make boards.py vars "fs_xx" instead of "spiffs_xx"
Apply most compatible changes needed to get the core compiling under GCC
7.2 to the main gcc 4.8 tree to ease porting for 3.0.0.
Update pgmspace.h with corrected and optimized unaligned pgm_read
macros. Now pgm_read_dword in the unaligned case gives proper results
even if optimization is enabled and is also written in assembly and only
1 instruction longer than the pgm_read_byte macro (which also has been
optimized to reduce 1 instruction). These changes should marginally
shrink code and speed up flash reads accordingly.
The toolchain should/will be rebuilt at a later time with this
optimization to ensure it's used in the libc.a/etc. files.
Undo the BearSSL RODATA->PROGMEM changes because there are some bad
performance regressions in EC server operations which can result in
timeouts and WDTs.
Keep the shrunked bearssl.a library as that is orthogonal to the PROGMEM
changes.
Rewrite all the integer math operations with const input parameters to
use PROGMEM properly (pgm_read_xx or memcpy_P), and move all the EC
order and generators and SHA OIDs to PROGMEM.
This frees around 1.2KB of heap for any SSL applications.
Also delete unneeded objects from the bearssl.a library to shrink the
GIT repo size.
As found by @mhightower83, umm_malloc was placed in flash during the
.c->.cpp conversion because of a missed linker change.
Adjust the link script to the new name .cpp
Move additional constants to flash and use _P/pgm_read routines to
access them. Minimal runtime impact, but remove variables from RODATA
and gives addition 484 bytes of heap to SSL applications.
Fixes#6005
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
The interrupt vectors in IRAM are omitted when there is a PROVIDE
statement in the linker control files when using the PIO method of
-Wl,-T<linkfile>.
Drop the PROVIDES (they're in RAM anyway and not ROM related), and
add the required "-u"s to the PIO build script.
Should have no impact on the Arduino side.
Fixes#6087
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
* Unaligned access support for pgm_read_word/dword
* Fix pgm_read_ptr_aligned() per #5735
* Allow users to use aligned-only via a #define
Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.
Default is still to allow unaligned accesses.
* Split IRAM into 2 linker sections to move std::fcn
Callbacks need to be placed in IRAM when being called from an IRQ (like
the SPISlave callbacks).
This can be done by hacking the std::functional header and making every
single specialization of the template into an IRAM section, which would
take a lot of space for no benefit in the majority of cases.
The alternate is to specify the single instantiation types/operators
required, but the problem is the flash segment matcher would match them
before the IRAM section was begun, and rules for them would just not be
applied.
Get around this by splitting the IRAM section definition into .text and
.text1. This is linker syntactic sugar and does not actually change the
on-chip layout. But it does allow us to put the exception vectors at
the required absolute addresses and add single functions to IRAM.
* Add .text1 segment to space used calculation in IDE
* All functional callers are now placed in IRAM
* Write out the .text1 segment to the BIN
The extra segment name needs to be placed into the output binary as
well, or else only the init code gets stored and none of the real app is
present. This leads to an infinite boot loop.
This change adds in the segment to the generated image.
In some cases the printf implememtation would call an internal puts()
implementation which did not use pgm_read_byte() to access the format
string. In many operating modes this would work, but in interrupts or
when flash was disabled you'd get crashes.
Updated newlib to use pgm_read_byte in that one spot and recompiled.
ref: d-a-v/esp82xx-nonos-linklayer#31
origin: #5902me-no-dev/ESPAsyncTCP#108
Following the links above is instructive.
To summarize:
* currently and from a long time lwIP tcp client connections always uses the same tcp source port number right after boot
* this port number is increased everytime a new one is needed (= new tcp client connection)
(to be noted, linux has the same increasing behavior)
* when connecting to the same server (right after boot), the triplet (esp-ip-address, source port, destination port) are the same, and may hit remote server list of sockets in time-wait-state (previous connection unproperly closed from the same esp). Consequently the new connection fails when it happens.
* this is happening only when debugging (esp reboots often, in less time than time-wait expiration), so the nasty effect is amplified especially when bugs are being chased
* efforts had been done when espressif's lwIP implementation wasn't open source, with WiFiClient::setLocalPortStart() #632 but it must be explicitely called with a different random number at every reboot. Efficient but not ideal.
This PR uses espressif firmware's r_rand() everytime a new local source port is needed. A different source port number is now showed by tcpdump right after boot. Source port range and duplication is verified everytime in lwIP's src/core/tcp.c:tcp_new_port(). It is implemented as a local patch for upstream lwIP so it is valid not only with WiFiClient but also with @me-no-dev's Async libraries (they don't use WiFiClient).
WiFiClient::setLocalPortStart() is still usable with the same effects as before.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
* dynamic WiFi.hostname("newname")
* WiFi.hostname() back to String return type
* no silent hostname fix but proceed with debug message and returning false
* Add full gdb support with uart/Serial integration
* Fix GDB merge errors
* Update to unpatched GDB protocol specification
It appears that Espressif patched the open source xtensa GDB port in
order to build their old GDB executable and their old gdbstub (basically
removing any register in a generic xtensa and only leaving those
present in the chip they synthesized). Their GDBStub also assumed this
behavior.
Unpatched upstream GNU GDB now expects all the registers in
xtensa-config.c to be sent/read on a 'g' command. Change the GDB stub
to send "xxxxxxxx"s (legal per the spec) for unimplemented registers.
This makes the 'g' response much longer, but it's results are cached
and in an interactive debugger it isn't noticeable.
* Fix .iram.literal to come before .iram.text for GDB
* Move functions to flash, call using wrappers
All functions which are not interrupt or exception called are now in
flash. A small IRAM wrapper enables flash when processing main GDB ops
by calling Cache_Read_Enable_New() and then jumping to the main flash
code. This seems to work for catching exceptions, data and code breaks,
and Ctrl-C.
The UART ISR handler and exception handler register-saving bits of
code in ASM are still in IRAM.
GDB IRAM usage is now about 670 bytes.
* Remove LWIP2 builder commit
* Add documentation and gdbstub_init header
Add some simple GDB documentation to the main tree showing a worked
example.
Adds the definition of `void gdbstub_init()` to <GDBStub.h>
* Clean up GDB include and library dir
Replace GDBstub.h with the version in the internal/ directory, and
adjust stub code accordingly. This way, only one copy of a file called
"GDBstub.h" will exist.
Update the gdbcommands and replace the obsolete ESPRESSIF readme with
@kylefleming's version since we're mainly doing serial, not TCP,
connected debugging.
Bump the library rev. number since this is a pretty big functionality
change.
Minor documentation tweak.
* Undo much of UART refactoring, set fifo IRQ to 16
Remove the refactoring of pin control and other little things not directly
related to GDB processing. Should greatly reduce the diff size in uart.c.
Should also remove any register value changes (intended or otherwise)
introduced in the original PR from @kylefleming.
Set the FIFO interrupt to 16 chars when in GDB mode, matching the latest
UART configuration for highest speed.
* Add architecture comments, cleanup uart.c code
Comments added to UART.c trying to explain (as best as I understand it)
the changes done to support GDB and how they interact with standard
operation.
Fix the uart_uninit to stop the ISR and then free appropriately.
Fix uart_isr_handle_data (GDB's shim for sending chars to the 8266 app)
to do the exact same thing as the standard UART handler including set
the overflow properly and either discard or overwrite in that case.
Fix serial reception when GDB enabled by enabling the user recv ISR.
Remove commented attributes from gdbstub, leftover from the move to
flash.
General logic cleanup per comments in the PR.
* Also set the UART flags for HW error in GDB
Ensure we also check the UART flags and set the uart status
appropriately when in GDB mode.
There is a bug in the BearSSL PEM decoder when Windows EOLs (\r\n) are
passed in. Avoid the issue by silenly discarding \rs as they are read
from the PEM source in the C code, to keep my sanity by avoiding reworking
the pseudo-Forth parser code.
Fixes#5591
PR #5538 made exceptions disabled by default and changed some file names
which didn't get updated in the linker file, resulting in exceptions ending up
back in IRAM.
Scripts, makefiles, and users who do no changes will not have exceptions
enabled during builds. This should avoid the sketch inflation issue for
users who are space constrained, while allowing users who care about
exceptions to enable them through the IDE.
The new non-exception libstdc++ was not referenced in the linker script,
allowing it to end up in IRAM when not needed. Add the line to match
and move it into IROM where it belongs.
* lwip2: better handling of ipv4_addr/t type + 3 sntp servers
* bump lwip2 version
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* Only with FEATURES=1: 3 sntp servers and AutoIP enabled (169.254 when dhcp server fails)
* local CI runner: select build type
* new ipv4_addr/t definition makes things easier for IPAddress
* update local CI runner
* lwip2 changes
* lwip2: port esp-ping and espconn
Remove the -fno-jump-tables since the new toolchain places these tables
in ROM now. Rebuild using the toolchain. Saves 1-3KB of flash and
has 0 RAM impact plus may make certain bits marginally faster by using
a LUT instead of a if-else-else chain.
The complete toolchain, including mkspiffs, esptool, C, C++, newlib,
and others (BearSSL excluded) is now built and uploaded with a single
command to ensure repeatability and minimize manual mistakes. All
OSes and architectures are built at a time.
Update to 2.5.0-2 throught the chain.
* Make exceptions a configurable menu
Add a menu, Exceptions, which allows exceptions to be disabled for ROM
sensitive scripts. Default is enabled.
* Update to latest JSON builder
* Fix the template.json with latest core patches
* Add 64-bit %ll printf format support
Adds support for %lld, %llx, etc. 64-bit integer printing, useful
for logging timestamps and other things.
Fixes#5430
* Remove unwanted updated JSON
* Add %z and %x to printf backend
%z is a C99 format used for size_t and was not included in any printf.
On the 8266 it's a no-op as size_t==int, so ignore it and things just
work.
%x lowercase support added back in (wasn't present in nano-printf).
* Update to toolchain built newlib, fix link error
Previous commit was a hand build and copy, this one used the full
toolchain and should not include atexit().
Looks like the pgm_read_(32bit) defines were not used in the main core, and
they contained syntax errors when invoked due to some bad bracket/parens.
Fix the macros
