Undo the BearSSL RODATA->PROGMEM changes because there are some bad
performance regressions in EC server operations which can result in
timeouts and WDTs.
Keep the shrunked bearssl.a library as that is orthogonal to the PROGMEM
changes.
Rewrite all the integer math operations with const input parameters to
use PROGMEM properly (pgm_read_xx or memcpy_P), and move all the EC
order and generators and SHA OIDs to PROGMEM.
This frees around 1.2KB of heap for any SSL applications.
Also delete unneeded objects from the bearssl.a library to shrink the
GIT repo size.
As found by @mhightower83, umm_malloc was placed in flash during the
.c->.cpp conversion because of a missed linker change.
Adjust the link script to the new name .cpp
Move additional constants to flash and use _P/pgm_read routines to
access them. Minimal runtime impact, but remove variables from RODATA
and gives addition 484 bytes of heap to SSL applications.
Fixes#6005
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
The interrupt vectors in IRAM are omitted when there is a PROVIDE
statement in the linker control files when using the PIO method of
-Wl,-T<linkfile>.
Drop the PROVIDES (they're in RAM anyway and not ROM related), and
add the required "-u"s to the PIO build script.
Should have no impact on the Arduino side.
Fixes#6087
* Upgrade to https: serving for JSON, links in docs
Fixes#5480
* Update boards.rst documentation
* Update more documentation http: refs to https:
* Remove obsolete staging info
* Drop obsolete versions from JSON programatically
After the final merge is done on the JSON, strip out any named versions
from the final product.
Removing 1.6.5-* and 2.5.0-beta(1,2,3) for now.
* Remove 2.4.0-rc(0/1) from JSON, too
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
* New menu option to minimize BSSL ROM with only RSA
Adds a menu option and define to limit BearSSL to older RSA connection
options. This saves ~45K program memory and can speed up connections
since EC, while more secure, is significantly slower on the chip.
The supported ciphers are identical to the ones that axTLS supported.
Fixes#6005
* Add default SSLFLAGS(blank) to platform.txt
* Fix unused variable warning
* Add clarifying comment to menu items
* Unaligned access support for pgm_read_word/dword
* Fix pgm_read_ptr_aligned() per #5735
* Allow users to use aligned-only via a #define
Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.
Default is still to allow unaligned accesses.
* Split IRAM into 2 linker sections to move std::fcn
Callbacks need to be placed in IRAM when being called from an IRQ (like
the SPISlave callbacks).
This can be done by hacking the std::functional header and making every
single specialization of the template into an IRAM section, which would
take a lot of space for no benefit in the majority of cases.
The alternate is to specify the single instantiation types/operators
required, but the problem is the flash segment matcher would match them
before the IRAM section was begun, and rules for them would just not be
applied.
Get around this by splitting the IRAM section definition into .text and
.text1. This is linker syntactic sugar and does not actually change the
on-chip layout. But it does allow us to put the exception vectors at
the required absolute addresses and add single functions to IRAM.
* Add .text1 segment to space used calculation in IDE
* All functional callers are now placed in IRAM
* Write out the .text1 segment to the BIN
The extra segment name needs to be placed into the output binary as
well, or else only the init code gets stored and none of the real app is
present. This leads to an infinite boot loop.
This change adds in the segment to the generated image.
In some cases the printf implememtation would call an internal puts()
implementation which did not use pgm_read_byte() to access the format
string. In many operating modes this would work, but in interrupts or
when flash was disabled you'd get crashes.
Updated newlib to use pgm_read_byte in that one spot and recompiled.
ref: d-a-v/esp82xx-nonos-linklayer#31
origin: #5902me-no-dev/ESPAsyncTCP#108
Following the links above is instructive.
To summarize:
* currently and from a long time lwIP tcp client connections always uses the same tcp source port number right after boot
* this port number is increased everytime a new one is needed (= new tcp client connection)
(to be noted, linux has the same increasing behavior)
* when connecting to the same server (right after boot), the triplet (esp-ip-address, source port, destination port) are the same, and may hit remote server list of sockets in time-wait-state (previous connection unproperly closed from the same esp). Consequently the new connection fails when it happens.
* this is happening only when debugging (esp reboots often, in less time than time-wait expiration), so the nasty effect is amplified especially when bugs are being chased
* efforts had been done when espressif's lwIP implementation wasn't open source, with WiFiClient::setLocalPortStart() #632 but it must be explicitely called with a different random number at every reboot. Efficient but not ideal.
This PR uses espressif firmware's r_rand() everytime a new local source port is needed. A different source port number is now showed by tcpdump right after boot. Source port range and duplication is verified everytime in lwIP's src/core/tcp.c:tcp_new_port(). It is implemented as a local patch for upstream lwIP so it is valid not only with WiFiClient but also with @me-no-dev's Async libraries (they don't use WiFiClient).
WiFiClient::setLocalPortStart() is still usable with the same effects as before.
No need to explicitly send in full paths for esptool and pyserial Python
libs because they're in well known locations relative to upload.py.
Make upload.py calculate these on-the-fly and clean up platform.txt.
Simply import the pyserial and esptool modules directly into upload.py
instead of trying to fake things with os.fork()s. Reduces code and is
more Pythonic.
This commit allows switching SDK firmware:
nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:
* Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)
* Overall performances seem to have decreased (#5513)
This PR restores sdk2.2.1 (as in core-2.4.2).
SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.
BREAKING
* new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`
* for external build systems: new library directory: `tools/sdk/lib/<version>/lib`
* PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.
Fix#5736
* Add esptool.py, pyserial, and python to JSON
Add installation of python on Win32/Win64, and on all systems install
esptool.py and pyserial.
* Initial esptool.py upload test
* First successfull esptool.py upload
* Patch in verbose flag operation
* Replace esptool-ck.exe with Python equivalent
Remove need for binary esptool-ck.exe by implementing the same logic as
esptool-ck uses in Python. Only image creation is supported, and only
in the Arduino standard mode (with its custom bootloader and ROM
layout).
* Remove all esptool-ck.exe, hook Windows Python
Remove all references to esptool-ck and use Python on Windows and Linux
for all recipes where possible.
* Use python to make core_version as well
Avoid ugly bash and CMD.exe tricks in platform.txt by using python to
make the core_version header.
* Rename conflicting script, clean up packager
* Windows test passes
Need to make sure Python2 and Python3 compatible and paths are munged
properly to avoid eaccidentally escaping things when calling esptool.py
Able to compile, build a BIN and upload via esptool.py on a Windows
machine without Python installed globally, only as part of the Arduino
tools package.
* Use github sources for pyserial
* Erase calibration or all flash before programming
Add back in erase support by calling esptool.py twice (since it does not
support chained operations like esptool-ck.exe).
* Make 460K default speed, remove 961K
961K doesn't seem to work with esptool, so make 460K the default upload
speed and remove 961K.
Even at this lower speed, esptool.py is much faster to upload (even
before taking into account the compression when doing things like SPIFFS
and code upload).
* Make erase and upload work again
Arduino does not support a upload.#.cmd pattern, so we need to do
everything in a single command line. Make it cleaner by introducing a
Python wrapper script which will run the same executable with different
sets of commands (since we need to erase a block w/a separate invocation
from the real upload).
Update boards.txt to use the new options format, placing the esptool
command as "version" when there is no "erase_flash" or "erase_region" to
be done to keep things simple.
* Move esptool/pyserial to submodules
Since esptool.py and pyserial are coming directly from github repos,
there is no need to include them as a tool in package.json.
* Restore 921K upload opt, silent downgrade to 460k
To enable full backward compatibility, restore the 921k option for
upload speed but silently change it to 460k in the upload.py script.
Add error checking on upload.py
Adding the Arduino repo as a git submodule can be very useful for version control. However, `git submodule update` is not enough, as the tools need to be downloaded through `get.py`. Wrapping the main code in that script into its own function allows an external Python script to import the module and execute its functionality without relying on the unsafe `execfile` (or `Popen`). The main advantage would be easy flow control in case of error (eg: issue #4464).
* dynamic WiFi.hostname("newname")
* WiFi.hostname() back to String return type
* no silent hostname fix but proceed with debug message and returning false
* Add full gdb support with uart/Serial integration
* Fix GDB merge errors
* Update to unpatched GDB protocol specification
It appears that Espressif patched the open source xtensa GDB port in
order to build their old GDB executable and their old gdbstub (basically
removing any register in a generic xtensa and only leaving those
present in the chip they synthesized). Their GDBStub also assumed this
behavior.
Unpatched upstream GNU GDB now expects all the registers in
xtensa-config.c to be sent/read on a 'g' command. Change the GDB stub
to send "xxxxxxxx"s (legal per the spec) for unimplemented registers.
This makes the 'g' response much longer, but it's results are cached
and in an interactive debugger it isn't noticeable.
* Fix .iram.literal to come before .iram.text for GDB
* Move functions to flash, call using wrappers
All functions which are not interrupt or exception called are now in
flash. A small IRAM wrapper enables flash when processing main GDB ops
by calling Cache_Read_Enable_New() and then jumping to the main flash
code. This seems to work for catching exceptions, data and code breaks,
and Ctrl-C.
The UART ISR handler and exception handler register-saving bits of
code in ASM are still in IRAM.
GDB IRAM usage is now about 670 bytes.
* Remove LWIP2 builder commit
* Add documentation and gdbstub_init header
Add some simple GDB documentation to the main tree showing a worked
example.
Adds the definition of `void gdbstub_init()` to <GDBStub.h>
* Clean up GDB include and library dir
Replace GDBstub.h with the version in the internal/ directory, and
adjust stub code accordingly. This way, only one copy of a file called
"GDBstub.h" will exist.
Update the gdbcommands and replace the obsolete ESPRESSIF readme with
@kylefleming's version since we're mainly doing serial, not TCP,
connected debugging.
Bump the library rev. number since this is a pretty big functionality
change.
Minor documentation tweak.
* Undo much of UART refactoring, set fifo IRQ to 16
Remove the refactoring of pin control and other little things not directly
related to GDB processing. Should greatly reduce the diff size in uart.c.
Should also remove any register value changes (intended or otherwise)
introduced in the original PR from @kylefleming.
Set the FIFO interrupt to 16 chars when in GDB mode, matching the latest
UART configuration for highest speed.
* Add architecture comments, cleanup uart.c code
Comments added to UART.c trying to explain (as best as I understand it)
the changes done to support GDB and how they interact with standard
operation.
Fix the uart_uninit to stop the ISR and then free appropriately.
Fix uart_isr_handle_data (GDB's shim for sending chars to the 8266 app)
to do the exact same thing as the standard UART handler including set
the overflow properly and either discard or overwrite in that case.
Fix serial reception when GDB enabled by enabling the user recv ISR.
Remove commented attributes from gdbstub, leftover from the move to
flash.
General logic cleanup per comments in the PR.
* Also set the UART flags for HW error in GDB
Ensure we also check the UART flags and set the uart status
appropriately when in GDB mode.
