* Replace the SDK's use of ets_intr_lock/unlock with nestable versions
Testing has shown that there are several paths in the SDK that result in nested
calls to ets_intr_lock() / ets_intr_unlock() which may be a problem.
These functions also do not preserve the enabled interrupt level and may
result in code running with interrupts enabled when that is not intended.
This issue has recently been fixed in the Arduino code by using
xt_rsil() / xt_wsr_ps() but still exists in the Espressif SDK code.
This commit is intended to fix that and should be used in addition to the above.
The maximum nesting I have seen is 2 and lock/unlock calls appear to be balanced.
A max of 7 levels of nesting leaves plenty of room for that to change.
* make ets_intr_lock_stack uint16_t and behave like the original on over/underflow
The PS register is 15 bits, we should store the whole thing as xt_wsr_ps()
writes the whole thing.
Also if there is an underflow, we should make sure interrupts are enabled.
Same goes for overflow making sure interrupts are disabled, although this
is less important.
* Rename ets_intr_(un)lock_nest to ets_intr_(un)lock
This saves having to modify libmain.a, libpp.a and libnet80211.a to use the
nested versions.
Adjusts fix_sdk_libs.sh accordingly.
* Remove ets_intr_(un)lock from the rom .ld as we no longer use them
* ets_post() wrapper to preserve interrupt state
Add a wrapper around the ets_post code in rom to preserve the interrupt enable state.
Rather than modifying the SDK libs, rename ets_post in the .ld file and call the
wrapper "ets_post" to replace it.
As far as I can establish, ets_post is the only rom function in use by our code or
the SDK libs we use that causes calls to ets_intr_(un)lock.
* Add IRAM_ATTR to ets_intr_(un)lock and ets_post wrappers.
* Throw in a few comments and make ets_intr_lock_stack* static.
Build a single INO under a Windows VM on Travis to ensure
that the Win32 toolchain works properly.
In build.py, instead of making a string with spaces and then
splitting on " ", just make the list itself with individual parameters.
This will allow spaces in paths to be supported properly.
* mDNS debug option + AP address is used by default when STA is also present
* mDNS: store network interface, checking it is up
* igmp: force on selected interface (avoid crash *sometimes*)
* fix for all lwip2 ipv4 ipv6 & lwip1
* mdns: IPAddress is not needed to reference associated interface
* mdns: debug: fix print warnings
* emulation: add ets_strncpy
* emulation: truly emulate AddrList (remove fake one)
* Move all scripts and documentation to Python3
Python 2 EOL is Jan 1, 2020. Migrate scripts to run under Python 3.
Under Windows, we're already running Python 3.7, by dumb luck. The
oddness is that the Windows standalone executable for Python 3 is called
"python" whereas under UNIX-like OSes it's called "python3" with
"python" always referring to the Python 2 executable. The ZIP needs to
be updated to include a Python3.exe (copy of Python.exe) so that we can
use the same command lines under Linux and Windows, and to preserve my
sanity.
Fixes#6376
* Add new Windows ZIP with python3.exe file
* Sort options in boards.txt generation for repeatability
The order of the board opts dict changes depending on the Python version
and machine, so sort the options before printing them to get a stable
ordering.
* Re-add Python2 compatibility tweaks
Most scripts can run as Python 2 or Python 3 with minimal changes, so
re-add (and fix, as necessary) compatibility tweaks to the scripts.
Dropped routines form libc.a which are present and usable in ROM:
`ar dv libc.a lib_a-strcmp.o lib_a-strlen.o lib_a-strncmp.o lib_a-strstr.o lib_a-memcmp.o lib_a-memcpy.o lib_a-memmove.o lib_a-memset.o lib_a-bzero.o`
Left strcpy and strncpy in libc.a because they silently support PROGMEM
accesses as required by GCC optimizations.
Saves ~628 bytes in AdvancedWebServer example, from IROM (not IRAM).
Also allows mem* and str* routines to be safely called from inside an
ISR. Prior to this PR, these routines were stored in flash and not
IRAM, so they were technically illegal to call while in an ISR.
Fixes#6430
* Correct critical section with interrupt level preserving and nest support
alternative. Replace ets_intr_lock()/ets_intr_unlock() with uint32_t
oldValue=xt_rsil(3)/xt_wrs(oldValue). Added UMM_CRITICAL_DECL macro to define
storage for current state. Expanded UMM_CRITICAL_... to use unique
identifiers. This helpt facilitate gather function specific timing
information.
Replace printf with something that is ROM or IRAM based so that a printf
that occurs during an ISR malloc/new does not cause a crash. To avoid any
reentry issue it should also avoid doing malloc lib calls.
Refactor realloc to avoid memcpy/memmove while in critical section. This is
only effective when realloc is called with interrupts enabled. The copy
process alone can take over 10us (when copying more than ~498 bytes with a
80MHz CPU clock). It would be good practice for an ISR to avoid realloc.
Note, while doing this might initially sound scary, this appears to be very
stable. It ran on my troublesome sketch for over 3 weeks until I got back from
vacation and flashed an update. Troublesome sketch - runs ESPAsyncTCP, with
modified fauxmo emulation for 10 devices. It receives lost of Network traffic
related to uPnP scans, which includes lots of TCP connects disconnects RSTs
related to uPnP discovery.
I have clocked umm_info critical lock time taking as much as 180us. A common
use for the umm_info call is to get the free heap result. It is common
to try and closely monitor free heap as a method to detect memory leaks.
This may result in frequent calls to umm_info. There has not been a clear
test case that shows an issue yet; however, I and others think they are or
have had crashes related to this.
I have added code that adjusts the running free heap number from _umm_malloc,
_umm_realloc, and _umm_free. Removing the need to do a long interrupts
disabled calculation via _umm_info.
Build optional, min/max time measurements for locks held while in info,
malloc, realloc, and free. Also, maintain a count of how many times each is
called with INTLEVEL set.
* Fixed. travis build complaint.
* Changes for https://github.com/esp8266/Arduino/pull/6274#pullrequestreview-259579883
* Added requested comment and missing comment for UMM_CRITICAL_PERIOD_ANALYZE.
* Updated comments and update xt_rsil()
* Moved xt_rsil&co (pulled in __STRINGIFY) definitions out of
Arduino.h, to cores/esp8266/core_esp8266_features.h
Added esp_get_cycle_count() to core_esp8266_features.h.
Updated umm_malloc and Esp.h to use new defines and location.
* Added "#ifndef CORE_MOCK" around conflicted area.
* Moved performance measurment and ESP specific definitions to
umm_performance.h/cpp. Removed testing asserts.
* Commented out umm analyze. Delay CRITICAL_SECTION_EXIT() in
umm_realloc() to avoid exposing a transient OOM condition to ISR.
* Missed file change. This commit has: Delay CRITICAL_SECTION_EXIT() in
umm_realloc() to avoid exposing a transient OOM condition to ISR.
* 2nd Path. Removed early release of critical section around memmove
to avoid a possible OOM for an ISR.
* improved variable name
* Resolved ISR OOM concern with `_umm_realloc()`
Updated realloc() to do a preliminary free() of unused space,
before performing a critical section exit and memmove.
This change was applied to the current _umm_realloc().
This change should reduce the risk of an ISR getting an
OOM, during a realloc memmove operation.
Added additional stats for verifying correct operation.
* Resolved ISR OOM concern in _umm_realloc()
Updated realloc() to do a preliminary free() of unused space,
before performing a critical section exit and memmove.
This change was applied to the current _umm_realloc().
This change should reduce the risk of an ISR getting an
OOM when interrupting an active realloc memmove operation.
Added additional stats for verifying correct operation.
Updated: for clarity and Travis-CI fail.
* Update to keep access to alternate printf in one file.
* Updated to use ISR safe versions of memmove, memcpy, and memset.
The library versions of memmove, memcpy, and memset were in flash.
Updated to use ROM functions ets_memmove, ets_memcpy, and ets_memset.
Additional note, the library version of memmove does not appear to
have been optimized. It took almost 10x longer than the ROM version.
Renamed printf macro to DBGLOG_FUNCTION and moved to umm_malloc_cfg.h.
Changed printf macro usage to use DBGLOG_FUNCTION.
* Update umm_malloc.cpp
Fix comment
* Upgrade to 2.5.0-4 toolchain w/improved pgm_read_x
Rebuild the entire toolchain (including standard libraries) with the
latest pgm_read_xxx headers included (which fix unaligned dword reads
from progmem and run faster/smaller, and a pgm_read_byte change which
removes an instruction on each read saving flash).
Pull in latest bearssl while we're at it, too, which speeds up EC
handshakes and reduced ROM usage, too.
* Fix C++ exceptions
Exception code now only does 32b aligned reads from progmem to access
the eh_table (some via -mforce-l32, some via hand-inserted pgm_read_x
macros).
Fixes#6151Fixes#6305Fixes#6198
Cleans up all warnings seen w/GCC 9.1 to allow it to track the main
branch more easily until 3.x.
Does not include Ticker.h "fix" of pragmas around a function cast we're
doing that GCC9 doesn't like, that will be addressed separately and
maybe only in the 3.0 branch.
Does not include GDB hook fix, either, because the pragmas required
to disable the GCC9.1 warnings don't exist in 4.8 at all.
* enable by default latest 2.2.x firmware, including fixed espnow
* LittleFS: avoid crash when FS size is 0
* flash size defaults: 1M for generic board, not empty FS for all
* Make SPIFFS be an integer number of blocks
boards.txt.py simply calculated the end and start using flash sizes, but
in cases where an 8K page was used (>512KB SPIFFS), this could leave a
4K half-block left at the end of SPIFFS.
mkspiffs and the SPIFFS code uses integer division to calculate the
maximum block number, so it worked fine in practice and the code simply
ignored the extra, fractional block.
Now actually take block size into account when calculating the end of
SPIFFS, ensuring no fractional blocks are passed in. Does not result in
data loss on pre-existing SPIFFS filesystems.
* Fix the 1m512 case and clean up code
Ensure that no SPIFFS_block in the LD files is modified from the
original to endure correct backwards compatibility
* Factor out common if, clean code
* Make boards.py vars "fs_xx" instead of "spiffs_xx"
Apply most compatible changes needed to get the core compiling under GCC
7.2 to the main gcc 4.8 tree to ease porting for 3.0.0.
Update pgmspace.h with corrected and optimized unaligned pgm_read
macros. Now pgm_read_dword in the unaligned case gives proper results
even if optimization is enabled and is also written in assembly and only
1 instruction longer than the pgm_read_byte macro (which also has been
optimized to reduce 1 instruction). These changes should marginally
shrink code and speed up flash reads accordingly.
The toolchain should/will be rebuilt at a later time with this
optimization to ensure it's used in the libc.a/etc. files.
Undo the BearSSL RODATA->PROGMEM changes because there are some bad
performance regressions in EC server operations which can result in
timeouts and WDTs.
Keep the shrunked bearssl.a library as that is orthogonal to the PROGMEM
changes.
Rewrite all the integer math operations with const input parameters to
use PROGMEM properly (pgm_read_xx or memcpy_P), and move all the EC
order and generators and SHA OIDs to PROGMEM.
This frees around 1.2KB of heap for any SSL applications.
Also delete unneeded objects from the bearssl.a library to shrink the
GIT repo size.
As found by @mhightower83, umm_malloc was placed in flash during the
.c->.cpp conversion because of a missed linker change.
Adjust the link script to the new name .cpp
Move additional constants to flash and use _P/pgm_read routines to
access them. Minimal runtime impact, but remove variables from RODATA
and gives addition 484 bytes of heap to SSL applications.
Fixes#6005
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* add regular scheduled functions, now also callable on `yield()`
added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.
Legacy schedule_function() is preserved
This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
(callable from yield())
- without the need to call the driver handling function
(transparent)
This may be also applicable with common libraries (mDNS, Webserver, )
The interrupt vectors in IRAM are omitted when there is a PROVIDE
statement in the linker control files when using the PIO method of
-Wl,-T<linkfile>.
Drop the PROVIDES (they're in RAM anyway and not ROM related), and
add the required "-u"s to the PIO build script.
Should have no impact on the Arduino side.
Fixes#6087
* Upgrade to https: serving for JSON, links in docs
Fixes#5480
* Update boards.rst documentation
* Update more documentation http: refs to https:
* Remove obsolete staging info
* Drop obsolete versions from JSON programatically
After the final merge is done on the JSON, strip out any named versions
from the final product.
Removing 1.6.5-* and 2.5.0-beta(1,2,3) for now.
* Remove 2.4.0-rc(0/1) from JSON, too
Fixes#5996
* Add extensions to probe message for EC, others
probeMFLN was failing on some connection attempts to servers which only
supported EC based ciphers because it did not include the proper TLS
handshake extensions to list what kinds of ECs it supported.
Add those to the probeMFLN ClientHello message to make probes pass.
* Add client.getMFLNStatus method, returns MFLN state
After a connection it is useful to check whether MFLN negotiation
succeeded. getMFLNStatus returns a bool (valid only after
client.connect() succeeds, of course) indicating whether the requested
buffer sizes were negotiated successfully.
* New menu option to minimize BSSL ROM with only RSA
Adds a menu option and define to limit BearSSL to older RSA connection
options. This saves ~45K program memory and can speed up connections
since EC, while more secure, is significantly slower on the chip.
The supported ciphers are identical to the ones that axTLS supported.
Fixes#6005
* Add default SSLFLAGS(blank) to platform.txt
* Fix unused variable warning
* Add clarifying comment to menu items
* Unaligned access support for pgm_read_word/dword
* Fix pgm_read_ptr_aligned() per #5735
* Allow users to use aligned-only via a #define
Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.
Default is still to allow unaligned accesses.
* Split IRAM into 2 linker sections to move std::fcn
Callbacks need to be placed in IRAM when being called from an IRQ (like
the SPISlave callbacks).
This can be done by hacking the std::functional header and making every
single specialization of the template into an IRAM section, which would
take a lot of space for no benefit in the majority of cases.
The alternate is to specify the single instantiation types/operators
required, but the problem is the flash segment matcher would match them
before the IRAM section was begun, and rules for them would just not be
applied.
Get around this by splitting the IRAM section definition into .text and
.text1. This is linker syntactic sugar and does not actually change the
on-chip layout. But it does allow us to put the exception vectors at
the required absolute addresses and add single functions to IRAM.
* Add .text1 segment to space used calculation in IDE
* All functional callers are now placed in IRAM
* Write out the .text1 segment to the BIN
The extra segment name needs to be placed into the output binary as
well, or else only the init code gets stored and none of the real app is
present. This leads to an infinite boot loop.
This change adds in the segment to the generated image.
In some cases the printf implememtation would call an internal puts()
implementation which did not use pgm_read_byte() to access the format
string. In many operating modes this would work, but in interrupts or
when flash was disabled you'd get crashes.
Updated newlib to use pgm_read_byte in that one spot and recompiled.
