* Deprecate SPIFFS, move examples to LittleFS
SPIFFS has been a great filesystem, but it has significant problems in
many cases (and it's also pretty slow). Development seems to have
slowed/stopped on the upstream version, and we're not able to provide
support or fix the known issues with it as-is.
Deprecate SPIFFS variable.
Update all examples to use LittleFS instead of SPIFFS.
Also, minor cleanup on very old examples which has obsolete delays
waiting for the Serial port to come up, or which were stuck at 9600 baud
because of their ancient AVR heritage.
Fixes#7095
* Remove leftover debug code
* Clean up comments in some examples
* Update documentation on SPIFFS deprecation
* Fix host tests to avoid deprecation warnings
* Fix cut-n-paste error
* Restore SpeedTest.ino, adjust to allow custom FSes
Co-authored-by: Develo <deveyes@gmail.com>
* Add way to force follow redirections in `HTTPClient`
* Follow other client implementations about `HTTP_CODE_FOUND`; Small rewrite of `sendRequest` function of `HTTPClient`
* Better names for follow redirection modes in `HTTPClient`
Also changed a bit order of the enums (0 element to be DISABLED)
* Rewrite `sendRequest` to remove recursion
Also got rid of unnecessary `redirectCount` field. Now redirect counting and limiting is handled in `sendRequest` directly.
* Use new `setFollowRedirects` of `HTTPClient` instead deprecated one.
* More explanatory comment for `followRedirects_t` in HTTPClient
* Resolved issue #3359
Made severing connections optional as per the patch
in the issue.
Also fixed a minor spacing issue.
* Renamed sever to close and added information to readme
Also my editor automatically removed some odd whitespace at the
end of a few lines.
* Allow GZIP compressed flash updates
Modified the bootloader to be able to take stored updates in compressed
GZIP format (i.e. the output of "gzip -9 xxx.bin") and decompress them
on-the-fly to their final destination. This can work for apps and for
filesystems (when used with the 2-step update option).
Allow eboot to be built using -Os/2 optimizations by fixing some portions
which failed when any optimizations were used. Add -Wall and use data
and function sections to reduce size. Use -Os to minimize size.
Remove obsolete esptool-ck calls to build a .ROM image, we don't use it.
Move all uninitted variables to RAM from IRAM, allowing 8-bit access.
Hook in @d-a-v and @pfalcon's uzlib port to actually do the
decompression. Do not use any CRC checking which saves space. Since we
have overwritten all of flash by the time we know id the CRC matches,
there's nothing we could have done anyway.
Adjust the Updater class to support GZIP files and not attempt to patch
them.
Bootloader builds to 0xd90 out of 0xfff bytes.
* Add @d-a-v's patch for httpupdate
https://github.com/esp8266/Arduino/pull/6820#pullrequestreview-326541014
* Update uzlib to point to pfalcon++
For now, because there are some self-test failures with @d-a-v's esp8266
branch (whose cool new features we don't actually use in eboot now)
start with pfalcon's 2.9 release and add the 2 patches (clcidx to code
from IRAM/RODATA, and the Windows test file renaming) needed to build
and run successfully.
* Add (c) notice for uzlib to README
Replaces abandoned #1817 and #2694
Add optional std::function callback (so it supports lambdas and normal
functions) via ::onStart, ::onEnd, ::onProgress, and ::onError methods.
Update example with their use.
From @baruch's original pull request:
The callback is called when the upgrade actually starts rather than just
the initial query so that the user can know that it will not take longer
and can also prepare for the upgrade by shutting down other works.
From @karlp's original pull request:
Incomplete: I've not updated any documentation yet. If this style looks
good, I'll happily go and update the documentation (likewise for the
examples)
There are actually several instances where we pass in read-only
parameters as pass-by-value, where in the case of String() that
is inefficient as it involves copy-constructor/temp string creations.
We can avoid that, similarly to single character string concatenations
done via string literals instead of char literals.
__This is a breaking change, but the header and example did warn
everyone that this API was in flux due to the incompatible SD and SPIFFS
File implementations.__
BearSSL CertStores now simply need a filesystem and the names of the
data (generated on-chip) and archive (uploaded by user) files on it.
No more need to roll your own virtual CertStoreFile class.
Update the library, examples, and device test.
* Add LittleFS as internal flash filesystem
Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.
LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.
LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling. This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.
Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.
**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ . The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.
The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.
Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next(). On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).
On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs. So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements. The returned filenames also only have the
filename returned, not full paths. So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS. This is consistent with
POSIX ideas about reading directories and more natural for a FS.
Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.
Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.
Subdirectories are made silently when they do not exist when you
try and create a file in a subdir. They are silently removed when
the last file in them is deleted. This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior. Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.
SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.
* Fix up merge blank line issue
* Merge in the FSConfig changs from SDFS PR
Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.
* Fix merge errors
* Update to use v2-alpha branch
The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).
* V2 of littlefs is now in upstream/master
* Update test to support non-creation-ordered files
In a directory, the order in which "readNextFile()" will return a name
is undefined. SPIFFS may return it in order, but LittleFS does not as
of V2. Update the test to look for files by name when doing
readNextFile() testing.
* Fix LittleFS.truncate implementation
* Fix SDFS tests
SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.
* Update to point to mklittlefs v2
Upgrade mklittlefs to V2 format support
* Remove extra FS::write(const char *s) method
This was removed in #5861 and erroneously re-introduced here.
* Minimize spurious differences from master
* Dramatically reduce memory usage
Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.
* Add @d-a-v's host emulation for LittleFS
* Fix SW Serial library version
* Fix free space reporting
Thanks to @TD-er for discovering the issue
* Update littlefs to latest upstream
* Remove sdfat version included by accident
* Update SDFAT to include MOCK changes required
* Update to include SD.h test of file append
* fix DEBUG macros
All fmt strings in flash
fix#5658
This also allows to avoid warnings and easy mistakes with (no brace):
if (something)
DEBUGV("blah");
* use newlib unaligned-compatible printf for DEBUGV
* remove useless putprintf since ::printf already uses ets_putc
* optionally allow redirects on http OTA updates
* Refactored HTTPClient::begin(url...) & setURL functions, now only beginInternal parses URL, sets ports
Added HTTPRedirect example.
* fix indentation for style check
* add space after while for style check
* don't use deprecated begin method in redirect example
* moved redirect handling code to HTTPClient.
only GET and HEAD requests are currently handled automatically
Redirects that fail to be automatically handled return the redirect code as before
* added support for POST/303 redirect
added device redirect tests
* add missing getLocation() implementation
* if the new location is only a path then only update the URI
Using a pluggable architecture, allow updates delivered via the Update
class to be verified as signed by a certificate. By using plugins, avoid
pulling either axTLS or BearSSL into normal builds.
A signature is appended to a binary image, followed by the size of the
signature as a 32-bit int. The updater takes a verification function
and checks this signature using whatever method it chooses, and if it
fails the update is not applied.
A SHA256 hash class is presently implemented for the signing hash (since
MD5 is a busted algorithm).
A BearSSLPublicKey based verifier is implemented for RSA keys. The
application only needs the Public Key, while to sign you can use
OpenSSL and your private key (which should never leave your control
or be deployed on any endpoints).
An example using automatic signing is included.
Update the docs to show the signing steps and how to use it in the
automatic and manual modes.
Also remove one debugging line from the signing tool.
Saves ~600 bytes when in debug mode by moving strings to PMEM
Windows can't run the signing script, nor does it normally have OpenSSL
installed. When trying to build an automatically signed binary, warn
and don't run the python.
Make HTTPClient take a WiFiClient parameter, allowing you to pass in a
simple HTTP WiFiClient or a BearSSL or axTLS WiFiClientSecure with
any desired verification options. Deprecate the older, TLSTraits methods.
Add basic HttpsClient example.
Add optional LED feedback to the Update class
BearSSL (https://www.bearssl.org) is a TLS(SSL) library written by
Thomas Pornin that is optimized for lower-memory embedded systems
like the ESP8266. It supports a wide variety of modern ciphers and
is unique in that it doesn't perform any memory allocations during
operation (which is the unfortunate bane of the current axTLS).
BearSSL is also absolutely focused on security and by default performs
all its security checks on x.509 certificates during the connection
phase (but if you want to be insecure and dangerous, that's possible
too).
While it does support unidirectional SSL buffers, like axTLS,
as implemented the ESP8266 wrappers only support bidirectional
buffers. These bidirectional buffers avoid deadlocks in protocols
which don't have well separated receive and transmit periods.
This patch adds several classes which allow connecting to TLS servers
using this library in almost the same way as axTLS:
BearSSL::WiFiClientSecure - WiFiClient that supports TLS
BearSSL::WiFiServerSecure - WiFiServer supporting TLS and client certs
It also introduces objects for PEM/DER encoded keys and certificates:
BearSSLX509List - x.509 Certificate (list) for general use
BearSSLPrivateKey - RSA or EC private key
BearSSLPublicKey - RSA or EC public key (i.e. from a public website)
Finally, it adds a Certificate Authority store object which lets
BearSSL access a set of trusted CA certificates on SPIFFS to allow it
to verify the identity of any remote site on the Internet, without
requiring RAM except for the single matching certificate.
CertStoreSPIFFSBearSSL - Certificate store utility
Client certificates are supported for the BearSSL::WiFiClientSecure, and
what's more the BearSSL::WiFiServerSecure can also *require* remote clients
to have a trusted certificate signed by a specific CA (or yourself with
self-signing CAs).
Maximum Fragment Length Negotiation probing and usage are supported, but
be aware that most sites on the Internet don't support it yet. When
available, you can reduce the memory footprint of the SSL client or server
dramatically (i.e. down to 2-8KB vs. the ~22KB required for a full 16K
receive fragment and 512b send fragment). You can also manually set a
smaller fragment size and guarantee at your protocol level all data will
fit within it.
Examples are included to show the usage of these new features.
axTLS has been moved to its own namespace, "axtls". A default "using"
clause allows existing apps to run using axTLS without any changes.
The BearSSL::WiFi{client,server}Secure implements the axTLS
client/server API which lets many end user applications take advantage
of BearSSL with few or no changes.
The BearSSL static library used presently is stored at
https://github.com/earlephilhower/bearssl-esp8266 and can be built
using the standard ESP8266 toolchain.
* Fix minor typo in spelling of failed. Removed s from http paths.
* add check if we are just updating spiffs to not reboot. example now works with this change.
