1
0
mirror of https://github.com/esp8266/Arduino.git synced 2025-04-19 23:22:16 +03:00

65 Commits

Author SHA1 Message Date
Earle F. Philhower, III
0920daf251
Add some more CI tests for String::replace (#6193) 2019-06-10 12:51:43 -07:00
Earle F. Philhower, III
78a1a66e6d
Make SSO support \0s, use memmove, add test (#6155)
Supercedes #6027

Make SSO more generic by keeping track of its length explicitly,
allowing for embedded \0s to exist in the String (just like the non-SSO
ones).

Use memmove/memcpy_P when we know the length of a string to save CPU
time.

Add tests to inject \0s in a String to ensure it is still working as
designed.
2019-06-05 08:11:53 -07:00
Earle F. Philhower, III
a389a995fb Add LittleFS as an optional filesystem, API compatible w/SPIFFS (but not on-flash-format compatible) (#5511)
* Add LittleFS as internal flash filesystem

Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.

LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.

LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling.  This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.

Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.

**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ .  The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.

The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.

Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next().  On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).

On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs.  So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements.  The returned filenames also only have the
filename returned, not full paths.  So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS.  This is consistent with
POSIX ideas about reading directories and more natural for a FS.

Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.

Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.

Subdirectories are made silently when they do not exist when you
try and create a file in a subdir.  They are silently removed when
the last file in them is deleted.  This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior.  Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.

SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.

* Fix up merge blank line issue

* Merge in the FSConfig changs from SDFS PR

Enable setConfig for LittleFS as well plys merge the SPIFFS changes
done in the SDFS PR.

* Fix merge errors

* Update to use v2-alpha branch

The V2-alpha branch supports small file optimizations which can help
increase the utilization of flash when small files are prevalent.
It also adds support for metadata, which means we can start adding
things like file creation times, if desired (not yet).

* V2 of littlefs is now in upstream/master

* Update test to support non-creation-ordered files

In a directory, the order in which "readNextFile()" will return a name
is undefined.  SPIFFS may return it in order, but LittleFS does not as
of V2.  Update the test to look for files by name when doing
readNextFile() testing.

* Fix LittleFS.truncate implementation

* Fix SDFS tests

SDFS, SPIFFS, and LittleFS now all share the same common set of tests,
greatly increasing the SDFS test coverage.

* Update to point to mklittlefs v2

Upgrade mklittlefs to V2 format support

* Remove extra FS::write(const char *s) method

This was removed in #5861 and erroneously re-introduced here.

* Minimize spurious differences from master

* Dramatically reduce memory usage

Reduce the program and read chunk sizes which impacts performance
minimally but reduces per-file RAM usage of 16KB to <1KB.

* Add @d-a-v's host emulation for LittleFS

* Fix SW Serial library version

* Fix free space reporting

Thanks to @TD-er for discovering the issue

* Update littlefs to latest upstream

* Remove sdfat version included by accident

* Update SDFAT to include MOCK changes required

* Update to include SD.h test of file append
2019-05-25 09:53:24 +02:00
david gauchard
b55199227b
add regular scheduled functions, now also callable on yield() (#6039)
* add regular scheduled functions, now also callable on `yield()`

added bool schedule_function_us(std::function<bool(void)> fn, uint32_t repeat_us)
lambda must return true to be not removed from the schedule function list
if repeat_us is 0, then the function is called only once.

Legacy schedule_function() is preserved

This addition allows network drivers like ethernet chips on lwIP to be regularly called
- even if some user code loops on receiving data without getting out from main loop
  (callable from yield())
- without the need to call the driver handling function
  (transparent)

This may be also applicable with common libraries (mDNS, Webserver, )
2019-05-23 22:03:53 +02:00
Earle F. Philhower, III
82adc95d64
Add explicit Print::write(char) (#6101)
W/o this change, file::write('a'); tries to use the template and fails
since char is a basic type.

The reason it is needed is due to pre 2.5.x behavior, File::write(char)
silently was cast to File::write(uint8_t).  With the template write,
though, this is not performed.

* Add Print::write tests and add'l overrides

Ensure that print::write does something sane and doesn't cause a compile
time error about templates when used for Files.

Test using SPIFFS file since Print is an abstract type.
2019-05-18 14:07:13 -07:00
Earle F. Philhower, III
968d6fcb0a
Make FILE_WRITE append in SD.h wrapper (#6106)
* Make FILE_WRITE append in SD.h wrapper

Fixes #6105

* Update host tests
2019-05-16 17:12:09 -07:00
Harald
a67aa569da Test for StreamString SSO bug (#6035) (#6041) 2019-05-03 07:09:32 -07:00
Earle F. Philhower, III
72c21feff6
Make chars unsigned by default, matching ESP8266 (#6013)
Force GCC to run with -funsigned-char during host tests  to make
the PC match the default behaviour used by the xtensa GCC port.

As noted in #6010.  Thanks @MichaelBrunn3r
2019-04-24 08:44:36 -07:00
david gauchard
2c36cfe04d
mock on host: compile without rtti (no dynamic_cast<>(), like on native) (#5994)
* mock on host: compile without rtti (no dynamic_cast<>(), like on native)
2019-04-17 18:27:30 +02:00
Earle F. Philhower, III
885276e75c
Unaligned access support for pgm_read_word/dword (#5692)
* Unaligned access support for pgm_read_word/dword

* Fix pgm_read_ptr_aligned() per #5735

* Allow users to use aligned-only via a #define

Adding -DPGM_READ_UNALIGNED=0 or #define PGM_READ_UNALIGNED 0 will
change the default at compile-time to only aligned (faster, but less
compatible) macro implementations.

Default is still to allow unaligned accesses.
2019-04-11 22:24:49 +03:00
Earle F. Philhower, III
9712170276
Fix String.replace overlapping strcpy (#5966)
* Fix String.replace overlapping strcpy

Fixes #5949

Adds a test from the issue above and fixes the problem valgrind found.

Additional pathological memcpy->memmove fixes
2019-04-10 17:21:15 +03:00
david gauchard
9a2ed274f3 polledTimeout: add option to use CPU count instead of millis() (#5870)
* polledTimeout: add option to use CPU count instead of millis()

* use more "using" alias

* more c++/clear code, using typename (thanks @devyte)

* rename class name to include unit, introduce timeMax() and check it with assert()

* remove useless defines

* improve api readability, add micro-second unit

* update example

* mock: emulate getCycleCount, add/fix polledTimeout CI test

* + nano-seconds, assert -> message, comments, host test

* allow 0 for timeout (enables immediate timeout, fix division by 0)

* typo, set member instead of local variable

* unify error message

* slight change on checkExpired() allows "never expired"
also removed printed message, add YieldAndDelay, simplify calculations

* remove traces of debug.h/cpp in this PR

* include missing <limits> header

* back to original expired test, introduce boolean _neverExpires, fix reset(), getTimeout() is invalid

* fix expiredOneShot with _timeout==0 check

* reenable getTimeout()

* expose checkExpired with unit conversion

* fix timing comments, move critical code to iram

* add member ::neverExpires and use it where relevant

* improve clarity

* remove exposed checkExpired(), adapt LEAmDNS with equivalent

* add API ::resetToNeverExpires(), use it in LEAmDNS

* remove offending constness from ::flagged() LEAmDNS (due do API fix in PolledTimeout)

* simplify "Fast" base classes

* minor variable rename

* Fix examples

* compliance with good c++ manners

* minor changes for consistency

* add missing const

* expired() and bool() moved to iram

* constexpr compensation computing

* add/update comments

* move neverExpires and alwaysExpired
2019-04-05 10:50:53 -03:00
Earle F. Philhower, III
54240d2cc5
Fix String::replace() garbage at end of string (#5897)
* Fix String::replace() 

Fixes #5883 and supercedes #5890

The replace() function was using len() while in the middle of buffer
operations.  In SSO mode len() is not stored separately and is a call to
strlen(), which may not be legal if you're in the middle of overwriting
the SSO buffer, as was the case in ::replace when the replacement string
was longer than the find string.  This caused potential garbage at the
end of the string when accessed.  Instead, just cache the length in a
local while doing the operation.

Add in test cases from #5890 as well as some new ones that fail on the
unmodified core.

* Fix stack smashing error on 64b

When pointers are 8 bytes long, the size of a String is larger than 16
chars.  Increase the allocated array we're using in the test to avoid a
"stack smashing" error.

* Manually call destructor in test

Just for clarity, manually call the destructor for the Strings() that
are "placement new'd" in the String tests.  It is a no-op for the
existing test, since thanks to SSO there are no memory allocations, but
will help in case someone adds tests later which include longer strings.
2019-03-20 06:18:04 -07:00
david gauchard
a3ea816e65
add optional espressif fw nonos-sdk 2.2.x (19.03.13) (#5873) 2019-03-14 13:35:26 +01:00
Earle F. Philhower, III
02f54e85fd
Reimplement SD.h write methods exactly in File (#5861)
* Reimplement SD.h write methods exactly in File

Replace the individual override with the existing SD.h File's
implementation for all methods of File::write.

Fixes #5846

* Add add'l tests

* Fix Print and File incompatible writes w/casts

Print and File have ambiguous resolutions for single-argument
write(0)s.

Fix by adding explicit methods.  A write of any integer will not be a
const char* write (i.e. won't write a string) but will instead just
write the integer truncated to 8 bits (as makes sense).

* Use 256byte chunks in ::write template

Reduce stack requirements for templated writes to 256bytes, matching the
size uses in WiFiClient/etc. (from 512bytes).  Reduces the chance of
stack overflow.

* Move write(int) methods up to Print.h

Remove some technical debt by moving the ::write(int/short/long) methods
out of FS and HardwareSerial and up into Print.h.
2019-03-12 13:58:00 -07:00
Earle F. Philhower, III
b1da9eda46
SD Filesystem compatible with 8266 File, using latest SdFat (#5525)
* Add a FAT filesystem for SD cards to Arduino FS

Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.

The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.

This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.

This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.

By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.

Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.

Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).

Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other.  Add a simple
bool FS::isDirectory() and bool FS::isFile() method.  SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.

Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories.  SPIFFS directory operations will, of course, fail
and return false.

Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.

Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls.  This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.

The only major visible change for users is that long filenames now are
fully supported and work without any code changes.  If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.

While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.

Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.

The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.

* Add a FSConfig and SDFSConfig param to FS.begin()

Allows for configuration values to be passed into a filesystem via the
begin method.  By default, a FS will receive a nullptr and should so
whatever is appropriate.

The base FSConfig class has one parameter, _autoFormat, set by the
default constructor to true.

For SPIFFS, you can now disable auto formatting on mount failure by
passing in a FSConfig(false) object.

For SDFS a SDFSConfig parameter can be passed into config specifying the
chip select and SPI configuration.  If nothing is passed in, the begin
will fail since there are no safe default values here.

* Add FS::setConfig to set FS-specific options

Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which
takes a FS-specific configuration object and copies any special settings
on a per-FS basis.  The call is only valid on unmounted filesystems, and
checks the type of object passed in matches the FS being configured.

Updates the docs and tests to utilize this new configuration method.

* Add ::truncate to File interface

Fixes #3846

* Use polledTimeout for formatting yields, cleanup

Use the new polledTimeout class to ensure a yield every 5ms while
formatting.

Add in default case handling and some debug messages when invalid inputs
specified.

* Make setConfig take const& ref, cleaner code

setConfig now can take a parameter defined directly in the call by using
a const &ref to it, leading to one less line of code to write and
cleaner reading of the code.

Also clean up SDFS implementation pointer definition.
2019-03-06 02:14:44 +00:00
david gauchard
61a8a6b14e
emulation on host: fix nasty delay (#5840)
emulation with socket: fix nasty delay
        properly check all return values (read() returning 0 indicates closed peer)
        ClientContext::state() triggers read() to check for closed peer

uart emulation on tty: add cmdline -b debug option to disable raw tty

hide annoying MOCK messages, add cmdline -v option to show them

uart emulation on tty: check more return values

emulation on host: tcp/udp port shifting: new cmdline -s(port Shift) option
2019-03-05 16:59:39 +01:00
Earle F. Philhower, III
82da5c76b9 Fix minor warning w/Ubuntu18.04 in host test build (#5829)
GCC reports the following warning about ignoring the return value of
write() (to the console/UART in this instance).

common/MockUART.cpp: In function ‘void uart_do_write_char(int, char)’:
common/MockUART.cpp:67:8: warning: ignoring return value of ‘ssize_t write(int, const void*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
   write(uart_nr + 1, &c, 1);
   ~~~~~^~~~~~~~~~~~~~~~~~~~

Add a simple warning printout to STDERR which may not be the same as the
UART stream and could succeed.
2019-03-02 14:37:03 -03:00
Clemens Kirchgatterer
21926a69b6 emulation on host: fix wrong tty_restore instead of restore_tty variable (#5809) 2019-02-23 11:30:44 +01:00
Clemens Kirchgatterer
ef44211eea emulation on host: Add full UART driver emulation. (#5785)
* emulation on host: Add full UART driver emulation.

  This PR replaces the high level Serial mock with a more complete
  UART driver. This way the HardwareSerial works without any
  modifications. Additionally the driver supports UART0 and UART1
  at the same time (UART0 is directed to stdout and UART1 writes
  to stderr). RX is implemented by switching the terminal into raw
  non-blocking mode and injecting each key-press directly into the
  FIFO of UART0. A new command line switch -c was added to ignore
  CTRL-C and send it via serial as well. The decumentation was
  updated accordingly.

  Reading and setting of GPIOs does only write to stderr, when compiled
  with D=1. But this is subject to be replaced with a proper GPIO emu
  anyway. Reading from GPIO0 now returns 1 instead of 0 because this is
  most likely a low active input.

* Fixed unused variable.

* Remove unused functions, as long as there are no debug macros using them.

* Move user_interface.cc from MOCK_CPP_FILES_EMU to MOCK_CPP_FILES.

* Move optimistic_yield() from user_interface.cpp to Arduino.cpp

* Remove atexit() weak declaration (fixes segfault when calling atexit).

* Improve resetting of terminal after program exit, convert \r to \r\n.

* Show error, when STDOUT is not a TTY, minor code cleanung.
2019-02-22 16:50:55 +01:00
david gauchard
f08346a73e
Fix host emulation (sdk changes) (#5802) 2019-02-21 17:04:17 +01:00
david gauchard
7745e99046
Revert to nonos-sdk 2.2.1, new sdk-switching option in IDE menu for generic board only (#5763)
This commit allows switching SDK firmware:

nonos-sdk-pre-v3 shipped with release 2.5.0 has issues:

    * Some boards show erratic behavior (radio connection is quickly lost), with an unknown cause.
      These boards work well with previous nonos-sdk-2.2.1 firmware (#5736)

    * Overall performances seem to have decreased (#5513)

This PR restores sdk2.2.1 (as in core-2.4.2).

SDK-pre-3.0 - which has brought long awaited fixes (WiFi sleep modes) - is still available through a menu option available only with generic board.

BREAKING

    * new define `-DNONOSDK221=1` or `-DNONOSDK3V0=1`

    * for external build systems: new library directory: `tools/sdk/lib/<version>/lib`

    * PIO: variable `PIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK3` is needed for sdk-pre-v3.


Fix #5736
2019-02-19 13:10:49 +01:00
Clemens Kirchgatterer
b93b37fb81 emulation on host: Fix optimistic_yield(interval_us) (#5772)
optimistic_yield() takes micro seconds not milliseconds as interval parameter
2019-02-17 23:11:40 +01:00
Clemens Kirchgatterer
9f9c661d99 emulation on host: Make non essential ARDUINO LIBS optional. (#5771)
This patch splits ARDUINO_LIBS into two variables. The second variable is called OPT_ARDUINO_LIBS and can be overridden by the caller as it uses the ?= assignment operator. Additionally it unifies and simplifies collecting the files in common/ by using the addprefix macro. All changes should be 100% backwards compatible.
2019-02-16 22:50:27 +01:00
Clemens Kirchgatterer
2be374cc6c emulation on host: Add base64 decode library to mock files (#5764)
Fixes missing symbols during linking of sketches that use them.
2019-02-16 21:39:30 +01:00
Clemens Kirchgatterer
9ba89ebaf4 emulation on host: Add missing ESP.*() mocking methods to EspClass. (#5765) 2019-02-16 00:05:18 +01:00
Clemens Kirchgatterer
d3a2ab9bb4 emulation on host: Add (some) missing macros for *_P() functions. (#5766) 2019-02-15 23:00:08 +01:00
Earle F. Philhower, III
7369133681
Small String Optimization (#5690)
Reduce String memory overhead from 24 bytes to 16 bytes by limiting the
maximum string length to <64Kbytes (which is larger than heap so no
effective problem).

Add Small String Optimization, SSO, which instead of allocating pointers
to small strings on the heap will store the string in place of the
pointer in the class.  This should reduce memory fragmentation as

Save up to 12 chars (11 + \0) in String itself by using the terminating
\0 in the inline string as a flag to identify if this is a SSO or a heap
string.

Add a host test that verifies that no memory is allocated until a
full 11 characters are assigned to a string, as well as checking all
intermediate values.

No user code changes should be required to work with this optimization.
2019-02-08 17:37:43 +00:00
Earle F. Philhower, III
f706c83b66 Move .C to .CPP in the code (#5696)
Use g++ to compile core files to get additional C++ checks on the code.

Also move libb64 constants to PROGMEM, saving ~128 bytes of heap when used.
2019-02-06 23:06:17 -03:00
david gauchard
8412d11b02
host emulation fix (#5704) 2019-02-04 11:37:39 +01:00
david gauchard
a0108fe69b
emulation on host: mock lwIP functions used in latest dhcp renew patch (#5691) 2019-01-29 16:59:17 +01:00
david gauchard
f42bfdfc0d
adaptations for lwIP-v1.4 (#5682)
* adaptations for lwIP-v1.4
* add lwIP-v1.4 in CI
2019-01-28 22:31:59 +01:00
david gauchard
bd11d026e8
fix host emulation on macOS (#5633) 2019-01-19 03:19:17 +01:00
Earle F. Philhower, III
6883beedec
Remove broken ltoa/ultoa, call itoa/utoa (#5625)
* Remove broken ltoa/ultoa, call itoa/utoa

Use the newlib integer-to-ASCII non-POSIX calls instead of rolling
our own.  Should be safe as sizeof(long) == sizeof(int).
The custom functions behaved differently from itoa when passed in
negative values in non-base-10.

Add host tests for negative non-base-10 int/longs
2019-01-17 02:08:19 +00:00
david gauchard
6bd26a3b4a
host emulation: improve udp, persistent spiffs (#5605) 2019-01-15 22:56:54 +01:00
david gauchard
dc03293d82
(re)introduce timeout in HardwareSerial::readBytes(buffer, size) (#5558)
(re)introduce timeout in HardwareSerial::readBytes(buffer, size), add HardwareSerial::read(buffer, size) + visual test
2019-01-08 04:01:21 +01:00
david gauchard
eb5d636f0d
use newlib api in new mDNS, fix host using mDNS (#5545) 2018-12-27 22:06:10 +01:00
david gauchard
228ad7ed75 tests/host: fixes and updates (#5537)
(LEAmDNS, broken pipe, non blocking accepted sockets, digitalRead)
2018-12-22 03:03:11 -03:00
david gauchard
4c8d8f1e8a uart: BW improvements (#4620)
* uart fixes and BW improvements

* uart: read_char straightly use hw buffer

* +attributes for functions called by ISR

* uart: BW improvements
read_char straightly use hw buffer (+ ~10%bw)
read by block (+ ~190%bw) (instead of generic Stream::readBytes)
attributes for functions called by ISR
remove overrun message
remove some ISR flags which were not honoured

* fix merge

* fix buffer overflow

* serial stress test sketch

* astyle

* serial stress example: interactive keyboard, stop reading, overrun

* serial device test: bandwidth & overrun

* update + HardwareSerial::hasError()

* interactive overrun in example

* astyle

* Test using @plerup's SoftwareSerial as submodule (tag 3.4.1)

* update upstream ref (fix warning)

* host mock uart/read(buf,size)

* reset style changes in submodules before style diff

* update build_boards_manager_package.sh for submodules

* trigger CI (removing space)

* cannot reproduce locally the CI issue, setting bash -x option to get live trace

* remove previously added (in this PR) 'set -e' in package builder (passes local tests, not real CI)
script-comment new recipe.hooks.core.prebuild.3 (along with already commented .1 and .2)
moved CI package test to be first on the test list
remove 'set -x', wish me luck
2018-12-10 10:35:11 -03:00
david gauchard
667c27f864 wip host build (#5431) 2018-12-04 14:55:10 -03:00
Earle F. Philhower, III
6280e98b03 Enable exceptions, update to optimized newlib, migrate to new toolchain (#5376)
* Move to PROGMEM aware libc, allow PSTR in printf()

A Newlib (libc) patch is in progress to move the _P functions from inside
Arduino into first-class citizens in libc.  This Arduino patch cleans up
code that's been migrated there.  Binaries for the new libs are included
because it seems they're part of the Arduino git tree, and should be
replaced with @igrr built ones when/if the Newlib changes are accepted.

Notable changes/additions for Arduino:
Allow for use of PROGMEM based format and parameter strings in all
*printf functions.  No need for copying PSTR()s into RAM before printing
them out (transparently saves heap space when using _P functions) and
makes it easier to print out constant strings for applications.

Add "%S" (capital-S) format that I've been told, but cannot verify,
is used in Arduino to specify a PROGMEM string parameter in printfs,
as an alias for "%s" since plain "%s" can now handle PROGMEM.

Optimized the memcpy_P, strnlen_P, and strncpy_P functions to use 32-bit
direct reads whenver possible (source and dest alignment mediated), but
there is still room for improvement in others.

Finally, move several constant arrays from RODATA into PROGMEM and
update their accessors.  Among these are the ctype array, ~260 bytes,
mprec* arrays, ~300 bytes, and strings/daycounts in the time
formatting functions, ~200 bytes.  All told, sketches will see from
300 to 800 additional RAM heap free on startup (depending on their
use of these routines).

* Fix merge error in #ifdef/#endif

* Fix host test using the newlib generic pgmspace.h

Host tests now use the sys/pgmspace.h for compiles instead of the
ESP8266-specific version.

* Update with rebuilt libraries using latest newlib

* Include binaries built directly from @igrr repo

Rebuild the binaries using a git clone of
https://github.com/igrr/newlib-xtensa

Build commands for posterity:
````
rm -rf ./xtensa-lx106-elf/
./configure --prefix=<DIR>/esp8266/tools/sdk/libc --with-newlib \
            --enable-multilib --disable-newlib-io-c99-formats \
            --disable-newlib-supplied-syscalls \
            --enable-newlib-nano-formatted-io --enable-newlib-reent-small \
            --enable-target-optspace \
            --program-transform-name="s&^&xtensa-lx106-elf-&" \
            --disable-option-checking --with-target-subdir=xtensa-lx106-elf \
            --target=xtensa-lx106-elf
rm -f etc/config.cache
CROSS_CFLAGS="-fno-omit-frame-pointer -DSIGNAL_PROVIDED -DABORT_PROVIDED"\
             " -DMALLOC_PROVIDED" \
  PATH=<DIR>/esp8266/tools/xtensa-lx106-elf/bin/:$PATH \
  make all install
````

* Fix merge define conflict in c_types.h

* Fix strlen_P misaligned source error

Include fix from newlib-xtensa/fix-strlen branch cleaning up misaligned
access on a non-aligned source string.

* Fix strlen_P and strcpy_P edge cases

Ran the included test suite on ESP8266 tstring.c with the following defines:
 #define MAX_1 50
 #define memcmp memcmp_P
 #define memcpy memcpy_P
 #define memmem memmem_P
 #define memchr memchr_P
 #define strcat strcat_P
 #define strncat strncat_P
 #define strcpy strcpy_P
 #define strlen strlen_P
 #define strnlen strnlen_P
 #define strcmp strcmp_P
 #define strncmp strncmp_P

Uncovered edge case and return value problems in the optimized versions of
the strnlen_P and strncpy_P functions.  Corrected.

* Fix memcpy_P return value

memcpy-1.c test suite showed error in return value of memcpy_P.  Correct it.

* Fix strnlen_P/strlen_P off-by-4 error

Random crashes, often on String constructors using a PSTR, would occur due
to the accelerated strnlen_P going past the end of the string. Would make
debug builds fail, too (ESP.getVersionString() failure).

Fix to fall through to normal copy on a word that's got a 0 byte anywhere
in it.

* Add device tests for libc functional verification

Add test suite used to debug libc optimized _P functions to the device
tests.

* Rebuild from igrr's repo (same source as prior)

Rebuild .a from igrr's repo at 347260af117b4177389e69fd4d04169b11d87a97

* WIP - add exceptions

* Fix exception to have 0-terminator

* Move some exception constants to TEXT from RODATA

* Remove throw stubs

* Move more exception stuff to ROM

* Enable exceptions in platform.io

* Remove atexit, is duplicated in rebuilt lib

Need to look at the quick-toolchain options, there seems to be a definition
for atexit defined there (libgcc?) that needs to be excised.  For now,
remove our local do-nothing copy.

* Update libgcc to remove soft-fp functions

The esp-quick-toolchain generated libgcc.a needed to have the soft-FP routines
that are in ROM removed from it.  Remove them in the new esp-quick-toolchain
and update.

* Fix merge typos in Makefile

* Add unhandled exception handler to postmortem

* Return our atexit() handler

* Latest stdc++, minimize exception emercengy area

* Remove atexit from newlib

atexit was defined in newlib strongly, but we also define a noop atexit in core.
Since we never exit, use the core's noop and delete the atexit from libc.a

Updated in esp-quick-toolchain as well.

* Move __FUNCTION__ static strings to PROGMEM

__FUNCTION__ is unlikely to be a timing sensitive variable, so move it to
PROGMEM and not RODATA (RAM) using linker magic.

asserts() now should take no RAM for any strings.

* Clean up linker file, update to latest stdc++

* Update to latest stdc++ which doesn't call strerror

* Update to GCC5.1 exception emergency allocator

Using GCC 5.1's emergency memory allocator for exceptions, much less
space is required in programs which do not use exceptions and when
space is allocated it is managed more efficiently.

* Initial try with new compiler toolchain

* Include newlib built from esp-quick-toolchain

* Update JSON with all new esp-quick-toolchain builds

* Use 64bit Windows compiler on 64bit Windows

* Dump std::exception.what() when possible

When doing the panic on unhandled exceptions, try and grab the
.what() pointer and dump it as part of the termination info.
Makes it easy to see mem errors (std::bad_alloc) or std::runtime_error
strings.

* Use scripted install from esp-quick-toolchain

Makes sure proper libraries and includes are present by using a
scripted installation from esp-quick-install instead of a manual
one.

* Update eqk to remove atexit, fix packaging diff
2018-12-03 03:37:14 -03:00
david gauchard
116da1881c update HTTPCLIENT_1_1_COMPATIBLE (#5389)
* update HTTPCLIENT_1_1_COMPATIBLE: #if instead if #ifdef, set to 1 by default
host emulation updates

* host CI: minor simplification

* revert -j
2018-11-29 10:42:44 -08:00
david gauchard
dc5e352676
fix host emulation (#5382) 2018-11-28 12:37:59 +01:00
Develo
a501d3ca3b
PolledTimeout Class for wrapping millis() loops (WIP) (#5198)
* PolledTimeout Class for wrapping millis() loops

* Add yield policies, improve reset, add host tests

* Fix copyright, comments

* adjust host tests for better time precision

* add fuzzyness to timing tests for CI jitter

* add blink example with polledTimeout

* improve namespace and type naming, add copyright, comments

* fix astyle
2018-11-26 10:57:49 -03:00
david gauchard
74ca42f829 Sketch emulation on host (#5342)
* WIP compile examples on host with 'make examples'

* WIP bufferize tcp input

* WIP Makefile

* WIP network to rework, tcp/udp to factorize, udp addresses broken

* minor changes to the core

* WIP basic udp working

* WIP mdns

* WIP mcast receiving, not sending

* WIP mdns OK

* beta version

* SSL + doc

* update travis host test command

* licenses

* typo

* doc: arduino builder is not around: declare functions before calling them

* fix with latest SSL PR, compile in 32 bits mode

* fix make clean

* make -m32 optional

* 32bits compiler ability tester

* WIP

* WIP (fix 1 vtable error, still another one to hunt with using spiffs)

* example astyle

* fix os_printf_plus

* load / save mock spiffs

* fix style

* fix using spiffs/mock

* don't mess ram

* update doc

* remove leftover

* optimization -Os except for CI, rename ARCH32 to FORCE32

* revert useless cast (not even compiled)

* remove unused function

* use proper type for pointer arithmetics

* makefile: sketch object and cpp file moved to bin/ directories
easier to clean, and IDE don't like them

* changes for review

* make use of %zd

* less verbose makefile by default (option)

* update readme
2018-11-20 18:51:45 -02:00
Earle F. Philhower, III
03ea61a32a Add valgrind and string tests to host_tests (#4939)
* Add valgrind and string tests to host_tests

Valgrind identified an error that was causing intermittent failures in
the host tests, and can check for subtle memory access and allocation bugs.
Add it to the standard host_test suite, and any errors will cause test
failure.

Also start adding string tests, since two undefined behaviors have been
found so far just by inspection.

* Add additional String tests

Looks like a possible bug in the concatenation operator, need to verify
expected behavior.

* Remove verbose from valgrind run

No need to be so chatty on the test.  Errors were a little hard to spot.
Go to normal verbosity.

* Add lcov and more string tests

LCOV and genhtml can generate nice HTML coverage charts hilighting test
coverage issues in a much simpler way than gcov text format.  Generate these
automatically from gcov output.

Add additional string creation and comparison tests.

* Move String coverage to >50%

Additional string tests and checks

* 66% test coverage in String

* Add allocation-unit-sized strings test

Ensure that strings that are right on the edge of the allocation
size are handled properly and account for trailing 0.
2018-07-27 13:10:55 -04:00
Ivan Grokhotkov
2c8b2b92ce tests: minor fixes, add README.md 2018-01-09 15:59:08 +08:00
Sven Eliasson
2404a602af Const-correctness for MD5Builder (#3222)
Resolves #1175
2017-05-12 01:03:14 -05:00
Ivan Grokhotkov
6d3109e8c7 Remove implementations of non-ISO libc functions which are present in newlib 2016-06-27 21:10:41 +08:00
Ivan Grokhotkov
b9dfe01903 Fix SPIFFS.openDir("") (#2143)
* Update spiffs_api.cpp

Fixes a bug where un-prefixed files are irretrievable with openDir(""). Described: https://github.com/esp8266/Arduino/issues/1818.

* Update FS test cases
2016-06-14 07:15:55 +08:00