Merge branch 'master' into bugfix/hardwareserial_swap

.gitignore

@@ -10,3 +10,4 @@ tools/sdk/lwip/src/build
 tools/sdk/lwip/src/liblwip_src.a
 
 *.pyc
+*.gch

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "lwip2"]
+	path = tools/sdk/lwip2/builder
+	url = https://github.com/d-a-v/esp82xx-nonos-linklayer.git

.travis.yml

@@ -3,8 +3,18 @@ language: bash
 os: linux
 dist: trusty
 
-env:
+matrix:
-  - BUILD_TYPE=build
+  include:
+    - env:
+      - BUILD_TYPE=build
+    - env:
+      - BUILD_TYPE=platformio
+    - env:
+      - BUILD_TYPE=docs
+    - env:
+      - BUILD_TYPE=package
+    - env:
+      - BUILD_TYPE=host_tests
 
 install:
   - pip install --user -r doc/requirements.txt
@@ -15,6 +25,7 @@ script:
 deploy:
   provider: releases
   prerelease: true
+  skip_cleanup: true
   api_key:
     secure: A4FBmqyhlzy33oPeZVolg2Q/A3ZcJ3WnRQqQJ3NAPy+qGM5xcboOYtwcLL9vKaHZGfUB7lUP9QVZFGou1Wrmo9DnPvAoe3+XvCaDRGzVMxeIpu7UStbBD4Knbh98tlbMvZCXYRlT4VcusI9bMLK6UWw4sMdPislBh2FEfglTiag=
   file_glob: true
@@ -24,6 +35,7 @@ deploy:
   on:
     repo: esp8266/Arduino
     tags: true
+    condition: "$BUILD_TYPE = package"
 
 notifications:
   email:

ISSUE_TEMPLATE.md

@@ -2,10 +2,10 @@
 Please fill the info fields, it helps to get you faster support ;)
 
 if you have a stack dump decode it:
-https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/stack_dump.md
+https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/stack_dump.rst
 
 for better debug messages:
-https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/debugging.md
+https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/debugging.rst
 
 ----------------------------- Remove above -----------------------------
 

POLICY.md

@@ -0,0 +1,67 @@
+This document describes rules that are in effect for this repository, meant for handling issues by contributors in the issue tracker and PRs.
+# Opening New Issues
+1.	The issue tracker is precisely that: a tool to track issues in the core code, and not a discussion forum. Opening an issue means that a problem has been found in the core code, and that it should be addressed by a contributor.
+2.	When opening an issue, a template is presented with fields to fill out. The requested information is important. If the template is ignored, or not enough info about the issue is provided, the issue may be closed due to lack of info. Example:
+      * Using WifiMulti and FS crashes with error. Why? (no basic info, no IDE settings, no sketch provided)
+3. Questions of type "How do I..." or "Can you please help me with..." or "Can the ESP do..." won't be handled here. Such questions should be directed at a discussion forum, like esp8266.com or stackoverflow. All issues of this type will be closed with a simple reference to the policy. Example:
+    * how do I connect to wifi
+    * how do I connect two ESPs
+    * can I send http data over a public network
+    * my wiring/project/code doesn't work, help!
+4. Issues that are obviously user error, programming language errors, lack of knowledge or experience with the use semantics of the core libs, or similar, will be closed with a reference to the policy. Examples: 
+    *	sketch crashes due to a char[] in it that is not null terminated
+    *	trying to use yield/delay, or libs that use yield/delay, from inside async callbacks
+    *	Use of new/malloc without matching delete/free (mem leak)
+5.	Issues about topics already handled in the documentation will be closed in a similar manner. Example:
+    *	can't flash with error espcomm failed
+6.	Issues must be provided with a minimalist sketch. Issues with an incomplete sketch, or a huge sketch, will be closed. Maximum effort must be put forth by the person opening the issue to reduce the relevant code that reproduces the issue, so that investigation can be taken up. MCVE is a must.
+7. Issues for unmerged PRs will be closed. If there is an issue with a PR, the explanation should be added to the PR itself.
+8. Issues with accompanied investigation that shows the root of the problem should be given priority
+9.	Duplicate issues will be closed with a reference to the original
+
+# Triaging
+1.	Any contributor of the project can participate in the triaging process, if he/she chooses to do so
+2.	An issue that needs to be closed, either due to not complying with this policy, or for other reasons, should be closed by a contributor
+3.	Issues that are accepted should be marked with appropriate labels, e.g.: component: xyz
+4.	If an issue is deemed to require specialized knowledge (e.g.: TLS, HTTP parser, SDK integration, etc), contributor(s) relevant to the affected code should be /cc’ed, as this can help grab attention
+5.	Severe issues should be assigned to the current milestone (i.e.: the next version to be released), or the milestone following. It is ok to push back issues depending on available resources.
+6.	Issues that could impact functionality for many users should be considered severe. 
+7.	Issues caused by the SDK or chip should not be marked severe, as there usually isn’t much to be done. Common sense should be applied when deciding. Such issues should be documented in a KID (Known Issues Document), possibly on the Wiki, for reference by users. Example:
+    * ARP issue
+    * Extra channel change beacon announced by the SoftAP
+    * Wakeup ROM bug in the ESP chip
+8.	Issues with feature requests should be discussed for viability/desirability. Example:
+    * Support for new board. If the new board is not widely used, doesn’t have a manufacturer webpage, etc, then it isn’t desirable to support it. If the new board is essentially a duplicate of another, it isn’t desirable to duplicate the existing one.
+9.	Feature requests or changes that are meant to address a very specific/limited use case, especially if at the expense of increased code complexity, may be denied, or may be required to be redesigned, generalized, or simplified
+10.	Feature requests that are not accompanied by a PR:
+    * could be closed immediately (denied)
+    * could be closed after some predetermined period of time (left as candidate for somebody to pick up)
+    * could be deemed interesting enough to work on, but without a specific project or target, and hence accumulated in a long-term feature request list. Such feature requests will in general not be targeted for a deadline or release.
+11.	In some cases, feedback may be requested from the issue reporter, either as additional info for clarification, additional testing, or other. If no feedback is provided after 30 days, the issue may be closed by a contributor.
+
+# Compatibility
+1.	Compatibility with the Arduino build system is first priority. Compatibility with PlatformIO and make are also maintained, but are second priority.
+2.	Feature requests should consider compatibility with Arduino
+    *	ESP-specific APIs should be added with care, and should be marked as such (Example: ESP8266WiFi)
+    *	APIs of common libraries should maintain compatibility (Example: Wire, SPI, Servo)
+    *	ESP-specific extensions to compatible APIs are ok, especially if required to fully use certain peripherals, but such functions should be clearly marked as ESP-specific
+3.	When making changes that are likely to impact PlatformIO or make, relevant people should be notified. Check whether some corresponding changes are needed on the build system side. When an issue related to one of these build systems is reported, redirect the issue reporter to the respective issue tracker.
+4.	The core libs are implemented as a wrapper layer over the Espressif SDK. Due to the requirements and limitations imposed by the SDK, there are inherent differences between the behavior of this core and the standard Arduino core (Example: using long delay()s is not allowed here). Compatibility can’t be maintained in such cases, and differences should be clearly documented.
+
+# Pull requests
+1.	All pull requests should undergo peer review by at least one contributor other than the creator
+2.	All pull requests should consider updates to the documentation
+3.	All pull requests should consider updates to regression tests, where possible
+4.	Pull requests that address an outstanding issue, particularly an issue deemed to be severe, should be given priority
+5.	If a PR is accepted, then it should undergo review and updated based on the feedback provided, then merged
+6.   Pull requests that don't meet the above will be denied and closed
+
+# Other
+A table should be maintained for relating maintainers and components. When triaging, this is essential to figure out if someone in particular should be consulted about specific changes.
+
+A stable release cadence should be established, e.g.: every 6 months.
+
+Regression testing should be revisited and streamlined with the release process. Running regression tests should be done before merging a PR to reduce overhead for a release. 
+
+
+

README.md

@@ -20,7 +20,7 @@ ESP8266 Arduino core comes with libraries to communicate over WiFi using TCP and
 
 Starting with 1.6.4, Arduino allows installation of third-party platform packages using Boards Manager. We have packages available for Windows, Mac OS, and Linux (32 and 64 bit).
 
-- Install Arduino 1.8.2 from the [Arduino website](http://www.arduino.cc/en/main/software).
+- Install the current upstream Arduino IDE at the 1.8 level or later. The current version is at the [Arduino website](http://www.arduino.cc/en/main/software).
 - Start Arduino and open Preferences window.
 - Enter ```http://arduino.esp8266.com/stable/package_esp8266com_index.json``` into *Additional Board Manager URLs* field. You can add multiple URLs, separating them with commas.
 - Open Boards Manager from Tools > Board menu and install *esp8266* platform (and don't forget to select your ESP8266 board from Tools > Board menu after installation).

cores/esp8266/Arduino.h

@@ -178,7 +178,7 @@ void ets_intr_unlock();
 #define _NOP() do { __asm__ volatile ("nop"); } while (0)
 #endif
 
-typedef unsigned int word;
+typedef uint16_t word;
 
 #define bit(b) (1UL << (b))
 #define _BV(b) (1UL << (b))
@@ -202,6 +202,7 @@ void analogWriteRange(uint32_t range);
 
 unsigned long millis(void);
 unsigned long micros(void);
+uint64_t micros64(void);
 void delay(unsigned long);
 void delayMicroseconds(unsigned int us);
 unsigned long pulseIn(uint8_t pin, uint8_t state, unsigned long timeout);
@@ -236,7 +237,7 @@ void optimistic_yield(uint32_t interval_us);
 #endif
 
 #ifdef __cplusplus
-
+#include <algorithm>
 #include "pgmspace.h"
 
 #include "WCharacter.h"
@@ -247,11 +248,10 @@ void optimistic_yield(uint32_t interval_us);
 #include "Updater.h"
 #include "debug.h"
 
-#ifndef _GLIBCXX_VECTOR
+using std::min;
-// arduino is not compatible with std::vector
+using std::max;
-#define min(a,b) ((a)<(b)?(a):(b))
+using std::isinf;
-#define max(a,b) ((a)>(b)?(a):(b))
+using std::isnan;
-#endif
 
 #define _min(a,b) ((a)<(b)?(a):(b))
 #define _max(a,b) ((a)>(b)?(a):(b))

cores/esp8266/Esp.cpp

@@ -396,23 +396,16 @@ struct rst_info * EspClass::getResetInfoPtr(void) {
 }
 
 bool EspClass::eraseConfig(void) {
-    bool ret = true;
+    const size_t cfgSize = 0x4000;
-    size_t cfgAddr = (ESP.getFlashChipSize() - 0x4000);
+    size_t cfgAddr = ESP.getFlashChipSize() - cfgSize;
-    size_t cfgSize = (8*1024);
 
-    noInterrupts();
+    for (size_t offset = 0; offset < cfgSize; offset += SPI_FLASH_SEC_SIZE) {
-    while(cfgSize) {
+        if (!flashEraseSector((cfgAddr + offset) / SPI_FLASH_SEC_SIZE)) {
-
+            return false;
-        if(spi_flash_erase_sector((cfgAddr / SPI_FLASH_SEC_SIZE)) != SPI_FLASH_RESULT_OK) {
-            ret = false;
         }
-
-        cfgSize -= SPI_FLASH_SEC_SIZE;
-        cfgAddr += SPI_FLASH_SEC_SIZE;
     }
-    interrupts();
 
-    return ret;
+    return true;
 }
 
 uint32_t EspClass::getSketchSize() {

cores/esp8266/FunctionalInterrupt.cpp

@@ -0,0 +1,24 @@
+#include <FunctionalInterrupt.h>
+
+
+// Duplicate typedefs from core_esp8266_wiring_digital_c
+typedef void (*voidFuncPtr)(void);
+
+// Helper functions for Functional interrupt routines
+extern "C" void ICACHE_RAM_ATTR __attachInterruptArg(uint8_t pin, voidFuncPtr userFunc, void*fp , int mode);
+
+// Structure for communication
+struct ArgStructure {
+	std::function<void(void)> reqFunction;
+};
+
+void interruptFunctional(void* arg)
+{
+	((ArgStructure*)arg)->reqFunction();
+}
+
+void attachInterrupt(uint8_t pin, std::function<void(void)> intRoutine, int mode)
+{
+	// use the local interrupt routine which takes the ArgStructure as argument
+	__attachInterruptArg (pin, (voidFuncPtr)interruptFunctional, new ArgStructure{intRoutine}, mode);
+}

cores/esp8266/FunctionalInterrupt.h

@@ -0,0 +1,15 @@
+#ifndef FUNCTIONALINTERRUPT_H
+#define FUNCTIONALINTERRUPT_H
+
+#include <stddef.h>
+#include <stdint.h>
+#include <functional>
+
+extern "C" {
+#include "c_types.h"
+#include "ets_sys.h"
+}
+
+void attachInterrupt(uint8_t pin, std::function<void(void)> intRoutine, int mode);
+
+#endif //INTERRUPTS_H

cores/esp8266/HardwareSerial.cpp

@@ -152,6 +152,9 @@ void HardwareSerial::flush()
     }
 
     uart_wait_tx_empty(_uart);
+    //Workaround for a bug in serial not actually being finished yet
+    //Wait for 8 data bits, 1 parity and 2 stop bits, just in case
+    delayMicroseconds(11000000 / uart_get_baudrate(_uart) + 1);
 }
 
 size_t HardwareSerial::write(uint8_t c)

cores/esp8266/Schedule.cpp

@@ -14,26 +14,6 @@ static scheduled_fn_t* sLastUnused = 0;
 
 static int sCount = 0;
 
-static void init_lists()
-{
-    (void) init_lists;
-    if (sCount != 0) {
-        return;
-    }
-    while (sCount < SCHEDULED_FN_INITIAL_COUNT) {
-        scheduled_fn_t* it = new scheduled_fn_t;
-        if (sCount == 0) {
-            sFirstUnused = it;
-        }
-        else {
-            sLastUnused->mNext = it;
-        }
-        sLastUnused = it;
-        ++sCount;
-    }
-    sLastUnused->mNext = NULL;
-}
-
 static scheduled_fn_t* get_fn() {
     scheduled_fn_t* result = NULL;
     // try to get an item from unused items list

cores/esp8266/WString.cpp

@@ -159,7 +159,6 @@ unsigned char String::changeBuffer(unsigned int maxStrLen) {
         buffer = newbuffer;
         return 1;
     }
-    buffer = newbuffer;
     return 0;
 }
 
@@ -478,6 +477,33 @@ unsigned char String::equalsIgnoreCase(const String &s2) const {
     return 1;
 }
 
+unsigned char String::equalsConstantTime(const String &s2) const {
+    // To avoid possible time-based attacks present function
+    // compares given strings in a constant time.
+    if(len != s2.len)
+        return 0;
+    //at this point lengths are the same
+    if(len == 0)
+        return 1;
+    //at this point lenghts are the same and non-zero
+    const char *p1 = buffer;
+    const char *p2 = s2.buffer;
+    unsigned int equalchars = 0;
+    unsigned int diffchars = 0;
+    while(*p1) {
+        if(*p1 == *p2)
+            ++equalchars;
+        else
+            ++diffchars;
+        ++p1;
+        ++p2;
+    }
+    //the following should force a constant time eval of the condition without a compiler "logical shortcut"
+    unsigned char equalcond = (equalchars == len);
+    unsigned char diffcond = (diffchars == 0);
+    return (equalcond & diffcond); //bitwise AND
+}
+
 unsigned char String::startsWith(const String &s2) const {
     if(len < s2.len)
         return 0;

cores/esp8266/WString.h

@@ -194,6 +194,7 @@ class String {
         unsigned char operator <=(const String &rhs) const;
         unsigned char operator >=(const String &rhs) const;
         unsigned char equalsIgnoreCase(const String &s) const;
+        unsigned char equalsConstantTime(const String &s) const;
         unsigned char startsWith(const String &prefix) const;
         unsigned char startsWith(const String &prefix, unsigned int offset) const;
         unsigned char endsWith(const String &suffix) const;

cores/esp8266/cbuf.cpp

@@ -39,7 +39,7 @@ size_t cbuf::resize(size_t newSize) {
 
 	// not lose any data
 	// if data can be lost use remove or flush before resize
-    if((newSize < bytes_available) || (newSize == _size)) {
+    if((newSize <= bytes_available) || (newSize == _size)) {
         return _size;
     }
 

cores/esp8266/core_esp8266_i2s.c

@@ -65,6 +65,10 @@ bool ICACHE_FLASH_ATTR i2s_is_empty(){
   return (i2s_slc_queue_len >= SLC_BUF_CNT-1);
 }
 
+int16_t ICACHE_FLASH_ATTR i2s_available(){
+	return (SLC_BUF_CNT - i2s_slc_queue_len) * SLC_BUF_LEN;
+}
+
 uint32_t ICACHE_FLASH_ATTR i2s_slc_queue_next_item(){ //pop the top off the queue
   uint8_t i;
   uint32_t item = i2s_slc_queue[0];

cores/esp8266/core_esp8266_postmortem.c

@@ -27,12 +27,14 @@
 #include "user_interface.h"
 #include "esp8266_peri.h"
 #include "cont.h"
+#include "pgmspace.h"
 
 extern void __real_system_restart_local();
 extern void gdb_do_break();
 
 extern cont_t g_cont;
 
+// These will be pointers to PROGMEM const strings
 static const char* s_panic_file = 0;
 static int s_panic_line = 0;
 static const char* s_panic_func = 0;
@@ -45,6 +47,8 @@ static void uart1_write_char_d(char c);
 static void print_stack(uint32_t start, uint32_t end);
 //static void print_pcs(uint32_t start, uint32_t end);
 
+bool __attribute((weak)) crash_for_gdb = 0;
+
 extern void __custom_crash_callback( struct rst_info * rst_info, uint32_t stack, uint32_t stack_end ) {
     (void) rst_info;
     (void) stack;
@@ -53,7 +57,16 @@ extern void __custom_crash_callback( struct rst_info * rst_info, uint32_t stack,
 
 extern void custom_crash_callback( struct rst_info * rst_info, uint32_t stack, uint32_t stack_end ) __attribute__ ((weak, alias("__custom_crash_callback")));
 
+static void ets_puts_P(const char *romString) {
+    char c = pgm_read_byte(romString++);
+    while (c) {
+        ets_putc(c);
+        c = pgm_read_byte(romString++);
+    }
+}
+
 void __wrap_system_restart_local() {
+    if (crash_for_gdb) *((int*)0) = 0;
     register uint32_t sp asm("a1");
 
     struct rst_info rst_info = {0};
@@ -68,17 +81,21 @@ void __wrap_system_restart_local() {
     ets_install_putc1(&uart_write_char_d);
 
     if (s_panic_line) {
-        ets_printf("\nPanic %s:%d %s\n", s_panic_file, s_panic_line, s_panic_func);
+        ets_puts_P(PSTR("\nPanic "));
+        ets_puts_P(s_panic_file);
+        ets_printf(":%d ", s_panic_line);
+        ets_puts_P(s_panic_func);
+        ets_puts_P(PSTR("\n"));
     }
     else if (s_abort_called) {
-        ets_printf("Abort called\n");
+        ets_puts_P(PSTR("Abort called\n"));
     }
     else if (rst_info.reason == REASON_EXCEPTION_RST) {
         ets_printf("\nException (%d):\nepc1=0x%08x epc2=0x%08x epc3=0x%08x excvaddr=0x%08x depc=0x%08x\n",
             rst_info.exccause, rst_info.epc1, rst_info.epc2, rst_info.epc3, rst_info.excvaddr, rst_info.depc);
     }
     else if (rst_info.reason == REASON_SOFT_WDT_RST) {
-        ets_printf("\nSoft WDT reset\n");
+        ets_puts_P(PSTR("\nSoft WDT reset\n"));
     }
 
     uint32_t cont_stack_start = (uint32_t) &(g_cont.stack);
@@ -100,11 +117,11 @@ void __wrap_system_restart_local() {
     }
 
     if (sp > cont_stack_start && sp < cont_stack_end) {
-        ets_printf("\nctx: cont \n");
+        ets_puts_P(PSTR("\nctx: cont \n"));
         stack_end = cont_stack_end;
     }
     else {
-        ets_printf("\nctx: sys \n");
+        ets_puts_P(("\nctx: sys \n"));
         stack_end = 0x3fffffb0;
         // it's actually 0x3ffffff0, but the stuff below ets_run
         // is likely not really relevant to the crash
@@ -123,7 +140,7 @@ void __wrap_system_restart_local() {
 
 
 static void print_stack(uint32_t start, uint32_t end) {
-    ets_printf("\n>>>stack>>>\n");
+    ets_puts_P(PSTR("\n>>>stack>>>\n"));
     for (uint32_t pos = start; pos < end; pos += 0x10) {
         uint32_t* values = (uint32_t*)(pos);
 
@@ -133,7 +150,7 @@ static void print_stack(uint32_t start, uint32_t end) {
         ets_printf("%08x:  %08x %08x %08x %08x %c\n",
             pos, values[0], values[1], values[2], values[3], (looksLikeStackFrame)?'<':' ');
     }
-    ets_printf("<<<stack<<<\n");
+    ets_puts_P(PSTR("<<<stack<<<\n"));
 }
 
 /*

cores/esp8266/core_esp8266_wiring.c

@@ -71,6 +71,13 @@ unsigned long ICACHE_RAM_ATTR micros() {
     return system_get_time();
 }
 
+uint64_t ICACHE_RAM_ATTR micros64() {
+    uint32_t low32_us = system_get_time();
+    uint32_t high32_us = micros_overflow_count + ((low32_us < micros_at_last_overflow_tick) ? 1 : 0);
+    uint64_t duration64_us = (uint64_t)high32_us << 32 | low32_us;
+    return duration64_us;
+}
+
 void ICACHE_RAM_ATTR delayMicroseconds(unsigned int us) {
     os_delay_us(us);
 }

cores/esp8266/core_esp8266_wiring_digital.c

@@ -105,12 +105,15 @@ extern int ICACHE_RAM_ATTR __digitalRead(uint8_t pin) {
 */
 
 typedef void (*voidFuncPtr)(void);
+typedef void (*voidFuncPtrArg)(void*);
 
 typedef struct {
   uint8_t mode;
   void (*fn)(void);
+  void * arg;
 } interrupt_handler_t;
 
+
 static interrupt_handler_t interrupt_handlers[16];
 static uint32_t interrupt_reg = 0;
 
@@ -127,39 +130,57 @@ void ICACHE_RAM_ATTR interrupt_handler(void *arg) {
     while(!(changedbits & (1 << i))) i++;
     changedbits &= ~(1 << i);
     interrupt_handler_t *handler = &interrupt_handlers[i];
-    if (handler->fn &&
+    if (handler->fn && 
-        (handler->mode == CHANGE ||
+        (handler->mode == CHANGE || 
          (handler->mode & 1) == !!(levels & (1 << i)))) {
       // to make ISR compatible to Arduino AVR model where interrupts are disabled
       // we disable them before we call the client ISR
-      uint32_t savedPS = xt_rsil(15); // stop other interrupts
+      uint32_t savedPS = xt_rsil(15); // stop other interrupts 
-      handler->fn();
+      if (handler->arg)
-      xt_wsr_ps(savedPS);
+      {
+    	  ((voidFuncPtrArg)handler->fn)(handler->arg);
+      }
+      else
+      {
+    	  handler->fn();
+      }
+       xt_wsr_ps(savedPS);
     }
   }
   ETS_GPIO_INTR_ENABLE();
 }
 
-extern void ICACHE_RAM_ATTR __attachInterrupt(uint8_t pin, voidFuncPtr userFunc, int mode) {
+extern void ICACHE_RAM_ATTR __attachInterruptArg(uint8_t pin, voidFuncPtr userFunc, void *arg, int mode) {
   if(pin < 16) {
+    ETS_GPIO_INTR_DISABLE();
     interrupt_handler_t *handler = &interrupt_handlers[pin];
     handler->mode = mode;
     handler->fn = userFunc;
+    handler->arg = arg;
     interrupt_reg |= (1 << pin);
     GPC(pin) &= ~(0xF << GPCI);//INT mode disabled
     GPIEC = (1 << pin); //Clear Interrupt for this pin
     GPC(pin) |= ((mode & 0xF) << GPCI);//INT mode "mode"
+    ETS_GPIO_INTR_ENABLE();
   }
 }
 
+extern void ICACHE_RAM_ATTR __attachInterrupt(uint8_t pin, voidFuncPtr userFunc, int mode )
+{
+	__attachInterruptArg (pin, userFunc, 0, mode);
+}
+
 extern void ICACHE_RAM_ATTR __detachInterrupt(uint8_t pin) {
   if(pin < 16) {
+    ETS_GPIO_INTR_DISABLE();
     GPC(pin) &= ~(0xF << GPCI);//INT mode disabled
     GPIEC = (1 << pin); //Clear Interrupt for this pin
     interrupt_reg &= ~(1 << pin);
     interrupt_handler_t *handler = &interrupt_handlers[pin];
     handler->mode = 0;
     handler->fn = 0;
+    handler->arg = 0;
+    ETS_GPIO_INTR_ENABLE();
   }
 }
 
@@ -176,7 +197,7 @@ void initPins() {
   for (int i = 12; i <= 16; ++i) {
     pinMode(i, INPUT);
   }
-
+  
   ETS_GPIO_INTR_ATTACH(interrupt_handler, &interrupt_reg);
   ETS_GPIO_INTR_ENABLE();
 }
@@ -186,3 +207,4 @@ extern void digitalWrite(uint8_t pin, uint8_t val) __attribute__ ((weak, alias("
 extern int digitalRead(uint8_t pin) __attribute__ ((weak, alias("__digitalRead")));
 extern void attachInterrupt(uint8_t pin, voidFuncPtr handler, int mode) __attribute__ ((weak, alias("__attachInterrupt")));
 extern void detachInterrupt(uint8_t pin) __attribute__ ((weak, alias("__detachInterrupt")));
+

cores/esp8266/coredecls.h

@@ -0,0 +1,20 @@
+
+#ifndef __COREDECLS_H
+#define __COREDECLS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// TODO: put declarations here, get rid of -Wno-implicit-function-declaration
+
+extern bool timeshift64_is_set;
+
+void tune_timeshift64 (uint64_t now_us);
+void settimeofday_cb (void (*cb)(void));
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif // __COREDECLS_H

cores/esp8266/debug.cpp

@@ -21,15 +21,16 @@
 #include "Arduino.h"
 #include "debug.h"
 
-void ICACHE_RAM_ATTR hexdump(uint8_t *mem, uint32_t len, uint8_t cols) {
+void ICACHE_RAM_ATTR hexdump(const void *mem, uint32_t len, uint8_t cols) {
-    os_printf("\n[HEXDUMP] Address: 0x%08X len: 0x%X (%d)", (size_t)mem, len, len);
+    const uint8_t* src = (const uint8_t*) mem;
+    os_printf("\n[HEXDUMP] Address: 0x%08X len: 0x%X (%d)", (ptrdiff_t)src, len, len);
     for(uint32_t i = 0; i < len; i++) {
         if(i % cols == 0) {
-            os_printf("\n[0x%08X] 0x%08X: ", (size_t)mem, i);
+            os_printf("\n[0x%08X] 0x%08X: ", (ptrdiff_t)src, i);
         	yield();
         }
-        os_printf("%02X ", *mem);
+        os_printf("%02X ", *src);
-        mem++;
+        src++;
     }
     os_printf("\n");
 }

cores/esp8266/debug.h

@@ -13,9 +13,9 @@
 #endif
 
 #ifdef __cplusplus
-void hexdump(uint8_t *mem, uint32_t len, uint8_t cols = 16);
+void hexdump(const void *mem, uint32_t len, uint8_t cols = 16);
 #else
-void hexdump(uint8_t *mem, uint32_t len, uint8_t cols);
+void hexdump(const void *mem, uint32_t len, uint8_t cols);
 #endif
 
 #ifdef __cplusplus

cores/esp8266/i2s.h

@@ -48,6 +48,7 @@ bool i2s_write_sample_nb(uint32_t sample);//same as above but does not block whe
 bool i2s_write_lr(int16_t left, int16_t right);//combines both channels and calls i2s_write_sample with the result
 bool i2s_is_full();//returns true if DMA is full and can not take more bytes (overflow)
 bool i2s_is_empty();//returns true if DMA is empty (underflow)
+int16_t i2s_available();// returns the number of samples than can be written before blocking
 
 #ifdef __cplusplus
 }

cores/esp8266/interrupts.h

@@ -11,7 +11,7 @@ extern "C" {
 // these auto classes wrap up xt_rsil so your code can be simplier, but can only be
 // used in an ino or cpp files. 
 
-// InterruptLock is used when you want to completely disable locks
+// InterruptLock is used when you want to completely disable interrupts
 //{
 //    {
 //      InterruptLock lock; 

cores/esp8266/libc_replacements.c

@@ -117,6 +117,11 @@ int ICACHE_RAM_ATTR putchar(int c) {
     return c;
 }
 
+void _exit(int status) {
+    (void) status;
+    abort();
+}
+
 #if 0
 
 int ICACHE_RAM_ATTR printf(const char* format, ...) {

cores/esp8266/pgmspace.cpp

@@ -24,6 +24,8 @@
 #include <stdarg.h>
 #include "pgmspace.h"
 
+extern "C" {
+
 size_t strnlen_P(PGM_P s, size_t size) {
     const char* cp;
     for (cp = s; size != 0 && pgm_read_byte(cp) != '\0'; cp++, size--);
@@ -147,6 +149,7 @@ void* memmem_P(const void* buf, size_t bufSize, PGM_VOID_P findP, size_t findPSi
 
 
 char* strncpy_P(char* dest, PGM_P src, size_t size) {
+    bool size_known = (size != SIZE_IRRELEVANT);
     const char* read = src;
     char* write = dest;
     char ch = '.';
@@ -156,6 +159,14 @@ char* strncpy_P(char* dest, PGM_P src, size_t size) {
         *write++ = ch;
         size--;
     }
+    if (size_known)
+    {
+        while (size > 0)
+        {
+            *write++ = 0;
+            size--;
+        }
+    }
 
     return dest;
 }
@@ -234,7 +245,7 @@ int printf_P(PGM_P formatP, ...) {
     char* format = new char[fmtLen + 1];
     strcpy_P(format, formatP);
 
-    ret = printf(format, arglist);
+    ret = vprintf(format, arglist);
 
     delete[] format;
 
@@ -277,3 +288,5 @@ int vsnprintf_P(char* str, size_t strSize, PGM_P formatP, va_list ap) {
 
     return ret;
 }
+
+} // extern "C"

cores/esp8266/pgmspace.h

@@ -4,16 +4,14 @@
 #include <stdint.h>
 #include <stdio.h>
 
-#ifdef __ets__
-
 #ifdef __cplusplus
 extern "C" {
 #endif
+
+#ifdef __ets__
+
 #include "ets_sys.h"
 #include "osapi.h"
-#ifdef __cplusplus
-}
-#endif
 
 #define PROGMEM     ICACHE_RODATA_ATTR
 #define PGM_P  		const char *
@@ -99,19 +97,23 @@ int	vsnprintf_P(char *str, size_t strSize, PGM_P formatP, va_list ap) __attribut
       :"1"(addr) \
       :);
 
-static inline uint8_t pgm_read_byte(const void* addr) {
+static inline uint8_t pgm_read_byte_inlined(const void* addr) {
   register uint32_t res;
   pgm_read_with_offset(addr, res);
   return (uint8_t) res;     /* This masks the lower byte from the returned word */
 }
 
 /* Although this says "word", it's actually 16 bit, i.e. half word on Xtensa */
-static inline uint16_t pgm_read_word(const void* addr) {
+static inline uint16_t pgm_read_word_inlined(const void* addr) {
   register uint32_t res;
   pgm_read_with_offset(addr, res);
   return (uint16_t) res;    /* This masks the lower half-word from the returned word */
 }
 
+// Make sure, that libraries checking existence of this macro are not failing
+#define pgm_read_byte(addr) pgm_read_byte_inlined(addr)
+#define pgm_read_word(addr) pgm_read_word_inlined(addr)
+
 #else //__ets__
 #define pgm_read_byte(addr)     (*reinterpret_cast<const uint8_t*>(addr))
 #define pgm_read_word(addr)     (*reinterpret_cast<const uint16_t*>(addr))
@@ -132,4 +134,8 @@ static inline uint16_t pgm_read_word(const void* addr) {
 #define pgm_read_float_far(addr) 	pgm_read_float(addr)
 #define pgm_read_ptr_far(addr)		pgm_read_ptr(addr)
 
+#ifdef __cplusplus
+}
+#endif
+
 #endif //__PGMSPACE_H_

cores/esp8266/sntp-lwip2.c

@@ -0,0 +1,463 @@
+/*
+ * sntp-lwip2.c - ESP8266-specific functions for SNTP and lwIP-v2
+ * Copyright (c) 2015 Espressif (license is tools/sdk/lwip/src/core/sntp.c's)
+ * Redistribution and use in source and binary forms, with or without modification, 
+ * are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *    this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ *    this list of conditions and the following disclaimer in the documentation
+ *    and/or other materials provided with the distribution.
+ * 3. The name of the author may not be used to endorse or promote products
+ *    derived from this software without specific prior written permission. 
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED 
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 
+ * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT 
+ * SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT 
+ * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 
+ * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY 
+ * OF SUCH DAMAGE.
+ *
+ *
+ * History:
+ * This code is extracted from lwip1.4-espressif's sntp.c
+ * which is a patched version of the original lwip1's sntp.
+ * (check the mix-up in tools/sdk/lwip/src/core/sntp.c)
+ * It is moved here as-is and cleaned for maintainability and
+ * because it does not belong to lwip.
+ *
+ * TODOs:
+ * settimeofday(): handle tv->tv_usec
+ * sntp_mktm_r(): review, fix DST handling (this one is currently untouched from lwip-1.4)
+ * implement adjtime()
+ */
+
+#include <lwip/init.h>
+#include <sys/time.h>
+#include <osapi.h>
+#include <os_type.h>
+#include "coredecls.h"
+
+static void (*_settimeofday_cb)(void) = NULL;
+
+void settimeofday_cb (void (*cb)(void))
+{
+    _settimeofday_cb = cb;
+}
+
+#if LWIP_VERSION_MAJOR == 1
+
+#include <pgmspace.h>
+
+static const char stod14[] PROGMEM = "settimeofday() can't set time!\n";
+
+int settimeofday(const struct timeval* tv, const struct timezone* tz)
+{
+    if (tz) /*before*/
+    {
+        sntp_set_timezone(tz->tz_minuteswest / 60);
+        // apparently tz->tz_dsttime is a bitfield and should not be further used (cf man)
+        sntp_set_daylight(0);
+    }
+    if (tv) /* after*/
+    {
+        // can't call lwip1.4's static sntp_set_system_time()
+        os_printf(stod14);
+
+        // reset time subsystem
+        timeshift64_is_set = false;
+        
+        return -1;
+    }
+    return 0;
+}
+
+#endif // lwip 1.4 only
+
+#if LWIP_VERSION_MAJOR == 2
+
+#include <lwip/apps/sntp.h>
+
+static uint32 realtime_stamp = 0;
+static uint16 dst = 0;
+static sint8 time_zone = 8; // espressif HQ's default timezone
+LOCAL os_timer_t sntp_timer;
+
+/*****************************************/
+#define SECSPERMIN	60L
+#define MINSPERHOUR	60L
+#define HOURSPERDAY	24L
+#define SECSPERHOUR	(SECSPERMIN * MINSPERHOUR)
+#define SECSPERDAY	(SECSPERHOUR * HOURSPERDAY)
+#define DAYSPERWEEK	7
+#define MONSPERYEAR	12
+
+#define YEAR_BASE	1900
+#define EPOCH_YEAR      1970
+#define EPOCH_WDAY      4
+#define EPOCH_YEARS_SINCE_LEAP 2
+#define EPOCH_YEARS_SINCE_CENTURY 70
+#define EPOCH_YEARS_SINCE_LEAP_CENTURY 370
+
+#define isleap(y) ((((y) % 4) == 0 && ((y) % 100) != 0) || ((y) % 400) == 0)
+
+int __tznorth;
+int __tzyear;
+char reult[100];
+static const int mon_lengths[2][12] = {
+  {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31},
+  {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}
+} ;
+
+static const int year_lengths[2] = {
+  365,
+  366
+} ;
+struct tm
+{
+  int	tm_sec;
+  int	tm_min;
+  int	tm_hour;
+  int	tm_mday;
+  int	tm_mon;
+  int	tm_year;
+  int	tm_wday;
+  int	tm_yday;
+  int	tm_isdst;
+};
+
+struct tm res_buf;
+typedef struct __tzrule_struct
+{
+  char ch;
+  int m;
+  int n;
+  int d;
+  int s;
+  time_t change;
+  int offset;
+} __tzrule_type;
+
+__tzrule_type sntp__tzrule[2];
+struct tm *
+sntp_mktm_r(const time_t * tim_p ,struct tm *res ,int is_gmtime)
+{
+  long days, rem;
+  time_t lcltime;
+  int y;
+  int yleap;
+  const int *ip;
+
+  /* base decision about std/dst time on current time */
+  lcltime = *tim_p;
+
+  days = ((long)lcltime) / SECSPERDAY;
+  rem = ((long)lcltime) % SECSPERDAY;
+  while (rem < 0)
+    {
+      rem += SECSPERDAY;
+      --days;
+    }
+  while (rem >= SECSPERDAY)
+    {
+      rem -= SECSPERDAY;
+      ++days;
+    }
+
+  /* compute hour, min, and sec */
+  res->tm_hour = (int) (rem / SECSPERHOUR);
+  rem %= SECSPERHOUR;
+  res->tm_min = (int) (rem / SECSPERMIN);
+  res->tm_sec = (int) (rem % SECSPERMIN);
+
+  /* compute day of week */
+  if ((res->tm_wday = ((EPOCH_WDAY + days) % DAYSPERWEEK)) < 0)
+    res->tm_wday += DAYSPERWEEK;
+
+  /* compute year & day of year */
+  y = EPOCH_YEAR;
+  if (days >= 0)
+    {
+      for (;;)
+	{
+	  yleap = isleap(y);
+	  if (days < year_lengths[yleap])
+	    break;
+	  y++;
+	  days -= year_lengths[yleap];
+	}
+    }
+  else
+    {
+      do
+	{
+	  --y;
+	  yleap = isleap(y);
+	  days += year_lengths[yleap];
+	} while (days < 0);
+    }
+
+  res->tm_year = y - YEAR_BASE;
+  res->tm_yday = days;
+  ip = mon_lengths[yleap];
+  for (res->tm_mon = 0; days >= ip[res->tm_mon]; ++res->tm_mon)
+    days -= ip[res->tm_mon];
+  res->tm_mday = days + 1;
+
+  if (!is_gmtime)
+    {
+      int offset;
+      int hours, mins, secs;
+
+//      TZ_LOCK;
+//      if (_daylight)
+//	{
+//	  if (y == __tzyear || __tzcalc_limits (y))
+//	    res->tm_isdst = (__tznorth
+//			     ? (*tim_p >= __tzrule[0].change && *tim_p < __tzrule[1].change)
+//			     : (*tim_p >= __tzrule[0].change || *tim_p < __tzrule[1].change));
+//	  else
+//	    res->tm_isdst = -1;
+//	}
+//      else
+      res->tm_isdst = -1;
+
+      offset = (res->tm_isdst == 1 ? sntp__tzrule[1].offset : sntp__tzrule[0].offset);
+
+      hours = offset / SECSPERHOUR;
+      offset = offset % SECSPERHOUR;
+
+      mins = offset / SECSPERMIN;
+      secs = offset % SECSPERMIN;
+
+      res->tm_sec -= secs;
+      res->tm_min -= mins;
+      res->tm_hour -= hours;
+
+      if (res->tm_sec >= SECSPERMIN)
+	{
+	  res->tm_min += 1;
+	  res->tm_sec -= SECSPERMIN;
+	}
+      else if (res->tm_sec < 0)
+	{
+	  res->tm_min -= 1;
+	  res->tm_sec += SECSPERMIN;
+	}
+      if (res->tm_min >= MINSPERHOUR)
+	{
+	  res->tm_hour += 1;
+	  res->tm_min -= MINSPERHOUR;
+	}
+      else if (res->tm_min < 0)
+	{
+	  res->tm_hour -= 1;
+	  res->tm_min += MINSPERHOUR;
+	}
+      if (res->tm_hour >= HOURSPERDAY)
+	{
+	  ++res->tm_yday;
+	  ++res->tm_wday;
+	  if (res->tm_wday > 6)
+	    res->tm_wday = 0;
+	  ++res->tm_mday;
+	  res->tm_hour -= HOURSPERDAY;
+	  if (res->tm_mday > ip[res->tm_mon])
+	    {
+	      res->tm_mday -= ip[res->tm_mon];
+	      res->tm_mon += 1;
+	      if (res->tm_mon == 12)
+		{
+		  res->tm_mon = 0;
+		  res->tm_year += 1;
+		  res->tm_yday = 0;
+		}
+	    }
+	}
+       else if (res->tm_hour < 0)
+	{
+	  res->tm_yday -= 1;
+	  res->tm_wday -= 1;
+	  if (res->tm_wday < 0)
+	    res->tm_wday = 6;
+	  res->tm_mday -= 1;
+	  res->tm_hour += 24;
+	  if (res->tm_mday == 0)
+	    {
+	      res->tm_mon -= 1;
+	      if (res->tm_mon < 0)
+		{
+		  res->tm_mon = 11;
+		  res->tm_year -= 1;
+		  res->tm_yday = 365 + isleap(res->tm_year);
+		}
+	      res->tm_mday = ip[res->tm_mon];
+	    }
+	}
+//      TZ_UNLOCK;
+    }
+  else
+    res->tm_isdst = 0;
+//  os_printf("res %d %d %d %d %d\n",res->tm_year,res->tm_mon,res->tm_mday,res->tm_yday,res->tm_hour);
+  return (res);
+}
+struct tm *
+sntp_localtime_r(const time_t * tim_p ,
+		struct tm *res)
+{
+  return sntp_mktm_r (tim_p, res, 0);
+}
+
+struct tm *
+sntp_localtime(const time_t * tim_p)
+{
+  return sntp_localtime_r (tim_p, &res_buf);
+}
+
+int sntp__tzcalc_limits(int year)
+{
+  int days, year_days, years;
+  int i, j;
+
+  if (year < EPOCH_YEAR)
+    return 0;
+
+  __tzyear = year;
+
+  years = (year - EPOCH_YEAR);
+
+  year_days = years * 365 +
+    (years - 1 + EPOCH_YEARS_SINCE_LEAP) / 4 - (years - 1 + EPOCH_YEARS_SINCE_CENTURY) / 100 +
+    (years - 1 + EPOCH_YEARS_SINCE_LEAP_CENTURY) / 400;
+
+  for (i = 0; i < 2; ++i)
+    {
+      if (sntp__tzrule[i].ch == 'J')
+	days = year_days + sntp__tzrule[i].d + (isleap(year) && sntp__tzrule[i].d >= 60);
+      else if (sntp__tzrule[i].ch == 'D')
+	days = year_days + sntp__tzrule[i].d;
+      else
+	{
+	  int yleap = isleap(year);
+	  int m_day, m_wday, wday_diff;
+	  const int *ip = mon_lengths[yleap];
+
+	  days = year_days;
+
+	  for (j = 1; j < sntp__tzrule[i].m; ++j)
+	    days += ip[j-1];
+
+	  m_wday = (EPOCH_WDAY + days) % DAYSPERWEEK;
+
+	  wday_diff = sntp__tzrule[i].d - m_wday;
+	  if (wday_diff < 0)
+	    wday_diff += DAYSPERWEEK;
+	  m_day = (sntp__tzrule[i].n - 1) * DAYSPERWEEK + wday_diff;
+
+	  while (m_day >= ip[j-1])
+	    m_day -= DAYSPERWEEK;
+
+	  days += m_day;
+	}
+
+      /* store the change-over time in GMT form by adding offset */
+      sntp__tzrule[i].change = days * SECSPERDAY + sntp__tzrule[i].s + sntp__tzrule[i].offset;
+    }
+
+  __tznorth = (sntp__tzrule[0].change < sntp__tzrule[1].change);
+
+  return 1;
+}
+
+char* sntp_asctime_r(struct tm *tim_p ,char *result)
+{
+  static const char day_name[7][4] = {
+	"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"
+  };
+  static const char mon_name[12][4] = {
+	"Jan", "Feb", "Mar", "Apr", "May", "Jun",
+	"Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
+  };
+  os_sprintf (result, "%s %s %02d %02d:%02d:%02d %02d\n",
+	   day_name[tim_p->tm_wday],
+	   mon_name[tim_p->tm_mon],
+	   tim_p->tm_mday, tim_p->tm_hour, tim_p->tm_min,
+	   tim_p->tm_sec, 1900 + tim_p->tm_year);
+  return result;
+}
+
+char* sntp_asctime(struct tm *tim_p)
+{
+    return sntp_asctime_r (tim_p, reult);
+}
+
+uint32 ICACHE_RAM_ATTR sntp_get_current_timestamp(void)
+{
+    return realtime_stamp;
+}
+
+char* sntp_get_real_time(time_t t)
+{
+    return sntp_asctime(sntp_localtime (&t));
+}
+
+sint8 sntp_get_timezone(void)
+{
+    return time_zone;
+}
+
+bool sntp_set_timezone(sint8 timezone)
+{
+    if(timezone >= -11 || timezone <= 13) {
+        time_zone = timezone;
+        return true;
+    } else {
+        return false;
+    }
+}
+
+void sntp_set_daylight(int daylight)
+{
+    dst = daylight;
+}
+
+void ICACHE_RAM_ATTR sntp_time_inc (void)
+{
+    realtime_stamp++;
+}
+
+static void sntp_set_system_time (uint32_t t)
+{
+    realtime_stamp = t + time_zone * 60 * 60 + dst;
+    os_timer_disarm(&sntp_timer);
+    os_timer_setfn(&sntp_timer, (os_timer_func_t *)sntp_time_inc, NULL);
+    os_timer_arm(&sntp_timer, 1000, 1);
+}
+
+int settimeofday(const struct timeval* tv, const struct timezone* tz)
+{
+    if (tz) /*before*/
+    {
+        sntp_set_timezone(tz->tz_minuteswest / 60);
+        // apparently tz->tz_dsttime is a bitfield and should not be further used (cf man)
+        sntp_set_daylight(0);
+    }
+    if (tv) /* after*/
+    {
+        // reset time subsystem
+        tune_timeshift64(tv->tv_sec * 1000000ULL + tv->tv_usec);
+
+        sntp_set_system_time(tv->tv_sec);
+
+        if (_settimeofday_cb)
+            _settimeofday_cb();
+    }
+    return 0;
+}
+
+#endif // lwip2 only

cores/esp8266/spiffs/LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2013-2015 Peter Andersson (pelleplutt1976<at>gmail.com)
+Copyright (c) 2013-2017 Peter Andersson (pelleplutt1976<at>gmail.com)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

cores/esp8266/spiffs/README.md

@@ -1,7 +1,9 @@
 # SPIFFS (SPI Flash File System) 
-**V0.3.4**
+**V0.3.7**
 
-Copyright (c) 2013-2016 Peter Andersson (pelleplutt1976 at gmail.com)
+[![Build Status](https://travis-ci.org/pellepl/spiffs.svg?branch=master)](https://travis-ci.org/pellepl/spiffs)
+
+Copyright (c) 2013-2017 Peter Andersson (pelleplutt1976 at gmail.com)
 
 For legal stuff, see [LICENSE](https://github.com/pellepl/spiffs/blob/master/LICENSE). Basically, you may do whatever you want with the source. Use, modify, sell, print it out, roll it and smoke it - as long as I won't be held responsible.
 
@@ -21,27 +23,36 @@ Spiffs is designed with following characteristics in mind:
  - Wear leveling
 
 
+## BUILDING
+
+`mkdir build; make`
+
+Otherwise, configure the `builddir` variable towards the top of `makefile` as something opposed to the default `build`. Sanity check on the host via `make test` and refer to `.travis.yml` for the official in-depth testing procedure. See the wiki for [integrating](https://github.com/pellepl/spiffs/wiki/Integrate-spiffs) spiffs into projects and [spiffsimg](https://github.com/nodemcu/nodemcu-firmware/tree/master/tools/spiffsimg) from [nodemcu](https://github.com/nodemcu) is a good example on the subject.
+
+
 ## FEATURES
 
 What spiffs does:
  - Specifically designed for low ram usage
  - Uses statically sized ram buffers, independent of number of files
  - Posix-like api: open, close, read, write, seek, stat, etc
- - It can be run on any NOR flash, not only SPI flash - theoretically also on embedded flash of an microprocessor
+ - It can run on any NOR flash, not only SPI flash - theoretically also on embedded flash of a microprocessor
- - Multiple spiffs configurations can be run on same target - and even on same SPI flash device
+ - Multiple spiffs configurations can run on same target - and even on same SPI flash device
  - Implements static wear leveling 
  - Built in file system consistency checks
+ - Highly configurable
  
 What spiffs does not:
  - Presently, spiffs does not support directories. It produces a flat structure. Creating a file with path *tmp/myfile.txt* will create a file called *tmp/myfile.txt* instead of a *myfile.txt* under directory *tmp*. 
- - It is not a realtime stack. One write operation might take much longer than another.
+ - It is not a realtime stack. One write operation might last much longer than another.
- - Poor scalability. Spiffs is intended for small memory devices - the normal sizes for SPI flashes. Going beyond ~128MB is probably a bad idea. This is a side effect of the design goal to use as little ram as possible.
+ - Poor scalability. Spiffs is intended for small memory devices - the normal sizes for SPI flashes. Going beyond ~128Mbyte is probably a bad idea. This is a side effect of the design goal to use as little ram as possible.
  - Presently, it does not detect or handle bad blocks.
+ - One configuration, one binary. There's no generic spiffs binary that handles all types of configurations.
 
  
 ## MORE INFO 
  
-See the [wiki](https://github.com/pellepl/spiffs/wiki) for configuring, integrating and using spiffs.
+See the [wiki](https://github.com/pellepl/spiffs/wiki) for [configuring](https://github.com/pellepl/spiffs/wiki/Configure-spiffs), [integrating](https://github.com/pellepl/spiffs/wiki/Integrate-spiffs), [using](https://github.com/pellepl/spiffs/wiki/Using-spiffs), and [optimizing](https://github.com/pellepl/spiffs/wiki/Performance-and-Optimizing) spiffs.
  
 For design, see [docs/TECH_SPEC](https://github.com/pellepl/spiffs/blob/master/docs/TECH_SPEC).
 
@@ -49,6 +60,61 @@ For a generic spi flash driver, see [this](https://github.com/pellepl/spiflash_d
 
 ## HISTORY
 
+### 0.3.7
+- fixed prevent seeking to negative offsets #158
+- fixed file descriptor offsets not updated for multiple fds on same file #157
+- fixed cache page not closed for removed files #156
+- fixed a lseek bug when seeking exactly to end of a fully indexed first level LUT #148
+- fixed wear leveling issue #145
+- fixed attempt to write out of bounds in flash #130, 
+- set file offset when seeking over end #121 (thanks @sensslen)
+- fixed seeking in virgin files #120 (thanks @sensslen)
+- Optional file metadata #128 (thanks @cesanta)
+- AFL testing framework #100 #143 (thanks @pjsg)
+- Testframe updates
+
+New API functions:
+- `SPIFFS_update_meta, SPIFFS_fupdate_meta` - updates metadata for a file
+
+New config defines:
+- `SPIFFS_OBJ_META_LEN` - enable possibility to add extra metadata to files
+
+### 0.3.6
+- Fix range bug in index memory mapping #98
+- Add index memory mapping #97
+- Optimize SPIFFS_read for large files #96
+- Add temporal cache for opening files #95
+- More robust gc #93 (thanks @dismirlian)
+- Fixed a double write of same data in certain cache situations
+- Fixed an open bug in READ_ONLY builds
+- File not visible in SPIFFS_readdir #90 (thanks @benpicco-tmp)
+- Cache load code cleanup #92 (thanks @niclash)
+- Fixed lock/unlock asymmetry #88 #87 (thanks @JackJefferson, @dpruessner)
+- Testframe updates
+
+New API functions:
+- `SPIFFS_ix_map` - map index meta data to memory for a file
+- `SPIFFS_ix_unmap` - unmaps index meta data for a file
+- `SPIFFS_ix_remap` - changes file offset for index metadata map
+- `SPIFFS_bytes_to_ix_map_entries` - utility, get length of needed vector for given amount of bytes
+- `SPIFFS_ix_map_entries_to_bytes` - utility, get number of bytes a vector can represent given length
+ 
+New config defines:
+- `SPIFFS_IX_MAP` - enable possibility to map index meta data to memory for reading faster 
+- `SPIFFS_TEMPORAL_FD_CACHE` - enable temporal cache for opening files faster
+- `SPIFFS_TEMPORAL_CACHE_HIT_SCORE` - for tuning the temporal cache
+
+### 0.3.5
+- Fixed a bug in fs check
+- API returns actual error codes #84) (thanks @Nails)
+- Fix compiler warnings for non-gcc #83 #81 (thanks @Nails)
+- Unable to recover from full fs #82 (thanks @rojer)
+- Define SPIFFS_O_* flags #80
+- Problem with long filenames #79 (thanks @psjg)
+- Duplicate file name bug fix #74 (thanks @igrr)
+- SPIFFS_eof and SPIFFS_tell return wrong value #72 (thanks @ArtemPisarenko)
+- Bunch of testframe updates #77 #78 #86 (thanks @dpreussner, @psjg a.o)  
+
 ### 0.3.4
 - Added user callback file func.
 - Fixed a stat bug with obj id.

cores/esp8266/spiffs/spiffs.h

@@ -54,6 +54,15 @@ extern "C" {
 #define SPIFFS_ERR_RO_ABORTED_OPERATION -10033
 #define SPIFFS_ERR_PROBE_TOO_FEW_BLOCKS -10034
 #define SPIFFS_ERR_PROBE_NOT_A_FS       -10035
+#define SPIFFS_ERR_NAME_TOO_LONG        -10036
+
+#define SPIFFS_ERR_IX_MAP_UNMAPPED      -10037
+#define SPIFFS_ERR_IX_MAP_MAPPED        -10038
+#define SPIFFS_ERR_IX_MAP_BAD_RANGE     -10039
+
+#define SPIFFS_ERR_SEEK_BOUNDS          -10040
+
+
 #define SPIFFS_ERR_INTERNAL             -10050
 
 #define SPIFFS_ERR_TEST                 -10100
@@ -104,7 +113,7 @@ typedef enum {
   SPIFFS_CHECK_FIX_LOOKUP,
   SPIFFS_CHECK_DELETE_ORPHANED_INDEX,
   SPIFFS_CHECK_DELETE_PAGE,
-  SPIFFS_CHECK_DELETE_BAD_FILE,
+  SPIFFS_CHECK_DELETE_BAD_FILE
 } spiffs_check_report;
 
 /* file system check callback function */
@@ -123,7 +132,7 @@ typedef enum {
   /* the file has been updated or moved to another page */
   SPIFFS_CB_UPDATED,
   /* the file has been deleted */
-  SPIFFS_CB_DELETED,
+  SPIFFS_CB_DELETED
 } spiffs_fileop_type;
 
 /* file system listener callback function */
@@ -131,7 +140,7 @@ typedef void (*spiffs_file_callback)(struct spiffs_t *fs, spiffs_fileop_type op,
 
 #ifndef SPIFFS_DBG
 #define SPIFFS_DBG(...) \
-    print(__VA_ARGS__)
+    printf(__VA_ARGS__)
 #endif
 #ifndef SPIFFS_GC_DBG
 #define SPIFFS_GC_DBG(...) printf(__VA_ARGS__)
@@ -145,20 +154,28 @@ typedef void (*spiffs_file_callback)(struct spiffs_t *fs, spiffs_fileop_type op,
 
 /* Any write to the filehandle is appended to end of the file */
 #define SPIFFS_APPEND                   (1<<0)
+#define SPIFFS_O_APPEND                 SPIFFS_APPEND
 /* If the opened file exists, it will be truncated to zero length before opened */
 #define SPIFFS_TRUNC                    (1<<1)
+#define SPIFFS_O_TRUNC                  SPIFFS_TRUNC
 /* If the opened file does not exist, it will be created before opened */
 #define SPIFFS_CREAT                    (1<<2)
+#define SPIFFS_O_CREAT                  SPIFFS_CREAT
 /* The opened file may only be read */
 #define SPIFFS_RDONLY                   (1<<3)
-/* The opened file may only be writted */
+#define SPIFFS_O_RDONLY                 SPIFFS_RDONLY
+/* The opened file may only be written */
 #define SPIFFS_WRONLY                   (1<<4)
-/* The opened file may be both read and writted */
+#define SPIFFS_O_WRONLY                 SPIFFS_WRONLY
+/* The opened file may be both read and written */
 #define SPIFFS_RDWR                     (SPIFFS_RDONLY | SPIFFS_WRONLY)
-/* Any writes to the filehandle will never be cached */
+#define SPIFFS_O_RDWR                   SPIFFS_RDWR
+/* Any writes to the filehandle will never be cached but flushed directly */
 #define SPIFFS_DIRECT                   (1<<5)
-/* If SPIFFS_CREAT and SPIFFS_EXCL are set, SPIFFS_open() shall fail if the file exists */
+#define SPIFFS_O_DIRECT                 SPIFFS_DIRECT
+/* If SPIFFS_O_CREAT and SPIFFS_O_EXCL are set, SPIFFS_open() shall fail if the file exists */
 #define SPIFFS_EXCL                     (1<<6)
+#define SPIFFS_O_EXCL                   SPIFFS_EXCL
 
 #define SPIFFS_SEEK_SET                 (0)
 #define SPIFFS_SEEK_CUR                 (1)
@@ -283,6 +300,9 @@ typedef struct {
   spiffs_obj_type type;
   spiffs_page_ix pix;
   u8_t name[SPIFFS_OBJ_NAME_LEN];
+#if SPIFFS_OBJ_META_LEN
+  u8_t meta[SPIFFS_OBJ_META_LEN];
+#endif
 } spiffs_stat;
 
 struct spiffs_dirent {
@@ -291,6 +311,9 @@ struct spiffs_dirent {
   spiffs_obj_type type;
   u32_t size;
   spiffs_page_ix pix;
+#if SPIFFS_OBJ_META_LEN
+  u8_t meta[SPIFFS_OBJ_META_LEN];
+#endif
 };
 
 typedef struct {
@@ -299,6 +322,21 @@ typedef struct {
   int entry;
 } spiffs_DIR;
 
+#if SPIFFS_IX_MAP
+
+typedef struct {
+  // buffer with looked up data pixes
+  spiffs_page_ix *map_buf;
+  // precise file byte offset
+  u32_t offset;
+  // start data span index of lookup buffer
+  spiffs_span_ix start_spix;
+  // end data span index of lookup buffer
+  spiffs_span_ix end_spix;
+} spiffs_ix_map;
+
+#endif
+
 // functions
 
 #if SPIFFS_USE_MAGIC && SPIFFS_USE_MAGIC_LENGTH && SPIFFS_SINGLETON==0
@@ -375,8 +413,8 @@ s32_t SPIFFS_creat(spiffs *fs, const char *path, spiffs_mode mode);
  * @param fs            the file system struct
  * @param path          the path of the new file
  * @param flags         the flags for the open command, can be combinations of
- *                      SPIFFS_APPEND, SPIFFS_TRUNC, SPIFFS_CREAT, SPIFFS_RD_ONLY,
+ *                      SPIFFS_O_APPEND, SPIFFS_O_TRUNC, SPIFFS_O_CREAT, SPIFFS_O_RDONLY,
- *                      SPIFFS_WR_ONLY, SPIFFS_RDWR, SPIFFS_DIRECT
+ *                      SPIFFS_O_WRONLY, SPIFFS_O_RDWR, SPIFFS_O_DIRECT, SPIFFS_O_EXCL
  * @param mode          ignored, for posix compliance
  */
 spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs_mode mode);
@@ -496,6 +534,24 @@ s32_t SPIFFS_close(spiffs *fs, spiffs_file fh);
  */
 s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *newPath);
 
+#if SPIFFS_OBJ_META_LEN
+/**
+ * Updates file's metadata
+ * @param fs            the file system struct
+ * @param path          path to the file
+ * @param meta          new metadata. must be SPIFFS_OBJ_META_LEN bytes long.
+ */
+s32_t SPIFFS_update_meta(spiffs *fs, const char *name, const void *meta);
+
+/**
+ * Updates file's metadata
+ * @param fs            the file system struct
+ * @param fh            file handle of the file
+ * @param meta          new metadata. must be SPIFFS_OBJ_META_LEN bytes long.
+ */
+s32_t SPIFFS_fupdate_meta(spiffs *fs, spiffs_file fh, const void *meta);
+#endif
+
 /**
  * Returns last error of last file operation.
  * @param fs            the file system struct
@@ -648,6 +704,85 @@ s32_t SPIFFS_tell(spiffs *fs, spiffs_file fh);
  */
 s32_t SPIFFS_set_file_callback_func(spiffs *fs, spiffs_file_callback cb_func);
 
+#if SPIFFS_IX_MAP
+
+/**
+ * Maps the first level index lookup to a given memory map.
+ * This will make reading big files faster, as the memory map will be used for
+ * looking up data pages instead of searching for the indices on the physical
+ * medium. When mapping, all affected indicies are found and the information is
+ * copied to the array.
+ * Whole file or only parts of it may be mapped. The index map will cover file
+ * contents from argument offset until and including arguments (offset+len).
+ * It is valid to map a longer range than the current file size. The map will
+ * then be populated when the file grows.
+ * On garbage collections and file data page movements, the map array will be
+ * automatically updated. Do not tamper with the map array, as this contains
+ * the references to the data pages. Modifying it from outside will corrupt any
+ * future readings using this file descriptor.
+ * The map will no longer be used when the file descriptor closed or the file
+ * is unmapped.
+ * This can be useful to get faster and more deterministic timing when reading
+ * large files, or when seeking and reading a lot within a file.
+ * @param fs      the file system struct
+ * @param fh      the file handle of the file to map
+ * @param map     a spiffs_ix_map struct, describing the index map
+ * @param offset  absolute file offset where to start the index map
+ * @param len     length of the mapping in actual file bytes
+ * @param map_buf the array buffer for the look up data - number of required
+ *                elements in the array can be derived from function
+ *                SPIFFS_bytes_to_ix_map_entries given the length
+ */
+s32_t SPIFFS_ix_map(spiffs *fs, spiffs_file fh, spiffs_ix_map *map,
+    u32_t offset, u32_t len, spiffs_page_ix *map_buf);
+
+/**
+ * Unmaps the index lookup from this filehandle. All future readings will
+ * proceed as normal, requiring reading of the first level indices from
+ * physical media.
+ * The map and map buffer given in function SPIFFS_ix_map will no longer be
+ * referenced by spiffs.
+ * It is not strictly necessary to unmap a file before closing it, as closing
+ * a file will automatically unmap it.
+ * @param fs      the file system struct
+ * @param fh      the file handle of the file to unmap
+ */
+s32_t SPIFFS_ix_unmap(spiffs *fs, spiffs_file fh);
+
+/**
+ * Moves the offset for the index map given in function SPIFFS_ix_map. Parts or
+ * all of the map buffer will repopulated.
+ * @param fs      the file system struct
+ * @param fh      the mapped file handle of the file to remap
+ * @param offset  new absolute file offset where to start the index map
+ */
+s32_t SPIFFS_ix_remap(spiffs *fs, spiffs_file fh, u32_t offs);
+
+/**
+ * Utility function to get number of spiffs_page_ix entries a map buffer must
+ * contain on order to map given amount of file data in bytes.
+ * See function SPIFFS_ix_map and SPIFFS_ix_map_entries_to_bytes.
+ * @param fs      the file system struct
+ * @param bytes   number of file data bytes to map
+ * @return        needed number of elements in a spiffs_page_ix array needed to
+ *                map given amount of bytes in a file
+ */
+s32_t SPIFFS_bytes_to_ix_map_entries(spiffs *fs, u32_t bytes);
+
+/**
+ * Utility function to amount of file data bytes that can be mapped when
+ * mapping a file with buffer having given number of spiffs_page_ix entries.
+ * See function SPIFFS_ix_map and SPIFFS_bytes_to_ix_map_entries.
+ * @param fs      the file system struct
+ * @param map_page_ix_entries   number of entries in a spiffs_page_ix array
+ * @return        amount of file data in bytes that can be mapped given a map
+ *                buffer having given amount of spiffs_page_ix entries
+ */
+s32_t SPIFFS_ix_map_entries_to_bytes(spiffs *fs, u32_t map_page_ix_entries);
+
+#endif // SPIFFS_IX_MAP
+
+
 #if SPIFFS_TEST_VISUALISATION
 /**
  * Prints out a visualization of the filesystem.

cores/esp8266/spiffs/spiffs_cache.c

@@ -20,12 +20,12 @@ static spiffs_cache_page *spiffs_cache_page_get(spiffs *fs, spiffs_page_ix pix)
     if ((cache->cpage_use_map & (1<<i)) &&
         (cp->flags & SPIFFS_CACHE_FLAG_TYPE_WR) == 0 &&
         cp->pix == pix ) {
-      SPIFFS_CACHE_DBG("CACHE_GET: have cache page %i for %04x\n", i, pix);
+      //SPIFFS_CACHE_DBG("CACHE_GET: have cache page "_SPIPRIi" for "_SPIPRIpg"\n", i, pix);
       cp->last_access = cache->last_access;
       return cp;
     }
   }
-  //SPIFFS_CACHE_DBG("CACHE_GET: no cache for %04x\n", pix);
+  //SPIFFS_CACHE_DBG("CACHE_GET: no cache for "_SPIPRIpg"\n", pix);
   return 0;
 }
 
@@ -39,17 +39,20 @@ static s32_t spiffs_cache_page_free(spiffs *fs, int ix, u8_t write_back) {
         (cp->flags & SPIFFS_CACHE_FLAG_TYPE_WR) == 0 &&
         (cp->flags & SPIFFS_CACHE_FLAG_DIRTY)) {
       u8_t *mem =  spiffs_get_cache_page(fs, cache, ix);
+      SPIFFS_CACHE_DBG("CACHE_FREE: write cache page "_SPIPRIi" pix "_SPIPRIpg"\n", ix, cp->pix);
       res = SPIFFS_HAL_WRITE(fs, SPIFFS_PAGE_TO_PADDR(fs, cp->pix), SPIFFS_CFG_LOG_PAGE_SZ(fs), mem);
     }
 
-    cp->flags = 0;
+#if SPIFFS_CACHE_WR
-    cache->cpage_use_map &= ~(1 << ix);
-
     if (cp->flags & SPIFFS_CACHE_FLAG_TYPE_WR) {
-      SPIFFS_CACHE_DBG("CACHE_FREE: free cache page %i objid %04x\n", ix, cp->obj_id);
+      SPIFFS_CACHE_DBG("CACHE_FREE: free cache page "_SPIPRIi" objid "_SPIPRIid"\n", ix, cp->obj_id);
-    } else {
+    } else
-      SPIFFS_CACHE_DBG("CACHE_FREE: free cache page %i pix %04x\n", ix, cp->pix);
+#endif
+    {
+      SPIFFS_CACHE_DBG("CACHE_FREE: free cache page "_SPIPRIi" pix "_SPIPRIpg"\n", ix, cp->pix);
     }
+    cache->cpage_use_map &= ~(1 << ix);
+    cp->flags = 0;
   }
 
   return res;
@@ -98,7 +101,7 @@ static spiffs_cache_page *spiffs_cache_page_allocate(spiffs *fs) {
       spiffs_cache_page *cp = spiffs_get_cache_page_hdr(fs, cache, i);
       cache->cpage_use_map |= (1<<i);
       cp->last_access = cache->last_access;
-      SPIFFS_CACHE_DBG("CACHE_ALLO: allocated cache page %i\n", i);
+      //SPIFFS_CACHE_DBG("CACHE_ALLO: allocated cache page "_SPIPRIi"\n", i);
       return cp;
     }
   }
@@ -130,10 +133,13 @@ s32_t spiffs_phys_rd(
   spiffs_cache_page *cp =  spiffs_cache_page_get(fs, SPIFFS_PADDR_TO_PAGE(fs, addr));
   cache->last_access++;
   if (cp) {
+    // we've already got one, you see
 #if SPIFFS_CACHE_STATS
     fs->cache_hits++;
 #endif
     cp->last_access = cache->last_access;
+    u8_t *mem =  spiffs_get_cache_page(fs, cache, cp->ix);
+    _SPIFFS_MEMCPY(dst, &mem[SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr)], len);
   } else {
     if ((op & SPIFFS_OP_TYPE_MASK) == SPIFFS_OP_T_OBJ_LU2) {
       // for second layer lookup functions, we do not cache in order to prevent shredding
@@ -142,22 +148,35 @@ s32_t spiffs_phys_rd(
 #if SPIFFS_CACHE_STATS
     fs->cache_misses++;
 #endif
+    // this operation will always free one cache page (unless all already free),
+    // the result code stems from the write operation of the possibly freed cache page
     res = spiffs_cache_page_remove_oldest(fs, SPIFFS_CACHE_FLAG_TYPE_WR, 0);
+
     cp = spiffs_cache_page_allocate(fs);
     if (cp) {
       cp->flags = SPIFFS_CACHE_FLAG_WRTHRU;
       cp->pix = SPIFFS_PADDR_TO_PAGE(fs, addr);
-    }
+      SPIFFS_CACHE_DBG("CACHE_ALLO: allocated cache page "_SPIPRIi" for pix "_SPIPRIpg "\n", cp->ix, cp->pix);
-    s32_t res2 = SPIFFS_HAL_READ(fs,
+
-        addr - SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr),
+      s32_t res2 = SPIFFS_HAL_READ(fs,
-        SPIFFS_CFG_LOG_PAGE_SZ(fs),
+          addr - SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr),
-        spiffs_get_cache_page(fs, cache, cp->ix));
+          SPIFFS_CFG_LOG_PAGE_SZ(fs),
-    if (res2 != SPIFFS_OK) {
+          spiffs_get_cache_page(fs, cache, cp->ix));
-      res = res2;
+      if (res2 != SPIFFS_OK) {
+        // honor read failure before possible write failure (bad idea?)
+        res = res2;
+      }
+      u8_t *mem =  spiffs_get_cache_page(fs, cache, cp->ix);
+      _SPIFFS_MEMCPY(dst, &mem[SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr)], len);
+    } else {
+      // this will never happen, last resort for sake of symmetry
+      s32_t res2 = SPIFFS_HAL_READ(fs, addr, len, dst);
+      if (res2 != SPIFFS_OK) {
+        // honor read failure before possible write failure (bad idea?)
+        res = res2;
+      }
     }
   }
-  u8_t *mem =  spiffs_get_cache_page(fs, cache, cp->ix);
-  memcpy(dst, &mem[SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr)], len);
   return res;
 }
 
@@ -186,7 +205,7 @@ s32_t spiffs_phys_wr(
     }
 
     u8_t *mem =  spiffs_get_cache_page(fs, cache, cp->ix);
-    memcpy(&mem[SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr)], src, len);
+    _SPIFFS_MEMCPY(&mem[SPIFFS_PADDR_TO_PAGE_OFFSET(fs, addr)], src, len);
 
     cache->last_access++;
     cp->last_access = cache->last_access;
@@ -241,6 +260,7 @@ spiffs_cache_page *spiffs_cache_page_allocate_by_fd(spiffs *fs, spiffs_fd *fd) {
   cp->flags = SPIFFS_CACHE_FLAG_TYPE_WR;
   cp->obj_id = fd->obj_id;
   fd->cache_page = cp;
+  SPIFFS_CACHE_DBG("CACHE_ALLO: allocated cache page "_SPIPRIi" for fd "_SPIPRIfd ":"_SPIPRIid "\n", cp->ix, fd->file_nbr, fd->obj_id);
   return cp;
 }
 
@@ -284,7 +304,7 @@ void spiffs_cache_init(spiffs *fs) {
 
   cache.cpage_use_map = 0xffffffff;
   cache.cpage_use_mask = cache_mask;
-  memcpy(fs->cache, &cache, sizeof(spiffs_cache));
+  _SPIFFS_MEMCPY(fs->cache, &cache, sizeof(spiffs_cache));
 
   spiffs_cache *c = spiffs_get_cache(fs);
 

cores/esp8266/spiffs/spiffs_check.c

@@ -182,7 +182,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
   if (((lu_obj_id == SPIFFS_OBJ_ID_DELETED) && (p_hdr->flags & SPIFFS_PH_FLAG_DELET)) ||
       ((lu_obj_id == SPIFFS_OBJ_ID_FREE) && (p_hdr->flags & SPIFFS_PH_FLAG_USED) == 0)) {
     // look up entry deleted / free but used in page header
-    SPIFFS_CHECK_DBG("LU: pix %04x deleted/free in lu but not on page\n", cur_pix);
+    SPIFFS_CHECK_DBG("LU: pix "_SPIPRIpg" deleted/free in lu but not on page\n", cur_pix);
     *reload_lu = 1;
     delete_page = 1;
     if (p_hdr->flags & SPIFFS_PH_FLAG_INDEX) {
@@ -199,14 +199,14 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
           // copy page to new place and re-write the object index to new place
           spiffs_page_ix new_pix;
           res = spiffs_rewrite_page(fs, cur_pix, p_hdr, &new_pix);
-          SPIFFS_CHECK_DBG("LU: FIXUP: data page not found elsewhere, rewriting %04x to new page %04x\n", cur_pix, new_pix);
+          SPIFFS_CHECK_DBG("LU: FIXUP: data page not found elsewhere, rewriting "_SPIPRIpg" to new page "_SPIPRIpg"\n", cur_pix, new_pix);
           SPIFFS_CHECK_RES(res);
           *reload_lu = 1;
-          SPIFFS_CHECK_DBG("LU: FIXUP: %04x rewritten to %04x, affected objix_pix %04x\n", cur_pix, new_pix, objix_pix);
+          SPIFFS_CHECK_DBG("LU: FIXUP: "_SPIPRIpg" rewritten to "_SPIPRIpg", affected objix_pix "_SPIPRIpg"\n", cur_pix, new_pix, objix_pix);
           res = spiffs_rewrite_index(fs, p_hdr->obj_id, p_hdr->span_ix, new_pix, objix_pix);
           if (res <= _SPIFFS_ERR_CHECK_FIRST && res > _SPIFFS_ERR_CHECK_LAST) {
             // index bad also, cannot mend this file
-            SPIFFS_CHECK_DBG("LU: FIXUP: index bad %i, cannot mend!\n", res);
+            SPIFFS_CHECK_DBG("LU: FIXUP: index bad "_SPIPRIi", cannot mend!\n", res);
             res = spiffs_page_delete(fs, new_pix);
             SPIFFS_CHECK_RES(res);
             res = spiffs_delete_obj_lazy(fs, p_hdr->obj_id);
@@ -229,7 +229,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
           // got a data page also, assume lu corruption only, rewrite to new page
           spiffs_page_ix new_pix;
           res = spiffs_rewrite_page(fs, cur_pix, p_hdr, &new_pix);
-          SPIFFS_CHECK_DBG("LU: FIXUP: ix page with data not found elsewhere, rewriting %04x to new page %04x\n", cur_pix, new_pix);
+          SPIFFS_CHECK_DBG("LU: FIXUP: ix page with data not found elsewhere, rewriting "_SPIPRIpg" to new page "_SPIPRIpg"\n", cur_pix, new_pix);
           SPIFFS_CHECK_RES(res);
           *reload_lu = 1;
           CHECK_CB(fs, SPIFFS_CHECK_LOOKUP, SPIFFS_CHECK_FIX_LOOKUP, p_hdr->obj_id, p_hdr->span_ix);
@@ -242,7 +242,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
   if (lu_obj_id != SPIFFS_OBJ_ID_FREE && lu_obj_id != SPIFFS_OBJ_ID_DELETED) {
     // look up entry used
     if ((p_hdr->obj_id | SPIFFS_OBJ_ID_IX_FLAG) != (lu_obj_id | SPIFFS_OBJ_ID_IX_FLAG)) {
-      SPIFFS_CHECK_DBG("LU: pix %04x differ in obj_id lu:%04x ph:%04x\n", cur_pix, lu_obj_id, p_hdr->obj_id);
+      SPIFFS_CHECK_DBG("LU: pix "_SPIPRIpg" differ in obj_id lu:"_SPIPRIid" ph:"_SPIPRIid"\n", cur_pix, lu_obj_id, p_hdr->obj_id);
       delete_page = 1;
       if ((p_hdr->flags & SPIFFS_PH_FLAG_DELET) == 0 ||
           (p_hdr->flags & SPIFFS_PH_FLAG_FINAL) ||
@@ -265,7 +265,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
               res = spiffs_rewrite_index(fs, p_hdr->obj_id, p_hdr->span_ix, new_pix, objix_pix);
               if (res <= _SPIFFS_ERR_CHECK_FIRST && res > _SPIFFS_ERR_CHECK_LAST) {
                 // index bad also, cannot mend this file
-                SPIFFS_CHECK_DBG("LU: FIXUP: index bad %i, cannot mend!\n", res);
+                SPIFFS_CHECK_DBG("LU: FIXUP: index bad "_SPIPRIi", cannot mend!\n", res);
                 res = spiffs_page_delete(fs, new_pix);
                 SPIFFS_CHECK_RES(res);
                 res = spiffs_delete_obj_lazy(fs, p_hdr->obj_id);
@@ -321,7 +321,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
               //   rewrite as obj_id_ph
               new_ph.obj_id = p_hdr->obj_id | SPIFFS_OBJ_ID_IX_FLAG;
               res = spiffs_rewrite_page(fs, cur_pix, &new_ph, &new_pix);
-              SPIFFS_CHECK_DBG("LU: FIXUP: rewrite page %04x as %04x to pix %04x\n", cur_pix, new_ph.obj_id, new_pix);
+              SPIFFS_CHECK_DBG("LU: FIXUP: rewrite page "_SPIPRIpg" as "_SPIPRIid" to pix "_SPIPRIpg"\n", cur_pix, new_ph.obj_id, new_pix);
               CHECK_CB(fs, SPIFFS_CHECK_LOOKUP, SPIFFS_CHECK_FIX_LOOKUP, p_hdr->obj_id, p_hdr->span_ix);
               SPIFFS_CHECK_RES(res);
               *reload_lu = 1;
@@ -330,7 +330,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
               //   got a data page for look up obj id
               //   rewrite as obj_id_lu
               new_ph.obj_id =  lu_obj_id | SPIFFS_OBJ_ID_IX_FLAG;
-              SPIFFS_CHECK_DBG("LU: FIXUP: rewrite page %04x as %04x\n", cur_pix, new_ph.obj_id);
+              SPIFFS_CHECK_DBG("LU: FIXUP: rewrite page "_SPIPRIpg" as "_SPIPRIid"\n", cur_pix, new_ph.obj_id);
               CHECK_CB(fs, SPIFFS_CHECK_LOOKUP, SPIFFS_CHECK_FIX_LOOKUP, p_hdr->obj_id, p_hdr->span_ix);
               res = spiffs_rewrite_page(fs, cur_pix, &new_ph, &new_pix);
               SPIFFS_CHECK_RES(res);
@@ -344,7 +344,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
       }
     } else if (((lu_obj_id & SPIFFS_OBJ_ID_IX_FLAG) && (p_hdr->flags & SPIFFS_PH_FLAG_INDEX)) ||
         ((lu_obj_id & SPIFFS_OBJ_ID_IX_FLAG) == 0 && (p_hdr->flags & SPIFFS_PH_FLAG_INDEX) == 0)) {
-      SPIFFS_CHECK_DBG("LU: %04x lu/page index marking differ\n", cur_pix);
+      SPIFFS_CHECK_DBG("LU: "_SPIPRIpg" lu/page index marking differ\n", cur_pix);
       spiffs_page_ix data_pix, objix_pix_d;
       // see if other data page exists for given obj id and span index
       res = spiffs_obj_lu_find_id_and_span(fs, lu_obj_id & ~SPIFFS_OBJ_ID_IX_FLAG, p_hdr->span_ix, cur_pix, &data_pix);
@@ -402,10 +402,10 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
       }
     }
     else if ((p_hdr->flags & SPIFFS_PH_FLAG_DELET) == 0) {
-      SPIFFS_CHECK_DBG("LU: pix %04x busy in lu but deleted on page\n", cur_pix);
+      SPIFFS_CHECK_DBG("LU: pix "_SPIPRIpg" busy in lu but deleted on page\n", cur_pix);
       delete_page = 1;
     } else if ((p_hdr->flags & SPIFFS_PH_FLAG_FINAL)) {
-      SPIFFS_CHECK_DBG("LU: pix %04x busy but not final\n", cur_pix);
+      SPIFFS_CHECK_DBG("LU: pix "_SPIPRIpg" busy but not final\n", cur_pix);
       // page can be removed if not referenced by object index
       *reload_lu = 1;
       res = spiffs_object_get_data_page_index_reference(fs, lu_obj_id, p_hdr->span_ix, &ref_pix, &objix_pix);
@@ -433,7 +433,7 @@ static s32_t spiffs_lookup_check_validate(spiffs *fs, spiffs_obj_id lu_obj_id, s
   }
 
   if (delete_page) {
-    SPIFFS_CHECK_DBG("LU: FIXUP: deleting page %04x\n", cur_pix);
+    SPIFFS_CHECK_DBG("LU: FIXUP: deleting page "_SPIPRIpg"\n", cur_pix);
     CHECK_CB(fs, SPIFFS_CHECK_LOOKUP, SPIFFS_CHECK_DELETE_PAGE, cur_pix, 0);
     res = spiffs_page_delete(fs, cur_pix);
     SPIFFS_CHECK_RES(res);
@@ -530,7 +530,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
       spiffs_page_ix cur_pix = SPIFFS_OBJ_LOOKUP_PAGES(fs) + SPIFFS_PAGES_PER_BLOCK(fs) * cur_block;
       while (!restart && cur_pix < SPIFFS_PAGES_PER_BLOCK(fs) * (cur_block+1)) {
         //if ((cur_pix & 0xff) == 0)
-        //  SPIFFS_CHECK_DBG("PA: processing pix %08x, block %08x of pix %08x, block %08x\n",
+        //  SPIFFS_CHECK_DBG("PA: processing pix "_SPIPRIpg", block "_SPIPRIbl" of pix "_SPIPRIpg", block "_SPIPRIbl"\n",
         //      cur_pix, cur_block, SPIFFS_PAGES_PER_BLOCK(fs) * fs->block_count, fs->block_count);
 
         // read header
@@ -589,7 +589,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                 || (rpix_within_range && SPIFFS_IS_LOOKUP_PAGE(fs, rpix))) {
 
               // bad reference
-              SPIFFS_CHECK_DBG("PA: pix %04x bad pix / LU referenced from page %04x\n",
+              SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg"x bad pix / LU referenced from page "_SPIPRIpg"\n",
                   rpix, cur_pix);
               // check for data page elsewhere
               spiffs_page_ix data_pix;
@@ -608,15 +608,15 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                 new_ph.span_ix = data_spix_offset + i;
                 res = spiffs_page_allocate_data(fs, new_ph.obj_id, &new_ph, 0, 0, 0, 1, &data_pix);
                 SPIFFS_CHECK_RES(res);
-                SPIFFS_CHECK_DBG("PA: FIXUP: found no existing data page, created new @ %04x\n", data_pix);
+                SPIFFS_CHECK_DBG("PA: FIXUP: found no existing data page, created new @ "_SPIPRIpg"\n", data_pix);
               }
               // remap index
-              SPIFFS_CHECK_DBG("PA: FIXUP: rewriting index pix %04x\n", cur_pix);
+              SPIFFS_CHECK_DBG("PA: FIXUP: rewriting index pix "_SPIPRIpg"\n", cur_pix);
               res = spiffs_rewrite_index(fs, objix_p_hdr->obj_id | SPIFFS_OBJ_ID_IX_FLAG,
                   data_spix_offset + i, data_pix, cur_pix);
               if (res <= _SPIFFS_ERR_CHECK_FIRST && res > _SPIFFS_ERR_CHECK_LAST) {
                 // index bad also, cannot mend this file
-                SPIFFS_CHECK_DBG("PA: FIXUP: index bad %i, cannot mend - delete object\n", res);
+                SPIFFS_CHECK_DBG("PA: FIXUP: index bad "_SPIPRIi", cannot mend - delete object\n", res);
                 CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_BAD_FILE, objix_p_hdr->obj_id, 0);
                 // delete file
                 res = spiffs_page_delete(fs, cur_pix);
@@ -640,7 +640,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                   rp_hdr.span_ix != data_spix_offset + i ||
                   (rp_hdr.flags & (SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_INDEX | SPIFFS_PH_FLAG_USED)) !=
                       (SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_INDEX)) {
-               SPIFFS_CHECK_DBG("PA: pix %04x has inconsistent page header ix id/span:%04x/%04x, ref id/span:%04x/%04x flags:%02x\n",
+               SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" has inconsistent page header ix id/span:"_SPIPRIid"/"_SPIPRIsp", ref id/span:"_SPIPRIid"/"_SPIPRIsp" flags:"_SPIPRIfl"\n",
                     rpix, p_hdr.obj_id & ~SPIFFS_OBJ_ID_IX_FLAG, data_spix_offset + i,
                     rp_hdr.obj_id, rp_hdr.span_ix, rp_hdr.flags);
                // try finding correct page
@@ -654,19 +654,19 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                SPIFFS_CHECK_RES(res);
                if (data_pix == 0) {
                  // not found, this index is badly borked
-                 SPIFFS_CHECK_DBG("PA: FIXUP: index bad, delete object id %04x\n", p_hdr.obj_id);
+                 SPIFFS_CHECK_DBG("PA: FIXUP: index bad, delete object id "_SPIPRIid"\n", p_hdr.obj_id);
                  CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_BAD_FILE, p_hdr.obj_id, 0);
                  res = spiffs_delete_obj_lazy(fs, p_hdr.obj_id);
                  SPIFFS_CHECK_RES(res);
                  break;
                } else {
                  // found it, so rewrite index
-                 SPIFFS_CHECK_DBG("PA: FIXUP: found correct data pix %04x, rewrite ix pix %04x id %04x\n",
+                 SPIFFS_CHECK_DBG("PA: FIXUP: found correct data pix "_SPIPRIpg", rewrite ix pix "_SPIPRIpg" id "_SPIPRIid"\n",
                      data_pix, cur_pix, p_hdr.obj_id);
                  res = spiffs_rewrite_index(fs, p_hdr.obj_id, data_spix_offset + i, data_pix, cur_pix);
                  if (res <= _SPIFFS_ERR_CHECK_FIRST && res > _SPIFFS_ERR_CHECK_LAST) {
                    // index bad also, cannot mend this file
-                   SPIFFS_CHECK_DBG("PA: FIXUP: index bad %i, cannot mend!\n", res);
+                   SPIFFS_CHECK_DBG("PA: FIXUP: index bad "_SPIPRIi", cannot mend!\n", res);
                    CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_BAD_FILE, p_hdr.obj_id, 0);
                    res = spiffs_delete_obj_lazy(fs, p_hdr.obj_id);
                  } else {
@@ -681,12 +681,12 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                 const u32_t rpix_byte_ix = (rpix - pix_offset) / (8/bits);
                 const u8_t rpix_bit_ix = (rpix & ((8/bits)-1)) * bits;
                 if (fs->work[rpix_byte_ix] & (1<<(rpix_bit_ix + 1))) {
-                  SPIFFS_CHECK_DBG("PA: pix %04x multiple referenced from page %04x\n",
+                  SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" multiple referenced from page "_SPIPRIpg"\n",
                       rpix, cur_pix);
                   // Here, we should have fixed all broken references - getting this means there
                   // must be multiple files with same object id. Only solution is to delete
                   // the object which is referring to this page
-                  SPIFFS_CHECK_DBG("PA: FIXUP: removing object %04x and page %04x\n",
+                  SPIFFS_CHECK_DBG("PA: FIXUP: removing object "_SPIPRIid" and page "_SPIPRIpg"\n",
                       p_hdr.obj_id, cur_pix);
                   CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_BAD_FILE, p_hdr.obj_id, 0);
                   res = spiffs_delete_obj_lazy(fs, p_hdr.obj_id);
@@ -725,7 +725,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
           if (bitmask == 0x1) {
 
             // 001
-            SPIFFS_CHECK_DBG("PA: pix %04x USED, UNREFERENCED, not index\n", cur_pix);
+            SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" USED, UNREFERENCED, not index\n", cur_pix);
 
             u8_t rewrite_ix_to_this = 0;
             u8_t delete_page = 0;
@@ -741,7 +741,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
               if (((rpix == (spiffs_page_ix)-1 || rpix > SPIFFS_MAX_PAGES(fs)) || (SPIFFS_IS_LOOKUP_PAGE(fs, rpix)))) {
                 // pointing to a bad page altogether, rewrite index to this
                 rewrite_ix_to_this = 1;
-                SPIFFS_CHECK_DBG("PA: corresponding ref is bad: %04x, rewrite to this %04x\n", rpix, cur_pix);
+                SPIFFS_CHECK_DBG("PA: corresponding ref is bad: "_SPIPRIpg", rewrite to this "_SPIPRIpg"\n", rpix, cur_pix);
               } else {
                 // pointing to something else, check what
                 spiffs_page_header rp_hdr;
@@ -752,12 +752,12 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                     ((rp_hdr.flags & (SPIFFS_PH_FLAG_INDEX | SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_USED | SPIFFS_PH_FLAG_FINAL)) ==
                         (SPIFFS_PH_FLAG_INDEX | SPIFFS_PH_FLAG_DELET))) {
                   // pointing to something else valid, just delete this page then
-                  SPIFFS_CHECK_DBG("PA: corresponding ref is good but different: %04x, delete this %04x\n", rpix, cur_pix);
+                  SPIFFS_CHECK_DBG("PA: corresponding ref is good but different: "_SPIPRIpg", delete this "_SPIPRIpg"\n", rpix, cur_pix);
                   delete_page = 1;
                 } else {
                   // pointing to something weird, update index to point to this page instead
                   if (rpix != cur_pix) {
-                    SPIFFS_CHECK_DBG("PA: corresponding ref is weird: %04x %s%s%s%s, rewrite this %04x\n", rpix,
+                    SPIFFS_CHECK_DBG("PA: corresponding ref is weird: "_SPIPRIpg" %s%s%s%s, rewrite this "_SPIPRIpg"\n", rpix,
                         (rp_hdr.flags & SPIFFS_PH_FLAG_INDEX) ? "" : "INDEX ",
                             (rp_hdr.flags & SPIFFS_PH_FLAG_DELET) ? "" : "DELETED ",
                                 (rp_hdr.flags & SPIFFS_PH_FLAG_USED) ? "NOTUSED " : "",
@@ -770,19 +770,19 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
                 }
               }
             } else if (res == SPIFFS_ERR_NOT_FOUND) {
-              SPIFFS_CHECK_DBG("PA: corresponding ref not found, delete %04x\n", cur_pix);
+              SPIFFS_CHECK_DBG("PA: corresponding ref not found, delete "_SPIPRIpg"\n", cur_pix);
               delete_page = 1;
               res = SPIFFS_OK;
             }
 
             if (rewrite_ix_to_this) {
               // if pointing to invalid page, redirect index to this page
-              SPIFFS_CHECK_DBG("PA: FIXUP: rewrite index id %04x data spix %04x to point to this pix: %04x\n",
+              SPIFFS_CHECK_DBG("PA: FIXUP: rewrite index id "_SPIPRIid" data spix "_SPIPRIsp" to point to this pix: "_SPIPRIpg"\n",
                   p_hdr.obj_id, p_hdr.span_ix, cur_pix);
               res = spiffs_rewrite_index(fs, p_hdr.obj_id, p_hdr.span_ix, cur_pix, objix_pix);
               if (res <= _SPIFFS_ERR_CHECK_FIRST && res > _SPIFFS_ERR_CHECK_LAST) {
                 // index bad also, cannot mend this file
-                SPIFFS_CHECK_DBG("PA: FIXUP: index bad %i, cannot mend!\n", res);
+                SPIFFS_CHECK_DBG("PA: FIXUP: index bad "_SPIPRIi", cannot mend!\n", res);
                 CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_BAD_FILE, p_hdr.obj_id, 0);
                 res = spiffs_page_delete(fs, cur_pix);
                 SPIFFS_CHECK_RES(res);
@@ -794,7 +794,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
               restart = 1;
               continue;
             } else if (delete_page) {
-              SPIFFS_CHECK_DBG("PA: FIXUP: deleting page %04x\n", cur_pix);
+              SPIFFS_CHECK_DBG("PA: FIXUP: deleting page "_SPIPRIpg"\n", cur_pix);
               CHECK_CB(fs, SPIFFS_CHECK_PAGE, SPIFFS_CHECK_DELETE_PAGE, cur_pix, 0);
               res = spiffs_page_delete(fs, cur_pix);
             }
@@ -803,7 +803,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
           if (bitmask == 0x2) {
 
             // 010
-            SPIFFS_CHECK_DBG("PA: pix %04x FREE, REFERENCED, not index\n", cur_pix);
+            SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" FREE, REFERENCED, not index\n", cur_pix);
 
             // no op, this should be taken care of when checking valid references
           }
@@ -813,7 +813,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
           if (bitmask == 0x4) {
 
             // 100
-            SPIFFS_CHECK_DBG("PA: pix %04x FREE, unreferenced, INDEX\n", cur_pix);
+            SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" FREE, unreferenced, INDEX\n", cur_pix);
 
             // this should never happen, major fubar
           }
@@ -823,14 +823,14 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
           if (bitmask == 0x6) {
 
             // 110
-            SPIFFS_CHECK_DBG("PA: pix %04x FREE, REFERENCED, INDEX\n", cur_pix);
+            SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" FREE, REFERENCED, INDEX\n", cur_pix);
 
             // no op, this should be taken care of when checking valid references
           }
           if (bitmask == 0x7) {
 
             // 111
-            SPIFFS_CHECK_DBG("PA: pix %04x USED, REFERENCED, INDEX\n", cur_pix);
+            SPIFFS_CHECK_DBG("PA: pix "_SPIPRIpg" USED, REFERENCED, INDEX\n", cur_pix);
 
             // no op, this should be taken care of when checking valid references
           }
@@ -838,7 +838,7 @@ static s32_t spiffs_page_consistency_check_i(spiffs *fs) {
       }
     }
 
-    SPIFFS_CHECK_DBG("PA: processed %04x, restart %i\n", pix_offset, restart);
+    SPIFFS_CHECK_DBG("PA: processed "_SPIPRIpg", restart "_SPIPRIi"\n", pix_offset, restart);
     // next page range
     if (!restart) {
       pix_offset += pages_per_scan;
@@ -898,7 +898,7 @@ static s32_t spiffs_object_index_consistency_check_v(spiffs *fs, spiffs_obj_id o
     if (p_hdr.span_ix == 0 &&
         (p_hdr.flags & (SPIFFS_PH_FLAG_INDEX | SPIFFS_PH_FLAG_FINAL | SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_IXDELE)) ==
         (SPIFFS_PH_FLAG_DELET)) {
-      SPIFFS_CHECK_DBG("IX: pix %04x, obj id:%04x spix:%04x header not fully deleted - deleting\n",
+      SPIFFS_CHECK_DBG("IX: pix "_SPIPRIpg", obj id:"_SPIPRIid" spix:"_SPIPRIsp" header not fully deleted - deleting\n",
           cur_pix, obj_id, p_hdr.span_ix);
       CHECK_CB(fs, SPIFFS_CHECK_INDEX, SPIFFS_CHECK_DELETE_PAGE, cur_pix, obj_id);
       res = spiffs_page_delete(fs, cur_pix);
@@ -954,7 +954,7 @@ static s32_t spiffs_object_index_consistency_check_v(spiffs *fs, spiffs_obj_id o
       }
 
       if (delete) {
-        SPIFFS_CHECK_DBG("IX: FIXUP: pix %04x, obj id:%04x spix:%04x is orphan index - deleting\n",
+        SPIFFS_CHECK_DBG("IX: FIXUP: pix "_SPIPRIpg", obj id:"_SPIPRIid" spix:"_SPIPRIsp" is orphan index - deleting\n",
             cur_pix, obj_id, p_hdr.span_ix);
         CHECK_CB(fs, SPIFFS_CHECK_INDEX, SPIFFS_CHECK_DELETE_ORPHANED_INDEX, cur_pix, obj_id);
         res = spiffs_page_delete(fs, cur_pix);
@@ -980,8 +980,8 @@ s32_t spiffs_object_index_consistency_check(spiffs *fs) {
   memset(fs->work, 0, SPIFFS_CFG_LOG_PAGE_SZ(fs));
   u32_t obj_id_log_ix = 0;
   CHECK_CB(fs, SPIFFS_CHECK_INDEX, SPIFFS_CHECK_PROGRESS, 0, 0);
-  res = spiffs_obj_lu_find_entry_visitor(fs, 0, 0, 0, 0, spiffs_object_index_consistency_check_v, &obj_id_log_ix,
+  res = spiffs_obj_lu_find_entry_visitor(fs, 0, 0, 0, 0, spiffs_object_index_consistency_check_v, 0, &obj_id_log_ix,
-      0, 0, 0);
+        0, 0);
   if (res == SPIFFS_VIS_END) {
     res = SPIFFS_OK;
   }

cores/esp8266/spiffs/spiffs_config.h

@@ -59,6 +59,44 @@ typedef uint8_t u8_t;
 #ifndef SPIFFS_CHECK_DBG
 #define SPIFFS_CHECK_DBG(...) //printf(__VA_ARGS__)
 #endif
+// Set spiffs debug output call for all api invocations.
+#ifndef SPIFFS_API_DBG
+#define SPIFFS_API_DBG(...) //printf(__VA_ARGS__)
+#endif
+
+// Defines spiffs debug print formatters
+// some general signed number
+#ifndef _SPIPRIi
+#define _SPIPRIi   "%d"
+#endif
+// address
+#ifndef _SPIPRIad
+#define _SPIPRIad  "%08x"
+#endif
+// block
+#ifndef _SPIPRIbl
+#define _SPIPRIbl  "%04x"
+#endif
+// page
+#ifndef _SPIPRIpg
+#define _SPIPRIpg  "%04x"
+#endif
+// span index
+#ifndef _SPIPRIsp
+#define _SPIPRIsp  "%04x"
+#endif
+// file descriptor
+#ifndef _SPIPRIfd
+#define _SPIPRIfd  "%d"
+#endif
+// file object id
+#ifndef _SPIPRIid
+#define _SPIPRIid  "%04x"
+#endif
+// file flags
+#ifndef _SPIPRIfl
+#define _SPIPRIfl  "%02x"
+#endif
 
 // Enable/disable API functions to determine exact number of bytes
 // for filedescriptor and cache buffers. Once decided for a configuration,
@@ -127,6 +165,20 @@ typedef uint8_t u8_t;
 #define SPIFFS_OBJ_NAME_LEN             (32)
 #endif
 
+// Maximum length of the metadata associated with an object.
+// Setting to non-zero value enables metadata-related API but also
+// changes the on-disk format, so the change is not backward-compatible.
+//
+// Do note: the meta length must never exceed
+// logical_page_size - (SPIFFS_OBJ_NAME_LEN + 64)
+//
+// This is derived from following:
+// logical_page_size - (SPIFFS_OBJ_NAME_LEN + sizeof(spiffs_page_header) +
+// spiffs_object_ix_header fields + at least some LUT entries)
+#ifndef SPIFFS_OBJ_META_LEN
+#define SPIFFS_OBJ_META_LEN             (0)
+#endif
+
 // Size of buffer allocated on stack used when copying data.
 // Lower value generates more read/writes. No meaning having it bigger
 // than logical page size.
@@ -142,6 +194,17 @@ typedef uint8_t u8_t;
 #define SPIFFS_USE_MAGIC                (1)
 #endif
 
+#if SPIFFS_USE_MAGIC
+// Only valid when SPIFFS_USE_MAGIC is enabled. If SPIFFS_USE_MAGIC_LENGTH is
+// enabled, the magic will also be dependent on the length of the filesystem.
+// For example, a filesystem configured and formatted for 4 megabytes will not
+// be accepted for mounting with a configuration defining the filesystem as 2
+// megabytes.
+#ifndef SPIFFS_USE_MAGIC_LENGTH
+#define SPIFFS_USE_MAGIC_LENGTH         (0)
+#endif
+#endif
+
 // SPIFFS_LOCK and SPIFFS_UNLOCK protects spiffs from reentrancy on api level
 // These should be defined on a multithreaded system
 
@@ -202,6 +265,67 @@ typedef uint8_t u8_t;
 #define SPIFFS_FILEHDL_OFFSET                 0
 #endif
 
+// Enable this to compile a read only version of spiffs.
+// This will reduce binary size of spiffs. All code comprising modification
+// of the file system will not be compiled. Some config will be ignored.
+// HAL functions for erasing and writing to spi-flash may be null. Cache
+// can be disabled for even further binary size reduction (and ram savings).
+// Functions modifying the fs will return SPIFFS_ERR_RO_NOT_IMPL.
+// If the file system cannot be mounted due to aborted erase operation and
+// SPIFFS_USE_MAGIC is enabled, SPIFFS_ERR_RO_ABORTED_OPERATION will be
+// returned.
+// Might be useful for e.g. bootloaders and such.
+#ifndef SPIFFS_READ_ONLY
+#define SPIFFS_READ_ONLY                      0
+#endif
+
+// Enable this to add a temporal file cache using the fd buffer.
+// The effects of the cache is that SPIFFS_open will find the file faster in
+// certain cases. It will make it a lot easier for spiffs to find files
+// opened frequently, reducing number of readings from the spi flash for
+// finding those files.
+// This will grow each fd by 6 bytes. If your files are opened in patterns
+// with a degree of temporal locality, the system is optimized.
+// Examples can be letting spiffs serve web content, where one file is the css.
+// The css is accessed for each html file that is opened, meaning it is
+// accessed almost every second time a file is opened. Another example could be
+// a log file that is often opened, written, and closed.
+// The size of the cache is number of given file descriptors, as it piggybacks
+// on the fd update mechanism. The cache lives in the closed file descriptors.
+// When closed, the fd know the whereabouts of the file. Instead of forgetting
+// this, the temporal cache will keep handling updates to that file even if the
+// fd is closed. If the file is opened again, the location of the file is found
+// directly. If all available descriptors become opened, all cache memory is
+// lost.
+#ifndef SPIFFS_TEMPORAL_FD_CACHE
+#define SPIFFS_TEMPORAL_FD_CACHE              1
+#endif
+
+// Temporal file cache hit score. Each time a file is opened, all cached files
+// will lose one point. If the opened file is found in cache, that entry will
+// gain SPIFFS_TEMPORAL_CACHE_HIT_SCORE points. One can experiment with this
+// value for the specific access patterns of the application. However, it must
+// be between 1 (no gain for hitting a cached entry often) and 255.
+#ifndef SPIFFS_TEMPORAL_CACHE_HIT_SCORE
+#define SPIFFS_TEMPORAL_CACHE_HIT_SCORE       4
+#endif
+
+// Enable to be able to map object indices to memory.
+// This allows for faster and more deterministic reading if cases of reading
+// large files and when changing file offset by seeking around a lot.
+// When mapping a file's index, the file system will be scanned for index pages
+// and the info will be put in memory provided by user. When reading, the
+// memory map can be looked up instead of searching for index pages on the
+// medium. This way, user can trade memory against performance.
+// Whole, parts of, or future parts not being written yet can be mapped. The
+// memory array will be owned by spiffs and updated accordingly during garbage
+// collecting or when modifying the indices. The latter is invoked by when the
+// file is modified in some way. The index buffer is tied to the file
+// descriptor.
+#ifndef SPIFFS_IX_MAP
+#define SPIFFS_IX_MAP                         1
+#endif
+
 // Set SPIFFS_TEST_VISUALISATION to non-zero to enable SPIFFS_vis function
 // in the api. This function will visualize all filesystem using given printf
 // function.

cores/esp8266/spiffs/spiffs_gc.c

@@ -11,7 +11,7 @@ static s32_t spiffs_gc_erase_block(
     spiffs_block_ix bix) {
   s32_t res;
 
-  SPIFFS_GC_DBG("gc: erase block %i\n", bix);
+  SPIFFS_GC_DBG("gc: erase block "_SPIPRIbl"\n", bix);
   res = spiffs_erase_block(fs, bix);
   SPIFFS_CHECK_RES(res);
 
@@ -122,19 +122,19 @@ s32_t spiffs_gc_check(
 
   u32_t needed_pages = (len + SPIFFS_DATA_PAGE_SIZE(fs) - 1) / SPIFFS_DATA_PAGE_SIZE(fs);
 //  if (fs->free_blocks <= 2 && (s32_t)needed_pages > free_pages) {
-//    SPIFFS_GC_DBG("gc: full freeblk:%i needed:%i free:%i dele:%i\n", fs->free_blocks, needed_pages, free_pages, fs->stats_p_deleted);
+//    SPIFFS_GC_DBG("gc: full freeblk:"_SPIPRIi" needed:"_SPIPRIi" free:"_SPIPRIi" dele:"_SPIPRIi"\n", fs->free_blocks, needed_pages, free_pages, fs->stats_p_deleted);
 //    return SPIFFS_ERR_FULL;
 //  }
   if ((s32_t)needed_pages > (s32_t)(free_pages + fs->stats_p_deleted)) {
-    SPIFFS_GC_DBG("gc_check: full freeblk:%i needed:%i free:%i dele:%i\n", fs->free_blocks, needed_pages, free_pages, fs->stats_p_deleted);
+    SPIFFS_GC_DBG("gc_check: full freeblk:"_SPIPRIi" needed:"_SPIPRIi" free:"_SPIPRIi" dele:"_SPIPRIi"\n", fs->free_blocks, needed_pages, free_pages, fs->stats_p_deleted);
     return SPIFFS_ERR_FULL;
   }
 
   do {
-    SPIFFS_GC_DBG("\ngc_check #%i: run gc free_blocks:%i pfree:%i pallo:%i pdele:%i [%i] len:%i of %i\n",
+    SPIFFS_GC_DBG("\ngc_check #"_SPIPRIi": run gc free_blocks:"_SPIPRIi" pfree:"_SPIPRIi" pallo:"_SPIPRIi" pdele:"_SPIPRIi" ["_SPIPRIi"] len:"_SPIPRIi" of "_SPIPRIi"\n",
         tries,
         fs->free_blocks, free_pages, fs->stats_p_allocated, fs->stats_p_deleted, (free_pages+fs->stats_p_allocated+fs->stats_p_deleted),
-        len, free_pages*SPIFFS_DATA_PAGE_SIZE(fs));
+        len, (u32_t)(free_pages*SPIFFS_DATA_PAGE_SIZE(fs)));
 
     spiffs_block_ix *cands;
     int count;
@@ -152,13 +152,13 @@ s32_t spiffs_gc_check(
 #endif
     cand = cands[0];
     fs->cleaning = 1;
-    //printf("gcing: cleaning block %i\n", cand);
+    //SPIFFS_GC_DBG("gcing: cleaning block "_SPIPRIi"\n", cand);
     res = spiffs_gc_clean(fs, cand);
     fs->cleaning = 0;
     if (res < 0) {
-      SPIFFS_GC_DBG("gc_check: cleaning block %i, result %i\n", cand, res);
+      SPIFFS_GC_DBG("gc_check: cleaning block "_SPIPRIi", result "_SPIPRIi"\n", cand, res);
     } else {
-      SPIFFS_GC_DBG("gc_check: cleaning block %i, result %i\n", cand, res);
+      SPIFFS_GC_DBG("gc_check: cleaning block "_SPIPRIi", result "_SPIPRIi"\n", cand, res);
     }
     SPIFFS_CHECK_RES(res);
 
@@ -188,7 +188,7 @@ s32_t spiffs_gc_check(
     res = SPIFFS_ERR_FULL;
   }
 
-  SPIFFS_GC_DBG("gc_check: finished, %i dirty, blocks %i free, %i pages free, %i tries, res %i\n",
+  SPIFFS_GC_DBG("gc_check: finished, "_SPIPRIi" dirty, blocks "_SPIPRIi" free, "_SPIPRIi" pages free, "_SPIPRIi" tries, res "_SPIPRIi"\n",
       fs->stats_p_allocated + fs->stats_p_deleted,
       fs->free_blocks, free_pages, tries, res);
 
@@ -226,7 +226,7 @@ s32_t spiffs_gc_erase_page_stats(
     } // per entry
     obj_lookup_page++;
   } // per object lookup page
-  SPIFFS_GC_DBG("gc_check: wipe pallo:%i pdele:%i\n", allo, dele);
+  SPIFFS_GC_DBG("gc_check: wipe pallo:"_SPIPRIi" pdele:"_SPIPRIi"\n", allo, dele);
   fs->stats_p_allocated -= allo;
   fs->stats_p_deleted -= dele;
   return res;
@@ -255,10 +255,7 @@ s32_t spiffs_gc_find_candidate(
   s32_t *cand_scores = (s32_t *)(fs->work + max_candidates * sizeof(spiffs_block_ix));
 
    // align cand_scores on s32_t boundary
-#pragma GCC diagnostic push
+  cand_scores = (s32_t*)(((intptr_t)cand_scores + sizeof(intptr_t) - 1) & ~(sizeof(intptr_t) - 1));
-#pragma GCC diagnostic ignored "-Wpointer-to-int-cast"
-  cand_scores = (s32_t*)(((ptrdiff_t)cand_scores + sizeof(ptrdiff_t) - 1) & ~(sizeof(ptrdiff_t) - 1));
-#pragma GCC diagnostic pop
 
   *block_candidates = cand_blocks;
 
@@ -297,7 +294,7 @@ s32_t spiffs_gc_find_candidate(
 
     // calculate score and insert into candidate table
     // stoneage sort, but probably not so many blocks
-    if (res == SPIFFS_OK && deleted_pages_in_block > 0) {
+    if (res == SPIFFS_OK /*&& deleted_pages_in_block > 0*/) {
       // read erase count
       spiffs_obj_id erase_count;
       res = _spiffs_rd(fs, SPIFFS_OP_C_READ | SPIFFS_OP_T_OBJ_LU2, 0,
@@ -317,7 +314,7 @@ s32_t spiffs_gc_find_candidate(
           used_pages_in_block * SPIFFS_GC_HEUR_W_USED +
           erase_age * (fs_crammed ? 0 : SPIFFS_GC_HEUR_W_ERASE_AGE);
       int cand_ix = 0;
-      SPIFFS_GC_DBG("gc_check: bix:%i del:%i use:%i score:%i\n", cur_block, deleted_pages_in_block, used_pages_in_block, score);
+      SPIFFS_GC_DBG("gc_check: bix:"_SPIPRIbl" del:"_SPIPRIi" use:"_SPIPRIi" score:"_SPIPRIi"\n", cur_block, deleted_pages_in_block, used_pages_in_block, score);
       while (cand_ix < max_candidates) {
         if (cand_blocks[cand_ix] == (spiffs_block_ix)-1) {
           cand_blocks[cand_ix] = cur_block;
@@ -359,6 +356,7 @@ typedef struct {
   spiffs_obj_id cur_obj_id;
   spiffs_span_ix cur_objix_spix;
   spiffs_page_ix cur_objix_pix;
+  spiffs_page_ix cur_data_pix;
   int stored_scan_entry_index;
   u8_t obj_id_found;
 } spiffs_gc;
@@ -378,15 +376,16 @@ typedef struct {
 //
 s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
   s32_t res = SPIFFS_OK;
-  int entries_per_page = (SPIFFS_CFG_LOG_PAGE_SZ(fs) / sizeof(spiffs_obj_id));
+  const int entries_per_page = (SPIFFS_CFG_LOG_PAGE_SZ(fs) / sizeof(spiffs_obj_id));
+  // this is the global localizer being pushed and popped
   int cur_entry = 0;
   spiffs_obj_id *obj_lu_buf = (spiffs_obj_id *)fs->lu_work;
-  spiffs_gc gc;
+  spiffs_gc gc; // our stack frame/state
   spiffs_page_ix cur_pix = 0;
   spiffs_page_object_ix_header *objix_hdr = (spiffs_page_object_ix_header *)fs->work;
   spiffs_page_object_ix *objix = (spiffs_page_object_ix *)fs->work;
 
-  SPIFFS_GC_DBG("gc_clean: cleaning block %i\n", bix);
+  SPIFFS_GC_DBG("gc_clean: cleaning block "_SPIPRIbl"\n", bix);
 
   memset(&gc, 0, sizeof(spiffs_gc));
   gc.state = FIND_OBJ_DATA;
@@ -395,12 +394,12 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
     // move free cursor to next block, cannot use free pages from the block we want to clean
     fs->free_cursor_block_ix = (bix+1)%fs->block_count;
     fs->free_cursor_obj_lu_entry = 0;
-    SPIFFS_GC_DBG("gc_clean: move free cursor to block %i\n", fs->free_cursor_block_ix);
+    SPIFFS_GC_DBG("gc_clean: move free cursor to block "_SPIPRIbl"\n", fs->free_cursor_block_ix);
   }
 
   while (res == SPIFFS_OK && gc.state != FINISHED) {
-    SPIFFS_GC_DBG("gc_clean: state = %i entry:%i\n", gc.state, cur_entry);
+    SPIFFS_GC_DBG("gc_clean: state = "_SPIPRIi" entry:"_SPIPRIi"\n", gc.state, cur_entry);
-    gc.obj_id_found = 0;
+    gc.obj_id_found = 0; // reset (to no found data page)
 
     // scan through lookup pages
     int obj_lookup_page = cur_entry / entries_per_page;
@@ -411,7 +410,7 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
       res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU | SPIFFS_OP_C_READ,
           0, bix * SPIFFS_CFG_LOG_BLOCK_SZ(fs) + SPIFFS_PAGE_TO_PADDR(fs, obj_lookup_page),
           SPIFFS_CFG_LOG_PAGE_SZ(fs), fs->lu_work);
-      // check each entry
+      // check each object lookup entry
       while (scan && res == SPIFFS_OK &&
           cur_entry - entry_offset < entries_per_page && cur_entry < (int)(SPIFFS_PAGES_PER_BLOCK(fs)-SPIFFS_OBJ_LOOKUP_PAGES(fs))) {
         spiffs_obj_id obj_id = obj_lu_buf[cur_entry-entry_offset];
@@ -420,21 +419,26 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
         // act upon object id depending on gc state
         switch (gc.state) {
         case FIND_OBJ_DATA:
+          // find a data page
           if (obj_id != SPIFFS_OBJ_ID_DELETED && obj_id != SPIFFS_OBJ_ID_FREE &&
               ((obj_id & SPIFFS_OBJ_ID_IX_FLAG) == 0)) {
-            SPIFFS_GC_DBG("gc_clean: FIND_DATA state:%i - found obj id %04x\n", gc.state, obj_id);
+            // found a data page, stop scanning and handle in switch case below
+            SPIFFS_GC_DBG("gc_clean: FIND_DATA state:"_SPIPRIi" - found obj id "_SPIPRIid"\n", gc.state, obj_id);
             gc.obj_id_found = 1;
             gc.cur_obj_id = obj_id;
+            gc.cur_data_pix = cur_pix;
             scan = 0;
           }
           break;
         case MOVE_OBJ_DATA:
+          // evacuate found data pages for corresponding object index we have in memory,
+          // update memory representation
           if (obj_id == gc.cur_obj_id) {
             spiffs_page_header p_hdr;
             res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU2 | SPIFFS_OP_C_READ,
                 0, SPIFFS_PAGE_TO_PADDR(fs, cur_pix), sizeof(spiffs_page_header), (u8_t*)&p_hdr);
             SPIFFS_CHECK_RES(res);
-            SPIFFS_GC_DBG("gc_clean: MOVE_DATA found data page %04x:%04x @ %04x\n", gc.cur_obj_id, p_hdr.span_ix, cur_pix);
+            SPIFFS_GC_DBG("gc_clean: MOVE_DATA found data page "_SPIPRIid":"_SPIPRIsp" @ "_SPIPRIpg"\n", gc.cur_obj_id, p_hdr.span_ix, cur_pix);
             if (SPIFFS_OBJ_IX_ENTRY_SPAN_IX(fs, p_hdr.span_ix) != gc.cur_objix_spix) {
               SPIFFS_GC_DBG("gc_clean: MOVE_DATA no objix spix match, take in another run\n");
             } else {
@@ -442,7 +446,7 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
               if (p_hdr.flags & SPIFFS_PH_FLAG_DELET) {
                 // move page
                 res = spiffs_page_move(fs, 0, 0, obj_id, &p_hdr, cur_pix, &new_data_pix);
-                SPIFFS_GC_DBG("gc_clean: MOVE_DATA move objix %04x:%04x page %04x to %04x\n", gc.cur_obj_id, p_hdr.span_ix, cur_pix, new_data_pix);
+                SPIFFS_GC_DBG("gc_clean: MOVE_DATA move objix "_SPIPRIid":"_SPIPRIsp" page "_SPIPRIpg" to "_SPIPRIpg"\n", gc.cur_obj_id, p_hdr.span_ix, cur_pix, new_data_pix);
                 SPIFFS_CHECK_RES(res);
                 // move wipes obj_lu, reload it
                 res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU | SPIFFS_OP_C_READ,
@@ -450,8 +454,10 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
                     SPIFFS_CFG_LOG_PAGE_SZ(fs), fs->lu_work);
                 SPIFFS_CHECK_RES(res);
               } else {
-                // page is deleted but not deleted in lookup, scrap it
+                // page is deleted but not deleted in lookup, scrap it -
-                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wipe objix %04x:%04x page %04x\n", obj_id, p_hdr.span_ix, cur_pix);
+                // might seem unnecessary as we will erase this block, but
+                // we might get aborted
+                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wipe objix "_SPIPRIid":"_SPIPRIsp" page "_SPIPRIpg"\n", obj_id, p_hdr.span_ix, cur_pix);
                 res = spiffs_page_delete(fs, cur_pix);
                 SPIFFS_CHECK_RES(res);
                 new_data_pix = SPIFFS_OBJ_ID_FREE;
@@ -460,16 +466,17 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
               if (gc.cur_objix_spix == 0) {
                 // update object index header page
                 ((spiffs_page_ix*)((u8_t *)objix_hdr + sizeof(spiffs_page_object_ix_header)))[p_hdr.span_ix] = new_data_pix;
-                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wrote page %04x to objix_hdr entry %02x in mem\n", new_data_pix, SPIFFS_OBJ_IX_ENTRY(fs, p_hdr.span_ix));
+                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wrote page "_SPIPRIpg" to objix_hdr entry "_SPIPRIsp" in mem\n", new_data_pix, (spiffs_span_ix)SPIFFS_OBJ_IX_ENTRY(fs, p_hdr.span_ix));
               } else {
                 // update object index page
                 ((spiffs_page_ix*)((u8_t *)objix + sizeof(spiffs_page_object_ix)))[SPIFFS_OBJ_IX_ENTRY(fs, p_hdr.span_ix)] = new_data_pix;
-                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wrote page %04x to objix entry %02x in mem\n", new_data_pix, SPIFFS_OBJ_IX_ENTRY(fs, p_hdr.span_ix));
+                SPIFFS_GC_DBG("gc_clean: MOVE_DATA wrote page "_SPIPRIpg" to objix entry "_SPIPRIsp" in mem\n", new_data_pix, (spiffs_span_ix)SPIFFS_OBJ_IX_ENTRY(fs, p_hdr.span_ix));
               }
             }
           }
           break;
         case MOVE_OBJ_IX:
+          // find and evacuate object index pages
           if (obj_id != SPIFFS_OBJ_ID_DELETED && obj_id != SPIFFS_OBJ_ID_FREE &&
               (obj_id & SPIFFS_OBJ_ID_IX_FLAG)) {
             // found an index object id
@@ -482,20 +489,24 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
             if (p_hdr.flags & SPIFFS_PH_FLAG_DELET) {
               // move page
               res = spiffs_page_move(fs, 0, 0, obj_id, &p_hdr, cur_pix, &new_pix);
-              SPIFFS_GC_DBG("gc_clean: MOVE_OBJIX move objix %04x:%04x page %04x to %04x\n", obj_id, p_hdr.span_ix, cur_pix, new_pix);
+              SPIFFS_GC_DBG("gc_clean: MOVE_OBJIX move objix "_SPIPRIid":"_SPIPRIsp" page "_SPIPRIpg" to "_SPIPRIpg"\n", obj_id, p_hdr.span_ix, cur_pix, new_pix);
               SPIFFS_CHECK_RES(res);
-              spiffs_cb_object_event(fs, 0, SPIFFS_EV_IX_UPD, obj_id, p_hdr.span_ix, new_pix, 0);
+              spiffs_cb_object_event(fs, (spiffs_page_object_ix *)&p_hdr,
+                  SPIFFS_EV_IX_MOV, obj_id, p_hdr.span_ix, new_pix, 0);
               // move wipes obj_lu, reload it
               res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU | SPIFFS_OP_C_READ,
                   0, bix * SPIFFS_CFG_LOG_BLOCK_SZ(fs) + SPIFFS_PAGE_TO_PADDR(fs, obj_lookup_page),
                   SPIFFS_CFG_LOG_PAGE_SZ(fs), fs->lu_work);
               SPIFFS_CHECK_RES(res);
             } else {
-              // page is deleted but not deleted in lookup, scrap it
+              // page is deleted but not deleted in lookup, scrap it -
-              SPIFFS_GC_DBG("gc_clean: MOVE_OBJIX wipe objix %04x:%04x page %04x\n", obj_id, p_hdr.span_ix, cur_pix);
+              // might seem unnecessary as we will erase this block, but
+              // we might get aborted
+              SPIFFS_GC_DBG("gc_clean: MOVE_OBJIX wipe objix "_SPIPRIid":"_SPIPRIsp" page "_SPIPRIpg"\n", obj_id, p_hdr.span_ix, cur_pix);
               res = spiffs_page_delete(fs, cur_pix);
               if (res == SPIFFS_OK) {
-                spiffs_cb_object_event(fs, 0, SPIFFS_EV_IX_DEL, obj_id, p_hdr.span_ix, cur_pix, 0);
+                spiffs_cb_object_event(fs, (spiffs_page_object_ix *)0,
+                    SPIFFS_EV_IX_DEL, obj_id, p_hdr.span_ix, cur_pix, 0);
               }
             }
             SPIFFS_CHECK_RES(res);
@@ -504,69 +515,88 @@ s32_t spiffs_gc_clean(spiffs *fs, spiffs_block_ix bix) {
         default:
           scan = 0;
           break;
-        }
+        } // switch gc state
         cur_entry++;
       } // per entry
-      obj_lookup_page++;
+      obj_lookup_page++; // no need to check scan variable here, obj_lookup_page is set in start of loop
     } // per object lookup page
-
     if (res != SPIFFS_OK) break;
 
     // state finalization and switch
     switch (gc.state) {
     case FIND_OBJ_DATA:
       if (gc.obj_id_found) {
+        // handle found data page -
         // find out corresponding obj ix page and load it to memory
         spiffs_page_header p_hdr;
         spiffs_page_ix objix_pix;
-        gc.stored_scan_entry_index = cur_entry;
+        gc.stored_scan_entry_index = cur_entry; // push cursor
-        cur_entry = 0;
+        cur_entry = 0; // restart scan from start
         gc.state = MOVE_OBJ_DATA;
         res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU2 | SPIFFS_OP_C_READ,
             0, SPIFFS_PAGE_TO_PADDR(fs, cur_pix), sizeof(spiffs_page_header), (u8_t*)&p_hdr);
         SPIFFS_CHECK_RES(res);
         gc.cur_objix_spix = SPIFFS_OBJ_IX_ENTRY_SPAN_IX(fs, p_hdr.span_ix);
-        SPIFFS_GC_DBG("gc_clean: FIND_DATA find objix span_ix:%04x\n", gc.cur_objix_spix);
+        SPIFFS_GC_DBG("gc_clean: FIND_DATA find objix span_ix:"_SPIPRIsp"\n", gc.cur_objix_spix);
         res = spiffs_obj_lu_find_id_and_span(fs, gc.cur_obj_id | SPIFFS_OBJ_ID_IX_FLAG, gc.cur_objix_spix, 0, &objix_pix);
+        if (res == SPIFFS_ERR_NOT_FOUND) {
+          // on borked systems we might get an ERR_NOT_FOUND here -
+          // this is handled by simply deleting the page as it is not referenced
+          // from anywhere
+          SPIFFS_GC_DBG("gc_clean: FIND_OBJ_DATA objix not found! Wipe page "_SPIPRIpg"\n", gc.cur_data_pix);
+          res = spiffs_page_delete(fs, gc.cur_data_pix);
+          SPIFFS_CHECK_RES(res);
+          // then we restore states and continue scanning for data pages
+          cur_entry = gc.stored_scan_entry_index; // pop cursor
+          gc.state = FIND_OBJ_DATA;
+          break; // done
+        }
         SPIFFS_CHECK_RES(res);
-        SPIFFS_GC_DBG("gc_clean: FIND_DATA found object index at page %04x\n", objix_pix);
+        SPIFFS_GC_DBG("gc_clean: FIND_DATA found object index at page "_SPIPRIpg"\n", objix_pix);
         res = _spiffs_rd(fs, SPIFFS_OP_T_OBJ_LU2 | SPIFFS_OP_C_READ,
             0, SPIFFS_PAGE_TO_PADDR(fs, objix_pix), SPIFFS_CFG_LOG_PAGE_SZ(fs), fs->work);
         SPIFFS_CHECK_RES(res);
+        // cannot allow a gc if the presumed index in fact is no index, a
+        // check must run or lot of data may be lost
         SPIFFS_VALIDATE_OBJIX(objix->p_hdr, gc.cur_obj_id | SPIFFS_OBJ_ID_IX_FLAG, gc.cur_objix_spix);
         gc.cur_objix_pix = objix_pix;
       } else {
+        // no more data pages found, passed thru all block, start evacuating object indices
         gc.state = MOVE_OBJ_IX;
         cur_entry = 0; // restart entry scan index
       }
       break;
     case MOVE_OBJ_DATA: {
-      // store modified objix (hdr) page
+      // store modified objix (hdr) page residing in memory now that all
+      // data pages belonging to this object index and residing in the block
+      // we want to evacuate
       spiffs_page_ix new_objix_pix;
       gc.state = FIND_OBJ_DATA;
-      cur_entry = gc.stored_scan_entry_index;
+      cur_entry = gc.stored_scan_entry_index; // pop cursor
       if (gc.cur_objix_spix == 0) {
         // store object index header page
-        res = spiffs_object_update_index_hdr(fs, 0, gc.cur_obj_id | SPIFFS_OBJ_ID_IX_FLAG, gc.cur_objix_pix, fs->work, 0, 0, &new_objix_pix);
+        res = spiffs_object_update_index_hdr(fs, 0, gc.cur_obj_id | SPIFFS_OBJ_ID_IX_FLAG, gc.cur_objix_pix, fs->work, 0, 0, 0, &new_objix_pix);
-        SPIFFS_GC_DBG("gc_clean: MOVE_DATA store modified objix_hdr page, %04x:%04x\n", new_objix_pix, 0);
+        SPIFFS_GC_DBG("gc_clean: MOVE_DATA store modified objix_hdr page, "_SPIPRIpg":"_SPIPRIsp"\n", new_objix_pix, 0);
         SPIFFS_CHECK_RES(res);
       } else {
         // store object index page
         res = spiffs_page_move(fs, 0, fs->work, gc.cur_obj_id | SPIFFS_OBJ_ID_IX_FLAG, 0, gc.cur_objix_pix, &new_objix_pix);
-        SPIFFS_GC_DBG("gc_clean: MOVE_DATA store modified objix page, %04x:%04x\n", new_objix_pix, objix->p_hdr.span_ix);
+        SPIFFS_GC_DBG("gc_clean: MOVE_DATA store modified objix page, "_SPIPRIpg":"_SPIPRIsp"\n", new_objix_pix, objix->p_hdr.span_ix);
         SPIFFS_CHECK_RES(res);
-        spiffs_cb_object_event(fs, 0, SPIFFS_EV_IX_UPD, gc.cur_obj_id, objix->p_hdr.span_ix, new_objix_pix, 0);
+        spiffs_cb_object_event(fs, (spiffs_page_object_ix *)fs->work,
+            SPIFFS_EV_IX_UPD, gc.cur_obj_id, objix->p_hdr.span_ix, new_objix_pix, 0);
       }
     }
     break;
     case MOVE_OBJ_IX:
+      // scanned thru all block, no more object indices found - our work here is done
       gc.state = FINISHED;
       break;
     default:
       cur_entry = 0;
       break;
-    }
+    } // switch gc.state
-    SPIFFS_GC_DBG("gc_clean: state-> %i\n", gc.state);
+    SPIFFS_GC_DBG("gc_clean: state-> "_SPIPRIi"\n", gc.state);
   } // while state != FINISHED
 
 

cores/esp8266/spiffs/spiffs_hydrogen.c

@@ -8,21 +8,13 @@
 #include "spiffs.h"
 #include "spiffs_nucleus.h"
 
-#if SPIFFS_FILEHDL_OFFSET
-#define SPIFFS_FH_OFFS(fs, fh)   ((fh) != 0 ? ((fh) + (fs)->cfg.fh_ix_offset) : 0)
-#define SPIFFS_FH_UNOFFS(fs, fh) ((fh) != 0 ? ((fh) - (fs)->cfg.fh_ix_offset) : 0)
-#else
-#define SPIFFS_FH_OFFS(fs, fh)   (fh)
-#define SPIFFS_FH_UNOFFS(fs, fh) (fh)
-#endif
-
 #if SPIFFS_CACHE == 1
 static s32_t spiffs_fflush_cache(spiffs *fs, spiffs_file fh);
 #endif
 
 #if SPIFFS_BUFFER_HELP
 u32_t SPIFFS_buffer_bytes_for_filedescs(spiffs *fs, u32_t num_descs) {
-  (void) fs;
+  (void)fs; // unused, avoid warning
   return num_descs * sizeof(spiffs_fd);
 }
 #if SPIFFS_CACHE
@@ -70,6 +62,7 @@ s32_t SPIFFS_format(spiffs *fs) {
 #if SPIFFS_USE_MAGIC && SPIFFS_USE_MAGIC_LENGTH && SPIFFS_SINGLETON==0
 
 s32_t SPIFFS_probe_fs(spiffs_config *config) {
+  SPIFFS_API_DBG("%s\n", __func__);
   s32_t res = spiffs_probe(config);
   return res;
 }
@@ -80,22 +73,31 @@ s32_t SPIFFS_mount(spiffs *fs, spiffs_config *config, u8_t *work,
     u8_t *fd_space, u32_t fd_space_size,
     void *cache, u32_t cache_size,
     spiffs_check_callback check_cb_f) {
+  SPIFFS_API_DBG("%s "
+                 " sz:"_SPIPRIi " logpgsz:"_SPIPRIi " logblksz:"_SPIPRIi " perasz:"_SPIPRIi
+                 " addr:"_SPIPRIad
+                 " fdsz:"_SPIPRIi " cachesz:"_SPIPRIi
+                 "\n",
+                 __func__,
+                 SPIFFS_CFG_PHYS_SZ(fs),
+                 SPIFFS_CFG_LOG_PAGE_SZ(fs),
+                 SPIFFS_CFG_LOG_BLOCK_SZ(fs),
+                 SPIFFS_CFG_PHYS_ERASE_SZ(fs),
+                 SPIFFS_CFG_PHYS_ADDR(fs),
+                 fd_space_size, cache_size);
   void *user_data;
   SPIFFS_LOCK(fs);
   user_data = fs->user_data;
   memset(fs, 0, sizeof(spiffs));
-  memcpy(&fs->cfg, config, sizeof(spiffs_config));
+  _SPIFFS_MEMCPY(&fs->cfg, config, sizeof(spiffs_config));
   fs->user_data = user_data;
   fs->block_count = SPIFFS_CFG_PHYS_SZ(fs) / SPIFFS_CFG_LOG_BLOCK_SZ(fs);
   fs->work = &work[0];
   fs->lu_work = &work[SPIFFS_CFG_LOG_PAGE_SZ(fs)];
   memset(fd_space, 0, fd_space_size);
-  // align fd_space pointer to pointer size byte boundary, below is safe
+  // align fd_space pointer to pointer size byte boundary
   u8_t ptr_size = sizeof(void*);
-#pragma GCC diagnostic push
+  u8_t addr_lsb = ((u8_t)(intptr_t)fd_space) & (ptr_size-1);
-#pragma GCC diagnostic ignored "-Wpointer-to-int-cast"
-  u8_t addr_lsb = ((u8_t)fd_space) & (ptr_size-1);
-#pragma GCC diagnostic pop
   if (addr_lsb) {
     fd_space += (ptr_size-addr_lsb);
     fd_space_size -= (ptr_size-addr_lsb);
@@ -103,11 +105,8 @@ s32_t SPIFFS_mount(spiffs *fs, spiffs_config *config, u8_t *work,
   fs->fd_space = fd_space;
   fs->fd_count = (fd_space_size/sizeof(spiffs_fd));
 
-  // align cache pointer to 4 byte boundary, below is safe
+  // align cache pointer to 4 byte boundary
-#pragma GCC diagnostic push
+  addr_lsb = ((u8_t)(intptr_t)cache) & (ptr_size-1);
-#pragma GCC diagnostic ignored "-Wpointer-to-int-cast"
-  addr_lsb = ((u8_t)cache) & (ptr_size-1);
-#pragma GCC diagnostic pop
   if (addr_lsb) {
     u8_t *cache_8 = (u8_t *)cache;
     cache_8 += (ptr_size-addr_lsb);
@@ -136,14 +135,14 @@ s32_t SPIFFS_mount(spiffs *fs, spiffs_config *config, u8_t *work,
   res = spiffs_obj_lu_scan(fs);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  SPIFFS_DBG("page index byte len:         %i\n", SPIFFS_CFG_LOG_PAGE_SZ(fs));
+  SPIFFS_DBG("page index byte len:         "_SPIPRIi"\n", (u32_t)SPIFFS_CFG_LOG_PAGE_SZ(fs));
-  SPIFFS_DBG("object lookup pages:         %i\n", SPIFFS_OBJ_LOOKUP_PAGES(fs));
+  SPIFFS_DBG("object lookup pages:         "_SPIPRIi"\n", (u32_t)SPIFFS_OBJ_LOOKUP_PAGES(fs));
-  SPIFFS_DBG("page pages per block:        %i\n", SPIFFS_PAGES_PER_BLOCK(fs));
+  SPIFFS_DBG("page pages per block:        "_SPIPRIi"\n", (u32_t)SPIFFS_PAGES_PER_BLOCK(fs));
-  SPIFFS_DBG("page header length:          %i\n", sizeof(spiffs_page_header));
+  SPIFFS_DBG("page header length:          "_SPIPRIi"\n", (u32_t)sizeof(spiffs_page_header));
-  SPIFFS_DBG("object header index entries: %i\n", SPIFFS_OBJ_HDR_IX_LEN(fs));
+  SPIFFS_DBG("object header index entries: "_SPIPRIi"\n", (u32_t)SPIFFS_OBJ_HDR_IX_LEN(fs));
-  SPIFFS_DBG("object index entries:        %i\n", SPIFFS_OBJ_IX_LEN(fs));
+  SPIFFS_DBG("object index entries:        "_SPIPRIi"\n", (u32_t)SPIFFS_OBJ_IX_LEN(fs));
-  SPIFFS_DBG("available file descriptors:  %i\n", fs->fd_count);
+  SPIFFS_DBG("available file descriptors:  "_SPIPRIi"\n", (u32_t)fs->fd_count);
-  SPIFFS_DBG("free blocks:                 %i\n", fs->free_blocks);
+  SPIFFS_DBG("free blocks:                 "_SPIPRIi"\n", (u32_t)fs->free_blocks);
 
   fs->check_cb_f = check_cb_f;
 
@@ -155,6 +154,7 @@ s32_t SPIFFS_mount(spiffs *fs, spiffs_config *config, u8_t *work,
 }
 
 void SPIFFS_unmount(spiffs *fs) {
+  SPIFFS_API_DBG("%s\n", __func__);
   if (!SPIFFS_CHECK_CFG(fs) || !SPIFFS_CHECK_MOUNT(fs)) return;
   SPIFFS_LOCK(fs);
   u32_t i;
@@ -178,10 +178,12 @@ s32_t SPIFFS_errno(spiffs *fs) {
 }
 
 void SPIFFS_clearerr(spiffs *fs) {
+  SPIFFS_API_DBG("%s\n", __func__);
   fs->err_code = SPIFFS_OK;
 }
 
 s32_t SPIFFS_creat(spiffs *fs, const char *path, spiffs_mode mode) {
+  SPIFFS_API_DBG("%s '%s'\n", __func__, path);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)path; (void)mode;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -189,13 +191,16 @@ s32_t SPIFFS_creat(spiffs *fs, const char *path, spiffs_mode mode) {
   (void)mode;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
+  if (strlen(path) > SPIFFS_OBJ_NAME_LEN - 1) {
+    SPIFFS_API_CHECK_RES(fs, SPIFFS_ERR_NAME_TOO_LONG);
+  }
   SPIFFS_LOCK(fs);
   spiffs_obj_id obj_id;
   s32_t res;
 
   res = spiffs_obj_lu_find_free_obj_id(fs, &obj_id, (const u8_t*)path);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
-  res = spiffs_object_create(fs, obj_id, (const u8_t*)path, SPIFFS_TYPE_FILE, 0);
+  res = spiffs_object_create(fs, obj_id, (const u8_t*)path, 0, SPIFFS_TYPE_FILE, 0);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   SPIFFS_UNLOCK(fs);
   return 0;
@@ -203,9 +208,13 @@ s32_t SPIFFS_creat(spiffs *fs, const char *path, spiffs_mode mode) {
 }
 
 spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs_mode mode) {
+  SPIFFS_API_DBG("%s '%s' "_SPIPRIfl "\n", __func__, path, flags);
   (void)mode;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
+  if (strlen(path) > SPIFFS_OBJ_NAME_LEN - 1) {
+    SPIFFS_API_CHECK_RES(fs, SPIFFS_ERR_NAME_TOO_LONG);
+  }
   SPIFFS_LOCK(fs);
 
   spiffs_fd *fd;
@@ -213,14 +222,14 @@ spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs
 
 #if SPIFFS_READ_ONLY
   // not valid flags in read only mode
-  flags &= ~SPIFFS_WRONLY | SPIFFS_CREAT | SPIFFS_TRUNC;
+  flags &= ~(SPIFFS_WRONLY | SPIFFS_CREAT | SPIFFS_TRUNC);
 #endif // SPIFFS_READ_ONLY
 
-  s32_t res = spiffs_fd_find_new(fs, &fd);
+  s32_t res = spiffs_fd_find_new(fs, &fd, path);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
   res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)path, &pix);
-  if ((flags & SPIFFS_CREAT) == 0) {
+  if ((flags & SPIFFS_O_CREAT) == 0) {
     if (res < SPIFFS_OK) {
       spiffs_fd_return(fs, fd->file_nbr);
     }
@@ -228,14 +237,14 @@ spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs
   }
 
   if (res == SPIFFS_OK &&
-      (flags & (SPIFFS_CREAT | SPIFFS_EXCL)) == (SPIFFS_CREAT | SPIFFS_EXCL)) {
+      (flags & (SPIFFS_O_CREAT | SPIFFS_O_EXCL)) == (SPIFFS_O_CREAT | SPIFFS_O_EXCL)) {
     // creat and excl and file exists - fail
     res = SPIFFS_ERR_FILE_EXISTS;
     spiffs_fd_return(fs, fd->file_nbr);
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   }
 
-  if ((flags & SPIFFS_CREAT) && res == SPIFFS_ERR_NOT_FOUND) {
+  if ((flags & SPIFFS_O_CREAT) && res == SPIFFS_ERR_NOT_FOUND) {
 #if !SPIFFS_READ_ONLY
     spiffs_obj_id obj_id;
     // no need to enter conflicting name here, already looked for it above
@@ -244,12 +253,12 @@ spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs
       spiffs_fd_return(fs, fd->file_nbr);
     }
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
-    res = spiffs_object_create(fs, obj_id, (const u8_t*)path, SPIFFS_TYPE_FILE, &pix);
+    res = spiffs_object_create(fs, obj_id, (const u8_t*)path, 0, SPIFFS_TYPE_FILE, &pix);
     if (res < SPIFFS_OK) {
       spiffs_fd_return(fs, fd->file_nbr);
     }
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
-    flags &= ~SPIFFS_TRUNC;
+    flags &= ~SPIFFS_O_TRUNC;
 #endif // !SPIFFS_READ_ONLY
   } else {
     if (res < SPIFFS_OK) {
@@ -263,7 +272,7 @@ spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs
   }
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 #if !SPIFFS_READ_ONLY
-  if (flags & SPIFFS_TRUNC) {
+  if (flags & SPIFFS_O_TRUNC) {
     res = spiffs_object_truncate(fd, 0, 0);
     if (res < SPIFFS_OK) {
       spiffs_fd_return(fs, fd->file_nbr);
@@ -280,13 +289,14 @@ spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs
 }
 
 spiffs_file SPIFFS_open_by_dirent(spiffs *fs, struct spiffs_dirent *e, spiffs_flags flags, spiffs_mode mode) {
+  SPIFFS_API_DBG("%s '%s':"_SPIPRIid " "_SPIPRIfl "\n", __func__, e->name, e->obj_id, flags);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
   SPIFFS_LOCK(fs);
 
   spiffs_fd *fd;
 
-  s32_t res = spiffs_fd_find_new(fs, &fd);
+  s32_t res = spiffs_fd_find_new(fs, &fd, 0);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
   res = spiffs_object_open_by_page(fs, e->pix, fd, flags, mode);
@@ -295,7 +305,7 @@ spiffs_file SPIFFS_open_by_dirent(spiffs *fs, struct spiffs_dirent *e, spiffs_fl
   }
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 #if !SPIFFS_READ_ONLY
-  if (flags & SPIFFS_TRUNC) {
+  if (flags & SPIFFS_O_TRUNC) {
     res = spiffs_object_truncate(fd, 0, 0);
     if (res < SPIFFS_OK) {
       spiffs_fd_return(fs, fd->file_nbr);
@@ -312,13 +322,14 @@ spiffs_file SPIFFS_open_by_dirent(spiffs *fs, struct spiffs_dirent *e, spiffs_fl
 }
 
 spiffs_file SPIFFS_open_by_page(spiffs *fs, spiffs_page_ix page_ix, spiffs_flags flags, spiffs_mode mode) {
+  SPIFFS_API_DBG("%s "_SPIPRIpg " "_SPIPRIfl "\n", __func__, page_ix, flags);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
   SPIFFS_LOCK(fs);
 
   spiffs_fd *fd;
 
-  s32_t res = spiffs_fd_find_new(fs, &fd);
+  s32_t res = spiffs_fd_find_new(fs, &fd, 0);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
   if (SPIFFS_IS_LOOKUP_PAGE(fs, page_ix)) {
@@ -341,7 +352,7 @@ spiffs_file SPIFFS_open_by_page(spiffs *fs, spiffs_page_ix page_ix, spiffs_flags
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
 #if !SPIFFS_READ_ONLY
-  if (flags & SPIFFS_TRUNC) {
+  if (flags & SPIFFS_O_TRUNC) {
     res = spiffs_object_truncate(fd, 0, 0);
     if (res < SPIFFS_OK) {
       spiffs_fd_return(fs, fd->file_nbr);
@@ -357,7 +368,7 @@ spiffs_file SPIFFS_open_by_page(spiffs *fs, spiffs_page_ix page_ix, spiffs_flags
   return SPIFFS_FH_OFFS(fs, fd->file_nbr);
 }
 
-s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
+static s32_t spiffs_hydro_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
   SPIFFS_LOCK(fs);
@@ -369,7 +380,7 @@ s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
   res = spiffs_fd_get(fs, fh, &fd);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  if ((fd->flags & SPIFFS_RDONLY) == 0) {
+  if ((fd->flags & SPIFFS_O_RDONLY) == 0) {
     res = SPIFFS_ERR_NOT_READABLE;
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   }
@@ -411,6 +422,16 @@ s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
   return len;
 }
 
+s32_t SPIFFS_read(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd " "_SPIPRIi "\n", __func__, fh, len);
+  s32_t res = spiffs_hydro_read(fs, fh, buf, len);
+  if (res == SPIFFS_ERR_END_OF_OBJECT) {
+    res = 0;
+  }
+  return res;
+}
+
+
 #if !SPIFFS_READ_ONLY
 static s32_t spiffs_hydro_write(spiffs *fs, spiffs_fd *fd, void *buf, u32_t offset, s32_t len) {
   (void)fs;
@@ -436,6 +457,7 @@ static s32_t spiffs_hydro_write(spiffs *fs, spiffs_fd *fd, void *buf, u32_t offs
 #endif // !SPIFFS_READ_ONLY
 
 s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd " "_SPIPRIi "\n", __func__, fh, len);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)fh; (void)buf; (void)len;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -452,15 +474,14 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
   res = spiffs_fd_get(fs, fh, &fd);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  if ((fd->flags & SPIFFS_WRONLY) == 0) {
+  if ((fd->flags & SPIFFS_O_WRONLY) == 0) {
     res = SPIFFS_ERR_NOT_WRITABLE;
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   }
 
-  if ((fd->flags & SPIFFS_APPEND)) {
+  if ((fd->flags & SPIFFS_O_APPEND)) {
     fd->fdoffset = fd->size == SPIFFS_UNDEFINED_LEN ? 0 : fd->size;
   }
-
   offset = fd->fdoffset;
 
 #if SPIFFS_CACHE_WR
@@ -469,7 +490,7 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
     fd->cache_page = spiffs_cache_page_get_by_fd(fs, fd);
   }
 #endif
-  if (fd->flags & SPIFFS_APPEND) {
+  if (fd->flags & SPIFFS_O_APPEND) {
     if (fd->size == SPIFFS_UNDEFINED_LEN) {
       offset = 0;
     } else {
@@ -483,7 +504,7 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
   }
 
 #if SPIFFS_CACHE_WR
-  if ((fd->flags & SPIFFS_DIRECT) == 0) {
+  if ((fd->flags & SPIFFS_O_DIRECT) == 0) {
     if (len < (s32_t)SPIFFS_CFG_LOG_PAGE_SZ(fs)) {
       // small write, try to cache it
       u8_t alloc_cpage = 1;
@@ -494,13 +515,13 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
             offset + len > fd->cache_page->offset + SPIFFS_CFG_LOG_PAGE_SZ(fs)) // writing beyond cache page
         {
           // boundary violation, write back cache first and allocate new
-          SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page %i for fd %i:%04x, boundary viol, offs:%i size:%i\n",
+          SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page "_SPIPRIi" for fd "_SPIPRIfd":"_SPIPRIid", boundary viol, offs:"_SPIPRIi" size:"_SPIPRIi"\n",
               fd->cache_page->ix, fd->file_nbr, fd->obj_id, fd->cache_page->offset, fd->cache_page->size);
           res = spiffs_hydro_write(fs, fd,
               spiffs_get_cache_page(fs, spiffs_get_cache(fs), fd->cache_page->ix),
               fd->cache_page->offset, fd->cache_page->size);
           spiffs_cache_fd_release(fs, fd->cache_page);
-          SPIFFS_API_CHECK_RES(fs, res);
+          SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
         } else {
           // writing within cache
           alloc_cpage = 0;
@@ -512,26 +533,37 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
         if (fd->cache_page) {
           fd->cache_page->offset = offset;
           fd->cache_page->size = 0;
-          SPIFFS_CACHE_DBG("CACHE_WR_ALLO: allocating cache page %i for fd %i:%04x\n",
+          SPIFFS_CACHE_DBG("CACHE_WR_ALLO: allocating cache page "_SPIPRIi" for fd "_SPIPRIfd":"_SPIPRIid"\n",
               fd->cache_page->ix, fd->file_nbr, fd->obj_id);
         }
       }
 
       if (fd->cache_page) {
         u32_t offset_in_cpage = offset - fd->cache_page->offset;
-        SPIFFS_CACHE_DBG("CACHE_WR_WRITE: storing to cache page %i for fd %i:%04x, offs %i:%i len %i\n",
+        SPIFFS_CACHE_DBG("CACHE_WR_WRITE: storing to cache page "_SPIPRIi" for fd "_SPIPRIfd":"_SPIPRIid", offs "_SPIPRIi":"_SPIPRIi" len "_SPIPRIi"\n",
             fd->cache_page->ix, fd->file_nbr, fd->obj_id,
             offset, offset_in_cpage, len);
         spiffs_cache *cache = spiffs_get_cache(fs);
         u8_t *cpage_data = spiffs_get_cache_page(fs, cache, fd->cache_page->ix);
-        memcpy(&cpage_data[offset_in_cpage], buf, len);
+#ifdef _SPIFFS_TEST
+        {
+          intptr_t __a1 = (u8_t*)&cpage_data[offset_in_cpage]-(u8_t*)cache;
+          intptr_t __a2 = (u8_t*)&cpage_data[offset_in_cpage]+len-(u8_t*)cache;
+          intptr_t __b = sizeof(spiffs_cache) + cache->cpage_count * (sizeof(spiffs_cache_page) + SPIFFS_CFG_LOG_PAGE_SZ(fs));
+          if (__a1 > __b || __a2 > __b) {
+            printf("FATAL OOB: CACHE_WR: memcpy to cache buffer ixs:%4ld..%4ld of %4ld\n", __a1, __a2, __b);
+            ERREXIT();
+          }
+        }
+#endif
+        _SPIFFS_MEMCPY(&cpage_data[offset_in_cpage], buf, len);
         fd->cache_page->size = MAX(fd->cache_page->size, offset_in_cpage + len);
         fd->fdoffset += len;
         SPIFFS_UNLOCK(fs);
         return len;
       } else {
         res = spiffs_hydro_write(fs, fd, buf, offset, len);
-        SPIFFS_API_CHECK_RES(fs, res);
+        SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
         fd->fdoffset += len;
         SPIFFS_UNLOCK(fs);
         return res;
@@ -540,22 +572,21 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
       // big write, no need to cache it - but first check if there is a cached write already
       if (fd->cache_page) {
         // write back cache first
-        SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page %i for fd %i:%04x, big write, offs:%i size:%i\n",
+        SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page "_SPIPRIi" for fd "_SPIPRIfd":"_SPIPRIid", big write, offs:"_SPIPRIi" size:"_SPIPRIi"\n",
             fd->cache_page->ix, fd->file_nbr, fd->obj_id, fd->cache_page->offset, fd->cache_page->size);
         res = spiffs_hydro_write(fs, fd,
             spiffs_get_cache_page(fs, spiffs_get_cache(fs), fd->cache_page->ix),
             fd->cache_page->offset, fd->cache_page->size);
         spiffs_cache_fd_release(fs, fd->cache_page);
-        SPIFFS_API_CHECK_RES(fs, res);
+        SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
-        res = spiffs_hydro_write(fs, fd, buf, offset, len);
+        // data written below
-        SPIFFS_API_CHECK_RES(fs, res);
       }
     }
   }
 #endif
 
   res = spiffs_hydro_write(fs, fd, buf, offset, len);
-  SPIFFS_API_CHECK_RES(fs, res);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   fd->fdoffset += len;
 
   SPIFFS_UNLOCK(fs);
@@ -565,6 +596,7 @@ s32_t SPIFFS_write(spiffs *fs, spiffs_file fh, void *buf, s32_t len) {
 }
 
 s32_t SPIFFS_lseek(spiffs *fs, spiffs_file fh, s32_t offs, int whence) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd " "_SPIPRIi " %s\n", __func__, fh, offs, (const char* []){"SET","CUR","END","???"}[MIN(whence,3)]);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
   SPIFFS_LOCK(fs);
@@ -573,27 +605,32 @@ s32_t SPIFFS_lseek(spiffs *fs, spiffs_file fh, s32_t offs, int whence) {
   s32_t res;
   fh = SPIFFS_FH_UNOFFS(fs, fh);
   res = spiffs_fd_get(fs, fh, &fd);
-  SPIFFS_API_CHECK_RES(fs, res);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
 #if SPIFFS_CACHE_WR
   spiffs_fflush_cache(fs, fh);
 #endif
 
+  s32_t file_size = fd->size == SPIFFS_UNDEFINED_LEN ? 0 : fd->size;
+
   switch (whence) {
   case SPIFFS_SEEK_CUR:
     offs = fd->fdoffset+offs;
     break;
   case SPIFFS_SEEK_END:
-    offs = (fd->size == SPIFFS_UNDEFINED_LEN ? 0 : fd->size) + offs;
+    offs = file_size + offs;
     break;
   }
-
+  if (offs < 0) {
-  if ((offs > (s32_t)fd->size) && (SPIFFS_UNDEFINED_LEN != fd->size)) {
+    SPIFFS_API_CHECK_RES_UNLOCK(fs, SPIFFS_ERR_SEEK_BOUNDS);
+  }
+  if (offs > file_size) {
+    fd->fdoffset = file_size;
     res = SPIFFS_ERR_END_OF_OBJECT;
   }
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  spiffs_span_ix data_spix = offs / SPIFFS_DATA_PAGE_SIZE(fs);
+  spiffs_span_ix data_spix = (offs > 0 ? (offs-1) : 0) / SPIFFS_DATA_PAGE_SIZE(fs);
   spiffs_span_ix objix_spix = SPIFFS_OBJ_IX_ENTRY_SPAN_IX(fs, data_spix);
   if (fd->cursor_objix_spix != objix_spix) {
     spiffs_page_ix pix;
@@ -611,19 +648,23 @@ s32_t SPIFFS_lseek(spiffs *fs, spiffs_file fh, s32_t offs, int whence) {
 }
 
 s32_t SPIFFS_remove(spiffs *fs, const char *path) {
+  SPIFFS_API_DBG("%s '%s'\n", __func__, path);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)path;
   return SPIFFS_ERR_RO_NOT_IMPL;
 #else
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
+  if (strlen(path) > SPIFFS_OBJ_NAME_LEN - 1) {
+    SPIFFS_API_CHECK_RES(fs, SPIFFS_ERR_NAME_TOO_LONG);
+  }
   SPIFFS_LOCK(fs);
 
   spiffs_fd *fd;
   spiffs_page_ix pix;
   s32_t res;
 
-  res = spiffs_fd_find_new(fs, &fd);
+  res = spiffs_fd_find_new(fs, &fd, 0);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
   res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)path, &pix);
@@ -650,6 +691,7 @@ s32_t SPIFFS_remove(spiffs *fs, const char *path) {
 }
 
 s32_t SPIFFS_fremove(spiffs *fs, spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)fh;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -664,7 +706,7 @@ s32_t SPIFFS_fremove(spiffs *fs, spiffs_file fh) {
   res = spiffs_fd_get(fs, fh, &fd);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  if ((fd->flags & SPIFFS_WRONLY) == 0) {
+  if ((fd->flags & SPIFFS_O_WRONLY) == 0) {
     res = SPIFFS_ERR_NOT_WRITABLE;
     SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
   }
@@ -702,13 +744,20 @@ static s32_t spiffs_stat_pix(spiffs *fs, spiffs_page_ix pix, spiffs_file fh, spi
   s->size = objix_hdr.size == SPIFFS_UNDEFINED_LEN ? 0 : objix_hdr.size;
   s->pix = pix;
   strncpy((char *)s->name, (char *)objix_hdr.name, SPIFFS_OBJ_NAME_LEN);
+#if SPIFFS_OBJ_META_LEN
+  _SPIFFS_MEMCPY(s->meta, objix_hdr.meta, SPIFFS_OBJ_META_LEN);
+#endif
 
   return res;
 }
 
 s32_t SPIFFS_stat(spiffs *fs, const char *path, spiffs_stat *s) {
+  SPIFFS_API_DBG("%s '%s'\n", __func__, path);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
+  if (strlen(path) > SPIFFS_OBJ_NAME_LEN - 1) {
+    SPIFFS_API_CHECK_RES(fs, SPIFFS_ERR_NAME_TOO_LONG);
+  }
   SPIFFS_LOCK(fs);
 
   s32_t res;
@@ -725,6 +774,7 @@ s32_t SPIFFS_stat(spiffs *fs, const char *path, spiffs_stat *s) {
 }
 
 s32_t SPIFFS_fstat(spiffs *fs, spiffs_file fh, spiffs_stat *s) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
   SPIFFS_LOCK(fs);
@@ -760,13 +810,13 @@ static s32_t spiffs_fflush_cache(spiffs *fs, spiffs_file fh) {
   res = spiffs_fd_get(fs, fh, &fd);
   SPIFFS_API_CHECK_RES(fs, res);
 
-  if ((fd->flags & SPIFFS_DIRECT) == 0) {
+  if ((fd->flags & SPIFFS_O_DIRECT) == 0) {
     if (fd->cache_page == 0) {
       // see if object id is associated with cache already
       fd->cache_page = spiffs_cache_page_get_by_fd(fs, fd);
     }
     if (fd->cache_page) {
-      SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page %i for fd %i:%04x, flush, offs:%i size:%i\n",
+      SPIFFS_CACHE_DBG("CACHE_WR_DUMP: dumping cache page "_SPIPRIi" for fd "_SPIPRIfd":"_SPIPRIid", flush, offs:"_SPIPRIi" size:"_SPIPRIi"\n",
           fd->cache_page->ix, fd->file_nbr,  fd->obj_id, fd->cache_page->offset, fd->cache_page->size);
       res = spiffs_hydro_write(fs, fd,
           spiffs_get_cache_page(fs, spiffs_get_cache(fs), fd->cache_page->ix),
@@ -784,6 +834,7 @@ static s32_t spiffs_fflush_cache(spiffs *fs, spiffs_file fh) {
 #endif
 
 s32_t SPIFFS_fflush(spiffs *fs, spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
   (void)fh;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
@@ -800,6 +851,7 @@ s32_t SPIFFS_fflush(spiffs *fs, spiffs_file fh) {
 }
 
 s32_t SPIFFS_close(spiffs *fs, spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
 
@@ -819,22 +871,27 @@ s32_t SPIFFS_close(spiffs *fs, spiffs_file fh) {
   return res;
 }
 
-s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *new) {
+s32_t SPIFFS_rename(spiffs *fs, const char *old_path, const char *new_path) {
+  SPIFFS_API_DBG("%s %s %s\n", __func__, old_path, new_path);
 #if SPIFFS_READ_ONLY
-  (void)fs; (void)old; (void)new;
+  (void)fs; (void)old_path; (void)new_path;
   return SPIFFS_ERR_RO_NOT_IMPL;
 #else
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
+  if (strlen(new_path) > SPIFFS_OBJ_NAME_LEN - 1 ||
+      strlen(old_path) > SPIFFS_OBJ_NAME_LEN - 1) {
+    SPIFFS_API_CHECK_RES(fs, SPIFFS_ERR_NAME_TOO_LONG);
+  }
   SPIFFS_LOCK(fs);
 
   spiffs_page_ix pix_old, pix_dummy;
   spiffs_fd *fd;
 
-  s32_t res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)old, &pix_old);
+  s32_t res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)old_path, &pix_old);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)new, &pix_dummy);
+  res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)new_path, &pix_dummy);
   if (res == SPIFFS_ERR_NOT_FOUND) {
     res = SPIFFS_OK;
   } else if (res == SPIFFS_OK) {
@@ -842,7 +899,7 @@ s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *new) {
   }
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  res = spiffs_fd_find_new(fs, &fd);
+  res = spiffs_fd_find_new(fs, &fd, 0);
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
   res = spiffs_object_open_by_page(fs, pix_old, fd, 0, 0);
@@ -851,7 +908,50 @@ s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *new) {
   }
   SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
 
-  res = spiffs_object_update_index_hdr(fs, fd, fd->obj_id, fd->objix_hdr_pix, 0, (const u8_t*)new,
+  res = spiffs_object_update_index_hdr(fs, fd, fd->obj_id, fd->objix_hdr_pix, 0, (const u8_t*)new_path,
+      0, 0, &pix_dummy);
+#if SPIFFS_TEMPORAL_FD_CACHE
+  if (res == SPIFFS_OK) {
+    spiffs_fd_temporal_cache_rehash(fs, old_path, new_path);
+  }
+#endif
+
+  spiffs_fd_return(fs, fd->file_nbr);
+
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  SPIFFS_UNLOCK(fs);
+
+  return res;
+#endif // SPIFFS_READ_ONLY
+}
+
+#if SPIFFS_OBJ_META_LEN
+s32_t SPIFFS_update_meta(spiffs *fs, const char *name, const void *meta) {
+#if SPIFFS_READ_ONLY
+  (void)fs; (void)name; (void)meta;
+  return SPIFFS_ERR_RO_NOT_IMPL;
+#else
+  SPIFFS_API_CHECK_CFG(fs);
+  SPIFFS_API_CHECK_MOUNT(fs);
+  SPIFFS_LOCK(fs);
+
+  spiffs_page_ix pix, pix_dummy;
+  spiffs_fd *fd;
+
+  s32_t res = spiffs_object_find_object_index_header_by_name(fs, (const u8_t*)name, &pix);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  res = spiffs_fd_find_new(fs, &fd, 0);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  res = spiffs_object_open_by_page(fs, pix, fd, 0, 0);
+  if (res != SPIFFS_OK) {
+    spiffs_fd_return(fs, fd->file_nbr);
+  }
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  res = spiffs_object_update_index_hdr(fs, fd, fd->obj_id, fd->objix_hdr_pix, 0, 0, meta,
       0, &pix_dummy);
 
   spiffs_fd_return(fs, fd->file_nbr);
@@ -864,7 +964,42 @@ s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *new) {
 #endif // SPIFFS_READ_ONLY
 }
 
+s32_t SPIFFS_fupdate_meta(spiffs *fs, spiffs_file fh, const void *meta) {
+#if SPIFFS_READ_ONLY
+  (void)fs; (void)fh; (void)meta;
+  return SPIFFS_ERR_RO_NOT_IMPL;
+#else
+  SPIFFS_API_CHECK_CFG(fs);
+  SPIFFS_API_CHECK_MOUNT(fs);
+  SPIFFS_LOCK(fs);
+
+  s32_t res;
+  spiffs_fd *fd;
+  spiffs_page_ix pix_dummy;
+
+  fh = SPIFFS_FH_UNOFFS(fs, fh);
+  res = spiffs_fd_get(fs, fh, &fd);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  if ((fd->flags & SPIFFS_O_WRONLY) == 0) {
+    res = SPIFFS_ERR_NOT_WRITABLE;
+    SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+  }
+
+  res = spiffs_object_update_index_hdr(fs, fd, fd->obj_id, fd->objix_hdr_pix, 0, 0, meta,
+      0, &pix_dummy);
+
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  SPIFFS_UNLOCK(fs);
+
+  return res;
+#endif // SPIFFS_READ_ONLY
+}
+#endif // SPIFFS_OBJ_META_LEN
+
 spiffs_DIR *SPIFFS_opendir(spiffs *fs, const char *name, spiffs_DIR *d) {
+  SPIFFS_API_DBG("%s\n", __func__);
   (void)name;
 
   if (!SPIFFS_CHECK_CFG((fs))) {
@@ -904,7 +1039,7 @@ static s32_t spiffs_read_dir_v(
   if (res != SPIFFS_OK) return res;
   if ((obj_id & SPIFFS_OBJ_ID_IX_FLAG) &&
       objix_hdr.p_hdr.span_ix == 0 &&
-      (objix_hdr.p_hdr.flags& (SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_FINAL | SPIFFS_PH_FLAG_IXDELE)) ==
+      (objix_hdr.p_hdr.flags & (SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_FINAL | SPIFFS_PH_FLAG_IXDELE)) ==
           (SPIFFS_PH_FLAG_DELET | SPIFFS_PH_FLAG_IXDELE)) {
     struct spiffs_dirent *e = (struct spiffs_dirent*)user_var_p;
     e->obj_id = obj_id;
@@ -912,13 +1047,16 @@ static s32_t spiffs_read_dir_v(
     e->type = objix_hdr.type;
     e->size = objix_hdr.size == SPIFFS_UNDEFINED_LEN ? 0 : objix_hdr.size;
     e->pix = pix;
+#if SPIFFS_OBJ_META_LEN
+    _SPIFFS_MEMCPY(e->meta, objix_hdr.meta, SPIFFS_OBJ_META_LEN);
+#endif
     return SPIFFS_OK;
   }
-
   return SPIFFS_VIS_COUNTINUE;
 }
 
 struct spiffs_dirent *SPIFFS_readdir(spiffs_DIR *d, struct spiffs_dirent *e) {
+  SPIFFS_API_DBG("%s\n", __func__);
   if (!SPIFFS_CHECK_MOUNT(d->fs)) {
     d->fs->err_code = SPIFFS_ERR_NOT_MOUNTED;
     return 0;
@@ -943,6 +1081,7 @@ struct spiffs_dirent *SPIFFS_readdir(spiffs_DIR *d, struct spiffs_dirent *e) {
   if (res == SPIFFS_OK) {
     d->block = bix;
     d->entry = entry + 1;
+    e->obj_id &= ~SPIFFS_OBJ_ID_IX_FLAG;
     ret = e;
   } else {
     d->fs->err_code = res;
@@ -952,12 +1091,14 @@ struct spiffs_dirent *SPIFFS_readdir(spiffs_DIR *d, struct spiffs_dirent *e) {
 }
 
 s32_t SPIFFS_closedir(spiffs_DIR *d) {
+  SPIFFS_API_DBG("%s\n", __func__);
   SPIFFS_API_CHECK_CFG(d->fs);
   SPIFFS_API_CHECK_MOUNT(d->fs);
   return 0;
 }
 
 s32_t SPIFFS_check(spiffs *fs) {
+  SPIFFS_API_DBG("%s\n", __func__);
 #if SPIFFS_READ_ONLY
   (void)fs;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -981,6 +1122,7 @@ s32_t SPIFFS_check(spiffs *fs) {
 }
 
 s32_t SPIFFS_info(spiffs *fs, u32_t *total, u32_t *used) {
+  SPIFFS_API_DBG("%s\n", __func__);
   s32_t res = SPIFFS_OK;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
@@ -1005,6 +1147,7 @@ s32_t SPIFFS_info(spiffs *fs, u32_t *total, u32_t *used) {
 }
 
 s32_t SPIFFS_gc_quick(spiffs *fs, u16_t max_free_pages) {
+  SPIFFS_API_DBG("%s "_SPIPRIi "\n", __func__, max_free_pages);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)max_free_pages;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -1024,6 +1167,7 @@ s32_t SPIFFS_gc_quick(spiffs *fs, u16_t max_free_pages) {
 
 
 s32_t SPIFFS_gc(spiffs *fs, u32_t size) {
+  SPIFFS_API_DBG("%s "_SPIPRIi "\n", __func__, size);
 #if SPIFFS_READ_ONLY
   (void)fs; (void)size;
   return SPIFFS_ERR_RO_NOT_IMPL;
@@ -1042,6 +1186,7 @@ s32_t SPIFFS_gc(spiffs *fs, u32_t size) {
 }
 
 s32_t SPIFFS_eof(spiffs *fs, spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
   s32_t res;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
@@ -1065,6 +1210,7 @@ s32_t SPIFFS_eof(spiffs *fs, spiffs_file fh) {
 }
 
 s32_t SPIFFS_tell(spiffs *fs, spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
   s32_t res;
   SPIFFS_API_CHECK_CFG(fs);
   SPIFFS_API_CHECK_MOUNT(fs);
@@ -1088,12 +1234,148 @@ s32_t SPIFFS_tell(spiffs *fs, spiffs_file fh) {
 }
 
 s32_t SPIFFS_set_file_callback_func(spiffs *fs, spiffs_file_callback cb_func) {
+  SPIFFS_API_DBG("%s\n", __func__);
   SPIFFS_LOCK(fs);
   fs->file_cb_f = cb_func;
   SPIFFS_UNLOCK(fs);
   return 0;
 }
 
+#if SPIFFS_IX_MAP
+
+s32_t SPIFFS_ix_map(spiffs *fs,  spiffs_file fh, spiffs_ix_map *map,
+    u32_t offset, u32_t len, spiffs_page_ix *map_buf) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd " "_SPIPRIi " "_SPIPRIi "\n", __func__, fh, offset, len);
+  s32_t res;
+  SPIFFS_API_CHECK_CFG(fs);
+  SPIFFS_API_CHECK_MOUNT(fs);
+  SPIFFS_LOCK(fs);
+
+  fh = SPIFFS_FH_UNOFFS(fs, fh);
+
+  spiffs_fd *fd;
+  res = spiffs_fd_get(fs, fh, &fd);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  if (fd->ix_map) {
+    SPIFFS_API_CHECK_RES_UNLOCK(fs, SPIFFS_ERR_IX_MAP_MAPPED);
+  }
+
+  map->map_buf = map_buf;
+  map->offset = offset;
+  // nb: spix range includes last
+  map->start_spix = offset / SPIFFS_DATA_PAGE_SIZE(fs);
+  map->end_spix = (offset + len) / SPIFFS_DATA_PAGE_SIZE(fs);
+  memset(map_buf, 0, sizeof(spiffs_page_ix) * (map->end_spix - map->start_spix + 1));
+  fd->ix_map = map;
+
+  // scan for pixes
+  res = spiffs_populate_ix_map(fs, fd, 0, map->end_spix - map->start_spix + 1);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  SPIFFS_UNLOCK(fs);
+  return res;
+}
+
+s32_t SPIFFS_ix_unmap(spiffs *fs,  spiffs_file fh) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd "\n", __func__, fh);
+  s32_t res;
+  SPIFFS_API_CHECK_CFG(fs);
+  SPIFFS_API_CHECK_MOUNT(fs);
+  SPIFFS_LOCK(fs);
+
+  fh = SPIFFS_FH_UNOFFS(fs, fh);
+
+  spiffs_fd *fd;
+  res = spiffs_fd_get(fs, fh, &fd);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  if (fd->ix_map == 0) {
+    SPIFFS_API_CHECK_RES_UNLOCK(fs, SPIFFS_ERR_IX_MAP_UNMAPPED);
+  }
+
+  fd->ix_map = 0;
+
+  SPIFFS_UNLOCK(fs);
+  return res;
+}
+
+s32_t SPIFFS_ix_remap(spiffs *fs, spiffs_file fh, u32_t offset) {
+  SPIFFS_API_DBG("%s "_SPIPRIfd " "_SPIPRIi "\n", __func__, fh, offset);
+  s32_t res = SPIFFS_OK;
+  SPIFFS_API_CHECK_CFG(fs);
+  SPIFFS_API_CHECK_MOUNT(fs);
+  SPIFFS_LOCK(fs);
+
+  fh = SPIFFS_FH_UNOFFS(fs, fh);
+
+  spiffs_fd *fd;
+  res = spiffs_fd_get(fs, fh, &fd);
+  SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+
+  if (fd->ix_map == 0) {
+    SPIFFS_API_CHECK_RES_UNLOCK(fs, SPIFFS_ERR_IX_MAP_UNMAPPED);
+  }
+
+  spiffs_ix_map *map = fd->ix_map;
+
+  s32_t spix_diff = offset / SPIFFS_DATA_PAGE_SIZE(fs) - map->start_spix;
+  map->offset = offset;
+
+  // move existing pixes if within map offs
+  if (spix_diff != 0) {
+    // move vector
+    int i;
+    const s32_t vec_len = map->end_spix - map->start_spix + 1; // spix range includes last
+    map->start_spix += spix_diff;
+    map->end_spix += spix_diff;
+    if (spix_diff >= vec_len) {
+      // moving beyond range
+      memset(&map->map_buf, 0, vec_len * sizeof(spiffs_page_ix));
+      // populate_ix_map is inclusive
+      res = spiffs_populate_ix_map(fs, fd, 0, vec_len-1);
+      SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+    } else if (spix_diff > 0) {
+      // diff positive
+      for (i = 0; i < vec_len - spix_diff; i++) {
+        map->map_buf[i] = map->map_buf[i + spix_diff];
+      }
+      // memset is non-inclusive
+      memset(&map->map_buf[vec_len - spix_diff], 0, spix_diff * sizeof(spiffs_page_ix));
+      // populate_ix_map is inclusive
+      res = spiffs_populate_ix_map(fs, fd, vec_len - spix_diff, vec_len-1);
+      SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+    } else {
+      // diff negative
+      for (i = vec_len - 1; i >= -spix_diff; i--) {
+        map->map_buf[i] = map->map_buf[i + spix_diff];
+      }
+      // memset is non-inclusive
+      memset(&map->map_buf[0], 0, -spix_diff * sizeof(spiffs_page_ix));
+      // populate_ix_map is inclusive
+      res = spiffs_populate_ix_map(fs, fd, 0, -spix_diff - 1);
+      SPIFFS_API_CHECK_RES_UNLOCK(fs, res);
+    }
+
+  }
+
+  SPIFFS_UNLOCK(fs);
+  return res;
+}
+
+s32_t SPIFFS_bytes_to_ix_map_entries(spiffs *fs, u32_t bytes) {
+  SPIFFS_API_CHECK_CFG(fs);
+  // always add one extra page, the offset might change to the middle of a page
+  return (bytes + SPIFFS_DATA_PAGE_SIZE(fs) ) / SPIFFS_DATA_PAGE_SIZE(fs);
+}
+
+s32_t SPIFFS_ix_map_entries_to_bytes(spiffs *fs, u32_t map_page_ix_entries) {
+  SPIFFS_API_CHECK_CFG(fs);
+  return map_page_ix_entries * SPIFFS_DATA_PAGE_SIZE(fs);
+}
+
+#endif // SPIFFS_IX_MAP
+
 #if SPIFFS_TEST_VISUALISATION
 s32_t SPIFFS_vis(spiffs *fs) {
   s32_t res = SPIFFS_OK;
@@ -1119,7 +1401,7 @@ s32_t SPIFFS_vis(spiffs *fs) {
           cur_entry - entry_offset < entries_per_page && cur_entry < (int)(SPIFFS_PAGES_PER_BLOCK(fs)-SPIFFS_OBJ_LOOKUP_PAGES(fs))) {
         spiffs_obj_id obj_id = obj_lu_buf[cur_entry-entry_offset];
         if (cur_entry == 0) {
-          spiffs_printf("%4i ", bix);
+          spiffs_printf(_SPIPRIbl" ", bix);
         } else if ((cur_entry & 0x3f) == 0) {
           spiffs_printf("     ");
         }
@@ -1147,7 +1429,7 @@ s32_t SPIFFS_vis(spiffs *fs) {
     SPIFFS_CHECK_RES(res);
 
     if (erase_count != (spiffs_obj_id)-1) {
-      spiffs_printf("\tera_cnt: %i\n", erase_count);
+      spiffs_printf("\tera_cnt: "_SPIPRIi"\n", erase_count);
     } else {
       spiffs_printf("\tera_cnt: N/A\n");
     }
@@ -1155,17 +1437,16 @@ s32_t SPIFFS_vis(spiffs *fs) {
     bix++;
   } // per block
 
-  spiffs_printf("era_cnt_max: %i\n", fs->max_erase_count);
+  spiffs_printf("era_cnt_max: "_SPIPRIi"\n", fs->max_erase_count);
-  spiffs_printf("last_errno:  %i\n", fs->err_code);
+  spiffs_printf("last_errno:  "_SPIPRIi"\n", fs->err_code);
-  spiffs_printf("blocks:      %i\n", fs->block_count);
+  spiffs_printf("blocks:      "_SPIPRIi"\n", fs->block_count);
-  spiffs_printf("free_blocks: %i\n", fs->free_blocks);
+  spiffs_printf("free_blocks: "_SPIPRIi"\n", fs->free_blocks);
-  spiffs_printf("page_alloc:  %i\n", fs->stats_p_allocated);
+  spiffs_printf("page_alloc:  "_SPIPRIi"\n", fs->stats_p_allocated);
-  spiffs_printf("page_delet:  %i\n", fs->stats_p_deleted);
+  spiffs_printf("page_delet:  "_SPIPRIi"\n", fs->stats_p_deleted);
+  SPIFFS_UNLOCK(fs);
   u32_t total, used;
   SPIFFS_info(fs, &total, &used);
-  spiffs_printf("used:        %i of %i\n", used, total);
+  spiffs_printf("used:        "_SPIPRIi" of "_SPIPRIi"\n", used, total);
-
-  SPIFFS_UNLOCK(fs);
   return res;
 }
 #endif

cores/esp8266/spiffs/spiffs_nucleus.h

@@ -116,13 +116,23 @@
 #define SPIFFS_ERR_CHECK_FLAGS_BAD      (SPIFFS_ERR_INTERNAL - 3)
 #define _SPIFFS_ERR_CHECK_LAST          (SPIFFS_ERR_INTERNAL - 4)
 
+// visitor result, continue searching
 #define SPIFFS_VIS_COUNTINUE            (SPIFFS_ERR_INTERNAL - 20)
+// visitor result, continue searching after reloading lu buffer
 #define SPIFFS_VIS_COUNTINUE_RELOAD     (SPIFFS_ERR_INTERNAL - 21)
+// visitor result, stop searching
 #define SPIFFS_VIS_END                  (SPIFFS_ERR_INTERNAL - 22)
 
-#define SPIFFS_EV_IX_UPD                0
+// updating an object index contents
-#define SPIFFS_EV_IX_NEW                1
+#define SPIFFS_EV_IX_UPD                (0)
-#define SPIFFS_EV_IX_DEL                2
+// creating a new object index
+#define SPIFFS_EV_IX_NEW                (1)
+// deleting an object index
+#define SPIFFS_EV_IX_DEL                (2)
+// moving an object index without updating contents
+#define SPIFFS_EV_IX_MOV                (3)
+// updating an object index header data only, not the table itself
+#define SPIFFS_EV_IX_UPD_HDR            (4)
 
 #define SPIFFS_OBJ_ID_IX_FLAG           ((spiffs_obj_id)(1<<(8*sizeof(spiffs_obj_id)-1)))
 
@@ -131,6 +141,22 @@
 #define SPIFFS_OBJ_ID_DELETED           ((spiffs_obj_id)0)
 #define SPIFFS_OBJ_ID_FREE              ((spiffs_obj_id)-1)
 
+
+
+#if defined(__GNUC__) || defined(__clang__)
+    /* For GCC and clang */
+#define SPIFFS_PACKED __attribute__((packed))
+#elif defined(__ICCARM__) || defined(__CC_ARM)
+    /* For IAR ARM and Keil MDK-ARM compilers */
+#define SPIFFS_PACKED 
+
+#else
+    /* Unknown compiler */
+#define SPIFFS_PACKED 
+#endif
+
+
+
 #if SPIFFS_USE_MAGIC
 #if !SPIFFS_USE_MAGIC_LENGTH
 #define SPIFFS_MAGIC(fs, bix)           \
@@ -228,6 +254,17 @@
 // object index span index number for given data span index or entry
 #define SPIFFS_OBJ_IX_ENTRY_SPAN_IX(fs, spix) \
   ((spix) < SPIFFS_OBJ_HDR_IX_LEN(fs) ? 0 : (1+((spix)-SPIFFS_OBJ_HDR_IX_LEN(fs))/SPIFFS_OBJ_IX_LEN(fs)))
+// get data span index for object index span index
+#define SPIFFS_DATA_SPAN_IX_FOR_OBJ_IX_SPAN_IX(fs, spix) \
+  ( (spix) == 0 ? 0 : (SPIFFS_OBJ_HDR_IX_LEN(fs) + (((spix)-1) * SPIFFS_OBJ_IX_LEN(fs))) )
+
+#if SPIFFS_FILEHDL_OFFSET
+#define SPIFFS_FH_OFFS(fs, fh)   ((fh) != 0 ? ((fh) + (fs)->cfg.fh_ix_offset) : 0)
+#define SPIFFS_FH_UNOFFS(fs, fh) ((fh) != 0 ? ((fh) - (fs)->cfg.fh_ix_offset) : 0)
+#else
+#define SPIFFS_FH_OFFS(fs, fh)   (fh)
+#define SPIFFS_FH_UNOFFS(fs, fh) (fh)
+#endif
 
 
 #define SPIFFS_OP_T_OBJ_LU    (0<<0)
@@ -272,26 +309,26 @@
 #define SPIFFS_API_CHECK_MOUNT(fs) \
   if (!SPIFFS_CHECK_MOUNT((fs))) { \
     (fs)->err_code = SPIFFS_ERR_NOT_MOUNTED; \
-    return -1; \
+    return SPIFFS_ERR_NOT_MOUNTED; \
   }
 
 #define SPIFFS_API_CHECK_CFG(fs) \
   if (!SPIFFS_CHECK_CFG((fs))) { \
     (fs)->err_code = SPIFFS_ERR_NOT_CONFIGURED; \
-    return -1; \
+    return SPIFFS_ERR_NOT_CONFIGURED; \
   }
 
 #define SPIFFS_API_CHECK_RES(fs, res) \
   if ((res) < SPIFFS_OK) { \
     (fs)->err_code = (res); \
-    return -1; \
+    return (res); \
   }
 
 #define SPIFFS_API_CHECK_RES_UNLOCK(fs, res) \
   if ((res) < SPIFFS_OK) { \
     (fs)->err_code = (res); \
     SPIFFS_UNLOCK(fs); \
-    return -1; \
+    return (res); \
   }
 
 #define SPIFFS_VALIDATE_OBJIX(ph, objid, spix) \
@@ -312,7 +349,7 @@
     if ((ph).span_ix != (spix)) return SPIFFS_ERR_DATA_SPAN_MISMATCH;
 
 
-// check id
+// check id, only visit matching objec ids
 #define SPIFFS_VIS_CHECK_ID     (1<<0)
 // report argument object id to visitor - else object lookup id is reported
 #define SPIFFS_VIS_CHECK_PH     (1<<1)
@@ -418,13 +455,23 @@ typedef struct {
   spiffs_span_ix cursor_objix_spix;
   // current absolute offset
   u32_t offset;
-  // current file descriptor offset
+  // current file descriptor offset (cached)
   u32_t fdoffset;
   // fd flags
   spiffs_flags flags;
 #if SPIFFS_CACHE_WR
   spiffs_cache_page *cache_page;
 #endif
+#if SPIFFS_TEMPORAL_FD_CACHE
+  // djb2 hash of filename
+  u32_t name_hash;
+  // hit score (score == 0 indicates never used fd)
+  u16_t score;
+#endif
+#if SPIFFS_IX_MAP
+  // spiffs index map, if 0 it means unmapped
+  spiffs_ix_map *ix_map;
+#endif
 } spiffs_fd;
 
 
@@ -433,7 +480,7 @@ typedef struct {
 // page header, part of each page except object lookup pages
 // NB: this is always aligned when the data page is an object index,
 // as in this case struct spiffs_page_object_ix is used
-typedef struct __attribute(( packed )) {
+typedef struct SPIFFS_PACKED {
   // object id
   spiffs_obj_id obj_id;
   // object span index
@@ -443,7 +490,7 @@ typedef struct __attribute(( packed )) {
 } spiffs_page_header;
 
 // object index header page header
-typedef struct __attribute(( packed ))
+typedef struct SPIFFS_PACKED
 #if SPIFFS_ALIGNED_OBJECT_INDEX_TABLES
                 __attribute(( aligned(sizeof(spiffs_page_ix)) ))
 #endif
@@ -458,10 +505,14 @@ typedef struct __attribute(( packed ))
   spiffs_obj_type type;
   // name of object
   u8_t name[SPIFFS_OBJ_NAME_LEN];
+#if SPIFFS_OBJ_META_LEN
+  // metadata. not interpreted by SPIFFS in any way.
+  u8_t meta[SPIFFS_OBJ_META_LEN];
+#endif
 } spiffs_page_object_ix_header;
 
 // object index page header
-typedef struct __attribute(( packed )) {
+typedef struct SPIFFS_PACKED {
  spiffs_page_header p_hdr;
  u8_t _align[4 - ((sizeof(spiffs_page_header)&3)==0 ? 4 : (sizeof(spiffs_page_header)&3))];
 } spiffs_page_object_ix;
@@ -612,7 +663,8 @@ s32_t spiffs_page_delete(
 s32_t spiffs_object_create(
     spiffs *fs,
     spiffs_obj_id obj_id,
-    const u8_t name[SPIFFS_OBJ_NAME_LEN],
+    const u8_t name[],
+    const u8_t meta[],
     spiffs_obj_type type,
     spiffs_page_ix *objix_hdr_pix);
 
@@ -622,13 +674,24 @@ s32_t spiffs_object_update_index_hdr(
     spiffs_obj_id obj_id,
     spiffs_page_ix objix_hdr_pix,
     u8_t *new_objix_hdr_data,
-    const u8_t name[SPIFFS_OBJ_NAME_LEN],
+    const u8_t name[],
+    const u8_t meta[],
     u32_t size,
     spiffs_page_ix *new_pix);
 
-void spiffs_cb_object_event(
+#if SPIFFS_IX_MAP
+
+s32_t spiffs_populate_ix_map(
     spiffs *fs,
     spiffs_fd *fd,
+    u32_t vec_entry_start,
+    u32_t vec_entry_end);
+
+#endif
+
+void spiffs_cb_object_event(
+    spiffs *fs,
+    spiffs_page_object_ix *objix,
     int ev,
     spiffs_obj_id obj_id,
     spiffs_span_ix spix,
@@ -704,7 +767,8 @@ s32_t spiffs_gc_quick(
 
 s32_t spiffs_fd_find_new(
     spiffs *fs,
-    spiffs_fd **fd);
+    spiffs_fd **fd,
+    const char *name);
 
 s32_t spiffs_fd_return(
     spiffs *fs,
@@ -715,6 +779,13 @@ s32_t spiffs_fd_get(
     spiffs_file f,
     spiffs_fd **fd);
 
+#if SPIFFS_TEMPORAL_FD_CACHE
+void spiffs_fd_temporal_cache_rehash(
+    spiffs *fs,
+    const char *old_path,
+    const char *new_path);
+#endif
+
 #if SPIFFS_CACHE
 void spiffs_cache_init(
     spiffs *fs);
@@ -748,4 +819,24 @@ s32_t spiffs_page_consistency_check(
 s32_t spiffs_object_index_consistency_check(
     spiffs *fs);
 
+// memcpy macro,
+// checked in test builds, otherwise plain memcpy (unless already defined)
+#ifdef _SPIFFS_TEST
+#define _SPIFFS_MEMCPY(__d, __s, __l) do { \
+    intptr_t __a1 = (intptr_t)((u8_t*)(__s)); \
+    intptr_t __a2 = (intptr_t)((u8_t*)(__s)+(__l)); \
+    intptr_t __b1 = (intptr_t)((u8_t*)(__d)); \
+    intptr_t __b2 = (intptr_t)((u8_t*)(__d)+(__l)); \
+    if (__a1 <= __b2 && __b1 <= __a2) { \
+      printf("FATAL OVERLAP: memcpy from %lx..%lx to %lx..%lx\n", __a1, __a2, __b1, __b2); \
+      ERREXIT(); \
+    } \
+    memcpy((__d),(__s),(__l)); \
+} while (0)
+#else
+#ifndef _SPIFFS_MEMCPY
+#define _SPIFFS_MEMCPY(__d, __s, __l) do{memcpy((__d),(__s),(__l));}while(0)
+#endif
+#endif //_SPIFFS_TEST
+
 #endif /* SPIFFS_NUCLEUS_H_ */

cores/esp8266/spiffs_hal.cpp

@@ -60,7 +60,7 @@ int32_t spiffs_hal_read(uint32_t addr, uint32_t size, uint8_t *dst) {
                 __LINE__, addr, size, alignedBegin, alignedEnd);
             return SPIFFS_ERR_INTERNAL;
         }
-        memcpy(dst, &tmp + 4 - nb, nb);
+        memcpy(dst, ((uint8_t*) &tmp) + 4 - nb, nb);
     }
 
     if (alignedEnd != alignedBegin) {

cores/esp8266/time.c

@@ -17,9 +17,10 @@
  */
 
 #include <time.h>
+#include <sys/time.h>
 #include <sys/reent.h>
 #include "sntp.h"
-
+#include "coredecls.h"
 
 #ifndef _TIMEVAL_DEFINED
 #define _TIMEVAL_DEFINED
@@ -31,25 +32,18 @@ struct timeval {
 
 extern char* sntp_asctime(const struct tm *t);
 extern struct tm* sntp_localtime(const time_t *clock);
+extern uint64_t micros64();
 
 // time gap in seconds from 01.01.1900 (NTP time) to 01.01.1970 (UNIX time)
 #define DIFF1900TO1970 2208988800UL
 
-static int s_daylightOffset_sec = 0;
+bool timeshift64_is_set = false;
-static long s_timezone_sec = 0;
+static uint64_t timeshift64 = 0;
-static time_t s_bootTime = 0;
 
-// calculate offset used in gettimeofday
+void tune_timeshift64 (uint64_t now_us)
-static void ensureBootTimeIsSet()
 {
-    if (!s_bootTime)
+     timeshift64 = now_us - micros64();
-    {
+     timeshift64_is_set = true;
-        time_t now = sntp_get_current_timestamp();
-        if (now)
-        {
-            s_bootTime =  now - millis() / 1000;
-        }
-    }
 }
 
 static void setServer(int id, const char* name_or_ip)
@@ -69,17 +63,17 @@ void configTime(int timezone, int daylightOffset_sec, const char* server1, const
     setServer(1, server2);
     setServer(2, server3);
 
-    s_timezone_sec = timezone;
-    s_daylightOffset_sec = daylightOffset_sec;
     sntp_set_timezone(timezone/3600);
+    sntp_set_daylight(daylightOffset_sec);
     sntp_init();
 }
 
 int clock_gettime(clockid_t unused, struct timespec *tp)
 {
     (void) unused;
-    tp->tv_sec  = millis() / 1000;
+    uint64_t m = micros64();
-    tp->tv_nsec = micros() * 1000;
+    tp->tv_sec = m / 1000000;
+    tp->tv_nsec = (m % 1000000) * 1000;
     return 0;
 }
 
@@ -99,9 +93,11 @@ int _gettimeofday_r(struct _reent* unused, struct timeval *tp, void *tzp)
     (void) tzp;
     if (tp)
     {
-        ensureBootTimeIsSet();
+        if (!timeshift64_is_set)
-        tp->tv_sec  = s_bootTime + millis() / 1000;
+            tune_timeshift64(sntp_get_current_timestamp() * 1000000ULL);
-        tp->tv_usec = micros() * 1000;
+        uint64_t currentTime_us = timeshift64 + micros64();
+        tp->tv_sec = currentTime_us / 1000000ULL;
+        tp->tv_usec = currentTime_us % 1000000ULL;
     }
     return 0;
 }

cores/esp8266/uart.c

@@ -108,6 +108,9 @@ int uart_peek_char(uart_t* uart)
 
 int uart_read_char(uart_t* uart)
 {
+    if(uart == NULL) {
+        return -1;
+    }
     int data = uart_peek_char(uart);
     if(data != -1) {
         uart->rx_buffer->rpos = (uart->rx_buffer->rpos + 1) % uart->rx_buffer->size;

cores/esp8266/umm_malloc/umm_malloc.c

@@ -493,6 +493,7 @@
 
 #include <stdio.h>
 #include <string.h>
+#include <pgmspace.h>
 
 #include "umm_malloc.h"
 
@@ -512,6 +513,9 @@
 #  define DBG_LOG_LEVEL DBG_LOG_LEVEL
 #endif
 
+// Macro to place constant strings into PROGMEM and print them properly
+#define printf(fmt, ...)  do { static const char fstr[] PROGMEM = fmt; char rstr[sizeof(fmt)]; for (size_t i=0; i<sizeof(rstr); i++) rstr[i] = fstr[i]; printf(rstr, ##__VA_ARGS__); } while (0)
+
 /* -- dbglog {{{ */
 
 /* ----------------------------------------------------------------------------
@@ -1628,9 +1632,9 @@ static void *_umm_realloc( void *ptr, size_t size ) {
 
     if( (ptr = _umm_malloc( size )) ) {
       memcpy( ptr, oldptr, curSize );
+      _umm_free( oldptr );
     }
 
-    _umm_free( oldptr );
   }
 
   /* Release the critical section... */

doc/boards.rst

@@ -58,7 +58,7 @@ This module is sold under many names for around $6.50 on AliExpress and it's one
 
 It's an open hardware design with an ESP-12E core and 4 MB of SPI flash.
 
-Acording to the manufacturer, "with a micro USB cable, you can connect NodeMCU devkit to your laptop and flash it without any trouble". This is more or less true: the board comes with a CP2102 onboard USB to serial adapter which just works, well, the majority of the time. Sometimes flashing fails and you have to reset the board by holding down FLASH +
+According to the manufacturer, "with a micro USB cable, you can connect NodeMCU devkit to your laptop and flash it without any trouble". This is more or less true: the board comes with a CP2102 onboard USB to serial adapter which just works, well, the majority of the time. Sometimes flashing fails and you have to reset the board by holding down FLASH +
 RST, then releasing FLASH, then releasing RST. This forces the CP2102 device to power cycle and to be re-numbered by Linux.
 
 The board also features a NCP1117 voltage regulator, a blue LED on GPIO16 and a 220k/100k Ohm voltage divider on the ADC input pin.
@@ -131,6 +131,15 @@ WifInfo integrates the ESP-12 or ESP-07+Ext antenna module with a 3.3v regulator
 
 For more information, please see WifInfo related `blog <http://hallard.me/category/wifinfo/>`__ entries, `github <https://github.com/hallard/WifInfo>`__ and `community <https://community.hallard.me/category/16/wifinfo>`__ forum.
 
+DigiStump Oak
+-------------
+
+The Oak requires an [adapter](#serial-adapter) for a serial connection or flashing; its micro USB port is only for power.
+
+To make a serial connection, wire the adapter's **TX to P3**, **RX to P4**, and **GND** to **GND**.  Supply 3.3v from the serial adapter if not already powered via USB.
+
+To put the board into bootloader mode, configure a serial connection as above, connect **P2 to GND**, then re-apply power.  Once flashing is complete, remove the connection from P2 to GND, then re-apply power to boot into normal mode.
+
 Generic ESP8266 modules
 -----------------------
 

doc/esp8266wifi/client-examples.rst

@@ -249,7 +249,7 @@ In case server's web address is incorrect, or server is not accessible, you shou
 Conclusion
 ~~~~~~~~~~
 
-With this simple example we have demonstrated how to set up a client program, connect it to a server, request a web page and retrieve it. Now you should be able to write your own client program for ESP8266 and move to more advanced dialogue with a server, like e.g. using `HTTPS <https://en.wikipedia.org/wiki/HTTPS>`__ protocol with the `Client Secure <readme.md#client-secure>`__.
+With this simple example we have demonstrated how to set up a client program, connect it to a server, request a web page and retrieve it. Now you should be able to write your own client program for ESP8266 and move to more advanced dialogue with a server, like e.g. using `HTTPS <https://en.wikipedia.org/wiki/HTTPS>`__ protocol with the :doc:`Client Secure <client-secure-examples>` .
 
 For more client examples please check
 

doc/esp8266wifi/client-secure-class.rst

@@ -5,6 +5,29 @@ Client Secure Class
 
 Methods and properties described in this section are specific to ESP8266. They are not covered in `Arduino WiFi library <https://www.arduino.cc/en/Reference/WiFi>`__ documentation. Before they are fully documented please refer to information below.
 
+Supported crypto
+~~~~~~~~~~~~~~~~
+
+In the background the library `axtls <http://axtls.sourceforge.net>`_ is used. The library supports only rsa certificates and no new eliptic curve certificates. TLSv1.2 is supported since SDK 2.4.0-rc1.
+
+The following ciphers and digests are supported by `specification <http://axtls.sourceforge.net/specifications.htm>`_:
+
+* Symmetric Ciphers
+    * AES128-SHA
+    * AES256-SHA
+    * AES128-SHA256
+    * AES256-SHA256
+* Asymmetric Ciphers
+    * RSA 512/1024/2048/4096 bit encryption/decryption.
+    * RSA signing/verification
+* Digests
+    * SHA1
+    * MD5
+    * SHA256/384/512
+    * HMAC-SHA1
+    * HMAC-MD5
+    * HMAC-SHA256
+
 loadCertificate
 ~~~~~~~~~~~~~~~
 

doc/esp8266wifi/client-secure-examples.rst

@@ -178,4 +178,4 @@ Conclusion
 Programming a secure client is almost identical as programming a non-secure client. The difference gets down to one extra step to verify server's identity. Keep in mind limitations due to heavy memory usage that depends on the strength of the key used by the server and whether server is willing to negotiate the `TLS buffer size <https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/>`__.
 
 For the list of functions provided to manage secure clients, please refer to the `Client Secure Class
-:arrow\_right: <client-secure-class.md>`__ documentation.
+:arrow\_right: <client-secure-class.rst>`__ documentation.

doc/esp8266wifi/generic-class.rst

@@ -79,4 +79,4 @@ Other Function Calls
 
 Documentation for the above functions is not yet prepared.
 
-For code samples please refer to separate section with `examples <generic-examples.md>`__ dedicated specifically to the Generic Class.
+For code samples please refer to separate section with `examples <generic-examples.rst>`__ dedicated specifically to the Generic Class.

doc/esp8266wifi/readme.rst

@@ -233,7 +233,13 @@ Some functions provide more than just a binary status information. A good exampl
 
     Serial.printf("Connection status: %d\n", WiFi.status());
 
-This function returns following codes to describe what is going on with Wi-Fi connection: \* 0 : ``WL_IDLE_STATUS`` when Wi-Fi is in process of changing between statuses \* 1 : ``WL_NO_SSID_AVAIL``\ in case configured SSID cannot be reached \* 3 : ``WL_CONNECTED`` after successful connection is established \* 4 : ``WL_CONNECT_FAILED`` if password is incorrect \* 6 : ``WL_DISCONNECTED`` if module is not configured in station mode
+This function returns following codes to describe what is going on with Wi-Fi connection: 
+
+* 0 : ``WL_IDLE_STATUS`` when Wi-Fi is in process of changing between statuses 
+* 1 : ``WL_NO_SSID_AVAIL``\ in case configured SSID cannot be reached 
+* 3 : ``WL_CONNECTED`` after successful connection is established 
+* 4 : ``WL_CONNECT_FAILED`` if password is incorrect 
+* 6 : ``WL_DISCONNECTED`` if module is not configured in station mode
 
 It is a good practice to display and check information returned by functions. Application development and troubleshooting will be easier with that.
 

doc/exception_causes.rst

@@ -1,124 +1,95 @@
 Exception Causes (EXCCAUSE)
 ===========================
 
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| EXCCAU | Cause Name | Cause Description                       | Required  | EXCVAD |
+| EXCCAUSE | Cause Name                     | Cause Description                       | Required    | EXCVADDR |
-| SE     |            |                                         | Option    | DR     |
+| Code     |                                |                                         | Option      | Loaded   |
-| Code   |            |                                         |           | Loaded |
++==========+================================+=========================================+=============+==========+
-+========+============+=========================================+===========+========+
+| 0        | IllegalInstructionCause        | Illegal instruction                     | Exception   | No       |
-| 0      | IllegalIns | Illegal instruction                     | Exception | No     |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | tructionCa |                                         |           |        |
+| 1        | SyscallCause                   | SYSCALL instruction                     | Exception   | No       |
-|        | use        |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 2        | InstructionFetchErrorCause     | Processor internal physical address or  | Exception   | Yes      |
-| 1      | SyscallCau | SYSCALL instruction                     | Exception | No     |
+|          |                                | data error during instruction fetch     |             |          |
-|        | se         |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 3        | LoadStoreErrorCause            | Processor internal physical address or  | Exception   | Yes      |
-| 2      | Instructio | Processor internal physical address or  | Exception | Yes    |
+|          |                                | data error during load or store         |             |          |
-|        | nFetchErro | data error during instruction fetch     |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | rCause     |                                         |           |        |
+| 4        | Level1InterruptCause           | Level-1 interrupt as indicated by set   | Interrupt   | No       |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                | level-1 bits in the INTERRUPT register  |             |          |
-| 3      | LoadStoreE | Processor internal physical address or  | Exception | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | rrorCause  | data error during load or store         |           |        |
+| 5        | AllocaCause                    | MOVSP instruction, if caller’s          | Windowed    | No       |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                | registers are not in the register file  | Register    |          |
-| 4      | Level1Inte | Level-1 interrupt as indicated by set   | Interrupt | No     |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | rruptCause | level-1 bits in the INTERRUPT register  |           |        |
+| 6        | IntegerDivideByZeroCause       | QUOS, QUOU, REMS, or REMU divisor       | 32-bit      | No       |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                | operand is zero                         | Integer     |          |
-| 5      | AllocaCaus | MOVSP instruction, if caller’s          | Windowed  | No     |
+|          |                                |                                         | Divide      |          |
-|        | e          | registers are not in the register file  | Register  |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 7        | Reserved for Tensilica         |                                         |             |          |
-| 6      | IntegerDiv | QUOS, QUOU, REMS, or REMU divisor       | 32-bit    | No     |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | ideByZeroC | operand is zero                         | Integer   |        |
+| 8        | PrivilegedCause                | Attempt to execute a privileged         | MMU         | No       |
-|        | ause       |                                         | Divide    |        |
+|          |                                | operation when CRING != 0               |             |          |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 7      | Reserved   |                                         |           |        |
+| 9        | LoadStoreAlignmentCause        | Load or store to an unaligned address   | Unaligned   | Yes      |
-|        | for        |                                         |           |        |
+|          |                                |                                         | Exception   |          |
-|        | Tensilica  |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 10..11   | Reserved for Tensilica         |                                         |             |          |
-| 8      | Privileged | Attempt to execute a privileged         | MMU       | No     |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | Cause      | operation when CRING != 0               |           |        |
+| 12       | InstrPIFDateErrorCause         | PIF data error during instruction fetch | Processor   | Yes      |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                |                                         | Interface   |          |
-| 9      | LoadStoreA | Load or store to an unaligned address   | Unaligned | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | lignmentCa |                                         | Exception |        |
+| 13       | LoadStorePIFDataErrorCause     | Synchronous PIF data error during       | Processor   | Yes      |
-|        | use        |                                         |           |        |
+|          |                                | LoadStore access                        | Interface   |          |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 10..11 | Reserved   |                                         |           |        |
+| 14       | InstrPIFAddrErrorCause         | PIF address error during instruction    | Processor   | Yes      |
-|        | for        |                                         |           |        |
+|          |                                | fetch                                   | Interface   |          |
-|        | Tensilica  |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 15       | LoadStorePIFAddrErrorCause     | Synchronous PIF address error during    | Processor   | Yes      |
-| 12     | InstrPIFDa | PIF data error during instruction fetch | Processor | Yes    |
+|          |                                | LoadStore access                        | Interface   |          |
-|        | taErrorCau |                                         | Interface |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | se         |                                         |           |        |
+| 16       | InstTLBMissCause               | Error during Instruction TLB refill     | MMU         | Yes      |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 13     | LoadStoreP | Synchronous PIF data error during       | Processor | Yes    |
+| 17       | InstTLBMultiHitCause           | Multiple instruction TLB entries        | MMU         | Yes      |
-|        | IFDataErro | LoadStore access                        | Interface |        |
+|          |                                | matched                                 |             |          |
-|        | rCause     |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 18       | InstFetchPrivilegeCause        | An instruction fetch referenced a       | MMU         | Yes      |
-| 14     | InstrPIFAd | PIF address error during instruction    | Processor | Yes    |
+|          |                                | virtual address at a ring level less    |             |          |
-|        | drErrorCau | fetch                                   | Interface |        |
+|          |                                | than CRING                              |             |          |
-|        | se         |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 19       | Reserved for Tensilica         |                                         |             |          |
-| 15     | LoadStoreP | Synchronous PIF address error during    | Processor | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | IFAddrErro | LoadStore access                        | Interface |        |
+| 20       | InstFetchProhibitedCause       | An instruction fetch referenced a page  | Region      | Yes      |
-|        | rCause     |                                         |           |        |
+|          |                                | mapped with an attribute that does not  | Protection  |          |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                | permit instruction fetch                | or MMU      |          |
-| 16     | InstTLBMis | Error during Instruction TLB refill     | MMU       | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | sCause     |                                         |           |        |
+| 21..23   | Reserved for Tensilica         |                                         |             |          |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 17     | InstTLBMul | Multiple instruction TLB entries        | MMU       | Yes    |
+| 24       | LoadStoreTLBMissCause          | Error during TLB refill for a load or   | MMU         | Yes      |
-|        | tiHitCause | matched                                 |           |        |
+|          |                                | store                                   |             |          |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 18     | InstFetchP | An instruction fetch referenced a       | MMU       | Yes    |
+| 25       | LoadStoreTLBMultiHitCause      | Multiple TLB entries matched for a load | MMU         | Yes      |
-|        | rivilegeCa | virtual address at a ring level less    |           |        |
+|          |                                | or store                                |             |          |
-|        | use        | than CRING                              |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 26       | LoadStorePrivilegeCause        | A load or store referenced a virtual    | MMU         | Yes      |
-| 19     | Reserved   |                                         |           |        |
+|          |                                | address at a ring level less than CRING |             |          |
-|        | for        |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | Tensilica  |                                         |           |        |
+| 27       | Reserved for Tensilica         |                                         |             |          |
-+--------+------------+-----------------------------------------+-----------+--------+
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-| 20     | InstFetchP | An instruction fetch referenced a page  | Region    | Yes    |
+| 28       | LoadProhibitedCause            | A load referenced a page mapped with an | Region      | Yes      |
-|        | rohibitedC | mapped with an attribute that does not  | Protectio |        |
+|          |                                | attribute that does not permit loads    | Protection  |          |
-|        | ause       | permit instruction fetch                | n         |        |
+|          |                                |                                         | or MMU      |          |
-|        |            |                                         | or MMU    |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 29       | StoreProhibitedCause           | A store referenced a page mapped with   | Region      | Yes      |
-| 21..23 | Reserved   |                                         |           |        |
+|          |                                | an attribute that does not permit       | Protection  |          |
-|        | for        |                                         |           |        |
+|          |                                |                                         | or MMU      |          |
-|        | Tensilica  |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
+| 30..31   | Reserved for Tensilica         |                                         |             |          |
-| 24     | LoadStoreT | Error during TLB refill for a load or   | MMU       | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | LBMissCaus | store                                   |           |        |
+| 32..39   | CoprocessornDisabled           | Coprocessor n instruction when cpn      | Coprocessor | No       |
-|        | e          |                                         |           |        |
+|          |                                | disabled. n varies 0..7 as the cause    |             |          |
-+--------+------------+-----------------------------------------+-----------+--------+
+|          |                                | varies 32..39                           |             |          |
-| 25     | LoadStoreT | Multiple TLB entries matched for a load | MMU       | Yes    |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-|        | LBMultiHit | or store                                |           |        |
+| 40..63   | Reserved                       |                                         |             |          |
-|        | Cause      |                                         |           |        |
++----------+--------------------------------+-----------------------------------------+-------------+----------+
-+--------+------------+-----------------------------------------+-----------+--------+
-| 26     | LoadStoreP | A load or store referenced a virtual    | MMU       | Yes    |
-|        | rivilegeCa | address at a ring level less than CRING |           |        |
-|        | use        |                                         |           |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 27     | Reserved   |                                         |           |        |
-|        | for        |                                         |           |        |
-|        | Tensilica  |                                         |           |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 28     | LoadProhib | A load referenced a page mapped with an | Region    | Yes    |
-|        | itedCause  | attribute that does not permit loads    | Protectio |        |
-|        |            |                                         | n         |        |
-|        |            |                                         | or MMU    |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 29     | StoreProhi | A store referenced a page mapped with   | Region    | Yes    |
-|        | bitedCause | an attribute that does not permit       | Protectio |        |
-|        |            | stores                                  | n         |        |
-|        |            |                                         | or MMU    |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 30..31 | Reserved   |                                         |           |        |
-|        | for        |                                         |           |        |
-|        | Tensilica  |                                         |           |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 32..39 | Coprocesso | Coprocessor n instruction when cpn      | Coprocess | No     |
-|        | rnDisabled | disabled. n varies 0..7 as the cause    | or        |        |
-|        |            | varies 32..39                           |           |        |
-+--------+------------+-----------------------------------------+-----------+--------+
-| 40..63 | Reserved   |                                         |           |        |
-+--------+------------+-----------------------------------------+-----------+--------+
 
 Infos from Xtensa Instruction Set Architecture (ISA) Reference Manual

doc/faq/a02-my-esp-crashes.rst

@@ -10,6 +10,7 @@ My ESP crashes running some code. How to troubleshoot it?
 -  `Exception <#exception>`__
 -  `Watchdog <#watchdog>`__
 -  `Check Where the Code Crashes <#check-where-the-code-crashes>`__
+-  `Other Causes for Crashes <#other-causes-for-crashes>`__
 -  `If at the Wall, Enter an Issue
    Report <#if-at-the-wall-enter-an-issue-report>`__
 -  `Conclusion <#conclusion>`__
@@ -229,6 +230,77 @@ dropped. The same procedure applies to crashes caused by exceptions.
     able to correctly decode the stack trace dropped by some other
     application not compiled and loaded from your Arduino IDE.
 
+
+Other Causes for Crashes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Interrupt Service Routines
+   By default, all functions are compiled into flash, which means that the 
+   cache may kick in for that code. However, the cache currently can't be used 
+   during hardware interrupts. That means that, if you use a hardware ISR, such as 
+   attachInterrupt(gpio, myISR, CHANGE) for a GPIO change, the ISR must have the 
+   ICACHE_RAM_ATTR attribute declared. Not only that, but the entire function tree 
+   called from the ISR must also have the ICACHE_RAM_ATTR declared.
+   Be aware that every function that has this attribute reduces available memory.
+
+   In addition, it is not possible to execute delay() or yield() from an ISR, 
+   or do blocking operations, or operations that disable the interrupts, e.g.: read
+   a DHT.
+
+   Finally, an ISR has very high restrictions on timing for the executed code, meaning
+   that executed code should not take longer than a very few microseconds. It is 
+   considered best practice to set a flag within the ISR, and then from within the loop()
+   check and clear that flag, and execute code.
+
+Asynchronous Callbacks
+   Asynchronous CBs, such as for the Ticker or ESPAsync* libs, have looser restrictions
+   than ISRs, but some restrictions still apply.
+   It is not possible to execute delay() or yield() from an asynchronous callback.
+   Timing is not as tight as an ISR, but it should remain below a few milliseconds. This
+   is a guideline. The hard timing requirements depend on the WiFi configuration and 
+   amount of traffic. In general, the CPU must not be hogged by the user code, as the
+   longer it is away from servicing the WiFi stack, the more likely that memory corruption
+   can happen.
+
+Memory, memory, memory
+   Running out of heap is the most common cause for crashes. Because the build process for
+   the ESP leaves out exceptions (they use memory), memory allocations that fail will do
+   so silently. A typical example is when setting or concatenating a large String. If 
+   allocation has failed internally, then the internal string copy can corrupt data, and 
+   the ESP will crash.
+
+   In addition, doing many String concatenations in sequence, e.g.: using operator+()
+   multiple times, will cause memory fragmentation. When that happens, allocations may
+   silently fail even though there is enough total heap available. The reason for the
+   failure is that an allocation requires finding a single free memory block that is large
+   enough for the size being requested. A sequence of String concatenations causes many
+   allocations/deallocations/reallocations, which makes "holes" in the memory map. After
+   many such operations, it can happen that all available holes are too small to comply
+   with the requested size, even though the sum of all holes is greater than the requested
+   size.
+
+   So why do these silent failures exist? On the one hand, there are specific interfaces that
+   must be adhered to. For example, the String object methods don't allow for error handling
+   at the user application level (i.e.: no old-school error returns).
+   On the other hand, some libraries don't have the allocation code accessible for
+   modification. For example, std::vector is available for use. The standard implementations
+   rely on exceptions for error handling, which is not available for the ESP, and in any
+   case there is no access to the underlying code.
+
+*Some techniques for reducing memory usage*
+
+   * Don't use const char * with literals. Instead, use const char[] PROGMEM. This is particularly true if you intend to, e.g.: embed html strings.
+   * Don't use global static arrays, such as uint8_t buffer[1024]. Instead, allocate dynamically. This forces you to think about the size of the array, and its scope (lifetime), so that it gets released when it's no longer needed. If you are not certain about dynamic allocation, use std libs (e.g.: std:vector, std::string), or smart pointers. They are slightly less memory efficient than dynamically allocating yourself, but the provided memory safety is well worth it.
+   * If you use std libs like std::vector, make sure to call its ::reserve() method before filling it. This allows allocating only once, which reduces mem fragmentation, and makes sure that there are no empty unused slots left over in the container at the end.
+
+Stack
+   The amount of stack in the ESP is tiny at only 4KB. For normal developement in large systems, it 
+   is good practice to use and abuse the stack, because it is faster for allocation/deallocation, the scope of the object is well defined, and deallocation automatically happens in reverse order as allocation, which means no mem fragmentation. However, with the tiny amount of stack available in the ESP, that practice is not really viable, at least not for big objects.
+      * Large objects that have internally managed memory, such as String, std::string, std::vector, etc, are ok on the stack, because they internally allocate their buffers on the heap.
+      * Large arrays on the stack, such as uint8_t buffer[2048] should be avoided on the stack and be dynamically allocated (consider smart pointers).
+      * Objects that have large data members, such as large arrays, should be avoided on the stack, and be dynamicaly allocated (consider smart pointers).
+
+
 If at the Wall, Enter an Issue Report
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/libraries.rst

@@ -11,7 +11,7 @@ ESP8266WiFi library has been developed basing on ESP8266 SDK, using naming conve
 Ticker
 ------
 
-Library for calling functions repeatedly with a certain period. Two examples included.
+Library for calling functions repeatedly with a certain period. `Two examples <https://github.com/esp8266/Arduino/tree/master/libraries/Ticker/examples>`__ included.
 
 It is currently not recommended to do blocking IO operations (network, serial, file) from Ticker callback functions. Instead, set a flag inside the ticker callback and check for that flag inside the loop function.
 
@@ -27,7 +27,7 @@ This is a bit different from standard EEPROM class. You need to call ``EEPROM.be
 
 EEPROM library uses one sector of flash located just after the SPIFFS.
 
-Three examples included.
+`Three examples <https://github.com/esp8266/Arduino/tree/master/libraries/EEPROM>`__  included.
 
 I2C (Wire library)
 ------------------

doc/ota_updates/readme.rst

@@ -57,7 +57,7 @@ Basic Requirements
 
 Flash chip size should be able to hold the old sketch (currently running) and the new sketch (OTA) at the same time.
 
-Keep in mind that the File system and EEPROM for example needs space too (one time) see `flash layout <../filesystem.md#flash-layout>`__.
+Keep in mind that the File system and EEPROM for example needs space too (one time) see `flash layout <../filesystem.rst#flash-layout>`__.
 
 .. code:: cpp
 
@@ -226,7 +226,7 @@ If this is the case, then most likely ESP module has not been reset after initia
 
 The most common causes of OTA failure are as follows: \* not enough physical memory on the chip (e.g. ESP01 with 512K flash memory is not enough for OTA), \* too much memory declared for SPIFFS so new sketch will not fit between existing sketch and SPIFFS – see `Update process - memory view <#update-process---memory-view>`__, \* too little memory declared in Arduino IDE for your selected board (i.e. less than physical size), \* not resetting the ESP module after initial upload using serial port.
 
-For more details regarding flash memory layout please check `File system <https://github.com/esp8266/Arduino/blob/master/doc/filesystem.md>`__. For overview where new sketch is stored, how it is copied and how memory is organized for the purpose of OTA see `Update process - memory view <#update-process---memory-view>`__.
+For more details regarding flash memory layout please check `File system <../filesystem.rst>`__. For overview where new sketch is stored, how it is copied and how memory is organized for the purpose of OTA see `Update process - memory view <#update-process---memory-view>`__.
 
 Web Browser
 -----------

doc/reference.rst

@@ -94,10 +94,11 @@ Serial
 ------
 
 ``Serial`` object works much the same way as on a regular Arduino. Apart
-from hardware FIFO (128 bytes for TX and RX) HardwareSerial has
+from hardware FIFO (128 bytes for TX and RX) ``Serial`` has
 additional 256-byte TX and RX buffers. Both transmit and receive is
 interrupt-driven. Write and read functions only block the sketch
-execution when the respective FIFO/buffers are full/empty.
+execution when the respective FIFO/buffers are full/empty. Note that
+the length of additional 256-bit buffer can be customized.
 
 ``Serial`` uses UART0, which is mapped to pins GPIO1 (TX) and GPIO3
 (RX). Serial may be remapped to GPIO15 (TX) and GPIO13 (RX) by calling
@@ -121,6 +122,9 @@ instead, call ``Serial1.setDebugOutput(true)``.
 You also need to use ``Serial.setDebugOutput(true)`` to enable output
 from ``printf()`` function.
 
+The method ``Serial.setRxBufferSize(size_t size)`` allows to define the
+receiving buffer depth. The default value is 256.
+
 Both ``Serial`` and ``Serial1`` objects support 5, 6, 7, 8 data bits,
 odd (O), even (E), and no (N) parity, and 1 or 2 stop bits. To set the
 desired mode, call ``Serial.begin(baudrate, SERIAL_8N1)``,
@@ -142,6 +146,8 @@ current speed. For example
     // Will print "Serial is 57600 bps"
     Serial.printf("Serial is %d bps", br);
 
+| ``Serial`` and ``Serial1`` objects are both instances of the
+  ``HardwareSerial`` class.
 | I've done this also for official ESP8266 `Software
   Serial <https://github.com/esp8266/Arduino/blob/master/doc/libraries.md#softwareserial>`__
   library, see this `pull

keywords.txt

@@ -0,0 +1,21 @@
+#######################################
+# Syntax Coloring Map
+#######################################
+
+#######################################
+# Datatypes (KEYWORD1)
+#######################################
+
+#######################################
+# Methods and Functions (KEYWORD2)
+#######################################
+analogWriteFreq	KEYWORD2
+analogWriteRange	KEYWORD2
+baudrate	KEYWORD2
+swap	KEYWORD2
+
+######################################
+# Constants (LITERAL1)
+#######################################
+INPUT_PULLDOWN_16	LITERAL1
+PWMRANGE	LITERAL1

libraries/ArduinoOTA/ArduinoOTA.cpp

@@ -229,7 +229,7 @@ void ArduinoOTAClass::_onRx(){
     String result = _challengemd5.toString();
 
     ota_ip.addr = (uint32_t)_ota_ip;
-    if(result.equals(response)){
+    if(result.equalsConstantTime(response)) {
       _state = OTA_RUNUPDATE;
     } else {
       _udp_ota->append("Authentication Failed", 21);

libraries/ArduinoOTA/keywords.txt

@@ -15,10 +15,10 @@ ArduinoOTA	KEYWORD1
 begin	KEYWORD2
 setup	KEYWORD2
 handle	KEYWORD2
-onStart KEYWORD2
+onStart	KEYWORD2
-onEnd   KEYWORD2
+onEnd	KEYWORD2
-onError KEYWORD2
+onError	KEYWORD2
-onProgress KEYWORD2
+onProgress	KEYWORD2
 
 #######################################
 # Constants (LITERAL1)

libraries/DNSServer/src/DNSServer.cpp

@@ -144,17 +144,9 @@ void DNSServer::replyWithIP()
 
 
 
-  #ifdef DEBUG
+  #ifdef DEBUG_ESP_DNS
-    DEBUG_OUTPUT.print("DNS responds: ");
+    DEBUG_ESP_PORT.printf("DNS responds: %s for %s\n",
-    DEBUG_OUTPUT.print(_resolvedIP[0]);
+            IPAddress(_resolvedIP).toString().c_str(), getDomainNameWithoutWwwPrefix().c_str() );
-    DEBUG_OUTPUT.print(".");
-    DEBUG_OUTPUT.print(_resolvedIP[1]);
-    DEBUG_OUTPUT.print(".");
-    DEBUG_OUTPUT.print(_resolvedIP[2]);
-    DEBUG_OUTPUT.print(".");
-    DEBUG_OUTPUT.print(_resolvedIP[3]);
-    DEBUG_OUTPUT.print(" for ");
-    DEBUG_OUTPUT.println(getDomainNameWithoutWwwPrefix());
   #endif
 }
 

libraries/EEPROM/EEPROM.cpp

@@ -56,16 +56,21 @@ void EEPROMClass::begin(size_t size) {
 
   size = (size + 3) & (~3);
 
-  if (_data) {
+  //In case begin() is called a 2nd+ time, don't reallocate if size is the same
+  if(_data && size != _size) {
     delete[] _data;
+    _data = new uint8_t[size];
+  } else if(!_data) {
+    _data = new uint8_t[size];
   }
 
-  _data = new uint8_t[size];
   _size = size;
 
   noInterrupts();
   spi_flash_read(_sector * SPI_FLASH_SEC_SIZE, reinterpret_cast<uint32_t*>(_data), _size);
   interrupts();
+
+  _dirty = false; //make sure dirty is cleared in case begin() is called 2nd+ time
 }
 
 void EEPROMClass::end() {
@@ -78,10 +83,11 @@ void EEPROMClass::end() {
   }
   _data = 0;
   _size = 0;
+  _dirty = false;
 }
 
 
-uint8_t EEPROMClass::read(int address) {
+uint8_t EEPROMClass::read(int const address) {
   if (address < 0 || (size_t)address >= _size)
     return 0;
   if(!_data)
@@ -90,7 +96,7 @@ uint8_t EEPROMClass::read(int address) {
   return _data[address];
 }
 
-void EEPROMClass::write(int address, uint8_t value) {
+void EEPROMClass::write(int const address, uint8_t const value) {
   if (address < 0 || (size_t)address >= _size)
     return;
   if(!_data)
@@ -131,6 +137,10 @@ uint8_t * EEPROMClass::getDataPtr() {
   return &_data[0];
 }
 
+uint8_t const * EEPROMClass::getConstDataPtr() const {
+  return &_data[0];
+}
+
 #if !defined(NO_GLOBAL_INSTANCES) && !defined(NO_GLOBAL_EEPROM)
 EEPROMClass EEPROM;
 #endif

libraries/EEPROM/EEPROM.h

@@ -32,15 +32,16 @@ public:
   EEPROMClass(void);
 
   void begin(size_t size);
-  uint8_t read(int address);
+  uint8_t read(int const address);
-  void write(int address, uint8_t val);
+  void write(int const address, uint8_t const val);
   bool commit();
   void end();
 
   uint8_t * getDataPtr();
+  uint8_t const * getConstDataPtr() const;
 
   template<typename T> 
-  T &get(int address, T &t) {
+  T &get(int const address, T &t) {
     if (address < 0 || address + sizeof(T) > _size)
       return t;
 
@@ -49,15 +50,22 @@ public:
   }
 
   template<typename T> 
-  const T &put(int address, const T &t) {
+  const T &put(int const address, const T &t) {
     if (address < 0 || address + sizeof(T) > _size)
       return t;
+    if (memcmp(_data + address, (const uint8_t*)&t, sizeof(T)) != 0) {
+      _dirty = true;
+      memcpy(_data + address, (const uint8_t*)&t, sizeof(T));
+    }
 
-    memcpy(_data + address, (const uint8_t*) &t, sizeof(T));
-    _dirty = true;
     return t;
   }
 
+  size_t length() {return _size;}
+
+  uint8_t& operator[](int const address) {return getDataPtr()[address];}
+  uint8_t const & operator[](int const address) const {return getConstDataPtr()[address];}
+
 protected:
   uint32_t _sector;
   uint8_t* _data;

libraries/ESP8266AVRISP/examples/Arduino_Wifi_AVRISP/Arduino_Wifi_AVRISP.ino

@@ -17,6 +17,7 @@ void setup() {
     Serial.println("Arduino AVR-ISP over TCP");
     avrprog.setReset(false); // let the AVR run
 
+    WiFi.mode(WIFI_STA);
     WiFi.begin(ssid, pass);
     while (WiFi.waitForConnectResult() != WL_CONNECTED);
 

libraries/ESP8266HTTPClient/examples/Authorization/Authorization.ino

@@ -31,6 +31,7 @@ void setup() {
         delay(1000);
     }
 
+    WiFi.mode(WIFI_STA);
     WiFiMulti.addAP("SSID", "PASSWORD");
 
 }

libraries/ESP8266HTTPClient/examples/BasicHttpClient/BasicHttpClient.ino

@@ -31,6 +31,7 @@ void setup() {
         delay(1000);
     }
 
+    WiFi.mode(WIFI_STA);
     WiFiMulti.addAP("SSID", "PASSWORD");
 
 }

libraries/ESP8266HTTPClient/examples/ReuseConnection/ReuseConnection.ino

@@ -34,6 +34,7 @@ void setup() {
         delay(1000);
     }
 
+    WiFi.mode(WIFI_STA);
     WiFiMulti.addAP("SSID", "PASSWORD");
 
     // allow reuse (if server supports it)

libraries/ESP8266HTTPClient/examples/StreamHttpClient/StreamHttpClient.ino

@@ -31,6 +31,7 @@ void setup() {
         delay(1000);
     }
 
+    WiFi.mode(WIFI_STA);
     WiFiMulti.addAP("SSID", "PASSWORD");
 
 }

libraries/ESP8266HTTPClient/library.properties

@@ -5,5 +5,5 @@ maintainer=Markus Sattler
 sentence=http Client for ESP8266
 paragraph=
 category=Communication
-url=https://github.com/Links2004/Arduino/tree/libraries/ESP8266HTTPClient
+url=https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient
 architectures=esp8266

libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp

@@ -44,6 +44,8 @@ public:
 
     virtual bool verify(WiFiClient& client, const char* host)
     {
+        (void)client;
+        (void)host;
         return true;
     }
 };
@@ -167,6 +169,7 @@ bool HTTPClient::beginInternal(String url, const char* expectedProtocol)
         _host = host;
     }
     _uri = url;
+
     if (_protocol != expectedProtocol) {
         DEBUG_HTTPCLIENT("[HTTP-Client][begin] unexpected protocol: %s, expected %s\n", _protocol.c_str(), expectedProtocol);
         return false;
@@ -351,6 +354,20 @@ int HTTPClient::PUT(String payload) {
     return PUT((uint8_t *) payload.c_str(), payload.length());
 }
 
+/**
+ * sends a patch request to the server
+ * @param payload uint8_t *
+ * @param size size_t
+ * @return http code
+ */
+int HTTPClient::PATCH(uint8_t * payload, size_t size) {
+    return sendRequest("PATCH", payload, size);
+}
+
+int HTTPClient::PATCH(String payload) {
+    return PATCH((uint8_t *) payload.c_str(), payload.length());
+}
+
 /**
  * sendRequest
  * @param type const char *     "GET", "POST", ....
@@ -823,6 +840,7 @@ bool HTTPClient::connect(void)
     }
 
     _tcp = _transportTraits->create();
+    _tcp->setTimeout(_tcpTimeout);
 
     if(!_tcp->connect(_host.c_str(), _port)) {
         DEBUG_HTTPCLIENT("[HTTP-Client] failed connect to %s:%u\n", _host.c_str(), _port);
@@ -837,8 +855,6 @@ bool HTTPClient::connect(void)
         return false;
     }
 
-    // set Timeout for readBytesUntil and readStringUntil
-    _tcp->setTimeout(_tcpTimeout);
 
 #ifdef ESP8266
     _tcp->setNoDelay(true);
@@ -857,7 +873,7 @@ bool HTTPClient::sendHeader(const char * type)
         return false;
     }
 
-    String header = String(type) + " " + _uri + F(" HTTP/1.");
+    String header = String(type) + " " + (_uri.length() ? _uri : F("/")) + F(" HTTP/1.");
 
     if(_useHTTP10) {
         header += "0";
@@ -894,6 +910,8 @@ bool HTTPClient::sendHeader(const char * type)
 
     header += _headers + "\r\n";
 
+    DEBUG_HTTPCLIENT("[HTTP-Client] sending request header\n-----\n%s-----\n", header.c_str());
+
     return (_tcp->write((const uint8_t *) header.c_str(), header.length()) == header.length());
 }
 

libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.h

@@ -158,6 +158,8 @@ public:
     int POST(String payload);
     int PUT(uint8_t * payload, size_t size);
     int PUT(String payload);
+    int PATCH(uint8_t * payload, size_t size);
+    int PATCH(String payload);
     int sendRequest(const char * type, String payload);
     int sendRequest(const char * type, uint8_t * payload = NULL, size_t size = 0);
     int sendRequest(const char * type, Stream * stream, size_t size = 0);

libraries/ESP8266HTTPUpdateServer/src/ESP8266HTTPUpdateServer.cpp

@@ -12,9 +12,9 @@ static const char serverIndex[] PROGMEM =
                   <input type='file' name='update'>
                   <input type='submit' value='Update'>
                </form>
-         </body></html>\n)";
+         </body></html>)";
 static const char successResponse[] PROGMEM = 
-  "<META http-equiv=\"refresh\" content=\"15;URL=\">Update Success! Rebooting...\n";
+  "<META http-equiv=\"refresh\" content=\"15;URL=/\">Update Success! Rebooting...\n";
 
 ESP8266HTTPUpdateServer::ESP8266HTTPUpdateServer(bool serial_debug)
 {

libraries/ESP8266LLMNR/ESP8266LLMNR.cpp

@@ -50,7 +50,7 @@ extern "C" {
 
 //#define LLMNR_DEBUG
 
-#define BIT(x) (1 << (x))
+//BIT(x) is defined in tools/sdk/c_types.h
 
 #define FLAGS_QR                BIT(15)
 #define FLAGS_OP_SHIFT          11
@@ -226,14 +226,15 @@ void LLMNRResponder::_process_packet() {
         Serial.println("(no matching RRs)");
 #endif
 
-    struct ip_info remote_ip_info;
+    ip_addr_t remote_ip;
-    remote_ip_info.ip.addr = _conn->getRemoteAddress();
+    remote_ip.addr = _conn->getRemoteAddress();
+
     struct ip_info ip_info;
     bool match_ap = false;
     if (wifi_get_opmode() & SOFTAP_MODE) {
         wifi_get_ip_info(SOFTAP_IF, &ip_info);
-        if (ip_info.ip.addr && ip_addr_netcmp(&remote_ip_info.ip, &ip_info.ip, &ip_info.netmask))
+    if (ip_info.ip.addr && ip_addr_netcmp(&remote_ip, &ip_info.ip, &ip_info.netmask))
-            match_ap = true;
+        match_ap = true;
     }
     if (!match_ap)
         wifi_get_ip_info(STATION_IF, &ip_info);
@@ -272,8 +273,8 @@ void LLMNRResponder::_process_packet() {
         };
         _conn->append(reinterpret_cast<const char*>(rr), sizeof(rr));
     }
-    _conn->setMulticastInterface(remote_ip_info.ip);
+    _conn->setMulticastInterface(remote_ip);
-    _conn->send(&remote_ip_info.ip, _conn->getRemotePort());
+    _conn->send(&remote_ip, _conn->getRemotePort());
 }
 
 #if !defined(NO_GLOBAL_INSTANCES) && !defined(NO_GLOBAL_LLMNR)

libraries/ESP8266LLMNR/examples/LLMNR_Web_Server/LLMNR_Web_Server.ino

@@ -77,6 +77,7 @@ void setup(void) {
   Serial.begin(115200);
 
   // Connect to WiFi network
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   Serial.println("");
 

libraries/ESP8266NetBIOS/ESP8266NetBIOS.cpp

@@ -17,6 +17,12 @@ extern "C" {
 
 #define NBNSQ_TYPE_NB (0x0020)
 #define NBNSQ_CLASS_IN (0x0001)
+#ifndef LWIP_PLATFORM_HTONS
+#define LWIP_PLATFORM_HTONS(_n)  ((u16_t)((((_n) & 0xff) << 8) | (((_n) >> 8) & 0xff)))
+#endif
+#ifndef LWIP_PLATFORM_HTONL
+#define LWIP_PLATFORM_HTONL(_n)  ((u32_t)( (((_n) & 0xff) << 24) | (((_n) & 0xff00) << 8) | (((_n) >> 8)  & 0xff00) | (((_n) >> 24) & 0xff) ))
+#endif
 
 // Definice struktury NBNS dotazu (alespon veci, ktere jsem vypozoroval):
 struct NBNSQUESTION {
@@ -140,7 +146,7 @@ bool ESP8266NetBIOS::begin(const char *name)
     }
 
     // presuneme jmeno zarizeni se soucasnou upravou na UPPER case
-    for (int i = 0; i < n; ++i) {
+    for (size_t  i = 0; i < n; ++i) {
         _name[i] = toupper(name[i]);
     }
     _name[n] = '\0';
@@ -168,8 +174,15 @@ void ESP8266NetBIOS::end()
     }
 }
 
-void ESP8266NetBIOS::_recv(udp_pcb *upcb, pbuf *pb, ip_addr_t *addr, u16_t port)
+#if LWIP_VERSION_MAJOR == 1 
+void ESP8266NetBIOS::_recv(udp_pcb *upcb, pbuf *pb, ip_addr_t *addr, uint16_t port)
+#else
+void ESP8266NetBIOS::_recv(udp_pcb *upcb, pbuf *pb, const ip_addr_t *addr, uint16_t port)
+#endif
 {
+    (void)upcb;
+    (void)addr;
+    (void)port;
     while(pb != NULL) {
         uint8_t * data = (uint8_t*)((pb)->payload);
         size_t len = pb->len;
@@ -256,7 +269,11 @@ void ESP8266NetBIOS::_recv(udp_pcb *upcb, pbuf *pb, ip_addr_t *addr, u16_t port)
     }
 }
 
+#if LWIP_VERSION_MAJOR == 1
 void ESP8266NetBIOS::_s_recv(void *arg, udp_pcb *upcb, pbuf *p, struct ip_addr *addr, uint16_t port)
+#else
+void ESP8266NetBIOS::_s_recv(void *arg, udp_pcb *upcb, pbuf *p, const ip_addr_t *addr, uint16_t port)
+#endif
 {
     reinterpret_cast<ESP8266NetBIOS*>(arg)->_recv(upcb, p, addr, port);
 }

libraries/ESP8266NetBIOS/ESP8266NetBIOS.h

@@ -2,6 +2,10 @@
 #ifndef __ESPNBNS_h__
 #define __ESPNBNS_h__
 
+extern "C" {
+#include "lwip/init.h" // LWIP_VERSION_
+#include <lwip/ip_addr.h>
+}
 #include <ESP8266WiFi.h>
 
 #define NBNS_PORT 137
@@ -15,7 +19,6 @@
 
 struct udp_pcb;
 struct pbuf;
-struct ip_addr;
 
 class ESP8266NetBIOS
 {
@@ -24,8 +27,14 @@ protected:
     char _name[NBNS_MAX_HOSTNAME_LEN + 1];
     void _getnbname(char *nbname, char *name, uint8_t maxlen);
     void _makenbname(char *name, char *nbname, uint8_t outlen);
+   
+#if LWIP_VERSION_MAJOR == 1 
     void _recv(udp_pcb *upcb, pbuf *pb, struct ip_addr *addr, uint16_t port);
     static void _s_recv(void *arg, udp_pcb *upcb, pbuf *p, struct ip_addr *addr, uint16_t port);
+#else
+    void _recv(udp_pcb *upcb, pbuf *pb, const ip_addr_t *addr, uint16_t port);
+    static void _s_recv(void *arg, udp_pcb *upcb, pbuf *p, const ip_addr_t *addr, uint16_t port);
+#endif
 public:
     ESP8266NetBIOS();
     ~ESP8266NetBIOS();

libraries/ESP8266NetBIOS/examples/ESP_NBNST/ESP_NBNST.ino

@@ -22,6 +22,7 @@ void setup()
     Serial.begin(115200);
 
     // Connect to WiFi network
+    WiFi.mode(WIFI_STA);
     WiFi.begin(ssid, password);
     Serial.println("");
 

libraries/ESP8266SSDP/ESP8266SSDP.cpp

@@ -25,7 +25,9 @@ License (MIT license):
   THE SOFTWARE.
 
 */
+#ifndef LWIP_OPEN_SRC
 #define LWIP_OPEN_SRC
+#endif
 #include <functional>
 #include "ESP8266SSDP.h"
 #include "WiFiUdp.h"
@@ -199,7 +201,7 @@ bool SSDPClass::begin(){
 
 void SSDPClass::_send(ssdp_method_t method){
   char buffer[1460];
-  uint32_t ip = WiFi.localIP();
+  IPAddress ip = WiFi.localIP();
 
   char valueBuffer[strlen_P(_ssdp_notify_template)+1];
   strcpy_P(valueBuffer, (method == NONE)?_ssdp_response_template:_ssdp_notify_template);
@@ -212,7 +214,7 @@ void SSDPClass::_send(ssdp_method_t method){
     _uuid,
     (method == NONE)?"ST":"NT",
     _deviceType,
-    IP2STR(&ip), _port, _schemaURL
+   ip[0], ip[1], ip[2], ip[3], _port, _schemaURL
   );
 
   _server->append(buffer, len);
@@ -242,11 +244,11 @@ void SSDPClass::_send(ssdp_method_t method){
 }
 
 void SSDPClass::schema(WiFiClient client){
-  uint32_t ip = WiFi.localIP();
+  IPAddress ip = WiFi.localIP();
   char buffer[strlen_P(_ssdp_schema_template)+1];
   strcpy_P(buffer, _ssdp_schema_template);
   client.printf(buffer,
-    IP2STR(&ip), _port,
+    ip[0], ip[1], ip[2], ip[3], _port,
     _deviceType,
     _friendlyName,
     _presentationURL,
@@ -287,7 +289,6 @@ void SSDPClass::_update(){
         case METHOD:
           if(c == ' '){
             if(strcmp(buffer, "M-SEARCH") == 0) method = SEARCH;
-            else if(strcmp(buffer, "NOTIFY") == 0) method = NOTIFY;
 
             if(method == NONE) state = ABORT;
             else state = URI;
@@ -328,7 +329,7 @@ void SSDPClass::_update(){
 #endif
                 }
                 // if the search type matches our type, we should respond instead of ABORT
-                if(strcmp(buffer, _deviceType) == 0){
+                if(strcasecmp(buffer, _deviceType) == 0){
                   _pending = true;
                   _process_time = millis();
                   state = KEY;

libraries/ESP8266WebServer/examples/AdvancedWebServer/AdvancedWebServer.ino

@@ -93,6 +93,7 @@ void setup ( void ) {
 	pinMode ( led, OUTPUT );
 	digitalWrite ( led, 0 );
 	Serial.begin ( 115200 );
+	WiFi.mode ( WIFI_STA );
 	WiFi.begin ( ssid, password );
 	Serial.println ( "" );
 

libraries/ESP8266WebServer/examples/FSBrowser/FSBrowser.ino

@@ -179,6 +179,7 @@ void setup(void){
   //WIFI INIT
   DBG_OUTPUT_PORT.printf("Connecting to %s\n", ssid);
   if (String(WiFi.SSID()) != String(ssid)) {
+    WiFi.mode(WIFI_STA);
     WiFi.begin(ssid, password);
   }
   

libraries/ESP8266WebServer/examples/HelloServer/HelloServer.ino

@@ -37,6 +37,7 @@ void setup(void){
   pinMode(led, OUTPUT);
   digitalWrite(led, 0);
   Serial.begin(115200);
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   Serial.println("");
 

libraries/ESP8266WebServer/examples/HttpAdvancedAuth/HttpAdvancedAuth.ino

@@ -0,0 +1,57 @@
+/*
+  HTTP Advanced Authentication example
+  Created Mar 16, 2017 by Ahmed El-Sharnoby.
+  This example code is in the public domain.
+*/
+
+#include <ESP8266WiFi.h>
+#include <ESP8266mDNS.h>
+#include <ArduinoOTA.h>
+#include <ESP8266WebServer.h>
+
+const char* ssid = "........";
+const char* password = "........";
+
+ESP8266WebServer server(80);
+
+const char* www_username = "admin";
+const char* www_password = "esp8266";
+ // allows you to set the realm of authentication Default:"Login Required"
+const char* www_realm = "Custom Auth Realm";
+// the Content of the HTML response in case of Unautherized Access Default:empty
+String authFailResponse = "Authentication Failed";
+
+void setup() {
+  Serial.begin(115200);
+  WiFi.mode(WIFI_STA);
+  WiFi.begin(ssid, password);
+  if(WiFi.waitForConnectResult() != WL_CONNECTED) {
+    Serial.println("WiFi Connect Failed! Rebooting...");
+    delay(1000);
+    ESP.restart();
+  }
+  ArduinoOTA.begin();
+
+  server.on("/", [](){
+    if(!server.authenticate(www_username, www_password))
+    //Basic Auth Method with Custom realm and Failure Response
+      //return server.requestAuthentication(BASIC_AUTH, www_realm, authFailResponse);
+    //Digest Auth Method with realm="Login Required" and empty Failure Response
+      //return server.requestAuthentication(DIGEST_AUTH);
+    //Digest Auth Method with Custom realm and empty Failure Response
+      //return server.requestAuthentication(DIGEST_AUTH, www_realm);
+     //Digest Auth Method with Custom realm and Failure Response
+      return server.requestAuthentication(DIGEST_AUTH, www_realm, authFailResponse);
+    server.send(200, "text/plain", "Login OK");
+  });
+  server.begin();
+
+  Serial.print("Open http://");
+  Serial.print(WiFi.localIP());
+  Serial.println("/ in your browser to see it working");
+}
+
+void loop() {
+  ArduinoOTA.handle();
+  server.handleClient();
+}

libraries/ESP8266WebServer/examples/SDWebServer/SDWebServer.ino

@@ -223,6 +223,7 @@ void setup(void){
   DBG_OUTPUT_PORT.begin(115200);
   DBG_OUTPUT_PORT.setDebugOutput(true);
   DBG_OUTPUT_PORT.print("\n");
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   DBG_OUTPUT_PORT.print("Connecting to ");
   DBG_OUTPUT_PORT.println(ssid);

libraries/ESP8266WebServer/examples/SimpleAuthentification/SimpleAuthentification.ino

@@ -95,6 +95,7 @@ void handleNotFound(){
 
 void setup(void){
   Serial.begin(115200);
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   Serial.println("");
 

libraries/ESP8266WebServer/keywords.txt

@@ -34,3 +34,4 @@ onNotFound	KEYWORD2
 HTTP_GET	LITERAL1
 HTTP_POST	LITERAL1
 HTTP_ANY	LITERAL1
+CONTENT_LENGTH_UNKNOWN	LITERAL1

libraries/ESP8266WebServer/src/ESP8266WebServer.cpp

@@ -94,6 +94,12 @@ void ESP8266WebServer::begin() {
     collectHeaders(0, 0);
 }
 
+String ESP8266WebServer::_exractParam(String& authReq,const String& param,const char delimit){
+  int _begin = authReq.indexOf(param);
+  if (_begin==-1) return "";
+  return authReq.substring(_begin+param.length(),authReq.indexOf(delimit,_begin+param.length()));
+}
+
 bool ESP8266WebServer::authenticate(const char * username, const char * password){
   if(hasHeader(AUTHORIZATION_HEADER)){
     String authReq = header(AUTHORIZATION_HEADER);
@@ -113,7 +119,7 @@ bool ESP8266WebServer::authenticate(const char * username, const char * password
         return false;
       }
       sprintf(toencode, "%s:%s", username, password);
-      if(base64_encode_chars(toencode, toencodeLen, encoded) > 0 && authReq.equals(encoded)){
+      if(base64_encode_chars(toencode, toencodeLen, encoded) > 0 && authReq.equalsConstantTime(encoded)) {
         authReq = String();
         delete[] toencode;
         delete[] encoded;
@@ -121,15 +127,106 @@ bool ESP8266WebServer::authenticate(const char * username, const char * password
       }
       delete[] toencode;
       delete[] encoded;
+    }else if(authReq.startsWith("Digest")){
+      authReq = authReq.substring(7);
+      #ifdef DEBUG_ESP_HTTP_SERVER
+      DEBUG_OUTPUT.println(authReq);
+      #endif
+      String _username = _exractParam(authReq,"username=\"");
+      if((!_username.length())||_username!=String(username)){
+        authReq = String();
+        return false;
+      }
+      // extracting required parameters for RFC 2069 simpler Digest
+      String _realm    = _exractParam(authReq,"realm=\"");
+      String _nonce    = _exractParam(authReq,"nonce=\"");
+      String _uri      = _exractParam(authReq,"uri=\"");
+      String _response = _exractParam(authReq,"response=\"");
+      String _opaque   = _exractParam(authReq,"opaque=\"");
+
+      if((!_realm.length())||(!_nonce.length())||(!_uri.length())||(!_response.length())||(!_opaque.length())){
+        authReq = String();
+        return false;
+      }
+      if((_opaque!=_sopaque)||(_nonce!=_snonce)||(_realm!=_srealm)){
+        authReq = String();
+        return false;
+      }
+      // parameters for the RFC 2617 newer Digest
+      String _nc,_cnonce;
+      if(authReq.indexOf("qop=auth") != -1){
+        _nc = _exractParam(authReq,"nc=",',');
+        _cnonce = _exractParam(authReq,"cnonce=\"");
+      }
+      MD5Builder md5;
+      md5.begin();
+      md5.add(String(username)+":"+_realm+":"+String(password));  // md5 of the user:realm:user
+      md5.calculate();
+      String _H1 = md5.toString();
+      #ifdef DEBUG_ESP_HTTP_SERVER
+      DEBUG_OUTPUT.println("Hash of user:realm:pass=" + _H1);
+      #endif
+      md5.begin();
+      if(_currentMethod == HTTP_GET){
+        md5.add("GET:"+_uri);
+      }else if(_currentMethod == HTTP_POST){
+        md5.add("POST:"+_uri);
+      }else if(_currentMethod == HTTP_PUT){
+        md5.add("PUT:"+_uri);
+      }else if(_currentMethod == HTTP_DELETE){
+        md5.add("DELETE:"+_uri);
+      }else{
+        md5.add("GET:"+_uri);
+      }
+      md5.calculate();
+      String _H2 = md5.toString();
+      #ifdef DEBUG_ESP_HTTP_SERVER
+      DEBUG_OUTPUT.println("Hash of GET:uri=" + _H2);
+      #endif
+      md5.begin();
+      if(authReq.indexOf("qop=auth") != -1){
+        md5.add(_H1+":"+_nonce+":"+_nc+":"+_cnonce+":auth:"+_H2);
+      }else{
+        md5.add(_H1+":"+_nonce+":"+_H2);
+      }
+      md5.calculate();
+      String _responsecheck = md5.toString();
+      #ifdef DEBUG_ESP_HTTP_SERVER
+      DEBUG_OUTPUT.println("The Proper response=" +_responsecheck);
+      #endif
+      if(_response==_responsecheck){
+        authReq = String();
+        return true;
+      }
     }
     authReq = String();
   }
   return false;
 }
 
-void ESP8266WebServer::requestAuthentication(){
+String ESP8266WebServer::_getRandomHexString(){
-  sendHeader("WWW-Authenticate", "Basic realm=\"Login Required\"");
+  char buffer[33];  // buffer to hold 32 Hex Digit + /0
-  send(401);
+  int i;
+  for(i=0;i<4;i++){
+    sprintf (buffer+(i*8), "%08x", RANDOM_REG32);
+  }
+  return String(buffer);
+}
+
+void ESP8266WebServer::requestAuthentication(HTTPAuthMethod mode, const char* realm, const String& authFailMsg){
+  if(realm==NULL){
+    _srealm = "Login Required";
+  }else{
+    _srealm = String(realm);
+  }
+  if(mode==BASIC_AUTH){
+    sendHeader("WWW-Authenticate", "Basic realm=\"" + _srealm + "\"");
+  }else{
+    _snonce=_getRandomHexString();
+    _sopaque=_getRandomHexString();
+    sendHeader("WWW-Authenticate", "Digest realm=\"" +_srealm + "\", qop=\"auth\", nonce=\""+_snonce+"\", opaque=\""+_sopaque+"\"");
+  }
+  send(401,"text/html",authFailMsg);
 }
 
 void ESP8266WebServer::on(const String &uri, ESP8266WebServer::THandlerFunction handler) {
@@ -179,51 +276,49 @@ void ESP8266WebServer::handleClient() {
     _statusChange = millis();
   }
 
-  if (!_currentClient.connected()) {
+  bool keepCurrentClient = false;
+  bool callYield = false;
+
+  if (_currentClient.connected()) {
+    switch (_currentStatus) {
+    case HC_WAIT_READ:
+      // Wait for data from client to become available
+      if (_currentClient.available()) {
+        if (_parseRequest(_currentClient)) {
+          _currentClient.setTimeout(HTTP_MAX_SEND_WAIT);
+          _contentLength = CONTENT_LENGTH_NOT_SET;
+          _handleRequest();
+
+          if (_currentClient.connected()) {
+            _currentStatus = HC_WAIT_CLOSE;
+            _statusChange = millis();
+            keepCurrentClient = true;
+          }
+        }
+      } else { // !_currentClient.available()
+        if (millis() - _statusChange <= HTTP_MAX_DATA_WAIT) {
+          keepCurrentClient = true;
+        }
+        callYield = true;
+      }
+      break;
+    case HC_WAIT_CLOSE:
+      // Wait for client to close the connection
+      if (millis() - _statusChange <= HTTP_MAX_CLOSE_WAIT) {
+        keepCurrentClient = true;
+        callYield = true;
+      }
+    }
+  }
+
+  if (!keepCurrentClient) {
     _currentClient = WiFiClient();
     _currentStatus = HC_NONE;
-    return;
+    _currentUpload.reset();
   }
 
-  // Wait for data from client to become available
+  if (callYield) {
-  if (_currentStatus == HC_WAIT_READ) {
+    yield();
-    if (!_currentClient.available()) {
-      if (millis() - _statusChange > HTTP_MAX_DATA_WAIT) {
-        _currentClient = WiFiClient();
-        _currentStatus = HC_NONE;
-      }
-      yield();
-      return;
-    }
-
-    if (!_parseRequest(_currentClient)) {
-      _currentClient = WiFiClient();
-      _currentStatus = HC_NONE;
-      return;
-    }
-    _currentClient.setTimeout(HTTP_MAX_SEND_WAIT);
-    _contentLength = CONTENT_LENGTH_NOT_SET;
-    _handleRequest();
-
-    if (!_currentClient.connected()) {
-      _currentClient = WiFiClient();
-      _currentStatus = HC_NONE;
-      return;
-    } else {
-      _currentStatus = HC_WAIT_CLOSE;
-      _statusChange = millis();
-      return;
-    }
-  }
-
-  if (_currentStatus == HC_WAIT_CLOSE) {
-    if (millis() - _statusChange > HTTP_MAX_CLOSE_WAIT) {
-      _currentClient = WiFiClient();
-      _currentStatus = HC_NONE;
-    } else {
-      yield();
-      return;
-    }
   }
 }
 
@@ -338,6 +433,9 @@ void ESP8266WebServer::sendContent(const String& content) {
   _currentClient.write(content.c_str(), len);
   if(_chunked){
     _currentClient.write(footer, 2);
+    if (len == 0) {
+      _chunked = false;
+    }
   }
 }
 
@@ -358,6 +456,9 @@ void ESP8266WebServer::sendContent_P(PGM_P content, size_t size) {
   _currentClient.write_P(content, size);
   if(_chunked){
     _currentClient.write(footer, 2);
+    if (size == 0) {
+      _chunked = false;
+    }
   }
 }
 
@@ -465,19 +566,27 @@ void ESP8266WebServer::_handleRequest() {
     }
 #endif
   }
-
+  if (!handled && _notFoundHandler) {
-  if (!handled) {
+    _notFoundHandler();
-    if(_notFoundHandler) {
+    handled = true;
-      _notFoundHandler();
+  }
-    }
+  if (!handled) {
-    else {
+    send(404, "text/plain", String("Not found: ") + _currentUri);
-      send(404, "text/plain", String("Not found: ") + _currentUri);
+    handled = true;
-    }
+  }
+  if (handled) {
+    _finalizeResponse();
   }
-
   _currentUri = String();
 }
 
+
+void ESP8266WebServer::_finalizeResponse() {
+  if (_chunked) {
+    sendContent("");
+  }
+}
+
 String ESP8266WebServer::_responseCodeToString(int code) {
   switch (code) {
     case 100: return F("Continue");

libraries/ESP8266WebServer/src/ESP8266WebServer.h

@@ -25,12 +25,14 @@
 #define ESP8266WEBSERVER_H
 
 #include <functional>
+#include <memory>
 #include <ESP8266WiFi.h>
 
 enum HTTPMethod { HTTP_ANY, HTTP_GET, HTTP_POST, HTTP_PUT, HTTP_PATCH, HTTP_DELETE, HTTP_OPTIONS };
 enum HTTPUploadStatus { UPLOAD_FILE_START, UPLOAD_FILE_WRITE, UPLOAD_FILE_END,
                         UPLOAD_FILE_ABORTED };
 enum HTTPClientStatus { HC_NONE, HC_WAIT_READ, HC_WAIT_CLOSE };
+enum HTTPAuthMethod { BASIC_AUTH, DIGEST_AUTH };
 
 #define HTTP_DOWNLOAD_UNIT_SIZE 1460
 
@@ -38,8 +40,8 @@ enum HTTPClientStatus { HC_NONE, HC_WAIT_READ, HC_WAIT_CLOSE };
 #define HTTP_UPLOAD_BUFLEN 2048
 #endif
 
-#define HTTP_MAX_DATA_WAIT 1000 //ms to wait for the client to send the request
+#define HTTP_MAX_DATA_WAIT 5000 //ms to wait for the client to send the request
-#define HTTP_MAX_POST_WAIT 1000 //ms to wait for POST data to arrive
+#define HTTP_MAX_POST_WAIT 5000 //ms to wait for POST data to arrive
 #define HTTP_MAX_SEND_WAIT 5000 //ms to wait for data chunk to be ACKed
 #define HTTP_MAX_CLOSE_WAIT 2000 //ms to wait for the client to close the connection
 
@@ -78,7 +80,7 @@ public:
   void stop();
 
   bool authenticate(const char * username, const char * password);
-  void requestAuthentication();
+  void requestAuthentication(HTTPAuthMethod mode = BASIC_AUTH, const char* realm = NULL, const String& authFailMsg = String("") );
 
   typedef std::function<void(void)> THandlerFunction;
   void on(const String &uri, THandlerFunction handler);
@@ -92,7 +94,7 @@ public:
   String uri() { return _currentUri; }
   HTTPMethod method() { return _currentMethod; }
   WiFiClient client() { return _currentClient; }
-  HTTPUpload& upload() { return _currentUpload; }
+  HTTPUpload& upload() { return *_currentUpload; }
 
   String arg(String name);        // get request argument value by name
   String arg(int i);              // get request argument value by number
@@ -140,6 +142,7 @@ template<typename T> size_t streamFile(T &file, const String& contentType){
 protected:
   void _addRequestHandler(RequestHandler* handler);
   void _handleRequest();
+  void _finalizeResponse();
   bool _parseRequest(WiFiClient& client);
   void _parseArguments(String data);
   static String _responseCodeToString(int code);
@@ -149,6 +152,10 @@ protected:
   uint8_t _uploadReadByte(WiFiClient& client);
   void _prepareHeader(String& response, int code, const char* content_type, size_t contentLength);
   bool _collectHeader(const char* headerName, const char* headerValue);
+  
+  String _getRandomHexString();
+  // for extracting Auth parameters
+  String _exractParam(String& authReq,const String& param,const char delimit = '"');
 
   struct RequestArgument {
     String key;
@@ -172,7 +179,7 @@ protected:
 
   int              _currentArgCount;
   RequestArgument* _currentArgs;
-  HTTPUpload       _currentUpload;
+  std::unique_ptr<HTTPUpload> _currentUpload;
 
   int              _headerKeysCount;
   RequestArgument* _currentHeaders;
@@ -182,6 +189,10 @@ protected:
   String           _hostHeader;
   bool             _chunked;
 
+  String           _snonce;  // Store noance and opaque for future comparison
+  String           _sopaque;
+  String           _srealm;  // Store the Auth realm between Calls
+
 };
 
 

libraries/ESP8266WebServer/src/Parsing.cpp

@@ -91,7 +91,7 @@ bool ESP8266WebServer::_parseRequest(WiFiClient& client) {
   String searchStr = "";
   int hasSearch = url.indexOf('?');
   if (hasSearch != -1){
-    searchStr = urlDecode(url.substring(hasSearch + 1));
+    searchStr = url.substring(hasSearch + 1);
     url = url.substring(0, hasSearch);
   }
   _currentUri = url;
@@ -166,6 +166,7 @@ bool ESP8266WebServer::_parseRequest(WiFiClient& client) {
           isEncoded = true;
         } else if (headerValue.startsWith("multipart/")){
           boundaryStr = headerValue.substring(headerValue.indexOf('=')+1);
+          boundaryStr.replace("\"","");
           isForm = true;
         }
       } else if (headerName.equalsIgnoreCase("Content-Length")){
@@ -205,6 +206,9 @@ bool ESP8266WebServer::_parseRequest(WiFiClient& client) {
         DEBUG_OUTPUT.println(plainBuf);
   #endif
         free(plainBuf);
+      } else {
+        // No content - but we can still have arguments in the URL.
+        _parseArguments(searchStr);
       }
     }
 
@@ -318,7 +322,7 @@ void ESP8266WebServer::_parseArguments(String data) {
     }
     RequestArgument& arg = _currentArgs[iarg];
     arg.key = data.substring(pos, equal_sign_index);
-	arg.value = data.substring(equal_sign_index + 1, next_arg_index);
+    arg.value = urlDecode(data.substring(equal_sign_index + 1, next_arg_index));
 #ifdef DEBUG_ESP_HTTP_SERVER
     DEBUG_OUTPUT.print("arg ");
     DEBUG_OUTPUT.print(iarg);
@@ -341,13 +345,13 @@ void ESP8266WebServer::_parseArguments(String data) {
 }
 
 void ESP8266WebServer::_uploadWriteByte(uint8_t b){
-  if (_currentUpload.currentSize == HTTP_UPLOAD_BUFLEN){
+  if (_currentUpload->currentSize == HTTP_UPLOAD_BUFLEN){
     if(_currentHandler && _currentHandler->canUpload(_currentUri))
-      _currentHandler->upload(*this, _currentUri, _currentUpload);
+      _currentHandler->upload(*this, _currentUri, *_currentUpload);
-    _currentUpload.totalSize += _currentUpload.currentSize;
+    _currentUpload->totalSize += _currentUpload->currentSize;
-    _currentUpload.currentSize = 0;
+    _currentUpload->currentSize = 0;
   }
-  _currentUpload.buf[_currentUpload.currentSize++] = b;
+  _currentUpload->buf[_currentUpload->currentSize++] = b;
 }
 
 uint8_t ESP8266WebServer::_uploadReadByte(WiFiClient& client){
@@ -449,21 +453,22 @@ bool ESP8266WebServer::_parseForm(WiFiClient& client, String boundary, uint32_t
               break;
             }
           } else {
-            _currentUpload.status = UPLOAD_FILE_START;
+            _currentUpload.reset(new HTTPUpload());
-            _currentUpload.name = argName;
+            _currentUpload->status = UPLOAD_FILE_START;
-            _currentUpload.filename = argFilename;
+            _currentUpload->name = argName;
-            _currentUpload.type = argType;
+            _currentUpload->filename = argFilename;
-            _currentUpload.totalSize = 0;
+            _currentUpload->type = argType;
-            _currentUpload.currentSize = 0;
+            _currentUpload->totalSize = 0;
+            _currentUpload->currentSize = 0;
 #ifdef DEBUG_ESP_HTTP_SERVER
             DEBUG_OUTPUT.print("Start File: ");
-            DEBUG_OUTPUT.print(_currentUpload.filename);
+            DEBUG_OUTPUT.print(_currentUpload->filename);
             DEBUG_OUTPUT.print(" Type: ");
-            DEBUG_OUTPUT.println(_currentUpload.type);
+            DEBUG_OUTPUT.println(_currentUpload->type);
 #endif
             if(_currentHandler && _currentHandler->canUpload(_currentUri))
-              _currentHandler->upload(*this, _currentUri, _currentUpload);
+              _currentHandler->upload(*this, _currentUri, *_currentUpload);
-            _currentUpload.status = UPLOAD_FILE_WRITE;
+            _currentUpload->status = UPLOAD_FILE_WRITE;
             uint8_t argByte = _uploadReadByte(client);
 readfile:
             while(argByte != 0x0D){
@@ -499,18 +504,18 @@ readfile:
 
               if (strstr((const char*)endBuf, boundary.c_str()) != NULL){
                 if(_currentHandler && _currentHandler->canUpload(_currentUri))
-                  _currentHandler->upload(*this, _currentUri, _currentUpload);
+                  _currentHandler->upload(*this, _currentUri, *_currentUpload);
-                _currentUpload.totalSize += _currentUpload.currentSize;
+                _currentUpload->totalSize += _currentUpload->currentSize;
-                _currentUpload.status = UPLOAD_FILE_END;
+                _currentUpload->status = UPLOAD_FILE_END;
                 if(_currentHandler && _currentHandler->canUpload(_currentUri))
-                  _currentHandler->upload(*this, _currentUri, _currentUpload);
+                  _currentHandler->upload(*this, _currentUri, *_currentUpload);
 #ifdef DEBUG_ESP_HTTP_SERVER
                 DEBUG_OUTPUT.print("End File: ");
-                DEBUG_OUTPUT.print(_currentUpload.filename);
+                DEBUG_OUTPUT.print(_currentUpload->filename);
                 DEBUG_OUTPUT.print(" Type: ");
-                DEBUG_OUTPUT.print(_currentUpload.type);
+                DEBUG_OUTPUT.print(_currentUpload->type);
                 DEBUG_OUTPUT.print(" Size: ");
-                DEBUG_OUTPUT.println(_currentUpload.totalSize);
+                DEBUG_OUTPUT.println(_currentUpload->totalSize);
 #endif
                 line = client.readStringUntil(0x0D);
                 client.readStringUntil(0x0A);
@@ -600,8 +605,8 @@ String ESP8266WebServer::urlDecode(const String& text)
 }
 
 bool ESP8266WebServer::_parseFormUploadAborted(){
-  _currentUpload.status = UPLOAD_FILE_ABORTED;
+  _currentUpload->status = UPLOAD_FILE_ABORTED;
   if(_currentHandler && _currentHandler->canUpload(_currentUri))
-    _currentHandler->upload(*this, _currentUri, _currentUpload);
+    _currentHandler->upload(*this, _currentUri, *_currentUpload);
   return false;
 }

libraries/ESP8266WebServer/src/detail/RequestHandlersImpl.h

@@ -3,6 +3,32 @@
 
 #include "RequestHandler.h"
 
+// Table of extension->MIME strings stored in PROGMEM, needs to be global due to GCC section typing rules
+static const struct {const char endsWith[16]; const char mimeType[32];} mimeTable[] ICACHE_RODATA_ATTR = {
+    { ".html", "text/html" },
+    { ".htm", "text/html" },
+    { ".css", "text/css" },
+    { ".txt", "text/plain" },
+    { ".js", "application/javascript" },
+    { ".json", "application/json" },
+    { ".png", "image/png" },
+    { ".gif", "image/gif" },
+    { ".jpg", "image/jpeg" },
+    { ".ico", "image/x-icon" },
+    { ".svg", "image/svg+xml" },
+    { ".ttf", "application/x-font-ttf" },
+    { ".otf", "application/x-font-opentype" },
+    { ".woff", "application/font-woff" },
+    { ".woff2", "application/font-woff2" },
+    { ".eot", "application/vnd.ms-fontobject" },
+    { ".sfnt", "application/font-sfnt" },
+    { ".xml", "text/xml" },
+    { ".pdf", "application/pdf" },
+    { ".zip", "application/zip" },
+    { ".gz", "application/x-gzip" },
+    { ".appcache", "text/cache-manifest" },
+    { "", "application/octet-stream" } };
+
 class FunctionRequestHandler : public RequestHandler {
 public:
     FunctionRequestHandler(ESP8266WebServer::THandlerFunction fn, ESP8266WebServer::THandlerFunction ufn, const String &uri, HTTPMethod method)
@@ -116,29 +142,18 @@ public:
     }
 
     static String getContentType(const String& path) {
-        if (path.endsWith(".html")) return "text/html";
+        char buff[sizeof(mimeTable[0].mimeType)];
-        else if (path.endsWith(".htm")) return "text/html";
+        // Check all entries but last one for match, return if found
-        else if (path.endsWith(".css")) return "text/css";
+        for (size_t i=0; i < sizeof(mimeTable)/sizeof(mimeTable[0])-1; i++) {
-        else if (path.endsWith(".txt")) return "text/plain";
+            strcpy_P(buff, mimeTable[i].endsWith);
-        else if (path.endsWith(".js")) return "application/javascript";
+            if (path.endsWith(buff)) {
-        else if (path.endsWith(".json")) return "application/json";
+                strcpy_P(buff, mimeTable[i].mimeType);
-        else if (path.endsWith(".png")) return "image/png";
+                return String(buff);
-        else if (path.endsWith(".gif")) return "image/gif";
+            }
-        else if (path.endsWith(".jpg")) return "image/jpeg";
+        }
-        else if (path.endsWith(".ico")) return "image/x-icon";
+        // Fall-through and just return default type
-        else if (path.endsWith(".svg")) return "image/svg+xml";
+        strcpy_P(buff, mimeTable[sizeof(mimeTable)/sizeof(mimeTable[0])-1].mimeType);
-        else if (path.endsWith(".ttf")) return "application/x-font-ttf";
+        return String(buff);
-        else if (path.endsWith(".otf")) return "application/x-font-opentype";
-        else if (path.endsWith(".woff")) return "application/font-woff";
-        else if (path.endsWith(".woff2")) return "application/font-woff2";
-        else if (path.endsWith(".eot")) return "application/vnd.ms-fontobject";
-        else if (path.endsWith(".sfnt")) return "application/font-sfnt";
-        else if (path.endsWith(".xml")) return "text/xml";
-        else if (path.endsWith(".pdf")) return "application/pdf";
-        else if (path.endsWith(".zip")) return "application/zip";
-        else if(path.endsWith(".gz")) return "application/x-gzip";
-        else if (path.endsWith(".appcache")) return "text/cache-manifest";
-        return "application/octet-stream";
     }
 
 protected:

libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

@@ -7,6 +7,11 @@
  *  esp8266/Arduino project continuous integration
  *  build.
  *
+ *  Limitations:
+ *    only RSA certificates
+ *    no support of Perfect Forward Secrecy (PFS)
+ *    TLSv1.2 is supported since version 2.4.0-rc1
+ *
  *  Created by Ivan Grokhotkov, 2015.
  *  This example is in public domain.
  */
@@ -29,6 +34,7 @@ void setup() {
   Serial.println();
   Serial.print("connecting to ");
   Serial.println(ssid);
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   while (WiFi.status() != WL_CONNECTED) {
     delay(500);

libraries/ESP8266WiFi/examples/HTTPSRequestCACert/CACert.ino

@@ -0,0 +1,86 @@
+
+const unsigned char caCert[] PROGMEM = {
+  0x30, 0x82, 0x03, 0xc5, 0x30, 0x82, 0x02, 0xad, 0xa0, 0x03, 0x02, 0x01,
+  0x02, 0x02, 0x10, 0x02, 0xac, 0x5c, 0x26, 0x6a, 0x0b, 0x40, 0x9b, 0x8f,
+  0x0b, 0x79, 0xf2, 0xae, 0x46, 0x25, 0x77, 0x30, 0x0d, 0x06, 0x09, 0x2a,
+  0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x05, 0x05, 0x00, 0x30, 0x6c,
+  0x31, 0x0b, 0x30, 0x09, 0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02, 0x55,
+  0x53, 0x31, 0x15, 0x30, 0x13, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13, 0x0c,
+  0x44, 0x69, 0x67, 0x69, 0x43, 0x65, 0x72, 0x74, 0x20, 0x49, 0x6e, 0x63,
+  0x31, 0x19, 0x30, 0x17, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13, 0x10, 0x77,
+  0x77, 0x77, 0x2e, 0x64, 0x69, 0x67, 0x69, 0x63, 0x65, 0x72, 0x74, 0x2e,
+  0x63, 0x6f, 0x6d, 0x31, 0x2b, 0x30, 0x29, 0x06, 0x03, 0x55, 0x04, 0x03,
+  0x13, 0x22, 0x44, 0x69, 0x67, 0x69, 0x43, 0x65, 0x72, 0x74, 0x20, 0x48,
+  0x69, 0x67, 0x68, 0x20, 0x41, 0x73, 0x73, 0x75, 0x72, 0x61, 0x6e, 0x63,
+  0x65, 0x20, 0x45, 0x56, 0x20, 0x52, 0x6f, 0x6f, 0x74, 0x20, 0x43, 0x41,
+  0x30, 0x1e, 0x17, 0x0d, 0x30, 0x36, 0x31, 0x31, 0x31, 0x30, 0x30, 0x30,
+  0x30, 0x30, 0x30, 0x30, 0x5a, 0x17, 0x0d, 0x33, 0x31, 0x31, 0x31, 0x31,
+  0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x5a, 0x30, 0x6c, 0x31, 0x0b,
+  0x30, 0x09, 0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02, 0x55, 0x53, 0x31,
+  0x15, 0x30, 0x13, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13, 0x0c, 0x44, 0x69,
+  0x67, 0x69, 0x43, 0x65, 0x72, 0x74, 0x20, 0x49, 0x6e, 0x63, 0x31, 0x19,
+  0x30, 0x17, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13, 0x10, 0x77, 0x77, 0x77,
+  0x2e, 0x64, 0x69, 0x67, 0x69, 0x63, 0x65, 0x72, 0x74, 0x2e, 0x63, 0x6f,
+  0x6d, 0x31, 0x2b, 0x30, 0x29, 0x06, 0x03, 0x55, 0x04, 0x03, 0x13, 0x22,
+  0x44, 0x69, 0x67, 0x69, 0x43, 0x65, 0x72, 0x74, 0x20, 0x48, 0x69, 0x67,
+  0x68, 0x20, 0x41, 0x73, 0x73, 0x75, 0x72, 0x61, 0x6e, 0x63, 0x65, 0x20,
+  0x45, 0x56, 0x20, 0x52, 0x6f, 0x6f, 0x74, 0x20, 0x43, 0x41, 0x30, 0x82,
+  0x01, 0x22, 0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d,
+  0x01, 0x01, 0x01, 0x05, 0x00, 0x03, 0x82, 0x01, 0x0f, 0x00, 0x30, 0x82,
+  0x01, 0x0a, 0x02, 0x82, 0x01, 0x01, 0x00, 0xc6, 0xcc, 0xe5, 0x73, 0xe6,
+  0xfb, 0xd4, 0xbb, 0xe5, 0x2d, 0x2d, 0x32, 0xa6, 0xdf, 0xe5, 0x81, 0x3f,
+  0xc9, 0xcd, 0x25, 0x49, 0xb6, 0x71, 0x2a, 0xc3, 0xd5, 0x94, 0x34, 0x67,
+  0xa2, 0x0a, 0x1c, 0xb0, 0x5f, 0x69, 0xa6, 0x40, 0xb1, 0xc4, 0xb7, 0xb2,
+  0x8f, 0xd0, 0x98, 0xa4, 0xa9, 0x41, 0x59, 0x3a, 0xd3, 0xdc, 0x94, 0xd6,
+  0x3c, 0xdb, 0x74, 0x38, 0xa4, 0x4a, 0xcc, 0x4d, 0x25, 0x82, 0xf7, 0x4a,
+  0xa5, 0x53, 0x12, 0x38, 0xee, 0xf3, 0x49, 0x6d, 0x71, 0x91, 0x7e, 0x63,
+  0xb6, 0xab, 0xa6, 0x5f, 0xc3, 0xa4, 0x84, 0xf8, 0x4f, 0x62, 0x51, 0xbe,
+  0xf8, 0xc5, 0xec, 0xdb, 0x38, 0x92, 0xe3, 0x06, 0xe5, 0x08, 0x91, 0x0c,
+  0xc4, 0x28, 0x41, 0x55, 0xfb, 0xcb, 0x5a, 0x89, 0x15, 0x7e, 0x71, 0xe8,
+  0x35, 0xbf, 0x4d, 0x72, 0x09, 0x3d, 0xbe, 0x3a, 0x38, 0x50, 0x5b, 0x77,
+  0x31, 0x1b, 0x8d, 0xb3, 0xc7, 0x24, 0x45, 0x9a, 0xa7, 0xac, 0x6d, 0x00,
+  0x14, 0x5a, 0x04, 0xb7, 0xba, 0x13, 0xeb, 0x51, 0x0a, 0x98, 0x41, 0x41,
+  0x22, 0x4e, 0x65, 0x61, 0x87, 0x81, 0x41, 0x50, 0xa6, 0x79, 0x5c, 0x89,
+  0xde, 0x19, 0x4a, 0x57, 0xd5, 0x2e, 0xe6, 0x5d, 0x1c, 0x53, 0x2c, 0x7e,
+  0x98, 0xcd, 0x1a, 0x06, 0x16, 0xa4, 0x68, 0x73, 0xd0, 0x34, 0x04, 0x13,
+  0x5c, 0xa1, 0x71, 0xd3, 0x5a, 0x7c, 0x55, 0xdb, 0x5e, 0x64, 0xe1, 0x37,
+  0x87, 0x30, 0x56, 0x04, 0xe5, 0x11, 0xb4, 0x29, 0x80, 0x12, 0xf1, 0x79,
+  0x39, 0x88, 0xa2, 0x02, 0x11, 0x7c, 0x27, 0x66, 0xb7, 0x88, 0xb7, 0x78,
+  0xf2, 0xca, 0x0a, 0xa8, 0x38, 0xab, 0x0a, 0x64, 0xc2, 0xbf, 0x66, 0x5d,
+  0x95, 0x84, 0xc1, 0xa1, 0x25, 0x1e, 0x87, 0x5d, 0x1a, 0x50, 0x0b, 0x20,
+  0x12, 0xcc, 0x41, 0xbb, 0x6e, 0x0b, 0x51, 0x38, 0xb8, 0x4b, 0xcb, 0x02,
+  0x03, 0x01, 0x00, 0x01, 0xa3, 0x63, 0x30, 0x61, 0x30, 0x0e, 0x06, 0x03,
+  0x55, 0x1d, 0x0f, 0x01, 0x01, 0xff, 0x04, 0x04, 0x03, 0x02, 0x01, 0x86,
+  0x30, 0x0f, 0x06, 0x03, 0x55, 0x1d, 0x13, 0x01, 0x01, 0xff, 0x04, 0x05,
+  0x30, 0x03, 0x01, 0x01, 0xff, 0x30, 0x1d, 0x06, 0x03, 0x55, 0x1d, 0x0e,
+  0x04, 0x16, 0x04, 0x14, 0xb1, 0x3e, 0xc3, 0x69, 0x03, 0xf8, 0xbf, 0x47,
+  0x01, 0xd4, 0x98, 0x26, 0x1a, 0x08, 0x02, 0xef, 0x63, 0x64, 0x2b, 0xc3,
+  0x30, 0x1f, 0x06, 0x03, 0x55, 0x1d, 0x23, 0x04, 0x18, 0x30, 0x16, 0x80,
+  0x14, 0xb1, 0x3e, 0xc3, 0x69, 0x03, 0xf8, 0xbf, 0x47, 0x01, 0xd4, 0x98,
+  0x26, 0x1a, 0x08, 0x02, 0xef, 0x63, 0x64, 0x2b, 0xc3, 0x30, 0x0d, 0x06,
+  0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x05, 0x05, 0x00,
+  0x03, 0x82, 0x01, 0x01, 0x00, 0x1c, 0x1a, 0x06, 0x97, 0xdc, 0xd7, 0x9c,
+  0x9f, 0x3c, 0x88, 0x66, 0x06, 0x08, 0x57, 0x21, 0xdb, 0x21, 0x47, 0xf8,
+  0x2a, 0x67, 0xaa, 0xbf, 0x18, 0x32, 0x76, 0x40, 0x10, 0x57, 0xc1, 0x8a,
+  0xf3, 0x7a, 0xd9, 0x11, 0x65, 0x8e, 0x35, 0xfa, 0x9e, 0xfc, 0x45, 0xb5,
+  0x9e, 0xd9, 0x4c, 0x31, 0x4b, 0xb8, 0x91, 0xe8, 0x43, 0x2c, 0x8e, 0xb3,
+  0x78, 0xce, 0xdb, 0xe3, 0x53, 0x79, 0x71, 0xd6, 0xe5, 0x21, 0x94, 0x01,
+  0xda, 0x55, 0x87, 0x9a, 0x24, 0x64, 0xf6, 0x8a, 0x66, 0xcc, 0xde, 0x9c,
+  0x37, 0xcd, 0xa8, 0x34, 0xb1, 0x69, 0x9b, 0x23, 0xc8, 0x9e, 0x78, 0x22,
+  0x2b, 0x70, 0x43, 0xe3, 0x55, 0x47, 0x31, 0x61, 0x19, 0xef, 0x58, 0xc5,
+  0x85, 0x2f, 0x4e, 0x30, 0xf6, 0xa0, 0x31, 0x16, 0x23, 0xc8, 0xe7, 0xe2,
+  0x65, 0x16, 0x33, 0xcb, 0xbf, 0x1a, 0x1b, 0xa0, 0x3d, 0xf8, 0xca, 0x5e,
+  0x8b, 0x31, 0x8b, 0x60, 0x08, 0x89, 0x2d, 0x0c, 0x06, 0x5c, 0x52, 0xb7,
+  0xc4, 0xf9, 0x0a, 0x98, 0xd1, 0x15, 0x5f, 0x9f, 0x12, 0xbe, 0x7c, 0x36,
+  0x63, 0x38, 0xbd, 0x44, 0xa4, 0x7f, 0xe4, 0x26, 0x2b, 0x0a, 0xc4, 0x97,
+  0x69, 0x0d, 0xe9, 0x8c, 0xe2, 0xc0, 0x10, 0x57, 0xb8, 0xc8, 0x76, 0x12,
+  0x91, 0x55, 0xf2, 0x48, 0x69, 0xd8, 0xbc, 0x2a, 0x02, 0x5b, 0x0f, 0x44,
+  0xd4, 0x20, 0x31, 0xdb, 0xf4, 0xba, 0x70, 0x26, 0x5d, 0x90, 0x60, 0x9e,
+  0xbc, 0x4b, 0x17, 0x09, 0x2f, 0xb4, 0xcb, 0x1e, 0x43, 0x68, 0xc9, 0x07,
+  0x27, 0xc1, 0xd2, 0x5c, 0xf7, 0xea, 0x21, 0xb9, 0x68, 0x12, 0x9c, 0x3c,
+  0x9c, 0xbf, 0x9e, 0xfc, 0x80, 0x5c, 0x9b, 0x63, 0xcd, 0xec, 0x47, 0xaa,
+  0x25, 0x27, 0x67, 0xa0, 0x37, 0xf3, 0x00, 0x82, 0x7d, 0x54, 0xd7, 0xa9,
+  0xf8, 0xe9, 0x2e, 0x13, 0xa3, 0x77, 0xe8, 0x1f, 0x4a
+};
+const unsigned int caCertLen = 969;
+

libraries/ESP8266WiFi/examples/HTTPSRequestCACert/HTTPSRequestCACert.ino

@@ -0,0 +1,133 @@
+/*
+    HTTP over TLS (HTTPS) example sketch
+
+    This example demonstrates how to use
+    WiFiClientSecure class to connect to a TLS server.
+
+    This example verifies server certificate using the
+    root CA certificate.
+
+    We fetch and display the status of
+    esp8266/Arduino project continuous integration
+    build.
+
+    Created by Ivan Grokhotkov, 2017.
+    This example is in public domain.
+*/
+
+#include <time.h>
+#include <ESP8266WiFi.h>
+#include <WiFiClientSecure.h>
+
+const char* ssid = "........";
+const char* password = "........";
+
+const char* host = "api.github.com";
+const int httpsPort = 443;
+
+// Root certificate used by api.github.com.
+// Defined in "CACert" tab.
+extern const unsigned char caCert[] PROGMEM;
+extern const unsigned int caCertLen;
+
+WiFiClientSecure client;
+
+void setup() {
+  Serial.begin(115200);
+  Serial.println();
+  Serial.print("connecting to ");
+  Serial.println(ssid);
+  WiFi.mode(WIFI_STA);
+  WiFi.begin(ssid, password);
+  while (WiFi.status() != WL_CONNECTED) {
+    delay(500);
+    Serial.print(".");
+  }
+  Serial.println("");
+  Serial.println("WiFi connected");
+  Serial.println("IP address: ");
+  Serial.println(WiFi.localIP());
+
+  // Synchronize time useing SNTP. This is necessary to verify that
+  // the TLS certificates offered by the server are currently valid.
+  Serial.print("Setting time using SNTP");
+  configTime(8 * 3600, 0, "pool.ntp.org", "time.nist.gov");
+  time_t now = time(nullptr);
+  while (now < 8 * 3600 * 2) {
+    delay(500);
+    Serial.print(".");
+    now = time(nullptr);
+  }
+  Serial.println("");
+  struct tm timeinfo;
+  gmtime_r(&now, &timeinfo);
+  Serial.print("Current time: ");
+  Serial.print(asctime(&timeinfo));
+
+  // Load root certificate in DER format into WiFiClientSecure object
+  bool res = client.setCACert_P(caCert, caCertLen);
+  if (!res) {
+    Serial.println("Failed to load root CA certificate!");
+    while (true) {
+      yield();
+    }
+  }
+}
+
+void loop() {
+  // Connect to remote server
+  Serial.print("connecting to ");
+  Serial.println(host);
+  if (!client.connect(host, httpsPort)) {
+    Serial.println("connection failed");
+    return;
+  }
+
+  // Verify validity of server's certificate
+  if (client.verifyCertChain(host)) {
+    Serial.println("Server certificate verified");
+  } else {
+    Serial.println("ERROR: certificate verification failed!");
+    return;
+  }
+
+  String url = "/repos/esp8266/Arduino/commits/master/status";
+  Serial.print("requesting URL: ");
+  Serial.println(url);
+
+  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
+               "Host: " + host + "\r\n" +
+               "User-Agent: BuildFailureDetectorESP8266\r\n" +
+               "Connection: close\r\n\r\n");
+
+  Serial.println("request sent");
+  while (client.connected()) {
+    String line = client.readStringUntil('\n');
+    if (line == "\r") {
+      Serial.println("headers received");
+      break;
+    }
+  }
+  String line = client.readStringUntil('\n');
+  if (line.startsWith("{\"state\":\"success\"")) {
+    Serial.println("esp8266/Arduino CI successfull!");
+  } else {
+    Serial.println("esp8266/Arduino CI has failed");
+  }
+  Serial.println("reply was:");
+  Serial.println("==========");
+  Serial.println(line);
+  Serial.println("==========");
+  Serial.println();
+
+  static int repeat = 0;
+  if (++repeat == 3) {
+    Serial.println("Done");
+    while (true) {
+      delay(1000);
+    }
+  }
+  delay(10000);
+}
+
+

libraries/ESP8266WiFi/examples/NTPClient/NTPClient.ino

@@ -49,6 +49,7 @@ void setup()
   // We start by connecting to a WiFi network
   Serial.print("Connecting to ");
   Serial.println(ssid);
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, pass);
   
   while (WiFi.status() != WL_CONNECTED) {

libraries/ESP8266WiFi/examples/WiFiClientBasic/WiFiClientBasic.ino

@@ -13,6 +13,7 @@ void setup() {
     delay(10);
 
     // We start by connecting to a WiFi network
+    WiFi.mode(WIFI_STA);
     WiFiMulti.addAP("SSID", "passpasspass");
 
     Serial.println();
@@ -53,11 +54,11 @@ void loop() {
     }
 
     // This will send the request to the server
-    client.print("Send this data to server");
+    client.println("Send this data to server");
 
     //read back one line from server
     String line = client.readStringUntil('\r');
-    client.println(line);
+    Serial.println(line);
 
     Serial.println("closing connection");
     client.stop();

libraries/ESP8266WiFi/examples/WiFiMulti/WiFiMulti.ino

@@ -12,6 +12,7 @@ void setup() {
     Serial.begin(115200);
     delay(10);
 	
+    WiFi.mode(WIFI_STA);
     wifiMulti.addAP("ssid_from_AP_1", "your_password_for_AP_1");
     wifiMulti.addAP("ssid_from_AP_2", "your_password_for_AP_2");
     wifiMulti.addAP("ssid_from_AP_3", "your_password_for_AP_3");
@@ -30,4 +31,4 @@ void loop() {
         Serial.println("WiFi not connected!");
         delay(1000);
     }
-}
+}

libraries/ESP8266WiFi/examples/WiFiTelnetToSerial/WiFiTelnetToSerial.ino

@@ -30,6 +30,7 @@ WiFiClient serverClients[MAX_SRV_CLIENTS];
 
 void setup() {
   Serial1.begin(115200);
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   Serial1.print("\nConnecting to "); Serial1.println(ssid);
   uint8_t i = 0;

libraries/ESP8266WiFi/examples/WiFiWebServer/WiFiWebServer.ino

@@ -30,6 +30,7 @@ void setup() {
   Serial.print("Connecting to ");
   Serial.println(ssid);
   
+  WiFi.mode(WIFI_STA);
   WiFi.begin(ssid, password);
   
   while (WiFi.status() != WL_CONNECTED) {

libraries/ESP8266WiFi/keywords.txt

@@ -17,7 +17,7 @@ WiFiClient	KEYWORD1
 WiFiServer	KEYWORD1
 WiFiUDP	KEYWORD1
 WiFiClientSecure	KEYWORD1
-
+ESP8266WiFiMulti	KEYWORD1
 #######################################
 # Methods and Functions (KEYWORD2)
 #######################################
@@ -130,9 +130,14 @@ verify	KEYWORD2
 verifyCertChain	KEYWORD2
 setCertificate	KEYWORD2
 setPrivateKey	KEYWORD2
+setCACert	KEYWORD2
+setCertificate_P	KEYWORD2
+setPrivateKey_P	KEYWORD2
+setCACert_P	KEYWORD2
 loadCertificate	KEYWORD2
 loadPrivateKey	KEYWORD2
 loadCACert	KEYWORD2
+allowSelfSignedCerts	KEYWORD2
 
 #WiFiServer
 hasClient	KEYWORD2

libraries/ESP8266WiFi/src/ESP8266WiFiAP.cpp

@@ -69,6 +69,12 @@ static bool softap_config_equal(const softap_config& lhs, const softap_config& r
     if(lhs.max_connection != rhs.max_connection) {
         return false;
     }
+    if(lhs.beacon_interval != rhs.beacon_interval) {
+        return false;
+    }
+    if(lhs.authmode != rhs.authmode) {
+        return false;
+    }
     return true;
 }
 
@@ -265,6 +271,7 @@ bool ESP8266WiFiAPClass::softAPdisconnect(bool wifioff) {
     struct softap_config conf;
     *conf.ssid = 0;
     *conf.password = 0;
+    conf.authmode = AUTH_OPEN;
     ETS_UART_INTR_DISABLE();
     if(WiFi._persistent) {
         ret = wifi_softap_set_config(&conf);
@@ -277,7 +284,7 @@ bool ESP8266WiFiAPClass::softAPdisconnect(bool wifioff) {
         DEBUG_WIFI("[APdisconnect] set_config failed!\n");
     }
 
-    if(wifioff) {
+    if(ret && wifioff) {
         ret = WiFi.enableAP(false);
     }
 

libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp

@@ -303,6 +303,13 @@ void ESP8266WiFiGenericClass::persistent(bool persistent) {
     _persistent = persistent;
 }
 
+/**
+ * gets the persistent state
+ * @return bool
+ */
+bool ESP8266WiFiGenericClass::getPersistent(){
+    return _persistent;
+}
 
 /**
  * set new mode
@@ -426,6 +433,8 @@ void wifi_dns_found_callback(const char *name, ip_addr_t *ipaddr, void *callback
 void wifi_dns_found_callback(const char *name, const ip_addr_t *ipaddr, void *callback_arg);
 #endif
 
+static bool _dns_lookup_pending = false;
+
 /**
  * Resolve the given hostname to an IP address.
  * @param aHostname     Name to be resolved
@@ -433,7 +442,14 @@ void wifi_dns_found_callback(const char *name, const ip_addr_t *ipaddr, void *ca
  * @return 1 if aIPAddrString was successfully converted to an IP address,
  *          else error code
  */
-int ESP8266WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResult) {
+int ESP8266WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResult)
+{
+    return hostByName(aHostname, aResult, 10000);
+}
+
+
+int ESP8266WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResult, uint32_t timeout_ms)
+{
     ip_addr_t addr;
     aResult = static_cast<uint32_t>(0);
 
@@ -448,7 +464,9 @@ int ESP8266WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResul
     if(err == ERR_OK) {
         aResult = addr.addr;
     } else if(err == ERR_INPROGRESS) {
-        esp_yield();
+        _dns_lookup_pending = true;
+        delay(timeout_ms);
+        _dns_lookup_pending = false;
         // will return here when dns_found_callback fires
         if(aResult != 0) {
             err = ERR_OK;
@@ -477,6 +495,9 @@ void wifi_dns_found_callback(const char *name, const ip_addr_t *ipaddr, void *ca
 #endif
 {
     (void) name;
+    if (!_dns_lookup_pending) {
+        return;
+    }
     if(ipaddr) {
         (*reinterpret_cast<IPAddress*>(callback_arg)) = ipaddr->addr;
     }

libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.h

@@ -98,7 +98,8 @@ class ESP8266WiFiGenericClass {
     public:
 
         int hostByName(const char* aHostname, IPAddress& aResult);
-
+        int hostByName(const char* aHostname, IPAddress& aResult, uint32_t timeout_ms);
+        bool getPersistent();
     protected:
 
         friend class ESP8266WiFiSTAClass;

libraries/ESP8266WiFi/src/ESP8266WiFiMulti.cpp

@@ -40,16 +40,33 @@ bool ESP8266WiFiMulti::addAP(const char* ssid, const char *passphrase) {
 
 wl_status_t ESP8266WiFiMulti::run(void) {
 
-    int8_t scanResult;
     wl_status_t status = WiFi.status();
     if(status == WL_DISCONNECTED || status == WL_NO_SSID_AVAIL || status == WL_IDLE_STATUS || status == WL_CONNECT_FAILED) {
 
-        scanResult = WiFi.scanComplete();
+        int8_t scanResult = WiFi.scanComplete();
+
         if(scanResult == WIFI_SCAN_RUNNING) {
-            // scan is running
+            // scan is running, do nothing yet
-            return WL_NO_SSID_AVAIL;
+            status = WL_NO_SSID_AVAIL;
-        } else if(scanResult > 0) {
+            return status;
-            // scan done analyze
+        } 
+
+        if(scanResult == 0) {
+            // scan done, no ssids found. Start another scan.
+            DEBUG_WIFI_MULTI("[WIFI] scan done\n");
+            DEBUG_WIFI_MULTI("[WIFI] no networks found\n");
+            WiFi.scanDelete();
+            DEBUG_WIFI_MULTI("\n\n");
+            delay(0);
+            WiFi.disconnect();
+            DEBUG_WIFI_MULTI("[WIFI] start scan\n");
+            // scan wifi async mode
+            WiFi.scanNetworks(true);
+            return status;
+        } 
+
+        if(scanResult > 0) {
+            // scan done, analyze
             WifiAPlist_t bestNetwork { NULL, NULL };
             int bestNetworkDb = INT_MIN;
             uint8 bestBSSID[6];
@@ -58,48 +75,44 @@ wl_status_t ESP8266WiFiMulti::run(void) {
             DEBUG_WIFI_MULTI("[WIFI] scan done\n");
             delay(0);
 
-            if(scanResult <= 0) {
+            DEBUG_WIFI_MULTI("[WIFI] %d networks found\n", scanResult);
-                DEBUG_WIFI_MULTI("[WIFI] no networks found\n");
+            for(int8_t i = 0; i < scanResult; ++i) {
-            } else {
-                DEBUG_WIFI_MULTI("[WIFI] %d networks found\n", scanResult);
-                for(int8_t i = 0; i < scanResult; ++i) {
 
-                    String ssid_scan;
+                String ssid_scan;
-                    int32_t rssi_scan;
+                int32_t rssi_scan;
-                    uint8_t sec_scan;
+                uint8_t sec_scan;
-                    uint8_t* BSSID_scan;
+                uint8_t* BSSID_scan;
-                    int32_t chan_scan;
+                int32_t chan_scan;
-                    bool hidden_scan;
+                bool hidden_scan;
 
-                    WiFi.getNetworkInfo(i, ssid_scan, sec_scan, rssi_scan, BSSID_scan, chan_scan, hidden_scan);
+                WiFi.getNetworkInfo(i, ssid_scan, sec_scan, rssi_scan, BSSID_scan, chan_scan, hidden_scan);
 
-                    bool known = false;
+                bool known = false;
-                    for(uint32_t x = 0; x < APlist.size(); x++) {
+                for(uint32_t x = 0; x < APlist.size(); x++) {
-                        WifiAPlist_t entry = APlist[x];
+                    WifiAPlist_t entry = APlist[x];
 
-                        if(ssid_scan == entry.ssid) { // SSID match
+                    if(ssid_scan == entry.ssid) { // SSID match
-                            known = true;
+                        known = true;
-                            if(rssi_scan > bestNetworkDb) { // best network
+                        if(rssi_scan > bestNetworkDb) { // best network
-                                if(sec_scan == ENC_TYPE_NONE || entry.passphrase) { // check for passphrase if not open wlan
+                            if(sec_scan == ENC_TYPE_NONE || entry.passphrase) { // check for passphrase if not open wlan
-                                    bestNetworkDb = rssi_scan;
+                                bestNetworkDb = rssi_scan;
-                                    bestChannel = chan_scan;
+                                bestChannel = chan_scan;
-                                    memcpy((void*) &bestNetwork, (void*) &entry, sizeof(bestNetwork));
+                                memcpy((void*) &bestNetwork, (void*) &entry, sizeof(bestNetwork));
-                                    memcpy((void*) &bestBSSID, (void*) BSSID_scan, sizeof(bestBSSID));
+                                memcpy((void*) &bestBSSID, (void*) BSSID_scan, sizeof(bestBSSID));
-                                }
                             }
-                            break;
                         }
+                        break;
                     }
-
-                    if(known) {
-                        DEBUG_WIFI_MULTI(" ---> ");
-                    } else {
-                        DEBUG_WIFI_MULTI("      ");
-                    }
-
-                    DEBUG_WIFI_MULTI(" %d: [%d][%02X:%02X:%02X:%02X:%02X:%02X] %s (%d) %c\n", i, chan_scan, BSSID_scan[0], BSSID_scan[1], BSSID_scan[2], BSSID_scan[3], BSSID_scan[4], BSSID_scan[5], ssid_scan.c_str(), rssi_scan, (sec_scan == ENC_TYPE_NONE) ? ' ' : '*');
-                    delay(0);
                 }
+
+                if(known) {
+                    DEBUG_WIFI_MULTI(" ---> ");
+                } else {
+                    DEBUG_WIFI_MULTI("      ");
+                }
+
+                DEBUG_WIFI_MULTI(" %d: [%d][%02X:%02X:%02X:%02X:%02X:%02X] %s (%d) %c\n", i, chan_scan, BSSID_scan[0], BSSID_scan[1], BSSID_scan[2], BSSID_scan[3], BSSID_scan[4], BSSID_scan[5], ssid_scan.c_str(), rssi_scan, (sec_scan == ENC_TYPE_NONE) ? ' ' : '*');
+                delay(0);
             }
 
             // clean up ram
@@ -146,15 +159,18 @@ wl_status_t ESP8266WiFiMulti::run(void) {
             } else {
                 DEBUG_WIFI_MULTI("[WIFI] no matching wifi found!\n");
             }
-        } else {
-            // start scan
-            DEBUG_WIFI_MULTI("[WIFI] delete old wifi config...\n");
-            WiFi.disconnect();
 
-            DEBUG_WIFI_MULTI("[WIFI] start scan\n");
+            return status;
-            // scan wifi async mode
-            WiFi.scanNetworks(true);
         }
+       
+       
+        // scan failed, or some other condition not handled above. Start another scan.
+        DEBUG_WIFI_MULTI("[WIFI] delete old wifi config...\n");
+        WiFi.disconnect();
+
+        DEBUG_WIFI_MULTI("[WIFI] start scan\n");
+        // scan wifi async mode
+        WiFi.scanNetworks(true);
     }
     return status;
 }

libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp

@@ -195,6 +195,14 @@ wl_status_t ESP8266WiFiSTAClass::begin() {
     return status();
 }
 
+static void
+swap(IPAddress &lhs, IPAddress &rhs)
+{
+  IPAddress tmp = lhs;
+  lhs = rhs;
+  rhs = tmp;
+}
+
 
 /**
  * Change IP configuration settings disabling the dhcp client
@@ -210,6 +218,22 @@ bool ESP8266WiFiSTAClass::config(IPAddress local_ip, IPAddress gateway, IPAddres
         return false;
     }
 
+    //Arduino has a different arg order: ip, dns, gateway, subnet. To allow compatibility, check first octet of 3rd arg. If 255, interpret as ESP order, otherwise Arduino order.
+    if(subnet[0] != 255)
+    {
+      //octet is not 255 => interpret as Arduino order
+
+      if(dns1[0] == 0)
+      {
+        //arg order is arduino and 4th arg not given => assign it arduino default
+        dns1 = IPAddress(255,255,255,0);
+      }
+
+      //current order is arduino:                   ip-dns-gway-subnet
+      swap(gateway, subnet); //after this, order is ip-gway-dns-subnet
+      swap(subnet, dns1);    //after this, order is ip-gway-subnet-dns (correct ESP order)
+    } 
+
     struct ip_info info;
     info.ip.addr = static_cast<uint32_t>(local_ip);
     info.gw.addr = static_cast<uint32_t>(gateway);