Improvements to the existing ETag implementation (#8227)

* WebServer eTag implementation improvements

libraries/ESP8266WebServer/examples/WebServer/README.md

@@ -0,0 +1,199 @@
+# WebServer example documentation and hints
+
+This example shows different techniques on how to use and extend the ESP8266WebServer for specific purposes
+
+It is a small project in it's own and has some files to use on the web server to show how to use simple REST based services.
+
+It requires some space for a filesystem and runs fine on ESP8266 NodeMCU board with 4 MByte flash using the following options:
+* Flash Size Option 4MB (FS:2MB)
+* Debug Port Serial
+* MMU 32+32 balanced
+
+It features
+
+* http access to the web server
+* deliver all files from the file system
+* deliver special built-in files
+* implement services (list files, sysinfo)
+* uploading files using drag & drop
+* listing and deleting files using a SPA application
+* Example of SPA and Web Service application 
+* Only files in the root folder are supported for simplicity - no directories.
+
+
+
+
+## Implementing a web server
+
+The ESP8266WebServer library offers a simple path to implement a web server on a ESP8266 board.
+
+The advantage on using the ESP8266WebServer instead of the plain simple WiFiServer is that the ESP8266WebServer
+takes much care about the http protocol conventions and features and allows easily access to parameters.
+It offers plug-in capabilities by registering specific functionalities that will be outlined below.
+
+
+### Initialization
+
+In the setup() function in the webserver.ino sketch file the following steps are implemented to make the webserver available on the local network.
+
+* Create a webserver listening to port 80 for http requests.
+* Initialize the access to the filesystem in the free flash memory (typically 2MByte).
+* Connect to the local WiFi network. Here is only a straight-forward implementation hard-coding network name and passphrase. You may consider to use something like the WiFiManager library.
+* Register the device in DNS using a known hostname.
+* Registering several plug-ins (see below).
+* Starting the web server.
+
+
+### Running
+
+In the loop() function the web server will be given time to receive and send network packages by calling
+`server.handleClient();`.
+
+
+
+## Registering simple functions to implement RESTful services
+
+Registering function is the simplest integration mechanism available to add functionality. The server offers the `on(path, function)` methods that take the URL and the function as parameters.
+
+There are 2 functions implemented that get registered to handle incoming GET requests for given URLs.
+
+The JSON data format is used often for such services as it is the "natural" data format of the browser using javascript.
+
+When the **handleSysInfo()** function is registered and a browser requests for <http://webserver/$sysinfo> the function will be called and can collect the requested information.
+
+> ```CPP
+> server.on("/$sysinfo", handleSysInfo);
+> ```
+
+The result in this case is a JSON object that is assembled in the result String variable and the returned as a response to the client also giving the information about the data format.
+
+You can try this request in a browser by opening <http://webserver/$sysinfo> in the address bar.
+
+> ```CPP
+> server.on("/$sysinfo", handleList);
+> ```
+
+The function **handleList()** is registered the same way to return the list of files in the file system also returning a JSON object including name, size and the last modification timestamp.
+
+You can try this request in a browser by opening <http://webserver/$list> in the address bar.
+
+
+## Registering a function to send out some static content from a String
+
+This is an example of registering a inline function in the web server.
+The 2. parameter of the on() method is a so called CPP lamda function (without a name) 
+that actually has only one line of functionality by sending a string as result to the client.
+
+> ```CPP
+> server.on("/$upload.htm", []() {
+>   server.send(200, "text/html", FPSTR(uploadContent));
+> });
+> ```
+
+Here the text from a static String with html code is returned instead of a file from the filesystem. 
+The content of this string can be found in the file `builtinfiles.h`. It contains a small html+javascript implementation
+that allows uploading new files into the empty filesystem.
+
+Just open <http://webserver/$upload.htm> and drag some files from the data folder on the drop area.
+
+
+## Registering a function to handle requests to the server without a path
+
+Often servers are addressed by using the base URL like <http://webserver/> where no further path details is given.
+Of course we like the user to be redirected to something usable. Therefore the `handleRoot()` function is registered:
+
+> ```CPP
+> server.on("/$upload.htm", handleRoot);
+> ```
+
+The `handleRoot()` function checks the filesystem for the file named **/index.htm** and creates a redirect to this file when the file exists.
+Otherwise the redirection goes to the built-in **/$upload.htm** web page.
+
+
+
+## Using the serveStatic plug-in
+
+The **serveStatic** plug in is part of the library and handles delivering files from the filesystem to the client. It can be customized in some ways.
+
+> ```CPP
+> server.enableCORS(true);
+> server.enableETag(true);
+> server.serveStatic("/", LittleFS, "/");
+> ```
+
+
+### Cross-Origin Ressource Sharing (CORS)
+
+The `enableCORS(true)` function adds a `Access-Control-Allow-Origin: *` http-header to all responses to the client
+to inform that it is allowed to call URLs and services on this server from other web sites.
+
+The feature is disabled by default (in the current version) and when you like to disable this then you should call `enableCORS(false)` during setup.
+
+* Web sites providing high sensitive information like online banking this is disabled most of the times.
+* Web sites providing advertising information or reusable scripts / images this is enabled.
+
+
+### ETag support
+
+The `enableETag(true)` function adds a ETag http header to the responses to the client that come from files from the filesystem
+to enable better use of the cache in the browser.
+
+When enabled by default the server reads the file content and creates a checksum using the md5 and base64 algorithm her called the ETag value
+that changes whenever the file contains something different.
+
+Once a browser has got the content of a file from the server including the ETag information it will add that ETag value is added in the following requests for the same resource.
+Now the server can answer with a 'use the version from the cache' when the new calculated ETag value is equal to the ETag value in the request.
+
+The calculation of the ETag value requires some time and processing but sending content is always slower.
+So when you have the situation that a browser will use a web server multiple times this mechanism saves network and computing and makes web pages more responsive.
+
+In the source code you can find another version of an algorithm to calculate a ETag value that uses the date&time from the filesystem.
+This is a simpler and faster way but with a low risk of dismissing a file update as the timestamp is based on seconds and local time.
+This can be enabled on demand, see inline comments.
+
+
+## Registering a full-featured handler as plug-in
+
+The example also implements the class `FileServerHandler` derived from the class `RequestHandler` to plug in functionality
+that can handle more complex requests without giving a fixed URL.
+It implements uploading and deleting files in the file system that is not implemented by the standard server.serveStatic functionality.
+
+This class has to implements several functions and works in a more detailed way:
+
+* The `canHandle()` method can inspect the given http method and url to decide weather the RequestFileHandler can handle the incoming request or not.
+
+  In this case the RequestFileHandler will return true when the request method is an POST for upload or a DELETE for deleting files.
+
+  The regular GET requests will be ignored and therefore handled by the also registered server.serveStatic handler.
+
+* The function `handle()` then implements the real deletion of the file.
+
+* The `canUpload()`and `upload()` methods work similar while the `upload()` method is called multiple times to create, append data and close the new file.
+
+
+## Registering a special handler for "file not found"
+
+Any other incoming request that was not handled by the registered plug-ins above can be detected by registering 
+
+> ```CPP
+> // handle cases when file is not found
+> server.onNotFound([]() {
+>   // standard not found in browser.
+>   server.send(404, "text/html", FPSTR(notFoundContent));
+> });
+> ```
+
+This allows sending back an "friendly" result for the browser. Here a sim ple html page is created from a static string.
+You can easily change the html code in the file `builtinfiles.h`.
+
+
+## customizations
+
+You may like to change the hostname and the timezone in the lines:
+
+> ```CPP
+> #define HOSTNAME "webserver"
+> #define TIMEZONE "CET-1CEST,M3.5.0,M10.5.0/3"
+> ```
+
+

libraries/ESP8266WebServer/examples/WebServer/WebServer.ino

@@ -0,0 +1,261 @@
+// @file WebServer.ino
+// @brief Example implementation using the ESP8266 WebServer.
+//
+// See also README.md for instructions and hints.
+//
+// Changelog:
+// 21.07.2021 creation, first version
+
+#include <Arduino.h>
+#include <ESP8266WebServer.h>
+
+#include "secrets.h" // add WLAN Credentials in here.
+
+#include <FS.h>       // File System for Web Server Files
+#include <LittleFS.h> // This file system is used.
+
+// mark parameters not used in example
+#define UNUSED __attribute__((unused))
+
+// TRACE output simplified, can be deactivated here
+#define TRACE(...) Serial.printf(__VA_ARGS__)
+
+// name of the server. You reach it using http://webserver
+#define HOSTNAME "webserver"
+
+// local time zone definition (Berlin)
+#define TIMEZONE "CET-1CEST,M3.5.0,M10.5.0/3"
+
+// need a WebServer for http access on port 80.
+ESP8266WebServer server(80);
+
+// The text of builtin files are in this header file
+#include "builtinfiles.h"
+
+
+// ===== Simple functions used to answer simple GET requests =====
+
+// This function is called when the WebServer was requested without giving a filename.
+// This will redirect to the file index.htm when it is existing otherwise to the built-in $upload.htm page
+void handleRedirect() {
+  TRACE("Redirect...");
+  String url = "/index.htm";
+
+  if (!LittleFS.exists(url)) {
+    url = "/$update.htm";
+  }
+
+  server.sendHeader("Location", url, true);
+  server.send(302);
+} // handleRedirect()
+
+
+// This function is called when the WebServer was requested to list all existing files in the filesystem.
+// a JSON array with file information is returned.
+void handleListFiles() {
+  Dir dir = LittleFS.openDir("/");
+  String result;
+
+  result += "[\n";
+  while (dir.next()) {
+    if (result.length() > 4) {
+      result += ",";
+    }
+    result += "  {";
+    result += " \"name\": \"" + dir.fileName() + "\", ";
+    result += " \"size\": " + String(dir.fileSize()) + ", ";
+    result += " \"time\": " + String(dir.fileTime());
+    result += " }\n";
+    // jc.addProperty("size", dir.fileSize());
+  } // while
+  result += "]";
+  server.sendHeader("Cache-Control", "no-cache");
+  server.send(200, "text/javascript; charset=utf-8", result);
+} // handleListFiles()
+
+
+// This function is called when the sysInfo service was requested.
+void handleSysInfo() {
+  String result;
+
+  FSInfo fs_info;
+  LittleFS.info(fs_info);
+
+  result += "{\n";
+  result += "  \"flashSize\": " + String(ESP.getFlashChipSize()) + ",\n";
+  result += "  \"freeHeap\": " + String(ESP.getFreeHeap()) + ",\n";
+  result += "  \"fsTotalBytes\": " + String(fs_info.totalBytes) + ",\n";
+  result += "  \"fsUsedBytes\": " + String(fs_info.usedBytes) + ",\n";
+  result += "}";
+
+  server.sendHeader("Cache-Control", "no-cache");
+  server.send(200, "text/javascript; charset=utf-8", result);
+} // handleSysInfo()
+
+
+// ===== Request Handler class used to answer more complex requests =====
+
+// The FileServerHandler is registered to the web server to support DELETE and UPLOAD of files into the filesystem.
+class FileServerHandler : public RequestHandler {
+  public:
+    // @brief Construct a new File Server Handler object
+    // @param fs The file system to be used.
+    // @param path Path to the root folder in the file system that is used for serving static data down and upload.
+    // @param cache_header Cache Header to be used in replies.
+    FileServerHandler() {
+      TRACE("FileServerHandler is registered\n");
+    }
+
+
+    // @brief check incoming request. Can handle POST for uploads and DELETE.
+    // @param requestMethod method of the http request line.
+    // @param requestUri request ressource from the http request line.
+    // @return true when method can be handled.
+    bool canHandle(HTTPMethod requestMethod, const String UNUSED &_uri) override {
+      return ((requestMethod == HTTP_POST) || (requestMethod == HTTP_DELETE));
+    } // canHandle()
+
+
+    bool canUpload(const String &uri) override {
+      // only allow upload on root fs level.
+      return (uri == "/");
+    } // canUpload()
+
+
+    bool handle(ESP8266WebServer &server, HTTPMethod requestMethod, const String &requestUri) override {
+      // ensure that filename starts with '/'
+      String fName = requestUri;
+      if (!fName.startsWith("/")) {
+        fName = "/" + fName;
+      }
+
+      if (requestMethod == HTTP_POST) {
+        // all done in upload. no other forms.
+
+      } else if (requestMethod == HTTP_DELETE) {
+        if (LittleFS.exists(fName)) {
+          LittleFS.remove(fName);
+        }
+      } // if
+
+      server.send(200); // all done.
+      return (true);
+    } // handle()
+
+
+    // uploading process
+    void upload(ESP8266WebServer UNUSED &server, const String UNUSED &_requestUri, HTTPUpload &upload) override {
+      // ensure that filename starts with '/'
+      String fName = upload.filename;
+      if (!fName.startsWith("/")) {
+        fName = "/" + fName;
+      }
+
+      if (upload.status == UPLOAD_FILE_START) {
+        // Open the file
+        if (LittleFS.exists(fName)) {
+          LittleFS.remove(fName);
+        } // if
+        _fsUploadFile = LittleFS.open(fName, "w");
+
+      } else if (upload.status == UPLOAD_FILE_WRITE) {
+        // Write received bytes
+        if (_fsUploadFile) {
+          _fsUploadFile.write(upload.buf, upload.currentSize);
+        }
+
+      } else if (upload.status == UPLOAD_FILE_END) {
+        // Close the file
+        if (_fsUploadFile) {
+          _fsUploadFile.close();
+        }
+      } // if
+    }   // upload()
+
+  protected:
+    File _fsUploadFile;
+};
+
+
+// Setup everything to make the webserver work.
+void setup(void) {
+  delay(3000); // wait for serial monitor to start completely.
+
+  // Use Serial port for some trace information from the example
+  Serial.begin(115200);
+  Serial.setDebugOutput(false);
+
+  TRACE("Starting WebServer example...\n");
+
+  TRACE("Mounting the filesystem...\n");
+  if (!LittleFS.begin()) {
+    TRACE("could not mount the filesystem...\n");
+    delay(2000);
+    ESP.restart();
+  }
+
+  // start WiFI
+  WiFi.mode(WIFI_STA);
+  if (strlen(ssid) == 0) {
+    WiFi.begin();
+  } else {
+    WiFi.begin(ssid, passPhrase);
+  }
+
+  // allow to address the device by the given name e.g. http://webserver
+  WiFi.setHostname(HOSTNAME);
+
+  TRACE("Connect to WiFi...\n");
+  while (WiFi.status() != WL_CONNECTED) {
+    delay(500);
+    TRACE(".");
+  }
+  TRACE("connected.\n");
+
+  // Ask for the current time using NTP request builtin into ESP firmware.
+  TRACE("Setup ntp...\n");
+  configTime(TIMEZONE, "pool.ntp.org");
+
+  TRACE("Register service handlers...\n");
+
+  // serve a built-in htm page
+  server.on("/$upload.htm", []() {
+    server.send(200, "text/html", FPSTR(uploadContent));
+  });
+
+  // register a redirect handler when only domain name is given.
+  server.on("/", HTTP_GET, handleRedirect);
+
+  // register some REST services
+  server.on("/$list", HTTP_GET, handleListFiles);
+  server.on("/$sysinfo", HTTP_GET, handleSysInfo);
+
+  // UPLOAD and DELETE of files in the file system using a request handler.
+  server.addHandler(new FileServerHandler());
+
+  // enable CORS header in webserver results
+  server.enableCORS(true);
+
+  // enable ETAG header in webserver results from serveStatic handler
+  server.enableETag(true);
+
+  // serve all static files
+  server.serveStatic("/", LittleFS, "/");
+
+  // handle cases when file is not found
+  server.onNotFound([]() {
+    // standard not found in browser.
+    server.send(404, "text/html", FPSTR(notFoundContent));
+  });
+
+  server.begin();
+  TRACE("hostname=%s\n", WiFi.getHostname());
+} // setup
+
+
+// run the server...
+void loop(void) {
+  server.handleClient();
+} // loop()
+
+// end.

libraries/ESP8266WebServer/examples/WebServer/builtinfiles.h

@@ -0,0 +1,63 @@
+/**
+ * @file builtinfiles.h
+ * @brief This file is part of the WebServer example for the ESP8266WebServer.
+ *  
+ * This file contains long, multiline text variables for  all builtin resources.
+ */
+
+// used for $upload.htm
+static const char uploadContent[] PROGMEM =
+R"==(
+<!doctype html>
+<html lang='en'>
+
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Upload</title>
+</head>
+
+<body style="width:300px">
+  <h1>Upload</h1>
+  <div><a href="/">Home</a></div>
+  <hr>
+  <div id='zone' style='width:16em;height:12em;padding:10px;background-color:#ddd'>Drop files here...</div>
+
+  <script>
+    // allow drag&drop of file objects 
+    function dragHelper(e) {
+      e.stopPropagation();
+      e.preventDefault();
+    }
+
+    // allow drag&drop of file objects 
+    function dropped(e) {
+      dragHelper(e);
+      var fls = e.dataTransfer.files;
+      var formData = new FormData();
+      for (var i = 0; i < fls.length; i++) {
+        formData.append('file', fls[i], '/' + fls[i].name);
+      }
+      fetch('/', { method: 'POST', body: formData }).then(function () {
+        window.alert('done.');
+      });
+    }
+    var z = document.getElementById('zone');
+    z.addEventListener('dragenter', dragHelper, false);
+    z.addEventListener('dragover', dragHelper, false);
+    z.addEventListener('drop', dropped, false);
+  </script>
+</body>
+)==";
+
+// used for $upload.htm
+static const char notFoundContent[] PROGMEM = R"==(
+<html>
+<head>
+  <title>Ressource not found</title>
+</head>
+<body>
+  <p>The ressource was not found.</p>
+  <p><a href="/">Start again</a></p>
+</body>
+)==";

libraries/ESP8266WebServer/examples/WebServer/data/files.htm

@@ -0,0 +1,65 @@
+<html>
+
+<head>
+  <title>Files</title>
+  <link Content-Type="text/css" href="/style.css" rel="stylesheet" />
+</head>
+
+<body>
+  <h1>Files on Server</h1>
+
+  <p>These files are available on the server to be opened or delete:</p>
+  <div id="list">
+  </div>
+
+  <script>
+    // load and display all files after page loading has finished
+    window.addEventListener("load", function () {
+      fetch('/$list')
+        .then(function (result) { return result.json(); })
+        .then(function (e) {
+          var listObj = document.querySelector('#list');
+          e.forEach(function (f) {
+            var entry = document.createElement("div");
+            var nameObj = document.createElement("a");
+            nameObj.href = '/' + f.name;
+            nameObj.innerText = '/' + f.name;
+            entry.appendChild(nameObj)
+            
+            entry.appendChild(document.createTextNode(' (' + f.size + ') '));
+
+            var timeObj = document.createElement("span");
+            timeObj.innerText = (new Date(f.time*1000)).toLocaleString();
+            entry.appendChild(timeObj)
+            entry.appendChild(document.createTextNode(" "));
+
+            var delObj = document.createElement("span");
+            delObj.className = 'deleteFile';
+            delObj.innerText = ' [delete]';
+            entry.appendChild(delObj)
+
+            listObj.appendChild(entry)
+          });
+
+        })
+        .catch(function (err) {
+          window.alert(err);
+        });
+    });
+
+    window.addEventListener("click", function (evt) {
+      var t = evt.target;
+      if (t.className === 'deleteFile') {
+        var fname = t.parentElement.innerText;
+        fname = '/'+ fname.split(' ')[0];
+        if (window.confirm("Delete " + fname + " ?")) {
+          fetch(fname, { method: 'DELETE' });
+          document.location.reload(false);
+        }
+      };
+    });
+
+  </script>
+</body>
+
+</html>

libraries/ESP8266WebServer/examples/WebServer/data/index.htm

@@ -0,0 +1,24 @@
+<html>
+
+<head>
+  <title>HomePage</title>
+  <link Content-Type="text/css" href="/style.css" rel="stylesheet" />
+</head>
+
+<body>
+  <h1>Homepage of the WebServer Example</h1>
+
+  <p>The following pages are available:</p>
+  <ul>
+    <li><a href="/index.htm">/index.htm</a> - This page</li>
+    <li><a href="/files.htm">/files.htm</a> - Manage files on the server</li>
+    <li><a href="/$upload.htm">/$upload.htm</a> - Built-in upload utility</a></li>
+  </ul>
+  
+  <p>The following REST services are available:</p>
+  <ul>
+    <li><a href="/$sysinfo">/$sysinfo</a> - Some system level information</a></li>
+    <li><a href="/$list">/$list</a> - Array of all files</a></li>
+  </ul>
+</body>
+</html>

libraries/ESP8266WebServer/examples/WebServer/data/style.css

@@ -0,0 +1,10 @@
+html, body {
+  color: #111111; font-family: Arial, ui-sans-serif, sans-serif; font-size: 1em; background-color: #f0f0f0;
+}
+
+#list > div {
+  margin: 0 0 0.5rem 0;
+}
+
+a { color: inherit; cursor: pointer; }
+

libraries/ESP8266WebServer/examples/WebServer/secrets.h

@@ -0,0 +1,13 @@
+// Secrets for your local home network
+
+// This is a "hard way" to configure your local WiFi network name and passphrase
+// into the source code and the uploaded sketch.
+// 
+// Using the WiFi Manager is preferred and avoids reprogramming when your network changes.
+// See https://homeding.github.io/#page=/wifimanager.md
+
+// ssid and passPhrase can be used when compiling for a specific environment as a 2. option.
+
+// add you wifi network name and PassPhrase or use WiFi Manager
+const char *ssid = "KHMH";
+const char *passPhrase = "hk-2012FD2926";