Add a CRC32 over progmem and ESP.checkFlashCRC (#6566)

* Add a CRC32 over progmem and ESP.checkFlashCRC

Automatically embed a CRC32 of the program memory (including bootloader
but excluding any filesystems) in all images in unused space in the
bootloader block.

Add a call, ESP.checkFlashCRC() which returns false if the calculated
CRC doesn't match the one stored in the image (i.e. flash corruption).

Fixes #4165

* Add example that corrupts itself, comments

Show CRC checking catch a 1-bit error in program code by corrupting a
large array, and then return it to clean and verify the CRC matches once
again.

Add comments to the CRC check routine

Clean up pylint complaints on crc32bin.py

* Check linker script for CRC space in bootsector

Add an assertion in the eboot linker file to guarantee that we have at
least 8 bytes of unused space at the end of the boot sector to patch in
the CRC.  If not, the eboot link will fail.

* Add note about what to do if CRC check fails

Per discussion with @d-a-v.

When the CRC check fails, you could *try* to do certain things (but may
not succeed since there is known flash corruption at that point).  List
a few ideas for application authors.

* Only single, flash/ram friendly crc32() function

* Combine the CRC calc and bin generation in 1 step

Per discussion w/@mcspr, combine the CRC calculation with the binary
generation, removing the additional build step.

tools/elf2bin.py

@@ -30,6 +30,9 @@ fmodeb = { 'dout': 3, 'dio': 2, 'qout': 1, 'qio': 0 }
 ffreqb = { '40': 0, '26': 1, '20': 2, '80': 15 }
 fsizeb = { '512K': 0, '256K': 1, '1M': 2, '2M': 3, '4M': 4, '8M': 8, '16M': 9 }
 
+crcsize_offset = 4088
+crcval_offset = 4092
+
 def get_elf_entry(elf, path):
     p = subprocess.Popen([path + "/xtensa-lx106-elf-readelf", '-h', elf], stdout=subprocess.PIPE, universal_newlines=True )
     lines = p.stdout.readlines()
@@ -94,6 +97,47 @@ def write_bin(out, elf, segments, to_addr, flash_mode, flash_size, flash_freq, p
             out.write(bytearray([0xaa]))
             total_size += 1
 
+def crc8266(ldata):
+    "Return the CRC of ldata using same algorithm as eboot"
+    crc = 0xffffffff
+    idx = 0
+    while idx < len(ldata):
+        byte = int(ldata[idx])
+        idx = idx + 1
+        for i in [0x80, 0x40, 0x20, 0x10, 0x08, 0x04, 0x02, 0x01]:
+            bit = crc & 0x80000000
+            if (byte & i) != 0:
+                if bit == 0:
+                    bit = 1
+                else:
+                    bit = 0
+            crc = int(crc << 1) & 0xffffffff
+            if bit != 0:
+                crc = int(crc ^ 0x04c11db7)
+    return crc
+
+def store_word(raw, offset, val):
+    "Place a 4-byte word in 8266-dependent order in the raw image"
+    raw[offset] = val & 255
+    raw[offset + 1] = (val >> 8) & 255
+    raw[offset + 2] = (val >> 16) & 255
+    raw[offset + 3] = (val >> 24) & 255
+    return raw
+
+def add_crc(out):
+    with open(out, "rb") as binfile:
+        raw = bytearray(binfile.read())
+
+    # Zero out the spots we're going to overwrite to be idempotent
+    raw = store_word(raw, crcsize_offset, 0)
+    raw = store_word(raw, crcval_offset, 0)
+    crc = crc8266(raw)
+    raw = store_word(raw, crcsize_offset, len(raw))
+    raw = store_word(raw, crcval_offset, int(crc))
+
+    with open(out, "wb") as binfile:
+        binfile.write(raw)
+
 def main():
     parser = argparse.ArgumentParser(description='Create a BIN file from eboot.elf and Arduino sketch.elf for upload by esptool.py')
     parser.add_argument('-e', '--eboot', action='store', required=True, help='Path to the Arduino eboot.elf bootloader')
@@ -113,6 +157,9 @@ def main():
     write_bin(out, args.app, ['.irom0.text', '.text', '.text1', '.data', '.rodata'], 0, args.flash_mode, args.flash_size, args.flash_freq, args.path)
     out.close()
 
+    # Because the CRC includes both eboot and app, can only calculate it after the entire BIN generated
+    add_crc(args.out)
+
     return 0