Use esptool.py to handle sketch upload, make python available on Windows, too (#5635)

* Add esptool.py, pyserial, and python to JSON

Add installation of python on Win32/Win64, and on all systems install
esptool.py and pyserial.

* Initial esptool.py upload test

* First successfull esptool.py upload

* Patch in verbose flag operation

* Replace esptool-ck.exe with Python equivalent

Remove need for binary esptool-ck.exe by implementing the same logic as
esptool-ck uses in Python.  Only image creation is supported, and only
in the Arduino standard mode (with its custom bootloader and ROM
layout).

* Remove all esptool-ck.exe, hook Windows Python

Remove all references to esptool-ck and use Python on Windows and Linux
for all recipes where possible.

* Use python to make core_version as well

Avoid ugly bash and CMD.exe tricks in platform.txt by using python to
make the core_version header.

* Rename conflicting script, clean up packager

* Windows test passes

Need to make sure Python2 and Python3 compatible and paths are munged
properly to avoid eaccidentally escaping things when calling esptool.py

Able to compile, build a BIN and upload via esptool.py on a Windows
machine without Python installed globally, only as part of the Arduino
tools package.

* Use github sources for pyserial

* Erase calibration or all flash before programming

Add back in erase support by calling esptool.py twice (since it does not
support chained operations like esptool-ck.exe).

* Make 460K default speed, remove 961K

961K doesn't seem to work with esptool, so make 460K the default upload
speed and remove 961K.

Even at this lower speed, esptool.py is much faster to upload (even
before taking into account the compression when doing things like SPIFFS
and code upload).

* Make erase and upload work again

Arduino does not support a upload.#.cmd pattern, so we need to do
everything in a single command line.  Make it cleaner by introducing a
Python wrapper script which will run the same executable with different
sets of commands (since we need to erase a block w/a separate invocation
from the real upload).

Update boards.txt to use the new options format, placing the esptool
command as "version" when there is no "erase_flash" or "erase_region" to
be done to keep things simple.

* Move esptool/pyserial to submodules

Since esptool.py and pyserial are coming directly from github repos,
there is no need to include them as a tool in package.json.

* Restore 921K upload opt, silent downgrade to 460k

To enable full backward compatibility, restore the 921k option for
upload speed but silently change it to 460k in the upload.py script.

Add error checking on upload.py

tools/boards.txt.py

@@ -851,7 +851,7 @@ macros = {
         ( '.upload.tool', 'esptool' ),
         ( '.upload.maximum_data_size', '81920' ),
         ( '.upload.wait_for_upload_port', 'true' ),
-        ( '.upload.erase_cmd', ''),
+        ( '.upload.erase_cmd', 'version'),
         ( '.serial.disableDTR', 'true' ),
         ( '.serial.disableRTS', 'true' ),
         ( '.build.mcu', 'esp8266' ),
@@ -1066,11 +1066,11 @@ macros = {
 
     'flash_erase_menu': collections.OrderedDict([
         ( '.menu.wipe.none', 'Only Sketch' ),
-        ( '.menu.wipe.none.upload.erase_cmd', '' ),
+        ( '.menu.wipe.none.upload.erase_cmd', 'version' ),
         ( '.menu.wipe.sdk', 'Sketch + WiFi Settings' ),
-        ( '.menu.wipe.sdk.upload.erase_cmd', '-ca "{build.rfcal_addr}" -cz 0x4000' ),
+        ( '.menu.wipe.sdk.upload.erase_cmd', 'erase_region "{build.rfcal_addr}" 0x4000' ),
         ( '.menu.wipe.all', 'All Flash Contents' ),
-        ( '.menu.wipe.all.upload.erase_cmd', '-ca 0x0 -cz "{build.flash_size_bytes}"' ),
+        ( '.menu.wipe.all.upload.erase_cmd', 'erase_flash' ),
         ]),
 
     }

tools/elf2bin.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python
+
+# Generate an Arduino compatible BIN file from bootloader and sketch ELF
+# Replaces esptool-ck.exe and emulates its behavior.
+#
+# Copyright (C) 2019 - Earle F. Philhower, III
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import re
+import os
+import subprocess
+import sys
+import tempfile
+
+fmodeb = { 'dout': 3, 'dio': 2, 'quot': 1, 'qio': 0 }
+ffreqb = { '40': 0, '26': 1, '20': 2, '80': 15 }
+fsizeb = { '512K': 0, '256K': 1, '1M': 2, '2M': 3, '4M': 4, '8M': 8, '16M': 9 }
+
+def get_elf_entry(elf, path):
+    p = subprocess.Popen([path + "/xtensa-lx106-elf-readelf", '-h', elf], stdout=subprocess.PIPE, universal_newlines=True )
+    lines = p.stdout.readlines()
+    for line in lines:
+        if 'Entry point address' in line:
+            words = re.split('\s+', line)
+            entry_point = words[-2]
+            return int(entry_point, 16)
+    raise Exception('Unable to find entry point in file "' + elf + '"')
+
+def get_segment_size_addr(elf, segment, path):
+    p = subprocess.Popen([path + '/xtensa-lx106-elf-objdump', '-h', '-j', segment,  elf], stdout=subprocess.PIPE, universal_newlines=True )
+    lines = p.stdout.readlines()
+    for line in lines:
+        if segment in line:
+            words = re.split('\s+', line)
+            size = int(words[3], 16)
+            addr = int(words[4], 16)
+            return [ size, addr ]
+    raise Exception('Unable to find size and start point in file "' + elf + '" for "' + segment + '"')
+
+def read_segment(elf, segment, path):
+    tmpfile, dumpfile = tempfile.mkstemp()
+    os.close(tmpfile)
+    p = subprocess.check_call([path + "/xtensa-lx106-elf-objcopy", '-O', 'binary', '--only-section=' + segment, elf, dumpfile], stdout=subprocess.PIPE)
+    binfile = open(dumpfile, "rb")
+    raw = binfile.read()
+    binfile.close()
+    return raw
+
+def write_bin(out, elf, segments, to_addr, flash_mode, flash_size, flash_freq, path):
+    entry = int(get_elf_entry( elf, path ))
+    header = [ 0xe9, len(segments), fmodeb[flash_mode], ffreqb[flash_freq] + 16 * fsizeb[flash_size],
+               entry & 255, (entry>>8) & 255, (entry>>16) & 255, (entry>>24) & 255 ]
+    out.write(bytearray(header))
+    total_size = 8
+    checksum = 0xef
+    for segment in segments:
+        [size, addr] = get_segment_size_addr(elf, segment, path)
+        seghdr = [ addr & 255, (addr>>8) & 255, (addr>>16) & 255, (addr>>24) & 255,
+                   size & 255, (size>>8) & 255, (size>>16) & 255, (size>>24) & 255]
+        out.write(bytearray(seghdr));
+        total_size += 8;
+        raw = read_segment(elf, segment, path)
+        if len(raw) != size:
+            raise Exception('Segment size doesn\'t match read data for "' + segment + '" in "' + elf + '"')
+        out.write(raw)
+        total_size += len(raw)
+        try:
+            for data in raw:
+                checksum = checksum ^ ord(data)
+        except:
+            for data in raw:
+                checksum = checksum ^ data
+    total_size += 1
+    while total_size & 15:
+        total_size += 1
+        out.write(bytearray([0]))
+    out.write(bytearray([checksum]))
+    if to_addr != 0:
+        while total_size < to_addr:
+            out.write(bytearray([0xaa]))
+            total_size += 1
+
+def main():
+    parser = argparse.ArgumentParser(description='Create a BIN file from eboot.elf and Arduino sketch.elf for upload by esptool.py')
+    parser.add_argument('-e', '--eboot', action='store', required=True, help='Path to the Arduino eboot.elf bootloader')
+    parser.add_argument('-a', '--app', action='store', required=True, help='Path to the Arduino sketch ELF')
+    parser.add_argument('-m', '--flash_mode', action='store', required=True, choices=['dout', 'dio', 'qout', 'qio'], help='SPI flash mode')
+    parser.add_argument('-f', '--flash_freq', action='store', required=True, choices=['20', '26', '40', '80'], help='SPI flash speed')
+    parser.add_argument('-s', '--flash_size', action='store', required=True, choices=['256K', '512K', '1M', '2M', '4M', '8M', '16M'], help='SPI flash size')
+    parser.add_argument('-o', '--out', action='store', required=True, help='Output BIN filename')
+    parser.add_argument('-p', '--path', action='store', required=True, help='Path to Xtensa toolchain binaries')
+
+    args = parser.parse_args()
+
+    print('Creating BIN file "' + args.out + '" using "' + args.app + '"')
+
+    out = open(args.out, "wb")
+    write_bin(out, args.eboot, ['.text'], 4096, args.flash_mode, args.flash_size, args.flash_freq, args.path)
+    write_bin(out, args.app, ['.irom0.text', '.text', '.data', '.rodata'], 0, args.flash_mode, args.flash_size, args.flash_freq, args.path)
+    out.close()
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

tools/makecorever.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python
+
+# Generate the core_version.h header per-build
+#
+# Copyright (C) 2019 - Earle F. Philhower, III
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import os
+import subprocess
+
+parser = argparse.ArgumentParser(description='Generate core_version.h')
+parser.add_argument('-b', '--build_path', action='store', required=True, help='build.path variable')
+parser.add_argument('-p', '--platform_path', action='store', required=True, help='platform.path variable')
+parser.add_argument('-v', '--version', action='store', required=True, help='version variable')
+
+args = parser.parse_args()
+
+core = args.build_path + '/core'
+try:
+    os.makedirs(core)
+except:
+    pass
+
+out = open(core + '/core_version.h', "w")
+
+try:
+    p = subprocess.Popen(['git', '--git-dir', args.platform_path + '/.git', 'rev-parse', '--short=8', 'HEAD'], stdout = subprocess.PIPE )
+    git_ver = '0x' + p.stdout.readlines()[0].strip()
+    p = subprocess.Popen(['git', '--git-dir', args.platform_path + '/.git', 'describe', '--tags'], stdout = subprocess.PIPE )
+    git_desc = p.stdout.readlines()[0].strip()
+except:
+    git_ver = '0xffffffff'
+    git_desc = args.version
+
+out.write('#define ARDUINO_ESP8266_GIT_VER  ' + git_ver + '\n')
+out.write('#define ARDUINO_ESP8266_GIT_DESC ' + git_desc + '\n')
+
+out.close()

tools/upload.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+
+# Wrapper for Arduino core / others that can call esptool.py possibly multiple times
+# Adds pyserial to sys.path automatically based on the path of the current file
+
+# First patameter is pyserial path, then a series of command arguments separated with --end
+# i.e. upload.py tools/pyserial tools/esptool/esptool.py erase_flash --end write_flash file 0x0 --end
+
+import inspect
+import os
+import sys
+
+sys.argv.pop(0) # Remove executable name
+try:
+    sys.path.append(sys.argv.pop(0).replace('\\', '/')) # Add pyserial dir to search path, in UNIX format
+    esptool = sys.argv.pop(0).replace('\\', '/') # Full path to esptool.py, in UNIX format
+except:
+    sys.stderr.write("Error in command line, need pyserial path as 1st arg and esptool path as 2nd.\n")
+    sys.exit(1)
+
+fakeargs = [];
+while len(sys.argv):
+    if sys.argv[0] == '--end':
+        pid  = os.fork()
+        if pid == 0:
+            sys.argv = ['esptool.py'] + fakeargs
+            sys.stderr.write("Running: " + " ".join(sys.argv) + "\n")
+            exec(open(esptool).read())
+            sys.exit(0)
+        else:
+            os.waitpid(pid, 0)
+        sys.argv.pop(0) # Remove --end
+        fakeargs = []
+    else:
+        # We silently replace the 921kbaud setting with 460k to enable backward
+        # compatibility with the old esptool-ck.exe.  Esptool.py doesn't seem
+        # work reliably at 921k, but is still significantly faster at 460kbaud.
+        thisarg = sys.argv.pop(0)
+        if thisarg == "921600":
+            thisarg = "460800"
+        fakeargs = fakeargs + [thisarg]