Upgrade to https: serving for JSON, links in docs (#5992)

* Upgrade to https: serving for JSON, links in docs

Fixes #5480

* Update boards.rst documentation

* Update more documentation http: refs to https:

* Remove obsolete staging info

* Drop obsolete versions from JSON programatically

After the final merge is done on the JSON, strip out any named versions
from the final product.

Removing 1.6.5-* and 2.5.0-beta(1,2,3) for now.

* Remove 2.4.0-rc(0/1) from JSON, too

package/README.md

@@ -56,7 +56,7 @@ Here is an overview of the release process. See the section below for detailed i
 
 4. Travis CI uploads boards manager package (.zip file) and package index (.json file) to Github Releases, creating a draft release at the same time.
 
-5. Travis CI also uploads package index .json file to `http://arduino.esp8266.com/stable/package_esp8266_index.json`, i.e. well-known URL used by most users. 
+5. Travis CI also uploads package index .json file to `https://arduino.esp8266.com/stable/package_esp8266_index.json`, i.e. well-known URL used by most users.
 
 6. When the draft release is created, maintainer edits release description and inserts changelog into the description field, unmarks the release as draft, and publishes the release.
 
@@ -123,7 +123,7 @@ The following points assume work in a direct clone of the repository, and not in
 
 6. Check that the new (draft) release has been created (no editing at this point!), see https://github.com/esp8266/Arduino/releases. Check that the boards manager package .zip file has been successfully uploaded as a release artifact.
 
-7. Check that the package index downloaded from http://arduino.esp8266.com/stable/package_esp8266com_index.json contains an entry for the new version (it may not be the first one).
+7. Check that the package index downloaded from https://arduino.esp8266.com/stable/package_esp8266com_index.json contains an entry for the new version (it may not be the first one).
 
 8. Navigate to release list in Github here https://github.com/esp8266/Arduino/releases, press "Edit" button to edit release description, paste release notes, and publish it.
 

package/build_boards_manager_package.sh

@@ -155,7 +155,8 @@ curl -L -o $old_json "https://github.com/esp8266/Arduino/releases/download/${bas
 new_json=package_esp8266com_index.json
 
 set +e
-python ../../merge_packages.py $new_json $old_json >tmp && mv tmp $new_json && rm $old_json
+# Merge the old and new, then drop any obsolete package versions
+python ../../merge_packages.py $new_json $old_json | python ../../drop_versions.py - platforms 1.6.5-947-g39819f0 2.5.0-beta1 2.5.0-beta2 2.5.0-beta3 2.4.0-rc1 2.4.0-rc2 >tmp && mv tmp $new_json && rm $old_json
 
 popd
 popd

package/drop_version.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+# This script drops one or multiple versions of a release
+#
+from __future__ import print_function
+import json
+import sys
+
+def load_package(filename):
+    if filename == "-":
+        pkg = json.load(sys.stdin)['packages'][0]
+    else:
+        pkg = json.load(open(filename))['packages'][0]
+    print("Loaded package {0} from {1}".format(pkg['name'], filename), file=sys.stderr)
+    print("{0} platform(s), {1} tools".format(len(pkg['platforms']), len(pkg['tools'])), file=sys.stderr)
+    return pkg
+
+# There's probably a lambda way of doing this, but I can't figure it out...
+def drop_version(todrop, obj):
+    out = [];
+    for o in obj:
+        version = o['version'].encode('ascii')
+        if version == todrop:
+            print("Dropping version {0}".format(todrop))
+        else:
+            out.append(o)
+    return out
+
+def main(args):
+    if len(args) < 3:
+        print("Usage: {0} <inpackage> <section> <version-to-remove> ...".format(args[0]), file=sys.stderr)
+        return 1
+
+    pkg = load_package(args[1])
+    section = args[2]
+    sub = pkg[section]
+    for ver in args[3:]:
+        sub = drop_version(ver, sub)
+    pkg[section] = sub
+
+    json.dump({'packages':[pkg]}, sys.stdout, indent=2)
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv))

package/esp8266-arduino-doc.bash

@@ -31,7 +31,7 @@ release_date=$(date "+%b_%d,_%Y") # format for badge link
 build_date=$(date "+%b %d, %Y")
 destination_path="$tmp_path/doc"
 doc_template_url="https://github.com/igrr/esp8266-arduino-docs.git"
-url="http://esp8266.github.io/Arduino"
+url="https://esp8266.github.io/Arduino"
 
 # control output
 echo "Arduino ESP8266 source dir: "$arduinoESP_src

package/package_esp8266com_index.template.json

@@ -6,7 +6,7 @@
          "websiteURL": "https://github.com/esp8266/Arduino",
          "email": "ivan@esp8266.com",
          "help": {
-            "online": "http://esp8266.com/arduino"
+            "online": "https://esp8266.com/arduino"
          },
          "platforms": [
             {