diff --git a/ext/ota/README.txt b/ext/ota/README.txt new file mode 100644 index 0000000000..2707f22f1d --- /dev/null +++ b/ext/ota/README.txt @@ -0,0 +1,142 @@ + +This file contains notes regarding the implementation of the OTA extension. +User documentation is in sqlite3ota.h. + +SQLite Hacks +------------ + +1) PRAGMA ota_mode: + + This is a new flag pragma. If the flag is set: + + * INSERT/DELETE/UPDATE commands are prevented from updating any but the main + b-tree for each table (the PK index for WITHOUT ROWID tables or the + rowid b-tree for others). + + * The above statements do not check UNIQUE constraints - except those enforced + by the main b-tree. + + * All non-temporary triggers are disabled. + + +2) PRAGMA pager_ota_mode: + + This pragma sets a flag on the pager associated with the main database only. In + a zipvfs system, this pragma is intercepted by zipvfs and the flag is set on + the lower level pager only. + + The flag can only be set when there is no open transaction and the pager does + not already have an open WAL file. Attempting to do so is an error. + + Once the flag has been set, it is not possible to open a regular WAL file. + If, when the next read-transaction is opened, a *-wal file is found or the + database header flags indicate that it is a wal-mode database, + SQLITE_CANTOPEN is returned. + + Otherwise, if no WAL file or flags are found, the pager opens the *-oal file + and uses it as a write-ahead-log with the *-shm data stored in heap-memory. + + The 8-bytes of "salt" at teh start of an *-oal file is a copy of the 8 bytes + starting at offset 24 of the database file header (the change counter and the + number of pages in the file). If the *-oal file already exists when it is + opened, SQLite checks that the salt still matches the database header fields. + If not, it concludes that the database file has been written by a rollback-mode + client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT error is + returned. No read-transaction can be opened in this case. + + A pager with the pager_ota_mode flag set never runs a checkpoint. + + Other clients see a rollback-mode database on which the pager_ota_mode client + is holding a SHARED lock. There are no locks to arbitrate between multiple + pager_ota_mode connections. If two or more such connections attempt to write + simultaneously, the results are undefined. + + +3) sqlite3_index_writer() + + This new API function is used to create VMs that can insert or delete entries + from individual index b-trees within the database. The VMs apply affinities + and check that UNIQUE constraints are not violated before updating index + b-trees. + + +The OTA extension +----------------- + +The OTA extension requires that the OTA update be packaged as an SQLite +database. The tables it expects to find are described in sqlite3ota.h. +Essentially, for each table xyz in the target database that the user wishes +to write to, a corresponding data_xyz table is created in the OTA database +and populated with one row for each row to update, insert or delete from +the target table. + +The OTA extension opens the target and OTA update databases using a single +database handle (the target database is "main", and the OTA update database is +attached as "ota"). It executes both the "pager_ota_mode" and "ota_mode" +pragmas described above. For each data_xyz table in then: + + * CREATEs an ota_xyz table in the OTA update database. + + * Loops through the data_xyz table, running the INSERT, UPDATE or DELETE + command on the corresponding target database table. Only the main b-tree + is updated by these statements. Modified pages are appended to the *-oal + file. + + Temporary triggers installed on the target database catch the old.* + values associated with any UPDATEd or DELETEd rows and store them in + the ota_xyz table (in the OTA update database). + + * For each index on the data_xyz table in the target database: + + Loop through a union of the data_xyz and ota_xyz tables in the order + specified by the data_xyz index. In other words, if the index is on + columns (a, b), read rows from the OTA update database using: + + SELECT * FROM data_xyz UNION ALL ota_xyz ORDER BY a, b; + + For each row visited, use an sqlite3_index_writer() VM to update the index + in the target database. + + * DROPs the ota_xyz table. + +At any point in the above, the process may be suspended by the user. In this +case the "ota_state" table is created in the OTA database, containing a single +row indicating the current table/index being processed and the number of updates +already performed on it, and the transaction on the target database is committed +to the *-oal file. The next OTA client will use the contents of the ota_state +table to continue the update from where this one left off. + +Alternatively, if the OTA update is completely applied, the transaction is +committed to the *-oal file and the database connection closed. sqlite3ota.c +then uses a rename() call to move the *-oal file to the corresponding *-wal +path. At that point it is finished - it does not take responsibility for +checkpointing the *-wal file. + + +Problems +-------- + +The rename() call might not be portable. And in theory it is unsafe if some +other client starts writing the db file. + +When state is saved, the commit to the *-oal file and the commit to the OTA +update database are not atomic. So if the power fails at the wrong moment they +might get out of sync. As the main database will be committed before the OTA +update database this will likely either just pass unnoticed, or result in +SQLITE_CONSTRAINT errors (due to UNIQUE constraint violations). + +If some client does modify the target database mid OTA update, or some other +error occurs, the OTA extension will keep throwing errors. It's not really +clear how to get out of this state. The system could just by delete the OTA +update database and *-oal file and have the device download the update again +and start over. + +At present, for an UPDATE, both the new.* and old.* records are collected in +the ota_xyz table. And for both UPDATEs and DELETEs all fields are collected. +This means we're probably writing a lot more data to disk when saving the +state of an ongoing update to the OTA update database than is strictly +necessary. + + + + diff --git a/manifest b/manifest index 8130805f1d..ebff66aeaf 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C Add\san\sAPI\sto\squery\san\sota\shandle\sfor\sthe\stotal\snumber\sof\skey/value\soperations\sperformed\sso\sfar. -D 2014-09-18T14:48:38.579 +C Add\snew\sfile\sext/ota/README.txt,\scontaining\snotes\sregarding\sthe\simplementation\sof\sthe\sota\sextension. +D 2014-09-18T15:22:48.861 F Makefile.arm-wince-mingw32ce-gcc d6df77f1f48d690bd73162294bbba7f59507c72f F Makefile.in cf57f673d77606ab0f2d9627ca52a9ba1464146a F Makefile.linux-gcc 91d710bdc4998cb015f39edf3cb314ec4f4d7e23 @@ -121,6 +121,7 @@ F ext/misc/totype.c 4a167594e791abeed95e0a8db028822b5e8fe512 F ext/misc/vfslog.c fe40fab5c077a40477f7e5eba994309ecac6cc95 F ext/misc/vtshim.c babb0dc2bf116029e3e7c9a618b8a1377045303e F ext/misc/wholenumber.c 784b12543d60702ebdd47da936e278aa03076212 +F ext/ota/README.txt cb11e39bfeba952ac8896dab860ada9d54731fb8 F ext/ota/ota.c c11a85af71dccc45976622fe7a51169a481caa91 F ext/ota/ota1.test 7cbf37a9f6cd29320f47b041cfeb0cc1d7eaa916 F ext/ota/ota2.test 716f9c66e8bf8b0ad2fe3a5d8323e6cf460a2e27 @@ -1205,7 +1206,7 @@ F tool/vdbe_profile.tcl 67746953071a9f8f2f668b73fe899074e2c6d8c1 F tool/warnings-clang.sh f6aa929dc20ef1f856af04a730772f59283631d4 F tool/warnings.sh 0abfd78ceb09b7f7c27c688c8e3fe93268a13b32 F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f -P 67ea2979d5831b6d0d55173bd9413b21644cf6a1 -R 38d6ede7d1ef519b3ce4a32ffbb6512b +P e3943fa7bbbfc5e16f73a494d8fa54d19e9cfcf9 +R fb02b3fe38b193b1aa6184b0012599dc U dan -Z 2cc714eb22b38452d154c6b43dc8544c +Z 4cd279ef72288cc868c417c33c545f0f diff --git a/manifest.uuid b/manifest.uuid index 38824d34b1..d10a6c3486 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -e3943fa7bbbfc5e16f73a494d8fa54d19e9cfcf9 \ No newline at end of file +3c6e1cbb4baaebc5958ab7276f27ff0ba14f4fa2 \ No newline at end of file