sqlite/autosetup

Maintaining Autosetup in the SQLite Tree

This document provides some tips and reminders for the SQLite developers regarding using and maintaining the Autosetup-based build infrastructure. It is not an Autosetup reference.

Table of Contents:

• Autosetup API Reference
• API Tips
• Ensuring TCL Compatibility
• Design Conventions
  • Symbolic Names of Feature Flags
  • Do Not Update Global Shared State
• Updating Autosetup
  • Patching Autosetup for Project-local changes
• Branch-specific Customization

---

Autosetup API Reference

The Autosetup API is quite extensive and can be read either in the files in the autosetup dir or using:

$ ./configure --reference | less

That will include any docs from any TCL files in the ./autosetup dir which contain certain (simple) markup defined by autosetup.

This project's own configuration-related TCL code is spread across the following files:

• proj.tcl: project-agnostic utility code for autosetup-driven projects. This file is designed to be shared between this project, other projects managed under the SQLite/Hwaci umbrella (e.g. Fossil), and personal projects of SQLite's developers. It is essentially an amalgamation of a decade's worth of autosetup-related utility code.
• sqlite-config.tcl: utility code which is too project-specific for proj.tcl. We split this out of auto.def so that it can be used by both auto.def and...
• auto.def: the primary driver for the ./configure process. When we talk about "the configure script," we're technically referring to this file, though it actually contains very little of the TCL code.
• autoconf/auto.def: the main driver script for the "autoconf" bundle's configure script. It is essentially a slightly trimmed-down version of the main auto.def file. The autoconf dir was ported from the Autotools to Autosetup in the 3.49.0 dev cycle but retains the "autoconf" name to minimize downstream disruption.

Autosetup API Tips

This section briefly covers only APIs which are frequently useful in day-to-day maintenance and might not be immediately recognized as such from a casual perusal of the relevant TCL files. The complete docs of those with proj- prefix can be found in proj.tcl and those with an sqlite- prefix are in sqlite-config.tcl. The others are part of Autosetup's core packages and are scattered around the TCL files in ./autosetup.

In (mostly) alphabetical order:

• file-isexec filename\
  Should be used in place of [file executable], as it will also check for ${filename}.exe on Windows platforms. However, on such platforms it also assumes that any existing file is executable.

• get-env VAR ?default?\
  Will fetch an "environment variable" from the first of either: (1) a KEY=VALUE passed to the configure script or (2) the system's environment variables. Not to be confused with getenv, which only does the latter and is rarely, if ever, useful in this tree.

  • proj-get-env VAR ?default?\
    Works like get-env but will, if that function finds no match, look for a file named ./.env-$VAR and, if found, return its trimmed contents. This can be used, e.g., to set a developer's local preferences for the default CFLAGS.\
    Tip: adding -O0 to .env-CFLAGS reduces rebuild times considerably at the cost of performance in make devtest and the like.

• proj-fatal msg\
  Emits $msg to stderr and exits with non-zero. Its differences from autosetup's user-error are purely cosmetic.

• proj-if-opt-truthy flag thenScript ?elseScript?\
  Evals thenScript if the given --flag is truthy, else it evals the optional elseScript.

• proj-indented-notice ?-error? ?-notice? msg\
  Breaks its msg argument into lines, trims them, and emits them with consistent indentation. Exactly how it emits depends on the flags passed to it (or not), as covered in its docs. This will stick out starkly from normal output and is intended to be used only for important notices.

• proj-opt-truthy flag\
  Returns 1 if --flag's value is "truthy," i.e. one of (1, on, enabled, yes, true).

• proj-opt-was-provided FLAG\
  Returns 1 if --FLAG was explicitly provided to configure, else 0. This distinction can be used to determine, e.g., whether --with-readline was provided or whether we're searching for readline by default. In the former case, failure to find it should be treated as fatal, where in the latter case it's not.\
  Unlike most functions which deal with --flags, this one does not validate that $FLAG is a registered flag so will not fail fatally if $FLAG is not registered as an Autosetup option.

• proj-val-truthy value\
  Returns 1 if $value is "truthy," See proj-opt-truthy for the definition of "truthy."

• proj-warn msg\
  Emits $msg to stderr. Closely-related is autosetup's user-notice (described below).

• sqlite-add-feature-flag ?-shell? FLAG...\
  Adds the given feature flag to the CFLAGS which are specific to building libsqlite3. It's intended to be passed one or more -DSQLITE_ENABLE_..., or similar, flags. If the -shell flag is used then it also passes its arguments to sqlite-add-shell-opt. This is a no-op if FLAG is not provided or is empty.

• sqlite-add-shell-opt FLAG...\
  The shell-specific counterpart of sqlite-add-feature-flag which only adds the given flag(s) to the CLI-shell-specific CFLAGS.

• sqlite-configure BUILD-NAME {script}\
  This is where all configure --flags are defined for all known build modes ("canonical" or "autoconf"). After processing all flags, this function runs $script, which contains the build-mode-specific configuration bits, and then runs any finalization bits which are common to all build modes. The auto.def files are intended to contain exactly two commands: use sqlite-config; sqlite-configure BUILD-NAME {script}

• user-notice msg\
  Queues $msg to be sent to stderr, but does not emit it until either show-notices is called or the next time autosetup would output something (it internally calls show-notices). This can be used to generate warnings between a "checking for..." message and its resulting "yes/no/whatever" message in such a way as to not spoil the layout of such messages.

Ensuring TCL Compatibility

One of the significant benefits of using Autosetup is that (A) this project uses many TCL scripts in the build process and (B) Autosetup comes with a TCL interpreter named JimTCL.

It is important that any TCL files used by the configure process and makefiles remain compatible with both JimTCL and the canonical TCL. Though JimTCL has outstanding compatibility with canonical TCL, it does have a few corners with incompatibilities, e.g. regular expressions. If a script runs in JimTCL without using any JimTCL-specific features, then it's a certainty that it will run in canonical TCL as well. The opposite, however, is not always the case.

When ./configure is run, it goes through a bootstrapping process to find a suitable TCL with which to run the autosetup framework. The first step involves finding or building a TCL shell. That will first search for an available tclsh (under several common names, e.g. tclsh8.6) before falling back to compiling the copy of jimsh0.c included in the source tree. i.e. it will prefer to use a system-installed TCL for running the configure script. Once it finds (or builds) a TCL shell, it then runs a sanity test to ensure that the shell is suitable before using it to run the main autosetup app.

There are two simple ways to ensure that running of the configure process uses JimTCL instead of the canonical tclsh, and either approach provides equally high assurances about configure script compatibility across TCL implementations:

1. Build on a system with no tclsh installed in the $PATH. In that case, the configure process will fall back to building the in-tree copy of JimTCL.

2. Manually build ./jimsh0 in the top of the checkout with:\
   cc -o jimsh0 autosetup/jimsh0.c\
   With that in place, the configure script will prefer to use that before looking for a system-level tclsh. Be aware, though, that make distclean will remove that file.

Note that ./jimsh0 is distinctly different from the ./jimsh which gets built for code-generation purposes. The latter requires non-default build flags to enable features which are platform-dependent, most notably to make its [file normalize] work. This means, for example, that the configure script and its utility APIs must not use [file normalize], but autosetup provides a TCL-only implementation of [file-normalize] (note the dash) for portable use in the configure script. Contrariwise, code-generation scripts invoked via make may use [file normalize], as they'll use ./jimsh or tclsh instead of ./jimsh0.

Known TCL Incompatibilities

A summary of known incompatibilities in JimTCL

• CRNL line endings: prior to 2025-02-05 fconfigure -translation ... was a no-op in JimTCL, and it emits CRNL line endings by default on Windows. Since then, it supports -translation binary, which is close enough to -translation lf for our purposes. When working with files using the open command, it is important to use mode "rb" or "wb", as appropriate, so that the output does not get CRNL-mangled on Windows.

• file copy does not support multiple source files. See for a workaround.

• Regular expressions:

  • Patterns treat \nnn octal values as back-references (which it does not support). Those can be reformulated as demonstrated in .

  • regsub does not support the \y flag. A workaround is demonstrated in .

Design Conventions

This section describes the motivations for the most glaring of the build's design decisions, in particular how they deviate from historical, or even widely-conventional, practices.

Symbolic Names of Feature Flags

Historically, the project's makefile has exclusively used UPPER_UNDERSCORE form for makefile variables. This build, however, primarily uses X.y format, where X is often a category label, e.g. CFLAGS, and y is the specific instance of that category, e.g. CFLAGS.readline.

When the configure script exports flags for consumption by filtered files, e.g. Makefile.in and the generated sqlite_cfg.h, it does so in the more conventional X_Y form because those flags get exported as as C #defines to sqlite_cfg.h, where dots are not permitted.

The X.y convention is used in the makefiles primarily because the person who did the initial port finds that considerably easier on the eyes and fingers. In practice, the X_Y form of such exports is used exactly once in Makefile.in, where it's translated from @X_Y@ into into X.y form for consumption by Makefile.in and main.mk. For example:

LDFLAGS.shobj = @SHOBJ_LDFLAGS@
LDFLAGS.zlib = @LDFLAGS_ZLIB@
LDFLAGS.math = @LDFLAGS_MATH@

(That first one is defined by autosetup, and thus applies "LDFLAGS" as the suffix rather than the prefix. Which is more legible is a matter of taste, for which there is no accounting.)

Do Not Update Global Shared State

In both the legacy Autotools-driven build and common Autosetup usage, feature tests performed by the configure script may amend global flags such as LIBS, LDFLAGS, and CFLAGS¹ . That's appropriate for a makefile which builds a single deliverable, but less so for makefiles which produce multiple deliverables. Drawbacks of that approach include:

• It's unlikely that every single deliverable will require the same core set of those flags.
• It can be difficult to determine the origin of any given change to that global state because those changes are hidden behind voodoo performed outside the immediate visibility of the configure script's maintainer.
• It can force the maintainers of the configure script to place tests in a specific order so that the resulting flags get applied at the correct time and/or in the correct order.\
  (A real-life example: before the approach described below was taken to collecting build-time flags, the test for -rpath had to come after the test for zlib because the results of the -rpath test implicitly modified global state which broke the zlib feature test. Because the feature tests no longer (intentionally) modify shared global state, that is not an issue.)

In this build, cases where feature tests modify global state in such a way that it may impact later feature tests are either (A) very intentionally defined to do so (e.g. the --with-wasi-sdk flag has invasive side-effects) or (B) are oversights (i.e. bugs).

This tree's configure script, utility APIs, Makefile.in, and main.mk therefore strive to separate the results of any given feature test into its own well-defined variables. For example:

• The linker flags for zlib are exported from the configure script as LDFLAGS_ZLIB, which Makefile.in and main.mk then expose as LDFLAGS.zlib.
• CFLAGS_READLINE (a.k.a. CFLAGS.readline) contains the CFLAGS needed for including libreadline, libedit, or linenoise, and LDFLAGS_READLINE (a.k.a. LDFLAGS.readline) is its link-time counterpart.

It is then up to the Makefile to apply and order the flags however is appropriate.

At the end of the configure script, the global CFLAGS ideally holds only flags which are either relevant to all targets or, failing that, will have no unintended side-effects on any targets. That said: clients frequently pass custom CFLAGS to ./configure or make to set library-level feature toggles, e.g. -DSQLITE_OMIT_FOO, in which case there is no practical way to avoid "polluting" the builds of arbitrary makefile targets with those. C'est la vie.

Updating Autosetup

Updating autosetup is, more often than not, painless. It requires having a checked-out copy of the autosetup git repository:

$ git clone https://github.com/msteveb/autosetup
$ cd autosetup
# Or, if it's already checked out:
$ git pull

Then, from the top-most directory of an SQLite checkout:

$ /path/to/autosetup-checkout/autosetup --install .
$ fossil status # show the modified files

Unless the upgrade made any incompatible changes (which is exceedingly rare), that's all there is to it. After that's done, apply a patch for the change described in the following section, test the configure process, and check it in.

Patching Autosetup for Project-local Changes

Autosetup reserves the flag name --debug for its own purposes, and its own special handling of --enable-... flags makes --debug an alias for --enable-debug. As this project has a long history of using --enable-debug, we patch autosetup to use the name --autosetup-debug in place of --debug. That requires (as of this writing) four small edits in , as demonstrated in check-in 3296c8d3.

If autosetup is upgraded and this patch is not applied the invoking ./configure will fail loudly because of the declaration of the debug flag in auto.def - duplicated flags are not permitted.

Branch-specific Customization

Certain vendor-specific branches require slight configure script customization. Rather than editing sqlite-config.tcl for this, which frequently leads to merge conflicts, the following approach is recommended:

In the vendor-specific branch, create a file named autosetup/sqlite-custom.tcl.

That file should contain the following content...

If flag customization is required, add:

proc sqlite-custom-flags {} {
  # If any existing --flags require different default values
  # then call:
  options-defaults {
    flag-name new-default-value
    ...
  }
  # ^^^ That will replace the default value but will not update
  # the --help text, which may lead to some confusion:
  # https://github.com/msteveb/autosetup/issues/77

  return {
   {*} {
     new-flag-name => {Help text}
     ...
   }
  }; #see below
}

That function must return either an empty string or a list in the form used internally by sqlite-config.tcl:sqlite-configure.

Next, define:

proc sqlite-custom-handle-flags {} {
  ... do any custom flag handling here ...
}

That function, if defined, will be called relatively late in the configure process, before any filtered files are generated but after all other significant processing.

---

1. But see this article for a detailed discussion of how autosetup currently deals specifically with CFLAGS: https://msteveb.github.io/autosetup/articles/handling-cflags/ ↩︎