* Add a FAT filesystem for SD cards to Arduino FS Arduino forked a copy of SD lib several years ago, put their own wrapper around it, and it's been languishing in our ESP8266 libraries ever since as SD. It doesn't support long file names, has class names which conflict with the ESP8266 internal names, and hasn't been updated in ages. The original author of the SD library has continued work in the meantime, and produced a very feature rich implementation of SdFat. It unfortunately also conflicts with the class names we use in ESP8266 Arduino and has a different API than the internal SPIFFS or proposed LittleFS filesystem objects. This PR puts a wrapper around the latest and greatest SdFat library, by forking it and wrapping its classes in a private namespace "sdfat," and making as thin a wrapper as possible around it to conform to the ESP8266 FS, File, and Dir classes. This PR also removes the Arduino SD.h class library and rewrites it using the new SDFS filesystem to make everything in the ESP8266 Arduino core compatible with each other. By doing so it lets us use a single interface for anything needing a file instead of multiple ones (see SDWebServer and how a different object is needed vs. one serving from SPIFFS even though the logic is all the same). Same for BearSSL's CertStores and probably a few others I've missed, cleaning up our code base significantly. Like LittleFS, silently create directories when a file is created with a subdirectory specifier ("/path/to/file.txt") if they do not yet exist. Adds a blacklist of sketches to skip in the CI process (because SdFat has many examples which do not build properly on the ESP8266). Now that LittleFS and SDFS have directory support, the FS needs to be able to communicate whether a name is one or the other. Add a simple bool FS::isDirectory() and bool FS::isFile() method. SPIFFS doesn't have directories, so if it's valid it's a file and reported as such. Add ::mkdir/::rmdir to the FS class to allow users to make and destroy subdirectories. SPIFFS directory operations will, of course, fail and return false. Emulate a 16MB SD card and allow test runner to exercise it by using a custom SdFat HOST_MOCK-enabled object. Throw out the original Arduino SD.h class and rewrite from scratch using only the ESP8266 native SDFS calls. This makes "SD" based applications compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations. The only major visible change for users is that long filenames now are fully supported and work without any code changes. If there are static arrays of 11 bytes for old 8.3 names in code, they will need to be adjusted. While it is recommended to use the more powerful SDFS class to access SD cards, this SD.h wrapper allows for use of existing Arduino libraries which are built to only with with that SD class. Additional helper functions added to ESP8266 native Filesystem:: classes to help support this portability. The rewrite is good enough to run the original SDWebServer and SD example code without any changes. * Add a FSConfig and SDFSConfig param to FS.begin() Allows for configuration values to be passed into a filesystem via the begin method. By default, a FS will receive a nullptr and should so whatever is appropriate. The base FSConfig class has one parameter, _autoFormat, set by the default constructor to true. For SPIFFS, you can now disable auto formatting on mount failure by passing in a FSConfig(false) object. For SDFS a SDFSConfig parameter can be passed into config specifying the chip select and SPI configuration. If nothing is passed in, the begin will fail since there are no safe default values here. * Add FS::setConfig to set FS-specific options Add a new call, FS::setConfig(const {SDFS,SPIFFS}Config *cfg), which takes a FS-specific configuration object and copies any special settings on a per-FS basis. The call is only valid on unmounted filesystems, and checks the type of object passed in matches the FS being configured. Updates the docs and tests to utilize this new configuration method. * Add ::truncate to File interface Fixes #3846 * Use polledTimeout for formatting yields, cleanup Use the new polledTimeout class to ensure a yield every 5ms while formatting. Add in default case handling and some debug messages when invalid inputs specified. * Make setConfig take const& ref, cleaner code setConfig now can take a parameter defined directly in the call by using a const &ref to it, leading to one less line of code to write and cleaner reading of the code. Also clean up SDFS implementation pointer definition.
13 KiB
Filesystem
Flash layout
Even though file system is stored on the same flash chip as the program, programming new sketch will not modify file system contents. This allows to use file system to store sketch data, configuration files, or content for Web server.
The following diagram illustrates flash layout used in Arduino environment:
|--------------|-------|---------------|--|--|--|--|--|
^ ^ ^ ^ ^
Sketch OTA update File system EEPROM WiFi config (SDK)
File system size depends on the flash chip size. Depending on the board which is selected in IDE, you have the following options for flash size:
Board | Flash chip size, bytes | File system size, bytes |
---|---|---|
Generic module | 512k | 64k, 128k |
Generic module | 1M | 64k, 128k, 256k, 512k |
Generic module | 2M | 1M |
Generic module | 4M | 1M, 2M, 3M |
Adafruit HUZZAH | 4M | 1M, 2M, 3M |
ESPresso Lite 1.0 | 4M | 1M, 2M, 3M |
ESPresso Lite 2.0 | 4M | 1M, 2M, 3M |
NodeMCU 0.9 | 4M | 1M, 2M, 3M |
NodeMCU 1.0 | 4M | 1M, 2M, 3M |
Olimex MOD-WIFI-ESP8266(-DEV) | 2M | 1M |
SparkFun Thing | 512k | 64k |
SweetPea ESP-210 | 4M | 1M, 2M, 3M |
WeMos D1 R1, R2 & mini | 4M | 1M, 2M, 3M |
ESPDuino | 4M | 1M, 2M, 3M |
WiFiduino | 4M | 1M, 2M, 3M |
Note: to use any of file system functions in the sketch, add the following include to the sketch:
#include "FS.h"
File system limitations
The filesystem implementation for ESP8266 had to accomodate the constraints of the chip, among which its limited RAM. SPIFFS was selected because it is designed for small systems, but that comes at the cost of some simplifications and limitations.
First, behind the scenes, SPIFFS does not support directories, it
just stores a "flat" list of files. But contrary to traditional
filesystems, the slash character '/'
is allowed in
filenames, so the functions that deal with directory listing (e.g.
openDir("/website")
) basically just filter the filenames
and keep the ones that start with the requested prefix
(/website/
). Practically speaking, that makes little
difference though.
Second, there is a limit of 32 chars in total for filenames. One
'\0'
char is reserved for C string termination, so that
leaves us with 31 usable characters.
Combined, that means it is advised to keep filenames short and not
use deeply nested directories, as the full path of each file (including
directories, '/'
characters, base name, dot and extension)
has to be 31 chars at a maximum. For example, the filename
/website/images/bird_thumbnail.jpg
is 34 chars and will
cause some problems if used, for example in exists()
or in
case another file starts with the same first 31 characters.
Warning: That limit is easily reached and if ignored, problems might go unnoticed because no error message will appear at compilation nor runtime.
For more details on the internals of SPIFFS implementation, see the SPIFFS readme file.
Uploading files to file system
ESP8266FS is a tool which integrates into the Arduino IDE. It adds a menu item to Tools menu for uploading the contents of sketch data directory into ESP8266 flash file system.
- Download the tool: https://github.com/esp8266/arduino-esp8266fs-plugin/releases/download/0.3.0/ESP8266FS-0.3.0.zip.
- In your Arduino sketchbook directory, create
tools
directory if it doesn't exist yet - Unpack the tool into
tools
directory (the path will look like<home_dir>/Arduino/tools/ESP8266FS/tool/esp8266fs.jar
) - Restart Arduino IDE
- Open a sketch (or create a new one and save it)
- Go to sketch directory (choose Sketch > Show Sketch Folder)
- Create a directory named
data
and any files you want in the file system there - Make sure you have selected a board, port, and closed Serial Monitor
- Select Tools > ESP8266 Sketch Data Upload. This should start
uploading the files into ESP8266 flash file system. When done, IDE
status bar will display
SPIFFS Image Uploaded
message.
File system object (SPIFFS)
setConfig
;
SPIFFSConfig cfg.setAutoFormat(false);
cfg.setConfig(cfg); SPIFFS
This method allows you to configure the parameters of a filesystem
before mounting. All filesystems have their own *Config
(i.e. SDFSConfig
or SPIFFSConfig
with their
custom set of options. All filesystems allow explicitly
enabling/disabling formatting when mounts fail. If you do not call this
setConfig
method before perforing begin()
, you
will get the filesystem's default behavior and configuration. By
default, SPIFFS will autoformat the filesystem if it cannot mount it,
while SDFS will not.
begin
.begin() SPIFFS
This method mounts SPIFFS file system. It must be called before any other FS APIs are used. Returns true if file system was mounted successfully, false otherwise. With no options it will format SPIFFS if it is unable to mount it on the first try.
end
.end() SPIFFS
This method unmounts SPIFFS file system. Use this method before updating SPIFFS using OTA.
format
.format() SPIFFS
Formats the file system. May be called either before or after calling
begin
. Returns true if formatting was
successful.
open
.open(path, mode) SPIFFS
Opens a file. path
should be an absolute path starting
with a slash (e.g. /dir/filename.txt
). mode
is
a string specifying access mode. It can be one of "r", "w", "a", "r+",
"w+", "a+". Meaning of these modes is the same as for fopen
C function.
r Open text file for reading. The stream is positioned at the
beginning of the file.
r+ Open for reading and writing. The stream is positioned at the
beginning of the file.
w Truncate file to zero length or create text file for writing.
The stream is positioned at the beginning of the file.
w+ Open for reading and writing. The file is created if it does
not exist, otherwise it is truncated. The stream is
positioned at the beginning of the file.
a Open for appending (writing at end of file). The file is
created if it does not exist. The stream is positioned at the
end of the file.
a+ Open for reading and appending (writing at end of file). The
file is created if it does not exist. The initial file
position for reading is at the beginning of the file, but
output is always appended to the end of the file.
Returns File object. To check whether the file was opened successfully, use the boolean operator.
= SPIFFS.open("/f.txt", "w");
File f if (!f) {
.println("file open failed");
Serial}
exists
.exists(path) SPIFFS
Returns true if a file with given path exists, false otherwise.
openDir
.openDir(path) SPIFFS
Opens a directory given its absolute path. Returns a Dir object.
remove
.remove(path) SPIFFS
Deletes the file given its absolute path. Returns true if file was deleted successfully.
rename
.rename(pathFrom, pathTo) SPIFFS
Renames file from pathFrom
to pathTo
. Paths
must be absolute. Returns true if file was renamed
successfully.
info
;
FSInfo fs_info.info(fs_info); SPIFFS
Fills FSInfo
structure with information about the file system. Returns
true
is successful, false
otherwise.
Filesystem information structure
struct FSInfo {
size_t totalBytes;
size_t usedBytes;
size_t blockSize;
size_t pageSize;
size_t maxOpenFiles;
size_t maxPathLength;
};
This is the structure which may be filled using FS::info method.
-totalBytes
— total size of useful data on the file system
-usedBytes
— number of bytes used by files -
blockSize
— SPIFFS block size - pageSize
—
SPIFFS logical page size - maxOpenFiles
— max number of
files which may be open simultaneously -maxPathLength
— max
file name length (including one byte for zero termination)
Directory object (Dir)
The purpose of Dir object is to iterate over files inside a
directory. It provides the methods: next()
,
fileName()
, fileSize()
, and
openFile(mode)
.
The following example shows how it should be used:
= SPIFFS.openDir("/data");
Dir dir while (dir.next()) {
.print(dir.fileName());
Serialif(dir.fileSize()) {
= dir.openFile("r");
File f .println(f.size());
Serial}
}
next
Returns true while there are files in the directory to iterate over.
It must be called before calling fileName()
,
fileSize()
, and openFile()
functions.
fileName
Returns the name of the current file pointed to by the internal iterator.
fileSize
Returns the size of the current file pointed to by the internal iterator.
openFile
This method takes mode argument which has the same meaning
as for SPIFFS.open()
function.
File object
SPIFFS.open()
and dir.openFile()
functions
return a File object. This object supports all the functions of
Stream, so you can use readBytes
,
findUntil
, parseInt
, println
, and
all other Stream methods.
There are also some functions which are specific to File object.
seek
.seek(offset, mode) file
This function behaves like fseek
C function. Depending
on the value of mode
, it moves current position in a file
as follows:
- if
mode
isSeekSet
, position is set tooffset
bytes from the beginning. - if
mode
isSeekCur
, current position is moved byoffset
bytes. - if
mode
isSeekEnd
, position is set tooffset
bytes from the end of the file.
Returns true if position was set successfully.
position
.position() file
Returns the current position inside the file, in bytes.
size
.size() file
Returns file size, in bytes.
name
= file.name(); String name
Returns file name, as const char*
. Convert it to
String for storage.
close
.close() file
Close the file. No other operations should be performed on
File object after close
function was called.