Spec-Driven

Spec-Driven Behavior: Traversal and File Selection

depth-first in filesystem order, 11 pruned names, and three admission tests

Synopsis:
This page covers how TextFinder reaches a file - the walk, and the gates an entry passes before it is searched.
  • Depth-first, in the order the platform's directory-reading facility yields. No sorting, no grouping of files ahead of directories.
  • An 11-name skip list prunes whole subtrees by directory basename. Pruning draws no announcement - it is work TextFinder chose not to do, not work it could not do.
  • Symbolic links are never opened. One met during traversal is passed over silently; one named as a root is announced.
One rule runs under several of these: an entry TextFinder finds on its own it may pass over in silence, and an entry the user typed it owes an explanation for.
  • That is why the skip list is not consulted for a root path, and why a root that is a link is announced.
  • /p matches the file's last dot-suffix, so .gitignore has extension gitignore - which is why the C++ spec rejects path::extension() by name.
  • Three admission tests in a fixed order: size from the filesystem, then no NUL byte, then valid UTF-8.
Two retreats are on the record, and Section 6 settles which gates the run counts.
  • Sorted sibling order gave way to filesystem order, and three places changed together so the document would not claim what it no longer delivered.
  • Read-then-test gave way to test-then-read, which had been reading a 200 MB file in order to refuse it.

1.  Depth-First in Filesystem Order

Traversal is depth-first: on reaching a directory that is not pruned, TextFinder descends into it and completes its subtree before returning to the next entry of the parent. Within a directory, entries are visited in the order the platform's directory-reading facility presents them, files and directories interleaved rather than grouped, so a directory is descended at the point it is reached in that order (§3.2). Two requirements fix that order, and §6's comparison of one implementation against another rests on both.
  1. An implementation enumerates a directory through the facility its platform provides - readdir on POSIX, FindFirstFileW and FindNextFileW on Windows, or whatever its own standard library wraps around them - and does not substitute a facility that imposes an order of its own.
  2. It visits entries in the order that facility yields. It does not sort them, and it does not group files ahead of directories or the reverse. Collecting a directory's entries before visiting any of them is permitted, since it changes no order; reordering them is not.
The visit order is therefore the filesystem's own, and it is the same order in every implementation on one machine against one tree. What that leaves guaranteed is narrower than sorted order would give, and §6 states the limit rather than claiming more: the total emission order is reproducible only across runs over the same tree on the same platform and filesystem. Elsewhere the match set still agrees and the order of matches from different directory entries may not. An earlier draft required ascending basename order, compared bytewise. The retreat to filesystem order was a deliberate trade, and the record states what each side buys. Filesystem order streams a level instead of buffering it and reads more cleanly, which counts for a teaching codebase. Sorted order buys cross-platform reproducibility, since NTFS returns uppercased-name index order while ext4 with dir_index returns hash order, and buys stability over time, since ext4 hash order shifts as entries are added and removed. Three places had to change together so the document would not claim what it no longer delivered: §3.2, §6, and the C++ traversal rule. Recursion is controlled by /s and is on by default. With /s false TextFinder searches the files directly within the root path and descends into none of its subdirectories. Multiple /P occurrences accumulate, and their subtrees are traversed in the order given.

2.  The Skip List

TextFinder maintains a list of directory names never entered during traversal. When a directory whose name appears in the list is met, its entire subtree is pruned. The default list holds 11 names, directories that typically carry version-control metadata or intermediate build output:
archive, .git, .svn, .hg, build, out, target, bin, obj, __pycache__, node_modules
A skip-list entry is compared against the directory's basename - the final component of its path - and matches when the two strings are equal. Matching is case-sensitive on POSIX and case-insensitive on Windows, following each platform's filesystem conventions. The list governs directory names only; file-name filtering by extension is /p. A pruned directory is not a failure and draws no announcement. That distinction matters: cannot open reports work TextFinder could not do, and pruning is work it chose not to do. The list governs directories met during traversal, never a root path the user supplied on /P. A root whose basename appears in the list is traversed like any other root, for the same reason a root that is a symbolic link is announced rather than passed over: the user named it explicitly, and returning nothing without a word would leave that unexplained. Pruning resumes immediately below the root, so a build directory found beneath a root named build is pruned as usual. That carve-out was removed once and restored. An intermediate version consulted the list for every directory including roots, so /P build was pruned before any entry was read; the current text exempts the root again. The C++ unit suite tests both halves - a root whose name is in the skip list is traversed, and the same directory reached during traversal is still pruned. Nothing on the command line and no configuration file extends the list. Extension happens at build time, through the one function §3.5 defines; the Contracts page covers it.

3.  Symbolic Links and Root Paths

Symbolic links are never opened. One met during traversal is passed over silently, since TextFinder makes no attempt to open it and so has no failure to report. A root path that is a symbolic link is announced with cannot open, because the user named it explicitly and it will not be searched. The asymmetry is the point rather than an oversight, and it follows the same rule the skip list follows: an entry TextFinder finds on its own it may pass over in silence, and an entry the user typed it owes an explanation for. A root path resolves three ways.
Root resolves to What happens
a directory Traversed, skip list not consulted for the root itself
a regular file Searched as that single file, filtered by /p like any other file
anything else Not searched, and reported with cannot open per §3.4
No root-path outcome affects the exit code. A run whose only root could not be opened announces that and exits 0, which is the §3.4 rule that an error announcement does not change the code. One more entry never gets as far as being selected. A name the implementation's string type cannot carry - bytes that are not valid UTF-8 on POSIX, unpaired surrogate code units on Windows - is announced with cannot open and skipped, and §3.4 places that test over every such entry rather than over the selected ones, so it fires whatever /p holds. It also runs after the symbolic-link test, so a link with such a name stays silent like any other link. The Overview page covers the announcement and the two costs §3.4 accepts for it. The path a block reports begins with the root path currently being traversed - the /P occurrence whose subtree holds the file, not the first /P given - and continues with the entry names descended through to reach it. The root's own text is part of the path, so a root of src yields src/foo.cpp rather than foo.cpp, and two roots holding files of the same name yield blocks that differ. A root of . is the one exception: it contributes no leading ./. Every path is rendered with / separators on every platform.

4.  Extension Rules

/p takes a quoted, comma-separated list of bare extensions. Four rules govern it, and the first is the one an implementer is likeliest to get wrong by reaching for a standard-library call.
  • An extension is the file's last dot-suffix, a leading dot on the name notwithstanding. .gitignore has extension gitignore, so a dot-file is searched like any other file and is excluded only by /p or by the skip list.
  • Each item is normalized. Trimmed of surrounding whitespace, then stripped of one leading dot if present, so cpp and .cpp are equivalent. Empty items are discarded, so "cpp,,rs" and "cpp, rs" name the same two extensions.
  • Comparison follows the platform, case-sensitive on POSIX and case-insensitive on Windows, as skip-list entries do.
  • The empty list and the non-empty list differ on extensionless files. An empty list searches every file, files with no extension included. A non-empty list excludes them.
The dot-file rule is why the C++ specification rejects std::filesystem::path::extension() by name: that function returns an empty string for .gitignore, where §5 gives it the extension gitignore. The warning is kept in the specification deliberately, because the wrong call is the plausible one.

5.  The Three Admission Tests

A selected file is tested before it is searched, and the order of the tests is part of the specification rather than an implementation detail.
  1. Size. Taken from the filesystem, it must not exceed 10 MB (10,485,760 bytes). A file above the limit is never read, and draws the error announcement too large.
  2. No NUL byte. This is the binary-file test of §7. A file within the size limit is read in full - so that one failing a later test is skipped entirely rather than searched in part - and rejected if its content holds a NUL byte.
  3. Valid UTF-8. Truncated sequences, overlong encodings, encoded surrogates, and scalar values above U+10FFFF all fail.
A file that fails test 2 or test 3 draws skipped, a file announcement, so /h can hide it. A file that fails test 1 draws too large, an error announcement, which /h cannot hide. The NUL test is there because UTF-8 validity alone does not serve: a UTF-16 file of ASCII text is valid UTF-8, so validation would admit it and the search would run fruitlessly over text the specification says not to search. The reason is stated in §3.3 rather than left implicit, and it survived the trimming passes for that reason. Test 1 reads metadata and tests 2 and 3 read bytes, and the ordering follows from that. An earlier draft said TextFinder reads the file in full and then admits it if it passes three tests, the first being size, which would have read a 200 MB file in order to refuse it. The current text is test-then-read: the filesystem-reported size is checked first, and the read-in-full justification attaches only to the NUL and UTF-8 tests. A leading UTF-8 BOM is consumed and does not belong to the first line. One case skips tests 2 and 3 entirely, and the Matching page covers it: with the default expression and neither /n nor /L, no file content is needed, so no file is opened and the size test runs alone.

6.  Which of These Gates Counts

§3.6 closes every traversing run with a line counting the files and directories it reached, and the gates this page describes are what decide the two numbers. The rule is one line each. A file counts when it passed the /p filter of Section 4 and TextFinder went on to take its size for test 1. A directory counts when TextFinder reads its entries or tries to. Put that against the sections above and the counts fall out without a second rule.
  • A file refused by /p is not counted. TextFinder learned its name and nothing more.
  • A symbolic link met during traversal is not counted, being passed over silently without an attempt to open it (Section 3).
  • Nothing beneath a directory the skip list pruned is counted, traversal never having reached it - and the pruned directory is not counted either, pruning meaning it was never entered (Section 2).
  • A root path the skip list would have pruned is counted, since Section 2 exempts roots from the list: the user named it explicitly.
  • Under /s false the root counts and no subdirectory does.
  • A file that passed /p and then failed test 1 is counted, the count standing ahead of the test. So is one whose size could not be read at all, which draws cannot open.
  • An entry refused before selection is not counted even though it draws cannot open: a name that will not render, which Section 3 announces whatever /p holds, and an entry that is neither a regular file nor a directory.
The last two items are the ones worth reading twice, because together they say the counts are of work attempted rather than of announcements written. A run can print more cannot open lines than its file count admits to, and that is correct: the two entries in the second item never reached the tests this page is about. The no-content case counts the same way it selects. It opens no file, yet every file it selects is counted, including a zero-length one that produced neither a block nor an announcement - the size test still ran on metadata, which is the whole of what the count requires.