Synopsis:
This page covers rust_textfinder_dirnav, which walks, selects, reads,
matches, and formats. It is the only component that touches file contents, and it writes
to no stream.
-
A struct generic over the
Output trait it declares and does not
implement, so the polymorphism is a compile-time bound rather than a runtime
dispatch.
-
Two shared borrows and one unique borrow are the whole of the access control this
component needs, and it is checked rather than documented.
-
Nothing in traversal panics - every fallible call yields an
io::Result
and every one is matched. No unwrap, no expect.
Four standard-library calls are rejected by name, each because the plausible call is
subtly wrong.
fs::metadata follows a link and would hide the case the specification requires be announced.
Path::extension returns nothing for .gitignore.
str::lines implements two of the three terminators, so a classic Mac OS file would arrive as one line.
- A directory-walking crate would own the descent this library must own.
Two details are worth reading for what the language decides rather than the specification.
str::from_utf8 supplies all four UTF-8 rejections, where C++ writes them out by hand for want of such a call.
- The file count is incremented at two call sites rather than inside
examine - the one place this implementation's shape forced a choice the other two did not face.
- The hand-written
Lines iterator is what lets a search stop reading at the first match.
1. Public Interface
rust_textfinder_dirnav walks a directory tree, reads each selected file,
evaluates the expression against each line, and formats every matching file into the block
Spec_TextFinder.md §3.4 fixes, emitting each of its lines as it is produced. It is the
only component that touches file contents, and it writes to no stream.
The crate root exports three things.
pub trait Output {
fn output(&mut self, text: &str);
}
pub type SkipList = Vec<String>;
pub struct Dirnav<'a, O: Output> { /* private */ }
impl<'a, O: Output> Dirnav<'a, O> {
pub fn new(out: &'a mut O, skips: &'a SkipList, commands: &'a ProgramCommands)
-> Result<Self, regex::Error>;
pub fn search(&mut self, root: &Path);
pub fn emit_run_summary(&mut self);
}
The struct declares the Output trait it does not implement, which is what puts
rust_textfinder_output downstream of it in the dependency chain. The generic
parameter makes the call direct, so the polymorphism is a compile-time bound rather than a
runtime dispatch.
new compiles commands.regex_text with
regex::Regex::new - the engine Spec_TextFinder.md §6.1 assigns to Rust -
and returns Err carrying the crate's own error when the expression will not
compile. The failure is a returned value rather than a panic, and the specification says why
in one sentence: a malformed pattern is something the user typed, so it is an ordinary
outcome of construction. rust_textfinder_cmdline guarantees the text is
non-empty, so new never sees an empty expression.
All three arguments are borrowed for the lifetime 'a and are owned by
rust_textfinder_entry, which the borrow checker obliges to keep them alive for
the lifetime of the Dirnav. The skip list and the commands are shared borrows
and cannot be modified through them; out is a unique borrow, since emitting a
line mutates the sink. That one distinction is the whole of the access control this
component needs, and it is checked rather than documented.
search returns nothing. Every failure it meets is announced through
Output, so the caller has nothing to report on its behalf. A single value is
reused across every root path, so the compiled expression is built once per run, and
search carries no state from one call to the next but for the two run counts of
Section 5.1, which accumulate across calls by design.
emit_run_summary writes the run summary Spec_TextFinder.md §3.6 requires.
It takes &mut self because emitting mutates the sink, takes no other
argument, and returns nothing; rust_textfinder_entry calls it once after the
last search returns. The split is deliberate: the counts belong here because
nothing else sees the entries, and the call belongs to the caller because nothing here knows
which root was the last. No accessor is exposed for either count - the line is the whole of
what they are for, and the unit suite reads them by reading that line through its own
Output.
2. Traversal
search resolves a root before walking it. A symbolic link, a kind that is
neither regular file nor directory, and a failed metadata query each draw
cannot open; a regular file is examined as a single file; a directory is
walked. The skip list is never consulted for the root, which Spec_TextFinder.md §3.2
exempts because the user named it explicitly.
The root's kind comes from std::fs::symlink_metadata and not
std::fs::metadata. The specification rejects the second by name: it follows a
symbolic link and would report the target's kind, hiding the very case §3.2 requires be
announced.
The recursion is written explicitly. walk iterates one directory and calls
itself on each subdirectory it decides to enter, so entries are handled as
std::fs::read_dir yields them, without being collected or reordered.
Two choices are stated in the specification rather than left to the implementer.
- No directory-walking crate.
walkdir and its kind own the
descent this library must own, since /s and the skip list both decide
whether to enter a directory, and such a crate may impose an order of its own.
read_dir enumerates a single level only. Its order is
unspecified by the standard library and is whatever readdir or
FindFirstFileW returns, which is the platform facility §3.2 requires
an implementation to enumerate through. Entries are neither sorted nor grouped, as
§3.2 forbids.
Every fallible call in the walk yields an io::Result and every one of them is
matched. read_dir failing announces the directory and returns; an entry yielded
as Err announces the directory and returns, so a directory that becomes
unreadable part way through is announced rather than silently truncated; a failed
file_type announces that entry and continues with the next. No
unwrap and no expect appear, which is what the specification means
when it says nothing in traversal panics.
The order of the entry tests matters, and §5 fixes it.
- Symbolic link, tested first. An entry that is a link is passed over
silently whatever its target, since no attempt is made to open it. The kind comes from
DirEntry::file_type, which does not follow a link;
std::fs::metadata is rejected by name here for the same reason it is
rejected for the root.
- Unrenderable name, tested second. An entry whose name is not valid
UTF-8 draws
cannot open and is neither entered nor searched.
OsStr::to_str reports the condition by returning None, so
nothing propagates and nothing is caught.
- Skip list, extension filter, and the open, after both.
Putting the name test before the skip-list test is what makes such a name announced whatever
/p holds, and Spec_TextFinder.md §3.4 writes that rule over every such
file rather than over the selected ones. This position is a correction: the Rust text
originally let the extension filter reach the entry first, which would have made the same
tree and the same command line produce different stdout from the two implementations. The
Process page covers how
that was found.
The link test staying ahead of the name test is a deliberate asymmetry all three implementations
carry: a link whose name cannot be rendered stays silent, because rule 1 turns on the
entry's kind, which costs no decoding, and §3.2 asks for silence there.
A pruned directory draws no announcement. Pruning is work the library chose not to do, and
cannot open reports work it could not do.
3. File Selection
Selection follows the /p rules of Spec_TextFinder.md §5, and the list
arrives from rust_textfinder_cmdline already normalized to bare extensions. An
empty list selects every file; a non-empty list selects a file whose extension matches an
entry.
std::path::Path::extension does not implement those rules, and the
specification rejects it by name: it returns None for .gitignore,
whereas §5 gives that file the extension gitignore. The extension is
therefore taken as the text after the last . in the file name, with
rfind('.') and a slice, no special case for a leading dot, and a name holding
no . at all has no extension.
Selection never meets a name it cannot read. The Section 2 gate has already refused every
entry whose name is not valid UTF-8, so selected takes a &str
rather than an OsStr and has no failure case of its own. Neither an empty nor a
non-empty /p list changes what happens to such a file.
Comparison goes through same_name, which is case-sensitive on POSIX and
case-insensitive on Windows per §3.2 and §5. It is written twice, once under
#[cfg(windows)] and once under #[cfg(not(windows))], and it is the
only place in the file where the target platform changes behavior. The same function serves
the skip list, since §3.2 and §5 fix one comparison for both, and the
specification names the function rather than restating the rule so that the two cannot
drift.
The Windows comparison uses eq_ignore_ascii_case, so it folds ASCII only. An
extension holding a non-ASCII character therefore compares case-sensitively on both
platforms, which the specification states as a consequence rather than leaving a reader to
discover it.
Selection applies uniformly: a root path that is a regular file is filtered by
/p like any other file. A root named on /P escapes the skip list
and does not escape the extension filter.
4. Admission and the No-Content Case
examine applies the three admission tests of Spec_TextFinder.md §3.3 in
the order the specification fixes. The size test reads the length
std::fs::Metadata::len reports, so a file above the limit is never read into
memory; the NUL and UTF-8 tests read bytes. The limit is 10,485,760 bytes, the number
§3.3 fixes, written 10_485_760 so that a reader can count the digits.
new records once, for the whole run, whether the no-content case applies:
let path_line_only =
commands.regex_text == "." && !commands.line_numbers && !commands.matched_line;
When it holds, a selected file that passes the size test and is not empty produces a block of
its path line alone, and the file is never opened. No file announcement accompanies that
block: searched reports a file that was read and matched nothing and
skipped reports one a content test rejected, and in this case neither happened,
so the library emits neither whatever /h says. A selected file of zero size
produces no block and draws no announcement either.
Both error announcements still work in this case, because both rest on metadata: a file
above the limit draws too large, and one whose metadata cannot be read draws
cannot open.
A file that needs reading is read in full with std::fs::read, so one failing a
later test is skipped entirely rather than searched in part. The NUL test is
bytes.contains(&0). The UTF-8 test is std::str::from_utf8,
which rejects truncated sequences, overlong encodings, encoded surrogates, and scalar values
above U+10FFFF - the four rejections §3.3 requires - so this implementation performs no
validation of its own and the standard library is the authority for what valid UTF-8 means.
The C++ implementation writes those four rejections out by hand, one line each, because it
has no such call to lean on.
A leading UTF-8 BOM is stripped with strip_prefix('\u{FEFF}') after the
admission tests, not before, so its three bytes count toward the size limit and toward the
NUL scan like any others.
5. Matching and Emission
Each line is evaluated with Regex::is_match, which gives the
anywhere-in-the-line match §3.3 requires and asks the engine for nothing more - no
match position, no matched substring, no capture group.
The line is passed as &str, so ., a character class, and a
class escape each match one Unicode scalar value. §6.1 records std::regex
over char as the one engine of the four that matches a byte instead, so on a
line holding a non-ASCII character this implementation agrees with the C# and Python
implementations and the C++ one is the outlier.
Three rules govern the emission, and all three are visible in a dozen lines of
examine:
- The path line goes out at the first match, ahead of the detail line
for that same match, and a
path_written flag makes sure it goes out once. A
file that never matches produces no line at all; a file that matches many produces its
path line once.
- With neither
/n nor /L, the loop returns as soon as
the path line is written. The block has no detail lines, so the first match
settles the file.
- Otherwise the loop runs to the end of the file, writing one detail
line - two spaces of indent, then the fields
/n and /L select
- as each matching line is evaluated.
Nothing is accumulated for the file. Output receives each line as it is
produced, which is the pipeline behavior §3.4's "as they occur" requires. Announcements
go through the same Output, at the point the failure or the finding is met, and
an error announcement is not gated on /h.
The /h gate is one function, file_announcement, and it reads as
the rule it implements: emit unless suppress_on_no_match. A file announcement
reports only a file that produced no block, so it never repeats a path the output already
carries - searched once the last line has been evaluated, which is the first
moment the library knows the file matched nothing, and skipped at the point of
rejection.
Path rendering is where this library does its own string work rather than the standard
library's. §3.4 requires / on every platform for the whole of
<path>, so a path is built by joining the root's own text with the entry
names descended through, and Path::display is rejected by name: it renders the
platform's separator, which on Windows would emit \ and make the same tree
produce different output on two platforms.
The root's own text is normalized too, since §3.4's rule covers all of
<path>. normalize replaces every \ with
/ before the first entry name is appended, so -P src\sub on
Windows yields src/sub/file.rs and not src\sub/file.rs. That was a
gap the audit found, and the integration suite now asserts it. A root path of .
contributes no leading ./, which walk arranges by taking an empty
prefix for that root and join by returning the name alone when the prefix is
empty.
One announcement carries a path that is not the path on disk. An entry name that is not
valid UTF-8 cannot be rendered as text, and §3.4 fixes the outcome: the file is not
searched and draws cannot open, naming it with U+FFFD substituted for each unit
that will not render. OsStr::to_string_lossy does exactly that, so this library
needs no substitution code. Every name that survives the Section 2 gate renders without
substitution, so a block's path line is always the name itself.
That is the one place the two implementations solve the same requirement with different
amounts of code. to_string_lossy is the whole of it here;
C++ writes a
renderPath helper that walks bytes on POSIX and UTF-16 units on Windows,
because the two calls it would otherwise use behave differently by platform and one of them
throws.
5.1 The Run Summary
Spec_TextFinder.md §3.6 puts the two run counts in this library, for every
implementation alike, and fixes the line they produce. Two usize fields hold
them, both zero from new, and neither is reset by search, so they
accumulate over every root the value is given.
- Directories are counted at the head of
walk, before
fs::read_dir, so a directory that cannot be enumerated is counted and
announced alike.
- Files are counted where the
/p test of Section 3 admits
one, at both of the two places that test is applied - the root-path arm of
search and the file arm of walk - and ahead of the metadata
call in each.
Putting the file increment at the two call sites rather than inside examine is
the one place this implementation's shape forced a choice the other two did not face.
walk announces cannot open and never calls examine
when entry.metadata() fails, and that file passed /p, so an
increment inside examine would miss it and this implementation would report one
file fewer than the others over the same tree. The specification says so in its own
§8.1, because the next reader's instinct will be to fold the two increments into one.
Everything §3.6 excludes is excluded by where those increments sit rather than by a
test of its own: an entry refused by /p, a symbolic link, an entry beneath a
pruned directory, an entry whose name is not valid UTF-8, and an entry that is neither a
file nor a directory all fail or bypass the /p test - though the last two draw
cannot open on the way past, which is why the file count can be smaller than
the number of announcements a run writes.
emit_run_summary composes the line with format! and sends it
through the same Output as every other line, in §3.6's fixed form and with
neither noun inflected:
accessed <files> files, <directories> directories
It is not gated on /h, which governs file announcements alone, and it names no
path, so this section's rendering rules do not reach it. Eight unit tests cover it, and the
one worth naming asserts accessed 1 files, 0 directories from a root that is a
regular file - the uninflected singular, asserted rather than left to be noticed.
6. Line Splitting Without str::lines
Spec_TextFinder.md §3.3 makes LF, CRLF, and a bare CR each terminate a line, and treats
a final unterminated run as a line. str::lines is rejected by name because it
implements two of those three: it splits on LF and strips a trailing CR, and it does not
treat a bare CR as a terminator, so a classic Mac OS file would arrive as one line.
The library carries a Lines iterator of its own, 30 lines including the
Iterator impl. It holds Option<&str> for the remainder,
finds the next \r or \n, consumes two bytes when the tail starts
with \r\n and one otherwise, and yields None when the remainder is
empty. Starting from None for empty input is what makes an empty file produce
no line rather than one empty line.
Writing it as an Iterator rather than as a function returning
Vec<&str> is what lets rule 2 of Section 5 stop reading at the first
match: the caller's for loop returns and the remaining bytes are never scanned.
Each item borrows the buffer rather than copying it, so a 10 MB file costs one allocation
for its contents and nothing per line.
enumerate supplies the line number, offset by one at the point of use. Line
numbers count every line, matching or not, which follows from numbering the iterator rather
than the matches.
7. Source
lib.rs in full - 261 lines, the largest file in the project. The comments cite
the Rust specification's sections, which cite Spec_TextFinder.md in turn.
Rust_Spec_driven_Dirnav/src/lib.rs
//! rust_textfinder_dirnav - directory navigation, matching, and block formatting.
//! Implements Spec_Rust_TextFinder_Dirnav.md, the Rust binding of Spec_TextFinder.md sections 3.2-3.4.
use regex::Regex;
use rust_textfinder_cmdline::ProgramCommands;
use std::fs::{self, Metadata};
use std::path::Path;
#[cfg(test)]
mod unit_tests;
pub trait Output {
fn output(&mut self, text: &str);
}
pub type SkipList = Vec<String>;
pub struct Dirnav<'a, O: Output> {
out: &'a mut O,
skips: &'a SkipList,
commands: &'a ProgramCommands,
expression: Regex,
path_line_only: bool,
files: usize,
directories: usize,
}
const SIZE_LIMIT: u64 = 10_485_760;
impl<'a, O: Output> Dirnav<'a, O> {
pub fn new(
out: &'a mut O,
skips: &'a SkipList,
commands: &'a ProgramCommands,
) -> Result<Self, regex::Error> {
let expression = Regex::new(&commands.regex_text)?;
let path_line_only =
commands.regex_text == "." && !commands.line_numbers && !commands.matched_line;
Ok(Dirnav { out, skips, commands, expression, path_line_only, files: 0, directories: 0 })
}
/// Section 8.1: the run summary of Spec_TextFinder.md section 3.6, written once
/// after the last root.
pub fn emit_run_summary(&mut self) {
let text = format!("accessed {} files, {} directories", self.files, self.directories);
self.out.output(&text);
}
pub fn search(&mut self, root: &Path) {
let display = normalize(&root.to_string_lossy());
let info = match fs::symlink_metadata(root) {
Ok(info) => info,
Err(_) => return self.announce("cannot open", &display),
};
let kind = info.file_type();
if kind.is_symlink() || !(kind.is_file() || kind.is_dir()) {
self.announce("cannot open", &display);
} else if kind.is_file() {
if self.selected(basename(&display)) {
self.files += 1; // section 8.1: after the /p test, ahead of every later outcome
self.examine(root, &display, &info);
}
} else {
self.walk(root, &display);
}
}
fn walk(&mut self, dir: &Path, display: &str) {
self.directories += 1; // section 8.1: counted before read_dir, so one that fails counts too
let prefix = if display == "." { "" } else { display };
let entries = match fs::read_dir(dir) {
Ok(entries) => entries,
Err(_) => return self.announce("cannot open", display),
};
for entry in entries {
let entry = match entry {
Ok(entry) => entry,
Err(_) => return self.announce("cannot open", display),
};
let raw = entry.file_name();
let kind = match entry.file_type() {
Ok(kind) => kind,
Err(_) => {
let child = join(prefix, &raw.to_string_lossy());
self.announce("cannot open", &child);
continue;
}
};
if kind.is_symlink() {
continue;
}
let name = match raw.to_str() {
Some(name) => name,
None => {
let child = join(prefix, &raw.to_string_lossy());
self.announce("cannot open", &child);
continue;
}
};
let child = join(prefix, name);
if kind.is_dir() {
if self.commands.recurse && !self.skips.iter().any(|skip| same_name(skip, name)) {
self.walk(&entry.path(), &child);
}
} else if kind.is_file() {
if self.selected(name) {
self.files += 1; // section 8.1
match entry.metadata() {
Ok(info) => self.examine(&entry.path(), &child, &info),
Err(_) => self.announce("cannot open", &child),
}
}
} else {
self.announce("cannot open", &child);
}
}
}
fn examine(&mut self, path: &Path, display: &str, info: &Metadata) {
if info.len() > SIZE_LIMIT {
return self.announce("too large", display);
}
if self.path_line_only {
if info.len() > 0 {
self.out.output(display);
}
return;
}
let bytes = match fs::read(path) {
Ok(bytes) => bytes,
Err(_) => return self.announce("cannot open", display),
};
if bytes.contains(&0) {
return self.file_announcement("skipped", display);
}
let text = match std::str::from_utf8(&bytes) {
Ok(text) => text,
Err(_) => return self.file_announcement("skipped", display),
};
let text = text.strip_prefix('\u{FEFF}').unwrap_or(text);
let details = self.commands.line_numbers || self.commands.matched_line;
let mut path_written = false;
for (number, line) in Lines::over(text).enumerate() {
if !self.expression.is_match(line) {
continue;
}
if !path_written {
self.out.output(display);
path_written = true;
}
if !details {
return;
}
let detail = self.detail(number + 1, line);
self.out.output(&detail);
}
if !path_written {
self.file_announcement("searched", display);
}
}
fn detail(&self, number: usize, line: &str) -> String {
let mut text = String::from(" ");
if self.commands.line_numbers {
text.push_str(&number.to_string());
if self.commands.matched_line {
text.push_str(" - ");
}
}
if self.commands.matched_line {
text.push_str(line);
}
text
}
fn selected(&self, name: &str) -> bool {
if self.commands.extensions.is_empty() {
return true;
}
match name.rfind('.') {
None => false,
Some(dot) => {
let extension = &name[dot + 1..];
self.commands.extensions.iter().any(|candidate| same_name(candidate, extension))
}
}
}
fn announce(&mut self, kind: &str, path: &str) {
self.out.output(&format!("{kind} {path}"));
}
fn file_announcement(&mut self, kind: &str, path: &str) {
if !self.commands.suppress_on_no_match {
self.announce(kind, path);
}
}
}
/// Spec_TextFinder.md section 3.4: every path is rendered with `/` on every platform.
fn normalize(text: &str) -> String {
text.replace('\\', "/")
}
fn join(prefix: &str, name: &str) -> String {
if prefix.is_empty() {
String::from(name)
} else if prefix.ends_with('/') {
format!("{prefix}{name}")
} else {
format!("{prefix}/{name}")
}
}
fn basename(display: &str) -> &str {
display.rsplit('/').next().unwrap_or(display)
}
#[cfg(windows)]
fn same_name(left: &str, right: &str) -> bool {
left.eq_ignore_ascii_case(right)
}
#[cfg(not(windows))]
fn same_name(left: &str, right: &str) -> bool {
left == right
}
/// Line splitting per Spec_TextFinder.md section 3.3: LF, CRLF, and bare CR terminate
/// a line, and a final unterminated run is a line. `str::lines` does neither.
struct Lines<'t> {
rest: Option<&'t str>,
}
impl<'t> Lines<'t> {
fn over(text: &'t str) -> Self {
Lines { rest: if text.is_empty() { None } else { Some(text) } }
}
}
impl<'t> Iterator for Lines<'t> {
type Item = &'t str;
fn next(&mut self) -> Option<&'t str> {
let rest = self.rest?;
match rest.find(|c| c == '\r' || c == '\n') {
None => {
self.rest = None;
Some(rest)
}
Some(at) => {
let (line, tail) = rest.split_at(at);
let consumed = if tail.starts_with("\r\n") { 2 } else { 1 };
let remainder = &tail[consumed..];
self.rest = if remainder.is_empty() { None } else { Some(remainder) };
Some(line)
}
}
}
}
Rust_Spec_driven_Dirnav/Cargo.toml
[package]
name = "rust_textfinder_dirnav"
version = "0.1.0"
edition = "2021"
rust-version = "1.70"
[lib]
path = "src/lib.rs"
[dependencies]
regex = "1"
rust_textfinder_cmdline = { path = "../Rust_Spec_driven_Cmdline" }
This manifest is the only one in the project that names a third-party crate. It does not
name rust_textfinder_output, which supplies the generic argument at the point
of use rather than at the point of definition, and that absence is the dependency direction
of Section 2 of the
Structure page stated
in a build file.
8. Prompt Records
This page carries none. Page_Structure.md §8 assigns it
Prompts_Spec_Rust_TextFinder_Dirnav.md and its Fix companion, and
neither was written: the Dirnav specification was produced in the same turn as the other
four documents, recorded at the project level. Three audit items reach this component - the
unnormalized root separators, the unnamed skip-list folding rule, and the gate's position -
and all three sit on the
Process page.