← In Class CS 326 · Week 05 · Code and output Space/j/k sections · click to pin · n/p example · i edit · l copy link

Week 05 · Streams of bytes: cat, wc, and grep

Every program in examples/src/bin/ beside the output it printed, and every file in examples/broken/ beside what rustc said about it. Each colored, numbered section of code sits in the same row as the output it produced. Click an output line to light up the println! that printed it, or click a println! to find its output. Click a section's number to copy a link straight to that section, to paste where students will follow it. Press i on a program to edit it and run it. On a phone the two columns do not fit, so a Code/Output switch shows one at a time.

The programs

ProgramThe one ideaThe line to point atSections
01fd_numbersa descriptor is the lowest free slotthe open after close(4) returning 4, not 65
02short_readsa short read is normal; only 0 ends itbuf[3..6] still saying "lo\n"5
03buffer_size_counts_callsthe buffer's size sets how many times you trap2,048 reads beside 1,048,576, and 8,192 on rv6 either way4
04read_loop_trlisten to both counts, read's and write'swrite_all(dst, &buf[..n])? beside the bare write that lost 10 bytes5
05exit_statusa later success does not erase a failurestatus = 1; // only ever set, never cleared6
06bytes_chars_strlen counts bytes, not characters"año".len() = 4 beside .chars().count() = 37
07state_across_chunksstate the caller keeps survives the end of a readafter_cr: &mut bool, and the two <- wrong rows5
08lines_in_a_fixed_bufferan unfinished line slides to the frontself.buf.copy_within(self.start..self.len, 0), then read(fd, &mut buf[6..]) -> 105
09while_letcall, match, and stop at the first misswhile let Some(line) = lines.next_line(), and Some("") for a blank line5
10fencepoststhe last start is len - k, and only ..= reaches it(0..3).contains(&3) = false for log in syslog6
11stop_earlywhat a program does not read costs nothing1,024 bytes read out of 10,000,000,0004

Programs that must not compile

ErrorWhat rustc saysThe fix
E0277Lines<'_> is not an iteratorwhile let Some(line) = lines.next_line()
E0308mismatched types expected u8, found charb'\t', a byte literal
E0499cannot borrow lines as mutable more than once at a timefinish with each line first, or copy out what you need
E0502cannot borrow buf[_] as immutable because it is also borrowed as mutableread through the line, or after the last use of lines
E0506cannot assign to buf[_] because it is borrowededit a copy, or write after the last use of lines
E0599no method named chars found for reference &[u8] in the current scopestay in bytes, or from_utf8 first

Keys

Space j ↓ →next section (dims the rest)
Shift+Space k ↑ ←previous section
0–9jump to that section
clickpin an output line and the code that printed it (click again to unpin)
click a numbercopy a link to that section, or to the whole example from the number beside its title
lcopy a link to the pinned line, else the section in focus, else this example
Escunpin, then leave section focus, and drop it from the link so a reload does not bring it back
n pnext or previous example
rnext run, for programs run more than once
eopen or close the explanation (broken files) or the build warnings
+ -larger or smaller text (remembered)
iedit this program and run it
⌘/Ctrl+↵run what you have edited
ccompare: what you changed in the code, and what changed in the output (press Esc first if you are typing)
oon a narrow screen, switch between the code and the output column

01 A descriptor is the lowest free slot in a table.

a descriptor is the lowest free slot · point at the open after close(4) returning 4, not 6

$ cargo run --bin 01_fd_numbers
//! 01 — A descriptor is the lowest free slot in a table.▸ 17 more lines
//!
//! `open` does not hand back a name or a pointer. It hands back an index into
//! a per-process table in the kernel, and it always picks the LOWEST free
//! slot. Three slots are taken before `main` runs:
//!
//! fd 0 1 2 3 4 5
//! [stdin ][stdout][stderr][ ][ ][ ] ...
//!
//! open -> 3 open -> 4 open -> 5
//! close(4) slot 4 is free again
//! open -> 4 not 6: the lowest free slot wins
//!
//! These are REAL descriptors from the real kernel, the one running this
//! program. `File::open` makes the `open` system call, `as_raw_fd()` shows
//! the number it returned, and dropping the `File` makes the `close`.
//!
//! Run: cargo run --bin 01_fd_numbers

use std::fs::File;
use std::io::{Read, Write};
use std::os::fd::AsRawFd;

/// rv6 gives each process 16 slots: fds 0 through 15.
const NOFILE: i32 = 16;

fn main() {
1 println!("== three are open before main runs ==");
println!("stdin is fd {}", std::io::stdin().as_raw_fd());
println!("stdout is fd {}", std::io::stdout().as_raw_fd());
println!("stderr is fd {}", std::io::stderr().as_raw_fd());
println!("nobody in this program opened them. They came with the process.");

1== three are open before main runs ==
stdin is fd 0
stdout is fd 1
stderr is fd 2
nobody in this program opened them. They came with the process.
2 println!("\n== open spends the lowest free slot ==");
let mut a = File::open("/dev/null").expect("/dev/null is always there");
let b = File::open("/dev/null").expect("/dev/null is always there");
let c = File::open("/dev/null").expect("/dev/null is always there");
let (fa, fb, fc) = (a.as_raw_fd(), b.as_raw_fd(), c.as_raw_fd());
println!("open(\"/dev/null\") -> {fa}");
println!("open(\"/dev/null\") -> {fb}");
println!("open(\"/dev/null\") -> {fc}");
println!("the same file three times, three different slots");

2== open spends the lowest free slot ==
open("/dev/null") -> 3
open("/dev/null") -> 4
open("/dev/null") -> 5
the same file three times, three different slots
3 println!("\n== close refunds the slot, and the next open takes it ==");
drop(b); // File's Drop is the close system call
println!("close({fb})");
let d = File::open("/dev/null").expect("/dev/null is always there");
let fd = d.as_raw_fd();
println!("open(\"/dev/null\") -> {fd}");
println!("{fd} again, not {}: the slot close gave back is the lowest free one", fc + 1);

3== close refunds the slot, and the next open takes it ==
close(4)
open("/dev/null") -> 4
4 again, not 6: the slot close gave back is the lowest free one
4 println!("\n== a leak: slots nobody refunds ==");
let mut held = Vec::new();
while let Ok(f) = File::open("/dev/null") {
let n = f.as_raw_fd();
held.push(f);
if n >= NOFILE - 1 {
break;
}
}
let last = held.last().map_or(-1, |f| f.as_raw_fd());
println!("opened {} more without closing any; the last one is fd {last}", held.len());
println!("on rv6 that is all {NOFILE} slots. The next open fails, although /dev/null");
println!("is right there. Nothing failed where the close was forgotten.");
let leaked = held.len();
drop(held); // closes every one of them
let e = File::open("/dev/null").expect("/dev/null is always there");
println!("close those {leaked}, then open(\"/dev/null\") -> {}", e.as_raw_fd());

4== a leak: slots nobody refunds ==
opened 10 more without closing any; the last one is fd 15
on rv6 that is all 16 slots. The next open fails, although /dev/null
is right there. Nothing failed where the close was forgotten.
close those 10, then open("/dev/null") -> 6
5 println!("\n== the number says nothing about what is behind it ==");
let mut buf = [0u8; 64];
println!("read(fd {fa}, 64 bytes) -> {:?} /dev/null is always at its end",
a.read(&mut buf));
let mut w = File::options().write(true).open("/dev/null").expect("writable");
println!("write(fd {}, 13 bytes) -> {:?} and swallows every write whole",
w.as_raw_fd(), w.write(b"into the void"));
println!("a file, a terminal, a pipe, /dev/null: all of them are read(fd, buf).");
5== the number says nothing about what is behind it ==
read(fd 3, 64 bytes) -> Ok(0) /dev/null is always at its end
write(fd 7, 13 bytes) -> Ok(13) and swallows every write whole
a file, a terminal, a pipe, /dev/null: all of them are read(fd, buf).
}
exit status 0

02 A short read is normal. Only 0 means the end.

a short read is normal; only 0 ends it · point at buf[3..6] still saying "lo\n"

$ cargo run --bin 02_short_reads
//! 02 — A short read is normal. Only 0 means the end.▸ 15 more lines
//!
//! `read(fd, buf)` is a request with a ceiling. It puts SOME bytes in `buf`,
//! anywhere from 0 to `buf.len()`, and returns how many. A terminal hands
//! over only what has been typed, one line per Enter:
//!
//! you type hello, Enter read -> 6 buf[..6] is "hello\n"
//! you type ok, Enter read -> 3 buf[..3] is "ok\n"; buf[3..6] still says "lo\n"
//! you press Ctrl-D read -> 0 end of input: three reads, two lines
//!
//! `Terminal` below replays what somebody typed, the way the real one
//! answers `cat` with no arguments. The buffer is 512 bytes and is never
//! full. 6 is not the end, 3 is not the end, and the bytes past 3 belong to
//! the line before.
//!
//! Run: cargo run --bin 02_short_reads

/// What a terminal does with `read`: one typed line per call, never more
/// than fits, the rest of a long line on the next call, and 0 after Ctrl-D.
struct Terminal {
typed: &'static [&'static [u8]], // lines still to come, each with its \n
pending: &'static [u8], // the part of a line that did not fit yet
}

impl Terminal {
fn new(typed: &'static [&'static [u8]]) -> Terminal {
Terminal { typed, pending: b"" }
}

fn read(&mut self, buf: &mut [u8]) -> usize {
if self.pending.is_empty() {
match self.typed.split_first() {
Some((line, rest)) => {
self.pending = line;
self.typed = rest;
}
None => return 0, // Ctrl-D: end of input
}
}
let n = self.pending.len().min(buf.len());
buf[..n].copy_from_slice(&self.pending[..n]);
self.pending = &self.pending[n..];
n
}
}

/// Bytes as Rust would write them in a string literal: "ok\n".
fn shown(bytes: &[u8]) -> String {
format!("{:?}", String::from_utf8_lossy(bytes))
}

/// A 1,100-byte file: everything was "typed" before the first read.
static FILE: [&[u8]; 1] = [&[b'x'; 1100]];

fn main() {
1 println!("== three reads, two lines ==");
let mut term = Terminal::new(&[b"hello\n", b"ok\n"]);
let mut buf = [0u8; 512];
loop {
let n = term.read(&mut buf);
println!("read -> {n} buf[..{n}] is {}", shown(&buf[..n]));
if n == 0 {
break;
}
}
println!("6 and 3 are both short of 512, and neither one is the end. 0 is.");

1== three reads, two lines ==
read -> 6 buf[..6] is "hello\n"
read -> 3 buf[..3] is "ok\n"
read -> 0 buf[..0] is ""
6 and 3 are both short of 512, and neither one is the end. 0 is.
2 println!("\n== the bytes past n are left over ==");
println!("buf[..3] is {:<12} the second read", shown(&buf[..3]));
println!("buf[3..6] is {:<12} the first read's tail: nothing cleared it", shown(&buf[3..6]));
println!("buf[..6] is {:<12} what a reader that ignores n passes on", shown(&buf[..6]));
println!("`lo` was never typed on the second line. Only n says which bytes are real.");

2== the bytes past n are left over ==
buf[..3] is "ok\n" the second read
buf[3..6] is "lo\n" the first read's tail: nothing cleared it
buf[..6] is "ok\nlo\n" what a reader that ignores n passes on
`lo` was never typed on the second line. Only n says which bytes are real.
3 println!("\n== the mistake: treat a short read as the end ==");
let mut term = Terminal::new(&[b"hello\n", b"ok\n"]);
let mut buf = [0u8; 512];
let mut reads = 0;
loop {
let n = term.read(&mut buf);
reads += 1;
println!("read -> {n} got {}", shown(&buf[..n]));
if n < buf.len() {
break; // WRONG: short is not the end
}
}
println!("{reads} read, then it quit. `ok` was typed and never read.");

3== the mistake: treat a short read as the end ==
read -> 6 got "hello\n"
1 read, then it quit. `ok` was typed and never read.
4 println!("\n== a buffer smaller than the line ==");
let mut term = Terminal::new(&[b"hello\n", b"ok\n"]);
let mut small = [0u8; 4];
loop {
let n = term.read(&mut small);
println!("read -> {n} small[..{n}] is {}", shown(&small[..n]));
if n == 0 {
break;
}
}
println!("the terminal kept `o\\n` for the next read. A full buffer is not a line either.");

4== a buffer smaller than the line ==
read -> 4 small[..4] is "hell"
read -> 2 small[..2] is "o\n"
read -> 3 small[..3] is "ok\n"
read -> 0 small[..0] is ""
the terminal kept `o\n` for the next read. A full buffer is not a line either.
5 println!("\n== a file ends with a short read too ==");
let mut file = Terminal::new(&FILE);
let mut buf = [0u8; 512];
let mut total = 0;
loop {
let n = file.read(&mut buf);
println!("read -> {n}");
if n == 0 {
break;
}
total += n;
}
let last = total % buf.len();
println!("{total} bytes. The read that got the last {last} said {last}, not 0.");
println!("Only the read after it, the one that returned 0, said the file was over.");
5== a file ends with a short read too ==
read -> 512
read -> 512
read -> 76
read -> 0
1100 bytes. The read that got the last 76 said 76, not 0.
Only the read after it, the one that returned 0, said the file was over.
}
exit status 0

03 The buffer's size sets how many times you trap.

the buffer's size sets how many times you trap · point at 2,048 reads beside 1,048,576, and 8,192 on rv6 either way

$ cargo run --bin 03_buffer_size_counts_calls
//! 03 — The buffer's size sets how many times you trap.▸ 15 more lines
//!
//! Every `read` is a trip into the kernel: a trap, a table lookup, a copy,
//! a return. That fixed cost does not shrink when the request does, so what
//! a bigger buffer buys is FEWER CALLS:
//!
//! copying 1 MiB (1,048,576 bytes), not counting the final read of 0
//! 1-byte buffer 1,048,576 reads
//! 512-byte buffer 2,048 reads 0.2% as many
//! 4 KiB buffer 256 reads and your whole stack
//!
//! `Source` below is a 1 MiB file that fills the buffer whenever it can, the
//! way the host does, and counts every call. Then it is a file on rv6, whose
//! kernel never returns more than 128 bytes per read, whatever you ask for.
//!
//! Run: cargo run --bin 03_buffer_size_counts_calls

const MIB: usize = 1024 * 1024;

/// A file of `left` bytes. Each read returns at most `cap` of them.
struct Source {
left: usize,
cap: usize,
calls: usize,
}

impl Source {
fn new(size: usize, cap: usize) -> Source {
Source { left: size, cap, calls: 0 }
}

fn read(&mut self, buf: &mut [u8]) -> usize {
self.calls += 1;
let n = self.left.min(buf.len()).min(self.cap);
buf[..n].fill(b'x');
self.left -= n;
n
}
}

/// Read to the end through `buf`. Returns (reads that returned data, all calls).
fn drain(src: &mut Source, buf: &mut [u8]) -> (usize, usize) {
let mut data = 0;
loop {
if src.read(buf) == 0 {
return (data, src.calls);
}
data += 1;
}
}

/// 1048576 -> "1,048,576"
fn commas(n: usize) -> String {
let s = n.to_string();
let mut out = String::new();
for (i, ch) in s.chars().enumerate() {
if i > 0 && (s.len() - i) % 3 == 0 {
out.push(',');
}
out.push(ch);
}
out
}

fn main() {
// One array. A buffer of B bytes is its first B bytes.
let mut big = [0u8; 65536];

1 println!("== 1 MiB through three buffers ==");
let mut counts = Vec::new();
for size in [1, 512, 4096] {
let (data, calls) = drain(&mut Source::new(MIB, usize::MAX), &mut big[..size]);
println!("{:>5}-byte buffer {:>9} reads, then the 0: {} calls",
size, commas(data), commas(calls));
counts.push(data);
}
println!("512 bytes is {:.1}% of the calls a 1-byte buffer makes",
100.0 * counts[1] as f64 / counts[0] as f64);

1== 1 MiB through three buffers ==
1-byte buffer 1,048,576 reads, then the 0: 1,048,577 calls
512-byte buffer 2,048 reads, then the 0: 2,049 calls
4096-byte buffer 256 reads, then the 0: 257 calls
512 bytes is 0.2% of the calls a 1-byte buffer makes
2 println!("\n== the first factor of 512 is nearly all of it ==");
let (big_reads, _) = drain(&mut Source::new(MIB, usize::MAX), &mut big[..65536]);
println!(" 1 -> 512 saves {} calls", commas(counts[0] - counts[1]));
println!(" 512 -> 4096 saves {} calls", commas(counts[1] - counts[2]));
println!(" 4096 -> 65536 saves {} calls ({} left)", commas(counts[2] - big_reads), big_reads);
println!("512 to 64 KiB removes {} of the last {} calls, at 128 times the memory",
commas(counts[1] - big_reads), commas(counts[1]));

2== the first factor of 512 is nearly all of it ==
1 -> 512 saves 1,046,528 calls
512 -> 4096 saves 1,792 calls
4096 -> 65536 saves 240 calls (16 left)
512 to 64 KiB removes 2,032 of the last 2,048 calls, at 128 times the memory
3 println!("\n== what the buffer costs: one 4 KiB stack page ==");
for size in [512, 1024, 4096, 8192] {
let verdict = match size {
s if s < 4096 => "fits, with room for everything else",
4096 => "the whole page: nothing left for any other local",
_ => "does not fit: sp runs off the page",
};
println!("{:>5} bytes = {:>5.1}% of the stack {verdict}",
size, 100.0 * size as f64 / 4096.0);
}
println!("that is why cat and wc use 512 bytes and grep uses 1,024.");

3== what the buffer costs: one 4 KiB stack page ==
512 bytes = 12.5% of the stack fits, with room for everything else
1024 bytes = 25.0% of the stack fits, with room for everything else
4096 bytes = 100.0% of the stack the whole page: nothing left for any other local
8192 bytes = 200.0% of the stack does not fit: sp runs off the page
that is why cat and wc use 512 bytes and grep uses 1,024.
4 println!("\n== on rv6: a kernel that returns at most 128 bytes ==");
for size in [512, 4096] {
let (data, _) = drain(&mut Source::new(MIB, 128), &mut big[..size]);
println!("{:>5}-byte buffer {:>9} reads", size, commas(data));
}
println!("past 128 the count stops falling: the kernel, not the buffer, sets it.");
println!("the loop is the same on both. Only the number of trips changes.");
4== on rv6: a kernel that returns at most 128 bytes ==
512-byte buffer 8,192 reads
4096-byte buffer 8,192 reads
past 128 the count stops falling: the kernel, not the buffer, sets it.
the loop is the same on both. Only the number of trips changes.
}
exit status 0

04 Read until 0, write every byte: tr a-z A-Z on a stingy kernel.

listen to both counts, read's and write's · point at write_all(dst, &buf[..n])? beside the bare write that lost 10 bytes

$ cargo run --bin 04_read_loop_tr
//! 04 — Read until 0, write every byte: `tr a-z A-Z` on a stingy kernel.▸ 17 more lines
//!
//! `tr a-z A-Z` copies its input to its output with each lowercase letter
//! made uppercase. The work is one line inside the loop every filter has:
//!
//! loop {
//! let n = read(src, &mut buf)?; maybe fewer than buf.len()
//! if n == 0 { break } the only end
//! buf[..n].make_ascii_uppercase(); this chunk, and only its n bytes
//! write_all(dst, &buf[..n])?; maybe several writes
//! }
//!
//! The folded `mod ulib` below is a 40-line stand-in: one input file on fd 3,
//! a `read` that returns at most READ_MAX bytes, and a `write` to fd 1 that
//! takes at most WRITE_MAX. Both return how many. Neither complains. Every
//! byte still arrives, because the loop listens to both numbers.
//!
//! Run: cargo run --bin 04_read_loop_tr

use ulib::{read, write, write_all, Error, Fd, INPUT, STDOUT};

// fold: ---- a 40-line ulib: one input file, one output, both stingy -----▸ 74 more lines
mod ulib {
use std::sync::Mutex;

pub type Fd = i32;
pub const STDOUT: Fd = 1;
pub const INPUT: Fd = 3; // a file somebody already opened

/// A failed call. rv6 says -1 and nothing more, so neither does this.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Error(pub i32);

struct State {
input: &'static [u8],
read_max: usize,
write_max: usize,
out: Vec<u8>,
reads: Vec<usize>,
writes: Vec<usize>,
}

static S: Mutex<State> = Mutex::new(State {
input: b"", read_max: 0, write_max: 0,
out: Vec::new(), reads: Vec::new(), writes: Vec::new(),
});

/// A fresh input, fresh limits, an empty output, no calls yet.
pub fn reset(input: &'static [u8], read_max: usize, write_max: usize) {
let mut s = S.lock().unwrap();
*s = State { input, read_max, write_max, out: Vec::new(), reads: Vec::new(), writes: Vec::new() };
}

/// Hands back at most READ_MAX bytes, and says how many. 0: used up.
pub fn read(fd: Fd, buf: &mut [u8]) -> Result<usize, Error> {
let mut s = S.lock().unwrap();
if fd != INPUT {
return Err(Error(-1));
}
let n = s.input.len().min(buf.len()).min(s.read_max);
buf[..n].copy_from_slice(&s.input[..n]);
s.input = &s.input[n..];
s.reads.push(n);
Ok(n)
}

/// Keeps at most WRITE_MAX bytes, and says how many.
pub fn write(fd: Fd, buf: &[u8]) -> Result<usize, Error> {
let mut s = S.lock().unwrap();
if fd != STDOUT {
return Err(Error(-1));
}
let n = buf.len().min(s.write_max);
s.out.extend_from_slice(&buf[..n]);
s.writes.push(n);
Ok(n)
}

/// Every byte, or an error. ulib/src/lib.rs, verbatim.
pub fn write_all(fd: Fd, mut buf: &[u8]) -> Result<(), Error> {
while !buf.is_empty() {
let n = write(fd, buf)?;
if n == 0 {
return Err(Error(-1));
}
buf = &buf[n..];
}
Ok(())
}

/// What reached fd 1, and what each call returned.
pub fn report() -> (String, Vec<usize>, Vec<usize>) {
let s = S.lock().unwrap();
(String::from_utf8_lossy(&s.out).into_owned(), s.reads.clone(), s.writes.clone())
}
}

/// `tr a-z A-Z`: returns how many bytes went through.
fn upcase(src: Fd, dst: Fd) -> Result<usize, Error> {
let mut buf = [0u8; 16];
let mut total = 0;
loop {
let n = read(src, &mut buf)?;
if n == 0 {
return Ok(total);
}
buf[..n].make_ascii_uppercase();
write_all(dst, &buf[..n])?;
total += n;
}
}

/// The same loop with a bare `write`. It compiles, and nothing warns.
fn upcase_bare(src: Fd, dst: Fd) -> Result<usize, Error> {
let mut buf = [0u8; 16];
let mut total = 0;
loop {
let n = read(src, &mut buf)?;
if n == 0 {
return Ok(total);
}
buf[..n].make_ascii_uppercase();
write(dst, &buf[..n])?; // the count it returns goes nowhere
total += n;
}
}

const TEXT: &[u8] = b"mount /\nstart the shell\nready\n";

fn main() {
1 println!("== the input: {} bytes in a file on fd {INPUT} ==", TEXT.len());
println!("input = {:?}", String::from_utf8_lossy(TEXT));
println!("the buffer is 16 bytes. read hands back at most 8; write takes at most 5.");

1== the input: 30 bytes in a file on fd 3 ==
input = "mount /\nstart the shell\nready\n"
the buffer is 16 bytes. read hands back at most 8; write takes at most 5.
2 println!("\n== the loop, on the stingy kernel ==");
ulib::reset(TEXT, 8, 5);
let r = upcase(INPUT, STDOUT);
let (out, reads, writes) = ulib::report();
println!("upcase(..) = {r:?}");
println!("read returned {reads:?}");
println!("write returned {writes:?}");
println!("fd 1 got {out:?}");
println!("8 bytes in, then write took 5 and write_all went back for the 3.");

2== the loop, on the stingy kernel ==
upcase(..) = Ok(30)
read returned [8, 8, 8, 6, 0]
write returned [5, 3, 5, 3, 5, 3, 5, 1]
fd 1 got "MOUNT /\nSTART THE SHELL\nREADY\n"
8 bytes in, then write took 5 and write_all went back for the 3.
3 println!("\n== the same loop, on a generous host ==");
ulib::reset(TEXT, usize::MAX, usize::MAX);
let r = upcase(INPUT, STDOUT);
let (out2, reads, writes) = ulib::report();
println!("upcase(..) = {r:?}");
println!("read returned {reads:?}");
println!("write returned {writes:?}");
println!("same output: {}. The kernel chose the calls; the loop chose nothing.", out2 == out);

3== the same loop, on a generous host ==
upcase(..) = Ok(30)
read returned [16, 14, 0]
write returned [16, 14]
same output: true. The kernel chose the calls; the loop chose nothing.
4 println!("\n== bare write: every call says how many, and nobody listens ==");
ulib::reset(TEXT, 8, 5);
let r = upcase_bare(INPUT, STDOUT);
let (out, _, writes) = ulib::report();
println!("upcase_bare(..) = {r:?}");
println!("write returned {writes:?}");
println!("fd 1 got {out:?}");
println!("{} of {} bytes arrived, and the function still said Ok.", out.len(), TEXT.len());
println!("on the host every write is taken whole, so every test of this passes.");

4== bare write: every call says how many, and nobody listens ==
upcase_bare(..) = Ok(30)
write returned [5, 5, 5, 5]
fd 1 got "MOUNTSTARTE SHEREADY"
20 of 30 bytes arrived, and the function still said Ok.
on the host every write is taken whole, so every test of this passes.
5 println!("\n== a write that takes 0 is an error, not a loop ==");
ulib::reset(TEXT, 8, 0);
let r = upcase(INPUT, STDOUT);
let (_, reads, writes) = ulib::report();
println!("upcase(..) = {r:?}");
println!("read returned {reads:?}");
println!("write returned {writes:?}");
println!("without the `n == 0` check, write_all would ask again forever.");
5== a write that takes 0 is an error, not a loop ==
upcase(..) = Err(Error(-1))
read returned [8]
write returned [0]
without the `n == 0` check, write_all would ask again forever.
}
exit status 0

05 One bad file, one exit status: a later success does not erase it.

a later success does not erase a failure · point at status = 1; // only ever set, never cleared

$ cargo run --bin 05_exit_status
//! 05 — One bad file, one exit status: a later success does not erase it.▸ 17 more lines
//!
//! $ cat gone.txt todo.txt
//! cat: gone.txt: No such file or directory <- fd 2, the complaint
//! buy milk <- fd 1, the data
//! $ echo $?
//! 1 <- the WHOLE run
//!
//! The `i32` a command returns is its exit status: 0 for success, anything
//! else for failure. This program replays that transcript against a
//! one-file "filesystem" in memory, three ways. Only one of them prints
//! `buy milk` AND ends with 1.
//!
//! None of the three is really `cat`. Each file arrives in one piece, with
//! no read loop (04 has the loop). The one idea is the status variable, and
//! where it is allowed to change.
//!
//! Run: cargo run --bin 05_exit_status

/// What the program wrote to fd 1 and fd 2.
#[derive(Default)]
struct Screen {
out: Vec<String>,
err: Vec<String>,
}

// fold: ---- the "filesystem": one file, and an open that can refuse ------▸ 9 more lines
/// `open` + `read` in one call, for this program only: the whole file, or
/// the news that there is none.
fn open_and_read(path: &str) -> Result<&'static str, &'static str> {
const FILES: [(&str, &str); 1] = [("todo.txt", "buy milk")];
FILES.iter()
.find(|(name, _)| *name == path)
.map(|(_, text)| *text)
.ok_or("No such file or directory")
}

/// Stops at the first failure.
fn gives_up(paths: &[&str], s: &mut Screen) -> i32 {
for p in paths {
match open_and_read(p) {
Ok(text) => s.out.push(text.to_string()),
Err(why) => {
s.err.push(format!("cat: {p}: {why}"));
return 1;
}
}
}
0
}

/// Every file gets its turn, and the LAST one decides the status.
fn forgets(paths: &[&str], s: &mut Screen) -> i32 {
let mut status = 0;
for p in paths {
match open_and_read(p) {
Ok(text) => {
s.out.push(text.to_string());
status = 0; // WRONG: this erases the failure before it
}
Err(why) => {
s.err.push(format!("cat: {p}: {why}"));
status = 1;
}
}
}
status
}

/// Every file gets its turn, and any failure sticks.
fn remembers(paths: &[&str], s: &mut Screen) -> i32 {
let mut status = 0;
for p in paths {
match open_and_read(p) {
Ok(text) => s.out.push(text.to_string()),
Err(why) => {
s.err.push(format!("cat: {p}: {why}"));
status = 1; // only ever set, never cleared
}
}
}
status
}

fn show(s: &Screen, status: i32) {
for line in &s.err {
println!(" fd 2 | {line}");
}
for line in &s.out {
println!(" fd 1 | {line}");
}
println!(" $? = {status}");
}

fn main() {
let paths = ["gone.txt", "todo.txt"];

1 println!("== gives up at the first failure ==");
let mut s = Screen::default();
let status = gives_up(&paths, &mut s);
show(&s, status);
println!("the status is right, and todo.txt was never printed.");

1== gives up at the first failure ==
fd 2 | cat: gone.txt: No such file or directory
$? = 1
the status is right, and todo.txt was never printed.
2 println!("\n== lets the last file decide ==");
let mut s = Screen::default();
let status = forgets(&paths, &mut s);
show(&s, status);
println!("both lines are right, and the 0 is a lie: a file was missing.");

2== lets the last file decide ==
fd 2 | cat: gone.txt: No such file or directory
fd 1 | buy milk
$? = 0
both lines are right, and the 0 is a lie: a file was missing.
3 println!("\n== remembers any failure ==");
let mut s = Screen::default();
let status = remembers(&paths, &mut s);
show(&s, status);
println!("the transcript, exactly. The 1 describes the run, not the last file.");

3== remembers any failure ==
fd 2 | cat: gone.txt: No such file or directory
fd 1 | buy milk
$? = 1
the transcript, exactly. The 1 describes the run, not the last file.
4 println!("\n== why the complaint goes to fd 2 ==");
println!("$ cat gone.txt todo.txt > both");
println!(" the screen gets fd 2: {:?}", s.err);
println!(" the file `both` gets fd 1: {:?}", s.out);
println!("the redirect moved fd 1 only, so `both` holds data and nothing else.");

4== why the complaint goes to fd 2 ==
$ cat gone.txt todo.txt > both
the screen gets fd 2: ["cat: gone.txt: No such file or directory"]
the file `both` gets fd 1: ["buy milk"]
the redirect moved fd 1 only, so `both` holds data and nothing else.
5 println!("\n== the shell reads the status ==");
for (status, meaning) in [(0, "some line matched"), (1, "nothing matched"), (2, "a file would not open")] {
let runs = if status == 0 { "runs" } else { "is skipped" };
println!(" grep -q ERROR log -> {status} ({meaning}): `&& echo read the log` {runs}");
}
println!("grep's 1 is the answer `no`, not a failure. A grep that always said 0");
println!("would make every `if grep ..` take the same branch.");

5== the shell reads the status ==
grep -q ERROR log -> 0 (some line matched): `&& echo read the log` runs
grep -q ERROR log -> 1 (nothing matched): `&& echo read the log` is skipped
grep -q ERROR log -> 2 (a file would not open): `&& echo read the log` is skipped
grep's 1 is the answer `no`, not a failure. A grep that always said 0
would make every `if grep ..` take the same branch.
6 println!("\n== one byte ==");
for count in [1, 3, 255, 256, 257] {
println!(" exit({count:>3}) -> $? = {}", count as u8);
}
println!("the status reaches the shell as one byte. Return the number of failed files");
println!("and the 256th failure reads as success. cat says 1 for any failure.");
6== one byte ==
exit( 1) -> $? = 1
exit( 3) -> $? = 3
exit(255) -> $? = 255
exit(256) -> $? = 0
exit(257) -> $? = 1
the status reaches the shell as one byte. Return the number of failed files
and the 256th failure reads as success. cat says 1 for any failure.
}
exit status 0

06 len counts bytes. A character may be several of them.

len counts bytes, not characters · point at "año".len() = 4 beside .chars().count() = 3

$ cargo run --bin 06_bytes_chars_str
//! 06 — `len` counts bytes. A character may be several of them.▸ 19 more lines
//!
//! Four types can hold text, and they promise different things:
//!
//! u8 1 byte nothing: any of 256 values
//! char 4 bytes one Unicode scalar value
//! &str 16 bytes (pointer, length) the bytes are valid UTF-8
//! &[u8] 16 bytes (pointer, length) nothing
//!
//! In UTF-8 a character takes 1 to 4 bytes, and ASCII takes exactly one. The
//! first byte says how long the character is. Every byte after it has the
//! form 10xxxxxx:
//!
//! a 0110_0001 61 top bit clear: ASCII
//! ñ 1100_0011 1011_0001 C3 B1 top bit set, both bytes
//!
//! So a byte-wise search for an ASCII byte, like b'\n', can never land
//! inside ñ. That is why cat, wc and grep can treat every file as bytes.
//!
//! Run: cargo run --bin 06_bytes_chars_str

use std::mem::size_of;
use std::str;

/// One byte against several patterns at once.
fn is_vowel(b: u8) -> bool {
matches!(b, b'a' | b'e' | b'i' | b'o' | b'u')
}

fn main() {
1 println!("== one char, two bytes ==");
println!("'ñ'.len_utf8() = {} one char, two bytes", 'ñ'.len_utf8());
println!("\"año\".len() = {} len counts bytes", "año".len());
println!("\"año\".chars().count() = {} chars decodes them first", "año".chars().count());
println!("size_of::<char>() = {} a char is not a byte", size_of::<char>());
println!("size_of::<&str>() = {} and a &str is a pointer and a length,", size_of::<&str>());
println!("size_of::<&[u8]>() = {} exactly like a &[u8]. Only the promise differs.",
size_of::<&[u8]>());

1== one char, two bytes ==
'ñ'.len_utf8() = 2 one char, two bytes
"año".len() = 4 len counts bytes
"año".chars().count() = 3 chars decodes them first
size_of::<char>() = 4 a char is not a byte
size_of::<&str>() = 16 and a &str is a pointer and a length,
size_of::<&[u8]>() = 16 exactly like a &[u8]. Only the promise differs.
2 println!("\n== the bytes of a file holding año and a newline ==");
let file = "año\n".as_bytes();
for &b in file {
println!(" {b:02X} {b:08b} {}", if b < 0x80 { "ASCII" } else { "part of a longer char" });
}
println!("{} bytes for 3 letters and a newline. `wc` reports {}.", file.len(), file.len());

2== the bytes of a file holding año and a newline ==
61 01100001 ASCII
C3 11000011 part of a longer char
B1 10110001 part of a longer char
6F 01101111 ASCII
0A 00001010 ASCII
5 bytes for 3 letters and a newline. `wc` reports 5.
3 println!("\n== one to four bytes ==");
for s in ["a", "ñ", "€", "🦀"] {
let bytes: Vec<String> = s.bytes().map(|b| format!("{b:08b}")).collect();
println!(" {s} {} byte(s) {}", s.len(), bytes.join(" "));
}
println!("a first byte of 0xxxxxxx is ASCII; 110, 1110 and 11110 say 2, 3 and 4 bytes.");
println!("every byte after the first starts 10.");

3== one to four bytes ==
a 1 byte(s) 01100001
ñ 2 byte(s) 11000011 10110001
€ 3 byte(s) 11100010 10000010 10101100
🦀 4 byte(s) 11110000 10011111 10100110 10000000
a first byte of 0xxxxxxx is ASCII; 110, 1110 and 11110 say 2, 3 and 4 bytes.
every byte after the first starts 10.
4 println!("\n== a search for b'\\n' cannot land inside a character ==");
let text = "año\nniño\n".as_bytes();
let newlines: Vec<usize> = (0..text.len()).filter(|&i| text[i] == b'\n').collect();
let high: Vec<usize> = (0..text.len()).filter(|&i| text[i] >= 0x80).collect();
println!("b'\\n' at {newlines:?}");
println!("bytes >= 0x80 at {high:?} the two ñ, two bytes each");
println!("0x0A has its top bit clear, and no byte of ñ does. They never collide.");

4== a search for b'\n' cannot land inside a character ==
b'\n' at [4, 10]
bytes >= 0x80 at [1, 2, 7, 8] the two ñ, two bytes each
0x0A has its top bit clear, and no byte of ñ does. They never collide.
5 println!("\n== b\"-n\" is bytes; \"-n\" is a str ==");
let flag: &[u8; 2] = b"-n";
let arg: &[u8] = b"-n"; // what argv hands you
println!("b\"-n\" = {flag:?} a &[u8; 2]");
println!("arg == b\"-n\" -> {}", arg == b"-n");
println!("arg == \"-n\".as_bytes() -> {} the same two bytes, reached from a str",
arg == "-n".as_bytes());
println!("b'a' = {}, 'a' as u8 = {}, 97u8 as char = {:?}", b'a', 'a' as u8, 97u8 as char);
println!("a u8 and a char never compare directly. RUN ./show-errors.sh e0308");

5== b"-n" is bytes; "-n" is a str ==
b"-n" = [45, 110] a &[u8; 2]
arg == b"-n" -> true
arg == "-n".as_bytes() -> true the same two bytes, reached from a str
b'a' = 97, 'a' as u8 = 97, 97u8 as char = 'a'
a u8 and a char never compare directly. RUN ./show-errors.sh e0308
6 println!("\n== matches! tests a byte against several patterns ==");
let word = b"rv6 boots";
let marks: String = word.iter()
.map(|&b| if is_vowel(b) { 'v' } else if matches!(b, b'0'..=b'9') { 'd' } else { '.' })
.collect();
println!(" {}", String::from_utf8_lossy(word));
println!(" {marks} v = a vowel, d = a digit");
println!("matches!(0xC3, b'a'..=b'z') = {} ñ's bytes are not letters to a byte test",
matches!(0xC3u8, b'a'..=b'z'));

6== matches! tests a byte against several patterns ==
rv6 boots
..d..vv.. v = a vowel, d = a digit
matches!(0xC3, b'a'..=b'z') = false ñ's bytes are not letters to a byte test
7 println!("\n== when you really want characters, you pay for a check ==");
let cut = &"año".as_bytes()[..2]; // a, then the first byte of ñ
println!("from_utf8(&\"año\".as_bytes()[..2]) = {:?}", str::from_utf8(cut));
println!("from_utf8_lossy of the same bytes = {:?}", String::from_utf8_lossy(cut));
println!("a read that ends between C3 and B1 hands you exactly that.");
println!("wc -m, which counts characters, has to decode every byte to answer.");
7== when you really want characters, you pay for a check ==
from_utf8(&"año".as_bytes()[..2]) = Err(Utf8Error { valid_up_to: 1, error_len: None })
from_utf8_lossy of the same bytes = "a�"
a read that ends between C3 and B1 hands you exactly that.
wc -m, which counts characters, has to decode every byte to answer.
}
exit status 0

07 State that must survive the end of a read.

state the caller keeps survives the end of a read · point at after_cr: &mut bool, and the two <- wrong rows

$ cargo run --bin 07_state_across_chunks
//! 07 — State that must survive the end of a read.▸ 17 more lines
//!
//! A stream arrives in chunks, and the kernel picks where each one ends.
//! Anything that spans two bytes can span two reads. This counter of
//! Windows line endings, `\r\n`, keeps the one fact about the past that the
//! next byte might need, in a variable the CALLER owns:
//!
//! fn feed(chunk: &[u8], after_cr: &mut bool, pairs: &mut usize)
//!
//! chunk 1 o n e \r after_cr = true pairs = 0
//! chunk 2 \n t w o \r \n after_cr = false pairs = 2
//! ^ pairs with the \r at the end of chunk 1
//!
//! `feed_forgetful` declares `after_cr` inside itself instead, so it starts
//! false on every call. It is right whenever no pair straddles two chunks,
//! and the program does not choose where the chunks end.
//!
//! Run: cargo run --bin 07_state_across_chunks

/// The lecture's counter. Called once per chunk; the caller keeps `after_cr`.
fn feed(chunk: &[u8], after_cr: &mut bool, pairs: &mut usize) {
for &b in chunk {
if *after_cr && b == b'\n' {
*pairs += 1;
}
*after_cr = b == b'\r';
}
}

/// The same loop, with the state declared in the wrong place.
fn feed_forgetful(chunk: &[u8], pairs: &mut usize) {
let mut after_cr = false; // reset on every call: the past is gone
for &b in chunk {
if after_cr && b == b'\n' {
*pairs += 1;
}
after_cr = b == b'\r';
}
}

/// Bytes as Rust would write them in a string literal: "one\r".
fn shown(bytes: &[u8]) -> String {
format!("{:?}", String::from_utf8_lossy(bytes))
}

fn main() {
let input: &[u8] = b"one\r\ntwo\r\n";

1 println!("== one feed of the whole input ==");
let (mut after_cr, mut pairs) = (false, 0);
feed(input, &mut after_cr, &mut pairs);
println!("feed({}) -> pairs = {pairs}", shown(input));

1== one feed of the whole input ==
feed("one\r\ntwo\r\n") -> pairs = 2
2 println!("\n== the same bytes in two chunks, state carried ==");
let (mut after_cr, mut pairs) = (false, 0);
for chunk in [&input[..4], &input[4..]] {
feed(chunk, &mut after_cr, &mut pairs);
println!("after {:<14} after_cr = {after_cr:<5} pairs = {pairs}", shown(chunk));
}
println!("2, the same as one feed. One row per chunk, each starting from the row above.");

2== the same bytes in two chunks, state carried ==
after "one\r" after_cr = true pairs = 0
after "\ntwo\r\n" after_cr = false pairs = 2
2, the same as one feed. One row per chunk, each starting from the row above.
3 println!("\n== the same chunks, state forgotten ==");
let mut pairs = 0;
for chunk in [&input[..4], &input[4..]] {
feed_forgetful(chunk, &mut pairs);
println!("after {:<14} pairs = {pairs}", shown(chunk));
}
println!("1. The \\n that opens chunk 2 met a fresh `false`, and that pair vanished.");

3== the same chunks, state forgotten ==
after "one\r" pairs = 0
after "\ntwo\r\n" pairs = 1
1. The \n that opens chunk 2 met a fresh `false`, and that pair vanished.
4 println!("\n== every place a read could end ==");
for k in 0..=input.len() {
let (a, b) = input.split_at(k);
let (mut after_cr, mut carried) = (false, 0);
feed(a, &mut after_cr, &mut carried);
feed(b, &mut after_cr, &mut carried);
let mut forgot = 0;
feed_forgetful(a, &mut forgot);
feed_forgetful(b, &mut forgot);
let mark = if forgot == carried { "" } else { " <- wrong" };
println!(" split at {k:>2}: {:>16} | {:<16} carried {carried} forgetful {forgot}{mark}",
shown(a), shown(b));
}
println!("the carried count is 2 at every split. The forgetful one is wrong only where");
println!("a read ends between \\r and \\n, and the kernel, not the program, picks that.");

4== every place a read could end ==
split at 0: "" | "one\r\ntwo\r\n" carried 2 forgetful 2
split at 1: "o" | "ne\r\ntwo\r\n" carried 2 forgetful 2
split at 2: "on" | "e\r\ntwo\r\n" carried 2 forgetful 2
split at 3: "one" | "\r\ntwo\r\n" carried 2 forgetful 2
split at 4: "one\r" | "\ntwo\r\n" carried 2 forgetful 1 <- wrong
split at 5: "one\r\n" | "two\r\n" carried 2 forgetful 2
split at 6: "one\r\nt" | "wo\r\n" carried 2 forgetful 2
split at 7: "one\r\ntw" | "o\r\n" carried 2 forgetful 2
split at 8: "one\r\ntwo" | "\r\n" carried 2 forgetful 2
split at 9: "one\r\ntwo\r" | "\n" carried 2 forgetful 1 <- wrong
split at 10: "one\r\ntwo\r\n" | "" carried 2 forgetful 2
the carried count is 2 at every split. The forgetful one is wrong only where
a read ends between \r and \n, and the kernel, not the program, picks that.
5 println!("\n== what the state costs ==");
println!("size_of::<bool>() + size_of::<usize>() = {} bytes, for a line or a terabyte",
std::mem::size_of::<bool>() + std::mem::size_of::<usize>());
println!("that is O(1) state: the same variables whatever the input's length.");
5== what the state costs ==
size_of::<bool>() + size_of::<usize>() = 9 bytes, for a line or a terabyte
that is O(1) state: the same variables whatever the input's length.
}
exit status 0

08 A line that straddles two reads, in a buffer that never grows.

an unfinished line slides to the front · point at self.buf.copy_within(self.start..self.len, 0), then read(fd, &mut buf[6..]) -> 10

$ cargo run --bin 08_lines_in_a_fixed_buffer
//! 08 — A line that straddles two reads, in a buffer that never grows.▸ 19 more lines
//!
//! Reads know nothing about lines, so a read can stop halfway through one.
//! With a heap you would grow the buffer until the line fits. Without one,
//! you slide the unfinished part to the front and read more BEHIND it. With
//! a 16-byte buffer, after `abc` and `def` came back:
//!
//! before [a b c \n d e f \n g h i j k l . .] start 8, len 14
//! compact [g h i j k l . . . . . . . . . .] start 0, len 6
//! refill read(fd, &mut buf[6..]) -> 10
//! [g h i j k l m n \n o p q r s t u] len 16
//! return "ghijklmn" start 9
//!
//! `Lines` below is `ulib::Lines` cut down to those three cases, and it
//! prints each row as it goes: a buffered `\n` returns a line with no read;
//! at end of input the leftover bytes are the last line; otherwise compact
//! and refill. (A line longer than the buffer is left out here. ulib cuts it
//! to the buffer's length and `truncated()` says so.)
//!
//! Run: cargo run --bin 08_lines_in_a_fixed_buffer

use kernel::Source;

// fold: ---- the kernel's side: reads of the sizes it chooses -------------▸ 30 more lines
mod kernel {
/// Hands out `data` in reads of `sizes[0]`, `sizes[1]`, ... bytes, then
/// whatever fits, then 0. Counts every call.
pub struct Source {
data: &'static [u8],
sizes: &'static [usize],
pub calls: usize,
}

impl Source {
pub fn new(data: &'static [u8], sizes: &'static [usize]) -> Source {
Source { data, sizes, calls: 0 }
}

pub fn read(&mut self, buf: &mut [u8]) -> usize {
self.calls += 1;
let want = match self.sizes.split_first() {
Some((&size, rest)) => {
self.sizes = rest;
size
}
None => usize::MAX,
};
let n = want.min(buf.len()).min(self.data.len());
buf[..n].copy_from_slice(&self.data[..n]);
self.data = &self.data[n..];
n
}
}
}

/// Lines out of a buffer the caller owns. It never allocates.
struct Lines<'b> {
src: Source,
buf: &'b mut [u8],
start: usize, // where the next line begins
len: usize, // bytes held in buf
eof: bool,
}

impl<'b> Lines<'b> {
fn new(src: Source, buf: &'b mut [u8]) -> Lines<'b> {
Lines { src, buf, start: 0, len: 0, eof: false }
}

fn next_line(&mut self) -> Option<&[u8]> {
loop {
// 1. A whole line is already buffered: no read at all.
if let Some(nl) = self.buf[self.start..self.len].iter().position(|&b| b == b'\n') {
let from = self.start;
self.start += nl + 1;
println!("return {:<35} start {}", shown(&self.buf[from..from + nl]), self.start);
return Some(&self.buf[from..from + nl]);
}
// 2. End of input: whatever is left is the last line.
if self.eof {
if self.start == self.len {
println!("return None");
return None;
}
let from = self.start;
self.start = self.len;
println!("return {:<35} no \\n, and still a line", shown(&self.buf[from..self.len]));
return Some(&self.buf[from..self.len]);
}
// 3. Part of a line: slide it to the front, then read behind it.
if self.start > 0 {
println!("before {} start {}, len {}", grid(self.buf, self.len), self.start, self.len);
self.buf.copy_within(self.start..self.len, 0);
self.len -= self.start;
self.start = 0;
println!("compact {} start 0, len {}", grid(self.buf, self.len), self.len);
}
debug_assert!(self.len < self.buf.len(), "a line longer than the buffer");
let n = self.src.read(&mut self.buf[self.len..]);
println!("refill read(fd, &mut buf[{}..]) -> {n}", self.len);
if n == 0 {
self.eof = true;
} else {
self.len += n;
println!(" {} len {}", grid(self.buf, self.len), self.len);
}
}
}
}

/// The buffer as the lecture draws it: held bytes, then dots.
fn grid(buf: &[u8], len: usize) -> String {
let cells: Vec<String> = buf.iter().enumerate().map(|(i, &b)| match b {
_ if i >= len => ".".to_string(),
b'\n' => "\\n".to_string(),
_ => (b as char).to_string(),
}).collect();
format!("[{}]", cells.join(" "))
}

/// Bytes as Rust would write them in a string literal.
fn shown(bytes: &[u8]) -> String {
format!("{:?}", String::from_utf8_lossy(bytes))
}

const INPUT: &[u8] = b"abc\ndef\nghijklmn\nopqrstuvw\nxyz";

fn main() {
1 println!("== the input, and a 16-byte buffer ==");
println!("{} {} bytes, five lines, the last with no \\n", shown(INPUT), INPUT.len());
println!("the kernel hands it over in reads of 14, then 10, then whatever is left.");
let mut buf = [0u8; 16];
let mut lines = Lines::new(Source::new(INPUT, &[14, 10]), &mut buf);

1== the input, and a 16-byte buffer ==
"abc\ndef\nghijklmn\nopqrstuvw\nxyz" 30 bytes, five lines, the last with no \n
the kernel hands it over in reads of 14, then 10, then whatever is left.
2 println!("\n== abc and def: one read, two lines ==");
lines.next_line(); // it prints its own rows
lines.next_line();
println!("the second call made no read. Its \\n was already in the buffer.");

2== abc and def: one read, two lines ==
refill read(fd, &mut buf[0..]) -> 14
[a b c \n d e f \n g h i j k l . .] len 14
return "abc" start 4
return "def" start 8
the second call made no read. Its \n was already in the buffer.
3 println!("\n== ghijklmn: the tail moves to the front, the read goes behind it ==");
lines.next_line();
println!("`ghijkl` came in one read and `mn` in the next. The caller sees one line.");

3== ghijklmn: the tail moves to the front, the read goes behind it ==
before [a b c \n d e f \n g h i j k l . .] start 8, len 14
compact [g h i j k l . . . . . . . . . .] start 0, len 6
refill read(fd, &mut buf[6..]) -> 10
[g h i j k l m n \n o p q r s t u] len 16
return "ghijklmn" start 9
`ghijkl` came in one read and `mn` in the next. The caller sees one line.
4 println!("\n== the rest, and a last line with no \\n ==");
while lines.next_line().is_some() {}
println!("the read that returned 0 did not end the lines. The call after the last one did.");

4== the rest, and a last line with no \n ==
before [g h i j k l m n \n o p q r s t u] start 9, len 16
compact [o p q r s t u . . . . . . . . .] start 0, len 7
refill read(fd, &mut buf[7..]) -> 6
[o p q r s t u v w \n x y z . . .] len 13
return "opqrstuvw" start 10
before [o p q r s t u v w \n x y z . . .] start 10, len 13
compact [x y z . . . . . . . . . . . . .] start 0, len 3
refill read(fd, &mut buf[3..]) -> 0
return "xyz" no \n, and still a line
return None
the read that returned 0 did not end the lines. The call after the last one did.
5 println!("\n== what it cost ==");
println!("{} reads for {} bytes, through 16 bytes of storage.", lines.src.calls, INPUT.len());
println!("each line was a slice of that one array. Nothing was copied out, and nothing grew.");
5== what it cost ==
4 reads for 30 bytes, through 16 bytes of storage.
each line was a slice of that one array. Nothing was copied out, and nothing grew.
}
exit status 0

09 while let: call, match, and stop at the first miss.

call, match, and stop at the first miss · point at while let Some(line) = lines.next_line(), and Some("") for a blank line

$ cargo run --bin 09_while_let
//! 09 — `while let`: call, match, and stop at the first miss.▸ 20 more lines
//!
//! while let Some(top) = stack.pop() {
//! ... runs once for each Some
//! } the first None ends the loop
//!
//! is exactly this, written shorter:
//!
//! loop {
//! match stack.pop() {
//! Some(top) => { ... }
//! _ => break,
//! }
//! }
//!
//! A `for` loop needs an `Iterator`, and `ulib::Lines` is not one. Each line
//! it returns borrows the buffer that the NEXT call may overwrite, and
//! `Iterator` has no way to say "this item dies when you ask for another".
//! So a program that reads lines drives them by hand, with `while let`.
//!
//! Run: cargo run --bin 09_while_let

/// Lines out of a byte slice, with ulib's signature: each line borrows
/// `self`, so nothing else may touch `self` while a line is in use.
struct Lines<'a> {
rest: &'a [u8],
calls: usize,
}

impl<'a> Lines<'a> {
fn new(text: &'a [u8]) -> Lines<'a> {
Lines { rest: text, calls: 0 }
}

fn next_line(&mut self) -> Option<&[u8]> {
self.calls += 1;
if self.rest.is_empty() {
return None;
}
let end = self.rest.iter().position(|&b| b == b'\n').unwrap_or(self.rest.len());
let (line, after) = self.rest.split_at(end);
self.rest = after.get(1..).unwrap_or(b""); // step over the \n
Some(line)
}
}

fn main() {
1 println!("== pop until None ==");
let mut stack = vec![10, 20, 30];
while let Some(top) = stack.pop() {
println!(" top = {top}");
}
println!("pop() answered None, so the loop ended. stack = {stack:?}");

1== pop until None ==
top = 30
top = 20
top = 10
pop() answered None, so the loop ended. stack = []
2 println!("\n== the same loop, spelled out ==");
let mut stack = vec!["init", "sh", "grep"];
loop {
match stack.pop() {
Some(top) => println!(" top = {top:?}"),
_ => break,
}
}
println!("one call, one pattern, per turn. The first value that fails it ends the loop.");

2== the same loop, spelled out ==
top = "grep"
top = "sh"
top = "init"
one call, one pattern, per turn. The first value that fails it ends the loop.
3 println!("\n== any pattern will do, not just Some ==");
let mut rest: &[u8] = b"rv6";
while let [first, tail @ ..] = rest {
println!(" first = {:?}, {} byte(s) after it", *first as char, tail.len());
rest = tail;
}
println!("`[first, tail @ ..]` fails on an empty slice, and that ended the loop.");

3== any pattern will do, not just Some ==
first = 'r', 2 byte(s) after it
first = 'v', 1 byte(s) after it
first = '6', 0 byte(s) after it
`[first, tail @ ..]` fails on an empty slice, and that ended the loop.
4 println!("\n== a line source, driven by hand ==");
let mut lines = Lines::new(b"ls\ncd /\n\nexit\n");
while let Some(line) = lines.next_line() {
println!(" {:<8} {} byte(s)", format!("{:?}", String::from_utf8_lossy(line)), line.len());
}
println!("{} calls for 4 lines: the last call is the None that ended the loop.", lines.calls);
println!("the empty line came back as Some(\"\"). Empty is a line; None is the end.");

4== a line source, driven by hand ==
"ls" 2 byte(s)
"cd /" 4 byte(s)
"" 0 byte(s)
"exit" 4 byte(s)
5 calls for 4 lines: the last call is the None that ended the loop.
the empty line came back as Some(""). Empty is a line; None is the end.
5 println!("\n== leaving early: break ==");
let mut lines = Lines::new(b"Host: rv6\nAccept: */*\n\nthe body, which nobody asks for\n");
while let Some(line) = lines.next_line() {
if line.is_empty() {
break; // the blank line ends the header block
}
println!(" header {:?}", String::from_utf8_lossy(line));
}
println!("{} calls, and the body was never asked for.", lines.calls);
println!("`break` and `return` leave a `while let` like any other loop.");
5== leaving early: break ==
header "Host: rv6"
header "Accept: */*"
3 calls, and the body was never asked for.
`break` and `return` leave a `while let` like any other loop.
}
exit status 0

10 Fenceposts: count the starts, and count both ends.

the last start is len - k, and only ..= reaches it · point at (0..3).contains(&3) = false for log in syslog

$ cargo run --bin 10_fenceposts
//! 10 — Fenceposts: count the starts, and count both ends.▸ 20 more lines
//!
//! A line matches if the needle occurs in it as one contiguous run of bytes:
//! `ear` is in `heart`, but not in `era`. Sliding a needle along a line is a
//! fencepost count, and both ends are posts:
//!
//! line h e a r t 5 bytes
//! starts 0 1 2 3 a 2-byte needle: 4 starts, the last is 5 - 2
//!
//! Rust has two range forms, and only one includes its end:
//!
//! 1..5 1, 2, 3, 4 stops BEFORE 5
//! 1..=5 1, 2, 3, 4, 5 up to and including 5
//!
//! Lengths are `usize`, which has no negative numbers: `2 - 5` panics in a
//! debug build, and `2usize.checked_sub(5)` answers None instead.
//!
//! The search itself is 13c's to write, so this program asks std's
//! `str::contains` and `str::find`, and only counts where a needle COULD start.
//!
//! Run: cargo run --bin 10_fenceposts

use std::hint::black_box;

fn main() {
1 println!("== two ranges ==");
let a: Vec<u32> = (1..5).collect();
let b: Vec<u32> = (1..=5).collect();
println!("(1..5) = {a:?} stops before 5");
println!("(1..=5) = {b:?} up to and including 5");

1== two ranges ==
(1..5) = [1, 2, 3, 4] stops before 5
(1..=5) = [1, 2, 3, 4, 5] up to and including 5
2 println!("\n== where a 2-byte needle can start in a 5-byte line ==");
let line = "heart";
let k = 2;
let letters: Vec<String> = line.chars().map(String::from).collect();
println!(" {}", letters.join(" "));
for start in 0..=line.len() - k {
let carets = format!("{}{}", " ".repeat(start), "^ ".repeat(k));
println!(" {:<11} start {start}", carets.trim_end());
}
println!("0..=(5 - 2) is {} starts. 0..(5 - 2) is {}, and never tries start 3.",
(0..=line.len() - k).count(), (0..line.len() - k).count());

2== where a 2-byte needle can start in a 5-byte line ==
h e a r t
^ ^ start 0
^ ^ start 1
^ ^ start 2
^ ^ start 3
0..=(5 - 2) is 4 starts. 0..(5 - 2) is 3, and never tries start 3.
3 println!("\n== contiguous: ear is in heart, not in era ==");
println!("\"heart\".contains(\"ear\") = {}", "heart".contains("ear"));
println!("\"heart\".find(\"ear\") = {:?} it starts at 1", "heart".find("ear"));
println!("\"era\".contains(\"ear\") = {} all three letters, not in one run",
"era".contains("ear"));

3== contiguous: ear is in heart, not in era ==
"heart".contains("ear") = true
"heart".find("ear") = Some(1) it starts at 1
"era".contains("ear") = false all three letters, not in one run
4 println!("\n== the last start is the one people skip ==");
let (hay, needle) = ("syslog", "log");
let last = hay.len() - needle.len();
println!("\"syslog\".find(\"log\") = {:?}", hay.find(needle));
println!("last start = {} - {} = {last}", hay.len(), needle.len());
println!("(0..{last}).contains(&{last}) = {} so a scan over 0..{last} misses `log`",
(0..last).contains(&last));
println!("(0..={last}).contains(&{last}) = {}", (0..=last).contains(&last));
println!("`logfile` matches either way. Only lines that END in the needle go missing.");

4== the last start is the one people skip ==
"syslog".find("log") = Some(3)
last start = 6 - 3 = 3
(0..3).contains(&3) = false so a scan over 0..3 misses `log`
(0..=3).contains(&3) = true
`logfile` matches either way. Only lines that END in the needle go missing.
5 println!("\n== two sizes to decide before counting ==");
println!("\"heart\".contains(\"\") = {} an empty needle is found in every line",
"heart".contains(""));
println!("\"\".contains(\"\") = {} even in an empty one", "".contains(""));
println!("\"he\".contains(\"heart\") = {} longer than the line: no start at all",
"he".contains("heart"));
println!("its last start would be 2 - 5, and a usize cannot hold -3:");
println!(" 5usize.checked_sub(2) = {:?}", 5usize.checked_sub(2));
println!(" 2usize.checked_sub(5) = {:?}", 2usize.checked_sub(5));
println!(" 2usize.saturating_sub(5) = {} not a fix: 0..=0 would still try start 0",
2usize.saturating_sub(5));

5== two sizes to decide before counting ==
"heart".contains("") = true an empty needle is found in every line
"".contains("") = true even in an empty one
"he".contains("heart") = false longer than the line: no start at all
its last start would be 2 - 5, and a usize cannot hold -3:
5usize.checked_sub(2) = Some(3)
2usize.checked_sub(5) = None
2usize.saturating_sub(5) = 0 not a fix: 0..=0 would still try start 0
6 println!("\n== what 2 - 5 does on usize, in a debug build ==");
std::panic::set_hook(Box::new(|info| {
println!(" panicked: {}", info.payload_as_str().unwrap_or("?"));
}));
let (len, needle_len) = (black_box(2usize), black_box(5usize));
let r = std::panic::catch_unwind(|| len - needle_len);
let _ = std::panic::take_hook();
println!("catch_unwind(|| 2 - 5) = {}", if r.is_err() { "Err: it panicked" } else { "Ok" });
println!("a release build wraps to a number near 2^64 instead, and the first index");
println!("that uses it panics anyway. On rv6 either one prints the single word panic.");
6== what 2 - 5 does on usize, in a debug build ==
panicked: attempt to subtract with overflow
catch_unwind(|| 2 - 5) = Err: it panicked
a release build wraps to a number near 2^64 instead, and the first index
that uses it panics anyway. On rv6 either one prints the single word panic.
}
exit status 0

11 What a program does not read costs nothing.

what a program does not read costs nothing · point at 1,024 bytes read out of 10,000,000,000

$ cargo run --bin 11_stop_early
//! 11 — What a program does not read costs nothing.▸ 15 more lines
//!
//! A `read` hands over one buffer's worth. Nothing makes the kernel fetch
//! the rest of a file until somebody asks for it. So the first line of a
//! 10 GB log costs one read, not ten billion bytes:
//!
//! the log |line 1|line 2| ... |line 51| ........................ 10 GB |
//! one read |<------ 1,024 bytes ------>|
//! the rest never asked for, so never read
//!
//! `head` is judged by exactly this: its cost is set by how many lines it
//! wants, not by how big the file is. The log below does not exist. Each
//! `read` makes up the bytes it returns and counts them, so the program can
//! say exactly how much it read before it stopped.
//!
//! Run: cargo run --bin 11_stop_early

use log::Log;

// fold: ---- a 10 GB log that exists only when read ------------------------▸ 38 more lines
mod log {
/// Fixed-width lines: "line 0000000001 ok.\n" is 20 bytes.
pub const LINE: u64 = 20;

pub struct Log {
pub size: u64,
offset: u64,
pub reads: u64,
pub served: u64,
}

impl Log {
pub fn new(size: u64) -> Log {
Log { size, offset: 0, reads: 0, served: 0 }
}

/// The byte at `pos`, made up on the spot.
fn byte_at(pos: u64) -> u8 {
let (line, col) = (pos / LINE + 1, (pos % LINE) as usize);
match col {
0..=4 => b"line "[col],
5..=14 => b'0' + (line / 10u64.pow(14 - col as u32) % 10) as u8,
_ => b" ok.\n"[col - 15],
}
}

pub fn read(&mut self, buf: &mut [u8]) -> usize {
self.reads += 1;
let n = (self.size - self.offset).min(buf.len() as u64) as usize;
for (i, b) in buf[..n].iter_mut().enumerate() {
*b = Log::byte_at(self.offset + i as u64);
}
self.offset += n as u64;
self.served += n as u64;
n
}
}
}

/// `yes`: "y\n" forever. Its read never returns 0.
struct Yes {
reads: u64,
}

impl Yes {
fn read(&mut self, buf: &mut [u8]) -> usize {
self.reads += 1;
for (i, b) in buf.iter_mut().enumerate() {
*b = if i % 2 == 0 { b'y' } else { b'\n' };
}
buf.len()
}
}

/// 10000000000 -> "10,000,000,000"
fn commas(n: u64) -> String {
let s = n.to_string();
let mut out = String::new();
for (i, ch) in s.chars().enumerate() {
if i > 0 && (s.len() - i) % 3 == 0 {
out.push(',');
}
out.push(ch);
}
out
}

fn main() {
let mut log = Log::new(10_000_000_000);
let mut buf = [0u8; 1024];

1 println!("== a 10 GB log, made up on demand ==");
println!("size {} bytes", commas(log.size));
println!("lines {} of {} bytes each", commas(log.size / log::LINE), log::LINE);
println!("read so far: {} bytes, in {} reads", log.served, log.reads);

1== a 10 GB log, made up on demand ==
size 10,000,000,000 bytes
lines 500,000,000 of 20 bytes each
read so far: 0 bytes, in 0 reads
2 println!("\n== the first line: one read ==");
let n = log.read(&mut buf);
let end = buf[..n].iter().position(|&b| b == b'\n').unwrap_or(n);
println!("read -> {n}");
println!("first line: {:?}", String::from_utf8_lossy(&buf[..end]));
println!("read so far: {} bytes, in {} read", commas(log.served), log.reads);
println!("the same read holds lines 2 through {} whole, so they would cost no more.",
n as u64 / log::LINE);
println!("that is {:.7}% of the file. The other {} bytes were never made.",
100.0 * log.served as f64 / log.size as f64, commas(log.size - log.served));

2== the first line: one read ==
read -> 1024
first line: "line 0000000001 ok."
read so far: 1,024 bytes, in 1 read
the same read holds lines 2 through 51 whole, so they would cost no more.
that is 0.0000102% of the file. The other 9,999,998,976 bytes were never made.
3 println!("\n== what reading to the end would cost ==");
let reads = log.size.div_ceil(buf.len() as u64);
println!("{} reads of 1,024 bytes, then one more that returns 0", commas(reads));
println!("a program that reads everything and prints only the first line gives the");
println!("same output as the one above, and pays for all {} of them.", commas(reads));

3== what reading to the end would cost ==
9,765,625 reads of 1,024 bytes, then one more that returns 0
a program that reads everything and prints only the first line gives the
same output as the one above, and pays for all 9,765,625 of them.
4 println!("\n== a source that never ends ==");
let mut yes = Yes { reads: 0 };
let n = yes.read(&mut buf);
println!("yes: read -> {n}, first line {:?}", String::from_utf8_lossy(&buf[..1]));
let mut zero_seen = false;
for _ in 0..1000 {
if yes.read(&mut buf) == 0 {
zero_seen = true;
break;
}
}
println!("after {} reads, has read ever said 0? {zero_seen}", yes.reads);
println!("`yes | head -n 1` finishes only because head stops asking.");
println!("a reader that waits for 0 here waits forever.");
4== a source that never ends ==
yes: read -> 1024, first line "y"
after 1001 reads, has read ever said 0? false
`yes | head -n 1` finishes only because head stops asking.
a reader that waits for 0 here waits forever.
}
exit status 0

E0277 Lines<'_> is not an iterator

for line in lines · fix: while let Some(line) = lines.next_line()

$ ./show-errors.sh e0277
// error[E0277]: `Lines<'_>` is not an iterator
//
// `for` calls `Iterator::next`, and `Lines` has no such method, only
// `next_line`. It cannot have one: `next(&mut self) -> Option<Self::Item>`
// gives `Item` no way to borrow from this one call, and every line is a
// slice of the buffer that the NEXT call may overwrite.
//
// FIX 1: `while let Some(line) = lines.next_line() { .. }` -- call, match,
// and stop at the first None.
// FIX 2: the same loop longhand: `loop { match lines.next_line() {
// Some(line) => .., None => break } }`.
struct Lines<'b> {
buf: &'b mut [u8],
start: usize,
}

impl<'b> Lines<'b> {
fn next_line(&mut self) -> Option<&[u8]> {
let nl = self.buf[self.start..].iter().position(|&b| b == b'\n')?;
let from = self.start;
self.start += nl + 1;
Some(&self.buf[from..from + nl])
}
}

fn main() {
let mut buf = *b"boot\nsh\n";
let lines = Lines { buf: &mut buf, start: 0 };
for line in lines {
println!("{} bytes", line.len());
}
}
Why, and 2 fixes e

for calls Iterator::next, and Lines has no such method, only next_line. It cannot have one: next(&mut self) -> Option<Self::Item> gives Item no way to borrow from this one call, and every line is a slice of the buffer that the NEXT call may overwrite.

  1. while let Some(line) = lines.next_line() { .. } -- call, match, and stop at the first None.
  2. the same loop longhand: loop { match lines.next_line() { Some(line) => .., None => break } }.
$ rustc --edition 2021 --emit=metadata -o /dev/null e0277_lines_is_not_an_iterator.rs
error[E0277]: `Lines<'_>` is not an iterator
--> e0277_lines_is_not_an_iterator.rs:29:17
|
29 | for line in lines {
| ^^^^^ `Lines<'_>` is not an iterator
|
help: the trait `Iterator` is not implemented for `Lines<'_>`
--> e0277_lines_is_not_an_iterator.rs:12:1
|
12 | struct Lines<'b> {
| ^^^^^^^^^^^^^^^^
= note: required for `Lines<'_>` to implement `IntoIterator`
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0277`.
exit status 1

E0308 mismatched types expected u8, found char

a u8 compared with the char '\t' · fix: b'\t', a byte literal

$ ./show-errors.sh e0308
// error[E0308]: mismatched types
// expected `u8`, found `char`
//
// Walking a `&[u8]` gives bytes. `'\t'` is a char: four bytes, one Unicode
// scalar value. A u8 and a char never compare, even when the byte is ASCII,
// because Rust will not guess an encoding for you.
//
// FIX 1: `b == b'\t'` -- a byte literal is a u8.
// FIX 2: `b as char == '\t'` also compiles, and reads each byte as a whole
// character: fine for ASCII, wrong for both bytes of ñ.
fn count_tabs(line: &[u8]) -> usize {
let mut tabs = 0;
for &b in line {
if b == '\t' {
tabs += 1;
}
}
tabs
}

fn main() {
println!("{}", count_tabs(b"pid\tstate\tname"));
}
Why, and 2 fixes e

Walking a &[u8] gives bytes. '\t' is a char: four bytes, one Unicode scalar value. A u8 and a char never compare, even when the byte is ASCII, because Rust will not guess an encoding for you.

  1. b == b'\t' -- a byte literal is a u8.
  2. b as char == '\t' also compiles, and reads each byte as a whole character: fine for ASCII, wrong for both bytes of ñ.
$ rustc --edition 2021 --emit=metadata -o /dev/null e0308_byte_is_not_a_char.rs
error[E0308]: mismatched types
--> e0308_byte_is_not_a_char.rs:14:17
|
14 | if b == '\t' {
| - ^^^^ expected `u8`, found `char`
| |
| expected because this is `u8`
|
help: if you meant to write a byte literal, prefix with `b`
|
14 | if b == b'\t' {
| +
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0308`.
exit status 1

E0499 cannot borrow lines as mutable more than once at a time

a line held across the next next_line · fix: finish with each line first, or copy out what you need

$ ./show-errors.sh e0499
// error[E0499]: cannot borrow `lines` as mutable more than once at a time
//
// A line is a slice of the buffer `lines` holds on loan, and the next call
// to `next_line` may slide or refill that buffer. So `first` keeps `lines`
// borrowed for as long as `first` is used, and the second call needs it
// again. C compiles this and prints whatever the refill left there.
//
// FIX 1: finish with each line before asking for the next: test it, print
// it, then call `next_line` again. `while let` does that for you.
// FIX 2: copy out what you need first -- a length, a flag, a few bytes into
// an array of your own -- and keep the copy, not the slice.
struct Lines<'b> {
buf: &'b mut [u8],
start: usize,
}

impl<'b> Lines<'b> {
fn next_line(&mut self) -> Option<&[u8]> {
let nl = self.buf[self.start..].iter().position(|&b| b == b'\n')?;
let from = self.start;
self.start += nl + 1;
Some(&self.buf[from..from + nl])
}
}

fn main() {
let mut buf = *b"boot\nsh\n";
let mut lines = Lines { buf: &mut buf, start: 0 };
let first = lines.next_line();
let second = lines.next_line();
println!("{first:?} then {second:?}");
}
Why, and 2 fixes e

A line is a slice of the buffer lines holds on loan, and the next call to next_line may slide or refill that buffer. So first keeps lines borrowed for as long as first is used, and the second call needs it again. C compiles this and prints whatever the refill left there.

  1. finish with each line before asking for the next: test it, print it, then call next_line again. while let does that for you.
  2. copy out what you need first -- a length, a flag, a few bytes into an array of your own -- and keep the copy, not the slice.
$ rustc --edition 2021 --emit=metadata -o /dev/null e0499_line_held_across_next_line.rs
error[E0499]: cannot borrow `lines` as mutable more than once at a time
--> e0499_line_held_across_next_line.rs:30:18
|
29 | let first = lines.next_line();
| ----- first mutable borrow occurs here
30 | let second = lines.next_line();
| ^^^^^ second mutable borrow occurs here
31 | println!("{first:?} then {second:?}");
| ----- first borrow later used here
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0499`.
exit status 1

E0502 cannot borrow buf[_] as immutable because it is also borrowed as mutable

reading buf[0] while lines has the buffer · fix: read through the line, or after the last use of lines

$ ./show-errors.sh e0502
// error[E0502]: cannot borrow `buf[_]` as immutable because it is also
// borrowed as mutable
//
// `Lines::new(&mut buf)` lends the WHOLE array to `lines` for as long as
// `lines` is in use, and it is used again on the next line. Reading buf[0]
// in between would read bytes that `next_line` may be moving.
//
// FIX 1: read through `lines`: the line it returned is the part of `buf`
// you may look at, and it is a slice of the same bytes.
// FIX 2: finish with `lines` first. After its last use, `buf` is yours
// again, and `buf[0]` compiles.
struct Lines<'b> {
buf: &'b mut [u8],
start: usize,
}

impl<'b> Lines<'b> {
fn new(buf: &'b mut [u8]) -> Lines<'b> {
Lines { buf, start: 0 }
}

fn next_line(&mut self) -> Option<&[u8]> {
let nl = self.buf[self.start..].iter().position(|&b| b == b'\n')?;
let from = self.start;
self.start += nl + 1;
Some(&self.buf[from..from + nl])
}
}

fn main() {
let mut buf = *b"boot\nsh\n";
let mut lines = Lines::new(&mut buf);
let first = lines.next_line().map(|l| l.len());
println!("buf[0] is {}", buf[0]);
let second = lines.next_line().map(|l| l.len());
println!("{first:?} then {second:?}");
}
Why, and 2 fixes e

Lines::new(&mut buf) lends the WHOLE array to lines for as long as lines is in use, and it is used again on the next line. Reading buf[0] in between would read bytes that next_line may be moving.

  1. read through lines: the line it returned is the part of buf you may look at, and it is a slice of the same bytes.
  2. finish with lines first. After its last use, buf is yours again, and buf[0] compiles.
$ rustc --edition 2021 --emit=metadata -o /dev/null e0502_read_buf_while_lines_has_it.rs
error[E0502]: cannot borrow `buf[_]` as immutable because it is also borrowed as mutable
--> e0502_read_buf_while_lines_has_it.rs:34:30
|
32 | let mut lines = Lines::new(&mut buf);
| -------- mutable borrow occurs here
33 | let first = lines.next_line().map(|l| l.len());
34 | println!("buf[0] is {}", buf[0]);
| ^^^^^^ immutable borrow occurs here
35 | let second = lines.next_line().map(|l| l.len());
| ----- mutable borrow later used here
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0502`.
exit status 1

E0506 cannot assign to buf[_] because it is borrowed

writing buf[5] while lines has the buffer · fix: edit a copy, or write after the last use of lines

$ ./show-errors.sh e0506
// error[E0506]: cannot assign to `buf[_]` because it is borrowed
//
// The same loan as e0502, written to instead of read. `lines` holds `buf`
// until its last use, and `next_line` trusts that the bytes it left in the
// buffer are still there. A write in between would change a line that
// `lines` has not handed out yet.
//
// FIX 1: change the line you were given, in an array of your own: copy the
// bytes out, then edit the copy.
// FIX 2: finish with `lines` first. After its last use, `buf` is yours to
// write again.
struct Lines<'b> {
buf: &'b mut [u8],
start: usize,
}

impl<'b> Lines<'b> {
fn new(buf: &'b mut [u8]) -> Lines<'b> {
Lines { buf, start: 0 }
}

fn next_line(&mut self) -> Option<&[u8]> {
let nl = self.buf[self.start..].iter().position(|&b| b == b'\n')?;
let from = self.start;
self.start += nl + 1;
Some(&self.buf[from..from + nl])
}
}

fn main() {
let mut buf = *b"boot\nsh\n";
let mut lines = Lines::new(&mut buf);
let first = lines.next_line().map(|l| l.len());
buf[5] = b'S';
let second = lines.next_line().map(|l| l.len());
println!("{first:?} then {second:?}, and buf[5] is {}", buf[5]);
}
Why, and 2 fixes e

The same loan as e0502, written to instead of read. lines holds buf until its last use, and next_line trusts that the bytes it left in the buffer are still there. A write in between would change a line that lines has not handed out yet.

  1. change the line you were given, in an array of your own: copy the bytes out, then edit the copy.
  2. finish with lines first. After its last use, buf is yours to write again.
$ rustc --edition 2021 --emit=metadata -o /dev/null e0506_write_buf_while_lines_has_it.rs
error[E0506]: cannot assign to `buf[_]` because it is borrowed
--> e0506_write_buf_while_lines_has_it.rs:34:5
|
32 | let mut lines = Lines::new(&mut buf);
| -------- `buf[_]` is borrowed here
33 | let first = lines.next_line().map(|l| l.len());
34 | buf[5] = b'S';
| ^^^^^^^^^^^^^ `buf[_]` is assigned to here but it was already borrowed
35 | let second = lines.next_line().map(|l| l.len());
| ----- borrow later used here
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0506`.
exit status 1

E0599 no method named chars found for reference &[u8] in the current scope

.chars() on a &[u8] · fix: stay in bytes, or from_utf8 first

$ ./show-errors.sh e0599
// error[E0599]: no method named `chars` found for reference `&[u8]` in the
// current scope
//
// `chars` decodes UTF-8, so it lives on `str`, whose bytes are PROMISED to
// be valid UTF-8. A `&[u8]` promises nothing: it might be half of a ñ, or a
// disk block. There is no char to hand back until somebody checks.
//
// FIX 1: stay in bytes: `line.first().is_some_and(|b| b.is_ascii_uppercase())`.
// Every command this week works this way.
// FIX 2: check first: `std::str::from_utf8(line)` returns a Result, and the
// `&str` inside an `Ok` has `.chars()`.
fn starts_upper(line: &[u8]) -> bool {
line.chars().next().is_some_and(|c| c.is_uppercase())
}

fn main() {
println!("{}", starts_upper(b"Boot ok"));
}
Why, and 2 fixes e

chars decodes UTF-8, so it lives on str, whose bytes are PROMISED to be valid UTF-8. A &[u8] promises nothing: it might be half of a ñ, or a disk block. There is no char to hand back until somebody checks.

  1. stay in bytes: line.first().is_some_and(|b| b.is_ascii_uppercase()). Every command this week works this way.
  2. check first: std::str::from_utf8(line) returns a Result, and the &str inside an Ok has .chars().
$ rustc --edition 2021 --emit=metadata -o /dev/null e0599_chars_on_bytes.rs
error[E0599]: no method named `chars` found for reference `&[u8]` in the current scope
--> e0599_chars_on_bytes.rs:13:10
|
13 | line.chars().next().is_some_and(|c| c.is_uppercase())
| ^^^^^ method not found in `&[u8]`
error: aborting due to 1 previous error
For more information about this error, try `rustc --explain E0599`.
exit status 1