On 28 December 2021, HPE Japan sent Kyoto University a two page apology letter. It opens with numbers. Between 14 December, 17:32, and 16 December, 12:43, a maintenance script deleted 34,011,293 files from /LARGE0, one of the storage volumes of the university’s supercomputer: about 77 TB. Fourteen research groups were affected. Four of them lost data that existed nowhere else. HPE stated the failure was 100% their responsibility.

No disk died, and nobody typed rm -rf. An engineer improved a backup script, then deployed the new version onto the live system. The letter’s own summary of what followed: the file was overwritten while the old version was still executing, bash partially re-read it, and a find command ran with undefined variables. Before reading further, take a minute and make a prediction. How does replacing the text of a shell script destroy 34 million files the script was never meant to touch?

This article rebuilds the chain and teaches the two pieces of systems knowledge it rests on: how an interpreted shell executes a script file, and what cp and mv actually do to a file. It is for people who use bash, cp, mv and find every week, students and junior engineers deploying scripts on machines that matter. I assume no prior knowledge of inodes or file descriptors. Veteran sysadmins already tell this story at parties and can skip to the checklist at the end. Every capture below is real output from my machine, and every experiment fits in a throwaway directory, so you can replay it all tonight.

One thing to keep straight from the start. HPE’s letter confirms the overwrite of the running script, the partial re-read by bash, and a find running with undefined variables. It never names the command that performed the overwrite. The full diagnosis below, a cp where a mv belonged plus an unquoted variable, is a reconstruction, assembled mostly in a Hacker News thread, that reproduces the published symptoms exactly. HPE has never confirmed it. The reconstructed links in the chain of events will be flagged as we reach them.

A mundane script

The letter describes the script in one sentence:

バックアップスクリプトには、find コマンドにより 10 日以上古いログファイル を削除する処理が含まれています。

“The backup script contains a step that deletes log files older than ten days using the find command.”

So the shape of the interesting part was something like this:

1
2
3
4
5

# Illustration of the cleanup step. The real script was never published;
# only the use of find, the ten day threshold and the variables are
# documented.
LOG_DIR="/LARGE0/ops/backup/logs"
find $LOG_DIR -mtime +10 -delete

Something of this shape ran unattended for a long time without hurting anyone. The change that triggered the disaster was cosmetic: per the letter, the variable handed to the deletion step was renamed for readability. Every element here is reasonable. Log rotation is hygiene, ten days is a sensible retention, clearer names are what code review keeps asking for.

How bash reads the script it is running

Here is the mental model most of us carry: a program is loaded into memory, then it runs; editing the file on disk afterwards changes the next run, never the current one. For a compiled binary, that model is close enough. But for a shell script, it is wrong. Bash opens your script, keeps the file descriptor open for the whole run, and reads commands from it as it goes, advancing its offset in the file as statements are executed. It does not read the whole file first. You can watch it do this with strace: bash read()s a chunk, parses one command, then seeks back to just after that command before executing it, so the next read starts exactly where parsing stopped (observed and discussed here).

Two versions of a small script make the mechanism visible. Version one works slowly, which leaves a window to swap the file mid-run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ mkdir -p /var/tmp/footgun && cd /var/tmp/footgun # This can be any directory
$ cat cleanup_v1.sh
#!/bin/bash
echo "v1: backup starting"
sleep 15
echo "v1: pruning old logs"
echo "v1: backup done"

$ cat cleanup_v2.sh
#!/bin/bash
# v2 of the cleanup script, now with better variable names
echo "v2: backup starting"
echo "v2: pruning old logs"
echo "v2: backup done"

Deploy v1, run it, and while it sleeps, deploy v2 over it with cp, an in-place overwrite like the incident’s release step:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

$ cp cleanup_v1.sh cleanup.sh && chmod +x cleanup.sh
$ stat -c 'inode %i' cleanup.sh
inode 7352159
$ ./cleanup.sh & PID=$!
v1: backup starting
$ grep pos /proc/$PID/fdinfo/255
pos:	48
$ readlink /proc/$PID/fd/255
/var/tmp/footgun/cleanup.sh
$ cp cleanup_v2.sh cleanup.sh
$ stat -c 'inode %i' cleanup.sh
inode 7352159
$ wait $PID; echo "exit: $?"
./cleanup.sh: line 4: better: command not found
v2: backup starting
v2: pruning old logs
v2: backup done
exit: 0

There is a lot in that capture. Bash keeps the script it is executing on file descriptor 255, and /proc lets us inspect it: while sleep 15 runs, the offset sits at byte 48. The arithmetic checks out: #!/bin/bash\n is 12 bytes, the first echo line 27, sleep 15\n another 9, total 48. After the cp, the name cleanup.sh still leads to the same inode (7352159, more on that word in a minute), so bash’s next read returns the new file’s bytes at the old offset. Byte 48 of v2 falls inside the long comment line, right before the word better. Bash executes better variable names as a command, fails, and carries on with the rest of v2. Two versions ran in one process: v1’s head, a line of garbage, then v2’s tail. Exit code 0. In the incident, this resumed-in-the-middle execution is what HPE reported, and it ran for 43 hours.

The intuition about programs being protected while they run is still half right, and the kernel itself draws the line. Try to write into a running ELF binary:

1
2
3
4
5
6
7
8

$ cp /usr/bin/sleep ./mysleep
$ ./mysleep 30 &
[1] 414478
$ echo "overwrite attempt" > ./mysleep
bash: ./mysleep: Text file busy
$ cp /usr/bin/sleep ./mysleep
cp: cannot create regular file './mysleep': Text file busy
$ kill %1

ETXTBSY, “text file busy”: Linux refuses to modify a binary that is being executed. This protection would have stopped the faulty deployment: cp exits with an error. Scripts get nothing of the sort, because as far as the kernel is concerned the thing being executed is /bin/bash and your script is just a file that bash happens to be reading.

What a file really is

To see why mv would have been safe where cp was fatal, we have to drop one level of abstraction. On a Unix filesystem, the name of a file is not the file. A directory entry maps a name to an inode number. The inode is the actual bookkeeping record: owner, permissions, timestamps, and pointers to the data blocks that hold the content. The name is one label stuck on that record, and there can be several (hard links), or, as we will see, zero.

This article will only explain the relevant parts for the story but you can check out Ethan Perruzza’s article Why rm doesn’t truly delete files if you want an in depth walk through of ext4.

The part that matters here is that open() resolves a name to an inode once, at open time. From then on, the file descriptor refers to the inode directly. So it does not care if the file is renamed or unlinked and it keeps reading the same inode. The kernel frees an inode only when no name links to it and no process holds it open. This is why a program can keep appending to a log that was deleted an hour ago, and why disk space sometimes refuses to come back until a process exits.

Two rules, then. Names point to inodes. File descriptors pin inodes, with or without a name. The next demo shows both live.

cp and mv at the inode level

cp cleanup_v2.sh cleanup.sh onto an existing destination creates nothing. It opens cleanup.sh with O_TRUNC, cuts the content to zero length, and writes v2’s bytes into the same inode. Every descriptor already pointing at that inode, bash’s fd 255 included, sees the new content. mv on the same filesystem is a different operation entirely: it is a single rename(2) call, which repoints the directory entry to v2’s inode and leaves the old inode untouched. The kernel guarantees the swap is atomic: a process opening the name gets the old file or the new one, never a mixture. And a process that already had the old file open keeps it, whole, until it closes it.

Same experiment as before, deploying with mv this time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ cp cleanup_v1.sh cleanup.sh
$ stat -c 'inode %i' cleanup.sh
inode 7352159
$ stat -c 'inode %i' cleanup_v2.sh
inode 7352158
$ ./cleanup.sh & PID=$!
v1: backup starting
$ mv cleanup_v2.sh cleanup.sh
$ stat -c 'inode %i' cleanup.sh
inode 7352158
$ readlink /proc/$PID/fd/255
/var/tmp/footgun/cleanup.sh (deleted)
$ wait $PID; echo "exit: $?"
v1: pruning old logs
v1: backup done
exit: 0

The name cleanup.sh now leads to inode 7352158, the new version, and the next run picks it up. Meanwhile /proc shows bash’s fd 255 pointing at a file with no name, (deleted): the old inode is pinned by the descriptor and thus lives. The running script finished v1 cleanly and exited 0. That single difference in inode numbers between the two examples is the whole story.

One caveat before you engrave “mv is atomic” anywhere is that it only holds true within one filesystem. Across filesystems, rename(2) fails with EXDEV, and mv silently falls back to copy-then-unlink.

The chain reaction

Now assemble the pieces, in the order the letter gives them:

bash は、シェルスクリプトの実行中に適時シェルスクリプトを読み込みます。 […] 実行中のスクリプトが存在している状態でスクリプトの上書きによりリリー スしてしまったことで、途中から修正したシェルスクリプトの再読み込みが発生 し、結果的に未定義の変数を含む find コマンドが実行されてしまいました。

“bash loads a shell script progressively while executing it. […] Because the release was performed by overwriting the script while it was running, the modified script was re-read from partway through, and in the end a find command containing undefined variables was executed.”

1. The new script’s bytes replaced the old ones in place, under a running bash. The overwrite is confirmed by the letter but is not said that the tool was cp specifically (an rsync, or a tar -x, would overwrite the inode just the same).
2. Bash re-read from its saved offset and resumed inside the new text, our better variable names moment, scaled up.
3. In the re-read tail, the deletion line used the new variable name. The assignments that had already executed were v1’s, under the old name. So the new name held nothing. This is the letter’s “undefined variables”.
4. An unquoted empty variable does not expand to an empty argument. It expands to no argument at all. The find process received only -mtime +10 -delete.
5. From GNU find’s manual: “If no paths are given, the current directory is used.” The working directory of the backup run sat inside /LARGE0. This last step is again reconstruction; the letter only says deletion hit /LARGE0 instead of the log directory.
6. For 43 hours, find deleted every file in reach older than ten days.

Watch steps 4 and 5 happen on a filesystem you can afford to lose:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ mkdir -p LARGE0/log LARGE0/research
$ touch -d '30 days ago' LARGE0/log/old.log LARGE0/research/precious_data.dat
$ touch LARGE0/research/recent.dat
$ cd LARGE0
$ unset LOG_DIR
$ find $LOG_DIR -mtime +10 -print
./log/old.log
./research/precious_data.dat
$ echo "exit: $?"
exit: 0

While precious_data.dat is outside the log directory, find listed it anyway, and reported success. The quotes would have changed everything:

1
2
3
4

$ find "$LOG_DIR" -mtime +10 -print
find: ‘’: No such file or directory
$ echo "exit: $?"
exit: 1

A quoted empty variable stays an argument: find receives an empty path, fails loudly, exits 1. With -delete in place of -print, the unquoted version goes through the same files in silence:

1
2
3
4
5

$ find $LOG_DIR -mtime +10 -delete
$ echo "exit: $?"
exit: 0
$ find . -type f
./research/recent.dat

The old research file is gone and the recent one survives. This brings us to the strongest corroboration of the reconstruction. -mtime +10 selects files at least 11 full days old, so a run started on 14 December at 17:32 draws the line at 3 December, 17:32. HPE’s letter states the deleted files were those not modified since, to the minute, 3 December 17:32. The published victim set is exactly what find -mtime +10, launched at the published start time, would select. It also rules out the popular “it deleted / instead of /log” retelling: everything outside /LARGE0 survived, and inside it, everything younger than the boundary did too.

How not to lose 77 TB

Every item below removes one link from the chain; the first two alone would have prevented the incident.

• Replace live scripts with mv, never with cp. Stage the new version in the same directory (same filesystem, so rename(2) applies), then mv it onto the name. Running readers keep the old inode, the next run gets the new one. install(1) is safe too: it unlinks the destination before writing (verified with strace on coreutils 9.11).
• Quote every expansion, "$LOG_DIR", even when you know it is set. A quoted empty variable is an argument that fails fast; an unquoted one is a disappearing argument that changes the meaning of the command around it.
• Make absence fatal where it matters:

1
2
3
4
5
6
7

$ cat guard.sh
#!/bin/bash
find "${LOG_DIR:?}" -mtime +10 -print
$ bash guard.sh
guard.sh: line 2: LOG_DIR: parameter null or not set
$ echo "exit: $?"
exit: 1

• Starting scripts with set -u makes bash fail when unset variables are met instead of becoming empty strings. Note that while it guards the expansion half of this incident, it does nothing against the re-read half.
• A synchronized mirror is not a backup. /LARGE0 had backups on /LARGE1, 49 TB of which survived for ten research groups. Four groups still lost everything, because the thing that failed was the backup pipeline itself. Copies that protect you from your own tooling are versioned, delayed, or offline.

Takeaway

Nothing in this story is exotic. Bash reading scripts incrementally is documented behavior older than most of its victims. cp writing into the destination inode is what O_TRUNC has always meant. find starting at . is in the manual. Renaming a variable for readability is good practice, and shipping the improved script is the job. Each step defensible, and 34 million files gone: the disaster lives entirely in the interaction between decisions that were individually sound. That is what makes it worth studying, and what makes “they were negligent” the wrong lesson to leave with. The right one fits in one sentence: a deployed script file is shared mutable state, and the two-letter difference between cp and mv decides whether a running reader observes the mutation.

Said once more, because it should survive every retelling: HPE confirmed the overwrite and the undefined variables. The cp and the missing quotes are the community’s reconstruction, unusually well supported by the published numbers, and still unconfirmed.

If you want to keep pulling the thread: run strace -e read,lseek bash yourscript.sh and watch the offset dance syscall by syscall; read how ETXTBSY is enforced in the kernel, and why interpreters get no equivalent; and Lustre itself, the filesystem serving petabytes to thousands of nodes under the same inode rules, deserves a dive of its own.

Sources and further reading

Primary sources:

• HPE Japan’s apology letter to Kyoto University (PDF, Japanese, 28 December 2021; archived). All incident figures and quotes in this article come from it.

Community reconstruction:

• The Hacker News thread where the mechanism was pieced together, including the strace of bash’s incremental reads, the pre-mmap history of this behavior and editor save strategies.
• A video retelling of the incident if you prefer the animated version.

Reference material:

• find(1) man page (man find): “If no paths are given, the current directory is used.”
• rename(2) and open(2) (O_TRUNC) man pages.
• Why rm doesn’t truly delete files, by Ethan Perruzza on this blog, for the on-disk side of inodes and deletion.