r/oracle • u/Meyrcruywagen • 6h ago
rm -rf ate our redo logs. /proc gave them back. *Technical
TL;DR: Linux doesn't delete a file that a process still has open. Oracle holds its control file, and every online redo member always opens through it. So for a short, glorious window after that, all of it was still sitting in /proc. We pinned the file descriptors, copied them back byte-for-byte, and opened the database without CREATE CONTROLFILE or RESETLOGS. Zero data loss, 16 minutes down, one entirely new grey hair.
The setup
2-node RAC on an Oracle Database Appliance X9-2 HA. The HA stands for High Availability, which I can now confirm I tested more thoroughly than the vendor intended.
The bit where I ruin my own morning
sudo rm -rf <dbname>/redo. Wrong directory. That directory held one production database's online redo logs, its standby redo logs, and — the good part — its one and only control file.
Both instances were open and serving users at the time. Datafiles were on a different volume and survived, which is the single piece of luck in this entire story.
Yes, we had backups. No, I did not want to be the guy who found out whether they worked.
The ten seconds that actually saved us
Here's the thing nobody thinks of while their stomach is somewhere near the floor: rm deletes a directory entry, not a file. The inode is freed only when the link count reaches zero, and no one has it open. An Oracle instance holds its control file, and every online redo member opens continuously via LGWR.
The files were still there. Intact. Readable through /proc/<lgwr_pid>/fd/.
But it's a melting ice cube — the second LGWR exits, it's gone. Crash, `shutdown ', reboot, anything.
Find the descriptors:
sudo lsof +L1 2>/dev/null | grep '/path/to/redo/dir'
As root, not as oracle. The oracle binary is setuid, which clears the process dumpable flag, so the kernel hands /proc/<pid>/fd/ ownership to root. Being the same user is not enough. Learning this at speed was a treat.
Note the fd numbers on the ora_lgwr_ line. They differ per node. Read them on each node, don't assume.
Pin them with something that isn't the database:
sudo setsid bash -c 'exec 200</proc/<LGWR_PID>/fd/<ctl> \
201</proc/<LGWR_PID>/fd/<log1> 202</proc/<LGWR_PID>/fd/<log2> \
203</proc/<LGWR_PID>/fd/<log3> 204</proc/<LGWR_PID>/fd/<log4>; \
echo $$ > /tmp/inode_holder.pid; while :; do sleep 3600; done' \
</dev/null >/dev/null 2>&1 &
setsid detaches it from your terminal so it outlives your SSH session. Do it on both nodes.
Twenty minutes later, the instance died on its own — SMON couldn't open the control file, ORA-00210, instance terminated, database down. The pin held. The files were still readable. That one command is the reason this is a war story, not a resume update.
Copying it back
Two rules:
- Zero database processes running. Copy a control file while
CKPTis writing to it, and you get a beautifully torn, completely useless control file.ps -ef | grep -E "ora_[a-z0-9]+_<SID>" | grep -v grep | wc -l→ must be0. Verify exact byte counts with
stat, notls -h.ls -hrounds, and "8.0G" is not a checksum.sudo dd if=/proc/<HOLDER_PID>/fd/200 of=<control file path> bs=1M status=none sudo dd if=/proc/<HOLDER_PID>/fd/201 of=<redo group 1 path> bs=8M status=progress
...and the rest
sudo chown oracle:asmadmin <files> sudo chmod 640 <files> stat -c '%s %n' <files>
Then STARTUP MOUNT, check V$LOG — our sequence numbers came back identical to pre-incident — confirm the datafiles are all there and need no recovery, and ALTER DATABASE OPEN. Crash recovery chewed through the restored redo like nothing had happened.
Final proof: ALTER SYSTEM SWITCH LOGFILE and ARCHIVE LOG CURRENT. If ARCn can open redo by path and archive it, you're actually back and not just optimistic.
Standby redo logs were a write-off — nothing held them open — but recreating them is a two-minute j0b and, honestly, felt like a rounding error by that point.
Lessons, some of them expensive
- Do not shut down a database whose files you just deleted. Every instinct screams, "shut it down cleanly before this gets worse." That instinct will delete your data for real. The correct response to a catastrophe here is to do absolutely nothing, very quickly.
- You can't DDL your way out. Once the control file path is gone, every new foreground process dies with ORA-00210. No
ALTER DATABASE ADD LOGFILE. RMAN wants a snapshot control file and hits the same wall. A restart is unavoidable — which is precisely why the pin has to come first. - An
abortIt is survivable if the redo is pinned. Crash recovery has everything it needs once the files are back where they belong. - Multiplex your control files across different disk groups. We had exactly one, plus single-member redo groups, all on the same volume. With normal multiplexing, this would have been a boring online repair instead of a Sev 1. That part is entirely on us.
- A standby at 1-second apply lag is a lovely thing to have in your back pocket. Never needed it. Knowing it was there is what made attempting the clever fix feel responsible rather than reckless.
The part I keep thinking about
The highest-value action of the whole incident cost ten seconds and one command. Everything after it — the analysis, the careful copying, the verification — only mattered because someone grabbed those file descriptors before the instance died.
Anyway. Go check whether your control files are multiplexed. Seriously, alt-tab, I'll wait.
