<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[gn000q's Dev Blog]]></title><description><![CDATA[gn000q's Dev Blog]]></description><link>https://gn000q.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Fri, 20 Nov 2026 19:26:42 GMT</lastBuildDate><atom:link href="https://gn000q.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Zero-Allocation PII Redaction in Go: Processing 787MB of Logs in Under 4 Minutes]]></title><description><![CDATA[A deep dive into building a zero-allocation PII redaction engine in Go that processes 780MB logs in under 3 minutes, with multi-jurisdiction complianc]]></description><link>https://gn000q.hashnode.dev/zero-allocation-pii-redaction-in-go-processing-787mb-of-logs-in-under-4-minutes</link><guid isPermaLink="true">https://gn000q.hashnode.dev/zero-allocation-pii-redaction-in-go-processing-787mb-of-logs-in-under-4-minutes</guid><dc:creator><![CDATA[gn000q]]></dc:creator><pubDate>Sun, 12 Apr 2026 16:08:37 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/69dbae8faadf1107e2efa59e/b6b7870c-e4bd-438a-af5d-5f35532d7adf.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Every team feeding logs to LLMs has the same dirty secret: those logs are full of emails, IP addresses, credit card numbers, and government IDs. I know because I built a tool to find them.</p>
<p>After scanning ~780MB of production logs, I found thousands of PII instances — emails, IPs, phone numbers — all sitting in plain text, waiting to be piped into ChatGPT or fine-tuning datasets.</p>
<p>So I built a local-first PII redaction engine in pure Go. No cloud. No API keys. No telemetry. This post breaks down the engineering decisions that made it fast.</p>
<hr />
<h2 id="heading-the-problem-pii-leaks-in-ai-pipelines">The Problem: PII Leaks in AI Pipelines</h2>
<p>The AI workflow looks like this:</p>
<pre><code>Production Logs → Pre-processing → LLM API / Fine-tuning
</code></pre><p>The gap is between step 1 and step 2. Most teams skip sanitization because:</p>
<ol>
<li><strong>Cloud DLP services</strong> (Google, AWS Macie) require uploading your data — defeating the purpose</li>
<li><strong>Python-based tools</strong> (Presidio, scrubadub) are slow on large log files and need heavy dependencies</li>
<li><strong>Manual regex</strong> is fragile and doesn't handle context (is <code>1.2.3.4</code> an IP or a version number?)</li>
</ol>
<p>I needed something that could:</p>
<ul>
<li>Process 787MB in under 4 minutes on a single machine</li>
<li>Run <strong>100% offline</strong> — no network calls, ever</li>
<li>Handle 11+ PII types across 7 jurisdictions (GDPR, HIPAA, CCPA, PIPL, APPI, PDPA)</li>
<li>Produce consistent tokenization for AI training (<code>user@test.com</code> → <code>[EMAIL_0001]</code> everywhere)</li>
</ul>
<hr />
<h2 id="heading-architecture-why-go-and-why-zero-allocation">Architecture: Why Go, and Why Zero-Allocation</h2>
<p>Go was chosen for one reason: <strong>predictable memory behavior at high throughput</strong>. No GC pauses, no JIT warmup, no pip dependency hell.</p>
<p><strong>CLI / GUI Entry</strong>
→ Fyne GUI (drag &amp; drop) | CLI Mode (batch processing)
→ <strong>Compliance Profiles</strong> (PIPL / GDPR / CCPA / HIPAA / APPI / PDPA)
→ <strong>Core Engine</strong> — pure <code>[]byte</code> pipeline:
  <code>PreFilter → Regex → Validate → Tokenize → Write</code>
  powered by <code>sync.Pool</code> · lock-free stats · streaming I/O</p>
<p>The engine never converts <code>[]byte</code> to <code>string</code> in the hot path. Here's why that matters:</p>
<h3 id="heading-trick-1-prefilter-byte-probes">Trick 1: PreFilter Byte Probes</h3>
<p>Before running regex (expensive), every line passes through a cheap byte probe:</p>
<pre><code class="lang-go"><span class="hljs-keyword">type</span> Pattern <span class="hljs-keyword">struct</span> {
    ID        <span class="hljs-keyword">string</span>
    Name      <span class="hljs-keyword">string</span>
    Regex     *regexp.Regexp
    PreFilter <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(line []<span class="hljs-keyword">byte</span>)</span> <span class="hljs-title">bool</span>  // ← <span class="hljs-title">fast</span> <span class="hljs-title">reject</span></span>
    Validate  <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(match []<span class="hljs-keyword">byte</span>)</span> <span class="hljs-title">bool</span> // ← <span class="hljs-title">context</span>-<span class="hljs-title">aware</span></span>
}
</code></pre>
<p>For example, the email pattern's PreFilter just checks if the line contains <code>@</code>:</p>
<pre><code class="lang-go">PreFilter: <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(line []<span class="hljs-keyword">byte</span>)</span> <span class="hljs-title">bool</span></span> {
    <span class="hljs-keyword">return</span> bytes.ContainsRune(line, <span class="hljs-string">'@'</span>)
}
</code></pre>
<p><strong>Result</strong>: ~80% of lines are skipped before regex runs. On a 787MB server log, this cuts significant processing time.</p>
<h3 id="heading-trick-2-syncpool-buffer-reuse">Trick 2: sync.Pool Buffer Reuse</h3>
<p>Every output line needs a buffer. Allocating and GC'ing millions of buffers kills throughput:</p>
<pre><code class="lang-go"><span class="hljs-keyword">var</span> bufPool = sync.Pool{
    New: <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span> <span class="hljs-title">interface</span></span>{} {
        b := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">byte</span>, <span class="hljs-number">0</span>, <span class="hljs-number">4096</span>)
        <span class="hljs-keyword">return</span> &amp;b
    },
}

<span class="hljs-comment">// In hot loop:</span>
bp := bufPool.Get().(*[]<span class="hljs-keyword">byte</span>)
buf := (*bp)[:<span class="hljs-number">0</span>] <span class="hljs-comment">// reset length, keep capacity</span>
<span class="hljs-comment">// ... write to buf ...</span>
bufPool.Put(bp) <span class="hljs-comment">// return to pool</span>
</code></pre>
<p><strong>Result</strong>: heap allocations drop from millions to ~50. GC pressure essentially zero.</p>
<h3 id="heading-trick-3-context-aware-validation">Trick 3: Context-Aware Validation</h3>
<p>The regex for IPv4 (<code>\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b</code>) matches version numbers like <code>1.2.3.4</code> and file paths like <code>data.2024.01.15</code>. The Validate callback handles this:</p>
<pre><code class="lang-go">Validate: <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(match []<span class="hljs-keyword">byte</span>)</span> <span class="hljs-title">bool</span></span> {
    <span class="hljs-comment">// Reject if preceded by "version", "v", "=" etc.</span>
    <span class="hljs-comment">// Reject if all octets &gt; 255</span>
    <span class="hljs-comment">// Reject if it looks like a date pattern</span>
    <span class="hljs-keyword">return</span> isLikelyIP(match)
}
</code></pre>
<p>This eliminates the vast majority of false positives without sacrificing recall.</p>
<h3 id="heading-trick-4-rwmutex-tokenization">Trick 4: RWMutex Tokenization</h3>
<p>For AI training data, you need consistent tokens: the same email should always map to <code>[EMAIL_0001]</code>. The tokenizer uses a read-write split:</p>
<pre><code class="lang-go"><span class="hljs-keyword">type</span> Tokenizer <span class="hljs-keyword">struct</span> {
    mu     sync.RWMutex
    tokens <span class="hljs-keyword">map</span>[<span class="hljs-keyword">string</span>]<span class="hljs-keyword">string</span>
    counts <span class="hljs-keyword">map</span>[<span class="hljs-keyword">string</span>]<span class="hljs-keyword">int</span>
}

<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(t *Tokenizer)</span> <span class="hljs-title">GetToken</span><span class="hljs-params">(typ, value <span class="hljs-keyword">string</span>)</span> <span class="hljs-title">string</span></span> {
    t.mu.RLock()
    <span class="hljs-keyword">if</span> tok, ok := t.tokens[key]; ok {
        t.mu.RUnlock()
        <span class="hljs-keyword">return</span> tok  <span class="hljs-comment">// fast path: read-only</span>
    }
    t.mu.RUnlock()

    t.mu.Lock()
    <span class="hljs-comment">// ... create new token ...</span>
    t.mu.Unlock()
    <span class="hljs-keyword">return</span> newToken  <span class="hljs-comment">// slow path: only for first occurrence</span>
}
</code></pre>
<p>In real logs, PII values repeat heavily. The RLock fast path handles ~95% of lookups with zero contention.</p>
<hr />
<h2 id="heading-benchmark-787mb-production-log">Benchmark: 787MB Production Log</h2>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Metric</td><td>Value</td></tr>
</thead>
<tbody>
<tr>
<td>Input size</td><td>787.3 MB</td></tr>
<tr>
<td>PII instances found</td><td>11,583,245</td></tr>
<tr>
<td>Processing time</td><td>3 min 38 sec</td></tr>
<tr>
<td>Throughput</td><td>~3.6 MB/s</td></tr>
<tr>
<td>PII types detected</td><td>Email (4.6M), IPv4 (4.6M), MAC (2.3M)</td></tr>
</tbody>
</table>
</div><p>Run on a consumer laptop (TOSHIBA). Pure Go, single binary, no GPU required.</p>
<hr />
<h2 id="heading-multi-jurisdiction-compliance">Multi-Jurisdiction Compliance</h2>
<p>The tool ships with 7 compliance profiles, each enabling only the PII patterns required by that jurisdiction:</p>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Profile</td><td>Jurisdiction</td><td>What It Catches</td></tr>
</thead>
<tbody>
<tr>
<td><code>default</code></td><td>Full scan</td><td>All 11 pattern types</td></tr>
<tr>
<td><code>pipl</code></td><td>China (PIPL)</td><td>ID Card, CN Mobile, Email, IPv4</td></tr>
<tr>
<td><code>gdpr</code></td><td>EU (GDPR)</td><td>Email, IPv4/v6, Credit Card</td></tr>
<tr>
<td><code>ccpa</code></td><td>California (CCPA)</td><td>Email, IP, Phone, Credit Card, SSN</td></tr>
<tr>
<td><code>hipaa</code></td><td>US Medical (HIPAA)</td><td>Email, Phone, SSN, IPv4</td></tr>
<tr>
<td><code>appi</code></td><td>Japan (APPI)</td><td>Email, Phone, My Number, IPv4</td></tr>
<tr>
<td><code>pdpa</code></td><td>Singapore/Thailand</td><td>Email, Phone, IPv4, ID Card</td></tr>
</tbody>
</table>
</div><p>Switch profiles with a single flag:</p>
<pre><code class="lang-bash">./pii_redactor --input server.log --profile gdpr --output clean.log
</code></pre>
<hr />
<h2 id="heading-audit-report">Audit Report</h2>
<p>Every run generates an audit report — essential for compliance documentation:</p>
<pre><code>╔══════════════════════════════════════════════════════════════╗
║           PII REDACTOR V2 — AUDIT REPORT                   ║
╚══════════════════════════════════════════════════════════════╝
  <span class="hljs-attr">Profile</span>:     Full Redaction
  <span class="hljs-attr">Duration</span>:    <span class="hljs-number">3</span>m37<span class="hljs-number">.751</span>s
  <span class="hljs-attr">Files</span>:       <span class="hljs-number">2</span> total, <span class="hljs-number">0</span> failed
──────────────────────────────────────────────────────────────
  [File <span class="hljs-number">2</span>/<span class="hljs-number">2</span>]
  <span class="hljs-attr">Input</span>:     piilog.txt
  <span class="hljs-attr">Size</span>:      <span class="hljs-number">787.3</span> MB
  <span class="hljs-attr">Encoding</span>:  UTF<span class="hljs-number">-8</span>
  <span class="hljs-attr">Duration</span>:  <span class="hljs-number">3</span>m37<span class="hljs-number">.747</span>s
  <span class="hljs-attr">Hits</span>:      <span class="hljs-number">11583224</span> total
  <span class="hljs-attr">Breakdown</span>:
    ipv4                 <span class="hljs-number">4633296</span>
    email                <span class="hljs-number">4633296</span>
    mac                  <span class="hljs-number">2316632</span>
──────────────────────────────────────────────────────────────
  Grand Total Hits: <span class="hljs-number">11583245</span>
══════════════════════════════════════════════════════════════
</code></pre><p>The tokenization map (<code>[EMAIL_0001]</code> ↔ original value) is kept in memory only during processing and never written to disk — zero data leakage by design.</p>
<hr />
<h2 id="heading-try-it">Try It</h2>
<p>The tool runs on Windows, macOS (Apple Silicon), and Linux. No dependencies, no Docker, no cloud account.</p>
<p><strong>GitHub</strong>: <a target="_blank" href="https://github.com/gn000q/pii_redactor">github.com/gn000q/pii_redactor</a></p>
<p><strong>Download pre-built binaries</strong>: <a target="_blank" href="https://gnqster.gumroad.com/l/pii-redactor-v2">PII Redactor V2 on Gumroad</a> — includes cross-platform binaries, sample test data, config templates, and a quick-start guide.</p>
<hr />
<h2 id="heading-whats-next">What's Next</h2>
<p>I'm considering adding:</p>
<ul>
<li>YAML/JSON structured log parsing (currently handles flat text)</li>
<li>Custom pattern loading from external config files</li>
<li>Streaming mode for piped input (<code>tail -f | pii_redactor</code>)</li>
</ul>
<p><strong>What does your PII cleanup workflow look like?</strong> I'd love to hear if you're dealing with similar issues — especially if you're feeding logs to AI APIs.</p>
]]></content:encoded></item></channel></rss>