Add anonymizer tool

.gitignore

@@ -2,3 +2,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 
 target
+
+# Local test PDF fixtures (not committed to repo; place plain PDFs here for offline tests)
+anonymizer_data/*.pdf

Cargo.toml

@@ -20,6 +20,9 @@ exclude = [
 [[bin]]
 name = "etradeTaxReturnHelper"
 path = "src/main.rs"
+[[bin]]
+name = "etradeAnonymizer"
+path = "src/anonymizer/anonymizer.rs"
 
 [[bin]]
 name = "gen_exchange_rates"
@@ -49,5 +52,4 @@ polars = "0.36.2"
 csv = "1.3.0"
 serde_json = { version = "=1.0.133", optional = true }
 holidays = { version = "0.1.0", default-features = false, features = ["PL"] }
-
-
+flate2 = "1.1.5"

README.md

@@ -1,5 +1,5 @@
 <!--
-SPDX-FileCopyrightText: 2022-2025 RustInFinance
+SPDX-FileCopyrightText: 2022-2026 RustInFinance
 SPDX-License-Identifier: BSD-3-Clause
 -->
 

src/anonymizer/README.md

@@ -0,0 +1,115 @@
+<!--
+SPDX-FileCopyrightText: 2025 RustInFinance
+SPDX-License-Identifier: BSD-3-Clause
+-->
+
+# etradeAnonymizer
+
+Minimal Rust tool for anonymizing E*TRADE / Morgan Stanley PDF statements by replacing personally identifiable information while preserving calculation-relevant data.
+
+## Usage
+
+Anonymize a PDF (creates `anonymous_statement.pdf` by default):
+```
+cargo run --bin etradeAnonymizer -- anonymize statement.pdf
+```
+
+Specify output file:
+```
+cargo run --bin etradeAnonymizer -- anonymize input.pdf output.pdf
+```
+
+List all text tokens (for debugging):
+```
+cargo run --bin etradeAnonymizer -- list statement.pdf
+```
+
+## Build & Test
+```
+cargo build --release --bin etradeAnonymizer
+# Tests live in the library target (there are no tests in the binary itself):
+cargo test --lib anonymizer
+```
+
+Resulting binary: `target/release/etradeAnonymizer`.
+
+## Anonymization Strategy
+
+The tool processes PDF FlateDecode streams and applies smart text replacement:
+
+1. **Preserve calculation-relevant strings**:
+   - Strings containing `CLIENT STATEMENT` or `For the Period`
+   - Everything between `CASH FLOW ACTIVITY BY DATE` and `NET CREDITS/(DEBITS)` (inclusive)
+
+2. **Replace all other text**:
+   - The current implementation replaces non-preserved strings with stable numeric tokens (`0`, `1`, `2`, ...).
+   - This removes PII while keeping structure and relative token identity for verification and testing.
+
+## Technical Specification (YAGNI Scope)
+
+This tool follows the **YAGNI (You Ain't Gonna Need It)** principle, focusing only on the subset of the PDF standard actually used in the target documents.
+
+### Supported Features
+- **PDF Standard:** Basic PDF 1.3 structure.
+- **Streams:** `stream` objects compressed with `/FlateDecode` or raw (uncompressed) with an explicit `/Length`.
+- **Text Blocks:** Data contained between `BT` (Begin Text) and `ET` (End Text) operators.
+- **Text Operators:** Extraction from `(...) Tj` and `[...] TJ` arrays.
+- **Unescaping:** Support for standard PDF escape sequences: `\n`, `\r`, `\t`, `\b`, `\f`, `\(`, `\)`, `\\`, and octal sequences `\ddd`.
+- **Encoding:** Text treated as ASCII/Latin-1 (internally handled as UTF-8).
+
+
+## Design Notes
+- **Regex-based Discovery:** The tool uses optimized regular expressions to scan the PDF binary for stream object headers. This allows for fast location of data without full PDF structure parsing.
+- Strict PDF header (`%PDF-1.3`) enforcement; files with any other header are rejected.
+- FlateDecode and uncompressed streams with an explicit `/Length` are processed.
+- Replacement recompresses; if no level fits original size, original compressed stream is kept.
+
+## Testing & Development
+
+### Running Tests
+Most unit tests run automatically with `cargo test`. Integration tests that require real PDF files are marked `#[ignore]` and will not run on CI.
+
+To run the integration tests locally:
+1. Place the plain PDF fixtures (`sample_statement.pdf`, `sample_statement_anonymized.pdf`) in the `anonymizer_data/` directory (git-ignored).
+2. Run:
+   ```bash
+   cargo test --lib anonymizer -- --ignored
+   # or without GUI dependencies:
+   cargo test --lib anonymizer --no-default-features -- --ignored
+   ```
+
+### Known Limitations (YAGNI Scope)
+- **Indirect Length Objects:** The scanner currently only supports streams with an explicit numeric `/Length` in the dictionary. It will skip streams where the length is a reference (e.g., `/Length 12 0 R`).
+- **Standard PDF Filters:** Only `/FlateDecode` is supported for text extraction. Image streams (e.g., `/DCTDecode`) and font files are intentionally ignored as they do not contain PII in target documents.
+
+### Why Padding? (Architecture Note)
+This tool avoids full PDF parsing and rebuilding. Instead, it modifies streams **in-place**.
+- PDF files rely on a Cross-Reference (XREF) table that stores the byte offset of every object.
+- If we changed the length of a stream object, all subsequent object offsets would shift, invalidating the XREF table.
+- To avoid rebuilding the XREF table, we ensure the modified stream is **exactly the same length** as the original.
+- If the new compressed data is smaller, we **pad** the remainder with null bytes (`0x00`).
+- If the new compressed data is larger than the original, we fall back to keeping the original stream to avoid file corruption.
+
+### Exact PDF object pattern searched
+The tool searches for PDF objects that exactly match the following pattern:
+
+```
+<number> <number> obj
+<<
+/Length <number>
+/Filter [/FlateDecode]
+>>
+stream
+<exactly Length bytes>
+endstream
+endobj
+```
+
+Note about `list` showing "0 tokens": the command always prints a stream dump header, but the token extractor only reports tokens when it finds PDF text-showing operators (e.g. `Tj`, `TJ`, `\'`, `"`). Non-text streams (images, fonts, etc.) will naturally show "0 tokens".
+
+## License
+See `BSD-3-Clause` in `LICENSES/` directory.
+
+## Disclaimer
+
+Please note: this tool attempts to detect and replace everything except the information required for calculation by analyzing tokens in PDF streams that are strictly defined, but there is no guarantee that all PII will be detected or removed. You must manually review the resulting file before sharing it.

src/anonymizer/anonymizer.rs

@@ -0,0 +1,67 @@
+// SPDX-FileCopyrightText: 2025 RustInFinance
+// SPDX-License-Identifier: BSD-3-Clause
+
+//! etradeAnonymizer - PDF anonymization tool for E*TRADE / Morgan Stanley statements.
+//!
+//! This tool provides two subcommands:
+//! - `list`: List all text tokens from FlateDecode streams in a PDF
+//! - `anonymize`: Anonymize PDF by replacing all PII except calculation-relevant data
+//!
+//! The tool operates on tightly structured PDF FlateDecode streams and preserves
+//! the original file structure by performing in-place replacements with exact-size matching.
+
+// Submodules are exported by `src/anonymizer/mod.rs` via the library crate.
+// Since this file is a binary entry point, reference them via `etradeTaxReturnHelper::anonymizer::...`.
+
+use clap::{Parser, Subcommand};
+use etradeTaxReturnHelper::anonymizer;
+use std::error::Error;
+use std::path::PathBuf;
+
+/// Tool for anonymizing PDF files by replacing specific strings in FlateDecode streams
+#[derive(Parser)]
+#[command(name = "etradeAnonymizer")]
+#[command(version, about, long_about = None)]
+struct Cli {
+    #[command(subcommand)]
+    command: Commands,
+}
+
+#[derive(Subcommand)]
+enum Commands {
+    /// List all text tokens from FlateDecode streams in the PDF
+    List {
+        /// Path to the input PDF file
+        input_file: PathBuf,
+    },
+    /// Anonymize PDF by replacing all text except calculation-relevant data
+    Anonymize {
+        /// Path to the input PDF file
+        input_file: PathBuf,
+        /// Path to the output PDF file (optional, defaults to anonymous_<input>.pdf)
+        output_file: Option<PathBuf>,
+    },
+}
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // Default to `warn` level; RUST_LOG env var overrides this if set.
+    simple_logger::SimpleLogger::new()
+        .with_level(log::LevelFilter::Warn)
+        .env()
+        .init()
+        .unwrap();
+
+    let cli = Cli::parse();
+
+    match cli.command {
+        Commands::List { input_file } => anonymizer::list::list_texts(&input_file),
+        Commands::Anonymize {
+            input_file,
+            output_file,
+        } => {
+            let output =
+                output_file.unwrap_or_else(|| anonymizer::path::anonymous_output_path(&input_file));
+            anonymizer::replace::replace_pii_smart(&input_file, &output)
+        }
+    }
+}

src/anonymizer/list.rs

@@ -0,0 +1,63 @@
+// SPDX-FileCopyrightText: 2025 RustInFinance
+// SPDX-License-Identifier: BSD-3-Clause
+
+//! Text listing module for anonymizer.
+//!
+//! This module provides functionality to extract and list all text tokens from
+//! FlateDecode and uncompressed streams (with an explicit /Length) in a PDF. Each
+//! token is printed with a global index, useful for understanding the structure and
+//! content of the PDF before anonymization.
+
+use super::pdf::{extract_texts_from_stream, read_pdf, stream_scanner};
+use log::warn;
+use std::error::Error;
+use std::path::Path;
+
+/// List all text tokens from FlateDecode and uncompressed streams in the PDF at `input_path`.
+///
+/// Prints each extracted token with a global index to stdout.
+/// Logs warnings for streams that fail to decompress or have invalid markers.
+///
+/// # Arguments
+/// * `input_path` - Path to the input PDF file.
+///
+/// # Returns
+/// `Ok(())` on success, or an error if the PDF cannot be read.
+pub fn list_texts(input_path: &Path) -> Result<(), Box<dyn Error>> {
+    let pdf_data = read_pdf(input_path)?;
+
+    let mut global_text_id = 0;
+    for (stream_id, stream) in stream_scanner(&pdf_data).enumerate() {
+        if !stream.valid_end_marker {
+            warn!(
+                "Skipping stream due to end-marker mismatch for object at {}",
+                stream.object_start
+            );
+            continue;
+        }
+
+        let extraction = extract_texts_from_stream(stream.compressed, stream.is_compressed);
+
+        match extraction {
+            Ok(extracted_texts) => {
+                println!("---------------------------------------------------------------");
+                println!("STREAM #{} ({} tokens)", stream_id, extracted_texts.len());
+                println!("---------------------------------------------------------------");
+                for (txt, _start, _end) in extracted_texts.iter() {
+                    // Sanitize token for console: escape non-printable bytes to avoid
+                    // terminal mojibake when streams contain binary data.
+                    let safe: String = txt.chars().flat_map(|c| c.escape_default()).collect();
+                    println!("  [{}] {}", global_text_id, safe);
+                    global_text_id += 1;
+                }
+            }
+            Err(e) => {
+                warn!(
+                    "Failed to extract texts from stream at {}: {}",
+                    stream.object_start, e
+                );
+            }
+        }
+    }
+    Ok(())
+}

src/anonymizer/mod.rs

@@ -0,0 +1,9 @@
+// SPDX-FileCopyrightText: 2025 RustInFinance
+// SPDX-License-Identifier: BSD-3-Clause
+
+//! Anonymizer module re-exports.
+
+pub mod list;
+pub mod path;
+pub mod pdf;
+pub mod replace;

src/anonymizer/path.rs

@@ -0,0 +1,64 @@
+// SPDX-FileCopyrightText: 2025 RustInFinance
+// SPDX-License-Identifier: BSD-3-Clause
+
+//! Path utility module for anonymizer.
+//!
+//! Provides helper functions for generating anonymized output file paths
+//! by prefixing filenames with `anonymous_` while preserving directory structure.
+
+use std::path::PathBuf;
+
+/// Build an output path by prefixing the input filename with `anonymous_`.
+///
+/// Preserves the parent directory if present and returns a `PathBuf`.
+///
+/// # Examples
+/// ```ignore
+/// use std::path::Path;
+/// let input = Path::new("data/statement.pdf");
+/// let output = anonymous_output_path(input);
+/// assert_eq!(output, Path::new("data/anonymous_statement.pdf"));
+/// ```
+pub fn anonymous_output_path(input_path: &std::path::Path) -> PathBuf {
+    let file_name = input_path
+        .file_name()
+        .map(|s| s.to_string_lossy().into_owned())
+        .unwrap_or_else(|| input_path.to_string_lossy().into_owned());
+
+    if let Some(parent) = input_path.parent() {
+        let mut pb = PathBuf::from(parent);
+        pb.push(format!("anonymous_{}", file_name));
+        pb
+    } else {
+        PathBuf::from(format!("anonymous_{}", file_name))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_anonymous_output_path_no_parent() {
+        let in_path = std::path::Path::new("statement.pdf");
+        let out = anonymous_output_path(in_path);
+        assert_eq!(out, std::path::PathBuf::from("anonymous_statement.pdf"));
+    }
+
+    #[test]
+    fn test_anonymous_output_path_with_parent() {
+        let in_path = std::path::Path::new("some/dir/statement.pdf");
+        let out = anonymous_output_path(in_path);
+        assert_eq!(
+            out,
+            std::path::PathBuf::from("some/dir/anonymous_statement.pdf")
+        );
+    }
+
+    #[test]
+    fn test_anonymous_output_path_unicode_filename() {
+        let in_path = std::path::Path::new("résumé.pdf");
+        let out = anonymous_output_path(in_path);
+        assert_eq!(out, std::path::PathBuf::from("anonymous_résumé.pdf"));
+    }
+}