Minor update to cli_redact for new local OCR model options. Updated app_settings.qmd, user_guide.qmd, and readme.md with descriptions of new features

README.md

@@ -10,7 +10,7 @@ license: agpl-3.0
 ---
 # Document redaction
 
-version: 1.5.0
+version: 1.5.1
 
 Redact personally identifiable information (PII) from documents (pdf, png, jpg), Word files (docx), or tabular data (xlsx/csv/parquet). Please see the [User Guide](#user-guide) for a full walkthrough of all the features in the app.
     
@@ -249,7 +249,6 @@ Now you have the app installed, what follows is a guide on how to use it for bas
 - [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
 
 ### Advanced user guide
-- [Advanced user guide](#advanced-user-guide)
 - [Fuzzy search and redaction](#fuzzy-search-and-redaction)
 - [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
     - [Using _for_review.pdf files with Adobe Acrobat](#using-_for_reviewpdf-files-with-adobe-acrobat)
@@ -261,7 +260,6 @@ Now you have the app installed, what follows is a guide on how to use it for bas
 - [Merging redaction review files](#merging-redaction-review-files)
 
 ### Features for expert users/system administrators
-- [Features for expert users/system administrators](#features-for-expert-userssystem-administrators)
 - [Advanced OCR options (Hybrid OCR)](#advanced-ocr-options-hybrid-ocr)
 - [Command Line Interface (CLI)](#command-line-interface-cli)
 
@@ -376,7 +374,17 @@ If you have used the AWS Textract option for extracting text, you may also see a
 
 ![Document upload alongside Textract](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/document_upload_with_textract.PNG)
 
-Similarly, if you have used the 'Local OCR method' to extract text, you may see a '..._ocr_results_with_words.json' file. This file works in the same way as the AWS Textract .json results described above, and can be uploaded alongside an input document to save time on text extraction in future in the same way.
+#### Additional outputs in the log file outputs
+
+On the Redaction settings tab, near the bottom of the pagethere is a section called 'Log file outputs'. This section contains the following files:
+
+You may see a '..._ocr_results_with_words... .json' file. This file works in the same way as the AWS Textract .json results described above, and can be uploaded alongside an input document to save time on text extraction in future in the same way.
+
+Also you will see a 'decision_process_table.csv' file. This file contains a table of the decisions made by the app for each page of the document. This can be useful for debugging and understanding the decisions made by the app.
+
+Additionally, if the option is enabled by your system administrator, on this tab you may see an image of the output from the OCR model used to extract the text from the document, an image ending with page number and '_visualisations.jpg'. A separate image will be created for each page of the document like the one below. This can be useful for seeing at a glance whether the text extraction process for a page was successful, and whether word-level bounding boxes are correctly positioned.
+
+![Text analysis output](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/example_complaint_letter_1_textract_visualisations.jpg)
 
 ### Downloading output files from previous redaction tasks
 
@@ -686,6 +694,7 @@ You can also write open text into an input box and redact that using the same me
 ### Redaction log outputs
 A list of the suggested redaction outputs from the tabular data / open text data redaction is available on the Redaction settings page under 'Log file outputs'.
 
+
 ## Identifying and redacting duplicate pages
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/duplicate_page_find_in_app/).
@@ -916,45 +925,91 @@ AWS_SECRET_KEY= your-secret-key
 
 The app should then pick up these keys when trying to access the AWS Textract and Comprehend services during redaction.
 
-Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
-
-## Advanced OCR options (Hybrid OCR)
+## Advanced OCR options
 
-The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by your system administrator.
+The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by changing the app_config.env file in your '/config' folder, or system environment variables in your system.
 
 ### Available OCR models
 
-- **Tesseract** (default): The standard OCR engine that works well for most documents
-- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise
-- **Hybrid**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text
+- **Tesseract** (default): The standard OCR engine that works well for most documents. Provides good word-level bounding box accuracy.
+- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise. Best for documents with clear, well-formatted text.
+- **Hybrid-paddle**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text regions.
+- **Hybrid-vlm**: Combines Tesseract with Vision Language Models (VLM) - uses Tesseract for initial extraction, then a VLM model (default: Dots.OCR) for re-extraction of low-confidence text.
+- **Hybrid-paddle-vlm**: Combines PaddleOCR with Vision Language Models - uses PaddleOCR first, then a VLM model for low-confidence regions.
 
 ### Enabling advanced OCR options
 
-To enable these options, your system administrator needs to modify the configuration file (`config.py`) and set:
+To enable these options, you need to modify the app_config.env file in your '/config' folder and set the following environment variables:
 
+**Basic OCR model selection:**
 ```
 SHOW_LOCAL_OCR_MODEL_OPTIONS = "True"
 ```
 
-Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between:
-- tesseract
-- hybrid  
-- paddle
+**To enable PaddleOCR options (paddle, hybrid-paddle):**
+```
+SHOW_PADDLE_MODEL_OPTIONS = "True"
+```
+
+**To enable Vision Language Model options (hybrid-vlm, hybrid-paddle-vlm):**
+```
+SHOW_VLM_MODEL_OPTIONS = "True"
+```
+
+Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between the available models based on what has been enabled.
 
-### Hybrid OCR configuration
+### OCR configuration parameters
 
-The hybrid OCR mode uses several configurable parameters:
+The following parameters can be configured by your system administrator to fine-tune OCR behavior:
 
-- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 65): Tesseract confidence score below which PaddleOCR will be used for re-extraction
-- **HYBRID_OCR_PADDING** (default: 1): Padding added to word bounding boxes before re-extraction
-- **SAVE_EXAMPLE_HYBRID_IMAGES** (default: False): Save comparison images when using hybrid mode
-- **SAVE_PAGE_OCR_VISUALISATIONS** (default: False): Save images with PaddleOCR bounding boxes overlaid
+#### Hybrid OCR settings
+
+- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 80): Tesseract confidence score below which the secondary OCR engine (PaddleOCR or VLM) will be used for re-extraction. Lower values mean more text will be re-extracted.
+- **HYBRID_OCR_PADDING** (default: 1): Padding (in pixels) added to word bounding boxes before re-extraction with the secondary engine.
+- **SAVE_EXAMPLE_HYBRID_IMAGES** (default: False): If enabled, saves comparison images showing Tesseract vs. secondary engine results when using hybrid modes.
+- **SAVE_PAGE_OCR_VISUALISATIONS** (default: False): If enabled, saves images with detected bounding boxes overlaid for debugging purposes.
+
+#### Tesseract settings
+
+- **TESSERACT_SEGMENTATION_LEVEL** (default: 11): Tesseract PSM (Page Segmentation Mode) level. Valid values are 0-13. Higher values provide more detailed segmentation but may be slower.
+
+#### PaddleOCR settings
+
+- **PADDLE_USE_TEXTLINE_ORIENTATION** (default: False): If enabled, PaddleOCR will detect and correct text line orientation.
+- **PADDLE_DET_DB_UNCLIP_RATIO** (default: 1.2): Controls the expansion ratio of detected text regions. Higher values expand the detection area more.
+- **CONVERT_LINE_TO_WORD_LEVEL** (default: False): If enabled, converts PaddleOCR line-level results to word-level for better precision in bounding boxes (not perfect, but pretty good).
+- **LOAD_PADDLE_AT_STARTUP** (default: False): If enabled, loads the PaddleOCR model when the application starts, reducing latency for first use but increasing startup time.
+
+#### Image preprocessing
+
+- **PREPROCESS_LOCAL_OCR_IMAGES** (default: True): If enabled, images are preprocessed before OCR. This can improve accuracy but may slow down processing.
+- **SAVE_PREPROCESS_IMAGES** (default: False): If enabled, saves the preprocessed images for debugging purposes.
+
+#### Vision Language Model (VLM) settings
+
+When VLM options are enabled, the following settings are available:
+
+- **SELECTED_MODEL** (default: "Dots.OCR"): The VLM model to use. Options include: "Nanonets-OCR2-3B", "Dots.OCR", "Qwen3-VL-2B-Instruct", "Qwen3-VL-4B-Instruct", "PaddleOCR-VL".
+- **MAX_SPACES_GPU_RUN_TIME** (default: 60): Maximum seconds to run GPU operations on Hugging Face Spaces.
+- **MAX_NEW_TOKENS** (default: 30): Maximum number of tokens to generate for VLM responses.
+- **MAX_INPUT_TOKEN_LENGTH** (default: 4096): Maximum number of tokens that can be input to the VLM.
+- **VLM_MAX_IMAGE_SIZE** (default: 1000000): Maximum total pixels (width × height) for images. Larger images are resized while maintaining aspect ratio.
+- **VLM_MAX_DPI** (default: 300.0): Maximum DPI for images. Higher DPI images are resized accordingly.
+- **USE_FLASH_ATTENTION** (default: False): If enabled, uses flash attention for improved VLM performance.
+- **SAVE_VLM_INPUT_IMAGES** (default: False): If enabled, saves input images sent to VLM for debugging.
+
+#### General settings
+
+- **MODEL_CACHE_PATH** (default: "./model_cache"): Directory where OCR models are cached.
+- **OVERWRITE_EXISTING_OCR_RESULTS** (default: False): If enabled, always creates new OCR results instead of loading from existing JSON files.
 
 ### When to use different OCR models
 
-- **Tesseract**: Best for general use, good balance of speed and accuracy
-- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision
-- **Hybrid**: Best for challenging documents where some text has low confidence scores, providing the benefits of both engines
+- **Tesseract**: Best for general use, providing a good balance of speed and accuracy with precise word-level bounding boxes.
+- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision.
+- **Hybrid-paddle**: Best for challenging documents where some text has low confidence scores, combining Tesseract's word-level precision with PaddleOCR's improved text recognition.
+- **Hybrid-vlm**: Best for very challenging documents with poor image quality or unusual text layouts, leveraging advanced vision models for difficult text.
+- **Hybrid-paddle-vlm**: Most comprehensive option, combining PaddleOCR's line-level detection with a VLM's advanced recognition capabilities.
 
 
 
@@ -1069,18 +1124,65 @@ python cli_redact.py --task textract --textract_action list
 
 ### Common CLI options
 
+#### General options
+
 - `--task`: Choose between "redact", "deduplicate", or "textract"
-- `--input_file`: Path to input file(s)
+- `--input_file`: Path to input file(s) - can specify multiple files separated by spaces
 - `--output_dir`: Directory for output files (default: output/)
-- `--page_min` / `--page_max`: Process only specific page range
-- `--ocr_method`: Choose text extraction method
-- `--pii_detector`: Choose PII detection method
-- `--local_redact_entities`: Specify local entities to redact
-- `--allow_list_file` / `--deny_list_file`: Custom lists
-- `--redact_whole_page_file`: List of pages to redact completely
-- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching
-- `--similarity_threshold`: Threshold for duplicate detection
-- `--anon_strategy`: Anonymization strategy for tabular data
+- `--input_dir`: Directory for input files (default: input/)
+- `--language`: Language of document content (e.g., "en", "es", "fr")
+- `--username`: Username for session tracking
+- `--pii_detector`: Choose PII detection method ("Local", "AWS Comprehend", or "None")
+- `--local_redact_entities`: Specify local entities to redact (space-separated list)
+- `--aws_redact_entities`: Specify AWS Comprehend entities to redact (space-separated list)
+- `--aws_access_key` / `--aws_secret_key`: AWS credentials for cloud services
+- `--aws_region`: AWS region for cloud services
+- `--s3_bucket`: S3 bucket name for cloud operations
+- `--cost_code`: Cost code for tracking usage
+
+#### PDF/Image redaction options
+
+- `--ocr_method`: Choose text extraction method ("AWS Textract", "Local OCR", or "Local text")
+- `--chosen_local_ocr_model`: Local OCR model to use (e.g., "tesseract", "paddle", "hybrid-paddle", "hybrid-vlm")
+- `--page_min` / `--page_max`: Process only specific page range (0 for max means all pages)
+- `--images_dpi`: DPI for image processing (default: 300.0)
+- `--preprocess_local_ocr_images`: Preprocess images before OCR (True/False)
+- `--compress_redacted_pdf`: Compress the final redacted PDF (True/False)
+- `--return_pdf_end_of_redaction`: Return PDF at end of redaction process (True/False)
+- `--allow_list_file` / `--deny_list_file`: Paths to custom allow/deny list CSV files
+- `--redact_whole_page_file`: Path to CSV file listing pages to redact completely
+- `--handwrite_signature_extraction`: Handwriting and signature extraction options for Textract ("Extract handwriting", "Extract signatures")
+- `--extract_forms`: Extract forms during Textract analysis (flag)
+- `--extract_tables`: Extract tables during Textract analysis (flag)
+- `--extract_layout`: Extract layout during Textract analysis (flag)
+
+#### Tabular/Word anonymization options
+
+- `--anon_strategy`: Anonymization strategy (e.g., "redact", "redact completely", "replace_redacted", "encrypt", "hash")
+- `--text_columns`: List of column names to anonymize (space-separated)
+- `--excel_sheets`: Specific Excel sheet names to process (space-separated)
+- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching (default: 1)
+- `--match_fuzzy_whole_phrase_bool`: Match fuzzy whole phrase (True/False)
+- `--do_initial_clean`: Perform initial text cleaning for tabular data (True/False)
+
+#### Duplicate detection options
+
+- `--duplicate_type`: Type of duplicate detection ("pages" for OCR files or "tabular" for CSV/Excel)
+- `--similarity_threshold`: Similarity threshold (0-1) to consider content as duplicates (default: 0.95)
+- `--min_word_count`: Minimum word count for text to be considered (default: 10)
+- `--min_consecutive_pages`: Minimum number of consecutive pages to consider as a match (default: 1)
+- `--greedy_match`: Use greedy matching strategy for consecutive pages (True/False)
+- `--combine_pages`: Combine text from same page number within a file (True/False)
+- `--remove_duplicate_rows`: Remove duplicate rows from output (True/False)
+
+#### Textract batch operations options
+
+- `--textract_action`: Action to perform ("submit", "retrieve", or "list")
+- `--job_id`: Textract job ID for retrieve action
+- `--extract_signatures`: Extract signatures during Textract analysis (flag)
+- `--textract_bucket`: S3 bucket name for Textract operations
+- `--poll_interval`: Polling interval in seconds for job status (default: 30)
+- `--max_poll_attempts`: Maximum polling attempts before timeout (default: 120)
 
 ### Output files
 

cli_redact.py

@@ -36,6 +36,7 @@ from tools.config import (
     FULL_ENTITY_LIST,
     IMAGES_DPI,
     INPUT_FOLDER,
+    LOCAL_OCR_MODEL_OPTIONS,
     LOCAL_PII_OPTION,
     OUTPUT_FOLDER,
     PADDLE_MODEL_PATH,
@@ -399,7 +400,7 @@ python cli_redact.py --task textract --textract_action list
     )
     pdf_group.add_argument(
         "--chosen_local_ocr_model",
-        choices=["tesseract", "hybrid-paddle", "paddle"],
+        choices=LOCAL_OCR_MODEL_OPTIONS,
         default=CHOSEN_LOCAL_OCR_MODEL,
         help="Local OCR model to use.",
     )

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "doc_redaction"
-version = "1.5.0"
+version = "1.5.1"
 description = "Redact PDF/image-based documents, Word, or CSV/XLSX files using a Gradio-based GUI interface"
 readme = "README.md"
 requires-python = ">=3.10"

src/app_settings.qmd

@@ -97,11 +97,11 @@ Configuration for input and output file handling.
     * **Description:** If set to `'True'`, the application will save output and input files into session-specific subfolders.
     * **Default Value:** `'False'`
 
-* **`OUTPUT_FOLDER`**
+* **`OUTPUT_FOLDER`** (environment variable: `GRADIO_OUTPUT_FOLDER`)
     * **Description:** Specifies the default output folder for generated files. Can be set to `"TEMP"` to use a temporary directory.
     * **Default Value:** `'output/'`
 
-* **`INPUT_FOLDER`**
+* **`INPUT_FOLDER`** (environment variable: `GRADIO_INPUT_FOLDER`)
     * **Description:** Specifies the default input folder for files. Can be set to `"TEMP"` to use a temporary directory.
     * **Default Value:** `'input/'`
 
@@ -225,6 +225,14 @@ Configurations for the Gradio UI, server behavior, and application limits.
     * **Description:** Maximum number of characters for open text input.
     * **Default Value:** `50000`
 
+* **`PAGE_BREAK_VALUE`**
+    * **Description:** Number of pages to process before breaking and restarting from the last finished page (not currently activated).
+    * **Default Value:** `99999`
+
+* **`MAX_TIME_VALUE`**
+    * **Description:** Maximum time value for processing operations.
+    * **Default Value:** `999999`
+
 * **`TLDEXTRACT_CACHE`**
     * **Description:** Path to the cache directory used by the `tldextract` library.
     * **Default Value:** `'tmp/tld/'`
@@ -263,6 +271,14 @@ Configurations related to text extraction, PII detection, and the redaction proc
     * **Description:** Controls whether local (Tesseract) or AWS (Textract) text extraction options are shown in the UI.
     * **Default Value:** `"True"` for both.
 
+* **`SELECTABLE_TEXT_EXTRACT_OPTION`**, **`TESSERACT_TEXT_EXTRACT_OPTION`**, **`TEXTRACT_TEXT_EXTRACT_OPTION`**
+    * **Description:** Labels for text extraction model options displayed in the UI. Customize the display names for "Local model - selectable text", "Local OCR model - PDFs without selectable text", and "AWS Textract service - all PDF types" respectively.
+    * **Default Values:** `"Local model - selectable text"`, `"Local OCR model - PDFs without selectable text"`, `"AWS Textract service - all PDF types"`
+
+* **`NO_REDACTION_PII_OPTION`**, **`LOCAL_PII_OPTION`**, **`AWS_PII_OPTION`**
+    * **Description:** Labels for PII detection model options displayed in the UI. Customize the display names for "Only extract text (no redaction)", "Local", and "AWS Comprehend" respectively.
+    * **Default Values:** `"Only extract text (no redaction)"`, `"Local"`, `"AWS Comprehend"`
+
 * **`SHOW_LOCAL_PII_DETECTION_OPTIONS`** / **`SHOW_AWS_PII_DETECTION_OPTIONS`**
     * **Description:** Controls whether local or AWS (Comprehend) PII detection options are shown in the UI.
     * **Default Value:** `"True"` for both.
@@ -309,7 +325,7 @@ Configurations related to text extraction, PII detection, and the redaction proc
 
 * **`HYBRID_OCR_CONFIDENCE_THRESHOLD`**
     * **Description:** In "hybrid-paddle" mode, this is the Tesseract confidence score below which PaddleOCR will be used for re-extraction.
-    * **Default Value:** `65`
+    * **Default Value:** `80`
 
 * **`HYBRID_OCR_PADDING`**
     * **Description:** In "hybrid-paddle" mode, padding added to the word's bounding box before re-extraction.
@@ -333,6 +349,76 @@ Configurations related to text extraction, PII detection, and the redaction proc
 
 * **`PREPROCESS_LOCAL_OCR_IMAGES`**
     * **Description:** If set to `"True"`, images will be preprocessed before local OCR. Can slow down processing.
+    * **Default Value:** `"True"`
+
+* **`SAVE_PREPROCESS_IMAGES`**
+    * **Description:** If set to `"True"`, saves the preprocessed images for debugging purposes.
+    * **Default Value:** `"False"`
+
+* **`SHOW_PADDLE_MODEL_OPTIONS`**
+    * **Description:** If set to `"True"`, allows the user to select PaddleOCR-related options (paddle, hybrid-paddle) from the UI.
+    * **Default Value:** `"False"`
+
+* **`MODEL_CACHE_PATH`**
+    * **Description:** Path to the directory where models are cached.
+    * **Default Value:** `"./model_cache"`
+
+* **`TESSERACT_SEGMENTATION_LEVEL`**
+    * **Description:** Tesseract PSM (Page Segmentation Mode) level to use for OCR. Valid values are 0-13.
+    * **Default Value:** `11`
+
+* **`CONVERT_LINE_TO_WORD_LEVEL`**
+    * **Description:** If set to `"True"`, converts PaddleOCR line-level OCR results to word-level for better precision.
+    * **Default Value:** `"False"`
+
+* **`LOAD_PADDLE_AT_STARTUP`**
+    * **Description:** If set to `"True"`, loads the PaddleOCR model at application startup.
+    * **Default Value:** `"False"`
+
+### Vision Language Model (VLM) Options
+
+* **`SHOW_VLM_MODEL_OPTIONS`**
+    * **Description:** If set to `"True"`, VLM (Vision Language Model) options will be shown in the UI.
+    * **Default Value:** `"False"`
+
+* **`SELECTED_MODEL`**
+    * **Description:** Selected vision model for OCR. Choose from: `"Nanonets-OCR2-3B"`, `"Dots.OCR"`, `"Qwen3-VL-2B-Instruct"`, `"Qwen3-VL-4B-Instruct"`, `"PaddleOCR-VL"`.
+    * **Default Value:** `"Dots.OCR"`
+
+* **`MAX_SPACES_GPU_RUN_TIME`**
+    * **Description:** Maximum number of seconds to run the GPU on Spaces (Hugging Face Spaces).
+    * **Default Value:** `60`
+
+* **`MAX_NEW_TOKENS`**
+    * **Description:** Maximum number of tokens to generate for VLM responses.
+    * **Default Value:** `30`
+
+* **`DEFAULT_MAX_NEW_TOKENS`**
+    * **Description:** Default maximum number of tokens to generate for VLM responses.
+    * **Default Value:** `30`
+
+* **`MAX_INPUT_TOKEN_LENGTH`**
+    * **Description:** Maximum number of tokens that can be input to the VLM.
+    * **Default Value:** `4096`
+
+* **`VLM_MAX_IMAGE_SIZE`**
+    * **Description:** Maximum total pixels (width * height) for images passed to VLM. Images with more pixels will be resized while maintaining aspect ratio.
+    * **Default Value:** `1000000` (1000x1000)
+
+* **`VLM_MAX_DPI`**
+    * **Description:** Maximum DPI for images passed to VLM. Images with higher DPI will be resized accordingly.
+    * **Default Value:** `300.0`
+
+* **`USE_FLASH_ATTENTION`**
+    * **Description:** If set to `"True"`, uses flash attention for the VLM, which can improve performance.
+    * **Default Value:** `"False"`
+
+* **`OVERWRITE_EXISTING_OCR_RESULTS`**
+    * **Description:** If set to `"True"`, always creates new OCR results instead of loading from existing JSON files.
+    * **Default Value:** `"False"`
+
+* **`SAVE_VLM_INPUT_IMAGES`**
+    * **Description:** If set to `"True"`, saves input images sent to VLM OCR for debugging purposes.
     * **Default Value:** `"False"`
 
 ### Entity and Search Options
@@ -367,6 +453,10 @@ Configurations related to text extraction, PII detection, and the redaction proc
     * **Description:** The default options selected for Textract's handwriting and signature detection.
     * **Default Value:** `['Extract handwriting']`
 
+* **`HANDWRITE_SIGNATURE_TEXTBOX_FULL_OPTIONS`**
+    * **Description:** Full list of available options for Textract's handwriting and signature detection. Can include `'Extract handwriting'`, `'Extract signatures'`, and optionally `'Extract forms'`, `'Extract layout'`, `'Extract tables'` if the corresponding include options are enabled.
+    * **Default Value:** `['Extract handwriting', 'Extract signatures']`
+
 * **`INCLUDE_FORM_EXTRACTION_TEXTRACT_OPTION`**
 * **`INCLUDE_LAYOUT_EXTRACTION_TEXTRACT_OPTION`**
 * **`INCLUDE_TABLE_EXTRACTION_TEXTRACT_OPTION`**
@@ -481,9 +571,105 @@ Settings for running the application from the command line (Direct Mode) or as a
     * **Description:** Path to the input file and output directory for the task.
     * **Default Values:** `''`, `output/`
 
-* **Other `DIRECT_MODE_*` variables:**
-    * **Description:** These variables allow for setting nearly all application options (e.g., `DIRECT_MODE_PII_DETECTOR`, `DIRECT_MODE_SIMILARITY_THRESHOLD`) directly for a single CLI run, overriding other configurations.
-    * **Default Value:** Defaults are inherited from the main application settings (e.g., `LOCAL_PII_OPTION`, `DEFAULT_DUPLICATE_DETECTION_THRESHOLD`).
+* **`DIRECT_MODE_DUPLICATE_TYPE`**
+    * **Description:** Type of duplicate detection for direct mode: `'pages'` or `'tabular'`.
+    * **Default Value:** `'pages'`
+
+* **`DIRECT_MODE_LANGUAGE`**
+    * **Description:** Language for document processing in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_LANGUAGE`
+
+* **`DIRECT_MODE_PII_DETECTOR`**
+    * **Description:** PII detection method for direct mode.
+    * **Default Value:** Inherits from `LOCAL_PII_OPTION`
+
+* **`DIRECT_MODE_OCR_METHOD`**
+    * **Description:** OCR method for PDF/image processing in direct mode.
+    * **Default Value:** `"Local OCR"`
+
+* **`DIRECT_MODE_PAGE_MIN`** / **`DIRECT_MODE_PAGE_MAX`**
+    * **Description:** First and last page to process in direct mode. `0` for max means process all pages.
+    * **Default Values:** Inherit from `DEFAULT_PAGE_MIN` / `DEFAULT_PAGE_MAX`
+
+* **`DIRECT_MODE_IMAGES_DPI`**
+    * **Description:** DPI for image processing in direct mode.
+    * **Default Value:** Inherits from `IMAGES_DPI`
+
+* **`DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL`**
+    * **Description:** Local OCR model choice for direct mode.
+    * **Default Value:** Inherits from `CHOSEN_LOCAL_OCR_MODEL`
+
+* **`DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES`**
+    * **Description:** If set to `"True"`, preprocesses images before OCR in direct mode.
+    * **Default Value:** Inherits from `PREPROCESS_LOCAL_OCR_IMAGES`
+
+* **`DIRECT_MODE_COMPRESS_REDACTED_PDF`**
+    * **Description:** If set to `"True"`, compresses the redacted PDF output in direct mode.
+    * **Default Value:** Inherits from `COMPRESS_REDACTED_PDF`
+
+* **`DIRECT_MODE_RETURN_PDF_END_OF_REDACTION`**
+    * **Description:** If set to `"True"`, returns a PDF at the end of redaction in direct mode.
+    * **Default Value:** Inherits from `RETURN_REDACTED_PDF`
+
+* **`DIRECT_MODE_EXTRACT_FORMS`**
+    * **Description:** If set to `"True"`, extracts forms during Textract analysis in direct mode.
+    * **Default Value:** `"False"`
+
+* **`DIRECT_MODE_EXTRACT_TABLES`**
+    * **Description:** If set to `"True"`, extracts tables during Textract analysis in direct mode.
+    * **Default Value:** `"False"`
+
+* **`DIRECT_MODE_EXTRACT_LAYOUT`**
+    * **Description:** If set to `"True"`, extracts layout during Textract analysis in direct mode.
+    * **Default Value:** `"False"`
+
+* **`DIRECT_MODE_EXTRACT_SIGNATURES`**
+    * **Description:** If set to `"True"`, extracts signatures during Textract analysis in direct mode.
+    * **Default Value:** `"False"`
+
+* **`DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL`**
+    * **Description:** If set to `"True"`, matches fuzzy whole phrases in direct mode.
+    * **Default Value:** `"True"`
+
+* **`DIRECT_MODE_ANON_STRATEGY`**
+    * **Description:** Anonymisation strategy for tabular data in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_TABULAR_ANONYMISATION_STRATEGY`
+
+* **`DIRECT_MODE_FUZZY_MISTAKES`**
+    * **Description:** Number of fuzzy spelling mistakes allowed in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_FUZZY_SPELLING_MISTAKES_NUM`
+
+* **`DIRECT_MODE_SIMILARITY_THRESHOLD`**
+    * **Description:** Similarity threshold for duplicate detection in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_DUPLICATE_DETECTION_THRESHOLD`
+
+* **`DIRECT_MODE_MIN_WORD_COUNT`**
+    * **Description:** Minimum word count for duplicate detection in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_MIN_WORD_COUNT`
+
+* **`DIRECT_MODE_MIN_CONSECUTIVE_PAGES`**
+    * **Description:** Minimum consecutive pages for duplicate detection in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_MIN_CONSECUTIVE_PAGES`
+
+* **`DIRECT_MODE_GREEDY_MATCH`**
+    * **Description:** If set to `"True"`, uses greedy matching for duplicate detection in direct mode.
+    * **Default Value:** Inherits from `USE_GREEDY_DUPLICATE_DETECTION`
+
+* **`DIRECT_MODE_COMBINE_PAGES`**
+    * **Description:** If set to `"True"`, combines pages for duplicate detection in direct mode.
+    * **Default Value:** Inherits from `DEFAULT_COMBINE_PAGES`
+
+* **`DIRECT_MODE_REMOVE_DUPLICATE_ROWS`**
+    * **Description:** If set to `"True"`, removes duplicate rows in tabular data in direct mode.
+    * **Default Value:** Inherits from `REMOVE_DUPLICATE_ROWS`
+
+* **`DIRECT_MODE_TEXTRACT_ACTION`**
+    * **Description:** Textract action for batch operations in direct mode.
+    * **Default Value:** `''`
+
+* **`DIRECT_MODE_JOB_ID`**
+    * **Description:** Job ID for Textract operations in direct mode.
+    * **Default Value:** `''`
 
 ### Lambda Configuration
 

src/user_guide.qmd

@@ -150,7 +150,17 @@ If you have used the AWS Textract option for extracting text, you may also see a
 
 ![Document upload alongside Textract](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/document_upload_with_textract.PNG)
 
-Similarly, if you have used the 'Local OCR method' to extract text, you may see a '..._ocr_results_with_words.json' file. This file works in the same way as the AWS Textract .json results described above, and can be uploaded alongside an input document to save time on text extraction in future in the same way.
+#### Additional outputs in the log file outputs
+
+On the Redaction settings tab, near the bottom of the pagethere is a section called 'Log file outputs'. This section contains the following files:
+
+You may see a '..._ocr_results_with_words... .json' file. This file works in the same way as the AWS Textract .json results described above, and can be uploaded alongside an input document to save time on text extraction in future in the same way.
+
+Also you will see a 'decision_process_table.csv' file. This file contains a table of the decisions made by the app for each page of the document. This can be useful for debugging and understanding the decisions made by the app.
+
+Additionally, if the option is enabled by your system administrator, on this tab you may see an image of the output from the OCR model used to extract the text from the document, an image ending with page number and '_visualisations.jpg'. A separate image will be created for each page of the document like the one below. This can be useful for seeing at a glance whether the text extraction process for a page was successful, and whether word-level bounding boxes are correctly positioned.
+
+![Text analysis output](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/example_complaint_letter_1_textract_visualisations.jpg)
 
 ### Downloading output files from previous redaction tasks
 
@@ -460,6 +470,7 @@ You can also write open text into an input box and redact that using the same me
 ### Redaction log outputs
 A list of the suggested redaction outputs from the tabular data / open text data redaction is available on the Redaction settings page under 'Log file outputs'.
 
+
 ## Identifying and redacting duplicate pages
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/duplicate_page_find_in_app/).
@@ -690,45 +701,91 @@ AWS_SECRET_KEY= your-secret-key
 
 The app should then pick up these keys when trying to access the AWS Textract and Comprehend services during redaction.
 
-Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
-
-## Advanced OCR options (Hybrid OCR)
+## Advanced OCR options
 
-The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by your system administrator.
+The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by changing the app_config.env file in your '/config' folder, or system environment variables in your system.
 
 ### Available OCR models
 
-- **Tesseract** (default): The standard OCR engine that works well for most documents
-- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise
-- **Hybrid**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text
+- **Tesseract** (default): The standard OCR engine that works well for most documents. Provides good word-level bounding box accuracy.
+- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise. Best for documents with clear, well-formatted text.
+- **Hybrid-paddle**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text regions.
+- **Hybrid-vlm**: Combines Tesseract with Vision Language Models (VLM) - uses Tesseract for initial extraction, then a VLM model (default: Dots.OCR) for re-extraction of low-confidence text.
+- **Hybrid-paddle-vlm**: Combines PaddleOCR with Vision Language Models - uses PaddleOCR first, then a VLM model for low-confidence regions.
 
 ### Enabling advanced OCR options
 
-To enable these options, your system administrator needs to modify the configuration file (`config.py`) and set:
+To enable these options, you need to modify the app_config.env file in your '/config' folder and set the following environment variables:
 
+**Basic OCR model selection:**
 ```
 SHOW_LOCAL_OCR_MODEL_OPTIONS = "True"
 ```
 
-Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between:
-- tesseract
-- hybrid  
-- paddle
+**To enable PaddleOCR options (paddle, hybrid-paddle):**
+```
+SHOW_PADDLE_MODEL_OPTIONS = "True"
+```
+
+**To enable Vision Language Model options (hybrid-vlm, hybrid-paddle-vlm):**
+```
+SHOW_VLM_MODEL_OPTIONS = "True"
+```
+
+Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between the available models based on what has been enabled.
 
-### Hybrid OCR configuration
+### OCR configuration parameters
 
-The hybrid OCR mode uses several configurable parameters:
+The following parameters can be configured by your system administrator to fine-tune OCR behavior:
 
-- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 65): Tesseract confidence score below which PaddleOCR will be used for re-extraction
-- **HYBRID_OCR_PADDING** (default: 1): Padding added to word bounding boxes before re-extraction
-- **SAVE_EXAMPLE_HYBRID_IMAGES** (default: False): Save comparison images when using hybrid mode
-- **SAVE_PAGE_OCR_VISUALISATIONS** (default: False): Save images with PaddleOCR bounding boxes overlaid
+#### Hybrid OCR settings
+
+- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 80): Tesseract confidence score below which the secondary OCR engine (PaddleOCR or VLM) will be used for re-extraction. Lower values mean more text will be re-extracted.
+- **HYBRID_OCR_PADDING** (default: 1): Padding (in pixels) added to word bounding boxes before re-extraction with the secondary engine.
+- **SAVE_EXAMPLE_HYBRID_IMAGES** (default: False): If enabled, saves comparison images showing Tesseract vs. secondary engine results when using hybrid modes.
+- **SAVE_PAGE_OCR_VISUALISATIONS** (default: False): If enabled, saves images with detected bounding boxes overlaid for debugging purposes.
+
+#### Tesseract settings
+
+- **TESSERACT_SEGMENTATION_LEVEL** (default: 11): Tesseract PSM (Page Segmentation Mode) level. Valid values are 0-13. Higher values provide more detailed segmentation but may be slower.
+
+#### PaddleOCR settings
+
+- **PADDLE_USE_TEXTLINE_ORIENTATION** (default: False): If enabled, PaddleOCR will detect and correct text line orientation.
+- **PADDLE_DET_DB_UNCLIP_RATIO** (default: 1.2): Controls the expansion ratio of detected text regions. Higher values expand the detection area more.
+- **CONVERT_LINE_TO_WORD_LEVEL** (default: False): If enabled, converts PaddleOCR line-level results to word-level for better precision in bounding boxes (not perfect, but pretty good).
+- **LOAD_PADDLE_AT_STARTUP** (default: False): If enabled, loads the PaddleOCR model when the application starts, reducing latency for first use but increasing startup time.
+
+#### Image preprocessing
+
+- **PREPROCESS_LOCAL_OCR_IMAGES** (default: True): If enabled, images are preprocessed before OCR. This can improve accuracy but may slow down processing.
+- **SAVE_PREPROCESS_IMAGES** (default: False): If enabled, saves the preprocessed images for debugging purposes.
+
+#### Vision Language Model (VLM) settings
+
+When VLM options are enabled, the following settings are available:
+
+- **SELECTED_MODEL** (default: "Dots.OCR"): The VLM model to use. Options include: "Nanonets-OCR2-3B", "Dots.OCR", "Qwen3-VL-2B-Instruct", "Qwen3-VL-4B-Instruct", "PaddleOCR-VL".
+- **MAX_SPACES_GPU_RUN_TIME** (default: 60): Maximum seconds to run GPU operations on Hugging Face Spaces.
+- **MAX_NEW_TOKENS** (default: 30): Maximum number of tokens to generate for VLM responses.
+- **MAX_INPUT_TOKEN_LENGTH** (default: 4096): Maximum number of tokens that can be input to the VLM.
+- **VLM_MAX_IMAGE_SIZE** (default: 1000000): Maximum total pixels (width × height) for images. Larger images are resized while maintaining aspect ratio.
+- **VLM_MAX_DPI** (default: 300.0): Maximum DPI for images. Higher DPI images are resized accordingly.
+- **USE_FLASH_ATTENTION** (default: False): If enabled, uses flash attention for improved VLM performance.
+- **SAVE_VLM_INPUT_IMAGES** (default: False): If enabled, saves input images sent to VLM for debugging.
+
+#### General settings
+
+- **MODEL_CACHE_PATH** (default: "./model_cache"): Directory where OCR models are cached.
+- **OVERWRITE_EXISTING_OCR_RESULTS** (default: False): If enabled, always creates new OCR results instead of loading from existing JSON files.
 
 ### When to use different OCR models
 
-- **Tesseract**: Best for general use, good balance of speed and accuracy
-- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision
-- **Hybrid**: Best for challenging documents where some text has low confidence scores, providing the benefits of both engines
+- **Tesseract**: Best for general use, providing a good balance of speed and accuracy with precise word-level bounding boxes.
+- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision.
+- **Hybrid-paddle**: Best for challenging documents where some text has low confidence scores, combining Tesseract's word-level precision with PaddleOCR's improved text recognition.
+- **Hybrid-vlm**: Best for very challenging documents with poor image quality or unusual text layouts, leveraging advanced vision models for difficult text.
+- **Hybrid-paddle-vlm**: Most comprehensive option, combining PaddleOCR's line-level detection with a VLM's advanced recognition capabilities.
 
 
 
@@ -843,18 +900,65 @@ python cli_redact.py --task textract --textract_action list
 
 ### Common CLI options
 
+#### General options
+
 - `--task`: Choose between "redact", "deduplicate", or "textract"
-- `--input_file`: Path to input file(s)
+- `--input_file`: Path to input file(s) - can specify multiple files separated by spaces
 - `--output_dir`: Directory for output files (default: output/)
-- `--page_min` / `--page_max`: Process only specific page range
-- `--ocr_method`: Choose text extraction method
-- `--pii_detector`: Choose PII detection method
-- `--local_redact_entities`: Specify local entities to redact
-- `--allow_list_file` / `--deny_list_file`: Custom lists
-- `--redact_whole_page_file`: List of pages to redact completely
-- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching
-- `--similarity_threshold`: Threshold for duplicate detection
-- `--anon_strategy`: Anonymization strategy for tabular data
+- `--input_dir`: Directory for input files (default: input/)
+- `--language`: Language of document content (e.g., "en", "es", "fr")
+- `--username`: Username for session tracking
+- `--pii_detector`: Choose PII detection method ("Local", "AWS Comprehend", or "None")
+- `--local_redact_entities`: Specify local entities to redact (space-separated list)
+- `--aws_redact_entities`: Specify AWS Comprehend entities to redact (space-separated list)
+- `--aws_access_key` / `--aws_secret_key`: AWS credentials for cloud services
+- `--aws_region`: AWS region for cloud services
+- `--s3_bucket`: S3 bucket name for cloud operations
+- `--cost_code`: Cost code for tracking usage
+
+#### PDF/Image redaction options
+
+- `--ocr_method`: Choose text extraction method ("AWS Textract", "Local OCR", or "Local text")
+- `--chosen_local_ocr_model`: Local OCR model to use (e.g., "tesseract", "paddle", "hybrid-paddle", "hybrid-vlm")
+- `--page_min` / `--page_max`: Process only specific page range (0 for max means all pages)
+- `--images_dpi`: DPI for image processing (default: 300.0)
+- `--preprocess_local_ocr_images`: Preprocess images before OCR (True/False)
+- `--compress_redacted_pdf`: Compress the final redacted PDF (True/False)
+- `--return_pdf_end_of_redaction`: Return PDF at end of redaction process (True/False)
+- `--allow_list_file` / `--deny_list_file`: Paths to custom allow/deny list CSV files
+- `--redact_whole_page_file`: Path to CSV file listing pages to redact completely
+- `--handwrite_signature_extraction`: Handwriting and signature extraction options for Textract ("Extract handwriting", "Extract signatures")
+- `--extract_forms`: Extract forms during Textract analysis (flag)
+- `--extract_tables`: Extract tables during Textract analysis (flag)
+- `--extract_layout`: Extract layout during Textract analysis (flag)
+
+#### Tabular/Word anonymization options
+
+- `--anon_strategy`: Anonymization strategy (e.g., "redact", "redact completely", "replace_redacted", "encrypt", "hash")
+- `--text_columns`: List of column names to anonymize (space-separated)
+- `--excel_sheets`: Specific Excel sheet names to process (space-separated)
+- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching (default: 1)
+- `--match_fuzzy_whole_phrase_bool`: Match fuzzy whole phrase (True/False)
+- `--do_initial_clean`: Perform initial text cleaning for tabular data (True/False)
+
+#### Duplicate detection options
+
+- `--duplicate_type`: Type of duplicate detection ("pages" for OCR files or "tabular" for CSV/Excel)
+- `--similarity_threshold`: Similarity threshold (0-1) to consider content as duplicates (default: 0.95)
+- `--min_word_count`: Minimum word count for text to be considered (default: 10)
+- `--min_consecutive_pages`: Minimum number of consecutive pages to consider as a match (default: 1)
+- `--greedy_match`: Use greedy matching strategy for consecutive pages (True/False)
+- `--combine_pages`: Combine text from same page number within a file (True/False)
+- `--remove_duplicate_rows`: Remove duplicate rows from output (True/False)
+
+#### Textract batch operations options
+
+- `--textract_action`: Action to perform ("submit", "retrieve", or "list")
+- `--job_id`: Textract job ID for retrieve action
+- `--extract_signatures`: Extract signatures during Textract analysis (flag)
+- `--textract_bucket`: S3 bucket name for Textract operations
+- `--poll_interval`: Polling interval in seconds for job status (default: 30)
+- `--max_poll_attempts`: Maximum polling attempts before timeout (default: 120)
 
 ### Output files