🚀 Automated Structured Content Generator (ASCGenerator)

Short Executive Summary: Enterprise-grade production microservice that ingests noisy, unstructured news payloads (raw text, HTML, RSS feeds) and transforms them into strictly validated, typed JSON records with zero runtime schema drift.

---

🎯 Key Features & Specifications

• ⚡ Zero-Setup Onboarding: Onboard & run in < 5 minutes (make install ──> ready to develop).
• 🔒 Type Safety & Quality Gates: 100% Mypy --strict coverage and Ruff linting enforced across all microservice modules.
• 🛡️ Defensive Security & Guardrails: Prompt injection defenses, XML tag isolation (<untrusted_news_payload>), and developer authority enforcement.
• 🧱 Pydantic V2 Validation Firewall: Strict contract validation, bounded ranges (e.g., confidence_score $\in [0.0, 1.0]$), and structured enum taxonomies.
• ⚡ Two-Tiered Fault Tolerance: Decorated Tenacity retry policy (exponential backoff on HTTP 429/5xx) paired with Instructor schema self-correction feedback loops.
• 📊 Integrated Monitoring Dashboard: Next.js frontend interface providing real-time schema inspection, batch execution testing, and metrics visualization.
• 🐳 Hardened Containerization: Multi-stage non-root Docker build (python:3.11-slim) ready for production container orchestrators.

---

🏗️ System Architecture Flow

┌──────────────────────────────────────────────────────────────────────────────────┐
│                                 INGESTION LAYER                                  │
│                 FastAPI Microservice (POST /v1/extract, /v1/extract/batch)       │
└────────────────────────┬─────────────────────────────────────────┘
                                         │
                                         ▼
                               ┌───────────────────┐
                               │  Config & Schemas │
                               │ (config.py,       │
                               │  schemas.py)      │
                               └─────────┬─────────┘
                                         │
                  ┌──────────────────────┼──────────────────────┐
                  │                      │                      │
                  ▼                      ▼                      ▼
        ┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐
        │ Defensive Prompt │   │ Deduplication    │   │ Structlog JSON   │
        │ Isolation Layer  │   │ Cache (SHA256)   │   │ FinOps Logger    │
        │ (security.py)    │   └────────┬─────────┘   │ (logger.py)      │
        └────────┬─────────┘            │              └────────┬─────────┘
                 │                      │ Hit                   │
                 │                      ├───────────────────────┤
                 ▼                      │ Miss                  │
        ┌───────────────────────────────┴───────────────────────┐
        │          LLM ENGINE & TWO-TIERED RESILIENCE           │
        │                  (engine.ExtractionEngine)            │
        │  ┌─────────────────────────────────────────────────┐  │
        │  │ Tier 1: Tenacity Exponential Backoff Retry       │  │
        │  │         (Network, Rate Limits 429, 5xx)         │  │
        │  └────────────────────────┬────────────────────────┘  │
        │                           │                           │
        │                           ▼                           │
        │  ┌─────────────────────────────────────────────────┐  │
        │  │ Instructor SDK Constrained LLM Inference        │  │
        │  │ (AsyncOpenAI / Anthropic / Mistral, temp=0.0)   │  │
        │  └────────────────────────┬────────────────────────┘  │
        │                           │ Raw Output                │
        │                           ▼                           │
        │  ┌─────────────────────────────────────────────────┐  │
        │  │ Tier 2: Pydantic V2 Validation Firewall &       │  │
        │  │         Schema Self-Correction Loop (Max 2)     │  │
        │  └────────────────────────┬────────────────────────┘  │
        └───────────────────────────┼───────────────────────────┘
                                    │ Validated Record
                                    ▼
                       ┌─────────────────────────┐
                       │ ArticleExtractionRecord │
                       │ (JSON Response / DB)    │
                       └─────────────────────────┘

---

📂 Repository Structure

.
├── docs/                       # Architecture & technical documentation
│   ├── architecture_en.md      # Software architecture specification & invariants
│   ├── specifications_en.md    # Product requirements & data model specification
│   ├── code_en.md              # Source code architecture reference
│   ├── glossary_en.md          # Domain & technical terminology glossary
│   └── code_fr.md / glossary_fr.md # French documentation variants
│
├── src/                        # Microservice source code
│   ├── main.py                 # FastAPI application setup, routes & exception handlers
│   ├── config.py               # Central configuration (Pydantic BaseSettings)
│   ├── engine.py               # Extraction engine, Instructor integration & retries
│   ├── schemas.py              # Pydantic V2 schemas & API DTOs
│   ├── security.py             # Prompt isolation & XML sanitization logic
│   └── logger.py               # Structlog JSON logging configuration
│
├── dashboard/                  # Monitoring & testing frontend (Next.js 14)
│   ├── src/                    # UI components & API bindings
│   └── package.json            # Dashboard dependencies
│
├── tests/                      # Automated test suite
│   ├── conftest.py             # Pytest fixtures & mock clients
│   ├── test_schemas.py         # Schema field validation unit tests
│   ├── test_engine.py          # Extraction engine & retry integration tests
│   └── test_security.py        # Defensive security & prompt injection suite
│
├── data/                       # Validation datasets
│   └── dataset_100_articles.jsonl # 100 sample article extraction benchmark
│
├── Dockerfile                  # Multi-stage non-root container configuration
├── docker-compose.yml          # Local microservice orchestration
├── Makefile                    # Unified command interface
├── pyproject.toml              # Poetry dependencies, Mypy strict & Ruff rules
└── README.md                   # Project overview & quickstart guide

---

🚀 Quickstart Guide

1. Installation & Environment Setup

# Clone repository and enter directory
cd projets/4_ASCGenerator

# Automatic onboarding (installs dependencies via Poetry)
make install

2. Configure Environment Variables

Copy .env.example to .env and set your LLM API keys.

Option A: Use OpenAI (default)

LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...
DEFAULT_MODEL=gpt-4o-mini
LOG_LEVEL=INFO

Option B: Use Google Gemini

LLM_PROVIDER=google
GOOGLE_API_KEY=your_google_gemini_api_key_here
DEFAULT_MODEL=gemini-2.0-flash
LOG_LEVEL=INFO

💡 Where to change the model: Set the DEFAULT_MODEL environment variable in your .env file.

• OpenAI models: gpt-4o-mini, gpt-4o, gpt-4-turbo, etc.
• Google Gemini models: gemini-2.0-flash, gemini-2.5-pro, gemini-1.5-pro, etc.

You can also override the model at runtime by passing model to ExtractionEngine(model="...").

🔑 Get a Gemini API key: Visit https://aistudio.google.com/app/apikey to create one.

3. Run Quality & Type Checks

make lint

4. Run Automated Test Suite

make test

5. Start the FastAPI Service

make dev

Open API documentation at http://localhost:8000/docs.

6. Launch the Next.js Monitoring Dashboard

make dashboard

Access the interactive dashboard at http://localhost:3000.

7. Build & Run via Docker

# Build container image
make docker-build

# Run microservice container
make docker-run

---

📊 Quality & FinOps Benchmarks

Metric	Target / Measured	Status
Mypy Strict Coverage	100%	✅ Passing
Schema Compliance Rate	100% (Zero Schema Drift)	✅ Passing
Prompt Injection Resilience	100% Pass Rate	✅ Passing
Network Retry Recovery (Tier 1)	Up to 3 attempts (Tenacity backoff)	✅ Passing
Schema Repair Rate (Tier 2)	Max 2 correction attempts	✅ Passing
Docker Container Size	< 250 MB	✅ Passing

---

📄 License & Contributing

Distributed under the MIT License. See LICENSE for more information.