CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Structure

This is a full-stack AI-powered crossword puzzle generator:
- **Python Backend** (`crossword-app/backend-py/`) - Primary implementation with dynamic word generation
- **React Frontend** (`crossword-app/frontend/`) - Modern React app with interactive crossword UI
- **Node.js Backend** (`backend/`) - Legacy implementation (deprecated)

Current deployment uses the Python backend with Docker containerization.

## Development Commands

### Frontend Development
```bash
cd crossword-app/frontend
npm install
npm run dev          # Start development server on http://localhost:5173
npm run build        # Build for production
npm run preview      # Preview production build
```

### Backend Development (Python - Primary)
```bash
cd crossword-app/backend-py

# Testing
python run_tests.py                                    # Run all tests
pytest test-unit/ -v                                  # Run unit tests
pytest test-integration/ -v                           # Run integration tests
python test_integration_minimal.py                    # Quick test without ML deps

# Development server
python app.py                                         # Start FastAPI server on port 7860

# Debug/development tools
python test_difficulty_softmax.py                     # Test difficulty selection
python test_softmax_service.py                       # Test word selection logic
python test_distribution_normalization.py            # Test distribution normalization across topics
```

### Backend Development (Node.js - Legacy)
```bash
cd backend
npm install
npm run dev          # Start Express server on http://localhost:3000
npm test             # Run tests
```

### Docker Deployment
```bash
# Build and run locally
docker build -t crossword-app .
docker run -p 7860:7860 -e NODE_ENV=production crossword-app

# Test deployment
curl http://localhost:7860/api/topics
curl http://localhost:7860/health
```

### Linting and Type Checking
```bash
# Python backend
cd crossword-app/backend-py
mypy src/           # Type checking (if mypy installed)
ruff src/           # Linting (if ruff installed)

# Frontend
cd crossword-app/frontend
npm run lint        # ESLint (if configured)
```

## Architecture Overview

### Full-Stack Components

**Frontend** (`crossword-app/frontend/`)
- React 18 with hooks and functional components
- Key components: `TopicSelector.jsx`, `PuzzleGrid.jsx`, `ClueList.jsx`, `DebugTab.jsx`
- Custom hook: `useCrossword.js` manages API calls and puzzle state
- Interactive crossword grid with cell navigation and solution reveal
- Debug tab for visualizing word selection process (when enabled)

**Python Backend** (`crossword-app/backend-py/` - Primary)
- FastAPI web framework serving both API and static frontend files
- AI-powered dynamic word generation using WordFreq + sentence-transformers
- No static word files - all words generated on-demand from 100K+ vocabulary
- WordNet-based clue generation with semantic definitions
- Comprehensive caching system for models, embeddings, and vocabulary

**Node.js Backend** (`backend/` - Legacy - Deprecated)
- Express.js with static JSON word files
- Original implementation, no longer actively maintained
- Used for comparison and fallback testing only

### Core Python Backend Components

**ThematicWordService** (`src/services/thematic_word_service.py`)
- Core AI-powered word generation engine using WordFreq database (100K+ words)
- Sentence-transformers (all-mpnet-base-v2) for semantic embeddings
- 10-tier frequency classification system with percentile-based difficulty selection
- Temperature-controlled softmax for balanced word selection randomness
- 50% word overgeneration strategy for better crossword grid fitting
- **Multi-topic intersection**: `_compute_multi_topic_similarities()` with vectorized soft minimum, geometric/harmonic means
- **Adaptive beta mechanism**: Automatically adjusts threshold (0.25→0.175→0.103...) to ensure 15+ word minimum
- **Performance optimized**: 40x speedup through vectorized operations over loop-based approach
- Key method: `generate_thematic_words()` - Returns words with semantic similarity scores and frequency tiers

**CrosswordGenerator** (`src/services/crossword_generator.py`)
- Main crossword generation algorithm using backtracking
- Integrates with ThematicWordService for AI word selection
- Sorts words by crossword suitability before grid placement
- Returns complete puzzle with grid, clues, and optional debug information

**WordNetClueGenerator** (`src/services/wordnet_clue_generator.py`)
- NLTK WordNet-based clue generation using semantic relationships
- Creates contextual crossword clues from word definitions
- Caches generated clues for performance optimization
- Handles multiple word senses and part-of-speech variations

**CrosswordGeneratorWrapper** (`src/services/crossword_generator_wrapper.py`)
- Wrapper service coordinating word generation and grid creation
- Manages integration between ThematicWordService and CrosswordGenerator
- Handles error recovery and fallback strategies

### Data Flow

1. **User Interaction** → React frontend (TopicSelector with topics/custom sentence/difficulty)
2. **API Request** → FastAPI backend (`src/routes/api.py`)
3. **Word Generation** → ThematicWordService (dynamic AI-powered word selection with multi-topic intersection)
4. **Clue Generation** → WordNetClueGenerator (semantic clue creation)
5. **Grid Generation** → CrosswordGenerator backtracking algorithm with word placement
6. **Response** → JSON with grid, clues, metadata, and optional debug data
7. **Frontend Rendering** → Interactive crossword grid with clues and debug visualization

### Critical Dependencies

**Frontend:**
- React 18, Vite (development/build)
- Node.js 18+ and npm 9+

**Python Backend (Primary):**
- FastAPI, uvicorn, pydantic (web framework)
- sentence-transformers, torch (AI word generation)
- wordfreq (vocabulary database)
- nltk (WordNet clue generation)
- scikit-learn (clustering and similarity)
- numpy (embeddings and mathematical operations)
- pytest, pytest-asyncio (testing)

**Node.js Backend (Legacy - Deprecated):**
- Express.js, cors, helmet
- JSON file-based word storage

The application requires AI dependencies for core functionality - no fallback to static word lists.

### API Endpoints

Python backend provides the following REST API:
- `GET /api/topics` - Returns 12 available topics (animals, geography, science, etc.)
- `POST /api/generate` - Generate crossword puzzle with topics/custom sentence/difficulty
- `POST /api/words` - Debug endpoint for testing word generation
- `GET /health` - Health check endpoint with service status
- `GET /api/topic/{topic}/words` - Generate words for specific topic (debug)

### Testing Strategy

**Python Backend Tests:**
- `test-unit/test_crossword_generator.py` - Grid generation logic and backtracking
- `test-unit/test_crossword_generator_wrapper.py` - Service integration testing
- `test-unit/test_api_routes.py` - FastAPI endpoints and request validation
- `test-integration/test_local.py` - End-to-end integration testing
- `test_integration_minimal.py` - Quick functionality test without heavy ML dependencies

**Multi-Topic Testing & Development Scripts:**
- `hack/test_soft_minimum_quick.py` - Quick soft minimum method verification
- `hack/test_optimized_soft_minimum.py` - Performance testing (40x speedup validation)
- `hack/debug_adaptive_beta_bug.py` - Adaptive beta mechanism debugging
- `hack/test_adaptive_fix.py` - Full vocabulary testing with adaptive beta
- `hack/test_simpler_case.py` - Compatible topic testing (animals + nature)
- All hack/ scripts use shared cache-dir for model loading consistency

**Frontend Tests:**
- Component testing with React Testing Library (if configured)
- E2E testing with Playwright/Cypress (if configured)

### Key Architecture Features

**Dynamic Word Generation:**
- No static word files - all words generated dynamically from WordFreq database
- 100K+ vocabulary with crossword-suitable filtering (3-12 letters, alphabetic only)
- AI-powered semantic similarity using sentence-transformers embeddings
- 10-tier frequency classification for difficulty-aware word selection

**Advanced Selection Logic:**
- Temperature-controlled softmax for balanced randomness
- 50% word overgeneration strategy to improve crossword grid fitting success
- Percentile-based difficulty mapping ensures consistent challenge levels
- Multi-theme vs single-theme processing modes for different puzzle styles

**Multi-Topic Intersection Methods:**
- **Soft Minimum (Default)**: Uses `-log(sum(exp(-beta * similarities))) / beta` formula to find words relevant to ALL topics
- **Adaptive Beta Mechanism**: Automatically adjusts beta parameter (10.0 → 7.0 → 4.9...) to ensure minimum word count (15+)
- **Alternative Methods**: geometric_mean, harmonic_mean, averaging for different intersection behaviors
- **Performance Optimized**: Vectorized implementation achieves 40x speedup over loop-based approach
- **Semantic Quality**: Filters problematic words like "ethology", "guns" for Art+Books, promotes true intersections like "literature"
- See `docs/multi_vector_word_finding.md` for detailed experimental analysis and method comparison

**Distribution Normalization:**
- **DISABLED BY DEFAULT** - Analysis shows non-normalized approach is better (see docs/distribution_normalization_analysis.md)
- Available normalization methods: similarity_range, composite_zscore, percentile_recentering
- Can be enabled with `ENABLE_DISTRIBUTION_NORMALIZATION=true` for experimentation
- When enabled, visible in debug tab with before/after comparison tooltips
- Non-normalized approach preserves natural semantic relationships and linguistic authenticity

**Comprehensive Caching:**
- Vocabulary, frequency, and embeddings cached for performance
- WordNet clue caching to avoid redundant semantic lookups
- Model cache shared across service instances

### Environment Configuration

**Python Backend (Production):**
```bash
NODE_ENV=production
PORT=7860
CACHE_DIR=/app/cache
THEMATIC_VOCAB_SIZE_LIMIT=100000
THEMATIC_MODEL_NAME=all-mpnet-base-v2
ENABLE_DEBUG_TAB=true
ENABLE_DISTRIBUTION_NORMALIZATION=false  # Default: disabled for better semantic authenticity
PYTHONPATH=/app/crossword-app/backend-py
PYTHONUNBUFFERED=1
```

**Frontend Development:**
```bash
VITE_API_BASE_URL=http://localhost:7860  # Points to Python backend
```

**Key Configuration Options:**
- `CACHE_DIR`: Directory for model cache, embeddings, and vocabulary files
- `THEMATIC_VOCAB_SIZE_LIMIT`: Maximum vocabulary size (default 100K)
- `ENABLE_DEBUG_TAB`: Enable debug visualization in frontend
- `THEMATIC_MODEL_NAME`: Sentence transformer model (default all-mpnet-base-v2)
- `ENABLE_DISTRIBUTION_NORMALIZATION`: Enable distribution normalization (default false - see analysis doc)
- `NORMALIZATION_METHOD`: Normalization method - similarity_range, composite_zscore, percentile_recentering (default similarity_range)

**Multi-Topic Intersection Configuration:**
- `MULTI_TOPIC_METHOD`: Multi-topic intersection method - soft_minimum, geometric_mean, harmonic_mean, averaging (default: soft_minimum)
- `SOFT_MIN_BETA`: Initial beta parameter for soft minimum method (default: 10.0)
- `SOFT_MIN_ADAPTIVE`: Enable adaptive beta mechanism for automatic threshold adjustment (default: true)
- `SOFT_MIN_MIN_WORDS`: Minimum words required before relaxing beta parameter (default: 15)
- `SOFT_MIN_MAX_RETRIES`: Maximum adaptive beta retries before giving up (default: 5)
- `SOFT_MIN_BETA_DECAY`: Beta decay factor per retry attempt (default: 0.7)

### Performance Notes

**Python Backend:**
- **Startup**: ~30-60 seconds (model download + cache creation)
- **Memory**: ~500MB-1GB (sentence-transformers + embeddings + vocabulary)
- **Response Time**: ~200-500ms (word generation + clue creation + grid fitting)
- **Cache Creation**: WordFreq vocabulary + embeddings generation is main startup bottleneck
- **Disk Usage**: ~500MB for full model cache (vocabulary, embeddings, models)

**Frontend:**
- **Development**: Hot reload with Vite (~200ms)
- **Build Time**: ~10-30 seconds for production build
- **Bundle Size**: Optimized with Vite tree-shaking

**Deployment:**
- Docker build time: ~5-10 minutes (includes frontend build + Python deps)
- Container size: ~1.5GB (includes ML models and dependencies)
- Hugging Face Spaces deployment: Automatic on git push

## Implementation Guidelines

### Development Priorities
- **No static word files** - All word/clue generation must be dynamic using AI services
- **No inference API solutions** - Use local model inference for better control and performance
- **Always run unit tests** after fixing bugs to ensure functionality
- **ThematicWordService is primary** - VectorSearchService is deprecated/unused
- **No fallback to static templates** - Application requires AI dependencies for core functionality

### Current Architecture Status
- ✅ **Fully AI-powered**: WordFreq + sentence-transformers + WordNet
- ✅ **Dynamic word generation**: 100K+ vocabulary with semantic filtering
- ✅ **Intelligent difficulty**: Percentile-based frequency classification
- ✅ **Multi-topic intersection**: Soft minimum method with adaptive beta for semantic quality
- ✅ **Performance optimized**: 40x speedup through vectorized operations
- ✅ **Debug visualization**: Optional debug tab for development/analysis
- ✅ **Comprehensive caching**: Models, embeddings, and vocabulary cached for performance
- ✅ **Modern stack**: FastAPI + React with Docker deployment ready
- the cache is present in root cache-dir/ folder. every program in hack folder should use this as the cache-dir for loading sentence transformer models