Hugging Face's logo

Spaces:
vimalk78
/
abc123
Running

App Files Community
abc123 / CLAUDE.md
vimalk78's picture
vimalk78
feat: add multi-topic intersection methods with adaptive beta for word selection
b05514b
preview code
|
raw
history blame contribute delete
13.9 kB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Structure

This is a full-stack AI-powered crossword puzzle generator:

  • Python Backend (crossword-app/backend-py/) - Primary implementation with dynamic word generation
  • React Frontend (crossword-app/frontend/) - Modern React app with interactive crossword UI
  • Node.js Backend (backend/) - Legacy implementation (deprecated)

Current deployment uses the Python backend with Docker containerization.

Development Commands

Frontend Development 

cd crossword-app/frontend
npm install
npm run dev          # Start development server on http://localhost:5173
npm run build        # Build for production
npm run preview      # Preview production build

Backend Development (Python - Primary) 

cd crossword-app/backend-py

# Testing
python run_tests.py                                    # Run all tests
pytest test-unit/ -v                                  # Run unit tests
pytest test-integration/ -v                           # Run integration tests
python test_integration_minimal.py                    # Quick test without ML deps

# Development server
python app.py                                         # Start FastAPI server on port 7860

# Debug/development tools
python test_difficulty_softmax.py                     # Test difficulty selection
python test_softmax_service.py                       # Test word selection logic
python test_distribution_normalization.py            # Test distribution normalization across topics

Backend Development (Node.js - Legacy) 

cd backend
npm install
npm run dev          # Start Express server on http://localhost:3000
npm test             # Run tests

Docker Deployment 

# Build and run locally
docker build -t crossword-app .
docker run -p 7860:7860 -e NODE_ENV=production crossword-app

# Test deployment
curl http://localhost:7860/api/topics
curl http://localhost:7860/health

Linting and Type Checking 

# Python backend
cd crossword-app/backend-py
mypy src/           # Type checking (if mypy installed)
ruff src/           # Linting (if ruff installed)

# Frontend
cd crossword-app/frontend
npm run lint        # ESLint (if configured)

Architecture Overview

Full-Stack Components

Frontend (crossword-app/frontend/)

  • React 18 with hooks and functional components
  • Key components: TopicSelector.jsx, PuzzleGrid.jsx, ClueList.jsx, DebugTab.jsx
  • Custom hook: useCrossword.js manages API calls and puzzle state
  • Interactive crossword grid with cell navigation and solution reveal
  • Debug tab for visualizing word selection process (when enabled)

Python Backend (crossword-app/backend-py/ - Primary)

  • FastAPI web framework serving both API and static frontend files
  • AI-powered dynamic word generation using WordFreq + sentence-transformers
  • No static word files - all words generated on-demand from 100K+ vocabulary
  • WordNet-based clue generation with semantic definitions
  • Comprehensive caching system for models, embeddings, and vocabulary

Node.js Backend (backend/ - Legacy - Deprecated)

  • Express.js with static JSON word files
  • Original implementation, no longer actively maintained
  • Used for comparison and fallback testing only

Core Python Backend Components

ThematicWordService (src/services/thematic_word_service.py)

  • Core AI-powered word generation engine using WordFreq database (100K+ words)
  • Sentence-transformers (all-mpnet-base-v2) for semantic embeddings
  • 10-tier frequency classification system with percentile-based difficulty selection
  • Temperature-controlled softmax for balanced word selection randomness
  • 50% word overgeneration strategy for better crossword grid fitting
  • Multi-topic intersection: _compute_multi_topic_similarities() with vectorized soft minimum, geometric/harmonic means
  • Adaptive beta mechanism: Automatically adjusts threshold (0.25→0.175→0.103...) to ensure 15+ word minimum
  • Performance optimized: 40x speedup through vectorized operations over loop-based approach
  • Key method: generate_thematic_words() - Returns words with semantic similarity scores and frequency tiers

CrosswordGenerator (src/services/crossword_generator.py)

  • Main crossword generation algorithm using backtracking
  • Integrates with ThematicWordService for AI word selection
  • Sorts words by crossword suitability before grid placement
  • Returns complete puzzle with grid, clues, and optional debug information

WordNetClueGenerator (src/services/wordnet_clue_generator.py)

  • NLTK WordNet-based clue generation using semantic relationships
  • Creates contextual crossword clues from word definitions
  • Caches generated clues for performance optimization
  • Handles multiple word senses and part-of-speech variations

CrosswordGeneratorWrapper (src/services/crossword_generator_wrapper.py)

  • Wrapper service coordinating word generation and grid creation
  • Manages integration between ThematicWordService and CrosswordGenerator
  • Handles error recovery and fallback strategies

Data Flow

  1. User Interaction → React frontend (TopicSelector with topics/custom sentence/difficulty)
  2. API Request → FastAPI backend (src/routes/api.py)
  3. Word Generation → ThematicWordService (dynamic AI-powered word selection with multi-topic intersection)
  4. Clue Generation → WordNetClueGenerator (semantic clue creation)
  5. Grid Generation → CrosswordGenerator backtracking algorithm with word placement
  6. Response → JSON with grid, clues, metadata, and optional debug data
  7. Frontend Rendering → Interactive crossword grid with clues and debug visualization

Critical Dependencies

Frontend:

  • React 18, Vite (development/build)
  • Node.js 18+ and npm 9+

Python Backend (Primary):

  • FastAPI, uvicorn, pydantic (web framework)
  • sentence-transformers, torch (AI word generation)
  • wordfreq (vocabulary database)
  • nltk (WordNet clue generation)
  • scikit-learn (clustering and similarity)
  • numpy (embeddings and mathematical operations)
  • pytest, pytest-asyncio (testing)

Node.js Backend (Legacy - Deprecated):

  • Express.js, cors, helmet
  • JSON file-based word storage

The application requires AI dependencies for core functionality - no fallback to static word lists.

API Endpoints

Python backend provides the following REST API:

  • GET /api/topics - Returns 12 available topics (animals, geography, science, etc.)
  • POST /api/generate - Generate crossword puzzle with topics/custom sentence/difficulty
  • POST /api/words - Debug endpoint for testing word generation
  • GET /health - Health check endpoint with service status
  • GET /api/topic/{topic}/words - Generate words for specific topic (debug)

Testing Strategy

Python Backend Tests:

  • test-unit/test_crossword_generator.py - Grid generation logic and backtracking
  • test-unit/test_crossword_generator_wrapper.py - Service integration testing
  • test-unit/test_api_routes.py - FastAPI endpoints and request validation
  • test-integration/test_local.py - End-to-end integration testing
  • test_integration_minimal.py - Quick functionality test without heavy ML dependencies

Multi-Topic Testing & Development Scripts:

  • hack/test_soft_minimum_quick.py - Quick soft minimum method verification
  • hack/test_optimized_soft_minimum.py - Performance testing (40x speedup validation)
  • hack/debug_adaptive_beta_bug.py - Adaptive beta mechanism debugging
  • hack/test_adaptive_fix.py - Full vocabulary testing with adaptive beta
  • hack/test_simpler_case.py - Compatible topic testing (animals + nature)
  • All hack/ scripts use shared cache-dir for model loading consistency

Frontend Tests:

  • Component testing with React Testing Library (if configured)
  • E2E testing with Playwright/Cypress (if configured)

Key Architecture Features

Dynamic Word Generation:

  • No static word files - all words generated dynamically from WordFreq database
  • 100K+ vocabulary with crossword-suitable filtering (3-12 letters, alphabetic only)
  • AI-powered semantic similarity using sentence-transformers embeddings
  • 10-tier frequency classification for difficulty-aware word selection

Advanced Selection Logic:

  • Temperature-controlled softmax for balanced randomness
  • 50% word overgeneration strategy to improve crossword grid fitting success
  • Percentile-based difficulty mapping ensures consistent challenge levels
  • Multi-theme vs single-theme processing modes for different puzzle styles

Multi-Topic Intersection Methods:

  • Soft Minimum (Default): Uses -log(sum(exp(-beta * similarities))) / beta formula to find words relevant to ALL topics
  • Adaptive Beta Mechanism: Automatically adjusts beta parameter (10.0 → 7.0 → 4.9...) to ensure minimum word count (15+)
  • Alternative Methods: geometric_mean, harmonic_mean, averaging for different intersection behaviors
  • Performance Optimized: Vectorized implementation achieves 40x speedup over loop-based approach
  • Semantic Quality: Filters problematic words like "ethology", "guns" for Art+Books, promotes true intersections like "literature"
  • See docs/multi_vector_word_finding.md for detailed experimental analysis and method comparison

Distribution Normalization:

  • DISABLED BY DEFAULT - Analysis shows non-normalized approach is better (see docs/distribution_normalization_analysis.md)
  • Available normalization methods: similarity_range, composite_zscore, percentile_recentering
  • Can be enabled with ENABLE_DISTRIBUTION_NORMALIZATION=true for experimentation
  • When enabled, visible in debug tab with before/after comparison tooltips
  • Non-normalized approach preserves natural semantic relationships and linguistic authenticity

Comprehensive Caching:

  • Vocabulary, frequency, and embeddings cached for performance
  • WordNet clue caching to avoid redundant semantic lookups
  • Model cache shared across service instances

Environment Configuration

Python Backend (Production):

NODE_ENV=production
PORT=7860
CACHE_DIR=/app/cache
THEMATIC_VOCAB_SIZE_LIMIT=100000
THEMATIC_MODEL_NAME=all-mpnet-base-v2
ENABLE_DEBUG_TAB=true
ENABLE_DISTRIBUTION_NORMALIZATION=false  # Default: disabled for better semantic authenticity
PYTHONPATH=/app/crossword-app/backend-py
PYTHONUNBUFFERED=1

Frontend Development:

VITE_API_BASE_URL=http://localhost:7860  # Points to Python backend

Key Configuration Options:

  • CACHE_DIR: Directory for model cache, embeddings, and vocabulary files
  • THEMATIC_VOCAB_SIZE_LIMIT: Maximum vocabulary size (default 100K)
  • ENABLE_DEBUG_TAB: Enable debug visualization in frontend
  • THEMATIC_MODEL_NAME: Sentence transformer model (default all-mpnet-base-v2)
  • ENABLE_DISTRIBUTION_NORMALIZATION: Enable distribution normalization (default false - see analysis doc)
  • NORMALIZATION_METHOD: Normalization method - similarity_range, composite_zscore, percentile_recentering (default similarity_range)

Multi-Topic Intersection Configuration:

  • MULTI_TOPIC_METHOD: Multi-topic intersection method - soft_minimum, geometric_mean, harmonic_mean, averaging (default: soft_minimum)
  • SOFT_MIN_BETA: Initial beta parameter for soft minimum method (default: 10.0)
  • SOFT_MIN_ADAPTIVE: Enable adaptive beta mechanism for automatic threshold adjustment (default: true)
  • SOFT_MIN_MIN_WORDS: Minimum words required before relaxing beta parameter (default: 15)
  • SOFT_MIN_MAX_RETRIES: Maximum adaptive beta retries before giving up (default: 5)
  • SOFT_MIN_BETA_DECAY: Beta decay factor per retry attempt (default: 0.7)

Performance Notes

Python Backend:

  • Startup: ~30-60 seconds (model download + cache creation)
  • Memory: ~500MB-1GB (sentence-transformers + embeddings + vocabulary)
  • Response Time: ~200-500ms (word generation + clue creation + grid fitting)
  • Cache Creation: WordFreq vocabulary + embeddings generation is main startup bottleneck
  • Disk Usage: ~500MB for full model cache (vocabulary, embeddings, models)

Frontend:

  • Development: Hot reload with Vite (~200ms)
  • Build Time: ~10-30 seconds for production build
  • Bundle Size: Optimized with Vite tree-shaking

Deployment:

  • Docker build time: ~5-10 minutes (includes frontend build + Python deps)
  • Container size: ~1.5GB (includes ML models and dependencies)
  • Hugging Face Spaces deployment: Automatic on git push

Implementation Guidelines

Development Priorities

  • No static word files - All word/clue generation must be dynamic using AI services
  • No inference API solutions - Use local model inference for better control and performance
  • Always run unit tests after fixing bugs to ensure functionality
  • ThematicWordService is primary - VectorSearchService is deprecated/unused
  • No fallback to static templates - Application requires AI dependencies for core functionality

Current Architecture Status

  • ✅ Fully AI-powered: WordFreq + sentence-transformers + WordNet
  • ✅ Dynamic word generation: 100K+ vocabulary with semantic filtering
  • ✅ Intelligent difficulty: Percentile-based frequency classification
  • ✅ Multi-topic intersection: Soft minimum method with adaptive beta for semantic quality
  • ✅ Performance optimized: 40x speedup through vectorized operations
  • ✅ Debug visualization: Optional debug tab for development/analysis
  • ✅ Comprehensive caching: Models, embeddings, and vocabulary cached for performance
  • ✅ Modern stack: FastAPI + React with Docker deployment ready
  • the cache is present in root cache-dir/ folder. every program in hack folder should use this as the cache-dir for loading sentence transformer models