RAG Email Search is an AI-powered application that enables semantic search through email conversations using advanced vector embeddings and retrieval-augmented generation (RAG) technology.
This repository is built using Turborepo for efficient monorepo management, allowing seamless development across multiple applications and shared packages.
web- A Next.js app for the frontend (Next.js Docs)api- An Express.js app for the backend (Express.js Docs)
@repo/ui- A React component library shared across the monorepo@repo/supabase- Supabase setup@repo/eslint-config- Shared ESLint configuration for consistent linting across packages@repo/typescript-config- Shared TypeScript configurations for consistency
- 🔍 AI-Powered Email Search: Semantic search through email conversations using vector embeddings
- 🧠 RAG (Retrieval-Augmented Generation): Intelligent question-answering based on email content
- ⚡ Real-time Streaming: Live search results with Server-Sent Events
- 🎯 Vector Search: Pinecone-powered semantic similarity matching
- 🤖 OpenAI Integration: Advanced AI responses using GPT models
- 📧 Email Thread Analysis: Search through question-answer pairs in email threads (With Mock data)
- 🛡️ Enterprise Security: SQL injection prevention, input validation, and CORS protection
- 🧪 Comprehensive Testing: 52+ tests covering models, services, middleware, and repositories
- 📊 Type Safety: Full TypeScript implementation across frontend and backend
- 🚀 Production Ready: Automated testing, pre-push hooks, and quality gates
┌─────────────────────────────────────┐
│ 🌐 API Layer │
│ (Controllers + Routes) │
├─────────────────────────────────────┤
│ 🛡️ Middleware Layer │
│ (Validation + Security) │
├─────────────────────────────────────┤
│ 🧠 Business Logic │
│ (Services) │
├─────────────────────────────────────┤
│ 🗃️ Data Access │
│ (Repositories) │
├─────────────────────────────────────┤
│ 🗄️ Database Layer │
│ (PostgreSQL) │
└─────────────────────────────────────┘
apps/api/src/
├── __tests__/ # 🧪 Testing Infrastructure
│ ├── controllers/ # Controller tests
│ ├── middleware/ # Middleware tests
│ ├── models/ # Model tests
│ ├── repositories/ # Repository tests
│ ├── services/ # Service tests
│ └── simple.test.ts # Basic functionality tests
│
├── controllers/ # 🎮 Request Handlers
│ └── searchController.ts # Search API endpoints
│
├── data/ # 📊 Static Data
│ └── email.json # Sample email data
│
├── db/ # 🗄️ Database Layer
│ └── pgClient.ts # PostgreSQL connection
│
├── errors/ # ⚠️ Error Handling
│ └── HttpException.ts # Custom error classes
│
├── middleware/ # 🛡️ Security & Validation
│ ├── validation.ts # Input validation & sanitization
│ └── rateLimiting.ts # Rate limiting middleware
│
├── models/ # 📋 Data Models
│ ├── metadata.ts # QAMetadata interface
│ ├── openai.ts # OpenAI model definitions
│ └── search.ts # SearchMatch & EmailSource interfaces
│
├── repositories/ # 🗃️ Data Access Layer
│ ├── messages.ts # Message database operations
│ └── threads.ts # Thread database operations
│
├── routes/ # 🛣️ API Routing
│ ├── index.ts # Main router configuration
│ └── searchRoutes.ts # Search-specific routes
│
├── scripts/ # 🔧 Utility Scripts
│ └── seed.ts # Database seeding script
│
├── services/ # 🧠 Business Logic
│ ├── mainSearchSevice.ts # Main search orchestration
│ ├── promptServices.ts # AI prompt generation
│ ├── ragServices.ts # RAG (Retrieval-Augmented Generation)
│ ├── responseServices.ts # Response formatting
│ └── searchServices.ts # Vector search operations
│
├── utils/ # 🔧 Utility Functions
│ ├── openai.ts # OpenAI API integration
│ └── vector.ts # Vector database (Pinecone) integration
│
├── index.ts # 🚀 Application Entry Point
└── mainServer.ts # 🌐 Express Server Configuration
┌─────────────────────────────────────┐
│ 🌐 App Layer │
│ (Next.js Pages) │
├─────────────────────────────────────┤
│ 🧩 Components │
│ (Features + UI) │
├─────────────────────────────────────┤
│ 🎣 Hooks Layer │
│ (State Management) │
├─────────────────────────────────────┤
│ 🔧 Utilities │
│ (Helpers + Types) │
└─────────────────────────────────────┘
apps/web/
├── app/ # 🌐 Next.js App Router
│ └── api/ # API Routes
│ └── chat/ # Chat API endpoint
│
├── components/ # 🧩 React Components
│ ├── __tests__/ # Component Tests
│ │ ├── features/ # Feature component tests
│ │ └── ui/ # UI component tests
│ ├── features/ # Feature Components
│ └── ui/ # Reusable UI Components
│
├── hooks/ # 🎣 Custom React Hooks
│
├── lib/ # 🔧 Utility Functions
│
├── public/ # 📁 Static Assets
│
├── types/ # 📋 TypeScript Types
│
└── Configuration Files # ⚙️ Project Setup
├── package.json
├── next.config.ts
├── tailwind.config.js
├── tsconfig.json
└── jest.config.js
Testing Categories:
- 🔬 Unit Tests - Isolated component testing
- 🔗 Integration Tests - Component interaction testing
- 🧪 Service Tests - Business logic validation
Before Testing Implementation:
- ❌ No validation of data structures
- ❌ Manual testing required for every change
- ❌ Security vulnerabilities undetected
- ❌ No regression detection
- ❌ Type mismatches
Now with Comprehensive Testing:
- ✅ 52+ tests automatically validate your code
- ✅ Model tests ensure data structure integrity
- ✅ Security validated through middleware tests
- ✅ Database operations tested with proper mocking
- ✅ Pre-push hooks prevent broken code
- ✅ Input validation comprehensive
- ✅ Regression prevention through automated tests
- ✅ Husky automated testing with Git hooks
components/
├── __tests__/ # All test files are grouped here
│ ├── features/ # Tests for feature-level components
│ └── ui/ # Tests for UI-level components
│
├── features/ # Business or feature-related components
│ └── chat-form.tsx
│
└── ui/ # Reusable UI components
├── accordion.tsx
├── button.tsx
├── chat-form.tsx
├── textarea.tsx
└── tooltip.tsx
Key Test Focus:
chat-form.test.tsx- Verifies the most important user flow: typing a message, submitting it, and seeing it rendered- Streaming Response Testing - Ensures streaming assistant responses are handled correctly
- User Experience Validation - Prevents silent breaks during future changes or refactors
yarn installThis section is for users who have just cloned the repository and need to set up the local database. This project uses Supabase for database management, with migrations handled through the Supabase CLI. Follow these steps to get your database up and running.
You can open up http://127.0.0.1:54323 to open supabase studio.
# for latest .env take a look at env.excample on each repo
DATABASE_URL=
SUPABASE_URL=http://localhost:54321
SUPABASE_KEY=
NODE_ENV=development
OPENAI_API_KEY=
Here are some useful scripts for development:
yarn dev- Start both the Next.js frontend and Express.js backendyarn build- Build all apps and packages
If you find a bug or have a feature request, please open an issue or submit a pull request.
This project is licensed under the MIT License.