121 lines
2.5 KiB
Markdown
121 lines
2.5 KiB
Markdown
# rmtPocketWatcher Backend
|
|
|
|
Backend scraping and API service for rmtPocketWatcher.
|
|
|
|
## Features
|
|
|
|
- Playwright-based web scraping for Eldorado and PlayerAuctions
|
|
- Automatic retry logic (3 attempts with exponential backoff)
|
|
- Scheduled scraping every 5 minutes (configurable)
|
|
- PostgreSQL + Prisma for data storage
|
|
- REST API + WebSocket for real-time updates
|
|
- Tracks all seller listings over time
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
# Install dependencies
|
|
npm install
|
|
|
|
# Install Playwright browsers
|
|
npx playwright install chromium
|
|
|
|
# Setup environment
|
|
cp .env.example .env
|
|
# Edit .env with your PostgreSQL connection string
|
|
|
|
# Generate Prisma client
|
|
npm run db:generate
|
|
|
|
# Run database migrations
|
|
npm run db:migrate
|
|
```
|
|
|
|
## Quick Start with Docker Compose
|
|
|
|
The easiest way to run the entire stack:
|
|
|
|
```bash
|
|
# From the project root
|
|
docker-compose up -d
|
|
|
|
# View logs
|
|
docker-compose logs -f backend
|
|
|
|
# Stop everything
|
|
docker-compose down
|
|
```
|
|
|
|
This will:
|
|
- Start PostgreSQL database
|
|
- Run database migrations
|
|
- Start the backend API server
|
|
- Begin scraping every 5 minutes
|
|
|
|
## Local Development Setup
|
|
|
|
If you want to run locally without Docker:
|
|
|
|
```bash
|
|
# Start PostgreSQL with Docker
|
|
docker-compose up -d postgres
|
|
|
|
# Install dependencies
|
|
cd backend
|
|
npm install
|
|
|
|
# Setup environment
|
|
cp .env.example .env
|
|
# Edit .env if needed
|
|
|
|
# Generate Prisma client and push schema
|
|
npm run db:generate
|
|
npm run db:push
|
|
|
|
# Install Playwright browsers
|
|
npx playwright install chromium
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Development mode with hot reload
|
|
npm run dev
|
|
|
|
# Manual scrape test (no database)
|
|
npm run scrape
|
|
|
|
# Build for production
|
|
npm run build
|
|
|
|
# Run production build
|
|
npm start
|
|
|
|
# Database management
|
|
npm run db:studio # Open Prisma Studio
|
|
npm run db:push # Push schema changes without migration
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Prices
|
|
- `GET /api/prices/latest` - Get all current listings
|
|
- `GET /api/prices/lowest` - Get current lowest price
|
|
- `GET /api/prices/by-seller?seller=&platform=` - Get seller's price history
|
|
- `GET /api/prices/by-platform?platform=` - Get all listings from a platform
|
|
- `GET /api/prices/history?from=&to=&seller=&platform=` - Historical data
|
|
|
|
### Index
|
|
- `GET /api/index/history?range=7d|30d|90d|all` - Lowest price over time
|
|
- `GET /api/index/ws` - WebSocket for real-time updates
|
|
|
|
### Health
|
|
- `GET /health` - Server health check
|
|
|
|
## Architecture
|
|
|
|
- `src/scrapers/` - Playwright scraping modules
|
|
- `src/api/` - Fastify REST + WebSocket API
|
|
- `src/database/` - Prisma client and repository
|
|
- `prisma/` - Database schema and migrations
|