Intial Version
This commit is contained in:
14
.env
Normal file
14
.env
Normal file
@@ -0,0 +1,14 @@
|
||||
# Database
|
||||
DATABASE_URL="postgresql://rmtpw:rmtpw_password@localhost:5432/rmtpocketwatcher?schema=public"
|
||||
|
||||
# Scraper Configuration
|
||||
SCRAPE_INTERVAL_MINUTES=5
|
||||
SCRAPER_HEADLESS=true
|
||||
SCRAPER_TIMEOUT=30000
|
||||
SCRAPER_MAX_RETRIES=3
|
||||
|
||||
# Server
|
||||
PORT=3000
|
||||
HOST=0.0.0.0
|
||||
NODE_ENV=development
|
||||
LOG_LEVEL=info
|
||||
22
.kiro/steering/product.md
Normal file
22
.kiro/steering/product.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Product Overview
|
||||
|
||||
rmtPocketWatcher is a cross-platform desktop application that tracks real-money trading (RMT) prices for Star Citizen AUEC (in-game currency). The app provides a Bloomberg-style terminal interface displaying a real-time AUEC Price Index derived from multiple vendor sources. It is developed and offered by the Lambda Banking Conglomerate a Star Citizen Organization
|
||||
|
||||
## Core Functionality
|
||||
|
||||
- Scrapes AUEC prices from Eldorado and PlayerAuctions every 5 minutes
|
||||
- Tracks all individual seller listings with platform, seller name, and USD per 1M AUEC
|
||||
- Computes lowest price across all vendors and sellers
|
||||
- Displays real-time price data with seller-level granularity
|
||||
- Historical charts showing price trends by seller, platform, or lowest overall
|
||||
- Vendor comparison table with sortable/filterable seller listings
|
||||
- Client-side alerts for price thresholds with native notifications
|
||||
- Long-term historical data storage for all listings (not just lowest)
|
||||
|
||||
## Key Constraints
|
||||
|
||||
- Desktop-only (no mobile or web versions)
|
||||
- No buying/selling functionality within the app
|
||||
- Alerts are evaluated client-side, not on the backend
|
||||
- Focus on data visualization and tracking, not RMT facilitation
|
||||
- Store all seller listings over time for historical comparison and trend analysis
|
||||
66
.kiro/steering/structure.md
Normal file
66
.kiro/steering/structure.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Project Structure
|
||||
|
||||
This is a monorepo-style project with separate backend and frontend applications.
|
||||
|
||||
## Expected Organization
|
||||
|
||||
```
|
||||
/
|
||||
├── backend/ # TypeScript backend server
|
||||
│ ├── src/
|
||||
│ │ ├── scrapers/ # Playwright scraping modules
|
||||
│ │ ├── api/ # REST endpoints
|
||||
│ │ ├── websocket/ # WebSocket handlers
|
||||
│ │ ├── database/ # Prisma/Drizzle schema & migrations
|
||||
│ │ ├── services/ # Business logic (price index calculation)
|
||||
│ │ └── utils/ # Shared utilities
|
||||
│ ├── tests/
|
||||
│ ├── Dockerfile
|
||||
│ └── package.json
|
||||
│
|
||||
├── electron-app/ # Electron + React frontend
|
||||
│ ├── src/
|
||||
│ │ ├── main/ # Electron main process
|
||||
│ │ ├── renderer/ # React UI components
|
||||
│ │ │ ├── components/
|
||||
│ │ │ ├── pages/
|
||||
│ │ │ ├── hooks/
|
||||
│ │ │ └── store/ # Zustand/Recoil state
|
||||
│ │ └── shared/ # IPC types & shared code
|
||||
│ ├── tests/
|
||||
│ └── package.json
|
||||
│
|
||||
├── shared/ # Shared TypeScript types/interfaces
|
||||
│ └── types/
|
||||
│
|
||||
├── .kiro/ # Kiro AI assistant configuration
|
||||
│ └── steering/ # Project guidance documents
|
||||
│
|
||||
├── PRD.md # Product Requirements Document
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
- **Backend**: Modular service-oriented architecture
|
||||
- Scrapers are isolated, containerized services
|
||||
- API layer is stateless for horizontal scaling
|
||||
- TimescaleDB handles time-series data efficiently
|
||||
|
||||
- **Frontend**: Component-based React architecture
|
||||
- Sandboxed renderer process for security
|
||||
- Secure IPC messaging between main and renderer
|
||||
- Client-side alert evaluation logic
|
||||
|
||||
- **Database Schema**: Three main tables
|
||||
- `raw_vendor_prices`: Individual vendor listings
|
||||
- `price_index`: Computed median index values
|
||||
- `scrape_log`: Monitoring and debugging
|
||||
|
||||
## Key Conventions
|
||||
|
||||
- All code in TypeScript with strict type checking
|
||||
- Scrapers include retry logic (3 attempts) and error handling
|
||||
- WebSocket auto-reconnect logic in Electron app
|
||||
- Signed binaries for all platform distributions
|
||||
- No remote code evaluation in Electron (security)
|
||||
55
.kiro/steering/tech.md
Normal file
55
.kiro/steering/tech.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Technology Stack
|
||||
|
||||
## Backend
|
||||
|
||||
- **Language**: TypeScript (Node.js)
|
||||
- **Framework**: Fastify
|
||||
- **Scraping**: Playwright (headless Chromium)
|
||||
- **Database**: PostgreSQL + TimescaleDB extension
|
||||
- **ORM**: Prisma or Drizzle
|
||||
- **Job Scheduling**: Node Scheduler or BullMQ
|
||||
- **WebSockets**: Native `ws` or Fastify WS plugin
|
||||
- **Deployment**: Docker containers
|
||||
|
||||
## Frontend (Electron Desktop App)
|
||||
|
||||
- **Framework**: Electron 30+
|
||||
- **UI Library**: NextJS? + TypeScript/TSX
|
||||
- **Build Tool**: Vite
|
||||
- **Styling**: TailwindCSS
|
||||
- **Charts**: Recharts, ECharts, or TradingView Charting Library
|
||||
- **State Management**: Zustand or Recoil
|
||||
- **Auto-updates**: electron-updater
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit Tests**: Jest
|
||||
- **E2E Tests**: Playwright
|
||||
- **API Integration**: Supertest or Pact
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
# Backend
|
||||
npm run dev # Start development server
|
||||
npm run build # Build TypeScript
|
||||
npm run test # Run Jest tests
|
||||
npm run scrape # Manual scrape trigger
|
||||
|
||||
# Frontend (Electron)
|
||||
npm run dev # Start Electron in dev mode
|
||||
npm run build # Build production app
|
||||
npm run package # Package for distribution
|
||||
npm run test # Run tests
|
||||
|
||||
# Database
|
||||
npm run migrate # Run database migrations
|
||||
npm run seed # Seed initial data
|
||||
```
|
||||
|
||||
## Performance Requirements
|
||||
|
||||
- API latency < 50ms for standard queries
|
||||
- Support 5000+ concurrent WebSocket connections
|
||||
- Desktop app must handle 50k+ chart datapoints smoothly
|
||||
- Scraping frequency: every 5 minutes (configurable)
|
||||
2
.vscode/settings.json
vendored
Normal file
2
.vscode/settings.json
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{
|
||||
}
|
||||
381
PRD.md
Normal file
381
PRD.md
Normal file
@@ -0,0 +1,381 @@
|
||||
Product Requirements Document (PRD)
|
||||
Star Citizen AUEC Price Index Tracker
|
||||
|
||||
Unified Modern TypeScript Stack + Electron Desktop Application
|
||||
|
||||
1. Product Overview
|
||||
|
||||
This product is a cross-platform desktop application (Electron + React/TSX) that displays real-money trading (RMT) prices for Star Citizen AUEC. A modern TypeScript-based backend performs scraping, normalization, time-series storage, and API hosting.
|
||||
|
||||
The goal is to generate a reliable AUEC Price Index, similar in spirit to a financial market index, and visualize it in a Bloomberg-style terminal UI that includes tables, charts, and alerts.
|
||||
|
||||
Alerts are evaluated client-side, based on data received from the backend.
|
||||
|
||||
2. Goals & Non-Goals
|
||||
Goals
|
||||
|
||||
Scrape AUEC prices from:
|
||||
|
||||
Eldorado
|
||||
|
||||
PlayerAuctions
|
||||
|
||||
Normalize them into a USD ↔ AUEC Price Index.
|
||||
|
||||
Provide REST + WebSocket API to clients.
|
||||
|
||||
Display Bloomberg-like UI using Electron + React.
|
||||
|
||||
Allow users to set local alerts (no backend evaluation).
|
||||
|
||||
Store long-term historical price data.
|
||||
|
||||
Non-Goals
|
||||
|
||||
Buying/selling AUEC inside the app.
|
||||
|
||||
Mobile or web app (desktop only).
|
||||
|
||||
Legal/regulatory RMT analysis.
|
||||
|
||||
Server-side alert evaluation.
|
||||
|
||||
3. System Architecture
|
||||
[Scraper Services - TypeScript + Playwright]
|
||||
|
|
||||
v
|
||||
[Backend API - Fastify/NestJS - TypeScript]
|
||||
- Normalize prices
|
||||
- Store historical data (PostgreSQL + TimescaleDB)
|
||||
- Expose REST + WebSocket
|
||||
|
|
||||
v
|
||||
[Electron Desktop App - React/TSX UI]
|
||||
- Bloomberg-style terminal interface
|
||||
- Client-side alerts
|
||||
- Charting & vendor tables
|
||||
|
||||
4. Detailed Requirements
|
||||
4.1 Scraping Module (Backend)
|
||||
Technology
|
||||
|
||||
TypeScript
|
||||
|
||||
Playwright for dynamic scraping
|
||||
|
||||
Node Scheduler or BullMQ for job scheduling
|
||||
|
||||
Deployed as containers (Docker)
|
||||
|
||||
Scraping Frequency
|
||||
|
||||
Every 5 minutes (configurable per environment)
|
||||
|
||||
Scraping Targets
|
||||
|
||||
Eldorado AUEC Market
|
||||
https://www.eldorado.gg/star-citizen-auec/g/141-0-0
|
||||
|
||||
PlayerAuctions AUEC Market
|
||||
https://www.playerauctions.com/star-citizen-auec/
|
||||
|
||||
Scraped Data Fields
|
||||
|
||||
Vendor name
|
||||
|
||||
Listing price (USD)
|
||||
|
||||
AUEC amount per listing
|
||||
|
||||
Derived:
|
||||
|
||||
USD per 1 million AUEC
|
||||
|
||||
AUEC per USD
|
||||
|
||||
Timestamp
|
||||
|
||||
Listing min/max spread (if available)
|
||||
|
||||
Scraper Requirements
|
||||
|
||||
Headless mode with Playwright Chromium
|
||||
|
||||
Randomized UA + viewport
|
||||
|
||||
Retry logic (3 attempts)
|
||||
|
||||
Automatically detect layout changes (optional heuristics)
|
||||
|
||||
Error Handling
|
||||
|
||||
Failures logged to DB
|
||||
|
||||
Continue schedule regardless of scrape results
|
||||
|
||||
4.2 Backend Server (TypeScript)
|
||||
Technology Stack
|
||||
|
||||
Node.js + TypeScript
|
||||
|
||||
Fastify or NestJS (dev’s choice)
|
||||
|
||||
WebSockets via native ws or Fastify WS plugin
|
||||
|
||||
PostgreSQL + TimescaleDB for time-series storage
|
||||
|
||||
Prisma or Drizzle ORM
|
||||
|
||||
Backend Responsibilities
|
||||
|
||||
Accept scraped data
|
||||
|
||||
Compute Price Index
|
||||
|
||||
Store time-series data
|
||||
|
||||
Serve historical & latest prices
|
||||
|
||||
Push WS updates to connected Electron clients
|
||||
|
||||
Price Index Formula
|
||||
|
||||
Default:
|
||||
|
||||
lowest_price = min(USD_per_million_AUEC_from_all_vendors)
|
||||
|
||||
|
||||
All individual listings are stored with seller and platform information for historical tracking and comparison.
|
||||
|
||||
REST API Endpoints
|
||||
|
||||
GET /prices/latest - Get all current listings with seller info
|
||||
|
||||
GET /prices/lowest - Get current lowest price across all vendors
|
||||
|
||||
GET /prices/by-seller?seller=&platform= - Get specific seller's price history
|
||||
|
||||
GET /prices/by-platform?platform= - Get all listings from a platform
|
||||
|
||||
GET /prices/history?from=&to=&seller=&platform= - Historical data with filters
|
||||
|
||||
GET /index/history?range=7d|30d|90d|all - Lowest price over time
|
||||
|
||||
WebSocket Channels
|
||||
|
||||
/ws/index → broadcast each new scrape
|
||||
|
||||
/ws/vendors → vendor-level updates
|
||||
|
||||
Performance Requirements
|
||||
|
||||
Handle 5000+ concurrent WS connections
|
||||
|
||||
API latency < 50ms for standard queries
|
||||
|
||||
4.3 Database Layer
|
||||
Database
|
||||
|
||||
PostgreSQL (core DB)
|
||||
|
||||
TimescaleDB extension for time-series compression & fast queries
|
||||
|
||||
Schema
|
||||
raw_vendor_prices
|
||||
field type
|
||||
id uuid
|
||||
timestamp timestamptz
|
||||
vendor text (eldorado/playerauctions)
|
||||
seller_name text
|
||||
usd_price numeric
|
||||
auec_amount numeric
|
||||
usd_per_million numeric
|
||||
delivery_time text
|
||||
url text
|
||||
price_index
|
||||
field type
|
||||
timestamp timestamptz
|
||||
lowest_price numeric
|
||||
vendor text
|
||||
seller_name text
|
||||
scrape_log
|
||||
field type
|
||||
timestamp timestamptz
|
||||
status text (success/failure)
|
||||
message text
|
||||
runtime_ms int
|
||||
|
||||
4.4 Electron Desktop App (React/TSX)
|
||||
Technology Stack
|
||||
|
||||
Electron 30+
|
||||
|
||||
React + TypeScript/TSX
|
||||
|
||||
Vite build tooling
|
||||
|
||||
TailwindCSS for styling
|
||||
|
||||
Recharts, ECharts, or TradingView Charting Library for visuals
|
||||
|
||||
Zustand/Recoil for state management
|
||||
|
||||
Main Features
|
||||
Dashboard (Bloomberg-style)
|
||||
|
||||
Large real-time Price Index number
|
||||
|
||||
Color-coded trend (% change)
|
||||
|
||||
Mini sparkline
|
||||
|
||||
Vendor Comparison Table
|
||||
|
||||
Columns:
|
||||
|
||||
Platform (Eldorado/PlayerAuctions)
|
||||
|
||||
Seller Name
|
||||
|
||||
USD per 1M AUEC
|
||||
|
||||
Delivery Time
|
||||
|
||||
Last updated
|
||||
|
||||
Sortable by price, seller, platform
|
||||
|
||||
Filter by platform
|
||||
|
||||
Auto-refresh from WebSocket events
|
||||
|
||||
Historical Chart
|
||||
|
||||
Select by:
|
||||
|
||||
Lowest price across all vendors (default)
|
||||
|
||||
Specific platform (Eldorado/PlayerAuctions)
|
||||
|
||||
Specific seller
|
||||
|
||||
Chart types:
|
||||
|
||||
Line (price over time)
|
||||
|
||||
Area
|
||||
|
||||
Multi-line comparison (compare multiple sellers)
|
||||
|
||||
Time ranges:
|
||||
|
||||
1h, 6h, 24h, 7d, 30d, All
|
||||
|
||||
Show all seller prices as separate lines for comparison
|
||||
|
||||
Client-Side Alerts
|
||||
|
||||
User selects:
|
||||
|
||||
Condition:
|
||||
|
||||
Price rises above X
|
||||
|
||||
Price falls below X
|
||||
|
||||
Threshold value
|
||||
|
||||
Stored locally (JSON or SQLite)
|
||||
|
||||
Trigger rules:
|
||||
|
||||
On each WS update, re-evaluate alerts
|
||||
|
||||
Trigger:
|
||||
|
||||
Windows/macOS/Linux native notification
|
||||
|
||||
Sound (toggle in settings)
|
||||
|
||||
UI popup/log entry
|
||||
|
||||
Settings Panel
|
||||
|
||||
Dark/light mode
|
||||
|
||||
Alert sounds on/off
|
||||
|
||||
Launch on startup
|
||||
|
||||
Chart theme
|
||||
|
||||
Electron-Specific Requirements
|
||||
|
||||
Auto-update (e.g., using electron-updater)
|
||||
|
||||
Secure IPC messaging (no remote code eval)
|
||||
|
||||
Sandboxed renderer process
|
||||
|
||||
Signed binaries (Windows .exe, macOS .dmg/.pkg)
|
||||
|
||||
5. Non-Functional Requirements
|
||||
Security
|
||||
|
||||
HTTPS for all API calls
|
||||
|
||||
No sensitive user data stored
|
||||
|
||||
Sanitized DB inputs
|
||||
|
||||
CSRF not needed (desktop app only)
|
||||
|
||||
Signed Electron builds
|
||||
|
||||
Scalability
|
||||
|
||||
Scrapers run in Docker for horizontal scaling
|
||||
|
||||
Backend stateless → easy to scale behind a load balancer
|
||||
|
||||
TimescaleDB handles time-series at scale
|
||||
|
||||
Reliability
|
||||
|
||||
WebSocket auto-reconnect
|
||||
|
||||
Graceful handling if server is offline
|
||||
|
||||
Retry logic in scraping pipeline
|
||||
|
||||
Performance
|
||||
|
||||
Desktop app must remain responsive when charts have 50k+ datapoints
|
||||
(handled with TimescaleDB downsampling + chart library performance)
|
||||
|
||||
Testing
|
||||
|
||||
Unit tests: Jest (backend & Electron renderer)
|
||||
|
||||
E2E tests: Playwright for UI
|
||||
|
||||
Integration tests for API via Supertest or Pact
|
||||
|
||||
6. Risks & Mitigation
|
||||
Risk Mitigation
|
||||
Vendor site layout changes break scrapers Monitoring + abstraction layers + fallback parsing
|
||||
Browser automation instability Playwright retries + container isolation
|
||||
WS disconnects Auto-reconnect logic in Electron
|
||||
Heavy Electron UI Use Vite + lazy-loaded components + chart virtualization
|
||||
TimescaleDB complexity Can be optional if scale is small
|
||||
7. Future Enhancements
|
||||
|
||||
Add more RMT vendors
|
||||
|
||||
Discord alerts via webhook
|
||||
|
||||
Predictive modeling (ML)
|
||||
|
||||
Add terminal-like command bar (e.g., /chart index 30d)
|
||||
|
||||
Desktop widgets and system tray “price ticker”
|
||||
159
README.md
159
README.md
@@ -1,2 +1,161 @@
|
||||
# rmtPocketWatcher
|
||||
|
||||
A cross-platform desktop application that tracks real-money trading (RMT) prices for Star Citizen AUEC. Provides a Bloomberg-style terminal interface displaying real-time AUEC price data from multiple vendors.
|
||||
|
||||
Developed by Lambda Banking Conglomerate - A Star Citizen Organization
|
||||
|
||||
## Features
|
||||
|
||||
- 🔄 Automated scraping from Eldorado and PlayerAuctions every 5 minutes
|
||||
- 📊 Track all seller listings with historical data
|
||||
- 💰 Real-time lowest price tracking across all vendors
|
||||
- 📈 Historical price charts by seller, platform, or overall lowest
|
||||
- 🔔 Client-side price alerts with native notifications
|
||||
- 🌐 REST API + WebSocket for real-time updates
|
||||
- 🗄️ PostgreSQL + Prisma for reliable data storage
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Docker Compose (Recommended)
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone <repository-url>
|
||||
cd rmtPocketWatcher
|
||||
|
||||
# Start everything
|
||||
docker-compose up -d
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f backend
|
||||
|
||||
# Access the API
|
||||
curl http://localhost:3000/health
|
||||
```
|
||||
|
||||
The backend will be available at `http://localhost:3000`
|
||||
|
||||
### Local Development
|
||||
|
||||
```bash
|
||||
# Start database only
|
||||
docker-compose up -d postgres
|
||||
|
||||
# Setup backend
|
||||
cd backend
|
||||
npm install
|
||||
npm run db:generate
|
||||
npm run db:push
|
||||
npx playwright install chromium
|
||||
|
||||
# Run development server
|
||||
npm run dev
|
||||
|
||||
# Or test scraper manually
|
||||
npm run scrape
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
rmtPocketWatcher/
|
||||
├── backend/ # TypeScript backend server
|
||||
│ ├── src/
|
||||
│ │ ├── scrapers/ # Playwright scraping modules
|
||||
│ │ ├── api/ # Fastify REST + WebSocket API
|
||||
│ │ ├── database/ # Prisma client and repository
|
||||
│ │ └── index.ts # Main server entry point
|
||||
│ ├── prisma/ # Database schema and migrations
|
||||
│ └── Dockerfile
|
||||
├── electron-app/ # Electron desktop app (coming soon)
|
||||
├── docker-compose.yml # Docker orchestration
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Prices
|
||||
- `GET /api/prices/latest` - Get all current listings
|
||||
- `GET /api/prices/lowest` - Get current lowest price
|
||||
- `GET /api/prices/by-seller?seller=&platform=` - Filter by seller
|
||||
- `GET /api/prices/by-platform?platform=` - Filter by platform
|
||||
- `GET /api/prices/history?from=&to=&seller=&platform=` - Historical data
|
||||
|
||||
### Index
|
||||
- `GET /api/index/history?range=7d|30d|90d|all` - Lowest price over time
|
||||
- `GET /api/index/ws` - WebSocket for real-time updates
|
||||
|
||||
### Health
|
||||
- `GET /health` - Server health check
|
||||
|
||||
## Configuration
|
||||
|
||||
Edit `.env` or set environment variables:
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql://rmtpw:rmtpw_password@localhost:5432/rmtpocketwatcher
|
||||
PORT=3000
|
||||
SCRAPE_INTERVAL_MINUTES=5
|
||||
SCRAPER_HEADLESS=true
|
||||
NODE_ENV=production
|
||||
```
|
||||
|
||||
## Technology Stack
|
||||
|
||||
**Backend:**
|
||||
- TypeScript + Node.js
|
||||
- Fastify (REST API + WebSocket)
|
||||
- Playwright (web scraping)
|
||||
- PostgreSQL + Prisma ORM
|
||||
- Node Scheduler (cron jobs)
|
||||
|
||||
**Frontend (Coming Soon):**
|
||||
- Electron 30+
|
||||
- React + TypeScript
|
||||
- TailwindCSS
|
||||
- Recharts/ECharts
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
npm test
|
||||
|
||||
# Lint and type check
|
||||
npx tsc --noEmit
|
||||
|
||||
# Database management
|
||||
npm run db:studio # Open Prisma Studio
|
||||
npm run db:push # Push schema changes
|
||||
npm run db:migrate # Create migration
|
||||
|
||||
# Manual scrape test
|
||||
npm run scrape
|
||||
```
|
||||
|
||||
## Docker Commands
|
||||
|
||||
```bash
|
||||
# Build and start
|
||||
docker-compose up --build
|
||||
|
||||
# Stop services
|
||||
docker-compose down
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f backend
|
||||
|
||||
# Restart backend only
|
||||
docker-compose restart backend
|
||||
|
||||
# Remove everything including volumes
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
## Disclaimer
|
||||
|
||||
This application is for informational and tracking purposes only. It does not facilitate any real-money trading transactions. Users are responsible for complying with all applicable laws and game terms of service.
|
||||
|
||||
171
SETUP.md
Normal file
171
SETUP.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# rmtPocketWatcher Setup Guide
|
||||
|
||||
## What's Been Created
|
||||
|
||||
✅ **Backend Scraper Service**
|
||||
- Playwright-based scrapers for Eldorado and PlayerAuctions
|
||||
- Automatic retry logic and error handling
|
||||
- Scheduled scraping every 5 minutes
|
||||
- Tracks all seller listings with platform, price, and delivery time
|
||||
|
||||
✅ **Database Layer**
|
||||
- PostgreSQL with Prisma ORM
|
||||
- Three tables: VendorPrice, PriceIndex, ScrapeLog
|
||||
- Stores all historical listings for trend analysis
|
||||
- Indexed for fast queries
|
||||
|
||||
✅ **API Layer**
|
||||
- Fastify REST API with 6 endpoints
|
||||
- WebSocket for real-time updates
|
||||
- Filter by seller, platform, date range
|
||||
- Historical price data
|
||||
|
||||
✅ **Docker Setup**
|
||||
- Docker Compose orchestration
|
||||
- PostgreSQL container with health checks
|
||||
- Backend container with auto-migration
|
||||
- Volume persistence for database
|
||||
|
||||
## Current Status
|
||||
|
||||
The Docker Compose stack is building. This will:
|
||||
1. Pull PostgreSQL 16 Alpine image
|
||||
2. Build the backend Node.js container
|
||||
3. Install Playwright and dependencies
|
||||
4. Generate Prisma client
|
||||
5. Start both services
|
||||
|
||||
## Once Build Completes
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
# View logs
|
||||
docker-compose logs -f backend
|
||||
|
||||
# Check if services are running
|
||||
docker ps
|
||||
```
|
||||
|
||||
### Test the API
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:3000/health
|
||||
|
||||
# Get latest prices (after first scrape)
|
||||
curl http://localhost:3000/api/prices/latest
|
||||
|
||||
# Get lowest price
|
||||
curl http://localhost:3000/api/prices/lowest
|
||||
|
||||
# Get price history
|
||||
curl "http://localhost:3000/api/index/history?range=7d"
|
||||
```
|
||||
|
||||
### Monitor Scraping
|
||||
The backend will automatically:
|
||||
- Scrape Eldorado and PlayerAuctions every 5 minutes
|
||||
- Save all listings to the database
|
||||
- Calculate and store the lowest price
|
||||
- Log all scrape attempts
|
||||
|
||||
Check logs to see scraping activity:
|
||||
```bash
|
||||
docker-compose logs -f backend | grep "scraping"
|
||||
```
|
||||
|
||||
## Database Access
|
||||
|
||||
### Using Prisma Studio
|
||||
```bash
|
||||
cd backend
|
||||
npm run db:studio
|
||||
```
|
||||
|
||||
### Using psql
|
||||
```bash
|
||||
docker exec -it rmtpw-postgres psql -U rmtpw -d rmtpocketwatcher
|
||||
```
|
||||
|
||||
## Stopping and Restarting
|
||||
|
||||
```bash
|
||||
# Stop services
|
||||
docker-compose down
|
||||
|
||||
# Start services
|
||||
docker-compose up -d
|
||||
|
||||
# Rebuild after code changes
|
||||
docker-compose up --build -d
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Remove everything including data
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Edit `.env` or `docker-compose.yml` to configure:
|
||||
|
||||
- `SCRAPE_INTERVAL_MINUTES` - How often to scrape (default: 5)
|
||||
- `SCRAPER_HEADLESS` - Run browser in headless mode (default: true)
|
||||
- `PORT` - API server port (default: 3000)
|
||||
- `DATABASE_URL` - PostgreSQL connection string
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Wait for build to complete** (~2-5 minutes)
|
||||
2. **Verify services are running**: `docker ps`
|
||||
3. **Check first scrape**: `docker-compose logs -f backend`
|
||||
4. **Test API endpoints**: See examples above
|
||||
5. **Build Electron frontend** (coming next)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Backend won't start
|
||||
```bash
|
||||
# Check logs
|
||||
docker-compose logs backend
|
||||
|
||||
# Restart backend
|
||||
docker-compose restart backend
|
||||
```
|
||||
|
||||
### Database connection issues
|
||||
```bash
|
||||
# Check postgres is healthy
|
||||
docker-compose ps
|
||||
|
||||
# Restart postgres
|
||||
docker-compose restart postgres
|
||||
```
|
||||
|
||||
### Scraper errors
|
||||
```bash
|
||||
# View detailed logs
|
||||
docker-compose logs -f backend | grep -A 5 "error"
|
||||
|
||||
# Check scrape log in database
|
||||
docker exec -it rmtpw-postgres psql -U rmtpw -d rmtpocketwatcher -c "SELECT * FROM scrape_log ORDER BY timestamp DESC LIMIT 10;"
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
rmtPocketWatcher/
|
||||
├── docker-compose.yml # Docker orchestration
|
||||
├── .env # Environment variables
|
||||
├── backend/
|
||||
│ ├── Dockerfile # Backend container definition
|
||||
│ ├── prisma/
|
||||
│ │ └── schema.prisma # Database schema
|
||||
│ ├── src/
|
||||
│ │ ├── scrapers/ # Scraping logic
|
||||
│ │ ├── api/ # REST + WebSocket API
|
||||
│ │ ├── database/ # Prisma client & repository
|
||||
│ │ └── index.ts # Main server
|
||||
│ └── package.json
|
||||
└── README.md
|
||||
```
|
||||
12
backend/.dockerignore
Normal file
12
backend/.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
node_modules
|
||||
dist
|
||||
.env
|
||||
.env.local
|
||||
*.log
|
||||
.DS_Store
|
||||
coverage
|
||||
.vscode
|
||||
.idea
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
14
backend/.env.example
Normal file
14
backend/.env.example
Normal file
@@ -0,0 +1,14 @@
|
||||
# Database
|
||||
DATABASE_URL="postgresql://user:password@localhost:5432/rmtpocketwatcher?schema=public"
|
||||
|
||||
# Scraper Configuration
|
||||
SCRAPE_INTERVAL_MINUTES=5
|
||||
SCRAPER_HEADLESS=true
|
||||
SCRAPER_TIMEOUT=30000
|
||||
SCRAPER_MAX_RETRIES=3
|
||||
|
||||
# Server
|
||||
PORT=3000
|
||||
HOST=0.0.0.0
|
||||
NODE_ENV=development
|
||||
LOG_LEVEL=info
|
||||
9
backend/.gitignore
vendored
Normal file
9
backend/.gitignore
vendored
Normal file
@@ -0,0 +1,9 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.env
|
||||
.env.local
|
||||
*.log
|
||||
.DS_Store
|
||||
coverage/
|
||||
.vscode/
|
||||
.idea/
|
||||
49
backend/Dockerfile
Normal file
49
backend/Dockerfile
Normal file
@@ -0,0 +1,49 @@
|
||||
FROM node:20-slim
|
||||
|
||||
# Install Playwright dependencies and PostgreSQL client
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libnss3 \
|
||||
libnspr4 \
|
||||
libatk1.0-0 \
|
||||
libatk-bridge2.0-0 \
|
||||
libcups2 \
|
||||
libdrm2 \
|
||||
libdbus-1-3 \
|
||||
libxkbcommon0 \
|
||||
libxcomposite1 \
|
||||
libxdamage1 \
|
||||
libxfixes3 \
|
||||
libxrandr2 \
|
||||
libgbm1 \
|
||||
libasound2 \
|
||||
libpango-1.0-0 \
|
||||
libcairo2 \
|
||||
postgresql-client \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files
|
||||
COPY package*.json ./
|
||||
COPY prisma ./prisma/
|
||||
|
||||
# Install dependencies (including dev dependencies for Prisma)
|
||||
RUN npm install
|
||||
|
||||
# Install Playwright browsers
|
||||
RUN npx playwright install chromium --with-deps
|
||||
|
||||
# Generate Prisma client
|
||||
RUN npx prisma generate
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Build TypeScript
|
||||
RUN npm run build
|
||||
|
||||
# Expose port
|
||||
EXPOSE 3000
|
||||
|
||||
# Start application (migrations will be run via docker-compose command)
|
||||
CMD ["npm", "start"]
|
||||
120
backend/README.md
Normal file
120
backend/README.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# rmtPocketWatcher Backend
|
||||
|
||||
Backend scraping and API service for rmtPocketWatcher.
|
||||
|
||||
## Features
|
||||
|
||||
- Playwright-based web scraping for Eldorado and PlayerAuctions
|
||||
- Automatic retry logic (3 attempts with exponential backoff)
|
||||
- Scheduled scraping every 5 minutes (configurable)
|
||||
- PostgreSQL + Prisma for data storage
|
||||
- REST API + WebSocket for real-time updates
|
||||
- Tracks all seller listings over time
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Install Playwright browsers
|
||||
npx playwright install chromium
|
||||
|
||||
# Setup environment
|
||||
cp .env.example .env
|
||||
# Edit .env with your PostgreSQL connection string
|
||||
|
||||
# Generate Prisma client
|
||||
npm run db:generate
|
||||
|
||||
# Run database migrations
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
## Quick Start with Docker Compose
|
||||
|
||||
The easiest way to run the entire stack:
|
||||
|
||||
```bash
|
||||
# From the project root
|
||||
docker-compose up -d
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f backend
|
||||
|
||||
# Stop everything
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
This will:
|
||||
- Start PostgreSQL database
|
||||
- Run database migrations
|
||||
- Start the backend API server
|
||||
- Begin scraping every 5 minutes
|
||||
|
||||
## Local Development Setup
|
||||
|
||||
If you want to run locally without Docker:
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL with Docker
|
||||
docker-compose up -d postgres
|
||||
|
||||
# Install dependencies
|
||||
cd backend
|
||||
npm install
|
||||
|
||||
# Setup environment
|
||||
cp .env.example .env
|
||||
# Edit .env if needed
|
||||
|
||||
# Generate Prisma client and push schema
|
||||
npm run db:generate
|
||||
npm run db:push
|
||||
|
||||
# Install Playwright browsers
|
||||
npx playwright install chromium
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Development mode with hot reload
|
||||
npm run dev
|
||||
|
||||
# Manual scrape test (no database)
|
||||
npm run scrape
|
||||
|
||||
# Build for production
|
||||
npm run build
|
||||
|
||||
# Run production build
|
||||
npm start
|
||||
|
||||
# Database management
|
||||
npm run db:studio # Open Prisma Studio
|
||||
npm run db:push # Push schema changes without migration
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Prices
|
||||
- `GET /api/prices/latest` - Get all current listings
|
||||
- `GET /api/prices/lowest` - Get current lowest price
|
||||
- `GET /api/prices/by-seller?seller=&platform=` - Get seller's price history
|
||||
- `GET /api/prices/by-platform?platform=` - Get all listings from a platform
|
||||
- `GET /api/prices/history?from=&to=&seller=&platform=` - Historical data
|
||||
|
||||
### Index
|
||||
- `GET /api/index/history?range=7d|30d|90d|all` - Lowest price over time
|
||||
- `GET /api/index/ws` - WebSocket for real-time updates
|
||||
|
||||
### Health
|
||||
- `GET /health` - Server health check
|
||||
|
||||
## Architecture
|
||||
|
||||
- `src/scrapers/` - Playwright scraping modules
|
||||
- `src/api/` - Fastify REST + WebSocket API
|
||||
- `src/database/` - Prisma client and repository
|
||||
- `prisma/` - Database schema and migrations
|
||||
15
backend/jest.config.js
Normal file
15
backend/jest.config.js
Normal file
@@ -0,0 +1,15 @@
|
||||
module.exports = {
|
||||
preset: 'ts-jest',
|
||||
testEnvironment: 'node',
|
||||
roots: ['<rootDir>/tests'],
|
||||
testMatch: ['**/*.test.ts'],
|
||||
collectCoverageFrom: [
|
||||
'src/**/*.ts',
|
||||
'!src/**/*.d.ts',
|
||||
'!src/**/index.ts',
|
||||
],
|
||||
moduleFileExtensions: ['ts', 'js', 'json'],
|
||||
transform: {
|
||||
'^.+\\.ts$': 'ts-jest',
|
||||
},
|
||||
};
|
||||
5200
backend/package-lock.json
generated
Normal file
5200
backend/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
39
backend/package.json
Normal file
39
backend/package.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"name": "rmtpocketwatcher-backend",
|
||||
"version": "1.0.0",
|
||||
"description": "Backend scraping and API service for rmtPocketWatcher",
|
||||
"main": "dist/index.js",
|
||||
"scripts": {
|
||||
"dev": "tsx watch src/index.ts",
|
||||
"build": "tsc",
|
||||
"start": "node dist/index.js",
|
||||
"test": "jest",
|
||||
"scrape": "tsx src/scrapers/manual-scrape.ts",
|
||||
"db:generate": "prisma generate",
|
||||
"db:migrate": "prisma migrate dev",
|
||||
"db:push": "prisma db push",
|
||||
"db:studio": "prisma studio"
|
||||
},
|
||||
"dependencies": {
|
||||
"fastify": "^4.25.0",
|
||||
"@fastify/websocket": "^10.0.0",
|
||||
"@fastify/cors": "^9.0.0",
|
||||
"playwright": "^1.40.0",
|
||||
"prisma": "^5.7.0",
|
||||
"@prisma/client": "^5.7.0",
|
||||
"node-schedule": "^2.1.1",
|
||||
"ws": "^8.16.0",
|
||||
"zod": "^3.22.4",
|
||||
"dotenv": "^16.3.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.10.0",
|
||||
"@types/node-schedule": "^2.1.5",
|
||||
"@types/ws": "^8.5.10",
|
||||
"typescript": "^5.3.0",
|
||||
"tsx": "^4.7.0",
|
||||
"jest": "^29.7.0",
|
||||
"@types/jest": "^29.5.0",
|
||||
"ts-jest": "^29.1.0"
|
||||
}
|
||||
}
|
||||
1
backend/prisma/migrations/.gitkeep
Normal file
1
backend/prisma/migrations/.gitkeep
Normal file
@@ -0,0 +1 @@
|
||||
# Migrations directory
|
||||
51
backend/prisma/schema.prisma
Normal file
51
backend/prisma/schema.prisma
Normal file
@@ -0,0 +1,51 @@
|
||||
// This is your Prisma schema file,
|
||||
// learn more about it in the docs: https://pris.ly/d/prisma-schema
|
||||
|
||||
generator client {
|
||||
provider = "prisma-client-js"
|
||||
}
|
||||
|
||||
datasource db {
|
||||
provider = "postgresql"
|
||||
url = env("DATABASE_URL")
|
||||
}
|
||||
|
||||
model VendorPrice {
|
||||
id String @id @default(uuid())
|
||||
timestamp DateTime @default(now()) @db.Timestamptz(3)
|
||||
vendor String // 'eldorado' or 'playerauctions'
|
||||
sellerName String? @map("seller_name")
|
||||
usdPrice Decimal @map("usd_price") @db.Decimal(12, 2)
|
||||
auecAmount BigInt @map("auec_amount")
|
||||
usdPerMillion Decimal @map("usd_per_million") @db.Decimal(12, 8)
|
||||
deliveryTime String? @map("delivery_time")
|
||||
url String
|
||||
|
||||
@@index([timestamp])
|
||||
@@index([vendor])
|
||||
@@index([sellerName])
|
||||
@@index([usdPerMillion])
|
||||
@@map("raw_vendor_prices")
|
||||
}
|
||||
|
||||
model PriceIndex {
|
||||
id String @id @default(uuid())
|
||||
timestamp DateTime @default(now()) @db.Timestamptz(3)
|
||||
lowestPrice Decimal @map("lowest_price") @db.Decimal(12, 8)
|
||||
vendor String
|
||||
sellerName String? @map("seller_name")
|
||||
|
||||
@@index([timestamp])
|
||||
@@map("price_index")
|
||||
}
|
||||
|
||||
model ScrapeLog {
|
||||
id String @id @default(uuid())
|
||||
timestamp DateTime @default(now()) @db.Timestamptz(3)
|
||||
status String // 'success' or 'failure'
|
||||
message String?
|
||||
runtimeMs Int? @map("runtime_ms")
|
||||
|
||||
@@index([timestamp])
|
||||
@@map("scrape_log")
|
||||
}
|
||||
55
backend/src/api/routes/index.ts
Normal file
55
backend/src/api/routes/index.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { FastifyPluginAsync } from 'fastify';
|
||||
import { PriceRepository } from '../../database/repository';
|
||||
|
||||
const repository = new PriceRepository();
|
||||
|
||||
export const indexRoutes: FastifyPluginAsync = async (server) => {
|
||||
// GET /api/index/history?range=7d|30d|90d|all
|
||||
server.get<{
|
||||
Querystring: { range?: string };
|
||||
}>('/history', async (request, reply) => {
|
||||
try {
|
||||
const { range = '7d' } = request.query;
|
||||
const history = await repository.getIndexHistory(range);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
data: history.map((h: any) => ({
|
||||
timestamp: h.timestamp,
|
||||
price: h.lowestPrice.toString(),
|
||||
vendor: h.vendor,
|
||||
seller: h.sellerName,
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
// WebSocket endpoint for real-time updates
|
||||
server.get('/ws', { websocket: true }, (connection: any, request: any) => {
|
||||
connection.socket.on('message', (message: any) => {
|
||||
// Echo back for now - will be replaced with real-time updates
|
||||
connection.socket.send(message.toString());
|
||||
});
|
||||
|
||||
// Send initial data
|
||||
repository.getLowestPrice().then(lowest => {
|
||||
if (lowest) {
|
||||
connection.socket.send(JSON.stringify({
|
||||
type: 'index',
|
||||
data: {
|
||||
timestamp: lowest.timestamp,
|
||||
price: lowest.lowestPrice.toString(),
|
||||
vendor: lowest.vendor,
|
||||
seller: lowest.sellerName,
|
||||
},
|
||||
}));
|
||||
}
|
||||
});
|
||||
});
|
||||
};
|
||||
158
backend/src/api/routes/prices.ts
Normal file
158
backend/src/api/routes/prices.ts
Normal file
@@ -0,0 +1,158 @@
|
||||
import { FastifyPluginAsync } from 'fastify';
|
||||
import { PriceRepository } from '../../database/repository';
|
||||
|
||||
const repository = new PriceRepository();
|
||||
|
||||
export const priceRoutes: FastifyPluginAsync = async (server) => {
|
||||
// GET /api/prices/latest - Get all current listings
|
||||
server.get('/latest', async (request, reply) => {
|
||||
try {
|
||||
const prices = await repository.getLatestPrices();
|
||||
return {
|
||||
success: true,
|
||||
data: prices.map((p: any) => ({
|
||||
id: p.id,
|
||||
timestamp: p.timestamp,
|
||||
vendor: p.vendor,
|
||||
seller: p.sellerName,
|
||||
usdPrice: p.usdPrice.toString(),
|
||||
auecAmount: p.auecAmount.toString(),
|
||||
usdPerMillion: p.usdPerMillion.toString(),
|
||||
deliveryTime: p.deliveryTime,
|
||||
url: p.url,
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
// GET /api/prices/lowest - Get current lowest price
|
||||
server.get('/lowest', async (request, reply) => {
|
||||
try {
|
||||
const lowest = await repository.getLowestPrice();
|
||||
if (!lowest) {
|
||||
return { success: true, data: null };
|
||||
}
|
||||
return {
|
||||
success: true,
|
||||
data: {
|
||||
timestamp: lowest.timestamp,
|
||||
price: lowest.lowestPrice.toString(),
|
||||
vendor: lowest.vendor,
|
||||
seller: lowest.sellerName,
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
// GET /api/prices/by-seller?seller=&platform=
|
||||
server.get<{
|
||||
Querystring: { seller: string; platform?: string };
|
||||
}>('/by-seller', async (request, reply) => {
|
||||
try {
|
||||
const { seller, platform } = request.query;
|
||||
if (!seller) {
|
||||
reply.code(400);
|
||||
return { success: false, error: 'seller parameter is required' };
|
||||
}
|
||||
|
||||
const prices = await repository.getPricesBySeller(seller, platform);
|
||||
return {
|
||||
success: true,
|
||||
data: prices.map((p: any) => ({
|
||||
timestamp: p.timestamp,
|
||||
vendor: p.vendor,
|
||||
seller: p.sellerName,
|
||||
usdPerMillion: p.usdPerMillion.toString(),
|
||||
deliveryTime: p.deliveryTime,
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
// GET /api/prices/by-platform?platform=
|
||||
server.get<{
|
||||
Querystring: { platform: string };
|
||||
}>('/by-platform', async (request, reply) => {
|
||||
try {
|
||||
const { platform } = request.query;
|
||||
if (!platform) {
|
||||
reply.code(400);
|
||||
return { success: false, error: 'platform parameter is required' };
|
||||
}
|
||||
|
||||
const prices = await repository.getPricesByPlatform(platform);
|
||||
return {
|
||||
success: true,
|
||||
data: prices.map((p: any) => ({
|
||||
timestamp: p.timestamp,
|
||||
seller: p.sellerName,
|
||||
usdPerMillion: p.usdPerMillion.toString(),
|
||||
deliveryTime: p.deliveryTime,
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
// GET /api/prices/history?from=&to=&seller=&platform=
|
||||
server.get<{
|
||||
Querystring: { from: string; to: string; seller?: string; platform?: string };
|
||||
}>('/history', async (request, reply) => {
|
||||
try {
|
||||
const { from, to, seller, platform } = request.query;
|
||||
if (!from || !to) {
|
||||
reply.code(400);
|
||||
return { success: false, error: 'from and to parameters are required' };
|
||||
}
|
||||
|
||||
const fromDate = new Date(from);
|
||||
const toDate = new Date(to);
|
||||
|
||||
if (isNaN(fromDate.getTime()) || isNaN(toDate.getTime())) {
|
||||
reply.code(400);
|
||||
return { success: false, error: 'Invalid date format' };
|
||||
}
|
||||
|
||||
const prices = await repository.getPriceHistory(fromDate, toDate, seller, platform);
|
||||
return {
|
||||
success: true,
|
||||
data: prices.map((p: any) => ({
|
||||
timestamp: p.timestamp,
|
||||
vendor: p.vendor,
|
||||
seller: p.sellerName,
|
||||
usdPerMillion: p.usdPerMillion.toString(),
|
||||
deliveryTime: p.deliveryTime,
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
reply.code(500);
|
||||
return {
|
||||
success: false,
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
};
|
||||
}
|
||||
});
|
||||
};
|
||||
153
backend/src/api/routes/websocket.ts
Normal file
153
backend/src/api/routes/websocket.ts
Normal file
@@ -0,0 +1,153 @@
|
||||
import { FastifyPluginAsync } from 'fastify';
|
||||
import { WebSocket } from 'ws';
|
||||
import { PriceRepository } from '../../database/repository';
|
||||
|
||||
interface WebSocketMessage {
|
||||
type: string;
|
||||
data?: any;
|
||||
}
|
||||
|
||||
interface WebSocketConnection {
|
||||
socket: WebSocket;
|
||||
}
|
||||
|
||||
const clients = new Set<WebSocketConnection>();
|
||||
const repository = new PriceRepository();
|
||||
|
||||
export const websocketRoutes: FastifyPluginAsync = async (server) => {
|
||||
server.log.info('WebSocket route registered at /ws');
|
||||
|
||||
server.get('/ws', { websocket: true }, (socket: any, request: any) => {
|
||||
server.log.info('WebSocket client connected');
|
||||
console.log('✓ WebSocket client connected');
|
||||
|
||||
const connection = { socket };
|
||||
clients.add(connection);
|
||||
|
||||
// Send connection confirmation
|
||||
socket.send(JSON.stringify({
|
||||
type: 'connection_status',
|
||||
data: { connected: true, message: 'Connected to rmtPocketWatcher' }
|
||||
}));
|
||||
|
||||
// Handle incoming messages
|
||||
socket.on('message', async (message: Buffer) => {
|
||||
try {
|
||||
const parsed: WebSocketMessage = JSON.parse(message.toString());
|
||||
|
||||
if (parsed.type === 'subscribe' && parsed.data?.channel === 'price_updates') {
|
||||
// Send latest price data on subscription
|
||||
const latestPrices = await repository.getLatestPrices();
|
||||
|
||||
if (latestPrices.length > 0) {
|
||||
const lowestPrice = latestPrices.reduce((min: any, p: any) =>
|
||||
Number(p.usdPerMillion) < Number(min.usdPerMillion) ? p : min
|
||||
);
|
||||
|
||||
socket.send(JSON.stringify({
|
||||
type: 'price_update',
|
||||
data: {
|
||||
timestamp: new Date(),
|
||||
lowestPrice: Number(lowestPrice.usdPerMillion),
|
||||
platform: lowestPrice.vendor,
|
||||
sellerName: lowestPrice.sellerName,
|
||||
allPrices: latestPrices.map((p: any) => ({
|
||||
id: p.id,
|
||||
platform: p.vendor,
|
||||
sellerName: p.sellerName,
|
||||
pricePerMillion: Number(p.usdPerMillion),
|
||||
timestamp: p.timestamp,
|
||||
url: p.url
|
||||
}))
|
||||
}
|
||||
}));
|
||||
}
|
||||
} else if (parsed.type === 'get_history') {
|
||||
// Handle historical data request
|
||||
const { range } = parsed.data || {};
|
||||
const now = new Date();
|
||||
let from = new Date();
|
||||
|
||||
switch (range) {
|
||||
case '6h':
|
||||
from = new Date(now.getTime() - 6 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '24h':
|
||||
from = new Date(now.getTime() - 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '3d':
|
||||
from = new Date(now.getTime() - 3 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '7d':
|
||||
from = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '1mo':
|
||||
from = new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case 'ytd':
|
||||
from = new Date(now.getFullYear(), 0, 1);
|
||||
break;
|
||||
default:
|
||||
from = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);
|
||||
}
|
||||
|
||||
const history = await repository.getPriceHistory(from, now);
|
||||
|
||||
const payload = {
|
||||
type: 'history_data',
|
||||
data: {
|
||||
range,
|
||||
from,
|
||||
to: now,
|
||||
prices: history.map((p: any) => ({
|
||||
id: p.id,
|
||||
platform: p.vendor,
|
||||
sellerName: p.sellerName,
|
||||
pricePerMillion: Number(p.usdPerMillion),
|
||||
timestamp: p.timestamp,
|
||||
url: p.url
|
||||
}))
|
||||
}
|
||||
};
|
||||
|
||||
server.log.info(`Sending history data: ${payload.data.prices.length} prices for range ${range}`);
|
||||
socket.send(JSON.stringify(payload));
|
||||
}
|
||||
} catch (error: any) {
|
||||
server.log.error({ error }, 'Error handling WebSocket message');
|
||||
socket.send(JSON.stringify({
|
||||
type: 'error',
|
||||
data: 'Invalid message format'
|
||||
}));
|
||||
}
|
||||
});
|
||||
|
||||
socket.on('close', () => {
|
||||
server.log.info('WebSocket client disconnected');
|
||||
clients.delete(connection);
|
||||
});
|
||||
|
||||
socket.on('error', (error: any) => {
|
||||
server.log.error({ error }, 'WebSocket error');
|
||||
clients.delete(connection);
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
// Broadcast price updates to all connected clients
|
||||
export function broadcastPriceUpdate(data: any): void {
|
||||
const message = JSON.stringify({
|
||||
type: 'price_update',
|
||||
data
|
||||
});
|
||||
|
||||
clients.forEach((connection) => {
|
||||
try {
|
||||
if (connection.socket.readyState === 1) { // OPEN
|
||||
connection.socket.send(message);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error broadcasting to client:', error);
|
||||
}
|
||||
});
|
||||
}
|
||||
36
backend/src/api/server.ts
Normal file
36
backend/src/api/server.ts
Normal file
@@ -0,0 +1,36 @@
|
||||
import Fastify from 'fastify';
|
||||
import cors from '@fastify/cors';
|
||||
import websocket from '@fastify/websocket';
|
||||
import { priceRoutes } from './routes/prices';
|
||||
import { indexRoutes } from './routes/index';
|
||||
import { websocketRoutes } from './routes/websocket';
|
||||
|
||||
const server = Fastify({
|
||||
logger: {
|
||||
level: process.env.LOG_LEVEL || 'info',
|
||||
},
|
||||
});
|
||||
|
||||
// Register plugins
|
||||
server.register(cors, {
|
||||
origin: true,
|
||||
});
|
||||
|
||||
server.register(websocket);
|
||||
|
||||
// Register routes
|
||||
server.register(priceRoutes, { prefix: '/api/prices' });
|
||||
server.register(indexRoutes, { prefix: '/api/index' });
|
||||
server.register(websocketRoutes);
|
||||
|
||||
// Health check
|
||||
server.get('/health', async () => {
|
||||
return { status: 'ok', timestamp: new Date().toISOString() };
|
||||
});
|
||||
|
||||
// Test endpoint to trigger scrape
|
||||
server.get('/api/test/scrape', async () => {
|
||||
return { message: 'Scrape will be triggered by scheduler' };
|
||||
});
|
||||
|
||||
export default server;
|
||||
7
backend/src/database/prisma.ts
Normal file
7
backend/src/database/prisma.ts
Normal file
@@ -0,0 +1,7 @@
|
||||
import { PrismaClient } from '@prisma/client';
|
||||
|
||||
const prisma = new PrismaClient({
|
||||
log: process.env.NODE_ENV === 'development' ? ['query', 'error', 'warn'] : ['error'],
|
||||
});
|
||||
|
||||
export default prisma;
|
||||
195
backend/src/database/repository.ts
Normal file
195
backend/src/database/repository.ts
Normal file
@@ -0,0 +1,195 @@
|
||||
import prisma from './prisma';
|
||||
import { VendorListing } from '../scrapers/types';
|
||||
|
||||
export class PriceRepository {
|
||||
async saveListings(listings: VendorListing[]): Promise<void> {
|
||||
// Use a single timestamp for all listings in this batch
|
||||
const batchTimestamp = new Date();
|
||||
|
||||
await prisma.vendorPrice.createMany({
|
||||
data: listings.map(listing => ({
|
||||
vendor: listing.vendor,
|
||||
sellerName: listing.seller,
|
||||
usdPrice: listing.priceUSD,
|
||||
auecAmount: listing.amountAUEC,
|
||||
usdPerMillion: listing.pricePerMillion,
|
||||
deliveryTime: listing.deliveryTime,
|
||||
url: listing.url,
|
||||
timestamp: batchTimestamp,
|
||||
})),
|
||||
});
|
||||
}
|
||||
|
||||
async savePriceIndex(lowestPrice: number, vendor: string, sellerName?: string): Promise<void> {
|
||||
await prisma.priceIndex.create({
|
||||
data: {
|
||||
lowestPrice,
|
||||
vendor,
|
||||
sellerName,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async logScrape(status: string, message?: string, runtimeMs?: number): Promise<void> {
|
||||
await prisma.scrapeLog.create({
|
||||
data: {
|
||||
status,
|
||||
message,
|
||||
runtimeMs,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async getLatestPrices() {
|
||||
// Get the most recent timestamp across all vendors
|
||||
const latest = await prisma.vendorPrice.findFirst({
|
||||
orderBy: { timestamp: 'desc' },
|
||||
select: { timestamp: true },
|
||||
});
|
||||
|
||||
if (!latest) return [];
|
||||
|
||||
// Get all prices from that timestamp (all vendors)
|
||||
return prisma.vendorPrice.findMany({
|
||||
where: { timestamp: latest.timestamp },
|
||||
orderBy: { usdPerMillion: 'asc' },
|
||||
});
|
||||
}
|
||||
|
||||
async getLatestPricesOld() {
|
||||
// Get the most recent timestamp for each vendor
|
||||
const latestEldorado = await prisma.vendorPrice.findFirst({
|
||||
where: { vendor: 'eldorado' },
|
||||
orderBy: { timestamp: 'desc' },
|
||||
select: { timestamp: true },
|
||||
});
|
||||
|
||||
const latestPlayerAuctions = await prisma.vendorPrice.findFirst({
|
||||
where: { vendor: 'playerauctions' },
|
||||
orderBy: { timestamp: 'desc' },
|
||||
select: { timestamp: true },
|
||||
});
|
||||
|
||||
// Get all prices from the latest scrape of each vendor
|
||||
const prices = [];
|
||||
|
||||
if (latestEldorado) {
|
||||
const eldoradoPrices = await prisma.vendorPrice.findMany({
|
||||
where: {
|
||||
vendor: 'eldorado',
|
||||
timestamp: latestEldorado.timestamp,
|
||||
},
|
||||
});
|
||||
prices.push(...eldoradoPrices);
|
||||
}
|
||||
|
||||
if (latestPlayerAuctions) {
|
||||
// Get all PlayerAuctions listings within 30 seconds of the latest timestamp
|
||||
const timeWindow = new Date(latestPlayerAuctions.timestamp.getTime() - 30000);
|
||||
const paPrices = await prisma.vendorPrice.findMany({
|
||||
where: {
|
||||
vendor: 'playerauctions',
|
||||
timestamp: {
|
||||
gte: timeWindow,
|
||||
lte: latestPlayerAuctions.timestamp,
|
||||
},
|
||||
},
|
||||
});
|
||||
prices.push(...paPrices);
|
||||
}
|
||||
|
||||
// Sort by price
|
||||
return prices.sort((a, b) =>
|
||||
Number(a.usdPerMillion) - Number(b.usdPerMillion)
|
||||
);
|
||||
}
|
||||
|
||||
async getLowestPrice() {
|
||||
const latest = await prisma.priceIndex.findFirst({
|
||||
orderBy: { timestamp: 'desc' },
|
||||
});
|
||||
|
||||
return latest;
|
||||
}
|
||||
|
||||
async getPricesBySeller(seller: string, platform?: string) {
|
||||
return prisma.vendorPrice.findMany({
|
||||
where: {
|
||||
sellerName: seller,
|
||||
...(platform && { vendor: platform }),
|
||||
},
|
||||
orderBy: { timestamp: 'desc' },
|
||||
take: 100,
|
||||
});
|
||||
}
|
||||
|
||||
async getPricesByPlatform(platform: string) {
|
||||
const latest = await prisma.vendorPrice.findFirst({
|
||||
where: { vendor: platform },
|
||||
orderBy: { timestamp: 'desc' },
|
||||
select: { timestamp: true },
|
||||
});
|
||||
|
||||
if (!latest) return [];
|
||||
|
||||
return prisma.vendorPrice.findMany({
|
||||
where: {
|
||||
vendor: platform,
|
||||
timestamp: latest.timestamp,
|
||||
},
|
||||
orderBy: { usdPerMillion: 'asc' },
|
||||
});
|
||||
}
|
||||
|
||||
async getPriceHistory(from: Date, to: Date, seller?: string, platform?: string) {
|
||||
return prisma.vendorPrice.findMany({
|
||||
where: {
|
||||
timestamp: {
|
||||
gte: from,
|
||||
lte: to,
|
||||
},
|
||||
...(seller && { sellerName: seller }),
|
||||
...(platform && { vendor: platform }),
|
||||
},
|
||||
orderBy: { timestamp: 'asc' },
|
||||
});
|
||||
}
|
||||
|
||||
async getIndexHistory(range: string) {
|
||||
const now = new Date();
|
||||
let from = new Date();
|
||||
|
||||
switch (range) {
|
||||
case '1h':
|
||||
from = new Date(now.getTime() - 60 * 60 * 1000);
|
||||
break;
|
||||
case '6h':
|
||||
from = new Date(now.getTime() - 6 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '24h':
|
||||
from = new Date(now.getTime() - 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '7d':
|
||||
from = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '30d':
|
||||
from = new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case '90d':
|
||||
from = new Date(now.getTime() - 90 * 24 * 60 * 60 * 1000);
|
||||
break;
|
||||
case 'all':
|
||||
from = new Date(0);
|
||||
break;
|
||||
default:
|
||||
from = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);
|
||||
}
|
||||
|
||||
return prisma.priceIndex.findMany({
|
||||
where: {
|
||||
timestamp: { gte: from },
|
||||
},
|
||||
orderBy: { timestamp: 'asc' },
|
||||
});
|
||||
}
|
||||
}
|
||||
91
backend/src/index.ts
Normal file
91
backend/src/index.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
import 'dotenv/config';
|
||||
import server from './api/server';
|
||||
import { ScraperScheduler } from './scrapers/scheduler';
|
||||
import { PriceRepository } from './database/repository';
|
||||
import { broadcastPriceUpdate } from './api/routes/websocket';
|
||||
|
||||
const PORT = parseInt(process.env.PORT || '3000', 10);
|
||||
const HOST = process.env.HOST || '0.0.0.0';
|
||||
const SCRAPE_INTERVAL = parseInt(process.env.SCRAPE_INTERVAL_MINUTES || '5', 10);
|
||||
|
||||
const repository = new PriceRepository();
|
||||
const scheduler = new ScraperScheduler(SCRAPE_INTERVAL);
|
||||
|
||||
// Handle scrape results
|
||||
scheduler.onScrapeComplete(async (results) => {
|
||||
const startTime = Date.now();
|
||||
|
||||
try {
|
||||
// Save all listings
|
||||
const allListings = results.flatMap(r => r.listings);
|
||||
if (allListings.length > 0) {
|
||||
await repository.saveListings(allListings);
|
||||
|
||||
// Find and save lowest price
|
||||
const lowestListing = allListings.reduce((min, listing) =>
|
||||
listing.pricePerMillion < min.pricePerMillion ? listing : min
|
||||
);
|
||||
|
||||
await repository.savePriceIndex(
|
||||
lowestListing.pricePerMillion,
|
||||
lowestListing.vendor,
|
||||
lowestListing.seller
|
||||
);
|
||||
|
||||
// Broadcast update to WebSocket clients
|
||||
const latestPrices = await repository.getLatestPrices();
|
||||
broadcastPriceUpdate({
|
||||
timestamp: new Date(),
|
||||
lowestPrice: lowestListing.pricePerMillion,
|
||||
platform: lowestListing.vendor,
|
||||
sellerName: lowestListing.seller,
|
||||
allPrices: latestPrices.map((p: any) => ({
|
||||
id: p.id,
|
||||
platform: p.vendor,
|
||||
sellerName: p.sellerName,
|
||||
pricePerMillion: Number(p.usdPerMillion),
|
||||
timestamp: p.timestamp,
|
||||
url: p.url
|
||||
}))
|
||||
});
|
||||
|
||||
console.log(`✓ Broadcasted price update to ${latestPrices.length} listings`);
|
||||
}
|
||||
|
||||
// Log successful scrape
|
||||
const runtimeMs = Date.now() - startTime;
|
||||
await repository.logScrape('success', `Saved ${allListings.length} listings`, runtimeMs);
|
||||
|
||||
console.log(`✓ Saved ${allListings.length} listings to database`);
|
||||
} catch (error) {
|
||||
const runtimeMs = Date.now() - startTime;
|
||||
const message = error instanceof Error ? error.message : 'Unknown error';
|
||||
await repository.logScrape('failure', message, runtimeMs);
|
||||
console.error('✗ Failed to save listings:', message);
|
||||
}
|
||||
});
|
||||
|
||||
// Start server
|
||||
const start = async () => {
|
||||
try {
|
||||
await server.listen({ port: PORT, host: HOST });
|
||||
console.log(`Server listening on ${HOST}:${PORT}`);
|
||||
|
||||
// Start scraper
|
||||
scheduler.start();
|
||||
console.log(`Scraper started (interval: ${SCRAPE_INTERVAL} minutes)`);
|
||||
} catch (err) {
|
||||
server.log.error(err);
|
||||
process.exit(1);
|
||||
}
|
||||
};
|
||||
|
||||
// Graceful shutdown
|
||||
process.on('SIGINT', async () => {
|
||||
console.log('\nShutting down gracefully...');
|
||||
await scheduler.close();
|
||||
await server.close();
|
||||
process.exit(0);
|
||||
});
|
||||
|
||||
start();
|
||||
92
backend/src/scrapers/base-scraper.ts
Normal file
92
backend/src/scrapers/base-scraper.ts
Normal file
@@ -0,0 +1,92 @@
|
||||
import { Browser, Page, chromium } from 'playwright';
|
||||
import { VendorListing, ScrapeResult, ScraperConfig } from './types';
|
||||
|
||||
export abstract class BaseScraper {
|
||||
protected config: ScraperConfig;
|
||||
protected browser: Browser | null = null;
|
||||
|
||||
constructor(config: Partial<ScraperConfig> = {}) {
|
||||
this.config = {
|
||||
maxRetries: 3,
|
||||
retryDelay: 2000,
|
||||
timeout: 30000,
|
||||
headless: true,
|
||||
...config,
|
||||
};
|
||||
}
|
||||
|
||||
abstract getVendorName(): 'eldorado' | 'playerauctions';
|
||||
abstract getTargetUrl(): string;
|
||||
abstract extractListings(page: Page): Promise<VendorListing[]>;
|
||||
|
||||
async scrape(): Promise<ScrapeResult> {
|
||||
const vendor = this.getVendorName();
|
||||
let lastError: string | undefined;
|
||||
|
||||
for (let attempt = 1; attempt <= this.config.maxRetries; attempt++) {
|
||||
try {
|
||||
const listings = await this.performScrape();
|
||||
|
||||
return {
|
||||
success: true,
|
||||
vendor,
|
||||
listings,
|
||||
scrapedAt: new Date(),
|
||||
};
|
||||
} catch (error) {
|
||||
lastError = error instanceof Error ? error.message : String(error);
|
||||
|
||||
if (attempt < this.config.maxRetries) {
|
||||
await this.delay(this.config.retryDelay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: false,
|
||||
vendor,
|
||||
listings: [],
|
||||
error: lastError,
|
||||
scrapedAt: new Date(),
|
||||
};
|
||||
}
|
||||
|
||||
private async performScrape(): Promise<VendorListing[]> {
|
||||
this.browser = await chromium.launch({ headless: this.config.headless });
|
||||
|
||||
try {
|
||||
const context = await this.browser.newContext({
|
||||
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
|
||||
});
|
||||
|
||||
const page = await context.newPage();
|
||||
page.setDefaultTimeout(this.config.timeout);
|
||||
|
||||
await page.goto(this.getTargetUrl(), { waitUntil: 'networkidle' });
|
||||
|
||||
const listings = await this.extractListings(page);
|
||||
|
||||
await context.close();
|
||||
|
||||
return listings;
|
||||
} finally {
|
||||
await this.browser?.close();
|
||||
this.browser = null;
|
||||
}
|
||||
}
|
||||
|
||||
protected calculatePricePerMillion(amountAUEC: number, priceUSD: number): number {
|
||||
return (priceUSD / amountAUEC) * 1_000_000;
|
||||
}
|
||||
|
||||
protected delay(ms: number): Promise<void> {
|
||||
return new Promise(resolve => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.browser) {
|
||||
await this.browser.close();
|
||||
this.browser = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
151
backend/src/scrapers/eldorado-scraper.ts
Normal file
151
backend/src/scrapers/eldorado-scraper.ts
Normal file
@@ -0,0 +1,151 @@
|
||||
import { Page } from 'playwright';
|
||||
import { BaseScraper } from './base-scraper';
|
||||
import { VendorListing } from './types';
|
||||
|
||||
export class EldoradoScraper extends BaseScraper {
|
||||
getVendorName(): 'eldorado' {
|
||||
return 'eldorado';
|
||||
}
|
||||
|
||||
getTargetUrl(): string {
|
||||
return 'https://www.eldorado.gg/star-citizen-auec/g/141-0-0';
|
||||
}
|
||||
|
||||
async extractListings(page: Page): Promise<VendorListing[]> {
|
||||
// Wait for page readiness
|
||||
await page.waitForSelector('text=Star Citizen aUEC', { timeout: 15000 }).catch(() => {});
|
||||
|
||||
// Wait for price elements to appear
|
||||
await page.waitForTimeout(3000);
|
||||
|
||||
const listings = await page.evaluate(() => {
|
||||
const results: Array<{
|
||||
amountAUEC: number;
|
||||
priceUSD: number;
|
||||
pricePerMillion: number;
|
||||
seller?: string;
|
||||
deliveryTime?: string;
|
||||
}> = [];
|
||||
|
||||
const bodyText = document.body.innerText;
|
||||
const lines = bodyText.split('\n').map(l => l.trim()).filter(l => l.length > 0);
|
||||
|
||||
// Track seen combinations to avoid duplicates
|
||||
const seenListings = new Set<string>();
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
|
||||
// Look for "$/M" pattern - this is the direct price per million
|
||||
// Examples: "$0.00007 / M", "$0.00018 / M", "$0.00007/M"
|
||||
const pricePerMMatch = line.match(/\$\s*([\d.]+)\s*\/?\s*\/\s*M/i) || line.match(/\$\s*([\d.]+)\s*\/\s*M/i);
|
||||
|
||||
if (pricePerMMatch) {
|
||||
const pricePerMillion = parseFloat(pricePerMMatch[1]);
|
||||
|
||||
|
||||
|
||||
// Look for "Min. qty" or "Min qty" nearby to get the quantity
|
||||
let minQtyM = 10000; // Default to 10000M
|
||||
for (let j = Math.max(0, i - 5); j < Math.min(lines.length, i + 5); j++) {
|
||||
const qtyLine = lines[j];
|
||||
// Match patterns like "Min. qty. 6000 M" or "Min qty: 16,000 M"
|
||||
const qtyMatch = qtyLine.match(/Min\.?\s*qty\.?\s*:?\s*([\d,]+)\s*M/i);
|
||||
if (qtyMatch) {
|
||||
minQtyM = parseFloat(qtyMatch[1].replace(/,/g, ''));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
const amountAUEC = minQtyM * 1_000_000;
|
||||
const priceUSD = pricePerMillion * minQtyM;
|
||||
|
||||
// Find seller name - look both backwards and forwards
|
||||
// For featured seller, name appears BEFORE the price
|
||||
// For other sellers, name appears in a structured list
|
||||
let seller: string | undefined;
|
||||
|
||||
// Search backwards first (for featured seller and some list items)
|
||||
for (let j = Math.max(0, i - 20); j < i; j++) {
|
||||
const sellerLine = lines[j];
|
||||
|
||||
// Skip common non-seller text
|
||||
if (
|
||||
sellerLine.includes('$') ||
|
||||
sellerLine.includes('Price') ||
|
||||
sellerLine.includes('qty') ||
|
||||
sellerLine.includes('stock') ||
|
||||
sellerLine.includes('Delivery') ||
|
||||
sellerLine.toLowerCase().includes('review') ||
|
||||
sellerLine.includes('Rating') ||
|
||||
sellerLine.includes('Offer') ||
|
||||
sellerLine.includes('Details') ||
|
||||
sellerLine.includes('FEATURED') ||
|
||||
sellerLine.includes('Other') ||
|
||||
sellerLine.includes('sellers') ||
|
||||
sellerLine.includes('aUEC') ||
|
||||
sellerLine === 'Star Citizen' || // Exclude the game title but not seller names
|
||||
sellerLine.includes('IMF') ||
|
||||
sellerLine.includes('in-game') ||
|
||||
sellerLine.includes('currency') ||
|
||||
sellerLine.length < 3 ||
|
||||
sellerLine.length > 30
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Match seller name patterns - alphanumeric with underscores/hyphens
|
||||
// Allow some special cases like "StarCitizen"
|
||||
if (/^[a-zA-Z][a-zA-Z0-9_-]*$/.test(sellerLine)) {
|
||||
seller = sellerLine;
|
||||
// Don't break - keep looking for a closer match
|
||||
}
|
||||
}
|
||||
|
||||
// Find delivery time
|
||||
let deliveryTime: string | undefined;
|
||||
for (let j = Math.max(0, i - 5); j < Math.min(lines.length, i + 5); j++) {
|
||||
const deliveryLine = lines[j];
|
||||
if (
|
||||
deliveryLine.match(/\d+\s*min/i) ||
|
||||
deliveryLine.match(/\d+\s*hour/i) ||
|
||||
deliveryLine.toLowerCase().includes('instant')
|
||||
) {
|
||||
deliveryTime = deliveryLine;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Create unique key to avoid duplicates
|
||||
const key = `${pricePerMillion}-${minQtyM}`;
|
||||
if (seenListings.has(key)) continue;
|
||||
seenListings.add(key);
|
||||
|
||||
results.push({
|
||||
amountAUEC,
|
||||
priceUSD,
|
||||
pricePerMillion,
|
||||
seller,
|
||||
deliveryTime,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
});
|
||||
|
||||
const scrapedAt = new Date();
|
||||
const url = this.getTargetUrl();
|
||||
|
||||
return listings.map(listing => ({
|
||||
vendor: 'eldorado' as const,
|
||||
amountAUEC: listing.amountAUEC,
|
||||
priceUSD: listing.priceUSD,
|
||||
pricePerMillion: listing.pricePerMillion,
|
||||
seller: listing.seller,
|
||||
deliveryTime: listing.deliveryTime,
|
||||
scrapedAt,
|
||||
url,
|
||||
}));
|
||||
}
|
||||
}
|
||||
6
backend/src/scrapers/index.ts
Normal file
6
backend/src/scrapers/index.ts
Normal file
@@ -0,0 +1,6 @@
|
||||
export { BaseScraper } from './base-scraper';
|
||||
export { EldoradoScraper } from './eldorado-scraper';
|
||||
export { PlayerAuctionsScraper } from './playerauctions-scraper';
|
||||
export { ScraperService } from './scraper-service';
|
||||
export { ScraperScheduler } from './scheduler';
|
||||
export type { VendorListing, ScrapeResult, ScraperConfig } from './types';
|
||||
49
backend/src/scrapers/manual-scrape.ts
Normal file
49
backend/src/scrapers/manual-scrape.ts
Normal file
@@ -0,0 +1,49 @@
|
||||
import { ScraperService } from './scraper-service';
|
||||
|
||||
async function main() {
|
||||
const scraperService = new ScraperService();
|
||||
|
||||
try {
|
||||
console.log('Scraping in progress...\n');
|
||||
|
||||
const results = await scraperService.scrapeAll();
|
||||
|
||||
// Show completion status for each vendor
|
||||
results.forEach(result => {
|
||||
if (result.success) {
|
||||
console.log(`✓ ${result.vendor.charAt(0).toUpperCase() + result.vendor.slice(1)} scraping done`);
|
||||
} else {
|
||||
console.log(`✗ ${result.vendor.charAt(0).toUpperCase() + result.vendor.slice(1)} scraping failed`);
|
||||
}
|
||||
});
|
||||
|
||||
// Show listings from each vendor
|
||||
console.log('');
|
||||
results.forEach(result => {
|
||||
console.log(`[${result.vendor.toUpperCase()}] Found ${result.listings.length} listings`);
|
||||
if (result.listings.length > 0) {
|
||||
result.listings.forEach((listing, i) => {
|
||||
console.log(` ${i + 1}. $${listing.pricePerMillion}/M (${listing.seller || 'Unknown'})`);
|
||||
});
|
||||
}
|
||||
console.log('');
|
||||
});
|
||||
|
||||
const allListings = results.flatMap(r => r.listings);
|
||||
const lowestPrice = scraperService.calculatePriceIndex(allListings);
|
||||
|
||||
console.log('=== LOWEST PRICE ===');
|
||||
if (lowestPrice) {
|
||||
console.log(`$${lowestPrice}/M`);
|
||||
} else {
|
||||
console.log('No listings found');
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
process.exit(1);
|
||||
} finally {
|
||||
await scraperService.close();
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
202
backend/src/scrapers/playerauctions-scraper.ts
Normal file
202
backend/src/scrapers/playerauctions-scraper.ts
Normal file
@@ -0,0 +1,202 @@
|
||||
import { Page } from 'playwright';
|
||||
import { BaseScraper } from './base-scraper';
|
||||
import { VendorListing } from './types';
|
||||
|
||||
export class PlayerAuctionsScraper extends BaseScraper {
|
||||
getVendorName(): 'playerauctions' {
|
||||
return 'playerauctions';
|
||||
}
|
||||
|
||||
getTargetUrl(): string {
|
||||
return 'https://www.playerauctions.com/star-citizen-auec/';
|
||||
}
|
||||
|
||||
async extractListings(page: Page): Promise<VendorListing[]> {
|
||||
// Wait for page readiness
|
||||
await page.waitForTimeout(3000);
|
||||
|
||||
// Close cookie popup if it exists
|
||||
try {
|
||||
const cookieClose = page.locator('[id*="cookie"] button, [class*="cookie"] button, button:has-text("Accept"), button:has-text("Close")').first();
|
||||
if (await cookieClose.isVisible({ timeout: 2000 }).catch(() => false)) {
|
||||
await cookieClose.click();
|
||||
await page.waitForTimeout(500);
|
||||
}
|
||||
} catch (e) {
|
||||
// No cookie popup or already closed
|
||||
}
|
||||
|
||||
// Find offer cards - they have class "offer-item"
|
||||
const offerCards = await page.locator('.offer-item, [class*="offer-item"]').all();
|
||||
|
||||
if (offerCards.length === 0) {
|
||||
return this.extractListingsAlternative(page);
|
||||
}
|
||||
|
||||
const listings: VendorListing[] = [];
|
||||
const targetQuantityM = 100000; // 10000 M = 10 billion AUEC (field is already in millions)
|
||||
|
||||
// Step 2-5: Process each offer card
|
||||
for (let i = 0; i < Math.min(offerCards.length, 20); i++) {
|
||||
try {
|
||||
const card = offerCards[i];
|
||||
|
||||
// Find the quantity input (shows number with "M" suffix, has +/- buttons)
|
||||
const qtyInput = card.locator('input[type="number"]').first();
|
||||
|
||||
if (!(await qtyInput.isVisible({ timeout: 1000 }).catch(() => false))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Set quantity to 10000 (which means 10000 M = 10 billion AUEC)
|
||||
await qtyInput.scrollIntoViewIfNeeded();
|
||||
await qtyInput.click({ force: true });
|
||||
await qtyInput.fill('');
|
||||
await qtyInput.pressSequentially(targetQuantityM.toString(), { delay: 10 });
|
||||
await qtyInput.press('Enter'); // Trigger update
|
||||
|
||||
// Wait for price to update (0.5-2 seconds as per instructions)
|
||||
await page.waitForTimeout(2500);
|
||||
|
||||
// Step 3: Extract the total price from the BUY NOW button area
|
||||
// Look for the price near the BUY NOW button - it's typically in a large font
|
||||
let totalPriceUSD = 0;
|
||||
|
||||
// Try to find the price element near BUY NOW button
|
||||
const buyNowButton = card.locator('button:has-text("BUY NOW"), [class*="buy"]').first();
|
||||
if (await buyNowButton.isVisible().catch(() => false)) {
|
||||
// Get the parent container and look for price nearby
|
||||
const priceContainer = buyNowButton.locator('xpath=..').first();
|
||||
const priceText = await priceContainer.textContent().catch(() => '');
|
||||
|
||||
// Extract price - should be like "$5.00" in large text
|
||||
if (priceText) {
|
||||
const priceMatch = priceText.match(/\$\s*([\d,]+\.\d{2})/);
|
||||
if (priceMatch) {
|
||||
totalPriceUSD = parseFloat(priceMatch[1].replace(/,/g, ''));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: look for price in the card, but exclude "Minutes" context
|
||||
if (totalPriceUSD === 0) {
|
||||
const cardText = await card.textContent().catch(() => '');
|
||||
if (cardText) {
|
||||
const lines = cardText.split('\n').map(l => l.trim());
|
||||
|
||||
for (const line of lines) {
|
||||
// Skip lines that contain time indicators
|
||||
if (line.includes('Minutes') || line.includes('Hours') || line.includes('Days')) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Look for price pattern with decimal
|
||||
const priceMatch = line.match(/\$\s*([\d,]+\.\d{2})/);
|
||||
if (priceMatch) {
|
||||
const price = parseFloat(priceMatch[1].replace(/,/g, ''));
|
||||
if (price > 0 && price < 100000) {
|
||||
totalPriceUSD = price;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (totalPriceUSD === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Step 4: Compute USD per 1M
|
||||
const pricePerMillion = totalPriceUSD / targetQuantityM;
|
||||
|
||||
// Extract seller name and delivery time from card text
|
||||
const fullCardText = await card.textContent().catch(() => '');
|
||||
const sellerMatch = fullCardText ? fullCardText.match(/([a-zA-Z0-9_-]{3,20})/) : null;
|
||||
const seller = sellerMatch ? sellerMatch[1] : 'Unknown';
|
||||
|
||||
const deliveryMatch = fullCardText ? fullCardText.match(/(\d+\s*(?:Minutes?|Hours?|Days?))/i) : null;
|
||||
const deliveryTime = deliveryMatch ? deliveryMatch[1] : undefined;
|
||||
|
||||
listings.push({
|
||||
vendor: 'playerauctions',
|
||||
amountAUEC: targetQuantityM * 1_000_000,
|
||||
priceUSD: totalPriceUSD,
|
||||
pricePerMillion,
|
||||
seller: seller.trim(),
|
||||
deliveryTime,
|
||||
scrapedAt: new Date(),
|
||||
url: this.getTargetUrl(),
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
// Skip this card
|
||||
}
|
||||
}
|
||||
|
||||
if (listings.length === 0) {
|
||||
return this.extractListingsAlternative(page);
|
||||
}
|
||||
|
||||
return listings;
|
||||
}
|
||||
|
||||
private async extractListingsAlternative(page: Page): Promise<VendorListing[]> {
|
||||
|
||||
const listings = await page.evaluate(() => {
|
||||
const results: Array<{
|
||||
amountAUEC: number;
|
||||
priceUSD: number;
|
||||
pricePerMillion: number;
|
||||
seller?: string;
|
||||
}> = [];
|
||||
|
||||
const bodyText = document.body.innerText;
|
||||
const lines = bodyText.split('\n').map(l => l.trim()).filter(l => l.length > 0);
|
||||
|
||||
const seenPrices = new Set<number>();
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
|
||||
// Look for "$1 = X M aUEC" pattern and convert
|
||||
const exchangeMatch = line.match(/\$1\s*=\s*([\d,]+(?:\.\d+)?)\s*M\s*(?:aUEC)?/i);
|
||||
|
||||
if (exchangeMatch) {
|
||||
const millionsPerDollar = parseFloat(exchangeMatch[1].replace(/,/g, ''));
|
||||
const pricePerMillion = 1 / millionsPerDollar;
|
||||
|
||||
if (seenPrices.has(pricePerMillion)) continue;
|
||||
seenPrices.add(pricePerMillion);
|
||||
|
||||
const targetQuantityM = 100000;
|
||||
const amountAUEC = targetQuantityM * 1_000_000;
|
||||
const priceUSD = pricePerMillion * targetQuantityM;
|
||||
|
||||
results.push({
|
||||
amountAUEC,
|
||||
priceUSD,
|
||||
pricePerMillion,
|
||||
seller: 'Unknown',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
});
|
||||
|
||||
const scrapedAt = new Date();
|
||||
const url = this.getTargetUrl();
|
||||
|
||||
return listings.map(listing => ({
|
||||
vendor: 'playerauctions' as const,
|
||||
amountAUEC: listing.amountAUEC,
|
||||
priceUSD: listing.priceUSD,
|
||||
pricePerMillion: listing.pricePerMillion,
|
||||
seller: listing.seller,
|
||||
deliveryTime: undefined,
|
||||
scrapedAt,
|
||||
url,
|
||||
}));
|
||||
}
|
||||
}
|
||||
80
backend/src/scrapers/scheduler.ts
Normal file
80
backend/src/scrapers/scheduler.ts
Normal file
@@ -0,0 +1,80 @@
|
||||
import * as schedule from 'node-schedule';
|
||||
import { ScraperService } from './scraper-service';
|
||||
import { ScrapeResult } from './types';
|
||||
|
||||
export type ScrapeCallback = (results: ScrapeResult[]) => Promise<void>;
|
||||
|
||||
export class ScraperScheduler {
|
||||
private scraperService: ScraperService;
|
||||
private job: schedule.Job | null = null;
|
||||
private callback: ScrapeCallback | null = null;
|
||||
private intervalMinutes: number;
|
||||
|
||||
constructor(intervalMinutes: number = 5) {
|
||||
this.scraperService = new ScraperService();
|
||||
this.intervalMinutes = intervalMinutes;
|
||||
}
|
||||
|
||||
onScrapeComplete(callback: ScrapeCallback): void {
|
||||
this.callback = callback;
|
||||
}
|
||||
|
||||
start(): void {
|
||||
if (this.job) {
|
||||
console.log('Scheduler already running');
|
||||
return;
|
||||
}
|
||||
|
||||
// Run immediately on start
|
||||
this.runScrape();
|
||||
|
||||
// Schedule recurring scrapes
|
||||
const rule = `*/${this.intervalMinutes} * * * *`;
|
||||
this.job = schedule.scheduleJob(rule, () => {
|
||||
this.runScrape();
|
||||
});
|
||||
|
||||
console.log(`Scraper scheduled to run every ${this.intervalMinutes} minutes`);
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
if (this.job) {
|
||||
this.job.cancel();
|
||||
this.job = null;
|
||||
console.log('Scraper scheduler stopped');
|
||||
}
|
||||
}
|
||||
|
||||
async runScrape(): Promise<void> {
|
||||
try {
|
||||
console.log(`[${new Date().toISOString()}] Running scheduled scrape...`);
|
||||
|
||||
const results = await this.scraperService.scrapeAll();
|
||||
|
||||
const successCount = results.filter(r => r.success).length;
|
||||
const totalListings = results.reduce((sum, r) => sum + r.listings.length, 0);
|
||||
|
||||
console.log(`Scrape complete: ${successCount}/${results.length} vendors successful, ${totalListings} listings`);
|
||||
|
||||
if (this.callback) {
|
||||
await this.callback(results);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error during scheduled scrape:', error);
|
||||
}
|
||||
}
|
||||
|
||||
setInterval(minutes: number): void {
|
||||
this.intervalMinutes = minutes;
|
||||
|
||||
if (this.job) {
|
||||
this.stop();
|
||||
this.start();
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.stop();
|
||||
await this.scraperService.close();
|
||||
}
|
||||
}
|
||||
59
backend/src/scrapers/scraper-service.ts
Normal file
59
backend/src/scrapers/scraper-service.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
import { EldoradoScraper } from './eldorado-scraper';
|
||||
import { PlayerAuctionsScraper } from './playerauctions-scraper';
|
||||
import { ScrapeResult, VendorListing } from './types';
|
||||
|
||||
export class ScraperService {
|
||||
private eldoradoScraper: EldoradoScraper;
|
||||
private playerAuctionsScraper: PlayerAuctionsScraper;
|
||||
|
||||
constructor() {
|
||||
this.eldoradoScraper = new EldoradoScraper();
|
||||
this.playerAuctionsScraper = new PlayerAuctionsScraper();
|
||||
}
|
||||
|
||||
async scrapeAll(): Promise<ScrapeResult[]> {
|
||||
const results = await Promise.allSettled([
|
||||
this.eldoradoScraper.scrape(),
|
||||
this.playerAuctionsScraper.scrape(),
|
||||
]);
|
||||
|
||||
return results.map((result, index) => {
|
||||
if (result.status === 'fulfilled') {
|
||||
return result.value;
|
||||
} else {
|
||||
const vendor = index === 0 ? 'eldorado' : 'playerauctions';
|
||||
return {
|
||||
success: false,
|
||||
vendor,
|
||||
listings: [],
|
||||
error: result.reason?.message || 'Unknown error',
|
||||
scrapedAt: new Date(),
|
||||
};
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async scrapeEldorado(): Promise<ScrapeResult> {
|
||||
return this.eldoradoScraper.scrape();
|
||||
}
|
||||
|
||||
async scrapePlayerAuctions(): Promise<ScrapeResult> {
|
||||
return this.playerAuctionsScraper.scrape();
|
||||
}
|
||||
|
||||
calculatePriceIndex(listings: VendorListing[]): number | null {
|
||||
if (listings.length === 0) return null;
|
||||
|
||||
const prices = listings.map(l => l.pricePerMillion);
|
||||
|
||||
// Return the lowest price
|
||||
return Math.min(...prices);
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
await Promise.all([
|
||||
this.eldoradoScraper.close(),
|
||||
this.playerAuctionsScraper.close(),
|
||||
]);
|
||||
}
|
||||
}
|
||||
25
backend/src/scrapers/types.ts
Normal file
25
backend/src/scrapers/types.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
export interface VendorListing {
|
||||
vendor: 'eldorado' | 'playerauctions';
|
||||
amountAUEC: number;
|
||||
priceUSD: number;
|
||||
pricePerMillion: number;
|
||||
seller?: string;
|
||||
deliveryTime?: string;
|
||||
scrapedAt: Date;
|
||||
url: string;
|
||||
}
|
||||
|
||||
export interface ScrapeResult {
|
||||
success: boolean;
|
||||
vendor: string;
|
||||
listings: VendorListing[];
|
||||
error?: string;
|
||||
scrapedAt: Date;
|
||||
}
|
||||
|
||||
export interface ScraperConfig {
|
||||
maxRetries: number;
|
||||
retryDelay: number;
|
||||
timeout: number;
|
||||
headless: boolean;
|
||||
}
|
||||
26
backend/test-ws.js
Normal file
26
backend/test-ws.js
Normal file
@@ -0,0 +1,26 @@
|
||||
const WebSocket = require('ws');
|
||||
|
||||
const ws = new WebSocket('ws://localhost:3000/ws');
|
||||
|
||||
ws.on('open', () => {
|
||||
console.log('✓ Connected to WebSocket');
|
||||
ws.send(JSON.stringify({ type: 'subscribe', data: { channel: 'price_updates' } }));
|
||||
});
|
||||
|
||||
ws.on('message', (data) => {
|
||||
console.log('✓ Received:', data.toString());
|
||||
});
|
||||
|
||||
ws.on('close', (code, reason) => {
|
||||
console.log(`✗ Connection closed: ${code} - ${reason}`);
|
||||
});
|
||||
|
||||
ws.on('error', (error) => {
|
||||
console.error('✗ WebSocket error:', error.message);
|
||||
});
|
||||
|
||||
setTimeout(() => {
|
||||
console.log('Closing connection...');
|
||||
ws.close();
|
||||
process.exit(0);
|
||||
}, 5000);
|
||||
55
backend/tests/scrapers/scraper-service.test.ts
Normal file
55
backend/tests/scrapers/scraper-service.test.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { ScraperService } from '../../src/scrapers/scraper-service';
|
||||
import { VendorListing } from '../../src/scrapers/types';
|
||||
|
||||
describe('ScraperService', () => {
|
||||
let service: ScraperService;
|
||||
|
||||
beforeEach(() => {
|
||||
service = new ScraperService();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await service.close();
|
||||
});
|
||||
|
||||
describe('calculatePriceIndex', () => {
|
||||
it('should return null for empty listings', () => {
|
||||
const result = service.calculatePriceIndex([]);
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('should calculate median for odd number of listings', () => {
|
||||
const listings: VendorListing[] = [
|
||||
{ pricePerMillion: 10 } as VendorListing,
|
||||
{ pricePerMillion: 20 } as VendorListing,
|
||||
{ pricePerMillion: 30 } as VendorListing,
|
||||
];
|
||||
|
||||
const result = service.calculatePriceIndex(listings);
|
||||
expect(result).toBe(20);
|
||||
});
|
||||
|
||||
it('should calculate median for even number of listings', () => {
|
||||
const listings: VendorListing[] = [
|
||||
{ pricePerMillion: 10 } as VendorListing,
|
||||
{ pricePerMillion: 20 } as VendorListing,
|
||||
{ pricePerMillion: 30 } as VendorListing,
|
||||
{ pricePerMillion: 40 } as VendorListing,
|
||||
];
|
||||
|
||||
const result = service.calculatePriceIndex(listings);
|
||||
expect(result).toBe(25);
|
||||
});
|
||||
|
||||
it('should handle unsorted listings', () => {
|
||||
const listings: VendorListing[] = [
|
||||
{ pricePerMillion: 30 } as VendorListing,
|
||||
{ pricePerMillion: 10 } as VendorListing,
|
||||
{ pricePerMillion: 20 } as VendorListing,
|
||||
];
|
||||
|
||||
const result = service.calculatePriceIndex(listings);
|
||||
expect(result).toBe(20);
|
||||
});
|
||||
});
|
||||
});
|
||||
20
backend/tsconfig.json
Normal file
20
backend/tsconfig.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "commonjs",
|
||||
"lib": ["ES2022", "DOM"],
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"moduleResolution": "node",
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist", "tests"]
|
||||
}
|
||||
50
docker-compose.yml
Normal file
50
docker-compose.yml
Normal file
@@ -0,0 +1,50 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: rmtpw-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: rmtpw
|
||||
POSTGRES_PASSWORD: rmtpw_password
|
||||
POSTGRES_DB: rmtpocketwatcher
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U rmtpw -d rmtpocketwatcher"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
backend:
|
||||
build:
|
||||
context: ./backend
|
||||
dockerfile: Dockerfile
|
||||
container_name: rmtpw-backend
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_URL: postgresql://rmtpw:rmtpw_password@postgres:5432/rmtpocketwatcher?schema=public
|
||||
PORT: 3000
|
||||
HOST: 0.0.0.0
|
||||
NODE_ENV: production
|
||||
SCRAPE_INTERVAL_MINUTES: 5
|
||||
SCRAPER_HEADLESS: "true"
|
||||
SCRAPER_TIMEOUT: 30000
|
||||
SCRAPER_MAX_RETRIES: 3
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
- ./backend:/app
|
||||
- /app/node_modules
|
||||
- /app/dist
|
||||
command: sh -c "npm run db:push && npm start"
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
5
electron-app/.env.example
Normal file
5
electron-app/.env.example
Normal file
@@ -0,0 +1,5 @@
|
||||
# WebSocket connection URL
|
||||
WS_URL=ws://localhost:3000/ws
|
||||
|
||||
# Development mode
|
||||
NODE_ENV=development
|
||||
62
electron-app/.gitignore
vendored
Normal file
62
electron-app/.gitignore
vendored
Normal file
@@ -0,0 +1,62 @@
|
||||
# Dependencies
|
||||
node_modules/
|
||||
package-lock.json
|
||||
|
||||
# Build outputs
|
||||
dist/
|
||||
dist-electron/
|
||||
out/
|
||||
build/
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
|
||||
# Logs
|
||||
logs/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
|
||||
# Editor directories and files
|
||||
.vscode/*
|
||||
!.vscode/extensions.json
|
||||
.idea/
|
||||
*.suo
|
||||
*.ntvs*
|
||||
*.njsproj
|
||||
*.sln
|
||||
*.sw?
|
||||
.DS_Store
|
||||
|
||||
# OS files
|
||||
Thumbs.db
|
||||
Desktop.ini
|
||||
|
||||
# Electron specific
|
||||
*.asar
|
||||
*.dmg
|
||||
*.exe
|
||||
*.AppImage
|
||||
*.deb
|
||||
*.rpm
|
||||
*.snap
|
||||
|
||||
# Release builds
|
||||
release/
|
||||
releases/
|
||||
|
||||
# TypeScript
|
||||
*.tsbuildinfo
|
||||
|
||||
# Testing
|
||||
coverage/
|
||||
.nyc_output/
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.temp
|
||||
.cache/
|
||||
61
electron-app/README.md
Normal file
61
electron-app/README.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# rmtPocketWatcher - Electron Desktop App
|
||||
|
||||
Cross-platform desktop application for tracking AUEC RMT prices in real-time.
|
||||
|
||||
## WebSocket Implementation
|
||||
|
||||
The app uses a robust WebSocket client with the following features:
|
||||
|
||||
### Main Process (src/main/)
|
||||
- **websocket-client.ts**: Core WebSocket client with auto-reconnect logic
|
||||
- Exponential backoff reconnection (max 10 attempts)
|
||||
- Event-based architecture for easy integration
|
||||
- Connection state management
|
||||
|
||||
- **ipc-handlers.ts**: IPC bridge between main and renderer processes
|
||||
- Forwards WebSocket events to renderer
|
||||
- Handles renderer commands (connect/disconnect/send)
|
||||
- Auto-connects on app startup
|
||||
|
||||
### Renderer Process (src/renderer/)
|
||||
- **hooks/useWebSocket.ts**: React hook for WebSocket integration
|
||||
- Manages connection status
|
||||
- Provides latest price updates
|
||||
- Exposes connect/disconnect/send methods
|
||||
|
||||
### Shared Types (src/shared/)
|
||||
- **types.ts**: TypeScript interfaces shared between processes
|
||||
- PriceIndex, VendorPrice, WebSocketMessage
|
||||
- Ensures type safety across IPC boundary
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Start Vite dev server and Electron
|
||||
npm run electron:dev
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
# Build for production
|
||||
npm run electron:build
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Copy `.env.example` to `.env` and configure:
|
||||
- `WS_URL`: WebSocket server URL (default: ws://localhost:3000/ws)
|
||||
|
||||
## Security
|
||||
|
||||
- Context isolation enabled
|
||||
- Sandbox mode enabled
|
||||
- No remote code evaluation
|
||||
- Secure IPC communication via preload script
|
||||
51
electron-app/index.html
Normal file
51
electron-app/index.html
Normal file
@@ -0,0 +1,51 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'">
|
||||
<title>rmtPocketWatcher</title>
|
||||
<style>
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html, body {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#root {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
overflow-y: scroll;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Custom scrollbar */
|
||||
#root::-webkit-scrollbar {
|
||||
width: 12px;
|
||||
}
|
||||
|
||||
#root::-webkit-scrollbar-track {
|
||||
background: #1a1f3a;
|
||||
}
|
||||
|
||||
#root::-webkit-scrollbar-thumb {
|
||||
background: #50e3c2;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
#root::-webkit-scrollbar-thumb:hover {
|
||||
background: #3cc9a8;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/renderer/index.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
61
electron-app/package.json
Normal file
61
electron-app/package.json
Normal file
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"name": "rmtpocketwatcher",
|
||||
"version": "1.0.0",
|
||||
"description": "Real-time AUEC price tracking desktop application",
|
||||
"main": "dist/main/index.js",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build:main": "tsc --project tsconfig.main.json && tsc --project tsconfig.preload.json",
|
||||
"build:renderer": "vite build",
|
||||
"build": "npm run build:main && npm run build:renderer",
|
||||
"electron:dev": "concurrently \"npm run dev\" \"wait-on http://localhost:5173 && cross-env NODE_ENV=development electron dist/main/index.js\"",
|
||||
"electron:build": "npm run build && electron-builder",
|
||||
"test": "jest"
|
||||
},
|
||||
"dependencies": {
|
||||
"electron": "^30.0.0",
|
||||
"electron-updater": "^6.1.0",
|
||||
"recharts": "^3.5.1",
|
||||
"ws": "^8.16.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.10.0",
|
||||
"@types/react": "^18.3.27",
|
||||
"@types/ws": "^8.5.10",
|
||||
"@vitejs/plugin-react": "^5.1.1",
|
||||
"concurrently": "^8.2.0",
|
||||
"cross-env": "^10.1.0",
|
||||
"electron-builder": "^24.9.0",
|
||||
"react": "^19.2.1",
|
||||
"react-dom": "^19.2.1",
|
||||
"typescript": "^5.3.0",
|
||||
"vite": "^5.0.0",
|
||||
"wait-on": "^7.2.0"
|
||||
},
|
||||
"build": {
|
||||
"appId": "com.lambdabanking.rmtpocketwatcher",
|
||||
"productName": "rmtPocketWatcher",
|
||||
"directories": {
|
||||
"output": "release"
|
||||
},
|
||||
"files": [
|
||||
"dist/**/*",
|
||||
"package.json"
|
||||
],
|
||||
"win": {
|
||||
"target": [
|
||||
"nsis"
|
||||
]
|
||||
},
|
||||
"mac": {
|
||||
"target": [
|
||||
"dmg"
|
||||
]
|
||||
},
|
||||
"linux": {
|
||||
"target": [
|
||||
"AppImage"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
53
electron-app/src/main/index.ts
Normal file
53
electron-app/src/main/index.ts
Normal file
@@ -0,0 +1,53 @@
|
||||
import { app, BrowserWindow } from 'electron';
|
||||
import * as path from 'path';
|
||||
import { setupIpcHandlers, cleanupIpcHandlers } from './ipc-handlers';
|
||||
|
||||
let mainWindow: BrowserWindow | null = null;
|
||||
|
||||
function createWindow(): void {
|
||||
mainWindow = new BrowserWindow({
|
||||
width: 1400,
|
||||
height: 900,
|
||||
webPreferences: {
|
||||
preload: path.join(__dirname, 'preload.js'),
|
||||
nodeIntegration: false,
|
||||
contextIsolation: true,
|
||||
sandbox: true,
|
||||
},
|
||||
title: 'rmtPocketWatcher',
|
||||
});
|
||||
|
||||
// Setup IPC handlers for WebSocket communication
|
||||
setupIpcHandlers(mainWindow);
|
||||
|
||||
// Load the app
|
||||
if (process.env.NODE_ENV === 'development') {
|
||||
mainWindow.loadURL('http://localhost:5173');
|
||||
mainWindow.webContents.openDevTools();
|
||||
} else {
|
||||
mainWindow.loadFile(path.join(__dirname, '../../renderer/index.html'));
|
||||
}
|
||||
|
||||
mainWindow.on('closed', () => {
|
||||
cleanupIpcHandlers();
|
||||
mainWindow = null;
|
||||
});
|
||||
}
|
||||
|
||||
app.whenReady().then(createWindow);
|
||||
|
||||
app.on('window-all-closed', () => {
|
||||
if (process.platform !== 'darwin') {
|
||||
app.quit();
|
||||
}
|
||||
});
|
||||
|
||||
app.on('activate', () => {
|
||||
if (BrowserWindow.getAllWindows().length === 0) {
|
||||
createWindow();
|
||||
}
|
||||
});
|
||||
|
||||
app.on('before-quit', () => {
|
||||
cleanupIpcHandlers();
|
||||
});
|
||||
73
electron-app/src/main/ipc-handlers.ts
Normal file
73
electron-app/src/main/ipc-handlers.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
import { ipcMain, BrowserWindow } from 'electron';
|
||||
import { WebSocketClient } from './websocket-client';
|
||||
import { PriceIndex } from '../shared/types';
|
||||
|
||||
let wsClient: WebSocketClient | null = null;
|
||||
|
||||
export function setupIpcHandlers(mainWindow: BrowserWindow): void {
|
||||
const wsUrl = process.env.WS_URL || 'ws://localhost:3000/ws';
|
||||
|
||||
// Initialize WebSocket client
|
||||
wsClient = new WebSocketClient(wsUrl);
|
||||
|
||||
// Forward WebSocket events to renderer
|
||||
wsClient.on('connected', () => {
|
||||
mainWindow.webContents.send('ws:connected');
|
||||
});
|
||||
|
||||
wsClient.on('disconnected', () => {
|
||||
mainWindow.webContents.send('ws:disconnected');
|
||||
});
|
||||
|
||||
wsClient.on('priceUpdate', (data: PriceIndex) => {
|
||||
mainWindow.webContents.send('ws:priceUpdate', data);
|
||||
});
|
||||
|
||||
wsClient.on('historyData', (data: any) => {
|
||||
mainWindow.webContents.send('ws:historyData', data);
|
||||
});
|
||||
|
||||
wsClient.on('error', (error: Error) => {
|
||||
mainWindow.webContents.send('ws:error', error.message);
|
||||
});
|
||||
|
||||
wsClient.on('maxReconnectAttemptsReached', () => {
|
||||
mainWindow.webContents.send('ws:maxReconnectAttemptsReached');
|
||||
});
|
||||
|
||||
// IPC handlers for renderer requests
|
||||
ipcMain.handle('ws:connect', async () => {
|
||||
wsClient?.connect();
|
||||
return { success: true };
|
||||
});
|
||||
|
||||
ipcMain.handle('ws:disconnect', async () => {
|
||||
wsClient?.disconnect();
|
||||
return { success: true };
|
||||
});
|
||||
|
||||
ipcMain.handle('ws:getStatus', async () => {
|
||||
return {
|
||||
connected: wsClient?.isConnected() || false,
|
||||
state: wsClient?.getConnectionState() || 'DISCONNECTED',
|
||||
};
|
||||
});
|
||||
|
||||
ipcMain.handle('ws:send', async (_event, message: any) => {
|
||||
wsClient?.sendMessage(message);
|
||||
return { success: true };
|
||||
});
|
||||
|
||||
// Auto-connect on startup
|
||||
wsClient.connect();
|
||||
}
|
||||
|
||||
export function cleanupIpcHandlers(): void {
|
||||
wsClient?.disconnect();
|
||||
wsClient = null;
|
||||
|
||||
ipcMain.removeHandler('ws:connect');
|
||||
ipcMain.removeHandler('ws:disconnect');
|
||||
ipcMain.removeHandler('ws:getStatus');
|
||||
ipcMain.removeHandler('ws:send');
|
||||
}
|
||||
145
electron-app/src/main/websocket-client.ts
Normal file
145
electron-app/src/main/websocket-client.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
import WebSocket from 'ws';
|
||||
import { EventEmitter } from 'events';
|
||||
import { PriceIndex, WebSocketMessage, HistoricalData } from '../shared/types';
|
||||
|
||||
export class WebSocketClient extends EventEmitter {
|
||||
private ws: WebSocket | null = null;
|
||||
private url: string;
|
||||
private reconnectInterval: number = 5000;
|
||||
private reconnectTimer: NodeJS.Timeout | null = null;
|
||||
private isIntentionallyClosed: boolean = false;
|
||||
private maxReconnectAttempts: number = 10;
|
||||
private reconnectAttempts: number = 0;
|
||||
|
||||
constructor(url: string) {
|
||||
super();
|
||||
this.url = url;
|
||||
}
|
||||
|
||||
connect(): void {
|
||||
if (this.ws?.readyState === WebSocket.OPEN) {
|
||||
console.log('WebSocket already connected');
|
||||
return;
|
||||
}
|
||||
|
||||
this.isIntentionallyClosed = false;
|
||||
console.log(`Connecting to WebSocket: ${this.url}`);
|
||||
|
||||
try {
|
||||
this.ws = new WebSocket(this.url);
|
||||
|
||||
this.ws.on('open', () => {
|
||||
console.log('WebSocket connected');
|
||||
this.reconnectAttempts = 0;
|
||||
this.emit('connected');
|
||||
this.sendMessage({ type: 'subscribe', data: { channel: 'price_updates' } });
|
||||
});
|
||||
|
||||
this.ws.on('message', (data: WebSocket.Data) => {
|
||||
try {
|
||||
const message: WebSocketMessage = JSON.parse(data.toString());
|
||||
this.handleMessage(message);
|
||||
} catch (error) {
|
||||
console.error('Failed to parse WebSocket message:', error);
|
||||
}
|
||||
});
|
||||
|
||||
this.ws.on('close', (code: number, reason: Buffer) => {
|
||||
console.log(`WebSocket closed: ${code} - ${reason.toString()}`);
|
||||
this.emit('disconnected');
|
||||
|
||||
if (!this.isIntentionallyClosed) {
|
||||
this.scheduleReconnect();
|
||||
}
|
||||
});
|
||||
|
||||
this.ws.on('error', (error: Error) => {
|
||||
console.error('WebSocket error:', error);
|
||||
this.emit('error', error);
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Failed to create WebSocket connection:', error);
|
||||
this.scheduleReconnect();
|
||||
}
|
||||
}
|
||||
|
||||
private handleMessage(message: WebSocketMessage): void {
|
||||
switch (message.type) {
|
||||
case 'price_update':
|
||||
this.emit('priceUpdate', message.data as PriceIndex);
|
||||
break;
|
||||
case 'history_data':
|
||||
this.emit('historyData', message.data as HistoricalData);
|
||||
break;
|
||||
case 'connection_status':
|
||||
this.emit('status', message.data);
|
||||
break;
|
||||
case 'error':
|
||||
this.emit('error', new Error(message.data));
|
||||
break;
|
||||
default:
|
||||
console.warn('Unknown message type:', message.type);
|
||||
}
|
||||
}
|
||||
|
||||
private scheduleReconnect(): void {
|
||||
if (this.reconnectTimer || this.isIntentionallyClosed) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.reconnectAttempts >= this.maxReconnectAttempts) {
|
||||
console.error('Max reconnection attempts reached');
|
||||
this.emit('maxReconnectAttemptsReached');
|
||||
return;
|
||||
}
|
||||
|
||||
this.reconnectAttempts++;
|
||||
const delay = Math.min(this.reconnectInterval * this.reconnectAttempts, 30000);
|
||||
|
||||
console.log(`Scheduling reconnect in ${delay}ms (attempt ${this.reconnectAttempts})`);
|
||||
|
||||
this.reconnectTimer = setTimeout(() => {
|
||||
this.reconnectTimer = null;
|
||||
this.connect();
|
||||
}, delay);
|
||||
}
|
||||
|
||||
sendMessage(message: any): void {
|
||||
if (this.ws?.readyState === WebSocket.OPEN) {
|
||||
this.ws.send(JSON.stringify(message));
|
||||
} else {
|
||||
console.warn('WebSocket not connected, cannot send message');
|
||||
}
|
||||
}
|
||||
|
||||
disconnect(): void {
|
||||
this.isIntentionallyClosed = true;
|
||||
|
||||
if (this.reconnectTimer) {
|
||||
clearTimeout(this.reconnectTimer);
|
||||
this.reconnectTimer = null;
|
||||
}
|
||||
|
||||
if (this.ws) {
|
||||
this.ws.close();
|
||||
this.ws = null;
|
||||
}
|
||||
}
|
||||
|
||||
isConnected(): boolean {
|
||||
return this.ws?.readyState === WebSocket.OPEN;
|
||||
}
|
||||
|
||||
getConnectionState(): string {
|
||||
if (!this.ws) return 'DISCONNECTED';
|
||||
|
||||
switch (this.ws.readyState) {
|
||||
case WebSocket.CONNECTING: return 'CONNECTING';
|
||||
case WebSocket.OPEN: return 'CONNECTED';
|
||||
case WebSocket.CLOSING: return 'CLOSING';
|
||||
case WebSocket.CLOSED: return 'DISCONNECTED';
|
||||
default: return 'UNKNOWN';
|
||||
}
|
||||
}
|
||||
}
|
||||
34
electron-app/src/preload.ts
Normal file
34
electron-app/src/preload.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
import { contextBridge, ipcRenderer } from 'electron';
|
||||
|
||||
// Expose protected methods that allow the renderer process to use
|
||||
// the ipcRenderer without exposing the entire object
|
||||
contextBridge.exposeInMainWorld('electron', {
|
||||
ipcRenderer: {
|
||||
invoke: (channel: string, ...args: any[]) => ipcRenderer.invoke(channel, ...args),
|
||||
on: (channel: string, func: (...args: any[]) => void) => {
|
||||
ipcRenderer.on(channel, (_event: any, ...args: any[]) => func(...args));
|
||||
},
|
||||
once: (channel: string, func: (...args: any[]) => void) => {
|
||||
ipcRenderer.once(channel, (_event: any, ...args: any[]) => func(...args));
|
||||
},
|
||||
removeAllListeners: (channel: string) => {
|
||||
ipcRenderer.removeAllListeners(channel);
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
// Type definitions for TypeScript
|
||||
export interface IElectronAPI {
|
||||
ipcRenderer: {
|
||||
invoke: (channel: string, ...args: any[]) => Promise<any>;
|
||||
on: (channel: string, func: (...args: any[]) => void) => void;
|
||||
once: (channel: string, func: (...args: any[]) => void) => void;
|
||||
removeAllListeners: (channel: string) => void;
|
||||
};
|
||||
}
|
||||
|
||||
declare global {
|
||||
interface Window {
|
||||
electron: IElectronAPI;
|
||||
}
|
||||
}
|
||||
303
electron-app/src/renderer/App.tsx
Normal file
303
electron-app/src/renderer/App.tsx
Normal file
@@ -0,0 +1,303 @@
|
||||
import { useState, useEffect, useMemo, useRef } from 'react';
|
||||
import { useWebSocket } from './hooks/useWebSocket';
|
||||
import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip, Legend, ResponsiveContainer } from 'recharts';
|
||||
|
||||
const TIME_RANGES = [
|
||||
{ label: '6 Hours', value: '6h' },
|
||||
{ label: '24 Hours', value: '24h' },
|
||||
{ label: '3 Days', value: '3d' },
|
||||
{ label: '7 Days', value: '7d' },
|
||||
{ label: '1 Month', value: '1mo' },
|
||||
{ label: 'YTD', value: 'ytd' },
|
||||
];
|
||||
|
||||
const COLORS = [
|
||||
'#8884d8', '#82ca9d', '#ffc658', '#ff7c7c', '#a28fd0', '#f5a623',
|
||||
'#50e3c2', '#ff6b9d', '#4a90e2', '#7ed321', '#d0021b', '#f8e71c'
|
||||
];
|
||||
|
||||
export function App() {
|
||||
const { status, latestPrice, historyData, requestHistory } = useWebSocket();
|
||||
const [selectedRange, setSelectedRange] = useState('7d');
|
||||
const [zoomLevel, setZoomLevel] = useState(1);
|
||||
const chartContainerRef = useRef<HTMLDivElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (status.connected) {
|
||||
requestHistory(selectedRange);
|
||||
}
|
||||
}, [status.connected, selectedRange, requestHistory]);
|
||||
|
||||
const chartData = useMemo(() => {
|
||||
if (!historyData?.prices) return [];
|
||||
|
||||
// Group by timestamp and seller
|
||||
const grouped = new Map<string, any>();
|
||||
|
||||
historyData.prices.forEach(price => {
|
||||
const time = new Date(price.timestamp).getTime();
|
||||
const key = time.toString();
|
||||
|
||||
if (!grouped.has(key)) {
|
||||
grouped.set(key, { timestamp: time, time: new Date(price.timestamp).toLocaleString() });
|
||||
}
|
||||
|
||||
const entry = grouped.get(key);
|
||||
const sellerKey = `${price.sellerName}`;
|
||||
entry[sellerKey] = price.pricePerMillion;
|
||||
});
|
||||
|
||||
return Array.from(grouped.values()).sort((a, b) => a.timestamp - b.timestamp);
|
||||
}, [historyData]);
|
||||
|
||||
const sellers = useMemo(() => {
|
||||
if (!historyData?.prices) return [];
|
||||
const uniqueSellers = new Set(historyData.prices.map(p => p.sellerName));
|
||||
return Array.from(uniqueSellers);
|
||||
}, [historyData]);
|
||||
|
||||
const { minY, maxY } = useMemo(() => {
|
||||
if (!historyData?.prices || historyData.prices.length === 0) {
|
||||
return { minY: 0, maxY: 1 };
|
||||
}
|
||||
|
||||
// Get all prices and sort them
|
||||
const allPrices = historyData.prices.map(p => p.pricePerMillion).sort((a, b) => a - b);
|
||||
|
||||
// Find the 90th percentile to focus on competitive prices
|
||||
const percentile90Index = Math.floor(allPrices.length * 0.90);
|
||||
const percentile90 = allPrices[percentile90Index];
|
||||
|
||||
// Get the minimum price
|
||||
const minPrice = allPrices[0];
|
||||
|
||||
// Base max Y on 90th percentile with padding
|
||||
const baseMaxY = Math.max(percentile90 * 1.1, minPrice * 2);
|
||||
|
||||
// Apply zoom level (higher zoom = smaller range)
|
||||
const zoomedMaxY = baseMaxY / zoomLevel;
|
||||
|
||||
console.log('Y-axis domain:', { minY: 0, maxY: zoomedMaxY, zoom: zoomLevel, base: baseMaxY });
|
||||
|
||||
return { minY: 0, maxY: zoomedMaxY };
|
||||
}, [historyData, zoomLevel]);
|
||||
|
||||
useEffect(() => {
|
||||
const chartContainer = chartContainerRef.current;
|
||||
if (!chartContainer) {
|
||||
console.log('Chart container ref not found');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log('Setting up wheel event listener on chart container');
|
||||
|
||||
const handleWheel = (e: WheelEvent) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
|
||||
console.log('Wheel event:', e.deltaY);
|
||||
|
||||
// Zoom in when scrolling up (negative deltaY), zoom out when scrolling down
|
||||
const zoomDelta = e.deltaY > 0 ? 0.9 : 1.1;
|
||||
|
||||
setZoomLevel(prev => {
|
||||
const newZoom = prev * zoomDelta;
|
||||
const clampedZoom = Math.max(0.1, Math.min(10, newZoom));
|
||||
console.log('Zoom level:', prev, '->', clampedZoom);
|
||||
return clampedZoom;
|
||||
});
|
||||
};
|
||||
|
||||
chartContainer.addEventListener('wheel', handleWheel, { passive: false });
|
||||
|
||||
return () => {
|
||||
console.log('Removing wheel event listener');
|
||||
chartContainer.removeEventListener('wheel', handleWheel);
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handleRangeChange = (range: string) => {
|
||||
setSelectedRange(range);
|
||||
};
|
||||
|
||||
const resetZoom = () => {
|
||||
setZoomLevel(1);
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{
|
||||
padding: '20px',
|
||||
fontFamily: 'system-ui',
|
||||
backgroundColor: '#0a0e27',
|
||||
color: '#fff',
|
||||
minHeight: '100%'
|
||||
}}>
|
||||
<h1 style={{ color: '#50e3c2', marginBottom: '10px' }}>rmtPocketWatcher</h1>
|
||||
<p style={{ color: '#888', fontSize: '14px', marginBottom: '20px' }}>Lambda Banking Conglomerate</p>
|
||||
|
||||
<div style={{ display: 'flex', gap: '20px', marginBottom: '20px' }}>
|
||||
<div style={{ flex: 1, backgroundColor: '#1a1f3a', padding: '15px', borderRadius: '8px' }}>
|
||||
<div style={{ fontSize: '12px', color: '#888', marginBottom: '5px' }}>CONNECTION</div>
|
||||
<div style={{ fontSize: '18px', fontWeight: 'bold', color: status.connected ? '#50e3c2' : '#ff6b9d' }}>
|
||||
{status.state}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{latestPrice && (
|
||||
<>
|
||||
<div style={{ flex: 1, backgroundColor: '#1a1f3a', padding: '15px', borderRadius: '8px' }}>
|
||||
<div style={{ fontSize: '12px', color: '#888', marginBottom: '5px' }}>LOWEST PRICE</div>
|
||||
<div style={{ fontSize: '24px', fontWeight: 'bold', color: '#50e3c2' }}>
|
||||
${latestPrice.lowestPrice.toFixed(4)}
|
||||
</div>
|
||||
<div style={{ fontSize: '12px', color: '#888' }}>per 1M AUEC</div>
|
||||
</div>
|
||||
|
||||
<div style={{ flex: 1, backgroundColor: '#1a1f3a', padding: '15px', borderRadius: '8px' }}>
|
||||
<div style={{ fontSize: '12px', color: '#888', marginBottom: '5px' }}>SELLER</div>
|
||||
<div style={{ fontSize: '18px', fontWeight: 'bold' }}>{latestPrice.sellerName}</div>
|
||||
<div style={{ fontSize: '12px', color: '#888' }}>{latestPrice.platform}</div>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div style={{ backgroundColor: '#1a1f3a', padding: '20px', borderRadius: '8px', marginBottom: '20px' }}>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: '20px' }}>
|
||||
<h2 style={{ margin: 0 }}>Price History</h2>
|
||||
<div style={{ display: 'flex', gap: '10px' }}>
|
||||
{TIME_RANGES.map(range => (
|
||||
<button
|
||||
key={range.value}
|
||||
onClick={() => handleRangeChange(range.value)}
|
||||
style={{
|
||||
padding: '8px 16px',
|
||||
backgroundColor: selectedRange === range.value ? '#50e3c2' : '#2a2f4a',
|
||||
color: selectedRange === range.value ? '#0a0e27' : '#fff',
|
||||
border: 'none',
|
||||
borderRadius: '4px',
|
||||
cursor: 'pointer',
|
||||
fontWeight: selectedRange === range.value ? 'bold' : 'normal',
|
||||
}}
|
||||
>
|
||||
{range.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
{historyData && historyData.prices && historyData.prices.length > 0 ? (
|
||||
chartData.length > 0 ? (
|
||||
<div>
|
||||
<div style={{ marginBottom: '10px', color: '#888', fontSize: '12px', display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<span>Scroll on chart to zoom. Extreme outliers may be clipped for better visibility.</span>
|
||||
<button
|
||||
onClick={resetZoom}
|
||||
style={{
|
||||
padding: '4px 12px',
|
||||
backgroundColor: '#2a2f4a',
|
||||
color: '#fff',
|
||||
border: '1px solid #50e3c2',
|
||||
borderRadius: '4px',
|
||||
cursor: 'pointer',
|
||||
fontSize: '11px'
|
||||
}}
|
||||
>
|
||||
Reset Zoom (×{zoomLevel.toFixed(1)})
|
||||
</button>
|
||||
</div>
|
||||
<div
|
||||
ref={chartContainerRef}
|
||||
style={{
|
||||
width: '100%',
|
||||
height: '500px',
|
||||
minHeight: '400px',
|
||||
cursor: 'ns-resize',
|
||||
border: '1px solid #2a2f4a'
|
||||
}}
|
||||
>
|
||||
<ResponsiveContainer width="100%" height="100%">
|
||||
<LineChart data={chartData} key={`chart-${zoomLevel}`}>
|
||||
<CartesianGrid strokeDasharray="3 3" stroke="#2a2f4a" />
|
||||
<XAxis
|
||||
dataKey="time"
|
||||
stroke="#888"
|
||||
tick={{ fill: '#888', fontSize: 10 }}
|
||||
angle={-45}
|
||||
textAnchor="end"
|
||||
height={80}
|
||||
/>
|
||||
<YAxis
|
||||
stroke="#888"
|
||||
tick={{ fill: '#888', fontSize: 12 }}
|
||||
label={{ value: 'USD per 1M AUEC', angle: -90, position: 'insideLeft', fill: '#888' }}
|
||||
domain={[0, yAxisDomain[1]]}
|
||||
allowDataOverflow={true}
|
||||
/>
|
||||
<Tooltip
|
||||
contentStyle={{ backgroundColor: '#1a1f3a', border: '1px solid #2a2f4a', borderRadius: '4px' }}
|
||||
labelStyle={{ color: '#fff' }}
|
||||
/>
|
||||
<Legend
|
||||
wrapperStyle={{ color: '#fff', maxHeight: '100px', overflowY: 'auto' }}
|
||||
iconType="line"
|
||||
/>
|
||||
{sellers.map((seller, index) => (
|
||||
<Line
|
||||
key={seller}
|
||||
type="monotone"
|
||||
dataKey={seller}
|
||||
stroke={COLORS[index % COLORS.length]}
|
||||
strokeWidth={2}
|
||||
dot={false}
|
||||
connectNulls
|
||||
/>
|
||||
))}
|
||||
</LineChart>
|
||||
</ResponsiveContainer>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ textAlign: 'center', padding: '40px', color: '#ff6b9d' }}>
|
||||
Data received but chart data is empty. Check console for details.
|
||||
</div>
|
||||
)
|
||||
) : (
|
||||
<div style={{ textAlign: 'center', padding: '40px', color: '#888' }}>
|
||||
{status.connected ? 'Loading historical data...' : 'Connect to view historical data'}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{latestPrice && (
|
||||
<div style={{ backgroundColor: '#1a1f3a', padding: '20px', borderRadius: '8px' }}>
|
||||
<h2>Current Listings ({latestPrice.allPrices.length})</h2>
|
||||
<div style={{ overflowX: 'auto', overflowY: 'auto', maxHeight: '400px' }}>
|
||||
<table style={{ width: '100%', borderCollapse: 'collapse' }}>
|
||||
<thead style={{ position: 'sticky', top: 0, backgroundColor: '#1a1f3a', zIndex: 1 }}>
|
||||
<tr style={{ borderBottom: '2px solid #2a2f4a' }}>
|
||||
<th style={{ textAlign: 'left', padding: '12px', color: '#888', fontWeight: 'normal' }}>Platform</th>
|
||||
<th style={{ textAlign: 'left', padding: '12px', color: '#888', fontWeight: 'normal' }}>Seller</th>
|
||||
<th style={{ textAlign: 'right', padding: '12px', color: '#888', fontWeight: 'normal' }}>Price/1M AUEC</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{latestPrice.allPrices.map((price, index) => (
|
||||
<tr key={price.id} style={{ borderBottom: '1px solid #2a2f4a' }}>
|
||||
<td style={{ padding: '12px' }}>{price.platform}</td>
|
||||
<td style={{ padding: '12px' }}>{price.sellerName}</td>
|
||||
<td style={{ textAlign: 'right', padding: '12px', color: '#50e3c2', fontWeight: 'bold' }}>
|
||||
${price.pricePerMillion.toFixed(4)}
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
96
electron-app/src/renderer/hooks/useWebSocket.ts
Normal file
96
electron-app/src/renderer/hooks/useWebSocket.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
import { useEffect, useState, useCallback } from 'react';
|
||||
import { PriceIndex, HistoricalData } from '../../shared/types';
|
||||
|
||||
interface WebSocketStatus {
|
||||
connected: boolean;
|
||||
state: string;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export function useWebSocket() {
|
||||
const [status, setStatus] = useState<WebSocketStatus>({
|
||||
connected: false,
|
||||
state: 'DISCONNECTED',
|
||||
});
|
||||
const [latestPrice, setLatestPrice] = useState<PriceIndex | null>(null);
|
||||
const [historyData, setHistoryData] = useState<HistoricalData | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
// Listen for WebSocket events from main process
|
||||
const handleConnected = () => {
|
||||
setStatus({ connected: true, state: 'CONNECTED' });
|
||||
};
|
||||
|
||||
const handleDisconnected = () => {
|
||||
setStatus({ connected: false, state: 'DISCONNECTED' });
|
||||
};
|
||||
|
||||
const handlePriceUpdate = (data: PriceIndex) => {
|
||||
setLatestPrice(data);
|
||||
};
|
||||
|
||||
const handleHistoryData = (data: HistoricalData) => {
|
||||
setHistoryData(data);
|
||||
};
|
||||
|
||||
const handleError = (error: string) => {
|
||||
setStatus((prev: WebSocketStatus) => ({ ...prev, error }));
|
||||
};
|
||||
|
||||
const handleMaxReconnect = () => {
|
||||
setStatus((prev: WebSocketStatus) => ({
|
||||
...prev,
|
||||
error: 'Maximum reconnection attempts reached. Please restart the application.'
|
||||
}));
|
||||
};
|
||||
|
||||
// Register IPC listeners
|
||||
window.electron.ipcRenderer.on('ws:connected', handleConnected);
|
||||
window.electron.ipcRenderer.on('ws:disconnected', handleDisconnected);
|
||||
window.electron.ipcRenderer.on('ws:priceUpdate', handlePriceUpdate);
|
||||
window.electron.ipcRenderer.on('ws:historyData', handleHistoryData);
|
||||
window.electron.ipcRenderer.on('ws:error', handleError);
|
||||
window.electron.ipcRenderer.on('ws:maxReconnectAttemptsReached', handleMaxReconnect);
|
||||
|
||||
// Get initial status
|
||||
window.electron.ipcRenderer.invoke('ws:getStatus').then((initialStatus) => {
|
||||
setStatus(initialStatus);
|
||||
});
|
||||
|
||||
// Cleanup
|
||||
return () => {
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:connected');
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:disconnected');
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:priceUpdate');
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:historyData');
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:error');
|
||||
window.electron.ipcRenderer.removeAllListeners('ws:maxReconnectAttemptsReached');
|
||||
};
|
||||
}, []);
|
||||
|
||||
const connect = useCallback(async () => {
|
||||
await window.electron.ipcRenderer.invoke('ws:connect');
|
||||
}, []);
|
||||
|
||||
const disconnect = useCallback(async () => {
|
||||
await window.electron.ipcRenderer.invoke('ws:disconnect');
|
||||
}, []);
|
||||
|
||||
const sendMessage = useCallback(async (message: any) => {
|
||||
await window.electron.ipcRenderer.invoke('ws:send', message);
|
||||
}, []);
|
||||
|
||||
const requestHistory = useCallback(async (range: string) => {
|
||||
await sendMessage({ type: 'get_history', data: { range } });
|
||||
}, [sendMessage]);
|
||||
|
||||
return {
|
||||
status,
|
||||
latestPrice,
|
||||
historyData,
|
||||
connect,
|
||||
disconnect,
|
||||
sendMessage,
|
||||
requestHistory,
|
||||
};
|
||||
}
|
||||
8
electron-app/src/renderer/index.tsx
Normal file
8
electron-app/src/renderer/index.tsx
Normal file
@@ -0,0 +1,8 @@
|
||||
import { createRoot } from 'react-dom/client';
|
||||
import { App } from './App';
|
||||
|
||||
const container = document.getElementById('root');
|
||||
if (!container) throw new Error('Root element not found');
|
||||
|
||||
const root = createRoot(container);
|
||||
root.render(<App />);
|
||||
3
electron-app/src/shared/types.js
Normal file
3
electron-app/src/shared/types.js
Normal file
@@ -0,0 +1,3 @@
|
||||
"use strict";
|
||||
// Shared types between main and renderer processes
|
||||
Object.defineProperty(exports, "__esModule", { value: true });
|
||||
43
electron-app/src/shared/types.ts
Normal file
43
electron-app/src/shared/types.ts
Normal file
@@ -0,0 +1,43 @@
|
||||
// Shared types between main and renderer processes
|
||||
|
||||
export interface VendorPrice {
|
||||
id: string;
|
||||
platform: 'eldorado' | 'playerauctions';
|
||||
sellerName: string;
|
||||
pricePerMillion: number;
|
||||
timestamp: Date;
|
||||
url?: string;
|
||||
}
|
||||
|
||||
export interface PriceIndex {
|
||||
timestamp: Date;
|
||||
lowestPrice: number;
|
||||
platform: string;
|
||||
sellerName: string;
|
||||
allPrices: VendorPrice[];
|
||||
}
|
||||
|
||||
export interface PriceAlert {
|
||||
id: string;
|
||||
type: 'below' | 'above';
|
||||
threshold: number;
|
||||
enabled: boolean;
|
||||
}
|
||||
|
||||
export interface WebSocketMessage {
|
||||
type: 'price_update' | 'connection_status' | 'error' | 'history_data' | 'get_history' | 'subscribe';
|
||||
data: any;
|
||||
}
|
||||
|
||||
export interface HistoricalData {
|
||||
range: string;
|
||||
from: Date;
|
||||
to: Date;
|
||||
prices: VendorPrice[];
|
||||
}
|
||||
|
||||
export interface ConnectionStatus {
|
||||
connected: boolean;
|
||||
lastUpdate?: Date;
|
||||
error?: string;
|
||||
}
|
||||
20
electron-app/tsconfig.json
Normal file
20
electron-app/tsconfig.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"module": "commonjs",
|
||||
"lib": ["ES2020", "DOM"],
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"moduleResolution": "node",
|
||||
"jsx": "react-jsx",
|
||||
"types": ["node", "electron"],
|
||||
"allowSyntheticDefaultImports": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
8
electron-app/tsconfig.main.json
Normal file
8
electron-app/tsconfig.main.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"extends": "./tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"outDir": "./dist/main",
|
||||
"rootDir": "./src"
|
||||
},
|
||||
"include": ["src/main/**/*", "src/shared/**/*"]
|
||||
}
|
||||
8
electron-app/tsconfig.preload.json
Normal file
8
electron-app/tsconfig.preload.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"extends": "./tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"outDir": "./dist/main",
|
||||
"rootDir": "./src"
|
||||
},
|
||||
"include": ["src/preload.ts", "src/shared/**/*"]
|
||||
}
|
||||
14
electron-app/vite.config.mjs
Normal file
14
electron-app/vite.config.mjs
Normal file
@@ -0,0 +1,14 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import react from '@vitejs/plugin-react';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
root: '.',
|
||||
build: {
|
||||
outDir: 'dist/renderer',
|
||||
emptyOutDir: true,
|
||||
},
|
||||
server: {
|
||||
port: 5173,
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user