update readme

README.md

@@ -1,6 +1,16 @@
 # Ingress Intel Report
 
-A command-line tool for fetching and analyzing game events from the Ingress Intel API.
+A comprehensive system for fetching, analyzing, and storing game events from the Ingress Intel API.
+
+## Overview
+
+The Ingress Intel Report is a multi-component system consisting of:
+
+- **CLI Tool** ([`main.py`](main.py:1)) - Command-line interface for querying events
+- **Web API** ([`app.py`](app.py:1)) - REST API for programmatic access
+- **Scheduler** ([`schedule.py`](schedule.py:1)) - Automated data collection every minute
+- **MongoDB** - Persistent storage for historical data
+- **Docker** - Containerized deployment with docker-compose
 
 ## Features
 
@@ -9,30 +19,152 @@ A command-line tool for fetching and analyzing game events from the Ingress Inte
 - **Player Tracking**: Filter events by specific players
 - **Timestamp Filtering**: Filter events by time range
 - **Coordinate Extraction**: Extract portal locations from events
+- **REST API**: JSON-based API for programmatic access
+- **MongoDB Persistence**: Store historical event data
+- **Scheduled Collection**: Automated data collection every minute
+- **Docker Deployment**: Containerized deployment with health checks
+- **Timezone Support**: Configurable timezone (default: Europe/Rome)
+
+## Architecture
+
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   CLI Tool  │     │  Web API    │     │  Scheduler  │
+│  (main.py)  │     │  (app.py)   │     │(schedule.py)│
+└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
+       │                   │                   │
+       └───────────────────┼───────────────────┘
+                           │
+                    ┌──────▼──────┐
+                    │ Ingress API │
+                    │ (ingress.py)│
+                    └──────┬──────┘
+                           │
+                    ┌──────▼──────┐
+                    │  MongoDB    │
+                    │  Database   │
+                    └─────────────┘
+```
 
 ## Installation
 
+### Option 1: Docker (Recommended)
+
+The easiest way to run the complete system is using Docker Compose:
+
 ```bash
-# Using uv (recommended)
+# Clone the repository
+git clone <repository-url>
+cd Ingress
+
+# Create environment file
+cp .env.example .env
+# Edit .env with your configuration
+
+# Start all services
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# Stop services
+docker-compose down
+```
+
+### Option 2: Local Development
+
+For local development without Docker:
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Or using uv (recommended)
 uv sync
 
-# Using pip
+# Set up MongoDB
+# Install MongoDB locally or use a cloud instance
+# Update MONGO_URI in .env file
+
+# Start the web API
+python app.py
+
+# In another terminal, start the scheduler
+python schedule.py
+```
+
+### Option 3: CLI Only
+
+If you only need the CLI tool:
+
+```bash
+# Install dependencies
 pip install -r requirements.txt
+
+# Set environment variables
+export INGRESS_COOKIE="your_cookie_here"
+export V="your_api_version"
+
+# Run the CLI
+python main.py --help
 ```
 
 ## Configuration
 
-Set your Ingress cookie as an environment variable:
+Create a `.env` file in the project root with the following variables:
+
+| Variable          | Required | Default     | Description                                                      |
+| ----------------- | -------- | ----------- | ---------------------------------------------------------------- |
+| `INGRESS_COOKIE`  | Yes      | -           | Ingress authentication cookie (includes csrftoken and sessionid) |
+| `V`               | Yes      | -           | Ingress API version string                                       |
+| `PORT`            | No       | 5000        | Web API server port                                              |
+| `MONGO_URI`       | Yes*     | -           | MongoDB connection string (required for scheduler)               |
+| `DB_NAME`         | Yes*     | -           | MongoDB database name (required for scheduler)                   |
+| `COLLECTION_NAME` | Yes*     | -           | MongoDB collection name (required for scheduler)                 |
+| `ENDPOINT_URL`    | Yes*     | -           | API endpoint for scheduler (e.g., http://web:7000/plexts)        |
+| `TZ`              | No       | Europe/Rome | Timezone for timestamps                                          |
+| `MIN_LAT`         | No       | 45470259    | Default minimum latitude (E6 format)                             |
+| `MIN_LNG`         | No       | 12244155    | Default minimum longitude (E6 format)                            |
+| `MAX_LAT`         | No       | 45480370    | Default maximum latitude (E6 format)                             |
+| `MAX_LNG`         | No       | 12298207    | Default maximum longitude (E6 format)                            |
+
+*Required for scheduler and web API with MongoDB integration
+
+### Example .env file
 
 ```bash
-export INGRESS_COOKIE="your_cookie_here"
+# Ingress API Configuration
-```
+INGRESS_COOKIE="csrftoken=your_token; sessionid=your_session"
+V="412c0ac7e784d6df783fc507bca30e23b3c58c55"
 
-Or hardcode it in [`main.py`](main.py:82) (not recommended).
+# Web API Configuration
+PORT=7000
+
+# MongoDB Configuration
+MONGO_URI="mongodb://ingress_root:password@localhost:27017/"
+DB_NAME=ingress
+COLLECTION_NAME=ingress
+
+# Scheduler Configuration
+ENDPOINT_URL="http://localhost:7000/plexts"
+
+# Timezone
+TZ=Europe/Rome
+
+# Geographic Bounds (Venice, Italy)
+MIN_LAT=45470259
+MIN_LNG=12244155
+MAX_LAT=45480370
+MAX_LNG=12298207
+```
 
 ## Usage
 
-### Basic Usage
+### CLI Tool
+
+The CLI tool provides a command-line interface for querying Ingress events.
+
+#### Basic Usage
 
 ```bash
 # Get all events in default area (Venice, Italy)
@@ -42,11 +174,11 @@ python main.py
 python main.py --help
 ```
 
-### Command Line Options
+#### Command Line Options
 
 | Option            | Type             | Default  | Description                                                     |
-| ----------------- | ---------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ----------------- | ---------------- | -------- | --------------------------------------------------------------- |
-| `--event-types`   | Multiple strings | None     | Filter by event types. Accepts one or more values from: `RESONATOR_DEPLOYED`, `RESONATOR_DESTROYED`, `PORTAL_CAPTURED`, `PORTAL_NEUTRALIZED`, `PORTAL_UNDER_ATTACK`, `LINK_CREATED`, `LINK_DESTROYED`, `CONTROL_FIELD_CREATED`, `UNKNOWN` |
+| `--event-types`   | Multiple strings | None     | Filter by event types (comma-separated or space-separated)      |
 | `--player-name`   | String           | None     | Filter events by a specific player name                         |
 | `--min-lat`       | Integer          | 45470259 | Minimum latitude (in microdegrees, E6 format)                   |
 | `--min-lng`       | Integer          | 12244155 | Minimum longitude (in microdegrees, E6 format)                  |
@@ -55,7 +187,7 @@ python main.py --help
 | `--min-timestamp` | Timestamp        | -1       | Minimum timestamp (milliseconds since epoch or ISO 8601 format) |
 | `--max-timestamp` | Timestamp        | -1       | Maximum timestamp (milliseconds since epoch or ISO 8601 format) |
 
-### Timestamp Filtering
+#### Timestamp Filtering
 
 The `--min-timestamp` and `--max-timestamp` options support two formats:
 
@@ -85,20 +217,20 @@ python main.py --min-timestamp "2026-01-12T10:00:00Z"
 python main.py --max-timestamp "2026-01-12T12:00:00Z"
 ```
 
-### Event Type Filtering
+#### Event Type Filtering
 
 ```bash
 # Filter by single event type
 python main.py --event-types PORTAL_CAPTURED
 
-# Filter by multiple event types
+# Filter by multiple event types (space-separated)
 python main.py --event-types PORTAL_CAPTURED LINK_CREATED CONTROL_FIELD_CREATED
 
 # Filter by resonator events
 python main.py --event-types RESONATOR_DEPLOYED RESONATOR_DESTROYED
 ```
 
-### Player Filtering
+#### Player Filtering
 
 ```bash
 # Filter by player name
@@ -108,7 +240,7 @@ python main.py --player-name Albicocca
 python main.py --player-name Albicocca --event-types PORTAL_CAPTURED
 ```
 
-### Geographic Filtering
+#### Geographic Filtering
 
 Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,000,000.
 
@@ -117,7 +249,7 @@ Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,0
 python main.py --min-lat 45470000 --max-lat 45480000 --min-lng 12240000 --max-lng 12300000
 ```
 
-### Combining Filters
+#### Combining Filters
 
 ```bash
 # Complex query with multiple filters
@@ -130,6 +262,259 @@ python main.py \
   --max-lat 45480000
 ```
 
+### Web API
+
+The web API provides a REST interface for programmatic access to Ingress events.
+
+#### Starting the Web API
+
+```bash
+# Development mode
+python app.py
+
+# Production mode with Gunicorn
+gunicorn -w 4 -b 0.0.0.0:7000 app:app
+
+# Or using the entrypoint script
+./entrypoint-web.sh
+```
+
+#### API Endpoints
+
+##### GET /
+
+Returns API information and available endpoints.
+
+**Response:**
+```json
+{
+  "name": "Ingress Intel API",
+  "version": "1.0.0",
+  "endpoints": {
+    "/plexts": {
+      "method": "GET",
+      "description": "Get plexts from the Ingress API",
+      "parameters": {
+        "event_types": "List of event types to filter by (comma-separated)",
+        "player_name": "Player name to filter by",
+        "min_lat": "Minimum latitude (default: 45470259)",
+        "min_lng": "Minimum longitude (default: 12244155)",
+        "max_lat": "Maximum latitude (default: 45480370)",
+        "max_lng": "Maximum longitude (default: 12298207)",
+        "min_timestamp": "Minimum timestamp (milliseconds or ISO 8601 format, default: -1)",
+        "max_timestamp": "Maximum timestamp (milliseconds or ISO 8601 format, default: -1)"
+      },
+      "event_types": [
+        "RESONATOR_DEPLOYED",
+        "RESONATOR_DESTROYED",
+        "PORTAL_CAPTURED",
+        "PORTAL_NEUTRALIZED",
+        "PORTAL_UNDER_ATTACK",
+        "LINK_CREATED",
+        "LINK_DESTROYED",
+        "CONTROL_FIELD_CREATED",
+        "UNKNOWN"
+      ]
+    }
+  }
+}
+```
+
+##### GET /plexts
+
+Get plexts from the Ingress API with optional filters.
+
+**Query Parameters:**
+
+| Parameter       | Type    | Default  | Description                                         |
+| --------------- | ------- | -------- | --------------------------------------------------- |
+| `event_types`   | String  | None     | Comma-separated list of event types to filter by    |
+| `player_name`   | String  | None     | Player name to filter by                            |
+| `min_lat`       | Integer | From env | Minimum latitude (E6 format)                        |
+| `min_lng`       | Integer | From env | Minimum longitude (E6 format)                       |
+| `max_lat`       | Integer | From env | Maximum latitude (E6 format)                        |
+| `max_lng`       | Integer | From env | Maximum longitude (E6 format)                       |
+| `min_timestamp` | String  | -1       | Minimum timestamp (milliseconds or ISO 8601 format) |
+| `max_timestamp` | String  | -1       | Maximum timestamp (milliseconds or ISO 8601 format) |
+
+**Response:**
+```json
+{
+  "count": 2,
+  "plexts": [
+    {
+      "id": "abc123",
+      "timestamp": 1736659207000,
+      "timestamp_formatted": "2026-01-12 11:00:07",
+      "text": "Albicocca captured L' Arboreto",
+      "team": "RESISTANCE",
+      "plext_type": "PLAYER_GENERATED",
+      "categories": 0,
+      "event_type": "PORTAL_CAPTURED",
+      "player_name": "Albicocca",
+      "portal_name": "L' Arboreto",
+      "coordinates": {
+        "lat": 45471652,
+        "lng": 12274703
+      },
+      "markup": [
+        {
+          "type": "PLAYER",
+          "plain": "Albicocca",
+          "team": "RESISTANCE",
+          "name": "Albicocca",
+          "address": "",
+          "latE6": 0,
+          "lngE6": 0
+        },
+        {
+          "type": "PORTAL",
+          "plain": "L' Arboreto",
+          "team": "RESISTANCE",
+          "name": "L' Arboreto",
+          "address": "Via Arboreto, Venice, Italy",
+          "latE6": 45471652,
+          "lngE6": 12274703
+        }
+      ]
+    }
+  ]
+}
+```
+
+**Examples:**
+
+```bash
+# Get all plexts
+curl "http://localhost:7000/plexts"
+
+# Filter by event type
+curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED,LINK_CREATED"
+
+# Filter by player
+curl "http://localhost:7000/plexts?player_name=Albicocca"
+
+# Filter by time range
+curl "http://localhost:7000/plexts?min_timestamp=2026-01-12T10:00:00Z&max_timestamp=2026-01-12T12:00:00Z"
+
+# Combine filters
+curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED&player_name=Albicocca&min_timestamp=2026-01-12T10:00:00Z"
+```
+
+### Scheduler
+
+The scheduler automatically collects Ingress events every minute and stores them in MongoDB.
+
+#### Starting the Scheduler
+
+```bash
+# Start the scheduler
+python schedule.py
+
+# Or using the entrypoint script
+./entrypoint-schedule.sh
+```
+
+#### How It Works
+
+1. The scheduler runs every minute using APScheduler
+2. It calculates a time range for the previous minute
+3. Fetches plexts from the web API endpoint
+4. Stores the results in MongoDB
+5. Logs the number of plexts collected
+
+#### Configuration
+
+The scheduler requires the following environment variables:
+
+- `ENDPOINT_URL` - The web API endpoint to fetch plexts from
+- `MONGO_URI` - MongoDB connection string
+- `DB_NAME` - Database name
+- `COLLECTION_NAME` - Collection name
+- `TZ` - Timezone for time calculations
+
+#### Example Output
+
+```
+[2026-01-12 11:00:00 CET] Starting job...
+Time range: 2026-01-12T10:59:00+01:00 to 2026-01-12T11:00:00+01:00
+Fetched 5 plexts
+Inserted 5 plexts to MongoDB
+Job completed successfully
+```
+
+### Docker Deployment
+
+#### Using Docker Compose
+
+The project includes a [`docker-compose.yml`](docker-compose.yml:1) file for easy deployment.
+
+**Services:**
+
+1. **MongoDB** - Database service
+   - Image: `mongo:7.0`
+   - Port: `27017`
+   - Health checks enabled
+   - Persistent volumes for data
+
+2. **Web** - Flask web API
+   - Image: `mrosati84/ingress-web:1.0`
+   - Port: `7000`
+   - Depends on MongoDB
+   - Health checks enabled
+
+3. **Schedule** - Scheduler service
+   - Image: `mrosati84/ingress-schedule:1.0`
+   - Depends on MongoDB and Web
+   - No exposed ports
+
+**Commands:**
+
+```bash
+# Start all services
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# View logs for specific service
+docker-compose logs -f web
+
+# Stop all services
+docker-compose down
+
+# Stop and remove volumes
+docker-compose down -v
+
+# Restart a service
+docker-compose restart web
+
+# Scale services
+docker-compose up -d --scale web=3
+```
+
+#### Building Custom Images
+
+If you want to build your own Docker images:
+
+```bash
+# Build web image
+docker build -f Dockerfile-web -t ingress-web .
+
+# Build schedule image
+docker build -f Dockerfile-schedule -t ingress-schedule .
+
+# Update docker-compose.yml to use your images
+```
+
+#### Health Checks
+
+All services include health checks:
+
+- **MongoDB**: Database ping test
+- **Web**: HTTP GET request to root endpoint
+- **Schedule**: No health check (depends on other services)
+
 ## Event Types
 
 | Event Type              | Description                        |
@@ -146,6 +531,8 @@ python main.py \
 
 ## Output Format
 
+### CLI Output
+
 Events are printed in the following format:
 
 ```
@@ -153,22 +540,190 @@ Events are printed in the following format:
 [2026-01-12 10:28:26] [LINK_CREATED] Albicocca linked from L' Arboreto to Parco San Giuliano - Coords: 45471652, 12274703
 ```
 
+### API Output
+
+The API returns JSON with the following structure:
+
+```json
+{
+  "count": 2,
+  "plexts": [
+    {
+      "id": "string",
+      "timestamp": 1736659207000,
+      "timestamp_formatted": "2026-01-12 11:00:07",
+      "text": "string",
+      "team": "RESISTANCE",
+      "plext_type": "PLAYER_GENERATED",
+      "categories": 0,
+      "event_type": "PORTAL_CAPTURED",
+      "player_name": "string",
+      "portal_name": "string",
+      "coordinates": {
+        "lat": 45471652,
+        "lng": 12274703
+      },
+      "markup": [...]
+    }
+  ]
+}
+```
+
 ## Project Structure
 
 ```
 ingress/
 ├── main.py                    # CLI entry point
-├── ingress.py        # API client
+├── app.py                     # Flask web API
-├── models.py         # Data models
+├── ingress.py                 # Ingress API client
-├── pyproject.toml     # Project configuration
+├── models.py                  # Data models (Plext, EventType, Markup)
+├── schedule.py                # Scheduled data collector
+├── pyproject.toml             # Project configuration (uv)
+├── requirements.txt           # Python dependencies
+├── docker-compose.yml         # Docker orchestration
+├── Dockerfile-web             # Web service Docker image
+├── Dockerfile-schedule        # Scheduler Docker image
+├── entrypoint-web.sh          # Web service entrypoint script
+├── entrypoint-schedule.sh     # Scheduler entrypoint script
+├── json_doc.md                # API documentation
+├── .python-version            # Python version specification
+├── .gitignore                 # Git ignore rules
+├── .env.example               # Environment variables template
 └── README.md                  # This file
 ```
 
 ## Dependencies
 
-- Python 3.13+
+### Core Dependencies
-- requests 2.31.0+
+
+- **Python 3.11+** (Docker images use Python 3.11)
+- **requests>=2.31.0** - HTTP client for API requests
+- **flask>=3.1.2** - Web framework for REST API
+- **gunicorn>=23.0.0** - Production WSGI server
+- **python-dotenv>=1.2.1** - Environment variable management
+- **pymongo>=4.10.0** - MongoDB driver
+- **apscheduler>=3.10.0** - Task scheduling
+- **ipython>=9.9.0** - Interactive Python shell (development)
+
+### Development Dependencies
+
+- **uv** - Fast Python package installer (optional)
+
+## Development
+
+### Local Development Setup
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Or using uv
+uv sync
+
+# Set up environment variables
+cp .env.example .env
+# Edit .env with your configuration
+
+# Start MongoDB (if not using Docker)
+# Install MongoDB locally or use a cloud instance
+
+# Start the web API
+python app.py
+
+# In another terminal, start the scheduler
+python schedule.py
+
+# Test the CLI
+python main.py --help
+```
+
+### Running Tests
+
+```bash
+# Run CLI tests
+python main.py --event-types PORTAL_CAPTURED --player-name test
+
+# Test API endpoints
+curl http://localhost:7000/
+curl http://localhost:7000/plexts
+```
+
+### Debugging
+
+The application includes comprehensive logging:
+
+- **CLI**: Logs to stdout
+- **Web API**: Logs to stdout (Gunicorn captures and forwards)
+- **Scheduler**: Logs to stdout
+- **Ingress API Client**: Detailed DEBUG level logging
+
+To enable debug logging:
+
+```bash
+# Set log level in environment
+export LOG_LEVEL=DEBUG
+
+# Or modify the logging configuration in the source files
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Issue: "No CSRF token found in cookie!"**
+- Solution: Ensure your `INGRESS_COOKIE` includes both `csrftoken` and `sessionid`
+
+**Issue: "MongoDB connection failed"**
+- Solution: Check that MongoDB is running and `MONGO_URI` is correct
+
+**Issue: "Scheduler not collecting data"**
+- Solution: Verify `ENDPOINT_URL` is accessible and web API is running
+
+**Issue: "Docker containers not starting"**
+- Solution: Check Docker logs: `docker-compose logs`
+
+**Issue: "Health check failing"**
+- Solution: Ensure all dependencies are running and ports are accessible
+
+### Debug Mode
+
+Enable debug mode for more detailed logging:
+
+```bash
+# For web API
+export FLASK_DEBUG=1
+python app.py
+
+# For scheduler
+export APSCHEDULER_DEBUG=1
+python schedule.py
+```
+
+### Logs
+
+View logs for different components:
+
+```bash
+# Docker logs
+docker-compose logs -f
+
+# Specific service logs
+docker-compose logs -f web
+docker-compose logs -f schedule
+docker-compose logs -f mongodb
+
+# Application logs (if running locally)
+# Logs are printed to stdout
+```
 
 ## License
 
 See project license file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## Support
+
+For issues and questions, please open an issue on the project repository.