From 49c5225027703f580ab4a4fa7a18b5072a3b3d03 Mon Sep 17 00:00:00 2001 From: Matteo Rosati Date: Thu, 15 Jan 2026 00:10:43 +0100 Subject: [PATCH] update readme --- README.md | 619 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 587 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 69e4eed..b68e245 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,16 @@ # Ingress Intel Report -A command-line tool for fetching and analyzing game events from the Ingress Intel API. +A comprehensive system for fetching, analyzing, and storing game events from the Ingress Intel API. + +## Overview + +The Ingress Intel Report is a multi-component system consisting of: + +- **CLI Tool** ([`main.py`](main.py:1)) - Command-line interface for querying events +- **Web API** ([`app.py`](app.py:1)) - REST API for programmatic access +- **Scheduler** ([`schedule.py`](schedule.py:1)) - Automated data collection every minute +- **MongoDB** - Persistent storage for historical data +- **Docker** - Containerized deployment with docker-compose ## Features @@ -9,30 +19,152 @@ A command-line tool for fetching and analyzing game events from the Ingress Inte - **Player Tracking**: Filter events by specific players - **Timestamp Filtering**: Filter events by time range - **Coordinate Extraction**: Extract portal locations from events +- **REST API**: JSON-based API for programmatic access +- **MongoDB Persistence**: Store historical event data +- **Scheduled Collection**: Automated data collection every minute +- **Docker Deployment**: Containerized deployment with health checks +- **Timezone Support**: Configurable timezone (default: Europe/Rome) + +## Architecture + +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ CLI Tool │ │ Web API │ │ Scheduler │ +│ (main.py) │ │ (app.py) │ │(schedule.py)│ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └───────────────────┼───────────────────┘ + │ + ┌──────▼──────┐ + │ Ingress API │ + │ (ingress.py)│ + └──────┬──────┘ + │ + ┌──────▼──────┐ + │ MongoDB │ + │ Database │ + └─────────────┘ +``` ## Installation +### Option 1: Docker (Recommended) + +The easiest way to run the complete system is using Docker Compose: + ```bash -# Using uv (recommended) +# Clone the repository +git clone +cd Ingress + +# Create environment file +cp .env.example .env +# Edit .env with your configuration + +# Start all services +docker-compose up -d + +# View logs +docker-compose logs -f + +# Stop services +docker-compose down +``` + +### Option 2: Local Development + +For local development without Docker: + +```bash +# Install dependencies +pip install -r requirements.txt + +# Or using uv (recommended) uv sync -# Using pip +# Set up MongoDB +# Install MongoDB locally or use a cloud instance +# Update MONGO_URI in .env file + +# Start the web API +python app.py + +# In another terminal, start the scheduler +python schedule.py +``` + +### Option 3: CLI Only + +If you only need the CLI tool: + +```bash +# Install dependencies pip install -r requirements.txt + +# Set environment variables +export INGRESS_COOKIE="your_cookie_here" +export V="your_api_version" + +# Run the CLI +python main.py --help ``` ## Configuration -Set your Ingress cookie as an environment variable: +Create a `.env` file in the project root with the following variables: + +| Variable | Required | Default | Description | +| ----------------- | -------- | ----------- | ---------------------------------------------------------------- | +| `INGRESS_COOKIE` | Yes | - | Ingress authentication cookie (includes csrftoken and sessionid) | +| `V` | Yes | - | Ingress API version string | +| `PORT` | No | 5000 | Web API server port | +| `MONGO_URI` | Yes* | - | MongoDB connection string (required for scheduler) | +| `DB_NAME` | Yes* | - | MongoDB database name (required for scheduler) | +| `COLLECTION_NAME` | Yes* | - | MongoDB collection name (required for scheduler) | +| `ENDPOINT_URL` | Yes* | - | API endpoint for scheduler (e.g., http://web:7000/plexts) | +| `TZ` | No | Europe/Rome | Timezone for timestamps | +| `MIN_LAT` | No | 45470259 | Default minimum latitude (E6 format) | +| `MIN_LNG` | No | 12244155 | Default minimum longitude (E6 format) | +| `MAX_LAT` | No | 45480370 | Default maximum latitude (E6 format) | +| `MAX_LNG` | No | 12298207 | Default maximum longitude (E6 format) | + +*Required for scheduler and web API with MongoDB integration + +### Example .env file ```bash -export INGRESS_COOKIE="your_cookie_here" -``` +# Ingress API Configuration +INGRESS_COOKIE="csrftoken=your_token; sessionid=your_session" +V="412c0ac7e784d6df783fc507bca30e23b3c58c55" -Or hardcode it in [`main.py`](main.py:82) (not recommended). +# Web API Configuration +PORT=7000 + +# MongoDB Configuration +MONGO_URI="mongodb://ingress_root:password@localhost:27017/" +DB_NAME=ingress +COLLECTION_NAME=ingress + +# Scheduler Configuration +ENDPOINT_URL="http://localhost:7000/plexts" + +# Timezone +TZ=Europe/Rome + +# Geographic Bounds (Venice, Italy) +MIN_LAT=45470259 +MIN_LNG=12244155 +MAX_LAT=45480370 +MAX_LNG=12298207 +``` ## Usage -### Basic Usage +### CLI Tool + +The CLI tool provides a command-line interface for querying Ingress events. + +#### Basic Usage ```bash # Get all events in default area (Venice, Italy) @@ -42,20 +174,20 @@ python main.py python main.py --help ``` -### Command Line Options +#### Command Line Options -| Option | Type | Default | Description | -| ----------------- | ---------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--event-types` | Multiple strings | None | Filter by event types. Accepts one or more values from: `RESONATOR_DEPLOYED`, `RESONATOR_DESTROYED`, `PORTAL_CAPTURED`, `PORTAL_NEUTRALIZED`, `PORTAL_UNDER_ATTACK`, `LINK_CREATED`, `LINK_DESTROYED`, `CONTROL_FIELD_CREATED`, `UNKNOWN` | -| `--player-name` | String | None | Filter events by a specific player name | -| `--min-lat` | Integer | 45470259 | Minimum latitude (in microdegrees, E6 format) | -| `--min-lng` | Integer | 12244155 | Minimum longitude (in microdegrees, E6 format) | -| `--max-lat` | Integer | 45480370 | Maximum latitude (in microdegrees, E6 format) | -| `--max-lng` | Integer | 12298207 | Maximum longitude (in microdegrees, E6 format) | -| `--min-timestamp` | Timestamp | -1 | Minimum timestamp (milliseconds since epoch or ISO 8601 format) | -| `--max-timestamp` | Timestamp | -1 | Maximum timestamp (milliseconds since epoch or ISO 8601 format) | +| Option | Type | Default | Description | +| ----------------- | ---------------- | -------- | --------------------------------------------------------------- | +| `--event-types` | Multiple strings | None | Filter by event types (comma-separated or space-separated) | +| `--player-name` | String | None | Filter events by a specific player name | +| `--min-lat` | Integer | 45470259 | Minimum latitude (in microdegrees, E6 format) | +| `--min-lng` | Integer | 12244155 | Minimum longitude (in microdegrees, E6 format) | +| `--max-lat` | Integer | 45480370 | Maximum latitude (in microdegrees, E6 format) | +| `--max-lng` | Integer | 12298207 | Maximum longitude (in microdegrees, E6 format) | +| `--min-timestamp` | Timestamp | -1 | Minimum timestamp (milliseconds since epoch or ISO 8601 format) | +| `--max-timestamp` | Timestamp | -1 | Maximum timestamp (milliseconds since epoch or ISO 8601 format) | -### Timestamp Filtering +#### Timestamp Filtering The `--min-timestamp` and `--max-timestamp` options support two formats: @@ -85,20 +217,20 @@ python main.py --min-timestamp "2026-01-12T10:00:00Z" python main.py --max-timestamp "2026-01-12T12:00:00Z" ``` -### Event Type Filtering +#### Event Type Filtering ```bash # Filter by single event type python main.py --event-types PORTAL_CAPTURED -# Filter by multiple event types +# Filter by multiple event types (space-separated) python main.py --event-types PORTAL_CAPTURED LINK_CREATED CONTROL_FIELD_CREATED # Filter by resonator events python main.py --event-types RESONATOR_DEPLOYED RESONATOR_DESTROYED ``` -### Player Filtering +#### Player Filtering ```bash # Filter by player name @@ -108,7 +240,7 @@ python main.py --player-name Albicocca python main.py --player-name Albicocca --event-types PORTAL_CAPTURED ``` -### Geographic Filtering +#### Geographic Filtering Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,000,000. @@ -117,7 +249,7 @@ Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,0 python main.py --min-lat 45470000 --max-lat 45480000 --min-lng 12240000 --max-lng 12300000 ``` -### Combining Filters +#### Combining Filters ```bash # Complex query with multiple filters @@ -130,6 +262,259 @@ python main.py \ --max-lat 45480000 ``` +### Web API + +The web API provides a REST interface for programmatic access to Ingress events. + +#### Starting the Web API + +```bash +# Development mode +python app.py + +# Production mode with Gunicorn +gunicorn -w 4 -b 0.0.0.0:7000 app:app + +# Or using the entrypoint script +./entrypoint-web.sh +``` + +#### API Endpoints + +##### GET / + +Returns API information and available endpoints. + +**Response:** +```json +{ + "name": "Ingress Intel API", + "version": "1.0.0", + "endpoints": { + "/plexts": { + "method": "GET", + "description": "Get plexts from the Ingress API", + "parameters": { + "event_types": "List of event types to filter by (comma-separated)", + "player_name": "Player name to filter by", + "min_lat": "Minimum latitude (default: 45470259)", + "min_lng": "Minimum longitude (default: 12244155)", + "max_lat": "Maximum latitude (default: 45480370)", + "max_lng": "Maximum longitude (default: 12298207)", + "min_timestamp": "Minimum timestamp (milliseconds or ISO 8601 format, default: -1)", + "max_timestamp": "Maximum timestamp (milliseconds or ISO 8601 format, default: -1)" + }, + "event_types": [ + "RESONATOR_DEPLOYED", + "RESONATOR_DESTROYED", + "PORTAL_CAPTURED", + "PORTAL_NEUTRALIZED", + "PORTAL_UNDER_ATTACK", + "LINK_CREATED", + "LINK_DESTROYED", + "CONTROL_FIELD_CREATED", + "UNKNOWN" + ] + } + } +} +``` + +##### GET /plexts + +Get plexts from the Ingress API with optional filters. + +**Query Parameters:** + +| Parameter | Type | Default | Description | +| --------------- | ------- | -------- | --------------------------------------------------- | +| `event_types` | String | None | Comma-separated list of event types to filter by | +| `player_name` | String | None | Player name to filter by | +| `min_lat` | Integer | From env | Minimum latitude (E6 format) | +| `min_lng` | Integer | From env | Minimum longitude (E6 format) | +| `max_lat` | Integer | From env | Maximum latitude (E6 format) | +| `max_lng` | Integer | From env | Maximum longitude (E6 format) | +| `min_timestamp` | String | -1 | Minimum timestamp (milliseconds or ISO 8601 format) | +| `max_timestamp` | String | -1 | Maximum timestamp (milliseconds or ISO 8601 format) | + +**Response:** +```json +{ + "count": 2, + "plexts": [ + { + "id": "abc123", + "timestamp": 1736659207000, + "timestamp_formatted": "2026-01-12 11:00:07", + "text": "Albicocca captured L' Arboreto", + "team": "RESISTANCE", + "plext_type": "PLAYER_GENERATED", + "categories": 0, + "event_type": "PORTAL_CAPTURED", + "player_name": "Albicocca", + "portal_name": "L' Arboreto", + "coordinates": { + "lat": 45471652, + "lng": 12274703 + }, + "markup": [ + { + "type": "PLAYER", + "plain": "Albicocca", + "team": "RESISTANCE", + "name": "Albicocca", + "address": "", + "latE6": 0, + "lngE6": 0 + }, + { + "type": "PORTAL", + "plain": "L' Arboreto", + "team": "RESISTANCE", + "name": "L' Arboreto", + "address": "Via Arboreto, Venice, Italy", + "latE6": 45471652, + "lngE6": 12274703 + } + ] + } + ] +} +``` + +**Examples:** + +```bash +# Get all plexts +curl "http://localhost:7000/plexts" + +# Filter by event type +curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED,LINK_CREATED" + +# Filter by player +curl "http://localhost:7000/plexts?player_name=Albicocca" + +# Filter by time range +curl "http://localhost:7000/plexts?min_timestamp=2026-01-12T10:00:00Z&max_timestamp=2026-01-12T12:00:00Z" + +# Combine filters +curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED&player_name=Albicocca&min_timestamp=2026-01-12T10:00:00Z" +``` + +### Scheduler + +The scheduler automatically collects Ingress events every minute and stores them in MongoDB. + +#### Starting the Scheduler + +```bash +# Start the scheduler +python schedule.py + +# Or using the entrypoint script +./entrypoint-schedule.sh +``` + +#### How It Works + +1. The scheduler runs every minute using APScheduler +2. It calculates a time range for the previous minute +3. Fetches plexts from the web API endpoint +4. Stores the results in MongoDB +5. Logs the number of plexts collected + +#### Configuration + +The scheduler requires the following environment variables: + +- `ENDPOINT_URL` - The web API endpoint to fetch plexts from +- `MONGO_URI` - MongoDB connection string +- `DB_NAME` - Database name +- `COLLECTION_NAME` - Collection name +- `TZ` - Timezone for time calculations + +#### Example Output + +``` +[2026-01-12 11:00:00 CET] Starting job... +Time range: 2026-01-12T10:59:00+01:00 to 2026-01-12T11:00:00+01:00 +Fetched 5 plexts +Inserted 5 plexts to MongoDB +Job completed successfully +``` + +### Docker Deployment + +#### Using Docker Compose + +The project includes a [`docker-compose.yml`](docker-compose.yml:1) file for easy deployment. + +**Services:** + +1. **MongoDB** - Database service + - Image: `mongo:7.0` + - Port: `27017` + - Health checks enabled + - Persistent volumes for data + +2. **Web** - Flask web API + - Image: `mrosati84/ingress-web:1.0` + - Port: `7000` + - Depends on MongoDB + - Health checks enabled + +3. **Schedule** - Scheduler service + - Image: `mrosati84/ingress-schedule:1.0` + - Depends on MongoDB and Web + - No exposed ports + +**Commands:** + +```bash +# Start all services +docker-compose up -d + +# View logs +docker-compose logs -f + +# View logs for specific service +docker-compose logs -f web + +# Stop all services +docker-compose down + +# Stop and remove volumes +docker-compose down -v + +# Restart a service +docker-compose restart web + +# Scale services +docker-compose up -d --scale web=3 +``` + +#### Building Custom Images + +If you want to build your own Docker images: + +```bash +# Build web image +docker build -f Dockerfile-web -t ingress-web . + +# Build schedule image +docker build -f Dockerfile-schedule -t ingress-schedule . + +# Update docker-compose.yml to use your images +``` + +#### Health Checks + +All services include health checks: + +- **MongoDB**: Database ping test +- **Web**: HTTP GET request to root endpoint +- **Schedule**: No health check (depends on other services) + ## Event Types | Event Type | Description | @@ -146,6 +531,8 @@ python main.py \ ## Output Format +### CLI Output + Events are printed in the following format: ``` @@ -153,22 +540,190 @@ Events are printed in the following format: [2026-01-12 10:28:26] [LINK_CREATED] Albicocca linked from L' Arboreto to Parco San Giuliano - Coords: 45471652, 12274703 ``` +### API Output + +The API returns JSON with the following structure: + +```json +{ + "count": 2, + "plexts": [ + { + "id": "string", + "timestamp": 1736659207000, + "timestamp_formatted": "2026-01-12 11:00:07", + "text": "string", + "team": "RESISTANCE", + "plext_type": "PLAYER_GENERATED", + "categories": 0, + "event_type": "PORTAL_CAPTURED", + "player_name": "string", + "portal_name": "string", + "coordinates": { + "lat": 45471652, + "lng": 12274703 + }, + "markup": [...] + } + ] +} +``` + ## Project Structure ``` ingress/ -├── main.py # CLI entry point -├── ingress.py # API client -├── models.py # Data models -├── pyproject.toml # Project configuration -└── README.md # This file +├── main.py # CLI entry point +├── app.py # Flask web API +├── ingress.py # Ingress API client +├── models.py # Data models (Plext, EventType, Markup) +├── schedule.py # Scheduled data collector +├── pyproject.toml # Project configuration (uv) +├── requirements.txt # Python dependencies +├── docker-compose.yml # Docker orchestration +├── Dockerfile-web # Web service Docker image +├── Dockerfile-schedule # Scheduler Docker image +├── entrypoint-web.sh # Web service entrypoint script +├── entrypoint-schedule.sh # Scheduler entrypoint script +├── json_doc.md # API documentation +├── .python-version # Python version specification +├── .gitignore # Git ignore rules +├── .env.example # Environment variables template +└── README.md # This file ``` ## Dependencies -- Python 3.13+ -- requests 2.31.0+ +### Core Dependencies + +- **Python 3.11+** (Docker images use Python 3.11) +- **requests>=2.31.0** - HTTP client for API requests +- **flask>=3.1.2** - Web framework for REST API +- **gunicorn>=23.0.0** - Production WSGI server +- **python-dotenv>=1.2.1** - Environment variable management +- **pymongo>=4.10.0** - MongoDB driver +- **apscheduler>=3.10.0** - Task scheduling +- **ipython>=9.9.0** - Interactive Python shell (development) + +### Development Dependencies + +- **uv** - Fast Python package installer (optional) + +## Development + +### Local Development Setup + +```bash +# Install dependencies +pip install -r requirements.txt + +# Or using uv +uv sync + +# Set up environment variables +cp .env.example .env +# Edit .env with your configuration + +# Start MongoDB (if not using Docker) +# Install MongoDB locally or use a cloud instance + +# Start the web API +python app.py + +# In another terminal, start the scheduler +python schedule.py + +# Test the CLI +python main.py --help +``` + +### Running Tests + +```bash +# Run CLI tests +python main.py --event-types PORTAL_CAPTURED --player-name test + +# Test API endpoints +curl http://localhost:7000/ +curl http://localhost:7000/plexts +``` + +### Debugging + +The application includes comprehensive logging: + +- **CLI**: Logs to stdout +- **Web API**: Logs to stdout (Gunicorn captures and forwards) +- **Scheduler**: Logs to stdout +- **Ingress API Client**: Detailed DEBUG level logging + +To enable debug logging: + +```bash +# Set log level in environment +export LOG_LEVEL=DEBUG + +# Or modify the logging configuration in the source files +``` + +## Troubleshooting + +### Common Issues + +**Issue: "No CSRF token found in cookie!"** +- Solution: Ensure your `INGRESS_COOKIE` includes both `csrftoken` and `sessionid` + +**Issue: "MongoDB connection failed"** +- Solution: Check that MongoDB is running and `MONGO_URI` is correct + +**Issue: "Scheduler not collecting data"** +- Solution: Verify `ENDPOINT_URL` is accessible and web API is running + +**Issue: "Docker containers not starting"** +- Solution: Check Docker logs: `docker-compose logs` + +**Issue: "Health check failing"** +- Solution: Ensure all dependencies are running and ports are accessible + +### Debug Mode + +Enable debug mode for more detailed logging: + +```bash +# For web API +export FLASK_DEBUG=1 +python app.py + +# For scheduler +export APSCHEDULER_DEBUG=1 +python schedule.py +``` + +### Logs + +View logs for different components: + +```bash +# Docker logs +docker-compose logs -f + +# Specific service logs +docker-compose logs -f web +docker-compose logs -f schedule +docker-compose logs -f mongodb + +# Application logs (if running locally) +# Logs are printed to stdout +``` ## License See project license file for details. + +## Contributing + +Contributions are welcome! Please feel free to submit a Pull Request. + +## Support + +For issues and questions, please open an issue on the project repository.