admin/Ingress
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4af1996fb4f0ee4e63450f1ccad38711b825174d
Ingress/README.md
Matteo Rosati 49c5225027 update readme
2026-01-15 00:10:43 +01:00

730 lines
20 KiB
Markdown
Raw Blame History

# Ingress Intel Report
A comprehensive system for fetching, analyzing, and storing game events from the Ingress Intel API.
## Overview
The Ingress Intel Report is a multi-component system consisting of:
- **CLI Tool** ([`main.py`](main.py:1)) - Command-line interface for querying events
- **Web API** ([`app.py`](app.py:1)) - REST API for programmatic access
- **Scheduler** ([`schedule.py`](schedule.py:1)) - Automated data collection every minute
- **MongoDB** - Persistent storage for historical data
- **Docker** - Containerized deployment with docker-compose
## Features
- **Geographic Filtering**: Query events within a bounding box
- **Event Classification**: Automatic event type detection (portal captures, links, resonators, etc.)
- **Player Tracking**: Filter events by specific players
- **Timestamp Filtering**: Filter events by time range
- **Coordinate Extraction**: Extract portal locations from events
- **REST API**: JSON-based API for programmatic access
- **MongoDB Persistence**: Store historical event data
- **Scheduled Collection**: Automated data collection every minute
- **Docker Deployment**: Containerized deployment with health checks
- **Timezone Support**: Configurable timezone (default: Europe/Rome)
## Architecture
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ CLI Tool │ │ Web API │ │ Scheduler │
│ (main.py) │ │ (app.py) │ │(schedule.py)│
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└───────────────────┼───────────────────┘
│
┌──────▼──────┐
│ Ingress API │
│ (ingress.py)│
└──────┬──────┘
│
┌──────▼──────┐
│ MongoDB │
│ Database │
└─────────────┘
```
## Installation
### Option 1: Docker (Recommended)
The easiest way to run the complete system is using Docker Compose:
```bash
# Clone the repository
git clone <repository-url>
cd Ingress
# Create environment file
cp .env.example .env
# Edit .env with your configuration
# Start all services
docker-compose up -d
# View logs
docker-compose logs -f
# Stop services
docker-compose down
```
### Option 2: Local Development
For local development without Docker:
```bash
# Install dependencies
pip install -r requirements.txt
# Or using uv (recommended)
uv sync
# Set up MongoDB
# Install MongoDB locally or use a cloud instance
# Update MONGO_URI in .env file
# Start the web API
python app.py
# In another terminal, start the scheduler
python schedule.py
```
### Option 3: CLI Only
If you only need the CLI tool:
```bash
# Install dependencies
pip install -r requirements.txt
# Set environment variables
export INGRESS_COOKIE="your_cookie_here"
export V="your_api_version"
# Run the CLI
python main.py --help
```
## Configuration
Create a `.env` file in the project root with the following variables:
| Variable | Required | Default | Description |
| ----------------- | -------- | ----------- | ---------------------------------------------------------------- |
| `INGRESS_COOKIE` | Yes | - | Ingress authentication cookie (includes csrftoken and sessionid) |
| `V` | Yes | - | Ingress API version string |
| `PORT` | No | 5000 | Web API server port |
| `MONGO_URI` | Yes* | - | MongoDB connection string (required for scheduler) |
| `DB_NAME` | Yes* | - | MongoDB database name (required for scheduler) |
| `COLLECTION_NAME` | Yes* | - | MongoDB collection name (required for scheduler) |
| `ENDPOINT_URL` | Yes* | - | API endpoint for scheduler (e.g., http://web:7000/plexts) |
| `TZ` | No | Europe/Rome | Timezone for timestamps |
| `MIN_LAT` | No | 45470259 | Default minimum latitude (E6 format) |
| `MIN_LNG` | No | 12244155 | Default minimum longitude (E6 format) |
| `MAX_LAT` | No | 45480370 | Default maximum latitude (E6 format) |
| `MAX_LNG` | No | 12298207 | Default maximum longitude (E6 format) |
*Required for scheduler and web API with MongoDB integration
### Example .env file
```bash
# Ingress API Configuration
INGRESS_COOKIE="csrftoken=your_token; sessionid=your_session"
V="412c0ac7e784d6df783fc507bca30e23b3c58c55"
# Web API Configuration
PORT=7000
# MongoDB Configuration
MONGO_URI="mongodb://ingress_root:password@localhost:27017/"
DB_NAME=ingress
COLLECTION_NAME=ingress
# Scheduler Configuration
ENDPOINT_URL="http://localhost:7000/plexts"
# Timezone
TZ=Europe/Rome
# Geographic Bounds (Venice, Italy)
MIN_LAT=45470259
MIN_LNG=12244155
MAX_LAT=45480370
MAX_LNG=12298207
```
## Usage
### CLI Tool
The CLI tool provides a command-line interface for querying Ingress events.
#### Basic Usage
```bash
# Get all events in default area (Venice, Italy)
python main.py
# Show help
python main.py --help
```
#### Command Line Options
| Option | Type | Default | Description |
| ----------------- | ---------------- | -------- | --------------------------------------------------------------- |
| `--event-types` | Multiple strings | None | Filter by event types (comma-separated or space-separated) |
| `--player-name` | String | None | Filter events by a specific player name |
| `--min-lat` | Integer | 45470259 | Minimum latitude (in microdegrees, E6 format) |
| `--min-lng` | Integer | 12244155 | Minimum longitude (in microdegrees, E6 format) |
| `--max-lat` | Integer | 45480370 | Maximum latitude (in microdegrees, E6 format) |
| `--max-lng` | Integer | 12298207 | Maximum longitude (in microdegrees, E6 format) |
| `--min-timestamp` | Timestamp | -1 | Minimum timestamp (milliseconds since epoch or ISO 8601 format) |
| `--max-timestamp` | Timestamp | -1 | Maximum timestamp (milliseconds since epoch or ISO 8601 format) |
#### Timestamp Filtering
The `--min-timestamp` and `--max-timestamp` options support two formats:
**Format 1: Milliseconds (integer)**
```bash
python main.py --min-timestamp 1736659200000 --max-timestamp 1736745600000
```
**Format 2: ISO 8601 datetime string**
```bash
python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"
```
**Examples:**
```bash
# Get events from last hour
python main.py --min-timestamp 1736659200000
# Get events for a specific day
python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"
# Get events from a specific time onwards
python main.py --min-timestamp "2026-01-12T10:00:00Z"
# Get events up to a specific time
python main.py --max-timestamp "2026-01-12T12:00:00Z"
```
#### Event Type Filtering
```bash
# Filter by single event type
python main.py --event-types PORTAL_CAPTURED
# Filter by multiple event types (space-separated)
python main.py --event-types PORTAL_CAPTURED LINK_CREATED CONTROL_FIELD_CREATED
# Filter by resonator events
python main.py --event-types RESONATOR_DEPLOYED RESONATOR_DESTROYED
```
#### Player Filtering
```bash
# Filter by player name
python main.py --player-name Albicocca
# Combine with event type filter
python main.py --player-name Albicocca --event-types PORTAL_CAPTURED
```
#### Geographic Filtering
Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,000,000.
```bash
# Custom geographic bounds
python main.py --min-lat 45470000 --max-lat 45480000 --min-lng 12240000 --max-lng 12300000
```
#### Combining Filters
```bash
# Complex query with multiple filters
python main.py \
--min-timestamp "2026-01-12T10:00:00Z" \
--max-timestamp "2026-01-12T12:00:00Z" \
--event-types PORTAL_CAPTURED LINK_CREATED \
--player-name Albicocca \
--min-lat 45470000 \
--max-lat 45480000
```
### Web API
The web API provides a REST interface for programmatic access to Ingress events.
#### Starting the Web API
```bash
# Development mode
python app.py
# Production mode with Gunicorn
gunicorn -w 4 -b 0.0.0.0:7000 app:app
# Or using the entrypoint script
./entrypoint-web.sh
```
#### API Endpoints
##### GET /
Returns API information and available endpoints.
**Response:**
```json
{
"name": "Ingress Intel API",
"version": "1.0.0",
"endpoints": {
"/plexts": {
"method": "GET",
"description": "Get plexts from the Ingress API",
"parameters": {
"event_types": "List of event types to filter by (comma-separated)",
"player_name": "Player name to filter by",
"min_lat": "Minimum latitude (default: 45470259)",
"min_lng": "Minimum longitude (default: 12244155)",
"max_lat": "Maximum latitude (default: 45480370)",
"max_lng": "Maximum longitude (default: 12298207)",
"min_timestamp": "Minimum timestamp (milliseconds or ISO 8601 format, default: -1)",
"max_timestamp": "Maximum timestamp (milliseconds or ISO 8601 format, default: -1)"
},
"event_types": [
"RESONATOR_DEPLOYED",
"RESONATOR_DESTROYED",
"PORTAL_CAPTURED",
"PORTAL_NEUTRALIZED",
"PORTAL_UNDER_ATTACK",
"LINK_CREATED",
"LINK_DESTROYED",
"CONTROL_FIELD_CREATED",
"UNKNOWN"
]
}
}
}
```
##### GET /plexts
Get plexts from the Ingress API with optional filters.
**Query Parameters:**
| Parameter | Type | Default | Description |
| --------------- | ------- | -------- | --------------------------------------------------- |
| `event_types` | String | None | Comma-separated list of event types to filter by |
| `player_name` | String | None | Player name to filter by |
| `min_lat` | Integer | From env | Minimum latitude (E6 format) |
| `min_lng` | Integer | From env | Minimum longitude (E6 format) |
| `max_lat` | Integer | From env | Maximum latitude (E6 format) |
| `max_lng` | Integer | From env | Maximum longitude (E6 format) |
| `min_timestamp` | String | -1 | Minimum timestamp (milliseconds or ISO 8601 format) |
| `max_timestamp` | String | -1 | Maximum timestamp (milliseconds or ISO 8601 format) |
**Response:**
```json
{
"count": 2,
"plexts": [
{
"id": "abc123",
"timestamp": 1736659207000,
"timestamp_formatted": "2026-01-12 11:00:07",
"text": "Albicocca captured L' Arboreto",
"team": "RESISTANCE",
"plext_type": "PLAYER_GENERATED",
"categories": 0,
"event_type": "PORTAL_CAPTURED",
"player_name": "Albicocca",
"portal_name": "L' Arboreto",
"coordinates": {
"lat": 45471652,
"lng": 12274703
},
"markup": [
{
"type": "PLAYER",
"plain": "Albicocca",
"team": "RESISTANCE",
"name": "Albicocca",
"address": "",
"latE6": 0,
"lngE6": 0
},
{
"type": "PORTAL",
"plain": "L' Arboreto",
"team": "RESISTANCE",
"name": "L' Arboreto",
"address": "Via Arboreto, Venice, Italy",
"latE6": 45471652,
"lngE6": 12274703
}
]
}
]
}
```
**Examples:**
```bash
# Get all plexts
curl "http://localhost:7000/plexts"
# Filter by event type
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED,LINK_CREATED"
# Filter by player
curl "http://localhost:7000/plexts?player_name=Albicocca"
# Filter by time range
curl "http://localhost:7000/plexts?min_timestamp=2026-01-12T10:00:00Z&max_timestamp=2026-01-12T12:00:00Z"
# Combine filters
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED&player_name=Albicocca&min_timestamp=2026-01-12T10:00:00Z"
```
### Scheduler
The scheduler automatically collects Ingress events every minute and stores them in MongoDB.
#### Starting the Scheduler
```bash
# Start the scheduler
python schedule.py
# Or using the entrypoint script
./entrypoint-schedule.sh
```
#### How It Works
1. The scheduler runs every minute using APScheduler
2. It calculates a time range for the previous minute
3. Fetches plexts from the web API endpoint
4. Stores the results in MongoDB
5. Logs the number of plexts collected
#### Configuration
The scheduler requires the following environment variables:
- `ENDPOINT_URL` - The web API endpoint to fetch plexts from
- `MONGO_URI` - MongoDB connection string
- `DB_NAME` - Database name
- `COLLECTION_NAME` - Collection name
- `TZ` - Timezone for time calculations
#### Example Output
```
[2026-01-12 11:00:00 CET] Starting job...
Time range: 2026-01-12T10:59:00+01:00 to 2026-01-12T11:00:00+01:00
Fetched 5 plexts
Inserted 5 plexts to MongoDB
Job completed successfully
```
### Docker Deployment
#### Using Docker Compose
The project includes a [`docker-compose.yml`](docker-compose.yml:1) file for easy deployment.
**Services:**
1. **MongoDB** - Database service
- Image: `mongo:7.0`
- Port: `27017`
- Health checks enabled
- Persistent volumes for data
2. **Web** - Flask web API
- Image: `mrosati84/ingress-web:1.0`
- Port: `7000`
- Depends on MongoDB
- Health checks enabled
3. **Schedule** - Scheduler service
- Image: `mrosati84/ingress-schedule:1.0`
- Depends on MongoDB and Web
- No exposed ports
**Commands:**
```bash
# Start all services
docker-compose up -d
# View logs
docker-compose logs -f
# View logs for specific service
docker-compose logs -f web
# Stop all services
docker-compose down
# Stop and remove volumes
docker-compose down -v
# Restart a service
docker-compose restart web
# Scale services
docker-compose up -d --scale web=3
```
#### Building Custom Images
If you want to build your own Docker images:
```bash
# Build web image
docker build -f Dockerfile-web -t ingress-web .
# Build schedule image
docker build -f Dockerfile-schedule -t ingress-schedule .
# Update docker-compose.yml to use your images
```
#### Health Checks
All services include health checks:
- **MongoDB**: Database ping test
- **Web**: HTTP GET request to root endpoint
- **Schedule**: No health check (depends on other services)
## Event Types
| Event Type | Description |
| ----------------------- | ---------------------------------- |
| `RESONATOR_DEPLOYED` | Player deploys resonator on portal |
| `RESONATOR_DESTROYED` | Resonator destroyed |
| `PORTAL_CAPTURED` | Portal captured by player |
| `PORTAL_NEUTRALIZED` | Portal neutralized |
| `PORTAL_UNDER_ATTACK` | Portal being attacked |
| `LINK_CREATED` | Link created between portals |
| `LINK_DESTROYED` | Link destroyed |
| `CONTROL_FIELD_CREATED` | Control field created |
| `UNKNOWN` | Unrecognized event type |
## Output Format
### CLI Output
Events are printed in the following format:
```
[2026-01-12 10:28:27] [PORTAL_CAPTURED] Albicocca captured L' Arboreto - Coords: 45471652, 12274703
[2026-01-12 10:28:26] [LINK_CREATED] Albicocca linked from L' Arboreto to Parco San Giuliano - Coords: 45471652, 12274703
```
### API Output
The API returns JSON with the following structure:
```json
{
"count": 2,
"plexts": [
{
"id": "string",
"timestamp": 1736659207000,
"timestamp_formatted": "2026-01-12 11:00:07",
"text": "string",
"team": "RESISTANCE",
"plext_type": "PLAYER_GENERATED",
"categories": 0,
"event_type": "PORTAL_CAPTURED",
"player_name": "string",
"portal_name": "string",
"coordinates": {
"lat": 45471652,
"lng": 12274703
},
"markup": [...]
}
]
}
```
## Project Structure
```
ingress/
├── main.py # CLI entry point
├── app.py # Flask web API
├── ingress.py # Ingress API client
├── models.py # Data models (Plext, EventType, Markup)
├── schedule.py # Scheduled data collector
├── pyproject.toml # Project configuration (uv)
├── requirements.txt # Python dependencies
├── docker-compose.yml # Docker orchestration
├── Dockerfile-web # Web service Docker image
├── Dockerfile-schedule # Scheduler Docker image
├── entrypoint-web.sh # Web service entrypoint script
├── entrypoint-schedule.sh # Scheduler entrypoint script
├── json_doc.md # API documentation
├── .python-version # Python version specification
├── .gitignore # Git ignore rules
├── .env.example # Environment variables template
└── README.md # This file
```
## Dependencies
### Core Dependencies
- **Python 3.11+** (Docker images use Python 3.11)
- **requests>=2.31.0** - HTTP client for API requests
- **flask>=3.1.2** - Web framework for REST API
- **gunicorn>=23.0.0** - Production WSGI server
- **python-dotenv>=1.2.1** - Environment variable management
- **pymongo>=4.10.0** - MongoDB driver
- **apscheduler>=3.10.0** - Task scheduling
- **ipython>=9.9.0** - Interactive Python shell (development)
### Development Dependencies
- **uv** - Fast Python package installer (optional)
## Development
### Local Development Setup
```bash
# Install dependencies
pip install -r requirements.txt
# Or using uv
uv sync
# Set up environment variables
cp .env.example .env
# Edit .env with your configuration
# Start MongoDB (if not using Docker)
# Install MongoDB locally or use a cloud instance
# Start the web API
python app.py
# In another terminal, start the scheduler
python schedule.py
# Test the CLI
python main.py --help
```
### Running Tests
```bash
# Run CLI tests
python main.py --event-types PORTAL_CAPTURED --player-name test
# Test API endpoints
curl http://localhost:7000/
curl http://localhost:7000/plexts
```
### Debugging
The application includes comprehensive logging:
- **CLI**: Logs to stdout
- **Web API**: Logs to stdout (Gunicorn captures and forwards)
- **Scheduler**: Logs to stdout
- **Ingress API Client**: Detailed DEBUG level logging
To enable debug logging:
```bash
# Set log level in environment
export LOG_LEVEL=DEBUG
# Or modify the logging configuration in the source files
```
## Troubleshooting
### Common Issues
**Issue: "No CSRF token found in cookie!"**
- Solution: Ensure your `INGRESS_COOKIE` includes both `csrftoken` and `sessionid`
**Issue: "MongoDB connection failed"**
- Solution: Check that MongoDB is running and `MONGO_URI` is correct
**Issue: "Scheduler not collecting data"**
- Solution: Verify `ENDPOINT_URL` is accessible and web API is running
**Issue: "Docker containers not starting"**
- Solution: Check Docker logs: `docker-compose logs`
**Issue: "Health check failing"**
- Solution: Ensure all dependencies are running and ports are accessible
### Debug Mode
Enable debug mode for more detailed logging:
```bash
# For web API
export FLASK_DEBUG=1
python app.py
# For scheduler
export APSCHEDULER_DEBUG=1
python schedule.py
```
### Logs
View logs for different components:
```bash
# Docker logs
docker-compose logs -f
# Specific service logs
docker-compose logs -f web
docker-compose logs -f schedule
docker-compose logs -f mongodb
# Application logs (if running locally)
# Logs are printed to stdout
```
## License
See project license file for details.
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## Support
For issues and questions, please open an issue on the project repository.
Reference in New Issue View Git Blame Copy Permalink