admin/Ingress
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
de53b760c43c417fdb9f9d42a898f31c65a7efa2
Ingress/README.md
Matteo Rosati de53b760c4 add center and radius params
2026-01-18 11:41:55 +01:00

23 KiB
Raw Blame History

Ingress Intel Report

A comprehensive system for fetching, analyzing, and storing game events from the Ingress Intel API.

Overview

The Ingress Intel Report is a multi-component system consisting of:

  • CLI Tool (main.py) - Command-line interface for querying events
  • Web API (app.py) - REST API for programmatic access
  • Scheduler (schedule.py) - Automated data collection every minute
  • MongoDB - Persistent storage for historical data
  • Docker - Containerized deployment with docker-compose

Features

  • Geographic Filtering: Query events within a bounding box
  • Event Classification: Automatic event type detection (portal captures, links, resonators, etc.)
  • Player Tracking: Filter events by specific players
  • Timestamp Filtering: Filter events by time range
  • Coordinate Extraction: Extract portal locations from events
  • REST API: JSON-based API for programmatic access
  • MongoDB Persistence: Store historical event data
  • Scheduled Collection: Automated data collection every minute
  • Docker Deployment: Containerized deployment with health checks
  • Timezone Support: Configurable timezone (default: Europe/Rome)

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   CLI Tool  │     │  Web API    │     │  Scheduler  │
│  (main.py)  │     │  (app.py)   │     │(schedule.py)│
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       └───────────────────┼───────────────────┘
                           │
                    ┌──────▼──────┐
                    │ Ingress API │
                    │ (ingress.py)│
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │  MongoDB    │
                    │  Database   │
                    └─────────────┘

Installation

Option 1: Docker (Recommended)

The easiest way to run the complete system is using Docker Compose:

# Clone the repository
git clone <repository-url>
cd Ingress

# Create environment file
cp .env.example .env
# Edit .env with your configuration

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

Option 2: Local Development

For local development without Docker:

# Install dependencies
pip install -r requirements.txt

# Or using uv (recommended)
uv sync

# Set up MongoDB
# Install MongoDB locally or use a cloud instance
# Update MONGO_URI in .env file

# Start the web API
python app.py

# In another terminal, start the scheduler
python schedule.py

Option 3: CLI Only

If you only need the CLI tool:

# Install dependencies
pip install -r requirements.txt

# Set environment variables
export INGRESS_COOKIE="your_cookie_here"
export V="your_api_version"

# Run the CLI
python main.py --help

Configuration

Create a .env file in the project root with the following variables:

Variable Required Default Description
INGRESS_COOKIE Yes - Ingress authentication cookie (includes csrftoken and sessionid)
V Yes - Ingress API version string
PORT No 5000 Web API server port
MONGO_URI Yes* - MongoDB connection string (required for scheduler)
DB_NAME Yes* - MongoDB database name (required for scheduler)
COLLECTION_NAME Yes* - MongoDB collection name (required for scheduler)
ENDPOINT_URL Yes* - API endpoint for scheduler (e.g., http://web:7000/plexts)
TZ No Europe/Rome Timezone for timestamps
MIN_LAT No 45470259 Default minimum latitude (E6 format)
MIN_LNG No 12244155 Default minimum longitude (E6 format)
MAX_LAT No 45480370 Default maximum latitude (E6 format)
MAX_LNG No 12298207 Default maximum longitude (E6 format)

*Required for scheduler and web API with MongoDB integration

Example .env file

# Ingress API Configuration
INGRESS_COOKIE="csrftoken=your_token; sessionid=your_session"
V="412c0ac7e784d6df783fc507bca30e23b3c58c55"

# Web API Configuration
PORT=7000

# MongoDB Configuration
MONGO_URI="mongodb://ingress_root:password@localhost:27017/"
DB_NAME=ingress
COLLECTION_NAME=ingress

# Scheduler Configuration
ENDPOINT_URL="http://localhost:7000/plexts"

# Timezone
TZ=Europe/Rome

# Geographic Bounds (Venice, Italy)
MIN_LAT=45470259
MIN_LNG=12244155
MAX_LAT=45480370
MAX_LNG=12298207

Usage

CLI Tool

The CLI tool provides a command-line interface for querying Ingress events.

Basic Usage

# Get all events in default area (Venice, Italy)
python main.py

# Show help
python main.py --help

Command Line Options

Option Type Default Description
--event-types Multiple strings None Filter by event types (comma-separated or space-separated)
--player-name String None Filter events by a specific player name
--min-lat Integer 45470259 Minimum latitude (in microdegrees, E6 format)
--min-lng Integer 12244155 Minimum longitude (in microdegrees, E6 format)
--max-lat Integer 45480370 Maximum latitude (in microdegrees, E6 format)
--max-lng Integer 12298207 Maximum longitude (in microdegrees, E6 format)
--min-timestamp Timestamp -1 Minimum timestamp (milliseconds since epoch or ISO 8601 format)
--max-timestamp Timestamp -1 Maximum timestamp (milliseconds since epoch or ISO 8601 format)

Timestamp Filtering

The --min-timestamp and --max-timestamp options support two formats:

Format 1: Milliseconds (integer)

python main.py --min-timestamp 1736659200000 --max-timestamp 1736745600000

Format 2: ISO 8601 datetime string

python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"

Examples:

# Get events from last hour
python main.py --min-timestamp 1736659200000

# Get events for a specific day
python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"

# Get events from a specific time onwards
python main.py --min-timestamp "2026-01-12T10:00:00Z"

# Get events up to a specific time
python main.py --max-timestamp "2026-01-12T12:00:00Z"

Event Type Filtering

# Filter by single event type
python main.py --event-types PORTAL_CAPTURED

# Filter by multiple event types (space-separated)
python main.py --event-types PORTAL_CAPTURED LINK_CREATED CONTROL_FIELD_CREATED

# Filter by resonator events
python main.py --event-types RESONATOR_DEPLOYED RESONATOR_DESTROYED

Player Filtering

# Filter by player name
python main.py --player-name Albicocca

# Combine with event type filter
python main.py --player-name Albicocca --event-types PORTAL_CAPTURED

Geographic Filtering

Coordinates are in E6 format (microdegrees): multiply decimal degrees by 1,000,000.

# Custom geographic bounds
python main.py --min-lat 45470000 --max-lat 45480000 --min-lng 12240000 --max-lng 12300000

Combining Filters

# Complex query with multiple filters
python main.py \
  --min-timestamp "2026-01-12T10:00:00Z" \
  --max-timestamp "2026-01-12T12:00:00Z" \
  --event-types PORTAL_CAPTURED LINK_CREATED \
  --player-name Albicocca \
  --min-lat 45470000 \
  --max-lat 45480000

Web API

The web API provides a REST interface for programmatic access to Ingress events.

Starting the Web API

# Development mode
python app.py

# Production mode with Gunicorn
gunicorn -w 4 -b 0.0.0.0:7000 app:app

# Or using the entrypoint script
./entrypoint-web.sh

API Endpoints

GET /

Returns API information and available endpoints.

Response:

{
  "name": "Ingress Intel API",
  "version": "1.0.0",
  "endpoints": {
    "/plexts": {
      "method": "GET",
      "description": "Get plexts from the Ingress API",
      "parameters": {
        "event_types": "List of event types to filter by (comma-separated)",
        "player_name": "Player name to filter by",
        "min_lat": "Minimum latitude (default: 45470259)",
        "min_lng": "Minimum longitude (default: 12244155)",
        "max_lat": "Maximum latitude (default: 45480370)",
        "max_lng": "Maximum longitude (default: 12298207)",
        "min_timestamp": "Minimum timestamp (milliseconds or ISO 8601 format, default: -1)",
        "max_timestamp": "Maximum timestamp (milliseconds or ISO 8601 format, default: -1)"
      },
      "event_types": [
        "RESONATOR_DEPLOYED",
        "RESONATOR_DESTROYED",
        "PORTAL_CAPTURED",
        "PORTAL_NEUTRALIZED",
        "PORTAL_UNDER_ATTACK",
        "LINK_CREATED",
        "LINK_DESTROYED",
        "CONTROL_FIELD_CREATED",
        "UNKNOWN"
      ]
    }
  }
}
GET /plexts

Get plexts from the Ingress API with optional filters.

Query Parameters:

Parameter Type Default Description
event_types String None Comma-separated list of event types to filter by
player_name String None Player name to filter by
min_lat Integer From env Minimum latitude (E6 format)
min_lng Integer From env Minimum longitude (E6 format)
max_lat Integer From env Maximum latitude (E6 format)
max_lng Integer From env Maximum longitude (E6 format)
min_timestamp String -1 Minimum timestamp (milliseconds or ISO 8601 format)
max_timestamp String -1 Maximum timestamp (milliseconds or ISO 8601 format)

Response:

{
  "count": 2,
  "plexts": [
    {
      "id": "abc123",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "Albicocca captured L' Arboreto",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "Albicocca",
      "portal_name": "L' Arboreto",
      "coordinates": {
        "lat": 45471652,
        "lng": 12274703
      },
      "markup": [
        {
          "type": "PLAYER",
          "plain": "Albicocca",
          "team": "RESISTANCE",
          "name": "Albicocca",
          "address": "",
          "latE6": 0,
          "lngE6": 0
        },
        {
          "type": "PORTAL",
          "plain": "L' Arboreto",
          "team": "RESISTANCE",
          "name": "L' Arboreto",
          "address": "Via Arboreto, Venice, Italy",
          "latE6": 45471652,
          "lngE6": 12274703
        }
      ]
    }
  ]
}

Examples:

# Get all plexts
curl "http://localhost:7000/plexts"

# Filter by event type
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED,LINK_CREATED"

# Filter by player
curl "http://localhost:7000/plexts?player_name=Albicocca"

# Filter by time range
curl "http://localhost:7000/plexts?min_timestamp=2026-01-12T10:00:00Z&max_timestamp=2026-01-12T12:00:00Z"

# Combine filters
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED&player_name=Albicocca&min_timestamp=2026-01-12T10:00:00Z"
GET /plexts/from-db

Get plexts from MongoDB with optional filters. Returns plexts sorted by timestamp (most recent first).

Query Parameters:

Parameter Type Default Description
player_name String None Player name to filter by
timestamp_from Integer None Minimum timestamp in milliseconds (optional)
timestamp_to Integer None Maximum timestamp in milliseconds (optional)
limit Integer 100 Maximum number of results to return
center String None Geographic center point in format "latitude,longitude" (optional, requires radius)
radius Integer None Search radius in meters (optional, requires center)

Geospatial Query:

When both center and radius parameters are provided, the endpoint returns plexts within the specified radius from the center point. The response includes a distance field (in meters) for each result.

Response with geospatial query:

{
  "count": 5,
  "plexts": [
    {
      "id": "abc123",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "Albicocca captured L' Arboreto",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "Albicocca",
      "portal_name": "L' Arboreto",
      "coordinates": {
        "type": "Point",
        "coordinates": [12.365208, 45.573661]
      },
      "distance": 125.5,
      "markup": [...]
    }
  ]
}

Note: Both center and radius parameters must be provided together. If neither is provided, the endpoint returns all plexts matching other filters without geospatial filtering.

Examples:

# Get all plexts from MongoDB
curl "http://localhost:7000/plexts/from-db"

# Filter by player name
curl "http://localhost:7000/plexts/from-db?player_name=Albicocca"

# Filter by time range
curl "http://localhost:7000/plexts/from-db?timestamp_from=17366592000000&timestamp_to=1736745600000"

# Geospatial query: Get plexts within 1km of a specific location
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000"

# Geospatial query with player filter
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&player_name=Albicocca"

# Geospatial query with time filter
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&timestamp_from=17366592000000"

# Combine all filters
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&player_name=Albicocca&timestamp_from=17366592000000&limit=10"

Authentication: Basic Auth required

Scheduler

The scheduler automatically collects Ingress events every minute and stores them in MongoDB.

Starting the Scheduler

# Start the scheduler
python schedule.py

# Or using the entrypoint script
./entrypoint-schedule.sh

How It Works

  1. The scheduler runs every minute using APScheduler
  2. It calculates a time range for the previous minute
  3. Fetches plexts from the web API endpoint
  4. Stores the results in MongoDB
  5. Logs the number of plexts collected

Configuration

The scheduler requires the following environment variables:

  • ENDPOINT_URL - The web API endpoint to fetch plexts from
  • MONGO_URI - MongoDB connection string
  • DB_NAME - Database name
  • COLLECTION_NAME - Collection name
  • TZ - Timezone for time calculations

Example Output

[2026-01-12 11:00:00 CET] Starting job...
Time range: 2026-01-12T10:59:00+01:00 to 2026-01-12T11:00:00+01:00
Fetched 5 plexts
Inserted 5 plexts to MongoDB
Job completed successfully

Docker Deployment

Using Docker Compose

The project includes a docker-compose.yml file for easy deployment.

Services:

  1. MongoDB - Database service

    • Image: mongo:7.0
    • Port: 27017
    • Health checks enabled
    • Persistent volumes for data

  2. Web - Flask web API

    • Image: mrosati84/ingress-web:1.0
    • Port: 7000
    • Depends on MongoDB
    • Health checks enabled

  3. Schedule - Scheduler service

    • Image: mrosati84/ingress-schedule:1.0
    • Depends on MongoDB and Web
    • No exposed ports

Commands:

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# View logs for specific service
docker-compose logs -f web

# Stop all services
docker-compose down

# Stop and remove volumes
docker-compose down -v

# Restart a service
docker-compose restart web

# Scale services
docker-compose up -d --scale web=3

Building Custom Images

If you want to build your own Docker images:

# Build web image
docker build -f Dockerfile-web -t ingress-web .

# Build schedule image
docker build -f Dockerfile-schedule -t ingress-schedule .

# Update docker-compose.yml to use your images

Health Checks

All services include health checks:

  • MongoDB: Database ping test
  • Web: HTTP GET request to root endpoint
  • Schedule: No health check (depends on other services)

Event Types

Event Type Description
RESONATOR_DEPLOYED Player deploys resonator on portal
RESONATOR_DESTROYED Resonator destroyed
PORTAL_CAPTURED Portal captured by player
PORTAL_NEUTRALIZED Portal neutralized
PORTAL_UNDER_ATTACK Portal being attacked
LINK_CREATED Link created between portals
LINK_DESTROYED Link destroyed
CONTROL_FIELD_CREATED Control field created
UNKNOWN Unrecognized event type

Output Format

CLI Output

Events are printed in the following format:

[2026-01-12 10:28:27] [PORTAL_CAPTURED] Albicocca captured L' Arboreto - Coords: 45471652, 12274703
[2026-01-12 10:28:26] [LINK_CREATED] Albicocca linked from L' Arboreto to Parco San Giuliano - Coords: 45471652, 12274703

API Output

The API returns JSON with the following structure:

{
  "count": 2,
  "plexts": [
    {
      "id": "string",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "string",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "string",
      "portal_name": "string",
      "coordinates": {
        "lat": 45471652,
        "lng": 12274703
      },
      "markup": [...]
    }
  ]
}

Project Structure

ingress/
├── main.py                    # CLI entry point
├── app.py                     # Flask web API
├── ingress.py                 # Ingress API client
├── models.py                  # Data models (Plext, EventType, Markup)
├── schedule.py                # Scheduled data collector
├── pyproject.toml             # Project configuration (uv)
├── requirements.txt           # Python dependencies
├── docker-compose.yml         # Docker orchestration
├── Dockerfile-web             # Web service Docker image
├── Dockerfile-schedule        # Scheduler Docker image
├── entrypoint-web.sh          # Web service entrypoint script
├── entrypoint-schedule.sh     # Scheduler entrypoint script
├── json_doc.md                # API documentation
├── .python-version            # Python version specification
├── .gitignore                 # Git ignore rules
├── .env.example               # Environment variables template
└── README.md                  # This file

Dependencies

Core Dependencies

  • Python 3.11+ (Docker images use Python 3.11)
  • requests>=2.31.0 - HTTP client for API requests
  • flask>=3.1.2 - Web framework for REST API
  • gunicorn>=23.0.0 - Production WSGI server
  • python-dotenv>=1.2.1 - Environment variable management
  • pymongo>=4.10.0 - MongoDB driver
  • apscheduler>=3.10.0 - Task scheduling
  • ipython>=9.9.0 - Interactive Python shell (development)

Development Dependencies

  • uv - Fast Python package installer (optional)

Development

Local Development Setup

# Install dependencies
pip install -r requirements.txt

# Or using uv
uv sync

# Set up environment variables
cp .env.example .env
# Edit .env with your configuration

# Start MongoDB (if not using Docker)
# Install MongoDB locally or use a cloud instance

# Start the web API
python app.py

# In another terminal, start the scheduler
python schedule.py

# Test the CLI
python main.py --help

Running Tests

# Run CLI tests
python main.py --event-types PORTAL_CAPTURED --player-name test

# Test API endpoints
curl http://localhost:7000/
curl http://localhost:7000/plexts

Debugging

The application includes comprehensive logging:

  • CLI: Logs to stdout
  • Web API: Logs to stdout (Gunicorn captures and forwards)
  • Scheduler: Logs to stdout
  • Ingress API Client: Detailed DEBUG level logging

To enable debug logging:

# Set log level in environment
export LOG_LEVEL=DEBUG

# Or modify the logging configuration in the source files

Troubleshooting

Common Issues

Issue: "No CSRF token found in cookie!"

  • Solution: Ensure your INGRESS_COOKIE includes both csrftoken and sessionid

Issue: "MongoDB connection failed"

  • Solution: Check that MongoDB is running and MONGO_URI is correct

Issue: "Scheduler not collecting data"

  • Solution: Verify ENDPOINT_URL is accessible and web API is running

Issue: "Docker containers not starting"

  • Solution: Check Docker logs: docker-compose logs

Issue: "Health check failing"

  • Solution: Ensure all dependencies are running and ports are accessible

Debug Mode

Enable debug mode for more detailed logging:

# For web API
export FLASK_DEBUG=1
python app.py

# For scheduler
export APSCHEDULER_DEBUG=1
python schedule.py

Logs

View logs for different components:

# Docker logs
docker-compose logs -f

# Specific service logs
docker-compose logs -f web
docker-compose logs -f schedule
docker-compose logs -f mongodb

# Application logs (if running locally)
# Logs are printed to stdout

License

See project license file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and questions, please open an issue on the project repository.

Reference in New Issue View Git Blame Copy Permalink