# Ingress Intel Report

A comprehensive system for fetching, analyzing, and storing game events from the Ingress Intel API.

## Overview

The Ingress Intel Report is a multi-component system consisting of:

- **CLI Tool** ([`main.py`](main.py:1)) - Command-line interface for querying events
- **Web API** ([`app.py`](app.py:1)) - REST API for programmatic access
- **Scheduler** ([`schedule.py`](schedule.py:1)) - Automated data collection every minute
- **MongoDB** - Persistent storage for historical data
- **Docker** - Containerized deployment with docker-compose

## Features

- **Geographic Filtering**: Query events within a bounding box
- **Event Classification**: Automatic event type detection (portal captures, links, resonators, etc.)
- **Player Tracking**: Filter events by specific players
- **Timestamp Filtering**: Filter events by time range
- **Coordinate Extraction**: Extract portal locations from events
- **REST API**: JSON-based API for programmatic access
- **MongoDB Persistence**: Store historical event data
- **Scheduled Collection**: Automated data collection every minute
- **Docker Deployment**: Containerized deployment with health checks
- **Timezone Support**: Configurable timezone (default: Europe/Rome)

## Architecture

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   CLI Tool  │     │  Web API    │     │  Scheduler  │
│  (main.py)  │     │  (app.py)   │     │(schedule.py)│
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       └───────────────────┼───────────────────┘
                           │
                    ┌──────▼──────┐
                    │ Ingress API │
                    │ (ingress.py)│
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │  MongoDB    │
                    │  Database   │
                    └─────────────┘
```

## Installation

### Option 1: Docker (Recommended)

The easiest way to run the complete system is using Docker Compose:

```bash
# Clone the repository
git clone <repository-url>
cd Ingress

# Create environment file
cp .env.example .env
# Edit .env with your configuration

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down
```

### Option 2: Local Development

For local development without Docker:

```bash
# Install dependencies
pip install -r requirements.txt

# Or using uv (recommended)
uv sync

# Set up MongoDB
# Install MongoDB locally or use a cloud instance
# Update MONGO_URI in .env file

# Start the web API
python app.py

# In another terminal, start the scheduler
python schedule.py
```

### Option 3: CLI Only

If you only need the CLI tool:

```bash
# Install dependencies
pip install -r requirements.txt

# Set environment variables
export INGRESS_COOKIE="your_cookie_here"
export V="your_api_version"

# Run the CLI
python main.py --help
```

## Configuration

Create a `.env` file in the project root with the following variables:

| Variable          | Required | Default     | Description                                                      |
| ----------------- | -------- | ----------- | ---------------------------------------------------------------- |
| `INGRESS_COOKIE`  | Yes      | -           | Ingress authentication cookie (includes csrftoken and sessionid) |
| `V`               | Yes      | -           | Ingress API version string                                       |
| `PORT`            | No       | 5000        | Web API server port                                              |
| `MONGO_URI`       | Yes*     | -           | MongoDB connection string (required for scheduler)               |
| `DB_NAME`         | Yes*     | -           | MongoDB database name (required for scheduler)                   |
| `COLLECTION_NAME` | Yes*     | -           | MongoDB collection name (required for scheduler)                 |
| `ENDPOINT_URL`    | Yes*     | -           | API endpoint for scheduler (e.g., http://web:7000/plexts)        |
| `TZ`              | No       | Europe/Rome | Timezone for timestamps                                          |
| `MIN_LAT`         | No       | 45470259    | Default minimum latitude (E6 format)                             |
| `MIN_LNG`         | No       | 12244155    | Default minimum longitude (E6 format)                            |
| `MAX_LAT`         | No       | 45480370    | Default maximum latitude (E6 format)                             |
| `MAX_LNG`         | No       | 12298207    | Default maximum longitude (E6 format)                            |

*Required for scheduler and web API with MongoDB integration

### Example .env file

```bash
# Ingress API Configuration
INGRESS_COOKIE="csrftoken=your_token; sessionid=your_session"
V="412c0ac7e784d6df783fc507bca30e23b3c58c55"

# Web API Configuration
PORT=7000

# MongoDB Configuration
MONGO_URI="mongodb://ingress_root:password@localhost:27017/"
DB_NAME=ingress
COLLECTION_NAME=ingress

# Scheduler Configuration
ENDPOINT_URL="http://localhost:7000/plexts"

# Timezone
TZ=Europe/Rome

# Geographic Bounds (Venice, Italy)
MIN_LAT=45470259
MIN_LNG=12244155
MAX_LAT=45480370
MAX_LNG=12298207
```

## Usage

### CLI Tool

The CLI tool provides a command-line interface for querying Ingress events.

#### Basic Usage

```bash
# Get all events in default area (Venice, Italy)
python main.py

# Show help
python main.py --help
```

#### Command Line Options

| Option            | Type             | Default  | Description                                                     |
| ----------------- | ---------------- | -------- | --------------------------------------------------------------- |
| `--event-types`   | Multiple strings | None     | Filter by event types (comma-separated or space-separated)      |
| `--player-name`   | String           | None     | Filter events by a specific player name                         |
| `--min-lat`       | Integer          | 45470259 | Minimum latitude (in microdegrees, E6 format)                   |
| `--min-lng`       | Integer          | 12244155 | Minimum longitude (in microdegrees, E6 format)                  |
| `--max-lat`       | Integer          | 45480370 | Maximum latitude (in microdegrees, E6 format)                   |
| `--max-lng`       | Integer          | 12298207 | Maximum longitude (in microdegrees, E6 format)                  |
| `--min-timestamp` | Timestamp        | -1       | Minimum timestamp (milliseconds since epoch or ISO 8601 format) |
| `--max-timestamp` | Timestamp        | -1       | Maximum timestamp (milliseconds since epoch or ISO 8601 format) |

#### Timestamp Filtering

The `--min-timestamp` and `--max-timestamp` options support two formats:

**Format 1: Milliseconds (integer)**
```bash
python main.py --min-timestamp 1736659200000 --max-timestamp 1736745600000
```

**Format 2: ISO 8601 datetime string**
```bash
python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"
```

**Examples:**

```bash
# Get events from last hour
python main.py --min-timestamp 1736659200000

# Get events for a specific day
python main.py --min-timestamp "2026-01-12T00:00:00Z" --max-timestamp "2026-01-13T00:00:00Z"

# Get events from a specific time onwards
python main.py --min-timestamp "2026-01-12T10:00:00Z"

# Get events up to a specific time
python main.py --max-timestamp "2026-01-12T12:00:00Z"
```

#### Event Type Filtering

```bash
# Filter by single event type
python main.py --event-types PORTAL_CAPTURED

# Filter by multiple event types (space-separated)
python main.py --event-types PORTAL_CAPTURED LINK_CREATED CONTROL_FIELD_CREATED

# Filter by resonator events
python main.py --event-types RESONATOR_DEPLOYED RESONATOR_DESTROYED
```

#### Player Filtering

```bash
# Filter by player name
python main.py --player-name Albicocca

# Combine with event type filter
python main.py --player-name Albicocca --event-types PORTAL_CAPTURED
```

#### Geographic Filtering

Coordinates are in **E6 format** (microdegrees): multiply decimal degrees by 1,000,000.

```bash
# Custom geographic bounds
python main.py --min-lat 45470000 --max-lat 45480000 --min-lng 12240000 --max-lng 12300000
```

#### Combining Filters

```bash
# Complex query with multiple filters
python main.py \
  --min-timestamp "2026-01-12T10:00:00Z" \
  --max-timestamp "2026-01-12T12:00:00Z" \
  --event-types PORTAL_CAPTURED LINK_CREATED \
  --player-name Albicocca \
  --min-lat 45470000 \
  --max-lat 45480000
```

### Web API

The web API provides a REST interface for programmatic access to Ingress events.

#### Starting the Web API

```bash
# Development mode
python app.py

# Production mode with Gunicorn
gunicorn -w 4 -b 0.0.0.0:7000 app:app

# Or using the entrypoint script
./entrypoint-web.sh
```

#### API Endpoints

##### GET /

Returns API information and available endpoints.

**Response:**
```json
{
  "name": "Ingress Intel API",
  "version": "1.0.0",
  "endpoints": {
    "/plexts": {
      "method": "GET",
      "description": "Get plexts from the Ingress API",
      "parameters": {
        "event_types": "List of event types to filter by (comma-separated)",
        "player_name": "Player name to filter by",
        "min_lat": "Minimum latitude (default: 45470259)",
        "min_lng": "Minimum longitude (default: 12244155)",
        "max_lat": "Maximum latitude (default: 45480370)",
        "max_lng": "Maximum longitude (default: 12298207)",
        "min_timestamp": "Minimum timestamp (milliseconds or ISO 8601 format, default: -1)",
        "max_timestamp": "Maximum timestamp (milliseconds or ISO 8601 format, default: -1)"
      },
      "event_types": [
        "RESONATOR_DEPLOYED",
        "RESONATOR_DESTROYED",
        "PORTAL_CAPTURED",
        "PORTAL_NEUTRALIZED",
        "PORTAL_UNDER_ATTACK",
        "LINK_CREATED",
        "LINK_DESTROYED",
        "CONTROL_FIELD_CREATED",
        "UNKNOWN"
      ]
    }
  }
}
```

##### GET /plexts

Get plexts from the Ingress API with optional filters.

**Query Parameters:**

| Parameter       | Type    | Default  | Description                                         |
| --------------- | ------- | -------- | --------------------------------------------------- |
| `event_types`   | String  | None     | Comma-separated list of event types to filter by    |
| `player_name`   | String  | None     | Player name to filter by                            |
| `min_lat`       | Integer | From env | Minimum latitude (E6 format)                        |
| `min_lng`       | Integer | From env | Minimum longitude (E6 format)                       |
| `max_lat`       | Integer | From env | Maximum latitude (E6 format)                        |
| `max_lng`       | Integer | From env | Maximum longitude (E6 format)                       |
| `min_timestamp` | String  | -1       | Minimum timestamp (milliseconds or ISO 8601 format) |
| `max_timestamp` | String  | -1       | Maximum timestamp (milliseconds or ISO 8601 format) |

**Response:**
```json
{
  "count": 2,
  "plexts": [
    {
      "id": "abc123",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "Albicocca captured L' Arboreto",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "Albicocca",
      "portal_name": "L' Arboreto",
      "coordinates": {
        "lat": 45471652,
        "lng": 12274703
      },
      "markup": [
        {
          "type": "PLAYER",
          "plain": "Albicocca",
          "team": "RESISTANCE",
          "name": "Albicocca",
          "address": "",
          "latE6": 0,
          "lngE6": 0
        },
        {
          "type": "PORTAL",
          "plain": "L' Arboreto",
          "team": "RESISTANCE",
          "name": "L' Arboreto",
          "address": "Via Arboreto, Venice, Italy",
          "latE6": 45471652,
          "lngE6": 12274703
        }
      ]
    }
  ]
}
```

**Examples:**

```bash
# Get all plexts
curl "http://localhost:7000/plexts"

# Filter by event type
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED,LINK_CREATED"

# Filter by player
curl "http://localhost:7000/plexts?player_name=Albicocca"

# Filter by time range
curl "http://localhost:7000/plexts?min_timestamp=2026-01-12T10:00:00Z&max_timestamp=2026-01-12T12:00:00Z"

# Combine filters
curl "http://localhost:7000/plexts?event_types=PORTAL_CAPTURED&player_name=Albicocca&min_timestamp=2026-01-12T10:00:00Z"
```

##### GET /plexts/from-db

Get plexts from MongoDB with optional filters. Returns plexts sorted by timestamp (most recent first).

**Query Parameters:**

| Parameter       | Type    | Default  | Description                                         |
| --------------- | ------- | -------- | --------------------------------------------------- |
| `player_name`   | String  | None     | Player name to filter by                            |
| `timestamp_from` | Integer | None     | Minimum timestamp in milliseconds (optional)            |
| `timestamp_to`   | Integer | None     | Maximum timestamp in milliseconds (optional)            |
| `limit`         | Integer | 100       | Maximum number of results to return                  |
| `center`        | String  | None     | Geographic center point in format "latitude,longitude" (optional, requires radius) |
| `radius`        | Integer | None     | Search radius in meters (optional, requires center) |

**Geospatial Query:**

When both `center` and `radius` parameters are provided, the endpoint returns plexts within the specified radius from the center point. The response includes a `distance` field (in meters) for each result.

**Response with geospatial query:**
```json
{
  "count": 5,
  "plexts": [
    {
      "id": "abc123",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "Albicocca captured L' Arboreto",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "Albicocca",
      "portal_name": "L' Arboreto",
      "coordinates": {
        "type": "Point",
        "coordinates": [12.365208, 45.573661]
      },
      "distance": 125.5,
      "markup": [...]
    }
  ]
}
```

**Note:** Both `center` and `radius` parameters must be provided together. If neither is provided, the endpoint returns all plexts matching other filters without geospatial filtering.

**Examples:**

```bash
# Get all plexts from MongoDB
curl "http://localhost:7000/plexts/from-db"

# Filter by player name
curl "http://localhost:7000/plexts/from-db?player_name=Albicocca"

# Filter by time range
curl "http://localhost:7000/plexts/from-db?timestamp_from=17366592000000&timestamp_to=1736745600000"

# Geospatial query: Get plexts within 1km of a specific location
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000"

# Geospatial query with player filter
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&player_name=Albicocca"

# Geospatial query with time filter
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&timestamp_from=17366592000000"

# Combine all filters
curl "http://localhost:7000/plexts/from-db?center=45.573661,12.365208&radius=1000&player_name=Albicocca&timestamp_from=17366592000000&limit=10"
```

**Authentication:** Basic Auth required

### Scheduler

The scheduler automatically collects Ingress events every minute and stores them in MongoDB.

#### Starting the Scheduler

```bash
# Start the scheduler
python schedule.py

# Or using the entrypoint script
./entrypoint-schedule.sh
```

#### How It Works

1. The scheduler runs every minute using APScheduler
2. It calculates a time range for the previous minute
3. Fetches plexts from the web API endpoint
4. Stores the results in MongoDB
5. Logs the number of plexts collected

#### Configuration

The scheduler requires the following environment variables:

- `ENDPOINT_URL` - The web API endpoint to fetch plexts from
- `MONGO_URI` - MongoDB connection string
- `DB_NAME` - Database name
- `COLLECTION_NAME` - Collection name
- `TZ` - Timezone for time calculations

#### Example Output

```
[2026-01-12 11:00:00 CET] Starting job...
Time range: 2026-01-12T10:59:00+01:00 to 2026-01-12T11:00:00+01:00
Fetched 5 plexts
Inserted 5 plexts to MongoDB
Job completed successfully
```

### Docker Deployment

#### Using Docker Compose

The project includes a [`docker-compose.yml`](docker-compose.yml:1) file for easy deployment.

**Services:**

1. **MongoDB** - Database service
   - Image: `mongo:7.0`
   - Port: `27017`
   - Health checks enabled
   - Persistent volumes for data

2. **Web** - Flask web API
   - Image: `mrosati84/ingress-web:1.0`
   - Port: `7000`
   - Depends on MongoDB
   - Health checks enabled

3. **Schedule** - Scheduler service
   - Image: `mrosati84/ingress-schedule:1.0`
   - Depends on MongoDB and Web
   - No exposed ports

**Commands:**

```bash
# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# View logs for specific service
docker-compose logs -f web

# Stop all services
docker-compose down

# Stop and remove volumes
docker-compose down -v

# Restart a service
docker-compose restart web

# Scale services
docker-compose up -d --scale web=3
```

#### Building Custom Images

If you want to build your own Docker images:

```bash
# Build web image
docker build -f Dockerfile-web -t ingress-web .

# Build schedule image
docker build -f Dockerfile-schedule -t ingress-schedule .

# Update docker-compose.yml to use your images
```

#### Health Checks

All services include health checks:

- **MongoDB**: Database ping test
- **Web**: HTTP GET request to root endpoint
- **Schedule**: No health check (depends on other services)

## Event Types

| Event Type              | Description                        |
| ----------------------- | ---------------------------------- |
| `RESONATOR_DEPLOYED`    | Player deploys resonator on portal |
| `RESONATOR_DESTROYED`   | Resonator destroyed                |
| `PORTAL_CAPTURED`       | Portal captured by player          |
| `PORTAL_NEUTRALIZED`    | Portal neutralized                 |
| `PORTAL_UNDER_ATTACK`   | Portal being attacked              |
| `LINK_CREATED`          | Link created between portals       |
| `LINK_DESTROYED`        | Link destroyed                     |
| `CONTROL_FIELD_CREATED` | Control field created              |
| `UNKNOWN`               | Unrecognized event type            |

## Output Format

### CLI Output

Events are printed in the following format:

```
[2026-01-12 10:28:27] [PORTAL_CAPTURED] Albicocca captured L' Arboreto - Coords: 45471652, 12274703
[2026-01-12 10:28:26] [LINK_CREATED] Albicocca linked from L' Arboreto to Parco San Giuliano - Coords: 45471652, 12274703
```

### API Output

The API returns JSON with the following structure:

```json
{
  "count": 2,
  "plexts": [
    {
      "id": "string",
      "timestamp": 1736659207000,
      "timestamp_formatted": "2026-01-12 11:00:07",
      "text": "string",
      "team": "RESISTANCE",
      "plext_type": "PLAYER_GENERATED",
      "categories": 0,
      "event_type": "PORTAL_CAPTURED",
      "player_name": "string",
      "portal_name": "string",
      "coordinates": {
        "lat": 45471652,
        "lng": 12274703
      },
      "markup": [...]
    }
  ]
}
```

## Project Structure

```
ingress/
├── main.py                    # CLI entry point
├── app.py                     # Flask web API
├── ingress.py                 # Ingress API client
├── models.py                  # Data models (Plext, EventType, Markup)
├── schedule.py                # Scheduled data collector
├── pyproject.toml             # Project configuration (uv)
├── requirements.txt           # Python dependencies
├── docker-compose.yml         # Docker orchestration
├── Dockerfile-web             # Web service Docker image
├── Dockerfile-schedule        # Scheduler Docker image
├── entrypoint-web.sh          # Web service entrypoint script
├── entrypoint-schedule.sh     # Scheduler entrypoint script
├── json_doc.md                # API documentation
├── .python-version            # Python version specification
├── .gitignore                 # Git ignore rules
├── .env.example               # Environment variables template
└── README.md                  # This file
```

## Dependencies

### Core Dependencies

- **Python 3.11+** (Docker images use Python 3.11)
- **requests>=2.31.0** - HTTP client for API requests
- **flask>=3.1.2** - Web framework for REST API
- **gunicorn>=23.0.0** - Production WSGI server
- **python-dotenv>=1.2.1** - Environment variable management
- **pymongo>=4.10.0** - MongoDB driver
- **apscheduler>=3.10.0** - Task scheduling
- **ipython>=9.9.0** - Interactive Python shell (development)

### Development Dependencies

- **uv** - Fast Python package installer (optional)

## Development

### Local Development Setup

```bash
# Install dependencies
pip install -r requirements.txt

# Or using uv
uv sync

# Set up environment variables
cp .env.example .env
# Edit .env with your configuration

# Start MongoDB (if not using Docker)
# Install MongoDB locally or use a cloud instance

# Start the web API
python app.py

# In another terminal, start the scheduler
python schedule.py

# Test the CLI
python main.py --help
```

### Running Tests

```bash
# Run CLI tests
python main.py --event-types PORTAL_CAPTURED --player-name test

# Test API endpoints
curl http://localhost:7000/
curl http://localhost:7000/plexts
```

### Debugging

The application includes comprehensive logging:

- **CLI**: Logs to stdout
- **Web API**: Logs to stdout (Gunicorn captures and forwards)
- **Scheduler**: Logs to stdout
- **Ingress API Client**: Detailed DEBUG level logging

To enable debug logging:

```bash
# Set log level in environment
export LOG_LEVEL=DEBUG

# Or modify the logging configuration in the source files
```

## Troubleshooting

### Common Issues

**Issue: "No CSRF token found in cookie!"**
- Solution: Ensure your `INGRESS_COOKIE` includes both `csrftoken` and `sessionid`

**Issue: "MongoDB connection failed"**
- Solution: Check that MongoDB is running and `MONGO_URI` is correct

**Issue: "Scheduler not collecting data"**
- Solution: Verify `ENDPOINT_URL` is accessible and web API is running

**Issue: "Docker containers not starting"**
- Solution: Check Docker logs: `docker-compose logs`

**Issue: "Health check failing"**
- Solution: Ensure all dependencies are running and ports are accessible

### Debug Mode

Enable debug mode for more detailed logging:

```bash
# For web API
export FLASK_DEBUG=1
python app.py

# For scheduler
export APSCHEDULER_DEBUG=1
python schedule.py
```

### Logs

View logs for different components:

```bash
# Docker logs
docker-compose logs -f

# Specific service logs
docker-compose logs -f web
docker-compose logs -f schedule
docker-compose logs -f mongodb

# Application logs (if running locally)
# Logs are printed to stdout
```

## License

See project license file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Support

For issues and questions, please open an issue on the project repository.