OSM-Notes-Monitoring

title

description

version

last_updated

author

OSM-Notes-Monitoring

Centralized Monitoring, Alerting, and API Security for OpenStreetMap Notes

This repository provides centralized monitoring, alerting, and security for the entire OSM Notes ecosystem. It monitors all components, provides unified dashboards, and protects the API against abuse and attacks.

Overview

OSM-Notes-Monitoring is the operational command center for the OSM Notes ecosystem, providing:

Centralized Monitoring: Single dashboard for all OSM Notes repositories
Unified Alerting: Email, Slack, and other alert channels
API Security: Rate limiting, DDoS protection, and abuse detection
Data Freshness: Monitor data freshness across all sources
Performance Tracking: Monitor performance metrics across all components
Security Monitoring: Detect and respond to security incidents

📚 Ecosystem Documentation

For shared documentation of the complete ecosystem, see:

OSM Notes Ecosystem - Ecosystem landing page
Global Glossary - Terms and definitions
Complete Installation Guide - Step-by-step installation of all projects
End-to-End Data Flow - Complete data flow
Decision Guide - Which project do I need?

OSM-Notes Ecosystem

This monitoring project is part of the OSM-Notes ecosystem, consisting of 8 interconnected projects. OSM-Notes-Ingestion is the base project - it was the first created and provides the foundation for all others. This Monitoring project monitors all other projects in the ecosystem.

Ecosystem Projects

OSM-Notes-Ingestion - Base project
- Downloads and synchronizes OSM notes from Planet and API
- Populates base PostgreSQL tables
- First project created, foundation for all others
OSM-Notes-Analytics
- ETL processes and data warehouse
- Generates analytics and datamarts
- Requires: OSM-Notes-Ingestion (reads from base tables)
OSM-Notes-API
- REST API for programmatic access
- Provides dynamic queries and advanced features
- Requires: OSM-Notes-Analytics (reads from data warehouse)
OSM-Notes-Viewer
- Web application for interactive visualization
- Consumes JSON data from OSM-Notes-Data (GitHub Pages)
- Requires: OSM-Notes-Data (which is generated by OSM-Notes-Analytics)
OSM-Notes-WMS
- Web Map Service for geographic visualization
- Publishes WMS layers for mapping applications
- Requires: OSM-Notes-Ingestion (uses same database)
OSM-Notes-Monitoring (this project)
- Centralized monitoring and alerting
- Monitors all ecosystem components
- Requires: Access to all other projects' databases/services
OSM-Notes-Common
- Shared Bash libraries and utilities
- Used as Git submodule by multiple projects
- Used by: Ingestion, Analytics, WMS, Monitoring (this project)
OSM-Notes-Data
- JSON data files exported from Analytics
- Served via GitHub Pages
- Requires: OSM-Notes-Analytics (generates and publishes the data)
- Consumed by: Viewer (primary consumer), API (optional)

Project Relationships

graph TB
    subgraph External["External Sources"]
        OSM[OSM Planet/API]
    end
    
    subgraph Base["Base Project"]
        INGESTION[OSM-Notes-Ingestion<br/>Base project]
    end
    
    subgraph Processing["Processing Layer"]
        ANALYTICS[OSM-Notes-Analytics<br/>ETL → Data Warehouse]
        WMS[OSM-Notes-WMS<br/>WMS layers]
    end
    
    subgraph Delivery["Delivery Layer"]
        DATA[OSM-Notes-Data<br/>JSON files<br/>GitHub Pages]
        API[OSM-Notes-API<br/>REST API]
        VIEWER[OSM-Notes-Viewer]
    end
    
    subgraph Support["Support Layer"]
        MONITORING[OSM-Notes-Monitoring<br/>Monitors all projects<br/>this project]
        COMMON[OSM-Notes-Common<br/>Shared libraries<br/>submodule<br/>used by Monitoring]
    end
    
    OSM -->|Downloads| INGESTION
    INGESTION -->|Base Tables| ANALYTICS
    INGESTION -->|Same Database| WMS
    ANALYTICS -->|JSON Export| DATA
    ANALYTICS -->|Data Warehouse| API
    DATA -->|JSON Files| VIEWER
    MONITORING -.->|Monitors| INGESTION
    MONITORING -.->|Monitors| ANALYTICS
    MONITORING -.->|Monitors| WMS
    MONITORING -.->|Monitors| API
    MONITORING -.->|Monitors| DATA
    MONITORING -.->|Monitors| VIEWER
    COMMON -.->|Used by| INGESTION
    COMMON -.->|Used by| ANALYTICS
    COMMON -.->|Used by| WMS
    COMMON -.->|Used by| MONITORING
    
    style OSM fill:#ADD8E6
    style INGESTION fill:#90EE90
    style ANALYTICS fill:#FFFFE0
    style WMS fill:#FFE4B5
    style DATA fill:#E0F6FF
    style API fill:#FFB6C1
    style VIEWER fill:#DDA0DD
    style MONITORING fill:#F0E68C
    style COMMON fill:#D3D3D3

Installation Order

When setting up the complete ecosystem, install projects in this order:

OSM-Notes-Ingestion - Install first (base project)
OSM-Notes-Analytics - Requires Ingestion
OSM-Notes-WMS - Requires Ingestion
OSM-Notes-Data - Requires Analytics (auto-generated by Analytics export script)
OSM-Notes-Viewer - Requires Data (consumes JSON from GitHub Pages)
OSM-Notes-API - Requires Analytics (reads from Analytics data warehouse)
OSM-Notes-Monitoring (this project) - Requires all others (monitors them) - Install last
OSM-Notes-Common - Used as submodule (no installation needed)

Monitored Components

This system monitors the following repositories and components:

OSM-Notes-Ingestion: Data ingestion status, processing health, data quality
- Integrates with existing monitoring scripts: notesCheckVerifier.sh, processCheckPlanetNotes.sh, analyzeDatabasePerformance.sh
- Monitors: Database health, API sync status, Planet processing, data freshness
- See Existing Monitoring Components for details
OSM-Notes-Analytics: DWH/ETL job status, data freshness, query performance
- Monitors: ETL execution, datamart updates, data warehouse health, query performance
OSM-Notes-API: API availability, rate limiting, security incidents
- Monitors: API uptime, response times, rate limiting events, security threats
OSM-Notes-Viewer: Service availability, GitHub Pages deployment status
- Monitors: GitHub Pages availability, data loading performance
OSM-Notes-WMS: Service availability, response times, tile generation
- Monitors: GeoServer status, WMS layer availability, tile generation performance
OSM-Notes-Data: Backup freshness, repository sync status
- Monitors: JSON file freshness, GitHub Pages sync, export timestamps
OSM-Notes-Common: Submodule version consistency
- Monitors: Submodule version alignment across projects
Infrastructure: Server resources, database health, network connectivity
- Monitors: CPU, memory, disk usage, database connections, network latency

Requirements

Application Requirements

For complete installation instructions, see Installation and Dependencies Guide.

PostgreSQL (for metrics storage in monitoring database)
Bash 4.0 or higher
Standard UNIX utilities: grep, awk, sed, curl
Email client: mutt (for email alerts)
HTTP client: curl (for HTTP health checks)

Internal Repository Requirements

OSM-Notes-Ingestion ⚠️ REQUIRED
- Access to Ingestion database (notes) for monitoring data quality and sync status
OSM-Notes-Analytics ⚠️ REQUIRED
- Access to Analytics database (osm_notes_dwh or notes_dwh) for monitoring ETL status
OSM-Notes-API (optional but recommended)
- Access to API service for monitoring availability and performance
OSM-Notes-WMS (optional but recommended)
- Access to WMS/GeoServer for monitoring service availability
OSM-Notes-Data (optional but recommended)
- Access to GitHub repository for monitoring data freshness

Note: Monitoring can be installed with access to only some projects, but full monitoring requires access to all monitored components.

Architecture Decision Records

Architecture Decision Records (ADRs) document important architectural decisions made in this project:

Entry Points

Main entry points for using and operating Monitoring:

Monitoring Script Entry Points

Monitor Ingestion (bin/monitor/monitorIngestion.sh)

# Monitor OSM-Notes-Ingestion component
./bin/monitor/monitorIngestion.sh

Monitor Analytics (bin/monitor/monitorAnalytics.sh)

# Monitor OSM-Notes-Analytics component
./bin/monitor/monitorAnalytics.sh

Monitor API (bin/monitor/monitorAPI.sh)

# Monitor OSM-Notes-API component
./bin/monitor/monitorAPI.sh

Monitor WMS (bin/monitor/monitorWMS.sh)

# Monitor OSM-Notes-WMS component
./bin/monitor/monitorWMS.sh

Monitor Data (bin/monitor/monitorData.sh)

# Monitor OSM-Notes-Data freshness
./bin/monitor/monitorData.sh

Monitor Infrastructure (bin/monitor/monitorInfrastructure.sh)

# Monitor system infrastructure (CPU, memory, disk)
./bin/monitor/monitorInfrastructure.sh

Metric Collection Entry Points

Collect Database Metrics (bin/monitor/collectDatabaseMetrics.sh)

# Collect database performance metrics
./bin/monitor/collectDatabaseMetrics.sh

Collect ETL Metrics (bin/monitor/collect_etl_metrics.sh)

# Collect ETL job metrics from Analytics
./bin/monitor/collect_etl_metrics.sh

Collect System Metrics (bin/monitor/collectSystemMetrics.sh)

# Collect system-level metrics
./bin/monitor/collectSystemMetrics.sh

Alerting Entry Points

Alert Manager (bin/alerts/alertManager.sh)

# Manage and send alerts
./bin/alerts/alertManager.sh

Send Alert (bin/alerts/sendAlert.sh)

# Send individual alert
./bin/alerts/sendAlert.sh [severity] [message]

Alert Rules (bin/alerts/alertRules.sh)

# Configure and manage alert rules
./bin/alerts/alertRules.sh

Security Entry Points

Rate Limiter (bin/security/rateLimiter.sh)

# Manage rate limiting for API
./bin/security/rateLimiter.sh

IP Blocking (bin/security/ipBlocking.sh)

# Manage IP whitelist/blacklist
./bin/security/ipBlocking.sh

Abuse Detection (bin/security/abuseDetection.sh)

# Detect and respond to abuse patterns
./bin/security/abuseDetection.sh

DDoS Protection (bin/security/ddosProtection.sh)

# DDoS detection and mitigation
./bin/security/ddosProtection.sh

Dashboard Entry Points

Update Dashboard (bin/dashboard/updateDashboard.sh)

# Update monitoring dashboard with latest metrics
./bin/dashboard/updateDashboard.sh

Generate Metrics (bin/dashboard/generateMetrics.sh)

# Generate metrics for dashboard visualization
./bin/dashboard/generateMetrics.sh

Configuration Entry Point

Monitoring Properties: etc/properties.sh - Main configuration file

# Copy example and configure
cp etc/properties.sh.example etc/properties.sh
vi etc/properties.sh

Alert Configuration: config/alerts.conf - Alert settings

export ADMIN_EMAIL="[email protected]"
export SEND_ALERT_EMAIL="true"

See docs/CONFIGURATION_REFERENCE.md for complete configuration options.

Quick Start

Installation

Clone the repository:

git clone https://github.com/OSM-Notes/OSM-Notes-Monitoring.git
cd OSM-Notes-Monitoring

Configure monitoring:

cp etc/properties.sh.example etc/properties.sh
# Edit etc/properties.sh with your configuration

Set up monitoring database:

# Create monitoring database (this project's own database)
# Development: osm_notes_monitoring
# Production: notes_monitoring
createdb osm_notes_monitoring

# Run initialization scripts
psql -d osm_notes_monitoring -f sql/init.sql

# Grant permissions to monitoring user (required!)
# Replace 'osm_notes_monitoring_user' with your actual database user
psql -d osm_notes_monitoring -c "GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE ON TABLES TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT EXECUTE ON FUNCTIONS TO osm_notes_monitoring_user;"
psql -d osm_notes_monitoring -c "GRANT USAGE ON SCHEMA public TO osm_notes_monitoring_user;"

# Apply query performance optimizations (recommended)
psql -d osm_notes_monitoring -f sql/optimize_queries.sql

Note: This project requires:

Its own database (osm_notes_monitoring / notes_monitoring) to store metrics, alerts, and security events
Access to monitored databases (notes for ingestion, notes_dwh for analytics) to read data for monitoring
Configure INGESTION_DBNAME and ANALYTICS_DBNAME in etc/properties.sh if different from defaults

Configure alerts:

# Edit config/alerts.conf
export ADMIN_EMAIL="[email protected]"
export SEND_ALERT_EMAIL="true"

Architecture

For detailed architecture documentation, see:

Monitoring Architecture Proposal
API Security Design
Monitoring Resumen Ejecutivo (Spanish)
Existing Monitoring Components: Integration with OSM-Notes-Ingestion monitoring scripts
Grafana Architecture: Dual Grafana deployment (API + Monitoring)
Query Performance Optimization: SQL query optimization guide
Security Audit Guide: Code-level security checks
Vulnerability Scanning Guide: Automated vulnerability detection
Penetration Testing Guide: Security penetration testing procedures
Code Coverage Instrumentation: Using bashcov for real coverage measurement
Code Coverage Explanation: Understanding estimated vs instrumented coverage and how to improve it

Features

Monitoring

Component Health: Monitor all OSM Notes repositories
Data Quality: Track data integrity and freshness
Performance: Monitor response times and resource usage
Dependencies: Track cross-repository dependencies

Security

Rate Limiting: Per-IP, per-API-key, per-endpoint limits
DDoS Protection: Automatic detection and mitigation
Abuse Detection: Pattern analysis and automatic blocking
IP Management: Whitelist, blacklist, temporary blocks

Alerting

Email Alerts: Immediate notifications for critical issues
Slack Integration: Team notifications
Escalation: Automatic escalation for critical alerts
Alert Management: Deduplication and alert history

Dashboards

Grafana Dashboards: Professional time-series visualization
Custom Dashboards: HTML-based dashboards for quick checks
CLI Tools: Command-line dashboards

Project Structure

OSM-Notes-Monitoring/
├── bin/                    # Executable scripts
│   ├── monitor/           # Monitoring scripts per component
│   ├── security/          # Security scripts (rate limiting, DDoS)
│   ├── alerts/            # Alerting system
│   ├── dashboard/         # Dashboard generation
│   └── lib/               # Shared library functions
├── sql/                   # SQL monitoring queries
│   ├── ingestion/         # Ingestion monitoring queries
│   ├── analytics/         # Analytics monitoring queries
│   ├── wms/               # WMS monitoring queries
│   ├── api/               # API monitoring queries
│   ├── data/              # Data freshness queries
│   ├── infrastructure/    # Infrastructure queries
│   └── init.sql           # Database initialization script
├── config/                # Configuration files
│   ├── monitoring.conf.example    # Monitoring configuration template
│   ├── alerts.conf.example        # Alert configuration template
│   ├── security.conf.example      # Security configuration template
│   └── dashboards/                # Dashboard configuration files
│       ├── grafana/               # Grafana dashboard configs
│       └── custom/                # Custom dashboard configs
├── dashboards/            # Dashboard files (Grafana JSON, HTML)
│   ├── grafana/           # Grafana dashboard JSON files
│   └── html/              # HTML dashboard files
├── metrics/               # Metrics storage (runtime data)
│   ├── ingestion/         # Ingestion metrics
│   ├── analytics/         # Analytics metrics
│   ├── wms/               # WMS metrics
│   ├── api/               # API metrics
│   └── infrastructure/    # Infrastructure metrics
├── etc/                   # Main configuration
│   └── properties.sh.example  # Main properties template
├── logs/                  # Monitoring logs (runtime)
├── docs/                  # Documentation
└── tests/                 # Test suite
    ├── unit/              # Unit tests
    ├── integration/       # Integration tests
    └── mock_commands/     # Mock commands for testing

Note:

config/dashboards/ contains configuration files for dashboards
dashboards/ contains the actual dashboard files (Grafana JSON, HTML)
etc/properties.sh is the main system configuration file (copy from .example)
config/monitoring.conf contains monitoring-specific settings (copy from .example)
config/alerts.conf and config/security.conf are component-specific configurations (copy from .example)

Usage

Monitor Ingestion

./bin/monitor/monitorIngestion.sh

Monitor Analytics

./bin/monitor/monitorAnalytics.sh

Monitor API Security

./bin/security/rateLimiter.sh check 192.168.1.100 /api/notes

View Dashboard

# Start Grafana (if configured)
# Or view HTML dashboard
open dashboards/html/overview.html

Code Coverage

This project uses two complementary methods to measure code coverage:

Quick Coverage Check (Estimated)

bash scripts/generate_coverage_report.sh
# Fast: Shows estimated coverage based on test file presence
# Output: coverage/coverage_report.txt

Detailed Coverage Analysis (Instrumented)

# Run in background (takes hours)
bash scripts/run_bashcov_background.sh start

# Monitor progress
bash scripts/monitor_bashcov.sh

# Check status
bash scripts/run_bashcov_background.sh status
# Output: coverage/coverage_report_instrumented.txt

Combined Report (Both Side by Side)

bash scripts/generate_coverage_combined.sh
# Shows estimated vs instrumented coverage comparison
# Output: coverage/coverage_report_combined.txt

Understanding the Numbers:

Estimated (80%): "We have tests for 80% of scripts" - Fast, optimistic
Instrumented (27%): "27% of code lines are executed during tests" - Slow, accurate
Gap (53%): Normal for unit tests with mocks - indicates tests exist but don't execute full code paths

For detailed explanation, see: Code Coverage Explanation

Testing

Quick Start

# Run all tests (master test runner)
./tests/run_all_tests.sh

# Run specific test suites
./tests/run_all_tests.sh --unit-only          # Unit tests only
./tests/run_all_tests.sh --integration-only  # Integration tests only
./tests/run_all_tests.sh --e2e-only          # End-to-end tests only

Master Test Runner: tests/run_all_tests.sh - Executes all test suites (unit, integration, e2e, SQL, scripts, performance)

For detailed testing information, see the test documentation in the tests/ directory.

Configuration

Quick Setup

Generate configuration files:

# Interactive mode (recommended)
./scripts/generate_config.sh -i

# Or generate with defaults
./scripts/generate_config.sh -a

Configuration Files

Main Config: etc/properties.sh - Database, intervals, logging
Monitoring Config: config/monitoring.conf - Component-specific settings
Alert Config: config/alerts.conf - Email, Slack, alert routing
Security Config: config/security.conf - Rate limiting, DDoS protection
Log Rotation: config/logrotate.conf - Log rotation configuration

Documentation

For complete configuration reference, see:

Configuration Reference: All configuration options documented
Logging Guide: Logging setup, rotation, and best practices
Logging Best Practices: Comprehensive logging guidelines and patterns
Adapting Scripts: Guide for adapting existing scripts to use shared libraries

Documentation

Getting Started

Quick Start Guide: Get up and running in 15 minutes
User Guide: Comprehensive user documentation
Setup Guide: Detailed setup instructions
Documentation Index: Complete index of all documentation

Configuration and Usage

Configuration Reference: All configuration options
Dashboard Guide: Using dashboards
Alerting Guide: Alert system usage
Grafana Setup Guide: Grafana dashboard setup

Component Monitoring

Ingestion Monitoring Guide: Monitor ingestion component
Analytics Monitoring Guide: Monitor analytics/DWH component
WMS Monitoring Guide: Monitor WMS service
Infrastructure Monitoring Guide: Monitor infrastructure
API Security Guide: API security features

Architecture and Development

Architecture Proposal: Complete system architecture
API Security Design: Security and protection mechanisms
Implementation Plan: Detailed implementation plan with testing and standards
Coding Standards: Coding standards and best practices
Resumen Ejecutivo: Executive summary (Spanish)
Existing Monitoring Components: Integration with OSM-Notes-Ingestion monitoring scripts

Production Deployment

Deployment Guide: General production deployment guide
Pre-Deployment Checklist: Checklist before production deployment

Related Repositories

For complete information about the OSM-Notes ecosystem and all 8 projects, see the OSM-Notes Ecosystem section above.

All monitored repositories:

OSM-Notes-Ingestion - Data ingestion (base project)
OSM-Notes-Analytics - DWH and analytics
OSM-Notes-API - REST API
OSM-Notes-Viewer - Web visualization
OSM-Notes-WMS - WMS service
OSM-Notes-Data - JSON data files
OSM-Notes-Common - Shared libraries

Contributing

See CONTRIBUTING.md for contribution guidelines.

GDPR Compliance

Important: This monitoring system accesses databases and services that process personal data from OpenStreetMap, including usernames and geographic locations. We are committed to GDPR compliance.

GDPR compliance mechanisms are implemented in the base project OSM-Notes-Ingestion. For detailed information about GDPR compliance, data processing, retention, and data subject rights, please refer to:

GDPR Privacy Policy: OSM-Notes-Ingestion/docs/GDPR_Privacy_Policy.md
GDPR Procedures: OSM-Notes-Ingestion/docs/GDPR_Procedures.md - Procedures for handling data subject requests

This Monitoring project monitors all ecosystem components and accesses data that is subject to GDPR compliance measures implemented in OSM-Notes-Ingestion.

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

Status

✅ Production Ready - All phases completed. System ready for production deployment.

Author: Andres Gomez (AngocA)
Version: 1.0.0
Last Updated: 2026-01-09

Name		Name	Last commit message	Last commit date
Latest commit History 288 Commits
.git-hooks		.git-hooks
.github/workflows		.github/workflows
.trunk		.trunk
bin		bin
config		config
dashboards		dashboards
docs		docs
etc		etc
metrics		metrics
scripts		scripts
sql		sql
tests		tests
.cursorrules		.cursorrules
.editorconfig		.editorconfig
.gitignore		.gitignore
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
.shfmt.toml		.shfmt.toml
.sqlfluff		.sqlfluff
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md

License

OSM-Notes/OSM-Notes-Monitoring

Folders and files

Latest commit

History

Repository files navigation

OSM-Notes-Monitoring

Overview

📚 Ecosystem Documentation

OSM-Notes Ecosystem

Ecosystem Projects

Project Relationships

Installation Order

Monitored Components

Requirements

Application Requirements

Internal Repository Requirements

Recommended Reading Path

For System Administrators

For Developers

Architecture Decision Records

Entry Points

Monitoring Script Entry Points

Metric Collection Entry Points

Alerting Entry Points

Security Entry Points

Dashboard Entry Points

Configuration Entry Point

Quick Start

Installation

Architecture

Features

Monitoring

Security

Alerting

Dashboards

Project Structure

Usage

Monitor Ingestion

Monitor Analytics

Monitor API Security

View Dashboard

Code Coverage

Quick Coverage Check (Estimated)

Detailed Coverage Analysis (Instrumented)

Combined Report (Both Side by Side)

Testing

Quick Start

Configuration

Quick Setup

Configuration Files

Documentation

Documentation

Getting Started

Configuration and Usage

Component Monitoring

Architecture and Development

Production Deployment

Related Repositories

Contributing

GDPR Compliance

License

Status

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages