OSM Notes API

title

description

version

last_updated

author

OSM Notes API

REST API for OSM Notes Analytics and Ingestion. Unified programmatic access to user profiles, country analytics, advanced search capabilities, rankings, comparisons, and real-time metrics. Extends OSM API 0.6 with specialized analytics features.

📋 Description

OSM Notes API provides programmatic access to OpenStreetMap notes analytics data, including:

User Profiles: Detailed statistics of contributors
Country Analytics: Aggregated metrics by country
Advanced Search: Complex filters and dynamic queries
Rankings: User and country classifications
Comparisons: Comparative analysis between entities
Trends: Temporal data analysis
Notes and Comments: Access to OSM notes and their comments

⚠️ Important Note

This API is COMPLEMENTARY to the static JSON system, NOT a replacement.

✅ JSON system maintained: The Viewer and other consumers continue using static JSON files
✅ API is additional: For use cases requiring dynamic queries or integrations
✅ Both coexist: Each system is used according to the specific use case

Requirements

For complete installation instructions, see Installation and Dependencies Guide.

Application Requirements

Node.js >= 18.0.0
npm >= 9.0.0
PostgreSQL 15+ (with access to osm_notes_dwh database)
Redis 7+ (optional but recommended for caching and rate limiting)

Internal Repository Requirements

OSM-Notes-Analytics ⚠️ REQUIRED
- This API reads from the Analytics data warehouse (osm_notes_dwh database)
- Required schemas: dwh (data warehouse with datamarts)
- Install Analytics FIRST before installing API

🚀 Quick Start

Installation

# Clone repository
git clone https://github.com/OSM-Notes/OSM-Notes-API.git
cd OSM-Notes-API

# Install dependencies
npm install

# Configure environment variables
cp .env.example .env
# Edit .env with your configurations

# Build TypeScript
npm run build

# Load mock data (optional, for testing)
# Requires OSM-Notes-Ingestion repository cloned
./scripts/load_mock_data.sh

# Start application
npm start

With Docker

# Start services
docker compose -f docker/docker-compose.yml up -d

# View logs
docker compose -f docker/docker-compose.yml logs -f api

See docs/INSTALLATION.md for detailed instructions.

Quick Usage Examples

Health Check:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     http://localhost:3000/health

Get a Note:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     http://localhost:3000/api/v1/notes/12345

Search Notes:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     "http://localhost:3000/api/v1/notes?status=open&country=42&limit=10"

Get User Profile:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     http://localhost:3000/api/v1/users/12345

Get Country Profile:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     http://localhost:3000/api/v1/countries/42

Get Global Analytics:

curl -H "User-Agent: MyApp/1.0 ([email protected])" \
     http://localhost:3000/api/v1/analytics/global

Interactive API Documentation:

Swagger UI: http://localhost:3000/docs
OpenAPI JSON: http://localhost:3000/docs/json

See docs/USAGE.md for complete usage guide.

Entry Points

Main entry points for using and operating the API:

Application Entry Points

Start the API Server

npm start              # Production mode (requires build first)
npm run dev            # Development mode with hot reload

Docker Entry Points

# Start all services (API + PostgreSQL + Redis)
docker compose -f docker/docker-compose.yml up -d

# Start with monitoring (Prometheus + Grafana)
docker compose -f docker/docker-compose.yml --profile monitoring up -d

API Endpoints (HTTP Entry Points)

Base URL: http://localhost:3000 (default)

Health Check: GET /health - Service health status
API Documentation: GET /docs - Swagger UI interactive documentation
OpenAPI Spec: GET /docs/json - OpenAPI JSON specification

Main API Endpoints:

GET /api/v1/users/:id - Get user profile
GET /api/v1/countries/:id - Get country profile
GET /api/v1/notes - Search notes
GET /api/v1/notes/:id - Get specific note
GET /api/v1/analytics/global - Global analytics

See docs/API.md for complete API reference.

Development Entry Points

Build and Type Check

npm run build          # Compile TypeScript
npm run type-check     # Verify types without building

Testing

npm test               # Run all tests (master test command)
npm run test:unit      # Unit tests only
npm run test:integration  # Integration tests only

Master Test Command: npm test - Executes all tests using Jest

Code Quality

npm run lint           # Check code style
npm run format         # Format code

Operations Entry Points

Health Monitoring
```
curl http://localhost:3000/health
```

View Logs

# Docker
docker compose -f docker/docker-compose.yml logs -f api

# PM2
pm2 logs osm-notes-api

Restart Service

# Docker
docker compose -f docker/docker-compose.yml restart api

# PM2
pm2 restart osm-notes-api

See docs/RUNBOOK.md for complete operations procedures.

📚 Documentation

Getting Started

Installation - Complete installation guide
Usage - API usage manual with examples
API Reference - Complete API reference documentation
API Reference (OpenAPI) - OpenAPI/Swagger documentation

Operations & Deployment

Deployment Guide - Production deployment instructions
Production Deployment (192.168.0.7) - Specific guide for production server
Operations Runbook - Operational procedures and runbook
Troubleshooting - Common issues and solutions
CI/CD Guide - Continuous Integration and Deployment

Performance & Monitoring

Performance Guide - Performance optimization and benchmarking
SLA/SLOs - Service Level Agreements and Objectives
Monitoring Guide - Prometheus and Grafana setup

Operations & Compliance

Disaster Recovery - Disaster recovery procedures
Backup Strategy - Backup and restore procedures
Capacity Planning - Capacity planning guide
Terms of Service - API terms of service
Privacy Policy - Privacy policy (GDPR compliant)

Development

Database Schema - Database schema and data requirements
Database User Setup - Create read-only database user
Redis Setup (Optional) - Redis configuration (optional but recommended)
Redis Troubleshooting - Fix Redis connection issues
Testing Guide - Testing documentation
Security Testing - OWASP ZAP security testing guide
Development Guide - Development guidelines
Contributing - Contributor guide
Architecture Decision Records - ADRs documenting technical decisions

Reference

Changelog - Change history
API Versioning - API versioning strategy

🏗️ Project Structure

OSM-Notes-API/
├── src/                    # Source code
│   ├── config/            # Configuration
│   ├── routes/            # API routes
│   ├── controllers/       # Controllers
│   ├── services/          # Business logic
│   ├── middleware/        # Custom middleware
│   ├── utils/             # Utilities
│   └── types/             # TypeScript types
├── tests/                  # Tests
│   ├── unit/              # Unit tests
│   ├── integration/       # Integration tests
│   └── load/              # Load tests
├── docs/                   # Documentation
│   ├── INSTALLATION.md    # Installation manual
│   ├── USAGE.md           # Usage manual
│   └── api/               # OpenAPI documentation
├── docker/                 # Docker configuration
└── package.json           # Dependencies and scripts

🛠️ Available Scripts

Development

npm run build          # Build TypeScript
npm start              # Run compiled application
npm run dev            # Development with hot reload
npm run type-check     # Verify TypeScript types

Testing

npm test               # Run all tests
npm run test:unit      # Unit tests only
npm run test:integration  # Integration tests only
npm run test:coverage  # Tests with coverage report
npm run test:watch     # Run tests in watch mode
npm run test:light     # Run unit tests with 1 worker (for performance)
npm run test:integration:single  # Run integration tests sequentially

Code Quality

npm run lint           # Run ESLint
npm run lint:fix       # Fix ESLint errors automatically
npm run format         # Format code with Prettier
npm run format:check   # Check code formatting

Performance & Database

# Run performance benchmarks
./scripts/run_benchmarks.sh

# Create database indexes
psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f scripts/create_indexes.sql

# Analyze query performance
psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f scripts/analyze_queries.sql

Load Testing (requires k6)

# Install k6 first (see tests/load/README.md)
k6 run tests/load/users.js
k6 run tests/load/notes.js
k6 run tests/load/analytics.js
k6 run tests/load/all-endpoints.js

🔒 Security

User-Agent required: All requests must include a valid User-Agent with format AppName/Version (Contact)
Rate Limiting: 50 requests/15min for anonymous users, 1000 req/hour for authenticated
Anti-abuse protection: Automatic blocking of known AIs and bots without OAuth
Security headers: Helmet.js configured with security best practices
Input validation: All inputs validated with Joi schemas
Optional OAuth: Available for advanced features (Phase 5)

See docs/USAGE.md and docs/security/ for more security details.

📈 Monitoring & Observability

Prometheus Metrics: Available at /metrics endpoint
Grafana Dashboards: Pre-configured dashboards for:
- Request rate and latency (P50, P95, P99)
- Error rates by endpoint
- Rate limiting events
- User-Agent statistics
Health Checks: /health endpoint with database and Redis status
Structured Logging: JSON logs with Winston
Alerting: Prometheus alerts configured for:
- High error rates
- High latency
- Frequent rate limiting
- Service downtime

See docs/MONITORING.md for monitoring setup.

⚡ Performance

Response Times:
- Simple endpoints (datamarts): P95 < 500ms
- Complex analytics: P95 < 2000ms
Caching: Redis-based response caching with configurable TTL
Database Optimization:
- Indexes created via scripts/create_indexes.sql
- Query analysis via scripts/analyze_queries.sql
Benchmarks: Performance benchmarks script available (scripts/run_benchmarks.sh)

See docs/PERFORMANCE.md for performance optimization guide.

📊 Project Status

Version: 0.1.0 (Production Ready - Phases 1-4 Complete)

Test Coverage: 87.54% statements, 93.93% branches, 87.56% functions

✅ 224+ tests passing (100% pass rate)
✅ 25+ test suites passing
✅ Unit tests: 100% pass rate
✅ Integration tests: 100% pass rate
✅ CI/CD pipeline configured with GitHub Actions
✅ Load testing scripts (k6) available

Implementation Phases:

✅ Phase 1: MVP (Core endpoints, basic features, documentation)
- ✅ Notes endpoints (get, search, comments)
- ✅ User profiles endpoint
- ✅ Country profiles endpoint
- ✅ Global analytics endpoint
- ✅ User-Agent validation
- ✅ Rate limiting
- ✅ Anti-abuse protection
- ✅ OpenAPI/Swagger documentation
- ✅ Comprehensive integration tests
- ✅ Test database configuration
✅ Phase 2: Basic Features (Rankings, search, caching)
- ✅ User and country rankings
- ✅ Basic search functionality
- ✅ Pagination support
- ✅ Redis caching implementation
- ✅ Prometheus metrics
✅ Phase 3: Advanced Features (Advanced search, monitoring)
- ✅ Advanced search with multiple filters
- ✅ Hashtags endpoints
- ✅ Analytics comparisons
- ✅ Trends analysis
- ✅ Grafana dashboards
- ✅ Prometheus alerts
- ✅ Load testing scripts (k6)
✅ Phase 4: Production (Documentation, operations, compliance)
- ✅ CI/CD pipeline configured
- ✅ Complete documentation (installation, usage, API reference)
- ✅ Operations runbook
- ✅ Disaster recovery plan
- ✅ Backup strategy
- ✅ Capacity planning
- ✅ SLA/SLOs defined
- ✅ Legal documentation (Terms of Service, Privacy Policy)
- ✅ Performance benchmarks
- ✅ Performance optimization scripts
⏳ Phase 5: Webhooks and Notifications (Future - if needed)
- ⏳ Webhook system
- ⏳ Event subscriptions
- ⏳ OAuth integration

🤝 Contributing

Contributions are welcome. Please read CONTRIBUTING.md for details on our code of conduct and pull request process.

GDPR Compliance

Important: This API processes and serves data that may contain personal information from OpenStreetMap, including usernames and geographic locations. We are committed to GDPR compliance.

This API has its own Privacy Policy that covers API-specific GDPR compliance. Additionally, GDPR compliance mechanisms are fully implemented in the base project OSM-Notes-Ingestion. For comprehensive GDPR documentation, including detailed procedures and SQL scripts for handling data subject requests, please refer to:

GDPR Privacy Policy: OSM-Notes-Ingestion/docs/GDPR_Privacy_Policy.md
GDPR Procedures: OSM-Notes-Ingestion/docs/GDPR_Procedures.md - Detailed procedures for handling data subject requests
GDPR SQL Scripts: OSM-Notes-Ingestion/sql/gdpr/README.md - SQL scripts to handle GDPR requests

This API reads data from OSM-Notes-Analytics (which originates from OSM-Notes-Ingestion) and is subject to the same GDPR compliance measures.

📝 License

This project is licensed under the GNU General Public License v3.0 - see LICENSE for details.

📚 Ecosystem Documentation

For shared documentation of the complete ecosystem, see:

OSM Notes Ecosystem - Ecosystem landing page
Global Glossary - Terms and definitions
Complete Installation Guide - Step-by-step installation of all projects
End-to-End Data Flow - Complete data flow
Decision Guide - Which project do I need?

🌐 OSM-Notes Ecosystem

This API project is part of the OSM-Notes ecosystem, consisting of 8 interconnected projects. OSM-Notes-Ingestion is the base project - it was the first created and provides the foundation for all others.

Ecosystem Projects

OSM-Notes-Ingestion - Base project
- Downloads and synchronizes OSM notes from Planet and API
- Populates base PostgreSQL tables
- First project created, foundation for all others
OSM-Notes-Analytics
- ETL processes and data warehouse
- Generates analytics and datamarts
- Requires: OSM-Notes-Ingestion (reads from base tables)
- This API reads from: Analytics data warehouse (osm_notes_dwh)
OSM-Notes-API (this project)
- REST API for programmatic access
- Provides dynamic queries and advanced features
- Requires: OSM-Notes-Analytics (reads from data warehouse)
- Complementary to: Static JSON system (OSM-Notes-Data)
OSM-Notes-Viewer
- Web application for interactive visualization
- Consumes JSON data from OSM-Notes-Data (GitHub Pages)
- Requires: OSM-Notes-Data (which is generated by OSM-Notes-Analytics)
- Alternative: Can also use this API for dynamic queries
OSM-Notes-WMS
- Web Map Service for geographic visualization
- Publishes WMS layers for mapping applications
- Requires: OSM-Notes-Ingestion (uses same database)
OSM-Notes-Monitoring
- Centralized monitoring and alerting
- Monitors all ecosystem components including this API
- Requires: Access to all other projects' databases/services
OSM-Notes-Common
- Shared Bash libraries and utilities
- Used as Git submodule by multiple projects
- Used by: Ingestion, Analytics, WMS, Monitoring
- Note: This API project (Node.js/TypeScript) does not use Common
OSM-Notes-Data
- JSON data files exported from Analytics
- Served via GitHub Pages
- Requires: OSM-Notes-Analytics (generates and publishes the data)
- Consumed by: Viewer (primary consumer), API (optional - can use Analytics DWH directly)

Project Relationships

graph TB
    subgraph External["External Sources"]
        OSM[OSM Planet/API]
    end
    
    subgraph Base["Base Project"]
        INGESTION[OSM-Notes-Ingestion<br/>Base project]
    end
    
    subgraph Processing["Processing Layer"]
        ANALYTICS[OSM-Notes-Analytics<br/>ETL → Data Warehouse]
        WMS[OSM-Notes-WMS<br/>WMS layers]
    end
    
    subgraph Delivery["Delivery Layer"]
        DATA[OSM-Notes-Data<br/>JSON files<br/>GitHub Pages]
        API[OSM-Notes-API<br/>REST API<br/>this project<br/>reads from Analytics DWH]
        VIEWER[OSM-Notes-Viewer<br/>Consumes JSON from Data]
    end
    
    subgraph Support["Support Layer"]
        MONITORING[OSM-Notes-Monitoring<br/>Monitors all projects<br/>including this API]
        COMMON[OSM-Notes-Common<br/>Shared libraries<br/>submodule<br/>not used by this API]
    end
    
    OSM -->|Downloads| INGESTION
    INGESTION -->|Base Tables| ANALYTICS
    INGESTION -->|Same Database| WMS
    ANALYTICS -->|JSON Export| DATA
    ANALYTICS -->|Data Warehouse| API
    DATA -->|JSON Files| VIEWER
    MONITORING -.->|Monitors| INGESTION
    MONITORING -.->|Monitors| ANALYTICS
    MONITORING -.->|Monitors| API
    COMMON -.->|Used by| INGESTION
    COMMON -.->|Used by| ANALYTICS
    COMMON -.->|Used by| WMS
    COMMON -.->|Used by| MONITORING
    
    style OSM fill:#ADD8E6
    style INGESTION fill:#90EE90
    style ANALYTICS fill:#FFFFE0
    style WMS fill:#FFE4B5
    style DATA fill:#E0F6FF
    style API fill:#FFB6C1
    style VIEWER fill:#DDA0DD
    style MONITORING fill:#F0E68C
    style COMMON fill:#D3D3D3

Installation Order

When setting up the complete ecosystem, install projects in this order:

OSM-Notes-Ingestion - Install first (base project)
OSM-Notes-Analytics - Requires Ingestion
OSM-Notes-WMS - Requires Ingestion
OSM-Notes-Data - Requires Analytics (auto-generated by Analytics export script)
OSM-Notes-Viewer - Requires Data (consumes JSON from GitHub Pages)
OSM-Notes-API (this project) - Requires Analytics (reads from Analytics data warehouse)
OSM-Notes-Monitoring - Requires all others (monitors them)
OSM-Notes-Common - Used as submodule (no installation needed, not used by this API)

API vs Static JSON

This API is complementary to the static JSON system:

Static JSON (OSM-Notes-Data): Pre-computed, fast, served via GitHub Pages
- Used by: Viewer (primary), simple integrations
- Advantages: No server needed, fast loading, CDN-ready
REST API (this project): Dynamic queries, real-time data, advanced features
- Used by: Applications requiring dynamic queries, integrations, real-time data
- Advantages: Flexible queries, filtering, sorting, pagination, real-time updates

Both systems coexist and serve different use cases.

External Resources

OpenStreetMap: https://www.openstreetmap.org

📧 Contact

For questions or support, please open an issue on GitHub.

Note: This project is part of the OSM Notes ecosystem and is designed to work together with the other projects in the ecosystem.

Name		Name	Last commit message	Last commit date
Latest commit History 109 Commits
.github		.github
.husky		.husky
.trunk		.trunk
docker		docker
docs		docs
logs		logs
scripts		scripts
src		src
tests		tests
.commitlintrc.json		.commitlintrc.json
.editorconfig		.editorconfig
.env.example		.env.example
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.nvmrc		.nvmrc
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
.python-version		.python-version
.shfmt.toml		.shfmt.toml
.sqlfluff		.sqlfluff
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
jest.config.js		jest.config.js
nodemon.json		nodemon.json
package-lock.json		package-lock.json
package.json		package.json
pyproject.toml		pyproject.toml
test-endpoints.sh		test-endpoints.sh
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json

License

OSM-Notes/OSM-Notes-API

Folders and files

Latest commit

History

Repository files navigation

OSM Notes API

📋 Description

⚠️ Important Note

Requirements

Application Requirements

Internal Repository Requirements

🚀 Quick Start

Installation

With Docker

Quick Usage Examples

Recommended Reading Path

For API Consumers

For Developers

For System Administrators

Entry Points

Application Entry Points

API Endpoints (HTTP Entry Points)

Development Entry Points

Operations Entry Points

📚 Documentation

Getting Started

Operations & Deployment

Performance & Monitoring

Operations & Compliance

Development

Reference

🏗️ Project Structure

🛠️ Available Scripts

Development

Testing

Code Quality

Performance & Database

Load Testing (requires k6)

🔒 Security

📈 Monitoring & Observability

⚡ Performance

📊 Project Status

🤝 Contributing

GDPR Compliance

📝 License

📚 Ecosystem Documentation

🌐 OSM-Notes Ecosystem

Ecosystem Projects

Project Relationships

Installation Order

API vs Static JSON

External Resources

📧 Contact

About

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages