tap-ms-graph

This is a Singer tap that produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md

This tap:

• Pulls raw data from the [MS_Graph API].

• Extracts the following resources:

  • Applications

  • AuditLogsDirectory

  • AuditLogsSignins

  • CalendarEvents

  • Channel

  • ChatMessages

  • Chats

  • ConditionalAccessPolicies

  • Contacts

  • DirectoryRoleMember

  • DirectoryRoleTemplates

  • DirectoryRoles

  • DriveItems

  • Drives

  • GroupMember

  • GroupOwner

  • Groups

  • MailMessages

  • ServicePrincipals

  • TeamMember

  • Teams

  • Users

• Outputs the schema for each resource

• FULL_TABLE replication pulls all records from the source every time

Streams

applications

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

audit_logs_directory

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

audit_logs_signins

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

calendar_events

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

channel

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

chat_messages

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

chats

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

conditional_access_policies

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

contacts

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

directory_role_member

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

directory_role_templates

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

directory_roles

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

drive_items

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

drives

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

group_member

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

group_owner

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

groups

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

mail_messages

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

service_principals

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

team_member

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

teams

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

users

• Data Key = value
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

Prerequisite

To use Microsoft Graph APIs for sync and data access, ensure your tenant has the required licenses.

Graph API / Feature	Required License	Notes
Mail, Calendar, Contacts	M365 Business Basic or higher	Access via Mail.Read, Calendars.Read, etc.
Azure AD Directory Sync	Azure AD Premium P1 or P2	Needed for advanced directory features
Audit Logs (30–365 days)	AAD P1/P2 or M365 E5	For extended retention and compliance
Teams Messages (Chats, Channels)	M365 E5 + Microsoft approval	Protected APIs; app access needs justification
Webhooks, Change Notifications	Any M365 license	For real-time sync triggers
Data Sync APIs (Delta Queries)	Any M365 license	Available for users, groups, mail, etc.

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-ms-graph
   > pip install -e .

2. Dependent libraries. The following dependent libraries were installed.

   > pip install singer-python
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file. The tap config file for this tap should include these entries:

   • start_date - the default value to use if no bookmark exists for an endpoint (rfc3339 date string)
   • user_agent (string, optional): Process and email for API logging purposes. Example: tap-ms-graph <api_user_email@your_company.com>
   • request_timeout (integer, 300): Max time for which request should wait to get a response. Default request_timeout is 300 seconds.

   {
       "tenant_id": "AZURE_TENANT_ID",
       "client_id": "AZURE_CLIENT_ID",
       "client_secret": "AZURE_CLIENT_SECRET",
       "scope": "AZURE_SCOPE"
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "users",
       "bookmarks": {
           "groups": "2019-09-27T22:34:39.000000Z",
           "teams": "2019-09-28T15:30:26.000000Z",
       }
   }

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-ms-graph --config config.json --discover > catalog.json

   See the Singer docs on discovery mode [here](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md

5. Run the Tap in Sync Mode (with catalog) and [write out to state file](https://github.com/singer-io/getting-started/blob/master/docs/RUNNING_AND_DEVELOPING.md

   For Sync mode:

   > tap-ms-graph --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-ms-graph --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-ms-graph --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   While developing the ms_graph tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve [code quality](https://github.com/singer-io/getting-started/blob/master/docs/BEST_PRACTICES.md

   > pylint tap_ms_graph -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

   Pylint test resulted in the following score:

   Your code has been rated at 9.67/10

   To [check the tap](https://github.com/singer-io/singer-tools

   > tap_ms_graph --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Unit Tests

   Unit tests may be run with the following.

   python -m pytest --verbose
   

   Note, you may need to install test dependencies.

   pip install -e .'[dev]'
   

---

Copyright © 2019 Stitch