[BREAKING] Python: WIP: Checkpoint refactor: encode/decode, checkpoint format, etc

[TaoChenOSU]

Motivation and Context

Closes: #3530, #3529, #1665

Description

1. Simplify checkpoint encoding and decoding strategies.
   • WorkflowCheckpoint now contains live objects, as opposed to serialized jsons. This makes working with a checkpoint much easier in code.
   • Serialization/deserialization is done when a checkpoint is saved to file or read from a file. This opens up possibilities to support different serialization protocols.
   • FileCheckpointStorage now uses pickle.
   • InMemoryCheckpointStorage now storages raw checkpoints (i.e. no serialization)
2. Remove workflow_id from checkpoints
3. Add previous_checkpoint_id to checkpoints
4. Rename APIs on checkpoint storages
5. Remove _checkpoint_summary.py and _conversation_state.py
6. Remove deprecated checkpoint hooks

TODO:

1. Tests and samples.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
observability.py	609	84	86%	336, 338–340, 343–345, 350–351, 357–358, 364–365, 372, 374–376, 379–381, 386–387, 393–394, 400–401, 408, 664, 667, 675–676, 679–682, 684, 687–689, 692–693, 721, 723, 734–736, 738–741, 745, 753, 854, 856, 1005, 1007, 1011–1016, 1018, 1021–1025, 1027, 1139–1140, 1142, 1193–1194, 1329, 1377–1378, 1494–1496, 1555, 1725, 1879, 1881
packages/core/agent_framework/_workflows
_agent_executor.py	163	22	86%	96, 144, 162–163, 214–215, 217–218, 248–250, 258–260, 270–272, 274, 278, 282, 286–287
_checkpoint.py	145	0	100%
_checkpoint_encoding.py	46	0	100%
_events.py	149	21	85%	90–91, 224, 228, 230, 262, 337, 352, 367, 382, 394–396, 408–410, 412–413, 415–416, 420
_runner.py	170	2	98%	272–273
_runner_context.py	148	11	92%	63, 77–78, 80–81, 83, 369, 388, 441, 454, 458
_workflow.py	255	25	90%	87, 267–269, 271–272, 290, 294, 322, 424, 599, 620, 646, 673, 675–676, 681–682, 688, 694, 699, 719–721, 734
_workflow_builder.py	266	35	86%	262, 532, 630, 637–638, 738, 741, 746, 748, 755, 758–762, 764, 825, 899, 902, 958, 976, 989, 1003–1010, 1012, 1015, 1017–1019
_workflow_executor.py	182	30	83%	94, 444, 467, 469, 477–478, 483, 485, 490, 492, 545, 573–579, 583–585, 593, 598, 609, 619, 623, 629, 633, 643, 647
packages/orchestrations/agent_framework_orchestrations
_group_chat.py	285	51	82%	172, 335, 342, 371, 382–383, 389, 394, 410, 437–442, 444, 477–480, 482, 487–491, 621, 624, 627, 630, 638, 651, 654, 667, 676, 682, 726–727, 731–732, 746–747, 749–750, 781–782, 848, 867, 875, 889, 899
_orchestration_state.py	27	5	81%	22, 30, 68, 83–84
TOTAL	16274	1979	87%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3943	225 💤	0 ❌	0 🔥	1m 6s ⏱️