
Intelligent
Transformation
Delivery
Orchestration
Components structure and Layers Architecture
Main Components
Transformer To transform input data from one collection to another collection. With plugin structure to support different transform type and different backends.
Transporter To manage collections with DDM backend.
Conductor To notify/schedule consumers to consume new transformed data in a fine-grained granularity.
Layer and Plugin Architecture
Abstract Layers Hide the complexity of different logics to implement general API
Plugins To support different transforms and backends
Restful interface Interface to communicate with clients
Experiment agnostic based on generation
Abstract for ochestration
Maintainability and extensibility with plugins
Applications that iDDS would like to serve
Split files into fine grained events and stream the events as inputs for EventService.
Aggregate tape stage-in requests and immediate processing of partially staged data.
Passive creation of analysis format data rather than proactively keeing them on DISK, reduction of expensive disk usage while increaing of cheap CPU usage.
Remotely transform/reduce data and then deliver only needed data to consumers.