Pipeline

Core Concepts

graph model { bgcolor=invis asset [shape=circle fillcolor=pink style=filled color=violetred3] asset_description [shape=record style="rounded,filled" fillcolor=snow label="Meaningful piece of\ldata that can be:\l\l • Identified\l • Versioned\l • Tracked\l \lContains metadata.\l\l"] asset -- asset_description [style=dashed] step [shape=triangle height=1 fillcolor=lightyellow style=filled color=khaki4] step_description [shape=record style="rounded,filled" fillcolor=snow label="Process in the pipeline\lthat outputs new assets.\l \lWhen growing in\lcomplexity, can be\lpromoted to area.\l \lMight receive assets as\linputs.\l"] step -- step_description [style=dashed] area [shape=box height=.75 fillcolor=lightskyblue1 color=dodgerblue4 style="filled,rounded"] area_description [shape=record style="rounded,filled" fillcolor=snow label="Contains steps that relate\lto each other.\l \lCan spawn multiple\linstances of the same\lsteps.\l \lManages assets I/O.\l\l"] area -- area_description [style=dashed] }

Relationships

digraph area_overview { bgcolor=invis newrank=true subgraph cluster_inputs { label=inputs color=invis margin=0 node [shape=circle label="" fillcolor=pink style=filled color=violetred3] input_asset1 input_asset2 input_asset3 input_asset1 -> input_asset2 -> input_asset3 [style=invis] } subgraph cluster_area { style="filled,rounded" label=area margin=5 fillcolor=lightskyblue1 color=dodgerblue4 sentinel [style=invis label=""] node [shape=triangle label="" fillcolor=lightyellow style=filled color=khaki4 height=.8] step2 step3 sentinel -> step2 -> step3 [style=invis] } subgraph cluster_outputs { label=outputs color=invis margin=0 node [shape=circle label="" fillcolor=pink style=filled color=violetred3] output_asset1 output_asset2 output_asset3 output_asset1 -> output_asset2 -> output_asset3 [style=invis] } subgraph cluster_descriptions { margin=1 style="rounded,filled" fillcolor=snow node [shape=none] assets_description [label="Assets are composed via\lUSD & can plug to as\lmany steps as required.\l"] steps_description [label="Steps don't know about\lothers of the same type.\l"] areas_description [label="Areas handle assets I/O\lwhile running steps.\l"] } {input_asset1 input_asset2 input_asset3} -> step2 input_asset3 -> step3 step2 -> output_asset1 step3 -> {output_asset2 output_asset3} {rank=same input_asset1 sentinel output_asset1 assets_description} {rank=same input_asset2 step2 output_asset2 steps_description} {rank=same input_asset3 step3 output_asset3 areas_description} }

Data Flow

digraph pipeline_overview { bgcolor=invis newrank=true rankdir=TB node [shape=box label="" style="filled,rounded" fillcolor=lightskyblue1 color=dodgerblue4] area1 area2 area3 area4 area5 area6 area7 area1 -> area1 [dir=back] area1 -> {area2 area3 area4} area2 -> area5 area3 -> {area4 area6 area7} area4 -> {area1 area6} area5 -> area3 subgraph cluster_descriptions { margin=1 style="rounded,filled" fillcolor=snow node [shape=none fillcolor=none] node_description [label="Areas act as nodes of\la graph.\l"] io_description [label="Inputs can come from\lanywhere.\l"] cyclic_description [label="Cyclic connections are\lallowed.\l"] } {rank=same area1 node_description} {rank=same area2 area4 io_description} {rank=same area5 area3 cyclic_description} }

Workflow Charts Legend

The Grill flow charts will use the following symbol legend:

digraph flowchart_symbols { bgcolor=invis node [style=filled fillcolor=snow] edge [style=invis] manual_op [shape=invtrapezium label="Manual\nOperation" height=.8 width=1.4 fixedsize=true] io [shape=parallelogram label="Input\nOutput" height=.8 width=1.2 fixedsize=true] decision [shape=diamond label=Decision fixedsize=true, height=1.2, width=1.2] process [shape=box label=Process height=0.7 width=1.2] terminator [shape=ellipse label=Terminator height=.8 width=1.6] connector [shape=circle label=Connector height=.8] storage [shape=folder label=Storage height=0.7 width=1.2] db [shape=cylinder label=Database height=0.7 width=1.2] document [shape=note label=Document height=0.7 width=1.2] merge [shape=invtriangle label=Merge fixedsize=true, height=1, width=1.2] display [shape=cds label=Display height=0.7 width=1.2] manual_op -> io -> decision process -> terminator -> connector document -> storage -> db merge -> display }

Data Creation

At the highest level, all asset data, from creation to finish, goes through the same life cycle:

digraph data_creation { bgcolor=invis newrank=true subgraph cluster_steps { color=invis margin=0 node [colorscheme=pastel19 style=filled] context [shape=invtrapezium fillcolor=3 label="1. Context\nSelection" height=.8 width=1.4 fixedsize=true] stage [shape=parallelogram label="2. Stage\nOpen" height=.8 width=1.3 fillcolor=5 fixedsize=true] opinion [shape=box label="3. Opinion\nAuthoring" height=0.7 width=1.3 fillcolor=2] persistency [shape=cylinder label="4. Persistency" height=0.7 width=1.2 fillcolor=8] context -> stage -> opinion -> persistency } subgraph cluster_descriptions{ color=invis margin=0 node [shape=box style="rounded,filled" fillcolor=snow] context_description [label="Contextual information for the\lprocesses about to manipulate assets.\l\lMinimum required is a pipeline area.\l"] stage_description [label="A corresponding USD stage for the\lcontext is opened.\l\lIf none exists but the context is valid,\lone will be created.\l"] opinion_description [label="USD opinions are authored by the\lpipeline steps from the context area.\l\l"] persistency_description [label="Created asset data is commited in\lorder for it to persist in the pipeline.\l\l"] } context -> context_description [style=invis] {rank=same context context_description} stage -> stage_description [style=invis] {rank=same stage stage_description} opinion -> opinion_description [style=invis] {rank=same opinion opinion_description} persistency -> persistency_description [style=invis] {rank=same persistency persistency_description} }