Skip to main content

Fork and Join Patterns

NeuroScript supports splitting data into multiple branches and merging them back together. This enables powerful patterns like residual connections, multi-path processing, and parallel computations.

The Fork Neuron​

Fork() duplicates its input into multiple identical copies. Combined with tuple unpacking, you can name each branch:

in -> Fork() -> (branch_a, branch_b)

This creates two references (branch_a and branch_b) that both point to the same data. You can then process each branch independently.

Basic Fork Example​

Simple Fork

Fork creates two identical copies that can be processed separately

NeuroScript
Loading editor...
PyTorch Output
Loading editor...

In this example:

  1. Fork() duplicates the input
  2. left goes through a Linear layer
  3. right is unchanged (identity path)
  4. Add() combines them element-wise

Residual Connection Pattern​

The most common fork/join pattern is the residual connection (skip connection):

Residual Block

The classic ResNet pattern: process main path, add skip connection

NeuroScript
Loading editor...
PyTorch Output
Loading editor...

Key points:

  • main path transforms the data
  • skip path preserves the original input
  • Add() merges them, enabling gradient flow
  • Both paths must have matching shapes at the merge point

Three-Way Fork​

Use Fork3() to create three branches:

Three-Way Processing

Fork3 creates three branches for parallel processing

NeuroScript
Loading editor...
PyTorch Output
Loading editor...

Shape Constraints at Merge Points​

When merging branches with Add(), Multiply(), or similar operations, all inputs must have the same shape:

# This works - both branches have shape [*, dim]
(processed, skip) -> Add() -> out

# This would fail - shape mismatch
main -> Linear(dim, dim * 2) -> processed  # [*, dim * 2]
(processed, skip) -> Add() -> out           # skip is [*, dim] - mismatch!

The compiler validates shape compatibility at merge points during validation.

Concat Instead of Add​

If you want to combine branches with different dimensions, use Concat():

Concatenation

Concat joins tensors along the last dimension

NeuroScript
Loading editor...
PyTorch Output
Loading editor...

Try It Yourself​

Experiment with these patterns:

  • Add more processing to the main branch
  • Try different merge operations (Add, Multiply, Concat)
  • Create deeper residual chains
  • Click "Show Analysis" to trace how shapes flow through branches