ADR-006: Distributed Content Architecture

Status

Superseded by ADR-007 | Proposed | Accepted | Complements ADR-005

Version History

• v1: 2025-04-29 - Initial proposal
• v2: 2025-04-29 - Updated with framework-specific considerations

Context

Building on the Content Orchestration Architecture described in ADR-005, we need to address scenarios where content may exist across multiple sources with different capabilities, availability characteristics, and consistency guarantees. This distributed content challenge extends beyond basic source coordination into the realm of multi-source synchronization and conflict management.

Key challenges we face include:

1. Content Source Diversity: Content may exist in multiple places (filesystem, memory, remote APIs, offline storage, etc.) with different access patterns.

2. Capability Heterogeneity: Sources have varied capabilities that aren’t inherently tied to their type (e.g., an HTTP source might be read-only or fully CRUD-capable).

3. Consistency Requirements: Different content types have different consistency needs, from strict consistency for configuration to eventual consistency for cached assets.

4. Offline-First Scenarios: Applications need to function when partially or fully disconnected from remote sources.

5. Conflict Resolution: When content is modified in multiple sources, we need strategies to detect and resolve conflicts.

While network-level solutions like CDNs and API gateways handle some distribution concerns, they don’t address the application-level challenges of working with diverse content sources simultaneously, particularly for complex client-side applications.

Decision

Implement a Distributed Content Architecture that extends the Content Orchestration Architecture with the following key components:

1. Source Groups: Enable content to be distributed across multiple sources with explicit policies.

2. Distribution Policies: Define how content is read from, written to, and synchronized across multiple sources.

3. Versioning Support: Provide tools for tracking content versions and detecting conflicts.

4. Synchronization Strategies: Implement mechanisms for maintaining consistency between content sources.

5. Offline Support: Add specific patterns for offline-first content handling.

Core Components

1. Source Groups

/**
 * A group of content sources that work together to serve content
 */
export interface SourceGroup {
  /**
   * Unique identifier for this group
   */
  id: string

  /**
   * Human-readable name
   */
  name: string

  /**
   * Description of what this group represents
   */
  description: string

  /**
   * Source IDs in this group
   */
  sources: string[]

  /**
   * The policies governing how this group operates
   */
  policy: DistributionPolicy
}

/**
 * Options for creating a source group
 */
export interface SourceGroupOptions {
  name: string
  description?: string
  sources: string[]
  policy: DistributionPolicy
}

2. Distribution Policies

/**
 * Policies governing how content is distributed across sources
 */
export interface DistributionPolicy {
  /**
   * Strategy for reading content
   */
  read: {
    /**
     * How to select sources for reading
     */
    strategy:
      | 'primary-only'
      | 'primary-with-fallback'
      | 'fastest-response'
      | 'aggregate'

    /**
     * Whether stale content is acceptable
     */
    acceptStale?: boolean

    /**
     * Maximum acceptable staleness in milliseconds
     */
    maxStaleness?: number

    /**
     * Source priorities for reading (higher is more preferred)
     */
    priorities?: Record<string, number>
  }

  /**
   * Strategy for writing content
   */
  write: {
    /**
     * How to handle writes across sources
     */
    strategy: 'primary-only' | 'all-sync' | 'primary-with-async-replication'

    /**
     * Validation level before write is considered successful
     */
    validationLevel?: 'none' | 'schema' | 'full'

    /**
     * Whether to wait for all sources to confirm write
     */
    waitForAll?: boolean

    /**
     * Timeout for write operations in milliseconds
     */
    timeout?: number
  }

  /**
   * Consistency requirements
   */
  consistency: {
    /**
     * Level of consistency required
     */
    level: 'eventual' | 'strong' | 'read-your-writes'

    /**
     * How to resolve conflicts
     */
    conflictResolution:
      | 'last-write-wins'
      | 'version-vector'
      | 'merge'
      | 'custom'

    /**
     * Custom conflict resolver function ID (if applicable)
     */
    conflictResolverFn?: string
  }

  /**
   * Synchronization configuration
   */
  synchronization: {
    /**
     * When to synchronize content between sources
     */
    mode: 'eager' | 'lazy' | 'scheduled'

    /**
     * Interval for scheduled synchronization in milliseconds
     */
    interval?: number

    /**
     * Maximum attempts for failed synchronization
     */
    maxRetries?: number

    /**
     * Whether to allow background synchronization
     */
    backgroundSync?: boolean
  }
}

3. Content Version Support

/**
 * Enhanced content metadata with versioning information
 */
export interface VersionedContentMetadata extends ContentMetadata {
  /**
   * Version identifier
   */
  version?: string

  /**
   * Vector clock for distributed version tracking
   */
  vectorClock?: Record<string, number>

  /**
   * Source that last modified this content
   */
  lastModifiedBy?: string

  /**
   * Sources this content has been synchronized to
   */
  syncedTo?: Record<
    string,
    {
      timestamp: number
      status: 'synced' | 'pending' | 'failed'
      version?: string
    }
  >

  /**
   * Whether this content has conflicts
   */
  hasConflicts?: boolean

  /**
   * Available conflict versions if conflicts exist
   */
  conflictVersions?: string[]
}

/**
 * Content with version information
 */
export interface VersionedContent<T = unknown> {
  /**
   * The content data
   */
  content: T

  /**
   * Versioning metadata
   */
  version: {
    id: string
    predecessorId?: string
    timestamp: number
    sourceId: string
    mergeBase?: string
  }

  /**
   * Standard content metadata
   */
  metadata: VersionedContentMetadata
}

4. Synchronization Manager

/**
 * Manages content synchronization between sources
 */
export interface SynchronizationManager extends Disposable {
  /**
   * Synchronize specific content across sources
   */
  synchronize(uri: string, options?: SyncOptions): Observable<SyncResult>

  /**
   * Synchronize all content matching a pattern
   */
  synchronizeAll(
    pattern: string,
    options?: SyncOptions
  ): Observable<SyncSummary>

  /**
   * Get synchronization status for content
   */
  getSyncStatus(uri: string): Observable<SyncStatus>

  /**
   * Observe synchronization events
   */
  observeSyncEvents(): Observable<SyncEvent>

  /**
   * Resolve a content conflict
   */
  resolveConflict(uri: string, resolution: ConflictResolution): Observable<void>
}

/**
 * Options for content synchronization
 */
export interface SyncOptions {
  /**
   * Source IDs to synchronize between (defaults to all applicable sources)
   */
  sources?: string[]

  /**
   * Direction of synchronization
   */
  direction?: 'push' | 'pull' | 'bidirectional'

  /**
   * Conflict resolution strategy
   */
  conflictStrategy?: 'auto' | 'manual' | 'prefer-source' | 'prefer-target'

  /**
   * Whether to force synchronization even with conflicts
   */
  force?: boolean
}

5. Offline Support

/**
 * Offline manager for content operations
 */
export interface OfflineManager extends Disposable {
  /**
   * Check if we're currently offline
   */
  isOffline(): boolean

  /**
   * Observe network status changes
   */
  observeNetworkStatus(): Observable<NetworkStatus>

  /**
   * Queue an operation for when online
   */
  queueOperation(operation: ContentOperation): Observable<QueuedOperationInfo>

  /**
   * Get pending operations
   */
  getPendingOperations(): Observable<QueuedOperationInfo[]>

  /**
   * Execute pending operations when online
   */
  executePendingOperations(
    options?: ExecuteOptions
  ): Observable<OperationResult<unknown>[]>

  /**
   * Update offline content policy
   */
  updateOfflinePolicy(policy: OfflinePolicy): void
}

/**
 * Network status information
 */
export interface NetworkStatus {
  online: boolean
  lastOnline?: number
  connectionType?: 'wifi' | 'cellular' | 'other' | 'unknown'
  downlink?: number // Mbps
  effectiveType?: 'slow-2g' | '2g' | '3g' | '4g'
}

/**
 * Offline policy configuration
 */
export interface OfflinePolicy {
  /**
   * Content URIs to prefetch for offline use
   */
  prefetchPatterns: string[]

  /**
   * Maximum offline storage in bytes
   */
  maxOfflineStorage?: number

  /**
   * How to handle operations when offline
   */
  offlineStrategy: 'queue' | 'reject' | 'simulate'

  /**
   * When to sync on reconnection
   */
  reconnectSync: 'immediate' | 'delayed' | 'manual'

  /**
   * Maximum queued operations
   */
  maxQueuedOperations?: number
}

Content Mediator Enhancements

The existing ContentMediator interface from ADR-005 is extended with distributed content capabilities:

export interface ContentMediator<T = RawContent> extends Disposable {
  // Source registration (from ADR-005)
  registerStream(
    stream: ContentStream<T>,
    options: StreamRegistrationOptions
  ): string

  // Source group management
  createSourceGroup(options: SourceGroupOptions): string
  updateSourceGroup(groupId: string, options: Partial<SourceGroupOptions>): void
  getSourceGroup(groupId: string): SourceGroup
  listSourceGroups(): SourceGroup[]

  // Content operations (extended from ADR-005)
  resolveContent(query: ContentQuery): Observable<ResolvedContent<T>>
  executeContentOperation(
    operation: ContentOperation
  ): Observable<OperationResult<T>>

  // Synchronization operations
  getSynchronizationManager(): SynchronizationManager

  // Offline support
  getOfflineManager(): OfflineManager

  // Other methods from ADR-005...
}

Enhanced Content Query

The Content Query model is extended to support distributed content operations:

export interface ContentQuery {
  // URI for content resolution
  uri: string

  // Query options
  options?: {
    // Existing options...

    // Source selection
    sourceStrategy?: {
      type: 'specific' | 'group' | 'capability-based' | 'auto'
      sourceId?: string
      groupId?: string
      requiredCapabilities?: Partial<ContentCapabilities>
    }

    // Distributed content options
    distribution?: {
      // Consistency requirements
      consistency?: 'eventual' | 'strong'

      // Whether to accept stale content
      acceptStale?: boolean

      // Maximum acceptable staleness
      maxStaleness?: number

      // Version constraints
      version?: {
        specific?: string
        minimum?: string
        branch?: string
      }

      // Synchronization behavior
      sync?: 'auto' | 'manual' | 'none'
    }

    // Offline options
    offline?: {
      // Whether to make this content available offline
      available: boolean

      // Priority for offline storage (higher is more important)
      priority?: number

      // Maximum age for offline content
      maxAge?: number

      // Dependencies to include with this content
      includeDependencies?: boolean
    }
  }
}

Architecture Diagram

+--------------------------------------------------+
|                   Application                     |
+--------------------------------------------------+
                       |
                       v
+--------------------------------------------------+
|                  Content Hooks                    |
+--------------------------------------------------+
                       |
                       v
+--------------------------------------------------+
|               Content Mediator                    |
|                                                   |
|  +----------------+        +-----------------+    |
|  | Source Registry |<----->| Capability      |    |
|  |                |        | Profiles        |    |
|  +----------------+        +-----------------+    |
|          |                         ^              |
|          v                         |              |
|  +----------------+        +-----------------+    |
|  | Source Groups  |<------>| Distribution    |    |
|  |                |        | Policies        |    |
|  +----------------+        +-----------------+    |
|          |                         |              |
|          v                         v              |
|  +----------------+        +-----------------+    |
|  | Content        |<------>| Synchronization |    |
|  | Pipeline       |        | Manager         |    |
|  +----------------+        +-----------------+    |
|                                   |              |
|                                   v              |
|                           +-----------------+    |
|                           | Offline         |    |
|                           | Manager         |    |
|                           +-----------------+    |
+--------------------------------------------------+
                       |
                       v
+--------------------------------------------------+
|                 Content Sources                   |
|                                                   |
|  +----------------+  +-------------+  +--------+  |
|  | Filesystem     |  | Memory      |  | Remote  |  |
|  | Source         |  | Source      |  | Sources |  |
|  +----------------+  +-------------+  +--------+  |
+--------------------------------------------------+

Practical Examples

Example 1: Content Available in Multiple Sources

// Create source group for distributed content
const blogGroupId = mediator.createSourceGroup({
  name: 'Blog Content Group',
  sources: [filesystemStreamId, remoteApiStreamId],
  policy: {
    read: {
      strategy: 'primary-with-fallback',
      acceptStale: true,
      maxStaleness: 3600000, // 1 hour
      priorities: { [filesystemStreamId]: 10, [remoteApiStreamId]: 5 },
    },
    write: {
      strategy: 'primary-with-async-replication',
      waitForAll: false,
    },
    consistency: {
      level: 'eventual',
      conflictResolution: 'last-write-wins',
    },
    synchronization: {
      mode: 'eager',
      maxRetries: 3,
      backgroundSync: true,
    },
  },
})

// Use in component
function BlogPost({ slug }) {
  const { content, loading, error } = useContent({
    uri: `blog:/posts/${slug}.mdx`,
    options: {
      sourceStrategy: {
        type: 'group',
        groupId: blogGroupId,
      },
      distribution: {
        acceptStale: true,
        maxStaleness: 300000, // 5 minutes
        sync: 'auto',
      },
    },
  })

  // Render component...
}

Example 2: Offline-First Content Editing

// Create source group for offline-first editing
const offlineFirstGroupId = mediator.createSourceGroup({
  name: 'Offline-First Editing',
  sources: [localStorageStreamId, remoteApiStreamId],
  policy: {
    read: {
      strategy: 'primary-with-fallback',
      priorities: { [localStorageStreamId]: 10, [remoteApiStreamId]: 5 }
    },
    write: {
      strategy: 'primary-only', // Write to local first
    },
    consistency: {
      level: 'eventual',
      conflictResolution: 'merge' // Try to merge changes
    },
    synchronization: {
      mode: 'lazy', // Sync when online
      backgroundSync: true
    }
  }
});

// Configure offline manager
mediator.getOfflineManager().updateOfflinePolicy({
  prefetchPatterns: ['doc:/templates/*', 'doc:/shared/*'],
  maxOfflineStorage: 50 * 1024 * 1024, // 50MB
  offlineStrategy: 'queue',
  reconnectSync: 'delayed',
  maxQueuedOperations: 100
});

// Editor component with sync control
function DocumentEditor({ documentId }) {
  const { content, loading, error, syncStatus } = useContent({
    uri: `doc:/${documentId}`,
    options: {
      sourceStrategy: {
        type: 'group',
        groupId: offlineFirstGroupId
      },
      distribution: {
        sync: 'manual' // Let user control synchronization
      },
      offline: {
        available: true,
        priority: 10
      }
    }
  });

  const { isOffline, networkStatus } = useNetworkStatus();
  const syncManager = useSyncManager();

  function handleSync() {
    syncManager.synchronize(`doc:/${documentId}`).subscribe({
      next: result => setStatus(`Sync completed: ${result.status}`),
      error: err => setStatus(`Sync failed: ${err.message}`)
    });
  }

  // Render editor with sync UI and offline indicator
  return (
    <EditorLayout>
      {isOffline && <OfflineBanner type={networkStatus.lastOnline ? "reconnecting" : "offline"} />}

      <TextEditor content={content} readOnly={loading} />

      <SyncStatusIndicator status={syncStatus} />

      <ButtonGroup>
        <SaveButton disabled={loading} />
        <SyncButton onClick={handleSync} disabled={isOffline || loading} />
      </ButtonGroup>
    </EditorLayout>
  );
}

Framework-Specific Considerations

For our targeted framework implementation, several specific aspects require attention:

1. RxJS Integration for Synchronization
   • Use proper RxJS patterns for synchronization events and status reporting
   • Ensure backpressure handling in synchronization pipelines
   • Implement retry/backoff strategies using RxJS operators

// Implementation example
class SynchronizationManagerImpl implements SynchronizationManager {
  synchronize(uri: string, options?: SyncOptions): Observable<SyncResult> {
    return this.getSources(uri, options).pipe(
      switchMap(sources => {
        return combineLatest(
          sources.map(source => this.synchronizeToSource(uri, source))
        )
      }),
      map(results => this.aggregateResults(results)),
      retryWhen(errors =>
        errors.pipe(
          tap(err =>
            this.logger.warn('Sync error, retrying', { uri, error: err })
          ),
          delayWhen((_, i) => timer(Math.min(1000 * 2 ** i, 30000))),
          take(options?.maxRetries || 3)
        )
      ),
      catchError(err => {
        this.logger.error('Sync failed', { uri, error: err })
        return of({ status: 'failed', error: err, uri })
      }),
      finalize(() => this.cleanupSyncResources(uri))
    )
  }
}

2. Memory Management in Offline Scenarios

   • Implement storage quota management for offline content
   • Use IndexedDB for structured offline storage via RxJS wrappers
   • Implement LRU eviction for offline cached content

3. Framework-Specific Offline Support

   • Integrate with ServiceWorker APIs when available
   • Implement framework-specific network detection
   • Support server-side rendering with hydration for offline content

4. Performance Optimization for Distributed Content

   • Implement request batching and deduplication
   • Use shared ReplaySubjects for content that is accessed frequently
   • Implement content preloading based on navigation patterns

5. Developer Experience Enhancements

   • Add React DevTools integration for inspecting distributed content
   • Create debugging tools for visualizing synchronization status
   • Implement content version visualization

Alternatives Considered

1. Pure Network-Based Distribution

   • Pros: Leverages existing CDN/gateway infrastructure, simpler client
   • Cons: Limited offline support, less control over capability detection
   • Outcome: Rejected as insufficient for our application-level requirements

2. Single Source of Truth with Caching

   • Pros: Simpler consistency model, clearer ownership
   • Cons: Limited offline capabilities, performance constraints
   • Outcome: Partially adopted for some scenarios, but insufficient for full requirements

3. Full Replication Everywhere

   • Pros: Maximum availability, simpler programming model
   • Cons: Storage overhead, complex conflict resolution
   • Outcome: Rejected as excessive for most content types

4. Manual Source Selection

   • Pros: Developer has full control, simpler implementation
   • Cons: Requires explicit source knowledge in components
   • Outcome: Supported as an option but not the default pattern

5. Dynamic Content Federation

   • Pros: Maximum flexibility, runtime adaptation
   • Cons: Complex implementation, harder to reason about
   • Outcome: Adapted core concepts but simplified for maintainability

Consequences

Benefits

1. Clear Separation of Capabilities and Sources: Decoupling capabilities from source types provides flexibility and clarity.

2. Support for Distributed Content: Explicit policies for reading, writing, and synchronizing across sources.

3. Offline-First Support: Architecture accommodates offline scenarios with appropriate synchronization.

4. Conflict Management: Built-in mechanisms for detecting and resolving conflicts.

5. Progressive Enhancement: Components can adapt based on available capabilities.

6. Developer Control: Explicit API for defining distribution policies while maintaining simple defaults.

7. Framework Integration: Tight integration with our target framework for optimal performance and developer experience.

Challenges

1. Additional Complexity: The distributed model adds conceptual complexity.

2. Performance Considerations: Synchronization and conflict resolution have performance costs.

3. Development Overhead: Requires more configuration and thought than simpler models.

4. Testing Complexity: Distributed systems are inherently harder to test.

5. Bundle Size: The full implementation may increase bundle size if not properly code-split.

6. Framework Dependency: Some optimizations may tie the implementation closely to our framework.

Implementation Strategy

The implementation will proceed in phases:

Phase 1: Core Infrastructure

1. Define source group interfaces and basic implementation
2. Implement distribution policies
3. Add simple synchronization manager

Phase 2: Versioning and Conflict Resolution

1. Implement version tracking for content
2. Add basic conflict detection
3. Implement simple conflict resolution strategies

Phase 3: Offline Support

1. Implement offline manager
2. Create persistent storage adapters
3. Add operation queueing system

Phase 4: Framework Integration

1. Implement framework-specific hooks and components
2. Add performance optimizations for the framework
3. Create developer tools for debugging

Phase 5: Advanced Features

1. Implement advanced conflict resolution strategies
2. Add background synchronization
3. Implement predictive prefetching

Conclusion

This Distributed Content Architecture complements the Content Orchestration Architecture from ADR-005 by addressing the specific challenges of content available across multiple sources with different capabilities. By providing explicit policies for distribution and integrating tightly with our target framework, we create a flexible system that supports a wide range of scenarios from simple single-source access to complex offline-first applications with synchronization.

The architecture balances the concerns of traditional content delivery networks with application-specific requirements for capability detection, source selection, and conflict management. It provides developers with both high-level abstractions for common patterns and fine-grained control when needed.

References

• CRDT: Conflict-Free Replicated Data Types
• CAP Theorem
• Offline-First Web Development
• Web Content Accessibility Guidelines (WCAG)
• Vector Clocks
• Progressive Enhancement
• RxJS Documentation
• IndexedDB API
• ADR-005: Content Orchestration Architecture