ADR-002: Real-time Content Watching Architecture

Status

Superseded by ADR-004 | Proposed | Accepted | Implemented

Version History

• v1: 2025-04-01 - Initial acceptance and implementation

Context

The content system needed to support real-time updates for several use cases:

1. Live previews of content during authoring
2. Development workflow with instant feedback
3. Content that changes based on external factors
4. Foundation for future collaborative editing

The existing system had several limitations:

• Content was loaded once and not updated
• Changes required manual refresh
• No way to synchronize content with filesystem changes
• No mechanism for reactive UI updates based on content changes

Decision

Implement a comprehensive real-time content watching system with these key components:

1. Adapter-Level Watching:

   • Extend ContentAdapter interface with watchContent method and supportsRealtime flag
   • Implement adapter-specific watching mechanisms (memory, filesystem)
   • Provide a consistent unsubscribe pattern across adapters

2. Registry-Level Integration:

   • Add watchContent method to registry interfaces
   • Implement watching in DynamicRegistry and UnifiedRegistry
   • Add central management of watchers to avoid duplication

3. Filesystem Adapter Implementation:

   • Create a new FilesystemAdapter for local development
   • Use the chokidar library for file system watching
   • Implement proper cleanup to avoid resource leaks

4. Unified Interface Design:

   • Registry capabilities for feature detection
   • Consistent callback patterns across layers
   • Proper type safety throughout the system

Alternatives Considered

1. Polling-Based Updates:

   • Pros: Simpler implementation, works with any content source
   • Cons: Less efficient, higher latency, resource intensive
   • Outcome: Rejected due to performance concerns and unnecessary CPU/memory usage

2. External Event Bus:

   • Pros: Decouples content updates from registry system
   • Cons: Additional complexity, potential consistency issues
   • Outcome: Rejected for initial implementation to reduce complexity

3. WebSockets for All Updates:

   • Pros: Unified approach for local and remote content
   • Cons: Overkill for local content, requires server infrastructure
   • Outcome: Partially implemented: Support planned for HTTP adapter in future

Consequences

Benefits

• Enables live preview capabilities for content authoring
• Improves development workflow with instant feedback
• Provides foundation for collaborative editing features
• Maintains clean separation of concerns across layers
• Offers flexible implementation across different content sources

Challenges

• Increased complexity in registry and adapter implementations
• Requires careful resource management to avoid memory leaks
• Different behaviors across content sources may be confusing
• Performance monitoring needed for large content sets
• Memory pressure with many active watchers

Implementation

The implementation follows a modular approach:

1. Core Implementation:

   • Adapter interfaces and implementations came first
   • Filesystem adapter with chokidar integration
   • Memory adapter with in-memory subscription system

2. API Design:

   • Registry interface extensions with clear semantics
   • Consistent callback and unsubscribe patterns
   • Type-safe event handling

3. System Integration:

   • Registry capabilities for feature detection
   • Unified watching experience from component perspective
   • Centralized cache invalidation when content changes

Key Components:

1. ContentAdapter Interface:

   export interface ContentAdapter {
     getContent(id: string): Promise<RawContent>
     readonly supportsRealtime: boolean
     watchContent?(
       id: string,
       callback: (content: RawContent) => void
     ): () => void
   }

2. Registry Implementation:

   export class DynamicRegistry implements ContentRegistry {
     // ... other methods ...
   
     watchContent(path: string, callback: ContentUpdateCallback): () => void {
       // Get adapter for path
       const adapter = this.getAdapter(path)
   
       // Check if adapter supports watching
       if (!adapter.supportsRealtime || !adapter.watchContent) {
         return () => {}
       }
   
       // Set up adapter-level watching
       return adapter.watchContent(path, async rawContent => {
         // Process content and notify callback
         const mdxData = await compileMdx(rawContent.content, {})
         callback(mdxData)
       })
     }
   }

3. FilesystemAdapter Implementation:

   export class FilesystemAdapter implements ContentAdapter {
     private watchers: Map<string, chokidar.FSWatcher> = new Map()
     private subscribers: Map<string, Set<ContentUpdate>> = new Map()
   
     get supportsRealtime(): boolean {
       return true
     }
   
     watchContent(id: string, callback: ContentUpdate): Unsubscribe {
       // Create file watcher
       const watcher = chokidar.watch(filePath, {
         /*options*/
       })
   
       // Handle file changes
       watcher.on('change', async () => {
         // Get updated content and notify subscribers
         const content = await this.getContent(id)
         subscribers.forEach(sub => sub(content))
       })
   
       // Return unsubscribe function
       return () => {
         // Clean up resources
       }
     }
   }

References

• Chokidar Documentation
• ReactiveX Pattern
• Observer Design Pattern
• Implementation in /src/lib/content/adapters/filesystem.ts and related files