Content Structure

This document explains the fundamental structure of content in the ReX system and how this structure enables flexibility and consistency across different environments.

Overview

Content structure refers to how content is organized, addressed, and represented in the system. The structure is designed to be simple yet flexible, allowing for a wide range of content types while maintaining a consistent programming interface.

Core Content Structure

At its most basic level, content in the system consists of three components:

1. Data: The actual content (text, binary data, structured data)
2. Content Type: The MIME type of the content (e.g., text/markdown, image/png)
3. Metadata: Descriptive information about the content

This structure is expressed in the Content<T> interface:

interface Content<T = string> {
  data: T
  contentType: string
  metadata: ContentMetadata
}

The generic type parameter T allows for different data representations:

• Content<string>: Text-based content (Markdown, HTML, etc.)
• Content<Uint8Array> or Content<Buffer>: Binary content (images, PDFs, etc.)
• Content<object>: Structured content (JSON, etc.)

Content Addressing

Content is addressed using a URI-based system that is:

1. Storage-Agnostic: The same URI format works across different storage backends
2. Path-Like: Uses familiar path notation for intuitive navigation
3. Hierarchical: Supports nested content organization

// Example URIs
const textUri = 'blog/posts/hello-world.md'
const imageUri = 'assets/images/logo.png'
const configUri = 'config/settings.json'

The URI system is implemented with the following components:

URI Parsing

interface ParsedContentURI {
  original: string // The original URI string
  segments: string[] // Path segments
  extension?: string // File extension (if any)
  contentType?: string // Content type inferred from extension
}

// Example
const parsed = parseContentUri('blog/posts/hello-world.md')
// {
//   original: 'blog/posts/hello-world.md',
//   segments: ['blog', 'posts', 'hello-world'],
//   extension: 'md',
//   contentType: 'text/markdown'
// }

URI Resolution

URIs can be relative or absolute within the content store:

// Resolving a relative URI against a base URI
const base = 'blog/posts/'
const relative = '../images/photo.jpg'
const resolved = resolveContentUri(relative, base)
// 'blog/images/photo.jpg'

// Normalizing a URI (removing . and .. segments)
const normalized = normalizeContentUri('blog/posts/../images/./photo.jpg')
// 'blog/images/photo.jpg'

URI Mapping

Different storage adapters map URIs to their native path formats:

// FileSystem adapter
const fsPath = mapUriToPath('blog/posts/hello.md', { root: '/content' })
// '/content/blog/posts/hello.md'

// HTTP adapter
const url = mapUriToPath('blog/posts/hello.md', {
  root: 'https://example.com/api',
})
// 'https://example.com/api/blog/posts/hello.md'

Content Collections

Content is often organized into collections with a predictable structure:

interface ContentCollection<T = string> {
  [uri: string]: Content<T>
}

Collections can be:

1. Homogeneous: All content has the same type

   const blogPosts: ContentCollection<string> = {
     'post-1.md': {
       /* Markdown content */
     },
     'post-2.md': {
       /* Markdown content */
     },
   }

2. Heterogeneous: Content can have different types

   const mixedCollection: ContentCollection<unknown> = {
     'post.md': { data: '# Post', contentType: 'text/markdown', metadata: {} },
     'image.png': { data: binaryData, contentType: 'image/png', metadata: {} },
     'config.json': {
       data: { key: 'value' },
       contentType: 'application/json',
       metadata: {},
     },
   }

Content Hierarchies

Content is organized hierarchically, mimicking a filesystem structure:

blog/
  posts/
    hello-world.md
    getting-started.md
  images/
    header.jpg
    profile.png
config/
  settings.json
  users.json

This hierarchical structure enables:

1. Logical Organization: Content is arranged in a meaningful way
2. Pattern-Based Access: Content can be accessed using glob patterns
3. Namespace Isolation: Different content areas are separated

Pattern Matching

The system supports glob-style pattern matching for content discovery:

// Find all Markdown files in the blog/posts directory
const posts = await store.list('blog/posts/*.md')

// Find all JSON files recursively
const jsonFiles = await store.list('**/*.json')

// Find specific file patterns
const imageFiles = await store.list('blog/images/{logo,banner}*.{png,jpg}')

Content References

Content can reference other content through a reference system:

interface ContentReference {
  uri: string
  type: 'embed' | 'link' | 'dependency'
  title?: string
  description?: string
}

// Example: Blog post with image references
const blogPost: Content<string> = {
  data: '# Post with Images\n\n![Logo](../images/logo.png)\n\n![Banner](../images/banner.jpg)',
  contentType: 'text/markdown',
  metadata: {
    title: 'Post with Images',
    references: [
      { uri: 'blog/images/logo.png', type: 'embed', title: 'Logo' },
      { uri: 'blog/images/banner.jpg', type: 'embed', title: 'Banner' },
    ],
  },
}

Reference types include:

1. Embed: Content directly embedded in the parent content
2. Link: Content referenced by the parent but not embedded
3. Dependency: Content required by the parent for functionality

Content Structure Patterns

The system supports several common content structure patterns:

Atomic Content

Individual pieces of content that stand alone:

// A single Markdown document
const document: Content<string> = {
  data: '# Title\n\nContent here...',
  contentType: 'text/markdown',
  metadata: {
    title: 'Title',
    createdAt: new Date(),
  },
}

Composite Content

Content that references other content to form a larger unit:

// A blog post that includes images
const post: Content<string> = {
  data: '# My Post\n\n![Image](images/photo.jpg)\n\n## Section\n\nMore content...',
  contentType: 'text/markdown',
  metadata: {
    title: 'My Post',
    references: [{ uri: 'images/photo.jpg', type: 'embed' }],
  },
}

Collection Content

Content that represents a collection of other content:

// A blog index
const blogIndex: Content<object> = {
  data: {
    title: 'My Blog',
    posts: [
      { uri: 'posts/post-1.md', title: 'First Post' },
      { uri: 'posts/post-2.md', title: 'Second Post' },
    ],
  },
  contentType: 'application/json',
  metadata: {
    title: 'Blog Index',
    references: [
      { uri: 'posts/post-1.md', type: 'link' },
      { uri: 'posts/post-2.md', type: 'link' },
    ],
  },
}

Physical Storage Structure

While the content structure provides a logical organization, the physical storage depends on the adapter:

Filesystem Adapter

/content-root/
  blog/
    posts/
      hello-world.md
      getting-started.md
    images/
      header.jpg

IndexedDB Adapter

Database: "ContentStore"
  ObjectStore: "Content"
    Key: "blog/posts/hello-world.md" -> Value: { data, contentType, metadata }
    Key: "blog/posts/getting-started.md" -> Value: { data, contentType, metadata }
    Key: "blog/images/header.jpg" -> Value: { data, contentType, metadata }

Memory Adapter

{
  "blog/posts/hello-world.md": { data, contentType, metadata },
  "blog/posts/getting-started.md": { data, contentType, metadata },
  "blog/images/header.jpg": { data, contentType, metadata }
}

Content Structure Benefits

This approach to content structure offers several benefits:

1. Consistency: The same structure works across different storage backends
2. Flexibility: Supports different content types and representations
3. Familiarity: Uses familiar concepts like paths and MIME types
4. Extensibility: Easily extended with additional metadata
5. Composability: Content can be composed through references

Working with Content Structure

Creating Structured Content

import { createContentStore } from '@lib/content'

const store = createContentStore()

// Create a directory structure
await store.write('blog/posts/hello.md', {
  data: '# Hello\n\nThis is my first post.',
  contentType: 'text/markdown',
  metadata: {
    title: 'Hello',
    createdAt: new Date(),
  },
})

await store.write('blog/images/photo.jpg', {
  data: imageData, // Uint8Array or Buffer
  contentType: 'image/jpeg',
  metadata: {
    title: 'Photo',
    width: 800,
    height: 600,
  },
})

Finding Content by Pattern

// List all blog posts
const posts = await store.list('blog/posts/*.md')

// List all images
const images = await store.list('blog/images/*')

// List everything recursively
const everything = await store.list('**/*')

Working with Content References

// Read content with references
const post = await store.read('blog/posts/hello.md', {
  includeReferences: true,
})

// Access references
const references = post.metadata.references || []
for (const ref of references) {
  console.log(`Referenced ${ref.type}: ${ref.uri}`)
}

// Follow a reference
if (references.length > 0) {
  const refContent = await store.read(references[0].uri)
  console.log(`Referenced content: ${refContent.metadata.title}`)
}

Related Concepts

• Content Model: The overall model for content representation
• Metadata System: How metadata enhances content
• URI System: Content addressing and resolution
• Content Operations: How content is manipulated