File Browser Upload and Conversion Design

Purpose: Design document for implementing file upload with automatic markdown conversion in the FileTree component
Created: 2025-01-09
Status: In Development

Overview

The FileTree component will be enhanced to support drag-and-drop file uploads with automatic conversion to markdown format. This enables users to easily add documents to their workspace while maintaining both original files and AI-optimized markdown versions side by side.

Architecture

┌─────────────────┐       ┌──────────────────┐       ┌─────────────────┐
│   FileTree UI   │──────►│  Upload Endpoint │──────►│   Document      │
│  (React D&D)    │       │  /api/projects/  │       │   Processor     │
└─────────────────┘       └──────────────────┘       └─────────────────┘
         │                         │                          │
         │                         ▼                          ▼
         │                ┌──────────────────┐       ┌─────────────────┐
         │                │  File Storage    │       │  Markdown       │
         │                │  (originals/)    │       │  Converter      │
         │                └──────────────────┘       └─────────────────┘
         │                         │                          │
         ▼                         ▼                          ▼
┌─────────────────┐       ┌──────────────────────────────────┐
│  File Browser   │◄──────│        Workspace Structure       │
│   (Refresh)     │       │  docs/local/originals/           │
└─────────────────┘       │  docs/local/converted/           │
                          └──────────────────────────────────┘

Directory Structure

Before Upload

~/.claude/projects/{projectName}/
└── docs/
    └── (other existing docs)

After Upload

~/.claude/projects/{projectName}/
└── docs/
    └── local/
        ├── originals/           # Original uploaded files
        │   ├── contract.pdf
        │   ├── employee-handbook.docx
        │   ├── financial-report.xlsx
        │   └── README.txt
        └── converted/           # Markdown conversions
            ├── contract.md
            ├── employee-handbook.md
            ├── financial-report.md
            └── README.md

Implementation Details

1. Frontend: FileTree Component

Current State

• Already has drag-and-drop zone UI
• Handlers exist but only log to console
• No actual upload functionality

Required Changes

// src/components/FileTree.jsx

// Add file input ref
const fileInputRef = useRef(null);

// Enhanced drop handler
const handleDrop = async (e) => {
  e.preventDefault();
  setIsDragging(false);
  
  const files = Array.from(e.dataTransfer.files);
  await uploadFiles(files);
};

// New upload function
const uploadFiles = async (files) => {
  setUploading(true);
  setUploadProgress({});
  
  const formData = new FormData();
  files.forEach(file => {
    formData.append('documents', file);
  });
  
  try {
    const response = await api.uploadDocuments(
      selectedProject.name, 
      formData,
      (progress) => updateProgress(progress)
    );
    
    if (response.ok) {
      const result = await response.json();
      showSuccessMessage(result);
      await fetchFiles(); // Refresh file list
    }
  } catch (error) {
    showErrorMessage(error);
  } finally {
    setUploading(false);
  }
};

UI Components

• Progress bar during upload/conversion
• Success/error toast notifications
• File type validation feedback
• Batch upload status list

2. Backend: Upload Endpoint

New Endpoint

// server/routes/files.js (new file)

router.post('/projects/:projectName/upload-documents', 
  authenticateToken, 
  upload.array('documents', 20), 
  async (req, res) => {
    const { projectName } = req.params;
    const userId = req.user.id;
    const files = req.files;
    
    // Get workspace path
    const workspacePath = getWorkspacePath(userId, projectName);
    const originalsPath = path.join(workspacePath, 'docs/local/originals');
    const convertedPath = path.join(workspacePath, 'docs/local/converted');
    
    // Ensure directories exist
    await fs.mkdir(originalsPath, { recursive: true });
    await fs.mkdir(convertedPath, { recursive: true });
    
    const results = [];
    
    for (const file of files) {
      // Save original
      const originalPath = path.join(originalsPath, file.originalname);
      await fs.rename(file.path, originalPath);
      
      // Convert to markdown
      const markdown = await convertDocumentToMarkdown({
        path: originalPath,
        originalname: file.originalname,
        mimetype: file.mimetype
      });
      
      // Save markdown
      const mdFilename = path.parse(file.originalname).name + '.md';
      const convertedFilePath = path.join(convertedPath, mdFilename);
      await fs.writeFile(convertedFilePath, markdown);
      
      results.push({
        original: file.originalname,
        converted: mdFilename,
        status: 'success'
      });
    }
    
    res.json({
      success: true,
      filesProcessed: results.length,
      results
    });
  }
);

3. Document Conversion Service

Reuse Existing Converter

// server/services/document-processor.js
// Already has convertDocumentToMarkdown() function

// Supported formats:
- PDF → Markdown
- Word (DOC/DOCX) → Markdown  
- Excel (XLS/XLSX) → Markdown tables
- Text/Markdown → Pass through
- PowerPoint → Basic text extraction

// Using @knowcode/convert-to-markdown package

4. API Integration

New API Method

// src/utils/api.js

uploadDocuments: async (projectName, formData, onProgress) => {
  return fetch(`${API_BASE}/projects/${projectName}/upload-documents`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${getToken()}`
    },
    body: formData,
    // Progress tracking if supported
    onUploadProgress: onProgress
  });
}

User Experience Flow

Upload Process

1. Initiate Upload

   • Drag files to dropzone OR
   • Click dropzone to browse files

2. File Validation

   • Check file types (PDF, DOCX, XLSX, TXT, MD)
   • Check file size (max 50MB per file)
   • Show validation errors if any

3. Upload Progress

   • Show progress bar
   • List files being processed
   • Real-time status updates

4. Conversion Process

   • "Converting to markdown..." status
   • Individual file progress indicators

5. Completion

   • Success message with file count
   • FileTree automatically refreshes
   • New files appear in both folders

File Browser Display

📁 docs/
  📁 local/
    📁 originals/               [Original Files]
      📄 contract.pdf           (2.3 MB)
      📄 handbook.docx          (456 KB)
      📊 data.xlsx              (89 KB)
    📁 converted/               [AI-Ready Markdown]
      📝 contract.md            (34 KB)
      📝 handbook.md            (78 KB)
      📝 data.md                (12 KB)

Visual Design

Dropzone States

Default State

┌─────────────────────────────────┐
│     📤                          │
│  Drop documents here to upload  │
│     or click to browse         │
│                                │
│  Supports PDF, DOCX, MD, TXT   │
└─────────────────────────────────┘

Dragging State

┌─────────────────────────────────┐
│     📥                          │
│    Release to upload files     │
│                                │
│         (Green border)          │
└─────────────────────────────────┘

Uploading State

┌─────────────────────────────────┐
│  Uploading 3 files...          │
│  ████████████░░░░░░ 75%        │
│                                │
│  ✓ document1.pdf               │
│  ⟳ document2.docx              │
│  ○ document3.xlsx              │
└─────────────────────────────────┘

Security Considerations

File Validation

• Whitelist allowed MIME types
• Maximum file size: 50MB per file
• Maximum files per upload: 20
• Filename sanitization

Access Control

• Authentication required
• Files scoped to user's workspace
• No cross-project access

Input Sanitization

// Sanitize filename
const sanitizeFilename = (filename) => {
  return filename
    .replace(/[^a-z0-9.-]/gi, '_')
    .replace(/_{2,}/g, '_')
    .toLowerCase();
};

Error Handling

Client-Side Errors

Error	User Message	Action
Invalid file type	"File type not supported"	Highlight invalid files
File too large	"File exceeds 50MB limit"	Prevent upload
Network error	"Upload failed. Please try again"	Retry option

Server-Side Errors

Error	User Message	Action
Conversion failed	"Could not convert {filename}"	Save original only
Disk full	"Insufficient storage space"	Show error
Permission denied	"Access denied"	Check workspace permissions

Performance Optimization

Batch Processing

• Process files in parallel where possible
• Limit concurrent conversions to 5
• Queue additional files

File Size Handling

// Stream large files
const streamLargeFile = async (filePath) => {
  const readStream = fs.createReadStream(filePath);
  const writeStream = fs.createWriteStream(outputPath);
  
  return pipeline(readStream, transformStream, writeStream);
};

Caching

• Cache conversion results
• Skip re-conversion of identical files
• Store conversion metadata

Testing Strategy

Unit Tests

describe('FileUpload', () => {
  it('should validate file types', () => {
    expect(isValidFileType('document.pdf')).toBe(true);
    expect(isValidFileType('script.exe')).toBe(false);
  });
  
  it('should handle multiple file uploads', async () => {
    const files = [file1, file2, file3];
    const result = await uploadFiles(files);
    expect(result.filesProcessed).toBe(3);
  });
});

Integration Tests

1. Upload single file → Verify both versions created
2. Upload multiple files → Verify batch processing
3. Upload unsupported type → Verify rejection
4. Upload duplicate → Verify handling

Manual Testing Checklist

• Drag and drop single file
• Drag and drop multiple files
• Click to browse files
• Upload various file types
• Verify conversions are accurate
• Check error handling
• Test progress indicators
• Verify file refresh

Migration and Rollback

Migration Steps

1. Deploy backend changes
2. Update frontend components
3. Test in development
4. Deploy to production

Rollback Plan

• Feature flag to disable upload
• Preserve existing file structure
• No breaking changes to existing code

Future Enhancements

Phase 2 Features

• OCR for scanned PDFs
• Image text extraction
• Automatic categorization
• Duplicate detection
• Version control for uploads
• Bulk operations (delete, move)
• Search within converted documents
• Preview before upload
• Conversion quality settings
• Scheduled re-conversion

Phase 3 Features

• AI-powered document summarization
• Automatic tagging and metadata
• Document relationship mapping
• Smart folder organization
• Conversion analytics

Success Metrics

Technical Metrics

• Upload success rate > 95%
• Conversion success rate > 90%
• Average upload time < 5 seconds per file
• Average conversion time < 10 seconds per file

User Experience Metrics

• Clear feedback at every step
• No data loss
• Intuitive drag-and-drop
• Accessible file organization

Conclusion

This implementation provides a seamless way for users to upload documents to their workspace with automatic markdown conversion. By maintaining both original and converted versions, users get the best of both worlds: preserved originals for reference and AI-optimized markdown for Claude to process effectively.