File Loading Optimization Ideas

Problem Statement

The "Loading files..." message appears too frequently in the file manager, creating a poor user experience. This happens because:

1. Files are refetched every time the Files tab becomes visible
2. Files are refetched when switching between projects
3. No caching mechanism exists to prevent unnecessary API calls

Root Causes

1. Tab Switching Mechanism

• The MainContent component uses activeTab === 'files' ? 'block' : 'hidden' to show/hide tabs
• This causes the FileTree component to re-render when the tab becomes visible
• The useEffect hook with [selectedProject] dependency triggers on every render

2. Component Lifecycle

• FileTree component doesn't persist state when hidden
• No memory of previously loaded files
• Always starts fresh when tab is selected

3. No Data Caching

• Every file fetch is a new API call
• No client-side caching of file tree data
• No staleness checking for data freshness

Proposed Solutions

Solution 1: Smart Caching with Timestamps

// Add to FileTree component
const [lastFetchTime, setLastFetchTime] = useState(null);
const [cachedProjectId, setCachedProjectId] = useState(null);

const CACHE_DURATION = 30000; // 30 seconds

const shouldRefetch = () => {
  if (!lastFetchTime) return true;
  if (cachedProjectId !== selectedProject?.id) return true;
  
  const now = Date.now();
  return (now - lastFetchTime) > CACHE_DURATION;
};

const fetchFiles = async (forceRefresh = false) => {
  if (!forceRefresh && !shouldRefetch()) {
    return; // Use cached data
  }
  
  setLoading(true);
  // ... fetch logic
  setLastFetchTime(Date.now());
  setCachedProjectId(selectedProject?.id);
};

Solution 2: Keep Components Mounted

Instead of using display: none approach, keep all tabs mounted but hidden:

// In MainContent.jsx
<div className={`h-full ${activeTab !== 'files' ? 'invisible absolute' : 'visible relative'}`}>
  <FileTree selectedProject={selectedProject} />
</div>

Solution 3: Global File State Management

Create a context or store for file data:

// FileDataContext.jsx
const FileDataContext = React.createContext();

export const FileDataProvider = ({ children }) => {
  const [fileCache, setFileCache] = useState({});
  
  const getFiles = (projectId) => {
    return fileCache[projectId]?.data || null;
  };
  
  const updateFiles = (projectId, data) => {
    setFileCache(prev => ({
      ...prev,
      [projectId]: {
        data,
        timestamp: Date.now()
      }
    }));
  };
  
  return (
    <FileDataContext.Provider value={{ getFiles, updateFiles }}>
      {children}
    </FileDataContext.Provider>
  );
};

Solution 4: Incremental Loading

Load files progressively:

1. Show cached data immediately (if available)
2. Fetch fresh data in background
3. Update UI only if data has changed

Solution 5: WebSocket Updates

Instead of refetching on tab switch, use WebSocket to listen for file changes:

• Server sends file update events
• Client updates file tree in real-time
• No need to refetch unless specifically requested

Implementation Priority

Phase 1: Quick Win (Minimal Changes)

1. Add simple timestamp-based caching (Solution 1)
2. Only refetch if data is >30 seconds old
3. Add "Refresh" button for manual updates

Phase 2: Better UX

1. Keep FileTree mounted (Solution 2)
2. Add loading skeleton instead of full loading state
3. Show stale data with refresh indicator

Phase 3: Optimal Solution

1. Implement global file state (Solution 3)
2. Add WebSocket file change notifications
3. Progressive/incremental loading

Additional Optimizations

1. Debounce Project Changes

const debouncedFetchFiles = useMemo(
  () => debounce(fetchFiles, 300),
  []
);

2. Optimistic UI Updates

• Update UI immediately for user actions (create, delete, rename)
• Sync with server in background
• Rollback on error

3. Virtual Scrolling

For large file trees, implement virtual scrolling to improve performance

4. Lazy Loading

Load directory contents only when expanded

Expected Benefits

• 80% reduction in API calls
• Instant tab switching
• Better perceived performance
• Reduced server load
• Improved user experience

Testing Considerations

• Test with large file trees (1000+ files)
• Test with slow network connections
• Test cache invalidation scenarios
• Test concurrent updates from multiple users

Metrics to Track

• API call frequency
• Time to interactive for Files tab
• User engagement with Files tab
• Server response times
• Cache hit/miss ratio