Long Bui Discovering new things. Data x Platform Ops

Multiple files has small size REALLY causes performance issue

I have the real problem with my file syncs between local and icloud Drive,

I am working in data engineering, I know the effection of small file causing low performance and network latency.

But I still had mistake that generate 90.000+ files in my synchronizing folder.

Even Wifi has 35Mb/s uploads, but it still got me xx hours and y restarted time to finish them.

Acctually, the upload bandwidth only xx kb/s.

My laptop, working repositories were stucked.

Again, small files => Performance issue while data reading.

graph TD A[Multiple Small Files] --> B[Network Latency] A --> C[I/O Operations Overhead] A --> D[Memory Usage Impact] A --> E[Storage Fragmentation] B --> F[Reduced Performance] C --> F D --> F E --> F F --> G[System Slowdown]

Subscribe to keep you posted the latest updates