Super-Fast and Secure Deduplication System based on Fixed Window Fixed Byte Chunking and Semantic Weight Poisson Process Filter in Cloud Storage

Abstract

As the demand for cloud data storage continues to surge, optimizing data management techniques for both efficiency and security has become increasingly critical. Traditional Content-Defined Chunking (CDC) methods have played a significant role in data deduplication, but they are not without their limitations. These approaches often involve variable window sizes and byte-based adjustments, which can lead to inefficiencies and increased computational complexity. Additionally, the reliance on hash-based deduplication methods necessitates complex boundary detection mechanisms, which can further exacerbate performance overheads and complicate implementation. To address these challenges, we propose a novel cloud storage system that incorporates three groundbreaking modules designed to enhance data deduplication and security. The first module introduces a Fixed Window Fixed Bytes Chunking method, which departs from the traditional variable window size approach. By employing a fixed-size window and byte-based chunking strategy, this module simplifies the chunking process, thereby reducing computational overhead and providing a more predictable and consistent chunk size. This improvement leads to enhanced storage efficiency and a reduction in system resource consumption. The second module employs a Semantic Weight-based Poisson Process Filter for deduplication. This innovative approach transcends the conventional boundary detection and hash value techniques. Instead, it utilizes semantic weighting to evaluate the significance of data chunks and applies a Poisson process filter to effectively identify and eliminate redundant data. This method not only improves the accuracy of deduplication but also minimizes false positives and enhances overall storage efficiency. The third module focuses on data security through Triple Indirect Level Cryptographic encryption. This advanced encryption technique ensures that only unique data is stored in the cloud, significantly enhancing data protection and safeguarding against unauthorized access. The triple indirect level approach adds multiple layers of encryption, providing robust security without compromising storage efficiency. Our system demonstrates superior performance across various metrics. Specifically, the Fixed Window Fixed Bytes Chunking (FWFB) method exhibits the lowest chunking time and highest chunking efficiency and throughput compared to other approaches. For instance, at a 2MB file size, FWFB achieves a chunking time of 1.2 seconds, an efficiency of 0.7, and a throughput of 0.71, outperforming the traditional methods in both speed and effectiveness. Overall, this innovative approach provides a more streamlined, effective, and secure solution for modern cloud data storage needs, addressing the key limitations of traditional methods.