Redundancy elimination in IoT oriented big data: a survey, schemes, open challenges and future applications

The Internet of Things (IoT), an enlarged Internet-based network, is a key element of the next information technology revolution. With the evolution of numerous IoT based smart applications like smart city, healthcare, huge amount of heterogenous data called big data with varying volume, velocity and variety is getting space on various storage systems. Data redundancy is a serious problem that wastes a lot of storage capacity and network bandwidth with less data security in setups that blend cloud integrated IoT data storage. Data deduplication/redundancy elimination strategies can effectively decrease and control this issue by removing duplicate data in cloud-integrated IoT storage systems. Security and privacy of data is another major concern. To maximise the storage effectively and extremely securely, with maintained data integrity, confidentiality, minimal storage cost and increased storage use, data deduplication (DD) over encrypted data is also a key problem in cloud integrated IoT storage and computing environment. With the dynamic nature of big data, the majority of current data deduplication techniques which primarily revolve around the static scenes like the backup and archive systems, are inappropriate to be applicable for real time scenarios in IoT environment. To overcome the aforementioned issues this survey presents an analysis of literature on conventional deduplication techniques. It is highlighting the need of deduplication in IoT oriented big data, its parameters and properties of effectiveness, with taxonomy for secure and conventional deduplication and scope of implementing it with new technologies; Blockchain. Further, it elaborates on issues and challenges along with future scope of deduplication schemes in various application domains.