Best Practices for Storing Large Data in Solidity

Blockchain storage is limited and block space is therefore expensive, especially at times of peak congestion. Solidity developers must employ strategies to optimise data storage whilst maintaining the integrity and accessibility of their smart contracts.

One of the most effective approaches is to leverage off-chain storage solutions. Instead of storing large data sets directly on the blockchain, developers can store a hash or reference to the data on-chain, whilst keeping the actual data in decentralised storage systems like IPFS. This method significantly reduces gas costs and allows for more efficient data management.

This is often seen with NFT drops where the image itself is stored on IPFS and linked to within the smart contract. Because storing 10,000 high res images on Ethereum mainnet would cost a generational fortune.

When on-chain storage is necessary, it’s crucial to optimise data structures, especially when dealing with large data sets. Mappings provide a key value pair lookups and don’t require iteration, making them ideal for scenarios where direct access to data is frequent but you don’t need to loop through that data which is often impossible as a data set grows due to gas limits.

Here’s a simple example of a user balance contract that can grow and scale as requried from the Solidity Snippets Github repo.

// SPDX-License-Identifier: MIT
pragma solidity >=0.8.0;

// Mapping Addresses To Balances
contract AddressMap {
    mapping(address => uint) public balances;

    function addFunds() public {
        balances[msg.sender] = 1000;
    }

    function checkFunds(address _addr) public view returns(uint) {
        return balances[_addr];
    }
}

This approach scales well as each new user covers the gas cost to store their account on chain when they call addFunds(), it minimises initial gas costs and reduces the risk of hitting block gas limits because at no point are we accessing anything other than a single point within a potentially large data set.

Solidity offers various data location options, including storage, memory, and calldata. Understanding and utilising these options appropriately can lead to significant optimisations. For instance, using calldata for function parameters that won’t be modified can save tx gas compared to using memory.

Storing large data in Solidity requires a thoughtful approach that balances on-chain and off-chain solutions, optimises data structures, and leverages Solidity’s unique features. By implementing these best practices, developers can create more efficient, cost-effective, and scalable smart contracts.