In smart grid infrastructure, multimedia communication plays an important role in various applications, for instance, load monitoring, automatic smart meter reading, and energy management. Energy management has gained widespread popularity with the increasing energy demand. The goal of this paper is to explore multiagent-based Reinforcement Learning (RL) for multicarrier (i.e., electricity and gas) Residential Energy Management (REM) systems with data security. It facilitates the possibility for a separate Demand Response Program (DRP) for every energy component accelerates with computational aspects of RL. This paper proposes Q-MSEM, a Q-learning-based Multiagent and Secure Energy Management scheme for the optimal solution of REM problems using RL and Ethereum Blockchain (EB) to reduce energy load and decrease energy costs. Then, Q-MSEM uses Ethereum Smart Contract (ESC) to covenants data security issues using off-chain storage, i.e., InterPlanetary File System (IPFS) to handle data storage costs issues. Experimental results disclose the effectiveness of the proposed Q-MSEM scheme in terms of reduction in load, energy cost (15.82%), and data storage cost.