In recent years, blockchain has been employed to realize decentralization and meanwhile provide transparency, immutability, and auditability for peer-to-peer (P2P) energy trading. Despite these advantages, blockchain-based P2P energy trading schemes grapple with issues concerning fair exchange and potential malpractices. Current solutions focus on either privacy or fair exchange, but rarely both. This paper introduces an innovative blockchain-based P2P energy trading scheme that harnesses the power of the Trusted Execution Environment (TEE) and smart contracts to boost privacy and fairness in energy trading processes, respectively. Specifically, energy aggregators who maintain the blockchain ledger and execute trading operations are equipped with TEE, which guarantees confidentiality and thereby enhances privacy. Furthermore, a two-stage atomic delivery scheme is implemented via a smart contract, designed to deter prosumers from committing fraudulent acts in energy trading, thus promoting fair exchange. Based on the security analysis and experimental tests, the proposed scheme preserves privacy and fair exchange with acceptable costs.