Molecular basis defining the selectivity of substituted isoquinolinones for the melatonin MT2 receptor

Melatonin MT1 and MT2 receptors represent attractive drug targets for the treatment of various disorders. However, the high conservation of the melatonin binding pocket has hindered the development of subtype-selective compounds. By leveraging on the recently resolved crystal structures of MT1 and MT2 receptors, this study aims to elucidate the structural basis of MT2-selectivity of a panel of isoquinolinone derivatives. Molecular modelling and ligand docking approaches were employed to predict residues involved in forming interactions with the MT2-selective isoquinolinones. Seven conserved residues (Asn(175), His(208), Trp(264), Asn(268), Gly(271), Tyr(294) and Tyr(298)) were selected as targets for site-directed mutagenesis. Ca2+ mobilization, cAMP inhibition, phosphorylation of extracellular signal-regulated kinase, and ligand binding assays were performed to functionally characterize the receptor mutants in transfected CHO cells. Unlike melatonin, isoquinolinones bearing a 3-methoxybenzyloxyl substituent were unaffected by alanine substitution at His(208) of MT2. Although alanine substitutions at Tyr(294) or Tyr(298) reduced the potency of melatonin and some isoquinolinones on MT2, similar mutations on MT1 allowed five hitherto ineffective isoquinolinones to act as agonists. An isoquinolinone antagonist bearing a 4-methoxybenzyloxyl moiety turned into an agonist at MT2 mutants with alanine substitutions at His(208), Tyr(294) or Tyr(298). A subset of residues is apparently involved in forming a hydrophobic binding cavity to confer selectivity upon the aromatic substituent of isoquinolinone compounds. Two conserved tyrosine residues on transmembrane helix 7 may confer ligand selectivity at MT1 and MT2 receptors, while a conserved histidine on transmembrane helix 5 is apparently involved in receptor activation.