Generative artificial intelligence algorithms have shownto besuccessful in exploring large chemical spaces and designing noveland diverse molecules. There has been considerable interest in developingpredictive models using artificial intelligence for drug-like properties,which can potentially reduce the late-stage attrition of drug candidatesor predict the properties of novel AI-designed molecules. Concurrently,it is important to understand the contribution of functional groupstoward these properties and modify them to obtain property-optimizedlead compounds. As a result, there is an increasing interest in thedevelopment of explainable property prediction models. However, currentexplainable approaches are mostly atom-based, where, often, only afraction of a fragment is shown to be significant. To address theabove challenges, we have developed a novel domain-aware molecularfragmentation approach termed post-processing of BRICS (pBRICS), whichcan fragment small molecules into their functional groups. Multitaskmodels were developed to predict various properties, including theabsorption, distribution, metabolism, excretion, and toxicity (ADMET)properties. The fragment importance was explained using the gradient-weightedclass activation mapping (Grad-CAM) approach. The method was validatedon data sets of experimentally available matched molecular pairs (MMPs).The explanations from the model can be useful for medicinal chemiststo identify the fragments responsible for poor drug-like propertiesand optimize the molecule. The explainability approach was also usedto identify the reason behind false positive and false negative MMPpredictions. Based on evidence from the existing literature and ouranalysis, some of these mispredictions were justified. We proposethat the quantity, quality, and diversity of the training data willimprove the accuracy of property prediction algorithms for novel molecules.