In this paper, a fractal–fractional mathematical model of convective fluid motion in rotating cavity is investigated inside the ellipsoid with inhomogeneous external heating. The fractal–fractional differential operators namely Caputo, Caputo–Fabrizio and Atangana–Baleanu Dτϵ1,τ1\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${\mathcal{D}}_{\tau }^{{\epsilon }_{1},{\tau }_{1}}$$\end{document}, Dτϵ2,τ2\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${\mathcal{D}}_{\tau }^{{\epsilon }_{2},{\tau }_{2}}$$\end{document} and Dτϵ3,τ3\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${\mathcal{D}}_{\tau }^{{\epsilon }_{3},{\tau }_{3}}$$\end{document}, respectively, are used in the non-linear mathematical model of convective fluid motion in rotating cavity. The numerical algorithms have been generated in terms of newly presented fractal–fractional differential operators on the basis of Adams–Bashforth method to compute the approximate solutions explicitly. The equilibrium points and stability analysis of the fractal–fractional Atangana–Baleanu, Caputo–Fabrizio and Caputo differential operators in Caputo sense have been investigated for non-linear mathematical model of convective fluid motion in rotating cavity. The numerical solutions are simulated in three types of variations (i) presence of fractional parameter without fractal parameter, (ii) presence of fractal parameter without fractional parameter, and (iii) presence of fractal parameter as well as fractional parameter. The chaotic behavior of convective fluid motion in rotating cavity based on each fractal–fractional differential operator has been highlighted as (a) projection on the x–y plane, (b) projection on the x–z plane, (c) projection on the y–z plane and (d) projection on the xyz\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$xyz$$\end{document} plane in three dimensions.