The distinguishing number D(G) of a graph G is the least integer d such that G has a vertex labeling with d labels that is preserved only by a trivial automorphism. We say that a graph G is d-distinguishing critical, if D(G) = d and D(H) not equal D(G), for every proper induced subgraph H of G. This generalizes the usual definition of a d-chromatic critical graph. While the investigation of d-critical graphs is a well established part of coloring theory, not much is known about d-distinguishing critical graphs. In this paper we determine all d-distinguishing critical graphs for d = 1, 2, 3 and observe that all of these kind of graphs are k-regular graph for some k <= d. Also, we state some results about the disconnected d-distinguishing critical graph with c connected components such that c >= d/2.