Nowadays, computer vision plays an essential role in disease detection, computer-aided diagnosis, and patient risk identification. This is especially true for skin cancer, which can be fatal if not diagnosed in its early stages. For this purpose, several computer-aided diagnostic and detection systems have been created in the past. They were limited in their performance because of the complicated visual characteristics of skin lesion images, which included inhomogeneous features and hazy borders. In this paper, we proposed two methods for detecting and classifying dermoscopic images into benign and malignant tumors. The first method is using k-nearest neighbor (KNN) as classifier when pretrained deep neural networks are used as feature extractors. The second one is AlexNet with grey wolf optimizer, that optimizes AlexNet's hyperparameters to get the best results. We also tested two approaches in classifying skin cancer images, which are machine learning (ML) and deep learning (DL). The used methods in ML approach are artificial neural network, KNN, support vector machine, Naive Bayes, and decision tree. The DL approach that we used contains convolutional neural network and pretrained DL networks: AlexNet, VGG-16, VGG-19, EfficientNet-b0, ResNet-18, ResNet-50, ResNet-101, DenseNet-201, Inception-v3, and MobileNet-v2. Our experiments are trained and tested on 4000 images from the ISIC archive dataset. The outcomes showed that the proposed methods outperformed the other tested approaches. Accuracy of first proposed method exceeded 99% in some models and second proposed method achieved 99%.