This study presents accuracy assessment of decision tree, random forest, support vector machine, neural network, and Naive Bayes classifiers used in thyroid classification problem. Utilising thyroid data from the University of California, Irvine repository, the study applied synthetic minority oversampling technique to resolve imbalanced dataset and avoid the likelihood of overfitting, reservoir sampling technique to split the augmented data into sample sizes, and 10-fold cross-validation to measure the unbiased accuracy of the models across the sample sizes in Weka. The random forest classifier yielded 99.075% accuracy, decision tree and support vector machine achieved 98.500% accuracy, neural network produced 98.375% accuracy, and the Naive Bayes classifier generated the least classification accuracy of 98.200%. The accuracy assessments across sample sizes are statically identical with each classifier beating the other classifiers on one of the datasets, which revealed the existence of a trade-off between classification accuracy and time complexities.