A novel method of microscopy observation combined with target recognition based on Artificial Neural Network (ANNs) was developed to study the gelatinization characteristics of starch, and the on-line detection of the gelatinization characteristics of potato starch was realized by machine learning algorithm. In this paper, a detector based on deep neural network (starch-SSD) was designed to monitor the morphological changes of potato starch with the increasing of temperature. The starch birefringence characteristics were automatically identified first and then, the number of starches with birefringence characteristics in each micrograph during the process of gelatinization was calculated. The gelatinization temperature of starch and the degree of gelatinization (DG) at different temperatures were determined according to the change of birefringence. Results showed that compared with the traditional methods based on artificial vision observation, the proposed method had the advantages of high precision and fast speed. In addition, this method overcame the uncertainty of artificial subjective judgments and provided a uniform standard for judging the gelatinization characteristics of starch by microscopic. This study would provide the possibility of on-line monitoring during the industrialized production of starch products.