Sediment heavy metals are significant for global seafood safety and human health, and there is no research detailing the characteristics and sources of sediment heavy metals in remote island areas. Sediments in the northern four-island area of Yantai, China were sampled. Enrichment factor (EF), Geoaccumulation index (I-geo), Potential ecological risk (ER) and Risk index (RI) were used to assess the contamination. Generally, sediment Cr, Cu, Pb, Zn, Co and Hg are no or minor enrichment (EF<3), no pollution (I-geo<0) and low potential ecological risk (ER<40). Besides, 20.87% Cd and 10.43% As are moderate enrichment (3<EF<5), 36.63% Cd are no to moderate pollution (0<I-geo<1), and 44.55% Cd and 7.92% Hg falls within a moderate potential risk (40<ER<80). However, the average sediment heavy metals are lower than those in shallow sea in China except Cd. All heavy metals in the samples fall within the safety standards referencing to the Oceanic Sediment Quality of China. Such facts indicate the sediments have low heavy metal levels because of the remote distance and fewer anthropogenic inputs. Correlation and Principal Component Analysis (PCA) are used to identify the sources of heavy metals. The prevailing culturing activities slow down ocean current and cause fine sediment, which deeply affects sediment Cr, Cu, Pb, Zn, Co, Cd and Hg levels. Additionally, the daily pollution discharge is another source of sediment Hg and Cd, and shipping is another source of sediment Pb. Sediment As is not influenced by grain sizes, but by the ocean current. The findings provide information on the scientific management, development and utilization of marine space in remote island areas.