The accuracy of short-term load forecasting in microgrids is crucial for their safe and economic operation. Microgrids have higher unpredictability than large power grids, making it more challenging to accurately predict short-term loads. To address this challenge, a novel approach that combines the time-varying filtered empirical mode decomposition (TVFEMD), Long Short Term Memory neural network (LSTM), and the simple moving average auto regressive model with additional inputs (ARMAX) methods is proposed. The TVFEMD is used to decompose the load sequences of microgrids, with the permutation entropy (PE) used to calculate the entropy values of subsequences. The model errors of ARMA and LSTM are verified to divide high and low frequencies, and weather and day patterns are selected as influencing factors. The LSTM model forecasts high frequency subsequences, while the ARMAX forecasts low frequency subsequences. The proposed TVFEMD-LSTM-ARMAX model is then applied to two microgrids in Taiyuan, China. The results show that permutation entropy method can accurately divide high and low frequencies, and the proposed TVFEMD-LSTM-ARMAX model can significantly improve the forecasting effect.