Magnolia little gem trees are very tough plants. These plants tend to grow better in acidic soil so add some fertilizer or coffee in the soil to make it a little acidic this can improve the health of your plant. There could also be the problem of under-nourishment since the plant is in a pot. Try planting it in a garden or giving it additional water and fertilizer etc.
"Little Gem" magnolias should be planted in the ground, at their oldest they only reach about 25' to 30' but if you want to keep it potted you need to repot into a bigger pot, maybe 2 sizes bigger than the one they are in and you need to use a good time-released fertilizer.