Yes, there are many factors but the main two are predisposition to stress, and metabolism.
The first one is (in non-nerd speak) the way someone reacts to stress. People will either eat more (comfort eaters) or eat less. In this busy world, stress plays a major role in body composition, and whether a person is a comfort eater or food avoider when stressed will determine their predisposition.
Metabolism is secondary.
If someone has a high metabolism (uses more energy to stay alive) then they will naturally lose weight faster than a person with a lower metabolism.
Age and gender are the major contributing factors to metabolism; a younger person has a faster metabolism than an older, and a man would have a faster metabolism than a woman.
In saying that, of every 100 people who lose weight quickly, 99 will put it back on. Consistency is far more important in a weight loss journey than speed