Are there cons to not having multiple farms?

I just unlocked the Elven farm and tbh, I am not a fan of having multiple farms. I'd rather just have my one farm because the upkeep of many farms seems like too much.

I know I can only grow certain crops on each farm. But is it necessary for the game to have all 3 farms? Can I buy a barn animal from the elf city and keep it on my sun haven farm?