My Boyfriend (M23) tried to remove my( 22F) pants

I (22F) and my boyfriend (23M) have been dating since 7 months. I am the type of person who dosen’t want to have sex for now. Recently we were cuddling and he tried to take off my clothes. I got uncomfortable about him trying to remove my pants. Idk if he’s right for me or not. He cares a lot about me but did this as well. Should I break up with him?