Hi, Brandon! I definitely think you should go for it! I’ve heard some excellent male harpists and have never thought they were “out of place”.
I’m really interested in the comment you made about playing the accordion. My sister plays it very well-she’s primarily a classical accordionist. In case you have never tried it, accordion and harp are fantastic together. Seems like a strange idea at first, but try it- it’s a lot of fun!