In pre-historic times, way back before C.B.G.B.'S, (when I was in High school), music was not so compartmentalized as it is now. Everything was lumped into one big giant category called rock. There were a few sub-genre's. There was not even a genre yet called "Heavy Metal" we used to refer to bands like Black Sabbath or Jimi Hendrix as "Acid Rock" or Hard Rock. Alice Cooper, Bowie, T.Rex, etc. were "Glitter Rock" (Not Glam). The California Hippie bands were called "Country Rock: & Bob Dylan was "Folk Rock".
The first time I heard the term "Punk Rock" was in around 74-75 when The NY Daily News did a story on the NY Underground Rock scene & used the term as their Byline. There was a picture of The Dictators on the cover of the Sunday entertainment section with those words as the headline. It stuck. As part of that scene, we started referring to it as "Punk Rock". It was way different from what it morphed into & what people nowdays think of as "Punk Rock".
Basically the idea came from an attitude. Not from an image. The subject matter in the song lyrics is what pieced the scene together. Not the look. That came way after.
It's why you could have so many diverse bands come from the same scene, though they looked different & sounded different, there was a common thread in the attitude.They were all non conformists. In 1974, it was quite revolutionary to hear a song with lyrics like "Beat On the Brat", (Ramones) "Rip Her To Shreds" (Blondie) "Psycho Killer (Talking Heads). The Clash fit right into that attitude with songs like "White Riot".
In the beginning, this was a relatively small local scene.I can remember early Ramones shows where there were maybe 20 people in the room. As the idea spread, first across the Atlantic & then out West, the kids in those scenes put their own spin on things until Punk became exactly what it started out to tear down. It's now a pre-conceived notion. A genre with a look & sound that people expect. Take for instance, the mosh pit. I went to C.B.G.B's or Max's just about every weekend from 1974-1983 &I never saw a Mosh Pit. The first time I did see a mosh pit was in San Francisco in 1981 at a Dead Kennedy's show. Now, when you think of Punk, you think Mosh pit. However, the "Punk Scene" had existed for 6 or 7 years before the Mosh Pit became the norm.
Some might argue that the 60's Garage Punk scene came before, but that genre was named "Garage Punk" retrospectively. it started to be called that after The Nuggets compilation came about, this was after the NY Underground scene was already being labeled "Punk Rock".
So this is why people like Patti Smith & Television are or were called "Punk Rock" and younger kids today do not understand this at all because, they see Punk as something completely different.
My best advice. Don't worry about it. if you like something who cares what label it falls under?