First and foremost, let's get one thing straight: Both are forms of vandalism. One does not supersede the other as a criminal act, and neither is "less destructive" than the other. All clear on that? Good. Let's proceed.
I consider myself to be somewhat of a street artist. I also dabble in graffiti here and there, but if I had to categorize myself I'd definitely claim the street art scene. That said, A couple of weeks back I was having a discussion on the subject with a friend of mine who happens to be a "writer" (writer meaning she graffs), and there were certain things we both agreed on that distinguishes the two art forms from eachother:
Graffiti is a subculture that's full of tradition and history. A lot of it has to do with "getting up", or writing your name with the means of gaining respect amongst other writers. It's very territory-driven, as writers use their art for the purpose of claiming a space and making their mark. It's something that's done on the spot, usually free-handed, with the traditional tools. Lastly, and I think this is the greatest distinction between the two cultures, graffiti can often be confusing to an outsider. Because tags are usually written in styles that're somewhat hard to read to untrained eyes, which means it takes a while to learn what's happening, it can be a little frightening or unfamiliar. Though there are sometimes exceptions to these rules, these are the basic general ideas behind the graffiti scene.