Social rules are formulated because people tend to find it easier to cooperate with each other when everyone treats each other in similar ways. It makes communication easier, because if people are following the rules, it is much clearer what someone means when they say or do something.
Generally, social rules are made explicit by the elites of a group of people. The elites have the highest status, and they use this status to make up rules so they can know what people are saying to them. The most important thing that following social rules helps us communicate is our level of respect for others.
There is another level of “formulation” of social rules that happens even if the rules are not made explicit, by, say, writing them down and teaching them in schools or passing them on from parents to children. These “rules” are not explicitly expressed, but they are still there. The only way to figure out these rules is to watch how others behave and to see if you can deduce these rules.
This is how most social “rules” started. They were ways that people developed organically (without conscious thought) to help people in the group get along more easily with each other. It is only later on, long after the rules have been developed in an unacknowledged way that people decide to make them explicit and try to formalize them and make sure they are immune to change.
Unfortunately for those who want the rules to never change, human culture is constantly changing, no matter how hard people try to enforce the rules as they prefer them. Dictionaries are one way that people try to standardize speech. Grammarians make up rules of speech and call one way the “proper” way to speak, and all the alternatives are signs of poor education or breeding. Thus there is more status in speaking properly.
Popular culture generally ignores these rules, and language changes all the time. The rules can never hold up for very long against the weight of popular culture, so we end up with constant sniping between people who are trying to fight a rearguard action to keep the rules of grammar and language the same, and the brute force of popular culture which ignores the rules, allowing language to live and change.
Social rules are a similar story. People try to enforce one set of rules as a means to enhance your status. Popular culture overwhelms the rules by sheer force of numbers of people changing the way they behave without any thought or planning. High status members of society continue to keep rules in place because it is a useful signifier of who belongs to the high status group, and it is, inevitably, one way people recognize who is high status and who isn’t.
Religions are another mechanism to try to formalize social rules and to indicate who belongs to the high status group (those with that particular religion) and the heathens or hoi polloi — those who don’t or can’t follow the formalized social rules of the religion.
The why behind social rules is twofold, I think. First it helps people sort themselves out into us and them. Second, it is a way of both identifying those with high status and conferring high status on someone. Notice that this is a sociological “why” there are formulated social rules. I am trying to describe the function these rules serve. I am not saying the rules are right or wrong, even though people who follow the rules want everyone else to believe their way of doing things is right and all other ways are wrong.
The rules, like words, are arbitrary. You can use any sound you want to express a particular idea. Once people agree that sound has a specific meaning, we tend to try to lock it to that meaning henceforth. Social rules are also arbitrary. People try to lock them in. When there are revolutions against older forms of social order, the new order often changes the social rules as a symbol that a new group of people are now in power.
For example, after the American revolution against England, Americans changed some rules of dining etiquette. The Brits keep a fork in their left hand and a knife in their right hand. Americans said that when you cut things, keep the fork in your left hand and knife in your right hand. However, when you feed yourself. switch the fork to the right hand. This was symbolic that a new regime was in power. It doesn’t make sense in terms of efficiency, but it creates a new standard for dining etiquette.
Of course, most people are not trained in these rules, so just by watching how a person handles their silverware, you can tell how educated and aculturated they are. The rules are arbitrary, but they have great symbolic meaning, and they do help people communicate more effectively. That’s both how and why social rules are formulated.