In common Nationalism is based on the concept of a common identity of a group of individuals and is aiming at the preservation of this identity, its traditions and the underlying values. If the preservation of values bolt there is a change that one's own identity is idolised which result in xenophobia.
On Ireland however the term Nationalism is related to Ireland and Being Irish. There are numerous Protestants participating in organisations strictly based on their common religion and genealogy, but only one percent of the Northern Irish Protestants think of themselves as being a Nationalist, against 60% of the Catholics. In the past this ratio was more balanced. The founders of the Society of United Irishmen for example were all Protestant.
As a rule of thumbs Northern Irish Nationalists are aiming at unification with the Republic of Ireland, but in contrast to Republicans they are more moderate and try to work within the existing political system. There are however also Nationalists who are satisfied when they can profess their religion, speak their own tongue and live their life with freedom of choice.
The Northern Irish Social Democratic and Labour Party (SDLP) can be labelled as a Nationalist party. The participation of Sinn Féin in the public political process on the other hand is rather fresh and their connections with Republican organisations are too obvious to ignore.
Usually the term Republican means a supporter of a republic as form of government, and is in modern times the counterpart of a Monarchist.
In Northern Ireland the definition must be narrowed. The Irish term Republican originates from those who supported and fought for the declaration of the Irish Republic during the War of Independence. Northern Irish Republicans are not aiming at overthrowing the monarch and the establishment of some sort of an Ulster Republic. Their goal is to include Northern Ireland in the Republic of Ireland. They are primary supporters of the Republic of Ireland and not particularly interested in the republican manner to govern a country.
Because Northern Irish Republicanism implies revolution rather than evolution the use of the term is restricted to parties operating at the edge or outside the public political spectrum. After all, one can not participate in a system you try to destroy.
All branches of the Irish Republican Army (IRA), such as the Provisional Irish Republican Army (pIRA) and the Real Irish Republican Army (rIRA), meet this definition.
The Irish National Liberation Army (INLA) goes one step further. This organisation pursuit a 32-county socialist republic.
In theory a Protestant can be a Republican, but in real life you will have a hard time finding one.