I imagine “Maces and Talons” is a game of the RTTE writers’ invention. However, I also suspect it is based off of real board games.
Most clearly to us modern viewers, “Maces and Talons” evokes something like chess. There are two players who try to conquer each others’ “army.” The different pieces on the board represent different types of fighters, and the different pieces appear to have different roles.
We know one of the pieces is the Viking chief, “the honorable Viking chief,” as Viggo calls him. When Dagur and Ryker play, their men actually acting it out, Dagur and Ryker also mention a Viking king (maybe the same thing as the chief?), a longship, and a dragon hunter’s kingship. Ryker says that there are eight hunters in the game, as well. This suggests roles like what we have in chess – knights, pawns, queens, etc.
According to this screencap, other game pieces appear to include a ship, a mace, a dragon, other warriors, and, of course, the traitor.
What’s interesting is that, historically, Vikings have had board games. Specifically, it’s suspected that Vikings might have had an early precursor to chess called Shatranj which they picked up from trade in Constantinople. Being able to play board games was considered to be something of an accomplishment and show of esteem – much like Fishlegs says, “Maces and Talons was designed to test the abilities and decision making expertise of Viking chiefs in the heat of battle.” There is a parallel that we could draw here to actual Viking epics where skill at board games was considered as great of an ability as physical strength.
It’s possible that the Viking board game hnefatafl (played around the fourth to twelfth centuries, during Hiccup’s theoretical lifetime) was used as some inspiration for “Maces and Talons.” This game is, to quote “a game of skill in which a king tries to escape, with the help of his men, from a numerous besieging army.” It is a chess-like strategy game, the concept similar to what we know about Maces and Talons. I think it is EXTREMELY possible that Maces and Talons is inspired by this old Viking strategy board game. Maces and Talons does seem to be a game where one side chooses to be a Viking chief or Viking king, while the other side – the side more “morally gray” according to Viggo – seems to be acting in a different role.
The game board for hnefatafl is maaaybe set up in a similar way to Maces and Talons:

Source: here.
If you peep into the box from the RTTE screencaps, Maces and Talons appears to be set up with four sides and then a central island. It’s somewhat a similar concept. You could argues there’s inspiration.
The last cool thing is the game pieces look slightly akin to some actual historical Viking chess pieces. This is “the honorable Viking chief” from RTTE:

And here are some late Twelfth Century game pieces from Scandinavia:

Source: here.
They’re not exactly alike, but there’s enough of a fun resemblance to consider there might have been some thought and research into the development of Maces and Talons.
This game from RTTE appears to be its own unique thing. Still, I imagine that they drew inspiration from Viking board games like the one I brought up.
Note: I am not a historian, nor sociologist, nor archaeologist, and essentially am not an expert in ancient culture. If someone with more expertise wants to weigh in with their ideas, I would love to learn!