After reaching Lunden I foolishly freed some NPCs in the Fort in Lundens upper westside. Well that was a decision I came to regret now as one of those Raiders keeps following me. I already teleported away, went to Norway and even restarted the game, but as soon as I come back to Lunden, he's appears quickly. And honestly, I think the only thing that kept him away in other places (when I teleported to Raventhorpe for example) is probably just the fact, that I went somewhere else before he could reach me there as well. Captured my stalker in the screenshot below
At least he fights my enemies as well.