I got the work visa but unable to go to Saudi Arabia

Please advise me on my situation. I am a Pakistani and I have got Saudi work visa of 90 days stamped on my passport but I had to refuse the offer because I cannot go to Saudi Arabia now due to personal reasons. Now I am concerned whether the company which gave me offer can ban me from working in Saudi Arabia? Please advise. Is it possible.

Dont worry nothing will happend just refuse them in a proper way