I’m sorry for super late reply.
The problem is not in Ubuntu or any server. It had something to do with my internet router settings. I figured out some the things which i had done in that period of time that could have affected the server connection with my system. So one thing i did was change the ports. I called my ISP and he fixed the router and after that i did not encounter this issue even once.
Thanks for being so active. If not this way then i would have eventually solved it through your help.