By default, the azure app will be unloaded if the site is not used for more than 20 minutes, and the site will take a long time to respond for the first request after unloading. This can be avoided by using "AlwayOn" settings in the portal, but this seems to be not working sometime.
To fix this issue, we need to provide the self requesting feature inside the report server. This feature will keep request the site every 2 minutes. This can be turned on and off at any time.