Yes, the temp will affect you POI. How much will depend on the powder you use and how temperature sensitive it is, the primer you use and how much your barrel changes due to the temperature change. So the answer to your original question is YES.
There are a couple of ways to deal with that information. One way is to go shoot groups during all weather events, from hot to cold in 10° increments. In rain, humidity, dryness, snow, that you would expect during your hunt. Don't forget if you plan to travel to shoot groups at all elevations in all conditions as well. Log it all carefully and at the end you will have definitive information....except you will now need a new barrel and you will have to start all over again, BUT....the short answer which I think you wanted is. YES you POI will change. Will that matter? That's up to you.