A quick test seems to show that these settings are only loaded at application startup.
//edit the config file now.
Console.ReadLine();
Console.WriteLine(ConfigurationManager.AppSettings["ApplicationName"].ToString());
Console.WriteLine("Press enter to redisplay");
//edit the config file again now.
Console.ReadLine();
Console.WriteLine(ConfigurationManager.AppSettings["ApplicationName"].ToString());
Console.ReadLine();
You’ll see that all outputs remain the same.