December 31, 1969Learning to Prompt in Unknown Environments: A POMDP Framework with Compositional Actions for Large Language Models