January 1, 1970Learning to Prompt in Unknown Environments: A POMDP Framework with Compositional Actions for Large Language Models