Lookup method interpreting result

I am using a lookup method which returns a password (using cyberark.. but that is not relevant for this issue description)

When the password happens to contain a curly brace and there is a variable in second lookup parameter, the result is interpreted by jinja.

Here is a simplified example.

Among the inventory variables I have the following values:

existing_var: 'just a string'
ansible_password: '{{ lookup("mylookup", "{{ existing_var }}") }}' 

Where mylookup is a test lookup method, that returns always a single string "x{g{%Y" such as below.
(this has to be stored in the lookup folder specified in the ansible config files, with name "mylookup.py".)

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
from ansible.errors import AnsibleError
from ansible.plugins.lookup import LookupBase

class LookupModule(LookupBase):
    def run(self, terms, variables=None, **kwargs):
        #we return always the same string without taking account of any parameter.
        return ["x{g{%Y"]

I don't expect the lookup result to be interpreted.

But it is.

This lead to an ansible interpretation error : "encountered unknown tag 'Y'.. string: {%Y".

Of course being a password in my real world case, the result is not meant to be interpreted.

I have noticed that this does not occur when there is no curly brace in the lookup second parameter.

ansible_password: '{{ lookup("mylookup", "simple string without curly braces.")}}' 

How to avoid this interpretation?

To solve this issue, I have tried to increase log verbosity, and activated debug. There was no useful log, ansible fails as soon as the lookup method is called.
I have stubbed the complex password lookup method to simplify the issue and present it here.
I have tried to use the filters "safe" and "string", to no avail.
I have removed the second parameter, and observed that in this case, the returned string is not interpreted.

Q: "How to avoid this interpretation?" (of the lookup plugin)

A: The problem is not caused by the lookup plugin. Ansible does not template lookups. See Unsafe or raw strings:

> "When handling values returned by lookup plugins, Ansible uses a data type called unsafe to block templating."

Create a mre project for testing

shell> tree .
.
├── ansible.cfg
├── hosts
├── pb.yml
└── plugins
    └── lookup
        └── mylookup.py

2 directories, 4 files

Configure the path to your custom lookup plugin DEFAULT_LOOKUP_PLUGIN_PATH

shell> cat ansible.cfg 
[defaults]
gathering = explicit
collections_path = $HOME/.local/lib/python3.9/site-packages/
inventory = $PWD/hosts
lookup_plugins = $PWD/plugins/lookup
retry_files_enabled = false
stdout_callback = yaml

Create the custom lookup plugin

shell> cat plugins/lookup/mylookup.py 
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

from ansible.plugins.lookup import LookupBase


class LookupModule(LookupBase):
    def run(self, terms, variables=None, **kwargs):
        return ["x{g{%Y"]

, simple inventory

shell> cat hosts
localhost

, and a playbook

shell> cat pb.yml 
- hosts: localhost

  vars:

    test_passwd1: "{{ lookup('mylookup') }}"
    test_passwd2: "{{ lookup('mylookup', 'simple string without curly braces') }}"

    test_passwd3: "{{ lookup('mylookup', existing_var) }}"

  tasks:

    - debug:
        var: test_passwd1
    - debug:
        var: test_passwd2

    - debug:
        var: test_passwd3
      vars:
        existing_var: !unsafe x{g{%Y

    - debug:
        var: test_passwd3
      vars:
        existing_var: x{g{%Y

Results:

• Ansible does not template lookups

  test_passwd1: "{{ lookup('mylookup') }}"

gives

  test_passwd1: x{g{%Y

• There is no difference if you ad a literal parameter. The custom lookup plugin mylookup doesn't require a parameter. We add it here to show the expansion of this additional parameter may cause the problem

test_passwd2: "{{ lookup('mylookup',
                         'simple string without curly braces') }}"

gives the same result

  test_passwd2: x{g{%Y

• You have to mark !unsafe a variable that comprises special characters like { or % if you want to expand it

  existing_var: !unsafe x{g{%Y
  test_passwd3: "{{ lookup('mylookup', existing_var) }}"

works fine

  test_passwd3: x{g{%Y

• If you omit !unsafe

  existing_var: x{g{%Y
  test_passwd3: "{{ lookup('mylookup', existing_var) }}"

the expansion fails:

> msg: 'An unhandled exception occurred while templating ''{{ lookup(''mylookup'', existing_var) }}''. Error was a <class ''ansible.errors.AnsibleError''>, original message: An unhandled exception occurred while templating ''x{g{%Y''. Error was a <class ''ansible.errors.AnsibleError''>, original message: template error while templating string: Encountered unknown tag ''Y''.. String: x{g{%Y. Encountered unknown tag ''Y''.'

<hr>

<sup>

Note: Test !unsafe. Given the file

shell> cat test_passwd.txt
x{g{%Y

• The play below works as expected because Ansible blocks the templating of the lookup plugins' output

shell> cat pb.yml 
- hosts: localhost

  vars:

    test_passwd: "{{ lookup('file', 'test_passwd.txt') }}"

  tasks:

    - debug:
        var: test_passwd

shell> ansible-playbook pb.yml 

PLAY [localhost] ******************************************************************************

TASK [debug] **********************************************************************************
ok: [localhost] => 
  test_passwd: x{g{%Y

PLAY RECAP ************************************************************************************
localhost: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

• The problem also does not appear when you declare a variable is !unsafe. For example,

    - debug:
        var: test_passwd
      vars:
        test_passwd: !unsafe x{g{%Y

gives

  test_passwd: x{g{%Y

• The task will fail when you omit !unsafe

    - debug:
        var: test_passwd
      vars:
        test_passwd: x{g{%Y

fails

> msg: 'An unhandled exception occurred while templating ''x{g{%Y''. Error was a <class ''ansible.errors.AnsibleError''>, original message: template error while templating string: Encountered unknown tag ''Y''.. String: x{g{%Y. Encountered unknown tag ''Y''.'

</sup>

The issue you're experiencing occurs because when the password returned by the mylookup lookup method contains curly braces, Jinja interprets it as a template tag, leading to errors.

To avoid this interpretation and ensure that the password string is treated as a literal, you can use the quote filter provided by Jinja. The quote filter escapes characters in a string to prevent them from being interpreted as template tags.

Here's an example of how you can modify your code to use the quote filter:

ansible_password: '{{ lookup("mylookup", existing_var | quote) }}'