fix: Support writing keys with invalid chars (#255)

[kkozik-amplify]

by @tmccombs:
This actually fixes a few of things related to expressions for object keys:

1. It allows they keys contain characters that aren't valid as identifiers, if the key is a plain string, for example: ":"
2. It removes the need for superflouous parenthesis around expresions in the keys
3. It no longer puts double quotes around an interpolated string in the key.

The second two were actually kind of side-affects of my fix for 1.

If we want to preserve the previous behavior for 2 and 3, I think it wouldn't be too hard to do though.

[kkozik-amplify]

hey @tmccombs
thank you for the fix.
2. and 3. are unwanted side effects though. The parser should take valid hcl2 and transform it into JSON faithfully, without semantic changes, so the parentheses and double quotes should remain in the JSON

[tmccombs]

The parser should take valid hcl2 and transform it into JSON faithfully, without semantic changes

They aren't semantic changes though.

"${(a + b)} is semantically equivalent to "${a + b}".

In hcl, if both "foo" = 1 and foo = 1 are valid, there is no semantic difference between them. (but the bare keyword isn't allowed if the key has special characters, and the quoted version isn't allowed at the top level).

And in both cases, it makes the deserialization of the key consistent with a string value. We don't add those parentheses or quotes when converting a string expression in a value position to json. And with the changes I made to how keys are converted back to hcl, no information is lost.

I could see an argument that those changes are backwards incompatible if there is code that relies on the current behavior. But I also think the current behavior is very surprising (and undocumented).

Also it looks like you made a new PR based on my original PR, so I'm not sure how I would go about changing this.

[kkozik-amplify]

As I said, The parser should take valid hcl2 and transform it into JSON faithfully, without semantic (or syntactic or other) changes, so the parentheses and double quotes should remain in the JSON.

Also it looks like you made a new PR based on my original PR, so I'm not sure how I would go about changing this.

You could open a PR from your fork to this branch.

But I also think the current behavior is very surprising (and undocumented).

https://github.com/amplify-education/python-hcl2/blob/main/README.md#using-inline-expression-as-an-object-key

[tmccombs]

without semantic (or syntactic or other) changes
As it is currently structured, that is an impossible goal, as the dict/json representation has no way to represent comments (actually some comments could be represented with a "//", key, but I don't know how you would preserve the location of the comments), and whitespace information is lost.

And there are many current cases where syntax isn't preserved, for example:

>>> hcl2.loads("""
... resource "a" {
...   tags = {
...     "example:name" = "my thing"
...     "example:usage" = "for my other thing"
...   }
... }
... """)
{'resource': [{'a': {'tags': {'example:name': 'my thing', 'example:usage': 'for my other thing'}}}]}

Notice how in this case the quotes aren't preserved. In fact, if they had been preserved, I wouldn't have found the original bug.

Also, this example:

>>> import hcl2
>>> hcl2.loads("""
... a {
...   x = merge({a = 2, b = "hello"}, var.more)
... }
... """)
{'a': [{'x': '${merge({"a": 2, "b": "hello"}, var.more)}'}]}

Notice how it adds quotes around the keys, and changes "=" to ":".

And there is this:

>>> hcl2.loads("""
...   a {
...     b = "hello ${var.world}"
...   }
...   """)
{'a': [{'b': 'hello ${var.world}'}]}

It could have output a.b as "${"hello ${var.world}"}", but it didn't . Why should we keep the quotes for the key, but not the value?

I think if this is a goal of python-hcl2, then instead of a dict using primitive types, it woudl probably be better to use richer types that preserve more information about the original structure. For example using an Expr object isntead of putting the expression in a string wrapped with ${ and }. Perhaps #203 would help with that.

[tmccombs]

If you feel really strongly about it, I'm ok with removing those changes, but personally I don't see the use in keeping the parentheses and quotes, and don't see them serving any purpose.