Python json dumps double backslash

Another situation where this unwanted escaping can happen is if you try to use json.dump[] on the pre-processed output of json.dumps[]. For example

import json, sys
json.dump[{"foo": json.dumps[[{"bar": 1}, {"baz": 2}]]},sys.stdout]

will result in

{"foo": "[{\"bar\": 1}, {\"baz\": 2}]"}

To avoid this, you need to pass dictionaries rather than the output of json.dumps[], e.g.

json.dump[{"foo": [{"bar": 1}, {"baz": 2}]},sys.stdout]

which outputs the desired

{"foo": [{"bar": 1}, {"baz": 2}]}

[Why would you pre-process the inner list with json.dumps[], you ask? Well, I had another function that was creating that inner list out of other stuff, and I thought it would make sense to return a json object from that function... Wrong.]

Replies: 2 suggested answers 2 replies

Add heading textAdd bold text, Add italic text,

Add a quote, Add code,

Insert Link

Add a link,

Add a bulleted list, Add a numbered list, Add a task list,

Directly mention a user or teamReference an issue or pull request

Add heading textAdd bold text, Add italic text, Add a bulleted list, Add a numbered list, Add a task list,

reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji

I am using jsonnet to write configurations for drone ci. Drone uses jsonnet to generate yaml file and use it as instruction for CI/CD.

For my needs I want to have the following output:

{ "some-field" : "${DRONE_BRANCH/\//-}" }

I tried the following jsonnet script:

{
  "some-field": "${DRONE_BRANCH/\//-}",
}

And it gave me:

{
   "some-field": "${DRONE_BRANCH///-}"
}

Then I tried:

{
   "some-field": "${DRONE_BRANCH/\\//-}"
}

It gave me:

{
   "some-field": "${DRONE_BRANCH/\\//-}"
}

I did not find a way to achieve what I want. Seems to me it's currently impossible.

P.S. I am using the latest version of jsonnet

$ jsonnet -v
Jsonnet commandline interpreter v0.13.0

alikhil, Raiondesu, prokop7, kevinkhanda, nickaleks, IvanovaChristina, Vsnegovik, gadhagod, and gaurav517 reacted with thumbs up emojiRaiondesu, kevinkhanda, Vsnegovik, and gaurav517 reacted with rocket emoji

+10 for this. How come single slashes are not allowed in a JSON?
This is ridiculous.

The thing is that in JSON "/" and "\/" are equivalent, i.e. they describe the same sequence of characters [i.e. a single character U+002F]. If I'm interpreting the JSON specification correctly, every compliant implementation should treat them the same per RFC 7159 8.3 String Comparison.

Similarly JSON strings "${DRONE_BRANCH/\//-}" and "${DRONE_BRANCH///-}" refer to exactly the same sequence of characters ${DRONE_BRANCH///-}. If you want a JSON string which denotes a string with a backslash there [e.g. because you feed it to a script which requires an escaped slash there], you should actually escape the backslash. So if you want a JSON meaning ${DRONE_BRANCH/\//-}, the right JSON is "${DRONE_BRANCH/\\//-}.

Python agrees btw:

>>> json.dumps[json.loads['"${DRONE_BRANCH/\\//-}"']] # double escaping
'"${DRONE_BRANCH///-}"'
>>> x = '"${DRONE_BRANCH/\\//-}"' # double escaping
>>> print[x]
"${DRONE_BRANCH/\//-}"
>>> print[json.loads[x]]
${DRONE_BRANCH///-}
>>> x = '"${DRONE_BRANCH/\\\\//-}"' # double escaping
>>> print[x]
"${DRONE_BRANCH/\\//-}"
>>> print[json.loads[x]]
${DRONE_BRANCH/\//-}

Escaping / is unnecessary, so Jsonnet doesn't put it there. Jsonnet generally goes for minimally escaped strings in the output.

Please let me know if that answers your question.

@sbarzowski, though it's a comprehensive answer, I suggest you pay more attention when reading the question.

Then I tried:

{
  "some-field": "${DRONE_BRANCH/\\//-}"
}

It gave me:

{
  "some-field": "${DRONE_BRANCH/\\//-}"
}

In other words, escaping the escaping backslash just produces two backslashes.

Therefore, the last bit of "example output" you've provided is not how jsonnet handles this.

Instead, it actually produces

>>> print[x]
"${DRONE_BRANCH/\\//-}"
>>> print[json.loads[x]]
${DRONE_BRANCH/\\//-}

Instead, it actually produces

Did you actually run it in Python? This was directly copy&pasted from Python shell. Here you can see it yourself: //www.onlinegdb.com/S1g978ilH.

In other words, escaping the escaping backslash just produces two backslashes.

Yes, because it also needs to be escaped in the output JSON. I'm arguing that it is the correct behavior. And that trying to get JSON output { "some-field" : "${DRONE_BRANCH/\//-}" } specifically and not { "some-field" : "${DRONE_BRANCH///-}" } is probably a wrong goal, because these two are equivalent. These two JSONs mean exactly the same thing, so why would you prefer the one with unnecessary escaping?

For completeness I'll add that you can use the "raw string" mode [-S option] to do your own serialization if you really want to. Then your program must produce a string, which will be printed without any additional escaping. But as long as your output is standard JSON you shouldn't have to.

In other words if you want it to be loaded with a single backslash you need these two backslashes in JSON.

E.g.:

➜ jsonnet -e '{
   "some-field": "${DRONE_BRANCH/\\//-}"
}' | python -c 'import json; import sys; print[json.load[sys.stdin]["some-field"]]'
${DRONE_BRANCH/\//-}

With a single backslash it won't work as expected [it will be interpreted as escaped / which is just /]:

➜ cat foo.json
{ "some-field" : "${DRONE_BRANCH/\//-}" }
➜ cat foo.json | python -c 'import json; import sys; print[json.load[sys.stdin]["some-field"]]'
${DRONE_BRANCH///-}

I'm going to assume that it's all clear now after my last explanation. If it's not, feel free to reopen.

Just came here to add that in YAML, you can specify a single character string containing a backslash as either:

or

In JSON, only the latter is supported. When Jsonnet generates YAML with std.manifestYAMLDoc, it chooses to emit YAML in the second form, because implementing that is simpler, and it's equivalent. Drone CI should interpret "\\" in the way that you want.

Oh and just for completeness -- the various ways of specifying it in Jsonnet:

[
  "\\",
  '\\',
  @"\",
  @'\',
  // Note, this last one also adds a \n, see #289
  |||
    \
  |||,
]

How do you escape a double quote in JSON Python?

Why does my JSON have backslash?

What is the difference between JSON dump and JSON dumps?

How do I ignore a double quote in Python?