Python json dumps escape new line

Valid json expects escaped newline characters to be encoded as '\\n', with two backslashes. I have data that contains newline characters that I want to save to a file. Here's a simplified version:

data = {'mystring': 'Line 1\nLine 2'}

I can encode it with json.dumps():

import json
json_data = json.dumps(data)
json_data
# -> '{"mystring": "Line 1\\nLine 2"}'

When I print it, the newline displays as '\n', not '\\n' (which I find odd but I can live with):

print(json_data)
# -> {"mystring": "Line 1\nLine 2"}

However (here's the problem) when I output it to a file, the content of the file no longer contains valid json:

f = open('mydata.json', 'w')
f.write(json_data)
f.close()

If I open the file and read it, it contains this:

{"mystring": "Line 1\nLine 2"}

but I was hoping for this:

{"mystring": "Line 1\\nLine 2"}

Oddly (I think), if I read the file using python's open(), the json data is considered valid:

f = open('mydata.json', 'r')
json_data = f.read()
f.close()
json_data
# -> '{"mystring": "Line 1\\nLine 2"}'

... and it decodes OK:

json.loads(json_data)
# -> {u'mystring': u'Line 1\nLine 2'}

My question is why is the data in the file not valid json? If I need another - non Python - application to read it it would probably be incorrect. If I copy and paste the file contents and use json.loads() on it it fails:

import json
json.loads('{"mystring": "Line 1\nLine 2"}')
# -> ValueError: Invalid control character at: line 1 column 21 (char 20)

Can anybody explain if this is the expected behaviour or am I doing something wrong?

This is a brief article that I hope will help somebody else in my situation to more easily utilize Python to generate valid JSON files that contain newline characters.

For example, say you’re building a JSON file that should be parsed to display a return (newline) or two when read by a client. You might have a JSON object that looks something like:

{"title":"Sample JSON File","description":"Hello!\n\n This is a new paragraph."}

Simple enough. A client may parse this JSON file and read the newline characters as returns, thereby creating a new paragraph after “Hello!”.

Now let’s say you’re using Python (though this technique could apply to many languages) to generate this file and so you create your string and write the string to the file. You realize that Python has interpreted the newline characters as such and has applied line returns to the file itself. Now your file looks something like:

{"title":"Sample JSON File","description":"Hello!This is a new paragraph."}

Uh oh! Now you have an invalid JSON string.

Let’s fix it.

One way to prevent Python from interpreting the newline characters and placing line returns within your file is to use a simple hack. First define a variable that looks similar to your newline string but instead of the letter “n”, you use something like a capital “X”.

hacky_string = '\X\X'

Awesome, so Python doesn’t see this as anything other than a string. Now let’s use a simple replace function to turn those “X”s back into “n”s.

hacky_string_converted_to_newlines = hacky_string.replace('X', 'n')

Now we’re getting somewhere!

Let’s insert the hacky newline string into our content string using f-string.

file_contents = f'{"title":"Sample JSON File","description":"Hello!{hacky_string_converted_to_newlines} This is a new paragraph."}'

And just like that we’ve injected newline characters into a string which can now be written to a file as valid JSON.

My only hope is that you found this article before giving up on coding altogether :) Toss a few claps my way if you found it useful, and please leave a comment if you know of any other ways to achieve the same thing.

Here is how the string looks like after json.dumps (notice \n in it):

"# HELP python_gc_objects_collected_total Objects collected during gc\n# TYPE python_gc_objects_collected_total counter\npython_gc_objects_collected_total{generation=\"0\"} 1807.0\npython_gc_objects_collected_total{generation=\"1\"} 943.0\npython_gc_objects_collected_total{generation=\"2\"} 19.0\n# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC\n# TYPE python_gc_objects_uncollectable_total counter\npython_gc_objects_uncollectable_total{generation=\"0\"} 0.0\npython_gc_objects_uncollectable_total{generation=\"1\"} 0.0\npython_gc_objects_uncollectable_total{generation=\"2\"} 0.0\n# HELP python_gc_collections_total Number of times this generation was collected\n# TYPE python_gc_collections_total counter\npython_gc_collections_total{generation=\"0\"} 190.0\npython_gc_collections_total{generation=\"1\"} 17.0\npython_gc_collections_total{generation=\"2\"} 1.0\n# HELP python_info Python platform information\n# TYPE python_info gauge\npython_info{implementation=\"CPython\",major=\"3\",minor=\"8\",patchlevel=\"11\",version=\"3.8.11\"} 1.0\n# HELP process_virtual_memory_bytes Virtual memory size in bytes.\n# TYPE process_virtual_memory_bytes gauge\nprocess_virtual_memory_bytes 7.4350592e+07\n# HELP process_resident_memory_bytes Resident memory size in bytes.\n# TYPE process_resident_memory_bytes gauge\nprocess_resident_memory_bytes 6.3066112e+07\n# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.\n# TYPE process_start_time_seconds gauge\nprocess_start_time_seconds 1.63845824433e+09\n# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.\n# TYPE process_cpu_seconds_total counter\nprocess_cpu_seconds_total 0.76\n# HELP process_open_fds Number of open file descriptors.\n# TYPE process_open_fds gauge\nprocess_open_fds 27.0\n# HELP process_max_fds Maximum number of open file descriptors.\n# TYPE process_max_fds gauge\nprocess_max_fds 1.048576e+06\n#

How do you escape a new line in JSON?

Can I use \n in JSON?

What is the difference between JSON dump and JSON dumps?

Should JSON files end with a newline?