I'm using multiprocessing to create a sub-process to my Python app. I would like to share data between my parent process and the child process. it's important to mention that I need to share this asynchronously, means that the child process and the parent process will update the data during the code running.
What would be the best way to perform that?
asked Feb 2, 2016 at 15:25
2
This is one simple example from python documentation -
from multiprocessing import Process, Queue
def f[q]:
q.put[[42, None, 'hello']]
if __name__ == '__main__':
q = Queue[]
p = Process[target=f, args=[q,]]
p.start[]
print q.get[] # prints "[42, None, 'hello']"
p.join[]
You can use pipe as well, Refer for more details - //docs.python.org/2/library/multiprocessing.html
answered Feb 2, 2016 at 15:36
AlokThakurAlokThakur
3,4591 gold badge18 silver badges31 bronze badges
4
Here's an example of multiprocess-multithread and sharing a couple variables:
from multiprocessing import Process, Queue, Value, Manager
from ctypes import c_bool
from threading import Thread
ps = []
def yourFunc[pause, budget]:
while True:
print[budget.value, pause.value]
##set value
pause.value = True
....
def multiProcess[threads, pause, budget]:
for _ in range[threads]:
t = Thread[target=yourFunc[], args=[pause, budget,]]
t.start[]
ts.append[t]
time.sleep[3]
if __name__ == '__main__':
pause = Value[c_bool, False]
budget = Value['i', 5000]
for i in range[2]:
p = Process[target=multiProcess, args=[2, pause, budget]]
p.start[]
ps.append[p]
answered May 5, 2021 at 6:07
grantrgrantr
4947 silver badges10 bronze badges
Source code: Lib/multiprocessing/shared_memory.py
New in version 3.8.
This module provides a class,
SharedMemory
, for the allocation and management of shared memory to be accessed by one or more processes on a multicore or symmetric multiprocessor [SMP] machine. To assist with the life-cycle management of shared memory especially across distinct processes, a
BaseManager
subclass, SharedMemoryManager
, is also provided in the multiprocessing.managers
module.
In this module, shared memory refers to “System V style” shared memory blocks [though is not necessarily implemented explicitly as such] and does not refer to “distributed shared memory”. This style of shared memory permits distinct processes to potentially read and write to a common [or shared] region of volatile memory. Processes are conventionally limited to only have access to their own process memory space but shared memory permits the sharing of data between processes, avoiding the need to instead send messages between processes containing that data. Sharing data directly via memory can provide significant performance benefits compared to sharing data via disk or socket or other communications requiring the serialization/deserialization and copying of data.
classmultiprocessing.shared_memory.
SharedMemory
[name=None, create=False, size=0]¶
Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name.
As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block
that might still be needed by other processes, the close[]
method should be called. When a shared memory block is no longer needed by any process, the
unlink[]
method should be called to ensure proper cleanup.
name is the unique name for the requested shared memory, specified as a string. When creating a new shared memory block, if None
[the default] is supplied for the name, a novel name will be generated.
create
controls whether a new shared memory block is created [True
] or an existing shared memory block is attached [False
].
size specifies the requested number of bytes when creating a new shared memory block. Because some platforms choose to allocate chunks of memory based upon that platform’s memory page size, the exact size of the shared memory block may be larger or equal to the size requested. When attaching to an existing shared memory block, the size
parameter is
ignored.
close
[]¶Closes access to the shared memory from this instance. In order to ensure proper cleanup of resources, all instances should call close[]
once the instance is no longer needed. Note that
calling close[]
does not cause the shared memory block itself to be destroyed.
unlink
[]¶Requests that the underlying shared memory block be destroyed. In order to ensure proper cleanup of
resources, unlink[]
should be called once [and only once] across all processes which have need for the shared memory block. After requesting its destruction, a shared memory block may or may not be immediately destroyed and this behavior may differ across platforms. Attempts to access data inside the shared memory block after unlink[]
has been called may result in memory access errors. Note: the last process relinquishing its hold on a shared memory block may call unlink[]
and
close[]
in either order.
buf
¶
A memoryview of contents of the shared memory block.
name
¶Read-only access to the unique name of the shared memory block.
size
¶Read-only access to size in bytes of the shared memory block.
The following example demonstrates low-level use of
SharedMemory
instances:
>>> from multiprocessing import shared_memory >>> shm_a = shared_memory.SharedMemory[create=True, size=10] >>> type[shm_a.buf] >>> buffer = shm_a.buf >>> len[buffer] 10 >>> buffer[:4] = bytearray[[22, 33, 44, 55]] # Modify multiple at once >>> buffer[4] = 100 # Modify single byte at a time >>> # Attach to an existing shared memory block >>> shm_b = shared_memory.SharedMemory[shm_a.name] >>> import array >>> array.array['b', shm_b.buf[:5]] # Copy the data into a new array.array array['b', [22, 33, 44, 55, 100]] >>> shm_b.buf[:5] = b'howdy' # Modify via shm_b using bytes >>> bytes[shm_a.buf[:5]] # Access via shm_a b'howdy' >>> shm_b.close[] # Close each SharedMemory instance >>> shm_a.close[] >>> shm_a.unlink[] # Call unlink only once to release the shared memory
The following example demonstrates a practical use of the SharedMemory
class with NumPy arrays, accessing the same numpy.ndarray
from two distinct Python shells:
>>> # In the first Python interactive shell >>> import numpy as np >>> a = np.array[[1, 1, 2, 3, 5, 8]] # Start with an existing NumPy array >>> from multiprocessing import shared_memory >>> shm = shared_memory.SharedMemory[create=True, size=a.nbytes] >>> # Now create a NumPy array backed by shared memory >>> b = np.ndarray[a.shape, dtype=a.dtype, buffer=shm.buf] >>> b[:] = a[:] # Copy the original data into shared memory >>> b array[[1, 1, 2, 3, 5, 8]] >>> type[b] >>> type[a] >>> shm.name # We did not specify a name so one was chosen for us 'psm_21467_46075' >>> # In either the same shell or a new Python shell on the same machine >>> import numpy as np >>> from multiprocessing import shared_memory >>> # Attach to the existing shared memory block >>> existing_shm = shared_memory.SharedMemory[name='psm_21467_46075'] >>> # Note that a.shape is [6,] and a.dtype is np.int64 in this example >>> c = np.ndarray[[6,], dtype=np.int64, buffer=existing_shm.buf] >>> c array[[1, 1, 2, 3, 5, 8]] >>> c[-1] = 888 >>> c array[[ 1, 1, 2, 3, 5, 888]] >>> # Back in the first Python interactive shell, b reflects this change >>> b array[[ 1, 1, 2, 3, 5, 888]] >>> # Clean up from within the second Python shell >>> del c # Unnecessary; merely emphasizing the array is no longer used >>> existing_shm.close[] >>> # Clean up from within the first Python shell >>> del b # Unnecessary; merely emphasizing the array is no longer used >>> shm.close[] >>> shm.unlink[] # Free and release the shared memory block at the very endclass
multiprocessing.managers.
SharedMemoryManager
[[address[,
authkey]]]¶A subclass of BaseManager
which can be used for the management of shared memory
blocks across processes.
A call to start[]
on a SharedMemoryManager
instance causes a new process to be started. This new
process’s sole purpose is to manage the life cycle of all shared memory blocks created through it. To trigger the release of all shared memory blocks managed by that process, call shutdown[]
on the instance. This triggers a SharedMemory.unlink[]
call on all of the
SharedMemory
objects managed by that process and then stops the process itself. By creating SharedMemory
instances through a SharedMemoryManager
, we avoid the need to manually track and trigger the freeing of shared memory resources.
This class provides methods for creating and returning
SharedMemory
instances and for creating a list-like object
[ShareableList
] backed by shared memory.
Refer to multiprocessing.managers.BaseManager
for a description
of the inherited address and authkey optional input arguments and how they may be used to connect to an existing SharedMemoryManager
service from other processes.
SharedMemory
[size]¶Create and return a new
SharedMemory
object with the specified size
in bytes.
ShareableList
[sequence]¶Create and return a new
ShareableList
object, initialized by the values from the input sequence
.
The following example demonstrates the basic mechanisms of a SharedMemoryManager
:
>>> from multiprocessing.managers import SharedMemoryManager >>> smm = SharedMemoryManager[] >>> smm.start[] # Start the process that manages the shared memory blocks >>> sl = smm.ShareableList[range[4]] >>> sl ShareableList[[0, 1, 2, 3], name='psm_6572_7512'] >>> raw_shm = smm.SharedMemory[size=128] >>> another_sl = smm.ShareableList['alpha'] >>> another_sl ShareableList[['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221'] >>> smm.shutdown[] # Calls unlink[] on sl, raw_shm, and another_sl
The following example depicts a potentially more convenient pattern for using SharedMemoryManager
objects via the with
statement to ensure that all shared memory blocks are released after they are no longer needed:
>>> with SharedMemoryManager[] as smm: ... sl = smm.ShareableList[range[2000]] ... # Divide the work among two processes, storing partial results in sl ... p1 = Process[target=do_work, args=[sl, 0, 1000]] ... p2 = Process[target=do_work, args=[sl, 1000, 2000]] ... p1.start[] ... p2.start[] # A multiprocessing.Pool might be more efficient ... p1.join[] ... p2.join[] # Wait for all work to complete in both processes ... total_result = sum[sl] # Consolidate the partial results now in sl
When using a SharedMemoryManager
in a with
statement, the shared memory blocks created using that manager are all released when the
with
statement’s code block finishes execution.
multiprocessing.shared_memory.
ShareableList
[sequence=None, *,
name=None]¶Provides a mutable list-like object where all values stored within are stored in a shared memory block. This constrains storable values to only the int
, float
, bool
, str
[less than 10M bytes each], bytes
[less than
10M bytes each], and None
built-in data types. It also notably differs from the built-in list
type in that these lists can not change their overall length [i.e. no append, insert, etc.] and do not support the dynamic creation of new ShareableList
instances via slicing.
sequence is used
in populating a new ShareableList
full of values. Set to None
to instead attach to an already existing ShareableList
by its unique shared memory name.
name is the unique name for the requested shared memory, as described in the definition for SharedMemory
. When attaching to an existing ShareableList
,
specify its shared memory block’s unique name while leaving sequence
set to None
.
count
[value]¶Returns the number of occurrences of value
.
index
[value]¶Returns first index position of value
. Raises ValueError
if value
is
not present.
format
¶Read-only attribute containing the struct
packing
format used by all currently stored values.
shm
¶The
SharedMemory
instance where the values are stored.
The following example demonstrates basic use of a
ShareableList
instance:
>>> from multiprocessing import shared_memory >>> a = shared_memory.ShareableList[['howdy', b'HoWdY', -273.154, 100, None, True, 42]] >>> [ type[entry] for entry in a ] [, , , , , , ] >>> a[2] -273.154 >>> a[2] = -78.5 >>> a[2] -78.5 >>> a[2] = 'dry ice' # Changing data types is supported as well >>> a[2] 'dry ice' >>> a[2] = 'larger than previously allocated storage space' Traceback [most recent call last]: ... ValueError: exceeds available storage for existing str >>> a[2] 'dry ice' >>> len[a] 7 >>> a.index[42] 6 >>> a.count[b'howdy'] 0 >>> a.count[b'HoWdY'] 1 >>> a.shm.close[] >>> a.shm.unlink[] >>> del a # Use of a ShareableList after call to unlink[] is unsupported
The following example depicts how one, two, or many processes may access the same
ShareableList
by supplying the name of the shared memory block behind it:
>>> b = shared_memory.ShareableList[range[5]] # In a first process >>> c = shared_memory.ShareableList[name=b.shm.name] # In a second process >>> c ShareableList[[0, 1, 2, 3, 4], name='...'] >>> c[-1] = -999 >>> b[-1] -999 >>> b.shm.close[] >>> c.shm.close[] >>> c.shm.unlink[]
The following examples demonstrates that ShareableList
[and underlying SharedMemory
] objects can be pickled and unpickled if needed. Note, that it will still be the same shared object. This happens, because the
deserialized object has the same unique name and is just attached to an existing object with the same name [if the object is still alive]:
>>> import pickle >>> from multiprocessing import shared_memory >>> sl = shared_memory.ShareableList[range[10]] >>> list[sl] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> deserialized_sl = pickle.loads[pickle.dumps[sl]] >>> list[deserialized_sl] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> sl[0] = -1 >>> deserialized_sl[1] = -2 >>> list[sl] [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] >>> list[deserialized_sl] [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
>>> sl.shm.close[] >>> sl.shm.unlink[]