Who convert the byte code instructions into machine code in python?
I'm a little confused as to how the PVM gets the cpu to carry out the bytecode instructions. I had read somewhere on StackOverflow that it doesn't convert byte-code to machine code, (alas, I can't find the thread now). Show Does it already have tons of pre-compiled machine instructions hard-coded that it runs/chooses one of those depending on the byte code? Thank you. asked Jul 28, 2017 at 18:36
It's a lot higher-level than machine language. There's a giant
The answered Jul 28, 2017 at 18:54
user2357112user2357112 242k26 gold badges393 silver badges470 bronze badges 5 If you mean Standard Python (CPython) by Python, then no! The byte-code (.pyc or .pyo files) are just a binary version of your code line by line, and is interpreted at run-time. But if you use pypy, yes! It has a JIT Compiler and it runs your byte-codeas like Java dn .NET (CLR). answered Jul 28, 2017 at 18:42
KamyarKamyar 2,1501 gold badge21 silver badges32 bronze badges Learn about disassembling Python bytecodeThe source code of a programming language can be executed using an interpreter or a compiler. In a compiled language, a compiler will translate the source code directly into binary machine code. This machine code is specific to that target machine since each machine can have a different operating system and hardware. After compilation, the target machine will directly run the machine code. In an interpreted language, the source code is not directly run by the target machine. There is another program called the interpreter that reads and executes the source code directly. The interpreter, which is specific to the target machine, translates each statement of the source code into machine code and runs it. Python is usually called an interpreted language, however, it combines compiling and interpreting. When we execute a source code (a file with a After compilation, the bytecode is sent for execution to the PVM. The PVM is an interpreter that runs the bytecode and is part of the Python system. The bytecode is platform-independent, but PVM is specific to the target machine. The default implementation of the Python programming language is CPython which is written in the C programming language. CPython compiles the python source code into the bytecode, and this bytecode is then executed by the CPython virtual machine. Generating bytecode files In Python, the bytecode is stored in a import file_name However, it will not be created if we don’t import another file in the source code. In that case, we can still manually create it. To compile the individual files python -m compileall file_1.py ... file_n.py All the generated We can also use the
We only focus on the first three arguments
which are required (the others are optional).
For example, to compile some Python statements we can write: s=''' or equivalently write: compile("a=5 \na+=1 \nprint(a)", "", "exec") To evaluate an expression we can write: compile("a+7", "", "eval") This mode gives an error if you don’t have an expression: # This does not work: Here compile("a=a+1", "", "single") But what is returned by
So what the Code object The a = 5 In a similar way, the bytecode generated by the compile function
is stored in the c = compile("a=a+1", "", "single") The code object contains not only the bytecode but also some other information necessary for the CPython to run the bytecode (they will be discussed later). A code object can be executed or evaluated by passing it to the exec(compile("print(5)", "", "single")) # Output is: 5 When you define a function in Python, it creates a code object for it and you can access it using the def f(n): And the output will be:
Like any other objects the code object has some attributes, and to get the bytecode stored in a code object, you can use its c = compile("print(5)", "", "single") The output is: b'e\x00d\x00\x83\x01F\x00d\x01S\x00' The result is a bytes literal which is prefixed with print(c.co_code[0]) gives: 101 since the first element has the decimal value of 101 and can be shown with the character print(c.co_code[4]) gives: 131 since the 4th element has the decimal value of 131. The hexadecimal value of 131 is 83. So this byte can be shown with a character whose character code is These sequences of bytes can be interpreted by CPython, but they are not human-friendly. So we need to understand how these bytes are mapped to the actual instructions that will be executed by CPython. In the next section, we are going to disassemble the byte code into some human-friendly instruction to see how the bytecode is executed by CPython. Bytecode details Before going into further details, it is important to note that the implementation detail of Bytecode usually changes between versions of Python. So what you see in this article may not be valid for all versions of Python. In fact, it includes the changes that happened in version 3.6, and some of the details may not be valid for older versions. The code in this article has been tested with Python 3.7. The bytecode can be thought of as a series of instructions or a low-level program for the Python interpreter. After version 3.6, Python uses 2 bytes for each instruction. One byte is for the code of that instruction which is called an opcode, and one byte is reserved for itsargumentwhich is called the oparg. Each opcode has a human-friendly name which is called the opname. The bytecode instructions have a general format like this: opcode oparg We already have the opcodes in our bytecode, and we just need to map them to their corresponding opname. There is a module called Some instructions do not need an argument, so they ignore the byte after the opcode. The opcodes which have a value below a certain number ignore their argument. This value is stored in For example, suppose that we
have a short bytecode bytecode = b'd\x00Z\x00d\x01S\x00' The output will be: 100 0 90 0 100 1 83 0 The first two bytes of the bytecode is dis.opname[100] and the result is LOAD_CONST 0 The last two bytes in the bytecode are RETURN_VALUE In addition, some of the instructions can have an argument too big to fit into the default one byte. There is a special opcode CALL_FUNCTION 260 However, the maximum number that a byte can store is 255, and 260 does not fit into a byte. So this opcode is prefixed with EXTENDED_ARG 1 When the interpreter executes extened_arg = 1 << 8 # same as 1 * 256 So the binary value extened_arg = extened_arg | 4 This is like adding
the value of the oparg to extened_arg = 256 + 4 = 260 and this value will be used as the actual oparg of EXTENDED_ARG 1 is interpreted as: EXTENDED_ARG 1 For each opcode, at most three prefixal Now we can focus on the oparg itself. What does it mean? Actually the meaning of each oparg depends on its opcode. As mentioned before, the code object stores
some information other than the bytecode. This information can be accessed using the different attributes of the code object, and we need some of these attributes to decipher the meaning of each oparg. These attributes are: Code object attributes I am going to explain the meaning of these attributes using an example. Suppose that you have the code object of this source code: # Listing 1 Now we can check what is stored in each of these attributes: 1- (5, 'text', So the literals s = "3 * a"
If you try to get the def f(x): The result will be 2- ('a', 'b', 'f') 3- def f(x): Now ('x', 'z', 'g') Why 4- ('t', 'x') 5- ('t', 'x') Now that we are familiar with these attributes, we can go back to the opargs. The meaning of each oparg depends on its opcode. We have different categories of opcodes, and for each category, the oparg has a different meaning. In the 1- LOAD_CONST 1 then the oparg is the element of LOAD_CONST 'text' Similarly, there are some other lists in the 2- 3- 4- 5- 6- The code object has one more important attribute that should be discussed here. It is called 1 0 LOAD_CONST 0 Now we have a mapping from bytecode offsets to line numbers like this table: The bytecode offset always starts at 0. The code object has an attribute named These two increment columns are zipped together in a sequence like this: 4 1 8 1 Each number is stored in a byte and the whole sequence is stored as a bytes literal in the b'\x04\x01\x08\x01' which is the bytes literal for the previous sequence. So by having the attributes Then the offset increment versus the line number increment should be: 139 is equal to 127 + 12. So the previous row should be written as: and should be stored as Disassembling the bytecode Now that we are familiar with the bytecode structure, we can write a simple disassembler program. We first write a generator function to unpack each instruction and yield the offset, opcode, and oparg: This function reads the next pair of bytes from the bytecode. The first byte is the opcode. By comparing this opcode with In the next iteration, this temporary variable will be added to the next oparg and adds one byte to it. This process continues if the next opcode is The It first divided the The The Now we can use all these functions to disassemble the bytecode. The It will first unpack the offset, opcode and oparg for each pair of bytes in the bytecode of the code object. Then it finds the corresponding source code line numbers, and checks if the offset is a jump target. Finally, it finds the opname and the meaning of the oparg and prints all the information. As mentioned before each function definition is stored in a separate code object. So at the end the function calls itself recursively to disassemble all the function definitions in the bytecode. Here is an example of using this function. Initially, we have this source code: a=0 We first store it in a string and compile it to get the object code. Then we use the s='''a=0 The output is: 1 0 LOAD_CONST 0 (0) So 4 lines of source code are converted into 38 bytes of bytecode or 19 lines of bytecode. In the next section, I will explain the meaning of these instructions and how they will be interpreted by CPython. The module Disassembling a pyc file As mentioned before, when the source code is compiled, the bytecode is stored in a Bytecode operationsSo far we learned how to disassemble the bytecode instructions. We can now focus on the meaning of these instructions and how they are executed by CPython. CPython which is the default implementation of Python uses a stack-based virtual machine. So first we should get familiar with the stack. Stack and heap Stack is a data structure with a LIFO (Last In First Out) order. It has two principal operations:
So the last element added or pushed to the stack is the first element to be removed or popped. The advantage of using stack to store data is that memory is managed for you. Reading from and writing to stack is very fast, however, the size of stack is limited. Data in Python is represented as objects stored on a private heap. Accessing the data on heap is a bit slower compared to stack, however, the size of heap is only limited by the size of virtual memory. The elements of heap have no dependencies with each other and can be accessed randomly at any time. Everything in Python is an object and objects are always stored on the heap. It’s only the reference (or the pointer) to the object that is stored in the stack. CPython uses the call stack for running a Python program. When a function is called in Python, a new frameis pushed onto the call stack, and every time a function call returns, its frame is popped off. The module in which the program runs has the bottom-most frame which is called the global frame or the module frame. Each frame has an evaluation stack where the execution of a Python function occurs. The function arguments and its local variables are pushed into this evaluation stack. CPython uses the evaluation stack to store the parameters required for any operations and also the result of those operations. Before starting that operation, all the required parameters are pushed onto the evaluation stack. Then the operation is started and it pops its parameters. When the operation is finished, it pushes the result back onto the evaluation stack. All the objects are stored on the heap and the evaluation stack in the frames deals with references to them. So the references to these objects can be pushed onto the evaluation stack temporarily to be used for the later operations. Most of Python’s bytecode instructions manipulate the evaluation stack in the current frame. In this article whenever we talk about the stack it means the evaluation stack in the current frame or the evaluation stack in the global frame if we are not in the scope of any functions. Let me start with a simple example, and disassemble the bytecode of the following source code: a=1 To do that we can write: s='''a=1 and we get: 1 0 LOAD_CONST 0 (1) In addition, we can check some other attributes of the code object: c.co_consts Here the code is running in the module, so we are inside the global frame. The first instruction is LOAD_CONST consti pushes the value of It is important to note that stack works with references to the objects. So whenever we say that an instruction pushes an object or the value of an object onto the stack, it means that a reference (or pointer) to that object is being pushed. The same thing happens when an object or its value is popped off the stack. Again its reference is popped. The interpreter knows how to retrieve or store the object's data using these references. The instruction STORE_NAME namei
pops the top of the stack and stores it into an object whose reference is stored in BINARY_ADD pops the top two elements of the stack ( RETURN_VALUE returns with the top of the stack to the caller of the function. Of course, here we are in the module scope and there is no caller function, so Functions, global and local variables Now let’s see what happens if we also have a function. We are going to disassemble the bytecode of a source code which has a function: #Listing 2 The output is: 1 0 LOAD_CONST 0 (1) In addition, we can check some other attributes of the code object: c.co_consts In the first line (offsets 0 and 2)
the constant In the second line, the constant STORE_GLOBAL namei pops the top of the stack and stores it into an object whose reference is stored in In the third line, the function MAKE_FUNCTION argc is used to create the function. It needs some parameters that should be pushed onto the stack. The name of the function should be on top of the stack and the function’s code object should be below it. In this example, its oparg is zero, but it can have other values. For example, if the function definition had a keyword argument like: def f(x=5): Then the disassembled bytecode for line 2 would be: 2 4 LOAD_CONST 5 ((5,)) An oparg of Now let’s looks inside the code object of LOAD_GLOBAL namei pushes a reference to the object referred by LOAD_FAST var_num pushes a reference to the object whose reference is ('x', 'y') So STORE_FAST var_num pops the
top of the stack and stores it into an object whose reference is stored in But how this function is called? If you look at the bytecode of line 8, first, CALL_FUNCTION argc calls a callable object with positional arguments. Its oparg, argc indicates the number of positional arguments. The top of the stack contains positional arguments, with the right-most argument on top. Below the arguments is the function callable object to call.
In our example, we only have one positional argument, so the instruction will be POP_TOP pops the item on top of the stack. That is because we do not need the returned value of the function anymore. Figure 2 shows all the bytecode operations with offsets 16 to 22. The bytecode instructions inside Built-in functions In line 9 of the disassembled bytecode of Listing 2, we want to Python uses its built-in functions to create data structures. For example, the following line: a = [1,2,3] will be converted to: 1 0 LOAD_CONST 0 (1) Initially, each element of the list is pushed onto the stack. Then the instruction BUILD_LIST count is called to
create the list using the count items from the stack and pushes the resulting list object onto the stack. Finally, the object on the stack will be popped and stored on the heap and EXTENDED_ARG As mentioned before, some of the instructions can have an argument too big to fit into the default one byte, and they will be prefixed by the instruction s= 'print(' + '"*",' * 260 + ')' Here 1 0 LOAD_NAME 0 (print) Here 522 EXTENDED_ARG 1 As mentioned before the oparg of EXTENDED_ARG will be left-shifted by eight bits or simply multiplied by 256 and will be added to the oparg of the next opcode. So the oparg of Conditional statements and jumps Consider the following source code which has an s='''a = 1 The disassembled bytecode is: 1 0 LOAD_CONST 0 (1) We have a few new instructions here. In line 2, the object that COMPARE_OP oparg performs a Boolean
operation. The operation name can be found in The instruction POP_JUMP_IF_FALSE target performs a conditional jump. First, it pops the top of the stack. If the element on top of the stack is false, it sets the bytecode counter to target. The bytecode counter shows the current bytecode offset which is being executed. So it jumps to the bytecode offset which is equal to target and the execution of bytecode continues from there. The offset 18 in the bytecode
is a jump target, so there is a JUMP_FORWARD delta increments the bytecode counter by delta. In the previous bytecode, the offset of this instruction is 16, and we know that each instruction takes 2 bytes. So when this instruction is finished, the bytecode counter is Now we can see how the Now let’s see a more complicated Boolean expression. Consider the following source code: s='''a = 1 Here we have a logical 1 0 LOAD_CONST 0 (1) In Python Loops and block stack As mentioned before, there is an evaluation stack inside each frame. In addition, in each frame, there is a block stack. It is used by CPython to keep track of certain types of control structures like the loops, Let’s see how loops are implemented in the bytecode. Consider the following code and its disassembled bytecode: s='''for i in range(3): The instruction SETUP_LOOP delta is executed before the loop starts. This instruction pushes a new item (which is also called a
block) onto the block stack. delta is added to the bytecode counter to determine the offset of the next instruction after the loop. Here the offset of After that, the function GET_ITER It takes the iterable on top of the stack and pushes an iterator of that. The instruction: FOR_ITER delta assumes that there is an iterator on top of the stack. It calls its JUMP_ABSOLUTE target sets the bytecode counter to target and jumps to the target offset. So it jumps to offset 10 and runs POP_BLOCK removes the current block from the top of the block stack. The offset of the next instruction after the loop is stored in the block (here it is 26). So the interpreter will jump to that offset and continue execution from there. Figure 3 shows the bytecode operations with offsets 0, 10, 24 and 26 as an example (In fact in Figures 1 and 2 we only showed the evaluation stack in each frame). Figure 3But what happens if we add a s='''for i in range(3): We have only added a BREAK_LOOP This opcode removes those extra items on the evaluation stack and pops the block from the top of the block stack. You should notice that the other instructions of the loop are still using the evaluation stack. So when the loop breaks, the items that belong to it should be popped off the evaluation stack. In this example, the iterator object is still on top of the stack. Remember that the block in the block stack stores the number of items that existed in the evaluation stack before starting the loop. So by knowing that number, Creating the code object The code object is an object of type types.CodeType(co_argcount, co_kwonlyargcount, The arguments form all the attributes of the code object. You are already familiar with some of these arguments (like def f(a, b, *args, c, **kwargs):
2 0 LOAD_CONST 1 (1) In line 2, one element is pushed onto the stack using the
Bytecode injection Now that we are completely familiar with the code object, we can start changing its bytecode. It is important to note that the code object is immutable. So once created we cannot change it. Suppose that we want to change the bytecode of the following function: def f(x, y): Here we cannot change the bytecode of the code object of the function directly. Instead, we need to create a new code object and then assign it to this function. To do that we need a few more functions. The We can try it on the previous function: disassembled_bytecode = disassemble_to_list(c) Now [['LOAD_FAST', 'x'], We can now change the instructions of this list easily. But we also need to assemble it back to the bytecode: The function The function disassembled_bytecode[2] = ['BINARY_MULTIPLY'] The instruction BINARY_MULTIPLY pops the top two elements of the stack, multiplies them together and pushes the result onto the stack. Now we assemble the modified disassembled bytecode: new_co_code= assemble(disassembled_bytecode, c.co_consts, After that we create a new code object: import types We use all the attributes of f(2,5) # Output is 10 not 7 Caution: The def func(x): Now if check its code object: c = func.__code__ In fact, this function has one nonlocal variable nc = types.CodeType(c.co_argcount, c.co_kwonlyargcount, But if we check the same attribute of the new code object nc.co_cellvars Output is: () It turns out to be empty. So Code optimization Understanding the bytecode instructions can help us with the optimization of the source code. Consider the following source code: setup1='''import math Here we define a function t1 = timeit.timeit(stmt="f()", setup=setup1, number=100000) You may get different numbers for In 7 24 LOAD_FAST 0 (total) But in 7 30 LOAD_FAST 2 (total) As you see in Local variables are stored in an array on each frame (which is not shown in the previous figures to make them simpler). We know that the name of local variables are stored in The global and builtins of the module are stored in a dictionary. We know that their names are stored in the Example: Defining constants in Python This example illustrates how to use the bytecode injection to change the behavior of functions. We are going to write a decorator which adds a const statement to Python. In some programming languages like C, C++, and JavaScript there is a const keyword. If a variable is declared as const using this keyword, then changing its value is illegal, and we cannot change the value of this variable in the source code anymore. Python does not have a const statement, and I do not claim that it is really necessary to have such a keyword in Python. In addition, defining constants can be also done without using
the bytecode injection. So this is just an example to show you how to put the bytecode injection into action. First, let me show how you can use it. The const keyword is provided using a function decorator named @const The variable @const When a variable is declared as const., it should be assigned to its initial value, and it will be a local variable of that function. Now let me show you how it has been implemented. Suppose that I define a function like this (without decoration): def f(x): It will be compiled properly. But if you try executing this function, you get an error: f(2) Now let's take a look at the disassembled bytecode of this function: 2 0 LOAD_CONST 1 (5) When Python tries to compile the function, it takes This function receives the list of bytecode instructions generated by NOP is a ‘Do nothing’ code. When the interpreter reaches to 2 0 LOAD_CONST 1 (5) Now line 2 is the equivalent of The second loop searches the list of bytecode instructions again to find any reassignment of the constant variables. Any instruction like As mentioned before, the bytecode should be changed before executing the code. So the function When we create the new code object, some of its attributes need to be modified. In the original function Understanding Python’s bytecode allows you to get familiar with the low-level implementation of the Python compiler and virtual machine. If you know how your source code is converted to the bytecode, you can make better decisions about writing and optimizing your code.
Bytecode injection is also a useful tool for code optimization and metaprogramming. I have only covered a small number of bytecode instructions in this article. You can refer to Who converts byte code to machine?The JVM, which is part of the Java Runtime Environment, interprets the bytecode and converts it to machine language specific to the intended platform. The JVM interpreter usually processes the bytecode instructions one instruction at a time, but a JVM can also support a just-in-time compiler.
How is Python code converted to byte code?CPython compiles the python source code into the bytecode, and this bytecode is then executed by the CPython virtual machine. All the generated pyc files will be stored in the __pycache__ folder. If you provide no file names after compileall, it will compile all the python source code files in the current folder.
Does bytecode get converted to machine code?JVM Converts Bytecode to Machine Code
JVM ( Java Virtual Machine ) receives this bytecode which is generated by Java Compiler. In JVM, there are two main components that perform all the jobs to convert the bytecode to native code, Classloader, and Execution Engine.
Who executes the bytecode and where in Python?Today we've learned that the CPython VM executes bytecode instructions one by one in an infinite loop. The loop contains a switch statement over all possible opcodes. Each opcode is executed in the corresponding case block.
|