[TriLUG] Do a lot of you program in Python?

[Igor Partola]

Kind of. Yes, it's specific to CPython, but the guarantees provided by the
GIL must be implemented in all implementations of Python since this has to
do with how the interpreter functions. Specifically, the idea is that when
you do:

    x = 1

the operation is atomic. The implementation detail here is that CPython
will not lock just the `x` variable, but the entire interpreter using a
global lock. So if one thread is doing `x = 1` and another one is doing `y
= 2`, there will be no parallelism.

This is subtle since this does not apply to things that happen inside C
modules, or to IO, so you can often have some parallelism, though not as
much as if you wrote the whole program in C (or Java, C++, C#, etc.) with
fine-grained locking. BTW, the GIL is a performance optimization: for
single-threaded programs fine-grained locking is a performance hit, so
having one global lock is better. Since most Python code is
single-threaded, the developers of CPython have made this choice.

Now, the interaction between threads and the GIL is where things get really
whacky. The problem is that CPython does not come with its own scheduler.
Instead, Python threads here are just OS threads, and rely on the OS's
scheduler to switch between them. However, the OS had no idea where the
interpreter is with respect to executing the Python bytecode, so it ends up
doing some very naive things when switching between threads. The effect is
that some threads get much more of the CPU time simply because the
scheduler is favoring them arbitrarily.

Long story short, for some things you can have lots of parallelism with
CPython + the threading module, except when you are doing heavy
computational work. See
http://www.dabeaz.com/python/UnderstandingGIL.pdffor a great
explanation of all this.

Igor

P.S.: PyPy in the meantime is experimenting with a completely different
locking strategy which may or may not produce much better results in the
end. It's called Software Transactional Memory and is detailed here
http://morepypy.blogspot.com/2013/10/update-on-stm.html


On Thu, Jan 9, 2014 at 11:48 AM, Randy Barlow
<randy at electronsweatshop.com>wrote:

> On Wed, Jan 08, 2014 at 08:01:39PM -0500, Igor Partola wrote:
> > The often cited GIL is actually a very specific and often misunderstood
> feature which should not prevent anyone from adopting the language. Other
> interpreters often have it too (Ruby being a prime example).
>
> Perhaps this is what you were hinting at by saying "very specific", but
> it's worth highlighting that the GIL is not a language feature, but
> instead is a CPython implementation detail. CPython is probably the most
> used Python implementation, but there are others (PyPy, Jython, etc.),
> and nothing in the langage spec demands that these others have the GIL.
>