Back on the struggle but with pip and Ansible! (what is _ctypes?)

TL;DR: If you're getting the following error:

from _ctypes import Union, Structure, Array
    ModuleNotFoundError: No module named '_ctypes'

you're probably missing libffi-dev on Ubuntu (or some variation on other OS's)

---

And we're back! Last time I was trying to install python v3.10.2 with pyenv and was missing a million dependencies.

This time I'm trying to install ansible using pip (managed with pyenv)

Collecting ansible
  Downloading ansible-5.5.0.tar.gz (42.0 MB)
     |████████████████████████████████| 42.0 MB 782 kB/s
    ERROR: Command errored out with exit status 1:
     command: /home/banana/.pyenv/versions/3.10.2/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-e10880nb
         cwd: /tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>
        from setuptools.dist import Distribution
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/dist.py", line 38, in <module>
        from setuptools import windows_support
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/ctypes/__init__.py", line 8, in <module>
        from _ctypes import Union, Structure, Array
    ModuleNotFoundError: No module named '_ctypes'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/b5/6e/5806eafaa71ab5471cfc390089a89c7815ab841657d6517fd72df3d41862/ansible-5.5.0.tar.gz#sha256=b8a76d737889c9bfd0e6b0f2276dcf8d836da667067a355776f3504d7a66d518 (from https://pypi.org/simple/ansible/) (requires-python:>=3.8). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This continues for a bit as pip tries to install versions of ansible in decreasing order.

What if I try to install this missing _ctypes module?

$ pip install _ctypes
ERROR: Invalid requirement: '_ctypes'
$ pip install ctypes
ERROR: Could not find a version that satisfies the requirement ctypes (from versions: none)
ERROR: No matching distribution found for ctypes

huh, weird

What is this _ctypes module that I'm missing? According to the docs:

ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

What does that even mean? My understanding is that the ctypes library allows python code to use stuff in DLLs (dynamic linked libraries, this is OS dependent) directly using python without needing custom python code. This is really powerful and you can read more about it in the Python docs.

But why doesn't _ctypes or ctypes exist in pip as an installable module? Well, _ctypes is a built-in library so there isn't any module to pip install. What we're missing is the libffi-dev Ubuntu package (thanks stackoverflow!):

Foreign Function Interface library (development files)

Let's hit up apt install libffi-dev. Then do another pip install ansible (if you want to save time you'd just try from ctypes import * in a python shell and find that the module still can't be found):

$ pip install ansible
Collecting ansible
  Using cached ansible-5.5.0.tar.gz (42.0 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Interesting. What is also interesting was the error I got last time when I was trying to install python:

Failed to build these modules:
_ctypes

After installing the zlib package previously I was still able to successfully compile python without libffi and also did not get an error. I'm wondering now if my installation of python 3.10.2 is a bit wonky now.

I've decided to take a gander at what modules I have and while I don't see any _ctypes.py or ctypes.py I did find a directory called ctype:

~/.pyenv/versions/3.10.2/lib/python3.10/ctypes$ ls
__init__.py  __pycache__  _aix.py  _endian.py  macholib  test  util.py  wintypes.py

I'm not sure if any of the above is suspicious but I thought I'd take a look anyway.

I also have a hunch that with libffi missing during my initial python install (failed silently??), the appropriate symbolic links may not have been created properly or python wasn't compiled properly so I'll do a pyenv uninstall 3.10.2 and pyenv install 3.10.2 to recompile.

Now to check if ctypes was successfully built:

Python 3.10.2 (main, Mar 21 2022, 00:28:55) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> libc = CDLL("libc.so.6")
>>> print(libc.time(None))
1647848336

yay success!

NOW, back to installing ansible:

$ pip install ansible
...
... lots of output
...
done
Successfully installed MarkupSafe-2.1.1 PyYAML-6.0 ansible-5.5.0 ansible-core-2.12.3 cffi-1.15.0 cryptography-36.0.2 jinja2-3.0.3 packaging-21.3 pycparser-2.21 pyparsing-3.0.7 resolvelib-0.5.4

Excellent, and all the dependencies were installed too!