DEV Community: milktea02

Misunderstanding Go Timer Resets

milktea02 — Mon, 10 Jun 2024 07:40:51 +0000

My teammate shared a snippet of code with me the other day which led to a fun rabbit hole.

TL;DR:

Remember to drain your timer channel before resetting it if the reset is not a result of the timer firing a notification (e.g. you read from the channel)

What is a Go Timer?

As the name implies, a timer waits a duration of time before firing an event (sending a signal to a channel).

Timers can be stopped, and reset. Just like in real life!

Here's an example:



t := time.NewTimer(10 * time.Second)

fmt.Println("Wait for 10 seconds for the timer to send a signal")
<- t.C
fmt.Println("It's been 10 seconds!)

Of course, you could always just use time.Sleep(10 * time.Second) in the above example. Unlike sleep, you can stop the Timer before the duration has been met.

Example:



t := time.NewTimer(10 * time.Second)

go func() {
   <- t.C
   fmt.Println("It's been 10 seconds!")
}()

fmt.Println("Never mind, don't wait 10 seconds for the timer")
t.Stop()

time.Sleep(15 * time.Second)
fmt.PrintLn("Slept for 15 seconds!")

And just like a kitchen timer, you can reset it:



t := time.NewTimer(10 * time.Second)

for {
    select {
    case <- t.C:
        // If you're feeling spicy you could reset with a random duration
        t.Reset(5 * time.Second)
    default:
        time.Sleep(time.Second)
    }
}

Everything makes sense until it doesn't

So, I thought I had Timers understood until my teammate sent me this snippet of code:



package main

import (
    "fmt"
    "time"
)

func main() {
    signal := make(chan int)
    go func() {
        for {
            signal <- 1 
        }
    }()

    t := time.NewTimer(time.Second)
    for {
        select {
        case <-signal:
            time.Sleep(2 * time.Second)
            t.Reset(time.Second)
            fmt.Print("1")
        case <-t.C:
            fmt.Print("!")
        default:
            fmt.Print(".")
        }
    }
}

Maybe you're an expert in timers, channels, and select blocks and can see the bug here.

At first glance, I would never expect to see ! printed. But here is example output:



$ go run main.go
.111!1!1!1111

Shouldn't the timer reset prevent the ! branch from ever getting executed? Have I been misunderstanding and using reset incorrectly?

Looking at the documentation for Reset, this is near the top:

For a Timer created with NewTimer, Reset should be invoked only on stopped or expired timers with drained channels.

D'oh! Even if the timer has been reset, it doesn't change the fact that it has already fired and so the channel contains an event! And select will pick a branch "randomly" if more than one branch is ready.

And just a few lines down, we are given instructions on how to use reset:

If a program has already received a value from t.C, the timer is known to have expired and the channel drained, so t.Reset can be used directly.
If a program has not yet received a value from t.C, however, the timer must be stopped and—if Stop reports that the timer expired before being stopped—the channel explicitly drained:



if !t.Stop() {
    <-t.C
}
t.Reset(d)

Putting it all together, correctly

Since we did not receive anything from the timer channel, we need to check if the timer has already went off, and if so we should drain the channel before resetting it. Changing the snippet as so:



for {
    select {
    case <-signal:
        time.Sleep(2 * time.Second)
        if !t.stop() {
            // drain the channel
            <-t.C
        }
        t.Reset(1 * time.Second)
        fmt.Printf("1")
    case <-t.C:
        fmt.Printf("!")
    default:
        fmt.Print(".")
    }
}

Now, all shall be well!

Example output:



$ go run main.go

.111111111111111111111

Resources:

As part of this little journey, we also discovered running this snippet on a MacBook Pro 2019 (intel), vs Ubuntu on WSL (intel) yielded consistently different outputs which I may or may not write about once I have all the facts straight.

Some things I thought about:

MacOS vs Ubuntu? MacOS vs Windows? Is this even relevant?
✨schedulers✨
Goroutines are not (os) threads. Repeat after me "goroutines are not (os) threads"

Consuming and deserializing avro messages with Apache kafka-console-consumer and AWS Glue as the Schema Registry

milktea02 — Sun, 29 Jan 2023 22:05:24 +0000

I had to do this recently and found little to no documentation on what was required to consume avro logs from kafka and deserialize them with the Glue schema registry so I'll document my findings here. For reference, Confluent has one that just works out of the box.

Requirements/Pre-requisites/Assumptions/?:

Apache Kafka
- I think I was using 3.3.1 with Scala 2.13
Kafka cluster with topics that have avro messages
- We are using an AWS MSK cluster with kafka v2.6.x
- This also assumes the messages are being serialized using the following headers
AWS Glue Registry with avro schema you're using
- You can find documentation on how to set this up with some schemas
jars from https://repo1.maven.org/maven2/org/apache/
this also assumes you have your iam role permissions/policies configured

Caveat: This is what worked with our setup and the jars we needed. I don't have any examples or screenshots.

Download required jars into <location of kafka>/libs/:

List of jars:

Group Id	Artifact Id	Version
`software.amazon.glue`	`schema-registry-common`	`1.1.14`
`software.amazon.glue`	`schema-registry-serde`	`1.1.14`
`software.amazon.awssdk`	`glue`	`2.17.12`
`software.amazon.awssdk`	`arns`	`2.17.12`
`software.amazon.awssdk`	`url-connection-client`	`2.17.12`
`org.apache.avro`	`avro`	`1.11.0`
`com.google.guava`	`guava`	`30.0-jre`
`com.google.guava`	`failureaccess`	`1.0.1`

Configuring kafka-console-consumer.sh:

You'll need to supply some properties to the glue deserializer; I've used a wrapper script kafka-glue-avro-console-consumer.sh:

#!/bin/sh

kafka-console-consumer.sh --value-deserializer com.amazonaws.services.schemaregistry.deserializers.GlueSchemaRegistryKafkaDeserializer \
        --property value.deserializer.region=<hardcoded-aws-region> \
        --property value.deserializer.dataFormat=AVRO \
        --property value.deserializer.avroRecordType=GENERIC_RECORD "$@"

You'll need to supply other properties such as --bootstrap-server <list of servers:port>, --topic <topic-name>, --consumer.config <config-file>.

Example:

$ ./kafka-glue-avro-console-consumer.sh --bootstrap-server broker-01:9098 --topic my-topic --consumer.config my-config --max-message 1 --from-beginning

If you want to specify the glue registry and schema you can do that with:

--property value.deserializer.registry.name=<registryName>
--property value.deserializer.schemaName=<schemaName>

You can check the source code for additional properties you can use: https://github.com/awslabs/aws-glue-schema-registry/blob/v1.1.14/common/src/main/java/com/amazonaws/services/schemaregistry/common/configs/GlueSchemaRegistryConfiguration.java

Here is a snippet of ansible code to grab all the jars:

- name: install artifacts from maven to consume AVRO records
  maven_artifact:
    group_id: "{{ item.group_id }}"
    artifact_id: "{{ item.artifact_id }}"
    version: "{{ item.version }}"
    dest: /home/ubuntu/kafka_{{ kafka_client_version }}/libs/{{ item.artifact_id }}-{{ item.version }}.jar
    group: ubuntu
    owner: ubuntu
    mode: 0664
  loop:
    - { group_id: 'software.amazon.glue', artifact_id: 'schema-registry-common', version: 1.1.14 }
    - { group_id: 'software.amazon.glue', artifact_id: 'schema-registry-serde', version: 1.1.14 }
    - { group_id: 'software.amazon.awssdk', artifact_id: 'glue', version: 2.17.122 }
    - { group_id: 'software.amazon.awssdk', artifact_id: 'arns', version: 2.17.122 }
    - { group_id: 'software.amazon.awssdk', artifact_id: 'url-connection-client', version: 2.17.122 }
    - { group_id: 'org.apache.avro', artifact_id: 'avro', version: 1.11.0 }
    - { group_id: 'com.google.guava', artifact_id: 'guava', version: 30.0-jre }
    - { group_id: 'com.google.guava', artifact_id: 'failureaccess', version: 1.0.1 }

(we are testing some things which is why everything is in the user's home directory!)

Upgrading to WSL2 (so I can install Rust)

milktea02 — Mon, 09 May 2022 03:58:28 +0000

TL;DR: Follow this guide someone has kindly posted.

Trying to install Rust and failing on WSL1 on Windows 11 led me to needing to upgrade from WSL1 to WSL2.

As you can see in my powershell I'm running version 1 of the wsl:

PS > wsl -l -v
  NAME      STATE           VERSION
* Ubuntu    Running         1

First, step is downloading and installing the new kernel package.

Secondly, enabling Virtual Machine feature:

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Remember to run powershell as an admin!

Error: 740

Elevated permissions are required to run DISM.
Use an elevated command prompt to complete these tasks.

Success:

Deployment Image Servicing and Management tool
Version: 10.0.22000.1

Image Version: 10.0.22000.556

Enabling feature(s)
[==========================100.0%==========================]
The operation completed successfully.

You maybe also need to enable in BIOS as well depending on your PC manufacturer. Windows 11 on Lenovo apparently has this enabled by default.

Now time to install the new kernel we downloaded in step 1:

It turns out I also needed to restart my computer for the virtualization enablement to take affect:

PS C:\WINDOWS\system32> wsl --set-default-version 2
Please enable the Virtual Machine Platform Windows feature and ensure virtualization is enabled in the BIOS.
For information please visit https://aka.ms/wsl2-install

Now to set to version 2:

PS C:\WINDOWS\system32> wsl --set-default-version 2
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
The operation completed successfully.

Oddly enough this didn't work - possibly because I needed to specify which linux distribution I would like to use wsl2?

PS C:\WINDOWS\system32> wsl -l -v
  NAME      STATE           VERSION
* Ubuntu    Stopped         1
PS C:\WINDOWS\system32> wsl --set-version ubuntu 2
Conversion in progress, this may take a few minutes...
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
Conversion complete.
PS C:\WINDOWS\system32> wsl -l -v
  NAME      STATE           VERSION
* Ubuntu    Stopped         2

Can I finally install Rust on Ubuntu on WSL 2?

Something something about running scripts through a cURL request but it's from rust-lang.org and you can always take a look at the script from https://sh.rustup.rs:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Rust is installed now. Great!

$ rustup -V
rustup 1.24.3 (ce5817a94 2021-05-31)
$ rustc --version
rustc 1.60.0 (7737e0b5c 2022-04-04)
$ cargo --version
cargo 1.60.0 (d1fd9fe 2022-03-01)

ayy!

Google Sheets to track Dividends (How I pack and search the data after webscraping it)

milktea02 — Fri, 08 Apr 2022 08:20:44 +0000

In my last post I wrote about using Google Sheets to webscrape dividend data from dividendhistory.org.

Today I'm going to share with you my adventures to improve efficiency and how to store data using sheets TRANSPOSE() and FLATTEN() functions and search using MATCH().

TOC

The improved scrape
Packing the data
Searching the data
- When am I going to get paid next?
- What about the other data?

The Improved Scrape

Before, to scrape the dividend history table I was using the IMPORTHTML() function but also noted that some times I may make up to two calls because the web page may only have one table instead of two causing essentially an index out of range error.

After reviewing the page source I found out the table has an id:



<table id='dividend_table' class='table table-striped table-bordered'>
  <thead>
    <tr>
      <th>Ex-Dividend Date</th>
      <th>Payout Date</th>
      <th>Cash Amount</th>
      <th>% Change</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><i>2022-10-29</i></td>
      <td><i>2022-11-26</i></td>
      <td><i>$1.33**</i></td>
      <td><i>unconfirmed/estimated</i></td>
    </tr>
    <tr>
      <td><i>2022-07-30</i></td>
      <td><i>2022-08-26</i></td>
      <td><i>$1.33**</i></td>
      <td><i>unconfirmed/estimated</i></td>
    </tr>   
    <tr>
      <td>2022-04-29</td>
      <td>2022-05-26</td>
      <td>$1.33</td>
      <td></td>
    </tr>
...
...
</table>

d'oh!

I can use IMPORTXML() and just query for this table directly!



=IMPORTXML("https://dividendhistory.org/payout/TSX/BMO", "//table[@id='dividend_table']/*/*")

And using that query looks like this in Sheets:

You can continue to use the INDEX(range, row, col) function to grab the exact cells you want but for myself I wanted to do a few fun things.

Packing the data

Now, I only really need some rows of data and I don't really care for the % Change column, so we use ARRAY_CONSTRAIN(range, rows, cols)



=ARRAY_CONSTRAIN(IMPORTXML("https://dividendhistory.org/payout/TSX/BMO", "//table[@id='dividend_table']/*/*"), 6, 3)

I want to further pack the data so that it occupies it's own row or column which can be done using TRANSPOSE() and FLATTEN() and you'll see that order will matter:

We want to apply TRANSPOSE() first then FLATTEN():



=FLATTEN(TRANSPOSE(ARRAY_CONSTRAIN(IMPORTXML("https://dividendhistory.org/payout/TSX/BMO", "//table[@id='dividend_table']/*/*"), 6, 3)))

Which makes it look like the columns are combined to a stack (hehe stack):

Searching the Data

If you ever had to take a course on algorithms and data structures you might remember doing a lot of stuff related to pointers and offsets and this is why I stored the data to fit one column!

When am I going to get paid next?

I can make use of the MATCH(key, range, [search_type]) formula to search for a relative offset.

The search_type is useful because I want to find the next payout date. To find the date that is equal to or greater than today I can set the search_type to -1 and luckily for us the dates are already sorted in descending order:

Sample of what we're going to do with the data:

As of writing, TODAY() is 2022-04-08 and we can see that the relative offset of the next payout date (2022-05-26) when searching from B14 to B18 is 3 from the top, so located in B16



=MATCH(TODAY(), B14:B18, -1)

We can also get the exact row where Payout Date starts at using ROW() (useful if you drag things here and there instead of hardcoding):



=ROW(B13)

So now we know that the next payout date is located in cell B(13 + 3 but how can we get the information from that cell address?

We can use the INDIRECT() function to reference a cell using an address that's in TEXT.

Basically we want a generic way to write:



=INDIRECT("B16")

The cool thing is we can use COLUMN and INDIRECT() also accepts address in RyCx format if we set the optional is_A1_notation flag to FALSE. Again I like using & to build strings in sheets:



=INDIRECT("R" & SUM(B1 + B3) & "C" & COLUMN(), FALSE)

And we can do the same for getting the next Cash Amount by using a different starting offset but same relative offset:



=INDIRECT("R" & SUM(B2 + B3) & "C" & COLUMN(), FALSE)

What about the other data?

In my last post I also scraped the yield. I also did this and with a bit of data massaging also tacked it onto the column as you can see here in my raw data spreadsheet example:

So ultimately I end up making only two calls to dividend history per ticker symbol which is much better than up-to two calls for the dividend amount, and one call for the yield. We also get all the useful data from the page.

Here is an example of what you can do with the raw data and only taking the relevant data:

There's all sorts of stuff you can do for example, taking the frequency and calculating how much dividend you could expect from a single share over a year. You can use IF() to decide if the frequency would be 12 (monthly) or 4 (quarterly).

And here is a link to the example if you would like to make a copy and check out the stuff for yourself. It's been pre-populated with Canadian banks and some American companies that pay dividends.

https://docs.google.com/spreadsheets/d/1CNfWRi7pY5zWcQ1htC5m557oBYS8-YKnMtTuStbhTaQ/edit?usp=drivesdk

Webscraping in Google Sheets (to do personal finance)

milktea02 — Fri, 01 Apr 2022 05:56:00 +0000

TL;DR: use IMPORTHTML() and IMPORTXML() to webscrape :)

TOC:

Bob's Your Uncle
My Solution
- Scraping for yield
- Scraping for dividend ($)
References

Did you know you can webscrape using Google Sheets? Me neither, but it's the year 2022 and we have 9 more months in this year to learn so many new random things.

Bob's your uncle

I enjoy reading about personal finance and one such person I enjoy reading posts from is Bob at Tawcan.com. He has a post detailing how he uses IMPORTHTML() and IMPORTXML() to scrape dividend data from Google Finance and Yahoo Finance. This is really useful since the built in GOOGLEFINANCE() function doesn't have this information.

I ended up modifying what he had and also had to learn about XPath and XQuery.

Here is one example of a cell function he uses to get the dividend and yield information from Yahoo Finance:

=SPLIT(INDEX(IMPORTHTML(concatenate("https://finance.yahoo.com/quote/",B2),"table",2),6,2)," ")

Using IMPORTHTML() you only get a choice of scraping for "table" or "list".

In the above example, B2 contains a stock ticker symbol like RY which is concatenated with the url to the page we want scraped. We end up getting the 2nd table on the page with IMPORTHTML(). Sheets ends up importing the table which isn't very useful for us so we further parse the table using INDEX() to get the exact cell we want. In this case the dividend yield information is the 6th row, and 2nd column with the index starting from 1. That returns a dividend and yield so we further SPLIT() that information.

If you check out Bob's post, you'll see that he constantly needs to update his formulas because Google Finance and Yahoo Finance keep updating layouts. He also uses IMPORTXML() at some point because the pages are dynamic and don't usually have that information statically available.

My Solution

I decided to find other sources for dividend and yield and came across dividend history. What's nice is that this source also has dividend and yield information for ETFs which isn't the case for Google and Yahoo Finance. I don't know how often the layout is updated but given how simple it is and how it serves a single purpose, I assume it won't be updated too often.

Scraping for Yield

=INDEX(SPLIT(IMPORTXML("https://dividendhistory.org/payout/"&IF($B2="NASDAQ", "", "TSX/")&$A2, "//p[contains(text(), 'Yield')]", "en_US"), ":%"), 1, 2)

In my use case, I like to keep ticker symbols in column A, and their exchanges in column B. This is important because Canadian stocks (TSX) are found in payout/TSX/<tickersymbol> whereas American stocks are in payout/<tickersymbol>. I also don't use CONCATENATE() and prefer to just use "<string>"&A1&"<string>".

The second argument in IMPORTXML() is the xquery_path.

//p[contains(text(), 'Yield')]

Thankfully HTML is basically just a very specific XML amirite? Anyway the query looks for any <p> that contains the text value 'Yield' between the tags.

Yield: 1.23%

And since I only want the numbers I SPLIT() at the : and % and take the second element (remember in Sheets, index starts at 1).

Scraping for Dividend ($)

=INDEX(IMPORTHTML("https://dividendhistory.org/payout/"&IF($B2="NASDAQ", "", "TSX/")&$A2, "table", 2, "us_EN"), 4, 3)

The dividend is easier since I can use IMPORTHTML() and select the exact cell from the 2nd table on the page.

This isn't perfect because some tickers don't have any news or announcements, as is the case for VGRO:

In this case, the dividend history table is actually the 1st table on the page. We end up with a reference error because I'm looking for a 2nd table that doesn't exist (out of bounds!):

#REF

My first solution to this problem was thinking maybe all ETFs or even REITs don't usually have announcements so I can track a column for the TYPE of investment and pick 1 or 2 accordingly. This proved to be an incorrect assumption.

My second solution was to make the first IMPORTHTML() request and if ISERR(<request>) then make a second request searching for the 1st table.

=IF(ISERR(G2),INDEX(IMPORTHTML("https://dividendhistory.org/payout/"&IF($B2="NASDAQ", "", "TSX/")&$A2, "table", 1, "us_EN"), 4, 3), G2)

This is really inefficient because it means making up to 2 requests for a single value. I stored the first request in column G and call it the "test" column:

Anyway, that was a fun little rabbit hole and I think I'll find many uses for webscraping on sheets that may or may not involve personal finance!

References

https://www.tawcan.com/using-google-spreadsheet-dividend-investment/#Even_Newer_Formula1
https://support.google.com/docs/answer/3093281?hl=en
https://www.w3schools.com/xml/xpath_syntax.asp

Back on the struggle but with pip and Ansible! (what is _ctypes?)

milktea02 — Mon, 21 Mar 2022 08:20:17 +0000

TL;DR: If you're getting the following error:

from _ctypes import Union, Structure, Array
    ModuleNotFoundError: No module named '_ctypes'

you're probably missing libffi-dev on Ubuntu (or some variation on other OS's)

And we're back! Last time I was trying to install python v3.10.2 with pyenv and was missing a million dependencies.

This time I'm trying to install ansible using pip (managed with pyenv)

Collecting ansible
  Downloading ansible-5.5.0.tar.gz (42.0 MB)
     |████████████████████████████████| 42.0 MB 782 kB/s
    ERROR: Command errored out with exit status 1:
     command: /home/banana/.pyenv/versions/3.10.2/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-e10880nb
         cwd: /tmp/pip-install-cfmis9e2/ansible_27dc97f765544a6381b36454d1ed76c5/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>
        from setuptools.dist import Distribution
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/dist.py", line 38, in <module>
        from setuptools import windows_support
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/home/banana/.pyenv/versions/3.10.2/lib/python3.10/ctypes/__init__.py", line 8, in <module>
        from _ctypes import Union, Structure, Array
    ModuleNotFoundError: No module named '_ctypes'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/b5/6e/5806eafaa71ab5471cfc390089a89c7815ab841657d6517fd72df3d41862/ansible-5.5.0.tar.gz#sha256=b8a76d737889c9bfd0e6b0f2276dcf8d836da667067a355776f3504d7a66d518 (from https://pypi.org/simple/ansible/) (requires-python:>=3.8). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This continues for a bit as pip tries to install versions of ansible in decreasing order.

What if I try to install this missing _ctypes module?

$ pip install _ctypes
ERROR: Invalid requirement: '_ctypes'
$ pip install ctypes
ERROR: Could not find a version that satisfies the requirement ctypes (from versions: none)
ERROR: No matching distribution found for ctypes

huh, weird

What is this _ctypes module that I'm missing? According to the docs:

ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

What does that even mean? My understanding is that the ctypes library allows python code to use stuff in DLLs (dynamic linked libraries, this is OS dependent) directly using python without needing custom python code. This is really powerful and you can read more about it in the Python docs.

But why doesn't _ctypes or ctypes exist in pip as an installable module? Well, _ctypes is a built-in library so there isn't any module to pip install. What we're missing is the libffi-dev Ubuntu package (thanks stackoverflow!):

Foreign Function Interface library (development files)

Let's hit up apt install libffi-dev. Then do another pip install ansible (if you want to save time you'd just try from ctypes import * in a python shell and find that the module still can't be found):

$ pip install ansible
Collecting ansible
  Using cached ansible-5.5.0.tar.gz (42.0 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Interesting. What is also interesting was the error I got last time when I was trying to install python:

Failed to build these modules:
_ctypes

After installing the zlib package previously I was still able to successfully compile python without libffi and also did not get an error. I'm wondering now if my installation of python 3.10.2 is a bit wonky now.

I've decided to take a gander at what modules I have and while I don't see any _ctypes.py or ctypes.py I did find a directory called ctype:

~/.pyenv/versions/3.10.2/lib/python3.10/ctypes$ ls
__init__.py  __pycache__  _aix.py  _endian.py  macholib  test  util.py  wintypes.py

I'm not sure if any of the above is suspicious but I thought I'd take a look anyway.

I also have a hunch that with libffi missing during my initial python install (failed silently??), the appropriate symbolic links may not have been created properly or python wasn't compiled properly so I'll do a pyenv uninstall 3.10.2 and pyenv install 3.10.2 to recompile.

Now to check if ctypes was successfully built:

Python 3.10.2 (main, Mar 21 2022, 00:28:55) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> libc = CDLL("libc.so.6")
>>> print(libc.time(None))
1647848336

yay success!

NOW, back to installing ansible:

$ pip install ansible
...
... lots of output
...
done
Successfully installed MarkupSafe-2.1.1 PyYAML-6.0 ansible-5.5.0 ansible-core-2.12.3 cffi-1.15.0 cryptography-36.0.2 jinja2-3.0.3 packaging-21.3 pycparser-2.21 pyparsing-3.0.7 resolvelib-0.5.4

Excellent, and all the dependencies were installed too!

struggle tweets to install python 3.10.2 with pyenv on Ubuntu 20.04 (WSL)

milktea02 — Sun, 13 Mar 2022 01:17:40 +0000

If you want to watch me struggle, continue below. Otherwise just go here: https://computingforgeeks.com/how-to-install-python-on-ubuntu-linux-system/

I just bought a new laptop and figured, setting up my WSL Ubuntu environment should be pretty straightforward right? All I need to do is git clone a few repos and setup my $PATH.

I was very wrong.

I'm not an expert in setting up Linux environments and it just so happened that Stackoverflow was going through some things (https://twitter.com/StackStatus/status/1502788125285879817).

Anyway, some of the things I needed to get my pyenv v2.2.4 to even install python v3.10.2

At this point I've already apt install the following:



gcc
make
build-essential

but I'm still getting the following error:



$ pyenv install 3.10.2
Downloading Python-3.10.2.tar.xz...
-> https://www.python.org/ftp/python/3.10.2/Python-3.10.2.tar.xz
Installing Python-3.10.2...

BUILD FAILED (Ubuntu 20.04 using python-build 2.2.4)

Inspect or clean up the working tree at /tmp/python-build.20220312160907.27360
Results logged to /tmp/python-build.20220312160907.27360.log

Last 10 log lines:
  File "/tmp/python-build.20220312160907.27360/Python-3.10.2/Lib/ensurepip/__init__.py", line 277, in _main
    return _bootstrap(
  File "/tmp/python-build.20220312160907.27360/Python-3.10.2/Lib/ensurepip/__init__.py", line 193, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
  File "/tmp/python-build.20220312160907.27360/Python-3.10.2/Lib/ensurepip/__init__.py", line 93, in _run_pip
    return subprocess.run([sys.executable, '-W', 'ignore::DeprecationWarning',
  File "/tmp/python-build.20220312160907.27360/Python-3.10.2/Lib/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/python-build.20220312160907.27360/Python-3.10.2/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpbfu6lr32/setuptools-58.1.0-py3-none-any.whl\', \'/tmp/tmpbfu6lr32/pip-21.2.4-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpbfu6lr32\', \'--root\', \'/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make: *** [Makefile:1280: install] Error 1

Looking at the logs provided, indeed I am missing a lott of tools:



$ cat /tmp/python-build.20220312160907.27360.log | grep -e "hecking.*no$" | wc -l
197

I'm just looking blindly through and find the following:



ModuleNotFoundError: No module named 'zlib'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen zipimport>", line 618, in _get_data
  File "<frozen zipimport>", line 573, in _get_decompress_func
zipimport.ZipImportError: can't decompress data; zlib not available



The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_dbm                  _gdbm                 _hashlib
_lzma                 _sqlite3              _ssl
_tkinter              _uuid                 readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  pwd                   time


Failed to build these modules:
_ctypes


Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

okay, what is zlib? https://packages.ubuntu.com/source/focal/zlib

Not too helpful, as there are a bunch of different packages here that I can install, but which one?

The internet seems to say zlib1g-dev.

So I suppose I'll need to get the dev versions of all these C library packages:



zlib1g-dev
libbz2-dev
libsqlite3-dev
libgdbm-dev
libssl-dev
libncurses5-dev
libreadline-dev
liblzma-dev

Anyway, since I know nothing and am just meandering through this on my own even though I can copy paste the list of dependencies someone else has posted on the internet, I decide to only install zlib1g-dev since the other ones are required for optional modules (which I'll probably need later and regret not installing now).

Okay so what happens when I try to pyenv install 3.10.2 now?



$ pyenv install 3.10.2
Downloading Python-3.10.2.tar.xz...
-> https://www.python.org/ftp/python/3.10.2/Python-3.10.2.tar.xz
Installing Python-3.10.2...
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?

Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems


BUILD FAILED (Ubuntu 20.04 using python-build 2.2.4)

Inspect or clean up the working tree at /tmp/python-build.20220312165507.7502
Results logged to /tmp/python-build.20220312165507.7502.log

Last 10 log lines:
Installing collected packages: setuptools, pip
  WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /home/banana/.pyenv/versions/3.10.2/include/python3.10/setuptools
  sysconfig: /tmp/python-build.20220312165507.7502/Python-3.10.2/Include/setuptools
  WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /home/banana/.pyenv/versions/3.10.2/include/python3.10/pip
  sysconfig: /tmp/python-build.20220312165507.7502/Python-3.10.2/Include/pip
  WARNING: The scripts pip3 and pip3.10 are installed in '/home/banana/.pyenv/versions/3.10.2/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.2.4 setuptools-58.1.0

Okay that seems like a more positive step. The only error that I see now is that ssl extensions was not compiled. Seems important. Let's install libssl-dev

Okay AGAIN:



Downloading Python-3.10.2.tar.xz...
-> https://www.python.org/ftp/python/3.10.2/Python-3.10.2.tar.xz
Installing Python-3.10.2...

WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.10.2 to /home/banana/.pyenv/versions/3.10.2

YAY!



$ pyenv versions
* system (set by /home/banana/.pyenv/version)
  3.10.2

We'll ignore all the warnings... until next time.