DEV Community

Kamal Mustafa
Kamal Mustafa

Posted on

5 1

Python urldecode on command line

I have some logs that contain url with encoded characters and I need to extract some data out of the urls. Searching around lead me to this stackexchange's answer:-

alias urldecode='python3 -c "import sys, urllib.parse as ul;print(ul.unquote_plus(sys.argv[1]))"'
Enter fullscreen mode Exit fullscreen mode

Then you can use it as:-

urldecode '%22status%22%3A%22SUCCESS%22'
"status":"SUCCESS"
Enter fullscreen mode Exit fullscreen mode

Now that's perfect. But the data that I want to decode must be passed through unix pipe, like this:-

grep 'status: 504' /var/log/local0* | awk '{ print $10}' | xargs urldecode
Enter fullscreen mode Exit fullscreen mode

But this won't work, as xargs won't see the alias. So we alias xargs as well:-

alias xargs='xargs '
Enter fullscreen mode Exit fullscreen mode

But that however only work for the first line. How to make xargs execute for each line?

alias xargs='xargs -L 1 '
Enter fullscreen mode Exit fullscreen mode

Now we can run urldecode for each line we piped from grep!

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (2)

Collapse
 
rrampage profile image
Raunak Ramakrishnan • Edited

We can also do it without xargs by using sys.stdin to read from pipe
e.g:

yes 'Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en' |\
 head -n 10 |\
 python3 -c "import sys, urllib.parse as ul; [print(ul.unquote_plus(l), end='') for l in sys.stdin]"
Collapse
 
k4ml profile image
Kamal Mustafa

Thanks for the tips!

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more