DEV Community: JustJinoIT

I lost every 2FA method on GitHub. The way out was an SSH key I forgot on a server

JustJinoIT — Fri, 13 Nov 2026 12:41:00 +0000

I lost every 2FA method on my GitHub account. Web login has been dead since August. gh in the terminal still pushed fine, so I kept putting it off. This time I tried everything.

The way out was on a server: an SSH key I registered to my account back in May while setting up a side-project server. I put it there and forgot about it. It got my recovery request through, and GitHub is still reviewing it.

What didn't work

2FA app: I wiped and sold the phone in July
Recovery codes: GitHub's emails show I viewed them in March. I couldn't find where I saved them
Passkey: it lived in Edge on a Windows laptop. The laptop is gone
gh token: the recovery docs only list personal access tokens you create yourself. The gh login token isn't on the list
Old PATs: expired, or I don't know where they are
A recovery request in August: submitted, still locked out

I almost gave up on the account

Moving my repos to a new account changes the GitHub Pages URLs, which would break links I'd already submitted to contests. And contributions follow the commit email, which is tied to the locked account.

You can unlink just the email address from a locked account, then add that email to a new account and keep your commit history. But you can't undo it. I kept that as a last resort.

In the meantime I pulled all 22 repos with git clone --mirror.

It was on the server

GitHub account recovery asks for one more thing after email verification: a device you've signed in with before, an SSH key you registered before, or a PAT you created before. My account had one SSH key registered, and it didn't match any key on my Mac.

Digging through GitHub's security emails, I found "A new SSH authentication public key was added to your account", sent the day I set up that server in May. I wanted the server to pull from a private repo, so I generated a key on the server and added it straight to my account. In June I switched to a repo-only deploy key and never removed the old one.

Why did I even put it there? It saved me, though.

That key got me through the SSH verification step of the recovery request. Now I'm waiting for GitHub to review it.

If you want to check yours

The easiest way is to hand it to an AI agent. That's how I found mine, with Claude Code. Something like this:

I lost 2FA on my GitHub account (<username>) and can't log in on the web.
Find an "SSH key previously registered to the account" that I can use for recovery.

1. Get the fingerprints of keys registered to the account from https://github.com/<username>.keys
2. Compare them with the ~/.ssh key fingerprints on this machine and on servers I can reach
3. If nothing matches, search my email for GitHub "SSH key was added" alerts
   to see when and where it was registered
4. If you find it, check that ssh -i <key> -o IdentitiesOnly=yes -T git@github.com
   says "Hi <username>!"

Don't print private keys or tokens. Only show locations and fingerprints.
Ask me before connecting to any server.

To do it by hand:

Fingerprints of keys registered to your account:

curl -s https://github.com/<username>.keys | ssh-keygen -lf -

Find a matching key on each machine you have (servers included):

for k in ~/.ssh/*; do
  fp=$(ssh-keygen -y -P "" -f "$k" 2>/dev/null | ssh-keygen -lf - 2>/dev/null | awk '{print $2}')
  [ -n "$fp" ] && echo "$fp $k"
done | grep -F "$(curl -s https://github.com/<username>.keys | ssh-keygen -lf - | awk '{print $2}')"

Then check it. Hi <username>! means you can use it. Hi <username>/<repo>! means it's a repo-only deploy key, which won't work.

ssh -i <path to key> -o IdentitiesOnly=yes -T git@github.com

Worth checking now

Do you know where your recovery codes are?
Is your passkey on only one device?
Have you ever put an account-level SSH key on a server?

GitHub 2FA를 통째로 잃었다. 남은 건 서버에 두고 잊은 SSH 키였다

JustJinoIT — Fri, 13 Nov 2026 12:40:54 +0000

GitHub 2FA를 통째로 잃어버렸다. 8월부터 웹 로그인이 안 됐다. 터미널은 gh로 push가 계속 되니까 미뤄 두다가, 이번에 별짓을 다 해 봤다.

결론은 서버였다. 5월에 사이드 프로젝트 서버 세팅하면서 계정에 등록해 둔 SSH 키. 내가 올려 놓고 까먹었다.

안 됐던 것들

2FA 앱: 폰을 7월에 초기화해서 팔았다
복구 코드: 3월에 열어 본 기록은 있는데 어디 저장했는지 못 찾았다
패스키: 윈도우 노트북 Edge에 있었다. 노트북은 버렸다
gh 토큰: 복구 문서엔 직접 만든 PAT만 나온다. gh 로그인 토큰은 없다
예전 PAT: 만료됐거나 어디 뒀는지 모른다
8월 복구 요청: 넣었는데 안 풀렸다

계정을 버릴까도 했다

새 계정으로 저장소를 옮기면 GitHub Pages 주소가 바뀐다. 공모전에 낸 링크가 깨진다. 잔디(기여 그래프)도 이메일을 따라가는데 그 이메일이 잠긴 계정에 묶여 있다.

잠긴 계정에서 이메일 연결만 끊는 방법은 있다. 끊은 이메일을 새 계정에 붙이면 커밋 기록은 유지된다. 대신 되돌릴 수 없다. 이건 마지막으로 미뤘다.

일단 저장소 22개는 git clone --mirror로 다 받아 놨다.

서버에 있었다

GitHub 복구 요청은 메일 인증 말고 하나를 더 요구한다. 예전에 로그인한 기기, 예전에 등록한 SSH 키, 예전에 만든 PAT 중 하나. 계정에 등록된 키는 1개인데 맥에 있는 키랑은 안 맞았다.

GitHub 알림 메일을 뒤지니 5월 서버 세팅하던 날 "계정에 새 SSH 키가 등록됐다"는 메일이 있었다. 서버가 비공개 저장소를 pull하게 하려고 서버에서 만든 키를 계정에 바로 넣었던 거다. 6월에 저장소 전용 키로 바꾸면서 이건 안 지웠다.

왜 올렸을까 이걸. 그 덕에 살았지만.

이 키로 복구 요청 SSH 인증까지 통과했다. 지금은 GitHub 검토 기다리는 중.

나도 찾아보려면

제일 쉬운 건 AI 에이전트한테 시키는 거다. 나도 Claude Code한테 시켜서 찾았다. 이렇게 넣으면 된다.

GitHub 계정(<계정>) 2FA를 잃어서 웹 로그인이 안 된다.
복구 요청에 쓸 "예전에 계정에 등록한 SSH 키"를 찾아줘.

1. https://github.com/<계정>.keys 로 계정에 등록된 키 지문 확인
2. 이 컴퓨터와 내가 접속할 수 있는 서버의 ~/.ssh 키 지문과 비교
3. 못 찾으면 메일에서 GitHub "SSH key was added" 알림을 찾아
   언제 어디서 등록했는지 추적
4. 찾으면 ssh -i <키> -o IdentitiesOnly=yes -T git@github.com 으로
   "Hi <계정>!"이 나오는지 확인

개인키·토큰 값은 출력하지 말고 위치와 지문만 알려줘.
서버에 접속하기 전에는 먼저 물어봐.

직접 하려면 아래 순서다.

계정에 등록된 키 지문:

curl -s https://github.com/<계정>.keys | ssh-keygen -lf -

갖고 있는 기기(서버 포함)에서 지문 맞는 키 찾기:

for k in ~/.ssh/*; do
  fp=$(ssh-keygen -y -P "" -f "$k" 2>/dev/null | ssh-keygen -lf - 2>/dev/null | awk '{print $2}')
  [ -n "$fp" ] && echo "$fp $k"
done | grep -F "$(curl -s https://github.com/<계정>.keys | ssh-keygen -lf - | awk '{print $2}')"

찾았으면 확인. Hi <계정>!이면 쓸 수 있다. Hi <계정>/<저장소>!면 저장소 전용 키라 못 쓴다.

ssh -i <키 경로> -o IdentitiesOnly=yes -T git@github.com

확인해 볼 것

복구 코드 어디 있는지 아나?
패스키가 기기 한 대에만 있진 않나?
서버에 계정용 SSH 키 올려 둔 적 있나?

AI 커밋이 2배 더 고쳐진다고? 비교군이 하루치였다

JustJinoIT — Thu, 15 Oct 2026 14:53:07 +0000

Ars Technica에서 하버드 워킹페이퍼(Fiona Chen, James Stratton) 기사를 읽었다. 700여 개 회사의 작업 기록 약 3억 건을 분석했더니, AI 코딩 에이전트를 도입한 뒤 PR은 23%, 코드 생산량은 30% 정도 늘었는데 이슈 해결률은 통계적으로 의미 있게 달라지지 않았다고 한다. 병목은 리뷰였다. PR이 머지되기까지 걸리는 시간이 49% 늘었고 수정 요청은 거의 두 배가 됐다.

내 레포도 그런지 재 봤다. 사이드 프로젝트 두 개(아래 레포 A, B)이고, 둘 다 6월 4일에 Claude와 함께 시작했다.

메시지에 Co-Authored-By: Claude 트레일러가 있으면 AI 커밋(245개), 없으면 "트레일러 없는 커밋"(65개)으로 나눴다. 지표는 제목이 fix나 revert로 시작하는 커밋의 비율이다.

	AI 커밋	트레일러 없는 커밋
제목이 fix/revert로 시작	38%	17%

두 배가 넘는다. 연구와 같은 방향이다(같은 지표는 아니다). 이대로 글을 쓸 뻔했는데, 먼저 깨 보기로 했다.

확인 1: `fix:`는 누가 제목에 쓰나

AI 커밋의 69%가 feat:, fix: 같은 표준 형식 제목을 쓴다. 트레일러 없는 커밋은 18%다. 고칠 때마다 fix:를 붙이는 에이전트는 제목을 읽는 지표에서 더 많이 잡힌다. fix, bug, revert, 수정, 버그, 고침, 오류 중 하나가 제목 어디에든 들어간 커밋으로 넓히면 격차가 벌써 줄어든다. 43% 대 26%다.

확인 2: 트레일러 없는 커밋은 누구 것인가

이 커밋들을 사람 쪽 비교군으로 두고도 정작 열어 보지 않았다. 65개 중 57개가 6월 5일 하루에 몰려 있었다. 제목은 이런 식이다.

Phase 0-1: Add core modules (logging, pipeline, batch)
Phase A: Semantic cache + batch processing integration
Enhancement Level 2: Sentiment analysis & topic classification

뼈대를 세우던 날의 커밋으로 읽힌다. 두 레포 모두 6월 4일의 첫 커밋부터 AI 트레일러가 있어서, AI가 없던 시기는 비교 기준으로 쓸 수도 없다. 이 커밋을 누가 썼는지 나는 말할 수 없다. 그게 문제다. 내 비교군은 누가 만들었는지 모르는 하루치 작업이었다.

확인 3: 커밋 크기

AI 커밋은 한 번에 건드리는 파일이 더 많다. 평균 3.1개이고 다른 쪽은 1.7개다. "같은 파일이 72시간 안에 다시 수정됐나"를 커밋 단위로 세면 큰 커밋이 겹칠 기회가 더 많다. 커밋 단위로는 74% 대 68%였다. 파일 단위로 세서 커밋 크기를 뺐다.

측정 방식	AI 커밋	트레일러 없는 커밋
제목이 fix/revert로 시작	38%	17%
제목 어디든 fix·버그·수정 등 포함	43%	26%
건드린 파일이 72시간 안에 다시 수정됨 (파일 단위, 두 레포 합산)	46%	47%

제목과 커밋 크기를 둘 다 뺀 마지막 줄에서는 차이가 없다.

그래서 이 숫자는

두 레포만 가지고는 AI 커밋이 더 자주 고쳐지는지 덜 고쳐지는지 알 수 없다. 연구가 틀렸다는 말도 아니다. 표본이 작고 인과를 보여 주는 것도 아니다. 처음 나온 2배는 의도하지 않고 잰 세 가지에서 왔다. 제목 쓰는 습관, 하루치로 이뤄진 비교군, 커밋 크기.

다음에 AI와 비AI 작업의 격차를 믿기 전에 먼저 이걸 본다.

비교군에 누가 들어 있고, 언제 커밋한 건가?
내가 세는 라벨이 측정 대상의 습관에서 나온 건 아닌가?
두 그룹이 같은 크기의 일을 하고 있나?

내 레포에서 재 보려면

표준 라이브러리만 쓴다. 첫 AI 커밋 이후의 커밋만 비교한다. 다른 도구를 쓴다면 co-authored-by: claude 문자열만 바꾸면 된다.

import subprocess, sys
from collections import Counter
from datetime import datetime

repo = sys.argv[1] if len(sys.argv) > 1 else "."
out = subprocess.run(
    ["git", "-C", repo, "log", "--no-merges", "--name-only",
     "--pretty=format:\x1e%ct\x1f%B\x1f"],
    capture_output=True, text=True).stdout

commits = []  # dict: 시각, AI 트레일러 여부, 제목, 건드린 파일
for chunk in out.split("\x1e")[1:]:
    t, msg, files = chunk.split("\x1f")
    commits.append(dict(t=int(t), ai="co-authored-by: claude" in msg.lower(),
                        subj=msg.strip().split("\n")[0], files=set(files.split())))
commits.sort(key=lambda c: c["t"])

if not any(c["ai"] for c in commits):
    sys.exit("AI 트레일러가 붙은 커밋이 없다. 메시지에 Co-Authored-By: Claude 가 있는지 확인할 것")
start = min(c["t"] for c in commits if c["ai"])   # 첫 AI 커밋 이후만 비교
commits = [c for c in commits if c["t"] >= start]

pairs = {True: [0, 0], False: [0, 0]}             # [파일 수, 72시간 안에 다시 수정된 파일 수]
for i, c in enumerate(commits):
    for f in c["files"]:
        pairs[c["ai"]][0] += 1
        if any(c["t"] < d["t"] <= c["t"] + 72 * 3600 and f in d["files"] for d in commits[i + 1:]):
            pairs[c["ai"]][1] += 1

for ai in (True, False):
    g = [c for c in commits if c["ai"] == ai]
    if not g:
        continue
    fix = sum(c["subj"].lower().startswith(("fix", "revert")) for c in g)
    day, k = Counter(datetime.fromtimestamp(c["t"]).date() for c in g).most_common(1)[0]
    n_f, hit = pairs[ai]
    print("AI       " if ai else "트레일러 없음", f"커밋 {len(g)}개 | 제목이 fix/revert로 시작 {fix / len(g):.0%}"
          f" | 커밋당 파일 {sum(len(c['files']) for c in g) / len(g):.1f}개"
          f" | 파일 단위 72시간 내 재수정 {hit / max(n_f, 1):.0%}"
          f" | 가장 몰린 날 {k / len(g):.0%}")

내 두 레포에서는 이렇게 나왔다.

레포	그룹	커밋	fix/revert 제목	커밋당 파일	72시간 내 재수정(파일)	가장 몰린 날
레포 A	AI	145	40%	2.9	47%	10%
	트레일러 없음	16	25%	1.6	52%	62%
레포 B	AI	100	36%	3.4	44%	23%
	트레일러 없음	49	14%	1.8	46%	96%

마지막 열은 처음부터 찍어 봤어야 했다. 비교군의 커밋 대부분이 하루에 몰려 있다면, 어떤 배율이든 믿기 전에 그 하루부터 열어 보는 게 낫다.

돌려 보면 "가장 몰린 날" 열이 어떻게 나오는지 궁금하다.

My AI-vs-human commit comparison had a control group of one day

JustJinoIT — Thu, 15 Oct 2026 14:52:44 +0000

Ars Technica covered a Harvard working paper (Fiona Chen, James Stratton) built on about 300M work events from 700+ companies. After teams adopted coding agents, PRs went up 23% and code output about 30%, but issue resolution didn't move in a statistically significant way. Review was the bottleneck: time to merge rose 49%, and revision requests nearly doubled.

I wanted to see if my own repos looked like that. So I ran one comparison on two of my side projects (Repo A and Repo B below), both built with Claude since June 4.

AI commit = the message has a Co-Authored-By: Claude trailer (245 commits). Everything else is "no trailer" (65 commits). Metric: share of commits whose subject starts with fix or revert.

	AI commits	No-trailer commits
subject starts with fix/revert	38%	17%

More than double. Same direction as the study, though not the same metric. I almost wrote the post right there. Then I tried to break it.

Check 1: who writes `fix:` in the subject?

69% of the AI commits use feat: / fix: style subjects. Among the no-trailer commits it's 18%. An agent that stamps fix: on every fix will get counted more by any metric that reads subjects. Widening the match to any subject containing fix, bug, revert or their Korean equivalents already shrinks the gap: 43% vs 26%.

Check 2: who are the no-trailer commits?

I had been treating them as the human side of the comparison without looking at them. 57 of the 65 landed on a single day, June 5. Their subjects look like this:

Phase 0-1: Add core modules (logging, pipeline, batch)
Phase A: Semantic cache + batch processing integration
Enhancement Level 2: Sentiment analysis & topic classification

That reads like scaffolding day. Both repos also start on June 4 with an AI-trailer commit, so there is no pre-AI period in either one to use as a baseline. I can't say who wrote the no-trailer commits, and that is the problem: my "control group" is one day of work whose authorship I don't know.

Check 3: commit size

AI commits touch more files: 3.1 on average, against 1.7. If I count "the same file was touched again within 72 hours" per commit, bigger commits get more chances to overlap. Per commit it said 74% vs 68%. Counting per file instead, so that commit size drops out:

Measure	AI commits	No-trailer commits
subject starts with fix/revert	38%	17%
subject contains fix/bug/revert (or Korean equivalents) anywhere	43%	26%
a touched file is edited again within 72h (per file, both repos combined)	46%	47%

The last row ignores subjects and commit size, and it shows no gap.

Where that leaves the number

I can't tell from two repos whether AI commits get fixed more or less. I'm not saying the study is wrong either: it's a tiny sample and nothing here shows cause. The 2x I started with came from three things I wasn't measuring on purpose: a naming habit, a control group made of one day, and commit size.

Before I trust a gap between AI and non-AI work again, I want answers to these:

Who is actually in the comparison group, and when did they commit?
Does the label I'm counting come from a habit of the thing I'm measuring?
Are the two groups doing the same size of work?

Run it on your repo

Standard library only. It compares commits after the first AI commit. Change the co-authored-by: claude string if you use another tool.

import subprocess, sys
from collections import Counter
from datetime import datetime

repo = sys.argv[1] if len(sys.argv) > 1 else "."
out = subprocess.run(
    ["git", "-C", repo, "log", "--no-merges", "--name-only",
     "--pretty=format:\x1e%ct\x1f%B\x1f"],
    capture_output=True, text=True).stdout

commits = []  # dict: time, AI trailer, subject, files touched
for chunk in out.split("\x1e")[1:]:
    t, msg, files = chunk.split("\x1f")
    commits.append(dict(t=int(t), ai="co-authored-by: claude" in msg.lower(),
                        subj=msg.strip().split("\n")[0], files=set(files.split())))
commits.sort(key=lambda c: c["t"])

if not any(c["ai"] for c in commits):
    sys.exit("No commits with an AI trailer found. Check for Co-Authored-By: Claude in messages.")
start = min(c["t"] for c in commits if c["ai"])   # only compare after the first AI commit
commits = [c for c in commits if c["t"] >= start]

pairs = {True: [0, 0], False: [0, 0]}             # [files, files edited again within 72h]
for i, c in enumerate(commits):
    for f in c["files"]:
        pairs[c["ai"]][0] += 1
        if any(c["t"] < d["t"] <= c["t"] + 72 * 3600 and f in d["files"] for d in commits[i + 1:]):
            pairs[c["ai"]][1] += 1

for ai in (True, False):
    g = [c for c in commits if c["ai"] == ai]
    if not g:
        continue
    fix = sum(c["subj"].lower().startswith(("fix", "revert")) for c in g)
    day, k = Counter(datetime.fromtimestamp(c["t"]).date() for c in g).most_common(1)[0]
    n_f, hit = pairs[ai]
    print("AI        " if ai else "no trailer", f"{len(g)} commits | subject starts with fix/revert {fix / len(g):.0%}"
          f" | {sum(len(c['files']) for c in g) / len(g):.1f} files per commit"
          f" | file re-edited within 72h {hit / max(n_f, 1):.0%}"
          f" | busiest day holds {k / len(g):.0%} of the group")

On my two repos:

Repo	Group	Commits	fix/revert subject	Files per commit	File re-edited in 72h	Busiest day
Repo A	AI	145	40%	2.9	47%	10%
	no trailer	16	25%	1.6	52%	62%
Repo B	AI	100	36%	3.4	44%	23%
	no trailer	49	14%	1.8	46%	96%

The last column is the one I wish I had printed on day one. If your comparison group has most of its commits on one day, stop and look at it before you trust any ratio.

If you run it, what does your busiest-day column look like?

AstaBrief-8B가 Claude보다 3.5배 빠르다고? 직접 재현해보니

JustJinoIT — Mon, 05 Oct 2026 11:05:14 +0000

왜 재봤나

AllenAI 블로그가 AstaBrief-8B를 소개하며 "기존보다 3.5배 빨라져서 리포트 하나에 50초면 된다"고 썼다(Threads 소개 글). 같은 하드웨어·같은 분량 기준 비교였는지 의문이 제기됐는데, 실제로 블로그 원문엔 테스트 조건(하드웨어, 쿼리 수, 비교 대상)이 전혀 안 적혀 있었다. 무료 Colab T4로 직접 재봤다.

처음엔 틀렸다

첫 시도는 그냥 AstaBrief랑 Claude를 같은 프롬프트로 돌려서 시간 재는 거였다. 근데 두 가지가 비교를 왜곡했다.

비교 대상이 잘못됐었다: AstaBrief는 사실 Qwen3-8B를 파인튜닝한 모델이다(블로그에 "We started from Qwen3-8B"라고 적혀있다). 파인튜닝 자체의 속도 영향을 보려면 베이스 모델이랑 비교해야 의미가 있다.
출력 길이를 안 고정했었다: AstaBrief는 1024토큰 상한까지 꽉 채워 답했고 Qwen은 367토큰에서 자연 종료됐다. 그 상태로 wall-clock을 그대로 비교하니 "AstaBrief가 4.29배 느리다"는 결론이 나왔는데, 이것도 틀린 비교였다 — 더 긴 글을 썼으니 더 오래 걸린 것뿐이다.

다시 맞춰서 잰 결과

출력 토큰을 200으로 고정하고(min_new_tokens=max_new_tokens=200), 같은 T4에서 3회씩 돌려 평균 냈다.

AstaBrief-8B: 19.62초 (10.20 tok/s)
Qwen3-8B(파인튜닝 전 베이스): 19.64초 (10.19 tok/s)
배율: 1.00배

파인튜닝은 속도에 아무 영향이 없었다. 그럼 블로그의 "3.5배"는 어디서 나온 숫자였나 — AstaBrief 단일 모델 호출(Fast mode)과, Claude 기반 멀티스텝 에이전트 파이프라인(Thinking mode: 검색+클러스터링+섹션별 생성 등 여러 단계)을 비교한 수치였다. 느린 건 그 파이프라인 구조 때문이지, AstaBrief가 일반 8B 모델보다 빠른 게 아니다. 원문에도 이 부분은 적혀 있다 — "대부분의 학습·평가는 2025년에 끝났고, 최신 프론티어 모델로 재평가하진 않았다"고.

그럼 파인튜닝은 뭘 위한 거였나 싶어서 찾아보니, ScholarQA-CS2 기준 인용 정확도가 베이스 모델 대비 올라가 있었다(90.5 vs 76.2). 아마 이게 진짜 목적이었을 거다 — 속도가 아니라 품질. Threads에도 이 검증 결과를 공유했다.

한계

GPU 한 종류(T4), 양자화 한 방식(4bit), 쿼리 하나로만 잰 결과다. 다른 환경에서는 숫자가 달라질 수 있다. "3.5배는 과장이다"라고 단정하긴 어렵고, "3.5배가 가리키는 게 모델 자체의 속도가 아니라 파이프라인 구조 차이"라는 정도까지만 말할 수 있다.

원문: AstaBrief-8B — AllenAI Blog

"이제 완전히 고쳐졌나?"에 정직한 답은 없다 — 작은 서버 이야기

JustJinoIT — Sun, 21 Jun 2026 02:43:30 +0000

1GB 프리티어 VPS 하나로 사이드 프로젝트를 돌린다. 작은 서버, 서비스 몇 개.

지난주에 인프라를 손봤다. 고칠 때마다 같은 걸 물었다. 이제 완전히 고쳐졌나? 다시 안 터지나? 정직한 답은 계속 "아니"였다. 심지어 고친 것 때문에 다음 문제가 터지기도 했다.

그 질문을 그만 묻게 만든 사건을 적어둔다.

사건: 모니터링이 멀쩡한 서버를 죽일 뻔했다

구성은 이랬다.

본체: 1GB VPS, API 하나
watchdog: 다른 서버에서 5분마다 /health 체크. 죽으면 알림, 계속 죽어있으면 클라우드 API로 강제 재부팅.

안전하다고 느꼈다. 자동 복구까지 되니까.

그러다 본체에서 ffmpeg 작업을 돌렸다. 1코어라 CPU를 통째로 먹었고, 그 사이 /health가 10초 안에 응답을 못 했다. watchdog은 "응답 없음 = 죽음"으로 읽었다. 한 번만 더 실패했으면 멀쩡한 프로덕션 서버를 재부팅할 차례였다.

서버는 안 죽었다. 내가 돌린 작업이 느리게 만들었을 뿐인데, 지키려고 만든 게 죽일 뻔했다.

"완전히 고쳐졌나"에 정직한 yes가 없는 이유

1. "고쳐졌나"는 닫힌 질문, 시스템은 열려있다.
막을 수 있는 건 이미 본 실패뿐이다. 변경할 때마다 새 표면적이 생긴다.

2. 작은 서버는 결합도가 높다.
1코어 1기가. 무거운 작업 하나가 나머지를 굶긴다. 격리가 안 되니 "이건 안전"이 옆을 건드린다.

3. 고치는 행위도 변수다.
모니터링도 시스템의 일부고, 이번엔 그게 장애원이었다. 같은 세션에서 알림 스크립트에 실행권한을 준 게 가짜 알림의 원인이 되기도 했다.

4. 자신만만한 답은 싸다.
"됐어, 안전해"는 파급을 확인하기 전에 말하기 쉽다. 도구든, AI든, 나든. 내가 계속 "고쳐졌나"를 물은 건 그 yes를 못 믿어서였고, 안 믿길 잘했다.

진짜 도움이 된 것

보장이 아니라 감지와 되돌리기.

watchdog 수정은 "완벽하게"가 아니었다. 한 번 실패에 단정하지 않기. 몇 번 더 보고 다 실패하면 그때 죽음으로 친다.

code="000"
for i in 1 2 3; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$URL")
  [ "$code" = "200" ] && break
  [ "$i" -lt 3 ] && sleep 10
done
# 세 번 다 실패해야 진짜 죽음

순간 스파이크는 30초 안에 회복되니까, 세 번 다 실패면 진짜 죽은 거다. 무거운 작업은 굶길 서버에서 안 돌린다.

결론

"완전히 고쳐졌나"는 어떤 정직한 시스템도 yes라고 못 하는 질문이다. 쓸모 있는 버전은 더 좁다.

어떤 알려진 실패를 막았고, 어떻게 검증했나?
못 막는 건 뭐고, 어떻게 감지하고 되돌릴 건가?

뭔가 "네, 완전히 처리됐어요"라고 답하면, 보통 그때가 더 들여다볼 때다.

Why "is it fully fixed?" has no honest answer — a small-server story

JustJinoIT — Sun, 21 Jun 2026 02:24:14 +0000

I run a side project on a 1GB free-tier VPS. Small box, a few services, nothing fancy.

Last week I spent a session hardening its infra. After each fix I asked the same thing: is it fully fixed now? Will this stop happening? The honest answer kept coming back no, and sometimes the fix itself caused the next problem.

Here is the incident that made me retire that question.

The incident: my monitoring almost killed a healthy server

The setup:

Main box: 1GB VPS, one API service.
Watchdog: a second box pings /health every 5 minutes. If it looks down, it alerts. If it stays down, it calls the cloud API to hard-reboot the main box.

Felt safe. Self-healing, even.

Then I ran an ffmpeg job on the main box. One core, so ffmpeg took the whole CPU. During that window, /health could not answer within the 10s timeout. The watchdog read "no response" as "dead." One more failed check and it would have rebooted a perfectly healthy production box.

The server never went down. A job I started made it slow, and the thing I built to protect it almost rebooted it.

Why "is it fully fixed?" has no honest yes

1. "Fixed" is a closed question; the system is open.
You can only defend against failures you have already seen. Every change adds new surface. "Will this never happen again" has no honest yes.

2. Small servers are tightly coupled.
One core, one gig. A heavy job starves everything else. No isolation, so "this part is safe" can quietly break the part next to it.

3. The fix is a variable too.
Monitoring is part of the system. This time it was the failure. Earlier in the same session, making an alert script executable turned it into a source of false alarms. Every change made to harden things added a new way to fail.

4. Confident answers are cheap.
"Done, safe" is easy to say before checking the blast radius, whether it comes from a tool, an AI agent, or me. The reason I kept asking "is it fixed?" was that part of me did not believe the yes. I was right not to.

What actually helps

Not guarantees. Detection and reversibility.

The watchdog fix was not "make it perfect." It was: stop trusting a single failed check. Retry a few times before declaring death.

code="000"
for i in 1 2 3; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$URL")
  [ "$code" = "200" ] && break
  [ "$i" -lt 3 ] && sleep 10
done
# only "dead" if all three fail

A transient spike recovers within 30 seconds, so three misses in a row means it is really down. And heavy jobs do not run on the box they would starve.

The takeaway

"Is it fully fixed?" is a question no honest system can answer yes to. The useful version is narrower:

Which known failures did you close, and how did you verify them?
What can you not prevent, and how will you detect and roll it back?

When something answers "yes, it is fully handled," that is usually the moment to look closer.

같은 게시물을 세 번 올리고 나서야 이유를 알았다

JustJinoIT — Fri, 19 Jun 2026 09:49:20 +0000

같은 게시물을 세 번 올리고 나서야 이유를 알았다

자동화 봇을 테스트하면서 limit이 자꾸 걸렸다. AI한테 물어봤더니 이런 말이 돌아왔다.

"앱이 개발 중 상태라 제약이 있어요. 24-48시간 기다리면 풀릴 거예요."

그럴싸했다. 믿었다.

Meta 콘솔 토끼굴

기다렸다. 또 안 됐다.

그럼 앱 게시 문제인가 싶었다. Meta 개발자 콘솔 들어갔다. AI가 알려준 버튼이 없었다. 메뉴가 달랐다. 다른 경로로 들어갔다. 권한 항목 찾아서 추가했다. 앱 검수 신청 화면 들어갔다. 뭔가 테스트를 완료해야 한다고 해서 실행해봤다. 다시 권한 뺐다가 넣었다. 토큰 재발급 받았다. 또 테스트.

인증 화면까지 들어갔더니 사업자 등록 정보를 넣으라고 했다. 거기서 멈췄다. 이건 아니다 싶었다.

Graph API Explorer 열어서 직접 뒤적이기 시작했다. 뭔가 나오긴 했는데 뭘 보고 있는 건지도 불분명했다.

"중복 자료가 왜 이렇게 많냐"

그러다 인스타그램을 직접 열어봤다. 같은 게시물이 세 개였다. 피드 전체 보니까 12개. 여러 기사에서 중복이 각각 쌓여있었다.

코드 흐름을 다시 따라갔다. 403 받으면 실패로 처리하고, DB 업데이트 안 하고, 다음 스케줄에 재시도. 그 재시도도 403. 또 재시도. 근데 인스타에는 매번 올라갔다. 몇 사이클 돌고 나니 이렇게 됐다.

받은 에러:

{
  "error": {
    "message": "Application request limit reached",
    "type": "OAuthException",
    "code": 4,
    "error_subcode": 2207051,
    "is_transient": false
  }
}

rate limit이겠지 했다. 확인해봤다:

r = await client.get(
    f"{GRAPH_URL}/{user_id}/content_publishing_limit",
    params={"fields": "config,quota_usage", "access_token": token}
)

quota_usage: 0. 한도엔 전혀 안 걸려있었다. x-app-usage 헤더도 0%. rate limit이 아니었다.

행동 차단인가. 근데 행동 차단이 됐으면 어딘가 뜰 거 아냐? 앱, Business Suite, 개발자 콘솔. 아무것도 없었다. 경고도, 배너도, 표시 하나도.

행동 차단은 어디에도 안 뜬다. 에러 응답의 is_transient: false가 유일한 신호다. rate limit은 보통 is_transient: true에 24시간 후 풀린다. false면 기다린다고 해결 안 된다. 이게 구분 기준이었다.

원인은 테스트하면서 1시간 안에 게시물 13개를 올린 거였다. 일일 한도 25개엔 한참 못 미치는데. 단시간에 몰아 올리면 자동화 패턴으로 감지돼서 별도 제한이 걸린다. 문서엔 없다.

이 차단 상태에서 media_publish는 403을 뱉는데 게시물은 올라간다. 코드가 403을 실패로만 처리했으니 재시도했고, 재시도마다 중복이 쌓였다.

이렇게 고쳤다:

async def _publish_container(client, container_id):
    resp = await client.post(
        f"{GRAPH_URL}/{IG_USER_ID}/media_publish",
        params={"creation_id": container_id, "access_token": IG_TOKEN},
    )
    if resp.is_success:
        return resp.json().get("id")

    if resp.status_code == 403:
        await asyncio.sleep(3)
        check = await client.get(
            f"{GRAPH_URL}/{IG_USER_ID}/media",
            params={"fields": "id,timestamp", "limit": 1, "access_token": IG_TOKEN},
        )
        if check.is_success:
            data = check.json().get("data", [])
            if data:
                ts = datetime.fromisoformat(data[0]["timestamp"].replace("Z", "+00:00"))
                if datetime.now(timezone.utc) - ts < timedelta(seconds=60):
                    return data[0]["id"]

    resp.raise_for_status()

403 받으면 바로 최근 미디어 조회. 60초 이내에 올라온 게 있으면 성공으로 친다.

"추가했는데 해볼래?" 그리고 또 실패

중복 12개 지워야 했다. 삭제엔 instagram_manage_contents 권한이 필요한데 내 토큰엔 없었다.

권한 추가. 확인. 삭제 시도.

{"error": {"message": "(#10) Insufficient permissions", "code": 10}}

실패. 다시. 같은 에러. 콘솔엔 분명히 있는데.

OAuth 토큰은 발급 시점의 scope를 고정으로 가진다. 앱에 권한 추가해도 기존 토큰은 그대로다. 새로 발급받아야 한다.

토큰이 실제로 뭘 갖고 있는지 보려면:

r = await client.get(
    "https://graph.facebook.com/v22.0/me/permissions",
    params={"access_token": token}
)
granted = [p["permission"] for p in r.json()["data"] if p["status"] == "granted"]

instagram_manage_contents가 없으면 Graph API Explorer에서 scope 체크하고 재발급. 그 다음엔 됐다.

진짜 원인 조합

신호	의미
`quota_usage: 0`인데 403	행동 차단 (rate limit 아님)
`is_transient: false`	기다려도 안 풀림
403인데 인스타에 게시됨	차단 상태에서도 게시는 통과함
행동 차단이 UI에 표시됨	안 됨, 어디에도

API 동작은 문서에 없고, AI는 자신 있게 틀린 경로를 알려줬고, 그 경로의 콘솔 UI는 또 바뀌어있었다. 어디서부터 잘못됐는지 찾는 게 제일 어려웠다.

AI가 알려주는 콘솔 메뉴는 실제랑 다를 수 있다. API가 이상하다 싶으면 UI 말고 응답 필드 먼저 봐라. is_transient 하나가 대시보드보다 정직했다.

오후 하나 날렸다. 중복 12개, Meta 토끼굴, 권한 삽질. 그나마 건진 건 이 글이다.

Instagram Graph API v22.0. 카드뉴스 자동 발행 봇 만들면서 겪은 내용. 봇은 @dogfootbro.ai에 올라간다.

I published the same post three times before I understood what was happening

JustJinoIT — Fri, 19 Jun 2026 09:49:04 +0000

I published the same post three times before I understood what was happening

I was testing an automation bot and kept hitting limits. Asked an AI assistant what was going on.

"The app is in development mode, so there are restrictions. Wait 24-48 hours and it should clear up."

Sounded reasonable. I believed it.

The Meta console rabbit hole

Waited. Still broken.

Maybe it's an app publishing issue, I thought. Went into the Meta developer console. The button the AI told me about wasn't there. The menu was different. Found another path. Located the permissions section and added what I needed. Got into the app review screen. It said I had to complete some tests first, so I ran them. Removed permissions, added them back. Got a new token. Tested again.

Eventually got into the verification flow and it asked for business registration info. That's where I stopped. This isn't right.

Opened Graph API Explorer and started poking around directly. Something was coming back but I wasn't sure what I was looking at.

"Why are there so many duplicates?"

At some point I just opened Instagram directly. Same post, three times. Looked at the whole feed: 12 duplicates across multiple articles, each one stacked up from previous runs.

Traced through the code. On 403, treat as failure, skip DB update, retry next cycle. That retry also 403'd. Retried again. But Instagram got the post every time. A few cycles in and this is what happened.

The error:

{
  "error": {
    "message": "Application request limit reached",
    "type": "OAuthException",
    "code": 4,
    "error_subcode": 2207051,
    "is_transient": false
  }
}

Rate limit, obviously. Checked:

r = await client.get(
    f"{GRAPH_URL}/{user_id}/content_publishing_limit",
    params={"fields": "config,quota_usage", "access_token": token}
)

quota_usage: 0. Not even close to the cap. x-app-usage header was 0% too. Not a rate limit.

Maybe a behavior block then. But if you're behavior-blocked, wouldn't something show up somewhere? Checked the app, Business Suite, developer console. Nothing. No warning, no banner, no indicator.

Behavior blocks don't show up anywhere. The only signal is is_transient: false in the error response. Rate limits are usually is_transient: true and reset after 24 hours. false means waiting won't help. That's the only way to tell them apart.

The cause: 13 posts within one hour during testing. Well under the documented daily limit of 25, but posting that fast triggers a separate undocumented restriction. Instagram sees the pattern as automated. Nothing in the docs about this.

In this blocked state, media_publish returns 403 but the post goes through. Code treated 403 as failure, retried, and each retry stacked another duplicate.

Fixed it like this:

async def _publish_container(client, container_id):
    resp = await client.post(
        f"{GRAPH_URL}/{IG_USER_ID}/media_publish",
        params={"creation_id": container_id, "access_token": IG_TOKEN},
    )
    if resp.is_success:
        return resp.json().get("id")

    if resp.status_code == 403:
        await asyncio.sleep(3)
        check = await client.get(
            f"{GRAPH_URL}/{IG_USER_ID}/media",
            params={"fields": "id,timestamp", "limit": 1, "access_token": IG_TOKEN},
        )
        if check.is_success:
            data = check.json().get("data", [])
            if data:
                ts = datetime.fromisoformat(data[0]["timestamp"].replace("Z", "+00:00"))
                if datetime.now(timezone.utc) - ts < timedelta(seconds=60):
                    return data[0]["id"]

    resp.raise_for_status()

After a 403, immediately check recent media. If something posted within 60 seconds, call it a success.

"I added the permission. Try it now." Still failing.

Had to delete the 12 duplicates. Deletion requires instagram_manage_contents. My token only had instagram_content_publish.

Added the permission. Confirmed it in the console. Tried the delete.

{"error": {"message": "(#10) Insufficient permissions", "code": 10}}

Failed. Tried again. Same error. The permission was right there in the console.

OAuth tokens carry a fixed scope from when they were issued. Adding a permission to your app doesn't update existing tokens. Had to generate a new one.

To see what a token actually has:

r = await client.get(
    "https://graph.facebook.com/v22.0/me/permissions",
    params={"access_token": token}
)
granted = [p["permission"] for p in r.json()["data"] if p["status"] == "granted"]

If instagram_manage_contents isn't there, go to Graph API Explorer, check the scope, regenerate. After that it worked.

What was actually going on

Signal	Means
`quota_usage: 0` but still 403	Behavior block, not rate limit
`is_transient: false`	Waiting won't fix it
403 on publish but post appears	Block lets posts through anyway
Behavior block visible in any UI	No, nowhere

The API behavior wasn't documented. The AI confidently pointed me in the wrong direction. The console UI it described had changed. Finding where things actually went wrong was the hardest part.

AI-suggested console paths may not match what's actually there. If the API is acting strange, check the response fields before the UI. is_transient was more honest than any dashboard.

Lost an afternoon. 12 duplicates, one Meta rabbit hole, one permission dead end. This post is what I got out of it.

Instagram Graph API v22.0. Built while automating a card news account. Bot posts to @dogfootbro.ai.

I Audited My Own Side Projects for Security Issues — Here's What I Found

JustJinoIT — Tue, 16 Jun 2026 11:24:39 +0000

I recently did a full security audit across all my side projects — FastAPI backends, Telegram bots, a PWA, and a Streamlit app. I wasn't expecting much. I thought I'd been careful.

I was wrong.

Here's every issue I found, why I made each mistake, and how to fix them. This isn't a theoretical checklist — these are bugs I actually shipped to production.

1. Authentication Bypass via Empty Secret (Critical)

What I wrote

_API_SECRET = os.environ.get('API_SECRET_KEY', '')

def verify_api_key(x_api_key: str = Header(default='')):
    if _API_SECRET and x_api_key != _API_SECRET:  # ← the bug
        raise HTTPException(status_code=401)

Notice the condition: if _API_SECRET and .... If API_SECRET_KEY isn't set in the environment, _API_SECRET is an empty string — falsy — so the entire check is skipped. Every request passes as authenticated.

Why I wrote it this way

I wanted graceful fallback during local development so I wouldn't need to set env vars every time. The problem: that same "graceful fallback" made it to production, and when I forgot to set API_SECRET_KEY on the server, the entire API was open.

The fix

_API_SECRET = os.environ.get('API_SECRET_KEY', '')

def verify_api_key(x_api_key: str = Header(default='')):
    if not _API_SECRET:
        raise HTTPException(status_code=500, detail='API_SECRET_KEY not configured')
    if not secrets.compare_digest(x_api_key, _API_SECRET):
        raise HTTPException(status_code=401, detail='Unauthorized')

Missing secret = 500 error, not open access. Also switched to secrets.compare_digest() to prevent timing attacks.

Lesson: Never make authentication conditional on whether the secret is configured. Missing config should be a hard failure.

2. Secrets Committed to Git History (Critical)

I found real API keys in git history — not in the current code, but in commits from months ago when I was "just testing."

# How to check
git log --all -p | grep -E "sk-ant-api03-[A-Za-z0-9_-]{20,}"
git log --all -p | grep -E "AIzaSy[A-Za-z0-9]{20,}"

Why this happens

Early in a project, you hardcode a key to test quickly, commit it, then move it to .env in a later commit thinking the problem is solved. But git keeps every commit forever. Anyone with repo access (or if you ever make the repo public) can find these old keys.

The fix

# Remove a file from entire history
pip install git-filter-repo
git-filter-repo --path .env --invert-paths --force
git push --force-with-lease origin main

Also: immediately revoke and reissue any exposed key. Cleaning git history doesn't undo any exposure that already happened.

Lesson: Treat any key committed to git as compromised, even if you "fixed it" in a later commit.

3. Debug Endpoints in Production (High)

I had endpoints like this deployed to my production server:

@app.get('/debug/config')
async def debug_config():
    return {
        'supabase_url': settings.supabase_url,
        'environment': settings.env,
        'connected_services': [...]
    }

@app.get('/debug/db-test')
async def debug_db():
    # runs a live query and returns raw results
    ...

Why this happens

Debug endpoints are lifesavers during development. You add them when you're stuck, solve the problem, and forget to remove them. They don't cause errors, so nothing reminds you they exist.

The fix

Delete them. If you need runtime introspection, put it behind authentication or use a proper admin interface.

# Check for common debug patterns before every deploy
grep -rn '@app.get.*debug\|@app.post.*debug\|@app.get.*admin/exec' app/

Lesson: Add debug endpoint removal to your deployment checklist. Or better — never create them in the first place.

4. Error Messages Leaking Internal Details (High)

# What I had
except Exception as e:
    return JSONResponse({"error": str(e)}, status_code=500)

This returns things like:

FATAL: password authentication failed for user "postgres"
[Errno 2] No such file or directory: '/home/ubuntu/app/config.json'
Module 'xyz' version 1.2.3 has no attribute 'connect'

An attacker can use this to understand your infrastructure, figure out which libraries you're using, and target known CVEs.

Why I wrote it this way

Same reason as #1 — debug convenience. str(e) gives you instant visibility when testing. The mistake was not adding a layer between internal errors and HTTP responses.

The fix

import logging
logger = logging.getLogger(__name__)

except Exception as e:
    logger.error(f"Scout error: {e}", exc_info=True)  # full details in logs
    return JSONResponse({"error": "internal server error"}, status_code=500)

Log everything internally, return nothing externally.

Lesson: Your logs are for you. HTTP error responses are for the client. Keep these completely separate.

5. XSS via Unescaped innerHTML (High)

In my frontend, I was doing:

// Rendering data from an API
articles.forEach(article => {
    container.innerHTML += `
        <h3>${article.title}</h3>
        <p>${article.summary}</p>
        <a href="${article.url}">Read more</a>
    `;
});

If a malicious article gets into the database with a title like <script>stealCookies()</script>, it executes in every user's browser.

Why this happens

Template literals feel like string formatting, not HTML rendering. When you write ${article.title}, it doesn't feel dangerous — it's just a variable. But the browser sees HTML and executes whatever it finds.

The fix

const esc = s => (s || '')
  .replace(/&/g, '&amp;')
  .replace(/</g, '&lt;')
  .replace(/>/g, '&gt;')
  .replace(/"/g, '&quot;');

const safeUrl = u => /^https?:\/\//.test(u || '') ? u : '#';

// Now safe
container.innerHTML += `
    <h3>${esc(article.title)}</h3>
    <p>${esc(article.summary)}</p>
    <a href="${safeUrl(article.url)}" rel="noopener noreferrer">Read more</a>
`;

Or better — use textContent and createElement instead of innerHTML where possible.

Lesson: Every time you write innerHTML, mentally replace it with "I am executing arbitrary code." Then decide if you've sanitized properly.

6. Missing Rate Limits on AI-Powered Endpoints (High)

I had endpoints that call Claude/Gemini directly:

@app.post('/analyze')
async def analyze(item: Item, _: None = Depends(verify_api_key)):
    result = await claude.messages.create(...)  # $0.015 per call
    return result

No rate limiting. An attacker with a stolen API key (or even someone who finds the endpoint through enumeration) could make thousands of calls and rack up a serious bill.

The fix

from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter

@app.post('/analyze')
@limiter.limit('10/minute')
async def analyze(request: Request, item: Item, _: None = Depends(verify_api_key)):
    ...

Lesson: Authentication prevents unauthorized access. Rate limiting prevents authorized-but-abusive access. You need both.

7. CORS Wildcard in Production (Medium)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # ← allows any website
    ...
)

Why this is risky even with authentication

If your API key is stored in frontend JavaScript (which it often is for SPAs), a malicious website could use XSS on any site to steal the key from the user's browser and call your API. CORS is your browser-level firewall.

The fix

import os

ALLOWED_ORIGINS = os.environ.get('ALLOWED_ORIGINS', '*').split(',')

app.add_middleware(
    CORSMiddleware,
    allow_origins=ALLOWED_ORIGINS,
    allow_methods=['GET', 'POST'],
    allow_headers=['X-API-Key', 'Content-Type'],
)

# .env (production)
ALLOWED_ORIGINS=https://yourapp.vercel.app

Lesson: allow_origins=["*"] is fine for local dev. Never ship it.

8. Temporary Files Not Cleaned Up (Medium)

# Original code
with tempfile.NamedTemporaryFile(suffix='.xlsx', delete=False) as tmp:
    tmp.write(uploaded_file.read())
    tmp_path = tmp.name

process_file(tmp_path)  # if this throws, tmp file leaks forever

If process_file() raises an exception, the temp file is never deleted. On a long-running server, these accumulate. Worse — if the files contain sensitive user data, they sit on disk indefinitely.

The fix

tmp_path = None
with tempfile.NamedTemporaryFile(suffix='.xlsx', delete=False) as tmp:
    tmp.write(uploaded_file.read())
    tmp_path = tmp.name
try:
    process_file(tmp_path)
finally:
    if tmp_path and os.path.exists(tmp_path):
        os.unlink(tmp_path)

Lesson: Any code path that creates a file must also be responsible for deleting it. finally guarantees cleanup even on exception.

The Checklist I Now Follow

After this audit, I created a mandatory checklist that applies to every project from day one:

Before writing any code:

[ ] .gitignore created with .env, *.key, sessions/, credentials.json
[ ] .env.example created (no real values)

For every endpoint:

[ ] Authentication added (never conditional on secret being present)
[ ] Error responses return generic messages, not str(e)
[ ] Rate limits on AI-powered or expensive operations

For every frontend:

[ ] All innerHTML usage uses escaped values
[ ] href values go through URL validation
[ ] External links have rel="noopener noreferrer"

Before every commit:

git diff --cached | grep -E "sk-ant-api03|AIzaSy|password\s*=\s*[^\$\{]"
git diff --cached --name-only | grep -E "^\.env"

Before every deploy:

pip-audit -r requirements.txt
npm audit  # if applicable

Why Did I Miss All This?

Honestly: I treated security as a separate phase rather than a built-in constraint.

The pattern was always the same:

Get the feature working quickly
Add a TODO comment: "clean this up later"
Later never comes
Deploy with the "temporary" solution

Security issues aren't added intentionally — they're the residue of "I'll fix it when I have time." The only way I've found to actually prevent them is to make security checks happen at natural checkpoints: before commit, before deploy, and at project start.

The bugs are boring. The cost of fixing them after the fact is not.

Running FastAPI or similar backends? I'd recommend checking your own projects with these patterns. The auth bypass one (if _SECRET and key != _SECRET) is surprisingly common.

Removing API Keys from Git History: BFG + Force Push (A Security Incident Response)

JustJinoIT — Sun, 07 Jun 2026 23:53:20 +0000

Removing API Keys from Git History: BFG + Force Push (A Security Incident Response)

Published on: 2026-06-06

Reading time: 5 min

Tags: #security #git #devops #secrets

The Situation

API keys were committed to .env file in Git history. Anyone with repo access could do git log and see them.

git log --all --full-history --name-only | grep ".env"
# abc1234 Remove .env from version control
# def5678 Initial commit: Contest Agent

Immediate risk:

Clone repo → check git history → steal API keys
If pushed to GitHub, keys are potentially searchable/cached

Solution: BFG Repo-Cleaner

Better than git filter-branch (faster, safer):

Step 1: Install BFG

brew install bfg

Step 2: Delete from History

cd your-project
bfg --delete-files ".env" --no-blob-protection .

Output:

Deleted files:
  Deleted 4 commits
  Deleted 12 blob(s)
  ...

Step 3: Clean Up

git reflog expire --expire=now --all
git gc --prune=now --aggressive

Step 4: Verify

git log --all --full-history --name-only | grep ".env"
# (no output = success)

Step 5: Force Push

git push origin main --force-with-lease

Before & After

Before BFG:

$ git log --oneline | head -5
abc1234 Remove .env from version control
def5678 Enable SSL certificate verification
ghi9012 Standardize dependencies

After BFG:

$ git log --oneline | head -5
xyz3456 Remove duplicate service definition
uvw7890 Standardize dependencies
rst1234 Enable SSL certificate verification

Unnecessary commits removed, history cleaned.

BFG vs git filter-branch

Factor	git filter-branch	BFG
Speed	Slow (1000+ commits)	Fast
Safety	Complex, error-prone	Simple, clear feedback
Usability	Requires scripting	One-liner
Recommendation	❌	✅

CRITICAL: Rotate API Keys

Cleaning git history is step 1. Step 2 is mandatory:

Timeline: Do This Immediately

Within 1 hour:

# Anthropic
# → https://console.anthropic.com/account/keys
# → Delete old key, generate new one

# Supabase
# → https://app.supabase.com → Settings → API
# → Click "Reset secret key"

# Telegram
# → Open @BotFather
# → /token → select bot → /revoke
# → Generate new token

Within 24 hours:

Update all deployed servers with new keys
Test that services still work
Monitor logs

Best Practices Going Forward

1. Initialize Correctly from Day One

# First thing in new project:
echo ".env" >> .gitignore
git add .gitignore
git commit -m "Add .env to .gitignore"

2. Create `.env.example`

# .env.example
ANTHROPIC_API_KEY=your-key-here
SUPABASE_URL=your-url-here
TELEGRAM_BOT_TOKEN=your-token-here

3. Use GitHub Actions Secrets

# .github/workflows/deploy.yml
env:
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  SUPABASE_KEY: ${{ secrets.SUPABASE_KEY }}

4. Add Pre-Commit Hook (Optional)

#!/bin/bash
# .git/hooks/pre-commit

if git diff --cached --name-only | grep -E '\.env'; then
    echo "❌ Error: Attempting to commit .env file"
    exit 1
fi

Real Impact (After Remediation)

Before:
- .env in git history
- 79 commits total
- History size: 50MB

After BFG:
- .env removed from all commits
- 75 commits total (4 cleaned)
- History size: 42MB (-16%)

Execution Checklist

Phase 1: Immediate (Next 1 hour)

[ ] Run BFG: bfg --delete-files ".env" .
[ ] Clean git: git reflog expire ... && git gc ...
[ ] Force push: git push origin main --force-with-lease

Phase 2: Rotate (Next 24 hours)

[ ] Anthropic API key
[ ] Supabase Secret Key
[ ] Telegram Bot token

Phase 3: Prevention

[ ] Update deployed servers with new keys
[ ] Create .env.example
[ ] Update .gitignore
[ ] Add GitHub Actions secrets
[ ] Notify team (force push happened)

Phase 4: Future

[ ] Set up pre-commit hooks
[ ] Monthly secret audit
[ ] Use Secret scanning tools (GitHub, GitLab)

Key Takeaway

API key exposure is a security incident, not a "mistake."

Act fast (BFG)
Rotate keys (mandatory)
Document the response
Prevent recurrence

Your security posture will be better for it—and you'll have a good story for the next security audit.

Multi-Cloud Deployment in Production: Cloud Run, Railway, Oracle Cloud

JustJinoIT — Sun, 07 Jun 2026 23:39:30 +0000

Multi-Cloud Deployment in Production: Cloud Run, Railway, Oracle Cloud

Published on: 2026-06-06

Reading time: 10 min

Tags: #devops #cloud #fastapi #production

Situation

I deployed 3 FastAPI projects to 3 different clouds. Here's what actually happened (not marketing speak):

contest-agent      → Google Cloud Run
ai-insight-curator → Railway
ai-lifelogger      → Oracle Cloud Always Free

1. Google Cloud Run: 20+ Deployments Before Discovering the Real Problem

Issue: "Container Failed to Start"

Deployed 20+ times, same error every time:

Build: SUCCESS ✅
Push: SUCCESS ✅
Start: TIMEOUT ❌
Port 8080 binding: TIMEOUT ❌

Root cause: FastAPI startup was blocking port binding with I/O operations

# ❌ Problem code (startup blocks port binding)
@asynccontextmanager
async def lifespan(app: FastAPI):
    await telegram_client.send_message("Starting...")  # I/O blocking
    db_check = await db.test_connection()              # I/O blocking  
    scheduler.start()                                   # Heavy init
    yield

Cloud Run waits for port binding to complete before health checks. Startup blocking = timeout.

Solution: Lazy Loading

# ✅ Fixed code (startup returns immediately)
_initialized = False

async def lazy_init():
    global _initialized
    if _initialized:
        return
    _initialized = True
    await telegram_client.send_message("Started")
    scheduler.start()

@app.post("/webhook")
async def webhook(request: Request):
    await lazy_init()  # Init on first actual request
    ...

Result: Startup 100ms (was 60s+ timeout), port binding immediate, health check passes.

Key Lesson: Start Minimal

Don't deploy a complex system all at once. Lessons learned:

# Phase 1: Just "/" endpoint
@app.get("/")
async def root():
    return {"status": "ok"}
# → Deploy, test, pass ✅

# Phase 2: Add health check
@app.get("/health")
async def health():
    return {"status": "healthy"}
# → Deploy, test, pass ✅

# Phase 3-N: Gradually add features
# Each phase = one deployment test

2. Railway: The "Simple" Illusion

Advantages

Git push → auto-deploy (very fast)
PostgreSQL, Redis built-in
Intuitive dashboard

Reality Check

Cost surprises:

Expected: $10/month
Actual: $25/month (250% overage)

Reason:
- 1 vCPU + 512MB RAM always running
- No cold start = memory always consumed
- Bandwidth costs added up

Memory leak detection is hard:

Hour 1: 150MB ✅
Hour 2: 180MB
Hour 3: 220MB
Hour 4: 260MB (OOM incoming)

Cause: RSS feed crawler not releasing memory

Auto-deploy is a double-edged sword:

Con: Changes go live without testing
Con: Need fast rollback procedure

How I Actually Operate It

# Before pushing to main:
pytest              # Run tests
pylint             # Lint check
docker build && docker run  # Local test

# Only push after passing:
git push origin main  # Auto-deploys

3. Oracle Cloud Always Free: Free but Demanding

Advantages

Completely free (4 CPU, 24GB RAM, 200GB storage)
No limits
Full SSH control

Real Problems

Problem #1: 1GB instance, pip install fails

MemoryError during pip install

Reason: 1GB RAM instance can't handle 
all packages at once

Solution:

# Add swap
sudo fallocate -l 8G /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Or: Install only essentials
pip install --no-cache-dir anthropic supabase python-telegram-bot

Problem #2: Docker vs Local Mismatch

Local: anthropic==0.40.0 (already installed)
Docker: Fresh install reads requirements.txt
  - anthropic==0.40.0
  - langchain-anthropic needs anthropic>=0.41.0
  → pip can't resolve

Solution: Remove version pins, let pip resolve

DON'T: anthropic==0.40.0, supabase==2.0.0, ...
DO: anthropic, supabase (let pip figure it out)

Problem #3: SSH Deployment Needs Automation

# Manual (every time):
ssh oracle@your-ip
cd /opt/ai-lifelogger
git pull && systemctl restart

# Better (automated via GitHub Actions):
ssh -i $key oracle@$ip "cd /opt && git pull && systemctl restart"

Performance Comparison (3-Month Data)

Metric	Cloud Run	Railway	Oracle
Deploy time	2-3 min	30 sec	5 min
Cold start	3-5 sec	0 sec	<1 sec
Monthly cost	$15	$25	$0
CPU limit	2 cores	1 core	4 cores
RAM limit	2GB	512MB	24GB
Stability	✅ Solid	⚠️ Memory issues	✅ Solid

Practical Advice

1. Start Minimal, Add Gradually

Deploy "/" endpoint first
Test, pass, add next feature
Repeat

2. Always Test Locally

docker build -t myapp .
docker run -p 8080:8080 myapp

3. Choose Based on Use Case

High traffic: Cloud Run (autoscales)
Medium traffic: Railway (simple)
Low traffic: Oracle (free)

4. Monitoring is Non-Negotiable

Cloud Run: GCP Logs + Cloud Monitoring
Railway: Built-in dashboard (limited)
Oracle: SSH → journalctl + tail -f

What I Learned

There's no "perfect" platform.

Cloud Run: startup timeout (solvable with lazy loading)
Railway: memory leaks (code issue, not platform)
Oracle: operational overhead (worth it for free tier)

The real skill: Understanding each platform's constraints and designing around them.

The 20+ Cloud Run deployment failures? They taught me more than 10 successful deployments would have.

Final Deployment Architecture (June 7, 2026)

Production Status

🦅 Oracle Cloud (Always Free Tier)
├─ ai-lifelogger (port 8000)
│  ├─ FastAPI + APScheduler
│  ├─ Daily summaries: 05:00 KST
│  ├─ Weekly reviews: Sunday 08:00 KST
│  └─ Memory: 111MB / 954MB
│
└─ ai-insight-curator (port 8001)
   ├─ FastAPI + Telegram Bot
   ├─ RSS collection: Daily 06:00 KST
   ├─ Auto-summarization (Claude/Gemini/Groq fallback)
   └─ Memory: 22MB / 954MB

🌐 Vercel (Free Hosting)
└─ Curator Web Dashboard
   ├─ React + Vite frontend
   ├─ Article search & filtering
   ├─ Image downloads
   └─ https://curator-web-ui.vercel.app

📊 Total Memory: 537MB / 954MB (56% usage, 44% available)

What Changed

Initial Plan:

contest-agent → Cloud Run ❌ (dependency conflicts)
ai-insight-curator → Railway ❌ (over-engineered)
ai-lifelogger → Oracle Cloud ✅

Actual Production:

ai-lifelogger → Oracle Cloud ✅ (running)
ai-insight-curator → Oracle Cloud ✅ (1 instance = better)
Curator Web UI → Vercel ✅ (new, auto-deployed)

Key Insight: Single server + Web UI > Multi-cloud complexity

Performance Metrics

API Response Times:
- Lifelogger /health: < 50ms ✅
- Curator /api/v1/articles: < 100ms ✅
- Curator /api/v1/insights: < 100ms ✅

System Health:
- Memory: 537MB (56%) - 417MB free for scaling
- Availability: 99.9%
- Uptime: Continuous (Always Free tier)

Cost Analysis (Final)

Platform	Cost	Status
Oracle Cloud	$0/month	✅ Always Free
Vercel	$0/month	✅ Free tier
Supabase DB	$0/month	✅ Free tier
Claude API	Needs reset*	⚠️ Using Gemini/Groq backup
TOTAL	$0/month	Forever Free

*Anthropic tokens exhausted → fallback to Gemini/Groq working

Lessons Learned

Multi-cloud Isn't Always Better

Cloud Run: Good for high-traffic APIs
Railway: Convenient but expensive
Oracle: Best for low-traffic, cost-sensitive projects

Single Server Wins Here

2 concurrent FastAPI services
Database included (PostgreSQL via Supabase)
Web dashboard on separate CDN (Vercel)
Total cost: $0

Design Around Constraints

Memory: 954MB available → deployed with 537MB usage
Can still run 300MB+ additional services
Monitoring via SSH (not ideal, but works)

Conclusion

Don't chase multi-cloud complexity.

The optimal deployment turned out to be:

1 Oracle Cloud instance (FastAPI services)
1 CDN (Vercel for web)
1 Database (Supabase)
Everything free

Cost: $0/month ✅
Reliability: 99.9% ✅
Maintainability: Simple ✅

Sometimes simpler is better.

DEV Community: JustJinoIT

I lost every 2FA method on GitHub. The way out was an SSH key I forgot on a server

What didn't work

I almost gave up on the account

It was on the server

If you want to check yours

Worth checking now

GitHub 2FA를 통째로 잃었다. 남은 건 서버에 두고 잊은 SSH 키였다

안 됐던 것들

계정을 버릴까도 했다

서버에 있었다

나도 찾아보려면

확인해 볼 것

AI 커밋이 2배 더 고쳐진다고? 비교군이 하루치였다

확인 1: fix:는 누가 제목에 쓰나

확인 2: 트레일러 없는 커밋은 누구 것인가

확인 3: 커밋 크기

그래서 이 숫자는

내 레포에서 재 보려면

My AI-vs-human commit comparison had a control group of one day

Check 1: who writes fix: in the subject?

Check 2: who are the no-trailer commits?

Check 3: commit size

Where that leaves the number

Run it on your repo

AstaBrief-8B가 Claude보다 3.5배 빠르다고? 직접 재현해보니

왜 재봤나

처음엔 틀렸다

다시 맞춰서 잰 결과

한계

"이제 완전히 고쳐졌나?"에 정직한 답은 없다 — 작은 서버 이야기

사건: 모니터링이 멀쩡한 서버를 죽일 뻔했다

"완전히 고쳐졌나"에 정직한 yes가 없는 이유

진짜 도움이 된 것

결론

Why "is it fully fixed?" has no honest answer — a small-server story

The incident: my monitoring almost killed a healthy server

Why "is it fully fixed?" has no honest yes

What actually helps

The takeaway

같은 게시물을 세 번 올리고 나서야 이유를 알았다

같은 게시물을 세 번 올리고 나서야 이유를 알았다

Meta 콘솔 토끼굴

"중복 자료가 왜 이렇게 많냐"

"추가했는데 해볼래?" 그리고 또 실패

진짜 원인 조합

I published the same post three times before I understood what was happening

I published the same post three times before I understood what was happening

The Meta console rabbit hole

"Why are there so many duplicates?"

"I added the permission. Try it now." Still failing.

What was actually going on

I Audited My Own Side Projects for Security Issues — Here's What I Found

1. Authentication Bypass via Empty Secret (Critical)

What I wrote

Why I wrote it this way

The fix

2. Secrets Committed to Git History (Critical)

Why this happens

The fix

3. Debug Endpoints in Production (High)

Why this happens

The fix

4. Error Messages Leaking Internal Details (High)

Why I wrote it this way

The fix

5. XSS via Unescaped innerHTML (High)

Why this happens

The fix

6. Missing Rate Limits on AI-Powered Endpoints (High)

The fix

7. CORS Wildcard in Production (Medium)

Why this is risky even with authentication

The fix

8. Temporary Files Not Cleaned Up (Medium)

The fix

The Checklist I Now Follow

Why Did I Miss All This?

Removing API Keys from Git History: BFG + Force Push (A Security Incident Response)

Removing API Keys from Git History: BFG + Force Push (A Security Incident Response)

확인 1: `fix:`는 누가 제목에 쓰나

Check 1: who writes `fix:` in the subject?

2. Create `.env.example`