First of all, there are rules to obey; if you don't want to get 429 or banned.
https://github.com/reddit-archive/reddit/wiki/API
I am using Node.js, where parallel async is easy; but the general rules should be applicable to any programming languages.
I am playing with Reddit while my Twitter requires official allowing...
Getting the OAuth token
You will need client ID and client secret from https://www.reddit.com/prefs/apps
- Client ID is below "Personal-use script".
- Secret is red underlined.
import axios from "axios";
import fs from "fs";
async function getToken(): Promise<{
access_token: string;
}> {
return axios
.post(
"https://www.reddit.com/api/v1/access_token",
"grant_type=client_credentials",
{
headers: {
Authorization: `Basic ${Buffer.from(
`${process.env.REDDIT_CLIENT_ID}:${process.env.REDDIT_CLIENT_SECRET}`
).toString("base64")}`,
"Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
},
params: {
scope: "read",
},
}
)
.then((r) => r.data);
}
if (require.main === module) {
getToken().then((data) =>
fs.writeFileSync("token.json", JSON.stringify(data))
);
}
The real access token is in data.access_token
.
Exploring the API
I recommend exploring the API on Postman. I feel it is more convenient than Insomnia, or just cURL with Terminal.
You don't have to login, to the website, nor the app. I feel that logging in is very annoying; but I can't find an alternative.
Another way to test, is go to any Reddit suburls on Firefox, and replace www.reddit.com
with api.reddit.com
.
Real scraping, while avoiding the troubles.
import axios from "axios";
import rateLimit from "axios-rate-limit";
const api = rateLimit(
axios.create({
baseURL: "https://oauth.reddit.com",
headers: {
Authorization: `Bearer ${
JSON.parse(fs.readFileSync("token.json", "utf-8")).access_token
}`,
},
}),
{
/**
* Clients connecting via OAuth2 may make up to 60 requests per minute.
*/
maxRequests: 60,
}
);
Helper function
declare global {
interface Array<T> {
mapAsync<U>(
callbackfn: (value: T, index: number, array: T[]) => Promise<U>,
thisArg?: any
): Promise<U[]>;
}
}
Array.prototype.mapAsync = async function (callbackfn, thisArg) {
return Promise.all(this.map(callbackfn, thisArg));
};
function dotProp<R>(o: any, p: string | string[], def?: R): R {
if (typeof o === "undefined") {
return def!;
}
const ps = typeof p === "string" ? p.split(".") : p;
if (!ps.length) {
return o;
}
if (o && typeof o === "object") {
if (Array.isArray(o)) {
return dotProp(o[parseInt(ps[0])], ps.slice(1), def);
}
return dotProp(o[ps[0]], ps.slice(1), def);
}
return def!;
}
Make use of Async Iterator
Of course, you can also send 1010 requests at once, but that would not only make unpredictable response time, but also get blocked.
function iterListing(apiPath = "/hot", count = 1000) {
const limit = 50;
const maxDepth = Math.ceil(count / limit);
return {
[Symbol.asyncIterator]() {
return {
depth: 0,
after: "",
async next() {
if (!this.after && this.depth) {
return { done: true };
}
if (this.depth < maxDepth) {
this.depth++;
const value = await api
.get(apiPath, {
params: {
after: this.after,
limit,
},
})
.then((r) => {
this.after = dotProp<string>(r, "data.data.after");
console.log(this.depth, this.after);
return dotProp<any[]>(r, "data.data.children", []).mapAsync(
async ({ data: { name } }) => {
return api
.get("/comments/" + name.split("_")[1])
.then((r) => {
const getComment = ({ data: { body = "", replies } }) =>
body +
"\n" +
(replies
? dotProp<any[]>(replies, "data.children")
.map((r) => getComment(r))
.join("\n")
: "");
return `${dotProp(
r,
"data.0.data.children.0.data.title",
""
)}\n${dotProp(
r,
"data.0.data.children.0.data.selftext",
""
)}\n${dotProp<any[]>(r, "data.1.data.children", [])
.map((r) => getComment(r))
.join("\n")}`;
});
}
);
});
return {
done: false,
value,
};
}
return {
done: true,
};
},
};
},
};
}
Don't write everything to file at once in Node.js
Learn to use stream. Stream is a very powerful concept in Node.js.
async function main() {
const outStream = fs.createWriteStream("raw/reddit.txt", {
encoding: "utf-8",
});
try {
for await (const out of iterListing()) {
if (out) {
out.map((it) => outStream.write(it + "\n"));
}
}
} catch (e) {
console.error(e.response || e);
}
outStream.close();
}
if (require.main === module) {
main();
}
Top comments (1)
Genius you are ! 🎖️