DEV Community: ImTheDeveloper

Paying Respect To Those Sites We Loved & Lost

ImTheDeveloper — Thu, 18 May 2023 05:00:24 +0000

Whilst posting my journey into development and architecture it brought me back to the mid 90s with my first exposure to the internet through AOL and such products.

Along the way I discovered some great websites that got me into the basics of HTML and ultimately sold my sould to JavaScript.

This morning I stumbled upon a website I was for sure convinced would be no longer with us:

http://dynamicdrive.com/

"DHTML" was a common search term to be able to find cool scripts I could put on my bolt.com profile and edit up websites on Geocities.

That said unfortunately it seems the latter is no longer with us, so please feel free to post some links I can run through the web archive and bring back the memories!

Dynamicdrive.com - one of the biggest libraries of random scripts. If you wanted snow flakes or a mouse cursor to shake and spam alert messages it could be found here!

Geocities.com - One of the earliest communities and free page hosting I can remember. Everyone had a page and became a hub for paid advertising and bios for people's pets

Homestead.com - Website page builder and where I built my first unofficial fan page for a singer. Drag and drop (a little bit) page building outside of frontpage and Dreamweaver whilst being online was a little mind blowing!

Where do you hire from?

ImTheDeveloper — Fri, 02 Apr 2021 08:01:12 +0000

A post to maybe relieve a little frustration but also to collate some good resources.

I'm working with a startup who are looking for:

Logo design
Branding
Help with setting up their e-commerce store (Bigcommerce)

The typical avenues so far of:

UpWork
Freelancer

Have both turned out to be quite hit or miss from my own experiences and what is seen as "brand design" or "logo design" appears to just be an exercise in who can use a logo generator or canva the best to spam listings until the customer succumbs into just picking whatever looks OK rather than something they actually like.

So a little call for help! Where do you usually go to for your own businesses and projects?

A design agency may break budgets at this stage but I can also appreciate the difference between doing something right and not.

ORM vs Query Builders vs Raw SQL

ImTheDeveloper — Tue, 17 Dec 2019 19:06:22 +0000

I've recently been working on an dated side project which is built in node.js coupled with an ORM querying against Postgres.

Since picking this back up, I'm starting to reach the point where ORM is getting in the way and I'm now reaching back to plain SQL statements against my DB.

This got me thinking, do I really need an ORM anymore? What are the benefits I aimed to achieve at the outset of the project and do they still seem relevant now?

I took some time to see how the sentiment looked on google and found a few (some quite recent) blog posts suggesting ORMs are to be avoided, ditch the ORMs and get back to basics:

https://blog.logrocket.com/why-you-should-avoid-orms-with-examples-in-node-js-e0baab73fa5/

https://eli.thegreenplace.net/2019/to-orm-or-not-to-orm/

Since I build in Node.js I then ended up looking into the world of SQL query builders, namely Knex.js which a few of the ORMs out there also use. Hmm ok, now query builders do I really need this in my life?

End of the day, all of this has me thinking. I'm proficient and happy to write raw SQL. I find ORMs provide good structure whilst prototyping early on and adjusting the model as I go, but this may just all be an illusion, since I equally would have little issue doing this work directly in my DB and all things said I would probably push more of the work and design correctly into my DB rather than polluting my application with such decisions.

I'd like to get some views from the Dev community:

Are ORMs still relevant?

If deemed that SQL queries in your application are fine, I'd be interested to understand whether you bundle these into a service / db access layer in your application or you find yourself peppering them across files.

Have you found benefit in using a query builder such as the features offered by Knex?

Do you feel as though you are using an ORM knowing you are going to be breaking out into native SQL at some point? In which case, is that OK with you?

Optimising Your Telegram Bot Response Times

ImTheDeveloper — Thu, 24 Oct 2019 14:18:00 +0000

Photo by Sonja Langford on Unsplash

Whilst the Modr8 bot platform is a SaaS package, requiring no code knowledge, we do want to share some optimisation techniques we have found to be crucial to running any service against the Telegram bot API.

An unresponsive or slow bot, no matter its purpose can turn users off from engagement and ultimately reduce your usage. If you are coding your own bot you may find the following tips quite useful.

Optimisation 1: Server location

Without a doubt one of the single most effective solutions for improving the speed your bot processes messages and returns a response to users comes down to the server location where it is hosted. This becomes most apparent when using webhooks as you should receive events in real-time from Telegram without any polling delays.

Telegram offers two protocols for their services:

The Mobile Protocol aka MTProto: This is served from global locations and your connection is likely to be routed through the closest server providing excellent support for desktop, mobile and browser based applications to receive events in acceptable time.
The Bot API: This is served from one location, hosted in Amsterdam and is the single point for API requests and responses that the Telegram Bot utilises.

When Modr8 started out we noticed that certain spam messages were being sent to chats and then within a second or two would disappear as our moderation engine removed them. Whilst the overall requirements of filtering such messages were met, the small lag wasn’t satisfactory.

If we run a ping from multiple global locations on the Bot API end-point https://api.telegram.org we return the following results:

Global pings against the Telegram Bot API

Amsterdam is blazing fast with its pong response and as we move outside of Europe we begin to see a slow down.

Whilst 150–200ms pings may be acceptable for some applications, often you can be making multiple API calls with a Telegram Bot, especially one that is moderating a chat so this soon bulks out to be a noticeable lag.

Modr8 began it’s life hosted in San Franciso and as you can see 150ms became a bottleneck when processing hundreds of messages a second. We decided to relocate closer to Amsterdam giving us on avg 12ms response times.

Optimisation 2: Cache Expensive Calls

Telegram has not published any hard rules on its API limits, but we have seen many open source/single token community bots that hit API limits quite easily as they are being used by many groups all compounding the number of messages to be processed and API calls.

Modr8 took a different approach, you bring your own token which alleviates the risk of hitting API limits with a shared bot. However, that being said there are still optimisations that can be made.

Certain calls such as getChatAdministrators, working with Files, getting member counts and restricting users can be expensive to continually call, especially those that require multiple in quick succession to produce the desired outcome. We found early that the behaviour of Telegram communities often means you are able to cache results in memory with an expiry time reducing the number of hits against the API.

A good example of this is caching the chat administrators. Assessing the permissions of a user executing commands and running messages through filters to display admin lists could result in numerous API calls, however we do this once and cache the results within Redis. Each chat has their admins cached with a TTL (time to live) expiry, if the results expired we call the API again and re-seed the cache.

Optimisation 3: Webhooks vs Polling

Whilst its easy to switch to a polling technique for pulling updates from the Bot API it is also a process which by its very nature reduces the responsiveness of your bot to the end user. Between each gap in requesting the getUpdates call your bot has dead time and the user is waiting.

Polling is an ideal setup for development and rapid prototyping, but if you start to receive heavier traffic you will want to switch to Webhooks. Bear in mind, this also reduces your calls per second to the API so you are getting back a few extra calls before reaching API limits.

Webhooks can be used to send and receive messages, for ease you may wish to simply receive webhook events and process the message and use the typical http request back to the API to offer your response. The choice is yours hear but receiving real time updates via webhook can drastically reduce loads and also improve responsiveness.

Optimisation 4: File Handling

If you are serving files to your users you mayg notice that there is a delay during the upload and optimisation process before your bot then displays the file. This is especially apparent when waiting for images sent as weblinks.

Telegram is pretty clever and has a built in CDN, if you are able to upload or serve the files earlier, especially coupled with caching the file links provided back by Telegram as per Optimisation 3 you will find the experience much more snappy for users.

In general though, we have actually found the process of handling files via the Bot API to be pretty clunky and it’s something which we prefer to handle ourselves from an external CDN and simply link users where needed to those files. This has definitely improved overall experience and allows us to take control of expiry times and compression ourselves.

Conclusion

Whilst the suggested optimisations may be obvious to many developers, sometimes the simplest methods bear the greatest rewards. For Modr8, caching and server location has been a game changer and allowed our platform to focus on horizontal scaleability to handle loads rather than dealing with hard stop bottlenecks due to round trip lags.

If you are looking for an all in one Telegram group management platform providing spam prevention, analytics and user management sign up for our free tier licence where all the functionality is available for you to test out!

Modr8 — The whitelabel Telegram group management engine

ImTheDeveloper — Thu, 24 Oct 2019 09:41:50 +0000

Modr8 — The whitelabel Telegram group management engine

Introduction

This post was originally shared on medium but Dev.to has been my home for tech reading now for quite a while so I wanted to show off the product we've built in the meantime. This posts will offer an insight into how the whitelabel telegram community moderation engine was born. This is the first, in a series of posts which will explain how we work with Telegram communities and also monitor the advancement of spammers/scammers methods to develop our product.

Lets begin…

Inspiration

Whilst being involved in Crypto projects during the 2017 bullrun we noticed the amount of Telegram based communities exploded with often tens of thousands of users all chatting within project communities and groups. It didn’t take long before scammers and spam became a major route to users being exploited and communities turning into cheap forums for bot based advertising.

With a background in Telegram Bot development, even back in the early beta days of the first API release by Telegram.org we set about automating the removal of suspicious media, user bots and also providing moderation commands for admins of chats to manage their community.

In the early days, this started out as small project with an individual bot built to the custom requirements of each community who needed our help. Over time though, we realised that this process would not scale and often the reason for using our services were to have a “personalised” and “on brand” bot that looked as though the community/project had developed the bot itself, without advertising another brands services in their chat. We decided it was time to scale up and offer a platform where users could generate a bot token utilising the official Telegram Bot registration process and then register this within our platform where our engine would handle everything else. This would give the scale and also self service management that many communities need.

From here the brand was born, Modr8 the white-label community bot platform. We ensure you have all the features needed to manage telegram communities, without having to detract from your own brand presence.

What the platform does

Through direct customer feedback and identifying the shortcomings of many free or open source projects we have developed a bot engine which handles everything from simple user management right through to polls, referrals and analytics.

Why personalised bots?

There are tens of bots out there which handle community management, we are well aware. However, these bots all have one thing in common, they all utilise the same Bot token and therefore all call the Telegram API under the registered token and username. Countless times we have seen these bots become unresponsive as they simply can not handle the inbound processing of messages as well as then sending all the outbound responses to the Telegram API. In no time at all you are faced with a flood or API limitation being put against the token and the bot no longer receives messages. If you are in the game of preventing spam and managing your community using your bot, the system has now been rendered useless.

In order to reduce the chance of this happening to our platform, we specifically chose not to go in the direction of simply producing a web platform which allows the management of a single generic bot under a single token that we own. We want users to “bring their own tokens” ensuring they can do 2 things:

Personalise their bot profiles using @BotFather to look, feel and be controlled as they wish.
Provide unique tokens which means every message they process is only their own, vastly reducing the chance of hitting API limitations.

All sounds great.. but honestly what does it do?

A breakdown of core functionality is listed below:

Fully customisable bot profiles, your token so your bot profile.

User management commands such as bans, kicks, mutes and warnings

Custom command manager allowing you to develop your own commands with your own content. If you want to create a command to show a list of your staff, or a link to your website it can all be done here.

Chat & User Analytics including everything from message sentiment through to most popular commands and active users.

Native Polls allowing you to issue and track the responses from polls created in the platform and issued to your chat by the bot.

Announcements send one off or scheduled and interval based announcements to your chat

Spam prevention turn on and off granular rules to detect and remove spam from your channel.

Referrals & Reputation full modules that allow you to generate referral links for users to invite others to your chat, including full tracking and leaderboards. We also have a reputation system with custom rankings allowing users to say thanks and gain levels in the leaderboards.

API we are the only platform to offer an API which has been utilised by a number of communities already to link to their internal systems, including swag giveaways and gift stores.

Translations responses to messages are now being translated and within the platform settings you can choose from a number of languages really making the bot fit into your community.

… there’s much more too!

How its been built

Modr8 started as a humble single node.js file utilising one of the many Telegram bot API npm packages for message inbound interactions and executing outbound instructions. Each bot had a JSON settings file and it was very much a single project for a single community.

In order to scale, a number of changes had to be made.

Splitting apart the mono project into:

Rules engine: Keeping with Node.js here, but each engine deployed inside docker containers allowing the engines to scale horizontally to keep up with message consumption.

Message producer: The consumer is a simple express App which receives webhook events from the Telegram API, these are then fed directly into the message queue along with parsing the Bot token allowing for the consumers to identify the bot messages they are processing.

Message queue: We spent a lot of time deciding whether to go with Kafka, ActiveMQ, Redis, RabbitMQ or a simple memory based message queue. In the end we fell for RabbitMQ mainly due to us wanting our consumers being “dumb” and therefore not needing too much logic other than a connection to the queue. Kafka is still on our roadmap, but I don’t believe we have hit any limitations in RabbitMQ to need the move yet.

Scheduler: This component similar to the engine is concerned with working with scheduled events. Whilst the rules engine reacts to in the moment inbound events the scheduler is looking for those which require a delay or specific execution time. This is built in node and right now simply polls the database for required events. This is one of the areas we are looking to improve, but right now sufficiently meets requirements for welcome message timeouts, scheduled announcements and clean up of messages marked for future deletion.

Database: At first the project started out with MongoDB purely as it had the easiest migration path from the JSON settings we had in the mono projects before. It also helped us to reduce the workload on data-modelling whilst discovering requirements as the project built out. Mongo is great for rapid prototyping and working with a flexible schema, it definitely hit the sweet spot for us during such heavy changes in the early days. We have however now moved to Postgres, mainly due to the consistency and capacity to handle heavier read/write scenarios. There is some usage of JSON documents within Postgres for a few flexible elements of our model but the vast majority of the platform is pretty well defined now and data structures are known.

Cache: Without a doubt here we have always defaulted to Redis for our cache needs. At one point we were looking at Redis to also handle our pub/sub and queue needs however as Rabbit won that battle we still had Redis supporting caching so it features regardless. The main usage for redis is storing some state for the rules engines, but also caching many of the costly/time consuming API calls that may occur multiple times during the lifecycle of a message being processed. An example of this is the Admin list in the community groups. The admins rarely change, so calling the API every message to see if the user has access to a rule or command is costly. Infact, there are some very strict (but also largely unknown) API limitations which we could easily hit if the chat were particularly busy.

Websites: the main www domain is serving a static site hosted by netlify. We love that service so much it's an amazing product and also powers our contact form without any hassle. The main platform domain is built on the sails.js framework. The simple MVC architecture is all that we needed to get running and sails has a really easy project structure to follow.

Reverse Proxy: We balance workloads amongst the rule engines using rabbit work queues but still we needed a reverse proxy for the platform website and consumer. Since all of the components apart from our www site are in docker containers we found traefik to be a great fit. The auto discovery and easy management of rules has been a perfect fit. Another highly recommended product if you are looking to remove the burden of nginx and haproxy configuration files when working with containers.

Challenges

One of the biggest challenges faced has easily been the switch from single silo projects to the generic platform we have today. It’s one thing personalising a single bot with custom actions, but then moving to a platform model where processes are decoupled, split and also offer the same flexibility for users to make changes and “personalise” the bot experience is a large leap.

With that being said, the pay off is huge and it makes long term plans much easier to attain. As an example we are now translating the bot into various community languages offering a true personalised experience for that locale which has been a major hit with our user base.

Accomplishments

We feel that the biggest accomplishment is offering a service which has truly hit our brief, we wanted to offer something which others in the market were not and that is the elements of “whitelabel” “saas” and “personalised” community management bots. When we receive feedback from our own community it is always humbling to see they notice the effort placed on quick turn around and focus on usability. There’s a few competitor services which are clunky to use and navigate, we have tried to make everything simple for the users to understand and, with that we have seen our bots embedded deeply into the communities and the platform has grown in popularity by recommendation rather than hard marketing.

What we’ve learnt

There have been a number of learning points, some of them have been related to code, but most of them have been around listening to customer feedback. The product we have today has had huge influence from our customers who are open and honest around specific issues they have faced managing communities, especially those who have a large user base and may need cross chat management. This really took the platform from its humble beginnings as a 1 bot 1 chat product, to now being built for enterprise usage and we can happily say we have a number of customers effectively managing chats running into 100k membership across multiple chats.

What’s next for Modr8

We’ve just released chat referrals which allows a user to generate a unique link via the bot and invite others to the community. Surprisingly this has been way more popular than we expected and has already been picked up heavily by the Cryptocurrency projects, especially exchanges who already have many thousand referrals to their chats and run competitions. This took quite a lot of work but it’s been great to see the feedback from customers.

For the future we have a backlog of items suggested by our customers, some of these are minor tweaks but the big hitters are:

Public polls: Right now we offer native telegram polls via the bot but these are anonymous by default. We collect the responses and display a nice historical table of the results but it would be great if we could offer the same functionality as the @PollBot or @Vote. This is a relatively simple change and will be coming in the next release.

Support Features: A long term goal has been to offer a F&Q or support type of feature to the engine. This would allow a community (especially those who have a product behind them) to offer a two way chat support feature in the bot. Much like we see on websites with chat bots, we feel there is a play to be made here in telegram too.

External Feeds: In the early days we did offer extended announcement support which allowed platform users to add their twitter accounts, medium account handles so that any new content would also be posted within the chat. Changes to the twitter API and also the number of feeds to subscribe to pushed this down the list of features to migrate over from the Alpha version. It’s something we want to add in as we see many community admins cross posting content which we can automate for them.

Want to get started?

Its easy to get started with the platform, we offer a free tier of 100 members of less so you can try out all of the features in a small chat before activating a bigger community.

Feel free to check out our website https://www.modr8.net

If you wish you could jump directly to the platform here: https://platform.modr8.net

Our support chat is always open with users who know the system inside and out to offer help with any questions: https://t.me/modr8support

My app is built.. but Docker deployments have me confused. Help!

ImTheDeveloper — Sun, 16 Dec 2018 07:48:09 +0000

I'm looking for those with docker experience to help me wrap my head around moving from development and into production. I would normally throw this type of stuff onto stackoverflow but I find the feedback and insights from our dev.to community being really constructive and useful for others to learn too.

Current situation:

I'm a first-time docker user and I've been working on a side project which I've got running fine in containers and lifted with docker-compose in development as a full stack. Theres a producer app, rabbitmq worker queue, consumer app, caching database and a sql database with a website front end amounting to 6 images. I have registered these within my private gitlab instance and I have a semi-decent setup for keeping my images up to date after changes.

Whenever I dip into the docker documentation it throws out things such as deploying with swarm, compose and machine. I've spent a couple of pain staking days getting docker-machine installed on a private vps to allow remote management since docker docs seem to suggest this is the way to go, however this is where the problems have been endless.

So far docker-machine is completely horrific to work with using the generic driver. I've hit problem after problem and spent a great deal of my time digging through various fixes in their github issues list. I'm not overly confident in whether it's going to run smoothly when my application is eventually deployed so I wanted to double check with the great dev.to community as to the route you typically follow.

At the moment I'm fine running on a single VPS which should hopefully simplify things. In the future, there are workers in my architecture which pull from a worker queue and if I need to scale it would most likely be spinning up more workers to get through the tasks.

As soon as I look outside of the docker documentation I start to read things like.. rancher, kubernetes, portainer etc. I'm not completely sure whether these are relevant or even needed in my case so it would be great to hear some feedback on those too.

Actually just found this .. what a mine field https://landscape.cncf.io/

Storing user customisations and settings. How do you do it?

ImTheDeveloper — Sat, 10 Nov 2018 18:40:14 +0000

I've recently embarked on scaling up a side hustle which entails storing a large amount of user settings and customisations that they can make to a web platform. The product is essentially white label and the customer can change any number of settings to brand, open and close off many features. The level of customisation can even go down to the text that is displayed to their users.

Currently, I've been building these one off for customers to test the water (will someone pay for this?) and storing settings/customisations in JSON files. It's been quick and easy, but now I'm looking to scale to a platform where there could be many multiples of users as well as sharing amongst teams. In short, the flat JSON files aren't going to scale and I'm looking into storing settings/customisation in a database.

Whilst it would be easy for me to look at document stores such as MongoDB to mimic closely the functionality I already have with my JSON files, I've also been pushing some relational issues into those JSON files which really, I would rather have sat in a relational database. Without wanting to proliferate the amount of databases I have in my architecture (albeit if that's the answer then why not!) I have been thinking long and hard about the problem and the potential options.

After some deep thought. I came up with 4 scenarios. But first a little bit of information about how a setting can look:

A single text item, typically for application names and titles.
An array of items, usually used for storing a number of selected tags or items.
A string, that has a default value, but can be customised by a user. Typically used for naming items or message text that can be changed.
Numerics: Usually stored for min and max limits or definitive values in the application.
Booleans: Typically stored for features to be turned on / off

The important part I cant forget though, is that there are always defaults. If a user does not adjust a setting, then a default value should be used. This is especially important for the text content shown on screens as well as messages pushed out to users.

Now, the methods and scenarios I've come up with after some brainstorming to potentially manage these settings and customisations:

A single row of settings per customer

In this scenario using a relational database I could store all of the settings required for each users application in a single table "settings", with each setting taking up a column and each user taking up a row.

Whilst this would be the simplest approach to get started, I don't feel like it has longevity. There are problems with ongoing changes I might need to make as well as "default values" which some users might not have based on the old schema. I'd need to go through lots of processing to update users and ensure there are no conflicts. However, as far as querying to return a users settings, this offers real simplicity.

JSON Storage

In this scenario I could take my existing JSON and push it into the relational database such as Postgres for storage. This would reduce the overhead of needing 2 databases (no-sql vs relational) whilst also keeping my model close to what I have. The big benefit is I dont need to worry about my schema ahead of time and I can add and remove setting as I wish without causing too many problems for other users.

With that flexibility though, doing any kind of big updates across my data are going to be a pain and trying to work with the data any further than simple selects might bring with it some overhead.

A property bag / key value approach

In this scenario I could create a very simple table containing an id, user_id, key, value in my database and ensure I have the flexibility to store settings with ease, whilst also being able to query for a user_id and pull back the whole set in one go.

My worry with this approach is the adherence to standards, if any of the keys became inconsistent I can end up with a bit of a mess. I also believe this can lead to some really bloated out table in my database. Assuming each customer had in the range of 200 settings, this can quite easily become a million rows with just 5000 users. I shouldn't be concerned with query performance on a million rows I guess.

An alternative?

My final option, well.. I feel like in each of the above scenarios I've tried to mention relational databases. But I have to face facts, each option actually is flirting with a nosql equivalent. Key value stores such as redis could be an option, although persistence and decomposing some long values might be a bit too inflexible. Document store such as MongoDb/Couch would match closely to my current model, giving each user their own settings document and pulling it back nice and easy. Whilst that sounds like a nice scenario, it means I'd probably also need to spin up a relational database to handle the features of the platform which are clearly relational. Do I want the extra infrastructure to look after and handling queries to two sources now instead of just one? Tough choice.

I'm interest to hear how others have approached this issue and also whether there are deemed any best practices? Whilst I've been quite wooly on the specifics of my project, I feel this is something others must have run into before!

Scalable, Available, Stable, Performant, and Intelligent System Design Patterns - An awesome resource!

ImTheDeveloper — Sat, 20 Oct 2018 18:15:58 +0000

Whilst working through some designs to scale my side project I stumbled upon this amazing resource related to the following:

Scalability
Availability
Stability
Performance
Intelligence
Architecture

CHECK IT OUT HERE: https://github.com/binhnguyennus/awesome-scalability

As soon as I saw all of the content and resources, I had to share it with the community.

This is why I LOVE the internet. Special thanks to the owner Binh Nguyen

Help me decide: Embed subdocument or have a new document collection?

ImTheDeveloper — Sat, 04 Aug 2018 04:42:11 +0000

I've been building a telegram bot recently which has an element of user tracking to allow for whitelisting users within my chat. I am now moving into an area of building out a reputation system. This will allow users to pass reputation to eachother, maybe via a command or maybe by just replying to them and saying thanks.

I'm using neDB right now which is saves to the local file system but behaves very similar to mongoDB, a document store database. I've modelled User, Commands and other objects so far with simple single documents for each case.

When designing the reputation system I'm hitting a bit of a fork in the road where I need to decide where I utilise a subdocument within my User document or I create a new Reputation collection, which might store a reference to my User.

Rough requirements for the reputation system are:

Each user can give reputation to any other user than themselves
Each user has a maximum of 10 reputation points to give per day
Each user can only add reputation to the same person 3 times a day
Reputation points are re-generated each day (back to 10 points)
Users want to see who has given them the most reputation
Users want to who has the most reputation
Admins can remove reputations points incrementally
Admins can remove all reputation points
Users want to see the number of reputation points they have

There are some other requirements but these are the most relevant to my question. So I've been toying with the following ways of doing this:

Utilise an array in my User collection to store the userID of the person who gave the reputation. I can then check array length to get total reputation and push/pop this array to add and remove reputation. My concern here is that the array could forever grow and is it then efficient to be querying a user to pull back an object with a large array attached?
Similar to the above except I hold an array plus a reputation_count this would mean I can filter out the large array when conducting my query and simply pull the count off the user. If I needed to get the "users want to see who has given the most reputation" I can pull the array info also. Once again, I'm not convinced just yet. This is a trade off between easy query vs large data retrieval.
Generate a new collection for reputation, this could be 1 reputation document per user with similar array scenario as before.
Generate a new collection for reputation, but with each document being 1 reputation point containing userID of the target and userID of the person who gave it.
Same scenario as 3 or 4 but also holding a reputation_count at the User document level. This allows me to grab the count very easy but then query the Reputation documents if I need further details.

All of the above as mentioned are really only options because I'm trying to reduce my number of queries dependent on the situation. I'd rather do one query and get the data I need but at the same time, I don't want to pull large documents if I dont need to. I think there are nuances in the requirements that might require me to carry out multiple queries whatever I do. I haven't even discussed in depth the checks around maximum number of times a user can give another use reputation in a single day too.

Any thoughts on how to approach this? maybe even just at a general level when thinking about these problems?

GDPR and Telegram Bots

ImTheDeveloper — Tue, 22 May 2018 06:34:32 +0000

I've been building telegram bots for some time now and with GDPR right around the corner I've had to turn a few off whilst I address any impact it's going to have since I feel that I don't have enough information to make a decision as to whether they are safe to continue running or not. In this post I'm looking for feedback on approaches and any additional information that can help me to deduce the compliance or changes that need to be made.

A set of bots I'm particularly concerned with are ones that I have built to manage large Telegram supergroups. For anyone familiar with group butler is very similar with features such as:

Removing bad links
Removing profanity
Muting or banning users
Custom commands to display group information.

Most recently I've been asked by a client if I could incorporate analytics into the bot so they can measure things such as:

Messages per day
Events
Sentiment
Number of banned users Etc.

The typical things I would expect a company to want to learn about their community. It's probably worth mentioning this particular group has over 12,000 users.

Whilst I began to build out the analytics module it struck me that this is not something outside of the scope of GDPR just because it's a bot sitting in a group acting on user interaction.

I decided to take a look into the rules and realised that actually it's not just the analytics which could pose a problem here. When an admin bans a user they give a time period to set the ban to expire and invites the user back to the chat if they wish. This requires the storage of a users Telegram ID. For anyone unfamiliar this is the data that a bot can generally see when interacting with a user on telegram:

{"ok":true,"result":{"id":276401636,"first_name":"SMTFirst","username":"SMTFirstBot"}}

The id above is unique to every user. The username is also unique and chosen by the user whenever they wish. It can also be changed at any time just like the first name.

I currently store the id along with the time frame for the ban and reason for the ban. I process this later to invite the users back.

When reading up on GDPR rules the telegram id is a piece of information that is unique to a person and along with other information can be used to discover their identify. Strike 1?

If I then delve into the requirements for analytics, it becomes a complete minefield. Whilst I could record an event, it appears the usage of any unique user ID being sent too can cause a lot of GDPR headaches.

Whilst I could go ahead and simply use timestamps, when it comes to attributing commands used, message counts etc without tieing these to some identifier for a particular user there won't be much hope for understanding user advocacy (even if I'm not bothered on knowing who exactly this is).

I had planned on using something like amplitude.com for analytics processing and whilst they are "GDPR ready" I'm going to be confident in saying that protects their offering and does little to help my side of the coin.

If I assume based on the above, that GDPR is most definitely relevant here, then how do I go about the following scenarios:

Informing users about the usage of their data. This is a chat app and unless a user interacts with the bot, it can not send them direct messages first to inform them.
Handling requests to be forgotten. Right now I could build out a command to do this and process the request quite easily, but once again is that even suitable.
Handling consents. This is one of the hardest parts. I've seen the blanket emails going around "We've updated our policy.. ya cool?😥". Translating this to a bot in a messaging app is probably going to be a lot of work, but right now it kind of feels like that's exactly what I'm going to have to do. Once again though, how the user is told to do this when the bot can not initiate a conversation privately first is going to be a problem.

Whilst the above scenarios for be handled with messages spamming into the chat every so often informing users of their rights and options, it's a actually precisely not what some clients are after. I'm also thinking a do not track option is easy enough to build out but does it's cover enough.

I've spent a good few hours now looking through Google and the best practice documents out there and I'm quite amazed for this has not been discussed before. Infact, right now I can list a number of bot accounts and services specifically used on Telegram which could be in for a lot more issues than I feel I'm up against.

Any thoughts on this one?

Telegram user responses to database/form. Any recommendations?

ImTheDeveloper — Sun, 13 May 2018 06:36:36 +0000

I'm building out a telegram bot that takes answers given to questions from a user and submits them "somewhere" for a client to read through, search, filter etc.

Obviously a database would normally make sense, however I'm really not wanting to build out a bunch of database integration and also then a nice little UI for a user to view the responses received.This is a proof of concept project.

I was thinking of using something like google sheets to act as the "database" and UI. Whilst this would work nicely, once again the integration and setup into Google is a slight pain generating keys or oAuth etc for a simple task.

I fully expect there is a service that will allow a nice and easy POST request to an API which will then also allow a user to view / search / edit the data behind the scenes without much integration work being needed.

This is really about wanting to keep to MVP and not go overboard straight away until we have confirmed that the customer would like to continue on, we can then look to enhancing.

Any recommendations?

Where do you store your apps flat file databases?

ImTheDeveloper — Fri, 23 Mar 2018 06:53:47 +0000

Hi All,

I've built a small node.js application which uses a local JSON file to persist data across restarts (nedb npm package for reference). I currently have this file set in my .gitignore and it is simply held within a Database folder in my project.

I'm using ansible to deploy my program to my production server, which will pull down the latest remote git repo and run the install/build and startup. I'm starting to wonder however where a sensible place would be to store the JSON file that is created upon first start up of the application.

Does storing it WITHIN the project folders make sense? I have noticed a few developers, especially on ubuntu save it to the users home directory and sometimes even as a hidden folder e.g. /home/user/.app/database.json

What is best practice in this case? My only worry is that potentially that project folder could get deleted, for whatever reason maybe to do a complete fresh install, but I would always like to keep the database file intact.

Happy for thoughts and discussions on this as I think people tend to do this in lots of different ways.